Example-based cross-modal denoising

10 years 2 months ago
Example-based cross-modal denoising
Widespread current cameras are part of multisensory systems with an integrated computer (smartphones). Computer vision thus starts evolving to cross-modal sensing, where vision and other sensors cooperate. This exists in humans and animals, reflecting nature, where visual events are often accompanied with sounds. Can vision assist in denoising another modality? As a case study, we demonstrate this principle by using video to denoise audio. Unimodal (audio-only) denoising is very difficult when the noise source is non-stationary, complex (e.g., another speaker or music in the background), strong and not individually accessible in any modality (unseen). Cross-modal association can help: a clear video can direct the audio estimator. We show this using an example-based approach. A training movie having clear audio provides cross-modal examples. In testing, cross-modal input segments having noisy audio rely on the examples for denoising. The video channel drives the search for relevant t...
Dana Segev, Yoav Y. Schechner, Michael Elad
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Where CVPR
Authors Dana Segev, Yoav Y. Schechner, Michael Elad
Comments (0)