Dual-Shutter Optical Vibration Sensing (CVPR 2022 Oral)

The approach allows audio to be individually extracted from several sources in a single video, such as multiple musicians each playing their own guitar, or even several speakers all playing different music.


The extracted audio isn’t as clear or high-fidelity as what a traditional microphone can capture, but the optical microphone could provide mixing engineers with an easy to way monitor individual instruments during a live performance, and over time there’s little doubt the quality of the extracted audio will continue to be improved. The system has other interesting applications outside of music. A video camera monitoring all the machines on a factory floor, or pointed at the engine of a running car, could determine when individual parts or components are making an abnormal sound, indicating maintenance may be required before a problem actually becomes a problem.