Sound Source Localization is the multimodal task of identifying the spatial location in a visual scene that corresponds to an observed sound — using audio-visual correlation to generate heatmaps or bounding boxes over video frames that pinpoint where a sound is originating from, such as localizing a speaking person, a playing instrument, or a barking dog by jointly analyzing audio spectral features and visual motion patterns.

What Is Sound Source Localization?

• Definition: Given a video with audio, determine which spatial region(s) in each video frame are producing the observed sound, outputting a localization map that highlights sound-producing areas.
• Audio-Visual Correlation: The model learns that visual regions whose appearance or motion correlates with the audio signal are likely sound sources — lip movements correlate with speech, string vibrations correlate with guitar sounds.
• Attention-Based Localization: Most methods compute cross-modal attention between audio features and spatial visual features, producing an attention map where high-attention regions indicate likely sound sources.
• Class-Agnostic: Unlike object detection, sound source localization doesn't require predefined object categories — it localizes any sound-producing region based on audio-visual correspondence.

Why Sound Source Localization Matters

• Robotics: Robots need to localize sound sources to orient toward speakers, identify alarm sounds, and navigate toward or away from audio events in their environment.
• Surveillance: Security systems can automatically focus cameras on sound-producing regions (breaking glass, gunshots, voices) for targeted monitoring.
• Video Editing: Automatic identification of sound sources enables intelligent audio-visual editing, such as isolating a speaker's audio track based on their visual location.
• Augmented Reality: AR systems need to spatially anchor virtual audio to real-world visual objects, requiring accurate sound source localization for immersive experiences.

Sound Source Localization Methods

• Attention and Activate (2018): Computes similarity between audio features and spatial visual features to produce a localization heatmap, trained with audio-visual correspondence as self-supervision.
• Learning to Localize Sound (LVS): Uses contrastive learning between audio and visual region features, with hard negative mining to improve localization precision.
• Mix-and-Localize: Trains on artificially mixed audio from multiple sources, learning to localize each source by separating the mixed audio conditioned on visual features.
• EZ-VSL (Easy Visual Sound Localization): Simplifies training with momentum-based pseudo-labels and achieves state-of-the-art localization without complex multi-stage training.

Sound source localization is the spatial grounding task of audio-visual AI — pinpointing where sounds originate in visual scenes through learned cross-modal correlations between audio spectral features and visual spatial features, enabling applications from robotics and surveillance to augmented reality that require machines to understand the spatial relationship between what they see and what they hear.