acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Spatial Grounding in Large Audio-Visual Retrieval Models

Abstract

Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information. To exploit this, we introduce LAIP (Localization via Audio-Informed Pooling), a framework that employs a lightweight Audio-informed Spatial Pooling (AiSP) to replace the standard global aggregation module. By querying intermediate visual tokens with audio aligned at the frame level, LAIP recovers localized spatial information that is otherwise discarded by the retrieval pipeline, with the largest gains observed for PE-AV, a stack with underlying temporal aggregation. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter, improving average CIoU from 13.21 to 26.22.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio visual learning, sound source localization, audio visual alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F5iiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAIP, which leverages a large-scale audio-visual retrieval model (PE-AV) for weakly supervised sound source localization. It replaces the [CLS] pooling between the intermediate layers of the frame encoder and the frozen video encoder with audio-query-based hierarchical pooling (AiSP), and uses the resulting attention maps as localization outputs. Training uses a FLAIR-style conditional sigmoid loss with two regularization terms, and the paper reports state-of-the-art performance on AVATAR, AVSBench, and ADE-SP without any spatial annotations.

Critical strengths
  1. The method shows large performance gains across all three benchmarks, including roughly 2× the CIoU of prior work on AVATAR.
  2. Reusing the pretrained temporal encoder as a source of training signal is an effective design choice, and this is supported experimentally (26.22 → 19.81 when trained from scratch).
  3. The experiments are broad, covering layer-wise analysis, application to other backbones, and computational overhead.
Critical weaknesses
  1. Retrieval compatibility amounts only to coexistence via explicit routing. Retrieval is performed with the original PE-AV, bypassing LAIP, so the two functions do not share representations. This kind of compatibility is a property of any method that attaches a separate head to a frozen backbone, and thus is hard to view as an advantage unique to LAIP.
  2. The novelty of the core mechanism is limited. Using audio to build similarity/attention maps over spatial features, pooling with them, and training with a global loss is the general paradigm followed by most SSL methods, starting with Senocak et al. (2018). The loss and the null token come from FLAIR. The actual contribution narrows down to inserting this mechanism in the middle of a pretrained stack and the hierarchical pooling, but this distinction is not made clear.
  3. The design is empirical and its motivations are not validated. Key premises, such as register token monopolization and the preservation of spatial information in intermediate layers, are not directly verified. The latter in particular conflicts with the description in Appendix D. The pooling schedule, head selection rule, and loss weights were determined by search, and the method is sensitive to some of these settings.
  4. The sources of the performance gains are insufficiently disentangled. The paper’s explanation of the improvement is inconsistent, alternating between attributing it to pretraining and claiming it is not due to pretraining. The attention pooling comparison in the main text uses a mismatched number of heads.
Questions for authors

Q1. Do the conditional representations make a practical contribution to retrieval (e.g., re-ranking)? Q2. What is LAIP’s unique contribution relative to existing audio-guided pooling and FLAIR? Q3. Is there any analysis that directly demonstrates register token monopolization and the spatial information in intermediate layers?

3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank Reviewer F5ii for the thorough, constructive review and detailed questions.

W1 & Q1 ("Do the conditional representations make a practical contribution to retrieval (e.g., re-ranking)?"):
We acknowledge this important point raised. To directly address whether LAIP contributes to retrieval beyond simple routing coexistence, we evaluate LAIP's conditioned representations in a practical two-stage re-ranking pipeline.
To directly test whether LAIP's conditional representations improve retrieval in practice, we ran a two-stage top-KK re-ranking experiment: frozen PE-AV first retrieves Top-KK candidates in O(N)O(N), and LAIP re-ranks only those KK candidates in O(NK)O(NK) via (1−α)SPE-AV(i,j)+α⟨O(i,j),Ag(j)⟩(1-\alpha)S_{\text{PE-AV}}(i,j) + \alpha \langle O^{(i,j)}, A_g^{(j)} \rangle (α=0.20\alpha=0.20) on VGGSound test clips (N=200N=200):

MethodTop-KKV2A R@1 (%) ↑A2V R@1 (%) ↑A2V R@5 (%) ↑A2V R@10 (%) ↑Mean R@1 (%) ↑
Frozen PE-AV (1st-stage O(N)O(N))—70.0075.5092.5094.0072.75
PE-AV + LAIP Re-ranking (Ours)572.50 (+2.50)76.50 (+1.00)90.5093.5074.50 (+1.75)
PE-AV + LAIP Re-ranking (Ours)1073.00 (+3.00)78.00 (+2.50)93.00 (+0.50)96.00 (+2.00)75.50 (+2.75)

LAIP's conditional representations make a clear practical contribution to retrieval—boosting PE-AV by +2.75% Mean R@1 (+3.00% V2A, +2.50% A2V at Top-10), with gains scaling up as KK increases from 5 to 10. We will include these re-ranking results in the paper.

W2 & Q2 ("The novelty of the core mechanism is limited... What is LAIP’s unique contribution relative to existing audio-guided pooling and FLAIR?"):
To clearly delineate LAIP's unique contribution from Senocak et al. (2018) and FLAIR, we isolate how standard audio-guided pooling performs on PE-AV and quantify each component we introduce using the ablations already in the paper (Tables 4 & 6, Fig. 7).
Directly applying 1-stage audio-guided pooling (Senocak / FLAIR) to PE-AV reaches only 14.34–20.89% CIoU (already in Tables 4 & 6). LAIP reaches 26.22% CIoU via three distinct contributions we will foreground in Sec. 1:

  1. Mid-stack intervention before a frozen temporal encoder (implicit distillation: +10.30 CIoU over final layer L24; +6.41 CIoU over a scratch temporal encoder; Table 4).
  2. 3-stage local convolutional pooling (AiSP) + LMRes\mathcal{L}_{\text{MRes}} (+11.88 CIoU over 1-head FLAIR; +5.33 CIoU over 8-head FLAIR; +8.70 CIoU over λ=0\lambda=0; Tables 4 & 6).
  3. Anti-collapse null-token regularization L∅\mathcal{L}_\emptyset (+3.21 CIoU over μ=0\mu=0, Table 4 & Fig. 7).

In short, standard 1-stage audio-guided pooling fails on ViT retrieval backbones (14.34% CIoU), and LAIP's hierarchical pooling + temporal distillation is specifically responsible for the +11.88% CIoU jump. We will clarify upfront in the paper.

AuthorsOpenReview ↗

Rebuttal 2/2

W3 & Q3 ("register token monopolization and the preservation of spatial information in intermediate layers, are not directly verified, and the latter conflicts with Appendix D"):
To show more clearly those two claims, we conducted new feature-level analyses on frozen PE-AV alongside our cross-dataset robustness checks (already in Table 12):

  • Appendix D consistency: The claims are not contradictory: at layer 16, most patches preserve local spatial structure while a few high-norm register tokens already encode global context (Darcet et al., 2024). Therefore, 1-stage attention is hurt by those outliers, whereas 3-stage AiSP suppresses them.
  • Direct diagnostics on frozen PE-AV:
    (1) Across layers, spatial locality decreases while video-encoder [CLS] alignment increases, crossing at Layer 16. Ideally the right layer to incorporate AiSP layers should have high spatial information while remaining compatible with the video encoder (so staying close to PE's original [CLS]). Therefore L16 seems to be the sweet spot (also shown by AVATAR performance in Table 11).
    (2) On on-screen AVATAR frames, we extracted frozen PE-AV Layer-16 tokens, identified background register tokens in each frame as patches outside the ground-truth sounding mask whose L2 norm lies in the top 5% of that frame, and measured both the share of spatial attention mass of those outlier tokens (BG Register Attn Mass) and the Signal-to-Register Ratio (attention mass placed inside the true sounding mask divided by attention mass placed on background registers). Comparing 1-stage pooling against LAIP shows that 3-stage AiSP cuts attention driven by background register tokens by 41% and more than doubles the Signal-to-Register ratio:
PE-AV LayerLayer 8Layer 14Layer 16 (Ours)Layer 18Layer 24 (Final)
Patch-to-[CLS] Homogenization (cos⁡(Vtℓ,ctℓ)\cos(V_t^\ell, c_t^\ell)) ↓-0.314-0.193-0.175-0.105+0.019
L2 Distance to Video-Encoder [CLS] (∥ctℓ−ct24∥2\lVert c_t^\ell - c_t^{24} \rVert_2) ↓54.5631.6028.2424.910.00
Method (at PE-AV Layer 16)BG Register Attn Mass (%) ↓Signal-to-Register Ratio (GT / Reg) ↑AVATAR Avg CIoU (%) ↑ (Tables 1 & 6)
FLAIR-style 1-stage AttnPool (8 heads)2.41%20.39×20.89
LAIP (Ours, 3-stage AiSP)1.43%46.31×26.22

These new feature-level diagnostics directly confirm both intermediate-layer spatial preservation and that our multi-stage method reduces impact of register-tokens. We will include these analyses and clarify Appendix D in the paper.

W4 ("The sources of the performance gain are insufficiently disentangled... attention pooling comparison in the main text uses a mismatched number of heads"):
To clearly disentangle the gain from pretraining vs. LAIP and address the head-count mismatch, our controlled ablations already reported in the paper (Tables 1, 4, 6, 9, and 10) explicitly isolate the effect of pretrained features, the AiSP module, the temporal encoder, and the number of attention heads:

  1. Pretraining vs. architecture: Pretrained PE-AV L16 features alone do not improve localization (EZ-VSL / FNAC / TAVLO: 12.10 / 11.14 / 6.93% CIoU in Tables 1 & 9vs. prior SOTA 13.21%). Adding AiSP alone (with a scratch temporal encoder) jumps to 19.81% (+6.60, Table 4) (and 16.68% on ImageBind, Table 10), and reusing the frozen pretrained temporal encoder adds another +6.41% to reach 26.22% (Table 4).
  2. Matched-head comparison (already in Appendix Table 6, moving to Table 4):
Pooling Architecture (Paper, Table 6)1 Head (CIoU / AUC %)8 Heads (CIoU / AUC %)
1-stage Attention Pooling (FLAIR-style)14.34 / 14.7420.89 / 21.12
3-stage Hierarchical AiSP (LAIP)21.00 / 21.28 (+6.66)26.22 / 26.34 (+5.33)

Pretrained features alone fail without LAIP (12.10% CIoU), whereas LAIP brings consistent gains both from its 3-stage AiSP architecture (+5.33 to +6.66% CIoU at matched head counts) and from reusing the frozen temporal encoder (+6.41% CIoU); we will clarify this disentanglement and move the matched-head comparison into Table 4 of the paper.

We hope that the new retrieval experiments and controlled ablations help clarify LAIP's contributions and address your main concerns. Could you please indicate whether the new evidence resolves the issues in your assessment?

Reviewer_F5iiOpenReview ↗

Response to rebuttal

I got confused for the previous comment, please ignore it.

Thank you for your detailed rebuttal. My questions regarding W2, the main weakness, have been answered. I appreciate authors for re-ranking test, but it does not resolve the mainly problem. I have raised my assessment from 2 to 3, but I am leaning slightly to acceptance.

Reviewer_swUKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Audio-informed Spatial Pooling modules to adapt pretrained PE-AV audio-visual retrieval model to weakly supervised sound-source localization. The paper also adds multi-resolution attention consistency and null token to prevent trivial or uninformative attention behavior.

Critical strengths

The paper is well written and easy to follow. The paper also addresses an important question of fine grained spatial grounding in large multimodal systems. Proposed methods also obtains good performance improvements over the baseline. The idea of using null token for pairs where audio/visual modality do not have mutual information is interesting.

Critical weaknesses

The proposed method is explicitly inspired by FLAIR, which introduces modality conditioned pooling of visual tokens using a globally trained multimodal model. LAIP extends this idea from text-image to audio-video localization, adding new improvements. What would be the performance of a flair style simpler baseline?

As mentioned in the paper: "audio-conditioned pooling prevents precomputation of visual representations." how does this affect the computational efficiency and inference/training time for the proposed method compared to other baselines?

Questions for authors

"We train for 10 epochs with a learning rate of 1e−4 and a batch size of 10." Why is the batch size so small? Usually contrastive objective benefit for larger batch sizes.

1 comment

We thank Reviewer swUK for the constructive review and questions.

W1 — Comparison with a simpler FLAIR-style baseline

To directly answer how a simpler FLAIR-style baseline performs in our exact setup, we point to the single-stage cross-attention ablations already reported in the paper.

This baseline is evaluated in Table 4 (Attention pooling) and Appendix Table 6. It replaces our 3-stage convolutional AiSP, LMResL_{\mathrm{MRes}}, and L∅L_{\varnothing} with single-stage audio-conditioned cross-attention under the same LSIGL_{\mathrm{SIG}} loss.

MethodAVATAR Avg CIoU (%)AVATAR Avg AUC (%)
FLAIR-style 1-stage AttnPool (1 head / 8 heads)14.34 / 20.8914.74 / 21.12
LAIP (Ours, 3-stage AiSP)26.2226.34

Full LAIP outperforms the 1-head and 8-head FLAIR-style baselines by 11.88 and 5.33 percentage points in CIoU, respectively. We thank the reviewer for highlighting the lack of clarity and will rename Attention pooling to FLAIR-style baseline in Table 4.

W2 — Computational efficiency and training/inference costs

To clarify how the O(B2)O(B^2) and O(N2)O(N^2) conditioning costs affect training and inference, we distinguish what can be precomputed in LAIP.

  • Training (quadratic pairwise conditioning): Unlike unconditioned methods (e.g., standard SigLIP or TAVLO), which run the temporal stack BB times per batch, LAIP processes all B2=100B^2 = 100 video-audio pairs for B=10B = 10 through AiSP and the 4-layer temporal encoder. Crucially, the heavy 24-layer PE-AV frame encoder and 28-layer audio encoder remain frozen, and their features are precomputed once on disk. Each training step takes approximately 4.7 seconds (about 1 hour/epoch, or 10 hours for 10 epochs on 10k VGGSound clips using a single GPU), compared with approximately 1–2 hours for non-conditioned MIL heads trained on the same cached tokens. This is substantially faster than end-to-end backbone training on 144k raw videos.

  • Inference (linear localization cost): The quadratic retrieval limitation discussed in Sec. 5 does not apply to localization. Each video is evaluated only with its matched audio pair (i,i)(i,i), requiring one pass per clip and bypassing the 4-layer temporal encoder entirely. Localization inference therefore scales as O(N)O(N), adding only 4.2% FLOPs and 4.0% peak memory over frozen PE-AV (Appendix F, Table 13).

Thus, precomputing the frozen visual and audio backbones keeps quadratic training practical on a single GPU (approximately 10 hours for 10 epochs), while localization inference remains linear with negligible overhead (+4.2% FLOPs). We will clarify this distinction in Sec. 5 and Appendix F.

Q1 — Why use a small batch size?

The choice of B=10B = 10 follows from the quadratic computational cost of conditioning and the properties of our pairwise sigmoid objective.

In standard dual-encoder contrastive learning, video and audio encoders run independently, and cross-modal interactions are computed through inexpensive dot products. In LAIP, audio conditions visual tokens before the 4-layer temporal video transformer, producing a distinct 16-frame token sequence c~t(i,j)\tilde{c}_t^{(i,j)} for every video-audio pair (i,j)(i,j).

Consequently, B=10B = 10 already requires forwarding B2=100B^2 = 100 conditioned video sequences (1,600 frames: 10 positive and 90 negative pairs) through AiSP and the temporal transformer at every step, saturating single-GPU memory.

Furthermore, unlike softmax InfoNCE, which generally benefits from larger batches, our pairwise sigmoid loss (LSIG\mathcal{L}_{\mathrm{SIG}}; Zhai et al., 2023) optimizes each of the 100 pairs as an independent binary logistic term, making it suitable for smaller batch sizes.

Thus, B=10B = 10 provides 100 conditioned temporal pairs (including 90 negatives) per update, saturating single-GPU VRAM while enabling stable training under our pairwise sigmoid objective. We will add this clarification in Sec. 4.

We hope these clarifications address your questions regarding the FLAIR baseline and computational efficiency. Do these complementary results adequately address your concerns, or is there anything else we could clarify?

Reviewer_S13aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes way to extend large-scale audio–visual retrieval model(PE-AV) for sound source localization. A temporally aligned, audio-conditioned visual token is computed from the image encoder’s intermediate-layer features and fed into the video encoder with small adapters. They train the model in weakly supervised way. They achieve sota performance on two benchmarks.

Critical strengths

Propose way to reuse audio-visual retrieval models for sound source localization, and shows strong performance, reaching sota with only a small amount of additional training.

Critical weaknesses

Major weakness

  1. About the contribution1 : The claim that intermediate visual tokens preserve spatial information has already been established in Perception Encoder: The Best Visual Embeddings Are Not at the Output of the Network (NeurIPS 2025). Plus, this observation concerns the properties of the visual encoder itself, rather than any characteristics specific to audio-visual models. While the authors provide layer-wise ablations in table 11, these experiments does not directly confirm the claim.
  2. About the contribution3 : The claim that global audio–video supervision is sufficient for sound source localization is the fundamental premise of weakly supervised sound source localization, dating back at least to Learning to Localize Sound Source in Visual Scenes (CVPR 2018). It's not a novel contribution. The authors should clarify what new insight is established beyond this existing learning paradigm.
  3. The motivation of hierarchical pooling: In line 193, the authors argue that the hierarchical pooling architecture is a way to mitigate the influence of register tokens. Although the ablation studies demonstrate that hierarchical pooling outperforms simpler attention pooling, they do not show that this improvement results from mitigating register-token interference. An analysis of attention distributions or register-token behavior would be necessary to support this motivation.
  4. The central idea of using a modality-conditioned query to pool spatial visual tokens under global contrastive supervision closely resembles FLAIR: VLM with Fine-grained Language-informed Image Representations (CVPR 2025).

Minor weakness

  1. The writing is not well organized: e.g., “PE-AV” (line 24) and B (line 262) are used without being defined.
Questions for authors
  1. VGG-Sound Source (VGG-SS) appears to be the standard benchmark in this area. Can you report VGG-SS results?
  2. The null token is motivated as a way for the audio query to attend away from the image when no visible region is relevant, i.e., the off-screen case. On AVATAR off-screen clips with their own audio, how much attention does the null token receive, and does high null attention correlate with correct rejection?
4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank Reviewer S13a for the constructive feedback and insightful questions. We appreciate the opportunity to clarify the contributions of LAIP and provide additional experimental evidence addressing the concerns raised.

W1 ("already been established in Perception Encoder... layer-wise ablations in Table 11 do not directly confirm their claim"):
To directly address both the difference from PE and the lack of direct verification behind Table 11, we clarify the Bolya et al. findings and ours, then measure layer-wise properties directly on frozen PE-AV tokens to explain more our results. Bolya et al. (2025) evaluated frozen intermediate ViT features using dense pixel/box labels in single-modality vision; they did not study hierarchical audio-visual retrieval stacks (where a frame ViT feeds a frozen temporal video encoder) or whether intermediate visual tokens—which have zero cross-modal alignment with audio (zero-shot similarity: 4.14% CIoU, already in Table 4)—can be grounded from weak clip-level audio-video pairs alone (validated across PE-AV, ImageBind, and LanguageBind, already in Tables 1 & 10).
Furthermore, we directly verify why Layer 16 is the optimal trade-off for sound source localization without training LAIP, by measuring on frozen PE-AV tokens: (i) Patch-to-[CLS] Distance per layer and (ii) L2 Distance to the Video-Encoder input [CLS] (∥ctℓ−ct24∥2\lVert c_t^\ell - c_t^{24} \rVert_2, compatibility with the frozen temporal encoder):

PE-AV LayerLayer 8Layer 14Layer 16 (Ours)Layer 18Layer 24 (Final)
Patch-to-[CLS] distance ↓-0.314-0.193-0.175-0.105+0.019
L2 Distance to Video-Encoder input [CLS] ↓54.5631.6028.2424.910.00
LAIP AVATAR Avg CIoU (%) ↑ (Table 11)18.6423.0126.2222.5615.92

As depth increases, spatial locality progressively decreases while alignment with the frozen video encoder's [CLS] space increases—making Layer 16 the clear sweet spot for sound source localization, which we will clarify in Contribution 1.

W2 ("global audio-video supervision is sufficient... since at least [CVPR 2018]... clarify what new insight is established"):
To clearly establish our new insight over Senocak et al. (2018) and prior SSL methods, we compare our design directly against both paradigms using the baselines already reported in the paper.
Prior weakly supervised SSL methods require dense output feature maps for region-level MIL losses (EZ-VSL, FNAC, SLAVC, TAVLO), which collapse on ViT tokens (6.93–12.10% CIoU, Tables 1 & 9). Conversely, single-stage global attention pooling—which corresponds directly to the pooling mechanism of Senocak et al. (2018) (who additionally relied on external semi-supervised labels)—is already evaluated in our Attention pooling baseline (Table 4) and reaches only 14.34% CIoU.
Our core architectural change over Senocak-style global pooling—hierarchical 3-stage local pooling injected before a frozen pretrained temporal video encoder (acting as implicit distillation)—is what unlocks 26.22% CIoU (+11.88%) from clip-level supervision alone; we will clarify this in the paper.

W3 ("do not demonstrate that this improvement comes from mitigating register-token interference"):
To directly verify that hierarchical pooling mitigates register interference, we measured attention distributions on high-norm outlier tokens.
On AVATAR, we identified high-norm background outlier ("register") tokens at PE-AV L16 (patches outside the GT sounding mask in the top-5% L2 norm) and measured its attention mass:

MethodBG Register Attn Mass (%) ↓Signal-to-Register Ratio (GT / Reg) ↑AVATAR Avg CIoU (%) ↑ (Tables 1 & 6)
FLAIR-style 1-stage AttnPool (8 heads)2.41%20.39×20.89
LAIP (Ours, 3-stage AiSP)1.43%46.31×26.22

Our 3-stage local pooling significantly reduces the impact of outlier register tokens, cutting background register attention by 41% and more than doubling the Signal-to-Register ratio, directly confirming our claim that our method reduces registers impact; we acknowledge the importance of this analysis, and will add it to the paper.

AuthorsOpenReview ↗

Rebuttal 2/2

W4 — Similarity to FLAIR

To clearly distinguish LAIP from FLAIR, we quantify the exact gains over a FLAIR adaptation on PE-AV (already evaluated in Tables 4 and 6 of the paper), and underline the main differences between them that create this gap.

We credit FLAIR for the use of the global loss, and the null token (Secs. 1–3). However, FLAIR uses single-stage attention pooling at the final layer of a static image encoder. Applied to PE-AV, this yields only 14.34% CIoU (1 head) or 20.89% (8 heads), versus 26.22% for LAIP (Tables 4 and 6). LAIP differs through:

  1. Mid-layer pooling from the frame encoder, fed into the frozen temporal encoder (+10.30 CIoU over L24).
  2. Three-stage local AiSP with LMRes\mathcal{L}_{\mathrm{MRes}} (+11.88 CIoU over single-stage pooling).
  3. Anti-collapse null regularization L∅\mathcal{L}_{\emptyset} (+3.21 CIoU).

Thus, while FLAIR fails to produce accurate segmentation maps on PE-AV, LAIP's adapted hierarchical design achieves +11.88% CIoU over a direct FLAIR adaptation, which we will highlight in the paper.

Q1 — VGG-Sound Source (VGG-SS) evaluation

We thank the reviewer for this suggestion. We additionally evaluated LAIP on the official VGG-SS test set, comprising 4,525 clips:

MethodVGG-SS cIoU (%)VGG-SS AUC (%)AVATAR Avg CIoU (%)
EZ-VSL (10k) / SLAVC (144k)28.86 / 29.0035.12 / 35.408.23 / 7.26
EZ-VSL on PE-AV (10k)18.9429.9412.10
FLAIR-style AttnPool (8 heads)32.3537.2520.89
LAIP (Ours, 10k)37.4439.1426.22

AVATAR results are from Tables 1, 6, and 9 of the submitted paper.

Under this coarse bounding-box evaluation on VGG-SS, LAIP outperforms prior SSL methods and PE-AV baselines by +5.09 to +18.50 percentage points in cIoU. We will add these results to the paper.

Q2 — Null-token attention and off-screen rejection

To directly answer both parts of this question (complementing Fig. 7 in Appendix C), we measured off-screen null attention and its correlation with rejection on AVATAR.

Across AVATAR test videos, the null token in full LAIP (μ=0.01\mu = 0.01) receives 50% more final-stage attention on off-screen frames than on on-screen frames (1.20% vs. 0.80%, averaged across heads).

An off-screen frame is correctly rejected when its peak heatmap value remains below the binarization threshold (max⁡M<τ\max M \lt \tau). We therefore computed the Spearman rank correlation between null-token attention (α∅\alpha_{\emptyset}) and peak heatmap suppression (−max⁡M-\max M), obtaining:

  • Off-screen frames: ρ=+0.446\rho = +0.446, p<10−100p \lt 10^{-100}.
  • All frames: ρ=+0.503\rho = +0.503.

These results provide quantitative evidence that higher null-token attention is associated with lower visual activation and correct off-screen rejection. We will include this analysis in the paper.

We hope that the additional experiments and clarifications address your concerns regarding novelty and empirical validation. Could you let us know whether these results sufficiently resolve your concerns, or if any points would benefit from further clarification?

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns, and I acknowledge the strong empirical results and improvements over the existing baselines.

However, my primary concern regarding novelty remains, particularly with respect to W1 and W2. Much of the novelty emphasized in the rebuttal appears to rely on the pretrained temporal video encoder, which is specific to the PE-AV architecture. As a result, while the proposed method demonstrates an effective adaptation of PE-AV for sound source localization, I am not yet fully convinced that it establishes the broader insights claimed in the paper.

I appreciate the authors' efforts and the additional clarifications. Nevertheless, I respectfully maintain my original rating.

We thank the Reviewer for the careful follow-up and for acknowledging both our efforts and the strength of the empirical results. We are also grateful for the concerns raised during this process, which have directly improved our manuscript, by factually addressing all his points: the new layer-wise analysis (Table 14), the register/interference analysis (Table 15), the VGG-SS evaluation (Table 2), and the null-token/off-screen analysis (Appendix D) are all now in the manuscript.

With the empirical and technical concerns now addressed, the remaining points therefore only concern the novelty of our contributions. We therefore provide below further objective, quantitative evidence to substantiate the methodological contributions and demonstrate the insights gained through our approach.

We respectfully offer three clarifications:

  1. Prior paradigms (PE + Senocak/FLAIR) fail when directly combined. If our insights were already established by Perception Encoder (PE, 2025; W1) and Senocak et al. (2018; W2), then applying Senocak/FLAIR-style 1-stage global pooling to PE-AV's intermediate Layer-16 tokens would already succeed. Instead, it reaches only 14.34–20.89% CIoU (vs. 26.22% for LAIP) even when assisted by PE-AV's frozen temporal encoder; and when trained with a from-scratch temporal encoder, it collapses completely to 2.29% Avg CIoU, whereas LAIP with the exact same from-scratch temporal encoder reaches 19.81% Avg CIoU (+17.52%). This proves our multi-stage local pooling is a genuine finding, not a restatement of prior work.

  2. The insight is not tied to PE-AV's temporal encoder. LAIP transfers to ImageBind and LanguageBind, which have no pretrained temporal stack, using a randomly initialized video encoder. It outperforms TAVLO on all three localization scenarios there. Intermediate layers also clearly outperform final layers on ImageBind (18.29 vs. 5.83 CIoU on single-source). The pretrained temporal encoder adds a further gain on PE-AV (19.81 -> 26.22), which we report as an additional benefit and, importantly, not as the source of the core insight.

  3. The relation to Bolya et al. is complementary. They show that intermediate features carry dense spatial information within a vision encoder. We show that this information can be grounded to audio without dense labels, and that a trade-off between spatiality and semanticity (as well as downstream compatibility when a temporal stack is present) determines the optimal layer. Audio-visual retrieval models do not provide this by default, since the zero-shot audio similarity of these tokens is only 4.14% CIoU.

We have revised the contribution statements to scope these claims precisely, so that readers see what is and is not new.

We sincerely thank the reviewer again for the constructive engagement. Given that the empirical and technical concerns have been addressed with additional experiments and new insights, and that the remaining disagreement concerns how broadly the contribution should be framed. We would therefore sincerely appreciate any specific suggestions on how the framing of W1 and W2 could be further sharpened. We also respectfully ask the reviewer to reconsider their rating in light of these additional results and clarifications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for their time and constructive feedback.

We wish to use this message to summarize our responses to the main concerns and questions raised across the reviews. Below, we group the central concerns shared by the reviewers (S13a, swUK, and F5ii) and outline the existing ablations and new quantitative experiments provided to address each of them:

  • Direct verification of intermediate-layer spatiality and register-token interference (S13a-W1/W3, F5ii-W3/Q3): A shared concern of S13a and F5ii was that our claims regarding spatial preservation in intermediate layers and register-token mitigation were only supported indirectly. We directly addressed both via two new feature-level experiments: (i) measuring patch-to-[CLS] homogenization vs. distance to the video-encoder [CLS] across frozen PE-AV layers showing more in depth why the 16th layer is optimal for audio visual alignment, and (ii) quantifying attention mass on high-norm background register tokens, showing quantitatively that our methods significantly reduces their impact.
  • Conceptual distinction from FLAIR / Senocak et al. and disentangling pretraining (S13a-W2/W4, swUK-W1, F5ii-W2/Q2/W4): A common question across all three reviewers concerned how LAIP differs from 1-stage audio-guided pooling / FLAIR and how much of the gain stems from pretraining vs. our architecture. We re-explained this quantitatively using controlled ablations already in the submitted paper (Tables 1, 4, 6, 9, and 10), comparing matched-head 1-stage FLAIR baselines, pretrained PE-AV features with prior SSL heads, AiSP with a scratch temporal encoder, and full LAIP.
  • Practical utility for retrieval (F5ii-W1/Q1): Addressing F5ii's concern that retrieval compatibility was only routing coexistence, we conducted a new two-stage Top-KK retrieval re-ranking experiment on VGGSound, and show that our method also allows boost in retrieval performance over the original PE-AV.
  • VGG-SS evaluation and off-screen null-token analysis (S13a-Q1/Q2): Finally, to answer S13a's specific benchmark and off-screen questions, we ran a new evaluation on the official VGG-SS test set (showing that here as well our method outperforms concurrent works) and computed a Spearman correlation test, showing that the null token is significantly more attended is off-screen scenarios.

We have uploaded a revised version of the paper, in which all changes are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.