- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LAIP, which leverages a large-scale audio-visual retrieval model (PE-AV) for weakly supervised sound source localization. It replaces the [CLS] pooling between the intermediate layers of the frame encoder and the frozen video encoder with audio-query-based hierarchical pooling (AiSP), and uses the resulting attention maps as localization outputs. Training uses a FLAIR-style conditional sigmoid loss with two regularization terms, and the paper reports state-of-the-art performance on AVATAR, AVSBench, and ADE-SP without any spatial annotations.
Critical strengths
- The method shows large performance gains across all three benchmarks, including roughly 2× the CIoU of prior work on AVATAR.
- Reusing the pretrained temporal encoder as a source of training signal is an effective design choice, and this is supported experimentally (26.22 → 19.81 when trained from scratch).
- The experiments are broad, covering layer-wise analysis, application to other backbones, and computational overhead.
Critical weaknesses
- Retrieval compatibility amounts only to coexistence via explicit routing. Retrieval is performed with the original PE-AV, bypassing LAIP, so the two functions do not share representations. This kind of compatibility is a property of any method that attaches a separate head to a frozen backbone, and thus is hard to view as an advantage unique to LAIP.
- The novelty of the core mechanism is limited. Using audio to build similarity/attention maps over spatial features, pooling with them, and training with a global loss is the general paradigm followed by most SSL methods, starting with Senocak et al. (2018). The loss and the null token come from FLAIR. The actual contribution narrows down to inserting this mechanism in the middle of a pretrained stack and the hierarchical pooling, but this distinction is not made clear.
- The design is empirical and its motivations are not validated. Key premises, such as register token monopolization and the preservation of spatial information in intermediate layers, are not directly verified. The latter in particular conflicts with the description in Appendix D. The pooling schedule, head selection rule, and loss weights were determined by search, and the method is sensitive to some of these settings.
- The sources of the performance gains are insufficiently disentangled. The paper’s explanation of the improvement is inconsistent, alternating between attributing it to pretraining and claiming it is not due to pretraining. The attention pooling comparison in the main text uses a mismatched number of heads.
Questions for authors
Q1. Do the conditional representations make a practical contribution to retrieval (e.g., re-ranking)? Q2. What is LAIP’s unique contribution relative to existing audio-guided pooling and FLAIR? Q3. Is there any analysis that directly demonstrates register token monopolization and the spatial information in intermediate layers?
3 comments
Rebuttal 2/2
W3 & Q3 ("register token monopolization and the preservation of spatial information in intermediate layers, are not directly verified, and the latter conflicts with Appendix D"):
To show more clearly those two claims, we conducted new feature-level analyses on frozen PE-AV alongside our cross-dataset robustness checks (already in Table 12):
- Appendix D consistency: The claims are not contradictory: at layer 16, most patches preserve local spatial structure while a few high-norm register tokens already encode global context (Darcet et al., 2024). Therefore, 1-stage attention is hurt by those outliers, whereas 3-stage AiSP suppresses them.
- Direct diagnostics on frozen PE-AV:
(1) Across layers, spatial locality decreases while video-encoder[CLS]alignment increases, crossing at Layer 16. Ideally the right layer to incorporate AiSP layers should have high spatial information while remaining compatible with the video encoder (so staying close to PE's original[CLS]). Therefore L16 seems to be the sweet spot (also shown by AVATAR performance in Table 11).
(2) On on-screen AVATAR frames, we extracted frozen PE-AV Layer-16 tokens, identified background register tokens in each frame as patches outside the ground-truth sounding mask whose L2 norm lies in the top 5% of that frame, and measured both the share of spatial attention mass of those outlier tokens (BG Register Attn Mass) and the Signal-to-Register Ratio (attention mass placed inside the true sounding mask divided by attention mass placed on background registers). Comparing 1-stage pooling against LAIP shows that 3-stage AiSP cuts attention driven by background register tokens by 41% and more than doubles the Signal-to-Register ratio:
| PE-AV Layer | Layer 8 | Layer 14 | Layer 16 (Ours) | Layer 18 | Layer 24 (Final) |
|---|---|---|---|---|---|
Patch-to-[CLS] Homogenization () ↓ | -0.314 | -0.193 | -0.175 | -0.105 | +0.019 |
L2 Distance to Video-Encoder [CLS] () ↓ | 54.56 | 31.60 | 28.24 | 24.91 | 0.00 |
| Method (at PE-AV Layer 16) | BG Register Attn Mass (%) ↓ | Signal-to-Register Ratio (GT / Reg) ↑ | AVATAR Avg CIoU (%) ↑ (Tables 1 & 6) |
|---|---|---|---|
| FLAIR-style 1-stage AttnPool (8 heads) | 2.41% | 20.39× | 20.89 |
| LAIP (Ours, 3-stage AiSP) | 1.43% | 46.31× | 26.22 |
These new feature-level diagnostics directly confirm both intermediate-layer spatial preservation and that our multi-stage method reduces impact of register-tokens. We will include these analyses and clarify Appendix D in the paper.
W4 ("The sources of the performance gain are insufficiently disentangled... attention pooling comparison in the main text uses a mismatched number of heads"):
To clearly disentangle the gain from pretraining vs. LAIP and address the head-count mismatch, our controlled ablations already reported in the paper (Tables 1, 4, 6, 9, and 10) explicitly isolate the effect of pretrained features, the AiSP module, the temporal encoder, and the number of attention heads:
- Pretraining vs. architecture: Pretrained PE-AV L16 features alone do not improve localization (
EZ-VSL / FNAC / TAVLO: 12.10 / 11.14 / 6.93% CIoU in Tables 1 & 9vs. prior SOTA13.21%). Adding AiSP alone (with a scratch temporal encoder) jumps to 19.81% (+6.60, Table 4) (and 16.68% on ImageBind, Table 10), and reusing the frozen pretrained temporal encoder adds another +6.41% to reach 26.22% (Table 4). - Matched-head comparison (already in Appendix Table 6, moving to Table 4):
| Pooling Architecture (Paper, Table 6) | 1 Head (CIoU / AUC %) | 8 Heads (CIoU / AUC %) |
|---|---|---|
| 1-stage Attention Pooling (FLAIR-style) | 14.34 / 14.74 | 20.89 / 21.12 |
| 3-stage Hierarchical AiSP (LAIP) | 21.00 / 21.28 (+6.66) | 26.22 / 26.34 (+5.33) |
Pretrained features alone fail without LAIP (12.10% CIoU), whereas LAIP brings consistent gains both from its 3-stage AiSP architecture (+5.33 to +6.66% CIoU at matched head counts) and from reusing the frozen temporal encoder (+6.41% CIoU); we will clarify this disentanglement and move the matched-head comparison into Table 4 of the paper.
We hope that the new retrieval experiments and controlled ablations help clarify LAIP's contributions and address your main concerns. Could you please indicate whether the new evidence resolves the issues in your assessment?
Response to rebuttal
I got confused for the previous comment, please ignore it.
Thank you for your detailed rebuttal. My questions regarding W2, the main weakness, have been answered. I appreciate authors for re-ranking test, but it does not resolve the mainly problem. I have raised my assessment from 2 to 3, but I am leaning slightly to acceptance.
Rebuttal 1/2
We thank Reviewer F5ii for the thorough, constructive review and detailed questions.
W1 & Q1 ("Do the conditional representations make a practical contribution to retrieval (e.g., re-ranking)?"):
We acknowledge this important point raised. To directly address whether LAIP contributes to retrieval beyond simple routing coexistence, we evaluate LAIP's conditioned representations in a practical two-stage re-ranking pipeline.
To directly test whether LAIP's conditional representations improve retrieval in practice, we ran a two-stage top-K re-ranking experiment: frozen PE-AV first retrieves Top-K candidates in O(N), and LAIP re-ranks only those K candidates in O(NK) via (1−α)SPE-AV(i,j)+α⟨O(i,j),Ag(j)⟩ (α=0.20) on VGGSound test clips (N=200):
LAIP's conditional representations make a clear practical contribution to retrieval—boosting PE-AV by +2.75% Mean R@1 (+3.00% V2A, +2.50% A2V at Top-10), with gains scaling up as K increases from 5 to 10. We will include these re-ranking results in the paper.
W2 & Q2 ("The novelty of the core mechanism is limited... What is LAIP’s unique contribution relative to existing audio-guided pooling and FLAIR?"):
To clearly delineate LAIP's unique contribution from Senocak et al. (2018) and FLAIR, we isolate how standard audio-guided pooling performs on PE-AV and quantify each component we introduce using the ablations already in the paper (Tables 4 & 6, Fig. 7).
Directly applying 1-stage audio-guided pooling (Senocak / FLAIR) to PE-AV reaches only 14.34–20.89% CIoU (already in Tables 4 & 6). LAIP reaches 26.22% CIoU via three distinct contributions we will foreground in Sec. 1:
In short, standard 1-stage audio-guided pooling fails on ViT retrieval backbones (14.34% CIoU), and LAIP's hierarchical pooling + temporal distillation is specifically responsible for the +11.88% CIoU jump. We will clarify upfront in the paper.