acceptodds
Under review as a conference paper at ICLR 2027

SARA: Stable Anchors and Rigorous Alignment for Fine-grained Audio-Text Retrieval

Abstract

Fine-grained Audio-Text Retrieval (ATR) requires precise token-level semantic alignment, yet existing prototype-based methods face two coupled limitations. First, without geometric constraints, learned prototypes collapse into redundant global representations, reducing fine-grained matching to global alignment. Second, mini-batch contrastive training limits the supply of hard negatives for distinguishing subtle semantic differences. We propose Stable Anchors and Rigorous Alignment (SARA) to address both jointly. To suppress prototype collapse and preserve inter-prototype diversity, SARA replaces dynamic pooling with learnable queries, applies orthogonal regularization to the resulting prototypes, and uses optimal transport to balance cross-modal prototype assignment so that every prototype contributes to matching. To further train these stabilized prototypes to distinguish fine-grained semantic differences, SARA mines hard negatives from a momentum-updated dynamic queue, focusing contrastive training on high-similarity negative audio-text pairs. Experiments on AudioCaps and Clotho under the unified encoder configuration show that SARA achieves the highest mean R@1 in both retrieval directions. On AudioCaps, SARA improves mean R@1 over the baseline (GPA) by 6.31 and 3.05 percentage points for audio-to-text and text-to-audio retrieval, respectively. Diagnostic analysis and case studies further show that SARA effectively mitigates prototype collapse and supports fine-grained semantic alignment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-text retrieval, fine-grained alignment, prototype learning, optimal transport, hard negative mining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_b7FxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to target two challenges in fine-grained audio-text retrieval which are prototype collapse and the lack of informative negatives. For these two challenges, this paper propose SARA to address these two issues using stable anchirs and more rigorous alignment. The experiments conducted on AudioCaps and Clotho show the performance improvement on the proposed methods.

Critical strengths

1.big perfomance gain is achieved on AudioCaps SARA ourperforms the strongest baseline by 5.31, which is a relatively big improvement.

2. compreshensive ablation studies and analysis. This paper provides ablation studies on orthogonal regularization, OT, hard negative mining, and prototype count. The analysis shows that reducing prototype collapse alone does not necessarily improve retrieval performance. The appendix also examine how model resonds to the changes in event order and content.

Critical weaknesses

1. The contribution of Stable Anchors is unclear.. Equation (1) introduce the basic linear-projection attention pooling, Equation (5) uses shared learnable queries. However, under the single-head formulation, the difference between them are not clear. The author should clarify this difference, and it would be better to see the comprison of using the original GPA pooling with the same orthognal loss, OT and negative queen. Moreover, as shown in Table 2, GPA+HN still suffers from collapse but achieves the same T2A R@1 as full SARA, and SARA without the queue make little improvement compared with GPA baseline. These make it unclear whether the gains come from better alignment or simply stronger constrasive learning.

2. The fine-grained semantic claims are not supported by sufficiently evidence. APCS and effective rank measure protoype geometry, but do not show whether the prototype capture the different events or relations. It is better to see more systematic evalutions, i.e. the order and attribute tests. Furthermore, the current analysis are mostly conducted on Audiocaps but the improvement on Clotho are very limited.

Questions for authors
  1. what is the main difference between Equation (1) and Equation (5)? Could the author clarify whether additional projections or attention head are used?
  2. Could the author provide more systematic test on the event order and/or attribute discrimination? it would be useful to compare SARA with other baselines under the same experimental setting. It would be interesting to see if reducing prototype improves semantic discrimination beyond the hard negative mining alone.
  3. the evluation settings, such as data splits, retrieval candidate pools and how checkpoints are selcted etc. are not well clarified in the paper. It would better to see the resutls on random seed given the limited improvements on Clotho.
Reviewer_9brQOpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper positions itself as a fine-grained audio-text retrieval (ATR) method and targets "prototype-based" fine-grained matching, in which a clip and a caption are each compressed into K prototypes that are matched locally. It argues that dynamically pooled prototypes collapse into a single direction and that mini-batch contrastive training supplies too few hard negatives. SARA combines (i) learnable cross-attention queries plus an orthogonality loss on prototypes (Sec. 3.1), (ii) Sinkhorn optimal-transport matching with equal marginals, and (iii) a momentum queue with Top-N hard-negative mining and MoCHi-style interpolation (Sec. 3.3). All baselines are re-run on a shared CED-base + BERT backbone with 3 seeds; SARA reports the highest mean R@1 on AudioCaps and Clotho (Tab. 1), geometric diagnostics (APCS, effective rank) showing collapse in GPA but not SARA, component ablations (Tab. 3–5), efficiency measurements, and qualitative prototype-attention case studies on two hand-picked samples (Fig. 3, App. G–J). The only quantitative evaluation is clip-level retrieval (R@1/R@5); no event-level or frame-level task is evaluated.

Critical strengths
  1. Controlled, re-run comparison with variance reporting (Tab. 1). All eight baselines are retrained on the same CED-base + BERT encoders with 3 seeds and std. On AudioCaps the A2T R@1 gain over GPA (+6.31, 55.84±1.15 vs 49.53±0.42) and over the strongest baseline THA-DCR (+5.31) is far outside the reported noise. → This is the main evidence that the method does something real on the retrieval benchmark, and the experimental hygiene (shared backbone, seeds, std, efficiency accounting in Tab. 6) is better than most papers in this area; the concerns below are about what the gain is attributed to, not about whether it exists.

  2. Diagnostic decoupling of geometry and retrieval (Tab. 2). Showing that GPA+HN gains retrieval accuracy while APCS hits 1.00 in both modalities, and that SARA-without-queue has perfect geometry (ρ = 1.00) but GPA-level retrieval, is an informative negative result about the "collapse ⇒ bad retrieval" assumption the paper itself starts from. → It is the one finding that goes beyond component stacking, and the APCS / effective-rank diagnostics are a reusable tool for anyone working with multi-prototype matching.

Critical weaknesses

1. The problem framing is narrower than claimed, and the central claim (fine-grained alignment) is never evaluated.

The abstract and introduction frame the problem around "existing prototype-based methods" in the plural, but the paper's own taxonomy (Tab. 1 and the related-work discussion) lists only one prototype-based prior method, GPA (Xie et al., 2024). The other fine-grained baselines are not prototype-based: FLAM is frame-wise contrastive, THA-DCR is hierarchical alignment with disentangled factors, and ASK injects out-of-batch knowledge. The collapse diagnosis (Tab. 2) is run on GPA alone, so the paper is in effect a GPA follow-up rather than a treatment of a general failure mode.

More importantly, although the title, abstract and conclusion are about fine-grained alignment, every quantitative result is clip-level retrieval R@1/R@5 (Tab. 1, Tab. 3–5). There is no event-level or frame-level evaluation of the kind FLAM reports (e.g., open-vocabulary sound event detection / localization on strongly labeled data), no text-to-audio grounding, and no dataset-level measurement of whether prototypes specialize to distinct events; the fine-grained claim rests on APCS/ρ geometry (Tab. 2) and attention maps for two hand-picked captions (Fig. 3). Even on clip-level retrieval, SARA is not uniformly best within the fine-grained family: it only ties FLAM on Clotho T2A R@5 (42.41). → For a paper whose stated contribution is fine-grained alignment, the absence of any fine-grained metric leaves the headline claim unsupported; what is supported is a modest clip-level retrieval improvement over GPA. → Where: abstract and introduction; Tab. 1 (baseline taxonomy and metrics); Tab. 2 (GPA-only diagnosis); Fig. 3 and the appendix case studies (qualitative only); related-work discussion.

2. The Stable Anchors package (learnable queries + orthogonality + OT) is shown to restore prototype geometry, but it yields no retrieval gain on its own and contributes only in combination with a known technique (a larger negative pool via a momentum queue).

The experiments demonstrate improved anchor diversity, but its contribution to retrieval remains unclear:

  • Geometry versus retrieval: Eq. 6 directly regularizes the quantity measured by APCS, so improved geometry alone does not independently establish a retrieval benefit. SARA without the HN queue performs similarly to GPA (49.60/39.66 vs. 49.53/40.04 R@1; Table 2).
  • Sensitivity to γ: At γ = 1.5–2.0, geometry improves further while retrieval falls below the γ = 1.0 peak (Table 4). This leaves unclear whether the “Quasi-Orthogonal State” explains a robust benefit beyond a particular hyperparameter setting.
  • Limited gain beyond the queue: Full SARA improves over GPA + HN by approximately 3.7 points in A2T, but shows no T2A gain at the reported precision (both 43.09 R@1; Table 2).
  • Missing component ablation: Learnable-query pooling is never independently ablated. The “w/o Ortho” variant retains queries, OT, and HN, so it cannot isolate the query mechanism’s contribution (Table 3).

Overall, the results suggest a possible interaction between the anchor design and the queue, but do not yet establish a robust benefit attributable to Stable Anchors. A controlled pooling ablation and repeated-seed evaluation across γ values and datasets would strengthen the central claim.

Questions for authors
  1. Which prior methods other than GPA do you consider "prototype-based", and has the collapse diagnosis been observed on any model other than GPA?
  2. Is there any quantitative fine-grained evaluation (event-level grounding, frame-level open-vocabulary SED on strongly labeled data as in FLAM, or a dataset-level prototype-specialization metric)? Without one, on what basis is the fine-grained alignment claim made beyond clip-level R@K and two case studies?
  3. Tab. 2 shows that GPA + HN matches full SARA on T2A R@1 (43.09). Is there any setting (Clotho, or R@5/R@10) where Stable Anchors improve T2A beyond GPA + HN? If not, please state explicitly in the main text that Stable Anchors improve geometry but not T2A retrieval on AudioCaps.
  4. Were γ, M, N, S and K selected on the AudioCaps/Clotho validation splits or on the test split? Please state the protocol and report validation R@1 for Tab. 4–5.
  5. Between γ = 0.5 and 1.0, R@1 rises by ~3 points together with a sharp change in geometry (audio APCS 0.95→0.21). At γ = 1.5, however, geometry improves slightly further while A2T R@1 drops back to 52.87 (Tab. 4). What explains this non-monotonicity? Please report per-seed numbers and a finer sweep over γ ∈ [0.5, 1.5].
Reviewer_MJp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that prototype-based methods for fine-grained audio-text retrieval have two problems: prototypes built by dynamic pooling collapse to a common direction, which reduces multi-prototype matching to global matching, and mini-batch contrastive training supplies few high-similarity negatives. SARA counters collapse with shared learnable queries, an orthogonal loss on the prototypes and optimal-transport matching under equal marginals, and it enlarges the negative pool with a momentum queue from which it mines and interpolates hard negatives. With all baselines re-run on the same encoders, SARA improves AudioCaps R@1 over GPA, while its Clotho margins over the strongest baselines fall within the standard deviations. Geometric diagnostics on AudioCaps indicate that collapse is mitigated, but restoring diversity alone does not improve retrieval, and most of the hard-negative gain comes from the queue and interpolation rather than from mining itself.

Critical strengths

The problem is clearly split into two parts, each with a measurable diagnostic. The paper separates a representation problem (prototype collapse) from an optimisation problem (scarce hard negatives). It measures collapse with two metrics: average pairwise cosine similarity (APCS) and normalised effective rank (ρ). The analysis in Eq. 3 shows that complete collapse reduces multi-prototype matching to a single global similarity, which makes the motivation easy to follow.

Critical weaknesses
  • The premise that fine-grained audio-text retrieval is best solved by prototype learning is not justified. The paper takes GPA’s paradigm (Section 2.1) as given. Each modality is compressed into K = 3 prototypes, and retrieval is scored by matching prototypes across modalities. The paper never asks whether this is the right formulation. It implicitly assumes that a matched audio-text pair can be broken into a small, fixed number of shared semantic units, with roughly one-to-one correspondence between the modalities.

  • The gain is not clearly attributed to the prototype component; the evidence points mainly to the negative queue. Several results suggest that the enlarged negative pool, not prototype stabilisation or hard-negative selection, drives most of the improvement

  • The collapse diagnostic is not fully convincing, and the results are sensitive to hyperparameters. The orthogonal loss (Eq. 6) directly minimises pairwise cosines between prototypes, so low APCS (which also averages signed cosines that can cancel) shows that the objective was met, not that the prototypes carry distinct semantic content. Showing that would need functional evidence, such as how well prototype attention aligns with the paper’s LLM event annotations across the test set, or how retrieval changes when individual prototypes are dropped at test time.

Questions for authors

Does “GPA w/ HN” use exactly SARA’s queue, mining and interpolation settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.