- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SAYAKA, a fine-grained audio-text retrieval framework. Its core innovation lies in being "fine-grained during training, coarse-grained during inference": during the training phase, it uses lightweight text parsing to extract semantic concepts for concept-level alignment, and employs an adaptive recombination module to integrate fine-grained signals into global representations. During inference, however, it reverts entirely to a standard dual-encoder setup, simply calculating the inner product of global vectors. Additionally, to address the issue of highly repeated concepts in datasets, the paper introduces a unidirectional C2A momentum contrast mechanism. Experiments show that this method achieves state-of-the-art performance on AudioCaps and Clotho, while maintaining inference efficiency on par with the baseline CLAP.
Critical strengths
1.The paper accurately addresses the pain point of the trade-off between "training supervision cost" and "inference efficiency" in fine-grained Audio-Text Retrieval (ATR). Through the Adaptive Audio Recomposition module, it ingeniously transforms fine-grained training signals into a global inference representation, requiring no modifications to the inference pipeline. This "fine-grained during training, coarse-grained during inference" design holds immense practical value for both industry and academia. 2.The paper keenly identifies the phenomenon of highly repetitive conceptual phrases in the AudioCaps dataset (Figure 2). Based on this observation, it reasonably restricts the concept-level momentum contrast to a unidirectional Concept-to-Audio (C2A) direction, effectively avoiding the supervision conflicts that would arise from using repetitive texts as negative samples in the Audio-to-Concept (A2C) direction. This analysis is logically rigorous and highly persuasive, demonstrating the authors' deep understanding of the data.
Critical weaknesses
1.The paper relies on spaCy for rule-based phrase extraction. For audio descriptions with complex grammatical structures, highly colloquial phrasing, or implicit semantics, rule-based methods may fail and trigger a fallback mechanism (using the entire text as a single concept). Although Appendix B provides the fallback ratio (approximately 0.14%–0.29%), the main text lacks a specific analysis of how these "parsing failure/degraded" samples impact retrieval performance. This somewhat limits the guarantees regarding the model's robustness when handling complex texts in open domains. 2.Although inference efficiency is on par with CLAP, Table 8 shows that SAYAKA incurs a significant increase in peak memory usage (Peak MiB) and per-step training time (e.g., training time is approximately 1.68× that of CLAP on AudioCaps and 1.18× on Clotho, alongside a substantial increase in memory consumption). For researchers with limited computational resources, this represents an important trade-off. However, the discussion in the main text is somewhat inadequate, as it fails to clearly define the cost-effectiveness boundary of this training overhead.
Questions for authors
1.Regarding the robustness of the fallback mechanism: When the fallback mechanism is triggered (i.e., spaCy fails to extract valid phrases and degrades to using the full caption as a single concept), how does SAYAKA perform on this specific subset? Are there any quantitative evaluations or case studies specifically targeting these samples? 2.Regarding the trade-off in training costs: Table 8 indicates an increase in training memory and time. How do the authors view this increased training overhead? On larger-scale datasets or with longer audio (e.g., >1 minute), will the cross-attention computation between concept tokens and long audio sequences become a memory bottleneck during training? 3.Regarding hyperparameter generalization: On the Clotho dataset, do the optimal hyperparameters, such as (soft positive sample weight) and , remain consistent with those used for AudioCaps? If the optimal values differ, does this discrepancy reflect fundamental differences in the captioning styles of the two datasets (e.g., Clotho's five independent descriptions versus AudioCaps's single-sentence crowdsourced descriptions)?
2 comments
Dear Reviewer rVVv,
Thank you again for your encouraging review. Before the discussion period closes, we would like to follow up to see if our response has addressed your questions on fallback captions, training cost and hyperparameter transfer. We would be happy to provide any further details.
Best regards,
Authors of Submission 12418
Response to Reviewer rVVv
We are grateful for your encouraging evaluation and for highlighting our train-fine/infer-coarse framework and C2A-only contrast analysis. For fair comparison, all additional evaluations below retain the CED/BERT backbones, global-only retrieval, and three seeds (mean ± std).
W1 & Q1: Fallback captions
When spaCy extracts no phrase, the full caption becomes a single concept. Such captions still train through alignment and recomposition (Appendix B), so fallback is a reduced but complete form of SAYAKA rather than a failure mode. We evaluate all AudioCaps test captions on which the parser falls back (44 captions, 41 clips) against the full candidate pool: A2T uses these clips as queries over all test captions, and T2A uses these captions as queries over all test clips (R@1):
Fallback captions are harder for both models, yet SAYAKA's advantage on them (+10.57/+6.06, positive for every seed) is no smaller than on the full test set (+6.76/+4.63). The whole-caption control in our response to Reviewer Arsj points the same way: even when every caption is reduced to a single query, the model does not fall below CLAP + Global MoCo. Parsing failures therefore do not pull SAYAKA below the global baseline.
W2 & Q2: Training cost and long audio
The extra cost is confined to training; retrieval time is identical to CLAP (Table 3). Most of it comes from the momentum encoder and the concept branch, while Separation and Recomposition add only ≈1–2 ms per step (per-module timing in our response to Reviewer o8TV); end-to-end ratios are in Table 8.
Cross-attention does not become the bottleneck for long audio. CED encodes audio in fixed-length chunks, so the token sequence read by the concept queries stays at ≈250 tokens regardless of duration. Forward + backward of full SAYAKA (batch 24, BF16, RTX 4090):
Memory and time grow roughly linearly with length, driven by the audio encoder, while the concept branch stays constant; a 70 s batch of 24 fits easily on a 24 GB GPU.
Q3: Hyperparameter generalization
We plan to integrate these empirical checks into the revision and trust they address your questions.