acceptodds
Under review as a conference paper at ICLR 2027

SAYAKA: Semantic Attention Yields Audio–Concept Alignment for Fine-Grained Audio–Text Retrieval

Abstract

Fine-grained audio–text retrieval requires models to precisely associate events described in captions with their corresponding acoustic evidence. Existing fine-grained alignment methods follow two paths. One introduces explicit frame-level supervision during training, relying on fixed event taxonomies that do not fully adapt to retrieval scenarios with open-vocabulary captions. The other learns implicit alignment between events and multiple fine-grained prototypes, but requires computing similarity scores with these prototypes at retrieval time, incurring noticeable overhead compared to the single-vector inner product of standard global audio–text retrieval. We propose SAYAKA, a fine-grained audio–text retrieval method that injects concept-level alignment structure only during training and fully reduces to a standard dual encoder at inference. SAYAKA parses captions into semantic concepts via lightweight text parsing; Semantically Conditioned Audio uses each concept as a query to produce conditioned audio tokens from a shared audio-token sequence through semantic attention. Soft-Positive Audio–Concept Alignment treats other concepts within the same caption as weak positives rather than negatives, while a concept separation constraint encourages distinct concepts to capture non-overlapping acoustic evidence. An adaptive audio recomposition module fuses multiple conditioned representations into a single vector aligned with the global audio embedding, enabling fine-grained retrieval using only a single vector at inference. We further construct momentum queues at both the sample and concept levels to continuously strengthen the model's ability to distinguish fine-grained conceptual differences across samples. Case studies and experimental results demonstrate that SAYAKA achieves fine-grained concept–audio alignment and surpasses all current audio–text retrieval methods on AudioCaps and Clotho in terms of R@1, while maintaining the retrieval efficiency of global audio–text retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-text retrieval, multimodal learning, contrastive learning, fine-grained alignment, audio representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rVVvOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SAYAKA, a fine-grained audio-text retrieval framework. Its core innovation lies in being "fine-grained during training, coarse-grained during inference": during the training phase, it uses lightweight text parsing to extract semantic concepts for concept-level alignment, and employs an adaptive recombination module to integrate fine-grained signals into global representations. During inference, however, it reverts entirely to a standard dual-encoder setup, simply calculating the inner product of global vectors. Additionally, to address the issue of highly repeated concepts in datasets, the paper introduces a unidirectional C2A momentum contrast mechanism. Experiments show that this method achieves state-of-the-art performance on AudioCaps and Clotho, while maintaining inference efficiency on par with the baseline CLAP.

Critical strengths

1.The paper accurately addresses the pain point of the trade-off between "training supervision cost" and "inference efficiency" in fine-grained Audio-Text Retrieval (ATR). Through the Adaptive Audio Recomposition module, it ingeniously transforms fine-grained training signals into a global inference representation, requiring no modifications to the inference pipeline. This "fine-grained during training, coarse-grained during inference" design holds immense practical value for both industry and academia. 2.The paper keenly identifies the phenomenon of highly repetitive conceptual phrases in the AudioCaps dataset (Figure 2). Based on this observation, it reasonably restricts the concept-level momentum contrast to a unidirectional Concept-to-Audio (C2A) direction, effectively avoiding the supervision conflicts that would arise from using repetitive texts as negative samples in the Audio-to-Concept (A2C) direction. This analysis is logically rigorous and highly persuasive, demonstrating the authors' deep understanding of the data.

Critical weaknesses

1.The paper relies on spaCy for rule-based phrase extraction. For audio descriptions with complex grammatical structures, highly colloquial phrasing, or implicit semantics, rule-based methods may fail and trigger a fallback mechanism (using the entire text as a single concept). Although Appendix B provides the fallback ratio (approximately 0.14%–0.29%), the main text lacks a specific analysis of how these "parsing failure/degraded" samples impact retrieval performance. This somewhat limits the guarantees regarding the model's robustness when handling complex texts in open domains. 2.Although inference efficiency is on par with CLAP, Table 8 shows that SAYAKA incurs a significant increase in peak memory usage (Peak MiB) and per-step training time (e.g., training time is approximately 1.68× that of CLAP on AudioCaps and 1.18× on Clotho, alongside a substantial increase in memory consumption). For researchers with limited computational resources, this represents an important trade-off. However, the discussion in the main text is somewhat inadequate, as it fails to clearly define the cost-effectiveness boundary of this training overhead.

Questions for authors

1.Regarding the robustness of the fallback mechanism: When the fallback mechanism is triggered (i.e., spaCy fails to extract valid phrases and degrades to using the full caption as a single concept), how does SAYAKA perform on this specific subset? Are there any quantitative evaluations or case studies specifically targeting these samples? 2.Regarding the trade-off in training costs: Table 8 indicates an increase in training memory and time. How do the authors view this increased training overhead? On larger-scale datasets or with longer audio (e.g., >1 minute), will the cross-attention computation between concept tokens and long audio sequences become a memory bottleneck during training? 3.Regarding hyperparameter generalization: On the Clotho dataset, do the optimal hyperparameters, such as betabeta (soft positive sample weight) and lambdaseplambda_{sep}, remain consistent with those used for AudioCaps? If the optimal values differ, does this discrepancy reflect fundamental differences in the captioning styles of the two datasets (e.g., Clotho's five independent descriptions versus AudioCaps's single-sentence crowdsourced descriptions)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rVVv

We are grateful for your encouraging evaluation and for highlighting our train-fine/infer-coarse framework and C2A-only contrast analysis. For fair comparison, all additional evaluations below retain the CED/BERT backbones, global-only retrieval, and three seeds (mean ±\pm std).

W1 & Q1: Fallback captions

When spaCy extracts no phrase, the full caption becomes a single concept. Such captions still train through alignment and recomposition (Appendix B), so fallback is a reduced but complete form of SAYAKA rather than a failure mode. We evaluate all AudioCaps test captions on which the parser falls back (44 captions, 41 clips) against the full candidate pool: A2T uses these clips as queries over all test captions, and T2A uses these captions as queries over all test clips (R@1):

SubsetMethodA2TT2A
FallbackCLAP32.52±7.4515.91±2.27
FallbackSAYAKA43.09±7.0421.97±4.73

Fallback captions are harder for both models, yet SAYAKA's advantage on them (+10.57/+6.06, positive for every seed) is no smaller than on the full test set (+6.76/+4.63). The whole-caption control in our response to Reviewer Arsj points the same way: even when every caption is reduced to a single query, the model does not fall below CLAP + Global MoCo. Parsing failures therefore do not pull SAYAKA below the global baseline.

W2 & Q2: Training cost and long audio

The extra cost is confined to training; retrieval time is identical to CLAP (Table 3). Most of it comes from the momentum encoder and the concept branch, while Separation and Recomposition add only ≈1–2 ms per step (per-module timing in our response to Reviewer o8TV); end-to-end ratios are in Table 8.

Cross-attention does not become the bottleneck for long audio. CED encodes audio in fixed-length chunks, so the token sequence read by the concept queries stays at ≈250 tokens regardless of duration. Forward + backward of full SAYAKA (batch 24, BF16, RTX 4090):

Audio lengthPeak memory (MiB)Time (ms)Cross-attention tokens
10 s2,836179248
30 s3,879383252
60 s5,581671252
70 s6,150765252

Memory and time grow roughly linearly with length, driven by the audio encoder, while the concept branch stays constant; a 70 s batch of 24 fits easily on a 24 GB GPU.

Q3: Hyperparameter generalization

  1. One configuration for all datasets. Clotho uses exactly the AudioCaps hyperparameters (β\beta, λsep\lambda_{\mathrm{sep}}, δ\delta, γ\gamma, λrec\lambda_{\mathrm{rec}}, queue settings); no dataset-specific tuning is performed. The same configuration is also transferred unchanged to held-out TACOS.
  2. Consistent trends. With this single configuration, SAYAKA achieves the best R@1 in both directions on AudioCaps and Clotho (Table 1), and on TACOS it improves over CLAP by +2.50/+3.59 with each concept objective contributing (response to Reviewer o8TV, W4). Clotho's five independent descriptions and TACOS's ≈1 caption per clip differ clearly from AudioCaps' style, so the objectives do not appear to depend on caption style.
  3. Low sensitivity. β\beta, λsep\lambda_{\mathrm{sep}}, δ\delta, γ\gamma and λrec\lambda_{\mathrm{rec}} are weights and shapes of auxiliary constraints on the conditioned tokens, not structural choices. Appendix A shows that across all tested values, A2T R@1 stays within 52.91–55.49 and T2A R@1 within 42.70–43.67, every setting well above CLAP (48.03/38.23), and the T2A values above every baseline in Table 1. T2A varies by less than 1 point, and the best values for R@1 and R@5 often differ, which indicates a flat region rather than a sharp optimum. On a harder dataset such as Clotho, where absolute scores are much lower, such small differences would be dominated by run-to-run noise, so a separate per-dataset sweep would add little information.

We plan to integrate these empirical checks into the revision and trust they address your questions.

Dear Reviewer rVVv,

Thank you again for your encouraging review. Before the discussion period closes, we would like to follow up to see if our response has addressed your questions on fallback captions, training cost and hyperparameter transfer. We would be happy to provide any further details.

Best regards,

Authors of Submission 12418

Reviewer_ArsjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAYAKA for audio–text retrieval. During training, it parses paired captions into concepts and uses these concepts as queries to read audio features through cross-attention. Soft-positive alignment, representation separation, adaptive recomposition, and momentum contrastive learning then improve the global retrieval representations. At inference, the concept branch is removed and retrieval uses only the similarity between global audio and text embeddings. Evaluation on AudioCaps and Clotho includes retrieval comparisons, ablations, concept analyses, and computational costs.

Critical strengths

1. The training design has clear practical value. Concept supervision is obtained directly from existing paired captions without additional event-boundary annotations. The concept branch is used only during training, while inference retains independent encoding and global-vector matching in a standard dual encoder. This provides finer training signals while keeping retrieval simple. The additional cost is primarily incurred during training; the reported matching and ranking times are the same as CLAP’s.

2. The shared-backbone comparisons and ablations support the value of concept training as a whole. The compared methods use the same CED/BERT backbone, and results include means and standard deviations over three runs. SAYAKA has the highest mean R@1 in both retrieval directions on both datasets (Table 1). On AudioCaps, its A2T and T2A gains over the strongest respective competitors are 1.88 and 0.86 percentage points. It also improves these directions by 3.49 and 1.32 points over the control retaining CLAP and global MoCo (Table 4), showing that the gains are not explained solely by adding a global queue.

Critical weaknesses

The incremental contribution over prior methods remains unclear. The paper should more clearly distinguish its approach from existing text-conditioning and phrase-level supervision methods [1–3]. The ablations support the overall value of concept training, but lack a control using the full caption as a single conditioning query. The evidence therefore does not sufficiently establish how much additional benefit concept decomposition, relation constraints, and recomposition provide over simpler full-caption conditioning.

Questions for authors
  1. Why is the proposed multi-concept design needed beyond prior methods [1–3]? Please identify the specific limitation that remains in existing text-conditioned or phrase-level auxiliary training and explain how concept decomposition, within-caption relation constraints, and recomposition address it. For example, is the main benefit better preservation of secondary sound information, less redundancy between concept representations, or more effective transfer of concept information into the global retrieval embedding? Please focus on the main new contribution, explain how it differs from prior approaches, and identify the existing experimental results that directly support this account.

  2. What additional benefit does the multi-concept scheme provide over auxiliary training conditioned on the full caption? Please add a direct control, at least on AudioCaps, that does not split the caption: encode the full caption as a single query, use the same cross-attention module to read audio features, and retain a training connection between the conditioned representation and the global audio embedding. Compare this control with full SAYAKA using the same audio–caption data and backbone and a comparable training budget, while keeping global contrast, global MoCo, and other shared settings consistent. Both models should use only global audio and text embeddings at evaluation. Please follow the repeated-run protocol of the main experiments and report the mean and standard deviation of R@1 in both retrieval directions.

    Since there are no other within-caption concepts in the single-query setting, please explain how the soft-positive target is adjusted, how the concept-separation loss is handled, and whether the recomposition module is retained. This comparison would assess the additional benefit of the multi-concept design and its associated objectives as a whole over simpler full-caption-conditioned auxiliary training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Arsj

We appreciate the reviewer's detailed feedback and positive remarks on our training-only concept design and shared-backbone comparisons. Every new result reported below uses the identical CED/BERT backbones, global embeddings only, and three random seeds (mean ±\pm std).

W1 & Q1: Difference from text-conditioned and phrase-level methods

As the references [1–3] are not listed, we discuss the closest work cited in the paper and would gladly add a direct comparison if the reviewer shares them. Existing methods follow two lines:

  • Phrase/frame-level supervision. FineLAP maps phrases to clusters from a fixed event ontology to build frame-level labels; FLAM obtains frame-level labels from synthesized mixtures and generated captions; MGA-CLAP aligns frames with words through a shared codebook. These rely on an external ontology, synthesized data, or word-level frame matching.
  • Text-conditioned representations or scoring. Text-Prompted CLAP (Li et al., 2026) learns text-conditioned audio representations; text-conditioned attention pooling, THA+DCR and GPA keep query-dependent interactions in the retrieval score.

SAYAKA differs in two respects.

  1. Supervision from the paired captions alone. Concepts are parsed from the existing captions, with no ontology, timestamps or synthesized data. Under the same backbone, it still outperforms these methods (Table 1, AudioCaps R@1):
MethodA2TT2A
FLAM49.74±0.5838.03±0.26
MGA-CLAP50.37±0.2842.15±0.40
THA+DCR51.93±3.9841.75±1.89
SAYAKA55.49±0.1043.01±0.24
  1. Conditioning as a training signal only. Concept conditioning shapes the global embedding through recomposition and is then discarded, so retrieval is a single inner product (Table 3, AudioCaps total time):
MethodRetrieval time (ms)
CLAP / SAYAKA1.264
GPA287.80
THA+DCR177,086.93

For Q1, the main gain is the transfer of secondary-sound information into the global embedding. Two results support this. (i) The whole-caption control below, which uses one query per caption, falls clearly behind. (ii) T2A gains over CLAP are larger for multi-concept captions: +2.55 for single-concept captions vs. +3.85/+7.41/+3.82/+4.57 for K=2/3/4/≥5K=2/3/4/\geq 5, positive for every seed.

W2 & Q2: Whole-caption single-query control

We ran exactly the requested control on AudioCaps. The only change from SAYAKA is the query set: the full caption is encoded as the single query (K=1K=1). Data, backbones, cross-attention, global contrast, Global MoCo, Concept MoCo (with the caption as the concept), all hyperparameters, and the 20-epoch budget are identical. Both models are evaluated with global embeddings only (seeds 41/42/6657, R@1):

MethodA2TT2A
CLAP + Global MoCo (Table 4)52.00±1.3641.69±0.52
Whole-caption single query52.14±1.0642.23±0.40
SAYAKA55.49±0.1043.01±0.24

Whole-caption conditioning adds little over CLAP + Global MoCo, whereas multi-concept decomposition adds +3.34/+0.78 on top of it, with gains for every seed and markedly lower variance. The benefit therefore comes from decomposing captions into concepts, not from conditioned auxiliary training per se.

How each objective behaves at K=1K=1. All objectives are defined over the set of concepts of a clip, so none needs redesign; they reduce naturally, exactly as for single-concept captions in normal training (Appendix B):

  • Soft-Positive Alignment: With no co-occurring concepts, the β\beta-weighted targets are empty, and the objective becomes standard contrastive alignment between each conditioned token and its own caption (α=1\alpha=1, β=0.5\beta=0.5 kept unchanged).
  • Separation: With no within-clip concept pairs, the loss is identically zero.
  • Recomposition: Retained; aggregation over one token is the identity, so the caption-conditioned token is aligned with the global audio embedding. This preserves the training path from the conditioned representation to the global embedding, as requested.

The control thus removes exactly what the multi-concept design adds: multiple queries, graded relations among co-occurring concepts, separation, and recomposition over several tokens.

These control experiments and discussions will be included in the revised manuscript; we hope they clearly establish the role of multi-concept decomposition.

Dear Reviewer Arsj,

Thank you again for your detailed and constructive review. As the discussion period is about to close, we are writing to briefly follow up on our rebuttal, in particular on the control experiment you requested.

Following your suggestion, we ran the whole-caption single-query control on AudioCaps with the same cross-attention, objectives, Global MoCo and training budget, evaluated with global embeddings only over three seeds, and explained how each objective reduces in the single-query setting. We would be glad to know whether this addresses your concern, and we are happy to add a direct comparison if you could share references [1–3].

Should the control and the accompanying discussion resolve your concerns about the incremental contribution, we would sincerely appreciate your reconsideration of the rating. We remain available for any further questions before the discussion closes.

Best regards,

Authors of Submission 12418

Reviewer_o8TVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an approach to improve audio-text alignment by aligning audio via a text-based “concept” regularization. Instead of directly aligning text and captions, they first deterministically break captions down into concepts (via spaCy), and then align the audio to these extracted concepts. They use the embeddings of these concepts as queries into audio tokens via cross-attention. Finally, a variety of additional losses and data sampling approaches are introduced to enhance the results.

The core claim or innovation seems to be 1) breaking text into concepts allows for clever loss design and sampling and 2) conditioning audio tokens on concepts instead of text is helpful for audio representation learning.

All relevant baselines are included for comparison, and the ablations cover most of the suggestions.

The method is overall quite complex with a lot of moving pieces. Some part of the manuscript focusing on the exact importance and intuition of each piece would be helpful. The complexity of the method only results in little gain, without new abilities for the underlying encoder (e.g. strong localization). Concepts allow for clever indexing and data strategies, which seem useful but do not result in large enough gains, I feel.

The most interesting part of the work I think is Figure 3, where the semantic conditioning results in the embedding attending to different parts of the spectrogram. However, it is a bit weak visually, the relationship, but it’s promising for future work.

The model performs well on all the datasets considered relative to many competing methods, which is a positive.

One thing to flag in this work is that the datasets for all ablations and the datasets for top-line reports all overlap. Mainly, the hyper parameter tuning, the ablation study, are all done on Clotho and AudioCaps. It would strengthen the paper to see results on a held-out data (e.g. TACOS or similar) to make sure they hold.

Critical strengths
  • SAYAKA has better retrieval accuracy, with out extra cost at search time.
  • Paper includes and compares favorably against well-chosen and strong baselines.
  • Fair experimental setup, all backbones are the same.
  • Concepts are a nice and lightweight idea for audio-text representation ability.
  • Concept conditioning is neat.
Critical weaknesses
  • Complex method, with a lot of moving pieces, with only a small gain.
  • No new ability for the encoder, such as localization.
  • Fig. 3 evidence is weak and isn't measured.
  • Hyperparameter tuning, ablations and the headline results all use the same datasets (AudioCaps, Clotho).
  • No held-out evaluation, such as in TACOS.
Questions for authors

Do the learned representations exhibit any more temporal ability than the default ones? Can they localize because of the multi-concept regularization? I see there’s some pattern in Figure 3, could these be expanded and measured?

T2A metrics do not move nearly as much as A2T with the given method. Is there any reason for this? TODO: Look in the paper for the reason?

2 comments
AuthorsOpenReview ↗

Response to Reviewer o8TV

Thank you for the thoughtful review and for recognizing our overhead-free accuracy gains, fair baselines, and lightweight concept design. Below, all new experiments use the same CED/BERT backbones, global embeddings only, and three seeds (mean ±\pm std).

W1: Complexity

SAYAKA's only new architectural module is Semantically Conditioned Audio: concepts query shared audio tokens through cross-attention to yield Conditioned Audio Tokens. For example, GAN training can combine adversarial objectives for the generator and discriminator with discriminator feature-matching losses and task-specific constraints, such as reconstruction or perceptual losses. Each of these objectives plays an important role. In SAYAKA, the objectives similarly impose complementary constraints on the shared Conditioned Audio Tokens:

  • Soft-Positive Alignment: Aligns tokens with their own concepts, with lower positive weights for co-occurring concepts in the caption.
  • Separation: Discourages collapse among different concepts' tokens within a clip.
  • Recomposition: Aligns aggregated tokens with the global audio embedding, connecting concept supervision to the retrieval representation.
  • Concept MoCo: Uses a momentum queue to expand negatives for concept-to-audio contrastive learning.

At inference all are removed, leaving a standard dual encoder. Their training cost is small (per-step time added when stacking modules onto CLAP; memory and end-to-end time in Table 8):

Added moduleAudioCapsClotho
Global MoCo+34.6 ms+78.4 ms
Concept branch + alignment+31.4 ms+30.9 ms
Separation≈+0.8 ms≈+1.0 ms
Recomposition≈+2.4 ms≈+1.6 ms
Concept MoCo+11.0 ms+12.1 ms

The design clearly outperforms simpler alternatives (AudioCaps R@1):

MethodA2TT2A
CLAP + Global MoCo52.00±1.3641.69±0.52
Whole-caption single query52.14±1.0642.23±0.40
SAYAKA55.49±0.1043.01±0.24

The single-query model shares the cross-attention, objectives and budget.

W2 & Q1: Localization

Trained only on clip-level captions, SAYAKA learns finer-grained representations than global training.

Single-event retrieval. The W4 models (clip-level training only) retrieve between 2,000 TACOS test clips and 4,884 real event-level strong descriptions (one sound event each, median 2.7 s).

R@1CLAPSAYAKA
A2T6.12±0.437.67±0.30
T2A4.96±0.586.17±0.40

Concept swap. We replace one concept of each AudioCaps test caption with a same-type concept from other clips, yielding 4 distractors that differ only in that concept (the new words never occur in the clip's references). The audio embedding must pick the original among the 5 (chance 20%).

Accuracy (%)CLAPSAYAKA
All70.15±0.4273.98±0.66
Action phrases48.46±1.0254.32±1.69

W3: Quantifying Fig. 3

A concept's attended region is its top-10% audio tokens by attention. We compute IoU between two concepts' regions on the same clip, so only the query differs. Rows 1–2 pair noun phrases from the clip's captions by whether they name the same sound source (same head word); row 3 pairs concepts within one caption; row 4 is chance.

Concept pair (same clip)IoU (%)
Same source ("a car" / "a passing car")54.28±0.91
Different sources ("a car" / "a bird")33.45±0.60
Two concepts of one caption29.47±0.78
Two random token sets≈5.3

Attended regions follow the query semantics, and concepts in the same caption read largely different regions.

W4 & W5: Held-out evaluation

On TACOS, we split the official dev set into training/validation and report the official test set, with clip-level captions only. All hyperparameters are frozen from Clotho (identical to AudioCaps), with no TACOS tuning (R@1):

MethodA2TT2A
CLAP14.05±0.5113.51±1.27
CLAP + Global MoCo15.95±0.1016.28±0.11
w/o Concept MoCo16.37±0.3816.77±0.47
w/o Separation16.38±0.4616.82±0.77
w/o Recomposition16.30±0.3016.64±0.44
SAYAKA16.55±0.1717.10±0.54

SAYAKA is best in both directions, and every concept objective contributes.

Q2: A2T vs. T2A

AudioCaps has ≈5 captions per test clip: an A2T query succeeds if any ranks first, while T2A has one target. Keeping one caption per clip (20 draws) makes both directions symmetric (Δ\Delta = SAYAKA −- CLAP, R@1):

ProtocolΔ\DeltaA2TΔ\DeltaT2A
Standard+6.76+4.63
One caption per clip+4.65+4.79

The gains are then nearly identical, so SAYAKA does not intrinsically favor A2T: the extra A2T gain arises when several references describe overlapping aspects of a scene. On TACOS (≈1 caption per clip), T2A gains more.

We will add these results to the revision and hope they resolve your concerns.

Dear Reviewer o8TV,

We hope you are doing well. As the discussion period is about to close, we wanted to check whether our rebuttal has addressed the concerns raised in your review.

We found your feedback very helpful and have made a dedicated effort to address each point, in particular with a held-out evaluation on TACOS using hyperparameters frozen from Clotho, a quantification of Fig. 3, event-level and concept-swap evaluations, and an analysis of the A2T/T2A difference. If any issue remains, we would be glad to clarify it in the remaining time.

If these additions resolve your concerns about held-out evaluation and the evidence behind Fig. 3, we would be grateful if you could consider updating your score. Thank you again for your thoughtful comments and your time.

Best regards,

Authors of Submission 12418

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and New Results

Dear Reviewers and Area Chair,

We thank the reviewers for their careful reading and constructive feedback. As the discussion period draws to a close, we summarize below the main concerns and the additional evidence provided in our individual responses. Unless otherwise stated, numbers are R@1 (A2T/T2A) computed with global embeddings only, using the same backbones and evaluation protocol as in the paper.

1. What does the multi-concept design add over whole-caption conditioning? (Arsj W2/Q2, o8TV W1)

Following Reviewer Arsj's request, we trained a control that encodes the full caption as a single query. It shares the cross-attention module, data, backbones, objectives, Global MoCo, hyperparameters and training budget with SAYAKA, and keeps the recomposition link to the global audio embedding.

AudioCaps R@1A2TT2A
CLAP + Global MoCo52.00±1.3641.69±0.52
Whole-caption single query52.14±1.0642.23±0.40
SAYAKA55.49±0.1043.01±0.24

Whole-caption conditioning performs on par with CLAP + Global MoCo, whereas the multi-concept design improves on it by +3.34/+0.78, consistently across seeds and with lower variance.

2. Do the conclusions hold on held-out data? (o8TV W4/W5)

TACOS was never used for method design or tuning. With all hyperparameters frozen from Clotho and only clip-level captions for training, SAYAKA reaches 16.55/17.10, compared with 14.05/13.51 for CLAP (+2.50/+3.59) and 15.95/16.28 for CLAP + Global MoCo, and attains the highest mean among all ablated variants.

3. Finer-grained representations and Fig. 3 (o8TV W2/W3/Q1)

Instead of adding a localization head, we examined whether the global embedding, trained only with clip-level captions, becomes more sensitive to fine-grained content:

  • Event-level retrieval. Between 2,000 TACOS test clips and 4,884 strong descriptions, each describing a single sound event (median 2.7 s), SAYAKA improves over CLAP by +1.55/+1.21.
  • Concept swap. When one concept in each AudioCaps caption is replaced (chance 20%), accuracy rises from 70.15 to 73.98, and by +5.86 on action phrases.
  • Quantifying Fig. 3. On the same clip, the IoU between two concepts' top-10% attended tokens is 54.28 when they name the same source, 33.45 when they name different sources, and 29.47 for two concepts of one caption (chance ≈5.3).

4. Why do T2A gains appear smaller? (o8TV Q2)

AudioCaps provides about five references per clip, so an A2T query counts as correct if any of them ranks first. When one caption per clip is kept (20 random draws), the gains over CLAP become +4.65 for A2T and +4.79 for T2A, i.e., nearly identical. On TACOS, with about one caption per clip, T2A likewise gains more.

5. Robustness to parser fallback (rVVv W1/Q1)

On all AudioCaps test captions for which spaCy extracts no phrase (44 captions, 41 clips), retrieved against the full candidate pool, SAYAKA reaches 43.09/21.97 versus 32.52/15.91 for CLAP (+10.57/+6.06, positive for every seed). This advantage is no smaller than on the full test set.

6. Complexity and training cost (o8TV W1, rVVv W2/Q2)

All concept components are used only during training, so inference remains a standard dual encoder with the same retrieval time as CLAP. Separation and Recomposition add only about 1–2 ms per training step, and Concept MoCo about 11–12 ms. Because the concept queries read about 250 audio tokens regardless of duration, a forward and backward pass on a 70 s batch of 24 takes 765 ms and 6.2 GB on an RTX 4090.

7. Hyperparameter transfer (rVVv Q3)

A single configuration is shared by AudioCaps and Clotho and transferred unchanged to TACOS. Across all values in Appendix A, A2T R@1 stays within 52.91–55.49 and T2A R@1 within 42.70–43.67, all well above CLAP (48.03/38.23).

8. Relation to prior work (Arsj W1/Q1)

Unlike phrase- or frame-level supervision (FineLAP, FLAM, MGA-CLAP) and text-conditioned representations or scoring (Text-Prompted CLAP, THA+DCR, GPA), SAYAKA draws supervision solely from the paired captions and uses conditioning only as a training signal. Under the same backbone, it outperforms these methods while retrieving as fast as CLAP. We would be glad to add a direct comparison with references [1–3] once they are available.

All of these results will be incorporated into the revision. We hope this summary helps the reviewers and the Area Chair assess how the concerns have been addressed, and we sincerely thank everyone for their time and effort.

Best regards,

Authors of Submission 12418

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.