- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses variable semantic correspondence between naturally co-occurring audio and visual signals. It proposes Caption Guided Alignment Aware Loss (CGAAL), which uses an LLM to estimate agreement between modality-specific captions and reweights contrastive training accordingly. The method also incorporates warm-up and gradual thresholding. Experiments demonstrate retrieval improvements across three contrastive baselines, under natural and additional synthetic misalignment, with zero-shot evaluation on three external datasets.
Critical strengths
- S1: The paper targets a meaningful limitation of audio-visual learning: co-occurring modalities can share little semantic content. Using modality-specific captions to estimate correspondence offers a practical way to incorporate external semantic information without requiring captions or an LLM at inference.
- S2: The empirical evidence is convincing. Comparisons use the same pretrained encoders, and CGAAL improves aggregate retrieval performance across CLIP, SLIP, and CMC under both training conditions. Zero-shot improvements across AVE, DiDeMo, and MSR-VTT further support transfer beyond the training dataset. The alignment-stratified analysis also helps explain where the gains arise.
Critical weaknesses
- W1: The evaluation does not sufficiently isolate the benefit of LLM-based caption scoring over alternative correspondence estimates. Section 2 discusses embedding-based weighting approaches, but the experiments primarily compare against methods without comparable alignment weighting. A controlled comparison using the same loss with simpler caption-similarity or pretrained audio-visual similarity scores would better establish the contribution of the proposed scoring procedure.
- W2: The evidence for gradual thresholding and warm-up is mixed. In Table 3, thresholding slightly reduces RSum without synthetic misalignment, from 220.8 to 220.3, while warm-up slightly reduces it with synthetic misalignment, from 194.4 to 194.0. These results do not undermine alignment weighting, but they provide limited support for consistently preferring the full curriculum. Without variability across runs, the small differences are difficult to interpret.
Questions for authors
- Q1 (W1): How does CGAAL compare with the same weighted objective using caption embedding similarity (like sentence embedding similarity) or pretrained audio-visual similarity as the weights? This would clarify whether the LLM-based scoring provides an advantage over simpler correspondence estimates.
- Q2 (W2): Can the authors report variability across runs for the weighting-only and curriculum variants in Table 3? This would help determine whether the additional components provide reliable benefits.
2 comments
Thank you for the additional experiments. The BERT comparison addresses my main question.
The new ablation shows gains from warm-up in both settings. Gradual thresholding helps only with synthetic misalignment. Without results across multiple runs with matched random seeds, it's still hard to judge how reliable the smaller gains are.
I'm keeping my original score (weak accept).
We thank the reviewer for their time, effort and constructive feedback. We are glad to hear that the reviewer appreciated the limitation of audio-visual learning that the paper targets and found the empirical results convincing. We address each of the remaining questions and concerns below.
Since submission, we have better tuned the hyperparameters of our approach, improving its performance. We thus report all new comparisons using this updated configuration.
Altneratives to LLaMa Alignment Scoring
To demonstrate the importance of our LLM based scoring method, we compute text embeddings of the vision and audio captions using BERT and compute similarity scores between them. We test our method using the BERT driven scores and show the results in the table below. The drop in RSum when using BERT driven scores is significant compared to using LLaMa scores with the synthetic misalignment domain. Futher, As suggested, we compare our method with a filtering-based approach, excluding training samples whose alignment score falls below a certain threshold. The table below reports results for thresholds of 0.7 and 0.4. Filtering at either threshold reduces retrieval performance relative to the unfiltered CLIP baseline, both with and without synthetic misalignment. In contrast, our method achieves considerable gains in RSum, demonstrating the benefit of modelling the degree of misalignment rather than discarding misaligned samples. The results are shown in the table below.
Alternatives to LLaMA-Based Scoring. We compare CGAAL using LLaMA-driven alignment scores against (i) filtering out weakly aligned samples whose alignment score falls below a threshold (0.7, 0.4), and (ii) using BERT-driven alignment scores.
Without synthetic misalignment
With synthetic misalignment
Evidence for gradual thresholding and warm-up
Since submission we have better optimized our method's hyperparameters which give better overall performance. The updated ablation provides clearer evidence for the benefit of warm-up, which increases RSum from 219.7 to 224.1 without synthetic misalignment and from 193.5 to 195.2 with it. Gradual thresholding is beneficial with greater misalignment, hence it improves RSum under synthetic misalignment, from 189.2 to 193.5, but it slightly reduces RSum without synthetic misalignment, from 220.8 to 219.7.
Model Ablation. Alignment weighting contributes most to our performance. Gradual thresholding gives a further gain with synthetic misalignment, while warm-up improves without it.
Without synthetic misalignment
With synthetic misalignment