acceptodds
Under review as a conference paper at ICLR 2027

Not All Pairs Align: Caption-Guided Audio-Visual Contrastive Learning

Abstract

We address the problem of learning audio-visual representations from co-occurring modalities. Multimodal contrastive learning typically assumes that paired observations provide shared views of the same underlying information.While this may be true for datasets collected in constrained environments, it does not always hold. For instance, co-occuring audio and visual signals can differ substantially in their content due to off-screen sounds. Nevertheless, standard contrastive objectives treat all paired samples as equally aligned. In this work, we propose Caption Guided Alignment Aware Loss (CGAAL), a model-agnostic objective that leverages modality-specific captions to estimate semantic agreement between audio and video. CGAAL uses these agreement scores to weight the contribution of each positive pair. A warm up period and gradual thresholding strategy progressively introduces this alignment-aware training signal. We demonstrate the generality of CGAAL by applying it to three contrastive baselines, consistently improving their performance on audio-visual retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal alignment, audio visual retrieval, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jEUxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses variable semantic correspondence between naturally co-occurring audio and visual signals. It proposes Caption Guided Alignment Aware Loss (CGAAL), which uses an LLM to estimate agreement between modality-specific captions and reweights contrastive training accordingly. The method also incorporates warm-up and gradual thresholding. Experiments demonstrate retrieval improvements across three contrastive baselines, under natural and additional synthetic misalignment, with zero-shot evaluation on three external datasets.

Critical strengths
  • S1: The paper targets a meaningful limitation of audio-visual learning: co-occurring modalities can share little semantic content. Using modality-specific captions to estimate correspondence offers a practical way to incorporate external semantic information without requiring captions or an LLM at inference.
  • S2: The empirical evidence is convincing. Comparisons use the same pretrained encoders, and CGAAL improves aggregate retrieval performance across CLIP, SLIP, and CMC under both training conditions. Zero-shot improvements across AVE, DiDeMo, and MSR-VTT further support transfer beyond the training dataset. The alignment-stratified analysis also helps explain where the gains arise.
Critical weaknesses
  • W1: The evaluation does not sufficiently isolate the benefit of LLM-based caption scoring over alternative correspondence estimates. Section 2 discusses embedding-based weighting approaches, but the experiments primarily compare against methods without comparable alignment weighting. A controlled comparison using the same loss with simpler caption-similarity or pretrained audio-visual similarity scores would better establish the contribution of the proposed scoring procedure.
  • W2: The evidence for gradual thresholding and warm-up is mixed. In Table 3, thresholding slightly reduces RSum without synthetic misalignment, from 220.8 to 220.3, while warm-up slightly reduces it with synthetic misalignment, from 194.4 to 194.0. These results do not undermine alignment weighting, but they provide limited support for consistently preferring the full curriculum. Without variability across runs, the small differences are difficult to interpret.
Questions for authors
  • Q1 (W1): How does CGAAL compare with the same weighted objective using caption embedding similarity (like sentence embedding similarity) or pretrained audio-visual similarity as the weights? This would clarify whether the LLM-based scoring provides an advantage over simpler correspondence estimates.
  • Q2 (W2): Can the authors report variability across runs for the weighting-only and curriculum variants in Table 3? This would help determine whether the additional components provide reliable benefits.
2 comments

We thank the reviewer for their time, effort and constructive feedback. We are glad to hear that the reviewer appreciated the limitation of audio-visual learning that the paper targets and found the empirical results convincing. We address each of the remaining questions and concerns below.

Since submission, we have better tuned the hyperparameters of our approach, improving its performance. We thus report all new comparisons using this updated configuration.

Altneratives to LLaMa Alignment Scoring

To demonstrate the importance of our LLM based scoring method, we compute text embeddings of the vision and audio captions using BERT and compute similarity scores between them. We test our method using the BERT driven scores and show the results in the table below. The drop in RSum when using BERT driven scores is significant compared to using LLaMa scores with the synthetic misalignment domain. Futher, As suggested, we compare our method with a filtering-based approach, excluding training samples whose alignment score falls below a certain threshold. The table below reports results for thresholds of 0.7 and 0.4. Filtering at either threshold reduces retrieval performance relative to the unfiltered CLIP baseline, both with and without synthetic misalignment. In contrast, our method achieves considerable gains in RSum, demonstrating the benefit of modelling the degree of misalignment rather than discarding misaligned samples. The results are shown in the table below.

Alternatives to LLaMA-Based Scoring. We compare CGAAL using LLaMA-driven alignment scores against (i) filtering out weakly aligned samples whose alignment score falls below a threshold (0.7, 0.4), and (ii) using BERT-driven alignment scores.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP15.938.350.51017.238.850.810211.5
CLIP + (0.7 threshold)3.912.818.8904.312.117.48869.2
CLIP + (0.4 threshold)11.629.239.42011.929.438.920160.4
CLIP (+Ours), BERT scores17.142.252.7917.941.752.49224.0
CLIP (+Ours), LLaMA scores18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP13.033.544.41413.333.043.516180.7
CLIP + (0.7 threshold)3.711.216.61013.510.116.010061.1
CLIP + (0.4 threshold)11.227.936.72311.027.037.424151.2
CLIP (+Ours), BERT scores13.834.445.71413.833.845.014186.5
CLIP (+Ours), LLaMA scores14.335.947.61214.836.046.613195.2

Evidence for gradual thresholding and warm-up

Since submission we have better optimized our method's hyperparameters which give better overall performance. The updated ablation provides clearer evidence for the benefit of warm-up, which increases RSum from 219.7 to 224.1 without synthetic misalignment and from 193.5 to 195.2 with it. Gradual thresholding is beneficial with greater misalignment, hence it improves RSum under synthetic misalignment, from 189.2 to 193.5, but it slightly reduces RSum without synthetic misalignment, from 220.8 to 219.7.

Model Ablation. Alignment weighting contributes most to our performance. Gradual thresholding gives a further gain with synthetic misalignment, while warm-up improves without it.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP15.938.350.51017.238.850.810211.5
+ alignment weighting18.040.952.0917.640.352.09220.8
+ gradual thresholding18.140.652.3917.239.951.69219.7
+ warm up18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP13.033.544.41413.333.043.516180.7
+ alignment weighting14.835.046.61313.734.344.814189.2
+ gradual thresholding14.435.747.31215.034.946.213193.5
+ warm up14.335.947.61214.836.046.613195.2

Thank you for the additional experiments. The BERT comparison addresses my main question.

The new ablation shows gains from warm-up in both settings. Gradual thresholding helps only with synthetic misalignment. Without results across multiple runs with matched random seeds, it's still hard to judge how reliable the smaller gains are.

I'm keeping my original score (weak accept).

Reviewer_WLQdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that co-occurring audio and video are often only partially aligned in content. It proposes CGAAL, which uses modality-specific captions (provided by VAST-27M) and an LLM (LLaMA-7B) with in-context examples to predict an alignment score in [0, 1] for each audio-visual pair, averaged over the 25 audio-vision caption combinations. These scores weight the per-sample terms of the InfoNCE loss, and they are introduced after a short warm-up through a decreasing threshold. The method is applied to CLIP, SLIP, and CMC-style objectives on a 100K subset of VAST-27M, with and without 15% synthetically shuffled audio. The paper reports consistent gains in audio-visual retrieval on the VAST test subset and in zero-shot retrieval on AVE, DiDeMo, and MSR-VTT.

Critical strengths
  1. The problem is real and well motivated, and the proposed fix is practical. The paper offers a model-agnostic way to inject a prior on pair quality without any cost at inference time. The score distribution in Fig. 5 is a useful observation in itself: even the unmodified VAST subset contains many pairs with low estimated agreement.

  2. The gains are consistent across settings. Adding the weighting improves RSum for all three contrastive objectives, with and without synthetic misalignment, and improves zero-shot RSum on all three unseen datasets. The improvements are largest for CLIP and in the misaligned setting, which matches the stated intuition.

Critical weaknesses
  1. The paper does not show that LLM-based caption scoring is needed, and the comparisons that would establish this are missing.

The core idea, reweighting positive pairs by an estimate of their correspondence, is close to existing soft-weighted contrastive objectives and to the noisy-correspondence literature. The synthetic setting, in which 15% of the audio is shuffled, is exactly the scenario those methods target, yet none of them is compared.

The related work dismisses embedding-based weights on the grounds that models trained under the redundancy assumption "will have already lost much of the nuance" of each pair. This argument is not tested. Several simple alternatives would be needed to support the design choice:

  • weights from a pretrained audio-visual or audio-text model,
  • cosine similarity between text embeddings of the two captions, with no LLM involved,
  • hard filtering of pairs below a score threshold.

Without these, it is unclear whether the gains come from the LLM judgment specifically or from any reasonable down-weighting of noisy pairs. This matters for practicality as well, since the method requires captions for every training sample and 25 LLM calls per sample (about 2.5M calls for the 100K subset).

  1. The experimental protocol leaves several of the claims insufficiently supported.

(a) Size of the effects. The gains for CMC are small (+1.7 RSum with synthetic misalignment, with V2A R@10 slightly lower than the baseline). In Table 3, the three configurations without synthetic misalignment differ by less than 2 RSum (220.8, 220.3 and 222.1), and with misalignment the full method is slightly below the variant without warm-up (194.0 vs. 194.4). The evidence for the thresholding and warm-up components is therefore weak.

(b) Comparison with CoMM and InfMasking. These methods learn fused representations for classification, and the paper does not explain how unimodal audio and video embeddings were extracted from them for retrieval. InfMasking reaches an RSum of 34.4 against 211.5 for CLIP with the same encoders, which suggests a setup mismatch rather than a meaningful comparison.

(c) Scale and setting. Training uses 100K clips, a batch size of 32, and a single visual frame per clip, and zero-shot recall is very low (e.g., MSR-VTT A2V R@1 of 3.9 to 4.7). It is not clear whether the gains persist at the scale at which audio-visual contrastive models are usually trained, or with video encoders that can capture motion-related correspondence such as walking or riding.

Questions for authors

Clarifications:

  1. During gradual thresholding, what weight do samples with scores below the current threshold receive: 1, 0, or something else? What is the initial threshold?

  2. On which data were warm-up length, Δe, and ΔT tuned for each model? Was there a validation split separate from the 1,927-sample test set, and were the baselines tuned with a comparable budget?

  3. How were audio and video embeddings obtained from CoMM and InfMasking for retrieval? Were their training recipes adapted to this setting?

  4. In Fig. 6, is retrieval performed within each alignment bin or against the full test gallery, with results grouped by the score of the query?

  5. For the synthetically shuffled pairs, are the alignment scores recomputed from the captions of the new audio-video pairing?

Suggestions that would affect my assessment:

  1. Comparisons with cheaper or established sources of pair weights: pretrained CLAP or ImageBind similarity, text-embedding similarity between the two captions, hard filtering by score, and at least one noisy-correspondence method (e.g., NCR) in the synthetic setting.

  2. Results at a larger training scale, or with a video encoder that uses multiple frames, to show that the gains are not specific to the current small-scale, single-frame setting.

2 comments

We thank the reviewer for their time, effort and constructive feedback. We are glad to hear that the reviewer appreciated the problem motivation, practical solution and consistent gains. We address each of the remaining questions and concerns below.

Since submission, we have better tuned the hyperparameters of our approach, improving its performance. We thus report all new comparisons using this updated configuration.

LLM Based Scoring Requirement

To demonstrate the importance of our LLM based scoring method, we compute text embeddings of the vision and audio captions using BERT and compute similarity scores between them. We test our method using the BERT driven scores and show the results in the table below. The drop in RSum when using BERT driven scores is significant compared to using LLaMa scores with the synthetic misalignment domain. Futher, As suggested, we compare our method with a filtering-based approach, excluding training samples whose alignment score falls below a certain threshold. The table below reports results for thresholds of 0.7 and 0.4. Filtering at either threshold reduces retrieval performance relative to the unfiltered CLIP baseline, both with and without synthetic misalignment. In contrast, our method achieves considerable gains in RSum, demonstrating the benefit of modelling the degree of misalignment rather than discarding misaligned samples. The results are shown in the table below.

Alternatives to LLaMA-Based Scoring. We compare CGAAL using LLaMA-driven alignment scores against (i) filtering out weakly aligned samples whose alignment score falls below a threshold (0.7, 0.4), and (ii) using BERT-driven alignment scores.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP15.938.350.51017.238.850.810211.5
CLIP + (0.7 threshold)3.912.818.8904.312.117.48869.2
CLIP + (0.4 threshold)11.629.239.42011.929.438.920160.4
CLIP (+Ours), BERT scores17.142.252.7917.941.752.49224.0
CLIP (+Ours), LLaMA scores18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP13.033.544.41413.333.043.516180.7
CLIP + (0.7 threshold)3.711.216.61013.510.116.010061.1
CLIP + (0.4 threshold)11.227.936.72311.027.037.424151.2
CLIP (+Ours), BERT scores13.834.445.71413.833.845.014186.5
CLIP (+Ours), LLaMA scores14.335.947.61214.836.046.613195.2

Size of the effects

We agree with the reviewer that the gain is relatively smaller on CMC with the synthetic misalignment setup (only +1.7 RSum and a 0.1 decrease on R@10). However, the broader results demonstrate improvements across contrastive objectives, training conditions and evaluation datasets. Across SLIP, CMC and CLIP, CGAAL improves 46 of the 48 recall values reported in Table 1 and increases RSum in all six method–training-condition combinations. For CLIP, the strongest baseline overall, RSum increases by 12.6 points without synthetic misalignment and 14.5 points with it. The gains also extend to zero-shot retrieval on three unseen datasets without further fine-tuning (Table 2).

Model Ablation. Alignment weighting contributes most to our performance. Gradual thresholding gives a further gain with synthetic misalignment, while warm-up improves without it.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP15.938.350.51017.238.850.810211.5
+ alignment weighting18.040.952.0917.640.352.09220.8
+ gradual thresholding18.140.652.3917.239.951.69219.7
+ warm up18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP13.033.544.41413.333.043.516180.7
+ alignment weighting14.835.046.61313.734.344.814189.2
+ gradual thresholding14.435.747.31215.034.946.213193.5
+ warm up14.335.947.61214.836.046.613195.2

Gradual Thresholding Hyperparameters

During gradual thresholding, samples with alignment scores below the current threshold receive a weight equal to their alignment score. Samples at or above the threshold retain a weight of 1. The threshold increases over training: it is initially set to 0.4 for the first two epochs after warm-up, it is then raised to 0.7 for the following two epochs and finally to 1 for the remaining epochs. At this final stage every sample's weight equals its alignment score. Thus, progressively more samples have their contribution adjusted according to their estimated alignment.

Hyperparameter Tuning

A separate validation set of 2000 samples was used to prevent overfitting and optimise hyperparameters before evaluating on the separate test set.

CoMM and InfMasking Adapation

We adapted CoMM and InfMasking using the same audio and visual encoders as our other comparisons. Both methods learn single joint representation of the input audio and video, but their training objectives also explicitly support single-modality inputs. For instance, CoMM constructs unimodal views by masking all other modalities and contrastively aligns their representations with augmented joint representations. Therefore, to perform retrieval, use these single modality representations, masking one modality at a time to generate separate embeddings. We then compare these single modality embeddings to perform A2V and V2A retrieval.

Figure 6 Clarification

The retrieval was performed against the full test gallery, with results grouped by the alignment scores.

Comparison with Noisy-Correspondence Methods

We compare our method to a noise correspondence method VDR [A]. We adapt VDR by replacing the text branch with audio branch and training and evaluating it under our experimental set up. As shown in the table below, our approach achieves better performance on all metrics both with and without synthetic misalignment. This indicates that our method is better able to capture partial correspondence compared to standard noise correspondence methods.

[A] J. Zhou, X. Li, L. Shang, X. Jiang, Q. Liu, and L. Chen. Retrieval-Based Disentangled Representation Learning with Natural Language Supervision. In ICLR, 2024.

NCR comparison. We compare our model to a noise correspondence baseline below.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
VDR13.133.446.71212.533.646.013185.3
CLIP (+Ours)18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
VDR12.632.043.01811.530.742.417172.2
CLIP (+Ours)14.335.947.61214.836.046.613195.2

Multi-Frame Evaluation

To demonstrate that our gains are not specific to the single frame setting, we report our methods performance when training with 5 frames for 24 epochs instead of 48. We observe consistent improvement on RSum when using our method over the baseline. Further, we observe improvement on V2A metrics despite training for a shorter period showing that our approach can benefit from additional visual context.

Scale up results. We compare our method when training with 5 frames.

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP - 5 frames10.929.740.81812.333.243.915170.8
CLIP (+Ours) - 5 frames13.533.645.31416.036.446.912191.7
Reviewer_TGhWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the challenge that audio-visual contrastive learning considers all audio and visual signals appearing together in the same clip as equally matched pairs, even though their contents may not actually correspond to each other, for example because of sounds coming from outside the camera view. To solve this challenge, the authors propose CGAAL. CGAAL compares the captions describing each modality with an LLM to estimate a semantic alignment score for each pair, and uses this score to control how much each positive pair contributes to the contrastive loss. The authors also propose a strategy that introduces the alignment scores into training step by step through warm-up and gradual thresholding. This was demonstrated through audio-visual retrieval experiments on a subset of VAST-27M, where the method is applied to three contrastive learning baselines, CLIP, SLIP, and CMC.

Critical strengths
  1. The proposed method has a simple form that adds a weight for each pair to the existing contrastive loss, does not require captions or an LLM at inference, and can be applied to various contrastive learning objectives.

  2. The proposed method improves the overall retrieval performance of all three contrastive learning baselines, and also shows overall improvements on datasets that were not used for training.

Critical weaknesses
  1. Novelty and motivation. The main contribution and the motivation of this paper are unclear.

    1-1. The authors raise, as the core problem, that the audio and visual signals from the same clip are not always semantically aligned. However, a similar problem has been addressed in a previous audio-visual contrastive learning work [R1], and the approach of reducing the contribution of weakly aligned pairs also appears to be similar. The authors need to clarify what advantage the proposed method provides over the previous work.

    1-2. he proposed method reflects the degree of semantic alignment only for the audio and visual pairs within the same clip. However, pairs from different clips may also have similar content, and such pairs are still trained to be pushed apart. An explanation is needed for why the degree of alignment is not reflected or considered for pairs from different clips.

  2. The reliability of the alignment scores is not verified. The proposed method relies on scores produced from automatically generated captions and an LLM to determine the direction of training, but there is no human evaluation or verification of how well these scores reflect the actual correspondence between audio and visual content. In addition, the experiments are conducted with only one caption source and one LLM, so it is not confirmed whether the proposed method remains effective when the model or the method for producing the scores changes. In other words, it is difficult to distinguish whether the performance comes from the proposed training approach or depends on the characteristics of the specific captions and LLM.

  3. It is not clear whether the performance improvement comes from the proposed alignment information. Since the weight of the proposed method reduces the overall training contribution of the corresponding sample, it can be seen as similar to a kind of data filtering that learns less from weakly aligned samples. However, there is no comparison with simply removing weakly aligned samples or assigning random scores.

  4. There is insufficient evidence of how the proposed method works. The authors claim that, with the proposed method, distances in the embedding space come to reflect the actual correspondence between audio and visual content. However, this is only shown through the performance and a few examples. There is no analysis of whether the distances between pairs actually change according to the degree of alignment after training, and qualitative analyses such as a similarity matrix, the feature distribution with t-SNE, or an attention map are also missing.

[R1] P. Morgado et al. Robust Audio-Visual Instance Discrimination. In CVPR, 2021.

Questions for authors
  1. What are the key differences and advantages of the proposed method compared to the previous work [R1], and why is the degree of alignment not considered for pairs from different clips? A clear explanation of these points would help clarify the contribution.

  2. A verification of how well the alignment scores reflect the actual correspondence between audio and visual content, or results with other captions or LLMs, would help support the effectiveness of the proposed method.

  3. A comparison with simply removing weakly aligned samples or assigning random scores would help clarify whether the performance improvement comes from the alignment information.

  4. The training objective is expressed differently in the figure and in the main text. Which one does the actual implementation follow?

Please refer to the weaknesses.

2 comments

We thank the reviewer for their time, effort and constructive feedback. We are glad to hear that the reviewer appreciated the simplicity of our approach and the performance improvement it gives. We address each of the remaining questions and concerns below.

Since submission, we have better tuned the hyperparameters of our approach, improving its performance. We thus report all new comparisons using this updated configuration.

Comparison with [R1]

We thank the reviewer for pointing us to [R1] (Morgado et al., CVPR 2021), which we agree is relevant. While both works recognise that audio and visual streams from the same clip are not always semantically aligned, the two approaches differ in formulation and methodology. We summarise the key differences below:

  1. Treatment of Negatives. [R1] corrects for both faulty positives and faulty negatives. We deliberately estimate alignment only for positive pairs, because positives are what directly shape the similarity structure that audio-visual retrieval relies on. A false negative, only adds a small repulsive term, and its effect is diluted across the many negatives in the batch. Its impact on the learned similarity, and therefore on retrieval, is correspondingly smaller. We therefore focus the correction where it most directly benefits the retrieval task. To verify, we compare R1 to our method in the table below.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
[R1] - 32 negatives13.230.339.92312.431.341.819168.9
[R1] - 1024 negatives17.136.044.91517.537.747.512200.7
CLIP15.938.350.51017.238.850.810211.5
CLIP (+Ours)18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
[R1] - 32 negatives10.926.033.6439.625.734.232140.0
[R1] - 1024 negatives11.827.635.43212.729.939.821157.2
CLIP13.033.544.41413.333.043.516180.7
CLIP (+Ours)14.335.947.61214.836.046.613195.2
  1. Memory Bank Dependence. In [R1], a memory bank is a fundamental part of the method. It supplies the negative samples, and its handling of both faulty positives and faulty negatives is built on this memory bank. In contrast, our method needs no memory bank. The alignment scores are computed directly from in-batch features and can therefore be added to a standard contrastive objective without the extra storage, hyperparameters (bank size, momentum), or stale-feature issues that come with a memory bank.

  2. Alignment Estimation. [R1] estimates the samples' similarities based on the audio and vision embedding learned which may already suppress modality-specific information due to the multiview redundancy assumption. In contrast, our method estimates alignment by comparing modality-specific captions, capturing both shared and modality-specific semantic content independent of the learned embeddings.

We will clarify these distinctions in the related work section in the revised paper.

Reliability of Alignment Scores

To assess whether caption-based LLM scores faithfully reflect audio-visual correspondence, we conducted a human verification study on 200 clips. Each clip was viewed with its audio and its semnatic alignment was rated using a continuous slider. The LLM scores correlate strongly with the human ratings (Pearson's r=0.957). Absolute differences also indicate close agreement: most samples differ by no more than 0.1, and only 11 of the 200 samples (5.5%) have a difference greater than 0.2. These results support the reliability of our caption-based scoring to estimate audio-visual alignment.

Dependence on Choice of LLM

To assess sensitivity to the choice of LLM, we replaced LLaMA with Gemini to score the same audio and visual caption pairs, using the same prompt and scoring protocol. CGAAL was then trained using these scores while keeping the remaining training setup unchanged.

Both LLaMA-based and Gemini-based scores improve retrieval performance over the CLIP baseline, with and without synthetic misalignment. These results indicate that our method's gains are not reliant on a particular LLM.

Gemini - LLaMa comparison. Comparing our method when using alignment scores from LLaMa and Gemini.

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP15.938.350.51017.238.850.810211.5
CLIP (+Ours) - LLaMa Scores18.441.152.8918.441.452.09224.1
CLIP (+Ours) - Gemini Scores17.641.853.0917.240.752.19222.4

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP13.033.544.41413.333.043.516180.7
CLIP (+Ours) - LLaMa Scores14.335.947.61214.836.046.613195.2
CLIP (+Ours) - Gemini Scores14.635.647.21214.635.046.013193.0

Comparison with removing weakly aligned samples

As suggested, we compare our method with a filtering-based approach, excluding training samples whose alignment score falls below a certain threshold. The table below reports results for thresholds of 0.7 and 0.4. Filtering at either threshold reduces retrieval performance relative to the unfiltered CLIP baseline, both with and without synthetic misalignment. In contrast, our method achieves considerable gains in RSum, demonstrating the benefit of modelling the degree of misalignment rather than discarding misaligned samples.

Filtering Samples. Comparing our method to filtering out weakly aligned samples by excluding samples with alignment scores below a threshold (0.7, 0.4).

Without synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP + (0.7 threshold)3.912.818.8904.312.117.48869.2
CLIP + (0.4 threshold)11.629.239.42011.929.438.920160.4
CLIP15.938.350.51017.238.850.810211.5
CLIP (+Ours)18.441.152.8918.441.452.09224.1

With synthetic misalignment

MethodA2V R1 ↑A2V R5 ↑A2V R10 ↑A2V MR ↓V2A R1 ↑V2A R5 ↑V2A R10 ↑V2A MR ↓RSum ↑
CLIP + (0.7 threshold)3.711.216.61013.510.116.010061.1
CLIP + (0.4 threshold)11.227.936.72311.027.037.424151.2
CLIP13.033.544.41413.333.043.516180.7
CLIP (+Ours)14.335.947.61214.836.046.613195.2

Training Objective Formula and Figure Discrepancy

Thank you for identifying this discrepancy. The objective specified in the main-text equation is the correct one used in our implementation and the figure one is overly simplified. We correct the figure to make it consistent with the equation and implementation.

Embedding Similarity Analysis

To examine how our method affects the learned representation, we compute the mean of the similarity scores of high aligned samples, moderately aligned samples and weakly samples when using CLIP and CLIP (+Ours) and show results in the table below. We report the comparison mean of similarities directly in the original embedding space to examine changes in cross-modal correspondence without introducing a two-dimensional projection of tSNE. We observe a correlation between the decrease in similarity between the samples when using CLIP(+Ours) compared to CLIP as the alignment score decreases.

Mean (± std) vision–audio cosine similarity on test set. Δ = CLIP (+Ours) − CLIP.

Alignment0.8–1.00.6–0.80.4–0.60.2–0.40.0–0.2
CLIP0.371 ± 0.1020.362 ± 0.1240.344 ± 0.1290.309 ± 0.1280.268 ± 0.132
CLIP (+Ours)0.349 ± 0.1020.324 ± 0.1110.304 ± 0.1200.269 ± 0.1160.229 ± 0.114
Δ (rel.)−0.023 (−6.1%)−0.037 (−10.3%)−0.040 (−11.6%)−0.039 (−12.7%)−0.039 (−14.4%)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive feedback and for recognising the importance of addressing partial semantic alignment in audio-visual learning, the practicality of our CGAAL method, and its consistent performance improvements. Below, we summarise the new results; detailed comparisons and analyses appear in our individual response and revised manuscript.

Updated results. Since submission we discovered a new set of hyperparameters that yield better performance, CGAAL continues to demonstrate consistent gains when applied to CLIP, SLIP and CMC. All new comparisons use this updated configuration. We clarified that we had been tuning the hyperparameters on a separate 2,000-sample validation set.

1. Reliability and effectiveness of alignment scoring (all reviewers)

  • New human validation (Figure 8): On 200 clips, our LLM scores show strong agreement with human ratings of the alignment between the vision and the audio (Pearson r = 0.957), with only 5.5% of clips differing by more than 0.2.
  • New scoring comparisons (Table 4): Gemini achieves comparable performance to LLaMA with and without synthetic misalignment, demonstrating robustness to LLM choice. BERT caption-embedding similarity matches LLaMA without synthetic misalignment but falls behind under synthetic misalignment (186.5 vs. 195.2 RSum), demonstrating the value of using LLMs where estimating alignment is more challenging.

Together, these results validate our alignment scores and support the benefit of LLM-based semantic scoring under stronger misalignment.

2. Reweighting vs. filtering (all reviewers) New experiments (Table 1) show that strongest hard filtering baseline substantially reduces performance below CLIP (151.2 vs. 180.7 RSum under synthetic misalignment), whereas CGAAL improves it to 195.2. This supports our central motivation of modelling degrees of semantic correspondence rather than discarding weakly aligned pairs.

3. Novelty and related work (TGhW, WLQd)

  • New comparison with [R1]: We clarify our distinctions in positive-pair weighting, memory-bank independence, and caption-based scoring and include a discussion in our related work. In a direct comparison, CGAAL outperforms [R1] with 1,024 negatives on all metrics with and without synthetic misalignment (224.1 vs. 200.7 RSum without synthetic misalignment and 195.2 vs. 157.2 RSum with synthetic misalignment).
  • New noisy-correspondence baseline [A]: The baseline was adapted by replacing the text branch with an audio one. CGAAL outperforms this noisy-correspondence baseline on all metrics with and without synthetic misalignment as in updated Table 1 (224.1 vs. 185.3 RSum without synthetic misalignment and 195.2 Vs 172.2 with synthetic misalignment).

[R1] P. Morgado et al. Robust Audio-Visual Instance Discrimination. In CVPR, 2021

[A] J. Zhou, X. Li, L. Shang, X. Jiang, Q. Liu, and L. Chen. Retrieval-Based Disentangled Representation Learning with Natural Language Supervision. In ICLR, 2024.

4. Contribution of each component (jEUx, WLQd) Updated ablations (Table 3) confirm alignment weighting as the main contributor (+9.3 / +8.5 RSum). Warm-up further improves both settings (+4.4 / +1.7), while gradual thresholding provides additional gains under synthetic misalignment (+4.3). The new ablation table provide stronger evidence for our design choices reflected by larger performance gains.

5. Generality and learned representations (WLQd, TGhW)

  • CGAAL consistently improves RSum across all three contrastive objectives and both training conditions (+12.6 / +14.5 for CLIP). In our new five-frame experiment, CGAAL substantially outperforms CLIP when training with synthetic misalignment (191.7 vs. 170.8 RSum) despite training for half the epochs, providing evidence of consistent results at larger scale training.
  • New embedding analysis (Table 5) shows a substantially larger similarity reductions for weakly aligned pairs (−14.4%) than highly aligned pairs (−6.1%) when comparing CLIP(+Ours) to CLIP, providing direct evidence that CGAAL better distinguishes semantic correspondence.

Manuscript revisions. We incorporated new experimental comparisons, human validation, updated ablations, embedding analysis and comparison with noise correspondence and weakly aligned samples filtering baselines. We also clarified the gradual-thresholding schedule and corrected Figure 2 to match the implemented objective.

We believe these additions address the reviewers' key concerns, as Reviewer jEUx kindly acknowledged that our new results address their main question. Although time constraints prevent multiple runs during rebuttal, we will assess variability across matched random seeds and report the results in the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.