acceptodds
Under review as a conference paper at ICLR 2027

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Abstract

Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio–visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model–benchmark combinations, while substantially improving the ability to track changing audio–visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio–visual question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omni-modal Models, Audio-Visual Temporal Correspondence, Representation Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p51ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SyncRA, a within-video symmetric InfoNCE objective applied to pooled audio and video tokens at a chosen intermediate layer during QA fine-tuning, with the projection discarded afterwards so that inference is unchanged. It also constructs a swap diagnostic that separately exchanges the audio window, the video window, or both.

Overall, I found the diagnostic genuinely valuable, but the method itself is not new and the claims run ahead of the evidence. Same-moment positives with other moments of the same video as negatives is a classic audio-visual objective, and it has already been applied to QA and inside omni models; what remains is placing it at an intermediate layer on the native time grid, with no experimental comparison against the closest alternatives.

Critical strengths

The swap diagnostic is valuable. Three perturbations give four versions, and the joint swap keeps the pairing and only moves it in time, so answering all four requires following the current pairing rather than global semantics or any single modality.

The core finding is important. After answer-only fine-tuning all four backbones still score low on all-four accuracy while aggregate accuracy is high, so aggregate metrics hide the missing ability, and supervising reasoning text does not fix it.

The method is minimal and portable. Supervision comes from the native time grid, the added projection is discarded, inference is free, and it works across dense and MoE backbones.

Critical weaknesses
  1. The novelty is an issue, and the closest alternatives are not compared. Treating same-moment audio and video as positives and other moments of the same video as negatives is a classic audio-visual self-supervision objective, used within videos by Korbar et al. [1] as early as 2018; a fine-grained version appears in CAV-MAE Sync [2]; AVAF-Net [3] already applies a temporal alignment contrastive loss to audio-visual QA; and LatentOmni [4] already aligns the two streams inside an omni model through shared timestamps. What remains is placing the loss at an intermediate layer on the processor's native grid, an implementation choice, and none of these methods is compared on the same data and backbones. The method is presented as a main contribution, but without these comparisons we cannot tell what it adds over existing objectives.

  2. A substantial part of the benchmark gain is not specific to correspondence supervision. On Qwen3-Omni and Nemotron, permuted pairs and fixed time codes capture a considerable share of SyncRA's benchmark gain, so a generic auxiliary contrastive loss already helps. Yet all twenty improved combinations are presented as evidence for correspondence supervision. The diagnostic does separate SyncRA from the controls, but the benchmark results mostly do not, and the paper leans on them heavily.

References

[1] Korbar, Tran & Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurIPS 2018.

[2] Araujo et al. CAV-MAE Sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment. CVPR 2025 (arXiv:2505.01237).

[3] Zhao, Wang & Jin. Audio-visual adaptive fusion network for question answering based on contrastive learning. AAAI 2025.

[4] Dai et al. LatentOmni: Rethinking omni-modal understanding via unified audio-visual latent reasoning. arXiv:2605.22012, 2026.

Questions for authors
  1. How does SyncRA compare with the objectives of Korbar et al. [1], CAV-MAE Sync [2], AVAF-Net [3], and LatentOmni [4] on the same data and backbones?
  2. Could you show per-backbone benchmark results for each control objective in the main text, split by whether a question needs audio-visual pairing? This would show which part of the gain is specific to correspondence.
  3. "Answer-only supervision cannot fix pairing" is a main conclusion, but it is tested only at equal budget. Does a larger answer-only budget (more data or epochs) close the gap?
  4. All-four accuracy rises with original-version accuracy. Does SyncRA's advantage hold after conditioning on original-version accuracy?
  5. Near-perfect retrieval at the supervised layer measures the training objective itself. Could you run the same retrieval analysis for the control objectives, and lead with final-layer retrieval and the delay control as the main mechanism evidence?

References

[1] Korbar, Tran & Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurIPS 2018.

[2] Araujo et al. CAV-MAE Sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment. CVPR 2025 (arXiv:2505.01237).

[3] Zhao, Wang & Jin. Audio-visual adaptive fusion network for question answering based on contrastive learning. AAAI 2025.

[4] Dai et al. LatentOmni: Rethinking omni-modal understanding via unified audio-visual latent reasoning. arXiv:2605.22012, 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer p51Z (1 of 4)

Thank you for the detailed review and for pointing to the closest prior objectives. We ran the comparison you asked for: we ported the objectives of Korbar et al. [1], CAV-MAE Sync [2], and AVAF-Net [3] to Qwen3-Omni with the same data, budget, and checkpoint selection as SyncRA. We also split the benchmark gain into a generic part and a pairing-specific part, extended the retrieval analysis to all control objectives and to the final layer, and tested larger answer-only budgets.

All new training runs are on Qwen3-Omni with a single seed. They follow the Run 1 protocol of Table 4, with the same 10K corpus and 2-epoch budget unless stated. All intervals are 95% CIs, from a paired per-video bootstrap (10,000 resamples) for diagnostic and retrieval numbers and from Welch intervals over three runs for Avg-5. Retrieval numbers come from a refreshed analysis pipeline and differ from Tables 25 and 28 by at most 0.1 points.

Response to weaknesses

1. In the same setting, SyncRA exceeds the strongest prior objective by 10.00 AllFour points.

"What remains is placing the loss at an intermediate layer on the processor's native grid, an implementation choice, and none of these methods is compared on the same data and backbones." (W1, Q1)

On the same data and backbone, SyncRA outperforms the strongest ported objective by 10.00 AllFour points (CI excludes 0) and the other two by 28.00–29.25 points. Korbar et al. and AVAF-Net are statistically indistinguishable from answer-only fine-tuning. The review describes the remaining difference as a choice of placement. This comparison holds placement fixed, because all four objectives act on the same block (36). AllFour still ranges from 33.25 to 62.50, so at a fixed placement the form of the objective decides whether pairing-following improves.

We re-implemented each auxiliary loss from its paper and ported it to Qwen3-Omni. Only the auxiliary loss changes. The data, answer loss, trainable parameters, supervision block (36), budget (2 epochs), NLL selection, and λ = 0.1 are identical to SyncRA. The projection sizes and the CAV-MAE Sync temperature are not specified in the original papers, and all other choices follow the source. The full porting settings are in Appendix B.9. All three losses converge (Korbar from 1.05 to 0.25, CAV-MAE Sync from 4.42 to 0.12, AVAF-Net from 2.01 to 0.09), so under-training does not explain the outcomes.

Objective (Qwen3-Omni, Run 1 protocol)AllFourΔ vs Vanilla [95% CI]Δ vs SyncRA [95% CI]P(all|O)R@1 (block 36)Daily-Omni
Vanilla SFT34.00–−28.50 [−33.25, −23.75]38.313.475.86
Korbar et al. (AVTS)33.25−0.75 [−3.75, +2.25]−29.25 [−33.75, −24.75]37.812.177.27
CAV-MAE Sync52.50+18.50 [+13.50, +23.50]−10.00 [−14.25, −5.75]57.253.678.36
AVAF-Net (TACL)34.50+0.50 [−2.75, +3.75]−28.00 [−32.25, −23.50]39.111.877.94
SyncRA62.50+28.50 [+23.75, +33.25]–67.096.978.95

Here P(all|O) is AllFour divided by original-version accuracy, and R@1 is unprojected same-interval retrieval at block 36. Daily-Omni is shown as an aggregate reference.

The ordering is consistent with three factors, each tested here or in the paper's ablations. The first is granularity: clip-level pooling gives R@1 13.9, against 96.9 at interval level. The second is the loss form: cosine-only gives 17.9 and pairwise ranking 27.5, against 96.9 for InfoNCE (Table 26). The third is the source of negatives: CAV-MAE Sync with cross-video negatives gives 53.6, against 96.9 for SyncRA with within-video negatives.

Korbar's result is consistent with its negatives. Between 75% and 100% of them come from other videos, so the margin loss is mostly satisfied by telling videos apart, and within-video retrieval stays at the Vanilla level (12.1 vs 13.4). The ported CAV-MAE Sync differs from SyncRA in two ways: its negatives come from other videos, and its loss runs in one direction. It lands in the middle on both R@1 (53.6) and AllFour (52.50). For AVAF-Net we report only the measured outcome: the loss converged to 0.09 while within-video R@1 stayed at 11.8. We are still examining why.

On novelty, we agree that the objective family is classic, and the paper places it in the synchrony line (§2.2, §7). The revision states the contribution in the terms that this comparison supports (§1): we identify the conditions under which synchrony supervision repairs pairing-following in omni-modal QA. These are same-video negatives, interval granularity, and the InfoNCE form. For depth, supervision at either block 24 or block 36 improves AllFour over Vanilla (Table 34), and we did not test other depths. The other two contributions are the swap diagnostic and the finding that answer-only supervision does not fix pairing. The comparison is now in §6.1 (Table 4) and Appendix B.9 (Table 15).

AuthorsOpenReview ↗

Response to Reviewer p51Z (2 of 4)

We did not run a numerical comparison with LatentOmni [4], for three reasons. First, the ICLR 2027 reviewer guidelines (FAQ on recent and non-peer-reviewed work) exempt papers available only on arXiv from required comparisons, and state that the lack of such a comparison "cannot be a basis for rejection." LatentOmni is available only on arXiv, and we cite and discuss it in §7 as the guidelines encourage. Second, the official pipeline cannot be run. At submission, the official repository contained only a README and the paper. As of November 17, 2026 (HEAD 7b2cb4e, November 7, 2026), it contains training and inference code for Qwen2.5-Omni only, the model weights are unreleased, and the dataset page is empty. Third, the loss cannot be ported on its own. The official temporal_sync_loss is defined on generated latent states (last-layer hidden states at latent positions) and is computed only during the latent-reasoning segment, so it depends on the full latent-reasoning pipeline and does not fit the answer-only setting shared by all comparisons above. §7 now states the conceptual difference: LatentOmni aligns extra latent states that it generates at inference, while SyncRA aligns the answering model's own intermediate states and leaves inference unchanged.

2. Part of the Avg-5 gain is generic, and the pairing-specific claims now rest on the diagnostic.

"A substantial part of the benchmark gain is not specific to correspondence supervision." (W2)

We agree, and we now quantify the decomposition. On Avg-5, the strongest control captures 73% of SyncRA's gain on Qwen3-Omni, 45% on Nemotron, and at most 25% on Qwen2.5-Omni. On MiniCPM-o 4.5, every control is below Vanilla. On AllFour, the strongest control captures 35–40% on all four backbones. The first table below gives the Avg-5 attribution over three runs with Welch CIs.

BackboneSyncRA − Vanilla [95% CI]SyncRA − strongest control [95% CI]Share captured by the three controls
Qwen3-Omni+1.93 [+1.65, +2.21]+0.52 [+0.36, +0.68] (Clip-level AV)46–73%
Nemotron+2.11 [+1.88, +2.34]+1.15 [+0.92, +1.38] (Fixed time codes)35–45%
Qwen2.5-Omni+2.27 [+2.00, +2.53]+1.70 [+1.38, +2.02] (Permuted pairs)≤25%
MiniCPM-o 4.5+0.96 [+0.57, +1.34]+2.03 [+1.65, +2.41] (Fixed time codes)controls below Vanilla

SyncRA exceeds the strongest control on all four backbones, with every CI excluding 0. On MiniCPM-o 4.5, even the strongest control is below Vanilla while SyncRA stays above it, so the generic component is not consistently positive across backbones. The second table gives the AllFour attribution with paired per-video bootstrap CIs.

BackboneVanillaPermuted pairsFixed time codesClip-level AVSyncRASyncRA − strongest control [95% CI]Control share
Qwen3-Omni34.0031.0044.7531.7562.50+17.75 [+13.00, +22.75]38%
Qwen2.5-Omni11.0010.2514.5011.2520.75+6.25 [+2.25, +10.25]36%
MiniCPM-o 4.557.2555.5059.0056.2562.25+3.25 [−0.50, +7.00]35%
Nemotron6.006.7510.257.5016.50+6.25 [+2.50, +10.25]40%

The two tables behave differently. On Avg-5, the share of the strongest control ranges from below zero to 73%, so an Avg-5 gain alone cannot be attributed to correspondence supervision. On AllFour, the share stays at 35–40%, and it comes from Fixed time codes, which supplies a shared time target. The two controls without a correct temporal target, Permuted pairs and Clip-level AV, stay within −3.00 to +1.50 points of Vanilla on every backbone. §6.1 notes that similar Daily-Omni accuracies hide different AllFour scores, and the revised §5.2 states the same for the 20/20 result: it presents the 20/20 improved combinations as a general QA gain with a generic component (Table 3), and the pairing-specific claims rest on the diagnostic.

On MiniCPM-o 4.5, SyncRA's AllFour advantage has a CI containing 0 only against the strongest control (Fixed time codes, +3.25 [−0.50, +7.00]). The CIs against Vanilla (+5.00 [+1.00, +9.00]), Clip-level AV (+6.00 [+2.25, +9.75]), and Permuted pairs (+6.75 [+2.75, +10.75]) all exclude 0. The Base model of MiniCPM-o 4.5 already tracks pairing best among the four backbones (35.50% AllFour), and Vanilla reaches 57.25%. This leaves less headroom, and the 100-video diagnostic gives a CI half-width of ±3.75, so statistical power is limited. We report each backbone separately because the effects differ in size (+3.25 to +17.75).

AuthorsOpenReview ↗

Response to Reviewer p51Z (3 of 4)

Response to questions

3. Per-control results are in Appendix F, and the pairing split is available only on the diagnostic.

"Could you show per-backbone benchmark results for each control objective in the main text, split by whether a question needs audio-visual pairing?" (Q2)

Per-benchmark results for every control objective and backbone are in Tables 41–44 (Appendix F). The revision brings them into the main text in condensed form as Table 3 (§5.2), with the shares in Table 21 (Appendix C.5). The AllFour control rows for the three backbones other than Qwen3-Omni are new. We cannot compute the exact split you ask for, because none of the five public benchmarks labels whether a question requires audio–visual pairing. The diagnostic is the closest substitute. Every cue requires pairing by construction, and the gain holds for both cue types: on-screen-speaker cues (220) rise from 37.27 for Vanilla to 65.91 for SyncRA, and off-screen-voice cues (180) rise from 30.00 to 58.33.

4. A larger answer-only budget does not close the gap.

"Does a larger answer-only budget (more data or epochs) close the gap?" (Q3)

It does not, within the budgets we tested. With 4× optimization steps or 3× data, Vanilla stays at 34.00–36.75 AllFour and SyncRA at 62.25–64.25. The gap holds at +27.25 to +28.50, with every CI excluding 0.

Run (NLL-selected)Vanilla AllFourSyncRA AllFourΔ [95% CI]Vanilla R@1SyncRA R@1Daily-Omni Δ
2 epochs (paper setting)34.0062.50+28.50 [+23.75, +33.25]13.496.9+3.09
4 epochs34.7562.75+28.00 [+23.50, +32.25]13.696.6+3.01
8 epochs35.0062.25+27.25 [+22.25, +31.75]13.896.4+2.92
30K data, 2 epochs36.7564.25+27.50 [+23.00, +32.25]14.897.2+3.01

The 2, 4, and 8-epoch runs are independent runs on the 10K corpus, at about 60, 120, and 240 GPU-hours. The LR schedule is rescaled to the total steps of each run, and checkpoints are selected by per-epoch validation NLL as in the paper. The 30K run uses about 30K questions from about 4,000 videos for 2 epochs.

Answer fitting saturates after the first epoch. The best validation answer NLL is 1.494, 1.492, and 1.491 for 2, 4, and 8 epochs, and the 8-epoch run overfits to 1.618 by its last epoch. SyncRA's validation answer NLL is slightly higher than Vanilla's at every matched point (for example, 1.541 vs 1.494 at 2 epochs), while its AllFour is far higher. The gain therefore does not come from better answer fitting.

Two further checks agree. In the 8-epoch run, Vanilla's best epoch reaches 35.75 (epoch 4) and SyncRA's worst epoch reaches 55.75 (epoch 1). If we let Vanilla pick its best epoch on the diagnostic itself, it still trails SyncRA's worst epoch by 20 points, and the gap is already +24.50 after the first epoch. Last-epoch comparisons, which depart from the NLL selection rule, give +27.75 at 4 epochs and +26.75 at 8 epochs. With 3× data, SyncRA's advantage changes by −1.00 [−5.25, +3.25]. The CI contains 0, so we see no measurable attenuation.

The comparison in point 1 supports this from another direction: at equal budget, the Korbar and AVAF-Net objectives also leave AllFour at the Vanilla level (at most 34.50). Within the tested budgets, the gap comes from the missing pairing signal. These results are in Appendix C.6 (Table 24).

5. The advantage holds after conditioning on original-version accuracy.

"Does SyncRA's advantage hold after conditioning on original-version accuracy?" (Q4)

It does. The advantage is unchanged on Qwen3-Omni and MiniCPM-o 4.5, and larger on Qwen2.5-Omni and Nemotron. We take the per-example predictions behind Table 17 and keep only the cues where both models answer the original version (O) correctly. On this set, we compare the joint accuracy on the A, V, and AV versions (Δjoint).

BackboneRaw ΔAllFourΔjoint [95% CI]Joint set size
Qwen3-Omni+28.50+28.86 [+23.68, +34.10]350
Qwen2.5-Omni+9.75+14.02 [+8.21, +19.84]264
MiniCPM-o 4.5+5.00+4.95 [+0.85, +9.10]364
Nemotron+10.50+14.03 [+9.19, +19.03]278

On Qwen3-Omni, SyncRA's Δjoint against the three control objectives ranges from +17.75 to +31.49, and all CIs exclude 0. On the same backbone, P(all four | original correct) rises from 38.3% for Vanilla to 67.0% for SyncRA. The AllFour gain is therefore not a side effect of higher original-version accuracy. Among cues that both models answer correctly on the original input, SyncRA follows the changed pairings far more often. This analysis is in Appendix C.6 (Table 22).

Reviewer_zhJUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes a framework to alleviate mismatches in associating audio and visual observations occurring at the same time in omni-modal question answering. Its key design introduces a temporal contrastive objective on intermediate model states during QA fine-tuning, using native timing information to align same-interval audio–visual representations while distinguishing those from different intervals within the same video. The method is evaluated through controlled audio–visual swap tests and benchmark comparisons. The proposed components are ablated to prove effectiveness.

Critical strengths
  • The overall framework is well-motivated (by carefully designed experiments to identity the hidden weakness of common approaches) and designed.

  • The performance gain is clear and consistent. And the results are comprehensively analyzed.

Critical weaknesses
  • The connection between representation alignment and behavioral improvement remains incompletely understood. The results show improvements in both temporal alignment and question answering, but provide limited evidence about how the learned alignment contributes to the model’s answer generation.
Questions for authors

See the weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zhJU (1 of 2)

We thank the reviewer for the positive assessment and for stating this concern clearly. The paper shows that temporal alignment and answering improve together, and you ask how the first contributes to the second. We ran two new analyses on Qwen3-Omni to test this link: activation patching and final-layer retrieval. We also compared alignment strength with swap-test accuracy across the 12 training configurations that we have on this backbone. All analyses use the Run 1 protocol of Table 4.

How the learned alignment contributes to answer generation

"The connection between representation alignment and behavioral improvement remains incompletely understood. The results show improvements in both temporal alignment and question answering, but provide limited evidence about how the learned alignment contributes to the model's answer generation."

In both Vanilla SFT and SyncRA, the answer depends on the hidden states at the time when the cue is spoken. SyncRA makes this dependence more selective, and its audio states at that time carry more of the simultaneous visual information. Across the 12 configurations, AllFour rises with the strength of the alignment. We give the three results below and then state what they do not show.

1. The answer depends on the hidden states at the cue time, and SyncRA makes this dependence more selective.

We run the model on two inputs: a clip from the swap test, and the same clip with its video windows exchanged (the video exchange of §4). The audio is identical in the two inputs, but a different scene is shown while the cue is spoken, so the correct answer is different. We then copy a chosen subset of hidden states from the second run into the first run, from block 4 to the last block, and measure how far the answer moves toward the answer of the second input. This procedure is activation patching. An effect of 0 means that the answer does not move. An effect of 1 means that it moves as far as when the model is run on the exchanged input itself.

We call the window in which the cue is spoken the cue window, and the window exchanged with it the partner window. We use 400 patching instances (50 source videos × 4 cues × 2 input pairs) and analyze the instances that each model answers correctly without patching (215 for Vanilla SFT, 289 for SyncRA).

Hidden states copied from the exchanged inputVanilla SFTSyncRASyncRA − Vanilla [95% CI]
Video tokens of the cue window and the partner window0.990.99—
Video tokens of the cue window only0.910.97+0.06 [+0.03, +0.08]
Video tokens of the partner window only0.120.02−0.10 [−0.13, −0.07]
Audio tokens of the cue window only0.320.46+0.14 [+0.11, +0.17]

The answer depends on the video states of the cue window. Copying only these states flips the answer in 93% of the instances for SyncRA (effect 0.97) and in 83% for Vanilla SFT (effect 0.91). SyncRA depends less on the other window: the effect of the partner window is 0.02 for SyncRA and 0.12 for Vanilla SFT.

In SyncRA, the audio states of the cue window also carry more visual information. The audio input is the same in the two runs, so these states differ only through what they have taken from the video. Copying them has an effect of 0.46 in SyncRA and 0.32 in Vanilla SFT.

The two models read these states at the same depth. When the copy starts at block 24 instead of block 4, about half of the effect remains (0.53 for Vanilla SFT, 0.52 for SyncRA). When it starts at block 36, the effect is about 0 for both. The answer therefore reads the cue-window states before block 36. SyncRA applies its loss at the output of block 36, so its gradient shapes the blocks in which this reading happens.

AuthorsOpenReview ↗

Response to Reviewer zhJU (2 of 2)

2. Across 12 training configurations, AllFour rises with the strength of the alignment.

If the alignment contributes to the answer, a configuration that produces stronger alignment should score higher on the swap test. The table below lists the 11 block-36 configurations that differ in the auxiliary objective or its weight, plus Vanilla SFT. It includes the controls and ablations of the paper, and three prior synchrony objectives (Korbar et al. 2018, CAV-MAE Sync, AVAF-Net) that we ported to the same data, budget, and supervision block during the rebuttal. Rows are sorted by same-interval retrieval at block 36 (R@1 on unprojected states, as in Table 25).

Training configurationR@1 at block 36 (%)AllFour (%)
AVAF-Net (ported)11.834.50
Korbar et al. (ported)12.133.25
Permuted pairs12.731.00
Vanilla SFT13.434.00
Clip-level AV13.931.75
Cosine-only17.940.50
Fixed time codes22.644.75
Pairwise ranking27.546.25
CAV-MAE Sync (ported)53.652.50
SyncRA, λ = 0.0390.958.00
SyncRA, λ = 0.393.758.75
SyncRA, λ = 0.1 (default)96.962.50

The table is now part of Table 4 (§6.1). R@1 for the three controls and all values for the ported objectives are new. Retrieval numbers in this response come from a refreshed analysis pipeline and differ from Tables 25 and 28 by at most 0.1 points.

The five configurations whose R@1 stays at the Vanilla level (11.8–13.9) have AllFour between 31.00 and 34.50. Above this level, AllFour increases with R@1, up to 62.50 at R@1 96.9. The interval-filtering variants in our response to Reviewer quya (R@1 93.8–95.9, AllFour 57.25–60.75) fall in the same range. Neighboring rows can differ by less than the bootstrap uncertainty of the diagnostic, so the evidence is the trend over the whole range.

This is a correlation across configurations and not an intervention. It also concerns one depth. With supervision at block 24 instead (Table 34), R@1 is 68.37 at block 24 and 39.87 at block 36, and AllFour is 60.00. This is consistent with the patching result that part of the reading happens before block 24, and it shows that block-36 R@1 is not the only depth that matters.

3. The alignment lasts to the final layer and follows the current time position.

The paper measures retrieval at the supervised block (§6.2). We repeated the analysis at the final layer (block 48) under the protocol of Appendix D.2, and added the three control objectives. SyncRA's R@1 at block 48 is 31.06%, against 3.44% for Vanilla SFT, 3.22% for Base, 3.40% for Permuted pairs, 3.68% for Clip-level AV, and 7.49% for Fixed time codes, the strongest control. The difference is +27.63 [+26.28, +28.98] against Vanilla SFT and +23.58 [+22.27, +24.87] against Fixed time codes (95% CIs, paired per-video bootstrap). Fixed time codes brings retrieval as close in time as SyncRA, with a mean time error of 14.24 s against 14.26 s, but it retrieves the exact interval at only 7.49%.

R@1 is lower at block 48 than at block 36 for every one of these models, including Vanilla SFT (13.4 → 3.44); Figure 4 shows the same decline. The patching result shows that the answer reads the cue-window states before block 36. We therefore use the final-layer result as evidence that the reorganization lasts in the model's own states, and not as a step in how the answer is computed.

The aligned states follow the current time position. In the delay control of the paper (Appendix D.3), the audio is delayed by two intervals. SyncRA then retrieves the interval at the current position at 96.44% and the interval where the content originally was at 0.01%. For Vanilla SFT, the two numbers are 7.92% and 7.79%. Among the controls, Fixed time codes shows the largest gap (17.81% vs 11.53%), and the other two differ by less than 0.5 points. The swap test asks for the same relation: which scene is shown while the cue is spoken.

Scope and revision

There are three limits. First, all analyses use Qwen3-Omni Run 1 checkpoints. For the other three backbones we have the AllFour gains (Table 1) and the heatmaps (Figure 6), but no patching. Second, patching uses the instances that each model answers correctly, so the two models are compared on different instance sets. Third, we have not removed the alignment from a trained SyncRA model to test whether the AllFour gain disappears. The link between alignment strength and AllFour therefore rests on the comparison across configurations.

The revision adds the patching study and the final-layer retrieval to §6.2 and Appendix D. §6.2 now leads with the final-layer retrieval and the delay control, then the patching study. The three limits above are in Appendix A.

We hope these results address your concern. If they do, we would appreciate it if you could take them into account in your assessment. Please let us know if any part of the link remains unclear.

I thank the authors for the comprehensive rebuttal, in which my concerns have been well addressed. Therefore, I would like to keep my original score.

Reviewer_quyaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether omni-modal models answer based on the audio–visual pairing currently in the input. It builds a swap test — four spoken cues per video, with audio and/or video windows exchanged — where a cue is credited only if all four versions are answered correctly (AllFour). Models often fail this even after answer-only fine-tuning. To fix it, SyncRA adds a small auxiliary loss during QA fine-tuning: audio and visual states are pooled within the processor's native time intervals at one probe-selected layer and aligned with a within-video contrastive loss, on free co-occurrence labels, with nothing added at inference. Across four backbones and five benchmarks, AllFour improves by +5.0 to +28.5 points and mean accuracy is higher in all 20 model–benchmark combinations.

Critical strengths
  1. A clean diagnostic for a real failure mode: the correct answer changes exactly when the audio–visual pairing changes.
  2. A cheap and practical method: no extra annotations, a training-only projection head, nothing added at inference.
  3. Clean attribution: wrong-pairing controls barely help on the swap metric, and internal analyses verify the mechanism.
  4. Careful statistics and consistent gains: three runs with paired intervals, and higher mean accuracy in every model–benchmark combination.
Critical weaknesses
  1. Limited objective novelty: the paper itself places the objective in the classic synchrony-supervision line (its §2.2 and related work), and the closest competitor, LatentOmni, is discussed but never compared. What is new is the attachment point and the diagnostic, not the objective, so the margin over the nearest alternative is argued rather than measured.
  2. Input-time binding, not semantic correspondence: the model learns to bind what is heard at time t with what is shown at time t, whether or not the two are actually related, as its own delay controls confirm. Narration referents, off-screen sound, and desync detection are therefore out of scope; the motivation citing sync verification and grounding overreaches, and claims should be narrowed.
  3. Narrow evaluation: the swap test is multiple-choice on 100 videos, open-ended tracking is untested, and all results sit at one data scale (10K-question fine-tuning). It is therefore unclear whether pairing-following survives in open-ended generation or at realistic data scales.
Questions for authors
  1. Attribution of the general QA gain: How much of the Avg-5 improvement is unique to correct pairing? On Qwen3-Omni the wrong-pairing control gains +1.35 of SyncRA's +1.93; please decompose the gain into a generic auxiliary-training effect and a pairing-specific effect.
  2. Behavior on mismatched inputs: When audio and video at the same moment do not correspond (desynced dubbing, narration referring elsewhere), does SyncRA improve, preserve, or reduce the ability to notice the mismatch?
  3. Noisy positives from narration and music: What fraction of training intervals has no real audio–visual link, and does filtering or down-weighting them change the results?

You can only answer the three questions above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer quya (1 of 4)

Thank you for the careful review. We agree with your characterization of what SyncRA learns: it binds what is heard at time t with what is shown at time t, and the revision narrows the claims to this scope. We also ran the experiments you asked for: a same-setting comparison with prior objectives, an open-ended version of the swap test, a larger data scale, a decomposition of the Avg-5 gain, a sync-judgment test, and an audit of the training intervals with filtering ablations.

All new training runs are on Qwen3-Omni with a single seed. They follow the Run 1 protocol of Table 4, with the same 10K corpus and 2-epoch budget unless stated. All intervals are 95% CIs, from a paired per-video bootstrap (10,000 resamples) for diagnostic numbers and from Welch intervals over three runs for Avg-5. Retrieval numbers come from a refreshed analysis pipeline and differ from Tables 25 and 28 by at most 0.1 points.

Response to weaknesses

1. We now measure the margin over three prior objectives, and LatentOmni is still not compared numerically.

"What is new is the attachment point and the diagnostic, not the objective, so the margin over the nearest alternative is argued rather than measured." (W1)

We re-implemented the auxiliary losses of Korbar et al. (AVTS), CAV-MAE Sync, and AVAF-Net (TACL; Zhao et al., AAAI 2025) from their papers and ported them to Qwen3-Omni. Only the auxiliary loss changes. The data, answer loss, trainable parameters, supervision block (36), budget (2 epochs), NLL selection, and λ = 0.1 are identical to SyncRA. All three losses converge (Korbar from 1.05 to 0.25, CAV-MAE Sync from 4.42 to 0.12, AVAF-Net from 2.01 to 0.09), so under-training does not explain the outcomes. Korbar uses a margin loss whose negatives come mostly from other videos, CAV-MAE Sync uses frame-level InfoNCE with cross-video negatives, and AVAF-Net uses temporal affinity consistency within each video. The porting settings are in Appendix B.9.

Objective (Qwen3-Omni, Run 1 protocol)AllFourΔ vs Vanilla [95% CI]Δ vs SyncRA [95% CI]P(all|O)R@1 (block 36)Daily-Omni
Vanilla SFT34.00–−28.50 [−33.25, −23.75]38.313.475.86
Korbar et al. (AVTS)33.25−0.75 [−3.75, +2.25]−29.25 [−33.75, −24.75]37.812.177.27
CAV-MAE Sync52.50+18.50 [+13.50, +23.50]−10.00 [−14.25, −5.75]57.253.678.36
AVAF-Net (TACL)34.50+0.50 [−2.75, +3.75]−28.00 [−32.25, −23.50]39.111.877.94
SyncRA62.50+28.50 [+23.75, +33.25]–67.096.978.95

Here P(all|O) is AllFour divided by original-version accuracy, and R@1 is unprojected same-interval retrieval at block 36. Daily-Omni is shown as an aggregate reference.

SyncRA exceeds the strongest ported objective, CAV-MAE Sync, by 10.00 AllFour points (CI excludes 0). Korbar and AVAF-Net are statistically indistinguishable from answer-only fine-tuning. We agree with your review that the objective family is classic and that the attachment point is new. This comparison holds the attachment point fixed, because all four objectives act on the same block (36). AllFour still ranges from 33.25 to 62.50, so at a fixed attachment point the form of the objective decides whether pairing-following improves.

The ordering is consistent with three factors, each supported by a comparison here or in the paper. The first is granularity: the Clip-level AV control, which pools over the whole clip, gives R@1 13.9, against 96.9 at interval level. The second is the loss form: cosine-only gives 17.9 and pairwise ranking 27.5, against 96.9 for InfoNCE (Table 26). The third is the source of negatives: CAV-MAE Sync with cross-video negatives gives 53.6, against 96.9 for SyncRA with within-video negatives. The ported CAV-MAE Sync also runs its loss in one direction, so this last contrast includes that difference. The revision states the contribution in these terms: we identify the conditions under which classic synchrony supervision repairs pairing-following in omni-modal QA. The comparison is now in §6.1 (Table 4) and Appendix B.9 (Table 15).

We did not run a numerical comparison with LatentOmni, for three reasons. First, the ICLR 2027 reviewer guidelines exempt arXiv-only papers such as LatentOmni from required comparisons. Second, the official pipeline cannot be run. As of November 17, 2026 (HEAD 7b2cb4e), the official repository contains code for Qwen2.5-Omni only, the model weights are unreleased, and the dataset page is empty. Third, the loss cannot be ported on its own. The official temporal_sync_loss is defined on generated latent states and is computed only during the latent-reasoning segment, so it does not fit our answer-only setting. §7 now states the conceptual difference: LatentOmni aligns extra latent states that it generates at inference, while SyncRA aligns the answering model's own intermediate states and leaves inference unchanged.

AuthorsOpenReview ↗

Response to Reviewer quya (2 of 4)

2. We agree that SyncRA learns input-time binding, and the revision narrows the claims to it.

"Input-time binding, not semantic correspondence: the model learns to bind what is heard at time t with what is shown at time t, whether or not the two are actually related..." (W2)

Three results support your reading. The first is the delay control in the paper (§6.2 and Table 29). After a two-interval audio delay, SyncRA retrieves the current interval at 96.44% R@1 and the content origin at 0.01%, so retrieval follows when, not what. The second is a new audit of the training intervals (point 6). Between 43% (automatic labels) and 60% (author blind labels) of the supervised intervals have no visible sound source. A positive pair is an audio segment and the scene shown at the same time, with or without the sound source in view. The third is a new filtering ablation (point 6). Restricting the loss to visible-source intervals lowers AllFour by 5.25 points [−9.25, −1.25], and the drop concentrates on off-screen-voice cues (from 58.33 to 47.78). This is consistent with the same reading.

The revision changes three places. §1 now states the scope: SyncRA targets local input-time correspondence between the audio and visual streams, and narration referents, off-screen sound semantics, and desync detection are outside this scope. The motivation in §1 that cites failures of synchronization verification and grounding is narrowed to the local-pairing failure that our diagnostic measures. §8 and the limitations in Appendix A carry the same scope statement.

3. Pairing-following holds in open-ended answers and at 3× data.

"It is therefore unclear whether pairing-following survives in open-ended generation or at realistic data scales." (W3)

For the open-ended test, we removed the options and asked each of the six Qwen3-Omni checkpoints to describe the scene visible while the cue is spoken. This gives 1,600 answers per checkpoint and 9,600 in total. Two text-only LLM judges (Qwen3-Omni and Qwen2.5-Omni) independently matched each answer to one of the four reference scenes or to "no correspondence", without knowing which checkpoint produced it. The two judges agree on 86.6% of the answers (κ = 0.83).

ModelOpen-ended AllFour (Qwen3 judge / Qwen2.5 judge / both judges)Multiple-choice AllFour
Base3.75 / 3.75 / 3.2513.25
Vanilla SFT13.00 / 11.75 / 9.2534.00
Permuted pairs14.00 / 13.50 / 11.0031.00
Fixed time codes19.25 / 18.25 / 15.2544.75
Clip-level AV15.75 / 14.25 / 11.2531.75
SyncRA33.75 / 33.75 / 26.7562.50

SyncRA exceeds the strongest control, Fixed time codes, under all three judging criteria: +14.50 [+9.50, +19.50], +15.50 [+10.25, +20.75], and +11.50 [+6.50, +16.50]. Open-ended scores are lower than multiple-choice scores for every model (33.75 against 62.50 for SyncRA), and SyncRA remains the highest. After conditioning on a correct original version, P(all four | original correct) under the Qwen3 judge is 43.4% for SyncRA, 27.6% for Fixed time codes, and 18.9% for Vanilla.

To check the judges, an author blind-labeled 200 answers (100 random and 100 on which the judges disagree). The author agrees with the two judges on 78% and 80% of these answers. A judge chose "no correspondence" where the author recognized the correct scene in 9–12 answers. A judge accepted an answer that the author rejected in 1–2 answers. The judges are therefore stricter than the author, and judge leniency does not explain SyncRA's lead.

For scale, we trained Vanilla and SyncRA on about 30K questions from about 4,000 videos, and for 4 and 8 epochs on the 10K corpus. All runs in the table below are NLL-selected and use a single seed.

RunVanilla AllFourSyncRA AllFourΔ [95% CI]
10K, 2 epochs (paper setting)34.0062.50+28.50 [+23.75, +33.25]
10K, 4 epochs34.7562.75+28.00 [+23.50, +32.25]
10K, 8 epochs35.0062.25+27.25 [+22.25, +31.75]
30K, 2 epochs36.7564.25+27.50 [+23.00, +32.25]

With 3× data, SyncRA's advantage changes by −1.00 [−5.25, +3.25]. The CI contains 0, so we see no measurable attenuation. With longer training, the best validation answer NLL is 1.494, 1.492, and 1.491 for 2, 4, and 8 epochs, so answer fitting saturates after the first epoch while the AllFour gap stays. We have not tested beyond 30K questions, and behavior at larger scales remains open. The results are in Appendix C.6 (Tables 23 and 24), and the first and fourth limitations in Appendix A now include them and this bound.

AuthorsOpenReview ↗

Response to Reviewer quya (3 of 4)

Response to questions

4. On Qwen3-Omni, the strongest control captures +1.41 of the +1.93 Avg-5 gain, and SyncRA adds +0.52 beyond it.

"How much of the Avg-5 improvement is unique to correct pairing?" (Q1)

On Qwen3-Omni, the strongest control (Clip-level AV) captures +1.41 of SyncRA's +1.93 Avg-5 gain, and SyncRA adds +0.52 [+0.36, +0.68] on top of it. The Permuted pairs control that you cite captures +1.35. The share of the strongest control differs widely across backbones on Avg-5: 73% on Qwen3-Omni, 45% on Nemotron, and at most 25% on Qwen2.5-Omni. On MiniCPM-o 4.5, every control is below Vanilla. On AllFour, the strongest control captures 35–40% on all four backbones. The first table below gives the Avg-5 attribution over three runs with Welch CIs.

BackboneSyncRA − Vanilla [95% CI]SyncRA − strongest control [95% CI]Share captured by the three controls
Qwen3-Omni+1.93 [+1.65, +2.21]+0.52 [+0.36, +0.68] (Clip-level AV)46–73%
Nemotron+2.11 [+1.88, +2.34]+1.15 [+0.92, +1.38] (Fixed time codes)35–45%
Qwen2.5-Omni+2.27 [+2.00, +2.53]+1.70 [+1.38, +2.02] (Permuted pairs)≤25%
MiniCPM-o 4.5+0.96 [+0.57, +1.34]+2.03 [+1.65, +2.41] (Fixed time codes)controls below Vanilla

SyncRA exceeds the strongest control on all four backbones, with every CI excluding 0. The second table gives the AllFour attribution with paired per-video bootstrap CIs. Its control columns for the three backbones other than Qwen3-Omni are new.

BackboneVanillaPermuted pairsFixed time codesClip-level AVSyncRASyncRA − strongest control [95% CI]Control share
Qwen3-Omni34.0031.0044.7531.7562.50+17.75 [+13.00, +22.75]38%
Qwen2.5-Omni11.0010.2514.5011.2520.75+6.25 [+2.25, +10.25]36%
MiniCPM-o 4.557.2555.5059.0056.2562.25+3.25 [−0.50, +7.00]35%
Nemotron6.006.7510.257.5016.50+6.25 [+2.50, +10.25]40%

The two tables behave differently. On Avg-5, the share of the strongest control ranges from below zero to 73%, so an Avg-5 gain alone cannot be attributed to correct pairing. On AllFour, the share stays at 35–40%, and it comes from Fixed time codes, which supplies a shared time target. The two controls without a correct temporal target, Permuted pairs and Clip-level AV, stay within −3.00 to +1.50 points of Vanilla on every backbone. The revised §5.2 (Table 3) presents the 20/20 improved combinations as a QA gain that is partly generic to auxiliary training, and the pairing-specific claims rest on the swap diagnostic.

On MiniCPM-o 4.5, SyncRA's AllFour advantage over the strongest control has a CI containing 0 (+3.25 [−0.50, +7.00]). The Base model of MiniCPM-o 4.5 already tracks pairing best among the four backbones (35.50% AllFour), and Vanilla reaches 57.25%. This leaves less headroom, and the 100-video diagnostic gives a CI half-width of ±3.75, so statistical power is limited.

5. SyncRA gives no measurable change in sync judgment, and this test cannot detect harm.

"...does SyncRA improve, preserve, or reduce the ability to notice the mismatch?" (Q2)

We built a sync-judgment test for the desync case. We did not test narration that refers to content elsewhere. We took 176 talking-face clips from WorldSense and made four versions of each: two in sync and two with the audio shifted by +2 s and −2 s. We ask "Is the audio in sync with the video?" and read the result in two ways. The first is the generated answer, scored by balanced accuracy. The second is a threshold-free logit readout that compares log P(out-of-sync) − log P(in-sync) across the versions of the same clip.

ModelBalanced accuracy (generated)Answers "in sync"Logit readout [95% CI]
Base49.497.3%51.1 [47.3, 55.0]
Vanilla SFT50.098.9%51.1 [47.2, 55.0]
Permuted pairs49.999.0%53.0 [49.0, 56.9]
Fixed time codes50.197.0%55.5 [51.8, 59.2]
Clip-level AV49.697.6%49.9 [46.1, 53.7]
SyncRA49.797.4%55.0 [51.1, 58.7]

Every model, including Base, answers "in sync" on 97% to 99% of the inputs, and balanced accuracy stays between 49.4 and 50.1. On the logit readout, SyncRA − Vanilla is +3.9 [−1.6, +9.4], and the CI contains 0.

The test has a floor effect. No model detects a 2 s shift in its generated answers, so the test shows no measurable change from SyncRA and cannot tell whether the ability is preserved or reduced. In a spot check of 30 clips, 80% show a clearly visible speaker, so some clips give weak visual evidence of the shift. We treat explicit sync judgment as a separate capability from input-time binding, which matches the narrowed scope in point 2. This result and its limitation are in Appendix E.4 (Table 37).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.