- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SyncRA, a within-video symmetric InfoNCE objective applied to pooled audio and video tokens at a chosen intermediate layer during QA fine-tuning, with the projection discarded afterwards so that inference is unchanged. It also constructs a swap diagnostic that separately exchanges the audio window, the video window, or both.
Overall, I found the diagnostic genuinely valuable, but the method itself is not new and the claims run ahead of the evidence. Same-moment positives with other moments of the same video as negatives is a classic audio-visual objective, and it has already been applied to QA and inside omni models; what remains is placing it at an intermediate layer on the native time grid, with no experimental comparison against the closest alternatives.
Critical strengths
The swap diagnostic is valuable. Three perturbations give four versions, and the joint swap keeps the pairing and only moves it in time, so answering all four requires following the current pairing rather than global semantics or any single modality.
The core finding is important. After answer-only fine-tuning all four backbones still score low on all-four accuracy while aggregate accuracy is high, so aggregate metrics hide the missing ability, and supervising reasoning text does not fix it.
The method is minimal and portable. Supervision comes from the native time grid, the added projection is discarded, inference is free, and it works across dense and MoE backbones.
Critical weaknesses
-
The novelty is an issue, and the closest alternatives are not compared. Treating same-moment audio and video as positives and other moments of the same video as negatives is a classic audio-visual self-supervision objective, used within videos by Korbar et al. [1] as early as 2018; a fine-grained version appears in CAV-MAE Sync [2]; AVAF-Net [3] already applies a temporal alignment contrastive loss to audio-visual QA; and LatentOmni [4] already aligns the two streams inside an omni model through shared timestamps. What remains is placing the loss at an intermediate layer on the processor's native grid, an implementation choice, and none of these methods is compared on the same data and backbones. The method is presented as a main contribution, but without these comparisons we cannot tell what it adds over existing objectives.
-
A substantial part of the benchmark gain is not specific to correspondence supervision. On Qwen3-Omni and Nemotron, permuted pairs and fixed time codes capture a considerable share of SyncRA's benchmark gain, so a generic auxiliary contrastive loss already helps. Yet all twenty improved combinations are presented as evidence for correspondence supervision. The diagnostic does separate SyncRA from the controls, but the benchmark results mostly do not, and the paper leans on them heavily.
References
[1] Korbar, Tran & Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurIPS 2018.
[2] Araujo et al. CAV-MAE Sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment. CVPR 2025 (arXiv:2505.01237).
[3] Zhao, Wang & Jin. Audio-visual adaptive fusion network for question answering based on contrastive learning. AAAI 2025.
[4] Dai et al. LatentOmni: Rethinking omni-modal understanding via unified audio-visual latent reasoning. arXiv:2605.22012, 2026.
Questions for authors
- How does SyncRA compare with the objectives of Korbar et al. [1], CAV-MAE Sync [2], AVAF-Net [3], and LatentOmni [4] on the same data and backbones?
- Could you show per-backbone benchmark results for each control objective in the main text, split by whether a question needs audio-visual pairing? This would show which part of the gain is specific to correspondence.
- "Answer-only supervision cannot fix pairing" is a main conclusion, but it is tested only at equal budget. Does a larger answer-only budget (more data or epochs) close the gap?
- All-four accuracy rises with original-version accuracy. Does SyncRA's advantage hold after conditioning on original-version accuracy?
- Near-perfect retrieval at the supervised layer measures the training objective itself. Could you run the same retrieval analysis for the control objectives, and lead with final-layer retrieval and the delay control as the main mechanism evidence?
References
[1] Korbar, Tran & Torresani. Cooperative learning of audio and video models from self-supervised synchronization. NeurIPS 2018.
[2] Araujo et al. CAV-MAE Sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment. CVPR 2025 (arXiv:2505.01237).
[3] Zhao, Wang & Jin. Audio-visual adaptive fusion network for question answering based on contrastive learning. AAAI 2025.
[4] Dai et al. LatentOmni: Rethinking omni-modal understanding via unified audio-visual latent reasoning. arXiv:2605.22012, 2026.
5 comments
Response to Reviewer p51Z (2 of 4)
We did not run a numerical comparison with LatentOmni [4], for three reasons. First, the ICLR 2027 reviewer guidelines (FAQ on recent and non-peer-reviewed work) exempt papers available only on arXiv from required comparisons, and state that the lack of such a comparison "cannot be a basis for rejection." LatentOmni is available only on arXiv, and we cite and discuss it in §7 as the guidelines encourage. Second, the official pipeline cannot be run. At submission, the official repository contained only a README and the paper. As of November 17, 2026 (HEAD 7b2cb4e, November 7, 2026), it contains training and inference code for Qwen2.5-Omni only, the model weights are unreleased, and the dataset page is empty. Third, the loss cannot be ported on its own. The official temporal_sync_loss is defined on generated latent states (last-layer hidden states at latent positions) and is computed only during the latent-reasoning segment, so it depends on the full latent-reasoning pipeline and does not fit the answer-only setting shared by all comparisons above. §7 now states the conceptual difference: LatentOmni aligns extra latent states that it generates at inference, while SyncRA aligns the answering model's own intermediate states and leaves inference unchanged.
2. Part of the Avg-5 gain is generic, and the pairing-specific claims now rest on the diagnostic.
"A substantial part of the benchmark gain is not specific to correspondence supervision." (W2)
We agree, and we now quantify the decomposition. On Avg-5, the strongest control captures 73% of SyncRA's gain on Qwen3-Omni, 45% on Nemotron, and at most 25% on Qwen2.5-Omni. On MiniCPM-o 4.5, every control is below Vanilla. On AllFour, the strongest control captures 35–40% on all four backbones. The first table below gives the Avg-5 attribution over three runs with Welch CIs.
| Backbone | SyncRA − Vanilla [95% CI] | SyncRA − strongest control [95% CI] | Share captured by the three controls |
|---|---|---|---|
| Qwen3-Omni | +1.93 [+1.65, +2.21] | +0.52 [+0.36, +0.68] (Clip-level AV) | 46–73% |
| Nemotron | +2.11 [+1.88, +2.34] | +1.15 [+0.92, +1.38] (Fixed time codes) | 35–45% |
| Qwen2.5-Omni | +2.27 [+2.00, +2.53] | +1.70 [+1.38, +2.02] (Permuted pairs) | ≤25% |
| MiniCPM-o 4.5 | +0.96 [+0.57, +1.34] | +2.03 [+1.65, +2.41] (Fixed time codes) | controls below Vanilla |
SyncRA exceeds the strongest control on all four backbones, with every CI excluding 0. On MiniCPM-o 4.5, even the strongest control is below Vanilla while SyncRA stays above it, so the generic component is not consistently positive across backbones. The second table gives the AllFour attribution with paired per-video bootstrap CIs.
| Backbone | Vanilla | Permuted pairs | Fixed time codes | Clip-level AV | SyncRA | SyncRA − strongest control [95% CI] | Control share |
|---|---|---|---|---|---|---|---|
| Qwen3-Omni | 34.00 | 31.00 | 44.75 | 31.75 | 62.50 | +17.75 [+13.00, +22.75] | 38% |
| Qwen2.5-Omni | 11.00 | 10.25 | 14.50 | 11.25 | 20.75 | +6.25 [+2.25, +10.25] | 36% |
| MiniCPM-o 4.5 | 57.25 | 55.50 | 59.00 | 56.25 | 62.25 | +3.25 [−0.50, +7.00] | 35% |
| Nemotron | 6.00 | 6.75 | 10.25 | 7.50 | 16.50 | +6.25 [+2.50, +10.25] | 40% |
The two tables behave differently. On Avg-5, the share of the strongest control ranges from below zero to 73%, so an Avg-5 gain alone cannot be attributed to correspondence supervision. On AllFour, the share stays at 35–40%, and it comes from Fixed time codes, which supplies a shared time target. The two controls without a correct temporal target, Permuted pairs and Clip-level AV, stay within −3.00 to +1.50 points of Vanilla on every backbone. §6.1 notes that similar Daily-Omni accuracies hide different AllFour scores, and the revised §5.2 states the same for the 20/20 result: it presents the 20/20 improved combinations as a general QA gain with a generic component (Table 3), and the pairing-specific claims rest on the diagnostic.
On MiniCPM-o 4.5, SyncRA's AllFour advantage has a CI containing 0 only against the strongest control (Fixed time codes, +3.25 [−0.50, +7.00]). The CIs against Vanilla (+5.00 [+1.00, +9.00]), Clip-level AV (+6.00 [+2.25, +9.75]), and Permuted pairs (+6.75 [+2.75, +10.75]) all exclude 0. The Base model of MiniCPM-o 4.5 already tracks pairing best among the four backbones (35.50% AllFour), and Vanilla reaches 57.25%. This leaves less headroom, and the 100-video diagnostic gives a CI half-width of ±3.75, so statistical power is limited. We report each backbone separately because the effects differ in size (+3.25 to +17.75).
Response to Reviewer p51Z (3 of 4)
Response to questions
3. Per-control results are in Appendix F, and the pairing split is available only on the diagnostic.
"Could you show per-backbone benchmark results for each control objective in the main text, split by whether a question needs audio-visual pairing?" (Q2)
Per-benchmark results for every control objective and backbone are in Tables 41–44 (Appendix F). The revision brings them into the main text in condensed form as Table 3 (§5.2), with the shares in Table 21 (Appendix C.5). The AllFour control rows for the three backbones other than Qwen3-Omni are new. We cannot compute the exact split you ask for, because none of the five public benchmarks labels whether a question requires audio–visual pairing. The diagnostic is the closest substitute. Every cue requires pairing by construction, and the gain holds for both cue types: on-screen-speaker cues (220) rise from 37.27 for Vanilla to 65.91 for SyncRA, and off-screen-voice cues (180) rise from 30.00 to 58.33.
4. A larger answer-only budget does not close the gap.
"Does a larger answer-only budget (more data or epochs) close the gap?" (Q3)
It does not, within the budgets we tested. With 4× optimization steps or 3× data, Vanilla stays at 34.00–36.75 AllFour and SyncRA at 62.25–64.25. The gap holds at +27.25 to +28.50, with every CI excluding 0.
| Run (NLL-selected) | Vanilla AllFour | SyncRA AllFour | Δ [95% CI] | Vanilla R@1 | SyncRA R@1 | Daily-Omni Δ |
|---|---|---|---|---|---|---|
| 2 epochs (paper setting) | 34.00 | 62.50 | +28.50 [+23.75, +33.25] | 13.4 | 96.9 | +3.09 |
| 4 epochs | 34.75 | 62.75 | +28.00 [+23.50, +32.25] | 13.6 | 96.6 | +3.01 |
| 8 epochs | 35.00 | 62.25 | +27.25 [+22.25, +31.75] | 13.8 | 96.4 | +2.92 |
| 30K data, 2 epochs | 36.75 | 64.25 | +27.50 [+23.00, +32.25] | 14.8 | 97.2 | +3.01 |
The 2, 4, and 8-epoch runs are independent runs on the 10K corpus, at about 60, 120, and 240 GPU-hours. The LR schedule is rescaled to the total steps of each run, and checkpoints are selected by per-epoch validation NLL as in the paper. The 30K run uses about 30K questions from about 4,000 videos for 2 epochs.
Answer fitting saturates after the first epoch. The best validation answer NLL is 1.494, 1.492, and 1.491 for 2, 4, and 8 epochs, and the 8-epoch run overfits to 1.618 by its last epoch. SyncRA's validation answer NLL is slightly higher than Vanilla's at every matched point (for example, 1.541 vs 1.494 at 2 epochs), while its AllFour is far higher. The gain therefore does not come from better answer fitting.
Two further checks agree. In the 8-epoch run, Vanilla's best epoch reaches 35.75 (epoch 4) and SyncRA's worst epoch reaches 55.75 (epoch 1). If we let Vanilla pick its best epoch on the diagnostic itself, it still trails SyncRA's worst epoch by 20 points, and the gap is already +24.50 after the first epoch. Last-epoch comparisons, which depart from the NLL selection rule, give +27.75 at 4 epochs and +26.75 at 8 epochs. With 3× data, SyncRA's advantage changes by −1.00 [−5.25, +3.25]. The CI contains 0, so we see no measurable attenuation.
The comparison in point 1 supports this from another direction: at equal budget, the Korbar and AVAF-Net objectives also leave AllFour at the Vanilla level (at most 34.50). Within the tested budgets, the gap comes from the missing pairing signal. These results are in Appendix C.6 (Table 24).
5. The advantage holds after conditioning on original-version accuracy.
"Does SyncRA's advantage hold after conditioning on original-version accuracy?" (Q4)
It does. The advantage is unchanged on Qwen3-Omni and MiniCPM-o 4.5, and larger on Qwen2.5-Omni and Nemotron. We take the per-example predictions behind Table 17 and keep only the cues where both models answer the original version (O) correctly. On this set, we compare the joint accuracy on the A, V, and AV versions (Δjoint).
| Backbone | Raw ΔAllFour | Δjoint [95% CI] | Joint set size |
|---|---|---|---|
| Qwen3-Omni | +28.50 | +28.86 [+23.68, +34.10] | 350 |
| Qwen2.5-Omni | +9.75 | +14.02 [+8.21, +19.84] | 264 |
| MiniCPM-o 4.5 | +5.00 | +4.95 [+0.85, +9.10] | 364 |
| Nemotron | +10.50 | +14.03 [+9.19, +19.03] | 278 |
On Qwen3-Omni, SyncRA's Δjoint against the three control objectives ranges from +17.75 to +31.49, and all CIs exclude 0. On the same backbone, P(all four | original correct) rises from 38.3% for Vanilla to 67.0% for SyncRA. The AllFour gain is therefore not a side effect of higher original-version accuracy. Among cues that both models answer correctly on the original input, SyncRA follows the changed pairings far more often. This analysis is in Appendix C.6 (Table 22).
Response to Reviewer p51Z (1 of 4)
Thank you for the detailed review and for pointing to the closest prior objectives. We ran the comparison you asked for: we ported the objectives of Korbar et al. [1], CAV-MAE Sync [2], and AVAF-Net [3] to Qwen3-Omni with the same data, budget, and checkpoint selection as SyncRA. We also split the benchmark gain into a generic part and a pairing-specific part, extended the retrieval analysis to all control objectives and to the final layer, and tested larger answer-only budgets.
All new training runs are on Qwen3-Omni with a single seed. They follow the Run 1 protocol of Table 4, with the same 10K corpus and 2-epoch budget unless stated. All intervals are 95% CIs, from a paired per-video bootstrap (10,000 resamples) for diagnostic and retrieval numbers and from Welch intervals over three runs for Avg-5. Retrieval numbers come from a refreshed analysis pipeline and differ from Tables 25 and 28 by at most 0.1 points.
Response to weaknesses
1. In the same setting, SyncRA exceeds the strongest prior objective by 10.00 AllFour points.
"What remains is placing the loss at an intermediate layer on the processor's native grid, an implementation choice, and none of these methods is compared on the same data and backbones." (W1, Q1)
On the same data and backbone, SyncRA outperforms the strongest ported objective by 10.00 AllFour points (CI excludes 0) and the other two by 28.00–29.25 points. Korbar et al. and AVAF-Net are statistically indistinguishable from answer-only fine-tuning. The review describes the remaining difference as a choice of placement. This comparison holds placement fixed, because all four objectives act on the same block (36). AllFour still ranges from 33.25 to 62.50, so at a fixed placement the form of the objective decides whether pairing-following improves.
We re-implemented each auxiliary loss from its paper and ported it to Qwen3-Omni. Only the auxiliary loss changes. The data, answer loss, trainable parameters, supervision block (36), budget (2 epochs), NLL selection, and λ = 0.1 are identical to SyncRA. The projection sizes and the CAV-MAE Sync temperature are not specified in the original papers, and all other choices follow the source. The full porting settings are in Appendix B.9. All three losses converge (Korbar from 1.05 to 0.25, CAV-MAE Sync from 4.42 to 0.12, AVAF-Net from 2.01 to 0.09), so under-training does not explain the outcomes.
Here P(all|O) is AllFour divided by original-version accuracy, and R@1 is unprojected same-interval retrieval at block 36. Daily-Omni is shown as an aggregate reference.
The ordering is consistent with three factors, each tested here or in the paper's ablations. The first is granularity: clip-level pooling gives R@1 13.9, against 96.9 at interval level. The second is the loss form: cosine-only gives 17.9 and pairwise ranking 27.5, against 96.9 for InfoNCE (Table 26). The third is the source of negatives: CAV-MAE Sync with cross-video negatives gives 53.6, against 96.9 for SyncRA with within-video negatives.
Korbar's result is consistent with its negatives. Between 75% and 100% of them come from other videos, so the margin loss is mostly satisfied by telling videos apart, and within-video retrieval stays at the Vanilla level (12.1 vs 13.4). The ported CAV-MAE Sync differs from SyncRA in two ways: its negatives come from other videos, and its loss runs in one direction. It lands in the middle on both R@1 (53.6) and AllFour (52.50). For AVAF-Net we report only the measured outcome: the loss converged to 0.09 while within-video R@1 stayed at 11.8. We are still examining why.
On novelty, we agree that the objective family is classic, and the paper places it in the synchrony line (§2.2, §7). The revision states the contribution in the terms that this comparison supports (§1): we identify the conditions under which synchrony supervision repairs pairing-following in omni-modal QA. These are same-video negatives, interval granularity, and the InfoNCE form. For depth, supervision at either block 24 or block 36 improves AllFour over Vanilla (Table 34), and we did not test other depths. The other two contributions are the swap diagnostic and the finding that answer-only supervision does not fix pairing. The comparison is now in §6.1 (Table 4) and Appendix B.9 (Table 15).