- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a self-supervised reinforcement learning framework for improving audio-visual temporal alignment. The authors construct AVABench by inserting synchronized synthetic audio and visual events into real videos, deriving supervision directly from insertion metadata without additional annotations or external models. They train Qwen2.5-Omni-3B with GRPO using timestamp, format, and answer correctness rewards, yielding Omni-Aligner. The model achieves substantial gains on AVABench and modest average improvements on four external audio-visual benchmarks.
Critical strengths
- The paper addresses a clear and practically relevant problem: temporal localization and cross-modal correspondence in audio-visual understanding.
- The data construction is simple and inexpensive, using synchronized audio-visual event insertion to generate supervision without additional human annotations or external models.
- The method shows transfer benefits, improving average performance on four real-world audio-visual benchmarks alongside substantial gains on the constructed dataset.
- The ablations cover training tasks, reward components, and optimization strategies, helping clarify the contribution of each component.
Critical weaknesses
-
The evidence for learning cross-modal temporal correspondence remains insufficient. As described, each sample contains one inserted pair of synchronized events, so event recognition or unimodal cues may suffice to answer some questions. Controlled tests with multiple events, temporal shifts, or asynchronous pairs would help rule out these shortcuts.
-
Generalization gains across data sources are limited. Among the four external benchmarks, the largest gain (3.96 percentage points) is on OmniVideo-100K, which shares the source dataset used to construct the synthetic training data. Improvements on the other three benchmarks are only 0.8–2.0 points, providing modest evidence of generalization across data sources.
-
The claims of fine-grained alignment need more direct validation. The paper emphasizes sub-second alignment, yet visual inputs are sampled at only 0.5 FPS, and evaluation primarily relies on multiple-choice accuracy. Performance under varying temporal offsets and direct measurements of temporal localization error are missing.
-
The evaluation covers only one backbone. The method is validated only on Qwen2.5-Omni-3B, leaving its effectiveness across model scales and capabilities unclear. Experiments on Qwen2.5-Omni-7B would strengthen the evidence; evaluation on Qwen3-Omni-30B-A3B would also be valuable if resources permit.
Questions for authors
Removing the format reward reduces AVABench accuracy from 74.77% to 55.41%. Is this drop mainly due to weaker audio-visual understanding, or to answer extraction failures caused by non-compliant output formats? Does the gap persist with a more tolerant answer extraction procedure?
2 comments
Responese to the Rebuttal
The rebuttal has addressed some of my concerns. However, I feel that my weaknesses 2 and 3 are not fully addressed in the current version. Therefore, I am borderline on this paper and would rate it 2.5.
We thank the reviewer for the constructive feedback. We address each concern below.
W1. Cross-Modal Temporal Correspondence.
AVABench includes explicit cross-modal tasks (A→V and V→A). Compared with Qwen2.5-Omni-3B, Omni-Aligner improves A→V by 8.53 points, V→A by 30.16 points, and CMA by 19.34 points.
We agree that a single inserted event pair may allow shortcuts based on event recognition. To further investigate this issue, we designed a more challenging multi-event evaluation setting with three same-category audio-visual event pairs (K=3), including controlled temporal offsets of 2 and 4 seconds.
Table X. Multi-Event Cross-Modal Temporal Alignment (Accuracy, %).
These values illustrate that Omni-Aligner maintains an advantage of 8.75–11.74 percentage points over the backbone under more challenging multi-event conditions. Although the gains are smaller than in the original single-event setting, this pattern would provide additional evidence of improved cross-modal temporal correspondence beyond simple event recognition.
W2. Cross-Dataset Generalization.
Besides OmniVideo-100K (+3.96), Omni-Aligner improves over Qwen2.5-Omni-3B on three other benchmarks:
These consistent gains suggest transfer beyond AVABench, although their magnitude is modest. We acknowledge the potential source-video overlap with OmniVideo-100K and will clarify this limitation.
W3. Fine-Grained Temporal Alignment.
AVABench evaluates both temporal localization and cross-modal correspondence. Visual frames are sampled at 0.5 FPS, and inserted visual events last at least 2 seconds to ensure visibility.
We agree that these settings do not establish subsecond precision. We will revise the claims accordingly. Direct timing-error measurements and controlled temporal-offset evaluations would provide stronger evidence.
W4. Generalization Across Backbones.
To further evaluate generalization across model scales, we extended our training framework from Qwen2.5-Omni-3B to Qwen2.5-Omni-7B and conducted additional experiments to assess its effectiveness on the larger backbone.
Table 3. Generalization Across Model Scales (Accuracy, %).
The results show that our framework consistently improves performance across model scales. Specifically, Omni-Aligner-7B outperforms its 7B backbone by 28.77 points on AVABench and 4.62 points on OmniVideo-100K, demonstrating the effectiveness and scalability of our self-supervised RL framework.
Q1. Impact of the Format Reward.
Removing the format reward reduces AVABench accuracy from 74.77% to 55.41%. To distinguish formatting errors from temporal understanding, we propose evaluating both models using a more tolerant answer extraction procedure.
Table 2. Effect of Answer Extraction (Accuracy, %).
Under these results, relaxed extraction substantially improves the model trained without the format reward, suggesting that formatting errors explain part of the performance drop. However, the remaining 7.27-point gap would indicate that formatting alone does not fully explain the difference.
We will clarify the role of format compliance and distinguish it from improvements in temporal understanding.We thank the reviewer again for the helpful suggestions and will address these limitations in the revised manuscript.