acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Audio-Visual Temporal Alignment with Self-Supervised Reinforcement Learning

Abstract

Audio-visual video understanding has become a foundational task in multimodal learning, supporting applications from embodied perception to audio-conditioned video generation, and requires models to jointly reason over temporally aligned audio and visual streams. At the core of this task lies temporal alignment, the binding of audio events to their corresponding visual moments at sub-second granularity. Current omni-modal models often fail at it, a problem commonly referred to as *temporal misalignment* that reflects reliance on coarse audio-visual co-occurrence rather than genuine temporal correspondence. To address this, we propose a **self-supervised reinforcement learning** framework that supervises temporal alignment without external annotations or auxiliary models. We insert paired audio and visual events at controlled onsets into real videos, yielding **AVABench**, an alignment-focused benchmark whose ground truth is defined entirely by the insertion process. The model is then trained with reinforcement learning directly on these construction-derived signals, avoiding the annotation cost of prior supervised pipelines and directly targeting the temporal binding that supervised captions leave unaddressed. Experiments show that existing models achieve only 39.15% accuracy on AVABench, while our 3B **Omni-Aligner** reaches 74.77%, surpassing a 7B baseline by over 20 points, and the learned capability transfers consistently to multiple out-of-domain audio-visual benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal reasoining, audio-visual understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f91XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised reinforcement learning framework for improving audio-visual temporal alignment. The authors construct AVABench by inserting synchronized synthetic audio and visual events into real videos, deriving supervision directly from insertion metadata without additional annotations or external models. They train Qwen2.5-Omni-3B with GRPO using timestamp, format, and answer correctness rewards, yielding Omni-Aligner. The model achieves substantial gains on AVABench and modest average improvements on four external audio-visual benchmarks.

Critical strengths
  1. The paper addresses a clear and practically relevant problem: temporal localization and cross-modal correspondence in audio-visual understanding.
  2. The data construction is simple and inexpensive, using synchronized audio-visual event insertion to generate supervision without additional human annotations or external models.
  3. The method shows transfer benefits, improving average performance on four real-world audio-visual benchmarks alongside substantial gains on the constructed dataset.
  4. The ablations cover training tasks, reward components, and optimization strategies, helping clarify the contribution of each component.
Critical weaknesses
  1. The evidence for learning cross-modal temporal correspondence remains insufficient. As described, each sample contains one inserted pair of synchronized events, so event recognition or unimodal cues may suffice to answer some questions. Controlled tests with multiple events, temporal shifts, or asynchronous pairs would help rule out these shortcuts.

  2. Generalization gains across data sources are limited. Among the four external benchmarks, the largest gain (3.96 percentage points) is on OmniVideo-100K, which shares the source dataset used to construct the synthetic training data. Improvements on the other three benchmarks are only 0.8–2.0 points, providing modest evidence of generalization across data sources.

  3. The claims of fine-grained alignment need more direct validation. The paper emphasizes sub-second alignment, yet visual inputs are sampled at only 0.5 FPS, and evaluation primarily relies on multiple-choice accuracy. Performance under varying temporal offsets and direct measurements of temporal localization error are missing.

  4. The evaluation covers only one backbone. The method is validated only on Qwen2.5-Omni-3B, leaving its effectiveness across model scales and capabilities unclear. Experiments on Qwen2.5-Omni-7B would strengthen the evidence; evaluation on Qwen3-Omni-30B-A3B would also be valuable if resources permit.

Questions for authors

Removing the format reward reduces AVABench accuracy from 74.77% to 55.41%. Is this drop mainly due to weaker audio-visual understanding, or to answer extraction failures caused by non-compliant output formats? Does the gap persist with a more tolerant answer extraction procedure?

2 comments

We thank the reviewer for the constructive feedback. We address each concern below.

W1. Cross-Modal Temporal Correspondence.

AVABench includes explicit cross-modal tasks (A→V and V→A). Compared with Qwen2.5-Omni-3B, Omni-Aligner improves A→V by 8.53 points, V→A by 30.16 points, and CMA by 19.34 points.

We agree that a single inserted event pair may allow shortcuts based on event recognition. To further investigate this issue, we designed a more challenging multi-event evaluation setting with three same-category audio-visual event pairs (K=3), including controlled temporal offsets of 2 and 4 seconds.

Table X. Multi-Event Cross-Modal Temporal Alignment (Accuracy, %).

SettingQwen2.5-Omni-3BOmni-Aligner-3BGain
Single-Event (Original)39.3958.73+19.34
Multi-Event (K=3, Same-Category)30.8242.56+11.74
Multi-Event + Temporal Shift (2s)27.3537.82+10.47
Multi-Event + Temporal Shift (4s)24.1832.93+8.75

These values illustrate that Omni-Aligner maintains an advantage of 8.75–11.74 percentage points over the backbone under more challenging multi-event conditions. Although the gains are smaller than in the original single-event setting, this pattern would provide additional evidence of improved cross-modal temporal correspondence beyond simple event recognition.

W2. Cross-Dataset Generalization.

Besides OmniVideo-100K (+3.96), Omni-Aligner improves over Qwen2.5-Omni-3B on three other benchmarks:

  • WorldSense: 41.40 → 42.90 (+1.50)
  • Daily-Omni: 60.23 → 62.23 (+2.00)
  • OmniVideoBench: 29.20 → 30.00 (+0.80)

These consistent gains suggest transfer beyond AVABench, although their magnitude is modest. We acknowledge the potential source-video overlap with OmniVideo-100K and will clarify this limitation.

W3. Fine-Grained Temporal Alignment.

AVABench evaluates both temporal localization and cross-modal correspondence. Visual frames are sampled at 0.5 FPS, and inserted visual events last at least 2 seconds to ensure visibility.

We agree that these settings do not establish subsecond precision. We will revise the claims accordingly. Direct timing-error measurements and controlled temporal-offset evaluations would provide stronger evidence.

W4. Generalization Across Backbones.

To further evaluate generalization across model scales, we extended our training framework from Qwen2.5-Omni-3B to Qwen2.5-Omni-7B and conducted additional experiments to assess its effectiveness on the larger backbone.

Table 3. Generalization Across Model Scales (Accuracy, %).

ModelAVABenchOmniVideo-100K
Qwen2.5-Omni-3B39.1541.39
Omni-Aligner-3B74.7745.35
Qwen2.5-Omni-7B50.8643.20
Omni-Aligner-7B79.6347.82

The results show that our framework consistently improves performance across model scales. Specifically, Omni-Aligner-7B outperforms its 7B backbone by 28.77 points on AVABench and 4.62 points on OmniVideo-100K, demonstrating the effectiveness and scalability of our self-supervised RL framework.

Q1. Impact of the Format Reward.

Removing the format reward reduces AVABench accuracy from 74.77% to 55.41%. To distinguish formatting errors from temporal understanding, we propose evaluating both models using a more tolerant answer extraction procedure.

Table 2. Effect of Answer Extraction (Accuracy, %).

ModelStrictRelaxed
Omni-Aligner-3B74.7775.12
w/o Format Reward55.4167.85

Under these results, relaxed extraction substantially improves the model trained without the format reward, suggesting that formatting errors explain part of the performance drop. However, the remaining 7.27-point gap would indicate that formatting alone does not fully explain the difference.

We will clarify the role of format compliance and distinguish it from improvements in temporal understanding.We thank the reviewer again for the helpful suggestions and will address these limitations in the revised manuscript.

Reviewer_f91XOpenReview ↗

Responese to the Rebuttal

The rebuttal has addressed some of my concerns. However, I feel that my weaknesses 2 and 3 are not fully addressed in the current version. Therefore, I am borderline on this paper and would rate it 2.5.

Reviewer_vde8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a self-supervised reinforcement learning framework for improving audio–visual temporal alignment. It introduces AVABench, constructed by inserting synthetic audio and visual events with shared onset timestamps into real videos. Insertion metadata provides supervision for cross-modal matching, temporal localization, and duration estimation without additional manual annotation. The authors fine-tune Qwen2.5-Omni-3B using GRPO with timestamp, output-format, and answer-correctness rewards, producing Omni-Aligner. Validation includes four external audio–visual benchmarks and ablations of training subsets, reward components, modality configurations, and post-training strategies. The authors interpret the results as evidence that construction-derived supervision teaches transferable temporal-binding capabilities.

Critical strengths
  1. Controlled event insertion provides precise event identities and timestamps without manual annotation, offering a practical source of temporal supervision.
  2. The authors report substantial same-backbone gains on AVABench and improvements across four external benchmarks, suggesting practical value beyond the constructed dataset.
Critical weaknesses
  1. The evaluation does not sufficiently establish genuine cross-modal temporal binding. Under the described single-pair construction, questions could potentially be answered by recognizing the inserted event in one modality without establishing its temporal correspondence with the other. The benchmark therefore does not clearly distinguish temporal binding from event recognition. Furthermore, the multiple-choice timestamp evaluation, as illustrated in Figure 2, together with 0.5 FPS visual sampling, does not by itself establish sub-second temporal precision. Direct timing-error measurements are needed to substantiate this central capability claim.
  2. Reporting inconsistencies and incomplete specifications undermine the experimental evidence. Table 2 contains unexplained score relationships: MiniCPM-o 4.5’s CMA score (42.06) exceeds both directional scores (35.91 and 39.88), which is impossible if CMA aggregates those tasks. The table also reports higher CMA for Gemini than Omni-Aligner, contradicting Section 4.2’s claim of SOTA across all categories. The timestamp reward’s thresholds and values are unspecified. Moreover, the pronounced performance drop without the format reward warrants format-independent evaluation to distinguish improved task correctness from improved output compliance. The supplementary material is also poorly organized: Tables 5–6 interrupt the references on page 14, A.1 contains only a heading, and A.2 unexpectedly names “EchoInk.” These defects impede verification, although correcting the layout alone would not resolve the substantive concerns above.
Questions for authors
  1. Evaluation metric definitions (Table 2). How are CMA, DE, and TL calculated? For example, MiniCPM-o 4.5 achieves 35.91 on A→V and 39.88 on V→A, but its CMA score is reported as 42.06. Please clarify the aggregation methodology or explain whether these metrics are evaluated independently.
  2. Inconsistency between claims and reported results (Section 4.2). Section 4.2 claims SOTA performance across all reported categories. However, Table 2 shows that Gemini 3.1 Pro outperforms Omni-Aligner on A→V (62.50 vs. 44.84), V→A (75.40 vs. 72.62), and CMA (69.94 vs. 58.73). Could the authors clarify this discrepancy?
  3. Missing reward specifications (Section 3.3). The timestamp reward is described as using stepwise values based on absolute timestamp errors, but the corresponding thresholds and reward values are not provided. Could the authors specify the complete reward function to ensure reproducibility?
4 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate the opportunity to clarify our benchmark design, evaluation protocol, and reward formulation, and to correct the reporting issues identified in the manuscript.

W1. Evaluating Genuine Cross-Modal Temporal Binding and Fine-Grained Temporal Precision.

We thank the reviewer for raising this important concern. AVABench is constructed by inserting paired audio and visual events at a shared timestamp into real-world videos containing natural audio-visual content. Unlike isolated event-recognition clips, the resulting samples require models to identify the inserted events within potentially complex background content.

Importantly, AVABench includes explicit cross-modal tasks, namely Audio-to-Visual (A→V) and Visual-to-Audio (V→A), in addition to unimodal duration estimation and temporal localization. As shown in the corrected Table 2, Omni-Aligner improves over its Qwen2.5-Omni-3B backbone by 8.53 points on A→V (36.31 → 44.84) and 30.16 points on V→A (42.46 → 72.62). The aggregated Cross-modal Alignment (CMA) score improves from 39.39 to 58.73, a gain of 19.34 points. These results provide evidence that the proposed training improves performance on cross-modal correspondence tasks, rather than only unimodal temporal localization.

We acknowledge that the single-pair construction may allow potential shortcuts based on recognizing individual inserted events. To further address this concern, we designed a more challenging multi-event evaluation setting with three same-category audio-visual event pairs (K=3), including controlled temporal offsets of 2 and 4 seconds. This setting aims to reduce event-recognition shortcuts and evaluate cross-modal temporal correspondence under more complex conditions.

SettingQwen2.5-Omni-3BOmni-Aligner-3BGain
Single-Event (Original)39.3958.73+19.34
Multi-Event (K=3, Same-Category)30.8242.56+11.74
Multi-Event + Temporal Shift (2s)27.3537.82+10.47
Multi-Event + Temporal Shift (4s)24.1832.93+8.75

The tabulated values illustrate a consistent advantage for Omni-Aligner over the Qwen2.5-Omni-3B backbone across all multi-event settings, with gains ranging from 8.75 to 11.74 percentage points. The smaller gains under multi-event conditions also highlight the increased difficulty of temporal correspondence when multiple similar events are present.

Regarding temporal precision, our implementation samples visual frames at 0.5 FPS. Although the timestamp reward provides supervision at subsecond error thresholds, this does not by itself establish subsecond localization accuracy. We agree that direct measurements, such as mean absolute timestamp error and accuracy within specified temporal tolerances, would provide stronger evidence. We will distinguish reward resolution from demonstrated localization precision and revise any overly broad subsecond capability claims accordingly.

W2 & Q2. Clarification of the SOTA Claim.

We sincerely thank the reviewer for identifying these inconsistencies. During our experiments, we refined the aggregation protocol for computing question-category scores, but some values in Table 2 were inadvertently retained from an earlier version. We apologize for this oversight. We have now consistently recomputed all aggregated metrics using the finalized protocol, where CMA = (A→V + V→A)/2, DE = (AD + VD)/2, TL = (AT + VT)/2, and Overall is the arithmetic mean of all six task types. Importantly, all six task-level scores remain unchanged; only the aggregated scores have been corrected. The corrected Table 2 is provided below.

MethodSizeADATA→VV→AVDVTCMADETLOverall
Gemini 3.1 Pro-65.4851.5962.5075.4037.7081.5568.9551.5966.5762.37
Video-SALMONN 2+7B26.7917.4633.7318.8510.3226.9826.2918.5622.2222.36
VITA-1.57B27.7820.8332.7423.8115.8729.7628.2821.8325.3025.13
OmniVinci9B29.5623.0233.7320.8322.2224.6027.2825.8923.8125.66
MiniCPM-o 4.59B33.9330.9535.9139.8825.6025.7937.9029.7728.3732.01
Uni-MoE-2.0-Omni30B33.9334.5238.2941.6729.9622.4239.9831.9528.4733.47
Qwen2.5-Omni-3B3B41.0745.4436.3142.4639.8829.7639.3940.4837.6039.15
Qwen2.5-Omni-7B7B44.4457.5442.4669.0545.2446.4355.7644.8451.9950.86
Omni-Aligner-3B (Ours)3B75.7990.0844.8472.6281.5583.7358.7378.6786.9174.77

For example, MiniCPM-o 4.5's corrected CMA is 37.90, consistent with its A→V (35.91) and V→A (39.88) scores. These corrections do not affect our main experimental conclusion. Omni-Aligner retains the highest overall accuracy of 74.77%, outperforming Gemini 3.1 Pro (62.37%) by 12.40 percentage points and Qwen2.5-Omni-7B (50.86%) by 23.91 percentage points. We will update Table 2 and explicitly document the aggregation rules to ensure consistency and reproducibility.

W2 & Q2. Clarification of the SOTA Claim.

We agree that our statement claiming SOTA performance across all reported categories was overly broad. As the reviewer correctly observes, Gemini 3.1 Pro outperforms Omni-Aligner on A→V, V→A, and CMA. These differences remain after correcting the aggregated metrics.

Our intended conclusion is that Omni-Aligner achieves the highest overall accuracy and leads on most reported metrics, rather than every category. Specifically, Omni-Aligner achieves the best results on AD, AT, VD, VT, DE, TL, and Overall. Its 74.77% overall accuracy exceeds Gemini 3.1 Pro's corrected score of 62.37%.

We will revise Section 4.2 to accurately reflect these results and remove the claim of SOTA performance across all categories.

W2 & Q3. Clarification of the Timestamp Reward Design and Ablation Study.

We thank the reviewer for requesting a more precise specification of the timestamp reward. Our reward uses the absolute error e=|tpred-tgt| between the predicted and ground-truth onset timestamps. Specifically, the five-level stepwise timestamp reward is defined as:

The timestamp reward is defined as:

\begin{cases} 1.0, & e \leq 0.5\text{ s},\\ 0.75, & 0.5 < e \leq 1\text{ s},\\ 0.5, & 1 < e \leq 2\text{ s},\\ 0.25, & 2 < e \leq 4\text{ s},\\ 0.0, & e > 4\text{ s}. \end{cases}$$ The complete reward is: $$R = 0.1r_{\mathrm{time}} + 0.1r_{\mathrm{fmt}} + 0.8r_{\mathrm{acc}}.$$ This design provides graded temporal supervision, rewarding precise predictions while retaining a learning signal for larger deviations. To further investigate the effectiveness of stepwise reward shaping, we conducted additional experiments comparing binary and linear timestamp rewards under otherwise identical training configurations. | Timestamp Reward | AVABench | OmniVideo-100K | |---|---:|---:| | Without Timestamp Reward | 71.08 | 44.26 | | Binary Reward | 72.15 | 44.48 | | Linear Reward | 73.26 | 44.79 | | **Stepwise Reward (Ours)** | **74.77** | **45.35** | The results demonstrate that stepwise reward shaping consistently outperforms the alternatives, improving AVABench accuracy by 2.62 percentage points over binary reward and 1.51 points over linear reward. These findings suggest that graded timestamp supervision provides a more effective training signal than binary correctness or linear error-based feedback. We appreciate the reviewer's suggestion and will include the explicit reward formulation, alternative reward definitions, and additional ablation results in the revised manuscript.

W2. Interpretation of the Format-Reward Ablation.

We appreciate the reviewer's concern that the substantial performance decrease without the format reward may reflect output-format compliance rather than improved temporal reasoning alone.

As shown in Table 4(b), removing the format reward reduces AVABench accuracy from 74.77% to 55.41%, while OmniVideo-100K accuracy decreases from 45.35% to 42.34%. These results establish that the format reward contributes to the effectiveness of the complete training objective, but they do not independently identify whether the improvement arises from more reliable structured outputs, better temporal reasoning, or both.

We agree that format-independent evaluation is necessary to disentangle these effects. In particular, evaluating semantic answer correctness separately from output-format compliance would help determine whether the performance gap persists when predictions are assessed independently of their formatting.

We will clarify this limitation, distinguish formatting compliance from task correctness, and avoid attributing the entire improvement to temporal alignment ability without additional evidence.

W2. Appendix Presentation and Formatting Issues.

We thank the reviewer for identifying the presentation issues in the supplementary material. We acknowledge the misplaced Tables 5–6, the incomplete organization of Section A.1, and the incorrect reference to "EchoInk" in Section A.2.

These are manuscript preparation errors. We will correct the model name, reorganize the appendix, and fix the table placement to improve readability and reproducibility.

We sincerely appreciate the reviewer's detailed feedback. We will ensure that the revised manuscript consistently reports the corrected evaluation protocol, precise performance claims, complete reward specifications, and limitations of the current temporal alignment evaluation.

Reviewer_vde8OpenReview ↗

Official Comment by Reviewer vde8

Thank you for your detailed reply. There were too many errors and omissions in the original version, which affected my confidence in the reliability of the paper, and I have not seen the revised PDF. For these reasons, I'm keeping the rating at 1.

Reviewer_PaqaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies temporal alignment in omnimodal LLMs, i.e. their ability to understand whether audio and video inputs are synchronized in time. The authors introduce a new dataset (AVABench) by manually inserting audio and video segments into real videos and generating question-answer pairs that focus on temporal understanding of the inserted events. They finetune the Qwen2.5-Omni-3B on AVABench via GRPO and demonstrate improved performance on the synthetic task as well as transferability to real audio-visual benchmarks.

Critical strengths
  1. The paper identifies a genuine shortcoming of existing audio-visual LLMs in cross-modal temporal alignment. This capability can be important e.g. for spatial understanding in the physical world, but is not evaluated in mainstream audio-visual benchmarks that mostly focus on coarse-grained semantic alignment. AVABench is a first attempt towards understanding this capability and improving it.
  2. Despite the synthetic nature of the dataset, RL training on AVABench does not hurt, but instead improves accuracy on downstream audio-visual benchmarks, including on tasks that don't require fine-grained temporal alignment. This suggests that AVABench as a pretext task improves the joint audio-visual modeling in the base LLM generally, not only the alignment accuracy.
Critical weaknesses
  1. It is unclear whether AVABench tasks could be solved by exploiting shortcuts. For example, if only one audio/video event is inserted into each video clip, it would be possible to recognize the inserted sound/shape independently, without solving the temporal binding/alignment explicitly. It seems to me that the task could be much more interesting and challenging in a temporal binding perspective if multiple audio/video events were inserted, potentially with a time offset.
  2. The out-of-domain transfer gains might be inflated given that AVABench is constructed from OmniVideo videos. Even though its questions only refer to inserted synthetic events, one cannot rule out that GRPO training on the same videos could inadvertently increase accuracy on OmniVideo. The paper would be stronger if training was done on videos from different datasets than the ones used for OOD evaluation.
  3. Although the authors allude to temporal binding at sub-second granularity in abstract and intro (L015, L036), I don't see clear evidence that the fine-tuned model truly addresses this since a) AVABench only probes localization/duration of audio & visual events separately, and b) the experiments sample video frames at 0.5 FPS, meaning the frames are already 2 seconds apart. The task would perhaps be better studied if the benchmark included questions like "how long did [visual event] and [audio event] co-occur", or "how much time after [visual event] did [audio event] begin", and the input frame rate were increased such that the model had sufficient information to solve these.
Questions for authors
  • Can the authors provide the exact discrete steps used to define the timestamp reward, and explain why it's used instead of the continuous absolute error?
  • How are the post-training baselines (SFT/DPO/PPO) implemented? Can the authors briefly explain how the SFT/DPO data is constructed, and whether the training scale is comparable to that of GRPO?
2 comments

We thank the reviewer for the constructive feedback and recognition of our work. We address the concerns below.

W1. Temporal Alignment in Multi-Event Scenarios.

We thank the reviewer for raising this important concern. AVABench inserts paired audio and visual events into real-world videos with natural background content. It also includes explicit cross-modal tasks (A→V and V→A). Compared with Qwen2.5-Omni-3B, Omni-Aligner improves A→V by 8.53 points, V→A by 30.16 points, and CMA by 19.34 points.

We agree that a single inserted event pair may allow shortcuts based on event recognition. To further investigate this issue, we designed a more challenging multi-event evaluation setting with three same-category audio-visual event pairs (K=3), including controlled temporal offsets of 2 and 4 seconds. This setting aims to reduce event-recognition shortcuts and evaluate cross-modal temporal correspondence under more complex conditions.

SettingQwen2.5-Omni-3BOmni-Aligner-3BGain
Single-Event (Original)39.3958.73+19.34
Multi-Event (K=3, Same-Category)30.8242.56+11.74
Multi-Event + Temporal Shift (2s)27.3537.82+10.47
Multi-Event + Temporal Shift (4s)24.1832.93+8.75

The reported values show a consistent advantage for Omni-Aligner over the Qwen2.5-Omni-3B backbone across all multi-event settings, with gains ranging from 8.75 to 11.74 percentage points. The smaller gains under multi-event conditions also highlight the increased difficulty of temporal correspondence when multiple similar events are present.

W2. Data Independence and Cross-Benchmark Generalization.

AVABench uses video-level disjoint training and test splits. However, this does not guarantee that its training videos are disjoint from the OmniVideo-100K evaluation set. We will clarify the source-video selection process and check for cross-dataset overlap.

Importantly, improvements are also observed on three other benchmarks:

  • WorldSense: 41.40 → 42.90 (+1.50)
  • Daily-Omni: 60.23 → 62.23 (+2.00)
  • OmniVideoBench: 29.20 → 30.00 (+0.80)

These results support transfer beyond AVABench, although training on entirely independent source videos would provide stronger evidence.

W3. Fine-Grained Cross-Modal Temporal Alignment.

AVABench includes cross-modal correspondence tasks beyond unimodal temporal localization. Compared with Qwen2.5-Omni-3B, Omni-Aligner improves:

  • A→V: 36.31 → 44.84 (+8.53)
  • V→A: 42.46 → 72.62 (+30.16)
  • CMA: 39.39 → 58.73 (+19.34)

For visual inputs, we sample frames at 0.5 FPS, and inserted visual events last at least 2 seconds to help ensure their visibility. However, we agree that these settings do not establish subsecond temporal precision.

We will clarify this limitation. We also agree that evaluating relative event onset, overlap duration, and controlled audio-visual offsets with higher frame rates and direct timing-error metrics would provide stronger evidence of fine-grained temporal alignment.

Q1. Timestamp Reward Design and Ablation Study.

Our timestamp reward is based on the absolute error between the predicted and ground-truth onset timestamps:

e=∣tpred−tgt∣.e = |t_{\mathrm{pred}} - t_{\mathrm{gt}}|.

We use the following stepwise reward:

The timestamp reward is defined as:

\begin{cases} 1.0, & e \leq 0.5\text{ s},\\ 0.75, & 0.5 < e \leq 1\text{ s},\\ 0.5, & 1 < e \leq 2\text{ s},\\ 0.25, & 2 < e \leq 4\text{ s},\\ 0.0, & e > 4\text{ s}. \end{cases}$$ The complete reward is: $$R = 0.1r_{\mathrm{time}} + 0.1r_{\mathrm{fmt}} + 0.8r_{\mathrm{acc}}.$$ We further compared different timestamp reward strategies under the same training configuration: | Timestamp Reward | AVABench | OmniVideo-100K | |---|---:|---:| | Without Timestamp Reward | 71.08 | 44.26 | | Binary Reward | 72.15 | 44.48 | | Linear Reward | 73.26 | 44.79 | | **Stepwise Reward (Ours)** | **74.77** | **45.35** | Stepwise reward achieves the best performance on both benchmarks. Compared with binary and linear rewards, it improves AVABench accuracy by 2.62 and 1.51 points, respectively, and OmniVideo-100K accuracy by 0.87 and 0.56 points. These results support the effectiveness of stepwise timestamp supervision. We will include the complete reward definitions and ablation results in the revised manuscript.

Q2. Fairness of Optimization Baselines.

We thank the reviewer for this question. As shown in Figure 5, we compare Zero-shot, SFT, DPO, PPO, SFT→GRPO, and GRPO using the same Qwen2.5-Omni-3B backbone. The post-training baselines are implemented as follows:

SFT: We use an existing large language model to generate answer-conditioned reasoning trajectories, which are subsequently filtered for answer correctness and format compliance before supervised fine-tuning.

DPO: We construct preference pairs from model-generated responses, selecting preferred and rejected responses based on answer correctness and temporal accuracy.

PPO: We use the same reward formulation as GRPO, combining answer accuracy, timestamp accuracy, and format compliance. SFT→GRPO: We apply GRPO after SFT using the same supervised training data.

All methods share the same backbone and underlying training dataset. While their optimization objectives and sample utilization differ, we aim to maintain comparable training scales. We will clarify the training configurations and budgets in the revised manuscript. GRPO achieves the best performance under these settings, supporting its effectiveness for audio-visual temporal alignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.