- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies dense temporal audio-visual captioning and introduces AVTime-50K, a large-scale dataset containing approximately 50K videos and 1.87M timestamped audio-visual events, AVTime with a Bidirectional Temporal-Semantic Alignment (BiTSA) reward, and AVTime-Bench for evaluating time-to-event, event-to-time, and event-to-event alignment. BiTSA combines semantic agreement at queried timestamps with temporal IoU of semantically matched events. Experiments on LongVALE and AVTime-Bench demonstrate consistent improvements over existing temporal and audio-visual baselines at both 7B and 30B scales.
Critical strengths
-
Comprehensive contribution. The paper presents a complete framework covering dataset construction, reward design, model training, and benchmark development for dense audio-visual temporal captioning.
-
Strong empirical performance. AVTime consistently outperforms the evaluated baselines on both LongVALE and AVTime-Bench, with gains observed at both 7B and 30B scales.
-
Useful ablation studies. The ablations show that AVTime-50K supervision and the two BiTSA reward components each contribute to the final performance, and that combining T2E and E2T gives the strongest results.
Critical weaknesses
-
Limited validation of annotation quality. Although the authors manually inspect 100 retained annotations, this is relatively small compared with the whole dataset. Since Qwen3-Omni-Thinking, which may not be sufficiently strong for fine-grained audio-visual temporal annotation, is used for both caption generation and filtering, a larger-scale human audit would better support the annotation quality claim.
-
The paper does not report results on OmniDCBench [1], which is a relevant benchmark for omni-modal dense captioning. Evaluating AVTime on OmniDCBench would provide useful evidence of its generalization beyond the proposed benchmark and LongVALE.
-
Potential model-family bias. The semantic evaluator and the AVTime backbone are both Qwen-based, which may introduce model-family bias in reward computation.
-
Limited benchmark scale. AVTime-Bench contains only 300 videos, which may be insufficient to ensure a stable and representative evaluation.
-
Potential distribution advantage. Although AVTime-Bench is held out, it shares the same data collection and annotation pipeline as AVTime-50K, which may favor AVTime over models trained on different distributions.
-
Judge reliability. AVTime-Bench relies on Qwen3.6-27B for semantic comparison and event matching. Although the paper reports correlation with human judgments, evaluation with alternative judges or broader human verification would further establish the robustness of the benchmark.
[1] TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions, ICML 2026
Questions for authors
Please see weaknesses.
7 comments
Comment 2 of 2
W4. Benchmark scale
AVTime-Bench contains only 300 videos, which may be insufficient to ensure a stable and representative evaluation.
The 300 videos span 18.2 hours and 10,344 reference events, giving 49,769 scored queries.
- Stability. With a video-level bootstrap (2,000 resamples), AVTime-30B-RL's Mean is 34.36 [32.78, 36.01]. Its margin over the strongest baseline is at least +5.3 [3.6, 7.2] on every metric, and it ranks first in every resample.
- Subsampling. With only 100 random videos, the system ranking stays close to the full one (Kendall's τ = 0.96 over 1,000 draws), and AVTime is first in every draw.
We will add the intervals to Table 2.
W5. Potential distribution advantage
Although AVTime-Bench is held out, it shares the same data collection and annotation pipeline as AVTime-50K, which may favor AVTime over models trained on different distributions.
- Other pipelines. AVTime is best on LongVALE and surpasses TimeChat-Captioner on OmniDCBench in SODA_m (W2); both were built by other groups.
- Comparable videos. On AVTime-Bench videos ≤90 s, TimeChat-Captioner is on par with AVTime (39.6 vs. 40.1). Its overall gap comes from videos longer than its one-minute training clips, not from annotation style.
W6. Judge reliability
AVTime-Bench relies on Qwen3.6-27B for semantic comparison and event matching. Although the paper reports correlation with human judgments, evaluation with alternative judges or broader human verification would further establish the robustness of the benchmark.
Alternative judge. We re-scored AVTime-Bench with MiMo-V2.6-Flash (non-Qwen); only the judge changes.
| Model | Mean (Qwen3.6) | Mean (MiMo) |
|---|---|---|
| AVTime-30B-RL | 34.36 | 29.94 |
| AVTime-SFT-7B | 17.27 | 14.76 |
| LongVALE-only SFT-30B | 17.04 | 14.41 |
| Qwen3-Omni-30B-Instruct | 13.06 | 12.46 |
| LongVALE-LLM-7B | 2.86 | 2.42 |
The ranking is identical (Kendall's τ = 1.0 on Mean), and AVTime is first on all six metrics under both judges.
Broader human verification. Human annotators also audited the references themselves: 98.8% of benchmark captions and 98.4% of boundaries are correct (W1).
We hope these responses address your concerns, and we are happy to answer any further questions.
Thank you very much for your thorough and thoughtful response, as well as the extensive additional experiments and clarifications provided in the revision.
I just have one quick follow-up question: why were the results for AVTime-7B-RL not included in Table 3 and Table 28? Since a 7B model is much more cost-effective to deploy in practice than a 30B model, I would like to better understand the relevant trade-offs.
Comment 1 of 2
Thank you for your time and constructive feedback. We address each comment below; the corresponding changes will be included in the revised PDF.
W1. Annotation quality
We ran a human audit about 20 times larger than the original check (1,966 training events vs. 100). Three annotators with good English proficiency checked each event against the video and audio.
On these 50 benchmark videos, dropping all queries that touch an event corrected by the annotators leaves the Table 2 ranking unchanged. We will report this audit in the appendix.
W2. Results on OmniDCBench
We evaluated on OmniDCBench (1,122 clips) with the official temporal metrics and SODA_m protocol.
* Output the OmniDCBench format (scene-level structured captions) instead of AVTime's percentage-based event timeline.
In addition, the reverse transfer is weaker: TimeChat-Captioner scores 17.17 on AVTime-Bench (AVTime-30B-RL: 34.36), because its predictions stop near 60 s.
W3. Model-family bias in reward computation