acceptodds
Under review as a conference paper at ICLR 2027

AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment

Abstract

Dense temporal audio-visual captioning is important for multimodal video understanding and generation. However, prior work is constrained by three main limitations: (1) the lack of a dataset combining high temporal resolution, longer videos, and bidirectional alignment between time and events; (2) the reliance of most existing RL methods on checklists; and (3) the lack of a benchmark that bidirectionally evaluates event semantic correctness and temporal accuracy. To address these challenges, we propose: (1) AVTime-50K, a high-quality 50K-scale dataset with temporally precise, semantically accurate captions and longer videos; (2) AVTime, a state-of-the-art model optimized with a novel Bidirectional Temporal-Semantic Alignment (BiTSA) reward that imposes explicit bidirectional constraints between time and events; and (3) AVTime-Bench, a specialized benchmark that introduces a novel bidirectional evaluation of temporal accuracy and semantic correctness. Code, models, and datasets will be made available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video understanding, audio-visual temporal understanding, reinforcement learning, dense video captioning, temporal grounding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t8WBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dense temporal audio-visual captioning and introduces AVTime-50K, a large-scale dataset containing approximately 50K videos and 1.87M timestamped audio-visual events, AVTime with a Bidirectional Temporal-Semantic Alignment (BiTSA) reward, and AVTime-Bench for evaluating time-to-event, event-to-time, and event-to-event alignment. BiTSA combines semantic agreement at queried timestamps with temporal IoU of semantically matched events. Experiments on LongVALE and AVTime-Bench demonstrate consistent improvements over existing temporal and audio-visual baselines at both 7B and 30B scales.

Critical strengths
  1. Comprehensive contribution. The paper presents a complete framework covering dataset construction, reward design, model training, and benchmark development for dense audio-visual temporal captioning.

  2. Strong empirical performance. AVTime consistently outperforms the evaluated baselines on both LongVALE and AVTime-Bench, with gains observed at both 7B and 30B scales.

  3. Useful ablation studies. The ablations show that AVTime-50K supervision and the two BiTSA reward components each contribute to the final performance, and that combining T2E and E2T gives the strongest results.

Critical weaknesses
  1. Limited validation of annotation quality. Although the authors manually inspect 100 retained annotations, this is relatively small compared with the whole dataset. Since Qwen3-Omni-Thinking, which may not be sufficiently strong for fine-grained audio-visual temporal annotation, is used for both caption generation and filtering, a larger-scale human audit would better support the annotation quality claim.

  2. The paper does not report results on OmniDCBench [1], which is a relevant benchmark for omni-modal dense captioning. Evaluating AVTime on OmniDCBench would provide useful evidence of its generalization beyond the proposed benchmark and LongVALE.

  3. Potential model-family bias. The semantic evaluator and the AVTime backbone are both Qwen-based, which may introduce model-family bias in reward computation.

  4. Limited benchmark scale. AVTime-Bench contains only 300 videos, which may be insufficient to ensure a stable and representative evaluation.

  5. Potential distribution advantage. Although AVTime-Bench is held out, it shares the same data collection and annotation pipeline as AVTime-50K, which may favor AVTime over models trained on different distributions.

  6. Judge reliability. AVTime-Bench relies on Qwen3.6-27B for semantic comparison and event matching. Although the paper reports correlation with human judgments, evaluation with alternative judges or broader human verification would further establish the robustness of the benchmark.

[1] TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions, ICML 2026

Questions for authors

Please see weaknesses.

7 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for your time and constructive feedback. We address each comment below; the corresponding changes will be included in the revised PDF.

W1. Annotation quality

Limited validation of annotation quality. Although the authors manually inspect 100 retained annotations, this is relatively small compared with the whole dataset ... a larger-scale human audit would better support the annotation quality claim.

We ran a human audit about 20 times larger than the original check (1,966 training events vs. 100). Three annotators with good English proficiency checked each event against the video and audio.

AuditVideosEventsCaption correctBoundary correctAudio content correctOrder correct
AVTime-50K (random training videos)501,96696.5%92.6%96.9%100%
AVTime-Bench501,69598.8%98.4%98.3%99.9%

On these 50 benchmark videos, dropping all queries that touch an event corrected by the annotators leaves the Table 2 ranking unchanged. We will report this audit in the appendix.

W2. Results on OmniDCBench

The paper does not report results on OmniDCBench [1] ... Evaluating AVTime on OmniDCBench would provide useful evidence of its generalization beyond the proposed benchmark and LongVALE.

We evaluated on OmniDCBench (1,122 clips) with the official temporal metrics and SODA_m protocol.

ModelF1mIoUSODA_m
TimeChat-Captioner-7B (released predictions)61.269.635.4
AVTime-30B-RL*60.068.941.0

* Output the OmniDCBench format (scene-level structured captions) instead of AVTime's percentage-based event timeline.

In addition, the reverse transfer is weaker: TimeChat-Captioner scores 17.17 on AVTime-Bench (AVTime-30B-RL: 34.36), because its predictions stop near 60 s.

W3. Model-family bias in reward computation

The semantic evaluator and the AVTime backbone are both Qwen-based, which may introduce model-family bias in reward computation.

  • Gains appear on metrics that use no LLM. All LongVALE metrics are judge-free (IoU, SODA-c, CIDEr, METEOR), and AVTime is best there (Table 1).
  • A non-Qwen evaluator agrees. Re-scored by MiMo-V2.6-Flash, AVTime-30B-RL remains first on all six AVTime-Bench metrics (W6).
  • The judge's role is narrow. It only compares caption text (a semantic score or an event-identity decision); all temporal quantities are computed by code.
AuthorsOpenReview ↗

Comment 2 of 2

W4. Benchmark scale

AVTime-Bench contains only 300 videos, which may be insufficient to ensure a stable and representative evaluation.

The 300 videos span 18.2 hours and 10,344 reference events, giving 49,769 scored queries.

  • Stability. With a video-level bootstrap (2,000 resamples), AVTime-30B-RL's Mean is 34.36 [32.78, 36.01]. Its margin over the strongest baseline is at least +5.3 [3.6, 7.2] on every metric, and it ranks first in every resample.
  • Subsampling. With only 100 random videos, the system ranking stays close to the full one (Kendall's τ = 0.96 over 1,000 draws), and AVTime is first in every draw.

We will add the intervals to Table 2.

W5. Potential distribution advantage

Although AVTime-Bench is held out, it shares the same data collection and annotation pipeline as AVTime-50K, which may favor AVTime over models trained on different distributions.

  • Other pipelines. AVTime is best on LongVALE and surpasses TimeChat-Captioner on OmniDCBench in SODA_m (W2); both were built by other groups.
  • Comparable videos. On AVTime-Bench videos ≤90 s, TimeChat-Captioner is on par with AVTime (39.6 vs. 40.1). Its overall gap comes from videos longer than its one-minute training clips, not from annotation style.

W6. Judge reliability

AVTime-Bench relies on Qwen3.6-27B for semantic comparison and event matching. Although the paper reports correlation with human judgments, evaluation with alternative judges or broader human verification would further establish the robustness of the benchmark.

Alternative judge. We re-scored AVTime-Bench with MiMo-V2.6-Flash (non-Qwen); only the judge changes.

ModelMean (Qwen3.6)Mean (MiMo)
AVTime-30B-RL34.3629.94
AVTime-SFT-7B17.2714.76
LongVALE-only SFT-30B17.0414.41
Qwen3-Omni-30B-Instruct13.0612.46
LongVALE-LLM-7B2.862.42

The ranking is identical (Kendall's τ = 1.0 on Mean), and AVTime is first on all six metrics under both judges.

Broader human verification. Human annotators also audited the references themselves: 98.8% of benchmark captions and 98.4% of boundaries are correct (W1).

We hope these responses address your concerns, and we are happy to answer any further questions.

Thank you very much for your thorough and thoughtful response, as well as the extensive additional experiments and clarifications provided in the revision.

I just have one quick follow-up question: why were the results for AVTime-7B-RL not included in Table 3 and Table 28? Since a 7B model is much more cost-effective to deploy in practice than a 30B model, I would like to better understand the relevant trade-offs.

Reviewer_XkTGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AVTime, a model optimized with a novel Bidirectional Temporal-Semantic Alignment reward. It also introduces AVTime-50K and AVTime-Bench to provide fine-grained supervision and evaluation of audio-visual temporal event understanding. Experiments demonstrate that the proposed method achieves SOTA performance on AVTime-Bench and the LongVALE benchmark.

Critical strengths

Overall, the proposed method is reasonable from an engineering perspective, and the design of a bidirectional reward is appropriate for this task. To evaluate the proposed algorithm, the authors also collect a new dataset, which could contribute to the community.

Critical weaknesses

The biggest concern with this work is its novelty. The core design of the bidirectional alignment reward is not entirely new. One similar work is MCCL [1], which also proposes a bidirectional approach that improves localization accuracy through semantic matching (similar to T2E) and refines event semantic perception through localization matching (similar to E2T). Although the tasks are different, the overall design philosophy is quite similar. The authors should compare their method with this work and clarify the core differences beyond the differences in tasks. [1] Xie, Zhuyang, et al. "Exploring temporal event cues for dense video captioning in cyclic co-learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.

Questions for authors
  1. Please refer to the weakness mentioned above.
  2. The LLM Judge uses a 0–4 scoring scale, but the distinction between scores of 2 and 3 is ambiguous. This design may also cause problems in multi-event scenarios. For example, if the ground truth is "a door opens while a dog is barking," while the model predicts "a door opens," should the judge assign a score of 2 or 3 given the missing event? The authors should better explain the rationale behind this design and consider how to handle multi-event scenarios.
  3. Following Question 2, what are the advantages of using an LLM Judge over other scoring methods, such as CLIPScore? It would be helpful to include such comparisons to validate the effectiveness of the proposed design.
  4. The analysis of the results on LongVALE and AVTime-Bench is relatively brief and provides limited insights. Simply repeating the results already presented in the tables is not very informative. Instead, it would be more insightful to provide a deeper analysis of these results. For example, the authors could explain why TimeChat underperforms despite introducing a similar T2E reward design, and what specific benefits the proposed method provides.
  5. For the Time-to-Event results on AVTime-Bench, there appears to be a large performance gap between the audio and visual modalities. Could the authors explain the reason for this gap?
4 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for your time and constructive feedback. We address each comment below; the corresponding changes will be included in the revised PDF.

W1 / Q1. Novelty relative to MCCL

The core design of the bidirectional alignment reward is not entirely new. One similar work is MCCL [1] ... The authors should compare their method with this work and clarify the core differences beyond the differences in tasks.

Thank you for pointing us to MCCL; we will cite and discuss it in Sec. 2.2. MCCL shows that semantic matching and location matching can supervise each other in a detection-style dense captioner. BiTSA shares this insight but starts from a different design.

Queries vs. time points. MCCL builds on PDVC, where N learned queries carry the events, and its central question is which query corresponds to which ground-truth event. A free-form MLLM timeline has no queries or slots. T2E therefore picks a time point and asks whether the model and the reference say the same thing at this moment; E2T asks where the model placed a given reference event. T2E compares caption sets instead of assigning slots, so a split event is still credited, while extra or duplicated events lower the score.

MCCLBiTSA (ours)
"Bidirectional"A training cycle between generator and localizerTwo independent checks of one generated timeline, averaged into a reward
UnitN learned queries, matched one-to-one to eventsTime points (T2E) and reference events (E2T)
Semantic comparisonQuery features vs. CLIP caption embeddingsGenerated text vs. reference text (LLM judge)
Learning signalDifferentiable supervised lossesScalar RL reward
RequirementDETR/PDVC-style architectureAny MLLM that outputs a text timeline

Q2. The 0–4 scale and multi-event cases

The LLM Judge uses a 0–4 scoring scale, but the distinction between scores of 2 and 3 is ambiguous. This design may also cause problems in multi-event scenarios ... if the ground truth is "a door opens while a dog is barking," while the model predicts "a door opens," should the judge assign a score of 2 or 3 given the missing event?

The 0–4 score applies to one (reference event, predicted event) pair; missing events are handled by code, not by the judge.

  • Reward. At each time point, active reference and predicted events are matched one-to-one and the total is normalized by |A| + |Â| (App. A.3, Eq. 2). If door-opening and barking are annotated as two separate reference events, "a door opens" matches the former (score 4), while barking remains unmatched, giving 2·1/(2+1) = 0.67. For a single compound reference event, the omission is assessed by the pairwise semantic rubric.
  • Benchmark. Audio and visual content are scored separately, so the visual branch gives credit for the door and the audio branch gives none for the barking.
  • Sensitivity. Merging the middle levels in three ways, or binarizing the score, leaves the Table 2 ranking unchanged (Kendall's τ = 1.0).

Q3. LLM judge vs. CLIPScore

Following Question 2, what are the advantages of using an LLM Judge over other scoring methods, such as CLIPScore?

  • Audio. CLIPScore compares images with text and cannot assess speech, music, or sound, which account for 40.6% of our T2E/E2T queries.
  • Event identity. E2T and E2E must decide which predicted events describe a reference event. Similarity scores are high for neighboring events in the same scene and need a hand-tuned threshold; the judge follows an explicit identity rule (Listing 10).
  • Robustness. A non-Qwen judge (MiMo-V2.6-Flash) reproduces the model ranking (Kendall's τ = 1.0 on Mean).
  • Comparison. We tested a text-only variant, CLIP similarity between reference and predicted captions, on the human-scored queries of Appendix E (50 videos). Its video-level Spearman correlation with human scores is 0.63 / 0.69 / 0.48 (T2E / E2T / E2E), lower than the judge's on all three (Table 12).
AuthorsOpenReview ↗

Comment 2 of 2

Q4. Deeper analysis: why TimeChat underperforms

The analysis of the results on LongVALE and AVTime-Bench is relatively brief ... the authors could explain why TimeChat underperforms despite introducing a similar T2E reward design, and what specific benefits the proposed method provides.

We will expand Sec. 4 with this analysis. The RL-trained model is TimeChat-Captioner, which is trained on one-minute clips (median 59 s); AVTime-50K videos have a median of 2.7 min. On AVTime-Bench its timeline stops near 60 s whatever the video length, covering 36% of the time axis (AVTime: 99.7%):

Mean of six metrics, by query positionFirst 60 s60–120 sAfter 120 s
TimeChat-Captioner-7B30.310.311.5
AVTime-30B-RL42.935.327.1

The table pools all 300 videos by query position; 278 of them are longer than 90 s, and they account for the first-column gap (29.2 vs. 43.1 in their first 60 s). On the 22 videos ≤90 s the two models are on par (39.6 vs. 40.1); the gap comes from longer videos.

Benefits of our method. From the Qwen3-Omni base to AVTime-30B-RL, the share of reference events that the model localizes rises from 41% to 81–83%, and the tIoU of matched events from 5–6 to 21–24. BiTSA also outperforms both single-direction rewards on LongVALE (TVG mIoU 17.76 vs. ≤16.83; DVC CIDEr 12.18 vs. ≤10.99; Table 18a).

Q5. The audio–visual gap in T2E

For the Time-to-Event results on AVTime-Bench, there appears to be a large performance gap between the audio and visual modalities. Could the authors explain the reason for this gap?

The gap reflects audio content, not timing: on E2T and E2E, audio and visual scores differ by about 1.5 points or less. Three factors explain it:

  1. Speech dominates. 85% of audio references involve speech and 48% contain a verbatim quote, so T2E-A largely measures transcription fidelity.
  2. The judge is stricter on audio. With the MiMo judge, AVTime's T2E-A rises from 14.3 to 27.6, while T2E-V changes little (44.3 → 39.1).
  3. Base models often omit sound. 44.5% of Qwen3-Omni's captions contain no audio cue (AVTime: 16.5%). AVTime raises T2E-A 7.4× over its base (1.94 → 14.34), the highest in Table 2.

We hope these responses address your concerns, and we are happy to answer any further questions.

thanks for the response. My concerns are addressed.

Thank you for carefully considering our response and for the updated assessment. We are glad that the additional experiments and clarifications addressed the concerns raised. Your constructive feedback has substantially improved the quality of the manuscript, and we have incorporated the corresponding revisions into the updated version.

Reviewer_64DQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the task of dense & temporally grounded video captioning. The authors first propose automatic methods to curate a large dataset of captioned & grounded videos, based on the idea that it is easier to caption small clips and post-hoc retore the full-video timeline. They then finetune a vision-language model to do the task with full video input. Models first use SFT training, and then use RL training with semantic and temporally grounded rewards. Results on a held-out test set and LongVALE demonstrate improvements over baselines and the previous SOTA.

Critical strengths

S1: the paper is clearly written, presents coherent novel ideas and demonstrates their utility with quantitative evaluation.

S2: the semantic and temporal RL rewards were clever and well-motivated. The ablations clearly demonstrate their utility. They can be useful inspiration for other research.

S3: the engagement with prior literature was good.

Critical weaknesses

W1: there is a certain circularity in auto-annotating data for the task, and then proposing a new automatic system for the same task. Especially as your "Quality control" states that the auto-annotator is perfect. Please can you provide a row for Table 2 that states the performance of the auto-annotator on the held-out set, prior to the human correction?

W2: I would have liked more discussion of task dependence on genre. Instructional videos, documentaries, cinematic videos, etc all seem to be included but naturally need different treatment. What is genre LongVALE testing? Can you run an OOD genre test suite?

Questions for authors

Q1: how much of the novelty in your work is due to the addition of auditory captioning (rather than just dense visual captioning)?

Q2: why are timestamps normalized to a percentage? Qwen receives as input second-level timestamps, so I would expect the same in its outputs.

Q3: please could you provide a brief motivation paragraph for why it is sensible to do the annotation without video context outside of one clip?

Q4: Have you defined notation for around L249?

Q5: Figure 3 order of the two rewards is opposite to the text, I think?

Q6: What are the LongVALE evaluation protocols? They need to be included in the main text in more detail.

Q7: What did you do with human judgements and alignment of the LLM judge? Neither from 4.5 or Appendix E can I understand your human methodology.

4 comments
AuthorsOpenReview ↗

Comment 1 of 2

Thank you for your time and constructive feedback. We address each comment below; the corresponding changes will be included in the revised PDF.

W1. Circularity of auto-annotation

there is a certain circularity in auto-annotating data for the task, and then proposing a new automatic system for the same task. Especially as your "Quality control" states that the auto-annotator is perfect. Please can you provide a row for Table 2 that states the performance of the auto-annotator on the held-out set, prior to the human correction?

  • The annotator is a pipeline instead of an end-to-end model. It splits the video at speech, scene, and sound boundaries and captions each short clip with dense frames and full audio. The same Qwen3-Omni-Thinking applied end-to-end to full videos scores 15.02 on AVTime-Bench (AVTime-30B-RL: 34.36). AVTime distills the pipeline's multi-pass output into one forward pass.
  • Requested row. Three annotators with good English proficiency checked and corrected the pipeline's annotations on 50 held-out benchmark videos (1,695 events):
Auto-annotator vs. human-corrected referencesCaption correctBoundary correctAudio content correctLocalization (tIoU × 100)
50 benchmark videos98.8%98.4%98.3%98.0

AVTime-30B-RL's E2T is 17.6 / 19.1, so the task is far from saturated.

  • Broader check of the training set. We (1) audited a larger sample and (2) asked annotators to inspect every event as closely as possible. On 50 random training videos (1,966 events), 96.5% of captions, 92.6% of boundaries, and 96.9% of audio descriptions are correct. We will report these numbers in Sec. 3.1.

W2. Dependence on genre

I would have liked more discussion of task dependence on genre ... What is genre LongVALE testing? Can you run an OOD genre test suite?

The table below reports results for 237 videos across four genre groups (mean of six metrics):

GenreVideosAVTime-30B-RLBest baseline
Cinematic (film clips, trailers)18236.217.2
News3432.022.5
Education & documentary1632.617.5
UGC & sports542.624.9

The remaining 63 videos come from sources without a genre label (AVTime 33.1 vs. 14.3).

Two benchmarks from other groups serve as out-of-distribution tests. LongVALE is built from open-domain web videos in ACAV-100M, and AVTime is best there (Table 1). OmniDCBench consists of movies and YouTube clips; AVTime-30B-RL, in its output format, reaches a SODA_m of 41.0 (TimeChat-Captioner: 35.4). We will add this table and discussion to the paper.

Q1. Novelty due to audio

how much of the novelty in your work is due to the addition of auditory captioning (rather than just dense visual captioning)?

The reward and the T2E/E2T/E2E protocol are modality-agnostic and would be new for visual-only dense captioning as well; the visual branch alone improves strongly over the base model (T2E-V 14.9 → 44.3). Audio adds joint events that bind speech, music, and sound to the visual timeline, and a separate audio branch in the benchmark, which reveals that current models rarely verbalize sound (T2E-A 1.9 for the base vs. 14.3 for AVTime).

AuthorsOpenReview ↗

Comment 2 of 2

Q2. Percentage timestamps

why are timestamps normalized to a percentage? Qwen receives as input second-level timestamps, so I would expect the same in its outputs.

(1) It matches LongVALE's output format, so one model serves both benchmarks. (2) The output vocabulary is fixed at 100 values, which keeps targets uniform from 1-minute to 10-minute videos. (3) The model sees at most 128 frames, so for a median 2.7-minute video one bucket (1.6 s) is already close to the frame spacing. Given the video duration, percentage timestamps can be converted to seconds at the resolution of the normalized temporal grid.

Q3. Annotating within one clip

please could you provide a brief motivation paragraph for why it is sensible to do the annotation without video context outside of one clip?

Short clips let the annotator see dense frames and the complete audio, which fine-grained events need; on full videos the same model is much coarser (15.02 on AVTime-Bench). Clip boundaries follow ASR sentences, scene cuts, and sound transitions, so events are rarely split, and captions are restricted to observable content, which needs little outside context. We will add this paragraph to Sec. 3.1.

Q4. Notation near L249

Have you defined notation for around L249?

We will revise the main text to define every symbol at first use: in the timelines Y={(sj,ej,cj)}Y = \{(s_j, e_j, c_j)\} (with MM events) and Y^={(s^i,e^i,c^i)}\hat Y = \{(\hat s_i, \hat e_i, \hat c_i)\} (with NN events), sjs_j and eje_j are the start and end percentage buckets of event jj, cjc_j is its caption, and MM and NN are the numbers of reference and predicted events. The sampled time points TT and the semantic score QsemQ_{\rm sem} will also be defined where they first appear.

Q5. Order in Figure 3

Figure 3 order of the two rewards is opposite to the text, I think?

Thank you. We have reordered Figure 3 to follow the text: T2E first, then E2T.

Q6. LongVALE evaluation protocols

What are the LongVALE evaluation protocols? They need to be included in the main text in more detail.

We follow the official LongVALE test set and scoring scripts for its three tasks: temporal video grounding ([email protected]/0.5/0.7, mIoU), dense video captioning (SODA-c, CIDEr, METEOR), and segment captioning (BLEU-4, ROUGE-L, CIDEr, METEOR). No metric uses an LLM judge. We will move this description into Sec. 4.1.

Q7. Human judgments and judge alignment

What did you do with human judgements and alignment of the LLM judge? Neither from 4.5 or Appendix E can I understand your human methodology.

  • Material. Queries are sampled from benchmark videos with a fixed seed, 20 per video, balanced over T2E/E2T/E2E and audio/visual, and independent of any score or model.
  • Blind setup. Three annotators with good English proficiency see only what the judge sees: the reference caption(s), the indexed candidate captions, and the same instructions. They see no model names, judge scores, or timestamps.
  • Task. The annotators answer the same requests as the judge: for T2E they give a 0–1 score; for E2T and E2E they select all predicted events that describe each reference event.
  • Scoring. Their answers go through the same evaluation code as the judge's, and we compare human-based and judge-based scores per query and per video.

We will add this description to Sec. 4.5 and Appendix E.

We hope these responses address your concerns, and we are happy to answer any further questions.

Reviewer_64DQOpenReview ↗

Follow-up

Thank you for your response. Regarding W1, you say "AVTime distills the pipeline's multi-pass output into one forward pass", which seems accurate to me. Are you then claiming that the problem addressed is to do with efficiency, and that you are in effect doing self-distillation? If so, this is quite different from the current framing of the paper, and would require quantitative evaluation of the computational requirements of the pipeline vs. end-to-end system.

Thank you for the follow-up, and apologies that our previous response was too brief on this point. We would like to clarify that efficiency is not the problem we set out to address, and that we do not see the method as self-distillation. We explain our reasoning below.

What the sentence refers to. It describes where the training annotations come from. The annotation pipeline (Section 3.1) is an offline data engine. It is run once to build AVTime-50K and is never used at inference. The pipeline's outputs are data: they are the annotations on which AVTime is first fine-tuned. We used "distills" in the everyday sense of condensing a multi-step annotation process into one model, not in the technical sense of knowledge distillation. (there are no soft targets or logits, and no model in the pipeline produces the full-video output that AVTime is trained to generate) To make this explicit, Section 3.1 of the revision now states: "The pipeline is used only offline to construct AVTime-50K; at inference, AVTime generates the full-video timeline in a single pass without segmentation or clip-level captioning."

How this differs from self-distillation. We understand why the sentence could be read that way. We hope the following points from the paper help to clarify the difference:

  1. The annotations do not come from a model performing AVTime's task. The pipeline first segments each video with ASR, scene-change, and acoustic cues, then captions each short clip separately, and finally assembles and filters the timeline. No model in the pipeline ever produces a full-video timeline. When the captioning model used in the pipeline is applied directly to full videos, it scores 15.02 on AVTime-Bench (Table 2), versus 34.36 for AVTime-30B-RL. The full-video capability therefore does not seem to be present in any single component of the pipeline.
  2. Training does not stop at fitting the annotations. After supervised fine-tuning, AVTime is optimized with GSPO under the BiTSA reward, which scores bidirectional time–event correspondence rather than token-level agreement with the annotations. This stage adds 6.92 points on AVTime-Bench over the SFT model (27.44 → 34.36, Section 4.3).
  3. Evaluation is independent of the pipeline's raw output. AVTime-Bench references are verified and refined by human annotators (Appendix D), and LongVALE uses its own annotations and official metrics.

Computational cost. Efficiency is always not our claim, but we agree the comparison is informative, so we measured it. Both systems were run on the same 40 AVTime-Bench videos and the same two GPUs, using our released pipeline code with default settings. Pipeline figures are averaged over the 36 videos (mean length 197 s) for which every clip was captioned successfully.

Per video (mean)Annotation pipelineAVTime-30B-RL
Captioning-model calls17.91
Additional models and toolsASR, scene detection, audio tagging, clip rendering–
Input tokens161.9k12.1k
Output tokens29.0k2.8k
GPU time: ASR and audio tagging37.5 s–
GPU time: clip-level captioning97.0 s–
GPU time: timeline filtering14.1 s–
GPU time: full-video inference–15.8 s
GPU time, total148.6 s15.8 s
CPU processing71.5 s–

The pipeline consumes 13.3× the input tokens and 9.4× the GPU time of a single AVTime-30B-RL pass. We report this as a property of the resulting model, not as a motivation of the work. The table and the measurement setup are given in the new Appendix P (Table 30).

Our contributions are unchanged. (1) AVTime-50K, a dataset of about 50K multi-minute videos and 1.87M temporally precise audio-visual event annotations; adding it to supervised training raises the AVTime-Bench mean from 17.04 to 27.44. (2) The BiTSA reward, which imposes explicit bidirectional constraints between time and events and adds a further 6.92 points. (3) AVTime-Bench, a human-verified benchmark that evaluates time-to-event, event-to-time, and event-to-event correspondence separately for audio and vision. The annotation pipeline is the means by which the first contribution is built; it is not the problem the paper addresses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

Summary of changes in the revised PDF

We thank all reviewers for their feedback. We have uploaded a revised PDF. All new or changed text is marked in blue, and the main text is 10 pages. The changes are listed below with the reviewer comments they address.

Main text

  • Sec. 1, 3.1: refers to the annotation model as a captioning model; states that the annotation pipeline is used only offline to construct AVTime-50K (follow-up discussion).
  • Sec. 2.2: cites MCCL (XkTG W1).
  • Sec. 3.1: explains clip-level annotation (64DQ Q3) and percentage timestamps (64DQ Q2); reports a human audit of 1,966 training events, which replaces the earlier 100-sample check (t8WB W1, 64DQ W1).
  • Sec. 3.2: defines all notation at first use (64DQ Q4); describes the T2E time probes and the caption-set score (XkTG Q2).
  • Figure 3: panels reordered to (a) Time-to-Event and (b) Event-to-Time (64DQ Q5).
  • Sec. 4.1: states the LongVALE test set and metrics (64DQ Q6).
  • Sec. 4.2: adds an analysis of TimeChat-Captioner and of the audio–visual gap (XkTG Q4, Q5); adds OmniDCBench results for AVTime-30B-RL and AVTime-7B-RL in the new Table 3 (t8WB W2, 64DQ W2; follow-up discussion); the caption of Table 2 points to confidence intervals (t8WB W4).
  • Sec. 4.5: describes the human evaluation protocol (64DQ Q7).

Appendix

  • A.1: same wording change as Sec. 1; points to the human audit and to Appendix P.
  • C.4: results by genre (64DQ W2).
  • E: human evaluation protocol (64DQ Q7).
  • H: comparison with MCCL (XkTG W1).
  • I: human audit of annotations (t8WB W1, 64DQ W1).
  • J, K: analysis of TimeChat-Captioner and of the audio–visual gap (XkTG Q4, Q5).
  • L: bootstrap confidence intervals and subsampling (t8WB W4).
  • M: evaluation with an alternative, non-Qwen judge, now including AVTime-7B-RL (XkTG Q3; t8WB W3, W6; follow-up discussion).
  • N: sensitivity to the judge's score scale (XkTG Q2).
  • O: OmniDCBench evaluation details (t8WB W2).
  • P: computational cost of the annotation pipeline and of AVTime inference (Table 30); comparison of AVTime-7B-RL and AVTime-30B-RL (Table 31, Figure 15) (follow-up discussion).

Tables 1 and 2 and Figures 1–14 keep their numbers. Table 3 and Figure 15 are new, and the appendix tables are renumbered because of the new tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.