acceptodds
Under review as a conference paper at ICLR 2027

How Video MLLMs Form Temporal Boundaries: Boundary Flow Tracing and Temporal Revisit

Abstract

Generative video temporal grounding enables video multimodal large language models (MLLMs) to directly produce temporal boundaries, yet their formation remains unclear. We investigate start and end timestamp formation through route interventions and representation probing. Boundary Flow Tracing reveals endpoint- and depth-dependent pathways in TimeLens, with visual-to-prompt effects concentrated at earlier depths than prompt-to-boundary effects. Qwen3-VL and InternVL2.5 also exhibit prompt-to-boundary dependence, with model-specific depth profiles. Restricted readouts recover temporal cues from intermediate visual states, while MiMo-VL probes provide additional evidence of boundary information before generation. These observations motivate Temporal Revisit, a learned intervention that returns intermediate visual evidence to later visual states of a frozen backbone. Broad Temporal Revisit (BTR) tests whether this evidence can improve native boundary generation. Hypothesis-Guided Temporal Revisit (HTR) further examines whether prediction-conditioned reuse with preservation training can improve grounding while limiting prediction changes. BTR improves mIoU on TimeLens-SFT, Qwen3-VL, and MiMo-VL across the three evaluated benchmarks. On TimeLens-SFT, the gains are 3.92, 2.85, and 2.19 percentage points on QVHighlights-TimeLens, Charades-TimeLens, and ActivityNet-TimeLens, respectively. After GRPO, HTR improves native mIoU while modifying fewer predictions than BTR. Post-intervention tracing on TimeLens-SFT shows stronger direct visual-to-boundary dependence after BTR. These results support learned reuse of recoverable temporal evidence, with benefits that depend on the host and intervention design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Multimodal Large Language Models, Mechanistic Interpretability, Intermediate Representations, Temporal Localization, Cross-Layer Evidence Reuse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_48dCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how video MLLMs generate the start and end timestamps of an event. Using attention interventions and representation probes, the authors examine where temporal information is retained and which pathways support timestamp generation. They then propose Temporal Revisit, which feeds intermediate visual features into later layers while keeping the original model frozen.

Critical strengths
  1. The paper addresses a useful question beyond benchmark performance: how temporal boundaries are formed inside video MLLMs. It carefully distinguishes information that can be recovered from hidden states from information actually used during generation.

  2. The analysis motivates a concrete intervention. The authors identify temporal cues in intermediate layers, introduce a module to reuse them, and examine how the intervention changes the model’s dependence on different information pathways.

  3. The results include useful controls. BTR adds approximately 270K trainable parameters and improves TimeLens-SFT mIoU by 3.92, 2.85, and 2.19 points across the three benchmarks. Temporal shuffling, same-data LoRA training, and an additional decoding pass provide reasonable checks against alternative explanations. Improvements on other backbones also support the usefulness of the approach.

Critical weaknesses
  1. The interventions provide only a partial account of boundary formation. Blocking prompt-to-boundary attention removes access to the query, instructions, and any visual information already incorporated into the prompt states. A drop in the probability of the original timestamp therefore does not necessarily indicate impaired temporal reasoning. Matched random-edge interventions and grounding results after intervention would help clarify what these effects measure.

  2. The gains on the stronger post-GRPO model are small. BTR provides little benefit and sometimes reduces performance. HTR improves mIoU by only 0.31, 0.53, and 0.25 points while requiring another inference pass. Multiple training seeds or confidence intervals for the main results would help establish whether these gains are reliable.

  3. Several design choices change together. BTR and HTR differ in readout depth, parameter count, training objectives, and inference passes, so their comparison does not isolate the effect of hypothesis-guided retrieval. Parameter-matched adapters and controls that transform late-layer features without reusing intermediate states would also help establish the specific benefit of temporal evidence reuse.

Questions for authors
  1. Please clarify whether the observed attention-knockout effects reflect disrupted temporal grounding or reduced access to query and instruction information. Matched random-edge controls and grounding results after intervention would help distinguish these explanations.

  2. Please provide confidence intervals or results across multiple training seeds for HTR’s small gains on the post-GRPO model, together with its inference overhead.

  3. Please isolate the effects of hypothesis-guided retrieval and preservation training under matched settings. A parameter-matched adapter without intermediate-feature reuse would also help establish the specific contribution of Temporal Revisit.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 48dC (Part 1)

We sincerely thank the reviewer for recognizing the value of our mechanistic analysis, the practical gains of Temporal Revisit, and the useful experimental controls. We also appreciate the constructive feedback and guidance that have helped us further improve the paper.

Q1&W1: Clarifying Attention-Knockout Effects on Temporal Grounding and Query/Instruction Access

Answer: We thank the reviewer for this constructive suggestion. First, we would like to clarify that our attention knockouts are applied to specific decoder-layer windows rather than across all layers. Thus, query and instruction information remains accessible through other layers.

In addition, to examine whether attention knockouts disrupt temporal grounding, we evaluate their impact on accurately localized examples. Compared with the no-knockout baseline, targeted P→Y knockout substantially disrupts temporal grounding, demonstrating that the attention knockouts actually affects localization performance.

To further investigate whether this performance drop is simply caused by reduced attention connectivity, we compare targeted P→Y knockout with a matched random-edge that blocks the same number of connections within the same layer window. The random-edge knockout has almost no effect on grounding performance, indicating that the degradation is not merely due to reduced attention connectivity, but more importantly, to the disruption of visual information.

Attention KnockoutGrounding mIoU (%)
TimeLens-SFT (No Knockout)95.06
Targeted P→Y24.48 (−70.57)
Random-edge95.11 (+0.05)

Accordingly, we have added a new paragraph titled “Grounding Effects of Pathway Interventions” in Section 2.1 (Table 1) of the revised manuscript (changes marked in blue).


Q2&W2: Reliability of HTR's Small Gains and Inference Overhead

Answer: Following the suggestion, we trained HTR based on TimeLens-GRPO baseline with three random seeds, keeping all other settings fixed. As shown in the following table, all three runs achieve positive mIoU gains on QVHighlights-TimeLens, with a mean gain of +0.33 mIoU points and a standard deviation of 0.07 points. Additionally, we evaluated paired video-cluster bootstrap to estimate the uncertainty of these gains. Specifically, we computed the 95% confidence intervals based on randomly sampled 2000 pairs, and their confidence intervals of perfromance gain lie entirely above zero for all runs.These results demonstrate the reliability of HTR's gains across independent training runs.

RunsΔmIoU
run 1+0.31
run 2+0.40
run 3+0.27
Mean ± Standard Deviation+0.33 ± 0.07
95% confidence intervals[+0.16, +0.49]

For inference overhead, we evaluated HTR based on Charades-TimeLens using a single NVIDIA H20 GPU. The additional pass for feature revisiting increases average inference latency from 0.75s to 1.34s, while peak GPU memory rises by only 0.06 GiB, from 20.89 GiB to 20.95 GiB. Overall, HTR improves grounding accuracy at the cost of around 0.5s in additional latency, with minimal impact on peak GPU memory.

Accordingly, we have added a new paragraph titled “HTR Reliability Across Training Seeds” and included the inference-cost measurements under “Inference Efficiency” in Appendix C.3 of the revised manuscript (changes marked in blue).

AuthorsOpenReview ↗

Response to Reviewer 48dC (Part 2)

Q3-1&W3: Isolating the Effects of Hypothesis-Guided Retrieval and Preservation Training under Matched Settings

Answer: Following your suggestion, we compared full HTR with variants that remove hypothesis-guided retrieval, preservation training, or both, using the same TimeLens-GRPO backbone and matched settings on QVHighlights-TimeLens. As shown in the table, each component improves grounding performance on its own, and full HTR with both components achieves the best result. These results show that both components improve performance and suggest that their contributions are complementary.

Hypothesis GuidancePreservation TrainingmIoU (%)
✗✗65.08
✓✗65.20
✗✓65.48
✓✓65.70

Accordingly, we have added a new paragraph titled “HTR Component Ablations” in Section 4.5 (Table 4) of the revised manuscript (changes marked in blue).


Q3-2&W3: Parameter-Matched Adapter without Intermediate-Feature Reuse

Answer: Following the suggestion, we compared our BTR using the intermediate-layer features against the counterpart without intermediate-feature reuse. Specifically, the counterpart reads L34 visual features and injects the transformed residuals through a capacity-matched adapter back into L34. As shown in the following table, full BTR outperforms the counterpart without intermediate-feature reuse. The results demonstrate the contribution of temporal revisit intermediate-layer features for video grounding.

MethodGrounding mIoU (%)
BTR (L21→L34)60.96
BTR without Intermediate-Feature (L34→L34)58.17

We have added a new paragraph titled “Parameter-Matched Adapter without Intermediate-Feature Reuse” in Appendix C.4 (Table 22) of the revised manuscript (changes marked in blue).

We appreciate the reviewer's careful evaluation and valuable suggestions, and hope that our additional experiments and analyses have addressed the concerns raised.

Reviewer_48dCOpenReview ↗

Response to Authors

Thank you for the detailed rebuttal and additional experiments. The random-edge control, multi-seed results with confidence intervals, and matched ablations address my main concerns. HTR's gains on the GRPO model remain modest relative to the extra inference pass, which is worth discussing in the main text. I will maintain my score.

Dear Reviewer 48dC,

Thank you for your follow-up and for acknowledging our additional experiments. We appreciate your constructive feedback and agree that the trade-off between HTR's modest grounding improvements and additional inference latency deserves discussion in the main text.

Following your suggestion, we have added a discussion of HTR's grounding gains and inference overhead in Section 4.2 of the revised manuscript(changes marked in blue). We now explicitly discuss the increased inference latency, the minimal additional GPU memory usage, and the limitations of HTR in latency-sensitive applications.

Thank you again for your time and valuable suggestions.

All the best,
The Authors

Reviewer_NTVcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies timestamp generation in video MLLMs using attention interventions and probes, then introduces Temporal Revisit to inject intermediate visual features into later layers. BTR uses one generation pass; HTR refines the native interval in a second pass with preservation training. Both freeze the host. BTR improves grounding across several hosts, while HTR yields smaller gains with fewer prediction changes on TimeLens-GRPO.

Critical strengths
  1. Internal evidence about timestamp generation. In TimeLens, visual-to-prompt dependence is concentrated earlier than prompt-to-timestamp dependence, with different profiles for start and end predictions. Probe performance drops after video replacement or temporal shuffling, supporting the presence of readable temporal cues in intermediate states. These findings provide concrete evidence about the layers and representations involved in timestamp generation (Figures 1–3, Table 4).
  2. Practical gains from BTR. With about 270K trainable parameters, BTR improves TimeLens-SFT's mIoU by 2.19–3.92 percentage points across the three TimeLens benchmarks. It also improves Qwen3-VL and MiMo-VL and outperforms the tested same-data LoRA baseline, supporting the practical value of this small adaptation module (Tables 1 and 2).
Critical weaknesses
  1. Limited explanatory and design value of the analysis. Table 4 provides controlled probe evidence that temporal cues are more accessible at intermediate depths under the chosen readout, but does not establish which computations determine particular boundaries or their errors. Similar visual-probe performance on SFT and GRPO coexists with substantially different BTR gains (Tables 1 and 4), which the analysis does not explain or predict. The diagnostics therefore provide an empirical basis for the design, but their explanatory value and ability to guide effective reuse remain limited.

  2. The added value of the core designs remains unclear. BTR lacks an independently trained, capacity-matched late-layer adapter control. HTR's 0.25–0.53-point mIoU gains on GRPO also lack paired confidence intervals and repeated-training results. These gaps leave the core designs' added value and the reliability of the gains unresolved.

Questions for authors
  1. Do probe advantages translate into revisit gains?. Table 4 compares probe performance across layers. With the host, injection layer, adapter architecture, data, and training budget fixed, do independently trained BTR adapters reproduce the intermediate-layer advantage in native grounding, including against a control using only injection-layer features?

  2. Reliability of HTR's small gains. Do HTR's gains on GRPO reproduce across independent training runs? Please report the mean gain, variation across runs, and paired video-cluster bootstrap confidence intervals.

  3. Can the diagnostics explain intervention outcomes?. Can the route and probe diagnostics explain the much larger BTR gains on SFT than GRPO despite similar visual-probe performance? What testable internal difference links these diagnostics to localization errors that revisiting can correct?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NTVc (Part 1)

We sincerely appreciate the reviewer's encouragement of our work, as well as the constructive feedback and guidance that have helped us further improve the paper.

Q1&W2(1): Do Probe Advantages Translate into Revisit Gains? and Does Intermediate-Feature Reuse Outperform Injection-Layer-Only Adaptation?

Answer: Following your suggestion, we have added controlled experiments to examine whether probe advantages translate into revisit gains and whether intermediate-feature reuse provides additional benefits over an adapter using only injection-layer features.

Firstly, we independently trained BTR adapters based on TimeLens-SFT using visual features from representative early (L4), intermediate (L21), and late (L30) decoder layers respectively. As shown in the following table, all three independently trained adapters improve native grounding model TimeLens-SFT, showing that independently trained BTR adapters can reproduce the intermediate-layer advantage in native grounding. Besides, revisiting layers with higher probe mIoU values generally achieve better grounding performance. The result demonstrates that the probe advantages can translate into revisit gains.

Readout → InjectionVisual-probe mIoU (%)Grounding mIoU (%)
TimeLens-SFT—57.04
L4 → L348.4060.12 (+3.08)
L21 → L3411.2060.96 (+3.92)
L30 → L349.7560.85 (+3.81)

Additionally, we compare our BTR using the intermediate-layer features against the counterpart using only injection-layer features with a capacity-matched adapter. Specifically, the counterpart reads L34 visual features and injects the transformed residuals back into L34. As shown in the following table, the gain from using only injection-layer features is smaller than that from reusing intermediate-layer features. These results demonstrate the intermediate-layer advantage against the injection-layer features.

MethodGrounding mIoU (%)
TimeLens-SFT57.04
BTR (L21 → L34)60.96 (+3.92)
Late-layer-only Adapter (L34→L34)58.17 (+1.13)

Accordingly, we have added a new paragraph titled "Probe Advantages and BTR Grounding Gains" in Appendix C.4 (Table 22) of the revised manuscript (changes marked in blue).


Q2&W2(2): Do HTR Gains Reproduce Across Independent Training Runs?

Answer: Thank you for the suggestion. We trained HTR based on TimeLens-GRPO baseline with three random seeds, keeping all other settings fixed. As shown in the table, all three runs achieve positive mIoU gains on QVHighlights-TimeLens, with a mean gain of +0.33 mIoU points and a standard deviation of 0.07 points. Additionally, we evaluated paired video-cluster bootstrap to estimate the uncertainty of these gains. Specifically, we computed the 95% confidence intervals based on randomly sampled 2000 pairs, and their confidence intervals of perfromance gain lie entirely above zero for all runs.These results demonstrate the reliability of HTR's gains across independent training runs.

RunsΔmIoU
run 1+0.31
run 2+0.40
run 3+0.27
Mean ± Standard Deviation+0.33 ± 0.07
95% confidence intervals[+0.16, +0.49]

Accordingly, we have added a new paragraph of "HTR reliability analysis across training seeds" in Appendix C.1 of the revised manuscript (changes marked in blue).

AuthorsOpenReview ↗

Response to Reviewer NTVc (Part 2)

Q3-1&W1: Explaining the Larger BTR Gains on SFT Than GRPO Despite Similar Visual-Probe Performance

Answer: Although the features injected in TimeLens-SFT and TimeLens-GRPO achieve similar visual-probe performance, two factors may explain why BTR yields larger gains on TimeLens-SFT.

1. TimeLens-SFT has more room for improvement. Before applying BTR, TimeLens-SFT achieves a much lower grounding mIoU than TimeLens-GRPO (57.04 vs. 65.39). This lower baseline may partly explain why TimeLens-SFT benefits more from injecting intermediate-layer features.

2. Intermediate-layer features may be more complementary in TimeLens-SFT. We conjecture that GRPO training makes intermediate- and late-layer visual features more similar, reducing the benefit of injecting intermediate-layer features. To examine it, we measured the similarity between visual features at layers L21 and L34 using Centered Kernel Alignment (CKA). The CKA score was smaller for TimeLens-SFT (0.7468) than for TimeLens-GRPO (0.7580). This result suggests that intermediate-layer features provide more complementary information in TimeLens-SFT than in TimeLens-GRPO, which also explain why TimeLens-SFT achieves larger gains with BTR.

Accordingly, we have added a new paragraph titled “Why BTR Gains Differ Between SFT and GRPO” in Appendix C.2 of the revised manuscript (changes marked in blue).


Q3-2: What testable internal difference links these diagnostics to localization errors that revisiting can correct?

Answer: Thank you for this question. Our diagnostics examine how different attention computations affect boundary generation. Specifically, we selectively block attention connections between visual, prompt, and boundary tokens within specific decoder-layer windows and measure the resulting changes in start- and end-boundary token probabilities. Using this diagnostic approach, we compare attention sensitivity before and after BTR on examples where BTR successfully corrects native localization errors. As shown in Appendix C.3 (Fig. 10a), blocking attention from boundary tokens to visual tokens has a stronger effect on both start- and end-boundary predictions after BTR. This indicates that boundary generation becomes more sensitive to visual information after BTR. These results demonstrate a measurable change in how the model uses visual information during boundary generation, providing an internal difference associated with BTR's localization improvements.


W1: Limited explanatory and design value of the analysis.

Answer: We thank the reviewer for this valuable comment. We actually conduct extensive analyses, including attention flow tracing, representation probing, controlled comparisons, and targeted attention interventions, to identify limitations in temporal boundary generation, guide the design of Temporal Revisit, and explain its effects.

1. Identifying limitations in boundary generation. We use Boundary Flow Tracing to examine how attention between video, prompt, and timestamp tokens affects boundary generation (Section 2.1, Fig. 2). We further analyze start/end predictions, localization errors, post-training changes, and different video MLLMs. These analyses highlight potential limitations in the direct use of visual information during boundary generation.

2. Providing direct guidance for method design. Motivated by the above findings, we investigate whether video MLLMs still retain useful temporal information. We further analyze temporal information in visual representations through layer-wise probing, with video-replacement and temporal-shuffling controls (Section 2.2, Fig. 3). These experiments reveal recoverable temporal information in intermediate visual states, directly motivating Temporal Revisit (Section 3) to reuse these features during boundary generation.

3. Explaining and validating the effects of Temporal Revisit. We conduct cross-model and cross-benchmark evaluations, prediction-change analyses, layer-sensitivity experiments, parameter-matched comparisons, and HTR component ablations to evaluate our method. We further use post-revisit Boundary Flow Tracing and targeted attention knockouts to analyze how Temporal Revisit changes boundary generation. Additional experiments assess reliability across training seeds and inference efficiency (Section 4; Appendix C).

We thank the reviewer again for the valuable feedback and hope that the reviewer could reconsider the assessment of our work.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer NTVc,

We sincerely appreciate your constructive feedback and valuable suggestions. In response to your concerns, we have conducted additional experiments to examine the relationship between probe performance and revisit gains, investigated why BTR achieves different gains on SFT and GRPO, and evaluated the reliability of HTR across independent training runs. These additional analyses have helped us further strengthen the manuscript.

With less than 24 hours remaining in the discussion period, we would greatly appreciate any further feedback you may have on our rebuttal. If any concerns remain unresolved, we would be happy to provide further clarification.

All the best,
The Authors

Reviewer_yn9EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses generative video temporal grounding for video MLLMs considering start and end timestamp formation for through route interventions and representation probing. The paper identifies the pathways that support start and end timestamps. Boundary Flow Tracing is used to reveal end point and depth-dependent pathways selected models. Boundary cues are located before timestamp generation. Restricted readouts recover temporal cues from intermediate visual states, while MiMo-VL probes provide additional evidence of boundary information before generation. The resultant observations is used improve native grounding with the design Temporal Revisit, a learned intervention that returns intermediate visual evidence to later visual states of a frozen backbone. Broad Temporal Revisit (BTR) tests whether this evidence can improve native boundary generation. Hypothesis-Guided Temporal Revisit (HTR) further examines whether prediction-conditioned reuse with preservation training can improve grounding while limiting prediction changes.

Critical strengths

The topic is interesting and can be useful in practice. The paper is also quite well presented - it is easy to read and follow and visuals are constructed well. The results seem good and improves on the baselines.

Critical weaknesses

No critical weakness.

Minor issue is that the appendix does some heave lifting, while the main paper can comes of more of a easy to digest technical overview. For example, Fig 9 that give some actual insight into the problem and the outcomes can be in main paper.

Questions for authors

I am positive about the paper, my issue around the construction of the paper cannot really be addressed in the rebuttal - unless there is some plan to give more of the lower level details and results in the main paper - and I do not have any specific technical question.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yn9E — Paper Organization

We sincerely thank the reviewer for the positive assessment of our work and the constructive suggestion, which has helped us improve the presentation and organization of the manuscript.

Q1&W1: Issue Around the Construction of the Paper

Answer: We agree that the original manuscript relied too heavily on the appendix. Following your suggestion, we have moved representative examples in Fig. 9 from the original Appendix to the main paper as Fig. 5 (Section 4.3). Additionally, we have also included more technical details and experimental results in the main paper, including outcome-conditioned Boundary Flow analysis (Appendix B.4→Fig. 2) and training details (Appendix D.3→Section 3.3), as summarized below.

Source locationMain-paper destinationAdded material
Original Appendix Fig. 9Fig. 5 · Section 4.3Representative examples
Appendix B.4Fig. 2 · Section 2.1Boundary Flow analysis
Appendix D.3Section 3.3Training details

These revisions make the main paper more self-contained, while the appendix provides supplementary results and implementation details (changes marked in blue).

Thanks again for your time and constructive suggestion.

Thank you for the response. I think this reorganisation is better. I was already quite positive on my assessment, so will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.