- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how video MLLMs generate the start and end timestamps of an event. Using attention interventions and representation probes, the authors examine where temporal information is retained and which pathways support timestamp generation. They then propose Temporal Revisit, which feeds intermediate visual features into later layers while keeping the original model frozen.
Critical strengths
-
The paper addresses a useful question beyond benchmark performance: how temporal boundaries are formed inside video MLLMs. It carefully distinguishes information that can be recovered from hidden states from information actually used during generation.
-
The analysis motivates a concrete intervention. The authors identify temporal cues in intermediate layers, introduce a module to reuse them, and examine how the intervention changes the model’s dependence on different information pathways.
-
The results include useful controls. BTR adds approximately 270K trainable parameters and improves TimeLens-SFT mIoU by 3.92, 2.85, and 2.19 points across the three benchmarks. Temporal shuffling, same-data LoRA training, and an additional decoding pass provide reasonable checks against alternative explanations. Improvements on other backbones also support the usefulness of the approach.
Critical weaknesses
-
The interventions provide only a partial account of boundary formation. Blocking prompt-to-boundary attention removes access to the query, instructions, and any visual information already incorporated into the prompt states. A drop in the probability of the original timestamp therefore does not necessarily indicate impaired temporal reasoning. Matched random-edge interventions and grounding results after intervention would help clarify what these effects measure.
-
The gains on the stronger post-GRPO model are small. BTR provides little benefit and sometimes reduces performance. HTR improves mIoU by only 0.31, 0.53, and 0.25 points while requiring another inference pass. Multiple training seeds or confidence intervals for the main results would help establish whether these gains are reliable.
-
Several design choices change together. BTR and HTR differ in readout depth, parameter count, training objectives, and inference passes, so their comparison does not isolate the effect of hypothesis-guided retrieval. Parameter-matched adapters and controls that transform late-layer features without reusing intermediate states would also help establish the specific benefit of temporal evidence reuse.
Questions for authors
-
Please clarify whether the observed attention-knockout effects reflect disrupted temporal grounding or reduced access to query and instruction information. Matched random-edge controls and grounding results after intervention would help distinguish these explanations.
-
Please provide confidence intervals or results across multiple training seeds for HTR’s small gains on the post-GRPO model, together with its inference overhead.
-
Please isolate the effects of hypothesis-guided retrieval and preservation training under matched settings. A parameter-matched adapter without intermediate-feature reuse would also help establish the specific contribution of Temporal Revisit.
4 comments
Response to Reviewer 48dC (Part 2)
Q3-1&W3: Isolating the Effects of Hypothesis-Guided Retrieval and Preservation Training under Matched Settings
Answer: Following your suggestion, we compared full HTR with variants that remove hypothesis-guided retrieval, preservation training, or both, using the same TimeLens-GRPO backbone and matched settings on QVHighlights-TimeLens. As shown in the table, each component improves grounding performance on its own, and full HTR with both components achieves the best result. These results show that both components improve performance and suggest that their contributions are complementary.
| Hypothesis Guidance | Preservation Training | mIoU (%) |
|---|---|---|
| ✗ | ✗ | 65.08 |
| ✓ | ✗ | 65.20 |
| ✗ | ✓ | 65.48 |
| ✓ | ✓ | 65.70 |
Accordingly, we have added a new paragraph titled “HTR Component Ablations” in Section 4.5 (Table 4) of the revised manuscript (changes marked in blue).
Q3-2&W3: Parameter-Matched Adapter without Intermediate-Feature Reuse
Answer: Following the suggestion, we compared our BTR using the intermediate-layer features against the counterpart without intermediate-feature reuse. Specifically, the counterpart reads L34 visual features and injects the transformed residuals through a capacity-matched adapter back into L34. As shown in the following table, full BTR outperforms the counterpart without intermediate-feature reuse. The results demonstrate the contribution of temporal revisit intermediate-layer features for video grounding.
| Method | Grounding mIoU (%) |
|---|---|
| BTR (L21→L34) | 60.96 |
| BTR without Intermediate-Feature (L34→L34) | 58.17 |
We have added a new paragraph titled “Parameter-Matched Adapter without Intermediate-Feature Reuse” in Appendix C.4 (Table 22) of the revised manuscript (changes marked in blue).
We appreciate the reviewer's careful evaluation and valuable suggestions, and hope that our additional experiments and analyses have addressed the concerns raised.
Response to Authors
Thank you for the detailed rebuttal and additional experiments. The random-edge control, multi-seed results with confidence intervals, and matched ablations address my main concerns. HTR's gains on the GRPO model remain modest relative to the extra inference pass, which is worth discussing in the main text. I will maintain my score.
Dear Reviewer 48dC,
Thank you for your follow-up and for acknowledging our additional experiments. We appreciate your constructive feedback and agree that the trade-off between HTR's modest grounding improvements and additional inference latency deserves discussion in the main text.
Following your suggestion, we have added a discussion of HTR's grounding gains and inference overhead in Section 4.2 of the revised manuscript(changes marked in blue). We now explicitly discuss the increased inference latency, the minimal additional GPU memory usage, and the limitations of HTR in latency-sensitive applications.
Thank you again for your time and valuable suggestions.
All the best,
The Authors
Response to Reviewer 48dC (Part 1)
We sincerely thank the reviewer for recognizing the value of our mechanistic analysis, the practical gains of Temporal Revisit, and the useful experimental controls. We also appreciate the constructive feedback and guidance that have helped us further improve the paper.
Q1&W1: Clarifying Attention-Knockout Effects on Temporal Grounding and Query/Instruction Access
Answer: We thank the reviewer for this constructive suggestion. First, we would like to clarify that our attention knockouts are applied to specific decoder-layer windows rather than across all layers. Thus, query and instruction information remains accessible through other layers.
In addition, to examine whether attention knockouts disrupt temporal grounding, we evaluate their impact on accurately localized examples. Compared with the no-knockout baseline, targeted P→Y knockout substantially disrupts temporal grounding, demonstrating that the attention knockouts actually affects localization performance.
To further investigate whether this performance drop is simply caused by reduced attention connectivity, we compare targeted P→Y knockout with a matched random-edge that blocks the same number of connections within the same layer window. The random-edge knockout has almost no effect on grounding performance, indicating that the degradation is not merely due to reduced attention connectivity, but more importantly, to the disruption of visual information.
Accordingly, we have added a new paragraph titled “Grounding Effects of Pathway Interventions” in Section 2.1 (Table 1) of the revised manuscript (changes marked in blue).
Q2&W2: Reliability of HTR's Small Gains and Inference Overhead
Answer: Following the suggestion, we trained HTR based on TimeLens-GRPO baseline with three random seeds, keeping all other settings fixed. As shown in the following table, all three runs achieve positive mIoU gains on QVHighlights-TimeLens, with a mean gain of +0.33 mIoU points and a standard deviation of 0.07 points. Additionally, we evaluated paired video-cluster bootstrap to estimate the uncertainty of these gains. Specifically, we computed the 95% confidence intervals based on randomly sampled 2000 pairs, and their confidence intervals of perfromance gain lie entirely above zero for all runs.These results demonstrate the reliability of HTR's gains across independent training runs.
For inference overhead, we evaluated HTR based on Charades-TimeLens using a single NVIDIA H20 GPU. The additional pass for feature revisiting increases average inference latency from 0.75s to 1.34s, while peak GPU memory rises by only 0.06 GiB, from 20.89 GiB to 20.95 GiB. Overall, HTR improves grounding accuracy at the cost of around 0.5s in additional latency, with minimal impact on peak GPU memory.
Accordingly, we have added a new paragraph titled “HTR Reliability Across Training Seeds” and included the inference-cost measurements under “Inference Efficiency” in Appendix C.3 of the revised manuscript (changes marked in blue).