acceptodds
Under review as a conference paper at ICLR 2027

When to Stop: Training-Free Video Temporal Grounding via Endpoint Activation Steering

Abstract

Video Temporal Grounding (VTG) aims to localize the start and end times of queried events in untrimmed videos. Despite substantial progress in event understanding, video large language models still produce unreliable timestamps. Existing improvements often rely on post-training with temporal annotations or additional inference, incurring annotation or computational costs. Contrastive activation steering offers a complementary training-free approach, but applying it to timestamp generation requires identifying the temporal components within activation differences that influence numerical boundary decisions. To address this question, we investigate how video evidence contributes to timestamp generation. We find that temporal information persists during generation, yet visual evidence provides weaker discrimination among end-time candidates than among start-time candidates. Models can recognize an event and locate its onset while still struggling to determine when it terminates. To identify the temporal evidence influencing this decision, we use endpoint-score Jacobians to locate directions that alter end-time preferences and project activation differences induced by temporal perturbations onto these directions. This construction selects components according to native endpoint sensitivity rather than activation differences alone. Aggregating the projected components across samples yields a low-dimensional subspace in late decoder layers. Causal interventions further establish that this subspace mediates the influence of temporal evidence on end-time prediction. Building on this intervention interface, we propose training-free endpoint activation steering. Using unlabeled samples, we extract a shared direction within the subspace and calibrate its relative strength offline for each model–dataset pair. At inference, we apply a single residual update immediately before the first end-time numeric token, preserving the generated start and native decoding without parameter updates or additional video forward passes. Evaluated across seven frozen backbones on TimeLens-Bench, our method improves mIoU by up to 3.62 points. Across the 4B and 8B variants of Qwen3-VL, InternVL3.5, and TimeLens2, it consistently improves both mIoU and [email protected] on all three datasets, with average gains of 2.19 and 1.46 points respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Activation steering, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4vRbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper finds that the video helps video VLMs choose the start time more than the end time. It then identifies a small set of directions in a late layer that affects the end-time prediction, and shows this with causal interventions. For deployment, the method averages the projected differences over 256 unlabeled samples to get one direction for each model and dataset. At inference, it adds this direction once to the residual stream before the first end-time token. On TimeLens-Bench, the method improves performance for six backbones by about 2 mIoU on average.

Critical strengths

A careful causal analysis. The paper tests the endpoint subspace with various steering tricks and analysis methods. The finding that the video helps the start time more than the end time is interesting for the design of grounding models.

Critical weaknesses
  1. The paper is very difficult to read, which is my main concern. The text uses many long, coined noun phrases without definition, for example "Jacobian-guided low-rank projection framework", "canonical model-own-start prefix" and "readout-aligned temporal contrasts". Key notation is defined only once (P0 and P2 in Sec. 3), P1 is never defined, and abbreviations such as MAE are never expanded. Most paragraphs in Sec. 3 and 4 contain several undefined terms in a row. As a result, I could not follow the basic argument without the appendix.
  2. The Qwen3-VL baselines are much weaker than published results, and TimeLens is not compared. On the same benchmarks, TimeLens2 (Zhu et al., 2026, arXiv:2607.17423, Table 1) reports native mIoU of 47.2 / 48.0 / 59.4 for Qwen3-VL-8B on Charades- / ActivityNet- / QVHighlights-TimeLens, and 49.4 / 50.7 / 62.3 for Qwen3-VL-4B. Table 2 of this paper reports only 34.48 / 38.43 / 47.24 and 41.41 / 39.81 / 54.16. Even after steering, Qwen3-VL-8B (36.52 / 39.92 / 49.19) stays about 8 to 11 mIoU below the published native results. In contrast, the TimeLens2 baselines in Table 2 agree with the published numbers within about 0.5 mIoU, so the test split alone is unlikely to explain the gap.
  3. The test split is small, and the main results have no confidence intervals. The method uses 15% of the videos of each benchmark for testing (App. D.1), which gives a few hundred queries per dataset. The main gains are approximately 1 to 3 points when absolute performance is at 60~70 scale, i.e., minimal.
  4. The method needs in-domain calibration data. Each model and dataset needs its own unlabeled calibration set. When the calibration comes from a different dataset, most of the gain disappears (Table 17). This limits the practical value of the training-free claim, e.g, when a random video is provided.
  5. The deployed update does not use the evidence of the current video. The paper motivates the method as restoring temporal evidence from the video (Sec. 3.3). But at test time, it adds one fixed vector, averaged over the calibration set, to every query of a dataset. This vector does not depend on the current video; only its size scales with the hidden state. Thus, it acts more like a fixed correction than like evidence from the video. The paper does not show what this correction does to the predictions, for example whether it moves end times later or makes intervals longer.
  6. The method cannot fix a wrong start time. The method changes only the end time and keeps the generated start. The paper does not report the start-time error of the native predictions, so it does not show how often the start is wrong. Also, the gains do not seem to come only from better end times. Queries that the native model misses completely (IoU = 0) also gain (Table 14), although their wrong start stays. With the ground-truth start, the gains for the 8B models are generally smaller (Table 5) than with the model's own start (Table 2). This makes the usability of the proposed method doubtful.
Questions for authors
  • Can you define all terms and notation at first use (for example P0, P1, P2, MAE and "canonical model-own-start prefix"), and describe the method in plain words before the formal construction ?
  • Does the analysis in Sec. 3 hold for other benchmarks such as QVH-TimeLens, Activitynet-TimeLens , and Ego4D-NLQ?
  • Why are your Qwen3-VL baselines about 8 to 13 mIoU below the results that TimeLens2 reports on the same benchmarks? Can you report native and steered results with the standard settings on the full TimeLens-Bench, and include TimeLens-8B?
  • Can you give confidence intervals or paired significance tests for Table 2?
  • What if the start timestamp is also wrong?
  • How would you handle a random video during real-life scenarios with your method?
16 comments
AuthorsOpenReview ↗

Clarity, notation, and method overview

We sincerely thank the reviewer for carefully reading our manuscript and for the thoughtful, constructive feedback. We appreciate the time and effort devoted to reviewing our work.

We have revised the Introduction and Sections 3–4 to explain the purpose of each operation before its mathematical construction, define the notation at first use, and clarify the prefix protocols in Appendix A.1. Revised text and figure captions are highlighted in red.

Method overview before the equations. Section 3.3 now gives a concise five-step overview: compare original/rolled-video states under a common native prefix; filter each contrast through its own query's endpoint-sensitive directions; aggregate the filtered components into a model-specific subspace; calibrate a shared direction and strength on unlabeled target-dataset samples; and apply one update before end-time generation. The first four steps are offline. The following paragraphs explain the roles of endpoint-score gradients, query-specific projection, and singular value decomposition (SVD). Section 4 distinguishes the fixed subspace from the calibrated direction and presents direction calibration, strength calibration, and deployment separately. Algorithm 1 follows the same sequence.

Definitions at first use. Section 3 now uses PansP_{\mathrm{ans}} for the position before the first answer token and PendP_{\mathrm{end}} for the position before the first end-time numeric token, replacing P0P_0 and P2P_2. These names directly identify the two positions. We also define the prefix, diagnostic probe, and mean absolute error (MAE), including normalization by video duration.

Canonical versus native prefixes. “Canonical” means standardized formatting of the model's own full-video-predicted start. For example, an illustrative native answer The relevant interval is [[8.24, 14.6]]. gives the canonical prefix [[8.2, , whereas the native prefix is The relevant interval is [[8.24, . Neither the predicted end nor ground-truth timestamps are included in the canonical prefix. A diagnostic probe uses the final prefix token's hidden state to predict the annotated end. In the original Charades diagnostics, full-video, no-video, and rolled-video probe controls reuse the same canonical prefix, fixing the start and formatting across conditions. Subspace construction, causal interventions, and deployed steering retain the exact native token prefix, including the generated start and original formatting. Section 3.1 states this distinction; Appendix A.1 specifies the original protocol and the native-prefix controls used in the added cross-dataset experiments.

Locations in the revised manuscript: Introduction (p. 2); definitions and probe protocol in Sections 3–3.1 (p. 3); method overview and construction in Section 3.3 (pp. 4–6); calibration and deployment in Section 4 (pp. 6–7); prefix protocols in Appendix A.1 (pp. 13–14); and Algorithm 1 (p. 19). These presentation revisions preserve the original results.

AuthorsOpenReview ↗

Experimental coverage

Choice of TimeLens2. We prioritized TimeLens2 because its published evaluation reports stronger temporal grounding performance than TimeLens at the same 8B scale (TimeLens2, Table 1). This choice tests whether our training-free intervention can further improve a strong model already specialized through temporal-grounding post-training. Our evaluation includes both TimeLens2-4B and TimeLens2-8B. In Table 2, steering improves both mIoU and [email protected] for both sizes across all three datasets. On TimeLens2-8B, the mIoU gains are +1.83, +1.08, and +0.84 points on Charades-TimeLens, QVHighlights-TimeLens, and ActivityNet-TimeLens, respectively. These results support the complementarity of endpoint activation steering with temporal post-training.

Does the Sec. 3 analysis extend to other benchmarks? We extended the core diagnostics to QVHighlights-TimeLens (QVH) and ActivityNet-TimeLens (AN), using Qwen3-VL-4B, TimeLens2-4B, and InternVL3.5-4B. The key findings persist across all six settings.

Temporal information remains decodable. Separately fitted probes retain end-time information at PendP_{\mathrm{end}}, whereas transferring an unchanged PansP_{\mathrm{ans}} probe increases normalized MAE in every setting. With Full/No-video/Roll probes fitted independently under the same native PendP_{\mathrm{end}} prefix, Full reduces normalized MAE by 0.04580–0.05576 relative to No-video and 0.00413–0.01969 relative to Roll. All 12 paired differences have 95% CIs above zero.

Start/end asymmetry persists. Within each model and comparison, we pair start and end on common eligible queries. Full uses mean numeric-token NLL; Visual uses the candidate-level cost difference Cfull−CqueryC_{\mathrm{full}}-C_{\mathrm{query}}. For each query and boundary, we compute Spearman correlation between cost and distance to the annotated boundary, then average the within-query difference Δρi=ρi,s−ρi,e\Delta\rho_i=\rho_{i,s}-\rho_{i,e}. The 95% CIs use 10,000 video-level bootstrap replicates with query pairing preserved. Positive gaps indicate stronger start discrimination.

DatasetModel (4B)Full start ρˉ\bar\rhoFull end ρˉ\bar\rhoFull Δρ‾\overline{\Delta\rho} [95% CI]Visual Δρ‾\overline{\Delta\rho} [95% CI]
QVHQwen3-VL0.505330.04468+0.46065 [0.35498, 0.56632]+0.107 [0.019, 0.195]
QVHTimeLens20.101100.07150+0.02960 [0.025060, 0.19460]+0.054 [0.005, 0.103]
QVHInternVL3.50.332270.09678+0.23549 [0.15209, 0.31889]+0.025 [0.002, 0.048]
ANQwen3-VL0.576010.43278+0.14323 [0.06342, 0.23042]+0.408 [0.291, 0.526]
ANTimeLens20.593250.58166+0.01159 [0.00040, 0.02280]+0.160 [0.043, 0.277]
ANInternVL3.50.392810.21414+0.17867 [0.05948, 0.30771]+0.255 [0.101, 0.409]

The visual start-minus-end gap is positive in all six settings, with every 95% CI excluding zero. Thus, weaker visual support for end-boundary discrimination extends beyond Charades-TimeLens.

Ego4D-NLQ. Due to time constraints, we were unable to complete the Ego4D-NLQ diagnostics within the rebuttal time window. The additional experiments above cover the other two benchmarks suggested by the reviewer, and we will consider including Ego4D-NLQ in future work.

Concurrent evidence. FinishStrong: Learning to End Well after Imperfect Start in Video Temporal Grounding, posted on arXiv on November 1, 2026, also reports systematically higher end MAE than start MAE across the evaluated model families (Fig. 1a; Appendix F). This concurrent observation supports the broader relevance of endpoint asymmetry. FinishStrong develops post-training for imperfect start prefixes; our work contributes a visual-evidence and activation-level diagnosis together with a training-free intervention in frozen models.

Changes in the revised manuscript. Section 3 (p. 3) now states the extended diagnostic scope. Section 3.1 (p. 3) summarizes the cross-dataset readout findings, and Figure 2(c) with Section 3.2 (p. 4) presents the visual-support gaps across all three datasets. Appendix A.1, Tables 7–8 (p. 14), gives the added readout and native-prefix video controls; Appendix A.2, Table 10 (p. 16), gives the full boundary comparisons and CIs, with the paired protocol on p. 15. Section 2 (p. 2) now cites FinishStrong. Revised text, tables, and figure captions are highlighted in red.

AuthorsOpenReview ↗

OMTG multi-interval boundary diagnostics

We extended the boundary-discrimination analysis in Sec. 3.2 to multi-interval queries from the RL subset (train split) of OMTG-56K, using frozen TimeLens2-4B. This adds a multi-interval setting to our cross-dataset diagnostics. We sampled 128 video sources across duration groups, selecting the first annotated event and one subsequent event per query. The analysis includes 206 eligible events from 125 sources. To compare the same event across conditions, we fixed preceding ground-truth intervals and the current interval's other ground-truth boundary.

We scored matched start/end candidates with and without video using the mean NLL of the moved boundary's numeric tokens. For each candidate, we computed the incremental visual cost Cvisual=Cfull−CqueryC_{\mathrm{visual}}=C_{\mathrm{full}}-C_{\mathrm{query}}, then its Spearman correlation ρ\rho with distance to the annotated boundary. Higher ρ\rho indicates stronger preference for candidates closer to that boundary. We computed start−end differences within events, averaged events within each source, and weighted sources equally. (Pointwise 95% CIs use 10,000 source-level paired bootstrap replicates.)

All three rows below use the same 206 events. The visual-contribution row measures the ranking of candidate-level cost differences.

Scored informationStart ρ\rho ↑End ρ\rho ↑Start−end gap [95% CI]
Video + query0.7510.708+0.043 [-0.008, 0.093]
Query only0.1150.251-0.136 [-0.227, -0.041]
Visual contribution0.6880.615+0.073 [0.016, 0.131]

Adding video raises the start correlation from 0.115 to 0.751 (+0.636) and the end correlation from 0.251 to 0.708 (+0.457). Under target-boundary scoring, the visual start−end gap is +0.073 [0.016, 0.131], reproducing the direction of our Sec. 3.2 finding.

With the second predefined scoring rule, which includes both start and end digits of the current interval, the overall visual gap is +0.012 [-0.047, 0.070]. For first intervals, the visual gaps are +0.115 [0.019, 0.213] and +0.121 [0.025, 0.221] under the two scoring rules, respectively.

These results extend the endpoint-focused diagnosis to multi-interval examples under fixed ground-truth history. Video evidence supports both boundaries, with stronger start discrimination under boundary-specific numeric scoring; the first-interval results agree across both scoring rules. The positive target-boundary gap supports the relevance of endpoint asymmetry beyond single-interval queries.

Reviewer_JznMOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how frozen video language models use temporal evidence when generating event endpoints. It identifies weaker visual support for end-time than start-time discrimination and constructs an endpoint-sensitive subspace by projecting temporal activation differences onto query-specific Jacobian directions. Unlabeled calibration produces a shared steering direction, applied once before end-time generation. Experiments report improvements across seven backbones and three datasets, supported by intervention studies, component ablations, and inference-cost measurements.

Critical strengths
  1. Targeted method design with convincing component validation. Filtering temporal contrasts through each query’s endpoint sensitivity provides a concrete refinement of activation steering. The comparisons against temporal-only, gradient-only, and shuffled-pairing constructions directly test this design. Additional controls on direction, magnitude, and intervention position strengthen the evidence that the gains depend on the proposed construction rather than arbitrary activation changes.

  2. Consistent improvements with a lightweight online intervention. Both mIoU and [email protected] improve across all 18 settings involving the Qwen3-VL, InternVL3.5, and TimeLens2 variants, with an average mIoU gain of 2.19 points. Improvements on temporally specialized models suggest complementarity with post-training. The single intervention preserves the generated start and requires no additional online video forward pass, making the method practically interesting.

Critical weaknesses
  1. Calibration dependence and evaluation scope limit the generality of the results. Each model–dataset pair requires separate calibration, and transferring calibration between datasets loses 1.15–1.90 mIoU relative to target-specific calibration. This is substantial compared with the reported gains. The evaluation also uses a new 70/15/15 video split, so absolute scores are not directly comparable with published full-benchmark results. The matched native-decoding comparisons remain informative, but the paper should more clearly delimit its deployment assumptions and external comparability. The large gap over FV-Action is less compelling because that baseline performs substantially below native decoding.

  2. The mechanistic interpretation requires qualification. The removal and restoration experiments support a causal influence of the selected activation components on endpoint predictions. However, they do not fully explain why visual support differs between start and end boundaries, or establish how much of this asymmetry arises from decoding order and conditioning on an already generated start. The paper should distinguish its well-supported finding of an effective intervention interface from a broader explanation of endpoint-generation failures.

Questions for authors
  1. Calibration and evaluation: Does the reported offline cost include Jacobian extraction and subspace construction, or only shared-direction calibration? Please also clarify which baseline results use the identical held-out split and preprocessing, particularly the additional comparisons in Appendix E.5.

  2. Mechanistic interpretation: What evidence separates an endpoint-specific weakness from effects of generation order and conditioning on the start timestamp? Please clarify the scope of the causal conclusions supported by the current controls.

  3. Scope discussion: Please clarify whether the intervention is intended only for single-interval outputs. A brief discussion of Towards One-to-Many Temporal Grounding (OMTG, ICML 2026) would help delineate its applicability: with multiple intervals, modifying an earlier endpoint may affect subsequent predictions and event-count consistency. This is a discussion suggestion.

Reviewer_8WVbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies endpoint prediction in frozen video language models. It finds that temporal information remains decodable during generation, while video evidence supports end-time discrimination less strongly than start-time discrimination. The proposed method projects each full-versus-temporally-rolled activation contrast onto that query’s endpoint-score Jacobian subspace, then aggregates the filtered contrasts into a model-specific low-rank basis. Unlabeled target-dataset calibration produces a shared direction and relative strength. At inference, one residual update is applied before the first end-time numeric token. The method preserves the generated start, keeps model weights frozen, and improves temporal grounding without an additional online video forward pass.

Critical strengths

I think the paper has the following strengths:

  1. I think it is a good way to build the connection between temporal activation and the final predicted numerical endpoint outputs. Table 4 provides detailed ablations across different baselines under different training-free injections. These controls provide relevant evidence that the proposed pairing contributes beyond generic low-rank filtering.

  2. This paper has extensive ablations to show the effectiveness of the proposed method in Table 3 and Table 4.

  3. The proposed method can achieve consistent improvements over major VLMs like Qwen3-VL, InternVL3.5, and TimeLens2 and across different datasets.

Critical weaknesses

However, I still have some concerns about the paper:

  1. The performance boost over the closest endpoint-gradient-only is minor. The endpoint-gradient-only baseline already achieves 43.26, 55.86, and 42.01 mIoU, compared with 43.93, 56.53, and 42.73 for the full method in Table 4. Thus, the additional query-specific contrast–Jacobian filtering contributes 0.67–0.72 points on the tested Qwen3-VL-4B backbone.

  2. The author only shows that the endpoint is less convincing. However, I think the start point is also important in temporal video grounding. I am somewhat curious: can the proposed method also be applied to the start point? The onset and offset should have the same weights in the evaluation.

Questions for authors

My question is mostly about the weaknesses site.

The most important question is: why is the onset different from the offset? Is it truly from the modeling (current VLM) side, or is it from the labeling part (annotation noise or other parts?)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.