acceptodds
Under review as a conference paper at ICLR 2027

Short-Moment Grounding in VideoLLMs: A Search-Space Limit Rather Than a Capability Limit

Abstract

Video large language models (VideoLLMs) have become the dominant approach to temporal grounding, and their progress is measured by averages over entire benchmarks. We find that these averages hide a near-total failure on short moments, which are rare in standard test sets but often matter most in practice. The failure appears in every model and dataset we examine, and we trace it to two sources: a bias toward predicting spans of a typical length and an imprecise estimate of where the moment lies. The bias is already present before any grounding training, and both sources persist across a wide range of fine-tuning interventions. The same frozen model, however, localizes short moments precisely once its input is narrowed around them, which suggests that the failure lies in how much of the video the model must search rather than in what it can perceive. We turn this finding into a training-free temporal zoom that gives a second, focused look only to moments the model itself predicts to be short, and it substantially improves short-moment grounding. Our results show that VideoLLMs can already do more than their average scores suggest, and that the cheapest progress lies at inference time.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video temporal grounding, video large language models, video moment retrieval, multimodal large language models, benchmarking and evaluation, failure analysis, training-free inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6HgtOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that VideoLLM grounders fail almost completely on moments shorter than two seconds, a failure that benchmark averages hide. Through length-stratified evaluation across three model families and three datasets, it traces the failure to two causes: a length-compression prior already present in the pretrained backbone, which grounding fine-tuning re-centers rather than removes, and a center-placement floor that no fine-tuning intervention evaluated here can move; it then turns the diagnosis into a training-free temporal zoom that gives short predicted moments a second, focused pass. The analysis is backed by deconfounding controls, a scale-fair metric check, an error taxonomy, and a falsification suite spanning corpus, capacity, objective, and architecture, plus a shipped scoring protocol and prediction package. However, the central search-space claim rests on interventions concentrated in one model family, and the deployable lever improves mainly the target stratum with real trade-offs elsewhere.

Critical strengths

S1. An important, previously unmeasured finding established with exceptional rigor. The short-moment collapse is real, large, and practically relevant, and the paper earns the conclusion the hard way: stratified evaluation on three model families and three corpora, deconfounding of video length and query difficulty, a metric-geometry analysis showing the deficit survives a scale-fair metric, and an error taxonomy showing misses are wrong-region errors. The shipped protocol, scorer, reference values, and raw predictions reproduce the diagnostic tables cell by cell, and I verified that the protocol document and code match the paper's stated strata, metrics, populations, and rounding conventions exactly. This sets a standard for evaluation-focused work and is the primary basis for my positive assessment.

S2. The falsification methodology is a model of how to support a negative mechanistic claim. Instead of proposing one fix, the paper rules out the obvious explanations one at a time: rebalancing, speed augmentation, natural-short supply, RL center rewards, reparameterization, and a soft-argmax head, across capacity and seeds. The temporal-zoom analysis then isolates placement of temporal samples as the active ingredient, with oracle and deployed variants and an honest account of what remains unseparated. The mechanism section is far more convincing than the single-intervention studies typical of this genre.

Critical weaknesses

W1. The central "search-space limit" claim is supported by conditioning interventions but not fully factorized, and the interventional evidence is narrow. The authors themselves note that the crop bundles range, distractors, and rescaling, so "search space" is a reading rather than a factorized mechanism; and nearly all intervention arms run on one model family, with only the natural-short arm replicated on a second. For a paper whose contribution is a mechanistic claim about VideoLLMs in general, the generality of the mechanism (not just of the collapse) is the crux, and the evidence there is thinner than for the diagnosis.

W2. The prescriptive payoff is narrower than the diagnosis, and the strongest claims are not covered by the shipped artifacts. The deployed zoom improves the short stratum but costs aggregate accuracy under a frozen threshold, and its gains concentrate where the first pass already suspects a short moment. Moreover, the supplementary package reproduces the diagnostic cells (Tables 1-2 populations and strata) but ships no code or predictions for the fine-tuning arms, the center interventions, or the zoom lever itself, so the paper's most consequential experiments (Tables 3-6) remain at the level of reported numbers.

Questions for authors
  1. Artifact coverage. The supplement reproduces the diagnostic tables but not the intervention arms or the zoom lever. Do you plan to release the fine-tuning configurations, the zoom implementation, and the corresponding predictions, so that the mechanism claims are as checkable as the diagnosis?
  2. Annotation noise versus the center floor. The analysis rests on re-annotated sub-2s boundaries, while the claimed center floor is on the order of one second. What is the inter-annotator agreement or re-annotation precision at this scale, and could annotation noise masquerade as part of the measured floor?
  3. Generality of the mechanism. Do the center floor and the zoom effect transfer to grounders outside the TimeLens lineage (e.g., TRACE or VideoChat-R1 as deployed levers), and to backbones with different timestamp representations or temporal token granularity?
  4. Tokenization as a confound. Could the floor be a consequence of timestamp token resolution or temporal patch granularity rather than search space per se? The frozen readout result suggests not, but a direct probe of internal representations at sub-second offsets would strengthen the claim.
  5. The router gate. The deployed threshold is selected on the reported sets, and the frozen variant loses aggregate accuracy beyond your preset bound. Is there a gate criterion (e.g., confidence-based) that keeps the short-stratum gain without the aggregate cost?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6Hgt (1/3): W1 and W2/Q1

W1: the mechanism is not factorized, and the interventions use one family

The central "search-space limit" claim is supported by conditioning interventions but not fully factorized, and the interventional evidence is narrow. The authors themselves note that the crop bundles range, distractors, and rescaling, so "search space" is a reading rather than a factorized mechanism

We extend the factorization of the crop to six models, taking the factors it bundles in turn.

  • Distractors. A mask that blacks out everything outside the window removes the distractors but keeps every frame, the full range and the original timeline. It lowers the mean center error on all six models in our general response's table: both TimeLens models, their two zero-shot backbones, TRACE (a different family) and VideoChat-R1. On QVHighlights it does so on all five models we ran there (Appendix X, Table A17).
  • Range. The mask keeps the uniform frames and so their full range, and still lowers the center error, so the gain does not need a shorter range. Foveation lowers it on five of the six models (within noise on VideoChat-R1) and on four of five on QVHighlights; the range its frames span narrows as the kernel tightens, so for foveation the two stay bundled. The same kernel aimed 8 s off the moment does worse than uniform sampling (median 5.60 against 1.05 s): what helps is frames on the moment, not a narrower span. With half of the frames held on the moment, where the other half fall does not hurt placement: gathering them all at one far point even lowers the error slightly (−0.42 s; Appendix X).
  • Rescaling. Rescaling time alone does not bring the center closer. Stretching the model's clock 4×, without moving any frame, shortens the answers on all five models we slow but leaves TimeLens-7B's center error where it was, and moves TimeLens-8B's by only 0.09 s, against 1.5 s for the crop.
  • What stays bundled. Inside the crop, range and rescaling still travel with frame concentration, and the revision says so. "Search space" now names the two parts we measure: the competing content and the share of frames on the moment.

nearly all intervention arms run on one model family, with only the natural-short arm replicated on a second

We broaden the input interventions beyond TimeLens; the training replication was already in the submission.

  • Input levers on four more models. Foveation and the crop lower the mean center error on zero-shot Qwen2.5-VL, on TRACE (a time encoder) and on zero-shot Qwen3-VL (native timestamps). On VideoChat-R1 the mask lowers it; foveation helps on most of its clips but not on the mean.
  • Training on a second paradigm, already in the submission. The submitted appendix repeats natural-short training on TimeLens-8B and on VideoChat-R1, an RL-tuned grounder on the same backbone family. On VideoChat-R1 it leaves the median center error above a second, at 1.45 s against 1.95 s before training.
  • The levers as deployed. Placed on each model's own first answer, on the same four models outside TimeLens: see our reply to your Q3.

W2/Q1: a narrower payoff, and artifact coverage

Do you plan to release the fine-tuning configurations, the zoom implementation, and the corresponding predictions, so that the mechanism claims are as checkable as the diagnosis?

Yes. The updated supplementary material holds the code, configurations and predictions for every experiment, with scripts that check every table value the paper prints against the released predictions (see the reproducibility statement).

  • It covers the finetuning configurations (the GRPO recipe included), the center interventions, the zoom, foveation, the router, the scorers and every new run.
  • It ships no videos, weights or annotation text.
  • On the payoff, the new deployment table (our reply to tJUY's Q2) shows where it pays and at what cost, per model and dataset (Appendix Z, Table A21).
  • TimeLens-8B gains on all three TimeLens-Bench sets and lifts overall mIoU there.

its gains concentrate where the first pass already suspects a short moment

Yes, by design. The zoom re-reads only clips whose first answer is shorter than 4 s. Its window lands on 60–61% of sub-2 s Charades moments with the TimeLens models, and on 2–43% with the five other models; where the first answer lies far from the moment, it cannot help (our reply to tJUY's W2). The frozen threshold's aggregate cost is in our reply to your Q5.

AuthorsOpenReview ↗

Response to Reviewer 6Hgt (2/3): Q2 and Q3

Q2: annotation noise against the center floor

What is the inter-annotator agreement or re-annotation precision at this scale

We ran no agreement study of our own; a published re-annotation of Charades places boundaries about a second apart. Annotators who re-marked Charades actions placed their boundaries about a second from the original ones at the median (Sigurdsson et al., ICCV 2017).

  • The re-annotation marks the core of the original events. For the 872 re-annotated pairs whose query matches an original Charades-STA pair, 95.2% of the sub-2 s moments nest inside the original span, and their centers sit a median 0.8 s from the original span's center. This is disagreement between two annotations of different extent, rather than the re-annotation's own noise.

could annotation noise masquerade as part of the measured floor?

Partly: noise can explain part of the floor, but not all of it, and none of the paired gain.

  • The crop bounds the noise below the floor. If all of the crop's 0.65 s were Gaussian label noise on the center, its SD would be at most 0.96 s, and a perfectly placed answer would sit at a median of 0.65 s, not the full video's 1.05 s.
  • Noise cannot create the paired gain. Label noise is shared by both conditions, while the center error falls from 1.05 to 0.65 s on the same labels.
  • Simulated noise leaves the gains intact. We add Gaussian noise to every true center, with SDs up to 2 s. Both paired contrasts hold in all 200 replicates at every level, and a 0.5 s SD moves the floor only from 1.05 to 1.15 s.
  • Most misses land outside the moment. On the full video, 78–83% of sub-2 s misses put their predicted midpoint outside the moment.
  • ActivityNet's original labels show the same collapse, so it does not depend on the re-annotation.

Reference: Sigurdsson, G. A., Russakovsky, O., and Gupta, A. What Actions are Needed for Understanding Human Actions in Videos? ICCV 2017.

Q3: generality outside TimeLens, and as deployed levers

Do the center floor and the zoom effect transfer to grounders outside the TimeLens lineage (e.g., TRACE or VideoChat-R1 as deployed levers)

The floor transfers: it holds on every VideoLLM we tested, now including UniTime at 1.40 s, and the five specialist methods do worse, at 2.0–8.0 s.

The zoom effect transfers as an oracle lever; deployed, it helps some models and not others.

  • As oracle levers, foveation and the crop lower the center error on three of the four outside VideoLLMs and the mask on all four, and the oracle 8 s window lowers it on all nine specialist runs.
  • Deployed, the zoom helps both zero-shot Qwen models, UniTime and RGNet trained on Charades: their mean center error falls and more clips improve than worsen. It does not help TRACE, VideoChat-R1, SnAG, DeCafNet's grounder, ReVisionLLM, CONE or RGNet's MAD model; for CONE and ReVisionLLM's MAD model the mean falls slightly but not clip by clip (Appendix Z, Tables A20 and A21).
  • It fails for two general reasons, and on VideoChat-R1 for a third. A window built from a wrong-region first answer cannot repair it, and that repair is most of the oracle's gain. Inside a window that does hold the moment, TRACE's answer gets no closer.
  • VideoChat-R1 often answers outside the crop. 45% of its crop answers fall outside the window, against 3–7% for TRACE and Qwen3-VL and 28% for zero-shot Qwen2.5-VL. Appendix A now reports this share.

and to backbones with different timestamp representations or temporal token granularity?

The floor appears under four ways of handling time: a time encoder (TRACE), native text timestamps (Qwen3-VL), interleaved text timestamps (UniTime, TimeLens) and whole-second output (TimeLens-8B). Within Qwen2.5-VL, our reply to Q4 varies the patch span; we tested no backbone with a different temporal patch size.

AuthorsOpenReview ↗

Response to Reviewer 6Hgt (3/3): Q4 and Q5

Q4: tokenization as a confound

Could the floor be a consequence of timestamp token resolution or temporal patch granularity rather than search space per se? […] a direct probe of internal representations at sub-second offsets would strengthen the claim.

Our tests find no evidence that either sets the floor. Crossing patch length with the crop on the same clips separates patch granularity from the search space (Appendix X).

  • Patch length does not set the floor. Qwen2.5-VL fuses two frames per patch, so a patch spans 1 s at 2 fps and 0.25 s at 8 fps.
  • On the full video, finer patches barely move the median center error: 1.05 → 0.98 s. Inside the 8 s crop at 2 fps, coarse 1 s patches reach 0.55 s.
  • Nor does timestamp resolution. TimeLens-8B answers in whole seconds and TimeLens-7B in tenths of a second, and both show the floor.
  • Stretching its clock 4× makes its whole-second answers 0.25 s apart in video time. That moves its center by only 0.09 s, while the crop moves it by 1.5 s.
  • The direct probe supports no conclusion. In the original paper, a regression head trained on the frozen model's features kept the median center error at or above 1.8 s. We also ran the suggested direct probe: a linear readout of TimeLens-7B's internal states, patch by patch, at 1 s and 0.25 s patches. Its positive control, recovering the model's own answer, reached an AUROC of 0.70 against our preset bar of 0.75 (0.72 after tuning), so we draw no conclusion from it (Appendix X).

Q5: how the threshold was chosen, and a gate without the aggregate cost

The deployed threshold is selected on the reported sets, and the frozen variant loses aggregate accuracy beyond your preset bound.

The threshold holds out of sample, and its held-out cost is 1.88 mIoU points.

  • Cross-fitted on Charades. We choose the threshold on one half of the videos and test it on the other, both ways, so every clip is held out once. Both halves pick the paper's frozen threshold, and the held-out short gain equals the in-sample one (10.5 points of [email protected]). On QVHighlights the held-out result also equals the in-sample one; on ActivityNet the held-out gain is small, +1.2 and +1.7 points for TimeLens-7B and 8B.
  • Most of the deployment table is out of sample. The same threshold runs unchanged on the five other models and on Ego4D NLQ (our reply to tJUY's Q2).
  • The cost against the bound. Held out, the aggregate cost is 1.88 mIoU points [1.10, 2.67]: at our preset 2-point bound, with the estimate inside it and the interval crossing it. The 2.7-point cost in the paper came from one half alone; Section 6 now reports the cross-fitted cost.

Is there a gate criterion (e.g., confidence-based) that keeps the short-stratum gain without the aggregate cost?

No gate we tried removes the aggregate cost, and confidence adds nothing (cross-fitted).

  • Token confidence is at chance for predicting where the zoom helps: AUROC 0.47–0.51 among short first answers.
  • Adding confidence to the length gate gains 10.7 points of short recall but costs 2.36 mIoU points, beyond the 2-point bound.
  • Gates that check the second answer. Accepting the second answer only when it stays off the window's edges keeps +4.9 points at a cost of 1.47 mIoU points. Requiring agreement with the first answer gains 8.0 points but costs 2.18 mIoU points, beyond the bound (Appendix Z).
Reviewer_tJUYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates short-moment grounding in VideoLLMs through duration-stratified evaluation. It identifies a bias toward typical predicted durations and persistent center-localization errors, and examines these behaviors through training and input-side interventions. Based on improved localization under focused video inputs, it proposes a training-free temporal zoom triggered by the model’s initial prediction.

Critical strengths
  1. Substantial diagnostic value. The paper shows that aggregate scores conceal severe failures on very short events and distinguishes predicting an appropriate duration from locating the event accurately. The analysis across models and datasets makes this a useful contribution to understanding how current grounders fail.

  2. Extensive and informative experimentation. The authors investigate multiple training interventions and control input factors such as sampling density, frame budget, and resolution. The held-out checks, video-level bootstrap analyses, and reporting of negative results strengthen the empirical study. Temporal zoom also provides a practical consequence of the analysis, with improvements in both short-event and aggregate performance for TimeLens-8B on two benchmarks.

Critical weaknesses
  1. The mechanism and training conclusions are stronger than the evidence supports. Improved performance under oracle windows demonstrates that focused inputs can enable finer localization, but does not establish that search-space limitations can be separated from limitations in the model’s ability to select relevant evidence. The input controls are valuable, yet Appendix A itself acknowledges that the mechanism is not fully isolated. The claim of a persistent center-error floor also requires qualification: Section 4 reports that short-event training reduces median center error on ActivityNet from 1.83 to 0.76 seconds. These results support conditional improvements and generalization trade-offs, rather than a general conclusion that training cannot resolve the localization deficit.

  2. The practical benefits are dependent on the model and dataset. With TimeLens-7B, the deployed router improves short-event recall on Charades at a cost to overall mIoU, while held-out strict-recall gains on QVHighlights and ActivityNet remain statistically uncertain. The stronger TimeLens-8B results are encouraging, but the method still depends on the initial prediction placing the target within the refinement window. The paper should state this applicability boundary more prominently and consistently distinguish oracle capability measurements from end-to-end improvements.

Questions for authors
  1. Interpretation: How should the proposed center-error floor be reconciled with the improvement to 0.76 seconds on ActivityNet? Please clarify which conclusions are specific to particular models or transfer settings, and which evidence supports the stronger distinction between search-space and capability limitations.

  2. Applicability: Can the authors consolidate the held-out deployment results by model and dataset, including short-event gains, aggregate performance changes, and additional inference cost? This would clarify where temporal zoom provides a reliable practical benefit and where initial localization remains the limiting factor.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tJUY (1/2): W1/Q1

W1/Q1: the mechanism and training claims, and ActivityNet's 0.76 s

We agree that both claims were stated more strongly than the evidence supports, and the revision narrows them: the mechanism claim to placement, and the training claim to what we tested.

Improved performance under oracle windows demonstrates that focused inputs can enable finer localization, but does not establish that search-space limitations can be separated from limitations in the model's ability to select relevant evidence.

We agree they cannot be fully separated, and the claim is now narrower. The mask and foveation both lower the mean center error on six models, four of them outside TimeLens (table in our general response), though foveation's change on VideoChat-R1 is within noise. Failing to pick the moment among competing content is a limit of the model in the broad sense. Our claim is that this limit can be removed at the input with the weights frozen, which the training we tried does only in distribution and at a cost to long moments. You note that Appendix A leaves the mechanism not fully isolated; the revision now measures part of it: across nine sampling conditions on the same clips, the more of the frame budget lands on the moment, the lower the center error (Appendix X).

How should the proposed center-error floor be reconciled with the improvement to 0.76 seconds on ActivityNet?

The 0.76 s is an in-distribution gain, it costs long moments, and it does not transfer. Trained and tested on ActivityNet, natural-short training reaches what the oracle crop reaches without any training (0.76 against 0.74 s). The price is long moments: [email protected] at 30 s and above falls from 0.229 to 0.092. Tested on Charades instead, the trained model's median center error stays where it was before training, at about a second. The Conclusion now reads "Neither yields to training without a cost elsewhere", and Contribution 2 is stated at the same scope.

Please clarify which conclusions are specific to particular models or transfer settings, and which evidence supports the stronger distinction between search-space and capability limitations.

Most findings hold across models; the full training study is on TimeLens-7B. On every model we tested, strict recall on sub-2 s moments is near zero, and predicted spans are compressed toward a typical length, a bias already present in the raw backbone. Every VideoLLM we tested also shows the center floor, now including UniTime. The training study's key run, natural-short training, is repeated in the submitted appendix on TimeLens-8B and on VideoChat-R1, an RL-tuned grounder; on VideoChat-R1 the median center error falls from 1.95 to 1.45 s but stays above a second. Two conclusions are setting-specific: training lowers the center error only in distribution (above), and the zoom pays on three of the seven models we deploy it on (our reply to your Q2).

Three results could have gone against the distinction and did not. If the floor were a resolution limit, sampling the whole video more finely would lower it; it barely moves (1.05 to 0.98 s). If more frames alone helped, the kernel aimed 8 s off the moment would help too; it does worse than uniform sampling. If training could teach placement, it would lower the floor across corpora; it does not.

AuthorsOpenReview ↗

Response to Reviewer tJUY (2/2): W2 and Q2

W2: the applicability boundary, and oracle against end-to-end

The paper should state this applicability boundary more prominently and consistently distinguish oracle capability measurements from end-to-end improvements.

The boundary is the first pass: a window can only repair what its first answer lets it see. The revision states it in the abstract, contribution 3 and Section 6 ("Coarse-pass reach"), and labels every oracle row.

You are right that TimeLens-7B's gains on QVHighlights and ActivityNet are statistically uncertain; both intervals include zero (Appendix Z, Table A21).

  • The zoom needs a first answer near the moment. On ActivityNet, 426 of the 1,837 windows the zoom re-reads miss the moment entirely.
  • Almost all of the oracle's gain lies out of the zoom's reach (four models outside TimeLens). It comes from clips whose first answer lies in the wrong region: −7.3 to −9.3 s there, on the three models where the oracle helps (Appendix Z).
  • Where the first answer lies within 4 s of the moment, a window built from it recovers most of the oracle's gain on both zero-shot Qwen models and half of it on UniTime.

Q2: consolidate deployment by model and dataset, with cost

Can the authors consolidate the held-out deployment results by model and dataset, including short-event gains, aggregate performance changes, and additional inference cost?

Yes. The revision adds this table for seven models (Appendix Z, Table A21). Each cell gives three changes when the zoom re-reads every clip whose first answer is shorter than 4 s: sub-2 s [email protected] and overall mIoU, in points, and the inference cost relative to one pass. The threshold was never chosen on the five other models or on Ego4D NLQ, so their cells are held out; for the TimeLens models [email protected] is cross-fitted, which changes it only for TimeLens-8B on ActivityNet-TL (+2.7 in-sample).

ModelCharades-TLQVHighlights-TLActivityNet-TLEgo4D NLQReach (Charades)
TimeLens-7B+10.5 / −1.9 / 1.42×+2.2 / 0.0 / 1.10×+1.2 / −0.2 / 1.16×+0.2 / +0.1 / 1.28×61%
TimeLens-8B+15.1 / +1.7 / 1.38×+21.7 / +0.9 / 1.12×+1.7 / +0.5 / 1.21×+0.2 / −0.1 / 1.09×60%
Qwen2.5-VL-7B, zero-shot+0.3 / +1.0 / 1.22×−1.1 / +0.9 / 1.22×−1.0 / +0.2 / 1.23×0.0 / 0.0 / 1.94×27%
Qwen3-VL-8B, zero-shot+3.6 / +0.5 / 1.24×+6.5 / +0.2 / 1.16×+0.6 / −0.2 / 1.22×+0.4 / +0.1 / 1.14×43%
UniTime+0.2 / 0.0 / 1.08×not runnot runnot run14%
TRACE+0.3 / +0.2 / 1.06×+2.2 / +0.2 / 1.15×+0.2 / −0.2 / 1.17×not run6%
VideoChat-R1−0.2 / 0.0 / 1.02×0.0 / 0.0 / 1.00×0.0 / 0.0 / 1.01×not run2%

Reach counts the 590 sub-2 s Charades moments that the zoom re-reads with its 8 s window containing them. UniTime already re-reads above 128 s, so we ran it on Charades only.

  • Where it pays. TimeLens-8B gains on all three TimeLens-Bench sets and lifts overall mIoU on each; TimeLens-7B gains on Charades at a cost of 1.9 mIoU points; zero-shot Qwen3-VL gains on Charades and QVHighlights. These gains' intervals exclude zero (Table A21).
  • Elsewhere it costs little and changes little. Overall mIoU moves by at most a point in the other cells, and rises on zero-shot Qwen2.5-VL. The cost is 1.00–1.42× one pass, except zero-shot Qwen2.5-VL on Ego4D NLQ (1.94×), whose short end-of-clip answers send 94% of queries to the zoom.
  • Reach explains most of the difference. UniTime, TRACE and VideoChat-R1 rarely give a short first answer, and their second answers stay 3–4 s long; zero-shot Qwen2.5-VL is reached more often, but inside the window its answer still lands about 1.5 s from the moment. On Ego4D NLQ the first answer lands a minute or more from a short moment, so the window holds it in under a third of the clips it re-reads (Appendix Y).
  • On the original ActivityNet split the first pass limits it, as you anticipated: sub-2 s [email protected] changes of +0.2 and −1.4 points for TimeLens-7B and 8B (Table A21).

Thank you for the detailed response and additional experiments. The revised claims, which focus specifically on placement error and better delimit the training conclusions, address much of my concern about the mechanism. The explanation of the 0.76 s ActivityNet result and the expanded cross-model deployment results are also helpful. I find the diagnosis interesting and the empirical study thorough. I therefore maintain my rating of 3.

Thank you for your follow-up and for your positive assessment of our work. We appreciate your time throughout the discussion.

Reviewer_fziyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the failure of VideoLLMs in grounding very short video moments, particularly those lasting less than two seconds. Through extensive experiments across multiple models and datasets, they identify two potential factors: a length-compression prior that biases predictions toward moments of typical durations, and a persistent center-localization error that remains difficult to eliminate through fine-tuning. Based on these findings, the authors propose Temporal Zoom, a training-free, two-stage inference strategy that first obtains a coarse prediction and then re-decodes a temporally cropped window around the predicted center.

Critical strengths
  1. The paper identifies an important and overlooked evaluation problem. The paper reveals that existing aggregate metrics can conceal severe failures in grounding very short moments. This is an important problem for practical applications. The proposed length-stratified evaluation provides a useful perspective for assessing VideoLLM grounding capabilities.

  2. Extensive and in-depth empirical analysis. The authors investigate several potential explanations for the observed failure, including temporal-length bias, center-localization errors, annotation and metric effects, and the impact of training data and optimization strategies. The experiments across multiple models and datasets provide useful evidence about the limitations of current VideoLLMs.

Critical weaknesses
  1. Limited methodological novelty. The proposed Temporal Zoom strategy follows the coarse-to-fine paradigm, in which a model first identifies a promising temporal region and then performs more precise localization within that region. This is already a common approach in long-video temporal grounding. For example, VideoLLM-based methods such as UniTime [1] and RevisionLLM [2], as well as non-VideoLLM methods such as CONE [3], have explored coarse-to-fine localization strategies. The paper should more clearly distinguish its method from these existing approaches.

[1] Universal Video Temporal Grounding with Generative Multi-modal Large Language Models [2] RevisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos [3] CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding

  1. Insufficient evidence for the central claim that the problem is a search-space limit rather than a capability limit. The title and central conclusion make a strong distinction between a search-space limitation and a capability limitation. However, the current experiments do not convincingly establish this distinction. As shown in Tables 5 and 6, reducing the temporal search space improves strict recall for moments shorter than two seconds, but the reported R@[email protected] remains only 0.202 under the oracle-window setting on Charades. This is still a low localization accuracy, even when the temporal window is guaranteed to contain the target. The result suggests that reducing the search space helps, but does not establish that the search space is the primary limitation or that the model's grounding capability is not a major bottleneck. Moreover, the cropping intervention changes several factors simultaneously, including the temporal range, the distribution of sampled frames, and the contextual information presented to the model. Table 6 also shows that foveated sampling over the full video can achieve even better center error and strict recall than temporal cropping, suggesting that narrowing the search space is not necessary for the observed improvement. The authors should provide stronger evidence to isolate the proposed mechanism and moderate their conclusion if the experiments cannot establish that search-space limitations dominate capability limitations.

  2. The paper attributes the observed failure to the short absolute duration of the GT moment. However, a potentially more fundamental factor may be the ratio between the target duration and the entire video duration. For example, is grounding a two-second event in a four-second video more difficult than grounding a four-second event in a one-hour video? Slowing down the original video can increase the absolute duration of the GT segment, but will this improve the model's performance?

  3. Insufficient comparison with and discussion of closely related work. The paper lacks a systematic discussion and comparison of prior work on grounding short moments in long videos. Several existing methods have explicitly investigated the challenge of localizing short temporal segments in hour-long videos, including RevisionLLM [2], UniTime [1], RGNet [4], and DeCafNet [5]. These works are particularly relevant because they also address the difficulty of locating brief events within a large temporal search space and employ strategies for efficient coarse-to-fine retrieval and grounding. [4] RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos [5] DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos

  4. The writing needs further improvement and it lacks a systematic related-work discussion. The paper introduces several diagnostic quantities, training interventions, and causal interpretations, but their relationships and respective roles are not sufficiently clear. In addition, the lack of a systematic related-work discussion makes it difficult to understand the paper's position relative to existing coarse-to-fine grounding methods and studies of VideoLLM temporal localization.

Questions for authors
  1. How does Temporal Zoom differ fundamentally from existing coarse-to-fine grounding methods such as UniTime, RevisionLLM, and CONE?

  2. Given that R@[email protected] remains only 0.202 under the oracle-window setting, what evidence supports the claim that search-space limitations, rather than the model's grounding capability, are the primary bottleneck?

  3. Can the authors disentangle the effects of absolute target duration and target-to-video duration ratio? Is grounding a two-second event in a four-second video more difficult than grounding a four-second event in a one-hour video? Slowing down the original video can increase the absolute duration of the GT segment, but will this improve the model's performance?

  4. How does the proposed approach compare with existing methods for short-moment grounding in long videos, particularly RevisionLLM, UniTime, RGNet, and DeCafNet?

5 comments
AuthorsOpenReview ↗

Response to Reviewer fziy (1/3): W1/Q1 and W4/Q4

W1/Q1: how does the zoom differ from coarse-to-fine methods?

The proposed Temporal Zoom strategy follows the coarse-to-fine paradigm, in which a model first identifies a promising temporal region and then performs more precise localization within that region. This is already a common approach in long-video temporal grounding.

You are right that re-reading a window is standard, and the revision says so. Re-reading a window is not our contribution: we contribute the diagnosis of why VideoLLMs miss short moments, and use the zoom as the instrument that tests it on a frozen model. A new main-text Related Work section places it among these methods (Section 7, Table 8), and Appendix B covers the wider literature.

How does Temporal Zoom differ fundamentally from existing coarse-to-fine grounding methods such as UniTime, RevisionLLM, and CONE?

It differs in three ways: it trains nothing, it acts on short videos, where three of these five coarse stages are inactive, and only the model's own short first answer triggers it. The table runs each method under our protocol (Appendix Z gives each run).

MethodTrained weightsCoarse stageCoarse stage active hereFull-video error (s)Oracle-window error (s)
UniTime✓coarse MLLM pass✗1.400.80
ReVisionLLM†✓recursive MLLM✗2.280.91
CONE†✓window filter✓4.240.56
RGNet†✓clip retriever✓4.450.80
DeCafNet†✓sidekick encoder✗2.151.05
TimeLens-7B + zoom✗own first pass✓1.050.65

Center errors are medians on sub-2 s moments of Charades-TimeLens, whose videos last at most 72 s. For the zoom, the full-video value is its own first pass; deployed, it reaches 0.85 s. †Trained on Charades-STA.

Where the coarse stage is inactive, their code shows why: UniTime re-reads only above 128 s, ReVisionLLM's finest level is a 125 s window, and DeCafNet's Charades setup has no sidekick. Inside an oracle window their placement improves, but their spans stay 3–8 s long, so their [email protected] on sub-2 s moments stays near zero (UniTime 0.005, CONE 0).

W4/Q4: comparison with RevisionLLM, UniTime, RGNet and DeCafNet

How does the proposed approach compare with existing methods for short-moment grounding in long videos, particularly RevisionLLM, UniTime, RGNet, and DeCafNet?

Under our protocol none of these methods places sub-2 s moments well. The models that come nearest, the released RGNet and ReVisionLLM, answer under 2 s for most queries, whatever the moment's length, and ground poorly overall. Hour-long video grounding, which their coarse stages target, is outside the scope of this paper: our claim concerns short moments in standard grounding benchmarks, where averages hide the failure, so we compare the methods there. The revision discusses all four in a new main-text Related Work section (Section 7).

MethodWeightsSub-2 s [email protected]Overall mIoU
SnAGreleased, Charades-STA0.0000.292
DeCafNet's groundertrained by us, Charades-STA0.0000.302
CONEtrained by us, Charades-STA0.0000.237
ReVisionLLMtrained by us, Charades-STA0.0000.231
RGNettrained by us, Charades-STA0.0020.254
RGNetreleased, MAD0.0810.098
ReVisionLLMreleased, MAD0.0490.044
ReVisionLLMreleased, VidChapters0.0360.042
UniTimereleased0.0000.416
TimeLens-7B, one pass → zoomreleased0.041 → 0.1460.490 → 0.471
TimeLens-8B, one pass → zoomreleased0.008 → 0.1590.553 → 0.570

Scored on Charades-TimeLens on the 1 s grid (Appendix Z, Table A20): [email protected] over its 590 sub-2 s moments, mIoU over all 3,363 queries. Charades-STA's training split has no moment under 2 s, and its labels differ from TimeLens's, so the overall column understates the specialists trained on it. We read the table as a description of this regime, not as a ranking.

  • None of the four papers reports results below 2 s or by moment length. DeCafNet reports Charades-STA overall only; the others report MAD, Ego4D or VidChapters.
  • Their main benchmarks cannot host a zoom: MAD ships features, not videos.
  • The original Charades-STA test split has no moment shorter than 2 s, so this failure cannot appear on it.
  • In hour-long videos the cost lies in finding the event, which the zoom does not address. In a 64-minute timeline the first answer lands a median 20 minutes from the event, so the window around it holds the event for only 1–2% of clips. Given an 8 s window that does hold it, the answer lands 0.70 s away, as in the clip alone.
AuthorsOpenReview ↗

Response to Reviewer fziy (2/3): W2/Q2

W2/Q2: what supports search space over capability, given 0.202?

Given that R@[email protected] remains only 0.202 under the oracle-window setting, what evidence supports the claim that search-space limitations, rather than the model's grounding capability, are the primary bottleneck?

You are right that 0.202 is low; the revision reports it as an open limit. The claim is about placement: the search space is the dominant and removable part of where the model places a short moment. On the same 590 sub-2 s clips, the oracle window lowers the median center error from 1.05 to 0.65 s with the weights frozen, and [email protected] rises five-fold, to 0.202. The mean center error, which counts far-off answers, falls from 2.64 to 1.21 s with the mask alone (table in our general response). The strict threshold is demanding: on a 1 s moment, IoU 0.7 needs the predicted center within 0.214 s of the truth. Strict recall needs both of the paper's two causes fixed at once: the right length, which the length-compression prior distorts (Section 4), and a center within a fraction of a second, which the center floor prevents (Section 5). Inside the oracle window, 71% of the remaining misses still have the wrong length and 95% a center off by half a second or more (Appendix D); the window narrows both causes but removes neither. At IoU 0.5, recall in the same window nearly doubles, to 0.386.

Moreover, the cropping intervention changes several factors simultaneously, including the temporal range, the distribution of sampled frames, and the contextual information presented to the model.

We separate the context from the distribution of frames, and removing the competing context helps on its own. A mask that blacks out everything outside the window removes the competing content but keeps every frame and the full range. It lowers the mean center error on all six models in the table in our general response, so the gain does not need a shorter range. Foveation changes both, since few of its frames fall outside the window, and lowers it on five; on VideoChat-R1 its change is within noise. For foveation the range stays bundled with that concentration, as the revision states. Where the frames fall also matters: across nine sampling conditions on the same clips, the more of the frame budget lands on the moment, the lower the center error.

Table 6 also shows that foveated sampling over the full video can achieve even better center error and strict recall than temporal cropping, suggesting that narrowing the search space is not necessary for the observed improvement.

Foveation narrows too: it narrows the frame budget rather than the time range. It keeps the whole video but concentrates its frames around the moment, so few fall outside the window: in the 31 s example of our general response, 58 of its 62 frames land inside the 8 s window. That is why it matches or slightly beats the crop in Table 6. The revision sharpens our definition of the search space to name this second part, and the same table tests it: the same kernel aimed 8 s off the moment gives a median center error of 5.60 s, against 1.05 s for uniform sampling. Both rows are oracle placements; placed on the model's own first answer, foveation and the crop change the center error by about the same amount on four models outside TimeLens (Appendix Z, Table A21). Strict recall also depends on how densely the moment is sampled: sampling the whole video four times more finely lifts [email protected] from 0.041 to 0.153 while barely moving the median center error, which is why we state the claim for placement.

The authors should provide stronger evidence to isolate the proposed mechanism and moderate their conclusion if the experiments cannot establish that search-space limitations dominate capability limitations.

For placement the experiments establish it; for boundary precision they do not, and there the revision moderates the conclusion. The mask above isolates the competing content and lowers the mean center error on all six models. The abstract, introduction and conclusion now state the claim for placement only (the abstract sentence is quoted in our general response), and the conclusion calls the search space "the dominant and removable part of the placement error, not of boundary precision". The conclusion and Appendix D report strict recall inside the correct window as an open limit.

AuthorsOpenReview ↗

Response to Reviewer fziy (3/3): W3/Q3 and W5

W3/Q3: absolute length or the ratio; would slowing down help?

Can the authors disentangle the effects of absolute target duration and target-to-video duration ratio?

On TimeLens-7B, placement follows the ratio; absolute length matters only through the geometry of IoU.

  • Around a fixed moment, the error grows with the input (Appendix R). A 1 s moment is held fixed while the input grows from a 4 s window to the full video, and the median center error rises from 0.55 to 1.05 s.
  • A short moment is hard because of how much video surrounds it (17,505 ActivityNet pairs). At a fixed moment length, the center error grows with video duration (elasticity 0.50: a 1% longer video gives about a 0.5% larger error; Appendix Y). Measured against the moment's own length, which is what IoU judges, it also shrinks as the moment gets longer. Placement therefore depends on the moment's share of the video, which is the search space, and narrowing the input is what raises that share.
  • At a fixed ratio, absolute length helps on Charades only. On ActivityNet, longer moments at a fixed ratio are recovered slightly less often (−0.075 log-odds [−0.123, −0.027]), while Charades shows +0.61 log-odds, so the revision presents the paper's threshold–center bound (Prop. 1) as IoU geometry only, not as a prediction that longer moments are found more often (Appendix Y).

Is grounding a two-second event in a four-second video more difficult than grounding a four-second event in a one-hour video?

No: the short event in the short video is far easier. We placed each clip's video at a random point in a timeline of other Charades-TimeLens videos, up to 64 minutes long. For TimeLens-7B, [email protected] is 0.189 for a 2 s event in a 4 s video and 0.000 for a 4 s event in the 64-minute timeline (Table A19). There the answer falls inside the event's own 30 s video only 3% of the time, so the hour's cost lies in finding the event. In the 4 s video, even a constant 2 s answer reaches an [email protected] of 0.31–0.37, because the model's answers there run about 1.2 s for a 2 s event. That case tests the answer's width, which our search-space claim does not cover, rather than search.

  • The collapse holds on natural long videos (Ego4D NLQ, 4,338 queries on 8-minute videos). Under 2 s, TimeLens-7B and 8B reach an [email protected] of 0.003 and 0.000, against 0.07–0.17 at 5 s and longer. Their first pass lands about a minute from a short moment (Appendix Y).

Slowing down the original video can increase the absolute duration of the GT segment, but will this improve the model's performance?

No. Neither a 4× slower clock, on five models, nor a 4× slower video, on TimeLens-7B, improves performance over the whole benchmark. It shortens the answers, which raises [email protected] below 2 s and lowers it at 5 s and longer (Section 5.3, Table 7; Table A18).

  • A slower clock shortens the answers but does not bring the center closer (five models). A 4× clock multiplies every timestamp the model reads by four and leaves the frames unchanged; TimeLens-7B's center error stays where it was (+0.14 s [−0.01, +0.29]). Its median answer shortens from 2.95 to 2.05 s, which raises sub-2 s [email protected] by 8.3 points (Table A18).
  • It gains no overall mIoU on any model. Over all 3,363 queries, the 4× clock leaves TimeLens-7B's mIoU unchanged (−0.6 points [−1.4, +0.1]) and lowers it by 3.6 to 9.7 points on the other four models. Actually slowing the video, 4× slower at 2 fps, costs TimeLens-7B 4.7 mIoU points overall and helps only below 2 s.

W5: writing and related work

The paper introduces several diagnostic quantities, training interventions, and causal interpretations, but their relationships and respective roles are not sufficiently clear.

The revision adds a main-text Related Work section (Section 7) and a one-paragraph map of what each diagnostic measures and which failure it isolates (Appendix C.2), pointed to from Section 2.1.

  • The length slope β, from regressing predicted on true length, measures length calibration: 1 is calibrated, below 1 compressive. Emission measures whether short spans are proposed at all.
  • The center error measures placement. The threshold–center bound (Prop. 1) states how close the center must be for [email protected]. Wrong-region misses count answers whose midpoint falls outside the moment.
  • Each quantity is tied to the section that uses it, and a duplicated Introduction paragraph is removed.

The training interventions test whether training moves either the length prior or the center error, and the input levers, with the weights frozen, test the search-space reading. Section 7 places the zoom among coarse-to-fine methods, and Appendix B places the diagnosis among VideoLLM grounders, audits of what grounding benchmarks measure, and earlier work on short moments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for close and careful readings. All three found the diagnosis useful. Below we answer the concerns that more than one reviewer raised, then list what the revision adds. The revised PDF marks every change in blue, and each reply gives the numbers for its reviewer's points.

Search space or capability? (fziy, tJUY, 6Hgt) fziy asks for evidence that search-space limitations dominate capability limitations. The added experiments show this for placement: the search space is the dominant and removable part of where a frozen VideoLLM places a short moment. We define that search space by its two parts: the content that competes with the moment, and the share of a fixed frame budget that reaches it. In a 31 s video with a 1 s moment, uniform sampling at 2 fps puts 2 of 62 frames on the moment. A mask that blacks out the 46 frames outside an 8 s window removes the competing content and leaves those 2; foveation moves the frames toward the moment, putting 12 of them onto it. The table below gives the center error under each lever on the same 590 sub-2 s Charades clips, as uniform → mask → foveation (Section 5.3, Table 7; VideoChat-R1 and the intervals in Appendix X, Table A17). The median shows the typical answer; the mean also counts far-off answers. The mask lowers the mean on all six models and the median on five; foveation lowers them on five and four (its mean change on VideoChat-R1 is within noise). TimeLens-8B's whole-second answers keep its median at 0.50 s under every input.

ModelMedian (s)Mean (s)
TimeLens-7B1.05 → 0.85 → 0.602.64 → 1.21 → 0.99
TimeLens-8B0.50 → 0.50 → 0.502.26 → 0.92 → 0.71
Qwen2.5-VL-7B, zero-shot3.45 → 2.45 → 1.655.65 → 3.96 → 2.38
Qwen3-VL-8B, zero-shot1.25 → 1.00 → 0.852.83 → 1.29 → 1.24
TRACE2.00 → 1.50 → 2.005.18 → 1.91 → 2.70
VideoChat-R13.95 → 3.55 → 3.455.01 → 4.05 → 4.65

The claim is about placement. An oracle 8 s window around the moment lowers TimeLens-7B's median center error from 1.05 to 0.65 s. Removable means at the input: an oracle window removes much of this error, and the deployed zoom removes part of it only where the model's own first answer lands near the moment.

Does it hold beyond TimeLens, and how do other methods compare? (fziy, 6Hgt) The center floor now holds on every VideoLLM we tested, including UniTime; the specialist methods fziy names, run under our protocol, do worse still, with median center errors of 2 s or more. The levers that lower the center error also work beyond TimeLens (table above). None of them places sub-2 s moments well, and the four papers fziy asks about report no results by moment length (Section 7, Table 8; Appendix Z, Table A20).

What the revision adds.

  • The claim, scoped in the paper itself: the abstract, introduction and conclusion now state it for placement. The abstract reads: "The same frozen model, however, localizes short moments far more precisely once its input is narrowed around them, which suggests that its placement error lies mainly in how much of the video the model must search rather than in what it can perceive."
  • The search-space claim: the mask lowers the center error on every model and dataset we test, and foveation on all but one model per dataset; with the frames on the moment held fixed, where the others fall does not hurt placement (Appendix X, Table A17).
  • Length and long videos: on all five models we slow, a slower clock raises strict recall below 2 s and lowers it on longer moments; on ActivityNet a short moment's placement depends on its share of the video; in hour-long timelines and on Ego4D NLQ the cost lies in finding the event (Appendix Y, Tables A18 and A19).
  • Other methods: we run UniTime, SnAG and the specialist methods fziy names, nine runs in all, under our protocol (Section 7, Table 8; Appendix Z, Table A20).
  • Deployment and cost: the zoom on seven models and up to five benchmarks, held out, at 1.0–1.4× the first pass's cost on TimeLens-Bench (Appendix Z, Table A21).

Also in the revision: a main-text Related Work section (Section 7); more precise training and analysis details throughout; and supplementary code, configurations and predictions, with scripts that check every table value against them.

Dear Reviewers,

Please kindly review the authors' rebuttal and engage in discussions if you have further questions.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.