- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that VideoLLM grounders fail almost completely on moments shorter than two seconds, a failure that benchmark averages hide. Through length-stratified evaluation across three model families and three datasets, it traces the failure to two causes: a length-compression prior already present in the pretrained backbone, which grounding fine-tuning re-centers rather than removes, and a center-placement floor that no fine-tuning intervention evaluated here can move; it then turns the diagnosis into a training-free temporal zoom that gives short predicted moments a second, focused pass. The analysis is backed by deconfounding controls, a scale-fair metric check, an error taxonomy, and a falsification suite spanning corpus, capacity, objective, and architecture, plus a shipped scoring protocol and prediction package. However, the central search-space claim rests on interventions concentrated in one model family, and the deployable lever improves mainly the target stratum with real trade-offs elsewhere.
Critical strengths
S1. An important, previously unmeasured finding established with exceptional rigor. The short-moment collapse is real, large, and practically relevant, and the paper earns the conclusion the hard way: stratified evaluation on three model families and three corpora, deconfounding of video length and query difficulty, a metric-geometry analysis showing the deficit survives a scale-fair metric, and an error taxonomy showing misses are wrong-region errors. The shipped protocol, scorer, reference values, and raw predictions reproduce the diagnostic tables cell by cell, and I verified that the protocol document and code match the paper's stated strata, metrics, populations, and rounding conventions exactly. This sets a standard for evaluation-focused work and is the primary basis for my positive assessment.
S2. The falsification methodology is a model of how to support a negative mechanistic claim. Instead of proposing one fix, the paper rules out the obvious explanations one at a time: rebalancing, speed augmentation, natural-short supply, RL center rewards, reparameterization, and a soft-argmax head, across capacity and seeds. The temporal-zoom analysis then isolates placement of temporal samples as the active ingredient, with oracle and deployed variants and an honest account of what remains unseparated. The mechanism section is far more convincing than the single-intervention studies typical of this genre.
Critical weaknesses
W1. The central "search-space limit" claim is supported by conditioning interventions but not fully factorized, and the interventional evidence is narrow. The authors themselves note that the crop bundles range, distractors, and rescaling, so "search space" is a reading rather than a factorized mechanism; and nearly all intervention arms run on one model family, with only the natural-short arm replicated on a second. For a paper whose contribution is a mechanistic claim about VideoLLMs in general, the generality of the mechanism (not just of the collapse) is the crux, and the evidence there is thinner than for the diagnosis.
W2. The prescriptive payoff is narrower than the diagnosis, and the strongest claims are not covered by the shipped artifacts. The deployed zoom improves the short stratum but costs aggregate accuracy under a frozen threshold, and its gains concentrate where the first pass already suspects a short moment. Moreover, the supplementary package reproduces the diagnostic cells (Tables 1-2 populations and strata) but ships no code or predictions for the fine-tuning arms, the center interventions, or the zoom lever itself, so the paper's most consequential experiments (Tables 3-6) remain at the level of reported numbers.
Questions for authors
- Artifact coverage. The supplement reproduces the diagnostic tables but not the intervention arms or the zoom lever. Do you plan to release the fine-tuning configurations, the zoom implementation, and the corresponding predictions, so that the mechanism claims are as checkable as the diagnosis?
- Annotation noise versus the center floor. The analysis rests on re-annotated sub-2s boundaries, while the claimed center floor is on the order of one second. What is the inter-annotator agreement or re-annotation precision at this scale, and could annotation noise masquerade as part of the measured floor?
- Generality of the mechanism. Do the center floor and the zoom effect transfer to grounders outside the TimeLens lineage (e.g., TRACE or VideoChat-R1 as deployed levers), and to backbones with different timestamp representations or temporal token granularity?
- Tokenization as a confound. Could the floor be a consequence of timestamp token resolution or temporal patch granularity rather than search space per se? The frozen readout result suggests not, but a direct probe of internal representations at sub-second offsets would strengthen the claim.
- The router gate. The deployed threshold is selected on the reported sets, and the frozen variant loses aggregate accuracy beyond your preset bound. Is there a gate criterion (e.g., confidence-based) that keeps the short-stratum gain without the aggregate cost?
5 comments
Response to Reviewer 6Hgt (2/3): Q2 and Q3
Q2: annotation noise against the center floor
What is the inter-annotator agreement or re-annotation precision at this scale
We ran no agreement study of our own; a published re-annotation of Charades places boundaries about a second apart. Annotators who re-marked Charades actions placed their boundaries about a second from the original ones at the median (Sigurdsson et al., ICCV 2017).
- The re-annotation marks the core of the original events. For the 872 re-annotated pairs whose query matches an original Charades-STA pair, 95.2% of the sub-2 s moments nest inside the original span, and their centers sit a median 0.8 s from the original span's center. This is disagreement between two annotations of different extent, rather than the re-annotation's own noise.
could annotation noise masquerade as part of the measured floor?
Partly: noise can explain part of the floor, but not all of it, and none of the paired gain.
- The crop bounds the noise below the floor. If all of the crop's 0.65 s were Gaussian label noise on the center, its SD would be at most 0.96 s, and a perfectly placed answer would sit at a median of 0.65 s, not the full video's 1.05 s.
- Noise cannot create the paired gain. Label noise is shared by both conditions, while the center error falls from 1.05 to 0.65 s on the same labels.
- Simulated noise leaves the gains intact. We add Gaussian noise to every true center, with SDs up to 2 s. Both paired contrasts hold in all 200 replicates at every level, and a 0.5 s SD moves the floor only from 1.05 to 1.15 s.
- Most misses land outside the moment. On the full video, 78–83% of sub-2 s misses put their predicted midpoint outside the moment.
- ActivityNet's original labels show the same collapse, so it does not depend on the re-annotation.
Reference: Sigurdsson, G. A., Russakovsky, O., and Gupta, A. What Actions are Needed for Understanding Human Actions in Videos? ICCV 2017.
Q3: generality outside TimeLens, and as deployed levers
Do the center floor and the zoom effect transfer to grounders outside the TimeLens lineage (e.g., TRACE or VideoChat-R1 as deployed levers)
The floor transfers: it holds on every VideoLLM we tested, now including UniTime at 1.40 s, and the five specialist methods do worse, at 2.0–8.0 s.
The zoom effect transfers as an oracle lever; deployed, it helps some models and not others.
- As oracle levers, foveation and the crop lower the center error on three of the four outside VideoLLMs and the mask on all four, and the oracle 8 s window lowers it on all nine specialist runs.
- Deployed, the zoom helps both zero-shot Qwen models, UniTime and RGNet trained on Charades: their mean center error falls and more clips improve than worsen. It does not help TRACE, VideoChat-R1, SnAG, DeCafNet's grounder, ReVisionLLM, CONE or RGNet's MAD model; for CONE and ReVisionLLM's MAD model the mean falls slightly but not clip by clip (Appendix Z, Tables A20 and A21).
- It fails for two general reasons, and on VideoChat-R1 for a third. A window built from a wrong-region first answer cannot repair it, and that repair is most of the oracle's gain. Inside a window that does hold the moment, TRACE's answer gets no closer.
- VideoChat-R1 often answers outside the crop. 45% of its crop answers fall outside the window, against 3–7% for TRACE and Qwen3-VL and 28% for zero-shot Qwen2.5-VL. Appendix A now reports this share.
and to backbones with different timestamp representations or temporal token granularity?
The floor appears under four ways of handling time: a time encoder (TRACE), native text timestamps (Qwen3-VL), interleaved text timestamps (UniTime, TimeLens) and whole-second output (TimeLens-8B). Within Qwen2.5-VL, our reply to Q4 varies the patch span; we tested no backbone with a different temporal patch size.
Response to Reviewer 6Hgt (3/3): Q4 and Q5
Q4: tokenization as a confound
Could the floor be a consequence of timestamp token resolution or temporal patch granularity rather than search space per se? […] a direct probe of internal representations at sub-second offsets would strengthen the claim.
Our tests find no evidence that either sets the floor. Crossing patch length with the crop on the same clips separates patch granularity from the search space (Appendix X).
- Patch length does not set the floor. Qwen2.5-VL fuses two frames per patch, so a patch spans 1 s at 2 fps and 0.25 s at 8 fps.
- On the full video, finer patches barely move the median center error: 1.05 → 0.98 s. Inside the 8 s crop at 2 fps, coarse 1 s patches reach 0.55 s.
- Nor does timestamp resolution. TimeLens-8B answers in whole seconds and TimeLens-7B in tenths of a second, and both show the floor.
- Stretching its clock 4× makes its whole-second answers 0.25 s apart in video time. That moves its center by only 0.09 s, while the crop moves it by 1.5 s.
- The direct probe supports no conclusion. In the original paper, a regression head trained on the frozen model's features kept the median center error at or above 1.8 s. We also ran the suggested direct probe: a linear readout of TimeLens-7B's internal states, patch by patch, at 1 s and 0.25 s patches. Its positive control, recovering the model's own answer, reached an AUROC of 0.70 against our preset bar of 0.75 (0.72 after tuning), so we draw no conclusion from it (Appendix X).
Q5: how the threshold was chosen, and a gate without the aggregate cost
The deployed threshold is selected on the reported sets, and the frozen variant loses aggregate accuracy beyond your preset bound.
The threshold holds out of sample, and its held-out cost is 1.88 mIoU points.
- Cross-fitted on Charades. We choose the threshold on one half of the videos and test it on the other, both ways, so every clip is held out once. Both halves pick the paper's frozen threshold, and the held-out short gain equals the in-sample one (10.5 points of [email protected]). On QVHighlights the held-out result also equals the in-sample one; on ActivityNet the held-out gain is small, +1.2 and +1.7 points for TimeLens-7B and 8B.
- Most of the deployment table is out of sample. The same threshold runs unchanged on the five other models and on Ego4D NLQ (our reply to tJUY's Q2).
- The cost against the bound. Held out, the aggregate cost is 1.88 mIoU points [1.10, 2.67]: at our preset 2-point bound, with the estimate inside it and the interval crossing it. The 2.7-point cost in the paper came from one half alone; Section 6 now reports the cross-fitted cost.
Is there a gate criterion (e.g., confidence-based) that keeps the short-stratum gain without the aggregate cost?
No gate we tried removes the aggregate cost, and confidence adds nothing (cross-fitted).
- Token confidence is at chance for predicting where the zoom helps: AUROC 0.47–0.51 among short first answers.
- Adding confidence to the length gate gains 10.7 points of short recall but costs 2.36 mIoU points, beyond the 2-point bound.
- Gates that check the second answer. Accepting the second answer only when it stays off the window's edges keeps +4.9 points at a cost of 1.47 mIoU points. Requiring agreement with the first answer gains 8.0 points but costs 2.18 mIoU points, beyond the bound (Appendix Z).
Response to Reviewer 6Hgt (1/3): W1 and W2/Q1
W1: the mechanism is not factorized, and the interventions use one family
We extend the factorization of the crop to six models, taking the factors it bundles in turn.
We broaden the input interventions beyond TimeLens; the training replication was already in the submission.
W2/Q1: a narrower payoff, and artifact coverage
Yes. The updated supplementary material holds the code, configurations and predictions for every experiment, with scripts that check every table value the paper prints against the released predictions (see the reproducibility statement).
Yes, by design. The zoom re-reads only clips whose first answer is shorter than 4 s. Its window lands on 60–61% of sub-2 s Charades moments with the TimeLens models, and on 2–43% with the five other models; where the first answer lies far from the moment, it cannot help (our reply to tJUY's W2). The frozen threshold's aggregate cost is in our reply to your Q5.