acceptodds
Withdrawn from ICLR 2027

TickTock: Teaching and Evaluating Video LLMs to Tell Time at Frame-Level Precision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advanced video large language models show temporal awareness: they can relate events to when they occur and answer multi-hop temporal questions. Despite this potential, they remain far from frame-level precision, especially for complex events whose boundaries unfold within a fraction of a second. A key obstacle is that their training labels and benchmark references are mostly recorded in whole seconds, so neither supervision nor evaluation can tell whether a prediction points to the right frame. In this work, we introduce TickTock, a frame-anchored annotation framework for teaching and evaluating frame-level temporal grounding. Instead of guessing a timestamp, the annotator, either a human or advanced MLLMs, selects the boundary frame from a numbered filmstrip, and returns its exact timestamp, so every label is traceable to a real frame. Building on this framework, we construct TickTock-5K, a training set of 5,000 videos spanning 11 domains from short videos and sports to lectures and screen tutorials, and release refined annotations for the TimeLens-100K and TimeLens2-93K training corpora. For evaluation, we build TickTock-Refined, a frame-level re-annotation from TimeLens-Bench and VUE-TR, complemented by TickTock-Point500, 500 human-annotated moments snapped to decoded frames. Across 28 open and closed video-LLMs, the strongest model places 79.2% of moments within five seconds but no model exceeds 21.6% within 0.1 s, a gap that IoU-based metrics largely conceal. Frame-anchored references reorder models and track human point accuracy more closely. Together, TickTock and its data and evaluation protocol lay a foundation for frame-level temporal grounding in Video-LLMs.

Primary area:
datasets and benchmarks
Keywords:
Video Large Language Models, MLLMs, Temporal Video Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m12vOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper says that labels in temporal grounding benchmarks have a resolution of one second. It gives TickTock, a method that attaches each label to a decoded video frame. A human or an MLLM selects the boundary frames on numbered contact sheets. The paper releases four resources: a training set, MLLM-refined TimeLens labels, a benchmark that Gemini re-annotated, and 500 human time points. The paper tests many Video LLMs which get very low performance with 0.1-level annotations.

Critical strengths

None.

Critical weaknesses
  1. **The paper does not show that sub-second temporal grounding is a meaningful application. ** It is not clear why improving timestamp precision from 1 s to 0.1 s matters, especially when this precision may exceed the consistency of human annotation. For example, the paper reports no human agreement data at sub-second precision, and no task is exemplified where 0.1 s accuracy changes the outcome meaningfully.
  2. The Newton example is confusing. It describes human perception at a low frame rate rather than a model task. It is also not a use case for sub-second grounding: the problem in the example is the input frame rate, but TickTock changes only the labels.
  3. The refinement of current benchmarks is limited. Building a temporal grounding benchmark involves both event search, which is the main challenge, and boundary finding, which current benchmarks already annotate at the 1 s level. The refinement moves endpoints by only a fraction of a second (median about 0.4 s, App. C.2), and the paper does not explain which applications need this change.
  4. The paper shows no benefit of the new data. The paper does not train a model on the released data. Thus, it is unclear whether the training data provides additional learning signal, or how well current models can learn sub-second temporal grounding. The word "Teaching" in the title is therefore not yet supported.
  5. The role of TickTock-5K is unclear. Based on the Fig. 2 caption and Sec. 3.2, TickTock-5K does not appear to contain frame-level annotations. It seems to be a curated set of 5K samples from existing data, which does not match the dataset name or the main claim of the paper. As a result, the dataset seems disconnected from the rest of the paper, and its purpose is unclear.
  6. TickTock-Refined is not human-verified. Gemini 3.1 Pro produced the TickTock-Refined boundaries, and no human checked them (App. C.4). LLM-generated labels can be acceptable for training data, but benchmark references should be verified by humans.
  7. The paper is difficult to read. Many engineering details, overlapping evaluation subsets and data statistics are spread across long paragraphs in the nine pages. These details obscure the main claims and the logic of the paper.
Questions for authors
  1. What is the proof that sub-second temporal grounding is significantly more useful than the current second-level practice?
  2. What is the point of TickTock5K if it is not even frame/sub-second-level annotations which is the main claim of the paper?
  3. Can the current MLLM models really benefit from the proposed training data?
Reviewer_9MBjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TickTock, a framework for frame-level temporal annotation and evaluation of video-LLMs. Instead of directly assigning timestamps, annotators select event boundary frames from numbered contact sheets, and the decoder maps these frames to their timestamps. The authors compile TickTock-5K, provide refined annotations for existing TimeLens training corpora, and construct two evaluation benchmarks, TickTock-Refined and TickTock-Point500. Experiments involving 28 video-LLMs show that current models struggle with sub-second temporal localization and that refined references can change model rankings.

Critical strengths
  1. The paper addresses a relevant limitation of existing temporal grounding benchmarks. The experiments demonstrate that IoU-based evaluation can conceal substantial errors in event boundaries.
  2. The frame-anchoring procedure provides traceable timestamps, and the evaluation of 28 video-LLMs offers useful evidence about current temporal localization capabilities.
Critical weaknesses
  1. The accuracy of the refined annotations is not sufficiently validated. TickTock-Refined relies on MLLM-selected boundaries without independent human boundary verification. Although the selected timestamps correspond to decoded frames, this does not guarantee that the frames represent the correct semantic boundaries.
  2. The contribution of the training data is insufficiently demonstrated. TickTock-5K is primarily compiled from existing MLLM-generated event annotations and retains their original timestamps, rather than applying the proposed frame-anchoring method to the full dataset. Furthermore, no model is trained on the proposed data to demonstrate its effectiveness.
Questions for authors
  1. Can the authors provide an independent human evaluation of a representative subset of TickTock-Refined, comparing the semantic accuracy of the refined boundaries with the original annotations?
  2. Can the authors provide experimental evidence that the proposed training resources improve frame-level temporal grounding compared with the original annotations?
  3. The reference list contains duplicate entries for Gao et al. (2017a, 2017b) on page 10 and Lei et al. (2021a, 2021b) on page 11. These entries refer to the same respective papers and should be consolidated. In addition, Table 1 on page 8 contains rather small text, and the appendix contents extend onto page 15 with substantial unused space. The authors are encouraged to improve these formatting issues.
Reviewer_1Q1WOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TickTock, a frame-anchored annotation and evaluation framework for temporal grounding in Video-LLMs. The authors construct TickTock-5K, and also refined annotations for TimeLens training corpora. They evaluate SOTA Video-LLMs and show that current models can often localize events at the second level but rarely at sub-second or frame-level precision.

Critical strengths
  1. The paper identifies a real limitation of existing temporal grounding benchmarks, and the fine-grained annotation is valuable.

  2. The evaluation is comprehensive, and the paper is easy to follow.

Critical weaknesses
  1. The methodological novelty is limited, and lack the discussion comparing frame-level annotation to more fine-grained timestamps, or relative timestamps (xx.x%) .

  2. External MLLM annotation error is not sufficiently evaluated. As the TickTock-Refined references are produced by MLLMs, will new annotation error introduced? Do you have some quantitive results?

  3. The training is somehow confusing -- are the labels frame-level or time-level? If the model can not sample target frames, will the timestamps change while training?

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.