acceptodds
Under review as a conference paper at ICLR 2027

Tracing Arrow-of-Time: Diagnosing and Addressing Temporal Information Loss in Video-LLMs

Abstract

The Arrow-of-Time (AoT) task requires distinguishing forward from backward video playback by recognizing temporally irreversible events. Prior work has revealed a substantial gap between humans and Video-LLMs on AoT in end-to-end evaluations. Here, we use AoT as a diagnostic lens to localize where this gap arises: do visual backbones fail to encode temporal information, or is such information lost when transferred to the LLM? We combine vision-encoder probing with downstream video question answering (VQA) to trace temporal information across the encoder, projector, and LLM. We find that video-centric encoders with explicit temporal modeling encode substantially stronger temporal signals than the evaluated frame-centric encoders. However, encoding alone is insufficient: projector design and token compression can disrupt temporal information before it reaches the LLM. Moreover, layer-wise analysis reveals the distinction between probe-level recoverability and LLM-level usability, underscoring the complementary roles of probing and VQA. Guided by these findings, we combine a temporally aware video encoder with a time-preserving projector and achieve 98.1% accuracy on AoT, surpassing human performance. Finally, we show that AoT supervision improves temporal reasoning across both short- and long-video benchmarks. Our results position AoT as both a diagnostic lens and an effective supervision for improving broader temporal reasoning in Video-LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Temporal Understanding, Video Understanding, VLM, Arrow-of-Time

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zMT2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates temporal information loss in Video-LLMs using the Arrow-of-Time (AoT) task, analyzing how temporal information is encoded by vision encoders, transferred through projectors, and utilized by LLMs. Based on these findings, the authors propose a temporally aware Video-LLM configuration that achieves strong AoT performance and demonstrate that AoT supervision can improve broader temporal reasoning tasks.

Critical strengths
  1. Important and Well-Motivated Research Problem. The paper addresses an important limitation of current Video-LLMs by investigating where temporal information is lost or becomes difficult to utilize. The research questions are clearly formulated and provide a systematic perspective on temporal information processing.

  2. Comprehensive Empirical Analysis. The authors conduct extensive experiments across different vision encoders, projector architectures, token compression strategies, and encoder layers. In particular, combining probing with end-to-end VQA evaluation provides useful insights into the discrepancy between temporal information recoverability and its actual utilization by LLMs.

  3. Strong AoT Performance and Evaluation of Generalization. The proposed configuration achieves 98.1% accuracy on AoT-PPB, surpassing the reported human baseline. Furthermore, the authors evaluate the effects of AoT supervision across multiple short- and long-video benchmarks, providing evidence of potential benefits beyond the AoT task itself.

Critical weaknesses

[W1] Potential Bias in Encoder Probing. The attentive probe may be inherently disadvantaged when applied to frame-centric encoders, as attention pooling without explicit temporal positional information can be insensitive to frame ordering. Thus, near-chance performance may reflect limitations of the probing architecture rather than the encoder itself.

[W2] Lack of Direct Evidence for Temporal Information Loss. The paper attributes poor AoT performance to information loss during vision-language projection. However, downstream VQA accuracy alone cannot distinguish information loss from difficulties in utilizing the projected representations.

[W3] The improvements from AoT supervision may partly result from the increased training data volume and optimization steps rather than the specific temporal supervision. The current experiments do not sufficiently disentangle these factors.

[W4] The temporal compression analysis primarily relies on average pooling, which can inherently discard temporal ordering information. Therefore, the results may not generalize to more sophisticated temporal compression approaches.

Questions for authors

Q1. Could the authors evaluate frame-centric encoders using temporal positional embeddings or order-aware probes (e.g., temporal Transformers) to ensure a fair comparison?

Q2. Could the authors compare against order-aware temporal compression methods, such as temporal convolutions or attention-based aggregation, under the same token budget?

Q3. Could the authors compare AoT supervision with additional general video instruction tuning under matched data volumes and training budgets?

Q4. Could the authors clarify whether Qwen3-VL's near-chance AoT performance, despite its enhanced temporal modeling mechanisms, stems from limitations in the vision encoder or from the LLM's inability to effectively utilize the encoded temporal information?

Reviewer_SLpfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses the Arrow-of-Time (AoT) task (distinguishing between forward and backward playback of a video) as a diagnostic lens to examine whether Video-LLMs have a human-like sense of time, and if not, which component (vision encoder, projector, or LLM) is responsible. To identify the source of failure, the paper conducts controlled experiments on the type of vision encoder (frame-centric vs. video-centric) using probing and VQA, the projector architecture (e.g., MLP, Q-Former), and visual token compression, and adopts a video-centric encoder with an MLP projector that applies only spatial pooling (preserving the temporal dimension). Finally, adding AoT data to instruction tuning with this design yields up to 98.1% on AoT_PPB, which transfers to small gains on TVBench, MVBench, MLVU, and Video-MME-Long (0–2%p).

Critical strengths

Controlled analysis of Video-LLM design choices. A controlled study of design choices (vision encoder, projector, and token compression) on a single well-defined task offers practical insights into how to extend language- and image-based backbones to Video-LLMs. Also, the study covers multiple video- and frame-centric encoders, five projector designs, two LLMs, and both short- and long-video benchmarks.

Critical weaknesses
  1. Limited new insight relative to prior work.

    • Most of the main take-aways appear to be already known in the community. Xue et al. [1] show that a representative LMM is largely insensitive to frame reversal and that AoT-aware training improves broader video QA. For design choices, Apollo [2] systematically compares video and image encoders as well as connectors for video-LMMs, V-JEPA 2 [3] shows that a video-centric encoder outperforms image encoders when coupled with an LLM, and PLLaVA [4] finds that spatial pooling is preferable to temporal pooling, which is closely related to the time-preserving projector result.
    • The new part beyond these works, i.e., AoT supervision for broader temporal reasoning (RQ3), also provides limited insight. Beyond AoT$_{PPB}$, the gains on other tasks are marginal, mostly within 0–2 points with some decreases (Table 2). Moreover, the AoT data (roughly 340k SSv2/MiT clips, based on Appendix E.2 and Table 3) appear to be added on top of the ~1M instruction-tuning set, so it is unclear whether the gains come from AoT supervision itself or partly from the increased amount of training data. In addition, the models trained with this recipe remain well below off-the-shelf Qwen2.5-VL-7B evaluated under the same protocol (e.g., MLVU 51.5 vs. 67.9 and Video-MME-Long 42.8 vs. 52.8 for the best configuration), while AoT fine-tuning of Qwen2.5-VL-7B itself stays at chance (Table 1). It is therefore unclear whether the benefit of AoT supervision transfers to competitive Video-LLMs.
  2. Validity of the experimental setup.

    • The attentive probe pools all tokens with a single learnable query, which is invariant to token order. Thus, for frame-centric encoders without temporal positional encodings, near-chance performance is expected by design, whereas the LLM can use its own positional encoding. This seems consistent with InternVideo2 stage2 being at chance under probing (49.5%) but reaching 73.1% in VQA, so I am not sure whether attentive probing is a suitable experimental design for this diagnosis.
    • Many SSv2 videos remain natural even when played in reverse (e.g., Opening vs. Closing), which makes the backward labels in the training data ambiguous.

[1] Z. Xue et al. Seeing the Arrow of Time in Large Multimodal Models. NeurIPS, 2025.

[2] O. Zohar et al. Apollo: An Exploration of Video Understanding in Large Multimodal Models. CVPR, 2025.

[3] M. Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985, 2025.

[4] L. Xu et al. PLLaVA: Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning. arXiv:2404.16994, 2024.

Questions for authors
  1. What are the contributions and novel findings of this work beyond prior works [1]–[4]?
  2. How do the probing results change with an order-aware probe (e.g., adding temporal positional embeddings to the tokens of frame-centric encoders before attentive pooling)? This would show whether the gap between frame- and video-centric encoders reflects the encoders or the probe design.
  3. How data-efficient is AoT supervision compared with adding the same amount of training data for other tasks (e.g., the same SSv2/MiT clips with action classification, or additional general instruction data)? Please also specify the exact composition of the ~1M instruction-tuning set and the number of added AoT samples.

Minor: Qwen2.5-VL-7B obtains 49.3% both before and after AoT fine-tuning (Table 1), so it would help to check whether its predictions collapse to one class; duplicated "✗" for Qwen3VL-8B in Table 1; the TVBench gain for Qwen2.5-7B + VJepa-H in Table 2 (37.3 → 38.5) is reported as +1.8.

Reviewer_uy9EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work analyzes the mechanism of time information loss in the Arrow-of-Time task by combining probing on a frozen vision encoder with end-to-end VQA to trace the encoding and transmission of temporal information across the encoder, projector, and LLM in Video-LLM. Based on this, it builds a video-centric encoder combined with a Time-Preserved MLP, effectively enhancing VideoLLM's temporal reasoning capabilities using AoT as supervision.

Critical strengths
  1. This work provides a reasonable analysis of AoT, a clean control design, clear logic, and effectively demonstrates the mechanism of information loss in AoT.
  2. This work addresses the current pain points of VideoLLM, and the proposed solution is helpful for further improving the spatio-temporal information processing capability of LLMs.
  3. The proposed LLM supervised training method based on AoT strategy has been proven to be effective on multiple benchmarks.
Critical weaknesses
  1. I understand that Q-former indeed has many limitations, but I suggest the authors can conduct a more systematic analysis and examination of the significant gap in Figure 2(a), otherwise it may be misinterpreted as incomplete or incorrect training of Q-former, thereby affecting the solidity of the conclusions.

  2. The gap between probe and VQA in Fig. 2(c) does not necessarily reflect whether the LLM can exploit the temporal information. The two experiments mixed differences such as time information, model capacity, and whether the input is pooled, which interferes with the attribution. It is suggested that the authors conduct a more in-depth analysis or add time position encoding to the probe in order to reduce the interference.

  3. Typo: Tab. 1, Qwen3VL-8B checkmark; Tab. 2 Video-MME-Long 33.2 (-1.2)

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.