- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates temporal information loss in Video-LLMs using the Arrow-of-Time (AoT) task, analyzing how temporal information is encoded by vision encoders, transferred through projectors, and utilized by LLMs. Based on these findings, the authors propose a temporally aware Video-LLM configuration that achieves strong AoT performance and demonstrate that AoT supervision can improve broader temporal reasoning tasks.
Critical strengths
-
Important and Well-Motivated Research Problem. The paper addresses an important limitation of current Video-LLMs by investigating where temporal information is lost or becomes difficult to utilize. The research questions are clearly formulated and provide a systematic perspective on temporal information processing.
-
Comprehensive Empirical Analysis. The authors conduct extensive experiments across different vision encoders, projector architectures, token compression strategies, and encoder layers. In particular, combining probing with end-to-end VQA evaluation provides useful insights into the discrepancy between temporal information recoverability and its actual utilization by LLMs.
-
Strong AoT Performance and Evaluation of Generalization. The proposed configuration achieves 98.1% accuracy on AoT-PPB, surpassing the reported human baseline. Furthermore, the authors evaluate the effects of AoT supervision across multiple short- and long-video benchmarks, providing evidence of potential benefits beyond the AoT task itself.
Critical weaknesses
[W1] Potential Bias in Encoder Probing. The attentive probe may be inherently disadvantaged when applied to frame-centric encoders, as attention pooling without explicit temporal positional information can be insensitive to frame ordering. Thus, near-chance performance may reflect limitations of the probing architecture rather than the encoder itself.
[W2] Lack of Direct Evidence for Temporal Information Loss. The paper attributes poor AoT performance to information loss during vision-language projection. However, downstream VQA accuracy alone cannot distinguish information loss from difficulties in utilizing the projected representations.
[W3] The improvements from AoT supervision may partly result from the increased training data volume and optimization steps rather than the specific temporal supervision. The current experiments do not sufficiently disentangle these factors.
[W4] The temporal compression analysis primarily relies on average pooling, which can inherently discard temporal ordering information. Therefore, the results may not generalize to more sophisticated temporal compression approaches.
Questions for authors
Q1. Could the authors evaluate frame-centric encoders using temporal positional embeddings or order-aware probes (e.g., temporal Transformers) to ensure a fair comparison?
Q2. Could the authors compare against order-aware temporal compression methods, such as temporal convolutions or attention-based aggregation, under the same token budget?
Q3. Could the authors compare AoT supervision with additional general video instruction tuning under matched data volumes and training budgets?
Q4. Could the authors clarify whether Qwen3-VL's near-chance AoT performance, despite its enhanced temporal modeling mechanisms, stems from limitations in the vision encoder or from the LLM's inability to effectively utilize the encoded temporal information?