acceptodds
Withdrawn from ICLR 2027

SITE: Surfacing Inter-frame Temporal Evidence to Recover VLMs' Dormant Temporal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) typically answer temporally discriminative video questions using uniformly sampled frames. This interface is efficient, but presents the same frame-wise input regardless of the temporal relation required by the question. Even when the frames contain the relevant visual observations, these relations remain implicit. Answering correctly therefore requires the VLM to both organize these observations into usable temporal evidence and reason over that evidence. These two operations are coupled within a single prediction, so final-answer accuracy alone cannot distinguish failures in evidence formation from failures in answer reasoning. We investigate whether explicitly presenting question-relevant temporal evidence can correct errors in frozen-VLM under this interface. We introduce SITE, a training-free Route–Surface–Corroborate framework. SITE routes each question to the required type of inter-frame evidence, constructs corresponding evidence views through visual organization and task-specific instructions, and combines their predictions through cross-view voting. Across three benchmarks and three frozen backbones, SITE improves accuracy by up to 23.0 percentage points over the sampled-frames baseline and outperforms matched-backbone training-free baselines. These results show that explicitly organizing relevant temporal evidence and corroborating predictions across views can correct temporal QA errors without updating model parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Reasoning, Video Question Answering, Vision-Language Models, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mCr2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SITE, a training-free framework designed to improve temporal reasoning in frozen Vision-Language Models (VLMs) by explicitly organizing question-relevant inter-frame evidence. SITE consists of three stages: Route, which classifies questions into temporal reasoning categories; Surface, which constructs task-specific evidence views through visual organization and prompting; and Corroborate, which aggregates predictions across multiple views using voting and a disagreement verifier.

Critical strengths
  • The paper identifies an interesting limitation of conventional frame-based video reasoning and presents the proposed framework in a clear, well-structured manner.
  • SITE achieves consistent accuracy improvements across three benchmarks and three frozen VLM backbones.
  • The stage-wise ablations, correctness-transition analysis, and evaluations under different frame sampling strategies provide useful evidence of the framework's practical effectiveness.
Critical weaknesses
  • The proposed framework primarily combines existing techniques, including question classification, task-specific visual prompting, optical-flow visualization, and prediction aggregation through voting. Although their integration is reasonable, the individual components are relatively conventional, and the paper does not establish a sufficiently distinctive algorithmic contribution beyond their combination.
  • The paper attributes performance improvements to recovering dormant temporal reasoning capabilities through explicit evidence formation. However, the current experiments do not clearly distinguish improvements in temporal evidence utilization from those resulting from better prompting, and increased inference computation.
  • While the stage-wise ablations are useful, the paper provides limited insight into router reliability, the contribution of individual evidence views, and the extent to which different views provide complementary information.
Questions for authors
  • What is the fundamental algorithmic contribution of SITE beyond combining existing visual prompting, question classification, and multi-view prediction aggregation?
  • Can the authors provide direct evidence that SITE improves temporal evidence formation rather than simply improving final-answer selection?
  • Since the optical-flow view uses information from the full video, how much of the improvement remains when this component is restricted to the same sampled frames as the baseline?
  • Could the authors also report the average number of VLM calls, end-to-end latency, and computational overhead across all three backbones?
Reviewer_2PLuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents SITE, a training-free method for temporal video QA. It involves a frozen VLM that first sorts each question into one of fife types then answers it using several hand-written prompts and layouts for that type (plus the use of optical-flow overlays for motion questiuons), and finally combines those answers by majority vote with LLM tie-breaker. It gives consistent gains across backbones and benchmarks but is essentially per-type prompt engineering plus ensampling and nothing tests the central claims that it separates failure in forming the evidence from failures in reasoning over it.

Critical strengths
  • The gains are consistent across 3 backbones and 3 benchmarks, including the strongest model (GPT-5.6-Terra). It fixes 38–57% of that model's baseline errors.
  • It includes two controls many papers skip: voting with the same number of calls (Table 6) and random question types (Table 7).
  • It matches the strongest training-free baseline, TimeLogic, using about 2.3× fewer tokens.
  • It's simple, needs no training, and works with any VLM.
Critical weaknesses
  • Every piece already exists. Sending different question types to different modules is the idea behind ViperGPT, MoReVQA and TraveLER. The voting is self-consistency. Drawing timestamps on frames is ViKey and set-of-mark prompting. Using optical flow as a cue is standard. Putting these together for temporal QA is engineering, not a new idea.

  • The headline claim isn't tested. The abstract argues that final-answer accuracy can't tell failures in forming the evidence from failures in reasoning, and that SITE separates the two. But nothing measures that separation since every result is still final accuracy. For an ICLR paper, that missing insight is the biggest problem.

  • The question types look fitted to the benchmarks. The 4 types are exactly the 4 subsets of VTL-Core, which the authors built themselves. Some prompts match particular benchmark question formats. The paper never says how the prompts were developed, or whether they were tuned on these same test sets. If they were, that's test-set leakage. It also means that this is not an approach that can scale since for every domain/benchmark you need to craft questions.

Questions for authors

In my personal opinion this paper does not present a contribution that is ICLR level. However, here are somethings that could improve the paper:

  • An experiment that actually separates forming the evidence from reasoning over it
  • A breakdown of each view's contribution, especially optical flow vs. prompts on the Kinematic questions.
  • A statement of how the prompts were written, plus results on a temporal benchmark that wasn't used to design them (it will be interesting to see how it applies to a different benchmark given that the questions are benchmark-specific).
Reviewer_re4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SITE, a training-free inference framework for temporal video QA that first routes each question to a temporal-relation category, constructs question-specific evidence views using prompts, visual layouts, timestamps, or optical-flow cues, and then aggregates predictions across these views. Experiments on 3 benchmarks and 3 frozen VLM backbones show consistent improvements over standard sampled-frame inference and stronger accuracy than the evaluated same-backbone training-free baselines. The results suggest that a nontrivial fraction of temporal QA errors can be recovered by changing how temporal evidence is presented at inference time rather than by updating model parameters.

Critical strengths
  • The paper demonstrated a useful empirical finding that question-conditioned evidence presentation (in temporal perspective) can substantially improve temporal reasoning in frozen VLMs without additional training.

  • The evaluation is relatively thorough, including multiple backbones and benchmarks, as well as ablations.

Critical weaknesses
  • The main comparisons do not control inference-time computation. Full SITE uses multiple VLM (up to 7? If Transition route, then 1 initial call + 4 calls + 1 highest-priority view if disagree + 1 verifier if tie) calls, and Kinematic route additionally needs computation for optical flow, making it difficult to determine whether SITE is better than competing SOTA methods under a matched compute budget.

  • The methodological novelty over existing training-free temporal reasoning SOTA methods is limited, since routing, temporal prompting/annotation, question-dependent evidence acquisition, visual prompting, and multi-sample aggregation have precedent, with SITE’s main novelty lying in their hand-designed relation-specific combination IIUC.

Questions for authors
  • Could the authors provide compute-matched comparisons against self-consistency, TimeLogic, and ViKey across more than one backbone and benchmark, reporting an accuracy-cost trade-off using total VLM calls/tokens and ideally latency or FLOPs, including the cost of optical-flow preprocessing?

  • Could the authors clarify what they view as the key technical distinction from methods such as TimeLogic and ViKey, and provide evidence that relation-specific evidence representations themselves (instead of primarily the increased test-time computation and heterogeneous ensembling) account for the advantage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.