acceptodds
Withdrawn from ICLR 2027

Beyond Rewatching: Learning Evidence-Driven Complex Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Complex video reasoning requires integrating visual evidence across entities, events, and time. Although recent models interleave textual reasoning with visual inspection, tool use alone does not ensure evidence-driven reasoning: models may inspect unnecessarily, fail to adapt unsuccessful searches, or retain interpretations contradicted by new observations. We address these limitations through three complementary training improvements. First, we propose Evidence-Conditioned Synthesis (ECS), which constructs verified reasoning continuations from intermediate states in executed video interactions, providing supervision for evidence assessment, selective inspection, recovery, and belief updating. Second, we introduce EviVideo-QA-20K, a dataset of 20K perception-centric questions that require composing visual evidence through conjunctive and sequential reasoning. Third, we design an Evidence Reward that complements answer correctness with an assessment of evidence use and an explicit inspection cost. Together, these components train EviVideo through supervised fine-tuning followed by reinforcement learning. Our 4B and 8B models improve over their Qwen3-VL backbones by average absolute accuracy gains of 7.6% and 7.9% across MINERVA, Minerva-Ego, and Video-Holmes, and by 4.0% and 3.6% across Video-MME and Video-MME-v2. Behavioral evaluations further indicate improved evidence-responsive inspection and revision. These results highlight the importance of teaching models not merely to invoke visual tools, but to let evidence guide reasoning and interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video reasoning, active visual reasoning, synthetic training data, supervised fine-tuning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hUroOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that tool-augmented ("thinking with videos") video reasoning models make genuine tool calls but are not evidence-driven. It names four failure modes: ungrounded reasoning, miscalibrated evidence sufficiency, inflexible inspection, and confirmation bias.

Critical strengths

None

Critical weaknesses

Seems totally AI generated. Several appendix tables are self-labelled mock/simulated/placeholder (Tables 5, 6, 8, 9, 11).

Questions for authors

Provenance of the reported results (C1). Several appendix tables are labelled by the authors as mock, hypothetical or simulated (Tables 5, 6, 8, 9, 11).

Reviewer_h6cLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies evidence-driven reasoning in tool-augmented video language models. It proposes Evidence-Conditioned Synthesis, which generates verified continuations from executed interaction prefixes; EviVideo-QA-20K, a dataset of compositional video questions; and an evidence-use reward combined with answer correctness and inspection cost. These components are used to train EviVideo through supervised fine-tuning and GRPO. The paper reports improvements on five video benchmarks, together with behavioral evaluations of inspection, stopping, recovery, and belief revision. However, several supplementary analyses are explicitly marked as hypothetical or unverified, substantially limiting the empirical conclusions that can be drawn from the submitted version.

Critical strengths
  1. The strongest contribution is the explicit treatment of intermediate evidence states. Generating continuations from unsuccessful inspections or unsupported interpretations, while masking the supplied prefix from the SFT loss, is a sensible way to teach recovery without directly imitating earlier errors. The distinction between executing a visual tool and appropriately using its observations is consequential for reliable video agents.
Critical weaknesses
  1. Tables 5, 6, and 8 are explicitly labeled as mock results; Table 9 contains draft placeholders; and Table 11 reports simulated paired comparisons rather than empirical evidence. Appendix E further states that the behavioral counts in Table 10 require linkage to per-state annotations before being treated as verified experimental results. This directly weakens the support for the behavioral claims in Section 5. There is also a concrete inconsistency in the dataset accounting: Section F.4 starts from 42,000 questions, while Table 15 defines successive retention rates of 72.5%, 65.0%, and 47.6%. These imply approximately 9,421 retained questions, rather than 20,000. The current manuscript does not provide a sufficiently reliable empirical foundation for acceptance.

2, The matched-QA comparison in Table 3 controls questions and supervised-token budgets, but explicitly allows different teacher models and synthesis procedures. It therefore supports the usefulness of the overall regenerated data pipeline, rather than isolating the effect of prefix-conditioned synthesis. Likewise, removing the Evidence Reward jointly removes the evidence-use score and inspection cost. An independently validated assessment of the learned behaviors is particularly important here, but the appendix describes a recommended annotation protocol rather than a fully documented completed evaluation.

Questions for authors
  1. Which reported results are completed experiments, and which are placeholders? Please provide the actual evaluation procedure and traceable per-state records underlying Figure 3 and Table 10, and reconcile the filtering counts in Table 15.

  2. With teacher identity, source questions, and supervised-token budget fixed, how does ECS compare with retaining only complete successful trajectories?

  3. What independent validation supports the evidence-use evaluator and behavioral labels? Please report the actual annotators or judges, agreement statistics, and separate effects of the evidence-use score and inspection penalty.

Reviewer_RgkgOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to train video reasoning models to seek evidence, recover from unsuccessful inspections, and revise unsupported interpretations. It combines Evidence-Conditioned Synthesis (ECS), which generates verified continuations from executed interaction prefixes, the compositional EviVideo-QA-20K dataset, and an RL reward for evidence use and inspection cost. The resulting 4B and 8B models are evaluated on five video benchmarks, with additional ablations and matched continuation tests.

Critical strengths
  1. ECS addresses a limitation of training only on successful trajectories. Branching from an unsuccessful interaction and masking the prefix loss allows the model to learn a correction without being trained to reproduce the preceding mistake. The verification rule that later observations cannot justify earlier claims is also well aligned with the stated objective (Section 3.2).

  2. The evaluation asks useful questions beyond final-answer accuracy. In particular, the matched evidence states in Figure 3 and the comparison of uniform sampling and adaptive inspection using the same checkpoint in Table 4 could distinguish evidence-responsive behavior from simply using more frames. These are appropriate tests for the paper's claims, provided the underlying results can be substantiated.

Critical weaknesses
  1. The status of the empirical evidence is my main concern. Tables 5, 6, and 8 are explicitly labeled as mock results, Table 9 contains illustrative placeholders, and Table 11 uses simulated pairings. More directly, Appendix E says that the author-supplied counts in Table 10 require linkage to per-state annotations before they can be treated as verified results. These counts reproduce the behavioral rates reported in Figure 3. The annotation procedure is described as a recommended protocol, with the numbers of distinct questions and videos still left as placeholders. This does not establish that Tables 1-4 are simulated, but it leaves a central behavioral claim without a documented empirical basis. The appendix also refers to Qwen2.5-VL-7B and EgoPlan-Bench2, whereas the main experiments use Qwen3-VL-4B/8B and Minerva-Ego.

  2. The contribution of ECS itself remains unclear. Table 3 controls the QA pairs and budget of supervised tokens, which is useful, but explicitly allows the teacher and synthesis procedure to differ. Removing all CoT SFT in Table 2 does not distinguish ECS from ordinary successful demonstrations generated by the same teacher. Likewise, the evidence-reward ablation removes both the evidence-use term and the cost penalty; the more detailed comparison in Table 8 is only illustrative. The reported gains therefore do not yet establish which of the proposed mechanisms is responsible.

Questions for authors
  1. Which results were actually measured? For Figure 3 and Table 10, please provide the paired state-level outcomes, distinct question/video counts, and the annotation procedure that was used, including agreement. Please also resolve the backbone and benchmark inconsistencies noted above.

  2. With the same teacher, QA pairs, budget of supervised tokens, and verification rules, how does training on complete successful trajectories compare with adding ECS continuations? This would directly test the main synthesis contribution.

  3. Are there measured reward ablations separating evidence-use supervision from the inspection-cost penalty? Such results would clarify what the combined ablation in Table 2 establishes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.