acceptodds
Withdrawn from ICLR 2027

VidDeduce: Enhancing Long-form Video Understanding via Deductive Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-form video understanding poses challenges in clue localization and logical reasoning. While agents enhance reasoning via adaptive region selection and evidence gathering, current paradigms either introduce query-agnostic redundancy via global text preprocessing, or neglect global context through localized iterative retrieval. These limitations lead agents to form misleading intermediate hypotheses, trapping them in unviable reasoning trajectories. To enable evidence-guided search refinement, we propose VidDeduce, a training-free multi-agent framework comprising a cognitive planner, visual observer, and a verifier based on Natural Language Inference (NLI). The planner integrates clue-relevant captions with prior verification feedback to update accumulated evidence and unresolved gaps, and formulates subsequent search strategies. Translating these strategies into actionable perception, the observer distills relevant captions and adaptively performs visual inspection to acquire observations. The verifier assesses new observations against the accumulated evidence and unresolved gaps via NLI, providing explicit feedback to guide continued investigation of useful clues or revise unproductive search directions, ensuring every exploratory step possesses a traceable logical rationale. Extensive experiments across four video understanding benchmarks demonstrate the effectiveness of VidDeduce, which outperforms strong agentic baselines including DVD and LensWalk on Video-MME, LongVideoBench, and EgoSchema.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-form Video Understanding, Multimodal Agents, Deductive Verification, Search Trajectory Regulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aiodOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VidDeduce is a training-free agent for long-video question answering. It builds a caption database, then uses a planner to track evidence gaps and an observer to retrieve captions and inspect video intervals at coarse or fine scale. An NLI verifier distinguishes full support, partial support, contradiction and neutral observations to guide the next search. Using GPT-5.3 and Qwen2.5-VL-72B, it reports the highest scores in Table 2 on Video-MME, LongVideoBench and EgoSchema, with component ablations and search analyses.

Critical strengths
  1. Distinguishing contradiction from missing or partial evidence is a sensible way to guide video search. The examples in Fig. 4 show why this matters: one requires abandoning a mistaken hypothesis, while the other requires examining the day sequence further. The planner–observer separation also makes the search decisions easier to follow.

  2. The complete system performs well on three of four benchmarks (Table 2), and the evaluation includes component ablations and different reasoning and visual backbones. The discussion of the nine-point LVBench deficit to DVD is useful. In particular, Table 17 shows concrete details lost by the coarser captions rather than only reporting successful cases.

Critical weaknesses
  1. The experiments do not isolate the benefit of deductive verification.

The only direct comparison replaces NLI with an unspecified confidence mechanism on Video-MME (Table 3). The 69.89 to 71.33 improvement is 13 questions out of 900, with no reported uncertainty. Without the confidence prompt and stopping rule, it is difficult to know whether this tests four-way feedback or simply different prompting. No binary-verifier or no-verifier control is reported. The main baselines do not resolve this: DVD differs in caption granularity, visual model and search budget (App. A.3), while LensWalk shares the visual backbone but has no documented matched end-to-end rerun.

The verifier also receives retrieved captions as part of both the accumulated evidence and the new observation. Although the planner requests direct visual inspection, the verifier is not required to base its label on the visual findings (Eqs. 2–5; Tables 9 and 11). Caption-only and visual-only controls would be particularly informative here. The label definitions need a rule for observations that both support one gap and contradict another premise; excluding contradiction from Neutral does not address this overlap. Overall, the paper shows a competitive agent, but the contribution of the verifier remains uncertain.

  1. Several evaluation details affect the interpretation of the results.

The answer schemas allow only A–D (Tables 12 and 14), although EgoSchema has five options [Mangalam et al., 2023] and LongVideoBench permits 4–5 [Wu et al., 2024]. No remapping or separate schema is documented. Algorithm 1 also passes K and U to final deduction without incorporating the last non-global observation. The initial caption-only shortcut bypasses verification, but its decision rule and exit rate are not given. These details need to be resolved before the reported evaluations can be reproduced.

The grounding and efficiency claims need clearer accounting. Fig. 3(c)'s Combined Hit of 61.2% cannot be a pooled average of the 54.3% coarse and 38.8% fine observation rates; the unit over which intervals are combined is unspecified. The subsequent-entailment rates are also judged by the same verifier rather than independent evidence annotations (Sec. 4.2). For efficiency, the 162.1 active-search frames per Video-MME question exclude full-video captioning, which requires 1,200–2,400 frames per question at the benchmark's three questions per video (Sec. 4.1; Table 5). This supports a claim about active search, but not the cost of the full pipeline.

Questions for authors
  1. What exactly is the confidence baseline in Table 3, including its prompt and stopping rule? Are paired predictions or repeated-run results available for the 13-question gain?
  2. How were option E answers handled on EgoSchema? Does the selected LongVideoBench split contain five-choice questions, and if so, how were E answers handled? Were benchmark-specific schemas used?
  3. Does the implementation use the final observation differently from Algorithm 1? How often does the initial caption-only shortcut answer the question?
  4. Which Table 2 comparisons use matched models, captions and budgets? How does the verification mechanism differ from Logic-in-Frames?
  5. What is the sample unit and denominator for Combined Hit in Fig. 3(c), and what is the total cost per question after including caption generation?
Reviewer_1YXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VidDeduce, a training-free multi-agent framework for long-form video question answering. A GPT-5.3-based planner tracks accumulated evidence and unresolved gaps, an observer retrieves captions and invokes coarse- or fine-scale visual inspection using Qwen2.5-VL-72B, and a GPT-5.3-based verifier labels each new observation as global entailment, partial entailment, contradiction, or neutral. This feedback is used to retain supported clues, refine the current search, or redirect the trajectory. VidDeduce is evaluated on LVBench, Video-MME Long, LongVideoBench Val-Long, and EgoSchema validation. It obtains 65.2%, 71.3%, 69.9%, and 78.6% accuracy, respectively. Component ablations on Video-MME Long show reductions of 1.44 points when NLI verification is replaced by confidence feedback, 1.55-2.11 points when one of the two visual scales is removed, and 3.33 points when temporal guidance is removed. Additional analyses examine backbone transfer, query adaptation, temporal overlap with annotated evidence, and inference efficiency.

Critical strengths
  1. The verification taxonomy maps naturally to different search responses. Treating partial support, contradiction, and neutral observations differently is more informative than a binary sufficiency or confidence check. The method presents a coherent mechanism for refining, redirecting, or terminating evidence acquisition.
  2. The empirical evaluation is broad and includes several useful diagnostic views. The paper evaluates four benchmarks, two visual backbones, several reasoning backbones, dual-scale visual ablations, temporal grounding, query adaptation, and qualitative trajectories. The prompts and benchmark-specific sampling settings are also documented in detail.
Critical weaknesses
  1. The contribution of NLI verification is weakly isolated and relatively small. The planner, observer scheduler, and verifier all use the same GPT-5.3 backbone; the “NLI verifier” is therefore a prompted role rather than an independent or calibrated entailment model. Replacing NLI feedback with confidence reduces Video-MME Long accuracy by only 1.44 points, and no paired significance test or repeated-run analysis is provided. The paper also lacks a budget-matched baseline using the same GPT-5.3 and Qwen2.5-VL models, the same captions, and the same number of calls, but a simpler non-NLI iterative controller. Consequently, the main results do not establish that deductive verification, rather than the strong reasoning model and multi-call scaffold, causes the improvement.
  2. The comparison and efficiency analyses omit important confounds. Main-table systems use different visual and reasoning backbones, captioning pipelines, search budgets, and input modalities. VidDeduce preprocesses every video into captions using Qwen2.5-VL-72B at 2 FPS, but the efficiency discussion counts only active-search frames and reasoning calls. In addition, the reported localization “Hit” metric increases when coarse planner windows and caption intervals are unioned with visual observations; because larger temporal coverage mechanically raises overlap probability, the 83.6% result is difficult to interpret without coverage-matched or random baselines. The query-adaptation score is likewise assigned by another language model without human validation or a demonstrated relationship to answer correctness.
Questions for authors
  1. Can the authors provide a budget-matched baseline using the same GPT-5.3/Qwen2.5-VL backbones, caption database, tools, number of calls, and token budget, but without the four-way NLI mechanism? Please report paired per-question outcomes and significance for the 1.44-point NLI-versus-confidence difference.
  2. How accurate are the verifier’s four labels? A blinded human audit of a representative sample, including confusion among partial entailment, contradiction, and neutral, would help establish that the verifier is performing meaningful evidence assessment rather than producing plausible self-reflection text.
  3. What are the complete preprocessing and inference costs, including full-video caption generation and embedding? For temporal grounding, can the authors report coverage-normalized recall/precision or IoU and compare against uniform, random, caption-only, and equal-duration retrieval baselines?
Reviewer_N5BLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VidDeduce, a training-free multi-agent framework for long-form video understanding. It combines caption-based retrieval, dual-scale visual inspection, and NLI-based verification in an iterative planning and observation process. The verifier provides structured feedback to refine search strategies, revise unsupported hypotheses, and determine when sufficient evidence has been acquired. Experiments on four video understanding benchmarks demonstrate competitive performance, with additional ablations and analyses of search behavior.

Critical strengths
  1. The proposed framework integrates global caption retrieval with adaptive visual inspection at different temporal scales, allowing the agent to update its search strategy based on accumulated evidence and unresolved questions.

  2. The experiments cover four video understanding benchmarks and include component ablations, backbone generalization experiments, and analyses of query adaptation and temporal grounding. The results demonstrate competitive performance and provide empirical support for the proposed design.

Critical weaknesses
  1. The novelty and independent effectiveness of NLI-guided verification are not sufficiently established. Existing video agents already support iterative reasoning, adaptive visual inspection, and feedback-based search refinement. The main methodological distinction of this paper is the structured NLI feedback, implemented through LLM prompting. However, Table 3 shows only a 1.44pt improvement over confidence-based feedback on Video-MME (Long). The paper also lacks direct evaluation of the reliability of NLI judgments or stronger evidence that these structured feedback categories offer substantial benefits beyond conventional LLM-based reflection.

  2. The computational cost of the proposed framework is not adequately evaluated. VidDeduce precomputes captions for the entire video using Qwen2.5-VL-72B and relies on multiple GPT-5.3 reasoning calls and additional visual inspections during inference. Although the paper reports LLM call counts and inspected frames, it does not provide end-to-end latency or computational costs including preprocessing. This makes it difficult to assess whether the performance improvements justify the additional complexity and resource consumption compared with existing agentic approaches.

Questions for authors
  1. Table 2 does not specify the reasoning and visual backbones used by the agent baselines. Can the authors clarify these model configurations? Given that VidDeduce uses GPT-5.3 for reasoning and Qwen2.5-VL-72B for visual perception, comparisons under matched model configurations and inference budgets would help establish the contribution of the proposed framework.

  2. Can the authors further isolate the benefits of the four NLI feedback categories compared with conventional LLM reflection under identical model and inference settings? How reliable are these judgments, particularly in identifying contradictions and unsupported hypotheses?

  3. What are the end-to-end wall-clock latency and computational costs, including offline video captioning, retrieval, and all LLM/VLM calls? How do these compare with existing agentic methods for both single-question inference and repeated queries over the same video?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.