acceptodds
Under review as a conference paper at ICLR 2027

EviHarness: Evidence-Grounded Learnable Harness for Long Video Understanding

Abstract

Long-video question answering must locate sparse, answer-critical evidence under a limited observation budget. Existing selective-acquisition methods allocate observations to find relevant intervals, but compressing them can discard details needed to answer; timestamp-based memory points to a location without identifying which unsupported claim needs further evidence. We address this gap by linking compact claims to executable source pointers and using verification diagnoses to guide targeted re-observation. We introduce EviHarness, a learnable harness with raw-addressable evidence records, a frozen verifier that diagnoses missing or conflicting support, and a budget-aware controller that searches, revisits sources, or answers. We train only the controller: supervised fine-tuning (SFT) learns from valid, correct, low-cost trajectories, and GRPO optimizes answer correctness and acquisition cost while observation and verification components remain fixed. With Qwen2.5-VL-7B under shared inference caps, SFT+RL improves over prompting on Video-MME, LVBench, and MLVU; on LVBench, independently audited source support is 77.1% versus 61.4% for timestamp-only memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
ong video understanding, video-language models, multimodal agents, active perception, evidence grounding, temporal reasoning, tool use, visual memory, verification, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_J5D1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EviHarness introduces a learnable agent harness for long-video question answering under limited observation budgets. It stores compact evidence records with source pointers, timestamps, and acquisition metadata to support targeted video revisits. A frozen verifier identifies missing or conflicting evidence and guides further observations. The controller learns when to search, revisit, or answer through supervised fine-tuning (SFT) and GRPO. Only the controller is trained, while the observation tools and verifier remain frozen. Experiments on Video-MME, LVBench, and MLVU show improved accuracy and source support over prompting and simplified memory baselines.

Critical strengths
  1. Practical evidence-grounded framework. EviHarness connects evidence records to original video sources, allowing the agent to recover missing details without retaining all observed frames in its context.

  2. Evidence-support evaluation. Beyond answer accuracy, the paper evaluates whether cited evidence supports the generated answers.

Critical weaknesses
  1. Incomplete baselines and modest learning gains. The Qwen3-VL-8B experiments lack base-model, prompting, and SFT-only baselines.

  2. Insufficient isolation of component contributions. The ablations do not clearly separate the effects of source revisits, verifier feedback, and structured evidence memory. It remains unclear whether the gains come from the proposed recovery mechanism or additional observations.

  3. Limited evaluation of challenging reasoning and grounding. The experiments omit harder benchmarks such as Video-MME-v2 and VideoZeroBench. The single OCR-based case study and limited tool analysis also provide insufficient evidence for complex temporal reasoning and conflict resolution.

  4. Insufficient comparisons with recent methods. The experiments omit closely related works, such as Video-o3, VideoTemp-o3, OmniAgent and Video-MTR (ICML 2026). This limits the evidence for the benefits of the proposed learning and evidence-grounding mechanisms.

Questions for authors
  1. Why does GRPO optimize only answer accuracy and acquisition cost, without an explicit evidence-quality reward? Would such a reward improve grounded accuracy?

  2. How does EviHarness compare with existing video agents in end-to-end latency, model calls, and GPU cost under matched computational budgets?

Reviewer_CV1TOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work addresses the challenge in long-video question answering (QA) of locating sparse, critical evidence under a limited observation budget. It notes that existing selective-retrieval methods often lose details due to evidence compression, while timestamp-based memory mechanisms can only point to a location, failing to specify which unsupported claim requires additional evidence. The authors propose "EviHarness," which links concise claims to executable source pointers. It incorporates a record of original, addressable evidence; a frozen validator that diagnoses missing or conflicting support; and a budget-aware controller that decides whether to search for new information, revisit existing sources, or formulate an answer.

Critical strengths

By linking claims to source pointers and diagnostic validation, the system enables targeted re-observation based on missing or conflicting support, a more evidence-centric approach than relying solely on timestamps.

The architecture is highly modular; the observation and validation components remain frozen, while only the budget-aware controller is trained. The training objective simultaneously accounts for both answer accuracy and the cost of acquiring evidence.

Beyond standard QA performance metrics, the study reports source support rates via independent audits, providing an additional dimension for evaluating evidence traceability and the quality of supporting information.

Critical weaknesses

The validator judges “enough evidence” using only observations saved earlier. It does not look at the original media itself. It does not promise the final answer is right. So it may believe the evidence supports the answer more than it truly does.

The training reward cares only about answer correctness and retrieval cost. It has no “enough evidence” cutoff and no special reward for the evidence itself. So training may not directly aim to make sure the evidence truly supports the answer.

The observation, evidence-building, validation, and tool-use parts are frozen. So training cannot fix errors that come from how these parts are designed.

The inference limit is 64 frames in total and 6 controller steps. This may be too small to find evidence that is sparse and spread widely across very long videos.

The evaluation does not use official subtitles or ASR. It uses only RGB frames and OCR. No one has checked whether evidence can be found from audio or subtitles. So the conclusions may not hold for all long videos with more than one modality.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.