acceptodds
Withdrawn from ICLR 2027

EviAgent: Evidence-Gain-Guided Active Evidence Acquisition for Long-Horizon Surgical Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon surgical video reasoning remains challenging because query-relevant evidence is often sparse, heterogeneous, and dynamically dependent on what has already been established. Existing long-video methods improve context coverage through compression, retrieval, or adaptive observation, but the relationship between established evidence, unresolved evidence, and the next acquisition decision often remains implicit. We therefore formulate long-horizon surgical video reasoning as active evidence acquisition and propose EviAgent, an evidence-seeking agent that explicitly targets unresolved, query-dependent evidence in long surgical videos. EviAgent maintains a structured evidence state and uses an evidence-gain-guided policy to adaptively select temporal retrieval, local verification, or stopping as reasoning progresses. We evaluate EviAgent on four surgical subsets of MedHorizon under procedure-level cross-fitted evaluation, covering both fine-grained surgical understanding and semantic reasoning. Anonymous code is available at: https://anonymous.4open.science/r/eviagent-CC46.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surgical Video, Long Video Reasoning, Evidence Seeking, Video Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uGUWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviAgent, an active evidence-acquisition framework for long-horizon surgical video reasoning. Its main contribution is a structured state that explicitly tracks acquired evidence, unresolved requirements, temporal proposals, and answer hypotheses. A policy trained with annotation-derived finite-horizon action values selects temporal retrieval, staged local verification, or stopping, accounting for the downstream value of acquiring additional evidence. Evaluation uses procedure-level two-fold cross-fitting on four MedHorizon surgical subsets comprising 207 videos and 868 multiple-choice questions. EviAgent achieves 50.46% overall accuracy, compared with 44.70% for the strongest reported existing baseline and 37.10% for an annotation-SFT control. Ablations support the contributions of structured states, finite-horizon acquisition, and staged verification.

Critical strengths
  1. A coherent approach to query-dependent evidence acquisition. EviAgent explicitly distinguishes established evidence from unresolved requirements and connects that distinction to retrieval, verification, and stopping decisions. This is well motivated by sparse and repeated events in long surgical videos. The structured-state ablation improves overall accuracy from 43.66% to 50.46% (Table 3), providing empirical support for a central design choice and making this more than a purely conceptual contribution.

  2. Promising gains with meaningful supervision controls and component ablations. EviAgent reaches 50.46% overall accuracy, outperforming the strongest reported existing baseline by 5.76 percentage points and the annotation-SFT control by 13.36 points. Procedure-level cross-fitting excludes annotations from each evaluated procedure, and MedHorizon question-answer labels are excluded from training. Additional improvements over one-step-gain supervision (46.31%, Table 4) and single-stage verification (41.13%, Table 5) strengthen the case that the proposed components contribute to the observed gains.

Critical weaknesses
  1. The central policy-supervision mechanism is not specified precisely enough to assess or reproduce. Section 3.5 defines evidence gain through an annotation-derived sufficiency function, Phi*, and describes a finite-horizon action value, Q*, without providing concrete task-specific sufficiency rules, an explicit action-value computation, the planning horizon, or acquisition-cost weights. Appendix C.2 adds auxiliary losses without complete definitions or coefficient values. These are central to the claimed evidence-gain contribution, rather than peripheral implementation details. Their omission prevents a clear assessment of what is learned and how the method differs from alternative acquisition heuristics.

  2. The evaluation does not sufficiently establish that the gains arise from acquiring reliable and sufficient evidence. Multiple-choice accuracy and qualitative trajectories do not directly measure evidence correctness, coverage, or premature stopping. This is particularly consequential for global counting and ranking tasks, where local positive observations cannot establish that unobserved regions would not change the answer; Appendix A.4 acknowledges this limitation. In addition, cumulative frame counts and acquisition costs are not reported, and Appendix C.3 notes that computational cost is not matched. Consequently, the current results leave unresolved whether the proposed policy improves evidence acquisition under comparable resources, and whether its stopping decisions are justified by the evidence obtained.

Questions for authors
  1. Clarification of the core algorithm: Could you provide explicit formulas or pseudocode for Phi* and Q*, including task-specific evidence-sufficiency rules, the planning horizon, acquisition costs, terminal rewards, and auxiliary-loss definitions and weights? These details would directly affect my assessment of technical soundness and reproducibility.

  2. Clarification of training-state construction: Are policy-training observations generated by the frozen learned visual modules or constructed from oracle annotations? How are retrieval misses, verification errors, and conflicting observations represented in the training trajectories? This would clarify the relationship between the supervised policy and the states encountered at inference.

  3. Targeted evidence needed to assess the central claim: For global counting and ranking questions, what makes STOP preferable to further acquisition when coverage is incomplete? Could you provide quantitative evidence on premature stopping or answer-supporting evidence correctness, and show how accuracy changes with acquisition budget? This would help determine whether the policy learns useful evidence sufficiency rather than primarily exploiting answer-selection regularities.

Reviewer_4zHGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EviAgent runs a RETRIEVE, VERIFY, and STOP loop over a structured evidence state (Eq. 3) that holds query conditions, supported or hypothesized evidence, unresolved requirements, temporal proposals, and answer hypotheses; the query conditions cover procedural context, action, instrument, and anatomical target. The policy is trained to imitate an annotation-derived finite-horizon action value built on the immediate evidence gain of Eq. 7, and both quantities are used only during training. The authors state a new formulation of long-horizon surgical video reasoning.

Critical strengths
  1. Figure 1 asks the right question for a long procedure: which requirement is still open, not how many frames were watched.
  2. On the same Qwen3-VL-8B-Instruct backbone, overall accuracy is 50.46, against 30.41 for the backbone and 37.10 for annotation SFT trained on the same fold labels.
Critical weaknesses
  1. The agents the paper says it builds on are never compared. Section 2.3 names VideoThinker, LensWalk, Active Video Perception, VideoARM, VideoHV-Agent, and MedScope, yet none of them appears in Table 2; the only agentic baseline is LongVideo-R1 (32.49 with Qwen3). Appendix C.3 also states that the comparison does not establish equal computational cost: EviAgent may sample up to 256 frames per retrieval and 64 per verification over several steps, while direct baselines receive one uniform sample under their input budgets. Table 2 therefore shows that EviAgent beats zero-shot VLMs and a direct-answer SFT model under a larger observation budget, not that its state and gain criterion beat existing evidence-seeking agents.
  2. The quantity in the title contributes the least. In Table 4, a fixed schedule over the same trained modules already reaches 42.74, which is 12.33 points above the backbone and above the SFT control. Replacing it with the one-step evidence-gain policy of Eq. 7 adds 3.57 points, and the remaining 4.15 points come from finite-horizon action-value supervision, which is the standard recipe of imitating an oracle multi-step value. By comparison, Appendix Table 5 shows that splitting verification into support and resolution stages is worth 9.33 points on its own (41.13 to 50.46). Most of the gain is thus explained by the trained retrieval and verification modules, not by evidence gain.
  3. The semantic-reasoning gains rest on very small subsets, with no measure of variance. Outside IR, the WR, RSR, and CPR accuracies in Table 2 are all multiples of 1/26, so each subset appears to contain about 26 questions. The RSR lead over Kimi-K3 (84.62 vs. 76.92) is two questions, CPR ties Video-R1-7B, WR (34.62) is below five baselines including Video-R1-7B, VideoRFT, and Hulu-Med-7B, and SD (40.18) is below Gemini-2.5-Flash (41.07). The paper reports no per-fold results, seeds, or confidence intervals for the 868 questions.
  4. Much of the reasoning is done by the parser and a fixed rule rather than over the video. In Figure 3, the parser turns the question into an answer specification such as "group by instrument × phase, maximize distinct-phase count", and a deterministic finalization procedure (Section 3.6) maps the terminal state to an answer. For such questions the video modules supply instrument–phase records and the cross-phase aggregation is a program written from the question text. No experiment removes or perturbs the parser's answer specification, so the share of the semantic-reasoning gain that comes from parsing questions into counting programs is unknown.
  5. The structured state is supported only against the weakest alternative. Table 3 compares it with raw accumulated action–observation history (43.66 vs. 50.46), not with the hierarchical memory of VideoARM or the hypothesis-verification state of VideoHV-Agent, which Section 2.3 cites. Its fields reuse existing surgical vocabularies (CholecT50's instrument–verb–target triplets and GraSP's phases, steps, instruments, and actions), so the ablation does not show that this particular state is better than existing structured agent states.
Questions for authors
  1. Could the authors re-run at least one of the agents cited in Section 2.3 on the same backbone with the same trained Retriever and Verifier, so the comparison isolates the evidence state and acquisition policy?
  2. Please report the acquisition budget and the average number of steps and frames per question for EviAgent, next to the frame budgets of the baselines.
  3. Please report per-fold results or confidence intervals, and the number of questions in each task of Table 2.
  4. What is the accuracy if the parser's answer specification is replaced by a generic one, or if the final answer is produced by the VLM from the terminal state instead of the deterministic finalization procedure?
Reviewer_dMvgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates long-horizon surgical video reasoning as active evidence acquisition process. Then, the authors propose EviAgent, an evidence-seeking agent that explicitly targets unresolved, query-dependent evidence in long surgical videos. The EviAgent is evaluated on four surgical subsets of MedHorizon and shows improved performance.

Critical strengths
  1. The problem formulation is interesting. Especially Fig. 1 clearly illustrates the differences between existing methods and the proposed method. It is an interesting idea to utilize an agent to actively aquire and articulate the evidence rather than reasoning individually over segments of long videos.
  2. The paper contains sufficient technical details and it is in general well written and easy to follow.
Critical weaknesses
  1. The most critical issue of this paper is that while the authors formulate the problem under the RL framework and solve it under the RL framework, there lacks significant key aspects or innovations on RL framework. Furthermore, Appendix C.2, states that "the policy is trained by supervised learning on oracle-derived action targets, rather than reinforcement learning." No environment interaction, reward feedback, or policy optimization is involved. The claimed evidence-gain-guided policy therefore reduces to behavior cloning on oracle-computed targets, and the proposed RL framework does not correspond to the actual implementation.
  2. The central claim in Fig. 1 that existing methods conduct reasoning over individual fragments is not accurate. In literature, there are many methods that perform adaptive observation, temporal retrieval, and iterative verification for long-video reasoning. The authors should better position their work in relate to the literature.
  3. The evaluation is conducted on only 868 QA pairs across four surgical subsets, with PitVQA containing merely 13 questions. Evaluation on such a small dataset does not lead to any solid conclusions.
Questions for authors
  1. I am curious to know why the authors frame and solve the problem utilizing the terminologies such as state, action, observations, policy and action value, but never formally admit it under an RL framework. In addition, it utilizes supervise learining rather than using reward, interaction with environment as usually does in RL agent. Could the authors justify more clearly on these design choices?
  2. Could the authors provide a detailed comparison with the most related agentic video methods to better position their method and substantiate the claimed novelty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.