- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EviAgent, an active evidence-acquisition framework for long-horizon surgical video reasoning. Its main contribution is a structured state that explicitly tracks acquired evidence, unresolved requirements, temporal proposals, and answer hypotheses. A policy trained with annotation-derived finite-horizon action values selects temporal retrieval, staged local verification, or stopping, accounting for the downstream value of acquiring additional evidence. Evaluation uses procedure-level two-fold cross-fitting on four MedHorizon surgical subsets comprising 207 videos and 868 multiple-choice questions. EviAgent achieves 50.46% overall accuracy, compared with 44.70% for the strongest reported existing baseline and 37.10% for an annotation-SFT control. Ablations support the contributions of structured states, finite-horizon acquisition, and staged verification.
Critical strengths
-
A coherent approach to query-dependent evidence acquisition. EviAgent explicitly distinguishes established evidence from unresolved requirements and connects that distinction to retrieval, verification, and stopping decisions. This is well motivated by sparse and repeated events in long surgical videos. The structured-state ablation improves overall accuracy from 43.66% to 50.46% (Table 3), providing empirical support for a central design choice and making this more than a purely conceptual contribution.
-
Promising gains with meaningful supervision controls and component ablations. EviAgent reaches 50.46% overall accuracy, outperforming the strongest reported existing baseline by 5.76 percentage points and the annotation-SFT control by 13.36 points. Procedure-level cross-fitting excludes annotations from each evaluated procedure, and MedHorizon question-answer labels are excluded from training. Additional improvements over one-step-gain supervision (46.31%, Table 4) and single-stage verification (41.13%, Table 5) strengthen the case that the proposed components contribute to the observed gains.
Critical weaknesses
-
The central policy-supervision mechanism is not specified precisely enough to assess or reproduce. Section 3.5 defines evidence gain through an annotation-derived sufficiency function, Phi*, and describes a finite-horizon action value, Q*, without providing concrete task-specific sufficiency rules, an explicit action-value computation, the planning horizon, or acquisition-cost weights. Appendix C.2 adds auxiliary losses without complete definitions or coefficient values. These are central to the claimed evidence-gain contribution, rather than peripheral implementation details. Their omission prevents a clear assessment of what is learned and how the method differs from alternative acquisition heuristics.
-
The evaluation does not sufficiently establish that the gains arise from acquiring reliable and sufficient evidence. Multiple-choice accuracy and qualitative trajectories do not directly measure evidence correctness, coverage, or premature stopping. This is particularly consequential for global counting and ranking tasks, where local positive observations cannot establish that unobserved regions would not change the answer; Appendix A.4 acknowledges this limitation. In addition, cumulative frame counts and acquisition costs are not reported, and Appendix C.3 notes that computational cost is not matched. Consequently, the current results leave unresolved whether the proposed policy improves evidence acquisition under comparable resources, and whether its stopping decisions are justified by the evidence obtained.
Questions for authors
-
Clarification of the core algorithm: Could you provide explicit formulas or pseudocode for Phi* and Q*, including task-specific evidence-sufficiency rules, the planning horizon, acquisition costs, terminal rewards, and auxiliary-loss definitions and weights? These details would directly affect my assessment of technical soundness and reproducibility.
-
Clarification of training-state construction: Are policy-training observations generated by the frozen learned visual modules or constructed from oracle annotations? How are retrieval misses, verification errors, and conflicting observations represented in the training trajectories? This would clarify the relationship between the supervised policy and the states encountered at inference.
-
Targeted evidence needed to assess the central claim: For global counting and ranking questions, what makes STOP preferable to further acquisition when coverage is incomplete? Could you provide quantitative evidence on premature stopping or answer-supporting evidence correctness, and show how accuracy changes with acquisition budget? This would help determine whether the policy learns useful evidence sufficiency rather than primarily exploiting answer-selection regularities.