acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Review: Selective Multi-Agent Verification for Surgical Video Understanding

Abstract

Reliable surgical video understanding requires accurate and mutually consistent predictions across perception (*e.g.*, instrument recognition) and interaction understanding (*e.g.*, triplet recognition) to support procedure-level reasoning and summarization (*e.g.*, surgical report generation). Existing approaches use Vision–Language Models to infer these surgical states, yet prediction errors can propagate into downstream reasoning and compromise the factual accuracy of generated reports. Additional model review offers an opportunity to correct these errors, but reviewing every observation incurs redundant computation and can alter predictions that were already correct. Addressing this trade-off requires deciding when further review is useful and how its evidence should guide prediction updates. To this end, we introduce SurgRoute, a selective multi-agent framework that couples evidence-guided review with controlled correction. Specifically, SurgRoute comprises three coordinated components. First, Joint Probe Gate constructs candidate hypotheses using procedural priors and jointly probes interaction and phase alternatives, learning from the resulting evidence whether additional review is likely to reduce prediction errors. Second, Multi-Agent Review and Repair reuses the probe response within a heterogeneous reviewer panel and controls revisions through explicit candidate-admission and phase-replacement rules, terminating review when further ratings cannot change the prescribed decision. Finally, Tracker-Assisted Correction integrates instrument-tracking evidence, task-specific label constraints, and temporal phase aggregation to refine predictions from both the skip-review and review branches. Experiments on the benchmark demonstrate that SurgRoute consistently improves upon base Vision–Language Models across all evaluated surgical tasks while improving inference efficiency.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Surgical Video Understanding, Vision–Language Models, Selective Multi-Agent Verification, Structured Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hKh1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SurgRoute, a selective multi-agent verification framework for surgical video understanding. The framework consists of three components: Joint Probe Gate (JPG), which learns when additional verification is beneficial using procedural priors and joint interaction-phase evidence; Multi-Agent Review and Repair (MAR), which aggregates feedback from multiple LLMs through constrained correction rules and early stopping; and Tracker-Assisted Correction (TC), which incorporates instrument tracking, label constraints, and temporal phase aggregation. Experiments on SurAgent-Bench using six VLM backbones demonstrate improvements in surgical recognition and report generation, alongside reduced token consumption compared with exhaustive multi-agent review.

Critical strengths
  1. The paper addresses a practically relevant problem in surgical video understanding, where prediction errors and inconsistencies across related tasks can affect downstream report generation.

  2. The framework demonstrates consistent improvements across six VLM backbones and includes diagnostic experiments examining review allocation, correction rules, and computational efficiency.

Critical weaknesses
  1. The methodological novelty is relatively limited. The proposed framework primarily combines established techniques, including gradient-boosted routing, multi-LLM rating aggregation, rule-based correction, and temporal voting. The paper does not sufficiently demonstrate a substantial methodological contribution beyond integrating these existing components.

  2. The experimental evidence does not convincingly support the central motivation for selective multi-agent verification. Always-review achieves only a marginal improvement over skip-review in triplet F1 (46.43 vs. 46.21), despite nearly doubling token consumption. Selective review provides an even smaller improvement over skip-review (46.40 vs. 46.21). Moreover, the substantial contribution of Tracker-Assisted Correction and the absence of a review-free baseline retaining the other correction mechanisms make it difficult to establish the necessity and independent effectiveness of multi-agent verification.

Questions for authors
  1. Could the authors provide an ablation of JPG + TC without MAR to isolate the contribution of multi-agent verification? Given the marginal improvement of always-review over skip-review, what evidence establishes the necessity of additional LLM-based review?

  2. Could the authors independently evaluate the contributions of procedural priors, instrument tracking, label correction, and temporal phase aggregation? The current component ablations combine multiple mechanisms, making it difficult to attribute performance improvements to individual components.

  3. Could the authors provide sensitivity analyses for the procedural prior thresholds (0.01 and 0.70), tracker confidence threshold (0.35), and temporal voting window (60 seconds)?

  4. Could the authors report end-to-end inference latency for skip-review, always-review, and selective-review settings to complement the token consumption analysis?

  5. How well does the framework generalize across different surgical datasets and procedures, particularly given its dependence on procedure-specific priors, manually defined correction rules, and tracker performance?

Reviewer_NSFDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SURGROUTE, a selective multi-agent framework for surgical video understanding. It adaptively decides when further review is useful and how review evidence should guide prediction updates. The proposed method integrates joint prediction gating, multi-agent review, and temporal consistency. It is evaluated on SurAgent-Bench, and shows improved performance.

Critical strengths
  1. This paper presents a good engineering solution, which combines JPG, MAR, and TC. Their effects are well ablated within the evaluated setting, demonstrating solid engineering rigor.
  2. The accuracy-efficiency trade-offs and error pattern analysis provide some diagnostic values for future research.
Critical weaknesses
  1. The authors should better position their paper. The core claim of motivations, the need for mutually consistent perception and interaction predictions, and the accuracy–efficiency trade-off, have been extensively studied in literature. In addition, the claimed novel designs, JPG, MAR and TC, are largely adapted from existing techniques. No new insight is offered as to why this particular combination is uniquely suited for surgical video understanding.
  2. Experiments are conducted on a single, unpublished benchmark without cross-dataset evaluation or statistical significance testing. The authors should also evaluate their method on publicly available datasets and compare directly against published results.
  3. The paper does not cite or discuss several directly comparable efforts that share the same overarching goal of perception–interaction consistency and procedure-level reasoning.
  4. The decision logics for the three agents are simple and largely hand-crafted. The authors did not provide sufficient theoretical or empirical justification on their design choice.
Questions for authors
  1. What are the motivations of the proposed method after a more thorough review of literature? What are the novel designs of the proposed method in regarding to the similar designs in literature?
  2. Can the authors evaluate their method on publicly available benchmarks and report statistical significance tests to support the claimed improvements? Will the dataset used in this paper be publicly released?
  3. Given the simple decision logics of the agents, what are their novel designs?
Reviewer_vTccOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a pipeline to correct predictions of a general-purpose VLM for surgical video understanding, in particular surgical action triplet recognition and surgical phase prediction, i.e. multi-label and multiclass prediction. The authors propose a combination of LLM based review and handcrafted filtering. Furthermore, to reduce the LLM overhead a learned gate is used to skip review when the predicted benefit is below a certain threshold. The pipeline consists of 5 steps: 1) a general-purpose VLM predicts preliminary labels for a given frame, 2) the initial prediction is expanded into a candidate pool, and phase-conditioned training statistics are used to filter unlikely triplets, 3) given a Qwen based scoring of the candidates, a learned gate predicts whether the candidates require LLM review or directly proceed to the final handcrafted filtering (step 5), 4) a panel of LLMs reviews and scores the candidates where edits are accepted based on reviewer agreement, 5) a set of tailored filter mechanisms are applied. In particular a frozen instrument detection model is used to overwrite the candidate instrument labels, hardcoded instrument-verb rules are applied to add task-specific verbs and the phase label is smoothed using a moving window. By evaluating their method on a mix of public and private data the authors show an increased F1 score compared to a single VLM baseline. The pipeline is evaluated for 6 different VLMs in the initial prediction step and compared against their respective single VLM baseline. The authors also ablate individual parts of the pipeline and show that the final stage of the pipeline (step 5) contributes the most to the overall performance gain.

Critical strengths

The method is well documented and the ablations, though incomplete, reveal its limitations.

Critical weaknesses

The mechanisms in the proposed pipeline are not novel and it seems like the method is tailored around the target dataset and task. It is hard to understand for what exact reason the authors made specific design decisions. That includes the choice of thresholds, rules and hyperparameters. Given the fact that the LLM review part of the pipeline contributes next to nothing to the increased scores (e.g. the no-review version achieves the highest accuracy for the phase prediction) the main reason for the improvements seems to be the handcrafted filters and the external instrument tracker. This raises the question to what extent the method can generalize to OOD data and evaluations in that regard are missing in the paper.

Questions for authors
  1. What do the scores look like in Table 2 for a TC-only version? What does the LLM review actually contribute?
  2. How does the method compare to models actually trained on the data? I.e. no pre-trained VLMs tested zero-shot on it.
  3. How does the method perform on other datasets?
  4. As the dataset contains private patient data, has any of it been sent to external APIs? I am missing an ethics statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.