Learning When to Review: Selective Multi-Agent Verification for Surgical Video Understanding

Reliable surgical video understanding requires accurate and mutually consistent predictions across perception (*e.g.*, instrument recognition) and interaction understanding (*e.g.*, triplet recognition) to support procedure-level reasoning and summarization (*e.g.*, surgical report generation). Existing approaches use Vision–Language Models to infer these surgical states, yet prediction errors can propagate into downstream reasoning and compromise the factual accuracy of generated reports. Additional model review offers an opportunity to correct these errors, but reviewing every observation incurs redundant computation and can alter predictions that were already correct. Addressing this trade-off requires deciding when further review is useful and how its evidence should guide prediction updates. To this end, we introduce SurgRoute, a selective multi-agent framework that couples evidence-guided review with controlled correction. Specifically, SurgRoute comprises three coordinated components. First, Joint Probe Gate constructs candidate hypotheses using procedural priors and jointly probes interaction and phase alternatives, learning from the resulting evidence whether additional review is likely to reduce prediction errors. Second, Multi-Agent Review and Repair reuses the probe response within a heterogeneous reviewer panel and controls revisions through explicit candidate-admission and phase-replacement rules, terminating review when further ratings cannot change the prescribed decision. Finally, Tracker-Assisted Correction integrates instrument-tracking evidence, task-specific label constraints, and temporal phase aggregation to refine predictions from both the skip-review and review branches. Experiments on the benchmark demonstrate that SurgRoute consistently improves upon base Vision–Language Models across all evaluated surgical tasks while improving inference efficiency.