- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
I used GPT to translate and polish my review. Below is the text I provided to the model: This paper studies surgical intent understanding from video. It introduces a dataset of 603 clips and 2,520 multiple-choice questions. The questions cover anatomy, instrument function, instrument coordination, and subsequent surgical objectives. The proposed multi-agent system retrieves knowledge, asks visual questions, and revises its answers. The agents are trained with supervised fine-tuning and preference optimization. The system achieves 67.62% test accuracy, outperforming the strongest fine-tuned baseline by 3.90 percentage points. Ablations show benefits from visual evidence training and preference optimization.
Critical strengths
1.The problem is worth studying. Surgeon involvement in annotation and review adds value to the dataset. Evaluating instrument roles, coordination, and subsequent objectives within the same clip also provides a concrete setting for studying surgical video understanding. 2.The evaluation covers several model families and includes component ablations. The DPO comparison is particularly useful. It shows an improvement while keeping the visual observer, prompts, and interaction limit unchanged. This supports the value of training the reasoning policy within the proposed system.
Critical weaknesses
1.The dataset’s provenance and annotation reliability need more detail. The paper does not report how many distinct surgeries, patients, or source videos underlie the 603 clips. It is also unclear whether the training and test sets share these sources. The authors describe expert review and adjudication, but do not report agreement before adjudication or the proportion of disputed cases. These details matter for assessing benchmark quality, especially for interpretive labels such as subsequent surgical objectives. 2.The methodological contribution is not yet clear enough. Prior work already uses hypothesis-based reasoning, evidence gathering, and verification. A direct comparison with the closest methods is missing. The full system also uses additional visual QA data, action preferences, and multiple rounds of inspection. It is therefore hard to separate the benefit of active verification from that of additional supervision and compute. The ablations are useful, but do not fully resolve this issue.
Questions for authors
1.How many distinct surgeries, patients, and source videos are included? How are they distributed across the training and test sets? If these sets share sources, can you report results on a split that keeps source videos or surgical cases separate? 2.How would a simple multi-round inspection baseline or the closest video-agent method perform with the same auxiliary training data and comparable visual input and inference budgets? Does adaptive questioning still provide a benefit? 3.Before and after DPO, how often does the system change an incorrect answer to a correct one, or a correct answer to an incorrect one? Does the gain over the strongest fine-tuned baseline remain reliable after accounting for correlations among questions from the same clip? 4.For interpretive questions such as subsequent objectives, can you report expert agreement before adjudication and the proportion of disputed cases? Please include examples with the full clinical context and all answer choices. These would help clarify how the correct answer is established and how other equally plausible interpretations are ruled out. I would reconsider my assessment and consider raising my score if the authors clarify the dataset’s provenance and annotation reliability, and show that the method’s gains hold under targeted, controlled comparisons.
9 comments
Q2: On Resource-Matched Multi-Round Baselines and Isolating Adaptive Probing (W2 & Q2)
To strictly separate our active hypothesis-verification mechanism from auxiliary supervision and inference compute, we conducted two targeted comparisons as you suggested:
-
Multi-Round Inspection Baseline with Matched VEA Data & Runtime Budget: We trained
Qwen3.5-4B (SFT)on the intent questions plus the exact same VEA visual QA pairs. We evaluated the same checkpoint under both single-pass inference and a runtime-calibrated multi-round inspection protocol using predefined clinical questions on anatomy, instrument roles and coordination, and tissue changes.Model / Inference Setting Mean Calls / Question Mean Cumulative Frames / Question Mean Online Time (s/question) Correct / Total Accuracy Qwen3.5-4B (SFT + VEA)— Single-pass1 16 3.26 419/667 62.82% Qwen3.5-4B (SFT + VEA)— Fixed multi-round6 96 79.29 383/667 57.42% SurgIntent-Agent(Full)6.81 73.11 78.04 451/667 67.62% -
Adaptive vs. Non-Adaptive Questioning within the Same Agent: Holding the
HA,VEA, andSIRAcheckpoints, initialHApredictions, and a 3-round budget strictly fixed, we compared Non-Adaptive Questioning (SIRAgenerates probes from the initial hypotheses without seeingVEA's prior visual responses) against Adaptive Questioning (SIRAconditions each next probe on newly acquired visual evidence). Both variants started with the same first probe and could use all three rounds of visual evidence for the final answer. Only the adaptive questioner could use earlier visual feedback to choose its next probe.Controlled Agent Variant (Same Weights, Same Initial HA, Fixed 3 Rounds) Correct Answers Accuracy (%) Non-Adaptive Questioning (No observation feedback in probe generation) 414/667 62.07% Adaptive Questioning (Closed-loop evidence-guided probing) 443/667 66.42% (+4.35%)
Under identical data, weights, and round budgets, adaptive questioning yields 29 more correct answers (), confirming that dynamically targeting competing clinical interpretations drives our framework's advance. We will add these controlled experiments to Section 5.3.
Q3: On DPO Answer-Revision Dynamics and Clip-Clustered Statistical Reliability (Q3)
- Before vs. After DPO Revision Matrix: Starting from the exact same
HAinitial predictions ( correct, incorrect), we tracked howSIRArevises answers with and without DPO:
| SIRA Policy | Initial Errors Corrected () | Correct Answers Flipped to Wrong () | Net Question Gain over HA | Final Accuracy |
|---|---|---|---|---|
| Without DPO | 54 / 263 (20.53%) | 41 / 404 (10.15%) | +13 Qs | 417 / 667 (62.52%) |
| With DPO | 66 / 263 (25.10%) | 19 / 404 (4.70%) | +47 Qs (+34 over w/o DPO) | 451 / 667 (67.62%) |
DPO improves error correction ( questions) and, crucially, reduces erroneous overturning of initially correct answers by more than half ( errors, from down to ).
- Accounting for Within-Clip Correlations: Using 10,000 clip-clustered bootstrap resamples (sampling the 156 clips as units) and a 100,000-iteration clip-level paired randomization test, the gain of
SurgIntent-Agent() overQwen3.5-4B (SFT)() has a 95% CI of and a Holm-adjusted paired (all Table 2 ablations also satisfy ), proving that the gain is statistically robust to within-clip correlations. We will incorporate both analyses into Section 5.3.
Q4: On Pre-Adjudication Agreement and Full Clinical Examples for Prospective Intent (W1 & Q4)
Pre-Adjudication Dispute Rates. Five surgeons wrote and cross-reviewed the annotations, and two senior surgeons conducted the final review. At first review, a second surgeon could see the proposed answer and assessed whether the video supported it. Across the released questions, () were accepted without dispute at first review.
| Intent Dimension | Released Questions | Disputed at First Review | Dispute Rate |
|---|---|---|---|
Anatomical (AM) | 201 | 0 | 0.00% |
Functional (IP) | 1,207 | 0 | 0.00% |
Coordinative (IC) | 509 | 5 | 0.98% |
Prospective (NG) | 603 | 29 | 4.81% |
| Overall | 2,520 | 34 | 1.35% |
Of the 34 disputed questions, 29 concerned prospective intent. The main point of disagreement was how far ahead the visible maneuver justified looking: the immediately ensuing step or a later operative objective. Surgeons revised disputed items by consensus and excluded those for which they could not reach agreement or identify sufficient visual support.
Concrete Examples of Ruling Out Plausible Prospective Distractors. Below are two representative NG items showing how surgeons distinguished between objectives that were clinically plausible within the same procedure.
Example 1. Laparoscopic-assisted left hepatectomy
Clinical context provided to the model. Laparoscopic-assisted left hepatectomy. The current operative phase was not provided.
Question. Based on the video evidence, which subsequent surgical objective is the current maneuver preparing for?
- (A) Release the remaining gallbladder-bed attachments while maintaining hemostasis and protecting previously treated structures.
- (B) [Correct] Divide the remaining round or falciform ligament attachments to increase left-lobe mobility and improve lateral exposure.
- (C) Pause dissection to control a focal bleeding point and restore a clear operative field before proceeding.
- (D) Continue pericholecystic dissection to define the gallbladder neck and the cystic duct and artery in the hepatocystic triangle.
- (E) After completing mobilization and planning vascular control, mark and begin the planned transection of the left hepatic parenchyma.
Candidate knowledge provided to the model. Dividing round and falciform ligament attachments can progressively mobilize the left liver and improve exposure. Once the ligamentous sheet is fully released, the lobe can be rotated to inspect the posterior plane. If the mobilization surface is oozing, local coagulation may be appropriate before proceeding.
Why B over E/A/C/D. Both B and E fit left hepatectomy. The video shows traction and energy dissection along a thin ligamentous band. As the exposed space widens, a residual attachment remains visible. This distinguishes B from E: the surgeon is still releasing the lobe, whereas E explicitly presupposes that mobilization is complete. A and D shift the operative target to the gallbladder region. C would require evidence of focal bleeding sufficient to interrupt the ongoing dissection.
Thank you for your insightful review, your high expertise in this area, and your encouraging feedback on our expert-annotated dataset and DPO ablations. We particularly appreciate your clear guidance at the end of your review on what is needed to reconsider your assessment. Following your exact suggestions, we have compiled full provenance and agreement statistics, evaluated an unseen surgical patient, implemented controlled multi-round/adaptive baselines, and analyzed DPO revision matrices. We detail these results below and will integrate all of them into the revised manuscript.
Q1: On Dataset Provenance, Split Distribution, and Unseen-Case Evaluation (W1 & Q1)
We sincerely thank you for asking how the source recordings were distributed across the training and test sets, and we apologize for omitting this breakdown from the original submission. The 603 clips were curated from ~15 hours of footage across five source recordings. The two clinical liver recordings came from two distinct patients, and the three lung recordings came from publicly available thoracoscopic surgery teaching videos.
Liver V01Liver V02Lung V01Lung V02Lung V03Within each recording, we assigned clips to the training and test sets by temporal block, keeping each clip and all its questions in a single split. All 156 test clips came from recordings that also contributed training clips. At boundaries with available timestamps, the gap between the nearest selected clips on opposite sides averaged 30.4 minutes, with a minimum of 24.2 minutes.
While the >24 min boundary gap prevents adjacent frame leakage, we fully agree that evaluating on an independent surgical case is vital to rule out case-specific overfitting. Therefore, we asked our surgical team to annotate a newly collected, unseen liver surgery patient (60 clips, 240 five-option questions across all four dimensions) using the exact same protocol:
Qwen3.5-4B (SFT)SurgIntent-Agent(Ours)On this independent surgical case,
SurgIntent-Agentsurpasses the strongest SFT baseline by +10.42% overall and +20.00% on Prospective Intent (32/60 vs. 20/60), demonstrating strong cross-patient generalization. We will add both tables to Sections 3.2 and 5.2.