acceptodds
Desk rejected at ICLR 2027

Towards Higher-Level Surgical Intelligence: A Comprehensive Benchmark and Multi-Agent Framework for Surgical Intent Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent years, surgical video understanding has progressed from instrument, action, and phase recognition to multimodal video question answering. However, to achieve higher-level surgical intelligence that can assist surgical maneuvers and support clinical decision-making, models need to understand the clinical intent underlying surgical actions: why an ongoing maneuver is performed and which subsequent surgical objective it prepares for. Therefore, we introduce the Surgical Intent Understanding task for endoscopic surgical videos, which jointly evaluates four clinical capabilities: anatomical localization, interpretation of instrument roles, interpretation of instrument coordination, and inference of subsequent surgical objectives. In consultation with experienced surgeons, we define four corresponding evaluation dimensions and construct the SurgIntent-VidQA dataset, comprising 603 video clips and 2,520 surgeon-annotated and reviewed five-option, single-answer questions across four procedure types in liver and lung surgery. Based on this dataset, we establish SurgIntent-Bench for systematic evaluation. We further propose SurgIntent-Agent, a knowledge-guided multi-agent framework that generates candidate intent hypotheses using surgical knowledge, acquires video evidence through targeted questions, and iteratively verifies and refines its predictions. We train the agents using supervised fine-tuning and direct preference optimization. Systematic evaluation on SurgIntent-Bench covers mainstream general-purpose, open-source video, medical, and task-specific fine-tuned multimodal large language models. SurgIntent-Agent achieves an overall accuracy of 67.62%, outperforming the strongest general-purpose model (61.32%) and task-specific SFT baseline (63.72%) by 6.30 and 3.90 percentage points, respectively. Ablation studies further support the contributions of surgical knowledge, visual evidence extraction, and preference optimization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surgical Video Understanding, Multimodal Large Language Models, Multimodal Reasoning, Agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_U7KZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I used GPT to translate and polish my review. Below is the text I provided to the model: This paper studies surgical intent understanding from video. It introduces a dataset of 603 clips and 2,520 multiple-choice questions. The questions cover anatomy, instrument function, instrument coordination, and subsequent surgical objectives. The proposed multi-agent system retrieves knowledge, asks visual questions, and revises its answers. The agents are trained with supervised fine-tuning and preference optimization. The system achieves 67.62% test accuracy, outperforming the strongest fine-tuned baseline by 3.90 percentage points. Ablations show benefits from visual evidence training and preference optimization.

Critical strengths

1.The problem is worth studying. Surgeon involvement in annotation and review adds value to the dataset. Evaluating instrument roles, coordination, and subsequent objectives within the same clip also provides a concrete setting for studying surgical video understanding. 2.The evaluation covers several model families and includes component ablations. The DPO comparison is particularly useful. It shows an improvement while keeping the visual observer, prompts, and interaction limit unchanged. This supports the value of training the reasoning policy within the proposed system.

Critical weaknesses

1.The dataset’s provenance and annotation reliability need more detail. The paper does not report how many distinct surgeries, patients, or source videos underlie the 603 clips. It is also unclear whether the training and test sets share these sources. The authors describe expert review and adjudication, but do not report agreement before adjudication or the proportion of disputed cases. These details matter for assessing benchmark quality, especially for interpretive labels such as subsequent surgical objectives. 2.The methodological contribution is not yet clear enough. Prior work already uses hypothesis-based reasoning, evidence gathering, and verification. A direct comparison with the closest methods is missing. The full system also uses additional visual QA data, action preferences, and multiple rounds of inspection. It is therefore hard to separate the benefit of active verification from that of additional supervision and compute. The ablations are useful, but do not fully resolve this issue.

Questions for authors

1.How many distinct surgeries, patients, and source videos are included? How are they distributed across the training and test sets? If these sets share sources, can you report results on a split that keeps source videos or surgical cases separate? 2.How would a simple multi-round inspection baseline or the closest video-agent method perform with the same auxiliary training data and comparable visual input and inference budgets? Does adaptive questioning still provide a benefit? 3.Before and after DPO, how often does the system change an incorrect answer to a correct one, or a correct answer to an incorrect one? Does the gain over the strongest fine-tuned baseline remain reliable after accounting for correlations among questions from the same clip? 4.For interpretive questions such as subsequent objectives, can you report expert agreement before adjudication and the proportion of disputed cases? Please include examples with the full clinical context and all answer choices. These would help clarify how the correct answer is established and how other equally plausible interpretations are ruled out. I would reconsider my assessment and consider raising my score if the authors clarify the dataset’s provenance and annotation reliability, and show that the method’s gains hold under targeted, controlled comparisons.

9 comments

Thank you for your insightful review, your high expertise in this area, and your encouraging feedback on our expert-annotated dataset and DPO ablations. We particularly appreciate your clear guidance at the end of your review on what is needed to reconsider your assessment. Following your exact suggestions, we have compiled full provenance and agreement statistics, evaluated an unseen surgical patient, implemented controlled multi-round/adaptive baselines, and analyzed DPO revision matrices. We detail these results below and will integrate all of them into the revised manuscript.

Q1: On Dataset Provenance, Split Distribution, and Unseen-Case Evaluation (W1 & Q1)

We sincerely thank you for asking how the source recordings were distributed across the training and test sets, and we apologize for omitting this breakdown from the original submission. The 603 clips were curated from ~15 hours of footage across five source recordings. The two clinical liver recordings came from two distinct patients, and the three lung recordings came from publicly available thoracoscopic surgery teaching videos.

Source RecordingTrain ClipsTrain QuestionsTest ClipsTest Questions
Liver V0111848639177
Liver V0212058139202
Lung V011246727
Lung V021866728
Lung V0317967464233
Total4471,853156667

Within each recording, we assigned clips to the training and test sets by temporal block, keeping each clip and all its questions in a single split. All 156 test clips came from recordings that also contributed training clips. At boundaries with available timestamps, the gap between the nearest selected clips on opposite sides averaged 30.430.4 minutes, with a minimum of 24.224.2 minutes.

While the >24 min>24\text{ min} boundary gap prevents adjacent frame leakage, we fully agree that evaluating on an independent surgical case is vital to rule out case-specific overfitting. Therefore, we asked our surgical team to annotate a newly collected, unseen liver surgery patient (60 clips, 240 five-option questions across all four dimensions) using the exact same protocol:

ModelFunctional (IP)Coordinative (IC)Prospective (NG)Anatomical (AM)Overall (Unseen Case)
Qwen3.5-4B (SFT)76.67%46.67%33.33%43.33%50.00% (120/240)
SurgIntent-Agent (Ours)83.33%48.33%53.33%56.67%60.42% (145/240, +10.42%)

On this independent surgical case, SurgIntent-Agent surpasses the strongest SFT baseline by +10.42%+10.42\% overall and +20.00%+20.00\% on Prospective Intent (32/6032/60 vs. 20/6020/60), demonstrating strong cross-patient generalization. We will add both tables to Sections 3.2 and 5.2.

Q2: On Resource-Matched Multi-Round Baselines and Isolating Adaptive Probing (W2 & Q2)

To strictly separate our active hypothesis-verification mechanism from auxiliary supervision and inference compute, we conducted two targeted comparisons as you suggested:

  1. Multi-Round Inspection Baseline with Matched VEA Data & Runtime Budget: We trained Qwen3.5-4B (SFT) on the 1,8531,853 intent questions plus the exact same 2,1762,176 VEA visual QA pairs. We evaluated the same checkpoint under both single-pass inference and a runtime-calibrated multi-round inspection protocol using predefined clinical questions on anatomy, instrument roles and coordination, and tissue changes.

    Model / Inference SettingMean Calls / QuestionMean Cumulative Frames / QuestionMean Online Time (s/question)Correct / TotalAccuracy
    Qwen3.5-4B (SFT + VEA) — Single-pass1163.26419/66762.82%
    Qwen3.5-4B (SFT + VEA) — Fixed multi-round69679.29383/66757.42%
    SurgIntent-Agent (Full)6.8173.1178.04451/66767.62%
  2. Adaptive vs. Non-Adaptive Questioning within the Same Agent: Holding the HA, VEA, and SIRA checkpoints, initial HA predictions, and a 3-round budget strictly fixed, we compared Non-Adaptive Questioning (SIRA generates probes from the initial hypotheses without seeing VEA's prior visual responses) against Adaptive Questioning (SIRA conditions each next probe on newly acquired visual evidence). Both variants started with the same first probe and could use all three rounds of visual evidence for the final answer. Only the adaptive questioner could use earlier visual feedback to choose its next probe.

    Controlled Agent Variant (Same Weights, Same Initial HA, Fixed 3 Rounds)Correct AnswersAccuracy (%)
    Non-Adaptive Questioning (No observation feedback in probe generation)414/66762.07%
    Adaptive Questioning (Closed-loop evidence-guided probing)443/66766.42% (+4.35%)

Under identical data, weights, and round budgets, adaptive questioning yields 29 more correct answers (+4.35%+4.35\%), confirming that dynamically targeting competing clinical interpretations drives our framework's advance. We will add these controlled experiments to Section 5.3.


Q3: On DPO Answer-Revision Dynamics and Clip-Clustered Statistical Reliability (Q3)

  • Before vs. After DPO Revision Matrix: Starting from the exact same HA initial predictions (404404 correct, 263263 incorrect), we tracked how SIRA revises answers with and without DPO:
SIRA PolicyInitial Errors Corrected (/263/263)Correct Answers Flipped to Wrong (/404/404)Net Question Gain over HAFinal Accuracy
Without DPO54 / 263 (20.53%)41 / 404 (10.15%)+13 Qs417 / 667 (62.52%)
With DPO66 / 263 (25.10%)19 / 404 (4.70%)+47 Qs (+34 over w/o DPO)451 / 667 (67.62%)

DPO improves error correction (+12+12 questions) and, crucially, reduces erroneous overturning of initially correct answers by more than half (−22-22 errors, from 10.15%10.15\% down to 4.70%4.70\%).

  • Accounting for Within-Clip Correlations: Using 10,000 clip-clustered bootstrap resamples (sampling the 156 clips as units) and a 100,000-iteration clip-level paired randomization test, the +3.90%+3.90\% gain of SurgIntent-Agent (67.62%67.62\%) over Qwen3.5-4B (SFT) (63.72%63.72\%) has a 95% CI of [1.04,6.79][1.04, 6.79] and a Holm-adjusted paired p=0.01259p = 0.01259 (all Table 2 ablations also satisfy pHolm≤0.0073p_{\text{Holm}} \le 0.0073), proving that the gain is statistically robust to within-clip correlations. We will incorporate both analyses into Section 5.3.

Q4: On Pre-Adjudication Agreement and Full Clinical Examples for Prospective Intent (W1 & Q4)

Pre-Adjudication Dispute Rates. Five surgeons wrote and cross-reviewed the annotations, and two senior surgeons conducted the final review. At first review, a second surgeon could see the proposed answer and assessed whether the video supported it. Across the 2,5202,520 released questions, 98.65%98.65\% (2,486/2,5202,486/2,520) were accepted without dispute at first review.

Intent DimensionReleased QuestionsDisputed at First ReviewDispute Rate
Anatomical (AM)20100.00%
Functional (IP)1,20700.00%
Coordinative (IC)50950.98%
Prospective (NG)603294.81%
Overall2,520341.35%

Of the 34 disputed questions, 29 concerned prospective intent. The main point of disagreement was how far ahead the visible maneuver justified looking: the immediately ensuing step or a later operative objective. Surgeons revised disputed items by consensus and excluded those for which they could not reach agreement or identify sufficient visual support.

Concrete Examples of Ruling Out Plausible Prospective Distractors. Below are two representative NG items showing how surgeons distinguished between objectives that were clinically plausible within the same procedure.

Example 1. Laparoscopic-assisted left hepatectomy

Clinical context provided to the model. Laparoscopic-assisted left hepatectomy. The current operative phase was not provided.

Question. Based on the video evidence, which subsequent surgical objective is the current maneuver preparing for?

  • (A) Release the remaining gallbladder-bed attachments while maintaining hemostasis and protecting previously treated structures.
  • (B) [Correct] Divide the remaining round or falciform ligament attachments to increase left-lobe mobility and improve lateral exposure.
  • (C) Pause dissection to control a focal bleeding point and restore a clear operative field before proceeding.
  • (D) Continue pericholecystic dissection to define the gallbladder neck and the cystic duct and artery in the hepatocystic triangle.
  • (E) After completing mobilization and planning vascular control, mark and begin the planned transection of the left hepatic parenchyma.

Candidate knowledge provided to the model. Dividing round and falciform ligament attachments can progressively mobilize the left liver and improve exposure. Once the ligamentous sheet is fully released, the lobe can be rotated to inspect the posterior plane. If the mobilization surface is oozing, local coagulation may be appropriate before proceeding.

Why B over E/A/C/D. Both B and E fit left hepatectomy. The video shows traction and energy dissection along a thin ligamentous band. As the exposed space widens, a residual attachment remains visible. This distinguishes B from E: the surgeon is still releasing the lobe, whereas E explicitly presupposes that mobilization is complete. A and D shift the operative target to the gallbladder region. C would require evidence of focal bleeding sufficient to interrupt the ongoing dissection.

Reviewer_fcQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines Surgical Intent Understanding along four dimensions (anatomical, functional, coordinative, prospective) and introduces SurgIntent-VidQA, a benchmark of 603 clips from four liver and lung procedures with 2,520 surgeon-written five-option questions, split into train and test by clip-level temporal blocks. It also proposes SurgIntent-Agent, a pipeline of a visual-evidence agent, a GPT-5.6-sol knowledge retriever, a hypothesis agent and a DPO-tuned reasoning agent that asks up to three follow-up visual questions. On the test set the agent outperforms a Qwen3.5-4B SFT baseline and GPT-5.6-sol, all given the same retrieved knowledge (Table 1). Table 1 also benchmarks 22 zero-shot MLLMs and three SFT models.

Critical strengths
  1. A genuine, expert-built resource with a careful protocol. Questions and answers are surgeon-written and double-reviewed, distractors are real surgeon answers matched for length and terminology, and all baselines receive the same retrieved knowledge. Expert-written surgical VQA above the recognition level is rare, and the dataset is the paper's most durable contribution.
  2. Transparent design and honest reporting. Agent interfaces are information-restricted, Table 3 reports text-only accuracy well above chance, and the paper states that cross-patient generalization is untested. The limitations are stated openly.
Critical weaknesses
  1. The agent's headline advantage is not established. The margin over the SFT baseline (Table 1) comes from a single run on the 667-question test set, with no seeds, intervals or paired test. The baseline is not matched for data or compute: the agent adds a separate GPT-generated, surgeon-reviewed VEA training set, a preference set and up to three inference rounds, while the baseline is trained once on the intent training set with the same retrieved knowledge. Without the DPO stage the agent scores below the SFT baseline (Table 2), so the whole margin rests on a brief DPO run (Table 7) whose effect is unexplained. The text layer of Figure 3 additionally contains an unrendered GRPO / reward-critic training description that contradicts the stated offline-DPO protocol.
  2. The split design limits what fine-tuned numbers mean. Train and test are separated by clip-level temporal blocks, with no statement that recordings are disjoint; the number of source recordings or patients is never given; and the authors concede that "cross-patient and cross-institution generalization remains to be established". Fine-tuned models can therefore learn patient-specific anatomy and step order, and SFT raises next-goal and coordinative accuracy sharply for the two models that have both a zero-shot and an SFT row (Table 1). No inter-annotator agreement or human ceiling is reported.
Questions for authors
  1. How many distinct source recordings and patients make up the 603 clips, and how many test clips come from recordings that also supply training clips? Please report the Qwen3.5-4B SFT baseline and SurgIntent-Agent under a recording-level or procedure-held-out split.
  2. Please report clip-clustered bootstrap confidence intervals and a paired test for the agent versus the Qwen3.5-4B SFT baseline and for each Table 2 row, together with seed variance for the trainable components.
  3. The text layer of Figure 3 describes a GRPO update with an answer reward and a gold-intent term. Was SIRA ever trained that way, and does the reported checkpoint come from the offline-DPO protocol described in the method section? How many preference pairs and optimizer steps did that DPO run use?
  4. Was the Qwen3.5-4B SFT baseline trained with the retrieved knowledge in its input, and what does it score when also trained on the VEA pairs and given a forward-pass budget matched to the agent's three rounds?
  5. Was the GPT-generated candidate knowledge audited for answer leakage, for example by lexical overlap with correct versus distractor options, or by accuracy from knowledge and question alone without video?
  6. What is the inter-annotator agreement on a doubly-annotated subset, and what is surgeon accuracy on a subset of the test set?
10 comments

Thank you for your exceptionally thorough and constructive review, and for recognizing SurgIntent-VidQA as a genuine, expert-built resource with transparent design. We are especially grateful to you for pointing out the residual GRPO text in Figure 3. Your question also prompted us to recheck the training record and identify the incorrect update count in Table 7. We fully agree that rigorous statistical testing, resource-matched baselines, and cross-patient evaluations are essential to substantiate our claims. Following your guidance, we report the additional experiments and analyses below and will revise the manuscript accordingly.

Q1: On Source Provenance and Independent Cross-Patient Generalization (W2 & Q1)

We apologize for omitting the source-recording breakdown in the initial text. The 603 clips were curated from ~15 hours of footage across five source recordings: Liver V01 (118 train / 39 test clips), Liver V02 (120 / 39), Lung V01 (12 / 7), Lung V02 (18 / 7), and Lung V03 (179 / 64). The two clinical liver recordings came from two distinct patients. The three lung recordings came from publicly available surgical teaching videos online.

Clips were separated by temporal blocks within each recording. At boundaries with available timestamps, the gap between the nearest selected clips on opposite sides averaged 30.430.4 minutes, with a minimum of 24.224.2 minutes. All 156156 test clips share source recordings with the training set. The original evaluation therefore measures how well models interpret unseen operative segments from the same source recordings. To directly evaluate case-held-out generalization, we asked our surgical team to annotate a newly recorded liver surgery case from a patient absent from training (60 clips, 240 five-option questions across the four dimensions):

ModelFunctional (IP)Coordinative (IC)Prospective (NG)Anatomical (AM)Overall (Unseen Patient)
Qwen3.5-4B (SFT)76.67%46.67%33.33%43.33%50.00% (120/240)
SurgIntent-Agent (Ours)83.33%48.33%53.33%56.67%60.42% (145/240)
  • While Qwen3.5-4B (SFT) drops to 50.00%50.00\% (120/240120/240) on the unseen patient, with 33.33%33.33\% accuracy on Prospective Intent, SurgIntent-Agent achieves 60.42%60.42\% (145/240145/240), outperforming the SFT baseline by 10.4210.42 percentage points overall (IP: 83.33%83.33\% vs. 76.67%76.67\%, IC: 48.33%48.33\% vs. 46.67%46.67\%, NG: 53.33%53.33\% vs. 33.33%33.33\%, AM: 56.67%56.67\% vs. 43.33%43.33\%). We will add the provenance table and this held-out patient evaluation to Sections 3.2 and 5.2.

Q2: On Clip-Clustered Confidence Intervals, Paired Tests, and Seed Variance (W1 & Q2)

We performed 10,000 clip-clustered bootstrap resamples (sampling the 156 test clips with replacement) and 100,000 clip-level randomizations per paired test, with Holm step-down correction across all five comparisons:

Model / Ablation RowAccuracy (%) [95% CI]Δ\Delta (Full −- Row, percentage points) [95% CI]Raw Paired ppHolm-Adjusted pp
SurgIntent-Agent (Full)67.62 [63.61, 71.62]———
Qwen3.5-4B (SFT) Baseline63.72 [59.72, 67.58]+3.90 [1.04, 6.79]0.012590.01259
w/o surgical knowledge56.67 [53.03, 60.24]+10.94 [7.69, 14.20]0.000010.00005
w/o SIRA (HA only)60.57 [56.77, 64.34]+7.05 [3.95, 10.12]0.000030.00009
w/o VEA-SFT58.77 [54.82, 62.71]+8.85 [6.31, 11.40]0.000010.00005
w/o SIRA-DPO62.52 [58.96, 66.67]+5.10 [1.75, 7.89]0.003630.00726

All improvements are statistically significant (pHolm<0.05p_{\text{Holm}} < 0.05) after accounting for within-clip correlations.

Training-seed variability. Seed 42 denotes the original reported run. For each additional run, we changed the training seed of the indicated component and kept the other two components at their original weights. The table reports SurgIntent-Agent accuracy on the liver subset, lung subset, and full test set.

Training ConfigurationSeedLiver Accuracy (%)Lung Accuracy (%)Overall Accuracy (%)
Original reported run4272.3061.4667.62
SIRA-DPO retrained12369.9261.8166.42
SIRA-DPO retrained202673.6160.4267.92
SIRA-DPO mean ±\pm sample SD42, 123, 202671.94±1.8771.94 \pm 1.8761.23±0.7261.23 \pm 0.7267.32±0.7967.32 \pm 0.79
VEA-SFT retrained10073.0959.0367.02

Across all three SIRA-DPO seeds, SurgIntent-Agent exceeded the reported Qwen3.5-4B (SFT) baseline of 63.72%63.72\% overall. We will include these intervals, pp-values, and seed variance analyses in Tables 1 & 2.

Q3: On the Mechanism of DPO, Figure 3 Text Layer, and Table 7 Steps (W1 & Q3)

  • Clarification & Apology on Fig. 3 and Table 7: We sincerely thank you for flagging the residual GRPO text in Figure 3 and prompting us to verify the training record. SIRA was never trained with GRPO. The reported checkpoint (67.62%67.62\%) comes strictly from the offline DPO protocol described in Section 4.3. An early draft of Figure 3 included a conceptual note on GRPO that was visually cropped out in our drawing software, but the PDF exporter inadvertently retained the invisible text layer. Additionally, the DPO training run used 1,6041,604 preference pairs (derived from 459459 intent questions) and 101101 optimizer steps. “20 updates” in Appendix Table 7 was a typographical error. We will re-export a clean Figure 3 and fix Table 7 in the revision.

  • Why DPO is Crucial (Transition Matrix): Without DPO, uncalibrated multi-round probing suffers from over-revision—changing initially correct HA answers when VEA returns ambiguous observations. Tracking the exact same 667667 initial HA predictions (404404 correct, 263263 incorrect), we compared how SIRA revised its answers with and without DPO.

SIRA TrainingInitial Errors Corrected (out of 263)Correct Initial Answers Changed to Wrong (out of 404)Final Accuracy
Without DPO54 (20.53%)41 (10.15%)417/667 (62.52%)
With DPO66 (25.10%)19 (4.70%)451/667 (67.62%)

DPO yielded 34 additional correct answers by correcting 12 more initial errors and making 22 fewer erroneous revisions. We will add this transition analysis to Section 5.3 to explain DPO’s exact behavioral role.


Q4: On Matching Knowledge, VEA Data, and Inference Budget for SFT (W1 & Q4)

Yes, the Qwen3.5-4B (SFT) baseline (63.72%63.72\%) received retrieved surgical knowledge during both training and evaluation for IP, IC, and NG. At evaluation, it received the same retrieved entries as SurgIntent-Agent for each question. Anatomical questions (AM) received no candidate knowledge.

We further augmented Qwen3.5-4B (SFT) with the same 2,1762,176 visual QA pairs used to train VEA. We evaluated the same SFT checkpoint under single-pass inference and fixed-question multi-round inspection, calibrating the multi-round budget against the agent’s online runtime.

Model / Inference SettingMean Calls / QuestionMean Cumulative Frames / QuestionMean Online Time (s/question)Correct / TotalAccuracy
Qwen3.5-4B (SFT + VEA) — Single-pass1163.26419/66762.82%
Qwen3.5-4B (SFT + VEA) — Fixed multi-round69679.29383/66757.42%
SurgIntent-Agent (Full)6.8173.1178.04451/66767.62%

Within SurgIntent-Agent, we also compared adaptive and non-adaptive probing with the same checkpoints, initial answers, and fixed three-round budget. Both variants used all three rounds of visual evidence for the final decision. Allowing earlier observations to guide the next probe increased accuracy from 62.07%62.07\% (414/667414/667) to 66.42%66.42\% (443/667443/667), yielding 29 more correct answers (+4.35 percentage points). We will include these controlled comparisons in Section 5.3.

Q5: On Auditing Candidate Knowledge vs. Video Contribution (Q5)

Table 3 reports the no-video check you suggested. We retained the question, answer options, and clinical context and measured accuracy with and without candidate knowledge. Adding candidate knowledge raises no-video accuracy from 37.78%37.78\% to 45.28%45.28\% for Qwen3.5-4B (SFT) and from 41.68%41.68\% to 52.32%52.32\% for GPT-5.6-sol. Because retrieval queries use an initial visual caption dd, the retrieved knowledge can carry scene-specific cues even when the model does not see the video itself.

Every baseline in Table 1 receives the same candidate knowledge. With that knowledge held fixed, restoring the video further improves accuracy by 18.4418.44 percentage points for Qwen3.5-4B (SFT) and 9.009.00 points for GPT-5.6-sol. Thus, candidate knowledge provides useful answer cues, and direct observation provides additional evidence for identifying the operative intent. We will expand this discussion in Section 5.3.


Q6: On Expert Agreement and Human Surgeon Performance (W2 & Q6)

During our two-stage review (five annotating surgeons cross-checking, two senior surgeons adjudicating), a second surgeon reviewed each proposed answer against the video. Since the reviewer could see the proposed answer, our records measure disagreement at first review rather than independent agreement between two blinded annotators. The pre-adjudication first-review dispute rate across all 2,5202,520 released questions was 1.35%1.35\% (34/2,52034/2,520): AM 0.00%0.00\% (0/2010/201), IP 0.00%0.00\% (0/1,2070/1,207), IC 0.98%0.98\% (5/5095/509), and NG 4.81%4.81\% (29/60329/603). Disputed items were revised by consensus or excluded if the surgeons could not agree or found insufficient visual evidence.

In our newly conducted human evaluation on a 100-question dimension-balanced test subset (25 per dimension), all three participants independently watched the videos and answered the same five-option questions without access to the reference answers. An independent practicing surgeon achieved 94.0%94.0\% (94/10094/100) accuracy, and two medical students scored 76.0%76.0\% and 83.0%83.0\%. We will report both statistics in Section 3.2.

We hope these comprehensive experiments and corrections fully address your concerns, and we sincerely appreciate your reconsideration of our work.

Reviewer_pL5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Surgical Intent Understanding for endoscopic surgical video and constructs SurgIntent-VidQA, an expert-annotated five-option benchmark of 603 clips and 2,520 questions covering anatomical, functional, coordinative, and prospective intent. It proposes SurgIntent-Agent, a knowledge-guided multi-agent framework with a Hypothesis Agent, a Surgical Intent Reasoning Agent, and a Visual Evidence Agent, trained via SFT and DPO. The framework reaches 67.62% overall accuracy, an absolute improvement of 3.9 and 6.3 percentage points over the strongest task-specific SFT baseline and GPT-5.6-sol, respectively, though the method is largely integrative and generalization remains limited.

Critical strengths

The paper defines a clinically meaningful Surgical Intent Understanding task and provides SurgIntent-VidQA, an expert-annotated five-option benchmark of 603 clips and 2,520 questions with a detailed annotation and quality-control protocol. This is consequential because it moves surgical video understanding beyond instrument/action recognition and offers a reusable evaluation resource. The proposed SurgIntent-Agent is also well motivated: its knowledge-guided hypothesis–probing–evidence loop directly targets the mismatch between clinically plausible explanations and the current video, and ablations show substantial drops when surgical knowledge (10.95 points), iterative verification (7.05), VEA-SFT (8.85), or SIRA-DPO (5.1) is removed, supporting acceptance as a solid resource-and-framework contribution despite moderate overall gains.

Critical weaknesses

The paper presents SurgIntent-Agent as a framework-level advance, but the evidence does not fully isolate the contribution of the multi-agent probing loop itself. The overall gain over the strongest task-specific SFT baseline is 3.9 percentage points, and the framework does not consistently outperform simpler or general models across the four dimensions, with anatomical intent at 44.23% versus 78.85% for Gemini-3-Pro. The existing ablations support separate contributions from knowledge, VEA-SFT, SIRA-DPO, and iterative verification, but they do not fully control for knowledge input, training data, and inference budget. This makes it difficult to attribute the 3.9-point gain specifically to the multi-agent probing design rather than to additional resources.

The benchmark is small and narrow: 603 clips totaling about 133 minutes from only four liver and lung procedures, with no cross-center, cross-patient, or cross-procedure generalization evidence, and no annotator-agreement statistic. This is consequential because the central claim is a new benchmark for surgical intent understanding, yet the current scale and coverage cannot establish that models are evaluated on clinically representative intent reasoning rather than a curated short-clip task. Without stronger generalization and annotation-reliability evidence, acceptance as a benchmark contribution remains premature.

Questions for authors

Could the authors clarify how much of the 3.9 point gain is attributable to the multi-agent probing loop itself rather than to additional knowledge input, training data, and inference budget? If a fully controlled comparison is not feasible, could they explain why the existing ablations are sufficient to support the framework-level claim?

Could the authors report annotator-agreement statistics and clarify the representativeness of the current benchmark, including whether the data support cross-center, cross-patient, or cross-procedure evaluation? If such evaluation is not feasible, what are the explicit boundaries of the benchmark’s clinical representativeness?

7 comments

Thank you for your constructive review and for highlighting the clinical significance of our task, the rigorous quality control of SurgIntent-VidQA, and the well-motivated design of SurgIntent-Agent. We fully agree that in the initial manuscript, our ablations did not sufficiently disentangle the multi-agent probing mechanism from auxiliary supervision/compute, and the dataset's provenance boundaries were not stated explicitly enough. Guided by your insightful questions, we have conducted strictly controlled experiments and an independent cross-patient evaluation, which we will incorporate into the revised paper.

Q1: On Disentangling Multi-Agent Probing from Auxiliary Supervision, Knowledge, and Inference Budget (W1 & Q1)

We appreciate this critical point. We respectfully clarify that the 3.903.90-point headline gain is not merely a byproduct of extra training data or compute, which we verify through two newly added, strictly controlled studies:

  • Matching Knowledge, VEA Supervision, and Multi-Round Compute for the SFT Baseline: All methods in Table 1 received the same retrieved surgical knowledge for each functional, coordinative, and prospective question at evaluation, and the original Qwen3.5-4B (SFT) baseline also received retrieved knowledge during training. Anatomical questions received no candidate knowledge. Both Qwen3.5-4B (SFT) and SurgIntent-Agent used the same 1,8531,853 intent training questions. The SFT baseline used all 1,8531,853 for direct answer prediction, while SurgIntent-Agent used 1,3941,394 to train HA and the remaining 459459 to construct action preferences for SIRA. To match auxiliary visual supervision, we trained Qwen3.5-4B (SFT) with the same 2,1762,176 VEA visual QA examples and evaluated this checkpoint under both single-pass answering and fixed-question multi-round inspection. We calibrated the multi-round budget using the online runtime of SurgIntent-Agent.
Model VariantMean Model Calls / QuestionMean Cumulative Frames / QuestionCorrect / TotalAccuracy (%)
Qwen3.5-4B (SFT) — Single-Pass Answering116419 / 66762.82%
Qwen3.5-4B (SFT) — Multi-Round Fixed Inspection696383 / 66757.42%
SurgIntent-Agent (Ours, Adaptive Closed-Loop)6.8173.11451 / 66767.62%

Mean online inference time was 78.0478.04 s/question for SurgIntent-Agent and 79.2979.29 s/question for the fixed multi-round SFT baseline. Adding the same auxiliary visual supervision and giving the baseline multiple rounds of inspection did not close the accuracy gap.

  • Isolating the Probing Loop under Identical Checkpoints and a Fixed Three-Round Budget: To strictly isolate our feedback-guided probing design, we froze the same HA, VEA, and SIRA checkpoints, fixed the initial HA answer and first probe for each of the 667667 test questions, and enforced an identical three-round budget. Both variants used all three rounds of visual evidence when selecting their final answer. They differed only in whether SIRA could use earlier visual feedback to choose its next probe. Comparing Non-Adaptive Probing (where SIRA generates its next probe without seeing VEA's prior visual responses or the resulting state updates) against Adaptive Probing (where SIRA uses this feedback to decide what to inspect next) shows that adaptive questioning improves accuracy from 62.07%62.07\% (414/667414/667) to 66.42%66.42\% (443/667443/667). With the same three rounds of inspection, adaptive probing yielded 2929 more correct answers, a gain of 4.354.35 percentage points. Regarding anatomical intent (AM), as you observed, Gemini-3-Pro leads on anatomical questions (78.85%78.85\% vs. 44.23%44.23\% for SurgIntent-Agent), while our 4B-based agent achieves the highest scores on functional intent (IP: 64.81%64.81\%) and prospective intent (NG: 80.13%80.13\%). We will add these controlled comparisons to Section 5.3 to further substantiate our framework-level claim.

Q2: On Annotation Reliability, Dataset Provenance, and Cross-Patient Generalization (W2 & Q2)

Thanks for raising this important concern regarding benchmark representativeness. While the 603 clips total about 133 minutes, they were distilled by five surgeons from about 15 hours of raw footage across five source recordings (two clinical liver recordings from two distinct patients and three public lung teaching recordings) to capture dense, clinically decisive maneuvers rather than repetitive idle phases. We respectfully clarify that, although the total video duration is modest, surgeons carefully selected each clip from lengthy surgical recordings to capture a maneuver representative of a specific operative stage, prioritizing expert clinical curation and the study of surgical intent over total footage length. To establish reliability and generalization boundaries transparently:

  • Expert Agreement & Human Performance: Our annotation protocol employed a rigorous two-stage consensus review (five annotating surgeons cross-checking, adjudicated by two chief surgeons). At first review, a second surgeon saw the proposed answer and assessed whether the video supported it. Across all 2,5202,520 released questions, the pre-adjudication first-review dispute rate was only 1.35%1.35\% (34/2,52034/2,520): 0.00%0.00\% on Anatomical (0/2010/201) and Functional (0/1,2070/1,207), 0.98%0.98\% on Coordinative (5/5095/509), and 4.81%4.81\% on Prospective (29/60329/603, primarily regarding the temporal horizon of the next goal, which was resolved via consensus). Furthermore, on a 100-question dimension-balanced test subset with 25 questions per dimension, an independent practicing surgeon achieved 94.0%94.0\% accuracy (and two medical students achieved 76.0%76.0\% and 83.0%83.0\%). All three independently answered the same questions without access to the reference answers.

  • Split Boundaries & New Independent Cross-Patient Evaluation: In our main benchmark, train and test clips were partitioned via non-overlapping temporal blocks within the five recordings. All 156156 test clips came from recordings that also contributed training clips. Each clip and its associated questions were assigned to a single split. At split boundaries with available timestamps, the gap between the nearest selected clips on opposite sides averaged 30.430.4 minutes, with a minimum of 24.224.2 minutes. The original evaluation therefore measures how well models interpret unseen operative segments from the same source recordings. To evaluate performance on an unseen patient, we collaborated with surgeons during the rebuttal period to annotate a new liver surgery case from a patient absent from training (60 clips, 240 questions across all four dimensions) following the exact same protocol:

ModelFunctional (IP)Coordinative (IC)Prospective (NG)Anatomical (AM)Overall (Unseen Patient)
Qwen3.5-4B (SFT)76.67%46.67%33.33%43.33%50.00% (120/240)
SurgIntent-Agent (Ours)83.33%48.33%53.33%56.67%60.42% (145/240, +10.42 points)

On this entirely unseen patient, SurgIntent-Agent answered 25 more questions correctly than the strongest SFT baseline (145/240145/240 vs. 120/240120/240, a gain of 10.4210.42 percentage points). The largest gain was on Prospective Intent, where the agent correctly inferred the subsequent objective on 32/6032/60 questions versus 20/6020/60 for SFT (+20.00+20.00 points). This extends our evaluation to one independently recorded liver case. Cross-center and cross-procedure generalization remain open questions. We will integrate the source provenance table, dispute statistics, human study, and this unseen-patient evaluation into Section 3.2 and Section 5.

We hope these new experiments and clarifications fully resolve your concerns, and we deeply appreciate your reconsideration of our work.

Reviewer_pL5yOpenReview ↗

Follow-up Assessment

Thank you for the detailed clarifications and the additional experiments. The adaptive-questioning comparison, the DPO revision analysis, and the clip-level statistical tests all meaningfully strengthen the empirical evidence.

I appreciate the clarification regarding data provenance as well as the additional evaluation on an unseen patient. While individual clips capture different surgical actions, the benchmark is built from five source recordings, and all of these are shared between training and testing. This leaves uncertainty about how the method would perform across independent surgical settings. The added evaluation on one unseen patient is encouraging and goes some way toward addressing this concern, but evidence that the approach generalizes to unseen surgeries is still limited.

The adaptive-questioning ablation supports the value of feedback-guided probing within the framework. However, the fixed multi-round baseline differs from the proposed method not only in its interaction design but also in how it is trained. SIRA receives dedicated preference optimization for probing and evidence assessment, whereas the baseline is trained with QA supervision. For this reason, the comparison does not fully disentangle the benefit of the framework from that of specialized policy training. In addition, the methodological advance over closely related approaches remains insufficiently established.

Overall, I acknowledge the improvements presented in the rebuttal, but the remaining concerns regarding benchmark evaluation and methodological contribution lead me to keep my score at 2.

Reviewer_KbMDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The research work indtroduces a new dataset for surgical intent understanding, to evaluate four clinical capabilities. The 603 annotated videos in the proposed SurgIntent benchmark within a multi-agent knowledge base aims to support advanced analysis in intent predictions.

Critical strengths

A new dataset on for surgical intent understanding, with 603 annotated videos.

Critical weaknesses

While the established new dataset on surgical intent understanding provides a welcome benchmark to the research community, it's not clear what key issues it support addressing. There is also a lack of justifications on why and how the four clinical capabilities are chosen as the main criteria. First results have been reported on this new benchmark. It's not clear what scientific questions these provide to answer which key questions in the area.

Questions for authors

See the above comments.

5 comments

Thank you for your thoughtful review and for recognizing the value of SurgIntent-Bench as a benchmark for the community. We apologize that our initial presentation in Section 1 did not articulate the scientific motivation, the clinical rationale behind the four dimensions, and our main empirical findings clearly enough. Inspired by your valuable comments, we address these questions below and will clarify the corresponding passages in the Introduction and Experimental Analysis.

Q1. On Key Issues Addressed by SurgIntent-Bench

Thanks for this foundational question. We respectfully clarify that SurgIntent-Bench is designed to bridge a critical gap in surgical AI by transitioning from low-level visual perception (what is visible) to high-level clinical decision support (why a maneuver is performed and what it prepares for). Existing benchmarks primarily evaluate instrument or action triplet recognition (Levels 1–2).

This distinction matters for AI-assisted surgical education. Trainees develop their own surgical judgment by observing experienced surgeons and learning why they choose a particular maneuver at a particular point in an operation. In Figure 1, for example, a trainee may recognize that an ultrasonic scalpel is dividing a ligamentous attachment. The teaching question is why the surgeon divides this attachment here. Dividing it helps mobilize the left liver lobe so that its lateral surface can be exposed.

Understanding operative goals is also relevant to decision-making in future surgical robots. A robot learning from expert demonstrations would need to relate the observed action and tissue state to the surgeon’s objective when deciding whether to continue dissection or prepare for the next step. SurgIntent-Bench evaluates one prerequisite for such reasoning. Surgeons write clip-specific questions and reference answers, with distractors drawn from their answers to the same type of question in other clips. The model must therefore identify which of several clinically plausible interpretations the current video supports. We will make this distinction explicit when introducing the Figure 1 example in Section 1.

Q2: On Justification and Selection of the Four Clinical Capabilities

We agree that the clinical grounding of our evaluation taxonomy should be presented more prominently. Rather than an arbitrary selection, the four dimensions were formulated through structured consultations with five practicing surgeons across multiple hospitals to mirror the exact cognitive workflow surgeons use during an operation:

  • Anatomical Intent (Where to operate): Many operative steps begin with the identification of a specific anatomical landmark. Surgeons use their knowledge of the procedure to seek out that landmark, establish the target’s relationship to surrounding structures, and determine where to operate. This is why anatomy belongs in an intent benchmark. The same instrument action can serve different purposes depending on the structure involved. Identifying the operative target provides a basis for interpreting the action.

  • Functional Intent (Why use this instrument here): Interpreting the specific therapeutic or dissective purpose of an individual tool within the immediate tissue state, beyond its generic textbook function.

  • Coordinative Intent (How instruments collaborate): Capturing bimanual synergy (e.g., one forceps maintaining opposing traction while an energy device divides the tensioned interface to widen a plane).

  • Prospective Intent (What next): Assessing operative progress and residual attachments to infer the subsequent surgical objective the current maneuver enables.

Together, these four capabilities form a closed clinical reasoning loop: from spatial grounding (Where), to single- and multi-instrument execution (Why & How), to forward surgical planning (What Next). We will add a dedicated justification paragraph and schematic mapping in Section 3.1 to make this clinical rationale crystal clear.

Q3: On Scientific Questions Answered by Our Experiments

We appreciate the opportunity to highlight the broader scientific insights revealed by SurgIntent-Bench, which we will emphasize in Section 5.2:

  • Can current general or medical MLLMs understand surgical intent, and does overall accuracy tell the whole story? No. Current medical MLLMs lag significantly (≤40.03%\le 40.03\%), and even frontier models reach only 61.32%61.32\%. Crucially, our four-dimensional breakdown exposes hidden capability trade-offs that overall scores obscure: for instance, GPT-5.6-sol and fine-tuned InternVL3.5-8B tie in overall accuracy (61.32%61.32\%), yet GPT-5.6-sol excels in anatomical grounding (67.31%67.31\% vs. 44.23%44.23\%) while InternVL3.5-8B dominates in prospective planning (74.36%74.36\% vs. 57.69%57.69\%). Similarly, Gemini-3-Pro leads in anatomy (78.85%78.85\%) but struggles in functional (55.25%55.25\%) and prospective (58.33%58.33\%) reasoning.

  • How do domain knowledge and direct visual observation interact in clinical reasoning? Surgeons draw on prior knowledge gained through experience to interpret the intent behind a maneuver they see on video, and our results show that models likewise benefit from surgical knowledge. While prior surgical knowledge narrows down clinically plausible hypotheses, our controlled input study (Table 3) proves that direct visual verification is irreplaceable—adding video over text+knowledge boosts accuracy by +18.44%+18.44\% (Qwen3.5-4B SFT) and +9.00%+9.00\% (GPT-5.6-sol), whereas explicit domain knowledge provides complementary gains (+11.40%+11.40\% for open-source SFT models) by bridging clinical rationale gaps.

We hope these clarifications address your concerns and sincerely appreciate your reconsideration of our work.

Reviewer KbMD: The discussion deadline is approaching. Could you please respond to the authors’ rebuttal? Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.