acceptodds
Under review as a conference paper at ICLR 2027

GATE: Can Vision-Language Models Ground, Audit, and Track Surgical Events?

Abstract

Vision-language models (VLMs) could one day assist surgeons, but only if they can reliably interpret visual evidence, verify their own interpretations, and follow procedures as they unfold. How well current VLMs support these abilities, and how training can improve them, remain open questions. We introduce GATE, an evaluation framework that integrates a surgical video benchmark with systematic training interventions to assess whether models can Ground, Audit, and Track surgical Events. We therefore establish a capability taxonomy of six categories and 15 tasks, supported by a benchmark of 13,715 human-reviewed questions over 36,264 video clips from eight datasets across six anatomical sites. We evaluate 11 leading proprietary and open-weight models, including the recently released GPT-6 Astra, which is the strongest model in our evaluation yet achieves only 49.0% overall accuracy. On average, models are stronger at recognition and spatial judgments than at workflow reasoning and joint description and workflow correction. This uneven profile makes procedural reasoning and consistent verification central priorities for advancing surgical intelligence. Training interventions offer a concrete path forward: across models adapted separately for each task from a backbone with two billion parameters, supervised fine-tuning followed by reinforcement learning raises pooled test accuracy from 2.3% to 38.9%, compared with 35.7% for supervised fine-tuning and 12.3% for direct reinforcement learning. Supervised fine-tuning accounts for most of the overall accuracy gain, with further improvement from subsequent reinforcement learning. GATE thus provides an assessment framework that quantifies current capability gaps, evaluates how training interventions address these gaps, and guides future research and model development in surgical intelligence.

Primary area:
datasets and benchmarks
Keywords:
Surgical Video Understanding, Vision-Language Models, Multimodal Benchmarks, Surgical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HBjYOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GATE, a benchmark and evaluation framework for assessing whether vision-language models can ground, audit, and track surgical events. GATE organizes surgical understanding into six capability categories and 15 tasks covering phase understanding, scene understanding, temporal reasoning, workflow reasoning, description correction, and joint description/workflow correction. The benchmark contains 13,715 human-reviewed questions constructed from eight existing surgical datasets across six anatomical sites. The paper evaluates 11 proprietary and open-weight VLMs and finds large performance differences across capability types, with models generally performing better on recognition and local spatial judgments than on workflow reasoning and correction tasks. The authors further study supervised fine-tuning, reinforcement learning, and sequential SFT+RL on Qwen3-VL-2B, together with cross-task, category-level, joint-training, text-only, answer-only, and error analyses. The paper also includes a small action-conditioned image/video generation study.

Critical strengths

A major strength of GATE is the breadth of the evaluation. Rather than focusing on a single recognition or VQA task, the benchmark covers local perception, temporal comparison, workflow reasoning, auditing, and correction across multiple procedures and datasets. The task definitions are generally clear, and the authors perform explicit visual quality control before retaining benchmark questions. I also appreciate the unusually extensive diagnostic analysis. The paper does not stop at reporting aggregate accuracy: it examines text-only performance, answer validity, cross-task transfer, joint training, repeated predictions, and visible error patterns. In particular, the analysis showing that some high task accuracies can arise from constant predictions or answer priors is useful and makes the benchmark study more informative than a simple model leaderboard.

Critical weaknesses

My main concern is that the “video” and temporal-reasoning claims are somewhat stronger than the actual input protocol supports. Each clip is represented by only four sampled frames arranged as a static montage, rather than a continuous video sequence. Several temporal and workflow tasks therefore test reasoning over sparse observations rather than full temporal video understanding. This distinction is particularly important because frame spacing varies across datasets and the paper itself notes that observed stability does not imply stability during unobserved gaps. A second concern is that the training-intervention results are difficult to interpret cleanly. The large Base-to-SFT gain partly reflects learning the required response format: most Base responses in the paired SFT pre-test are invalid, while SFT nearly eliminates this failure mode. In addition, the cross-task, category-level, and joint-training experiments allow image overlap across training and test through different tasks. The paper appropriately acknowledges this, but the reported joint-training advantage therefore cannot be interpreted as clean evidence of transferable surgical reasoning. The Related Work section appears rather late for a benchmark paper. Moving the comparison with existing surgical VLM benchmarks earlier could make GATE's novelty easier to assess. The authors may also consider a more domain-specific benchmark name such as Surgical-GATE or SurgGATE, since “GATE” alone is fairly generic.

Questions for authors

Main-text-driven

Q1:GATE is presented as a surgical video benchmark, but each clip is represented by only four sampled frames arranged as a static montage rather than being provided to the model as a native video sequence. This setup may not fully evaluate the temporal modeling capabilities of video-capable VLMs. Could the authors evaluate a representative subset of the temporal or workflow tasks using native video input and compare the results with the current four-frame montage setting?

Q2: The benchmark is a central contribution of the paper, but I could not find a clear data-release plan. Will the authors release the 13,715 questions, reference answers, benchmark splits, prompts, construction/evaluation code, and quality-control metadata? If the original surgical images/videos cannot be redistributed, will identifiers and reconstruction scripts be provided so that researchers with access to the source datasets can reproduce GATE?

Q3: The text-only analysis indicates that several tasks can be partially solved from textual structure or answer priors, and Description and Workflow Correction even performs better without images. Could the authors provide an image-shuffling or mismatched-image control to better quantify how much each task actually depends on visual evidence?

Q4: The benchmark relies on human review to verify that each automatically constructed question is answerable from the displayed visual evidence. Could the authors clarify the background of the reviewers, whether each candidate was independently assessed by more than one reviewer, and whether any inter-reviewer agreement was measured? This seems important for establishing the reliability of the benchmark quality-control process.

Appendix-driven

Q5: The per-task splits are image-disjoint, but surgical cases can overlap, and the cross-task/category/joint experiments contain substantial train–test image overlap across tasks. Could the authors provide a globally image-disjoint or case-disjoint evaluation for the joint-training setting, or otherwise quantify how much the reported 48.23% joint-training result may benefit from prior exposure to test images through other tasks?

7 comments
AuthorsOpenReview ↗

Responses to Reviewer HBjY (1/5)

We thank Reviewer HBjY for the constructive feedback and for recognizing GATE's broad coverage, clear task definitions, visual quality control, and extensive analyses of model behavior and answer priors.

In response to your comments and questions, we present additional experiments, analyses, and clarifications below, addressing each question and weakness in turn. We hope this evidence addresses your concerns and welcome further questions.

[W1, Q1] Sparse observations versus full temporal video understanding

GATE evaluates temporal and workflow reasoning from ordered surgical video frames. Our additional comparison finds that supplying the same frames separately improves accuracy by only 1.35–3.70 percentage points, with all four models retaining the same ranking (Table H1). The sampling density reflects the available task-specific annotations, as explained below.

1. GATE evaluates surgical video understanding from temporally ordered frames.

GATE evaluates surgical video understanding from sampled frames, including temporal and workflow reasoning. Each clip is represented by four frames in chronological order. The temporal and workflow tasks require models to compare surgical events, track changes, and reason about procedural order. We will clarify the sampling protocol and the scope of the temporal conclusions in the main text.

The available annotations constrain how densely frames can be sampled with verified labels. For Cholec80 and CholecT50 state-change clips, Endoscapes2023 CVS clips, and the two Cataract1K correction tasks, we use consecutive annotated frames within each clip. These clips retain the source annotation cadence without further temporal subsampling. Their one-second or approximately five-second spacing therefore reflects the availability of the required labels, rather than an additional choice to skip annotated frames. Even if more video frames are available, the corresponding task-specific labels are not.

The task-specific sampling rules are described in Appendix A.3.

We will report the applicable sampling rule alongside each task and state in the main text that stability in the sampled frames does not establish stability during unobserved gaps.

2. Frame montages are an established input representation for video understanding.

Combining each clip's four frames into one image accommodates image-count limits in some models and keeps the sampled frames and layout consistent across models. IG-VLM evaluates image grids of sampled frames on zero-shot video question-answering benchmarks. Video Panels combines frames into panels and evaluates this representation on long-video understanding benchmarks. These studies support using composite-frame inputs for video understanding. We test the effect of this input format in GATE below.

3. Separate-frame input yields only modest absolute accuracy gains, with model rankings unchanged.

On a stratified random 10% sample of the updated benchmark (1,108 questions, 66-86 per task), we compared tiled montages with the same frames supplied as separate images in temporal order.

Table H1. Accuracy with tiled versus separate-frame inputs (%).

ModelTiledSeparate framesChange (pp)
GPT-6 Astra48.1949.55+1.35
GPT-5.6 Sol34.7537.73+2.98
GPT-6 Luna25.2728.97+3.70
GPT-5.6 Luna23.3825.27+1.90

The absolute accuracy gains are limited to 1.35-3.70 percentage points across the four models, and their ranking remains unchanged. Thus, presenting the same frames separately provides only a modest improvement in overall accuracy on this sample, without changing which models perform better.

4. Correctly assigning frames to clips matters.

In our within-question shuffling control on the same 1,108-question sample and four GPT models, reassigning frames across clips reduces multi-clip accuracy from 28.9% to 12.4%, below the 16.9% no-image result. This supports the importance of matching visual evidence to the question's clip structure, even under sparse sampling.

AuthorsOpenReview ↗

Responses to Reviewer HBjY (2/5)

[W2] Response-format learning confounds training gains

The Base-to-SFT accuracy gain remains 30.75 percentage points with fallback parsing, and accuracy also improves on questions already scorable before training (Tables H2–H4). Thus, the improvement extends beyond learning the required output format.

All reported answer accuracies, both in benchmark evaluation and across training experiments, use the same fallback-enabled answer parser. It extracts explicit final answers from tagged outputs, JSON, and supported plain-text responses, allowing correct answers to receive credit even when the requested response format is not followed. The strict-only scores below are an additional comparison. The reported results already include fallback extraction.

We compare Qwen3-VL-2B-Instruct before and after per-task SFT on the updated case-disjoint split, using 2,218 test questions across 15 tasks. Strict scoring requires the requested response format and a legal answer structure before checking correctness. Fallback scoring applies the existing parser to the exact same saved outputs. Both accuracies use all test questions as the denominator.

Table H2. Overall accuracy (%)

StageStrictWith fallback
Before SFT0.002.21
After SFT32.9632.96
Gain (pp)+32.96+30.75

Table H3. Per-task accuracy (%)

TaskBefore: strictBefore: fallbackAfter: strictAfter: fallback
Phase Recognition0.0010.6726.6726.67
Phase Consistency0.001.4935.0735.07
Action Description Matching0.000.009.229.22
Shared CVS Correction0.000.0012.7512.75
Spatial Repair0.000.0096.6796.67
Reference Alignment0.0010.0019.3319.33
Temporal Order Repair0.000.0012.6712.67
CVS Milestone Tracking0.000.0053.0253.02
State Change Tracking0.002.008.008.00
Spatial Relation Tracking0.000.6719.3319.33
Workflow Ordering0.008.0033.3333.33
Workflow Path Recovery0.000.0047.3047.30
Workflow Repair0.000.0022.3022.30
Description Correction0.000.0057.3357.33
Description and Workflow Correction0.000.0040.2740.27

Table H4. Same questions scorable both before and after SFT

QuestionsCorrect before SFTCorrect after SFTAccuracy beforeAccuracy afterGain (pp)
4824913710.17%28.42%+18.26

The improvement on this subset is not caused by previously unscorable responses becoming scorable: both outputs can already be scored.

The large measured Base-to-SFT improvement remains when the existing fallback parser is applied: 2.21% to 32.96% (+30.75 percentage points).

On the same 482 questions scorable before and after training, answer accuracy improves from 10.17% to 28.42%.

[W4] Related-work placement

Thank you for the suggestion. We will move the comparison with existing surgical VLM benchmarks earlier in the paper to make GATE’s novelty clearer.

[W5] Benchmark naming

Thank you for the suggestion. We will consider a more domain-specific name, such as Surgical-GATE or SurgGATE, in the revision.

[Q4] Human reviewer background and agreement

Three reviewers independently assessed a shared subset and achieved 98.06% unanimous agreement. Human review checked visual answerability, while automatic validation checked consistency with source annotations and task rules.

Reviewers. Three team members with computer science backgrounds and familiarity with the source datasets conducted the review using a dedicated annotation platform.

Review criteria. Human review checked whether the images were clear and whether the displayed visual evidence was sufficient to answer the question. Automatic checks verified consistency among the questions, reference answers, source annotations, and task rules (Appendix A.3).

Reviewer agreement. The three reviewers independently rated the same subset of questions, achieving 98.06% unanimous inter-annotator agreement (IAA).

We will add the reviewer backgrounds and agreement results to Appendix A.3 and clarify the respective roles of human review and automatic validation.

AuthorsOpenReview ↗

Responses to Reviewer HBjY (3/5)

[W3, Q5] Cross-task image overlap confounds transfer conclusions (1/2)

We have removed the documented cross-task overlap by using one global case-disjoint split and rerunning the cross-task, category-level, and joint-training experiments. The original per-task splits allowed surgical cases to recur across partitions and images to recur when tasks were pooled. Tables H5–H6 report results under the revised protocol.

1. One case split shared by all tasks.

Each source surgical case belongs to one partition across all tasks. Every video clip in a question must belong to that partition.

Cross-task checks found zero shared case groups, question IDs, image paths, or identical image contents between train, validation, and test.

After QC, the revised dataset contains 7,756 training, 1,108 validation, and 2,218 test questions. Per-task, answer-only, cross-task, category, and joint experiments all use this global split.

2. Updated evaluation of the original 11 models (Table H5, accuracy, %).

TaskGPT-6 AstraGPT-6 LunaGPT-5.6 SolGPT-5.6 LunaClaude Opus 5.5Claude Sonnet 5Kimi K3GLM-5.3-FlashInternVL3.5Nemotron-3 OmniGemma 4 31B IT
Phase Recognition65.8236.1853.8140.5961.9535.1140.0549.8033.1130.3146.09
Phase Consistency59.9738.7450.7943.0451.0834.7246.7245.0538.5930.8546.31
Action Description Matching26.063.2410.995.4922.112.547.4010.565.210.565.14
Shared CVS Correction24.7014.8420.6512.6918.3512.5510.2016.7312.282.028.45
Spatial Repair91.6067.6091.8776.2780.2730.4086.7360.2725.078.5363.75
Reference Alignment53.4728.1341.8729.8739.7321.6032.3125.6022.9317.3332.12
Temporal Order Repair67.6526.3143.0926.7149.5315.0329.5017.8511.285.5016.48
CVS Milestone Tracking61.6827.5533.9919.5844.3432.0318.2228.536.434.4821.66
State Change Tracking10.136.136.675.3311.734.674.216.804.930.806.39
Spatial Relation Tracking63.4749.8747.4743.2066.5323.4740.0038.538.533.8740.67
Workflow Ordering49.0026.4435.6526.1739.2520.8328.1428.9717.8917.2224.97
Workflow Path Recovery38.224.9814.404.0431.493.368.669.292.832.158.37
Workflow Repair49.0513.9227.8413.1141.4912.7019.5720.278.653.9217.91
Description Correction29.6023.4723.2020.4036.2722.8022.2720.0012.130.4024.67
Description and Workflow Correction34.8612.1119.9222.0745.637.0027.8317.5018.5710.0914.97

3. Summary of rerun training results: Qwen3-VL-2B-Instruct (Table H6, test accuracy, %).

Training / evaluationBaseSFTRLSFT + RL
Per-task / same task2.2132.9610.6034.76
Answer-only / same task5.9529.767.4830.93
Cross-task / other tasks2.215.563.205.65
Category / within category2.2135.3513.3936.43
Category / outside category2.216.893.447.06
Joint / all tasks2.2141.7910.7339.27

The revised protocol removes the documented case/image overlap across partitions and evaluates training gains on held-out surgical cases. We will update Appendix A.10 and the affected result tables accordingly.

Reviewer_rXgoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GATE, an evaluation framework to assess whether models can Ground, Audit, and Track surgical Events. A dataset of 13,715 human-reviewed questions over 36,264 video clips is constructed from eight datasets. 11 models are evaluated on this dataset, with a special focus on training interventions.

Critical strengths
  1. The paper addresses a unique set of correction-oriented tasks, which is not studied in literature.
  2. A focused study on training-intervention offers practical evidence on how training protocols affect surgical video understanding.
Critical weaknesses
  1. Although the dataset is reasonably sized for surgical VQA, it remains relatively small compared to contemporary large-scale video-language benchmarks. In addition, all VQA data are pooled from eight existing benchmarks with no new surgical videos or original annotations collected. This further reduces the novelty and significance of this work. In addition, the authors did not provide sufficient details in quality control, which is critical to a high-quality benchmark, e.g., inter-annotator agreement scores, exclusion criteria, or any per-task verification details. Furthermore, the proposed 15-task taxonomy lacks theoretical or empirical justification. In summary, the constructed dataset carries some values but not significant enough for ICLR.
  2. This paper offers very limted new findings beyond recent surgical video understanding benchmarks. The authors need to make much more efforts to uniquely position their work in regards to the growing literature in surgical video understanding. So far, the most distinctive element, correction-oriented tasks, reproduces the observation that models tend to over-correct, but reported only as aggregate accuracy without deeper characterization. Other key findings mostly echo findings already established in prior work.
  3. The experimental validation is not fully convincing. Only 11 models are evaluated, without medical/surgical-specific methods. Only aggregated accuracy is reported, without further metrics to reveal findings beyond this aggregated accuracy. Furthermore, the reported over-correction phenomenon remains as accuracy values, lacking qualitative failure cases, error decomposition, or analyses tied to specific tasks.
Questions for authors
  1. Most of the information is assembled from 8 existing datasets. What is the new value added by this paper and what is the most challenging problem in this process? How does the proposed framework solve or address this problem? In addition, could the authors share more insights on the QC challenges for this work?
  2. Beyond the aggregated accruacy, could the authors provide more metrics to illustrate the unique values or findings that this paper could offer beyond the known findings in literature?
7 comments
AuthorsOpenReview ↗

Responses to Reviewer rXgo (1/6)

We thank Reviewer rXgo for the constructive feedback and for recognizing GATE's distinctive correction-oriented tasks and the practical evidence provided by its training-intervention study.

GATE contributes tests of surgical correction and evidence about why models fail them. Our new analyses identify incorrect repairs after correct recognition, unnecessary edits consistent with mistaken descriptions, and a bias against recognizing achieved safety criteria. We support these findings with controlled analyses, task-specific error rates, and concrete failure cases below.

We organize the responses around Q1 and Q2, covering the related weaknesses as follows.

ResponseRelated weaknessFocus
Q1W1Dataset scale, reused sources, added value, and construction/QC challenges and solutions.
Q1W2Quality-control process, exclusion criteria, and reviewer agreement.
Q2 (1/3–2/3)W3New findings and deeper characterization of over-correction.
Q2 (2/3–3/3)W4Medical and surgical models, task-specific error metrics, and qualitative failure cases.

[Q1] New value, construction challenges, and QC challenges (1/3)

Related concerns: W1 — dataset scale, reused sources, and added value, W2 — quality-control details.

Our new contribution is a benchmark that tests whether models can correct supplied surgical descriptions and workflows while preserving correct information. The paper’s Table 3 compares GATE with 19 surgical benchmarks. None includes these correction tasks or multiple surgical video clips as separate inputs to one question. GATE also supplies verifiable reference corrections and experiments that distinguish recognition errors from failures to apply workflow rules.

HELM (Liang et al., TMLR 2023) emphasizes broad capability coverage, standardized comparisons, and transparent analysis as central principles of model evaluation. These principles motivate assessing a benchmark by what it measures and reveals, alongside its size. GATE applies them to surgical reasoning by defining what to assess, constructing verifiable tests, and revealing where current models fail.

Alaa et al. (ICML 2025) argue that medical benchmarks should use experiments to show that they measure the intended capabilities. GATE follows this principle: removing or shuffling images tests models' reliance on visual evidence, while examining repairs after correct recognition identifies failures in applying workflow rules.

1. GATE provides a systematic assessment of surgical capabilities.

The revised benchmark contains 11,082 questions across 15 tasks, six capability categories, and six anatomical sites. Its scope and evaluation design serve complementary purposes:

Evaluation principleHow GATE implements it
Capability coverageTasks assess recognition, spatial relations, temporal changes, workflow reasoning, and correction of descriptions and workflows.
Standardized comparisonsGATE ranks among leading surgical VLM benchmarks in the number of models evaluated. Proprietary and open-weight VLMs, including the most recent models such as GPT-6 Astra and Claude Opus 5.5, are assessed with common task definitions, visual inputs, and scoring rules.
Analysis beyond aggregate scoresVisual-input controls, error analysis, and SFT/RL interventions examine what models rely on, where they fail, and which capabilities improve.
ReproducibilityReference answers are traceable to source annotations. The revised training experiments use a global case-disjoint split. Questions, splits, prompts, and construction/evaluation code will be released.

Evaluation scale. GATE contains 11,082 questions across 15 tasks, placing it within the scale of existing surgical and video-understanding benchmarks. Examples include SurgeryVideoQA (2,692 questions), REAL-Colon-VQA (5,200), EndoBench (6,832), PSI-AVA-VQA (10,291), EndoVis18-VQA (11,783 classification QA pairs), and SurgCoTBench (14,256). General video-understanding benchmarks also use focused evaluation sets, including MVBench (4,000 questions) and Video-MME (2,700). These precedents, together with GATE’s capability coverage, human review, and diagnostic analyses, support its scale for systematic evaluation of surgical capabilities.

AuthorsOpenReview ↗

Responses to Reviewer rXgo (2/6)

[Q1] New value, construction challenges, and QC challenges (2/3)

2. The new contribution is the surgical assessment built on existing videos.

Published benchmarks establish that existing videos and annotations can support new assessments. AGQA (CVPR 2021) constructs compositional reasoning questions from Charades action annotations and Action Genome scene graphs. STAR (NeurIPS 2021) builds situated reasoning questions on Charades videos using structured representations and question-generation programs.

GATE constructs new questions and reference answers from surgical annotations. Our domain-specific contribution is deciding which observations support a surgical judgment and how to test it reliably. For example, CVS Milestone Tracking converts individual safety annotations into questions about when the criteria are first satisfied and remain satisfied across consecutive observations.

These tasks require explicit construction rules, checks that the annotations support the answer, and human verification that the displayed images are sufficient to answer the question.

3. Surgical data scale depends on access to usable annotations and ethically shareable recordings.

GATE brings together eight established surgical datasets, covering nearly all available sources with the annotations and usage permissions required for these assessments. Obtaining additional surgical footage involves patient privacy, consent, de-identification, and permission for the intended use. For example, Endoscapes2023 documents ethics approval, patient consent, de-identification, and CVS assessments by three trained surgeons. Some surgical datasets also prohibit uploading images or videos to external VLM endpoints for evaluation. More footage alone does not supply the verified phase, spatial, or safety labels needed to construct these tests. Using established sources lets other researchers obtain the same inputs and reconstruct the assessment under the source datasets' access conditions.

4. GATE extends assessment to model behavior and visual prediction.

Our broader analyses establish additional findings, including but not limited to the following: Recognition does not ensure correct repair: among 591 selected responses correctly describing all four surgical video clips on questions with workflow errors, 131 (22.2%) still choose the wrong repair (Table R1). World-model probes expose action failures: 133/150 primary video errors concern wrong actions, unrequested events, or unsupported scene changes (Appendix F). More tokens do not ensure higher accuracy: output length and validation accuracy do not consistently rise together during training (Appendix C.7).

GATE contributes a reproducible way to measure these gaps and evaluate interventions across surgical capabilities. We hope these distinctions clarify its novelty and support reconsideration of its significance.

AuthorsOpenReview ↗

Responses to Reviewer rXgo (3/6)

[Q1] New value, construction challenges, and QC challenges (3/3)

5. Construction and quality-control challenges and solutions.

Integrating different sources. The source datasets cover different anatomical sites and procedures and provide different types of annotations. We use task-specific construction rules to turn these annotations into consistently structured questions with verifiable reference answers.

Balancing uneven data availability. The datasets differ in size, and the available annotations support different numbers of questions for each task. We consider both task coverage and source representation when selecting questions, and check their distributions after filtering.

Checking whether the visual evidence is sufficient. A question can agree with the source annotations while still being difficult to answer from the displayed frames. The central QC challenge is determining whether those frames clearly show the information needed to answer that particular question. Reviewers therefore inspect candidate questions individually and exclude those with unclear images or insufficient visual evidence.

Quality-control details (W2).

Three reviewers independently assessed a shared subset and achieved 98.06% unanimous agreement. We used human review to verify visual answerability and automatic checks to validate the constructed questions and reference answers.

Reviewers. Three team members with computer science backgrounds and familiarity with the source datasets conducted the review using a dedicated annotation platform.

Review criteria. Human review checked whether the images were clear and whether the displayed visual evidence was sufficient to answer the question. Automatic checks verified consistency among the questions, reference answers, source annotations, and task rules (Appendix A.3).

Reviewer agreement. The three reviewers independently rated the same subset of questions, achieving 98.06% unanimous inter-annotator agreement (IAA).

Exclusion criteria. Candidates were excluded for unclear images or insufficient visual evidence to answer the question. Automatic checks also filtered out missing images, duplicate questions, and inconsistencies between annotations, questions, and reference answers.

Task-specific verification. Checks followed each task's construction rules. For example, Phase Recognition answers were matched to source phase labels. Spatial Repair required the reference relations to hold across all four frames and rejected ambiguous geometry. CVS Milestone Tracking derived milestone positions from the ordered CVS annotations. Task-specific rules are detailed in Appendices A.4–A.9.

We will add the reviewer backgrounds and agreement results to Appendix A.3, make the exclusion criteria explicit, and clarify the respective roles of human review and automatic validation, with references to the task-specific checks.

Reviewer_KpZLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GATE for assessing vision-language models (VLMs) in surgical video understanding.

Critical strengths
  • Introduction of six capability categories encompassing 15 tasks.

  • Integrates eight existing datasets covering six anatomical sites.

Critical weaknesses
  • The evidence that the benchmark genuinely evaluates surgical reasoning rather than recognition and rule-based manipulation is limited. In this paper, many tasks are constructed by combining existing categorical annotations, introducing artificial errors, or rearranging video clips.

  • In Appendix A.10, page 20, although training, validation, and test questions are disjoint within each task, the authors explicitly acknowledge that surgical cases may overlap across partitions.

  • Validation of human-reviewed benchmark quality is insufficient.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

## Responses to Reviewer KpZL (1/4)

We thank Reviewer KpZL for the constructive feedback and for recognizing GATE's six capability categories encompassing 15 tasks and its integration of eight datasets across six anatomical sites.

In response to your comments and questions, we present additional experiments, analyses, and clarifications below, addressing each question and weakness in turn. We hope this evidence addresses your concerns and welcome further questions.

Below, we clarify what GATE assesses beyond recognition, report evaluation and training results under the revised data split, and provide details of human review and reviewer agreement.

[W1] Surgical reasoning versus recognition and rule-based manipulation

GATE tests whether models can apply surgical knowledge after recognizing what the video clips show. Our new analysis finds that 131 of 591 selected responses (22.2%) choose an incorrect workflow repair despite correctly describing every clip and receiving the expected workflow in the question (Table K1). This directly identifies failures beyond recognition.

1. The tasks ask models to do more than name what they see.

Phase Recognition provides the recognition baseline. It tests whether models identify the surgical phase shown in a clip.

Compare events over time. Temporal Order Repair uses clips from the same surgical phase. Knowing their phase therefore does not tell the model which came first. CVS Milestone Tracking asks when all three CVS criteria are first met and when they are met in consecutive clips (Appendix A.6).

Decide what needs correcting. Workflow Repair asks for the smallest change that fixes a workflow. Description and Workflow Correction also asks models to fix incorrect descriptions while leaving correct information unchanged (Appendices A.7 and A.9).

We introduce errors whose correct repairs are known, so we can check whether models find and fix them. Questions without errors test whether models make unnecessary changes. Retained questions pass automatic checks and human review (Appendix A.3). AGQA and STAR also construct video reasoning questions from existing annotations.

2. Models can describe every clip correctly and still choose the wrong repair.

Across all models, we select Description and Workflow Correction responses to questions with an erroneous sequence and an explicitly provided expected workflow. We retain the subset in which the model correctly describes all four surgical video clips, including their phases, and check its repair against the reference answer.

Table K1. Workflow-repair errors after correct clip descriptions.

Responses with all four clips correctly describedOf these, incorrect workflow repairsError rate
59113122.2%

Of the 131 wrong repairs, 69 choose the wrong operation and 59 miss the workflow error. Correct descriptions and a supplied workflow rule still do not ensure a correct repair.

3. Some workflow repairs contradict the model's own descriptions.

We analyzed 8,909 sampled wrong responses across all models and tasks, each describing every clip and assessable by an LLM judge (GPT-6 Luna). All were already incorrect against the reference answers. The LLM judge checked whether the model's final answer contradicted its own descriptions of the video clips, given the task rules. The LLM judge saw the question, rules, descriptions, explanation, and answer, but no images, source annotations, reference answers, or model identities. The same LLM judge rechecked flagged contradictions.

Across all tasks: In 385 of 8,909 wrong responses (4.3%), the LLM judge found that the final answer contradicted the model's own descriptions of the video clips, given the task rules.

Within Description and Workflow Correction: wrong repairs that contradict the explanation. This task accounts for 274 of the 8,909 responses. In 99 of these 274 (36.1%), the proposed repair did not follow from the model's own descriptions under the task rules. Even if the model's descriptions were true, these repairs would still violate the task rules. Such contradictions occurred in most models.

We will explain these task differences and add the analyses in the revision. The main finding is that correctly describing surgical video clips does not guarantee correctly applying workflow rules.

AuthorsOpenReview ↗

Responses to Reviewer KpZL (2/4)

[W2] Surgical-case overlap across partitions (1/2)

We have replaced the overlapping splits with one global case-disjoint split and rerun the training experiments. The original per-task splits allowed surgical cases to recur across partitions and images to recur when tasks were pooled. The revised split separates cases and images across all tasks. Tables K2–K3 report the updated evaluation and training results.

1. One case split shared by all tasks.

Each source surgical case belongs to one partition across all tasks. Every video clip in a question must belong to that partition.

Cross-task checks found zero shared case groups, question IDs, image paths, or identical image contents between train, validation, and test.

After QC, the revised dataset contains 7,756 training, 1,108 validation, and 2,218 test questions. Per-task, answer-only, cross-task, category, and joint experiments all use this global split.

2. Updated evaluation of the original 11 models (Table K2, accuracy, %).

TaskGPT-6 AstraGPT-6 LunaGPT-5.6 SolGPT-5.6 LunaClaude Opus 5.5Claude Sonnet 5Kimi K3GLM-5.3-FlashInternVL3.5Nemotron-3 OmniGemma 4 31B IT
Phase Recognition65.8236.1853.8140.5961.9535.1140.0549.8033.1130.3146.09
Phase Consistency59.9738.7450.7943.0451.0834.7246.7245.0538.5930.8546.31
Action Description Matching26.063.2410.995.4922.112.547.4010.565.210.565.14
Shared CVS Correction24.7014.8420.6512.6918.3512.5510.2016.7312.282.028.45
Spatial Repair91.6067.6091.8776.2780.2730.4086.7360.2725.078.5363.75
Reference Alignment53.4728.1341.8729.8739.7321.6032.3125.6022.9317.3332.12
Temporal Order Repair67.6526.3143.0926.7149.5315.0329.5017.8511.285.5016.48
CVS Milestone Tracking61.6827.5533.9919.5844.3432.0318.2228.536.434.4821.66
State Change Tracking10.136.136.675.3311.734.674.216.804.930.806.39
Spatial Relation Tracking63.4749.8747.4743.2066.5323.4740.0038.538.533.8740.67
Workflow Ordering49.0026.4435.6526.1739.2520.8328.1428.9717.8917.2224.97
Workflow Path Recovery38.224.9814.404.0431.493.368.669.292.832.158.37
Workflow Repair49.0513.9227.8413.1141.4912.7019.5720.278.653.9217.91
Description Correction29.6023.4723.2020.4036.2722.8022.2720.0012.130.4024.67
Description and Workflow Correction34.8612.1119.9222.0745.637.0027.8317.5018.5710.0914.97

3. Summary of rerun training results: Qwen3-VL-2B-Instruct (Table K3, test accuracy, %).

Training / evaluationBaseSFTRLSFT + RL
Per-task / same task2.2132.9610.6034.76
Answer-only / same task5.9529.767.4830.93
Cross-task / other tasks2.215.563.205.65
Category / within category2.2135.3513.3936.43
Category / outside category2.216.893.447.06
Joint / all tasks2.2141.7910.7339.27

The revised protocol removes the documented case/image overlap across partitions and evaluates training gains on held-out surgical cases. We will update Appendix A.10 and the affected result tables accordingly.

AuthorsOpenReview ↗

Responses to Reviewer KpZL (3/4)

[W2] Surgical-case overlap across partitions (2/2)

Detailed per-task training and transfer results.

Table K4. Per-task training with reasoning.

Test taskBaseSFTRLSFT + RL
Phase Recognition10.6726.6725.3328.00
Phase Consistency1.4935.0723.8842.54
Action Description Matching0.009.220.0010.64
Shared CVS Correction0.0012.759.4014.09
Spatial Repair0.0096.6712.0096.00
Reference Alignment10.0019.3315.3318.67
Temporal Order Repair0.0012.677.3310.67
CVS Milestone Tracking0.0053.020.0055.70
State Change Tracking2.008.000.676.67
Spatial Relation Tracking0.6719.3317.3324.67
Workflow Ordering8.0033.3312.6734.00
Workflow Path Recovery0.0047.300.0041.89
Workflow Repair0.0022.304.7332.43
Description Correction0.0057.332.0059.33
Description and Workflow Correction0.0040.2728.8645.64

Table K5. Answer-only per-task training.

Test taskBaseSFTRLSFT + RL
Phase Recognition25.3340.0023.3344.00
Phase Consistency20.9037.3119.4042.54
Action Description Matching0.0014.890.0014.89
Shared CVS Correction12.7512.7512.7514.09
Spatial Repair0.0092.670.0092.00
Reference Alignment16.0014.0016.6710.67
Temporal Order Repair0.0010.000.0011.33
CVS Milestone Tracking0.0034.900.0036.24
State Change Tracking1.334.001.331.33
Spatial Relation Tracking0.0037.330.0036.67
Workflow Ordering12.0020.0018.6716.00
Workflow Path Recovery0.688.780.6815.54
Workflow Repair1.3514.1920.2720.27
Description Correction0.0063.330.0063.33
Description and Workflow Correction0.0041.610.0044.97

Table K6. Cross-task transfer.

Training taskBaseSFTRLSFT + RL
Phase Recognition1.604.212.134.40
Phase Consistency2.266.003.506.19
Action Description Matching2.368.382.707.32
Shared CVS Correction2.375.414.495.36
Spatial Repair2.376.484.216.77
Reference Alignment1.643.971.453.53
Temporal Order Repair2.376.724.166.62
CVS Milestone Tracking2.374.782.274.83
State Change Tracking2.223.870.634.16
Spatial Relation Tracking2.323.536.193.68
Workflow Ordering1.793.872.514.01
Workflow Path Recovery2.375.893.096.62
Workflow Repair2.375.025.316.38
Description Correction2.378.173.488.99
Description and Workflow Correction2.377.011.885.80

Detailed category and joint training results.

Table K7. Category training.

Training categoryTest tasksBaseSFTRLSFT + RL
Phase UnderstandingWithin6.3444.0121.1345.42
Phase UnderstandingOutside1.603.101.033.36
Scene UnderstandingWithin0.0041.144.0941.59
Scene UnderstandingOutside2.769.903.1510.91
Temporal ReasoningWithin2.5418.5618.5620.29
Temporal ReasoningOutside2.046.198.786.47
Workflow ReasoningWithin2.6943.277.6241.93
Workflow ReasoningOutside2.096.553.277.22
Description CorrectionWithin0.0057.332.0059.33
Description CorrectionOutside2.378.323.688.90
Description and Workflow CorrectionWithin0.0040.2728.8645.64
Description and Workflow CorrectionOutside2.377.202.085.65

Table K8. Joint training.

Test taskBaseSFTRLSFT + RL
Phase Recognition10.6762.6727.3373.33
Phase Consistency1.4955.2222.3950.00
Action Description Matching0.0015.600.0017.73
Shared CVS Correction0.0012.7512.0812.08
Spatial Repair0.0096.0030.6787.33
Reference Alignment10.0025.3316.6726.67
Temporal Order Repair0.0015.338.6720.00
CVS Milestone Tracking0.0034.902.0130.87
State Change Tracking2.008.672.008.00
Spatial Relation Tracking0.6731.3310.0034.67
Workflow Ordering8.0052.0014.0050.00
Workflow Path Recovery0.0050.004.7339.86
Workflow Repair0.0038.5110.8133.11
Description Correction0.0070.000.0063.33
Description and Workflow Correction0.0058.390.0041.61

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their time and constructive feedback. GATE’s contribution is a benchmark that tests whether models can use surgical observations to compare events, track changes, and correct descriptions and workflows while preserving valid information. Our experiments identify where models fail on these tasks and how training changes their performance.

We appreciate the reviewers’ recognition of GATE’s 15 tasks across six capability categories, integrating eight datasets across six anatomical sites (KpZL, HBjY). The reviewers also recognized its distinctive correction-oriented tasks and practical training-intervention study (rXgo). They further highlighted its clear task definitions, explicit visual quality control, and extensive diagnostic analyses, including constant predictions and answer priors (HBjY).

During the rebuttal period, we rebuilt globally case-disjoint splits and reran evaluation and training experiments. We added visual-input controls and medical and surgical model evaluations, examined whether SFT gains extend beyond response-format improvements, and expanded analyses of recognition, reasoning consistency, and correction errors. We also measured agreement among independent human reviewers.

We devoted considerable time and care to these additional experiments and analyses. We would be grateful if you could review the evidence presented in our responses, and we welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.