- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GATE, a benchmark and evaluation framework for assessing whether vision-language models can ground, audit, and track surgical events. GATE organizes surgical understanding into six capability categories and 15 tasks covering phase understanding, scene understanding, temporal reasoning, workflow reasoning, description correction, and joint description/workflow correction. The benchmark contains 13,715 human-reviewed questions constructed from eight existing surgical datasets across six anatomical sites. The paper evaluates 11 proprietary and open-weight VLMs and finds large performance differences across capability types, with models generally performing better on recognition and local spatial judgments than on workflow reasoning and correction tasks. The authors further study supervised fine-tuning, reinforcement learning, and sequential SFT+RL on Qwen3-VL-2B, together with cross-task, category-level, joint-training, text-only, answer-only, and error analyses. The paper also includes a small action-conditioned image/video generation study.
Critical strengths
A major strength of GATE is the breadth of the evaluation. Rather than focusing on a single recognition or VQA task, the benchmark covers local perception, temporal comparison, workflow reasoning, auditing, and correction across multiple procedures and datasets. The task definitions are generally clear, and the authors perform explicit visual quality control before retaining benchmark questions. I also appreciate the unusually extensive diagnostic analysis. The paper does not stop at reporting aggregate accuracy: it examines text-only performance, answer validity, cross-task transfer, joint training, repeated predictions, and visible error patterns. In particular, the analysis showing that some high task accuracies can arise from constant predictions or answer priors is useful and makes the benchmark study more informative than a simple model leaderboard.
Critical weaknesses
My main concern is that the “video” and temporal-reasoning claims are somewhat stronger than the actual input protocol supports. Each clip is represented by only four sampled frames arranged as a static montage, rather than a continuous video sequence. Several temporal and workflow tasks therefore test reasoning over sparse observations rather than full temporal video understanding. This distinction is particularly important because frame spacing varies across datasets and the paper itself notes that observed stability does not imply stability during unobserved gaps. A second concern is that the training-intervention results are difficult to interpret cleanly. The large Base-to-SFT gain partly reflects learning the required response format: most Base responses in the paired SFT pre-test are invalid, while SFT nearly eliminates this failure mode. In addition, the cross-task, category-level, and joint-training experiments allow image overlap across training and test through different tasks. The paper appropriately acknowledges this, but the reported joint-training advantage therefore cannot be interpreted as clean evidence of transferable surgical reasoning. The Related Work section appears rather late for a benchmark paper. Moving the comparison with existing surgical VLM benchmarks earlier could make GATE's novelty easier to assess. The authors may also consider a more domain-specific benchmark name such as Surgical-GATE or SurgGATE, since “GATE” alone is fairly generic.
Questions for authors
Main-text-driven
Q1:GATE is presented as a surgical video benchmark, but each clip is represented by only four sampled frames arranged as a static montage rather than being provided to the model as a native video sequence. This setup may not fully evaluate the temporal modeling capabilities of video-capable VLMs. Could the authors evaluate a representative subset of the temporal or workflow tasks using native video input and compare the results with the current four-frame montage setting?
Q2: The benchmark is a central contribution of the paper, but I could not find a clear data-release plan. Will the authors release the 13,715 questions, reference answers, benchmark splits, prompts, construction/evaluation code, and quality-control metadata? If the original surgical images/videos cannot be redistributed, will identifiers and reconstruction scripts be provided so that researchers with access to the source datasets can reproduce GATE?
Q3: The text-only analysis indicates that several tasks can be partially solved from textual structure or answer priors, and Description and Workflow Correction even performs better without images. Could the authors provide an image-shuffling or mismatched-image control to better quantify how much each task actually depends on visual evidence?
Q4: The benchmark relies on human review to verify that each automatically constructed question is answerable from the displayed visual evidence. Could the authors clarify the background of the reviewers, whether each candidate was independently assessed by more than one reviewer, and whether any inter-reviewer agreement was measured? This seems important for establishing the reliability of the benchmark quality-control process.
Appendix-driven
Q5: The per-task splits are image-disjoint, but surgical cases can overlap, and the cross-task/category/joint experiments contain substantial train–test image overlap across tasks. Could the authors provide a globally image-disjoint or case-disjoint evaluation for the joint-training setting, or otherwise quantify how much the reported 48.23% joint-training result may benefit from prior exposure to test images through other tasks?
7 comments
Responses to Reviewer HBjY (2/5)
[W2] Response-format learning confounds training gains
The Base-to-SFT accuracy gain remains 30.75 percentage points with fallback parsing, and accuracy also improves on questions already scorable before training (Tables H2–H4). Thus, the improvement extends beyond learning the required output format.
All reported answer accuracies, both in benchmark evaluation and across training experiments, use the same fallback-enabled answer parser. It extracts explicit final answers from tagged outputs, JSON, and supported plain-text responses, allowing correct answers to receive credit even when the requested response format is not followed. The strict-only scores below are an additional comparison. The reported results already include fallback extraction.
We compare Qwen3-VL-2B-Instruct before and after per-task SFT on the updated case-disjoint split, using 2,218 test questions across 15 tasks. Strict scoring requires the requested response format and a legal answer structure before checking correctness. Fallback scoring applies the existing parser to the exact same saved outputs. Both accuracies use all test questions as the denominator.
Table H2. Overall accuracy (%)
| Stage | Strict | With fallback |
|---|---|---|
| Before SFT | 0.00 | 2.21 |
| After SFT | 32.96 | 32.96 |
| Gain (pp) | +32.96 | +30.75 |
Table H3. Per-task accuracy (%)
| Task | Before: strict | Before: fallback | After: strict | After: fallback |
|---|---|---|---|---|
| Phase Recognition | 0.00 | 10.67 | 26.67 | 26.67 |
| Phase Consistency | 0.00 | 1.49 | 35.07 | 35.07 |
| Action Description Matching | 0.00 | 0.00 | 9.22 | 9.22 |
| Shared CVS Correction | 0.00 | 0.00 | 12.75 | 12.75 |
| Spatial Repair | 0.00 | 0.00 | 96.67 | 96.67 |
| Reference Alignment | 0.00 | 10.00 | 19.33 | 19.33 |
| Temporal Order Repair | 0.00 | 0.00 | 12.67 | 12.67 |
| CVS Milestone Tracking | 0.00 | 0.00 | 53.02 | 53.02 |
| State Change Tracking | 0.00 | 2.00 | 8.00 | 8.00 |
| Spatial Relation Tracking | 0.00 | 0.67 | 19.33 | 19.33 |
| Workflow Ordering | 0.00 | 8.00 | 33.33 | 33.33 |
| Workflow Path Recovery | 0.00 | 0.00 | 47.30 | 47.30 |
| Workflow Repair | 0.00 | 0.00 | 22.30 | 22.30 |
| Description Correction | 0.00 | 0.00 | 57.33 | 57.33 |
| Description and Workflow Correction | 0.00 | 0.00 | 40.27 | 40.27 |
Table H4. Same questions scorable both before and after SFT
| Questions | Correct before SFT | Correct after SFT | Accuracy before | Accuracy after | Gain (pp) |
|---|---|---|---|---|---|
| 482 | 49 | 137 | 10.17% | 28.42% | +18.26 |
The improvement on this subset is not caused by previously unscorable responses becoming scorable: both outputs can already be scored.
The large measured Base-to-SFT improvement remains when the existing fallback parser is applied: 2.21% to 32.96% (+30.75 percentage points).
On the same 482 questions scorable before and after training, answer accuracy improves from 10.17% to 28.42%.
[W4] Related-work placement
Thank you for the suggestion. We will move the comparison with existing surgical VLM benchmarks earlier in the paper to make GATE’s novelty clearer.
[W5] Benchmark naming
Thank you for the suggestion. We will consider a more domain-specific name, such as Surgical-GATE or SurgGATE, in the revision.
[Q4] Human reviewer background and agreement
Three reviewers independently assessed a shared subset and achieved 98.06% unanimous agreement. Human review checked visual answerability, while automatic validation checked consistency with source annotations and task rules.
Reviewers. Three team members with computer science backgrounds and familiarity with the source datasets conducted the review using a dedicated annotation platform.
Review criteria. Human review checked whether the images were clear and whether the displayed visual evidence was sufficient to answer the question. Automatic checks verified consistency among the questions, reference answers, source annotations, and task rules (Appendix A.3).
Reviewer agreement. The three reviewers independently rated the same subset of questions, achieving 98.06% unanimous inter-annotator agreement (IAA).
We will add the reviewer backgrounds and agreement results to Appendix A.3 and clarify the respective roles of human review and automatic validation.
Responses to Reviewer HBjY (3/5)
[W3, Q5] Cross-task image overlap confounds transfer conclusions (1/2)
We have removed the documented cross-task overlap by using one global case-disjoint split and rerunning the cross-task, category-level, and joint-training experiments. The original per-task splits allowed surgical cases to recur across partitions and images to recur when tasks were pooled. Tables H5–H6 report results under the revised protocol.
1. One case split shared by all tasks.
Each source surgical case belongs to one partition across all tasks. Every video clip in a question must belong to that partition.
Cross-task checks found zero shared case groups, question IDs, image paths, or identical image contents between train, validation, and test.
After QC, the revised dataset contains 7,756 training, 1,108 validation, and 2,218 test questions. Per-task, answer-only, cross-task, category, and joint experiments all use this global split.
2. Updated evaluation of the original 11 models (Table H5, accuracy, %).
| Task | GPT-6 Astra | GPT-6 Luna | GPT-5.6 Sol | GPT-5.6 Luna | Claude Opus 5.5 | Claude Sonnet 5 | Kimi K3 | GLM-5.3-Flash | InternVL3.5 | Nemotron-3 Omni | Gemma 4 31B IT |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Phase Recognition | 65.82 | 36.18 | 53.81 | 40.59 | 61.95 | 35.11 | 40.05 | 49.80 | 33.11 | 30.31 | 46.09 |
| Phase Consistency | 59.97 | 38.74 | 50.79 | 43.04 | 51.08 | 34.72 | 46.72 | 45.05 | 38.59 | 30.85 | 46.31 |
| Action Description Matching | 26.06 | 3.24 | 10.99 | 5.49 | 22.11 | 2.54 | 7.40 | 10.56 | 5.21 | 0.56 | 5.14 |
| Shared CVS Correction | 24.70 | 14.84 | 20.65 | 12.69 | 18.35 | 12.55 | 10.20 | 16.73 | 12.28 | 2.02 | 8.45 |
| Spatial Repair | 91.60 | 67.60 | 91.87 | 76.27 | 80.27 | 30.40 | 86.73 | 60.27 | 25.07 | 8.53 | 63.75 |
| Reference Alignment | 53.47 | 28.13 | 41.87 | 29.87 | 39.73 | 21.60 | 32.31 | 25.60 | 22.93 | 17.33 | 32.12 |
| Temporal Order Repair | 67.65 | 26.31 | 43.09 | 26.71 | 49.53 | 15.03 | 29.50 | 17.85 | 11.28 | 5.50 | 16.48 |
| CVS Milestone Tracking | 61.68 | 27.55 | 33.99 | 19.58 | 44.34 | 32.03 | 18.22 | 28.53 | 6.43 | 4.48 | 21.66 |
| State Change Tracking | 10.13 | 6.13 | 6.67 | 5.33 | 11.73 | 4.67 | 4.21 | 6.80 | 4.93 | 0.80 | 6.39 |
| Spatial Relation Tracking | 63.47 | 49.87 | 47.47 | 43.20 | 66.53 | 23.47 | 40.00 | 38.53 | 8.53 | 3.87 | 40.67 |
| Workflow Ordering | 49.00 | 26.44 | 35.65 | 26.17 | 39.25 | 20.83 | 28.14 | 28.97 | 17.89 | 17.22 | 24.97 |
| Workflow Path Recovery | 38.22 | 4.98 | 14.40 | 4.04 | 31.49 | 3.36 | 8.66 | 9.29 | 2.83 | 2.15 | 8.37 |
| Workflow Repair | 49.05 | 13.92 | 27.84 | 13.11 | 41.49 | 12.70 | 19.57 | 20.27 | 8.65 | 3.92 | 17.91 |
| Description Correction | 29.60 | 23.47 | 23.20 | 20.40 | 36.27 | 22.80 | 22.27 | 20.00 | 12.13 | 0.40 | 24.67 |
| Description and Workflow Correction | 34.86 | 12.11 | 19.92 | 22.07 | 45.63 | 7.00 | 27.83 | 17.50 | 18.57 | 10.09 | 14.97 |
3. Summary of rerun training results: Qwen3-VL-2B-Instruct (Table H6, test accuracy, %).
| Training / evaluation | Base | SFT | RL | SFT + RL |
|---|---|---|---|---|
| Per-task / same task | 2.21 | 32.96 | 10.60 | 34.76 |
| Answer-only / same task | 5.95 | 29.76 | 7.48 | 30.93 |
| Cross-task / other tasks | 2.21 | 5.56 | 3.20 | 5.65 |
| Category / within category | 2.21 | 35.35 | 13.39 | 36.43 |
| Category / outside category | 2.21 | 6.89 | 3.44 | 7.06 |
| Joint / all tasks | 2.21 | 41.79 | 10.73 | 39.27 |
The revised protocol removes the documented case/image overlap across partitions and evaluates training gains on held-out surgical cases. We will update Appendix A.10 and the affected result tables accordingly.
Responses to Reviewer HBjY (1/5)
We thank Reviewer HBjY for the constructive feedback and for recognizing GATE's broad coverage, clear task definitions, visual quality control, and extensive analyses of model behavior and answer priors.
In response to your comments and questions, we present additional experiments, analyses, and clarifications below, addressing each question and weakness in turn. We hope this evidence addresses your concerns and welcome further questions.
[W1, Q1] Sparse observations versus full temporal video understanding
GATE evaluates temporal and workflow reasoning from ordered surgical video frames. Our additional comparison finds that supplying the same frames separately improves accuracy by only 1.35–3.70 percentage points, with all four models retaining the same ranking (Table H1). The sampling density reflects the available task-specific annotations, as explained below.
1. GATE evaluates surgical video understanding from temporally ordered frames.
GATE evaluates surgical video understanding from sampled frames, including temporal and workflow reasoning. Each clip is represented by four frames in chronological order. The temporal and workflow tasks require models to compare surgical events, track changes, and reason about procedural order. We will clarify the sampling protocol and the scope of the temporal conclusions in the main text.
The available annotations constrain how densely frames can be sampled with verified labels. For Cholec80 and CholecT50 state-change clips, Endoscapes2023 CVS clips, and the two Cataract1K correction tasks, we use consecutive annotated frames within each clip. These clips retain the source annotation cadence without further temporal subsampling. Their one-second or approximately five-second spacing therefore reflects the availability of the required labels, rather than an additional choice to skip annotated frames. Even if more video frames are available, the corresponding task-specific labels are not.
The task-specific sampling rules are described in Appendix A.3.
We will report the applicable sampling rule alongside each task and state in the main text that stability in the sampled frames does not establish stability during unobserved gaps.
2. Frame montages are an established input representation for video understanding.
Combining each clip's four frames into one image accommodates image-count limits in some models and keeps the sampled frames and layout consistent across models. IG-VLM evaluates image grids of sampled frames on zero-shot video question-answering benchmarks. Video Panels combines frames into panels and evaluates this representation on long-video understanding benchmarks. These studies support using composite-frame inputs for video understanding. We test the effect of this input format in GATE below.
3. Separate-frame input yields only modest absolute accuracy gains, with model rankings unchanged.
On a stratified random 10% sample of the updated benchmark (1,108 questions, 66-86 per task), we compared tiled montages with the same frames supplied as separate images in temporal order.
Table H1. Accuracy with tiled versus separate-frame inputs (%).
The absolute accuracy gains are limited to 1.35-3.70 percentage points across the four models, and their ranking remains unchanged. Thus, presenting the same frames separately provides only a modest improvement in overall accuracy on this sample, without changing which models perform better.
4. Correctly assigning frames to clips matters.
In our within-question shuffling control on the same 1,108-question sample and four GPT models, reassigning frames across clips reduces multi-clip accuracy from 28.9% to 12.4%, below the 16.9% no-image result. This supports the importance of matching visual evidence to the question's clip structure, even under sparse sampling.