acceptodds
Withdrawn from ICLR 2027

SurgVSI: A Benchmark for 3D Spatial Reasoning in Surgical Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robust 3D spatial reasoning is essential for understanding complex surgical scenes, where accurate perception of geometric relationships, depth, and motion is critical for downstream decision-making. However, existing benchmarks largely focus on general-domain settings and do not capture the fine-grained geometric interactions and dynamic constraints present in real surgical environments. In this work, we introduce SurgVSI, a benchmark dataset for evaluating 3D spatial reasoning in surgical environments. SurgVSI is constructed from StereoMIS, EndoVis17, and EndoVis18, and comprises a diverse set of tasks that assess both static and dynamic spatial understanding, including depth estimation, distance estimation, camera ego-motion analysis, and instrument path tracking and completion. The dataset contains 5,998 evaluation instances and 147,386 training instances, with metric annotations derived from stereo reconstruction and camera calibration. We conduct extensive evaluations across representative models to characterize current capabilities on SurgVSI, revealing substantial challenges in accurate geometric reasoning and dynamic spatial understanding. Our benchmark establishes a standardized testbed for studying 3D spatial reasoning in surgical contexts and provides a foundation for future research in perception, reasoning, and robotic systems operating in complex real-world environments.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Dataset, 3D Spatial Reasoning, Surgical Environments, Scene Understanding, Robotic Surgery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J1MdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SURGVSI, a new benchmark dataset for spatial reasoning in surgical environments. The new dataset is constructed from other existing datasets. The authors then provide several tasks and benchmark on the new dataset.

Critical strengths

The paper is mostly easy to understand. The writing is clear and the benmarking in several tasks can be useful for other work.

Critical weaknesses
  • My main concern is that the real-world medical application of "3D spatial reasoning" in the paper is not well justified. The paper proposes a new dataset for these "spatial reasoning" tasks. Yet, I don't see how these tasks can be useful in real-world medical domains (e.g., hospitals). For example, a lot of LLM models are used for experiments (Table 2, Table 3) - it's just not easy to apply any of those results to real-world problems.
  • The new dataset is constructed from StereoMIS, EndoVis17, and EndoVis18. No new data were collected, so it somehow limits the contribution of a "really new" dataset.
  • There is no evaluation of the data construction pipeline (Section 3.2). How can we guarantee that the constructed ground truth is of good quality?
Questions for authors

I mainly have concerns about the application of the dataset in the real world, and also the quality of the constructed ground truth.

Reviewer_StuYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SurgVSI turns StereoMIS, EndoVis17, and EndoVis18 into six question types with numeric or structured answers for vision-language models. The labels are produced by the paper's own pipeline: stereo depth, instrument tip localization, Euclidean distance, SIFT-based ego-motion with PnP, and trajectory linking. The authors position this package as a benchmark for 3D spatial reasoning in the surgical domain.

Critical strengths

The stereo pipeline is documented and provides metric targets on real porcine endoscopy, which 2D surgical visual question answering cannot provide.

Critical weaknesses
  1. The labels are generated by stereo matching, instrument tip tracking, and SIFT-based pose estimation. A VLM is therefore scored on agreement with that pipeline, which measures metric estimation rather than 3D reasoning.
  2. The underlying stereo data already exist: StereoMIS was released by Hayoz et al. for depth estimation and reconstruction, and EndoVis17 and EndoVis18 with calibration and instrument annotations. The six tasks are readouts of the paper's reconstruction and tracking pipeline on these data (Section 3.2), so the new artifact is a question-answer format over geometric quantities rather than a new task.
  3. Some tasks do not separate the models. On path completion, MeanDev for the five general-purpose VLMs lies between 19.40 and 19.52 mm and LenDev is about 103 mm for all of them (Section 4.3), even though the same models differ widely on every other task. Euler-magnitude RMSE on camera motion is similarly compressed (5.59 to 5.82 degrees). Near-identical errors suggest that the models return a similar default answer, such as a path between the given endpoints or near-zero rotation. The paper reports no trivial reference baseline for any task, so it is impossible to tell whether these scores measure anything beyond that default.
  4. The Limitations acknowledge that validation covers only internal consistency and that independent references are needed. For one task, such a reference is already available: StereoMIS is distributed with camera poses (Table 6), yet the ego-motion labels are re-estimated with SIFT and PnP and never compared with them. Without any external check, it is unclear whether the 6–11 mm translation errors of the general-purpose VLMs exceed the error of the labels themselves.
  5. The introduction presents SurgVSI as a foundation for "advancing surgical AI safety," and the abstract as a foundation for robotic systems. The release contains geometric questions and answers only, with no action, planning, or safety labels, and the paper itself states that it "does not establish deployment or clinical safety." The positioning should be narrowed accordingly.
Questions for authors
  1. Could the authors report simple reference baselines for each task, such as endpoint linear interpolation for path completion, zero motion for camera ego-motion, and the training-set mean for the depth and distance tasks?
  2. Could the authors compare the SIFT–PnP ego-motion labels with the camera poses distributed with StereoMIS, to give at least one external estimate of label error?
  3. Since all six labels are computed by the pipeline in Section 3.2, would the authors consider describing the contribution as metric 3D estimation on surgical stereo data? I think this would still be a useful framing.
Reviewer_HTr4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SurgVSI for evaluating 3D spatial reasoning capabilities of vision-language models (VLMs) in surgical environments.

Critical strengths

The paper identifies the gap between existing general-domain spatial reasoning benchmarks and the requirements of surgical scene understanding.

Critical weaknesses
  • The paper presents validation of the geometric ground-truth annotations. It derives ground-truth measurements from stereo matching, segmentation masks, camera calibration, and geometric reconstruction. However, these measurements are treated as reference answers without sufficient independent validation against physical measurements.

  • Instrument-tip localization relies on heuristic morphological operations.

  • This paper claims to evaluate 3D spatial reasoning, but presented tasks seem to emphasize numerical prediction from visual inputs rather than reasoning about geometric relationships.

  • This paper only consists of a bit more than 8 pages, with insufficient technical contents.

Questions for authors

See Critical Weaknesses.

Comments on the paper

1 comment

Dear reviewers,

As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.