- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents SURGVSI, a new benchmark dataset for spatial reasoning in surgical environments. The new dataset is constructed from other existing datasets. The authors then provide several tasks and benchmark on the new dataset.
Critical strengths
The paper is mostly easy to understand. The writing is clear and the benmarking in several tasks can be useful for other work.
Critical weaknesses
- My main concern is that the real-world medical application of "3D spatial reasoning" in the paper is not well justified. The paper proposes a new dataset for these "spatial reasoning" tasks. Yet, I don't see how these tasks can be useful in real-world medical domains (e.g., hospitals). For example, a lot of LLM models are used for experiments (Table 2, Table 3) - it's just not easy to apply any of those results to real-world problems.
- The new dataset is constructed from StereoMIS, EndoVis17, and EndoVis18. No new data were collected, so it somehow limits the contribution of a "really new" dataset.
- There is no evaluation of the data construction pipeline (Section 3.2). How can we guarantee that the constructed ground truth is of good quality?
Questions for authors
I mainly have concerns about the application of the dataset in the real world, and also the quality of the constructed ground truth.
Dear reviewers,
As the rebuttal period is coming to an end, I kindly remind you to check the authors' responses. If you have not yet responded, please indicate whether your concerns have been adequately addressed. Many thanks!
Best, AC