- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SAR-IDE, a surgical motion representation model trained on heterogeneous robotic surgical data. It uses inverse dynamics with embodiment-specific action heads, together with a tool-centric forward dynamics objective based on instrument masks, point tracks, and optical flow. The model is mainly pretrained on OpenH with synchronized kinematics, with additional intraoperative videos, and is evaluated on action regression, action recognition, point tracking, unseen-embodiment transfer, and downstream robot control.
Critical strengths
-
The paper studies the problem of learning shared representations across surgical robots with different action spaces. Using embodiment-specific action heads with a shared visual encoder is a reasonable design choice for handling heterogeneous kinematics.
-
The paper evaluates the learned representation on several motion-related tasks, including action regression, action recognition, point tracking, few-shot transfer to an unseen robot embodiment, and a real-robot policy experiment, providing evidence of transfer across different motion-focused settings.
Critical weaknesses
1.The "surgical video foundation model" claim is broader than the current evaluation and baseline coverage support. Most downstream tasks are closely related to motion or action: action regression, action recognition, point tracking, and robot control. These results support SAR-IDE as a surgical motion/action representation model, but do not yet demonstrate broad surgical video understanding. A foundation-model evaluation should also cover more diverse tasks, such as surgical phase recognition and semantic segmentation. In addition, the comparison with recent surgical foundation models is incomplete. SurgVISTA [1] is cited but not included as an experimental baseline, and its reference should be updated from the arXiv version to the published npj Digital Medicine paper (2026). Also, the recent surgical foundation model LemonFM [2] (CVPR 2026) should also be included in the comparison.
[1] Large-scale self-supervised video foundation model for intelligent surgery. npj Digital Medicine, 2026.
[2] LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings. CVPR 2026.
2.Downstream comparisons are confounded by target-domain exposure during pretraining. SAR-IDE uses training videos from SAR-RARP50 and GraSP during pretraining through tool-centric visual supervision, and both datasets are later used for downstream action recognition. The authors exclude the corresponding validation and test videos, however, SAR-IDE still receives additional in-domain exposure that may not be available to the compared frozen encoders. Therefore, the gains on RARP50 and GraSP may reflect both the proposed method and additional target-domain pretraining.
3. The source of the performance gains over external baselines is not fully disentangled. SAR-IDE is initialized from the same V-JEPA 2.1 checkpoint used as one of the main baselines, and is then further pretrained with domain-specific surgical data and tool-centric supervision. Therefore, the improvement over V-JEPA 2.1 may reflect both the proposed learning framework and the additional surgical pretraining recipe. A controlled comparison with matched data, supervision, and training budget would help isolate the contribution of the proposed method. It would also be informative to report a from-scratch SAR-IDE variant to show how much the final performance depends on the V-JEPA 2.1 initialization.
Questions for authors
Questions:
- Can the authors evaluate SAR-IDE on broader surgical video tasks, such as phase recognition or semantic segmentation, and compare with recent surgical foundation models such as SurgVISTA and LemonFM?
- How much do the RARP50 and GraSP results depend on using their training videos during pretraining? Could the authors report results without this target-domain exposure?
- How much of the improvement over V-JEPA 2.1 comes from the proposed method versus additional surgical pretraining? A matched-data comparison would help clarify this.
- What is the full computational cost of the SAR-IDE pretraining pipeline, including the generation of instrument masks, point tracks, and optical flow? Reporting the preprocessing cost in addition to the main training cost would help clarify the overall efficiency and practical scalability of the method.
Addressing these questions could lead me to increase my rating.
3 comments
Thank you for the clarifications and additional experiments.
However, my main concern about the surgical video foundation model claim remains. The authors propose narrowing the scope to surgical motion and action representations, but this change is not yet reflected in a revised manuscript. Evaluating a general surgical video foundation model requires broader task coverage than evaluating a surgical domain-specific representation model focused on motion and action.
The new phase recognition evaluation uses GraSP, which contains only 13 surgical videos, and SAR-IDE achieves lower accuracy than LemonFM. Broader evaluations, such as Cholec80 phase recognition, and an important surgical task such as semantic segmentation, are still missing. Without these evaluations, the current evidence remains insufficient to support the broader surgical video foundation model claim.
Thank you for raising this distinction. We agree that broader phase-recognition and semantic-segmentation evaluations would be necessary to support general surgical-video understanding, but this is definitely beyond the scope of this work. Our intended claim is narrower: SAR-IDE learns generalizable representations of surgical instrument motion and action from heterogeneous robot kinematics. To mitigate any potential misunderstanding, we have now reflected this scope in the revised manuscript by replacing “surgical video foundation model” with “surgical action representation model” in the Abstract, Introduction, Contributions, and Conclusion, with the revised terms marked in blue. Thanks for your suggestion!
Accordingly, our main evaluations focus on tasks that directly require instrument-motion or action information: action regression, atomic action recognition, instrument point tracking, few-shot transfer to an unseen embodiment, and downstream policy learning. In contrast, phase recognition describes procedure-level states and can rely strongly on anatomy, appearance, scene context, and temporal workflow, while semantic segmentation primarily evaluates single-frame spatial semantics. These tasks may benefit from motion information, but they do not directly measure whether a representation captures generalizable instrument motion across robot embodiments. We report the GraSP phase-recognition result transparently, but do not use it to claim uniformly stronger general surgical-video understanding. To avoid any misunderstanding of our core motivation of learning transferable motion representations from surgical robotics video, in the revised manuscript, we no longer make the broader surgical video foundation model claim.
Thank you for identifying the distinction between a general surgical video model and a motion-focused representation model. We address evaluation scope, target-domain exposure, the V-JEPA control, and compute below.
1. Scope across surgical video tasks
Thanks for pointing out the missed SOTA LemonFM. SurgVISTA has not released the model weights so we tested LemonFM on our task and added new step recognition task on PitVQA, and phase recognition on GraSP. We use SAR-IDE trained only on OpenH for phase recognition, so that neither model has seen GraSP videos.
LemonFM is pretrained on 938 h of surgical video, roughly twice ours. Even so, SAR-IDE reaches comparable phase accuracy with the best F1, and outperforms finer-grained step recognition. SAR-IDE also leads by large margins on other tasks, we therefore position SAR-IDE as a surgical motion and action representation rather than a general surgical video foundation model, and will revise the wording accordingly in the final manuscript.
2. Target-domain exposure on RARP50 and GraSP
We evaluated a SAR-IDE checkpoint trained only on OpenH video–kinematics data:
Performance does not decrease significantly when target-domain training videos are removed.
3. Framework contribution vs additional surgical pretraining
We evaluated a V-JEPA 2.1 checkpoint continued on the surgical data with its original objective. Kindly refer to the Answer to Question 1 raised by Reviewer gPys The conclusion is that exposure to more surgical video under the original V-JEPA objective is insufficient to produce the SAR-IDE improvement.
4. Training and preprocessing cost
The main SAR-IDE run uses 32 GPUs for 12,000 optimizer steps and approximately 33 hours of summed wall time. Preprocessing requires approximately 1.35k GPU-hours under our recorded configuration. Since videos and clips are processed independently, its wall-clock time can be substantially reduced through multi-GPU parallelism, while larger per-GPU batch sizes can further shorten required time. Mask, point-track, and optical-flow generation run offline and shard independently by video, with one worker per GPU and modest memory (~3.6 GiB for SAM 3.1, 4.9–7.0 GiB for SEA-RAFT; CoTracker3 fits on a single GPU), so they run on commodity GPUs. Annotations are cached and reused, so this one-time cost adds no overhead to training or inference.