- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a joint visual-trajectory prediction framework for surgical world modeling, with applications in surgical motion planning. The proposed framework models tool trajectories using spatial features and temporally aligns them with visual representations. The aligned features are then injected into an existing world model to guide video prediction. Experiments are conducted on an existing surgical benchmark, with the generated videos evaluated for visual quality and trajectory alignment. The paper also introduces two additional metrics to quantify the influence of trajectory conditioning on video generation.
Critical strengths
-
The idea of separating trajectory control from visual modeling is interesting. Controllable generation is particularly important in surgery, given the variability in anatomy, surgical instruments, and scene dynamics. The proposed approach moves in a useful direction by combining explicit tool-tip trajectories with visual content for future video prediction.
-
The idea of identifying limitations in existing surgical world models and building upon them is appreciated. Given the limited availability of surgical data, improving the controllability of existing pretrained models is a useful direction. This could potentially improve the applicability of such models without requiring an entirely new surgical video generation framework.
Critical weaknesses
-
Limited evaluation of practical utility for surgical applications: The main limitation of this work is the lack of evaluation showing the practical usefulness of the proposed trajectory conditioning. Improving controllability in world models is an established research direction [1-3], and its application to surgery requires further justification. The paper motivates surgical action anticipation but primarily evaluates visual quality and future tool trajectories. A correctly predicted trajectory does not necessarily imply that the generated video captures meaningful surgical interactions or is useful for downstream tasks. For example, evaluating whether the generated videos improve surgical action anticipation, predict potentially unsafe tool-tissue interactions, or support robotic policy learning would provide stronger evidence of the contribution. Without such experiments, the practical benefit of the proposed method for surgical applications remains unclear.
-
Relationship between trajectory alignment and generation quality. The qualitative results in Figure 3 raise questions about the proposed evaluation metrics. For baselines such as V-JEPA2 and TSWAM, the deviation of the predicted tool trajectories from the ground truth appears relatively small. However, VideoVLA appears to show greater deviations in some examples while achieving better scores on the proposed TrajAlign and TrajGain metrics. The paper should explain this discrepancy and clarify what these metrics capture. More importantly, it remains unclear whether improvements in trajectory alignment translate to better modeling of surgical actions or improved downstream performance.
-
Information of eval dataset. The paper uses data from SurgWMBench, but the corresponding surgical video annotations do not appear to be publicly available. It is therefore unclear how the authors obtained the required trajectory annotations for training and evaluation. The source of these annotations, their generation procedure, and any validation performed should be clearly described. This information is important for assessing the reliability and reproducibility of the experimental results.
-
Limited training data and potential overfitting. The training dataset consists of only 2,024 clips, each containing 20 frames. Given the two-stage training procedure, this appears to be a relatively small dataset for learning a trajectory-conditioned video generation framework. The reported improvements could potentially reflect adaptation to the motion patterns and scene distributions in this dataset rather than generalizable trajectory control. The paper should provide evidence that the model has not overfitted, preferably through evaluation on unseen surgical scenes, procedures, or tool trajectories.
-
Unclear use of future-frame information during training. Line 308 describes an initial trajectory training stage followed by joint visual-trajectory training. Appendix A further mentions that the first stage uses future frames and ground-truth trajectories to train the spatial trajectory conditioner while keeping the video generation model frozen. It is unclear how the future-frame features are used and whether the same information is available during inference. Although using future frames as supervision during training is reasonable, using features derived from target frames as conditioning inputs could introduce a train-test mismatch or unintended information leakage. The paper should clarify the training and inference information flow and explain why the video reconstruction loss is needed during this stage.
-
Handling of the original action conditioning The Cosmos-H Surgical Simulator is an action-conditioned world model that takes robotkinematics as input. However, the paper does not clearly explain what happens to this action-conditioning mechanism when the proposed trajectory module is introduced. Is the original action conditioner retained, frozen, modified, or removed? This is important because action conditioning is a central component of the original world model.
-
Backbone choice and missing surgical baselines. The motivation for using Cosmos Predict 2.5 as the backbone is unclear when a surgical model such as Cosmos-H-Predict is already available. The latter has been adapted to surgical data and could provide stronger domain-specific representations for the proposed trajectory conditioning. Adding the proposed module to this surgical backbone would be a relevant comparison for determining whether the improvements arise from trajectory conditioning or from the characteristics of the selected video generation backbone.
Minor comments (Not necessary for the final rating)
-
The description of the trajectory encoder is incomplete. The architecture, parameter count, and implementation details should be provided. This is particularly important if the training code and dataset are not planned for public release, as the current description limits reproducibility.
-
Section 3.2 refers to the Cosmos-H Surgical Simulator [4], which is an action-conditioned surgical world model. However, the baseline evaluation also mentions Cosmos-H-Surgical [3], while other parts of the paper refer to Cosmos Predict 2.5. The authors should clearly identify which model is used as the backbone, which models serve as baselines, and how the proposed method differs from each of them.
References:
- Rapuri et al., SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation, MICCAI 2026
- Sivakumar et al., SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation, MICCAI 2026
- He et al., Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling, 2025
- Cosmos-H-Surgical-Simulatore, NVIDIA (https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical-Simulator)
Questions for authors
-
Can the authors provide results that improved trajectory alignment benefits a surgical application, such as action anticipation, tool-tissue interaction prediction, or robotic policy learning? What conclusions can currently be drawn about surgical usefulness from the proposed evaluation metrics?
-
Why do some qualitative examples in Figure 3 appear to show smaller trajectory deviations for V-JEPA2 and TSWAM, despite the reported improvements in TrajAlign and TrajGain for VideoVLA? Could the authors clarify how these metrics are computed and add additional video comparisons.
-
How were the trajectory annotations from SurgWMBench obtained? Given the relatively small training dataset, what evidence supports the generalization of the proposed method to unseen scenes and trajectories?
-
How are future-frame features used during the first training stage, and what information is available to the trajectory conditioner during inference? How do the authors ensure that target-frame information is not unintentionally used as a conditioning signal?
-
Why was Cosmos Predict 2.5 preferred over the surgical domain-adapted Cosmos-H-Predict model? Would the proposed trajectory module provide similar improvements when integrated into an existing surgical world model?