acceptodds
Withdrawn from ICLR 2027

Causal Visual–Trajectory Joint Prediction for Surgical Motion Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective surgical motion planning requires jointly anticipating future visual states and instrument trajectories, together with an explicit causal relationship between the predicted motion plan and the generated visual future. Existing surgical world models typically treat video generation and trajectory prediction as separate objectives or provide limited evidence that predicted trajectories meaningfully affect visual dynamics. To solve this problem, we propose a joint visual–trajectory prediction framework that establishes an explicit trajectory-to-visual pathway through spatially grounded motion conditioning. The framework transforms predicted trajectories into temporally aligned spatial representations and associates them with observed local appearance to guide future visual dynamics. We further introduce a visual–trajectory causal evaluation framework that measures local visual responses at spatial positions and time steps associated with instrument trajectories. Trajectory-Grounded Motion Alignment (TrajAlign) and Trajectory-Grounded Response Gain (TrajGain) characterize the directional consistency and relative strength of these responses, enabling quantitative assessment of how trajectory conditioning influences future visual dynamics. Experiments demonstrate improved trajectory accuracy and visual–trajectory consistency while maintaining competitive visual quality. Controlled response analysis provides quantitative evidence that trajectory conditioning induces directionally consistent changes in local visual motion, making the trajectory–visual relationship both explicit and measurable.

Primary area:
generative models
Keywords:
Surgical World Models, Visual–Trajectory Joint Prediction, Causal Video Generation, Surgical Motion Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JMszOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a joint visual-trajectory prediction framework for surgical world modeling, with applications in surgical motion planning. The proposed framework models tool trajectories using spatial features and temporally aligns them with visual representations. The aligned features are then injected into an existing world model to guide video prediction. Experiments are conducted on an existing surgical benchmark, with the generated videos evaluated for visual quality and trajectory alignment. The paper also introduces two additional metrics to quantify the influence of trajectory conditioning on video generation.

Critical strengths
  1. The idea of separating trajectory control from visual modeling is interesting. Controllable generation is particularly important in surgery, given the variability in anatomy, surgical instruments, and scene dynamics. The proposed approach moves in a useful direction by combining explicit tool-tip trajectories with visual content for future video prediction.

  2. The idea of identifying limitations in existing surgical world models and building upon them is appreciated. Given the limited availability of surgical data, improving the controllability of existing pretrained models is a useful direction. This could potentially improve the applicability of such models without requiring an entirely new surgical video generation framework.

Critical weaknesses
  1. Limited evaluation of practical utility for surgical applications: The main limitation of this work is the lack of evaluation showing the practical usefulness of the proposed trajectory conditioning. Improving controllability in world models is an established research direction [1-3], and its application to surgery requires further justification. The paper motivates surgical action anticipation but primarily evaluates visual quality and future tool trajectories. A correctly predicted trajectory does not necessarily imply that the generated video captures meaningful surgical interactions or is useful for downstream tasks. For example, evaluating whether the generated videos improve surgical action anticipation, predict potentially unsafe tool-tissue interactions, or support robotic policy learning would provide stronger evidence of the contribution. Without such experiments, the practical benefit of the proposed method for surgical applications remains unclear.

  2. Relationship between trajectory alignment and generation quality. The qualitative results in Figure 3 raise questions about the proposed evaluation metrics. For baselines such as V-JEPA2 and TSWAM, the deviation of the predicted tool trajectories from the ground truth appears relatively small. However, VideoVLA appears to show greater deviations in some examples while achieving better scores on the proposed TrajAlign and TrajGain metrics. The paper should explain this discrepancy and clarify what these metrics capture. More importantly, it remains unclear whether improvements in trajectory alignment translate to better modeling of surgical actions or improved downstream performance.

  3. Information of eval dataset. The paper uses data from SurgWMBench, but the corresponding surgical video annotations do not appear to be publicly available. It is therefore unclear how the authors obtained the required trajectory annotations for training and evaluation. The source of these annotations, their generation procedure, and any validation performed should be clearly described. This information is important for assessing the reliability and reproducibility of the experimental results.

  4. Limited training data and potential overfitting. The training dataset consists of only 2,024 clips, each containing 20 frames. Given the two-stage training procedure, this appears to be a relatively small dataset for learning a trajectory-conditioned video generation framework. The reported improvements could potentially reflect adaptation to the motion patterns and scene distributions in this dataset rather than generalizable trajectory control. The paper should provide evidence that the model has not overfitted, preferably through evaluation on unseen surgical scenes, procedures, or tool trajectories.

  5. Unclear use of future-frame information during training. Line 308 describes an initial trajectory training stage followed by joint visual-trajectory training. Appendix A further mentions that the first stage uses future frames and ground-truth trajectories to train the spatial trajectory conditioner while keeping the video generation model frozen. It is unclear how the future-frame features are used and whether the same information is available during inference. Although using future frames as supervision during training is reasonable, using features derived from target frames as conditioning inputs could introduce a train-test mismatch or unintended information leakage. The paper should clarify the training and inference information flow and explain why the video reconstruction loss is needed during this stage.

  6. Handling of the original action conditioning The Cosmos-H Surgical Simulator is an action-conditioned world model that takes robotkinematics as input. However, the paper does not clearly explain what happens to this action-conditioning mechanism when the proposed trajectory module is introduced. Is the original action conditioner retained, frozen, modified, or removed? This is important because action conditioning is a central component of the original world model.

  7. Backbone choice and missing surgical baselines. The motivation for using Cosmos Predict 2.5 as the backbone is unclear when a surgical model such as Cosmos-H-Predict is already available. The latter has been adapted to surgical data and could provide stronger domain-specific representations for the proposed trajectory conditioning. Adding the proposed module to this surgical backbone would be a relevant comparison for determining whether the improvements arise from trajectory conditioning or from the characteristics of the selected video generation backbone.

Minor comments (Not necessary for the final rating)

  1. The description of the trajectory encoder is incomplete. The architecture, parameter count, and implementation details should be provided. This is particularly important if the training code and dataset are not planned for public release, as the current description limits reproducibility.

  2. Section 3.2 refers to the Cosmos-H Surgical Simulator [4], which is an action-conditioned surgical world model. However, the baseline evaluation also mentions Cosmos-H-Surgical [3], while other parts of the paper refer to Cosmos Predict 2.5. The authors should clearly identify which model is used as the backbone, which models serve as baselines, and how the proposed method differs from each of them.

References:

  1. Rapuri et al., SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation, MICCAI 2026
  2. Sivakumar et al., SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation, MICCAI 2026
  3. He et al., Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling, 2025
  4. Cosmos-H-Surgical-Simulatore, NVIDIA (https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical-Simulator)
Questions for authors
  1. Can the authors provide results that improved trajectory alignment benefits a surgical application, such as action anticipation, tool-tissue interaction prediction, or robotic policy learning? What conclusions can currently be drawn about surgical usefulness from the proposed evaluation metrics?

  2. Why do some qualitative examples in Figure 3 appear to show smaller trajectory deviations for V-JEPA2 and TSWAM, despite the reported improvements in TrajAlign and TrajGain for VideoVLA? Could the authors clarify how these metrics are computed and add additional video comparisons.

  3. How were the trajectory annotations from SurgWMBench obtained? Given the relatively small training dataset, what evidence supports the generalization of the proposed method to unseen scenes and trajectories?

  4. How are future-frame features used during the first training stage, and what information is available to the trajectory conditioner during inference? How do the authors ensure that target-frame information is not unintentionally used as a conditioning signal?

  5. Why was Cosmos Predict 2.5 preferred over the surgical domain-adapted Cosmos-H-Predict model? Would the proposed trajectory module provide similar improvements when integrated into an existing surgical world model?

Reviewer_vj9BOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies joint prediction of future surgical video and future instrument-tip trajectories, with the goal of making the trajectory→visual relationship explicit and measurable rather than treating the two as parallel heads. The model is built on top of Cosmos-H-Surgical-Simulator. Given 5 observed frames and the corresponding tip coordinates, a Transformer encoder–decoder planner predicts 15 future tip positions by accumulating bounded per-step displacements. These are converted into per-step spatial motion fields (Gaussian maps carrying displacement components and magnitude, plus a numerical descriptor), temporally aligned to the 4 future latent slots using an offline frame-replacement probe of the frozen video encoder, and combined with a local appearance descriptor sampled around the last observed tip. The fused features are injected as zero-initialized residuals into future latent slots at four backbone layers.

Critical strengths
  1. The problem is well-motivated and the gap is real. The observation that a joint model can produce an accurate trajectory and a plausible video while the two remain behaviorally disconnected is a genuine failure mode of parallel-head formulations, and it is rarely made explicit in the surgical world-model literature.
  2. The VAE temporal-alignment probe is a clean idea. Recovering frame↔latent-temporal correspondence by individually replacing each future frame with its predecessor on a frozen encoder and measuring the response is a principled, cheap alternative to the hand-specified contiguous grouping most trajectory-conditioned generators use. Initializing the resampler from these measured responses while keeping the support fixed is sensible.
  3. The evaluation protocol is the paper's most valuable contribution. Paired interventions with fixed history and fixed noise isolate the effect of the conditioning signal in a way that standard PSNR/ADE reporting cannot. The Appendix A.5 shuffled-condition ablation is a particularly well-designed control: it shows image fidelity is essentially unchanged while TrajAlign collapses from 0.41 to 0.04, cleanly decoupling "good video" from "responsive to the plan." I expect this protocol to be reused.
Critical weaknesses
  1. W1 — "Causal" is stronger than what is measured. What is demonstrated is that intervening on the conditioning input of a fixed generator produces directionally consistent changes in its output. There is no counterfactual ground truth: we do not know what the scene should look like under a 60°-rotated trajectory, and a large fraction of the sampled interventions (especially ±60° and 2×) are likely physically implausible for the anatomy. TrajGain therefore measures responsiveness, not correctness. I would ask the authors to either soften the terminology to "interventional controllability / plan-responsiveness" throughout, or add an explicit statement of this limitation. As written, a reader will over-interpret the headline claim.
  2. W2 — Effect size is modest, and the absolute numbers are weak. TrajAlign = 0.411 is far from 1, and TrajGain = 0.393 means the generated local motion realizes only ~39% of the requested displacement along the trajectory direction. Describing 0.393 as "the TrajGain closest to unity" is technically true but rhetorically generous. More importantly, the marginal gain over the configuration that already inherits the base model's global trajectory condition (row 1 of Table 2, TrajAlign 0.326) is +0.085 — the paper should foreground this number, since it is the honest measure of what the proposed spatial pathway adds. A dose–response analysis (response magnitude as a function of intervention magnitude: 0.5× vs 1.5× vs 2×, 30° vs 60°) would substantially strengthen the causal story; a single pooled number does not distinguish a genuine graded response from a saturated one.
  3. W3 — The metrics rest on an unvalidated motion estimator. TrajAlign/TrajGain are computed from Farnebäck dense optical flow on generated 288×512 frames, averaged over a 49×49 ROI. Farnebäck is known to degrade badly under large displacement, motion blur, and occlusion — precisely the artifacts generated surgical video exhibits. Nowhere is the flow estimator validated (e.g., against synthetic warps, or against GT trajectory-derived flow on real frames), nor is its sensitivity reported against a modern estimator (RAFT/FlowFormer). Additionally, the ROI response includes global endoscopic camera motion, which is a systematic confound: the paper should report a control at off-trajectory locations, or camera-motion-compensated flow.
  4. W4 — Baseline comparison is confounded by resolution. VideoVLA, Cosmos-Predict2.5, TSWAM and V-JEPA2 generate 256×256 and are anisotropically resized to 288×512. This is not a neutral operation — it changes aspect ratio — and it affects PSNR/SSIM/LPIPS and the flow-based metrics in ways whose direction is unclear. Either evaluate all methods at their native resolution, or additionally report the proposed method at 256×256. Relatedly, the two baselines that most directly condition generation on motion — SurgSora and SAW — are discussed in related work but never compared, which makes the Table 1 TrajAlign comparison much weaker than it appears: models that never feed the trajectory into generation (e.g., Cosmos-Predict2.5, per Table 5) trivially score ≈0. The informative controllability comparison is Table 4, not Table 1.
Questions for authors
  1. Can you report TrajAlign∆/TrajGain∆ separately per intervention type and magnitude, to show the response is graded rather than saturated?
  2. What is TrajAlign/TrajGain of the ground-truth future video measured with the same protocol? This upper bound is essential for interpreting 0.41/0.39.
  3. Are the 64 intervention clips disjoint from the 64 validation clips used for model selection?
Reviewer_L1ZBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper jointly predicts 15 future surgical frames and a 2D instrument-tip trajectory from 5 observed frames and their tip coordinates. A Transformer predicts the future trajectory from past coordinates only. Each trajectory step is turned into a Gaussian motion field. These are grouped into the backbone's 4 latent time slots using a temporal prior measured by probing the video encoder. They are then combined with local appearance sampled at the last tip position and added as zero-initialized residuals at 4 layers of a frozen Cosmos-H-Surgical-Simulator. During joint training, the video flow-matching loss is backpropagated into the trajectory predictor.

Two metrics are proposed. TrajAlign is the cosine between local optical flow and the trajectory displacement. TrajGain is the ratio of projected flow magnitude to displacement magnitude. Together with paired interventions (rotating or rescaling the trajectory), they quantify how strongly the video responds to the trajectory. On SurgWMBench, the method reports the best ADE, FDE and TrajAlign among joint baselines, with slightly lower visual quality than the best video-only model.

Critical strengths
  1. The problem is well motivated. The surgery tasks require only learning from the expert demonstration without exploration. And it wants to learn a world model for the surgery and predict the action.
  2. The paired-intervention protocol is sound. History and noise are fixed, and confidence intervals use a bootstrap grouped by source video. The missing- and shuffled-trajectory controls (Table 4) are useful.
  3. The engineering is careful: zero-initialized residuals, a frozen backbone, a full 2³ component ablation, documented baseline adaptations, and train/test splits disjoint by source video.
Critical weaknesses

W1. Motion planning is not demonstrated.

  • The action planner sees only 5 past tip coordinates. It has no access to the images or anatomy, and it is trained by regression to the demonstrated future.
  • The world model is never used to evaluate, rank or optimize candidate trajectories. Moreover, you can use the world model to do model-predictive control.
  • The title, abstract and introduction ("assessing candidate motions and anticipating risks") overclaim. The authors should either demonstrate planning (sample candidates, roll them out, select, and show a gain) or reframe the work as trajectory-conditioned forecasting.

W2. The benefit of joint learning is marginal.

  • Visual feedback improves ADE from 18.76 to 18.53 px, and the method is essentially tied with TSWAM (18.53 vs 18.55).
  • In Table 2, the model with all components disabled reaches ADE 18.61, better than "Ours (w/o visual feedback)" at 18.76. Enabling only the spatial component worsens ADE to 19.52. Run-to-run variance therefore looks comparable to the claimed gains.

W3. Novelty over existing conditioning methods is limited, and baselines are missing.

  • Gaussian trajectory maps (DragAnything), spatiotemporal motion injection into transformer blocks (Tora) and zero-initialized residual control (ControlNet) are all established.
  • The temporal prior has a negligible effect on its own (Table 2).
  • There is no comparison with Tora- or DragAnything-style conditioning attached to the same frozen backbone and fed the same predicted trajectory.

W4. Generalization and dependence on the backbone are not tested.

  • Visual quality is below the video-only Cosmos-H Predict, and the frozen backbone with no added modules is not reported.
Questions for authors

See weakness.

  • Why is predicting video necessary for better trajectories? If you compare with (i) image features given to the predictor, (ii) auxiliary representation losses without a generator, and (iii) GRPO or RL using the world model as a simulator, it can better support the claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.