acceptodds
Under review as a conference paper at ICLR 2027

A Composed State-Space World Model for Ultrasound-Guided Spinal Surgery

Abstract

Ultrasound-guided pedicle screw placement is studied as two disconnected problems: bring the probe to a standard imaging plane, and drill a planned corridor once you are there. The second is always evaluated with the probe teleported into place from ground truth, so no published number says whether the pair composes. We solve each problem independently with a state-space world model, then compose the two in sequence. Each is a learned readout of the tool's pose in a physically named frame, propagated by a small transition and planned by CEM-MPC over their respective objectives. The composition replaces the SonoGym simulator's privileged probe placement with a navigated one. In isolation each half is competitive: while learning its dynamics the drilling controller reaches 3.00 mm insertion and 2.80 mm lateral at 90% safety, beating both reported reinforcement-learning baselines on both position axes (their 11.9-12.3 and 5.07-5.42), and exact kinematics improve that again to 2.1/2.0 mm; the probe controller reaches and holds the plane in 0.95 of episodes. Composed, safety falls from 90% to 0% and success to 0.00, and it fails in one place, not everywhere: drill orientation survives a 32 deg probe azimuth error while lateral offset does not. Instead of building a better navigator, we inject known amounts of probe error and measure how much the drilling controller absorbs. The pose estimator turns out not to degrade at the new probe position but to stop responding to its input, reporting the same large error wherever the probe is, which is what an estimator does when both its image and its own output are far outside the distribution it was fitted on. Post-training it at the position where it is actually used repairs this. The composed system then recovers most of what the teleport gave it: safety returns to 85% against the teleported arm's 90% with the exact propagator and to 75% at matched propagator. Auditing the forecasts the planner already computes and discards splits its 13.2 mm error into a propagator term and a perception term: two independent routes put the propagator's share at 9.2 and 9.4 mm, and correcting it leaves a perception-limited 3.8 mm. Two further probes put a hand-designed alignment cap, not the planner, in control inside the corridor. Three lessons about measuring such compositions, demonstrated here rather than established in general: one-channel tolerances do not compose, the joint failing where each channel alone passes; the tightest tolerance is on a probe axis neither task commands, failing by leaving the intervention undone while the safety metric still reads safe; and the controller's own introspective signals invert, the arm with the worst forecast controlling best and the one that fails completely proving the most self-consistent.

Primary area:
reinforcement learning
Keywords:
world models, model-based planning, CEM-MPC, robotic ultrasound, surgical robotics, evaluation protocols

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PFzPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed a state-space world model for simulated ultrasound-guided robotic drilling. Specifically, a model predicts the kinematic state from the observations, and the states are propagated through a closed-form model or an MLP. The paper analyzes proposed methods for US navigation, US-guided drilling, and their sequential combination. On the methodology side, the pipeline is similar to the existing image-based MPC methods, and the main contributions are unclear. On the presentation side, the paper needs significant improvement.

Critical strengths
  1. The studied application is challenging but meaningful
  2. The paper observed a limitation of SonoGym, which is valid.
Critical weaknesses
  1. The overall presentation is below the ICLR conference standard
  • The paper is not organized in a standard and structured way, with nested methods and results, without clear methodology and experiment sections.
  • The text is generally hard to follow, with too dense, unstructured information. Some paragraphs are too long, including the abstract and several captions for several figures and tables
  • The figure quality requires significant improvement: Figure 1 is unpolished with very small text; Figure 2 has inverted arrow directions
  • The bolded text in the tables is misleading. In some tables, the best numbers are highlighted, and in others the proposed method is highlighted.
  1. The main contribution of the proposed work is unclear. Especially, predicting state from images and then applying MPC has been investigated in many existing papers, for instance: “Vision-Based High Speed Driving with a Deep Dynamic Observer” and “Robust-Sub-Gaussian Model Predictive Control for Safe Ultrasound-Image-Guided Robotic Spinal Surgery”. Here, the second paper even focuses on the same application problem and provides more solid theoretical guarantees and analysis.

  2. Applying the proposed method only to one specific simulated task limits the impact of the work

Questions for authors
  1. What is the main contribution of this work compared to “Robust-Sub-Gaussian Model Predictive Control for Safe Ultrasound-Image-Guided Robotic Spinal Surgery”?
  2. Why is the closed-form dynamic model wrong in line 205? Is it because of the tracking error from the robot, or the pose estimation uncertainty from the US images? While existing works focus on quantifying these uncertainties, the benefits of learning the pose propagation with an MLP are unclear.
  3. What is the conclusion of the paper? It seems the analytic propagation still achieves higher accuracy compared to MLP-based state propagation in most of the cases. Does it mean that model-based propagation is still preferred in practice?
1 comment

We thank the reviewer for a careful review; we agree with much of it. Unless stated, numbers are 4 seeds × 100 environments (400 episodes) for the arm with the learned (MLP) propagator, the paper's world model. SAFE (Eq. 2) is the fraction of episodes never unsafe at any step; SAFE-state is SonoGym's own measure, the fraction of steps that are safe, as its baselines report. ins/side/rot: insertion, lateral (mm) and angle (°) errors; succ: SUCCESS.

Corrections to the submitted numbers (LB-IDT, n = 20 → 400)

conditioninssiderotSAFE-stateSAFEsucc
drilling, teleportedsubmitted3.002.804.20–90.00.75
now3.363.654.0099.383.00.537
composition, teleportsubmitted4.95.76.6–900.20
now3.933.974.6499.589.10.472
composition, repairedsubmitted5.66.97.5–750.20
now5.436.115.3196.280.60.388

Navigation: 8.3 mm / 5.2° submitted, 1.1 mm / 8.3° now. The claim "costs nothing" rested on the two equal 0.20s and does not survive: the repair recovers 90% of the teleport's SAFE (p = 0.033) and 82% of its success (p = 0.030), so a gap remains, and we withdraw the claim.

Contribution. We agree that image-to-state-then-MPC is established (Vision-Based High-Speed Driving with a Deep Dynamic Observer is an instance), and we do not claim it as our contribution. The contribution is what happens when two such controllers are composed. SonoGym places the drilling probe from ground truth, with at most 5 mm of tangential randomisation. A navigated probe hands off 7.85 mm and 32.4° of azimuth error, a channel that randomisation never covers:

probe placed byinssiderotSAFE-stateSAFEsucc
teleport (oracle)3.933.974.6499.589.10.472
navigator, unrepaired11.5324.0913.4845.20.00.000
navigator, repaired5.436.115.3196.280.60.388

Drilling tolerates only 10–15° of azimuth error (submitted §6), so larger errors lead to catastrophic failure. The navigator's readout saturates, and retraining it where it is used recovers most of the gap. On four more vertebrae, rerun from scratch, the pattern holds: the unrepaired composition is safe in at most 0.3% of episodes, and the repair recovers 88–100% of the teleport's SAFE. Full per-vertebra tables will be in the revised appendix; we are happy to share them here if helpful.

Presentation. This is fair. In the revised manuscript, we will follow standard paper-writing practice and address the concerns raised.

Q1. Robust Sub-Gaussian MPC. We will cite it. It models image-based estimation error as sub-Gaussian with bounded mean and gives closed-loop guarantees for MPC on linear dynamics; since it covers biased estimates, we do not claim our failure lies outside its noise model. The works are complementary: its guarantee is conditional on error bounds estimated in the nominal setup, where probe placement is not part of the problem. We measure what those bounds must cover once a learned navigator places the probe: a near-constant 32.4° azimuth error against a 10–15° tolerance. A bias set that wide would be very conservative, and that paper itself notes that a fully robust treatment can be infeasible. After our repair, the handoff error is 5.5°.

Q2. Why is the closed form wrong? The cause is the plant: the drill controller realises about 9% of the commanded translation per step and damps lateral and angular motion about 5× in the corridor. The learned propagator, fit to achieved poses, absorbs this. At h = 1, its depth-forecast error is 4.0 mm, nearly all of it perception; the closed form's 13.2 mm is 3.9 mm perception and 9.2 mm propagator. Because the closed form's error is a single scalar gain and the controller re-plans every step, the error rescales commands rather than misdirecting them, so the closed form controls better despite forecasting worse (41 vs 5.5 mm at h = 10): objective mismatch (submitted §7).

Q3. Is model-based propagation preferred? Where exact kinematics exist, yes; we include that arm to measure the learned propagator against it:

propagatorinssiderotSAFE-stateSAFEsucc
learned MLP3.363.654.0099.383.00.537
closed form2.682.972.1398.886.00.723

The difference in SAFE is not significant (p = 0.28); the difference is in accuracy. The learned propagator also transfers to new vertebrae without refitting (refitting per vertebra buys nothing consistent; ablation in the revised appendix), and every composition result holds with it.

One simulated task. Agreed. We would only add that the setting combines two SonoGym tasks, now tested in all four of its settings and on five vertebrae. We make no clinical claim.

Reviewer_eLCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a state-space world model that integrates ultrasound probe navigation and robotic drilling into a unified surgical workflow using model predictive control. It demonstrates that independently successful controllers can fail when composed, primarily due to out-of-distribution pose estimation errors.

Critical strengths

Strengths:

  1. It addresses an important limitation of existing surgical simulation benchmarks, where drilling controllers rely on privileged probe placement. Evaluating the complete navigation-to-drilling pipeline provides a more realistic assessment of robotic surgical autonomy.

  2. It systematically investigate composition failures through controlled perturbations, error decomposition, and estimator analysis. Their findings reveal important interactions between position and orientation errors that cannot be captured through independent evaluations.

  3. The proposed post-training strategy substantially improves composed-system safety from 0% to 85%, approaching the 90% oracle-probe baseline. The analysis also provides valuable insights into the relationship between prediction accuracy, control performance, and safety constraints.

Critical weaknesses

Weakness:

  1. All experiments are conducted in a single simulator using a fixed anatomy family and synthetic ultrasound images. The small evaluation sample of 20 episodes per condition further limits statistical confidence and generalization to realistic clinical environments.

  2. The proposed recovery simultaneously modifies the estimator's training distribution and expands the navigation workspace. Without independently evaluating these changes, it remains difficult to determine their respective contributions to performance improvements.

  3. The framework primarily combines established techniques, including supervised pose estimation, DAgger, and CEM-MPC, while relying heavily on handcrafted safety constraints. Moreover, the analytical propagator outperforms the learned alternatives, raising questions about the necessity and contribution of learned dynamics.

Questions for authors
  1. How well does the composed framework generalize to previously unseen anatomies, ultrasound noise levels, and imaging distributions? Would the proposed post-training strategy remain effective without additional environment-specific supervision?

  2. Can the authors provide separate ablations for workspace expansion and estimator post-training? This would help establish whether the performance recovery primarily results from improved perception or relaxed navigation constraints.

  3. Given that the analytical propagator consistently outperforms the learned models in closed-loop control, what advantages justify incorporating learned dynamics? Could a simpler analytical controller achieve comparable performance with lower computational and training costs?

  4. How does the controller behave under simultaneous probe position, azimuth, and roll errors beyond the evaluated perturbation ranges?

1 comment

We thank the reviewer for a careful review and would like to address its concerns with updated numbers. Unless stated, numbers are 4 seeds × 100 environments (400 episodes) for the arm with the learned (MLP) propagator. SAFE (Eq. 2) is the fraction of episodes never unsafe at any step; SAFE-state is SonoGym's own measure, the fraction of steps that are safe. ins/side/rot: insertion, lateral (mm) and angle (°) errors; succ: SUCCESS.

Corrections to the submitted numbers (LB-IDT, n = 20 → 400)

conditioninssiderotSAFE-stateSAFEsucc
drilling, teleportedsubmitted3.002.804.20–90.00.75
now3.363.654.0099.383.00.537
composition, teleportsubmitted4.95.76.6–900.20
now3.933.974.6499.589.10.472
composition, repairedsubmitted5.66.97.5–750.20
now5.436.115.3196.280.60.388

Navigation: 8.3 mm / 5.2° submitted, 1.1 mm / 8.3° now. The repair recovers 90% of the teleport's SAFE (p = 0.033) and 82% of its success (p = 0.030), so a gap remains, and we withdraw the "costs nothing" claim.

W1 / Q1. Generalisation. We evaluate all four SonoGym settings (LB: GAN renderer, MB: ray-cast; ODT: held-out GAN instance on LB, unseen patient on MB); the propagator is fit once and never refit.

settingdrilling: inssiderotSAFE-stateSAFEnavigation: final pos / rotcomposition succ: teleport / unrepaired / repaired
LB-IDT3.363.654.0099.383.01.12 / 8.260.472 / 0 / 0.388
LB-ODT4.686.814.9592.563.412.56 / 4.980.193 / 0 / 0.068
MB-IDT5.132.754.9399.589.80.89 / 8.430.518 / 0 / 0.545
MB-ODT14.8310.858.2584.84.5– (not defined)–

Imaging distribution and noise can be modelled as held-out GAN instance (LB-IDT → LB-ODT)

Anatomy. Per-patient retraining works: rerun from scratch on four more vertebrae, the unrepaired composition is safe in at most 0.3% of episodes on every one, and the repair recovers 88–100% of the teleport's SAFE. On the unseen patient (MB-ODT), most episodes over-drill (the depth readout reads ~2 mm shallow), so the recipe is re-run per patient. Full per-vertebra tables will be in the revised appendix; we are happy to share them here if helpful.

Without environment-specific supervision? As specified, post-training uses privileged labels. We conducted an additional label-free alternative experiment(test-time adaptation of BatchNorm scale/shift on the controller's own rollouts) cuts unseen-patient insertion error from 14.83 to 3.65 mm, raises SAFE from 4.5% to 38.8% and raises SAFE-state from 84.8% to 89.7%. Full results will be in the revised appendix.

W2 / Q2. Workspace vs post-training. Done: the old readout with only the workspace widened (voxel scale 100 → 92, in both the environment and the planner), with paired start states:

readout / workspacehandoff (mm / °)succ
old / stock7.64 / 32.400/100
old / widened only11.80 / 32.720/100
post-trained / widened6.32 / 5.540.388

Widening alone makes the handoff worse; the azimuth error stays the same because it comes from the readout. The one-seed 95% upper bound on success is 0.037, so the recovery comes from where the readout is trained. The fourth cell cannot be run, since the drilling probe pose lies outside the stock workspace. We used one seed because the old readout's azimuth error is 32.36–32.40° on all four seeds.

W3 / Q3. Why learned dynamics? Could an analytical controller do as well? This comparison is already in the paper: the closed-form row, included to measure the learned propagator against it:

propagatorinssiderotSAFE-stateSAFEsucc
learned MLP3.363.654.0099.383.00.537
closed form2.682.972.1398.886.00.723

The difference in SAFE is not significant (p = 0.28); the difference is in accuracy. That controller still needs the learned pose readout, since the drill's pose comes from ultrasound; the only choice is whether to learn the dynamics too. Learned dynamics also transfer to unseen vertebrae without refitting.

Q4. Joint position, azimuth and roll errors. The position × azimuth grid (submitted §6) shows that the two channels interact: where independence predicts 0.56 safety, we measure 0.30. Swept alone, roll is 4× more damaging than azimuth, and the drill stops short instead of breaching the wall. We have not run roll jointly; we expect a third channel to tighten the safe region further and will state this limitation. In simulation, roll never crosses the interface; on hardware, it would.

Reviewer_kne7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper develops world-model controllers for ultrasound probe navigation and pedicle screw drilling, then evaluates them together. Although both perform well independently, their composition fails because probe placement errors expose a pose estimator that stops responding to its input. Controlled error experiments reveal this failure and show why evaluating each stage separately can hide critical weaknesses.

Critical strengths
  • The idea is straightforward and easy to understand.
  • The experiments provide interesting practical insights, showing how individually effective controllers can fail when combined.
Critical weaknesses
  • The paper addresses a surgical robotics problem and may be better suited to a medical robotics conference.
  • The experiments are conducted entirely in simulation, with insufficient discussion of how faithfully the simulator represents real surgical conditions. No clinical or real-world validation is provided to support the method’s transferability.
  • While the approach is interesting, its components are well established in the machine learning community, limiting the methodological novelty.
  • The medical application is the paper’s most compelling aspect, but the lack of real-world clinical validation makes its practical relevance difficult to assess.
  • Learning is limited to perception and dynamics, while corridor geometry and control rules are predefined.
  • Validation on a single simulated patient leaves real-world performance uncertain.
Questions for authors

I mainly have concerns about two aspects: 1. the novelty of the proposed method for ML community,, and 2. the effectiveness of the method in real-world medical domains.

1 comment

We thank the reviewer for the careful review and answer both concerns with new experiments. Unless stated, numbers are 4 seeds × 100 environments (400 episodes) for the arm with the learned (MLP) propagator. SAFE (Eq. 2) is the fraction of episodes never unsafe at any step; SAFE-state is SonoGym's own measure, the fraction of steps that are safe, as its baselines report. ins/side/rot: insertion, lateral (mm) and angle (°) errors; succ: SUCCESS.

1. Novelty for the ML community

Established components. We agree that the parts (supervised readout, DAgger, CEM-MPC) are standard, and we do not claim them. We claim three things.

(a) A new task: the composition. SonoGym studies navigation and drilling separately, placing the drilling probe from ground truth with at most 5 mm of tangential randomisation. In our composed task, the drilling controller receives its probe pose from a navigator, which carries 32.4° of azimuth error, a channel that randomisation never exercises. As far as we know, this has not been studied before.

(b) Each half is competitive. Against SonoGym's published baselines (LB-IDT):

navigationfinal pos (mm)final rot (°)
PPO1.87.3
ACT2.88.4
diffusion policy0.3 ± 10.28.6
ours1.18.3
drillingsideinsrotSAFE-stateSAFE
PPO5.4212.33.7093.1–
PPO + safety filter5.0711.93.9395.3–
ACT5.341.621.0186.0–
ours3.653.364.0099.383.0

The baselines report only SAFE-state.

(c) Failure analysis and a recovery path. Composed, the two competitive halves fail completely (LB-IDT):

probe placed byinssiderotSAFE-stateSAFEsucc
teleport (oracle)3.933.974.6499.589.10.472
navigator, unrepaired11.5324.0913.4845.20.00.000
navigator, repaired5.436.115.3196.280.60.388

Injecting known probe error shows that drilling tolerates only 10–15° of azimuth error, so azimuth is the failing channel. The navigator's readout turns out to saturate, reporting roughly the same pose wherever the probe is, which points to the repair: retraining the readout where it is used recovers 90% of the teleport's SAFE. The analysis also shows that tolerances measured one channel at a time do not combine: where independence predicts 0.56 safety, we measure 0.30.

2. Effectiveness in real-world medical domains

We make no clinical claim. Our point is that this kind of evaluation must come before hardware, because a benchmark that places the probe from ground truth would have passed this system.

Single patient. We reran the recipe from scratch on four more vertebrae (the drilling propagator was never refit). Drilling alone keeps 93–99% SAFE-state on all five (SAFE 44–91%; B and I are the hardest). Navigation reaches the plane on all five (success 0.55–1.00). The composition failure and its repair reproduce on every one: unrepaired, at most 0.3% of episodes are safe; repaired, the composition recovers 88–100% of the teleport's SAFE. Full per-vertebra tables will be in the revised appendix; we are happy to share them here if helpful.

Simulator fidelity. SonoGym runs on Isaac Lab with real CT anatomy (TotalSegmentator) and ultrasound from either a ray-cast model (MB) or a pix2pix GAN trained on seven ex-vivo specimens (LB). Patients are static rigid bodies (a stated SonoGym limitation), so our evaluation says nothing about tissue deformation, breathing or contact force. We expect the effect to matter more on a real system, since probe roll, the tightest tolerance we measured, never crosses the interface in simulation but would on hardware.

Distribution shift. We test every shift SonoGym defines (ODT: held-out GAN instance on LB, unseen patient on MB):

settingmethodsideinsrotSAFE-stateSAFE
LB-ODTPPO / +filter / ACT6.41 / 4.56 / 6.2626.9 / 27.2 / 1.723.76 / 3.59 / 1.8193.1 / 96.0 / 80.3–
ours6.814.684.9592.563.4
MB-IDTPPO / +filter / ACT2.32 / 2.17 / 5.3816.0 / 13.9 / 2.925.11 / 5.36 / 0.6299.9 / 100 / 65.9–
ours2.755.134.9399.589.8
MB-ODTPPO / +filter / ACT5.66 / 8.94 / 18.420.5 / 27.3 / 14.54.54 / 4.55 / 5.9888.3 / 89.8 / 65.3–
ours10.8514.838.2584.84.5

The unseen patient is our weakest setting: most episodes over-drill because the depth readout reads about 2 mm shallow. We also tested label-free test-time adaptation on the controller's own rollouts, which cuts insertion error from 14.83 to 3.65 mm, raises SAFE from 4.5% to 38.8% and raises SAFE-state from 84.8% to 89.7%; full results will be in the revised appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.