acceptodds
Under review as a conference paper at ICLR 2027

SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models

10 views

Abstract

Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments: physical phantoms and acoustic simulators exhibit a substantial domain gap from clinical images, and trial-and-error exploration on patients is unsafe. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an electromagnetic (EM) sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive generation with selectively caching eviction tokens at inference time. We evaluate controllability through pose-following fidelity, round-trip consistency, and axis-specific control under synthetic actions on held-out tracked scans, and further validate it with a user study. In model-based action discrimination, SonoWorld correctly ranks the target-directed probe motion over an equal-magnitude opposite motion in up to 74.8% of trials. To our knowledge, SonoWorld is the first causal, pose-controllable ultrasound world model learned primarily from freehand clinical videos, providing a data-driven interactive environment for robotic ultrasound data acquisition and policy evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Model, Ultrasound Imaging, Medical AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gCkgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work presented a pose-controllable ultrasound world model. The training includes 3 stages: i) pretraining with large-scale US videos without pose annotation, ii) post-training with small-scale tracked US data, and iii) self-force distillation to ensure causality. The claimed contributions include: i) enabling synthetic realistic robotic ultrasound data collection, ii) leveraging large-scale untracked US data, and iii) preserving data quality for causal autoregressive generation.

Critical strengths
  1. The proposed pose-controllable ultrasound world model is an interesting approach, particularly given its potential to leverage large-scale untracked ultrasound data.
  2. Preserving data quality during autoregressive generation is a meaningful practical contribution.
  3. The paper presents a broad set of experiments to validate the proposed method and its effectiveness.
Critical weaknesses
  1. An important line of prior work is not discussed, which may weaken the paper's claimed contributions. For example, SonoGym explores ultrasound simulation through a pipeline combining 3D CT data, pose-based 2D CT slicing, and learning-based CT-to-ultrasound generation. This approach addresses a closely related problem: generating realistic, pose-controllable ultrasound images. It may also offer advantages in computational efficiency and geometric consistency. The paper should more clearly articulate the contributions beyond this established line of work.
  2. The model is post-trained on a relatively small subset of EM-tracked ultrasound data. While the experiments show its ability to potentially generalize to action trajectories outside the distribution covered by this dataset, the underlying mechanism that allows this remains unclear.
  3. Although the learnable latent action model appears effective during pretraining, it is unclear whether this representation facilitates or complicates subsequent pose-conditioned post-training. A comparison of post-training results in Section 4.3 would better clarify the benefits of the latent actions.
Questions for authors
  1. How are the contributions of this work positioned against SonoGym?
  2. Is it possible to train robotic ultrasound policies with the proposed world model?
  3. Given the relatively small tracked ultrasound dataset used for post-training, how does the method generalize to action distributions and trajectories not observed during training? What mechanisms, if any, help ensure reliable predictions beyond the tracked data distribution?
  4. Are the learnable latent action tokens learned during pretraining beneficial for subsequent pose-conditioned post-training?
  5. What is the maximum sequence length the proposed world model can reliably simulate?
3 comments

We thank the reviewer for the constructive comments. Please find our responses below.

W1 & Q1: Contributions beyond SonoGym.

We thank the reviewer for highlighting SonoGym (Ao et al., 2025), a closely related work that we will discuss in Section 2. SonoGym provides efficient, geometry-grounded ultrasound simulation by extracting 2D slices from patient-specific 3D CT volumes and segmentation maps, followed by physics-based rendering or CT-to-ultrasound translation. However, it relies on static anatomical models without soft-tissue deformation, and its GAN-based generation does not explicitly enforce temporal consistency, as acknowledged by the authors.

SonoWorld takes a complementary, video-driven approach: (1) No patient-specific CT: SonoWorld learns from ~6.9M unlabeled clinical ultrasound frames and grounds the dynamics to 6-DoF probe actions with limited EM-tracked data, requiring no 3D anatomical model. (2) Clinical appearance and dynamics: SonoWorld learns ultrasound appearance, speckle statistics, and probe-induced image variations directly from real clinical videos rather than CT-based image formation. (3) Causal temporal modeling: SonoWorld generates each observation conditioned on the action and its generated history rather than synthesizing frames independently.

W2 & Q3: Mechanism on generalization to unseen action trajectories.

SonoWorld models action-conditioned local transitions autoregressively rather than memorizing complete trajectories. Actions are relative, egocentric 6-DoF motions in the preceding probe frame (Eqs. 3–4), and at each step the model predicts the next observation conditioned on the generated history and the action increment. Unseen trajectories are thus generated by sequentially composing local increments learned during training. Generalization may degrade when actions or anatomy fall far outside the tracked training distribution.

W3 & Q4: Benefits of latent-action pretraining.

We ran three comparisons: (i) Wan 2.1 without ultrasound pretraining, (ii) ultrasound pretraining without latent actions, and (iii) ultrasound pretraining with latent actions (ours). Causal-inference results are below (SSIM / LPIPS / PSNR).

TaskWan 2.1w/o latent actionsw/ latent actions (Ours)
Original pose0.283 / 0.534 / 16.320.375 / 0.447 / 15.690.386 / 0.439 / 16.09
Zero motion0.204 / 0.596 / 15.530.398 / 0.394 / 17.550.409 / 0.390 / 17.76
Speed×20.357 / 0.484 / 16.860.375 / 0.453 / 15.680.388 / 0.445 / 16.10
Round trip0.240 / 0.578 / 15.710.337 / 0.470 / 14.710.349 / 0.465 / 15.17
Long (8 s)0.214 / 0.609 / 15.270.382 / 0.444 / 15.950.394 / 0.436 / 16.37

First, latent-action pretraining consistently improves all three metrics over action-agnostic ultrasound pretraining across all five trajectory settings, confirming that it benefits the final pose-conditioned model. Second, ultrasound-domain pretraining substantially improves SSIM and LPIPS over direct Wan 2.1 initialization across all trajectories, highlighting the importance of adapting the video model to ultrasound appearance and dynamics.

Q2: Can robotic ultrasound policies be trained with SonoWorld?

Section 5.2 is a first step toward policy learning: given a current observation and a target frame, SonoWorld predicts the observation resulting from each candidate probe action and scores it by similarity to the target, correctly identifying the target-directed action in up to 74.8% of trials. Two extensions follow naturally. (1) Sampling-based planning: CEM or MPPI could optimize candidate action sequences through SonoWorld rollouts, execute the first action, and replan. (2) Model-based policy learning: a policy could interact with SonoWorld and learn from generated trajectories using task-specific rewards. We plan to validate full closed-loop policy training and sim-to-real transfer in future work.

Q5: Maximum generation sequence length.

We extended autoregressive generation to 18 s, the longest horizon our validation data support, on nine held-out EM-tracked sequences.

PolicyMetric3 s8 s12 s16 s18 s
LocalPSNR↑16.5816.2616.3016.1116.13
SSIM↑0.3980.3870.3930.3830.382
LPIPS↓0.4290.4400.4410.4470.443
SonoWorld (Ours)PSNR↑16.5816.3716.4916.3416.41
SSIM↑0.3980.3940.4060.3980.398
LPIPS↓0.4290.4360.4300.4360.435

With our selective-eviction KV cache, quality remains stable from 3 s to 18 s (PSNR −0.17 dB, SSIM unchanged, LPIPS +0.006), whereas a local sliding window under the same memory budget degrades steadily and is worse at every horizon beyond the training length. SonoWorld thus reliably generates ultrasound video up to 18 s, which we attribute to selectively retaining attention-salient early frames.

Thank you again for the constructive comments. In our response above, we discussed how SonoWorld differs from SonoGym and added a long-horizon evaluation up to 18 s, among other results. We would greatly appreciate any further questions about the distinction from SonoGym or the new results and will address them promptly.

Thank you for the detailed response. Since all my major concerns have been addressed, I will raise my score.

Reviewer_LTASOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SonoWorld learns an ultrasound video model from approximately 6.9 million untracked clinical frames, then grounds generation in relative 6-DoF probe motions using EM-tracked thyroid scans from 33 training subjects. A latent-action-conditioned diffusion model is adapted to physical actions and distilled into a causal generator with bounded history. Experiments include trajectory replay on nine held-out subjects, modified motions, eight-second rollouts, expert ratings and binary action discrimination. The intended contribution is an interactive learned environment that reduces dependence on tracked data for ultrasound simulation.

Critical strengths

Learning from untracked clinical video is a well-motivated route around the scarcity of calibrated scans. The probe-tip calibration and subject-disjoint tracked evaluation make this a useful ultrasound application of latent-action pretraining and causal video modeling. Its value lies in bringing these components into a setting where a frame reveals only a thin slice of anatomy.

The causal-training experiment provides concrete support for the implementation. On eight-second rollouts, Table 3 reports LPIPS of 0.436 with self-forcing versus 0.515 with teacher forcing, and the model is trained on three-second clips. This is encouraging evidence that streaming generation can extend beyond its training horizon.

Critical weaknesses

Whether latent-action pretraining improves physical control is unresolved. Tables 2 and 11 compare generation with and without codes inferred from target videos, rather than final pose-conditioned models after identical tracked-data training. A control initialized from Wan or action-agnostic ultrasound pretraining is needed to establish the contribution of the central transfer strategy.

The control evaluation also leaves metric pose following uncertain. Table 1 lacks a static-frame prediction baseline or mismatched-action control and reports whole-sequence averages for round trips, obscuring return-to-start error. Expert ratings in Table 5 assess plausibility without an independent geometric reference. Table 6 supplies useful directional evidence but omits trial counts, threshold units and uncertainty. These gaps limit confidence in the model as an environment for policy evaluation.

Questions for authors

Could you compare final pose control after identical EM post-training, starting from Wan, action-agnostic ultrasound pretraining and latent-action pretraining? Please also clarify whether the nine held-out tracked subjects are excluded from the untracked pretraining corpus.

For Table 6, what are the threshold units, trial counts per axis, aggregation rule and uncertainty across subjects? For the Table 1 trajectories, how do matched actions compare with shuffled actions or a frozen initial frame, and what is the error at the round-trip endpoint? These checks would help distinguish useful action sensitivity from dependable metric control.

2 comments

We thank the reviewer for providing constructive and insightful comments. We try our best to address your comments as follows:

1. Latent-action pretraining.

We ran three comparisons: (i) Wan 2.1 without ultrasound pretraining, (ii) ultrasound pretraining without latent actions, and (iii) ultrasound pretraining with latent actions (ours). Causal-inference results are below (SSIM / LPIPS / PSNR).

TaskWan 2.1w/o latent actionsw/ latent actions (Ours)
Original pose0.283 / 0.534 / 16.320.375 / 0.447 / 15.690.386 / 0.439 / 16.09
Zero motion0.204 / 0.596 / 15.530.398 / 0.394 / 17.550.409 / 0.390 / 17.76
Speed×20.357 / 0.484 / 16.860.375 / 0.453 / 15.680.388 / 0.445 / 16.10
Round trip0.240 / 0.578 / 15.710.337 / 0.470 / 14.710.349 / 0.465 / 15.17
Long (8 s)0.214 / 0.609 / 15.270.382 / 0.444 / 15.950.394 / 0.436 / 16.37

The results yield two observations. First, latent-action pretraining consistently improves all three metrics over action-agnostic ultrasound pretraining across all five trajectory settings, confirming that it benefits the final pose-conditioned model. Second, ultrasound-domain pretraining substantially improves SSIM and LPIPS over direct Wan 2.1 initialization across all trajectories, highlighting the importance of adapting the video model to ultrasound appearance and dynamics.

2. Whether the nine held-out tracked subjects are excluded from the untracked pretraining corpus.

Yes. The EM-tracked dataset was collected from a separate cohort, independent of the untracked pretraining corpus, so the nine held-out subjects appear in neither the pretraining corpus nor the post-training set.

3. Table 1: Static-frame baseline, matched actions vs. shuffled actions, and round-trip endpoint error.

We thank the reviewer for suggesting these meaningful baselines. For shuffled actions, each clip is driven by the action sequence of a different clip. For the round trip, we report the whole-sequence average and the return-to-start error.

ConditionPSNR (dB) ↑SSIM ↑LPIPS ↓
Frozen initial frame17.370.4320.371
Matched actions16.090.3860.439
Shuffled actions15.440.3470.478
Round trip (sequence)15.170.3490.465
Round trip (endpoint)14.810.3330.465

Matched actions outperform shuffled actions on all three metrics, confirming that the model uses the provided probe motions for future-frame prediction. The frozen initial frame scores highest because consecutive frames are highly correlated, so copying the first frame incurs modest pixel error, whereas generated frames are anatomically correct under the commanded pose but not pixel-identical to the ground truth. To verify this, we compute thyroid segmentation Dice only within regions that change between consecutive ground-truth frames: matched actions achieve 0.363 versus 0.310 for the frozen frame, confirming that generation tracks motion-induced anatomical changes. For the round trip, the endpoint error is only modestly higher than the sequence average and LPIPS is unchanged, indicating stable generation over the longer rollout with no divergence at the return point.

4. Table 5: User study with geometric reference.

We agree that expert ratings primarily assess perceptual plausibility rather than quantitative geometric accuracy. As complementary evidence, we reconstruct the 3D thyroid from generated ultrasound sequences by segmenting each frame and placing the masks in 3D using the corresponding 6-DoF probe poses (Figs. 1 and 10). The resulting reconstruction exhibits a coherent, anatomically plausible gland structure, providing qualitative evidence of cross-view spatial consistency. We will clarify this point in the revised manuscript.

5. Table 6: Threshold, trial counts, aggregation, and uncertainty.

Thresholds are in mm for translation (xx, yy) and degrees for rotation (rxr_x, ryr_y, rzr_z). Regarding trial counts, we update Table 6 to include the number of trials per axis, reported as success rate (successes/trials). In terms of aggregation, Avg. is the fraction of successful trials among all trials.

Thresholdxxyyrxr_xryr_yrzr_zAvg.
1573.9 (17/23)78.3 (18/23)61.5 (8/13)87.0 (20/23)66.7 (14/21)74.8 (77/103)
2088.9 (8/9)77.8 (7/9)66.7 (2/3)77.8 (7/9)44.4 (8/18)66.7 (32/48)

To quantify uncertainty, we additionally report the per-trial PSNR margin, i.e., the PSNR gain of the target-directed action over the negated action; a positive margin corresponds to a success in Table 6. At threshold 15, the mean margin is +0.72 dB with a 95% CI of [+0.47, +0.97] (one-sided Wilcoxon p = 2.2e-8). At threshold 20, it is +0.64 dB with a 95% CI of [+0.28, +1.01] (p = 6.2e-4). These results provide statistically significant evidence of directional action sensitivity.

Thank you again for the constructive comments. In our response above, we added a comparison of pretraining settings, shuffled action results, the round-trip endpoint error, and the details for Tables 5 and 6. We would greatly appreciate any further questions and will address them promptly.

Reviewer_1L6rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a controllable ultrasound world model for ultrasound video generation, with applications in synthetic data generation and robotic policy evaluation. The paper introduces a three-stage training framework consisting of latent action modeling using a large collection of unlabeled clinical ultrasound videos, probe-pose-conditioned post-training, and distillation for long-horizon video generation. The proposed approach is evaluated using an in-house dataset, with experiments investigating action sensitivity, the effect of latent action pretraining, and the practical utility of the generated videos. The evaluation also includes human assessments of generated video quality and experiments involving robotic navigation policies.

Critical strengths
  1. The paper is very well written. The introduction clearly motivates the problem, and the contributions are well positioned within the existing literature. The three training stages are explained clearly, with appropriate figures illustrating the overall framework.

  2. The adaptation of latent action modeling from DreamDojo to ultrasound imaging is an interesting and innovative direction. The three-stage training approach is well designed, making use of unlabeled ultrasound videos to learn motion dynamics before introducing explicit pose conditioning. The following distillation stage for long-horizon generation is also a useful component for developing a controllable ultrasound world model.

  3. The experimental section is thorough and evaluates several important aspects of a controllable world model. In particular, the action sensitivity experiments are appreciated, as they help establish whether the model responds to the provided control signals rather than relying primarily on learned video priors. The evaluation also goes beyond generation quality by investigating the usefulness of the world model for robotic navigation policies. This provides practical evidence for the proposed approach.

  4. The inclusion of error bars in the experimental results is appreciated, as it helps illustrate the variability of the reported performance and makes the comparisons more informative. I find this particularly important for medical imaging tasks.

Critical weaknesses

The following concerns are not major ones and more of a clarification and open ended question

  1. The use of a large collection of clinical ultrasound videos is an important aspect of the proposed training framework. Making the dataset publicly available would be beneficial for the community, particularly for developing future ultrasound world models. Similarly, releasing the pretrained model weights and training pipeline would improve reproducibility and facilitate further research.

  2. The paper shows the usefulness of probe-pose conditioning for controllable ultrasound video generation. However, it would be interesting to understand whether the generated videos also preserve anatomical consistency under different probe movements and during long-horizon generation. Given that the model is trained on a large collection of clinical videos, it could potentially support other applications, such as synthetic data generation for disease classification, anatomical detection, or segmentation. Although these experiments are not essential for the current work, a discussion of these possibilities and the associated limitations would strengthen the paper.

Questions for authors
  1. Are there plans to publicly release the ultrasound dataset, pretrained model weights, or training pipeline? If the clinical dataset cannot be released due to privacy restrictions, would releasing the model weights and training code be possible?

  2. How well does the model preserve anatomical consistency when generating videos under different probe poses, particularly during long-horizon generation? Have the authors observed any anatomical distortions or inconsistencies?

  3. Could the authors comment on the potential use of the proposed world model for downstream clinical tasks, such as disease classification or anatomical segmentation?

3 comments

Thank you so much for taking the time to review our paper and providing valuable comments. Kindly see our response to your questions below.

Q1: Open-source datasets, model weights, and training code.

We thank the reviewer for this important question. As our model is trained on clinical ultrasound data, releasing both the original data and the generative model weights requires careful consideration of patient privacy and institutional regulations, including extensive IRB review and approval procedures. We are currently actively working on addressing the administrative issues and pursuing the possible release options. As a more feasible near-term step, we are exploring the possibility of releasing a subset of the test data, subject to the required approvals. In the meantime, we will provide more detailed descriptions of the training configurations, model architecture, action-conditioning mechanism, and evaluation protocols to facilitate independent reproduction of the proposed methodology.

Q2: Anatomical consistency over long horizons.

We thank the reviewer for this important question. We interpret anatomical consistency as follows: over long horizons, the depicted anatomy should not drift or morph significantly as the rollout proceeds.

The user study (Table 5) directly scores probe–anatomy alignment over long autoregressive rollouts (8 seconds), which measures whether the generated frame is consistent with where the probe is in the anatomy. Probe–anatomy alignment received the highest scores of the three criteria (3.3–4.4 on a 5-point scale), indicating that the generated views remain anatomically plausible across different probe movements. We reconstruct the 3D thyroid from the generated sequences by segmenting each generated frame and placing it in 3D according to the pose (Figs. 1 and 10). We observe that the reconstructed gland forms an anatomically plausible 3D structure.

However, we do observe distortions in some cases, typically under large out-of-plane motions or far from the initial view, which highlights the remaining challenge of long-horizon consistency. We expect that expanding the EM-tracked training data to more diverse anatomy and trajectories would improve robustness.

Q3: Potential downstream tasks.

We thank the reviewer for this insightful suggestion. Beyond robotic ultrasound, SonoWorld has the potential to serve as a controllable ultrasound data generation engine for downstream clinical tasks, including anatomical segmentation, detection, and disease classification. For segmentation and detection, SonoWorld can take a single annotated frame and generate the neighboring views seen as the probe moves, expanding the training distribution with realistic viewpoint variation while reducing the need for additional data acquisition. For disease classification, the pretrained video DiT learns ultrasound appearance and dynamics from ~6.9M frames spanning multiple organs (abdomen, heart, and thyroid), and its features could serve as initialization or a frozen backbone for downstream models.

One consideration is that these tasks require task-specific supervision. While SonoWorld generates ultrasound images, supervised tasks such as segmentation and classification additionally require anatomical annotations or diagnostic labels, which could be obtained through manual annotation or pseudo-labeling. Generating pathology-specific samples would further require conditioning on diagnostic labels and clinical validation. We view the integration of such task-specific supervision with SonoWorld as a promising direction for future work.

Reviewer_1L6rOpenReview ↗

Official comment by reviewer 1L6r

I thank the authors for the rebuttal which provides clear explanation on the raised concerns. The authors have addressed my concerns and I will stay with my rating of the paper.

Few follow up questions:

  1. It feels a bit disappointing to hear about the open source constrains that lie forward for this work. It is understandable that privacy concerns are very serious and indeed needs time before data or model can be released to the public. However, in my experience, releasing either a small subset of the test data or even clear explanation of the architecture has not helped the community in reproducing a work. Even with the details about the architecture, the model cannot be trained and compared fairly against a follow up as the data is lacking. I will leave this decision to the AC,SAC to consider when they discuss this work.

  2. I agree with the useful of downstream tasks. The authors could indeed perform such downstream tasks and release those models, which could be beneficial for the medical community to use this work. Just as a note, I think reproducible/open-source weights would be the best for researchers to build on this work.

We sincerely thank the reviewer for confirming that our response addressed the concerns. We deeply understand and agree that open-sourcing datasets and model weights would faciliate comparison and improve reproducibility. We will do our utmost to obtain the required approvals and make them available to the public.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive comments. We are encouraged that they find the paper well written, the three-stage training well designed, learning from untracked clinical video well motivated, and the experiments thorough. We summarize our responses below; details are in the individual replies.

Additional experiments

  1. Latent-action pretraining. A three-way comparison (Wan 2.1, ultrasound pretraining without latent actions, with latent actions) shows latent-action pretraining improves all metrics under all trajectories, and ultrasound pretraining substantially outperforms Wan 2.1 initialization.
  2. Table 1 baselines. We add a frozen-initial-frame baseline, shuffled actions, and the round-trip endpoint error. Matched actions outperform shuffled actions; within regions that change between consecutive ground-truth frames, segmentation Dice is 0.363 for matched actions versus 0.310 for the frozen frame.
  3. Long-horizon generation. Rollouts extended to 18 s (6× the training clip) remain stable with our selective KV-cache eviction (PSNR −0.17 dB, SSIM unchanged), while a local-window cache degrades steadily.
  4. Table 6 details. We report threshold units, per-axis trial counts, aggregation, and the per-trial PSNR margin with 95% CIs, which is significantly positive at both thresholds.

Clarifications

  • We position SonoWorld relative to SonoGym and articulate its distinct contributions.
  • We confirm subject-level separation between the pretraining corpus and the EM-tracked data.
  • We explain the mechanism behind generalization to unseen action trajectories.
  • We discuss anatomical consistency under probe motion and over long horizons, supported by 3D reconstruction from generated sequences.
  • We discuss potential downstream clinical applications and policy training with SonoWorld, along with their limitations.
  • We describe our plans for data and model release subject to institutional approval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.