acceptodds
Under review as a conference paper at ICLR 2027

UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

Abstract

World models can support autonomous ultrasound scanning by predicting probe-motion outcomes from local observations. Learning this action–observation relationship typically requires synchronized video–pose pairs, which are costly to collect at scale and generally absent from routine clinical recordings. World models must also account for ultrasound's cross-sectional sampling geometry to follow scanning actions reliably. We present UltraWorld, a self-distillation recipe that transfers clinical video priors into interactive ultrasound world models without real action annotations. Using clinical videos, we adapt a video foundation model into an ultrasound generator conditioned on reference images and masks. The resulting world model predicts from local observations and actions without masks or assets at inference. Our Acoustic Sampling Map (AsMap) encodes probe poses and imaging settings as pixel-wise 3D sampling positions, beam directions, and depths. Experiments show improved prediction fidelity and action following. Across nine simulated closed-loop local planning episodes, UltraWorld reduces mean final distance to the goal and orientation error by 29% and 38%, respectively, compared with visual servoing. Code: https://github.com/sephohi/UltraWorld.

Primary area:
generative models
Keywords:
Ultrasound World Model, Pose Encoding, Spatial Grounding, Geometry Map, Self-distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MaXnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a recipe for training an interactive ultrasound world model with a large-scale clinical dataset without pose data. Specifically, a semantic mask-conditioned US video generation model is first pretrained using the large-scale untracked US segmentation dataset. Then a pose-conditioned world model is distilled from the video generator based on synthetic pose-to-semantic mask maps simulated from segmented MR images. The main novelty is utilizing the large-scale untracked US data to train a pose-conditioned world model through self-distillation. This has practical impact, as collecting tracked US data with synchronized poses is often tedious and time-consuming.

Critical strengths
  1. The paper is well-organized and easy to follow, with sufficient visual demonstrations
  2. The proposed method is novel and practical, removing the practical burden of collecting tracked ultrasound data
  3. The experimental validation is solid, with sufficient quantitative and qualitative results, as well as end-to-end robotic ultrasound planning evaluation.
Critical weaknesses
  1. Specific training and implementation details require further clarification
  • How many clinical ultrasound image frames and MRIs in total are used for training?
  • How long does generating each frame take? Can the world model achieve real-time inference for robotic applications?
  • What is the per-step motion scale?
  1. The robotic ultrasound experiment initializes the probe already close to the target (16mm). In practice, the probe is often initialized further from the target, but it is unclear whether the proposed world model still works well. The effectiveness of the planning is also constrained by the limited prediction horizon length of the world model.
Questions for authors
  1. See weakness 1 for questions on implementation details
  2. The target application of the proposed world model is visual servoing. Then what are the benefits of the proposed pipeline compared to training a world model in the mask/anatomy space? Specifically, an anatomy world model can be trained to predict the segmentation masks given the motion sequence, while initial and goal US frames can both be converted to segmentation masks with a US segmentation model. Such an anatomic world model can be trained with various segmented 3D CT/MR datasets. And the segmentation model can be trained with the clinical US dataset like CAMUS. This alternative approach might be more reasonable in practice, as the goal positions for robotic ultrasound are often specified as a 2D plane on patient anatomy instead of a US image.
  3. In the proposed world model, ultrasound generation is conditioned purely on the motion. But the real ultrasound images also depend on anatomical geometry. Do you observe variance in generated US images by your world model given the same motion condition?
  4. How diverse are the US pose trajectories in the unlabeled dataset for prior learning? Does it have to cover all possible motion trajectories that the world model needs to simulate?
1 comment

We sincerely thank the reviewer for the positive assessment of our work and the constructive suggestions. We address the concerns below.

[W1, Q1, Q4] Training and Implementation Details

To address these questions, we have added detailed implementation information in the revised manuscript (Appendices E.3 and F.6), including:

  • Training data: 40,471 clinical ultrasound training frames (46,824 in total) and 67 training MRI/CT volumes (78 in total).
  • Inference speed: Generating a 17-frame sequence takes 2.408 seconds on an NVIDIA H200 GPU. Future work will explore few-step and autoregressive distillation toward real-time inference.
  • Motion scale: Median per-step translations are 0.489, 0.376, and 0.677 mm, with rotations of 0.451°, 0.388°, and 0.116°, for breast, neck, and lower limb, respectively.
  • Trajectory diversity: Since clinical videos are untracked, their motion distributions cannot be directly measured. Exhaustive coverage of all possible trajectories is not required in the prior-learning stage, which focuses on learning ultrasound appearance and temporal priors. Instead, we construct action supervision using programmable trajectories informed by clinical scanning practices, including sweeps, slides, rocking, fanning, revisits, and their combinations. These provide representative action patterns for world-model training, although generalization to arbitrary unseen trajectories is not guaranteed.

[W2] Robotic Planning with Larger Initial Distances

The appropriate scanning range varies across anatomical domains. We have expanded the evaluation to 24 simulated tasks across three domains, including lower-limb tasks with an average initial distance of approximately 43 mm, substantially larger than the previously evaluated 16 mm. Closed-loop replanning enables the model to approach goals beyond a single prediction horizon. For larger-scale navigation, coarse localization using external RGB cameras could also be combined with our world model for fine-grained ultrasound guidance.

[W2] Limited Prediction Horizon

We have additionally tested 49-frame long-horizon predictions. We encourage the reviewer to watch the ▶ Long-Horizon Prediction Videos, which illustrate the model's ability to maintain coherent anatomical evolution over extended trajectories.

Future work could explore integrating 3D anatomical reconstruction with world modeling to support longer-horizon prediction.

[Q2] Why Not an Anatomy World Model?

We agree that an anatomy-only world model is a promising and potentially more lightweight solution for geometry-based planning. However, ultrasound images contain richer information than segmentation masks, including tissue appearance and acoustic characteristics. Our image-space world model aims to support broader downstream applications, such as image-quality assessment, image-based guidance, and policy learning, beyond anatomical alignment alone.

[Q3] Variability Under Identical Motion Conditions

Our world model is conditioned on both the initial ultrasound observation and the prescribed motion, rather than motion alone. Therefore, the same motion applied to different initial observations can produce different anatomical and appearance predictions, reflecting the underlying imaging context.

Reviewer_ccBVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UltraWorld, a controllable ultrasound world model that learns action-conditioned video prediction without requiring tracked clinical ultrasoundvideos. The proposed framework first trains a mask-conditioned ultrasound video generator using untracked clinical videos. Anatomical masks obtained from 3D MRI volumes are then sampled with predefined probe trajectories to generate synthetic action-video pairs. The paper further introduces a mapping, which encodes probe poses and imaging configurations using pixel-wise spatial information to improve action conditioning. Experiments are conducted to evaluate generation quality, action following, and generalization to unseen clinical observations. The paper also shows the usefulness of the model for closed-loop robotic ultrasound planning in simulation.

Critical strengths
  1. The paper addresses an important problem in ultrasound world modeling. Collecting synchronized ultrasound videos and corresponding probe poses is a tedious task. The idea of using untracked clinical videos to build controllable world models is interesting and could help address the limited availability of action-conditioned medical datasets.

  2. The proposed AsMap is an interesting contribution. Unlike conventional pose-based conditioning, it represents the physical sampling positions, beam directions, and depths associated with ultrasound acquisition. This is particularly relevant for ultrasound imaging, where changes in probe position and imaging depth affect the observed anatomical structures. Explicitly incorporating these factors into the conditioning representation is well motivated.

  3. The approach of learning ultrasound appearance from clinical videos and obtaining action supervision independently from 3D anatomical structures is appreciated. The use of synthetic action-video pairs provides a reasonable way to introduce controllability without requiring real probe trajectories during training. This could be useful for scaling ultrasound world models using existing clinical datasets.

  4. The paper is well written, with figures clearly explaining the proposed training pipeline, conditioning representation, and experimental results. The appendix also provides useful implementation details and additional qualitative evaluations.

Critical weaknesses
  1. Reliance on synthetic data for evaluating action following. One of the main limitations is that the proposed world model is trained and primarily evaluated using synthetic action-video pairs generated by the mask-conditioned video generator. Although these pairs provide known trajectories, they do not necessarily represent the true visual changes observed during real ultrasound scanning. The evaluation therefore shows how well the world model reproduces the synthetic trajectories, rather than directlyshowing the accuracy of its predictions under real probe movements.

  2. Limited validation on real clinical ultrasound. The generalization experiments on BUSI and BUV are appreciated. However, the zero-action and round-trip evaluations primarily measure observation stability and recovery consistency. A model could perform well on these metrics without accurately predicting the anatomical changes caused by a prescribed probe motion. While the difficulty of obtaining tracked clinical recordings is understandable, even a small evaluation using real ultrasound videos with known probe movements would provide stronger evidence for the proposed method. The current results do not fully show whether the learned action control transfers to real ultrasound settings.

  3. Limited robotic planning evaluation. The application to robotic ultrasound planning is interesting, but the evaluation is limited to nine simulated episodes. Furthermore, the comparison is conducted against an intensity-based visual servoing approach rather than other world-model-based planning methods. Since the proposed method uses model predictive control with CEM, it is unclear how much of the improvement comes from the world model itself and how much comes from the planning strategy. A comparison using the same planner with alternative conditioning representations, such as Fourier or MLP conditioning, would help isolate the contribution of AsMap.

  4. Unclear contribution of self-distillation and the different training stages. The paper proposes self-distillation as a central component for transferring ultrasound priors into an action-conditioned world model. However, Table 2 shows that initialization from the original Wan2.1 checkpoint achieves performance relatively close to the full model when trained on the same synthetic trajectories. This suggests that the generated action-video supervision may be more important than inheriting the ultrasound generator weights. The current ablation isolates the initialization strategy but does not fully show the individual contributions of the prior learning and synthetic supervision stages. A more detailed discussion or ablation would help clarify which components are essential for the reported improvements.

  5. Anatomical consistency and limitations of synthetic supervision. The proposed framework generates action-conditioned training data using anatomical masks obtained from 3D MRI volumes. However, the generated ultrasound appearance is learned independently from clinical videos. It is unclear how well this process preserves the relationship between anatomical structure, probe motion, and ultrasound-specific appearance changes. For example, changes in tissue scattering, attenuation, and acoustic artifacts may not be accurately represented by mask conditioning alone. This is particularly important when generating previously unseen anatomical cross-sections. The paper should clarify how these limitations affect the reliability of the generated trajectories and whether the synthetic videos were assessed for realistic acquisition behaviour.

  6. Generalization across anatomical domains. The video generator is trained using breast ultrasound and echocardiography datasets, while the action-conditioned trajectories are constructed using breast MRI volumes from I-SPY1. The real-clinical evaluation also focuses on breast ultrasound datasets. It is therefore unclear whether the resulting world model can generalize to other anatomical domains or probe configurations. Since the proposed framework is motivated as a scalable approach for ultrasound world modeling, the paper should clarify the extent to which the current method is specific to breast imaging and what would be required to extend it to other applications.

Minor comments (not necessary for the final rating)

  1. Table 1 shows clear improvements in prediction fidelity, but the improvements in motion error and delta-LPIPS are relatively smaller. A discussion of whether these improvements are practically meaningful for robotic scanning would be helpful.
  2. The AsMap representation contains sampling position, acoustic beam direction, and depth information. An ablation of these individual components could provide additional insight into which factors are most important for action following.
  3. The paper should clarify the quality of the segmentation masks obtained from clinical ultrasound videos during the initial training stage. Errors in these masks could affect both the learned video prior and the synthetic action-video supervision.
  4. It would be beneficial to report the limitations in the main section of the paper itself, as it a part of the main work and not an extra result to be added to the appendix.
Questions for authors
  1. How do the authors validate that the synthetic action-video trajectories reflect the actual changes observed during real ultrasound probe movements? Could the authors clarify whether any comparison with tracked real ultrasound acquisitions was performed?

  2. The real-clinical experiments evaluate zero-action and round-trip consistency. How do the authors distinguish a model that accurately follows probe actions from one that preserves the initial anatomy but does not respond correctly to intermediate actions?

  3. Could the authors evaluate robotic planning using the same MPC and CEM setup with alternative conditioning representations? This would help determine whether the improved planning results are primarily due to AsMap rather than the planning strategy.

  4. How is anatomical consistency maintained when the generated ultrasound appearance is obtained from a learned video prior while the acquisition trajectories are defined using MRI-derived masks? Have the authors evaluated whether this introduces unrealistic anatomical or acoustic changes during probe motion?

  5. Could the authors comment on the computational requirements of the proposed world model during robotic planning? Since the planner evaluates multiple candidate trajectories using a diffusion-based video model, the inference time per planning step would be useful for assessing its practicality in robotic ultrasound systems.

6 comments

We sincerely thank the reviewer for the thorough evaluation and constructive suggestions. We appreciate the concerns regarding real-world action following, robotic planning, and the role of self-distillation. Following these suggestions, we have substantially strengthened the experimental evaluation, added new analyses, and clarified the scope and limitations of our method.

[W1, Q1] Zero-Shot Evaluation on Real Tracked Ultrasound

We built a robotic ultrasound acquisition system and collected synchronized ultrasound–pose data from breast phantom and in-vivo neck scans involving three volunteers (Appendix E.2). Together with a public lower-limb dataset, we conducted zero-shot evaluation on 3,920 real tracked ultrasound frames across three anatomical domains (Sec. 5.4). Table 3 demonstrates UltraWorld's strong action-following performance on real probe trajectories, without training or fine-tuning on real tracked action–video pairs.

Table 3. Zero-shot transfer evaluation on real tracked ultrasound.

DomainMethodPSNR ↑LPIPS ↓Latent ℓ₂ ↓Δ-LPIPS ↓Motion Err. ↓MDA ↑
Breast phantomWan2.117.3700.3630.2990.60937.47265.630
AsMap (Ours)21.9250.2250.1290.5645.925100.000
NeckWan2.118.1680.4240.3150.67865.23327.040
AsMap (Ours)26.5010.2070.0970.5348.81578.480
Lower limbWan2.120.1720.2890.2040.61814.81356.090
AsMap (Ours)22.2790.2980.0990.4778.92476.110

[W2, Q2] Real-World Action Following and Counterfactual Predictions

We further validate whether UltraWorld genuinely follows prescribed probe actions through two complementary evaluations:

(1) Real-world action following. On real tracked ultrasound, we condition the model on the initial observation and recorded probe trajectory, then compare its predictions against the corresponding acquired frames (Sec. 5.4, Table 3). This directly evaluates whether the predicted anatomical motion agrees with actual probe movements.

(2) Counterfactual action prediction. In Fig. 15, we apply different prescribed actions to the same initial observation and visualize the resulting predictions. The model generates distinct, action-dependent anatomical changes, demonstrating that its predictions respond to the specified actions rather than merely preserving or extrapolating the initial observation.

We encourage the reviewer to watch the ▶ Real-Scan Prediction Videos for qualitative comparisons and counterfactual predictions.

[W3, Q3, Q5] Robotic Ultrasound Planning

We have expanded the robotic planning evaluation from 9 to 24 simulated tasks across three anatomical domains.

To isolate the contribution of AsMap from the planning strategy, we additionally compare Fourier, MLP, and PRoPE conditioning under the same closed-loop planning framework, alongside an online Jacobian-based visual-servoing baseline. UltraWorld achieves the lowest overall mean final position error of 4.84 mm, compared with 9.15 mm for the strongest baseline, representing a 47.1% reduction (Fig. 9).

Regarding computational efficiency, generating a 17-frame sequence requires 2.408 seconds using 25 Euler sampling steps on an NVIDIA H200 GPU. Future work will explore few-step and autoregressive distillation toward real-time inference.

In the revised evaluation, we adopt a greedy closed-loop planner instead of CEM. Candidate trajectories are evaluated by predicted anatomical similarity to the goal, followed by execution and replanning. This simpler and shared planning protocol helps isolate the effect of action conditioning from planner-specific optimization.

[W4] Role of Self-Distillation and Training Stages

Table 2 isolates the effect of weight initialization under identical synthetic action–video supervision. Initializing from Wan2.1 achieves nearly identical performance to generator-weight initialization, slightly outperforming it on four of six metrics. This suggests that synthetic supervision and the training recipe are more consequential than inherited generator weights, consistent with the findings of Causal Forcing (Zhu et al., 2026).

Importantly, both initialization variants learn from trajectories synthesized by the ultrasound-adapted generator. Thus, the learned ultrasound prior remains central to constructing action–video supervision, regardless of student initialization. The two-stage adaptation first aligns the AsMap encoder with the frozen backbone, then jointly fine-tunes both (Sec. 4.3).

References

Zhu, H., Zhao, M., He, G., Su, H., Li, C., and Zhu, J. (2026). Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. arXiv preprint arXiv:2602.02214.

[W5, Q4] Anatomical Consistency and Limitations of Synthetic Supervision

Anatomical consistency is maintained by sampling mask trajectories from the same 3D anatomical volume under prescribed probe motions. We further assess whether the generated ultrasound videos follow these anatomical constraints. Table 7reports an overall Dice of 0.7819 between generated segmentations and conditioning masks, and 0.8091 between generated and reference segmentations, demonstrating anatomical controllability. Generation fidelity is additionally evaluated in Table 6

Our motivation is that clinical ultrasound videos implicitly capture correlations between anatomical changes and ultrasound appearance, including effects associated with scattering, attenuation, and acoustic artifacts. Although these effects are not separately supervised, we expect the learned ultrasound prior to capture aspects of these coupled relationships and transfer them to the synthesized trajectories. The real tracked evaluation (Table 3) provides complementary evidence of the resulting model's action-following capability on real acquisitions.

[W6] Generalization Across Anatomical Domains

We have expanded the framework and evaluation beyond breast imaging to investigate its applicability across different anatomical domains.

Specifically, the ultrasound video prior is learned from four domains: breast, cardiac, neck, and lower limb, using 46,824 clinical ultrasound frames. For synthetic action supervision, we use 78 anatomical MRI/CT volumes across breast, neck, and lower-limb anatomy, rather than relying only on breast MRI.

The revised experiments demonstrate consistent improvements across all three action-supervised domains in synthetic prediction (Table 1), real tracked ultrasound evaluation (Table 3), and simulated robotic planning (Sec. 5.5). We acknowledge that generalization to entirely unseen anatomical domains and different probe configurations remains an open challenge and may require additional domain-specific visual priors, anatomical assets, and acquisition calibration.

[W4] Role of Self-Distillation and Training Stages

We have added Appendix B: Recipe-Guided Self-Distillation: Formulation and Analysis to clarify the self-distillation mechanism and the contributions of different training stages. Specifically, the ultrasound-adapted teacher combines learned ultrasound priors with anatomy-guided trajectories to generate synthetic action–video supervision. The student then learns observation- and action-conditioned prediction from these trajectories through flow matching, without requiring anatomical masks or 3D assets at inference.

Table 2 isolates the effect of weight initialization under identical synthetic supervision. Initializing from Wan2.1 achieves nearly identical performance to generator-weight initialization, slightly outperforming it on four of six metrics. Consistent with the initialization analysis in Causal Forcing (Zhu et al., 2026, Appendix C.3 and Figure 9), these results suggest that recipe-generated synthetic supervision plays a more critical role in knowledge transfer than direct teacher-weight inheritance.

Importantly, both initialization variants learn from trajectories generated by the ultrasound-adapted teacher. Thus, prior learning remains essential to the proposed data-generation pipeline, even when its weights are not directly inherited. Appendix B further formalizes this data-mediated knowledge transfer and clarifies the complementary roles of prior learning, trajectory synthesis, and student adaptation, including the encoder-only and joint fine-tuning phases (Sec. 4.3).

References

Zhu, H., Zhao, M., He, G., Su, H., Li, C., and Zhu, J. (2026). Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. arXiv preprint arXiv:2602.02214.

Reviewer_DgPnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper decomposes ultrasound world model generation into 2 steps. First, they train an ultrasound-video generation model that takes as input a reference image and a sequence of desired anatomic segmentation images; their model then generates a sequence of ultrasound images that represent the initial image "deforming" over time to match the provided segmentations. Second, they use simulated ultrasound trajectories to train their model to predict future ultrasound images conditioned on an action sequence. The first part is fairly well validated for breast and cardiac ultrasound. The second part uses human-segmented 3D MRI volumes of breast tissue as a basis for sampling cross-sectional tomographic anatomic slices from a sequence of probe trajectories and corresponding ultrasound prameters; they then simulate ultrasound from these 2D tomographic slices of anatomical label maps. The key deficiencies are their dependence on per-pixel human annotations of 3D breast MRIs and their subsequent focus on breast-ultrasound validation of their fully trained world model. They have not demonstrated a general ability to learn ultrasound world models. They have only demonstrated the ability to learn a breast ultrasound world model and an ability to learn an anatomic-map conditioned cardiac ultrasound generation model.

Critical strengths

They have 2 strong contributions: (1) decoupling world model generation into self-supervised anatomic-conditioned generation and then learning action conditioning based on simulations derived from human-labled MRI volumes. (2) Their AsMap encodes more ultrasound pose and acquisition parameters into each pixel than the representations used by prior ultrasound world models.

Critical weaknesses

Training and validation are limited to cardiac and breast ultrasound, and the final world model is only validated on breast ultrasound. This is a huge limitation for the claims of the paper. Second, they critically depend on precisely annotated 3D MRI volumes for each anatomic region of interest, so they still require substantial data (even if they avoid the need for precise ultrasound tracking); it is doubtful their current method could truly scale in the way expected of a self-supervised approach. Third, as far as I can tell they NEVER validated their method against any REAL data from tracked/robotic clinical ultrasound.

Questions for authors

This paper would be much stronger with validation against real tracked ultrasound data, even if only for 2 or 3 human subjects. Have you performed any such testing? If so, even limited anecdotal real-world evidence is much better than no real-world evidence. I'd love to see you add this, either to the main paper or to the Appendix. Also, please soften your claims. I suggest changing the tile to something like "Learning an Interactive Breast-Ultrasound World Model from Untracked Clinical Videos and Non-Corresponding Human-Labeled MRI Volumes."

Update: Thank you for addressing many of my concerns. I've increased my rating for this paper.

5 comments

We thank the reviewer for the comments and suggestions. In the revised manuscript, we have added real tracked-ultrasound evaluation, expanded action-conditioned validation to multiple anatomical domains, and clarified the use of 3D anatomical supervision.

[W3, Q1] Validation on Real Tracked Ultrasound

Following the reviewer's suggestion, we have added zero-shot evaluation on 3,920 real tracked ultrasound frames: 60 in-vivo neck scans from three volunteers, 14 breast phantom scans, and 13 lower-limb scans (Sec. 5.4). The model predicts future frames from a real initial observation and its recorded probe trajectory, without training or fine-tuning on tracked ultrasound–pose pairs. The results directly assess agreement with the corresponding acquired frames.

Table 3. Zero-shot transfer evaluation on real tracked ultrasound. Motion Error is in pixels; MDA is in percent.

DomainMethodPSNR ↑LPIPS ↓Latent L2 ↓Δ-LPIPS ↓Motion Err. ↓MDA ↑
Breast phantomWan2.117.3700.3630.2990.60937.47265.630
AsMap (Ours)21.9250.2250.1290.5645.925100.000
NeckWan2.118.1680.4240.3150.67865.23327.040
AsMap (Ours)26.5010.2070.0970.5348.81578.480
Lower limbWan2.120.1720.2890.2040.61814.81356.090
AsMap (Ours)22.2790.2980.0990.4778.92476.110

We encourage the reviewer to watch the following videos for direct comparisons with recorded ground truth:

▶ WATCH REAL-SCAN PREDICTION VIDEOS

[W1] Generalization Across Anatomical Domains

We have expanded the experiments beyond breast ultrasound. Action-conditioned world models are trained and evaluated on breast, neck, and lower limb, using breast/neck MRI and lower-limb CT anatomy. Across all three evaluated domains, AsMap improves action following over alternative conditioning methods:

Table 1. Comparison of action-conditioning representations across anatomical domains. Motion Error is measured in pixels, and MDA in percentage.

DomainConditioningPSNR ↑LPIPS ↓Latent ℓ2 ↓Δ-LPIPS ↓Motion Err. ↓MDA ↑
BreastFourier Enc.22.7420.2850.1550.51014.08577.97
MLP Enc.21.6740.2990.1780.51415.47474.16
PRoPE22.1350.2900.1790.53215.05874.41
AsMap (Ours)23.7590.2640.1290.47910.82984.33
NeckFourier Enc.27.3110.1640.0780.53914.23874.27
MLP Enc.26.1180.1710.0920.54613.80769.91
PRoPE25.9140.1780.0970.55715.59865.92
AsMap (Ours)27.9700.1510.0690.50411.10780.56
Lower limbFourier Enc.23.7560.2180.0930.48913.10278.51
MLP Enc.23.1730.2150.1030.48112.12874.57
PRoPE23.1330.2220.1110.50712.94472.49
AsMap (Ours)24.3660.2060.0850.47310.52381.42

[W2] Scalability of 3D Anatomical Supervision

Each annotated 3D volume can be reused to generate many programmable probe trajectories; no paired MRI–ultrasound acquisitions or dedicated tracked-ultrasound collection are required. MRI/CT scans are routinely acquired and archived in clinical care, so existing anatomical datasets can be repurposed without adding a new acquisition step to the ultrasound workflow. Semi-automatic segmentation tools can also assist annotation, although anatomical labels remain necessary.

Our action supervision uses 67 training MRI/CT volumes across three domains (78 volumes including held-out evaluation cases):

3D anatomical sourceTraining volumesHeld-out volumes
Breast MRI (I-SPY1)305
Neck MRI (SegThy)122
Lower-limb CT (VSD)254
Total6711

These assets are independent of the 40,471 untracked clinical ultrasound training frames, so the approach avoids requiring case-by-case MRI–ultrasound correspondence or probe tracking.

[Q2]: Scope of Claims and Paper Title

We appreciate the suggestion to align the title with the demonstrated scope. However, the revised action-conditioned evaluations now span breast, neck, and lower-limb ultrasound, including real tracked acquisitions. A breast-specific title would therefore no longer reflect the experimental evidence. We have clarified the evaluated domains and the remaining requirements for 3D anatomical supervision in the revised manuscript.

Reviewer_DgPnOpenReview ↗

Vastly Improved Experiments and Validation

Thank you for contributing the neck and limb validations with real tracked ultrasound of real anatomy. These new results vastly improve the paper and substantiate the claims. I'd still like to see a clarifier about MRI in the title, e.g. "UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map and a Small Set of Human-Labeled MRI Volumes"

Thank you for your positive feedback and valuable suggestion. We agree that explicitly clarifying the use of annotated 3D medical data in the title would help readers better understand our training setting. Following your suggestion, we have revised the title to:

UltraWorld: Learning Interactive Ultrasound World Models from Untracked Videos and Unpaired Labled 3D Volumes via Acoustic Sampling Maps

We believe the revised title more accurately reflects the training data and setting of our work. Thank you again for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.