acceptodds
Under review as a conference paper at ICLR 2027

CrossWorld: Learning Contact-Rich Robotic Ultrasound Policies from Complementary Partial Worlds

Abstract

Learning robotic ultrasound policies typically relies on complete real-world expert demonstrations that jointly capture spatial probe motion, contact force, and ultrasound observations, making data collection costly and difficult to scale. Simulation offers extensive and efficient spatial exploration but lacks faithful contact-dependent dynamics that are essential for robotic ultrasound scanning. Conversely, such dynamics can be captured through local force-controlled real interactions without requiring complete expert demonstrations. Since neither domain alone provides the full experience required for policy learning, this raises a natural question: can two complementary, low-cost partial worlds be composed to reduce the need for costly complete real-world demonstrations? We present CrossWorld, a cross-domain world-modeling framework that learns from a real contact world and a simulated spatial world through cross-world factor completion. Specifically, contact completion transfers appearance dynamics learned from local real interactions to complement simulated observations, while spatial completion transfers action-conditioned spatial dynamics learned in simulation to complement sparse real observations. The policy then jointly perceives each factual B-mode observation and its cross-world completed view through a paired co-observation interface, enabling effective policy learning with a limited number of complete real-world expert demonstrations. On L4 standard-plane navigation, CrossWorld with OpenVLA achieves 70.0% cross-subject success in simulation, compared with 67.5% for simulation-only training. With local contact interactions, 50 phantom expert demonstrations, and a fixed set of 60 human demonstrations, CrossWorld achieves 75.0% success on phantom scanning, compared with 25.0% for direct mixing of simulated and expert demonstrations. These results demonstrate that composing complementary partial worlds can substantially reduce reliance on costly real-world expert demonstrations for robotic ultrasound policy learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic ultrasound, sim-to-real transfer

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_UWMNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets robotic ultrasound navigation to the longitudinal L4 standard plane in a setting where complete real expert demonstrations (probe motion + force + B-mode) are expensive. The central assumption is that the dynamics behind image formation split into a spatial part (how probe motion changes the pose relative to anatomy) and a contact part (coupling and deformation as a function of force), and that each part can be learned from a cheaper, partial source. A spatial world model (ViT estimator + transition + DiT generator) is trained in SonoGym with a privileged 30-d spatial descriptor. A contact world model is trained on 512 local force-controlled phantom interactions. During policy training, the contact model generates a "contact-completed" view for every simulated frame, and the spatial model generates a "spatially-completed" view for every real frame from a K=10 history. The policy (ACT, Diffusion Policy, or OpenVLA) always gets the factual frame, the completed view, and a 9-d robot state.

Results: in simulation, success rises by 7.5-10 pp within- and cross-subject for all three backbones (160/40 rollouts). On the real robot with 50 phantom and 60 human demos, CrossWorld reaches 65-75% phantom success (N=20) against at most 25% for direct mixing, and 47-60% on within-participant human trials (N=15, the same 3 participants who supplied demonstrations). The appendix has ablations on completion direction, contact-control strategy, context length K, measured contact conditioning, view dropout, a contact-data pretraining control, and three training seeds for OpenVLA in simulation.

Critical strengths
  1. The problem framing is well motivated, and the real-robot study is more careful than most in this area. Splitting supervision so that simulation provides spatial structure and cheap local real interaction provides contact appearance suits ultrasound, where contact-dependent appearance is the hard part to simulate.

    • Real-robot success is judged by two clinicians blinded to the policy (kappa = 0.82 before consensus), and evaluation order is randomized in blocks.
    • The contact-data pretraining control (Table 13: the same D_contact used for image-to-wrench pretraining only gives 20 -> 25%) and the contact-control strategy comparison (Table 10) were especially useful to me.
    • This matters for my assessment because it shows that simply having D_contact doesn't explain the gain.
  2. The real-robot effect at the 50-demonstration budget is large and holds across three quite different backbones. If the mechanism holds up, this is a practically meaningful result for a domain where data really is expensive.

    • Table 2: 65/70/75% vs. 20/25/25% for direct mixing on the phantom.
    • Even with N=20, a gap of 15/20 vs. 5/20 is unlikely to be noise.
Critical weaknesses
  1. The completed view mixes "cross-world factor completion" with two simpler signals, and none of the controls separate them. Since the paper's contribution is the factor-completion mechanism, not just the final success rate, I see this as the main issue.

    (a) Temporal context. CrossWorld policies get frame history that the factual baselines never see.

    • The contact-completed view is G_c(o_{t-1}, a_{t-1}, c~_t) (Eq. 9), a learned function of the previous frame and action. The spatially-completed view at deployment is generated from a K=10 history (Eq. 10). The factual-only baselines get only o_t and r_t.
    • The context-length ablation (Fig. 10: 45% at K=1, 75% at K=10, 80% at K=16) fits the idea that much of the real-robot gain comes from history.
    • The existing controls don't address this: the augmented view in Table 8 is T(o_t), current frame only, and the independent-view variant (Fig. 5, B.1) uses only o_t at deployment. A factual + frame-stack baseline (o_{t-K..t} and past actions, same backbone and slot interface) is the obvious missing control.
    • This also explains a result I otherwise found odd. In simulation the contact condition c~ is sampled from the real bank (Eq. 9; A.2 describes this for training, and I assume evaluation does the same), so it's unclear what contact information the C-view adds at sim test time. Yet it gives +7.5-10 pp in sim, including cross-subject.

    (b) Domain-style alignment. At deployment the S slot effectively receives a learned real-to-sim translation of the current scene. That's a valid and useful trick, but it's a different claim from "simulation-derived spatial dynamics complement real observations."

    • In training, the S slot always holds simulator-style images (factual sim frames) and the C slot holds real-style images (G_c output, trained on real phantom images). At deployment, the S slot holds G_s output, a generator trained only on sim frames, and the C slot holds the real frame.
    • The paper should compare against a plain image-translation baseline (e.g., an unpaired real-to-sim translator feeding the same slot), and show the S-view next to sim frames.
    • Relatedly, the spatial estimator E_s is trained only on sim images and applied to real ones, which is exactly the sim-to-real gap the introduction calls hard. There's no check that s^_t on real data is meaningful. I understand there's no ground-truth registration, but even a qualitative or phantom-fiducial check would help.
  2. The statistical support is thin for several claims, and the cost argument doesn't quite add up. The headline phantom gap is real, but most of the design-choice comparisons in the appendix rest on differences of one or two rollouts.

    • All real-robot numbers come from one trained model per configuration (Sec. A.4: "Each model configuration is trained once using a fixed random seed"), one phantom, N=20 rollouts, and N=15 human rollouts on the same three participants who supplied the 60 human demonstrations.
    • A 95% interval for 15/20 is about 50-90%, and for 5/20 about 10-50%. By contrast: 30% vs. 40% in Table 9 (6/20 vs. 8/20), 55% vs. 60% in the K ablation (which is also non-monotonic), and 65% vs. 75% in Table 10. The text treats these as clear evidence for each design choice.
    • The super-additivity in Table 9 (25 -> 40 and 25 -> 30, but both together 75) is striking enough that I'd want to see it replicated with another seed.
    • On cost: Fig. 7 shows the 512 local contact trajectories took 3.74 h of wall-clock collection, while all 100 phantom expert demonstrations took 1.42 h. Fig. 4 suggests the CrossWorld advantage shrinks or disappears at 100 phantom demonstrations. So in wall-clock terms, 50 more expert demos look cheaper than D_contact.
    • The argument may still hold if contact collection doesn't need a sonographer, but the paper should state expert-hours for each source, since "reducing reliance on costly expert demonstrations" is the main motivation.
  3. The introduction doesn't explain clearly how the two world models are combined. After reading the contribution list I still didn't know whether they are chained, merged into one model, or used separately. It only becomes clear in Sections 3.5-3.6 that they never interact directly: each produces a second image for the other domain's data, and the policy sees both. One sentence and a pointer to Fig. 2 in the introduction would fix this. Beyond that, the main text reads well, but several facts needed to interpret the results are only in the appendix: the two-slot ordering (sim obs in the S slot, real obs in the C slot; Table 9), the fact that simulated force targets and wrenches are sampled or retrieved from the real contact bank because the sim force channel is a zero placeholder (A.2), and which backbone each appendix ablation uses (Tables 9-11 appear to be OpenVLA, Tables 12-13 ACT, but this is not stated consistently).

Questions for authors

Clarifications that would change my assessment:

  1. Can you add a factual-view baseline that gets the same temporal context as CrossWorld (frame stack of K frames + past actions, same backbone, same two-slot interface)? This is the control I most want to see, in sim and on the phantom.

  2. What does G_s produce on real inputs? Figs. 6b, 11b and 13 show small S-view thumbnails, but does the S-view look sim-like when placed next to actual sim frames? Did you compare against a pure real-to-sim image translation baseline in the S slot? If translation alone recovers most of the gain, the factor-completion framing needs revising.

  3. In simulation evaluation (Table 1), what information do you think the C-view adds, given that the contact condition is sampled from the real bank and the simulator doesn't model contact? Is the gain still there if the C-view is generated from o_t instead of o_{t-1}?

  4. A.3 says the ViT encoders are "further pretrained to learn compact ultrasound representations before world-model training." On which data? If it includes real phantom images (expert or contact), the factual baselines should get the same encoder.

  5. Please give the Fig. 4 numbers in a table for all budgets and backbones. The text says ACT+CW at 50 "approaches ACT's 70.0% with 100 demonstrations". Is 70% the direct-mixing ACT at n=100? If so, ACT jumps from 20% to 70% between 50 and 100 demos, which I'd like to understand.

  6. Does collecting D_contact need a sonographer or other operator in the loop? Please report expert-hours separately from wall-clock time.

  7. Can you report at least the main phantom comparison (OpenVLA direct mixing vs. OpenVLA+CW) over 2-3 training seeds, or give confidence intervals for all real-robot results?

  8. Simulated force targets f_t^{sim,*} are sampled from the empirical force distribution independently of state (A.2). So the force dimension of the sim action labels is noise relative to the observation. Did you try masking the force loss on sim samples? How does this interact with the learned-force result in Table 10?

  9. Related work: the idea of learning the hard-to-simulate contact/low-level part from real interaction and the rest in simulation has a clear precedent in legged locomotion, e.g. the ANYmal work from Hutter's group, where an actuator model learned from real data is combined with policy training in simulation (Hwangbo et al., Science Robotics 2019), and later hierarchical schemes with a sim-trained high level over a separately obtained low-level controller. How does CrossWorld differ from these, beyond the ultrasound image modality? A short discussion would help position the contribution.

Suggestions (not affecting my score): evaluating on held-out human participants, even 2-3, would make the human results much more convincing. Right now they measure within-participant performance only.

10 comments
AuthorsOpenReview ↗

Response to Reviewer UWMN [ Q1/W1(a) ]

Response to Reviewer UWMN

We thank the reviewer for the careful review, for naming the controls that would change the assessment, and for finding the real-robot study more careful than most in this area. We agree that the contribution is the factor-completion mechanism, not only the final success rate. For each question we give the existing evidence first and then the new controls. New results and revisions are marked in blue\color{blue}{\text{blue}}, both in this response and in the updated PDF.

▶\color{BrickRed}{\blacktriangleright} Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/


[ Q1 / W1(a) ]. Does temporal context explain the gain?

No. An existing ablation holds the test-time history fixed and still shows most of the gain, and a new control gives the policy the same raw history without reproducing it.

(1) Existing ablation (Appendix B.3). In Table 9, the spatial-only and full variants receive identical deployment inputs, including the S-view generated from the same K=10K=10 history. The two differ only in whether simulated training samples received contact completion. If history explained the gain, they would perform alike. Instead, success is 30.0% vs. 75.0% (6/20 vs. 15/20, two-sided Fisher exact test, p = 0.010), and the history-based S-view alone stays close to no completion (25.0%).

{}\\\\[-1em] \begin{array}{c} \textbf{Table 9 (excerpt).}\text{ Completion directions on the phantom.}\\\\ \begin{array}{lccc} \hline \textbf{Configuration}&\text{Sim. input }(S,C)&\text{Real input }(S,C)&\text{Ph. Succ.}\uparrow\\\\ \hline \text{No completion}&(o_t^\mathrm{sim}, \mathbf{0})&(\mathbf{0}, o_t^\mathrm{real})&25.0\\\\ \text{Spatial only}&(o_t^\mathrm{sim}, \mathbf{0})&(\hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}}, o_t^\mathrm{real})&30.0\\\\ \rowcolor[gray]{0.92} \textbf{Full CrossWorld}&(o_t^\mathrm{sim}, \hat{o}_t^{\mathrm{sim}\leftarrow\mathrm{real}})&(\hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}}, o_t^\mathrm{real})&\mathbf{75.0}\\\\ \hline \end{array}\\\\ \small\mathbf{0}\text{: masked view.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) New frame-stack control ($\color{blue}{\textbf{Appendix B.11}}$).** While $\color{blue}{\text{Fig. 12}}$ (originally Fig. 10) shows that spatial completion benefits from temporal context, we test whether directly providing the same raw history to the policy reproduces the gain. The factual policy keeps its backbone (ACT) and two-slot interface. Slot 1 holds $o_t$, Slot 2 holds the $K=10$ past frames $o_{t-10:t-1}$, and the past actions $a_{t-10:t-1}$ are appended to $r_t$. Training data match the factual baselines. $$\def\B{\color{blue}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} {\color{blue}{\textbf{Table 15.}}}\text{ Effect of raw temporal context. ACT success }(\\%)\text{.}\\\\ \begin{array}{lccc} \hline \textbf{Policy input}&\text{Sim. within}\uparrow&\text{Sim. cross}\uparrow&\text{Phantom}\uparrow\\\\ \hline \text{Factual view}&76.9&\underline{50.0}&\underline{20.0}\\\\ \B{\text{Factual + frame stack }(K=10)}&\B{\underline{81.3}}&\B{47.5}&\B{10.0}\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld}&\mathbf{85.0}&\mathbf{57.5}&\mathbf{65.0}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ Raw history does not reproduce the gain. In simulation it raises within-subject success from 76.9% to 81.3%, but not cross-subject success (47.5% vs. 50.0%). On the phantom it reaches 10.0% (2/20), far below CrossWorld's 65.0% (13/20, p < 0.001) and not different from the factual view (p = 0.66). Together, these results show that temporal context alone does not explain CrossWorld's gains; contact completion during training is also critical. The simulation part of W1(a), what the C-view adds when the simulator has no contact, is answered under Q3.
AuthorsOpenReview ↗

Response to Reviewer UWMN [ Q2/W1(b) ]

[ Q2 / W1(b) ]. Is the S-view a real-to-sim translation?

No. The S-view keeps the real appearance, a real-to-sim translator in the same slot does not reproduce the gain, and the spatial state behind the S-view tracks real probe motion.

(1) The S-view is not a style translation. Both generators are trained within one domain: GsG_s predicts simulated frames from the simulated state sts_t, and GcG_c predicts real frames from the measured contact ctc_t. Neither is trained to map one domain onto the other, which is what CycleGAN learns. On held-out real phantom frames, the S-view is as far from simulation as the raw real frames (KID 204.5 vs. 208.2), while CycleGAN output is close to it (27.6). The S-view also keeps the anatomy of oto_t better, although it never sees oto_t and CycleGAN does. ▶\color{BrickRed}{\blacktriangleright} Side-by-side examples: https://crossworld-5zb.pages.dev/#wm-comparison

(2) A translator does not replace spatial completion. Placing CycleGAN$(o_t)$ in the S slot, with contact completion and all else unchanged, gives 40.0% (8/20), the same as leaving the slot empty, vs. 75.0% with the S-view.

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{S-view vs. real-to-sim translation.}\\\\ \begin{array}{lccc} \hline \textbf{S slot at deployment}&\text{KID to sim }(\times10^{3})\downarrow&\text{Grad. corr. with }o_t\uparrow&\text{Phantom succ.}\uparrow\\\\ \hline \text{Empty (contact only, Table 9)}&\text{–}&\text{–}&40.0\\\\ \text{CycleGAN}(o_t)&\B{27.6}&\B{0.25}&\B{40.0}\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld S-view}&\B{204.5}&\B{0.59}&\mathbf{75.0}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ **(3) $\hat{s}_t$ carries spatial information on real data.** On held-out real frames, we regenerate $o_t$ with only $\hat{s}_t$ replaced by the mean state, another scan's state at the same step, or the same scan's state 2 s later, keeping $o_{t-1}$ and the noise fixed. On the third of frames with the most probe motion, the scan's own state reconstructs the real frame better than every replacement. On the third with the least motion, the replacements make no difference. $$\def\B{\color{blue}}\def\G{\cellcolor[gray]{0.92}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Spatial-state intervention on real ultrasound.}\ \Delta\text{PSNR (dB).}\\\\ \begin{array}{lcc} \hline \textbf{Replaced by}&\text{Static frames}&\G\text{Moving frames}\\\\ \hline \text{Mean state}&\B{-0.04}&\G\B{+0.71\ [0.61,0.89]}\\\\ \text{Other scan, same step}&\B{-0.02}&\G\B{+0.53\ [0.40,0.63]}\\\\ \text{Same scan, 2 s later}&\B{+0.02}&\G\B{+0.24\ [0.12,0.36]}\\\\ \hline \end{array}\\\\ \small\text{Positive: the scan's own state reconstructs better.} \end{array}\\\\[-1em] {} \end{multline*}$$ Independently, the longitudinal component of $\hat{s}_t$ follows the cranio-caudal probe displacement measured by the calibrated robot kinematics (Eq. 21), after controlling for lateral motion (median partial $\rho=0.73$, $p<0.005$ against a circular-shift null).
AuthorsOpenReview ↗

Response to Reviewer UWMN [Q3]

[ Q3 ]. What does the C-view add in simulation, and what if it is generated from oto_t?

The C-view adds variation in contact appearance, which the simulator does not provide. When the C-view is generated from oto_t, the cross-subject gain remains, while most of the within-subject gain disappears.

(1) What does the C-view add in simulation? Variation in contact appearance. SonoGym renders each frame with a pix2pix network [1] that maps CT slices to ultrasound images, trained on scans from seven ex-vivo spine specimens, with no contact input. Every simulated anatomy is therefore rendered under one fixed, implicit contact condition (Appendix A.2), and a policy trained only on factual frames can fit to it. The C-view renders the same simulated state under varied real contact sampled through c~t\tilde{c}_t: coupling, texture and attenuation change while the anatomy stays in place (Fig. 15\color{blue}{\text{Fig. 15}}, originally Fig. 12). Training with this variation makes the policy less sensitive to contact-dependent appearance, which matters most on held-out anatomies.

(2) Does generating the C-view from oto_t preserve the gain? Partly. We generate the C-view from the current frame oto_t instead of ot−1o_{t-1}, with the same trained GcG_c and sampled contact, and retrain OpenVLA in simulation.

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{C-view from }o_{t-1}\textbf{ vs. }o_t.\text{ OpenVLA simulation success }(\\%)\text{.}\\\\ \begin{array}{lcc} \hline \textbf{Policy input}&\text{Within-subject}\uparrow&\text{Cross-subject}\uparrow\\\\ \hline \text{Factual view}&78.8&67.5\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld, C-view from }o_{t-1}&\mathbf{86.2}&\underline{77.5}\\\\ \B{\text{C-view from }o_t}&\B{\underline{80.0}}&\B{\mathbf{82.5}}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ With the C-view generated from $o_t$, cross-subject success stays at the CrossWorld level (82.5% vs. 77.5%), so the gain on held-out anatomies is preserved, consistent with the contact variation in (1). Within-subject success is 80.0%, close to the factual view (78.8%). We use $o_{t-1}$ because $G_c$ is trained on consecutive tuples $(o_{t-1}, a_{t-1}, c_t, o_t)$, the only kind our continuous contact collection provides. A model trained for same-frame completion would need the same pose under several contacts, which requires force sweeps at fixed poses plus spatial registration. We will add this control to $\color{blue}{\textbf{Appendix B}}$. **References** - [1] Y. Ao, M. Moghani, M. Mittal, M. Prajapat, L. Wu, F. Giraud, F. Carrillo, A. Krause, and P. Fürnstahl. SonoGym: High performance simulation for challenging surgical tasks with robotic ultrasound. arXiv:2507.01152, 2025.
Reviewer_yCbGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of building a robotic policy for contact-rich ultrasound scanning task. Since data collection for such tasks is expensive and existing world model cannot predict ultrasound image accurately, authors combine a spatial world model with a contact world model to predict the actual ultrasound image and use it to generate data. Authors show that this could be applied to actual data collection in ultrasound tasks.

Critical strengths
  • The problem this paper tackles is very practical in the real world and also very interesting.
  • The code is provided in the repo.
Critical weaknesses
  • Writing could be significantly improved. Specific points are listed below
  • A explanation supplementary video will be helpful for understanding this work quickly.
  • It is very hard to capture the insight behind the design. Accoding to Figure 2 (4), there are four observations into policy (o_sim, o_real, o_sim2real, o_real2sim), and there are two separate world models in addition to sim and real dataset. It is hard for me to understand what each model tries to learn and why combining them can lead to a better result.
  • The complicated design is not well-justified. The paper proposes a complicated design by introducing a bidirectional world model and a co-observation policy learning. But authors do not analyze or visualize the output of these world models. For example, what kind of information does these world models learn to predict?
  • It would be better to have a video of real robot rollout.
Questions for authors
  • I am not sure whethee "ultrasound policy" is a well-established terminology in medical robotics. But I thought it is a policy taking ultrasound as inputs. After reading a bit, it looks to me as a policy for ultrasound scanning. Maybe renaming this terminology if this is not already a well-adopted terminology in medical robotcs. According to related works, it seems more works call it "robotics ultrasound imaging" or "robotic ultrasound scanning".
  • Since most of the readers might not have the medical background, it would be better to avoid using medical terms. For example, "factual B-mode observation" might be "ultrasound scanning observation". "L4 standard-plane navigation" might remind of people of the autonomous driving, which is confusing.
  • Some terminology is introduced in the abstract without describing it first, such as "phantom scanning". This can create a lot of confusions for the readers.
  • The method figure and description is overly complicated. Figure 2 contains too dense information. I think authors try to pass all details into reader in the method section. However, this might lose the focus of the description. The high-level formulaton already contains quite dense information, including o, a, s, r, c for sim and real version, and a can be random data or expert data. This is already quite dense to process, but in section 3.3, more details and annotations are further introduced. But some details can be put into the supp, such as E, T, G, and so on.
  • I am also confused by Equation 11. To derive z^{CW,sim}_t and z^{CW,real}_t, do we need an action a. If yes, is action a exactly a_t^*? If yes, that will make the design a bit weird, as action a appears as both input and output supervision.
4 comments
AuthorsOpenReview ↗

Response to Reviewer yCbG [ W1/Q1–Q3 ]

Response to Reviewer yCbG

We thank the reviewer for finding the problem practical and interesting, and for the careful reading and concrete writing suggestions. We answer the points in the order of the review and group related weaknesses and questions. All revisions are marked in blue\color{blue}{\text{blue}} in the updated PDF.

▶\color{BrickRed}{\blacktriangleright} Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/


[ W1 / Q1–Q3 ]. Writing and terminology

We agree with these suggestions. Several terms were used before they were defined, which makes the paper harder to follow for readers without a medical-imaging background. We have revised the terminology as follows. The method figure and description (Q4) are addressed together with W3 below.

(1) "Ultrasound policy" (Q1). We agree and have replaced the term with "robotic ultrasound scanning policy"\color{blue}{\text{"robotic ultrasound scanning policy"}} throughout the main text.

(2) Medical terms (Q2). We have replaced "factual B-mode observation" with "observed ultrasound image"\color{blue}{\text{"observed ultrasound image"}} and defined B-mode once, at its first use in the Fig. 1 caption, as the standard 2D ultrasound image. The Abstract and the Introduction now define "L4 standard-plane navigation" at first use as moving the probe to the standard longitudinal view of the fourth lumbar vertebra (L4).

(3) Terms in the Abstract (Q3). The Abstract now introduces the phantom at first use as a tissue-mimicking spine phantom and explains cross-subject success as success on held-out simulated anatomies, together with the definition of L4 standard-plane navigation above.


[ W2 / W5 ]. Explanation and real-robot videos

We provide an anonymized overview video of about three minutes with English subtitles, together with videos of CrossWorld policy rollouts, on an anonymous project page.

▶\color{BrickRed}{\blacktriangleright} Overview video: https://crossworld-5zb.pages.dev/#overview

▶\color{BrickRed}{\blacktriangleright} Policy rollouts: https://crossworld-5zb.pages.dev/#real-robot

(1) Explanation video (W2). The video explains the scanning task, the two partial worlds, what each world model learns, how the completed views are built, and the main real-robot results.

(2) Real-robot rollouts (W5). The rollout videos show the CrossWorld policy controlling the robot in closed loop, on the real robot and in simulation. Each clip pairs the observed ultrasound image with its completed view at every step, up to the final scan.

AuthorsOpenReview ↗

Response to Reviewer yCbG [ W3/Q4 ]

[ W3 / Q4 ]. Design insight and the method figure

(1) What the policy sees. Each training sample gives the policy two images, not four: an observed image and its completed view. Put simply, simulation shows where the probe is but not how contact changes the image. Real local interactions show contact but carry no navigation labels. Each world model fills in what the other world is missing. Fig. 2(4) shows the two kinds of samples side by side, and the table below summarizes the two world models.

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Roles of the spatial and contact world models.}\\\\ \begin{array}{l|l|l} \hline &\G\textbf{Spatial world model}&\G\textbf{Contact world model}\\\\ \hline \text{Trained on}&\G \text{Simulation, }\mathcal{D}_\mathrm{sim}&\G \text{Real local interactions, }\mathcal{D}_\mathrm{contact}\\\\ \text{Supervision}&\G \text{Spatial state }s_t\text{ (simulation only)}&\G \text{Measured contact }c_t\text{ (real only)}\\\\ \text{Learns}&\G \text{How probe motion changes the view}&\G \text{How contact force changes the image}\\\\ \text{Applied to}&\G \text{Real samples (Eq. 10)}&\G \text{Simulated samples (Eq. 9)}\\\\ \text{Policy input }(S,C)&\G (\hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}},o_t^{\mathrm{real}})&\G (o_t^{\mathrm{sim}},\hat{o}_t^{\mathrm{sim}\leftarrow\mathrm{real}})\\\\ \text{Used at deployment}&\G \text{Yes}&\G \text{No (training only)}\\\\ \hline \end{array}\\\\ \small\text{Hats: completed views. }S/C\text{: spatial/contact slot of the policy input.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) Why combining the two helps.** Each model needs a label that only one world provides: the spatial state $s_t$ exists only in simulation, and measured contact $c_t$ only in real interactions. Completion gives every sample both factors, so the policy learns from simulation and reality in the same two-factor format. Table 9 shows the effect: either direction alone gives 40.0% or 30.0% phantom success, while both together give 75.0%. **(3) Simplifying the method presentation (Q4).** We agree that Fig. 2 and **Section 3** carry too much detail and hide this insight. We have added this insight in three sentences to the **Introduction**, with a pointer to Fig. 2, and **Section 3.1** now summarizes each model's inputs and outputs in plain language. We will also simplify Fig. 2 so that it shows the two sample types and the two world models as single blocks, and move the internal estimator–transition–generator structure ($E$, $T$, $G$), including the two model-detail figures now in **Section 3** of the updated PDF, to **Appendix A.3**, keeping one equation per model in the main text.
AuthorsOpenReview ↗

Response to Reviewer yCbG [ W4 ]

[ W4 ]. Is the design justified, and what do the world models learn?

Each component is tested by an ablation, both world models are visualized in Appendix C.2, and we now also measure them directly.

(1) Ablations. The following ablations evaluate the contributions of key design components.

{}\\\\[-1em] \begin{array}{c} \textbf{Ablations of the design.}\text{ Success }(\\%)\text{ without and with each component.}\\\\ \begin{array}{llccl} \hline \textbf{Component}&\textbf{Setting}&\text{Without}&\cellcolor[gray]{0.92}\textbf{With}&\textbf{Source}\\\\ \hline \text{Spatial completion (vs. contact only)}&\text{Phantom, OpenVLA}&40.0&\cellcolor[gray]{0.92}\mathbf{75.0}&\text{Table 9}\\\\ \text{Contact completion (vs. spatial only)}&\text{Phantom, OpenVLA}&30.0&\cellcolor[gray]{0.92}\mathbf{75.0}&\text{Table 9}\\\\ \mathcal{D}_\mathrm{contact}\text{ for world modeling (vs. pretraining)}&\text{Phantom, ACT}&25.0&\cellcolor[gray]{0.92}\mathbf{65.0}&\text{Table 13}\\\\ \text{Paired views (vs. extra samples)}&\text{Sim, OpenVLA}&81.9/67.5&\cellcolor[gray]{0.92}\mathbf{86.2/77.5}&{\color{blue}{\text{Fig. 7}}}\text{ (originally Fig. 5)}\\\\ \text{Measured contact input }c_{t-1}&\text{Phantom, OpenVLA}&45.0&\cellcolor[gray]{0.92}\mathbf{75.0}&\text{Table 11}\\\\ \text{Spatial context }K=10\text{ (vs. }K=1\text{)}&\text{Phantom, OpenVLA}&45.0&\cellcolor[gray]{0.92}\mathbf{75.0}&{\color{blue}{\text{Fig. 12}}}\text{ (originally Fig. 10)}\\\\ \text{View dropout}&\text{Sim, ACT}&80.6/52.5&\cellcolor[gray]{0.92}\mathbf{85.0/57.5}&\text{Table 12}\\\\ \hline \end{array}\\\\ \small\text{Sim: within/cross-subject success.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) What the models learn (Appendix C.2).** In $\color{blue}{\text{Fig. 15}}$ (originally Fig. 12), the contact model changes near-field coupling, soft-tissue texture, bone-interface visibility and posterior attenuation as the force condition goes from 5 to 20 N, while the anatomy stays in place. In $\color{blue}{\text{Fig. 16}}$ (originally Fig. 13), the spatial model starts from real frames and moves bone and posterior structures along the executed action sequence; red boxes mark where long rollouts begin to drift. We will move a compact version of these two figures into the main text. $\color{BrickRed}{\blacktriangleright}$ **World-model outputs: [https://crossworld-5zb.pages.dev/#world-models](https://crossworld-5zb.pages.dev/#world-models)** **(3) Direct measurements.** We also measured the generated views directly. For each world model, we generate two views from the same input and change only its own condition: the force for the contact model and the spatial state for the spatial model. We then measure how far the anatomy moves (landmark displacement) and how much the texture changes (speckle decorrelation, $D_\mathrm{speckle}$). Consecutive real and simulated frames serve as references. $$\def\B{\color{blue}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Factor change between paired images.}\text{ Median [95\\% CI].}\\\\ \begin{array}{lcc} \hline \textbf{Source}&\text{Spatial change (mm) }(\times10^{2})&\text{Texture change }D_\mathrm{speckle}\ (\times10^{3})\\\\ \hline \text{Real B-mode}&\B{16.7\ [12.7,23.2]}&\B{16\ [14,21]}\\\\ \text{Simulated B-mode}&\B{4.6\ [4.1,6.0]}&\B{0.4\ [0.3,0.6]}\\\\ \rowcolor[gray]{0.92} \textbf{Spatially-completed}&\B{12.3\ [11.1,12.8]}&\B{0.4\ [0.4,0.6]}\\\\ \rowcolor[gray]{0.92} \textbf{Contact-completed}&\B{4.1\ [2.7,5.6]}&\B{12\ [10,17]}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ Changing only the force changes the texture of the contact-completed view by about as much as real consecutive frames (0.012 vs. 0.016), while the anatomy moves no more than between simulated frames. Changing only the spatial state moves the anatomy (0.123 mm, vs. 0.167 mm between real frames) while the texture stays at the simulated level. On held-out real scans, the estimated spatial state also follows the measured probe displacement (median partial $\rho=0.73$).
AuthorsOpenReview ↗

Response to Reviewer yCbG [ Q5 ]

[ Q5 ]. Is the target action at∗a_t^{*} used to build ztCWz_t^{\mathrm{CW}} in Eq. 11?

An action is needed, but it is the previous executed action at−1a_{t-1}, never the current action ata_t or the target at∗a_t^{*}. Conditioning on the previously executed action is standard in world-model dynamics (DreamerV3 [1]), and VLA controllers also use previously committed actions to maintain continuity across action chunks (Real-Time Chunking [2]). The braces below mark what each quantity is built from:

{}\\\\[-1em] \begin{array}{rll} \hat{o}_t^{\mathrm{sim}\leftarrow\mathrm{real}}&=G_c\big(\underbrace{o_{t-1}^{\mathrm{sim}},\ \bbox[LightCyan,1pt]{a_{t-1}^{\mathrm{sim}}}}_{\text{previous step}},\ \underbrace{\tilde{c}_t^{\mathrm{real}}}_{\text{real contact}}\big)&\qquad\text{(Eq. 9)}\\\\[12pt] \hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}}&=G_s\big(o_{t-1}^{\mathrm{real}},\ T_s\big(E_s(\underbrace{\mathcal{H}_{t-1}^{\mathrm{real}}}_{\text{up to }t-1}),\ \underbrace{\bbox[LightCyan,1pt]{a_{t-1}^{\mathrm{real}}}}_{\text{previous action}}\big)\big)&\qquad\text{(Eq. 10)}\\\\[12pt] \mathcal{D}_{\mathrm{pol}}&=\big\\{(\underbrace{z_t^{\mathrm{CW}}}_{\text{input}},\ \underbrace{\bbox[MistyRose,1pt]{a_t^{*}}}_{\text{label}})\big\\}&\qquad\text{(Eq. 11)} \end{array}\\\\[-1em] {} \end{multline*}$$ **(1) Training.** Both world models use only past observations and executed actions, without the target frame $o_t$ or the current action $a_t$ (**Sections 3.3–3.4**). The real contact condition in Eq. 9 is also retrieved with a key built from $a_{t-1}$ and a sampled force (**Appendix A.2**). The policy input $z_t^{\mathrm{CW}}=(o_t,\hat{o}_t,r_t)$ therefore contains no action from step $t$, and $a_t^{*}$ appears only as the label in Eq. 12. **(2) Deployment.** At test time, $a_{t-1}$ is the action the robot has just executed (**Section 3.6**), so the same input is available. We have added this clarification directly below Eq. 11. **References** - [1] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse control tasks through world models. *Nature*, 640:647–653, 2025. - [2] K. Black, M. Y. Galliker, and S. Levine. Real-time execution of action chunking flow policies. In *Advances in Neural Information Processing Systems (NeurIPS)*, 2025. In summary, we have revised the terminology and the method presentation, added the videos, and shown with ablations and direct measurements what each world model contributes. We are happy to answer further questions.
Reviewer_cgJFOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CrossWorld, a framework for learning robotic ultrasound navigation policies by combining complementary information from simulation and real-world contact interactions. The authors train separate spatial and contact world models and use bidirectional cross-world completion to generate paired observations for policy learning. Experiments on L4 ultrasound navigation show improvements across ACT, Diffusion Policy, and OpenVLA, including substantial gains in real-robot evaluations.

Critical strengths
  • The problem seems to be well-motivated. The idea of combining spatial information from simulation with realistic contact observations from the real world is interesting and potentially applicable to other contact-rich robotic tasks.

  • The experimental results are promising. CrossWorld consistently improves performance across three policy backbones, with particularly large gains in real-robot evaluations.

Critical weaknesses

The paper mainly relies on downstream policy performance to demonstrate their effectiveness. It is unclear whether the generated observations actually preserve the intended spatial/contact factors.

Real-world generalization is limited. The evaluation uses one phantom and only three human participants, who are also included in the training data. Generalization to unseen subjects remains unclear.

There's also no video attached despite being a robotics paper. I'd strongly encourage doing so otherwise it is hard to sell a robotics paper.

Questions for authors
  • Can the authors quantitatively evaluate whether the spatial and contact world models preserve the intended information in their generated observations?

  • Have the authors evaluated, or can they comment on, generalization to unseen human subjects?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cgJF [ W1 / Q1 ]

Response to Reviewer cgJF

We thank the reviewer for the positive assessment, for noting that the idea may apply to other contact-rich tasks, and for recognizing the consistent gains across three policy backbones. We address the three points below. All revisions are marked in blue\color{blue}{\text{blue}} in the updated PDF.

▶\color{BrickRed}{\blacktriangleright} Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/


[ W1 / Q1 ]. Do the generated observations preserve the intended factors?

Yes. Direct measurements on the generated views show that the two factors are separated (1) and that the spatial effect is correct on real scans (2).

(1) The factors are separated. If a model keeps its intended factor, changing its condition should change only that factor. For each world model, we generate two views from the same input and noise, changing only its own condition (s^t\hat{s}_t for GsG_s, the force in c~t\tilde{c}_t for GcG_c), and measure the anatomy shift (landmark displacement) and the texture change (speckle decorrelation Dspeckle=1−NCCD_\mathrm{speckle}=1-\mathrm{NCC} after registration). Consecutive real and simulated frames serve as references.

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Factor change between paired images.}\text{ Median [95\\% CI].}\\\\ \begin{array}{lcc} \hline \textbf{Source}&\text{Spatial change (mm) }(\times10^{2})&\text{Texture change }D_\mathrm{speckle}\ (\times10^{3})\\\\ \hline \text{Real B-mode}&\B{16.7\ [12.7,23.2]}&\B{16\ [14,21]}\\\\ \text{Simulated B-mode}&\B{4.6\ [4.1,6.0]}&\B{0.4\ [0.3,0.6]}\\\\ \rowcolor[gray]{0.92} \textbf{Spatially-completed}&\B{12.3\ [11.1,12.8]}&\B{0.4\ [0.4,0.6]}\\\\ \rowcolor[gray]{0.92} \textbf{Contact-completed}&\B{4.1\ [2.7,5.6]}&\B{12\ [10,17]}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ Spatially-completed views move the anatomy almost as much as consecutive real frames, while their texture stays at the simulated level. Contact-completed views show the reverse: their texture changes almost as much as between real frames, while the anatomy moves no more than between simulated frames. The intervals of the two completed rows do not overlap on either measure. **(2) The spatial effect is correct.** A correctness check needs a reference image, which only spatial completion has. Contact completion is applied to simulated frames, for which no real image under the sampled contact exists, so for the contact model we rely on (1) and the force sweeps in $\color{blue}{\text{Fig. 15}}$ (originally Fig. 12). Spatial completion is applied to real scans, where the robot kinematics and the next real frame serve as references, and both of its links hold. *The spatial state tracks the probe.* Using the calibrated robot kinematics (Eq. 21), the longitudinal component of $\hat{s}_t$ follows the measured cranio-caudal probe displacement after controlling for lateral motion (median partial $\rho=0.73$, $p<0.005$ against a circular-shift null). *The generated view uses the state.* On held-out real frames, we regenerate $o_t$ with only $\hat{s}_t$ replaced by the mean state, another scan's state at the same step, or the same scan's state 2 s later, keeping the previous frame and the sampling noise fixed. $$\def\B{\color{blue}}\def\G{\cellcolor[gray]{0.92}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Spatial-state intervention on real ultrasound.}\ \Delta\text{PSNR (dB).}\\\\ \begin{array}{lcc} \hline \textbf{Replaced by}&\text{Static frames}&\G\text{Moving frames}\\\\ \hline \text{Mean state}&\B{-0.04}&\G\B{+0.71\ [0.61,0.89]}\\\\ \text{Other scan, same step}&\B{-0.02}&\G\B{+0.53\ [0.40,0.63]}\\\\ \text{Same scan, 2 s later}&\B{+0.02}&\G\B{+0.24\ [0.12,0.36]}\\\\ \hline \end{array}\\\\ \small\text{Positive: the scan's own state reconstructs better.} \end{array}\\\\[-1em] {} \end{multline*}$$ On the third of frames with the most probe motion, the scan's own state reconstructs the real frame better than every replacement. On the third with the least motion, the differences are not significant. These measurements complement the spatial rollouts in $\color{blue}{\text{Fig. 16}}$ (originally Fig. 13) and the ablations in Tables 9 and 11. We will add both measurements to $\color{blue}{\textbf{Appendix C.2}}$.
AuthorsOpenReview ↗

Response to Reviewer cgJF [ W2/Q2/W3 ]

[ W2 / Q2 ]. Generalization to unseen human subjects

We have not yet evaluated unseen participants on the robot. Our human results are within-participant: the three participants who provided human demonstrations are evaluated again from newly randomized starts. Section 4.3 and Appendix D.1 state this scope, and we will also state it in the Abstract. Two results bear on generalization to unseen subjects.

(1) Held-out anatomies in simulation. On two held-out anatomies, S9–S10, CrossWorld improves all three backbones (Table 1: ACT and DP +7.5 points, OpenVLA 67.5% → 77.5%), and the OpenVLA gain holds across three training seeds (Table 14: 68.3% → 78.3%).

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Tables 1 and 14 (excerpt).}\text{ Cross-subject success }(\\%)\text{ on held-out anatomies.}\\\\ \begin{array}{lcl} \hline \textbf{Backbone}&\text{Simulation only}&\cellcolor[gray]{0.92}\textbf{+ CrossWorld}\\\\ \hline \text{ACT}&50.0&\cellcolor[gray]{0.92}57.5\up{7.5}\\\\ \text{DP}&52.5&\cellcolor[gray]{0.92}60.0\up{7.5}\\\\ \text{OpenVLA}&\underline{67.5}&\cellcolor[gray]{0.92}\mathbf{77.5}\up{10.0}\\\\ \hline \text{OpenVLA, 3 seeds}&68.3\pm3.8&\cellcolor[gray]{0.92}\mathbf{78.3\pm3.8}\\\\ \hline \end{array}\\\\ \small\text{3 seeds: mean }{\pm}\text{ std. Small numbers: gain over simulation only.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) Phantom-trained contact model.** Although the contact world model is trained only on phantom interactions, CrossWorld improves human success from at most 6.7% with direct mixing to 46.7–60.0% (Table 2), using the same 60 human expert demonstrations. **(3) Held-out participants.** Testing on held-out participants requires recruiting new participants through our institutional ethics review and informed-consent process (**Ethics statement**). This process cannot be completed within the discussion period, so we could not add new participants for this response. We agree that held-out participants are the direct test of human generalization, and we will complete real-robot evaluations on additional, unseen participants once this process is finished. --- > ## **[ W3 ]. Video** We agree. We have added an overview video and videos of CrossWorld policy rollouts, on the real robot and in simulation, to an anonymous project page. $\color{BrickRed}{\blacktriangleright}$ **Policy rollouts: [https://crossworld-5zb.pages.dev/#real-robot](https://crossworld-5zb.pages.dev/#real-robot)** $\color{BrickRed}{\blacktriangleright}$ **Overview video (3 min, English subtitles): [https://crossworld-5zb.pages.dev/#overview](https://crossworld-5zb.pages.dev/#overview)** The rollout videos show the CrossWorld policy controlling the robot in closed loop, on the real robot and in simulation. Each clip shows the robot scene, the observed ultrasound image and its completed view. The overview video covers the task, the method and the real-robot results. We hope these additions address your concerns, and we thank you again for the constructive suggestions.
Reviewer_8hWwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes CrossWorld to address the high cost of real-world robot data and the difficulty of obtaining certain types of information in real-world settings. The method trains separate world models on simulation and real-world data, then uses cross-generated visual features and images to train a robot policy. The proposed approach achieves substantial improvements over the evaluated baselines.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The paper addresses a well-defined problem, and the experiments demonstrate the effectiveness of the proposed method on the evaluated tasks.
  3. I particularly appreciate the analysis in Section 4.5, which provides insight into the policy’s behavior beyond aggregate success rates.
Critical weaknesses

Although I find the method interesting and novel, its contribution should be further clarified, as several relevant works are missing from the discussion and comparisons. My main concerns relate to sim-real co-training and world models.

  1. Sim-real co-training. The paper does not discuss this line of research. Many studies have investigated how to combine simulation data with costly and limited real-world data, including empirical analyses [1,2], mechanistic analyses [3], and methods based on representation alignment or reinforcement learning (e.g., [4,5]). These studies address broader robotic tasks, but the paper neither discusses nor compares against them. The authors should further discuss the relationship to these works and clarify their contribution beyond sim-real co-training itself.

  2. World models. Incorporating world-model-generated observations or features into robot policy training is not itself a new contribution. For example, VPP [6] uses predictive visual features for policy learning. Discussion of and comparisons with this line of work are also missing, even though such approaches may offer a simpler alternative to CrossWorld. These discussions and comparisons should be included.

[1] Maddukuri et al. Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation. 2025.

[2] Wei et al. Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels. 2025.

[3] Lei et al. A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies. 2026.

[4] Cheng et al. Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training. 2026.

[5] Shi et al. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models. 2026.

[6] Hu et al. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. 2025.

Questions for authors

I am also interested in the generalizability of the proposed method. The experiments in this paper focus on the medical domain, whereas broader robotic manipulation tasks are more commonly used for comparison. Since the paper does not discuss this, could the authors clarify whether the method is also applicable to general robotic manipulation tasks? Do the authors have any experimental results in such settings?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8hWw [W1]

Response to Reviewer 8hWw

We thank the reviewer for finding the paper clearly written, the method interesting and novel, and the analysis in Section 4.5 insightful. We agree that the positioning should be explicit, and we clarify it below. New results and revisions are marked in blue\color{blue}{\text{blue}}, both in this response and in the updated PDF. References [1–6] follow the numbering in the review.

▶\color{BrickRed}{\blacktriangleright} Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/


[ W1 ]. Relation to sim-real co-training

In short, conventional sim-real co-training [1–5] combines task-supervised data from both domains, while CrossWorld additionally exploits complementary partial-world data that lack task supervision. Here, the cheap real data (Dcontact\mathcal{D}_\mathrm{contact}, 512 local interactions) have no navigation labels, so conventional supervised co-training cannot directly use them for policy learning. The simulator renders no contact-dependent appearance. Alignment approaches [3, 4] address cross-domain differences but do not explicitly reconstruct the missing contact-dependent observations. CrossWorld fills each missing factor from the domain that observes it. We have added this discussion of [1–5] to Section 2.

(1) Basic co-training. Direct mixing (Dsim∪Dexpert\mathcal{D}_\mathrm{sim} \cup \mathcal{D}_\mathrm{expert}, 95:5 sampling, same as CrossWorld) reaches at most 25.0% phantom and 6.7% human success, vs. 65.0–75.0% and 46.7–60.0% for CrossWorld (Table 2). All six comparisons are significant (two-sided Fisher exact test, Holm-adjusted p ≤ 0.021). Section 4.1 states the 95:5 ratio, and we will add the ratio and the test to the Table 2 caption.

\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Table 2 (excerpt).}\text{ Real-robot success }(\\%)\text{.}\\\\ \begin{array}{lll} \hline \textbf{Method}&\text{Phantom}\uparrow&\text{Human}\uparrow\\\\ \hline \text{ACT direct mixing}&20.0&0.0\\\\ \rowcolor[gray]{0.92} \textbf{ACT + CW}&65.0\up{45.0}&46.7\up{46.7}\\\\ \hline \text{DP direct mixing}&25.0&0.0\\\\ \rowcolor[gray]{0.92} \textbf{DP + CW}&\underline{70.0}\up{45.0}&\underline{53.3}\up{53.3}\\\\ \hline \text{OpenVLA direct mixing}&25.0&6.7\\\\ \rowcolor[gray]{0.92} \textbf{OpenVLA + CW}&\mathbf{75.0}\up{50.0}&\mathbf{60.0}\up{53.3}\\\\ \hline \end{array}\\\\ \small\text{CW: CrossWorld. Small numbers: gain over direct mixing.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) Tuned co-training.** Sweeping the sampling ratio from 99:1 to 50:50 as in [1] raises direct mixing only to 25.0% (ACT) and 30.0% (OpenVLA), vs. 65.0% and 75.0% for untuned CrossWorld (p = 0.025, 0.010). For ACT, no ratio beats expert-only training (35.0%). We have added this sweep to $\color{blue}{\textbf{Appendix B.10}}$ ($\color{blue}{\text{Figure 13}}$). $\color{BrickRed}{\blacktriangleright}$ **Sampling-ratio sweep: [https://crossworld-5zb.pages.dev/#mixing-ratio](https://crossworld-5zb.pages.dev/#mixing-ratio)** $$\def\B{\color{blue}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} {\color{blue}{\textbf{Figure 13}}}\textbf{ (as a table).}\text{ Phantom success }(\\%)\text{ vs. sim:real sampling ratio.}\\\\ \begin{array}{llcc} \hline \textbf{Training}&\textbf{Sim:real}&\textbf{ACT}&\textbf{OpenVLA}\\\\ \hline \text{Expert only}&\text{0:100}&35.0&10.0\\\\ \hline \text{Direct mixing}&\B{\text{99:1}}&\B{10.0}&\B{5.0}\\\\ &\text{95:5}&20.0&25.0\\\\ &\B{\text{90:10}}&\B{20.0}&\text{–}\\\\ &\B{\text{75:25}}&\B{\underline{25.0}}&\text{–}\\\\ &\B{\text{50:50}}&\B{\underline{25.0}}&\B{\underline{30.0}}\\\\ \hline \rowcolor[gray]{0.92} \textbf{CrossWorld}&\text{95:5}&\mathbf{65.0}&\mathbf{75.0}\\\\ \hline \end{array}\\\\ \small\text{Underline: best direct-mixing ratio. –: not trained.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(3) Contact data without completion.** Using $\mathcal{D}_\mathrm{contact}$ for encoder pretraining gives 25.0% vs. 65.0% (Table 13). The gain comes from how the data are used, not from more data. $$\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Table 13 (excerpt).}\text{ Alternative uses of local contact data.}\\\\ \begin{array}{lc} \hline \textbf{Configuration}&\textbf{Phantom success}\\\\ \hline \text{ACT + contact-data pretraining}&25.0\\%\\\\ \rowcolor[gray]{0.92} \textbf{ACT + CrossWorld}&\mathbf{65.0\\%}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ CrossWorld complements co-training: $\mathcal{D}_\mathrm{pol}$ (Eq. 11) is itself a sim-real mixture, so ratio tuning [1] or alignment [4] can be applied on top.
AuthorsOpenReview ↗

Response to Reviewer 8hWw [W2]

[ W2 ]. Relation to world-model policies such as VPP

In short, VPP [6] uses predictive visual representations for policy learning, while CrossWorld learns complementary factors from different supervision sources and applies each model across domains. Feeding world-model predictions to a policy is not new, and we do not claim it. We have added this discussion of VPP [6] to Section 2.

(1) Methodological difference. VPP conditions the policy on future representations predicted inside a video diffusion model. In robotic ultrasound, no single domain shows both factors. Simulation shows spatial progress but no contact-dependent appearance, and local real interactions show contact but no task-directed navigation. A predictor trained within either domain has no supervision for the missing factor. CrossWorld differs in three respects:

  • Supervision. The spatial model is supervised by the spatial state sts_t, available only in simulation, and the contact model by measured contact ctc_t, available only on the real robot.
  • Application. Each model is applied to the other domain: contact completion for simulated frames, spatial completion for real frames.
  • Use by the policy. The completed view is paired with the factual frame of the same step, in both training and deployment, so simulated and real samples share one two-factor format.

(2) Simpler world models or domain augmentation. VPP fine-tunes a pretrained video foundation model on large human and robot manipulation datasets. Robotic ultrasound has no dataset of comparable scale. CrossWorld starts from generic image backbones, trains both predictors on in-domain data, and runs only the spatial model at deployment (Section 3.6). We tested the simpler alternatives available in this setting, and none reaches the full result:

  • A single world model. One completion direction gives 30.0% (spatial only) or 40.0% (contact only) phantom success, vs. 75.0% with both (Table 9; vs. spatial only p = 0.010, vs. no completion p = 0.004).
  • Learned domain translation. A CycleGAN real-to-sim translation of oto_t in the S slot, with contact completion unchanged, gives 40.0%, the same as an empty S slot.
  • Conventional augmentation. An augmented second view in the same paired slot gives 77.5/62.5% within-/cross-subject success, vs. 78.8/67.5% for factual-only training (Table 8).
  • Generated views as extra samples. Independent-view training raises within-subject success to 81.9% but leaves cross-subject success at 67.5%, vs. 86.2/77.5% for CrossWorld (Fig. 7\color{blue}{\text{Fig. 7}}, originally Fig. 5).
\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Simpler alternatives to cross-world completion.}\text{ Success }(\\%)\text{.}\\\\ \begin{array}{llccc} \hline \textbf{Setting}&\textbf{Method / Alternative}&\text{Phantom}\uparrow&\text{Sim. within}\uparrow&\text{Sim. cross}\uparrow\\\\ \hline \text{Real robot}&\text{No completion}&25.0&\text{–}&\text{–}\\\\ &\text{Spatial-only WM}&30.0&\text{–}&\text{–}\\\\ &\text{Contact-only WM}&\underline{40.0}&\text{–}&\text{–}\\\\ &\B{\text{CycleGAN + contact completion}}&\B{\underline{40.0}}&\text{–}&\text{–}\\\\ \rowcolor[gray]{0.92} &\textbf{Full CrossWorld}&\mathbf{75.0}&\text{–}&\text{–}\\\\ \hline \text{Simulation}&\text{Factual-only}&\text{–}&78.8&\underline{67.5}\\\\ &\text{Conventional augmented view}&\text{–}&77.5&62.5\\\\ &\text{Independent-view training}&\text{–}&\underline{81.9}&\underline{67.5}\\\\ \rowcolor[gray]{0.92} &\textbf{Full CrossWorld}&\text{–}&\mathbf{86.2}&\mathbf{77.5}\\\\ \hline \end{array}\\\\ \end{array}\\\\[-1em] {} \end{multline*}$$ These results point to two requirements: both factors must be completed, and each completed view must be paired with its factual frame at decision time. A VPP-style factor-agnostic predictor, trained on all three datasets without $s_t$ or $c_t$, is the most direct further comparison, and we will add it once its robot evaluation is complete.
AuthorsOpenReview ↗

Response to Reviewer 8hWw [Q1]

[ Q1 ]. Applicability beyond ultrasound

The cross-world factor-completion principle is not ultrasound-specific, but we have not yet evaluated it outside robotic ultrasound. It relies on three conditions: (i) simulation renders the spatial factor well but not the contact factor; (ii) local real interactions expose the contact factor through a measured signal such as force; (iii) a few complete demonstrations provide task supervision.

Contact-rich tasks such as wiping, insertion, or deformable-object handling plausibly meet these conditions. Collecting complementary, factor-specific data may therefore help these tasks as well. We plan to explore such extensions in future work, as they are beyond the scope of this study. Within the tested setting, the method is not tied to one policy class: gains hold for ACT, Diffusion Policy, and OpenVLA (Tables 1–2). We have added these conditions to the Section 5 Conclusion.

In summary, CrossWorld goes beyond conventional co-training by leveraging real interactions without navigation labels, and beyond same-domain world-model policies through cross-domain completion (Tables 2, 9, and 13). We are happy to discuss further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary to All Reviewers

Response Summary to All Reviewers

We thank the Area Chair and all reviewers for their careful reading and constructive comments. The reviewers found the problem practical and the idea of composing complementary partial worlds interesting and novel, noted the consistent gains across three policy backbones, and considered the real-robot study careful. New results and revisions are marked in blue\color{blue}{\text{blue}} in the updated PDF. The major revisions are summarized below; details are given in our response to each reviewer.

▶\color{BrickRed}{\blacktriangleright} Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/


(1) Relation to co-training and world-model policies (Reviewers 8hWw, UWMN). Section 2 now discusses sim-and-real co-training, world-model policies such as VPP, and learned actuator models, and states that conventional supervised co-training cannot use local contact interactions without navigation labels.

(2) Tuned co-training baseline (Reviewer 8hWw). Appendix B.10\color{blue}{\textbf{Appendix B.10}} sweeps the sim:real sampling ratio of direct mixing from 99:1 to 50:50 in Fig. 13\color{blue}{\text{Fig. 13}}. The best ratio reaches 25.0% (ACT) and 30.0% (OpenVLA) phantom success, versus 65.0% and 75.0% for CrossWorld (Section 4.3).

(3) Controls for the source of the gain (Reviewers UWMN, 8hWw). (i) Raw temporal context: the same K=10K=10 history given to the factual ACT policy reaches 10.0% phantom success, versus 65.0% for CrossWorld (Table 15\color{blue}{\text{Table 15}}, Appendix B.11\color{blue}{\textbf{Appendix B.11}}). (ii) Real-to-sim translation: a CycleGAN in the S slot reaches 40.0%, versus 75.0% with the spatially completed view (Table 18\color{blue}{\text{Table 18}}, Appendix B.13\color{blue}{\textbf{Appendix B.13}}; examples in Fig. 18\color{blue}{\text{Fig. 18}}). (iii) A contact-completed view generated from oto_t (Appendix B.2\color{blue}{\textbf{Appendix B.2}}). (iv) The world-model encoder used as the policy encoder (Appendix B.8\color{blue}{\textbf{Appendix B.8}}), with its pretraining data stated in Appendix A.3.

(4) Direct measurements of the world models (Reviewers cgJF, yCbG, UWMN). Appendix C.2\color{blue}{\textbf{Appendix C.2}} shows that changing only the condition of each world model changes mainly its own factor (Table 19\color{blue}{\text{Table 19}}), and that on real scans the estimated spatial state follows the measured probe displacement and is used by the generator during probe motion (Table 20\color{blue}{\text{Table 20}}).

(5) Statistical reporting (Reviewer UWMN). The OpenVLA phantom comparison now covers three training seeds (Table 14, Appendix B.9\color{blue}{\textbf{Appendix B.9}}). Success counts, 95% Wilson intervals and Holm-corrected tests are reported for Table 2 (Table 16\color{blue}{\text{Table 16}}), intervals are added to Tables 9–11 and Fig. 12\color{blue}{\text{Fig. 12}} (originally Fig. 10), and small differences are described as trends. The values of Fig. 6\color{blue}{\text{Fig. 6}} (originally Fig. 4) are listed in Table 17\color{blue}{\text{Table 17}}.

(6) Data cost and simulated force labels (Reviewer UWMN). Section 4.1 and Fig. 9\color{blue}{\text{Fig. 9}} (originally Fig. 7) report sonographer-hours separately from wall-clock time; Dcontact\mathcal{D}_\mathrm{contact} needs no sonographer. Section 4.1 also states how simulated force targets are sampled, and Appendix B.4\color{blue}{\textbf{Appendix B.4}} adds a control that masks their loss.

(7) Clarity of the method (Reviewers yCbG, UWMN). The paper now uses "robotic ultrasound scanning policy" and "observed ultrasound image", and defines B-mode, L4 standard-plane navigation, the phantom and cross-subject evaluation at first use. The Introduction states that each training sample contains an observed image and its completed view and that the two world models do not interact. Section 3.1 summarizes the inputs and outputs of each model, Fig. 2 is simplified, and Figs. 3 and 4\color{blue}{\text{Figs. 3 and 4}} show the two world models. Section 3.6 clarifies that the completed view uses only information up to t−1t-1 and states the fixed slot order.

(8) Scope and applicability (Reviewers cgJF, UWMN, 8hWw). The Abstract states that the human results are within-participant, and Section 5 states the conditions under which cross-world completion applies.

(9) Videos (Reviewers cgJF, yCbG). An anonymous project page, linked on the first page of the PDF, provides a three-minute overview video and closed-loop rollouts of the CrossWorld policy on the real robot and in simulation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.