- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets robotic ultrasound navigation to the longitudinal L4 standard plane in a setting where complete real expert demonstrations (probe motion + force + B-mode) are expensive. The central assumption is that the dynamics behind image formation split into a spatial part (how probe motion changes the pose relative to anatomy) and a contact part (coupling and deformation as a function of force), and that each part can be learned from a cheaper, partial source. A spatial world model (ViT estimator + transition + DiT generator) is trained in SonoGym with a privileged 30-d spatial descriptor. A contact world model is trained on 512 local force-controlled phantom interactions. During policy training, the contact model generates a "contact-completed" view for every simulated frame, and the spatial model generates a "spatially-completed" view for every real frame from a K=10 history. The policy (ACT, Diffusion Policy, or OpenVLA) always gets the factual frame, the completed view, and a 9-d robot state.
Results: in simulation, success rises by 7.5-10 pp within- and cross-subject for all three backbones (160/40 rollouts). On the real robot with 50 phantom and 60 human demos, CrossWorld reaches 65-75% phantom success (N=20) against at most 25% for direct mixing, and 47-60% on within-participant human trials (N=15, the same 3 participants who supplied demonstrations). The appendix has ablations on completion direction, contact-control strategy, context length K, measured contact conditioning, view dropout, a contact-data pretraining control, and three training seeds for OpenVLA in simulation.
Critical strengths
-
The problem framing is well motivated, and the real-robot study is more careful than most in this area. Splitting supervision so that simulation provides spatial structure and cheap local real interaction provides contact appearance suits ultrasound, where contact-dependent appearance is the hard part to simulate.
- Real-robot success is judged by two clinicians blinded to the policy (kappa = 0.82 before consensus), and evaluation order is randomized in blocks.
- The contact-data pretraining control (Table 13: the same D_contact used for image-to-wrench pretraining only gives 20 -> 25%) and the contact-control strategy comparison (Table 10) were especially useful to me.
- This matters for my assessment because it shows that simply having D_contact doesn't explain the gain.
-
The real-robot effect at the 50-demonstration budget is large and holds across three quite different backbones. If the mechanism holds up, this is a practically meaningful result for a domain where data really is expensive.
- Table 2: 65/70/75% vs. 20/25/25% for direct mixing on the phantom.
- Even with N=20, a gap of 15/20 vs. 5/20 is unlikely to be noise.
Critical weaknesses
-
The completed view mixes "cross-world factor completion" with two simpler signals, and none of the controls separate them. Since the paper's contribution is the factor-completion mechanism, not just the final success rate, I see this as the main issue.
(a) Temporal context. CrossWorld policies get frame history that the factual baselines never see.
- The contact-completed view is G_c(o_{t-1}, a_{t-1}, c~_t) (Eq. 9), a learned function of the previous frame and action. The spatially-completed view at deployment is generated from a K=10 history (Eq. 10). The factual-only baselines get only o_t and r_t.
- The context-length ablation (Fig. 10: 45% at K=1, 75% at K=10, 80% at K=16) fits the idea that much of the real-robot gain comes from history.
- The existing controls don't address this: the augmented view in Table 8 is T(o_t), current frame only, and the independent-view variant (Fig. 5, B.1) uses only o_t at deployment. A factual + frame-stack baseline (o_{t-K..t} and past actions, same backbone and slot interface) is the obvious missing control.
- This also explains a result I otherwise found odd. In simulation the contact condition c~ is sampled from the real bank (Eq. 9; A.2 describes this for training, and I assume evaluation does the same), so it's unclear what contact information the C-view adds at sim test time. Yet it gives +7.5-10 pp in sim, including cross-subject.
(b) Domain-style alignment. At deployment the S slot effectively receives a learned real-to-sim translation of the current scene. That's a valid and useful trick, but it's a different claim from "simulation-derived spatial dynamics complement real observations."
- In training, the S slot always holds simulator-style images (factual sim frames) and the C slot holds real-style images (G_c output, trained on real phantom images). At deployment, the S slot holds G_s output, a generator trained only on sim frames, and the C slot holds the real frame.
- The paper should compare against a plain image-translation baseline (e.g., an unpaired real-to-sim translator feeding the same slot), and show the S-view next to sim frames.
- Relatedly, the spatial estimator E_s is trained only on sim images and applied to real ones, which is exactly the sim-to-real gap the introduction calls hard. There's no check that s^_t on real data is meaningful. I understand there's no ground-truth registration, but even a qualitative or phantom-fiducial check would help.
-
The statistical support is thin for several claims, and the cost argument doesn't quite add up. The headline phantom gap is real, but most of the design-choice comparisons in the appendix rest on differences of one or two rollouts.
- All real-robot numbers come from one trained model per configuration (Sec. A.4: "Each model configuration is trained once using a fixed random seed"), one phantom, N=20 rollouts, and N=15 human rollouts on the same three participants who supplied the 60 human demonstrations.
- A 95% interval for 15/20 is about 50-90%, and for 5/20 about 10-50%. By contrast: 30% vs. 40% in Table 9 (6/20 vs. 8/20), 55% vs. 60% in the K ablation (which is also non-monotonic), and 65% vs. 75% in Table 10. The text treats these as clear evidence for each design choice.
- The super-additivity in Table 9 (25 -> 40 and 25 -> 30, but both together 75) is striking enough that I'd want to see it replicated with another seed.
- On cost: Fig. 7 shows the 512 local contact trajectories took 3.74 h of wall-clock collection, while all 100 phantom expert demonstrations took 1.42 h. Fig. 4 suggests the CrossWorld advantage shrinks or disappears at 100 phantom demonstrations. So in wall-clock terms, 50 more expert demos look cheaper than D_contact.
- The argument may still hold if contact collection doesn't need a sonographer, but the paper should state expert-hours for each source, since "reducing reliance on costly expert demonstrations" is the main motivation.
-
The introduction doesn't explain clearly how the two world models are combined. After reading the contribution list I still didn't know whether they are chained, merged into one model, or used separately. It only becomes clear in Sections 3.5-3.6 that they never interact directly: each produces a second image for the other domain's data, and the policy sees both. One sentence and a pointer to Fig. 2 in the introduction would fix this. Beyond that, the main text reads well, but several facts needed to interpret the results are only in the appendix: the two-slot ordering (sim obs in the S slot, real obs in the C slot; Table 9), the fact that simulated force targets and wrenches are sampled or retrieved from the real contact bank because the sim force channel is a zero placeholder (A.2), and which backbone each appendix ablation uses (Tables 9-11 appear to be OpenVLA, Tables 12-13 ACT, but this is not stated consistently).
Questions for authors
Clarifications that would change my assessment:
-
Can you add a factual-view baseline that gets the same temporal context as CrossWorld (frame stack of K frames + past actions, same backbone, same two-slot interface)? This is the control I most want to see, in sim and on the phantom.
-
What does G_s produce on real inputs? Figs. 6b, 11b and 13 show small S-view thumbnails, but does the S-view look sim-like when placed next to actual sim frames? Did you compare against a pure real-to-sim image translation baseline in the S slot? If translation alone recovers most of the gain, the factor-completion framing needs revising.
-
In simulation evaluation (Table 1), what information do you think the C-view adds, given that the contact condition is sampled from the real bank and the simulator doesn't model contact? Is the gain still there if the C-view is generated from o_t instead of o_{t-1}?
-
A.3 says the ViT encoders are "further pretrained to learn compact ultrasound representations before world-model training." On which data? If it includes real phantom images (expert or contact), the factual baselines should get the same encoder.
-
Please give the Fig. 4 numbers in a table for all budgets and backbones. The text says ACT+CW at 50 "approaches ACT's 70.0% with 100 demonstrations". Is 70% the direct-mixing ACT at n=100? If so, ACT jumps from 20% to 70% between 50 and 100 demos, which I'd like to understand.
-
Does collecting D_contact need a sonographer or other operator in the loop? Please report expert-hours separately from wall-clock time.
-
Can you report at least the main phantom comparison (OpenVLA direct mixing vs. OpenVLA+CW) over 2-3 training seeds, or give confidence intervals for all real-robot results?
-
Simulated force targets f_t^{sim,*} are sampled from the empirical force distribution independently of state (A.2). So the force dimension of the sim action labels is noise relative to the observation. Did you try masking the force loss on sim samples? How does this interact with the learned-force result in Table 10?
-
Related work: the idea of learning the hard-to-simulate contact/low-level part from real interaction and the rest in simulation has a clear precedent in legged locomotion, e.g. the ANYmal work from Hutter's group, where an actuator model learned from real data is combined with policy training in simulation (Hwangbo et al., Science Robotics 2019), and later hierarchical schemes with a sim-trained high level over a separately obtained low-level controller. How does CrossWorld differ from these, beyond the ultrasound image modality? A short discussion would help position the contribution.
Suggestions (not affecting my score): evaluating on held-out human participants, even 2-3, would make the human results much more convincing. Right now they measure within-participant performance only.
10 comments
Response to Reviewer UWMN [ Q2/W1(b) ]
[ Q2 / W1(b) ]. Is the S-view a real-to-sim translation?
No. The S-view keeps the real appearance, a real-to-sim translator in the same slot does not reproduce the gain, and the spatial state behind the S-view tracks real probe motion.
(1) The S-view is not a style translation. Both generators are trained within one domain: predicts simulated frames from the simulated state , and predicts real frames from the measured contact . Neither is trained to map one domain onto the other, which is what CycleGAN learns. On held-out real phantom frames, the S-view is as far from simulation as the raw real frames (KID 204.5 vs. 208.2), while CycleGAN output is close to it (27.6). The S-view also keeps the anatomy of better, although it never sees and CycleGAN does. Side-by-side examples: https://crossworld-5zb.pages.dev/#wm-comparison
(2) A translator does not replace spatial completion. Placing CycleGAN$(o_t)$ in the S slot, with contact completion and all else unchanged, gives 40.0% (8/20), the same as leaving the slot empty, vs. 75.0% with the S-view.
\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{S-view vs. real-to-sim translation.}\\\\ \begin{array}{lccc} \hline \textbf{S slot at deployment}&\text{KID to sim }(\times10^{3})\downarrow&\text{Grad. corr. with }o_t\uparrow&\text{Phantom succ.}\uparrow\\\\ \hline \text{Empty (contact only, Table 9)}&\text{–}&\text{–}&40.0\\\\ \text{CycleGAN}(o_t)&\B{27.6}&\B{0.25}&\B{40.0}\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld S-view}&\B{204.5}&\B{0.59}&\mathbf{75.0}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ **(3) $\hat{s}_t$ carries spatial information on real data.** On held-out real frames, we regenerate $o_t$ with only $\hat{s}_t$ replaced by the mean state, another scan's state at the same step, or the same scan's state 2 s later, keeping $o_{t-1}$ and the noise fixed. On the third of frames with the most probe motion, the scan's own state reconstructs the real frame better than every replacement. On the third with the least motion, the replacements make no difference. $$\def\B{\color{blue}}\def\G{\cellcolor[gray]{0.92}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{Spatial-state intervention on real ultrasound.}\ \Delta\text{PSNR (dB).}\\\\ \begin{array}{lcc} \hline \textbf{Replaced by}&\text{Static frames}&\G\text{Moving frames}\\\\ \hline \text{Mean state}&\B{-0.04}&\G\B{+0.71\ [0.61,0.89]}\\\\ \text{Other scan, same step}&\B{-0.02}&\G\B{+0.53\ [0.40,0.63]}\\\\ \text{Same scan, 2 s later}&\B{+0.02}&\G\B{+0.24\ [0.12,0.36]}\\\\ \hline \end{array}\\\\ \small\text{Positive: the scan's own state reconstructs better.} \end{array}\\\\[-1em] {} \end{multline*}$$ Independently, the longitudinal component of $\hat{s}_t$ follows the cranio-caudal probe displacement measured by the calibrated robot kinematics (Eq. 21), after controlling for lateral motion (median partial $\rho=0.73$, $p<0.005$ against a circular-shift null).Response to Reviewer UWMN [Q3]
[ Q3 ]. What does the C-view add in simulation, and what if it is generated from ?
The C-view adds variation in contact appearance, which the simulator does not provide. When the C-view is generated from , the cross-subject gain remains, while most of the within-subject gain disappears.
(1) What does the C-view add in simulation? Variation in contact appearance. SonoGym renders each frame with a pix2pix network [1] that maps CT slices to ultrasound images, trained on scans from seven ex-vivo spine specimens, with no contact input. Every simulated anatomy is therefore rendered under one fixed, implicit contact condition (Appendix A.2), and a policy trained only on factual frames can fit to it. The C-view renders the same simulated state under varied real contact sampled through : coupling, texture and attenuation change while the anatomy stays in place (, originally Fig. 12). Training with this variation makes the policy less sensitive to contact-dependent appearance, which matters most on held-out anatomies.
(2) Does generating the C-view from preserve the gain? Partly. We generate the C-view from the current frame instead of , with the same trained and sampled contact, and retrain OpenVLA in simulation.
\begin{multline*} {}\\\\[-1em] \begin{array}{c} \textbf{C-view from }o_{t-1}\textbf{ vs. }o_t.\text{ OpenVLA simulation success }(\\%)\text{.}\\\\ \begin{array}{lcc} \hline \textbf{Policy input}&\text{Within-subject}\uparrow&\text{Cross-subject}\uparrow\\\\ \hline \text{Factual view}&78.8&67.5\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld, C-view from }o_{t-1}&\mathbf{86.2}&\underline{77.5}\\\\ \B{\text{C-view from }o_t}&\B{\underline{80.0}}&\B{\mathbf{82.5}}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ With the C-view generated from $o_t$, cross-subject success stays at the CrossWorld level (82.5% vs. 77.5%), so the gain on held-out anatomies is preserved, consistent with the contact variation in (1). Within-subject success is 80.0%, close to the factual view (78.8%). We use $o_{t-1}$ because $G_c$ is trained on consecutive tuples $(o_{t-1}, a_{t-1}, c_t, o_t)$, the only kind our continuous contact collection provides. A model trained for same-frame completion would need the same pose under several contacts, which requires force sweeps at fixed poses plus spatial registration. We will add this control to $\color{blue}{\textbf{Appendix B}}$. **References** - [1] Y. Ao, M. Moghani, M. Mittal, M. Prajapat, L. Wu, F. Giraud, F. Carrillo, A. Krause, and P. Fürnstahl. SonoGym: High performance simulation for challenging surgical tasks with robotic ultrasound. arXiv:2507.01152, 2025.
Response to Reviewer UWMN [ Q1/W1(a) ]
Response to Reviewer UWMN
We thank the reviewer for the careful review, for naming the controls that would change the assessment, and for finding the real-robot study more careful than most in this area. We agree that the contribution is the factor-completion mechanism, not only the final success rate. For each question we give the existing evidence first and then the new controls. New results and revisions are marked in blue, both in this response and in the updated PDF.
▶ Overview video and CrossWorld policy rollouts: https://crossworld-5zb.pages.dev/
No. An existing ablation holds the test-time history fixed and still shows most of the gain, and a new control gives the policy the same raw history without reproducing it.
(1) Existing ablation (Appendix B.3). In Table 9, the spatial-only and full variants receive identical deployment inputs, including the S-view generated from the same K=10 history. The two differ only in whether simulated training samples received contact completion. If history explained the gain, they would perform alike. Instead, success is 30.0% vs. 75.0% (6/20 vs. 15/20, two-sided Fisher exact test, p = 0.010), and the history-based S-view alone stays close to no completion (25.0%).
{}\\\\[-1em] \begin{array}{c} \textbf{Table 9 (excerpt).}\text{ Completion directions on the phantom.}\\\\ \begin{array}{lccc} \hline \textbf{Configuration}&\text{Sim. input }(S,C)&\text{Real input }(S,C)&\text{Ph. Succ.}\uparrow\\\\ \hline \text{No completion}&(o_t^\mathrm{sim}, \mathbf{0})&(\mathbf{0}, o_t^\mathrm{real})&25.0\\\\ \text{Spatial only}&(o_t^\mathrm{sim}, \mathbf{0})&(\hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}}, o_t^\mathrm{real})&30.0\\\\ \rowcolor[gray]{0.92} \textbf{Full CrossWorld}&(o_t^\mathrm{sim}, \hat{o}_t^{\mathrm{sim}\leftarrow\mathrm{real}})&(\hat{o}_t^{\mathrm{real}\leftarrow\mathrm{sim}}, o_t^\mathrm{real})&\mathbf{75.0}\\\\ \hline \end{array}\\\\ \small\mathbf{0}\text{: masked view.} \end{array}\\\\[-1em] {} \end{multline*}$$ **(2) New frame-stack control ($\color{blue}{\textbf{Appendix B.11}}$).** While $\color{blue}{\text{Fig. 12}}$ (originally Fig. 10) shows that spatial completion benefits from temporal context, we test whether directly providing the same raw history to the policy reproduces the gain. The factual policy keeps its backbone (ACT) and two-slot interface. Slot 1 holds $o_t$, Slot 2 holds the $K=10$ past frames $o_{t-10:t-1}$, and the past actions $a_{t-10:t-1}$ are appended to $r_t$. Training data match the factual baselines. $$\def\B{\color{blue}} \begin{multline*} {}\\\\[-1em] \begin{array}{c} {\color{blue}{\textbf{Table 15.}}}\text{ Effect of raw temporal context. ACT success }(\\%)\text{.}\\\\ \begin{array}{lccc} \hline \textbf{Policy input}&\text{Sim. within}\uparrow&\text{Sim. cross}\uparrow&\text{Phantom}\uparrow\\\\ \hline \text{Factual view}&76.9&\underline{50.0}&\underline{20.0}\\\\ \B{\text{Factual + frame stack }(K=10)}&\B{\underline{81.3}}&\B{47.5}&\B{10.0}\\\\ \rowcolor[gray]{0.92} \textbf{CrossWorld}&\mathbf{85.0}&\mathbf{57.5}&\mathbf{65.0}\\\\ \hline \end{array} \end{array}\\\\[-1em] {} \end{multline*}$$ Raw history does not reproduce the gain. In simulation it raises within-subject success from 76.9% to 81.3%, but not cross-subject success (47.5% vs. 50.0%). On the phantom it reaches 10.0% (2/20), far below CrossWorld's 65.0% (13/20, p < 0.001) and not different from the factual view (p = 0.66). Together, these results show that temporal context alone does not explain CrossWorld's gains; contact completion during training is also critical. The simulation part of W1(a), what the C-view adds when the simulator has no contact, is answered under Q3.