- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies what geometric future a world-action model should predict and how that prediction should enter action generation. Using LingBot-VA as a fixed backbone, it compares feature alignment, reconstruction supervision, auxiliary prediction, and explicit conditioning, as well as different geometric signals, camera views, and latent encodings. These comparisons lead to a design that predicts world-frame pointmaps and scene flow for the main view and depth for the wrist view, encodes them through the pretrained video VAE, and conditions actions on the completed predictions. The resulting model improves the matched LingBot-VA baseline on LIBERO-Plus Long from 73.73% to 82.73%, with the largest gains under appearance and viewpoint perturbations, and also improves out-of-distribution performance on real-robot tasks.
Critical strengths
-
Robustness. Under a carefully matched comparison with the same backbone, training budget, and evaluation episodes, the final model substantially improves over LingBot-VA. The gains are concentrated under appearance and viewpoint perturbations, which is consistent with the paper's motivation.
-
Views. The paper shows that geometric supervision should depend on the reliability of each view: full 4D supervision on the wrist camera hurts performance, whereas simple wrist depth helps. The teacher-error visualization and gradient analysis offer a plausible explanation, making this a practically useful finding for multi-camera WAMs.
-
Practicality. Geometric teachers are used only to construct offline training targets, so the deployed policy requires only RGB sensing and no online reconstruction model. The real-robot experiments cover observer motion, moving objects, and deformable manipulation, extending the evaluation beyond simulation.
Critical weaknesses
-
Technical contribution. The paper does not discuss Flex-π[1], which already encodes pointmaps with the same frozen video VAE used for RGB, generates future pointmaps jointly with actions that attend to them, and shows that geometric streams help both as training supervision and when generated at inference. This prior work anticipates the paper's first finding and its encoding result, leaving world-frame scene flow, view-specific supervision, and conditioning on completed predictions as the main distinctions. These distinctions, along with related work such as PointAction, should be discussed explicitly, and Flex-π should be evaluated as a baseline on LIBERO-Plus to establish what the proposed design adds.
-
Mechanism. The paper attributes its robustness gains to actions reading a geometric future that is unaffected by appearance changes, but its own results point to a different explanation. Removing action access to imagined RGB performs worse than the RGB-only baseline, auxiliary-only prediction recovers most of the gain, and even pure-noise geometry still outperforms RGB-only (75.35% vs. 73.73%), suggesting that much of the improvement may come from geometric supervision shaping shared representations while actions still rely primarily on RGB. A per-perturbation breakdown of these variants is needed to test this: if explicit geometry drives the gains, they should degrade most under lighting, background, and sensor-noise perturbations.
-
Claims. The abstract states that the value of 4D "depends critically on its form and interface," yet all explicit-geometry variants perform similarly and far above the RGB-only baseline. The matched depth + optical-flow alternative comes close to the final model, and both-view native 4D underperforms several projective variants. The evidence therefore supports the value of dense geometric prediction in general more strongly than a specific advantage of world-frame pointmaps and scene flow, and the claims should be calibrated accordingly.
-
Presentation. The paper is overly dense: the introduction compresses four findings into one long paragraph, and Figure 1 combines architecture diagrams, attention masks, benchmark plots, and rollouts at a scale that is hard to read. Because the experiments switch among native, pseudo-RGB, both-view, and main-view configurations, it is difficult to tell which conclusions were verified on the final model. A clearer statement of the main takeaways and a single table summarizing each experiment's configuration would substantially improve readability.
[1] Yan, Ge, et al. "Flex-$\pi $: A Multi-Stream World-Action Model with Compute Flexibility." arXiv preprint arXiv:2608.10860 (2026).
I think these weaknesses can be addressed during the rebuttal period. I lean to a positive rating if all my concerns are addressed
Questions for authors
-
How do the authors interpret the corruption study, where heavy noise on the geometric condition barely affects success? Does this indicate that actions depend little on the specific content of the predicted geometry?
-
What motivates conditioning actions on completed geometric predictions rather than jointly denoising them with actions, as in Flex-π?
7 comments
W2 / Q1: Geometric supervision and inference-time use of geometry
We added a geometry-swap test and a Full-benchmark breakdown. They support both geometric supervision and inference-time access to scene-matched content.
Does action use geometry beyond its training benefit? In the Long native-4D controls, geometric supervision alone raises success from 73.73% to 77.54%; allowing action to read predicted geometry further raises it to 79.77%. The Full generalist shows a similar 2.19-point explicit-access gain in Table R12, reproducing the supervision-plus-access pattern beyond the Long-specialist setting.
We replaced all geometric streams at each action chunk with another episode's complete prediction from the same task but a different initial state. The recipient retains its RGB imagination and history; geometry keys and values are recomputed in that context. With checkpoint, sampling, cases, and evaluation seed fixed, Long success falls from 79.77% to 69.91% (1,761/2,519). This 9.86-point drop shows that action uses scene-matched geometric content (Appendix A.2.7, Table 13(b) of the revised manuscript).
Why does pure-noise geometry still permit useful control? We train action with noisy geometry alongside RGB and history. The moderate-noise plateau is consistent with this design. In the separate probe using teacher geometry reconstructed from predicted future RGB, pure noise reduces success from 80.41% to 75.35%, a 5.06-point loss (Figure 7; Table 13(a)). The retained RGB pathway and geometric training help explain why performance remains above RGB-only; tolerance to corruption coexists with content dependence.
Removing RGB-future access gives 71.86% on Long, consistent with the importance of LingBot-VA's pretrained RGB-to-action interface. This variant retains observed history, RGB generation, and RGB-to-geometry conditioning. The best control uses both future pathways.
Table R12 provides the requested perturbation breakdown for the Full generalist over all 10,030 episodes, separate from the Long interventions above.
Table R12. Geometric supervision and future access on LIBERO-Plus Full. Success rates are percentages; the last column is the explicit-access gain over auxiliary-only supervision in percentage points.
| Perturbation | RGB-only SR (%) | Auxiliary-only SR (%) | Explicit native 4D SR (%) | No RGB-future access SR (%) | Explicit − auxiliary (pp) |
|---|---|---|---|---|---|
| Background | 73.14 | 73.02 | 73.02 | 63.69 | 0.00 |
| Camera | 47.15 | 59.20 | 60.14 | 53.93 | +0.94 |
| Language | 91.18 | 93.16 | 92.11 | 83.14 | −1.05 |
| Lighting | 81.53 | 84.67 | 87.59 | 84.86 | +2.92 |
| Layout | 88.28 | 84.29 | 86.87 | 73.46 | +2.58 |
| Robot initialization | 87.66 | 86.77 | 90.59 | 79.30 | +3.82 |
| Sensor noise | 57.82 | 67.26 | 72.84 | 68.51 | +5.58 |
| Overall (pooled) | 74.82 | 78.15 | 80.34 | 72.19 | +2.19 |
Explicit access helps most under sensor noise, with gains under lighting, robot initialization, and layout changes. Background shows no additional gain, while language decreases slightly. Thus, benefits vary by perturbation and extend beyond appearance changes. The noise and lighting results, alongside recognizable predicted geometry in Figure 12, support geometry as a complementary structural and motion reference under degraded RGB appearance. Section 4.2 and Appendices A.2.6–A.2.7 and A.3.3 contain the interventions and interpretation.
W3: Why geometric representations and interfaces matter
We agree that depth and optical flow are strong alternatives. We revised the abstract and discussion to distinguish the overall geometry benefit from the further gains of representation and encoding choices.
The both-view native result shows why supervision reliability matters. We observe spurious wrist motion on stationary objects and noisier wrist gradients that conflict with main-view geometry (Figure 5; Appendix A.3.1). Replacing wrist 4D with wrist depth raises success from 79.77% to 81.68% with native main-view 4D fixed. The original gradient probes also show stronger geometric signals and better early-layer alignment with RGB and action under pseudo-RGB encoding (Appendix A.3.2), offering an optimization explanation for that interface.
We added the controls in Table R13. Common GT derives both main-view representations from the same simulator geometry and correspondences, fixing wrist depth and pseudo-RGB encoding; we trained each method once. For encoding, we retrained native models with aligned tokens, timestep sampling, loss totals, and training/sampling budgets, retaining the original pseudo-RGB references.
Table R13. Representation and encoding controls on LIBERO-Plus Long. Gains are in percentage points. Repeated-training entries are mean ± SD across three training-seed means, each averaging three evaluation seeds.
| Comparison | Setting | Reference SR (%) | Selected SR (%) | Gain (pp) |
|---|---|---|---|---|
| Pointmap + scene flow vs. depth + optical flow | Common simulator GT | 82.22 | 84.08 | +1.87 |
| Pointmap + scene flow vs. depth + optical flow | Repeated training, wrist depth | 81.48 ± 0.31 | 83.04 ± 0.43 | +1.56 |
| Pseudo-RGB vs. native | Aligned encoding, no wrist geometry | 80.35 | 81.42 | +1.07 |
| Pseudo-RGB vs. native | Aligned encoding, wrist depth | 81.78 | 82.73 | +0.95 |
The common-GT control and repeated-training comparison both favor pointmaps and scene flow with wrist depth. The repeated representation gain is substantially larger than the observed training-seed SDs, supporting a reproducible benefit. Budget alignment also preserves the encoding advantage in both wrist settings. We therefore retain world-frame pointmaps and scene flow with reliable view supervision and a compatible encoding to provide useful action conditions. The common-GT and aligned-encoding controls are described in Appendix A.2.5, Table 12 of the revised manuscript.
W1: Relationship to Flex-π and PointAction
We added Flex-π and PointAction to the discussion, recognizing their precedents for geometric futures and frozen video-VAE encoding. Our contribution examines three choices that make these futures effective for control:
These controls use the same 2,519 LIBERO-Plus Long cases and support the design choices within our backbone. Appendix A.6, Table 15 distinguishes the shared choices and interfaces.
We added the Full comparison in Table R11: our policy succeeds on 8,287 of 10,030 episodes using RGB visual observations. Flex-π's reported LIBERO-Plus configuration additionally uses camera-frame pointmaps from simulator depth, which is optional in its broader framework.
Table R11. Full LIBERO-Plus comparison under released instruction handling. Both rows use filename-derived instructions for non-language perturbations; Flex-π's score is its reported full-joint result.
Our RGB-observation policy exceeds this published result under shared instruction handling. Flex-π separately reports 88.6% after switching to each task's language annotation (Appendix E); we distinguish the protocols in Appendix A.1.4 and Table 10. This comparison demonstrates competitive system performance, while our internal controls explain the value of the design choices.