acceptodds
Under review as a conference paper at ICLR 2027

What Should a World-Action Model Imagine? 4D Geometry as an Intermediate Representation for Control

Abstract

World Action Models (WAMs) imagine future observations to guide control, yet it remains unclear what a model should imagine for action. Most existing WAMs predict RGB futures, where physical structure and motion are entangled with texture, illumination and observer motion. We systematically study 4D geometry as a predictive intermediate representation for control. Rather than asking whether geometric supervision is useful, we ask when and how an imagined geometric future should participate in action generation. We represent the observable 4D state with world-frame pointmaps and scene flow, which jointly describe where visible surfaces are and how the same physical points move, together with wrist-view depth for complementary local geometry. Our controlled studies reveal several consistent findings. First, geometry is most useful when explicitly imagined and made directly accessible to action, outperforming feature alignment and auxiliary-only geometric supervision. Second, geometric and RGB futures are complementary rather than substitutable: removing either condition degrades control. Third, the value of 4D depends critically on its form and interface: world-frame pointmaps and scene flow outperform projective depth-and-optical-flow cues; and geometric supervision should be selected by view-specific reliability. These findings translate into substantial robustness gains. The resulting WAM improves a matched LingBot-VA baseline from 73.73% to 82.73%, with the largest gains under sensor noise, lighting, background, and viewpoint changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, 4D representation, Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VJ2HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what geometric future a world-action model should predict and how that prediction should enter action generation. Using LingBot-VA as a fixed backbone, it compares feature alignment, reconstruction supervision, auxiliary prediction, and explicit conditioning, as well as different geometric signals, camera views, and latent encodings. These comparisons lead to a design that predicts world-frame pointmaps and scene flow for the main view and depth for the wrist view, encodes them through the pretrained video VAE, and conditions actions on the completed predictions. The resulting model improves the matched LingBot-VA baseline on LIBERO-Plus Long from 73.73% to 82.73%, with the largest gains under appearance and viewpoint perturbations, and also improves out-of-distribution performance on real-robot tasks.

Critical strengths
  1. Robustness. Under a carefully matched comparison with the same backbone, training budget, and evaluation episodes, the final model substantially improves over LingBot-VA. The gains are concentrated under appearance and viewpoint perturbations, which is consistent with the paper's motivation.

  2. Views. The paper shows that geometric supervision should depend on the reliability of each view: full 4D supervision on the wrist camera hurts performance, whereas simple wrist depth helps. The teacher-error visualization and gradient analysis offer a plausible explanation, making this a practically useful finding for multi-camera WAMs.

  3. Practicality. Geometric teachers are used only to construct offline training targets, so the deployed policy requires only RGB sensing and no online reconstruction model. The real-robot experiments cover observer motion, moving objects, and deformable manipulation, extending the evaluation beyond simulation.

Critical weaknesses
  1. Technical contribution. The paper does not discuss Flex-π[1], which already encodes pointmaps with the same frozen video VAE used for RGB, generates future pointmaps jointly with actions that attend to them, and shows that geometric streams help both as training supervision and when generated at inference. This prior work anticipates the paper's first finding and its encoding result, leaving world-frame scene flow, view-specific supervision, and conditioning on completed predictions as the main distinctions. These distinctions, along with related work such as PointAction, should be discussed explicitly, and Flex-π should be evaluated as a baseline on LIBERO-Plus to establish what the proposed design adds.

  2. Mechanism. The paper attributes its robustness gains to actions reading a geometric future that is unaffected by appearance changes, but its own results point to a different explanation. Removing action access to imagined RGB performs worse than the RGB-only baseline, auxiliary-only prediction recovers most of the gain, and even pure-noise geometry still outperforms RGB-only (75.35% vs. 73.73%), suggesting that much of the improvement may come from geometric supervision shaping shared representations while actions still rely primarily on RGB. A per-perturbation breakdown of these variants is needed to test this: if explicit geometry drives the gains, they should degrade most under lighting, background, and sensor-noise perturbations.

  3. Claims. The abstract states that the value of 4D "depends critically on its form and interface," yet all explicit-geometry variants perform similarly and far above the RGB-only baseline. The matched depth + optical-flow alternative comes close to the final model, and both-view native 4D underperforms several projective variants. The evidence therefore supports the value of dense geometric prediction in general more strongly than a specific advantage of world-frame pointmaps and scene flow, and the claims should be calibrated accordingly.

  4. Presentation. The paper is overly dense: the introduction compresses four findings into one long paragraph, and Figure 1 combines architecture diagrams, attention masks, benchmark plots, and rollouts at a scale that is hard to read. Because the experiments switch among native, pseudo-RGB, both-view, and main-view configurations, it is difficult to tell which conclusions were verified on the final model. A clearer statement of the main takeaways and a single table summarizing each experiment's configuration would substantially improve readability.

[1] Yan, Ge, et al. "Flex-$\pi $: A Multi-Stream World-Action Model with Compute Flexibility." arXiv preprint arXiv:2608.10860 (2026).

I think these weaknesses can be addressed during the rebuttal period. I lean to a positive rating if all my concerns are addressed

Questions for authors
  1. How do the authors interpret the corruption study, where heavy noise on the geometric condition barely affects success? Does this indicate that actions depend little on the specific content of the predicted geometry?

  2. What motivates conditioning actions on completed geometric predictions rather than jointly denoising them with actions, as in Flex-π?

7 comments
AuthorsOpenReview ↗

W1: Relationship to Flex-π and PointAction

We added Flex-π and PointAction to the discussion, recognizing their precedents for geometric futures and frozen video-VAE encoding. Our contribution examines three choices that make these futures effective for control:

  1. World-frame structure and motion. Pointmaps locate surfaces; scene flow explicitly supervises displacement of corresponding physical points. Under common simulator GT, they reach 84.08% versus 82.22% for depth and optical flow, retaining an advantage without main-view teacher differences (Appendix A.2.5, Table 12).
  2. View-specific supervision. With native main-view 4D fixed, wrist 4D, no wrist geometry, and wrist depth give 79.77%, 80.52%, and 81.68%. Reconstruction and gradient diagnostics motivate retaining local depth while avoiding unreliable wrist-motion targets (Table 5(b); Appendix A.3.1).
  3. Direct access to completed scene futures. Our completed-future interface reaches 82.33% versus 72.41% for a retrained joint-denoising control, with 20 steps per modality (Appendix A.2.4, Table 11). Flex-π reads evolving visual futures. PointAction also completes its rollout first, then segments robot points for an embodiment-specific decoder. Our action model directly reads completed RGB and scene-geometry latents alongside history.

These controls use the same 2,519 LIBERO-Plus Long cases and support the design choices within our backbone. Appendix A.6, Table 15 distinguishes the shared choices and interfaces.

We added the Full comparison in Table R11: our policy succeeds on 8,287 of 10,030 episodes using RGB visual observations. Flex-π's reported LIBERO-Plus configuration additionally uses camera-frame pointmaps from simulator depth, which is optional in its broader framework.

Table R11. Full LIBERO-Plus comparison under released instruction handling. Both rows use filename-derived instructions for non-language perturbations; Flex-π's score is its reported full-joint result.

MethodFull SR (%)
Flex-π, full joint (reported)80.9
Ours82.62

Our RGB-observation policy exceeds this published result under shared instruction handling. Flex-π separately reports 88.6% after switching to each task's language annotation (Appendix E); we distinguish the protocols in Appendix A.1.4 and Table 10. This comparison demonstrates competitive system performance, while our internal controls explain the value of the design choices.

AuthorsOpenReview ↗

W2 / Q1: Geometric supervision and inference-time use of geometry

We added a geometry-swap test and a Full-benchmark breakdown. They support both geometric supervision and inference-time access to scene-matched content.

Does action use geometry beyond its training benefit? In the Long native-4D controls, geometric supervision alone raises success from 73.73% to 77.54%; allowing action to read predicted geometry further raises it to 79.77%. The Full generalist shows a similar 2.19-point explicit-access gain in Table R12, reproducing the supervision-plus-access pattern beyond the Long-specialist setting.

We replaced all geometric streams at each action chunk with another episode's complete prediction from the same task but a different initial state. The recipient retains its RGB imagination and history; geometry keys and values are recomputed in that context. With checkpoint, sampling, cases, and evaluation seed fixed, Long success falls from 79.77% to 69.91% (1,761/2,519). This 9.86-point drop shows that action uses scene-matched geometric content (Appendix A.2.7, Table 13(b) of the revised manuscript).

Why does pure-noise geometry still permit useful control? We train action with noisy geometry alongside RGB and history. The moderate-noise plateau is consistent with this design. In the separate probe using teacher geometry reconstructed from predicted future RGB, pure noise reduces success from 80.41% to 75.35%, a 5.06-point loss (Figure 7; Table 13(a)). The retained RGB pathway and geometric training help explain why performance remains above RGB-only; tolerance to corruption coexists with content dependence.

Removing RGB-future access gives 71.86% on Long, consistent with the importance of LingBot-VA's pretrained RGB-to-action interface. This variant retains observed history, RGB generation, and RGB-to-geometry conditioning. The best control uses both future pathways.

Table R12 provides the requested perturbation breakdown for the Full generalist over all 10,030 episodes, separate from the Long interventions above.

Table R12. Geometric supervision and future access on LIBERO-Plus Full. Success rates are percentages; the last column is the explicit-access gain over auxiliary-only supervision in percentage points.

PerturbationRGB-only SR (%)Auxiliary-only SR (%)Explicit native 4D SR (%)No RGB-future access SR (%)Explicit − auxiliary (pp)
Background73.1473.0273.0263.690.00
Camera47.1559.2060.1453.93+0.94
Language91.1893.1692.1183.14−1.05
Lighting81.5384.6787.5984.86+2.92
Layout88.2884.2986.8773.46+2.58
Robot initialization87.6686.7790.5979.30+3.82
Sensor noise57.8267.2672.8468.51+5.58
Overall (pooled)74.8278.1580.3472.19+2.19

Explicit access helps most under sensor noise, with gains under lighting, robot initialization, and layout changes. Background shows no additional gain, while language decreases slightly. Thus, benefits vary by perturbation and extend beyond appearance changes. The noise and lighting results, alongside recognizable predicted geometry in Figure 12, support geometry as a complementary structural and motion reference under degraded RGB appearance. Section 4.2 and Appendices A.2.6–A.2.7 and A.3.3 contain the interventions and interpretation.

AuthorsOpenReview ↗

W3: Why geometric representations and interfaces matter

We agree that depth and optical flow are strong alternatives. We revised the abstract and discussion to distinguish the overall geometry benefit from the further gains of representation and encoding choices.

The both-view native result shows why supervision reliability matters. We observe spurious wrist motion on stationary objects and noisier wrist gradients that conflict with main-view geometry (Figure 5; Appendix A.3.1). Replacing wrist 4D with wrist depth raises success from 79.77% to 81.68% with native main-view 4D fixed. The original gradient probes also show stronger geometric signals and better early-layer alignment with RGB and action under pseudo-RGB encoding (Appendix A.3.2), offering an optimization explanation for that interface.

We added the controls in Table R13. Common GT derives both main-view representations from the same simulator geometry and correspondences, fixing wrist depth and pseudo-RGB encoding; we trained each method once. For encoding, we retrained native models with aligned tokens, timestep sampling, loss totals, and training/sampling budgets, retaining the original pseudo-RGB references.

Table R13. Representation and encoding controls on LIBERO-Plus Long. Gains are in percentage points. Repeated-training entries are mean ± SD across three training-seed means, each averaging three evaluation seeds.

ComparisonSettingReference SR (%)Selected SR (%)Gain (pp)
Pointmap + scene flow vs. depth + optical flowCommon simulator GT82.2284.08+1.87
Pointmap + scene flow vs. depth + optical flowRepeated training, wrist depth81.48 ± 0.3183.04 ± 0.43+1.56
Pseudo-RGB vs. nativeAligned encoding, no wrist geometry80.3581.42+1.07
Pseudo-RGB vs. nativeAligned encoding, wrist depth81.7882.73+0.95

The common-GT control and repeated-training comparison both favor pointmaps and scene flow with wrist depth. The repeated representation gain is substantially larger than the observed training-seed SDs, supporting a reproducible benefit. Budget alignment also preserves the encoding advantage in both wrist settings. We therefore retain world-frame pointmaps and scene flow with reliable view supervision and a compatible encoding to provide useful action conditions. The common-GT and aligned-encoding controls are described in Appendix A.2.5, Table 12 of the revised manuscript.

Reviewer_3YJsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how predicted geometry should enter a world-action model. Starting from LingBot-VA, it predicts future RGB, main-view pointmaps and scene flow, and wrist-view depth, and conditions action generation on the completed RGB and geometric predictions. Geometric targets are produced offline; deployment uses RGB observations without running the reconstruction teacher. The study compares auxiliary supervision, feature alignment, explicit geometric prediction, attention interfaces, geometric representations, and encoding choices. The strongest controlled result is an increase from 73.73% to 82.73% on LIBERO-Plus Long. Broader evaluations cover standard LIBERO, LIBERO-Plus, RoboTwin, and three physical-robot tasks.

Critical strengths
  1. The distinction between learning from geometric supervision and actually using predicted geometry for action is useful and experimentally addressed. In Table 4, explicit native-4D prediction reaches 79.77%, compared with 77.54% when geometry-to-action attention is removed and 73.73% for the RGB-only baseline. Removing current RGB-future conditioning also substantially hurts performance. These controls make the contribution more informative than simply adding another prediction target.

  2. The robustness improvement over the matched RGB baseline is substantial and is broken down by perturbation type: +9.00 percentage points overall on LIBERO-Plus Long, with particularly large gains for sensor noise, lighting, background, and camera changes. The real-robot experiments add useful evidence across observer motion, external object motion, and deformable manipulation. I also appreciate that the paper distinguishes external benchmark scores from its controlled comparisons and explicitly acknowledges the weaker layout result. These observations support a useful contribution even if some finer-grained design claims remain provisional.

Critical weaknesses
  1. The evidence for the individual representation/interface design principles is less decisive than the overall improvement over RGB-only control. In Table 5, pointmaps plus scene flow beat depth plus optical flow by only 0.22 points without wrist depth and 1.08 points with it; the encoding gains are about one point. No repeated-training uncertainty is supplied for these comparisons. Moreover, Appendix A.2.2 states that the native and pseudo-RGB paths differ in tokenization, stream layout, and timestep sampling, while A.2.3 notes that the direct-head attachment layer and parameter budget were not retained. Different reconstruction teachers also complicate attributing the 3D-versus-projective result specifically to the coordinate representation. The appendix is appropriately candid, but the main text sometimes reads as establishing general design principles more strongly than these experiments allow. The gradient diagnostics are suggestive rather than an isolation of the cause of improved control.

  2. The cost and reproducibility of explicit geometric imagination are not characterized sufficiently to assess its practical advantage over the auxiliary-only alternative. Appendix A.5.1 reports approximately 0.36 s per chunk on an RTX 4090, but does not provide a matched latency, memory, or compute comparison for the RGB-only, auxiliary-only, and final models, or enough detail about the optimized sampler to reproduce that number. The auxiliary-only model can omit geometry at inference and is only 2.23 points below the native explicit model. This tradeoff matters to the central question of whether geometry should be imagined at test time. The real-robot evidence is encouraging but cannot resolve small method differences: there is one training run for Mobile/Dynamic and 50 trials per condition, with configuration-correlated repeats acknowledged in the appendix.

Questions for authors
  1. Can you provide repeated-training results for the key Table 4 attention comparison and Table 5 representation comparison, together with paired episode outcomes or uncertainty estimates? In particular, do the 0.22-1.08 point representation differences persist across training seeds?

  2. Can the native versus pseudo-RGB comparison be repeated with identical timestep sampling and a matched token/compute budget? For the 3D versus depth/optical-flow comparison, what evidence separates the effect of the representation from differences in teacher accuracy or adaptation? If such controls are unavailable, which conclusions should be restricted to the tested implementations?

  3. Please report the world/action denoising steps, precision, token counts, and measured end-to-end chunk latency and memory for the RGB-only, auxiliary-only, and final models on the same hardware. Does explicit geometry remain preferable at a matched inference budget? These are clarification/validation requests for the current claims, rather than requests for a new application domain.

7 comments
AuthorsOpenReview ↗

Q1 / W1: Repeated training of the key ablations

We repeated the key attention and representation configurations from the manuscript's Tables 4 and 5 with three training seeds, each evaluated with three evaluation seeds on the same 2,519 LIBERO-Plus Long cases. Each evaluation pools all cases. We average the three evaluations within each training seed, then report mean ± SD across the three training-seed means in Tables R4 and R5.

Table R4. Repeated-training attention comparison. Success rates are mean ± SD (%).

ConfigurationSR (%)
RGB-only LingBot-VA73.29 ± 0.55
Auxiliary-only 4D supervision77.90 ± 0.34
Explicit native 4D conditioning80.07 ± 0.32
Explicit native 4D, no RGB-future → action72.10 ± 0.37

Explicit access retains a 2.17-point advantage over auxiliary-only, close to the original 2.23 points. Removing the RGB future again reduces performance below RGB-only. We therefore find that geometric supervision and direct geometric access both help, while RGB remains a complementary future condition.

Table R5. Repeated-training representation and encoding comparison. Success rates are mean ± SD (%).

Main-view signal and encodingNo wrist geometry (%)With wrist depth (%)
Depth + optical flow, pseudo-RGB80.95 ± 0.3481.48 ± 0.31
Pointmap + scene flow, native80.41 ± 0.3381.55 ± 0.32
Pointmap + scene flow, pseudo-RGB81.25 ± 0.3583.04 ± 0.43

Table R6 compares the original design-specific differences with the repeated-training mean differences.

Table R6. Original and repeated-training differences. All differences are in percentage points.

ContrastOriginal difference (pp)Repeated mean difference (pp)
Pointmap + scene flow vs. depth + optical flow, no wrist+0.22+0.30
Pointmap + scene flow vs. depth + optical flow, wrist depth+1.08+1.56
Pseudo-RGB vs. native, no wrist+0.90+0.84
Pseudo-RGB vs. native, wrist depth+1.05+1.49

With wrist depth, the representation gain is 1.56 points, compared with 1.08 originally, and the encoding gain is 1.49 points; the corresponding training-seed SDs are 0.31–0.43 points. Without wrist depth, the representation difference remains small at 0.30 points.

We reproduce the main improvements across independent training and evaluation repeats, including the complete model's 9.75-point mean gain over RGB-only. The overall, explicit-access, and wrist-depth design gains substantially exceed the observed training variability. This provides strong evidence that the main improvements are stable and reproducible, reflecting a systematic benefit of geometric future conditioning rather than a favorable original training run.

AuthorsOpenReview ↗

W1 / Q2: Controlling reconstruction-teacher quality

We added a common-GT control to separate the representation choice from main-view reconstruction-teacher quality. We derive world-frame pointmaps and scene flow, and camera-space depth and optical flow, from the same simulator geometry and physical-point correspondences before normalization.

Both models share the pseudo-RGB interface, frozen Wan VAE, 48 geometry tokens per latent frame, wrist-depth teacher targets, and LingBot-VA initialization. We train one model per method for 10K optimizer updates with global batch size 32, then evaluate each on the same 2,519 LIBERO-Plus Long cases with evaluation seed 0, 20 imagination denoising steps, and 50 action steps. GT supplies training targets only; at evaluation, the policy predicts geometry from RGB. Table R7 reports the results.

Table R7. Common-GT representation comparison on LIBERO-Plus Long. The gain is calculated from success counts before rounding.

Main-view supervisionSuccesses / 2,519SR (%)Gain (pp)
GT depth + optical flow2,07182.22—
GT pointmap + scene flow2,11884.08+1.87

We retain a 1.87-point advantage with common main-view labels, exceeding the original 1.08-point difference with wrist depth. The representation benefit thus persists after controlling the teacher difference, supporting world-frame structure and physical displacement as useful control targets under the shared encoding interface.

We added this control to Section 4.2 and Appendix A.2.5, Table 12.

AuthorsOpenReview ↗

W1 / Q2: Native versus pseudo-RGB encoding under aligned budgets

We retrained native encoding without wrist geometry and with wrist depth, aligning its token counts, timestep-sampling rule, loss totals, and training/sampling budgets to the corresponding original pseudo-RGB references.

Both interfaces share the data, teacher, LingBot-VA initialization, 10K optimizer updates, global batch size 32, BF16, 20 imagination denoising steps, and 50 action steps. Main-view geometry uses 32 tokens per latent frame, with 16 additional wrist-depth tokens when present. Timesteps are sampled independently per latent frame and actual geometric stream with the same scheduler; main-view loss coefficients total 0.2, plus 0.1 for wrist depth. Native retains a joint main-view stream and its bridge; pseudo-RGB retains separate pointmap and scene-flow streams. We train each native configuration once with seed 42 and evaluate with seed 0 on the same 2,519 LIBERO-Plus Long cases. Table R8 compares these runs with the original pseudo-RGB results.

Table R8. Aligned native encoding versus the original pseudo-RGB references.

Wrist geometryNative successes / 2,519Native SR (%)Pseudo-RGB SR (%)Pseudo-RGB gain (pp)
None2,02480.3581.42+1.07
Depth2,06081.7882.73+0.95

Aligning these settings changes native performance by less than 0.2 points and preserves an approximately one-point pseudo-RGB advantage in both wrist settings. We therefore retain the complete pseudo-RGB interface. Our original gradient probes offer an optimization explanation: pseudo-RGB has a 31× stronger geometry-to-action gradient ratio in block 14 and block-0 alignment with RGB (cosine 0.47) and action (0.26), consistent with compatibility with the pretrained backbone (Figure 6; Appendix A.3.2).

The aligned control appears in Section 4.2 and Appendix A.2.5, Table 12.

Reviewer_M7j3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how explicit geometric imagination can improve the robustness of World-Action Models (WAMs) for robotic control. Building upon LingBot-VA, the proposed method jointly predicts future RGB observations and geometric representations, including world-frame pointmaps, 3D scene flow, and wrist-view depth, which are directly accessible to the action generation module. Experiments show that this approach outperforms other approaches, i.e., auxiliary geometric supervision and feature alignment, while RGB and geometric predictions provide complementary information. The method improves the matched LingBot-VA baseline from 73.73% to 82.73% on LIBERO-Plus Long, with notable gains under sensor noise, lighting changes, background variations, and viewpoint shifts.

Critical strengths
  • Explicitly predicting geometric information and making it directly accessible to the action module is reasonable. The ablation studies provide empirical evidence that this design can improve control beyond auxiliary geometric supervision alone.

  • The paper explores various geometric representations, conditioning strategies, and perturbation settings, including real-robot tasks. The results provide useful insights into the benefits of geometric imagination, particularly under appearance and viewpoint changes.

Critical weaknesses
  • Some parts of the paper, particularly the architectural details, attention patterns, and experimental variants, are a little bit difficult to follow. Clearer explanations of the key concepts, notation, and motivations behind the design choices would improve readability.

  • The proposed method demonstrates meaningful improvements over LingBot-VA, but it remains unclear whether the approach generalizes to other state-of-the-art WAM architectures. Evaluating it on another backbone would help establish whether the benefits reflect a broadly applicable.

  • Although explicitly predicting geometry and making it accessible to the action module is intuitive, some architectural choices appear primarily motivated by empirical exploration. A more unified conceptual explanation of why these choices are effective, and when they should generalize, would strengthen the contribution.

Questions for authors
  • Could the authors report the additional inference latency, memory requirements, or computational cost compared with LingBot-VA? This would help clarify the trade-off between computational overhead and performance gains, especially for real-time robotic control.
6 comments
AuthorsOpenReview ↗

W1: Architecture, attention, and notation

We reorganized Section 3 around what to imagine, how to encode it, and how action accesses it, and clarified the notation and experimental variants.

1. Architecture. We predict main-view world-frame pointmaps and scene flow for structure and motion, together with wrist depth for local geometry; wrist depth avoids the unreliable wrist-motion targets illustrated in Figure 5. The final model encodes these targets as pseudo-RGB through the same frozen Wan VAE as RGB. Separate modality projections feed a shared backbone, jointly trained for RGB, geometry, and action. Offline teachers construct the training targets; the deployed policy uses RGB observations and generated latent futures.

2. Attention and notation. In Figure 3, zz, gg, and aa denote RGB, geometry, and action token blocks; subscripts distinguish history from the current chunk. Primes mark prediction streams. During training, action prediction a1′a'_1 reads RGB conditioning z1z_1 and noisy geometry g1′g'_1, alongside history and self-attention. We retain z1z_1 as LingBot-VA's visual conditioning interface, including its noise augmentation. Reading g1′g'_1 trains action under varied geometric noise levels, encouraging tolerance to imperfect finite-step predictions. The current action conditioning copy a1a_1 supplies later chunks, but is hidden from a1′a'_1.

Geometry reads the evolving RGB prediction z1′z'_1: each RGB pass supplies features to geometry within the same world-denoising loop. We block geometry-to-RGB attention to avoid direct interference with the pretrained RGB pathway; geometric losses still update the shared backbone. Current actions cannot condition world prediction. At inference, completed RGB and geometry fill the future-conditioning roles, and action is denoised against these fixed predictions and history. Appendix A.4.4 gives the full query/key sets and training–inference correspondence.

3. What the variants test. Table R1 separates geometric supervision from action access. These native-4D controls retain identical observed RGB/action history; the final pseudo-RGB model uses the full access pattern with the encoding and targets described above. Appendix A.2.2 maps all configurations.

Table R1. Geometric supervision and future access on LIBERO-Plus Long. Native-4D attention controls and the RGB-only reference, corresponding to Table 4(b) in the revised manuscript.

VariantGeometry supervisionFutures read by actionSR (%)
RGB-only, Fig. 9(a)NoRGB73.73
Auxiliary-only, Fig. 9(b)YesRGB77.54
Without RGB-future access, Fig. 9(c)YesGeometry71.86
Full native 4D, Fig. 3YesRGB and geometry79.77

Supervision alone improves control, while direct geometric access adds another 2.23 percentage points. Removing RGB-future access reduces success, even though RGB is still generated and read by geometry. We therefore let action read both complementary futures. Section 3.3 and Appendix A.2.3 now connect these comparisons to the attention choices and their motivations.

AuthorsOpenReview ↗

W2: Transfer to OpenWAM

We adapted our 4D-future design to OpenWAM-α, retaining its video DiT and separate ActionDiT. We added main-view world-frame pointmaps, scene flow, and wrist depth through OpenWAM's own frozen video VAE, and jointly trained RGB, geometry, and action. At inference, ActionDiT reads completed RGB–geometry futures. Deployment uses RGB observations and latent conditions, without online teachers or geometry decoding.

We trained one adapted model on clean demonstrations using OpenWAM's LIBERO fine-tuning recipe and evaluated all 10,030 LIBERO-Plus cases: one trial per case, evaluation seed 10000, and official_global initialization. Table R2 compares it with the published OpenWAM-α LIBERO-Plus release.

Table R2. Transfer of geometric future conditioning to OpenWAM-α on full LIBERO-Plus.

PerturbationOpenWAM-α, published SR (%)Ours (OpenWAM-α based), SR (%)Gain (percentage points)
Camera33.840.90+7.10
Robot76.182.45+6.35
Language88.091.67+3.67
Light97.097.99+0.99
Background87.193.40+6.30
Noise39.855.72+15.92
Layout77.577.64+0.14
Overall (7,541 / 10,030)69.275.18+5.98

We observe gains across all seven OpenWAM perturbation categories. The key trends agree with LingBot-VA (revised Table 3): noise benefits most, camera and background also improve substantially, and layout benefits little (−1.92 points on LingBot-VA, +0.14 here). OpenWAM's small light gain is consistent with its already high 97.0% baseline. Geometry particularly improves its weakest categories, camera and noise, raising overall success by 5.98 points.

These gains in a shared-backbone WAM and a WAM with a separate action module support the generality of our 4D-future design across the two evaluated architectures. Section 4.1 and Table 1 now report the transfer results, with adaptation and evaluation details in Appendix A.1.5.

AuthorsOpenReview ↗

W3: Complementary future representations for control

We added a unified interpretation in Section 5: RGB and geometry form complementary generative pathways on a shared backbone. RGB predicts appearance and semantic state; geometry predicts surface positions and motion. Action uses both imagined futures.

Why generate geometry? Denoising makes future structure and motion explicit learning targets across noise levels, and produces a concrete condition for action. The native-4D controls on LIBERO-Plus Long support both roles. Explicit imagination reaches 79.77%, compared with 75.63% for direct reconstruction supervision and 77.42% for Geometry Forcing (revised Table 4(a)). With the same geometric supervision, enabling action access improves success from 77.54% to 79.77%; replacing that geometry with another episode's prediction lowers it to 69.91% while retaining RGB (Tables 4(b) and 13(b)). We therefore find value in both learning geometry and reading its scene-matched future content.

Why does this help robustness? RGB prediction must model textures, illumination, and shadows alongside manipulation-relevant structure. Pointmaps and scene flow directly describe where surfaces are and how they move. Our interpretation is that this complementary prediction task gives action a more stable structural reference when appearance changes. Figure 12 illustrates the distinction: predicted pointmaps preserve recognizable cup and arm structure under zoom blur, while scene flow separates the withdrawing hand from the released cup under changed lighting. The large gains under sensor noise, lighting, and background changes in Table 3 agree with this interpretation.

This reasoning also guides the interface choices. We use main-view 4D and wrist depth for reliable global motion and local structure, avoiding the spurious wrist-motion targets in Figure 5. Pseudo-RGB connects these targets to the pretrained Wan encoding interface, and completing RGB and geometry before action provides fixed future conditions throughout action denoising. The resulting design makes geometric learning and geometric content directly useful for control. We expect its strongest robustness benefit when appearance changes but the underlying physical interaction remains recoverable; Appendix A.3.3 provides the qualitative cases.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall revision and new evidence

We sincerely thank all reviewers for their time, careful reading, and constructive feedback. Their comments have helped us improve the clarity of the paper and strengthen its experimental support.

We have organized the revisions around the paper's central question: how to shape a WAM's internal representations to better capture physical structure and motion, and how to make these representations useful for action generation. The main revisions clarify this argument and strengthen its experimental and practical foundations.

1. Clearer narrative and contribution (Reviewers M7j3 and VJ2H).

We revised the introduction, method, and discussion to connect representation learning with action generation. Geometric prediction provides structured supervision to shape the shared backbone's internal representations of physical structure and motion. At inference, the generated geometric future latents provide explicit, scene-matched conditions for action alongside RGB. These two roles guide the design questions: what geometry to predict, how to encode it, and how action should access it. Revised figures, notation, and a configuration matrix make this reasoning and the factors varied in each comparison easier to follow. Addressing VJ2H's contribution concern, we clarify the shared foundations with Flex-π and PointAction and focus our contribution on world-frame structure and motion targets, view-specific supervision, and direct action access to completed complementary futures.

2. Controlled analysis of representation and action conditioning (Reviewers 3YJs and VJ2H).

For representation and encoding, we added common simulator ground truth to control main-view teacher differences and aligned token counts, timestep sampling, and training and sampling budgets for the native versus pseudo-RGB comparison. Pointmaps and scene flow retain an advantage over depth and optical flow, while the pseudo-RGB advantage remains after budget alignment.

For action conditioning, swapping in another episode's geometry reduces success, showing that action benefits from scene-matched geometric content. The perturbation breakdown shows how supervision and explicit-access gains vary across conditions, while completed-future conditioning outperforms the retrained joint-denoising variant. These analyses support both the geometric targets and encoding used to shape the WAM's representations, and the action interface that makes scene-matched, completed RGB–geometry futures useful for control.

3. Generalization across WAM architectures (Reviewer M7j3).

We adapted the geometric targets, encoding, and completed-future conditioning to OpenWAM-α and trained the resulting model while retaining its separate action module. It reaches 75.18% on Full LIBERO-Plus, versus the published 69.2% reference, a gain of 5.98 percentage points. Gains in both the shared-backbone LingBot-VA and OpenWAM support the generality of geometric future conditioning across the two evaluated WAM architectures.

4. Inference cost and practical deployment (Reviewers M7j3 and 3YJs).

We added matched latency and peak allocated memory measurements for RGB-only, auxiliary-only, native explicit geometry, and the final model under common RTX 4090 deployment settings. Relative to RGB-only LingBot-VA, the final model adds 30.0 ms (9.14%) and 0.513 GiB. Its mean cycle fits the existing action-buffer window, giving a favorable performance–cost trade-off while retaining RGB-only sensing and asynchronous execution.

5. Consistency across training runs (Reviewer 3YJs).

We repeated the key simulation ablations and the Dynamic real-robot comparison across independent training runs. The main gains remain in the repeated-training averages, reinforcing the consistency of the findings beyond the original trained models.

Revisions are marked in blue.

We hope the revised manuscript and our detailed responses address the reviewers' concerns. We welcome any further questions or suggestions and would be happy to provide additional clarification. We look forward to your feedback and continued discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.