acceptodds
Withdrawn from ICLR 2027

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies have shown strong performance in language-conditioned robotic manipulation, but standard action imitation provides limited supervision for learning explicit 3D geometry and future scene evolution. We propose GaussianDream, a feed-forward 3D Gaussian world model that introduces structured spatial and temporal supervision into VLA policy learning. GaussianDream inserts a set of learnable queries into the policy encoder to form a latent GaussianDream prefix. During training, the prefix is decoded by a current reconstruction head and a future prediction head to model the observed 3D scene and its short-horizon evolution. The current branch is supervised with RGB rendering and pseudo depth, while the future branch additionally uses future RGB, pseudo depth, and pseudo 3D scene-flow supervision. At inference time, all auxiliary reconstruction and prediction heads are removed, and only the learned GaussianDream prefix is retained for action generation, requiring no test-time Gaussian reconstruction or future rollout. GaussianDream achieves 98.4% success on LIBERO, 87.0% on LIBERO-Plus, and 54.8% on RoboCasa Human-50. We further validate GaussianDream on real robots using both single-arm and dual-arm platforms, achieving 50.0% average success on the single-arm tasks and 40.0% success on the higher-precision dual-arm manipulation task.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Feed-Forward Gaussian Representation, Robotic Manipulation, VLA, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XavBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GaussianDream extends π0.5 with 1024 extra prefix tokens. A 12-block temporal module (TGE) builds these tokens from frozen VGGT features of three past frames. During training, two heads decode the tokens. One head predicts per-pixel Gaussians for the current frame, and the other predicts their 3D displacements for the next 1–5 steps. The heads are supervised with RGB rendering, pseudo-depth and pseudo scene flow. At inference the heads are dropped, but VGGT, TGE and the extra tokens are kept.

Critical strengths
  1. Good and reliable gains over π0.5, including on a real robot. This matters because it shows the overall recipe works beyond saturated simulation benchmarks, so the paper has real practical value.

  2. A practical, training-only design. The pseudo-depth and pseudo-flow targets come from off-the-shelf models, so the recipe needs no simulator access and transfers to real data. The paper also reports results that complicate its story and releases detailed code. This makes the work easy to build on.

Critical weaknesses
  1. The central claim, that the Gaussian world model drives the gains, is not supported. Compared with π0.5, the model adds VGGT features, a frame history, 1024 extra tokens and the Gaussian losses. Every ablation in Table 4 keeps the first three. The variants that each drop one loss still reach 85.9–86.2% on LIBERO-Plus, against 87.0% for the full model and 84.6% for π0.5, so most of the gain survives whichever loss is removed. The remaining 0.8–1.1-point differences come from single runs and are smaller than the spread across checkpoints in Table 5 (85.6–87.0%). The flow loss carries by far the largest weight (15, against 0.1–0.2 for the other losses) but is never ablated alone. Better world modelling also does not help: the higher-PSNR checkpoint and exact simulator geometry (GaussianDream-Sim) both give lower success. This weakness is decisive because the title, method and contributions all rest on this attribution. If a VGGT prefix trained with the action loss alone matches GaussianDream, the paper's main idea is not the source of its results.

  2. Described accurately, the method is close to prior work that it does not cite. The Gaussians are pixel-aligned, rendered back to a single view, and their base colours are copied from the input. The method therefore reduces to depth and scene-flow regression. The future head is not conditioned on actions and is unused at inference. Because it is trained on expert demonstrations, it learns where things move under the expert, which is closer to auxiliary action supervision than to a world model. The Robot dimension of LIBERO-Plus fits this reading: GaussianDream falls 4.5 points below π0.5 there (72.1 vs. 76.6). Under this description, the closest precedents are uncited. ManiGaussian (ECCV 2024) supervises a policy with an action-conditioned dynamic Gaussian world model. GAF (ICRA 2026) predicts feed-forward Gaussians plus per-Gaussian displacements that reuse the current appearance, which is the paper's second contribution. DreamVLA (NeurIPS 2025) adds future depth and motion prediction to a VLA on LIBERO. GAP (CVPR 2026) jointly predicts actions and future 3D geometry. This weakness matters because, without the world-model framing, the contribution over these works is not established.

Questions for authors

Requests that could change my rating:

  1. This question is decisive for my rating. What do the no_future_tokens preset (VGGT prefix, no auxiliary losses) and the base_vla preset achieve on LIBERO-Plus and RoboCasa, under the same training budget as the full model and with 2–3 seeds?
  2. What do the no_flow preset and a direct depth-and-flow regression without Gaussians achieve? This would show whether the Gaussian parameterization matters.

Clarifications:

  1. In Table 2, the baselines appear to be averaged over the 24 tasks, but GaussianDream over the three categories. Computed per task, GaussianDream scores 53.8, which is on par with Being-H0.5 (53.9). Please confirm this.
  2. Table 1 lists Spatial Forcing at 97.6, but the original paper reports 98.5. Table 1 also leaves out Cosmos-Policy and LingBot-VA, which both reach 98.5 in Table 3. Where does each baseline number come from? Does the claim of the best LIBERO result still hold?
  3. What training budget did π0.5 and the ablations use? App. B says 60K steps, while the ablation script uses 30K. How was the 60K checkpoint selected?
  4. Why does performance drop under robot initial-state perturbations on LIBERO-Plus?

Suggestions (not required for my assessment): An action-conditioned future head would test whether world modelling, rather than motion supervision, helps. Reframing the paper as an analysis of which auxiliary signals help VLAs would match the evidence better. The depth probe in Fig. 4a should report metrics over many images, and it would ideally compare against models whose features were not trained to predict the same pseudo-depth.

Reviewer_mTAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GaussianDream augments a VLA policy with a shared 3D Gaussian representation. It reconstructs the current scene and predicts future primitive displacements using RGB, pseudo-depth, and scene-flow supervision during training; the auxiliary decoding heads are removed at inference. The paper evaluates the approach on LIBERO, LIBERO-Plus, RoboCasa, and physical single-arm and dual-arm robots.

Critical strengths
  1. The shared Gaussian representation gives the visual prediction tasks a coherent geometric basis. Predicting displacement of the same primitives links current scene structure to future motion, which is a sensible way to provide training signals for manipulation.
  2. The empirical evaluation spans several simulation benchmarks and two physical robot settings. The reported gain is particularly meaningful on RoboCasa, where success rises from 40.1% for the base policy to 54.8%, and on single-arm tasks, where the category average rises from 34.4% to 50.0%.
  3. The paper examines current reconstruction, future prediction, and individual supervision signals, and it removes auxiliary decoding heads for deployment. These analyses make the design reasonably transparent and reveal that better reconstruction quality does not automatically translate into better control.
Critical weaknesses
  1. The methodological novelty relative to the closest predictive Gaussian approaches remains moderate. Prior work already uses Gaussian prediction as auxiliary supervision and removes its decoder at inference. The specific advantage of evolving a shared set of primitives, rather than combining established geometry and motion objectives, needs clearer isolation.
  2. The ablations do not fully attribute the improvement to Gaussian supervision. A matched control retaining the geometry encoder, temporal inputs, aggregation module, and training budget without the proposed losses is missing. The reported future-prediction gain is about one percentage point on LIBERO-Plus, so the separate effect of this component remains uncertain.
  3. The evidence for small improvements would be stronger with repeated training runs and a clearer checkpoint-selection protocol. The paper reports single-point success rates for the main comparisons, while reconstruction quality and task success do not improve together across the examined checkpoints. This makes the claimed link between representation quality and control less conclusive.
  4. The practical cost is incompletely characterized. Removing auxiliary heads reduces deployment work, but reported inference latency remains 531 ms, compared with 268 ms for the base policy. The contribution of the retained geometry and temporal modules, as well as preprocessing and training cost, should be stated to support the efficiency claim.
Questions for authors
  1. What precisely distinguishes the shared-primitive prediction from the closest predictive Gaussian supervision methods, including GeoPredict? Please identify the contribution that remains when the backbone and auxiliary training signals are held fixed.
  2. Can the authors provide a control that retains VGGT, history frames, temporal aggregation, and an equal training budget while removing Gaussian supervision? What is the additional benefit of future displacement prediction under this control?
  3. How stable are the smaller gains across training seeds, and how was the reported checkpoint chosen without using test performance? Please clarify which baselines were retrained under matched data and optimization settings.
  4. Do the model size and latency measurements include online geometry encoding and temporal processing? What are the full training, preprocessing, and inference costs relative to the base policy?
Reviewer_RxjXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GaussianDream, which uses 3D scene reconstruction and future prediction to improve VLA policies. It provides spatial and temporal supervision through a shared Gaussian representation, while removing the auxiliary heads at inference. Experiments on LIBERO, LIBERO-Plus, RoboCasa, and real robots show improvements over the base policy.

Critical strengths
  1. Predicting how a scene changes is useful for manipulation, and using the 3D Gaussians for current reconstruction and future motion provides an intuitive way to supervise VLA representations.
  2. The method derives depth and scene-flow supervision from RGB demonstrations, avoiding the need for ground-truth depth or scene-flow labels.
Critical weaknesses
  1. The added value beyond GeoPredict and ManiGaussian is unclear, as the ablations do not isolate Gaussian supervision from VGGT, TGE, and observation history.

  2. The averaging in Table 2 appears inconsistent: task weighting gives approximately 53.8% for GaussianDream, rather than 54.8%, which may affect the reported ranking.

Questions for authors
  1. A clearer comparison with GeoPredict and ManiGaussian would help identify the new components and the evidence supporting their benefits.
  2. A comparison that feeds projected VGGT tokens and queries directly into the VLM, without TGE but retaining Gaussian supervision, would clarify TGE's contribution.
  3. Results without Gaussian supervision during training, while retaining VGGT and TGE under a comparable training budget, would clarify the contribution of the auxiliary objectives.
  4. A single-frame model trained and evaluated under otherwise matched conditions would help quantify the benefit of the three-frame context.
  5. The averaging rule in Table 2 needs clarification, including how 54.8% is obtained and whether the same weighting is applied to all methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.