acceptodds
Withdrawn from ICLR 2027

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, World Models, Vision-Language-Action Models, Semantic 3D Scene Modeling, Action-Conditioned Future Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jD2jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForeAct3D augments a vision-language-action policy with auxiliary prediction of current and future depth, three-class semantic masks, and wrist-camera pose, with future queries conditioned on the policy-generated action chunk. Background staticity, instance-level rigid alignment, and kinematic pose supervision regularize the shared policy representation; future predictions are not required at inference. Without robot pretraining, the method reports 98.3% average success on LIBERO, 3.73 average task length on CALVIN ABC→D, and 37.8% average success across three real-world tasks.

Critical strengths
  1. Tables 1–2 report 96.6% → 98.3% on LIBERO and 2.94 → 3.73 average task length on CALVIN, with matched training configurations stated in Sections 4.1 and 4.3. Evaluation covers 2,000 LIBERO episodes and 1,000 CALVIN sequences, with CALVIN testing on unseen environment D (Appendix A.3), providing useful evidence across both task diversity and environmental shift.

  2. The real-world experiments corroborate the overall benefit with explicit outcome definitions, where with 50 demonstrations per task, total successes increase from 3/45 to 17/45 across placement, insertion, and sequential manipulation. Distinguishing target entry from final retention, and intermediate completion from full-task success, makes the practical improvement more informative than a single aggregate success rate.

Critical weaknesses
  1. Sections 3.1 and 3.4 condition forecasting on policy-generated actions. But they do not establish whether future targets come from executing those actions or from the recorded demonstration, nor whether forecasting gradients pass through the action output. If a demonstration endpoint supervises a different generated chunk, the target need not represent that chunk’s consequence; Tables 4(b) and 9 demonstrate the usefulness of the action-feature pathway without resolving this issue or testing different executed actions from the same state. This is a substantial, potentially fixable gap in the central scientific interpretation.

  2. Equation (3) and Appendix A.1 fit Kabsch transforms between identical pixel coordinates sharing an instance ID, although these generally need not observe the same material point after motion. Instances with fewer than 32 overlapping pixels are excluded. The authors acknowledge this approximation, but provide no moving-instance coverage or correspondence-sensitive validation showing that the loss preserves rigid geometry under the evaluated motions.

  3. Appendix A.4 selects predicted and target points at ground-truth future instance pixels, thereby supplying the object’s future image support, and excludes instances with insufficient valid predicted depth from recall without penalty. The predicted-mask variant still retains ground-truth instance IDs and lacks established identical filtering across conditions (Appendix A.5). On CALVIN, its reported error rises from 1.25 to 3.49 cm and recall falls from 92.8% to 65.9% (Table 8). These transparently disclosed qualifications substantially limit how strongly the localization scores validate the proposed future-state representatio.

  4. The contribution beyond established geometric auxiliary learning is insufficient. DreamVLA already uses query-based future depth and semantic forecasting with training-only decoders, while GeoPredict combines predictive robot kinematics and 3D geometry with training-only geometric decoding.

  5. No training-seed variation is reported, although the LIBERO closure and action-conditioning increments are only 0.6 and 0.4 percentage points; each real-world cell contains 15 trials, and the long-task action-conditioning gain is 1/15 → 3/15.

Questions for authors
  1. What actions generate each future training target, and how are discrepancies between recorded and policy-generated chunks handled?

  2. What fraction of moving instances receives the rigidity penalty, and does it improve geometric consistency when measured using valid temporal correspondences? Does the kinematic term impose any constraint beyond increasing the coefficient of the existing labeled pose loss?

  3. Are forecasting comparisons evaluated on identical held-out observation/action/target windows? Can the main action-conditioning comparison report fixed instance populations, excluded-instance coverage, and performance relative to copying the current scene, especially for objects that move?

  4. How does a matched current-only depth/semantic/pose auxiliary baseline compare with current-plus-future supervision?

  5. What are the exact backbone checkpoint and action-head configuration, and how stable are the principal policy and component gains across training seeds?

Reviewer_ZGCDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ForeAct3D introduces action-conditioned future 3D scene supervision into vision-language-action (VLA) policy training. Learnable geometric queries decode depth, three-class semantic segmentation (background/robot/object), and wrist-camera pose at the current time and at the end of the action chunk rom shared policy representations. The future-prediction branch additionally receives the policy-generated action chunk. Background staticity, instance rigidity, and end-effector kinematics regularize the predicted current and future geometry. Future-scene prediction serves as an auxiliary training objective and is not required during deployment. The paper reports improved policy performance on LIBERO, CALVIN, and three real-robot manipulation tasks, alongside geometric evaluations and component ablations.

Critical strengths
  1. Semantic 3D prediction, physical-consistency regularization, and action conditioning all serve the goal of improving action-relevant shared representations. Their roles are well motivated and reasonably integrated. Rather than supervising only future images or latent features, the paper explicitly structures the predicted 3D geometry and links future prediction to the policy's own actions.
  2. Relative to StarVLA-OFT, average success rises from 96.6% to 98.3% on LIBERO, average consecutive task completion from 2.94 to 3.73 on CALVIN, and total real-robot successes from 3/45 to 17/45 (Tables 1, 2, and 5). All four LIBERO suites improve. The CALVIN StarVLA-OFT score of 2.94 is reproduced by the authors (marked ∗ in Table 2), as discussed further in m1.
  3. Future 3D prediction serves mainly as training-time auxiliary supervision. At deployment, the VLA directly predicts actions without running the future-prediction branch or performing world-model-based planning. This is a practical design choice.
Critical weaknesses
  1. The future-prediction branch is conditioned on a policy-generated action chunk, whereas its target is the future state observed in the dataset trajectory. If the predicted action differs from the demonstrated action, the observed future may not correspond to the consequence of the predicted action. The paper does not sufficiently specify which output of the Action Head is used during training, whether the action features are detached, or whether the geometric prediction loss propagates gradients back through the action-conditioning pathway into the Action Head.
  2. The paper lacks a simple persistence baseline that assumes current scene geometry remains unchanged at the future time in the fixed external-camera view or world frame. For example, one could copy the relevant geometric components of the current predicted state, and evaluate them against the actual future under the stated protocol. This would help separate performance on genuinely changing scene content from performance attributable to largely static content.
  3. The paper does include an action-removal ablation showing that action inputs improve future-geometry accuracy on the evaluated data (Table 4(b), Appendix A.6). However, these results primarily measure prediction of observed futures. They do not directly show whether, from the same initial scene, varying valid action chunks produces distinct predictions that match the corresponding outcomes of actual execution.
Questions for authors
  1. Which action output of the Action Head conditions the future queries during training? How is potential mismatch with the observed demonstration future handled when the predicted action differs from the demonstrated action? Does the future-geometry loss backpropagate through action features into the Action Head, or is stop-gradient applied?
  2. Could the authors hold several initial scenes fixed, input multiple valid action chunks with distinct physical consequences for each scene, and compare predictions with the outcomes of executing those actions? This would be more direct than simply removing action conditioning. Zero-action or unrelated-action substitutions may serve as supplementary sensitivity checks, but could introduce out-of-distribution inputs and should not replace comparisons among valid actions.
  3. Could the authors add a persistence baseline (copying predicted current geometry into the future in a fixed-camera or world-coordinate representation) and stratify errors by actual object displacement? Could they also analyze the association between prediction quality and task success, rather than treating a difference between segmentation IoU and success rate as evidence of no relationship?
Reviewer_PwAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ForeAct3D, a VLA training framework in which predicting future geometry serves as an auxiliary learning objective. A readout from the policy's shared features reconstructs the current scene and predicts a future state, both represented by depth, three-class semantic masks, and wrist-camera pose. To condition the forecast on the planned motion, the future readout also receives the policy-generated action chunk. Background-staticity and instance-rigidity losses relate the two scene states, while robot kinematics provides supervision for the wrist-camera pose. These objectives train the shared policy representation without requiring future prediction at inference. Experiments on LIBERO, CALVIN, and three real-world manipulation tasks show improvements over the base policy.

Critical strengths
  1. Policy improvements are observed in simulation and on real hardware.

The full model improves upon StarVLA-OFT on LIBERO and CALVIN, with real-world experiments covering placement, insertion, and sequential manipulation. The gains in task success support geometric prediction as a useful auxiliary training signal in these settings.

  1. Targeted ablations support the consistency losses.

Removing individual consistency losses while retaining geometric prediction and action conditioning reduces policy performance, most notably for instance rigidity. These ablations provide evidence for the contribution of geometric regularization.

Critical weaknesses
  1. The contribution of future prediction itself is not isolated.

The cumulative ablation introduces current and future geometric supervision together, without a matched current-state-only baseline. The reported gains therefore do not isolate the benefit of forecasting from that of geometric auxiliary supervision, leaving the central future-modeling claim insufficiently supported.

  1. The roles of the semantic-3D components remain unclear.

Depth, segmentation, and pose supervision are introduced together, leaving their individual contributions unclear, particularly the value of coarse three-class segmentation.

Appendix A.1 states that scene supervision and the kinematics term share the same pose error with additive coefficients. Removing the kinematics term thus changes the effective pose-loss weight rather than removing pose supervision; it does not isolate the pose branch's contribution. The lack of reported variability across independent training runs further limits the interpretation of small incremental gains.

Questions for authors
  1. What scope of physical consistency is supported by the proposed constraints?

The introduction motivates the method through contact and object motion, but the constraints relate two predicted endpoints without directly assessing the intervening interaction. The rigidity loss also assumes same-pixel instance correspondences, with acknowledged limitations under substantial motion. Could the authors clarify the scope of their physical-consistency claim and assess the constraints' reliability and coverage under motion and occlusion?

  1. What accounts for the benefit of action conditioning?

Are gradients stopped at the policy-generated action chunk before it enters the future branch? Otherwise, this branch also introduces a gradient path to the action head. Do the predictions respond appropriately to changes in the action chunk when the initial observation is held fixed? Comparing predictions for different executable action chunks with their observed outcomes would help assess this. Please also report the downstream policy ablation on CALVIN, where the current action-conditioning ablation evaluates future prediction.

  1. Which comparisons support geometric future targets over alternative prediction targets?

Comparisons with DreamVLA and CLOVER use different prediction horizons and do not isolate the contributions of prediction targets and constraints. Could the authors clarify the advantages over RGB-future and latent-future prediction and provide a matched comparison to establish the contribution of geometric targets and constraints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.