acceptodds
Under review as a conference paper at ICLR 2027

DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

Abstract

Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Vision-Language-Action Models, World Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6J3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DexForesight, a framework that transfers predictive spatial representations into dexterous policies. An action-conditioned world model uses training-time depth and kinematic hand geometry to predict persistent hand-point displacements and visual feature changes. Its frozen query features supervise policy queries, which also condition action generation. Deployment requires only RGB, language, and robot state, without world-model execution or future decoding. Experiments on 11 simulation tasks and three physical-robot tasks report improved success rates, supported by component ablations and linear probing of future hand-motion information in deployed policy features.

Critical strengths
  1. The predictive targets combine articulated hand motion with observed scene changes. Persistent points derived from robot kinematics provide supervision for occluded fingers, complementing scene depth and visual features.

  2. The framework transfers predictive representations into action-conditioning features without retaining the world model or future decoders at deployment. This provides a practical way to exploit richer training-time information while preserving a lightweight control pipeline.

  3. Controlled ablations compare query alignment with direct prediction, vary action conditioning, and examine depth and hand-motion supervision. Architecture-matched linear probing further supports the claim that future hand-motion information remains accessible in the deployed policy.

Critical weaknesses
  1. Physical-robot evaluation is limited to three tasks with 20 trials per task, and success-rate uncertainty is not reported. The results are encouraging, but provide limited evidence for the stability of the 8.3-point advantage over the strongest baseline or generalization beyond the tested variations.

  2. Comparisons with external WAM baselines involve different architectures and training procedures. Although the internal ablations are controlled, these comparisons do not isolate whether query alignment is more effective than alternative predictive-training strategies under comparable training resources.

  3. The teacher relies on offline depth and kinematics-derived hand targets, but sensitivity to errors in these training signals is not evaluated. The no-depth ablation establishes the benefit of including depth, but does not show how reliably the method transfers predictive features when geometric supervision is imperfect.

Questions for authors
  1. How sensitive is downstream policy performance to the quality of offline depth and kinematics-derived hand targets? A controlled perturbation of these training signals would help establish whether the reported benefits remain reliable under imperfect geometric supervision.

  2. How much does visual-dynamics supervision contribute beyond hand-flow supervision in the teacher? A hand-flow-only teacher under the same training settings would help distinguish the benefit of learning joint hand–scene dynamics from that of learning articulated hand motion alone.

Reviewer_jq4QOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose DexForesight, an imitation learning method that includes a world model. The trained policy considers multi-view RGB, language, and robot state, while the world model additionally has depth, and hand geometry and robot history. The method was evaluated on both simulated and real robot tasks.

Critical strengths

The problem is very relevant, and the authors evaluated it on real-world robot tasks. The method shows better performance on interesting tasks in comparison to the baselines.

Critical weaknesses

It is difficult to understand the contributions and the novelty of the papers. The proposed approach combines several, mostly existing, approaches that are mostly naturally fit together.

Questions for authors
  1. Could you please clarify the novelty of the proposed method?
  2. Why are some of the baselines considered in real-robot tasks not used in the simulated evaluations?
  3. Have the baselines for the simulated tasks also been fined on the collected?
Reviewer_XVpPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DexForesight, a method that enables dexterous manipulation policies to acquire spatial foresight, i.e., the ability to anticipate how the hand and the surrounding scene change under the robot’s actions. The authors identify two challenges: parts of the hand are often occluded from the cameras, and existing approaches either depend on depth estimation at deployment to exploit spatial information or rely on explicit future generation, which adds computation. DexForesight first trains an action-conditioned spatial world model that additionally takes scene depth estimated offline and fixed points on the hand computed through the robot’s forward kinematics, which cover occluded fingers. Given the demonstrated actions, the world model predicts 3D hand-point displacements and visual feature changes at future timesteps. The world model is then frozen, and the policy’s latent representations are aligned with those of the world model and used to condition action generation. At deployment, the policy operates only on RGB observations, language, and robot state, without the world model or depth input. The method is evaluated on 11 DexJoCo simulation tasks and three real-robot tasks, where the authors report success rates of 60.5% in simulation and 58.3% on the real robot, compared with 50.9% and 50.0% for the strongest respective baselines among existing VLA and world action models.

Critical strengths

1. Practical use of training-time signals with minimal deployment cost.

DexForesight uses offline scene depth, kinematic hand geometry (including occluded fingers), and recorded future states only to train the world model, and transfers its representations into the policy through latent alignment. The deployed policy requires neither depth input nor future generation, yet improves simulation success by 11.1 percentage points over its backbone, while the reported latency increase on the real robot is less than 5% relative to the base policy. This is consequential because it moves the cost of spatial and predictive information from deployment to training, in a form applicable to existing VLA policies.

2. Controlled evidence that what is aligned matters.

Shuffling the teacher's action conditioning reduces success below the baseline without alignment (41.5% vs. 49.4%), aligning with pre-action world queries yields only 52.7%, and directly attaching prediction heads with the same data and update budget reaches at most 53.1%, compared with 60.5% for DexForesight. This is consequential because it shows that the action-conditioned content of the teacher representation matters, distinguishing the method from simply adding auxiliary prediction objectives.

Critical weaknesses

1. The contributions of the architectural change and the alignment content are not disentangled.

Beyond the alignment loss, DexForesight adds a policy-query pathway that reads multiple VLM layers and feeds the resulting tokens to the action expert. Even without alignment, this pathway is trained by the action objective and gives GR00T's action expert, which otherwise reads a single VLM layer, additional information and parameters, so it may improve performance on its own. However, the simulation baseline (49.4%) lacks this pathway, and the same architecture trained without the alignment loss is not reported. The shuffled-action and pre-action-query experiments show that what is aligned matters, but not how much of the overall gain comes from the architectural change; shuffled conditioning (41.5%) reflects misleading rather than absent supervision. It is also unclear whether "QwenVLA action-only" in the linear probe, described as sharing the same architecture, is the QwenVLA reported in Table 2. Without this reference point, the central claim that the gains come from transferring action-conditioned predictive representations is not fully supported.

2. The key motivations are not directly validated.

The method is motivated by (i) the need for spatial information beyond visible observations due to finger occlusion, and (ii) the computational cost of explicit future generation at deployment. Neither is directly validated.

First, both the deployed policy and the world model already receive joint states, from which the hand points are deterministically computed through forward kinematics. The contribution of kinematic hand geometry thus seems to lie less in providing the positions of occluded fingers than in explicitly representing hand–scene relations in 3D. Yet there is no ablation removing the hand-geometry input, no analysis comparing occluded and visible segments, and no failure analysis supporting the claim in Section 4.2 that failures reflect finger occlusion.

Second, no baseline is confirmed to generate the future at deployment: Fast-WAM skips future generation at test time, all real-robot methods deploy without it, and the setting of Cosmos3 is not specified. Fast-WAM's controlled comparison shows that imagine-then-execute variants achieve similar or slightly higher success at 3–4× the latency, yet where DexForesight lies on this success–latency trade-off for dexterous manipulation is not shown.

Consequently, the experiments support a contribution closer to "which training-time information the aligned teacher representation is learned from" than to "resolving finger occlusion" or "the advantage of implicit over explicit future modeling."

Questions for authors

1. Policy-query pathway without alignment. What is the success rate of a policy that keeps the same policy-query pathway but is trained only with the action objective, without the alignment loss? If "QwenVLA action-only" in the linear probe corresponds to such a model, is it the same model as QwenVLA in Table 2 (30.0%)?

2. Comparison with explicit future generation at deployment. Have the authors compared DexForesight on DexJoCo with world action models that generate the future at deployment (e.g., the Joint or IDM variants of Fast-WAM) in terms of both success rate and latency? In addition, does Cosmos3 generate the future at deployment in the simulation experiments?

3. Role of hand geometry and occlusion. What is the result when the hand-geometry input is removed from the world model and only the robot-state history is used? Have the authors analyzed performance or probe error separately for segments where fingers are occluded versus visible, or categorized failure cases by cause?

4. Context depth tokens. Why does using context depth tokens (42.0%) perform worse than using no offline depth (49.7%)?

5. Information about occluded objects. In dexterous manipulation, the state of objects occluded by the hand is also important. Hand points cover occluded fingers through forward kinematics, but no corresponding information is available for objects: depth covers only surfaces visible from the main camera view, and the visual target captures only features of visible regions. How is the state of occluded objects represented in the world model? What information do the authors believe accounts for the improvement on Insert Battery, which is described as testing alignment under object occlusion?

Minor comments

  • Figures 1 and 2 share the same caption title ("DexForesight overview"). Distinct titles reflecting each figure's role would be clearer.
  • In Figure 2, the demonstrated actions are labeled "G.T. Action." Since they serve as conditioning inputs rather than prediction targets in the world model, "demonstrated controls," as in the main text, would be more accurate. In addition, the figure does not show how the actions enter the query adapter to produce the action-conditioned world queries or how they are also provided to the two prediction experts, making it difficult to see where the actions are used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.