acceptodds
Under review as a conference paper at ICLR 2027

DexForesight: Self-Distilled Foresight for Dexterous Vision-Language-Action Models

10 views

Abstract

Task-oriented dexterous manipulation requires coordinated arm–hand actions whose quality is determined by the physical and visual consequences that follow them. Demonstration trajectories therefore contain more supervision than the demonstrated actions alone: they also record how objects move, articulate, and appear after each interaction. We introduce DexForesight, a training framework built on a single observation: a demonstrated future can supervise a causal policy in two fundamentally different ways—a predicted future teaches the representation what interaction state to anticipate, while the realized future teaches the action generator how its current prediction should change. For representation learning, we adapt V-JEPA 2 to demonstration trajectories and design an action-conditioned latent predictor with structured arm–hand controls; its predicted future latent supervises learnable query tokens in the vision-language model (VLM) representation. For action generation, we execute the same policy under causal and privileged future context; their velocity difference defines a future-conditioned flow correction learned by a lightweight residual adapter. Future observations and simulator object states are used only during training and are removed at deployment. Across the 11 single-arm and bimanual tasks of DexJoCo, DexForesight improves the average success rate of the DexJoCo π0.5 baseline by 10.1 percentage points (62.6% vs. 52.5%). Both supervision paths improve the policy in isolation, and their combination performs best; combining physical and visual privileged future context also achieves the strongest average performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Vision-Language-Action Models, Self-Distillation, Foresight Learning, Predictive Representation Learning, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eDodOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DexForesight trains a π0.5-based dexterous VLA with two training-only future signals: an alignment loss that pushes learnable VLM query tokens toward the future latent predicted by an action-conditioned V-JEPA 2 predictor, and a residual adapter distilled from the velocity difference between a privileged forward pass (future object state plus future visual features) and a causal one. On the 11 DexJoCo tasks, the method raises average success from 52.5% to 62.6% under rand-obj, with no extra inputs at deployment.

Critical strengths

The paper separates two ways a demonstrated future can supervise a causal policy: predicted futures shaping the representation, and realized futures shaping action generation.

All privileged inputs, the predictive modules, and the query tokens are removed at deployment. Inference overhead is below 1%, so the method drops into existing π0.5 pipelines easily.

Gains are consistent across rand-obj, rand-full, and multi-task settings. The path and privileged-modality ablations help show where the improvements come from.

Critical weaknesses

The flow-distillation target has no information the causal policy cannot already learn. At the flow-matching optimum, v_base(x_τ, τ | o_t) is the conditional expectation of v_priv given the causal inputs, so E[r_flow | x_τ, τ, o_t] = 0, and an adapter that sees only causal features can only regress toward that zero mean. Any gain must therefore come from v_base being under-fit, meaning the adapter acts as extra capacity or extra training signal rather than distilled foresight; capacity-matched adapters and longer base training are needed to rule this out.

The alignment target is a stack of approximations whose value is not justified. The query tokens regress a frozen predictor’s output, which is conditioned on ground-truth future actions, which in turn is defined in a space trained to mimic pretrained future-frame features. Because the Action Expert is masked from the queries, the loss acts only as an indirect regularizer on shared VLM weights, and there is no ablation against simpler targets such as E_dex(I_{t+Δ}) or direct auxiliary action or state prediction.

The empirical evidence does not support the novelty claim. The action path closely follows PFD and the alignment path resembles FLARE and FutureVLA, yet none of these are compared, and the rand-full and multi-task baselines are copied from the DexJoCo paper rather than reproduced.

Author should do real machine experiments to test model performance

Questions for authors

See Weakness

Reviewer_9kWPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on the observation that a demonstration records not only the actions that were taken but also what happened to the scene afterwards, and it proposes to use this recorded future as additional supervision when fine-tuning a VLA policy (pi0.5) for dexterous single-arm and bimanual manipulation.

The future is used in two complementary ways. On the representation side, the authors adapt a V-JEPA 2 encoder to dexterous videos and train an action-conditioned predictor on top of it, and the future latent produced by this predictor serves as a target for a set of learnable query tokens inside the VLM, which encourages the backbone to encode where the interaction is heading. On the action side, the same policy is run twice during training, once with only the current observation and once with privileged information about the future, and the difference between the two outputs is distilled into a small residual adapter that only sees the current observation.

Second, privileged future flow distillation. The same policy is run with and without privileged future context: simulator object state plus adapted-encoder features of the frame ahead, under shared noise and flow time. The velocity difference is distilled into a zero-initialized residual adapter that sees only causal features.

Since all privileged inputs are removed at test time, the deployed policy keeps the standard pi0.5 interface, and on the DexJoCo tasks it raises the average success rate, showing its effectiveness.

Critical strengths
  1. The method is practical and easy to deploy, because nothing beyond the usual observations is needed at test time and the residual adapter adds less than 1% to the inference cost. I also found it a clean design choice to share the noise, the flow time, and the weights between the two forward passes, since this makes the correction target well defined and avoids confounding it with other sources of variation.

  2. The empirical improvements are consistent across the benchmark. DexForesight improves 10 of the 11 tasks under rand-obj, the largest gains appear on the harder bimanual tasks such as Unlock iPad (+25.3) and Hanoi (+16.7), and the average also improves under the rand-full and multi-task settings. In addition, Appendix A describes the training setup and gradient routing in enough detail that I would expect the results to be reproducible.

Critical weaknesses

1. The primary concern is the limited novelty of the proposed method, as both of its components closely follow existing work. The action-level path is, by the paper's own description, an application of the principle introduced in PFD, which already compares a shared policy with and without true future observations and distills the resulting action-denoising difference into a causal residual, and the representation-level path is closely related to FLARE and FutureVLA, which likewise align policy features with future-aware embeddings. The novel elements are therefore the combination of the two paths and the dexterity-specific components, namely the Stage-1 adaptation of V-JEPA and the grouped conditioning on arm, wrist, and finger actions. However, neither of these components is ablated, and plain pi0.5 is the only baseline considered. Consequently, it remains unclear whether the improvements stem from contributions beyond the application of established techniques to a new benchmark, and the characterization of the work as a "unified training principle" is not adequately supported by the experiments. Comparisons with a PFD-style residual baseline and a FLARE-style alignment baseline, together with ablations of the two dexterity-specific components, are necessary to establish the contribution of the paper.

2. In addition, the role of the residual adapter is not sufficiently analyzed. The adapter is trained to predict the discrepancy between the two branches from causal policy features alone, so it can at best recover the expectation of this discrepancy over the futures consistent with those features, and under idealized conditions, where both branches fit the data well, this expectation would be close to zero. Moreover, the training loss on the privileged branch also updates the shared policy through the privileged branch, which suggests that part of the Flow-Path-Only improvement may be attributable to this auxiliary objective acting on the shared weights rather than to the correction produced by the adapter. These two explanations could be disentangled with a few inexpensive analyses, namely evaluating the Flow-Path-Only model with the adapter disabled, reporting the magnitude of the predicted correction relative to the original policy output, and reporting the performance of the privileged branch when it is provided with the true future.

Questions for authors

Clarification requests

  1. Are the three seeds evaluation seeds of a single trained model or independent training runs, and how was the evaluated checkpoint selected, given that checkpoints are saved every 10K steps? In addition, could the authors report the standard deviations of the average success rates and of the improvements over the baseline in the ablation tables (Tables 2 and 3), which are currently reported without them?

  2. Were the rand-full and multi-task baselines, which are taken from the DexJoCo paper, evaluated under the same protocol as DexForesight, and does the reproduced rand-obj baseline match the originally reported number?

Analyses that could change my assessment

  1. How does the Flow-Path-Only model perform when the adapter is disabled at inference, how well does the privileged branch perform when it is given the true future, and how large is the predicted correction compared with the original output of the policy? These results would clarify whether the improvement comes from the learned correction or from the auxiliary loss on the privileged branch.

  2. How does DexForesight compare with a PFD-style residual baseline and a FLARE-style alignment baseline, and what is the effect of removing the Stage-1 adaptation or the action grouping? Do the main ablation trends in Tables 2 and 3 also hold across multiple training seeds, at least for the base, Flow-Path-Only, and full models?

  3. Some DexJoCo tasks appear to require information about the task context or progress that is not fully observable from the current frame, such as keeping track of which digits of a password have already been entered in Unlock iPad, and the largest gains are indeed observed on tasks of this kind, such as Unlock iPad (+25.3) and Hanoi (+16.7). This raises the question of whether the improvement reflects a general benefit of future supervision or mainly compensates for missing context in these particular tasks. Could the authors evaluate the method on other commonly used manipulation benchmarks in which the task state is largely observable from the current frame, so that these two explanations can be distinguished?

Reviewer_saZUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose DexForesight, a VLA that uses the V-JEPA 2 world model in combination with a VLM for predicting future states. This generated data is used for finetuning π0.5\pi_{0.5}. The method is evaluated on simulated tasks from DexJoCo.

Critical strengths

None

Critical weaknesses

It is really challenging to understand the contribution and especially the novelty of the paper. The figures are too dense, and writing is convoluted. The paper has been evaluated only in simulation, and more critically, only compared to one baseline, namely the VLA that they fine-tuned.

Questions for authors
  1. Could you clarify what the main novelty of the proposed method?
  2. Have you evaluated the tasks with other baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.