- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a method called JEPA-VA to learn to supervise the consistency over the consequences to facilitate the policy learning. In stage-1, it learns an action-conditioned predictor (JEPA-style) and in stage-2, it freezes the predictor and then use the difference between the predicted action's change for the future latent features and the expert actions' change to the future latent features as the additional signals to back-propagate the gradients to the policy alongside the imitation loss (l1 loss). The policy is evaluated with other baselines on LIBERO-Plus, Robotwin and real world.
Critical strengths
- The paper proposes to supervise actions not only from the errors between predictions and expert actions but also from the visual consequences differences in the latent feature space. The framework is well-motivated and explained clearly in the paper.
- The paper has conducted extensive studies to study the encoder, the policy capacity and the verifier on Robotwin and LIBEROPlus benchmarks.
Critical weaknesses
- The ablations are not comprehensive enough to support the proposed method. The stage-1 training is not evaluated alone and the predictor is used in the stage-2 and most results focus on the policy performance. It will help the readers better understand the quality of the predictor. Moreover the predictor is only trained on demonstrations not policy rollouts or random actions. This may cause some problems for stage-2 training. For example, the stage-2 policy's action may be out of distribution for the predictor to predict accurately what is the future latent features.
- The real world tasks are limited to only two simple pick and place tasks which picks objects from the table to put in the basket/box. Since those two locations have large tolerance, the task is relatively easy to achieve. Therefore, the task is short-horizon and not complex enough to support the claim of the proposed method. In the real world baselines, the paper didn't compare to the most relevant baselines. Moreover, it also didn't compare to the common state-of-the-art VLA baseline of pi0.5 which has been widely shown good at those pick and place tasks.
Questions for authors
- Can you provide more quantitative metrics on the stage-1 training? Can you also provide some counterfactual actions as inputs to see how the prediction changes with different action inputs?
- The cosine objective aligns only the direction of the predicted and demonstrated residuals in feature space and is invariant to their magnitude, yet magnitude often matters for manipulation. Could the authors explain, intuitively or empirically, why a direction-only signal still provides useful supervision to the policy?
- why the policy is optimized with l1-loss for the actions? Currently most robot policies are optimized with diffusion loss or flow-matching loss and using l1-loss to optimize is uncommon.
6 comments
W1 — The predictor is trained only on demonstrations; policy actions may fall outside its training distribution.
From the same state, we ran both policy and demo action chunks and compared their prediction errors. We used the LIBERO Base policy trained w/o verifier, so Ψ have no influence on its actions. The policy actions increased the prediction error by only +0.003 / +0.004 compared with demo actions. On held-out tasks, policy actions had higher errors at 52.1% of points—close to half. These results show no clear additional prediction error on policy actions.
| Split | Demo action | Policy action (w/o verifier) | Policy − demo | Points where policy is worse |
|---|---|---|---|---|
| In-sample | 0.283 | 0.287 | +0.003 [−0.001, 0.008] | 54.5% |
| Held-out (LIBERO-90) | 0.419 | 0.423 | +0.004 [−0.003, 0.011] | 52.1% |
We agree that adding rollout or failure data could help close the gap between training and unseen tasks. We thank the reviewer for this helpful suggestion. Given the challenges of collecting enough failure data, we leave this promising direction for future work.
W2 — The real-world tasks are simple pick-and-place tasks; comparisons with leading baselines, including π0.5, are missing.
To address this limitation and the missing comparison with π0.5, we added a two-arm bean-pouring task on a physical Unitree G1D robot. We evaluated π0.5, FastWAM, ACT, and JEPA-VA with and without the verifier (see below). This task requires lifting and coordinating two cups, controlling their relative positions and orientations to transfer the beans, and placing the cups back down.
As you noted, π0.5 with robot pretraining performs strongly on grasping. On the more precise pouring step, however, JEPA-VA with the verifier achieved the same success rate. JEPA-VA without the verifier achieved the same pickup results but fewer successful pours, supporting the verifier’s benefit for fine manipulation.
| Policy | S1 (Pick up one cup) | S2 (Pick up both cups) | S3 (Successful pour) | Stage score |
|---|---|---|---|---|
| FastWAM | 1/20 | 0/20 | 2/20 | 0.06 |
| ACT | 6/20 | 0/20 | 3/20 | 0.15 |
| JEPA-VA w/o verifier | 8/20 | 5/20 | 3/20 | 0.24 |
| JEPA-VA w/ verifier | 8/20 | 5/20 | 5/20 | 0.29 |
| π0.5 w/ robot pretraining | 16/20 | 11/20 | 5/20 | 0.46 |
( The three stages were weighted 25% / 25% / 50%.)
The table below also summarizes published simulation results for π0 and π0.5 under the evaluation protocols in Tables 1–2 of our paper. We acknowledge that JEPA-VA still falls behind π0.5 in generalization, while using no large-scale robot-data pretraining.
| Method | Pretraining beyond the benchmark demo | LIBERO | LIBERO-Plus | RoboTwin 2.0 Easy (clean) | RoboTwin 2.0 Hard (randomized) |
|---|---|---|---|---|---|
| π0 | VLM + large-scale multi-robot data | 94.15 | 70.7 (ACoT-VLA) | 46.42 (one model per task) | 16.34 (one model per task) |
| π0.5 | VLM + large-scale multi-robot data | 96.85 | 86.5 | 70.66 | 46.0 |
| JEPA-VA w/o verifier (Table 4) | Self-supervised video encoder only | 92.95 | 61.89 / 59.32 | 77.20 | 12.00 |
| JEPA-VA | Self-supervised video encoder only | 95.00 | 72.17 / 70.60 | 77.50 | 17.80 |
W1 / Q1 — Quantitative evaluation of Stage I and counterfactual action inputs.
We conducted 3 experiments to test whether Ψ uses action rather than relying solely on images.
Together, these results show that Ψ does more than predict the future from the image alone. Its predictions change with the action chunk and match what those actions actually lead to, on both training and unseen tasks.