acceptodds
Under review as a conference paper at ICLR 2027

JEPA-VA: JEPA Is a Good Visual Action Verifier for Generalizable Robot Policies

Abstract

Vision-language-action (VLA) policies are commonly trained to align predicted actions with demonstrations, while world-action models (WAMs) jointly predict future video and action; neither explicitly checks whether an action produces its intended visual consequence. We study this complementary training signal: does a policy's proposed action lead to the expected visual change? We introduce \method, a JEPA visual verifier that evaluates proposed action chunks through their predicted feature changes relative to demonstrated transitions. Conditioned on the current observation and an action chunk, the verifier predicts the feature change at the end of the chunk. During policy training, a change-weighted cosine objective aligns the predicted change direction with the demonstrated one, sending a consequence-consistency gradient through the verifier's action input back to the policy. The verifier is not needed at deployment and adds no inference cost. Verifier-guided policy learning raises LIBERO-Plus success from 62.68% to 71.73% and clean-to-randomized RoboTwin success from 12.00% to 17.80%, suggesting that JEPA-VA helps policies learn more generalizable robot manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Policy Generalization, Visual Consequence Verification, Imitation Learning, World Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ifCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method called JEPA-VA to learn to supervise the consistency over the consequences to facilitate the policy learning. In stage-1, it learns an action-conditioned predictor (JEPA-style) and in stage-2, it freezes the predictor and then use the difference between the predicted action's change for the future latent features and the expert actions' change to the future latent features as the additional signals to back-propagate the gradients to the policy alongside the imitation loss (l1 loss). The policy is evaluated with other baselines on LIBERO-Plus, Robotwin and real world.

Critical strengths
  1. The paper proposes to supervise actions not only from the errors between predictions and expert actions but also from the visual consequences differences in the latent feature space. The framework is well-motivated and explained clearly in the paper.
  2. The paper has conducted extensive studies to study the encoder, the policy capacity and the verifier on Robotwin and LIBEROPlus benchmarks.
Critical weaknesses
  1. The ablations are not comprehensive enough to support the proposed method. The stage-1 training is not evaluated alone and the predictor is used in the stage-2 and most results focus on the policy performance. It will help the readers better understand the quality of the predictor. Moreover the predictor is only trained on demonstrations not policy rollouts or random actions. This may cause some problems for stage-2 training. For example, the stage-2 policy's action may be out of distribution for the predictor to predict accurately what is the future latent features.
  2. The real world tasks are limited to only two simple pick and place tasks which picks objects from the table to put in the basket/box. Since those two locations have large tolerance, the task is relatively easy to achieve. Therefore, the task is short-horizon and not complex enough to support the claim of the proposed method. In the real world baselines, the paper didn't compare to the most relevant baselines. Moreover, it also didn't compare to the common state-of-the-art VLA baseline of pi0.5 which has been widely shown good at those pick and place tasks.
Questions for authors
  1. Can you provide more quantitative metrics on the stage-1 training? Can you also provide some counterfactual actions as inputs to see how the prediction changes with different action inputs?
  2. The cosine objective aligns only the direction of the predicted and demonstrated residuals in feature space and is invariant to their magnitude, yet magnitude often matters for manipulation. Could the authors explain, intuitively or empirically, why a direction-only signal still provides useful supervision to the policy?
  3. why the policy is optimized with l1-loss for the actions? Currently most robot policies are optimized with diffusion loss or flow-matching loss and using l1-loss to optimize is uncommon.
6 comments
AuthorsOpenReview ↗

W1 / Q1 — Quantitative evaluation of Stage I and counterfactual action inputs.

We conducted 3 experiments to test whether Ψ uses action rather than relying solely on images.

  1. Fixing image and changing the action. Under both in-sample and held-out conditions, we tested 4 action substitutions: (1) adding noise, (2) action from another episode, (3) zero-action, and (4) zero-action with the gripper open. From following table, the prediction change is positively correlated with the action change and the demo action explained the future better than the alternatives in 93–100% of the windows. We also provide the PCA plots in Figs. G1–G3 of revised PDF which show the same pattern: predictions follow the demo actions, shift when actions are changed, and stay unchanged with zero actions.
action a′Prediction change (held-out)Demo better (held-out)Prediction change (in-sample)Demo better (in-sample)
Noise0.3893.1%0.45100.0%
Action from another episode0.7298.5%0.82100.0%
Zero action0.96100.0%1.08100.0%
Zero action w/ gripper open0.7199.1%0.82100.0%
  1. Running different actions from the same simulation state. The following 6×6 table compares Ψ’s predictions with what actually happened after each action. In every column, the matching action has the lowest error. This shows that Ψ can tell different actions’ effects apart, rather than just favoring the demo action.
Ψ input \ Executed chunkDemoNoiseOther taskZeroNo motionPolicy
Demo0.2830.3430.5360.5790.6780.340
Noise0.3500.3140.5370.5820.6810.381
Action from another episode0.5230.5210.3870.5780.6630.520
Zero action0.5480.5470.5600.4310.6460.546
Zero action w/ gripper open0.5980.5980.5940.5920.5710.595
Policy0.3420.3750.5340.5770.6750.287
  1. Comparing action-conditioned predictions with actual execution outcomes. We ran different actions from the same simulation state and checked: (i) how often Ψ(a′) matched the actual result of a′ better than Ψ(demo), and (ii) how often Ψ(demo) matched the actual demo result better than Ψ(a′). When changing the actions clearly changed the outcome, Ψ usually predicted that change correctly, on both training and unseen tasks. This shows that Ψ uses the actions, not just the image, and can do so on unseen tasks too.
Chunk a′(i) Own-action prediction closer to actual outcome (in-sample / held-out)(ii) Demo prediction closer to demo outcome (in-sample / held-out)
Noise78.2% / 66.0%98.5% / 90.3%
Action from another episode96.5% / 97.8%99.0% / 98.8%
Zero action97.0% / 96.8%100.0% / 98.5%
Zero action w/ gripper open92.6% / 87.8%100.0% / 99.8%
Policy92.1% / 93.3%95.0% / 98.3%

Together, these results show that Ψ does more than predict the future from the image alone. Its predictions change with the action chunk and match what those actions actually lead to, on both training and unseen tasks.

AuthorsOpenReview ↗

W1 — The predictor is trained only on demonstrations; policy actions may fall outside its training distribution.

From the same state, we ran both policy and demo action chunks and compared their prediction errors. We used the LIBERO Base policy trained w/o verifier, so Ψ have no influence on its actions. The policy actions increased the prediction error by only +0.003 / +0.004 compared with demo actions. On held-out tasks, policy actions had higher errors at 52.1% of points—close to half. These results show no clear additional prediction error on policy actions.

SplitDemo actionPolicy action (w/o verifier)Policy − demoPoints where policy is worse
In-sample0.2830.287+0.003 [−0.001, 0.008]54.5%
Held-out (LIBERO-90)0.4190.423+0.004 [−0.003, 0.011]52.1%

We agree that adding rollout or failure data could help close the gap between training and unseen tasks. We thank the reviewer for this helpful suggestion. Given the challenges of collecting enough failure data, we leave this promising direction for future work.

AuthorsOpenReview ↗

W2 — The real-world tasks are simple pick-and-place tasks; comparisons with leading baselines, including π0.5, are missing.

To address this limitation and the missing comparison with π0.5, we added a two-arm bean-pouring task on a physical Unitree G1D robot. We evaluated π0.5, FastWAM, ACT, and JEPA-VA with and without the verifier (see below). This task requires lifting and coordinating two cups, controlling their relative positions and orientations to transfer the beans, and placing the cups back down.

As you noted, π0.5 with robot pretraining performs strongly on grasping. On the more precise pouring step, however, JEPA-VA with the verifier achieved the same success rate. JEPA-VA without the verifier achieved the same pickup results but fewer successful pours, supporting the verifier’s benefit for fine manipulation.

PolicyS1 (Pick up one cup)S2 (Pick up both cups)S3 (Successful pour)Stage score
FastWAM1/200/202/200.06
ACT6/200/203/200.15
JEPA-VA w/o verifier8/205/203/200.24
JEPA-VA w/ verifier8/205/205/200.29
π0.5 w/ robot pretraining16/2011/205/200.46

( The three stages were weighted 25% / 25% / 50%.)

The table below also summarizes published simulation results for π0 and π0.5 under the evaluation protocols in Tables 1–2 of our paper. We acknowledge that JEPA-VA still falls behind π0.5 in generalization, while using no large-scale robot-data pretraining.

MethodPretraining beyond the benchmark demoLIBEROLIBERO-PlusRoboTwin 2.0 Easy (clean)RoboTwin 2.0 Hard (randomized)
π0VLM + large-scale multi-robot data94.1570.7 (ACoT-VLA)46.42 (one model per task)16.34 (one model per task)
π0.5VLM + large-scale multi-robot data96.8586.570.6646.0
JEPA-VA w/o verifier (Table 4)Self-supervised video encoder only92.9561.89 / 59.3277.2012.00
JEPA-VASelf-supervised video encoder only95.0072.17 / 70.6077.5017.80
Reviewer_SU9YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an action-conditioned visual verifier for robot policies using a frozen V-JEPA model. Instead of generating future video, it evaluates proposed action chunks by predicting feature changes during training via a change-weighted cosine objective. This provides consistency gradients that significantly improve robustness under distribution shifts, as tested on LIBERO-Plus and RoboTwin, while adding zero inference latency.

Critical strengths

The method is very straightforward, although not strikingly new. The evaluations are performed on both simulated and real-world settings. The results seem to validate the effectiveness of the verifier for policy training.

Critical weaknesses

Conceptually, the verifier acts as a feature-space world model, which has been studied extensively. So, the novelty of the paper is limited. But I do like the simplicity and if the result /gain reported in this paper can be verified by other studies, I consider it a very nice piece of work.

Questions for authors

since the verifier is trained on demonstration data, it would be useful to discuss the quality requirements of the demonstration data, e.g., as shown in Figure 2, the target delta Z is generated by comparing two views, how do we ensure that the difference between these views are due to the actions taken, instead of other factors, such as the changes of camera position, lighting, etc.

2 comments
AuthorsOpenReview ↗

Response to Reviewer SU9Y

Thank you for recognizing the simplicity and effectiveness of our work !

We agree that Ψ is an action-conditioned latent dynamics model, in the same family as V-JEPA 2-AC, DINO-WM, and the world-action models we cite. JEPA-VA’s contribution lies in how we use it: once trained, Ψ is frozen and used only during policy training as a differentiable verifier, passing gradients to the policy’s actions alongside the imitation loss. It is never used for rollouts, planning, or deployment, so it adds no inference overhead. It also predicts residual directions rather than image frames. Table 4 isolates the benefit of this use by comparing the same policy with and without the verifier. We will revise Section 2 to make this positioning explicit.

Response: How do we know that ΔZ reflects actions rather than camera or lighting changes?

We conducted 3 experiments to test whether Ψ uses action rather than relying solely on images.

  1. Fixing image and changing the action. Under both in-sample and held-out conditions, we tested 4 action substitutions: (1) adding noise, (2) action from another episode, (3) zero-action, and (4) zero-action with the gripper open. From following table, the prediction change is positively correlated with the action change and the demo action explained the future better than the alternatives in 93–100% of the windows. We also provide the PCA plots in Figs. G1–G3 of revised PDF which show the same pattern: predictions follow the demo actions, shift when actions are changed, and stay unchanged with zero actions.
action a′Prediction change (held-out)Demo better (held-out)Prediction change (in-sample)Demo better (in-sample)
Noise0.3893.1%0.45100.0%
Action from another episode0.7298.5%0.82100.0%
Zero action0.96100.0%1.08100.0%
Zero action w/ gripper open0.7199.1%0.82100.0%
  1. Running different actions from the same simulation state. The following 6×6 table compares Ψ’s predictions with what actually happened after each action. In every column, the matching action has the lowest error. This shows that Ψ can tell different actions’ effects apart, rather than just favoring the demo action.
Ψ input \ Executed chunkDemoNoiseOther taskZeroNo motionPolicy
Demo0.2830.3430.5360.5790.6780.340
Noise0.3500.3140.5370.5820.6810.381
Action from another episode0.5230.5210.3870.5780.6630.520
Zero action0.5480.5470.5600.4310.6460.546
Zero action w/ gripper open0.5980.5980.5940.5920.5710.595
Policy0.3420.3750.5340.5770.6750.287
  1. Comparing action-conditioned predictions with actual execution outcomes. We ran different actions from the same simulation state and checked: (i) how often Ψ(a′) matched the actual result of a′ better than Ψ(demo), and (ii) how often Ψ(demo) matched the actual demo result better than Ψ(a′). When changing the actions clearly changed the outcome, Ψ usually predicted that change correctly, on both training and unseen tasks. This shows that Ψ uses the actions, not just the image, and can do so on unseen tasks too.
Chunk a′(i) Own-action prediction closer to actual outcome (in-sample / held-out)(ii) Demo prediction closer to demo outcome (in-sample / held-out)
Noise78.2% / 66.0%98.5% / 90.3%
Action from another episode96.5% / 97.8%99.0% / 98.8%
Zero action97.0% / 96.8%100.0% / 98.5%
Zero action w/ gripper open92.6% / 87.8%100.0% / 99.8%
Policy92.1% / 93.3%95.0% / 98.3%

Together, these results show that Ψ does more than predict the future from the image alone. Its predictions change with the action chunk and match what those actions actually lead to, on both training and unseen tasks.

Response: Can others verify these gains?

With the final version, we will release checkpoints for both simulation and real-robot experiments, evaluation scripts, real-robot deployment and evaluation code and configurations, and the exact data splits so others can reproduce our results.

Thank you again for your review. We would appreciate your feedback on our responses and additional experiments, particularly whether they address your main concerns. We are happy to clarify any remaining questions.

Reviewer_BabSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

JEPA-VA trains a predictor to estimate how an action chunk changes the scene. It then uses this predictor to help train a policy alongside action imitation. The predictor is removed at deployment. Experiments include simulation and two real-world tasks.

Critical strengths

s1. The idea is clear: train actions to match both the demonstrated commands and their visual effects.

s2. The method adds no inference cost.

s3. Comparisons with and without the predictor show improvements on several simulation settings.

Critical weaknesses

w1. It is unclear whether the predictor actually checks the actions or mainly guesses the future from the current image. The paper lacks tests with incorrect or perturbed actions.

w2. The predictor only learns from successful demonstrations, but later guides actions generated by the policy. Its predictions may be wrong when these actions differ from the demonstrations.

w3. The visual loss matches the direction of feature changes but not their size. It may also focus on large robot movements rather than important object changes.

Questions for authors

q1. With the image fixed, does changing the action produce the correct change in the prediction? Can the authors test this against actual execution?

q2. How well does the predictor work on the policy’s own actions? Would rollout or failure data help?

q3. Since cosine loss only matches change direction, could you compare it with a loss that also matches magnitude? Moreover, could you also visualize the patch weights to show whether they emphasize robot motion or object interactions?

q4. Can the authors add a real-world comparison with and without the predictor loss, keeping everything else fixed?

6 comments
AuthorsOpenReview ↗

Response to w1 / q1 — Does Ψ evaluate actions, or merely predict the future from images?

We conducted 3 experiments to test whether Ψ uses action rather than relying solely on images.

  1. Fixing image and changing the action. Under both in-sample and held-out conditions, we tested 4 action substitutions: (1) adding noise, (2) action from another episode, (3) zero-action, and (4) zero-action with the gripper open. From following table, the prediction change is positively correlated with the action change and the demo action explained the future better than the alternatives in 93–100% of the windows. We also provide the PCA plots in Figs. G1–G3 of revised PDF which show the same pattern: predictions follow the demo actions, shift when actions are changed, and stay unchanged with zero actions.
action a′Prediction change (held-out)Demo better (held-out)Prediction change (in-sample)Demo better (in-sample)
Noise0.3893.1%0.45100.0%
Action from another episode0.7298.5%0.82100.0%
Zero action0.96100.0%1.08100.0%
Zero action w/ gripper open0.7199.1%0.82100.0%
  1. Running different actions from the same simulation state. The following 6×6 table compares Ψ’s predictions with what actually happened after each action. In every column, the matching action has the lowest error. This shows that Ψ can tell different actions’ effects apart, rather than just favoring the demo action.
Ψ input \ Executed chunkDemoNoiseOther taskZeroNo motionPolicy
Demo0.2830.3430.5360.5790.6780.340
Noise0.3500.3140.5370.5820.6810.381
Action from another episode0.5230.5210.3870.5780.6630.520
Zero action0.5480.5470.5600.4310.6460.546
Zero action w/ gripper open0.5980.5980.5940.5920.5710.595
Policy0.3420.3750.5340.5770.6750.287
  1. Comparing action-conditioned predictions with actual execution outcomes. We ran different actions from the same simulation state and checked: (i) how often Ψ(a′) matched the actual result of a′ better than Ψ(demo), and (ii) how often Ψ(demo) matched the actual demo result better than Ψ(a′). When changing the actions clearly changed the outcome, Ψ usually predicted that change correctly, on both training and unseen tasks. This shows that Ψ uses the actions, not just the image, and can do so on unseen tasks too.
Chunk a′(i) Own-action prediction closer to actual outcome (in-sample / held-out)(ii) Demo prediction closer to demo outcome (in-sample / held-out)
Noise78.2% / 66.0%98.5% / 90.3%
Action from another episode96.5% / 97.8%99.0% / 98.8%
Zero action97.0% / 96.8%100.0% / 98.5%
Zero action w/ gripper open92.6% / 87.8%100.0% / 99.8%
Policy92.1% / 93.3%95.0% / 98.3%

Together, these results show that Ψ does more than predict the future from the image alone. Its predictions change with the action chunk and match what those actions actually lead to, on both training and unseen tasks.

AuthorsOpenReview ↗

Response to w2 / q2 — Ψ learns only from successful demonstrations; is it reliable on the policy’s actions? Would rollout or failure data help?

From the same state, we ran both policy and demo action chunks and compared their prediction errors. We used the LIBERO Base policy trained w/o verifier, so Ψ have no influence on its actions. The policy actions increased the prediction error by only +0.003 / +0.004 compared with demo actions. On held-out tasks, policy actions had higher errors at 52.1% of points—close to half. These results show no clear additional prediction error on policy actions.

SplitDemo actionPolicy action (w/o verifier)Policy − demoPoints where policy is worse
In-sample0.2830.287+0.003 [−0.001, 0.008]54.5%
Held-out (LIBERO-90)0.4190.423+0.004 [−0.003, 0.011]52.1%

We agree that adding rollout or failure data could help close the gap between training and unseen tasks. We thank the reviewer for this helpful suggestion. Given the challenges of collecting enough failure data, we leave this promising direction for future work.

AuthorsOpenReview ↗

Response to w3 / q3 — How do direction and magnitude relate? Do the patch weights focus on robot motion or object interactions?

We visualized the patch weights used in the training loss (Fig. G4 in Appendix G of the revised PDF). The weights mainly focus on the robot arm, gripper, and manipulated objects, with little weight on the static background.

The cosine objective does not completely ignore magnitude. Each patch is weighted by its observed change, and a prediction of no change receives a discrepancy of 1—the same as an orthogonal prediction. Thus, actions that produce no effect cannot reduce the loss. To test whether matching magnitude helps, we retrained Stage I with the relative L2 loss, Σ‖ΔẐ−ΔZ‖² / Σ‖ΔZ‖². We evaluated both predictors using noise, action chunks from another episode, and zero-action chunks:

Stage-I lossCosine discrepancy ↓ (held-out / in-sample)Relative L2 ↓ (held-out / in-sample)Norm ratio ‖ΔẐ‖/‖ΔZ‖ (held-out / in-sample)Demo better: noise / other episode / zero chunk
Change-weighted cosine (paper)0.403 / 0.2180.892 / 0.5541.10 / 1.1172.7% / 98.5% / 100%
Relative L2 (magnitude-aware)0.412 / 0.2380.623 / 0.3280.74 / 0.7776.0% / 98.2% / 99.2%

Relative L2 training reduced the in-sample relative L2 error, but increased the cosine discrepancy. The norm ratio changed from 1.10 to 0.74, shifting from overestimating to underestimating the change magnitude. Action discrimination did not improve consistently: it increased by 3.3 percentage points for small noise but decreased slightly for chunks from another episode and zero-action chunks. Thus, adding a magnitude constraint improved the targeted L2 metric without consistently improving action discrimination.

We also trained a Stage-II policy using the relative-L2 Ψ as the verifier. It improved success by 0.79% over the cosine verifier, with a paired 95% confidence interval of [−1.01, +2.58]. We therefore found no clear additional benefit from matching magnitude in this experiment. In comparison, removing the verifier reduced success by 8.55% relative to the cosine verifier. Under this evaluation setting, the difference between the two verifiers was much smaller than the difference between using and removing a verifier.

PolicyPlus-7 success rateΔ vs. cosine
Relative-L2 verifier72.33 / 71.07+0.79 [−1.01, +2.58]
Cosine verifier (paper)71.54 / 70.06—
No verifier62.99 / 59.88−8.55 [−10.56, −6.51]

(The two Plus-7 values report success rates under a uniform 520-step limit and LIBERO’s suite-specific step limits, respectively.)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank all reviewers for their thoughtful comments and helpful suggestions.

In response, we added quantitative evaluations of Ψ with counterfactual actions and actual execution outcomes, including policy-generated actions on held-out tasks. Appendix G of the revised PDF includes PCA visualizations of predictions under different action inputs (Figs. G1–G3) and visualizations of the patch weights used in training (Fig. G4). We also compared cosine and magnitude-aware objectives, and added a real-robot bean-pouring task with π0.5, FastWAM, ACT, and JEPA-VA with and without the verifier. These experiments help clarify how Ψ uses actions and where the verifier improves policy performance. We also clarify JEPA-VA’s relationship to latent dynamics models and our choice of the L1 policy loss.

Detailed responses are provided below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.