acceptodds
Under review as a conference paper at ICLR 2027

JACT: Learning to Act Through Frozen JEPA Dynamics

Abstract

Visual world models can guide control through online planning, but repeated pre- diction and search add deployment cost. We propose JACT (JEPA-guided Action- Consequence Training), which moves the use of predictive dynamics into offline policy training. A direct policy acts recursively inside a frozen joint-embedding predictive architecture (JEPA), conditions subsequent decisions on predicted con- sequences, and learns to align the resulting trajectory with observed future em- beddings. Gradients pass through the fixed dynamics into the actor, while local supervision anchors its actions. Deployment uses only the encoder and policy, with no predictive rollout, search, or retrieval. Across 11 visual control settings, JACT combines broad task coverage with efficient direct execution. It achieves 97.25% average success on the four-task comparison, the highest among the listed methods with complete coverage. Controlled ablations and repeated actor training support the value of recursive consequence supervision, while identifying task- dependent limits. JACT offers a simple route from pretrained JEPA dynamics to direct visual control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
jepa

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K1tpOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents JACT, an offline goal-conditioned policy optimization method using JEPA for future embedding alignment and consequence training. It shows improvement compared to GC-IDM in some tasks and reduces the deployment consumption compared to online planning methods.

Critical strengths

None.

Critical weaknesses
  1. The novelty of the method proposed is limited. This is basically an offline model-based imitation where policy is optimized towards alignment between predictions and observed future embeddings. This framework resembles [1]. The waypoint prediction in 3.2 is incremental since decoder preserves states and goals, and it supposes that task progression is near linear and limits its applicability to non-linear progressed tasks.

  2. The results in Tab.1 is mostly dependent on reported results but not reproduced results. Comparable baselines only include one method (GC-IDM) which is totally insufficient. Improvements in Tab.2 also seems marginal as is in the error bar. Ablations in Tab.3 seems not consistent especially on Cube-double task. Results in Fig.2 lacks error bar across seeds. Direct control degrades from 99/65 to 30/32 compared to CEM as reported in 4.5, which does not compensate for the decision budget that is saved. Related to Q2, ablation on waypoint prediction is missing.

  3. The writing is overall tedious with over-detailed descriptions on prior knowledge. There are numerous notations which contain actually little information which makes the paper hard to follow. Fig.1 is basically AI-generated and contains meaningless text. The ablations in 4.3 require more detailed descriptions on experimental setup, model modifications, etc. Lastly, the paper needs to adjust the header and "manuscript in preparation" which seems to be totally not ready to me.

[1] Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models

Questions for authors
  1. What distinguishes JACT from model-based imitation learning with differentiable trajectory matching?

  2. What about removing the waypoint prediction in eq.7? How much does it contribute to the final performance?

  3. Results in Tab.2 is selected among epochs 1-10, how sensitive is the comparison to this selection? It would be helpful if scores among epochs are presented.

Reviewer_LaTeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JACT (JEPA-guided Action-Consequence Training), a two-stage framework that leverages pretrained JEPA dynamics to improve goal-conditioned visual control without online planning. The first stage trains an actor through supervised action and waypoint prediction. The second stage recursively rolls out actor-generated actions through a frozen JEPA and aligns predicted future representations with observed trajectory embeddings. Gradients are propagated through the frozen dynamics to update only the actor. At inference time, JACT requires only the encoder and actor, enabling efficient direct execution.

Critical strengths
  • Clear Motivation and Practical Significance. The paper addresses a meaningful trade-off between computationally expensive online planning and efficient direct policies that do not fully exploit pretrained dynamics.
  • Recursive Action-Consequence Training. By conditioning subsequent decisions on predicted states, JACT captures dependencies between successive actions and propagates future prediction errors to earlier decisions, extending beyond independent forward consistency.
  • Effective Use of Frozen Pretrained World Models. The framework leverages pretrained dynamics to provide action-sensitive supervision without additional environment interaction, reward modeling, or world model fine-tuning, while maintaining efficient inference.
  • Well-Designed Ablation Studies. The paper investigates alternative supervision strategies, recursive conditioning, rollout depth, and feedback gradients. These controlled comparisons provide evidence for the effectiveness of recursive consequence training, particularly on PointMaze and Puzzle-4×4.
Critical weaknesses
  • Strong Dependence on World Model Accuracy. The effectiveness of consequence supervision relies on the fidelity of frozen dynamics. Prediction errors may produce misleading policy gradients, and improved latent alignment does not necessarily translate into better execution. The reported prediction–execution discrepancies further highlight this concern.
  • Limited Methodological Novelty. Differentiating through learned dynamics and enforcing future-state consistency are established techniques. The primary novelty lies in combining these principles through recursive policy–predictor coupling with a frozen JEPA, rather than introducing a fundamentally new learning paradigm.
  • Consequence Matching May Over-Constrain the Policy. Aligning predicted states with a single recorded trajectory may penalize alternative yet valid behaviors. Consequently, minimizing trajectory-level representation errors does not necessarily correspond to maximizing task success.
  • High Training Computational Cost. Recursive rollouts and backpropagation through the frozen predictor introduce substantial computational and memory overhead. This raises scalability concerns, particularly for larger world models or longer prediction horizons.
  • Large Performance Gap Compared with Online Planning on Navigation. Despite efficient inference, JACT substantially underperforms CEM planning on PointMaze (30% vs. 99%) and Wall (32% vs. 65%). Although CEM incurs significantly higher inference costs, these results suggest that direct consequence-trained policies cannot consistently replace online planning in challenging navigation tasks.
Questions for authors

See strengths and weaknesses

Reviewer_ZUZgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JACT (JEPA-guided Action-Consequence Training), a two-stage method that leverages a frozen action-conditioned JEPA world model to supervise a direct, goal-conditioned policy during offline training. The key idea is to let the policy act inside the frozen predictor, recursively conditioning later actions on predicted contexts, and to align the predicted futures with encodings of observed future images; only the policy is updated and the world model remains fixed. At deployment the agent uses only the encoder and policy without searching, and empirical results across 11 visual-control settings and controlled ablations indicate consistent gains on several tasks, competitive four-task averages relative to strong prior methods, and favorable compute–performance trade-offs versus online planning.

Critical strengths

Demonstrates a practical route to distill predictive world-model guidance into a direct policy, removing online rollouts/search at test time while retaining competitive or superior success rates in several benchmarks.

Critical weaknesses
  1. The approach relies heavily on the accuracy and coverage of the frozen predictor; gradients can in principle reinforce model-specific biases or encourage policies that exploit predictor idiosyncrasies rather than true environment dynamics.

  2. Rollout depth is short (K=3 by default), and longer-rollout behavior is not explored; stability and benefits at larger K remain unclear.

Questions for authors
  1. Have you tried deeper rollouts (K > 3) with stronger regularization (e.g., KL or consistency on intermediate states) to balance signal vs. model-error accumulation? How does performance and stability change?

  2. Can you report ablations with stronger or different frozen predictors to test whether better dynamics improve JACT’s gradients and final policy quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.