acceptodds
Under review as a conference paper at ICLR 2027

Procedural Planning with JEPA World Model

Abstract

In this work, we introduce JEPA-Pro, a Joint-Embedding Predictive Architecture (JEPA)-based world model trained end-to-end for procedural planning, whose unit of prediction is a procedural step and in which text is used only to define actions and goals. World models predict how the state of the world changes under an action, which makes them a natural fit for model-based planning. Current world models, however, predict at a fixed temporal granularity which suits low-level control but is misaligned with the unit of decision in procedural tasks such as assembling furniture or cleaning a house. The challenge in procedural planning is twofold: (1) steps have variable duration ("wash the floor" may take seconds or minutes), and (2) they are rarely contiguous, since people often performs unrelated actions in between. To address this, our model is composed of two core components: an action-conditioned causal predictor trained jointly with the state encoder, and a learned cost function that estimates how far a predicted state is from the goal. At inference time, JEPA-Pro produces plans by test-time search over candidate action sequences in latent space, selecting the sequence whose predicted trajectory minimizes the cost, without generating pixels nor text. We evaluate JEPA-Pro on world modeling, procedural understanding and procedural planning benchmarks. We establish a new state-of-the-art on WorldPrediction-WM () and on EgoPlan-Bench (). On ENACT, the world model in zero-shot outperforms 23 of 30 VLM baseline, and all caption-free baselines on WorldPrediction-PP () using up to two order of magnitude less parameters. We show that finetuning the cost with few trajectories enables the world model to adapt to unseen tasks while naturally enabling test-time scaling during planning, outperforming zero-shot VLM policies on ALFRED and BEHAVIOR-1K.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Procedural Planning, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_VdfNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces JEPA-Pro, a JEPA-based world model for procedural planning. The architecture follows the Causal Temporal Prediction (CTP)-JEPA paradigm: a state estimation module aggregates visual observations into state embeddings, an action-conditioned causal predictor autoregressively predicts future state embeddings, and a cost module evaluates predicted states against goal embeddings. Planning is performed via test-time search over candidate action sequences in latent space, selecting sequences that minimize cumulative cost. The model is trained on 4.68M annotated steps from 2.51M procedural trajectories spanning egocentric human activities and robot recordings. Experiments demonstrate new state-of-the-art results on WorldPrediction-WM (71.8%) and EgoPlan-Bench (58.7%), and outperformance of 23/30 VLM baselines on ENACT. On ALFRED and BEHAVIOR-1K, the model achieves 64.6% and 10.0% success rates respectively with test-time search.

Critical strengths

a. The paper evaluates on five benchmarks (ENACT, WorldPrediction, EgoPlan-Bench, ALFRED, BEHAVIOR-1K) and demonstrates strong performance on several. The new state-of-the-art results on WorldPrediction-WM (71.8%) and EgoPlan-Bench (58.7%) are notable.

b. The paper shows that performance improves monotonically with beam width (Figure 6), demonstrating that the world model benefits from additional test-time compute. This is a useful finding for understanding the tradeoff between inference cost and planning quality.

c. The paper demonstrates that JEPA-Pro outperforms 23/30 VLM baselines on ENACT, including much larger models (GPT-4, Gemini). This suggests that the world model approach provides advantages over language-only reasoning for procedural planning.

Critical weaknesses

a. JEPA-Pro is a straightforward combination of existing components: VL-JEPA for representation learning, autoregressive transformer for state prediction, cosine similarity for cost, and beam search for planning. The CTP-JEPA formulation is just standard causal autoregressive prediction, there is no fundamental algorithmic advance. The paper dresses up these components with elaborate notation and a "world model" narrative, but the underlying technical contribution is thin.

b. The model predicts state embeddings, not world states. There's no analysis of whether the embeddings capture genuine world dynamics (e.g., object positions, physical properties). The paper shows the embeddings are useful for planning but doesn't establish they represent world state meaningfully. This is a representation learning approach, not a world model.

c. The paper compares against general-purpose VLMs but not against methods specifically designed for procedural planning (e.g., VidAssist, other search-based planning methods). This makes it unclear whether JEPA-Pro is the best approach or just better than general-purpose VLMs on this task.

d. Cosine similarity between state and goal embeddings is a crude measure of progress. It doesn't capture task structure, subgoal ordering, or the complexity of multi-step procedures. The paper doesn't analyze whether this cost function is sufficient for challenging tasks.

e. The paper only plans 4 steps ahead and all experiments are on simulated benchmarks. For a paper claiming to enable "autonomous agents that act in the physical world," this is a significant gap. The method's applicability to real-world procedural tasks with longer horizons is untested.

f. The paper doesn't analyze when the method fails or what types of tasks are challenging. Understanding failure modes is critical for assessing practical utility. The paper also doesn't discuss the computational cost of test-time search, which could be prohibitive for real-time applications.

Questions for authors
  1. Can you provide a comparison with dedicated procedural planning methods (e.g., VidAssist, other search-based planning approaches) to establish whether JEPA-Pro is competitive with methods specifically designed for this task?

  2. Can you analyze whether the learned state embeddings capture genuine world dynamics? For example, do they preserve object positions, velocities, or other physical properties?

  3. Can you extend the planning horizon beyond T=4 steps? How does performance degrade with longer horizons, and what modifications would be needed to handle tasks requiring 10+ steps?

  4. Can you analyze failure cases? What types of tasks does JEPA-Pro struggle with, and why?

  5. Can you discuss the computational cost of test-time search? For beam-4 with T=4, the model evaluates 256 candidate sequences — is this practical for real-time applications?

  6. Can you compare against VL-JEPA and other JEPA variants to establish whether the CTP-JEPA formulation provides advantages over existing approaches?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VdfN: contribution scope, learned cost and procedure-specific baselines (part 1)

We thank the reviewer for the detailed review. Before the point-by-point answers, we first address weakness a & d together and use that opportunity to clarify the scope of our work. We revised our submission regarding all the points mentioned by the reviewer in either the main body or the supplementary materials.

W-a, W-d, claim and scope. End-to-end latent world models are an established paradigm at the low level: LeWorldModel [1] and PLDM [2], as stated in Related Work, learn latent dynamics in fixed, controlled environments with low-level action spaces, and their value is not beating specialist policies on their benchmarks but showing that a learned dynamics model enables planning by search. Our paper carries this paradigm to the high level, with semantically abstract procedural steps, open-vocabulary text actions and real egocentric video, and trains it end to end, which to our knowledge no prior system had done. We want first to clarify that 1) the claim is an end-to-end-trained high-level world model that enables planning by search, in a noisier setting than fixed-camera manipulation and at a higher level of temporal abstraction; 2) VLMs are the baselines we report against because generating plans in token space toward a language goal is their natural ground, not because the goal is beating specialist models. Low-level world models are not judged that way either. We see high-level world modeling as one necessary building block toward enabling autonomous agents to act in the physical world, just as low-level world models are for current manipulation tasks.

The contribution is therefore not the invention of each block but making this joint training work at the high level, and we ran controlled ablations that bear directly on both weaknesses. On the combination claim: training a predictor on top of the frozen pretrained encoder (a DINO-WM [3] style recipe), under an identical adaptation budget, reaches 27.8 on EgoPlan-Bench vs 54.3 for our end-to-end model. On the cost: it is not raw cosine between pretrained embeddings but a learned metric with two trained heads projecting into a shared space, trained with a goal-contrastive loss and a temporal progress hinge. Removing the goal-contrastive term drops ALFRED beam-4 offline SR from 54.3 to 41.5, and removing the progress term drops BEHAVIOR from 6.3 to 4.3. We agree the submitted version did not separate the claim from the evidence clearly enough: these ablations, and several more in our answers to the questions below, should have been in the paper. The revision adds them and tightens the contribution statement accordingly.

W-c / Q1 (procedural-planning-specific baselines). The paper does compare against procedure-specific methods where benchmark protocols allow: VLWM-critic (the strongest published procedure-specific planner on WorldPrediction-PP, 45.4, which we discuss as the caption-based ceiling) and OEPP (36.8), alongside the VLM baselines. For procedural planning specifically, we implemented a VidAssist-style search framework as requested by the reviewer: VLM candidate proposal and VLM self-evaluation under beam search, refining the one-shot plan with a final visual re-verification, on the same Qwen3-VL backbone used for our zero-shot baselines, and ran it under our exact offline planning protocol:

Planner (identical windows)ALFRED SRmAccBEHAVIOR SRmAcc
Qwen3-VL-8B, one-shot (paper protocol)23.837.85.517.7
Qwen3-VL-8B + VidAssist-style search (beam 4)25.342.16.119.1
JEPA-Pro (beam 4, same windows)61.273.38.020.9

The search framework does improve the VLM: it refines 38–44% of plans and gains on every metric on both datasets (ALFRED SR +1.5, mAcc +4.3; BEHAVIOR SR +0.6, mAcc +1.4), yet it closes only a small part of the gap to JEPA-Pro on the identical windows. The revised version includes this baseline in the planning section, with the full protocol and metrics in the appendix.

[1] Maes et al., "LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels," arXiv:2603.19312, 2026. [2] Sobal et al., "Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models," NeurIPS 2026. [3] Zhou et al., "DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning," ICML 2025.

AuthorsOpenReview ↗

Response to Reviewer VdfN: state probing, longer horizons (part 2)

W-b / Q2 (do the embeddings capture world state?). In high-level world modeling, we argue that the state should capture the consequence of a macro action, not the physical detail of its execution. After "pick up the mug," the state must reflect that the mug is now in hand and off the table; how fast it was grasped, with which hand, along which trajectory is deliberately abstracted away unless explicitly specified in the action. The estimator learns this abstraction because it is trained on procedural data where the same action is executed in vastly different scenes and styles, so what the embedding should capture is what the action changes. Poses, velocities and contact dynamics belong to the low-level controller of a hierarchical system, and our action vocabulary carries no low-level content for the state to track. We tested directly whether this symbolic state is decodable from the embeddings, probing ground-truth state on ALFRED (probes fit on unseen-train, evaluated on unseen-test; the probed embeddings are vision-only):

Probe targetRandom baseline (shuffled labels)Linear, predicted state ŝ
Held object (48 classes)41.478.2
Agent location (53 classes)13.874.7

The predicted states carry linearly decodable possession and location state. The residual errors come from partial observability, not from the transitions themselves. The possession transition itself is read nearly perfectly (0.2% error on empty-hand steps); the remaining error is fine-grained identity of the in-hand object among look-alike classes, peaking one step after the grasp, when the object is occluded in the hand. At that point the identity is no longer visible in the observation. Likewise, location error grows during extended close-up manipulation when the scene is out of view, and prediction restores exactly this: at the location-change step the probe errs 12.9% on the predicted state vs 26.0% when reading the current observation's embedding. We agree the embeddings represent only a partially observable state, and the revision states this scope alongside the probe table.

W-e / Q3 (horizon T=4 and "simulated-only"). T=4 is the standard published Procedural Planning task definition (Chang et al. [4]; Patel et al. [5]), which we adopt for comparability. Two of our benchmarks already probe longer horizons on real video: ENACT requires ordering up to 10 steps (JEPA-Pro is one of only five systems above chance at L=10 forward), and three of five benchmarks (ENACT, WorldPrediction, EgoPlan) are real egocentric video, not simulation; only the planning adaptation study is simulated. We additionally ran the requested extension: the unchanged planner (same heads, same beam search) rolled out to T ∈ {4, 6, 8} on both simulators, evaluated on mAcc on the fixed subpopulation of test windows with ≥8 future reference steps, so horizons are compared on identical queries:

TALFREDBEHAVIORrefs containing a repeat (ALFRED / BEHAVIOR)
424.813.18% / 27%
623.011.530% / 51%
821.59.965% / 73%

Performance declines proportionally with horizon. Most of the loss at added depths comes from repeated actions: as the last column shows, long references increasingly re-execute earlier actions, which our planner is designed to avoid. For 10+ step tasks we see two requirements, which we discuss in the revision: action representations that distinguish re-executing an action in a new context from a degenerate repeat, and shorter-horizon replanning rather than deeper open-loop rollout. At 0.6–1.6 s per search (see Q5), replanning after every executed step is already practical and probably an easy lever for longer horizons.

AuthorsOpenReview ↗

Response to Reviewer VdfN: failure analysis, search cost, VL-JEPA comparison (part 3)

W-f / Q4 (failure analysis). We ran this analysis on the retained beam-4 predictions for procedural planning (ViT-L reproduction; ALFRED 1,106 / BEHAVIOR 8,075 windows); it is in the revised appendix. While long-horizon drift can be an issue, in the current setup one common failure case is propagation of early mistakes. When the planner fails to output the correct first step, it very rarely recovers in the following 2–4 steps of the same window. Grounding the very next step, not long-horizon drift, dominates; this is why we report nAcc alongside SR. Only ~26% of first-wrong windows recover any later position:

ALFREDBEHAVIOR-1K
failed windows already wrong at position 1 (%)6381
accuracy at positions 2–4 when position 1 is wrong (%)2111
accuracy at positions 2–4 when position 1 is correct (%)74–9015–34

Many exact-match "failures" are procedurally plausible plans. A recurring pattern is the correct procedure shifted by one step (pick → microwave → heat → navigate vs reference navigate → pick → microwave → heat), which exact match scores 0. A blinded VLM judge (Qwen3-VL-8B, given goal and executed history, with model and reference plans judged unmarked under the identical protocol):

Plans rated structurally executableALFREDBEHAVIOR-1K
our failed plans46.7%26.0%
hidden reference plans (judge calibration)73.3%51.7%

Relative to the judge's own ceiling, one half to two thirds of exact-match failures are coherent continuations rather than incoherent plans; plans judged executable also overlap the reference more (mIoU 0.47 vs 0.32).

W-f / Q5 (search cost). The search does not enumerate 256 full sequences: beam search expands width × pool candidates per depth with shared prefixes, and all candidate expansions at each depth are scored in a single batched forward. Measured on one H200: a complete beam-4, four-step plan costs 0.55 s on ALFRED and 1.40 s on BEHAVIOR end to end (plus 0.07–0.18 s perception), within 5–10 GiB of GPU memory, faster than a single zero-shot Qwen3-VL-8B generation for the same query on the same GPU. On both datasets, step-level decisions occur at multi-second intervals, so test-time search at these widths is practical in real time. The search budget is also a deployment parameter: a cheap reactive proposal can serve as the first attempt, with full world-model search engaged only when the plan score signals uncertainty. We also measured how the cost scales with the action space (BEHAVIOR-1K):

Pool sizebeam 1beam 2beam 3beam 4
1000.11 s0.20 s0.28 s0.37 s
2000.19 s0.34 s0.50 s0.65 s
3000.25 s0.45 s0.66 s0.86 s
4000.32 s0.60 s0.87 s1.14 s
491 (full)0.39 s0.73 s1.06 s1.40 s

Latency grows linearly in pool size and beam width. For scale: the average BEHAVIOR-1K step lasts 17.8 s (median 14.0 s), so even the most expensive configuration (beam 4 over the full 491-action pool) runs much faster than the time it takes to execute a single step. Test-time search is comfortably real-time at procedural timescales.

Q6 (comparison with VL-JEPA and JEPA variants). This comparison is in the paper for world modeling, and we have now extended it to EgoPlan-Bench with new results. It is the most controlled comparison available, because VL-JEPA is our initialization: every delta isolates what CTP training adds. The WorldPrediction comparison already reports VL-JEPA (BASE 63.9 / SFT 65.7) on WM against JEPA-Pro 71.8 under the same caption-free protocol (+6.1 over the strongest existing JEPA-family model). We emphasize what VL-JEPA structurally cannot do: it is a vision-text alignment model with no action-conditioned transition model, so it cannot predict future states; world-model scoring and procedural planning are undefined for it, and it can only be used reactively. We therefore evaluated VL-JEPA on EgoPlan-Bench in the two ways it can be used, under matched ViT-L scale and identical budgets:

EgoPlan-Bench, controlled ViT-L comparisonMacro acc.
VL-JEPA frozen + fresh predictor adapted on top (same budget)27.8
VL-JEPA frozen features + reactive probe (its only native mode)43.7
CTP-JEPA frozen features + identical reactive probe50.1
JEPA-Pro, world-model scoring (full)54.3

The results show CTP training is needed on both axes: reactively, CTP features beat the initialization by +6.4 under the identical probe, and an action-conditioned transition model cannot be recovered post hoc from the frozen alignment features. These numbers are now a table in the main text of the revision.

[4] Chang et al., "Procedure Planning in Instructional Videos," ECCV 2020.
[5] Patel et al., "Pretrained Language Models as Visual Planners for Human Assistance," ICCV 2023.

Reviewer_DxgtOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a JEPA-based world model for high-level procedural planning. It jointly trains a state encoder and an action-conditioned predictor to model procedural state transitions in latent space, together with a learned goal-conditioned cost function for evaluating candidate action sequences through test-time search. Extensive experiments on diverse datasets and evaluation tasks demonstrate its strong performance in procedural understanding and planning.

Critical strengths

A. Novel Paradigm for Procedural Planning. The paper introduces an insightful world model-based approach that extends action-conditioned state prediction from low-level control signals to high-level procedural actions. It presents a complete framework covering world model training, goal-conditioned cost estimation, and test-time planning, offering a promising new direction for procedural planning.

B. Comprehensive Experimental Validation. The extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of the proposed approach. The strong results provide convincing evidence for the potential of world models in high-level procedural understanding and planning, offering valuable insights for future research.

Critical weaknesses

No major weakness was found.

Questions for authors

A. In procedural planning, multiple valid action sequences may achieve the same goal. Therefore, a predicted sequence that differs from the ground truth does not necessarily indicate planning failure. How do the authors think this limitation could be addressed?

B. The proposed model represents actions using textual embeddings. However, in real-world scenarios, different executions of the same action may lead to different state transitions. How could the proposed framework account for such variations, for example, though alternative action representations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer bxgt: order-insensitive metrics and action-representation extensions

Thank you for the positive assessment. Both questions touch issues we care about, and we answer each below.

A (multiple valid sequences vs exact-match metrics). We agree; exact-match SR is a conservative lower bound that follows the original task definition to stay comparable with prior work. To expose exactly the effect the reviewer describes, we report two complementary metrics computed on the same retained predictions: next-action accuracy (nAcc) and order-insensitive plan overlap (mIoU), the latter directly discounting valid reorderings (ViT-L reproduction, beam-4, all windows):

SRmAccnAccmIoU
ALFRED (1,106 windows)61.273.375.576.4
BEHAVIOR (8,075 windows)7.720.525.139.3

The SR-to-mIoU gap (7.7 vs 39.3 on BEHAVIOR) quantifies how much of the "failure" mass is plans containing the right actions outside the single reference order. The principled resolution is closed-loop evaluation (scoring goal achievement rather than agreement with one reference), which our world-model formulation directly supports: the cost is goal-conditioned, not reference-conditioned. We flag this as the next step. A second mitigation is scoring against the set of reference trajectories reaching the same goal where datasets provide them. The revised version makes this limitation explicit: the conclusion now states that offline SR is a conservative lower bound, supported by a blinded plausibility audit of exact-match failures in the appendix.

B (action representation under execution variation). Text embeddings give an open-vocabulary, composable action space, and two training mechanisms absorb execution variability: the predictor is trained across many executions of the same textual action in different contexts, so the predicted next state models the distribution's central tendency given context, and counterfactual action negatives force predictions to depend on the action rather than the context alone. The framework itself is agnostic to the action encoder: richer representations such as continuous robot actions, action + duration tokens, or multimodal (text + exemplar clip) embeddings can plug in without architectural change, and extending to stochastic prediction heads for multimodal outcomes is a natural extension we now mention in the discussion.

Reviewer_1WL4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JEPA-Pro, an action-conditioned latent world model for procedural planning at the level of semantically meaningful steps. A causal state estimator aggregates video history, while a jointly trained predictor forecasts successor-state embeddings conditioned on actions. Training combines contrastive prediction, variance–covariance regularization, and an action-recovery auxiliary objective. A goal-conditioned cost scores imagined states, enabling search over action sequences. The model is trained on 4.68 million annotated action steps and evaluated on ENACT, WorldPrediction, and EgoPlan-related benchmarks. The paper also reports adaptation and search-width gains on offline four-action prediction tasks derived from ALFRED and BEHAVIOR-1K. These latter metrics measure agreement with reference action sequences rather than closed-loop task completion.

Critical strengths

1.The problem is practically relevant, and the modeling granularity is well aligned with high-level decisions. Organizing transitions around semantic actions rather than fixed temporal intervals is a reasonable approach to variable-duration procedures. Combining historical state aggregation with action-conditioned prediction is also appropriate for partially observed egocentric videos. Predicting intermediate embeddings provides an attractive alternative to generating images or textual state descriptions at every imagined step.

2.The experiments cover complementary offline tasks and report competitive results. JEPA-Pro-giant achieves 71.8% on WorldPrediction-WM and 58.7% macro accuracy on the EgoPlan-related three-level evaluation. Results on WorldPrediction-PP and ENACT, together with improvements from wider search across model sizes, indicate empirical promise. Adapting small modules while freezing the world model is also an interesting direction.

Critical weaknesses

1.The incremental contribution over existing model-based procedural planning is insufficiently established. The overall combination of action-conditioned state prediction, a learned critic, and search has precedents: VLWM, which the paper itself discusses, already employs this structure using textual state representations. JEPA-Pro’s more specific contribution therefore lies in step-level continuous-state prediction and joint learning of the state encoder and predictor. However, the experiments do not sufficiently isolate the value of these choices. Section 5.4 adapts both a predicted-state cost and an action-history prior, without prior-only, state-cost-only, or matched-supervision direct-policy controls, while the principal VLM baselines remain zero-shot. The Table 3 gains may therefore reflect additional supervision, procedural priors, and search, rather than learned state transitions specifically. Key controls, such as frozen versus jointly trained encoders and a context-free action-semantic predictor, are also absent, as are computational measurements substantiating the claimed efficiency advantage over textual-state rollouts. These gaps do not negate the potential novelty of continuous-representation planning, but limit the evidence for its independent contribution and claimed advantages over existing approaches.

2.The correctness conditions and implementation of key evaluation algorithms are insufficiently specified. State estimation and prediction in Section 3.1 explicitly depend on history, whereas Section 5.1 claims exact subset dynamic programming and exact linear assignment. Standard formulations require suitable optimal substructure and independently decomposable assignment scores, respectively; history-dependent state and action scores do not automatically satisfy these conditions. The paper provides no recurrence, assignment-matrix construction, or evaluation-time context restrictions establishing the claimed decomposition. Furthermore, the stated IDM recovers an outgoing action from the current state, leaving its temporal relationship to the transition equation unclear, despite directly contributing to benchmark scores. These ambiguities do not establish an implementation error, but prevent verification of important results from the current description.

Questions for authors

1.ENACT search: Please provide the dynamic-programming state and recurrence for forward ordering and the assignment-matrix construction for inverse ordering. How are history-dependent state estimates and dependencies on previously selected actions handled? If context is truncated or fixed during evaluation, please specify this and the precise objective for which search is exact.

2.IDM and temporal indexing: Please clarify which video segment and state transition each action labels, the actual IDM inputs, and the action-recovery term used in WorldPrediction scoring. Does the head recover an executed action from its resulting state change, or predict an upcoming action from the current state?

3.Limited attribution experiments: Under identical adaptation data and search budgets, how do an action-prior-only model, a world-model-cost-only model, their combination, and a direct action-prediction baseline compare? How much performance can a context-free action-embedding predictor explain?

4.Data and reproducibility: Please specify task-, trajectory-, and source-video-level train/validation/test separation and overlap exclusions. Please also provide the actual number of Cost_full trajectories, principal loss weights, the progress-loss definition, and the formula combining state cost and action prior.

5.Efficiency: Can the authors report end-to-end inference latency or computation, including visual encoding, candidate scoring, and search, and compare planning quality under matched compute budgets? Parameter counts and beam widths alone do not establish practical efficiency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1WL4: search exactness, IDM indexing, attribution (part 1)

We thank the reviewer for the in-depth review. We have run each of the requested controls and specify the evaluation procedures below exactly as implemented. We revised our submission regarding all the points mentioned by the reviewer in either the main body or the supplementary materials.

Q1 (ENACT search: recurrence, assignment construction, history handling). The reviewer is right that history-dependent scoring would not decompose. The resolution is that we do not use history in this evaluation: each observation is encoded independently, and each transition is scored independently. The edge score is e(t,i→j)=12[cos⁡(s^(si,at),sj)+cos⁡(a^ij,at)]e(t, i \to j) = \tfrac{1}{2}\left[\cos(\hat{s}(s_i, a_t), s_j) + \cos(\hat{a}_{ij}, a_t)\right], which depends only on (t,i,j)(t, i, j), never on the rest of the ordering.

  • Forward ordering. Because the edge scores are path-independent, we can order the candidates exactly with dynamic programming. The state is (set of already-placed images MM, current image ii), and V(M∪{j},j)=max⁡i[V(M,i)+e(∣M∣,i→j)]V(M \cup \{j\}, j) = \max_i \left[V(M, i) + e(|M|, i \to j)\right], starting from image 0, each candidate used once. This is exact for the summed edge-score objective.
  • Inverse ordering. The observations are already in order, so the transition slots are fixed. Scoring each action tt in each slot gives an n×nn \times n matrix S[t,slot]S[t, \mathrm{slot}], and the problem becomes a linear assignment, which we solve exactly.

Search is therefore exact for the decomposed objective defined by this evaluation-time context restriction; it is not a claim of exact search under full-history scoring. We state the restriction, the recurrence and the matrix construction in the appendix of the revised version (the evaluation code implements exactly the above).

Q2 (IDM temporal indexing). The IDM recovers an executed action from its consequence. Writing the data as interleaved sts_t, ata_t, st+1s_{t+1}, the IDM head maps st+1s_{t+1} → a^t\hat{a}_t: given the state an action produced, it reads out which action must have been executed to reach it. It never sees or predicts an upcoming action, so there is no future leakage. Unlike the standard inverse-dynamics formulation (st,st+1)(s_t, s_{t+1}) → ata_t, ours takes st+1s_{t+1} alone: states are encoded from short video segments rather than single frames, so st+1s_{t+1} already contains the visual change the action produced, and the pair input is unnecessary. In WorldPrediction-WM scoring, we encode each candidate clip and apply the IDM to its encoded state to get that candidate's action embedding a^\hat{a}, then match it against the query action by cosine similarity. The reported score per candidate is 12[cos⁡(s^1,s1)+cos⁡(a^,a)]\frac{1}{2} [\cos(\hat{s}_1, s_1) + \cos(\hat{a}, a)]. We have made the indexing explicit in Sec. 3.1 and in the WM protocol description in the revised version.

Q3 (attribution: prior-only / cost-only / combination / direct policy / context-free predictor). We ran exactly this decomposition on the reproduced ViT-L planner: identical frozen world model, identical candidate menus, histories, horizons and tie-breaks, beam-4. The reactive head is the direct-policy control: a goal-conditioned next-action scorer reading the current estimated state, the action history and the candidate action, trained on the same frozen features with the same adaptation trajectories and the same update budget as the cost head, and decoded with the same beam search. It never imagines a future state.

PlannerALFREDBEHAVIOR
cost + prior (full)61.27.7
cost only54.36.3
prior only (no rollout)10.60.6
direct policy (no rollout)20.32.1

Scoring imagined successors is the dominant term: every planner without rollout stays at or below 20.3 / 2.1 (26.3 / 2.6 with the prior added), while the identical cost head reaches 54.3 / 6.3 once it scores predicted next states; the prior adds ~7 / 1.4 on top. Supervision, priors and search alone do not explain our procedural-planning results; the predicted transitions do. On the context-free predictor: it cannot be dropped into the planning pipeline as-is, because planning also needs the cost and prior heads trained on the predictor's state space, and training new heads on it would measure adaptation rather than the initialization. We therefore compare where the predictor is used on its own, with no adapted heads: on WorldPrediction's direct transition-matching protocol the pretrained (context-free) initialization scores 61.8 vs 64.1 for the CTP-trained model, and frozen-feature reactive probes improve 43.7 → 50.1 with CTP training. The gain comes from the jointly trained representation, not the initialization.

AuthorsOpenReview ↗

Response to Reviewer 1WL4: reproducibility details, end-to-end efficiency (part 2)

Q4 (data and reproducibility). Thank you for raising this. We agree every one of these details belongs in the paper, and we have added a reproducibility appendix in the revised version covering each point. Concretely, per sub-question:

  • Split construction. No trajectory appears in more than one of seen/train, unseen/train and unseen/test. The seen/unseen task split is exact on BEHAVIOR-1K (none of the 50 unseen goal strings appears in seen/train) and holds for 169 of 189 ALFRED test tasks (20 templated goal strings also occur in seen/train scenes; we state this in the appendix). unseen/test shares goals with unseen/train by design: the cost adapts to the task distribution while test trajectories are held out.
  • Cost-adaptation cohorts. Cost_full uses 1,217 unseen-train trajectories on ALFRED (8,078 action steps, 356 unique goals) and 9,500 on BEHAVIOR-1K (162,238 action steps, 50 goals).
  • Pretraining loss. L=LInfoNCE+λVCLVCReg+λIDMLIDML = L_{\mathrm{InfoNCE}} + \lambda_{\mathrm{VC}} L_{\mathrm{VCReg}} + \lambda_{\mathrm{IDM}} L_{\mathrm{IDM}}, with λVC=0.1\lambda_{\mathrm{VC}} = 0.1 and λIDM=1.0\lambda_{\mathrm{IDM}} = 1.0. The prediction InfoNCE is symmetric with in-batch negatives and temperature τ=0.05\tau = 0.05, plus one counterfactual action negative per step; the IDM loss is a cosine loss.
  • Progress loss. A hinge on consecutive costs, Lprog=mean[ReLU(0.05+ct+1−ct)]L_{\mathrm{prog}} = \mathrm{mean}[\mathrm{ReLU}(0.05 + c_{t+1} - c_t)] over advancing pairs, with c=1−cos⁡(proj(s),proj(g))c = 1 - \cos(\mathrm{proj}(s), \mathrm{proj}(g)), applied equally to predicted and estimated states.
  • Plan score. Cumulative rollout cost + 0.5×+\, 0.5 \times prior cost summed over the horizon; the weight is fixed and shared across both environments.

Q5 (efficiency). We agree that parameter counts and beam widths alone do not establish practical efficiency, so we measured it end to end (one NVIDIA H200, bf16, 100 seeded queries per dataset, warmup excluded, fresh search state per query):

Per-query latencyALFRED (179 actions)BEHAVIOR (491 actions)
Search, beam 10.17 s0.39 s
Search, beam 20.30 s0.73 s
Search, beam 30.42 s1.06 s
Search, beam 40.55 s1.40 s
Visual state estimation~0.066 s (~16 frames @384px)~0.18 s (~40 frames)
Peak GPU memory5.0 GiB10.2 GiB

Visual state estimation encodes one 4-frame segment per executed step; the frame counts differ because the median ALFRED query carries 3 prior steps of history and the median BEHAVIOR query 9. Search stays cheap because latent rollouts are batched candidate scoring, not sequential generation (beam 4 scores on average 1,483 / 5,714 predictor and cost rows per query). The practical reference point is the environments' own pace: the median ground-truth step lasts 4.6 s on ALFRED and 14.0 s on BEHAVIOR (mean 17.8 s), so even beam-4 search plus perception (~0.6 s / ~1.6 s) completes well within a single step, fast enough not only to plan once per window but to replan at every step. We have added the table and these specifications to the appendix of the revised version.

Thank you for the author's additions. I have two points I would like to clarify: 1.Could you use a frozen pre-trained context-free predictor to retrain the cost head and prior head, and compare the planning results under the same data and search settings? If you have not separately compared the frozen encoder and joint training, it is recommended to limit the attribution of the gains to the overall training of CTP. 2.Could you separately report the results of the 169 tasks in ALFRED whose goals do not overlap with those in seen/train, and specify the deduplication rules for source videos and overlapping segments? Since unseen/train and unseen/test share the same goals, please also clarify which training stage "unseen" refers to.

Reviewer_BEeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends JEPA World Models from low-level control to high-level procedural planning, where JEPA-Pro uses semantically meaningful procedural steps as the units of prediction. In addition, the authors propose CTP-JEPA, which learns predictive state representations through joint training. Rather than relying entirely on a frozen visual encoder, CTP-JEPA jointly optimizes the State Encoder and Causal Predictor, enabling the learned representations to better support subsequent dynamic prediction tasks.

Critical strengths
  1. Experimental results demonstrate that JEPA-Pro, particularly JEPA-Pro-Giant, achieves substantial improvements in accuracy.
  2. The method is clearly illustrated.
Critical weaknesses
  1. Insufficient validation of the motivation for high-level procedural planning. JEPA-Pro defines each procedural step using a textual action description and predicts the next state based on the corresponding video clip. However, it remains unclear whether the model can effectively learn state transitions involving variable action durations and non-contiguous interactions. The authors should provide further experimental evidence to validate this key motivation.
  2. Novelty. Insufficient differentiation from existing JEPA-based world models. The paper proposes CTP-JEPA, which jointly trains the State Encoder and Causal Predictor using InfoNCE, VICReg, and Inverse Dynamics objectives to learn predictive state representations. However, existing works such as DINO-WM [1] and PLDM [2] have already explored latent state prediction and world model learning. The authors should further clarify the fundamental differences between CTP-JEPA and these approaches in terms of network architecture, training strategy, and state representation learning.
  3. Whether the temporal structure of the training data supports long-horizon procedural planning? The paper uses 4.68M annotated action steps and 2.51M procedural trajectories for training. However, as reported on Page 5, 93.4% of the trajectories are single-step trajectories, while only 6.6% contain multiple consecutive steps. Although these multi-step trajectories account for approximately half of the action steps, their proportion remains relatively small. Given the goal of learning high-level procedural transitions and supporting long-horizon planning, could this data distribution limit the model's ability to capture complex state dependencies and cross-step interactions? Can the authors provide further analysis of this issue?
  4. Some mistakes:

​ (1) Introduction, Page 1, "people often performs unrelated actions in between" should be "people often perform unrelated actions in between".

​ (2) Introduction, Page 2, "The encoder and predictor of JEPA-Pro is optimized at the same time" should be "The encoder and predictor of JEPA-Pro are optimized jointly".

​ (3) Related Work, Page 3, "but the encoder remain frozen" should be "but the encoders remain frozen".

​ (4) Methodology, Page 4, "An action-conditioned state prediction module predict the embedding" should be "An action-conditioned state prediction module predicts the embedding"

​ (5) Page 7, WorldPrediction, "a clip depict the same actions" should be "a clip depicting the same actions"

​ (6) Page 9, Results, "as shown in Figure 6)" should be "as shown in Figure 6"

​ (7) Page 9, Limitations, "Absolute performance on BEHAVIOR-1K remain relatively low" should be "Absolute performance on BEHAVIOR-1K remains relatively low".

[1] Zhou, G., Pan, H., LeCun, Y., & Pinto, L. (2025). DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 79115–79135.

[2] Sobal, U., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G. J., & LeCun, Y. (2025). Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models. Advances in Neural Information Processing Systems (NeurIPS 2025), 38.

Questions for authors

As noted in the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BEeV: clarifications and revised pdf

Thank you for the careful reading and the constructive assessment. We revised our submission regarding all the points mentioned by the reviewer in either the main body or the supplementary materials, and we address each point below.

W1 (variable action durations and non-contiguous interactions). Several of our design choices and evaluations target exactly this setting:

  • Since the prediction stride is semantic, each training segment is inherently a variable-length clip, so duration variability is absorbed directly at the data level.
  • We add random context dropout during training, so the model learns to predict even when intermediate states are missing, which is the non-contiguous case.
  • In ENACT's ordering tasks, the model must order up to 9 observations of extended interactions, where consecutive observations are separated by long, variable gaps. JEPA-Pro attains non-zero forward accuracy at L=10L=10, matched by only 4 of the 30 public VLM baselines.
  • In WorldPrediction, candidates are action equivalents (the same semantic action drawn from a different video context), so the clips conditioning the predictor are non-contiguous and of variable duration by construction.

We further stratified WorldPrediction by action duration (ViT-L reproduction):

Accuracy by duration quartile (%)Q1 (1–3.6 s)Q2 (3.6–6.6 s)Q3 (6.6–13.6 s)Q4 (13.6 s+)
WP-WM, text protocol65.367.668.959.4
WP-WM, vision protocol60.865.365.162.3
WP-PP, by clip duration42.040.136.644.1

WM accuracy is stable across transition durations spanning 1–59 s with only a mild dip in the longest quartile, and PP shows no degradation trend: the model's capability does not depend on a fixed band of action durations. This analysis is included in the revised appendix.

W2 (differentiation from DINO-WM and PLDM). We have made this comparison explicit in Related Work in the revision. The differences are substantive on all three axes mentioned:

  • Domain and abstraction level (the main difference). DINO-WM is not a CTP-JEPA: it trains a predictor on top of a frozen DINOv2 encoder, and the representation itself is never trained for prediction. PLDM is a CTP-JEPA (as is LeWorldModel [1]), but trained at the frame level for low-level control (continuous actions in simple simulated environments). JEPA-Pro jointly trains the state estimator and the action-conditioned predictor at the level of semantic procedural steps, conditioned on open-vocabulary text actions, on noisy real egocentric video of human activities. Demonstrating that end-to-end world-model training works at this level is our central claim.
  • Architecture. Operating at step level requires a trained state-abstraction stack (block-causal projector + attentive pooler, one state vector per semantic step, history aggregation under partial observability) that neither prior work has, or needs, at the frame granularity where they operate directly on patch-level representations.
  • Training strategy. The frozen-vs-joint distinction is important: reproducing the frozen-encoder recipe on our own foundation (fresh predictor on frozen features, identical budget) reaches 27.8 EgoPlan macro accuracy vs 54.3 for the CTP-trained model, and freezing the estimator with mean pooling costs 26.0 points (54.3 → 28.3). The recipe for joint training at the semantic-step level (symmetric InfoNCE + VCReg + IDM + counterfactual action negatives) is part of the technical contribution; we are not aware of prior work providing one at this level.

W3 (trajectory-length distribution of the training data). The 93.4% figure counts trajectories, but it does not reflect training exposure. One training sample is a (context, action, next state) triplet over steps, so the relevant ratio is at the step level: 93.4% of 2.51M trajectories contribute one step each (~2.34M of the 4.68M total), meaning multi-step trajectories account for the other half of all training samples. Each multi-step trajectory is additionally packed into windows of up to 16 steps, so cross-step structure is seen in every one of those samples. The exposure is therefore roughly balanced (~50/50), not 93/7. The single-step pairs are also not filler: they are dense supervision for transition semantics (the context-free case the predictor must also learn), while the multi-step half supplies the cross-step structure. This structure is directly evaluated by ENACT: ordering up to 9 shuffled steps collapses to chance if cross-step dependencies are not modeled.

W4 (typos). Thank you for pointing out these mistakes; all seven are fixed in the revision.

[1] Maes et al., "LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels," arXiv:2603.19312, 2026.

Comments on the paper

1 comment

Dear Reviewers,

Authors posted their responses to your queries. Please read these messages and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.