- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces JEPA-Pro, a JEPA-based world model for procedural planning. The architecture follows the Causal Temporal Prediction (CTP)-JEPA paradigm: a state estimation module aggregates visual observations into state embeddings, an action-conditioned causal predictor autoregressively predicts future state embeddings, and a cost module evaluates predicted states against goal embeddings. Planning is performed via test-time search over candidate action sequences in latent space, selecting sequences that minimize cumulative cost. The model is trained on 4.68M annotated steps from 2.51M procedural trajectories spanning egocentric human activities and robot recordings. Experiments demonstrate new state-of-the-art results on WorldPrediction-WM (71.8%) and EgoPlan-Bench (58.7%), and outperformance of 23/30 VLM baselines on ENACT. On ALFRED and BEHAVIOR-1K, the model achieves 64.6% and 10.0% success rates respectively with test-time search.
Critical strengths
a. The paper evaluates on five benchmarks (ENACT, WorldPrediction, EgoPlan-Bench, ALFRED, BEHAVIOR-1K) and demonstrates strong performance on several. The new state-of-the-art results on WorldPrediction-WM (71.8%) and EgoPlan-Bench (58.7%) are notable.
b. The paper shows that performance improves monotonically with beam width (Figure 6), demonstrating that the world model benefits from additional test-time compute. This is a useful finding for understanding the tradeoff between inference cost and planning quality.
c. The paper demonstrates that JEPA-Pro outperforms 23/30 VLM baselines on ENACT, including much larger models (GPT-4, Gemini). This suggests that the world model approach provides advantages over language-only reasoning for procedural planning.
Critical weaknesses
a. JEPA-Pro is a straightforward combination of existing components: VL-JEPA for representation learning, autoregressive transformer for state prediction, cosine similarity for cost, and beam search for planning. The CTP-JEPA formulation is just standard causal autoregressive prediction, there is no fundamental algorithmic advance. The paper dresses up these components with elaborate notation and a "world model" narrative, but the underlying technical contribution is thin.
b. The model predicts state embeddings, not world states. There's no analysis of whether the embeddings capture genuine world dynamics (e.g., object positions, physical properties). The paper shows the embeddings are useful for planning but doesn't establish they represent world state meaningfully. This is a representation learning approach, not a world model.
c. The paper compares against general-purpose VLMs but not against methods specifically designed for procedural planning (e.g., VidAssist, other search-based planning methods). This makes it unclear whether JEPA-Pro is the best approach or just better than general-purpose VLMs on this task.
d. Cosine similarity between state and goal embeddings is a crude measure of progress. It doesn't capture task structure, subgoal ordering, or the complexity of multi-step procedures. The paper doesn't analyze whether this cost function is sufficient for challenging tasks.
e. The paper only plans 4 steps ahead and all experiments are on simulated benchmarks. For a paper claiming to enable "autonomous agents that act in the physical world," this is a significant gap. The method's applicability to real-world procedural tasks with longer horizons is untested.
f. The paper doesn't analyze when the method fails or what types of tasks are challenging. Understanding failure modes is critical for assessing practical utility. The paper also doesn't discuss the computational cost of test-time search, which could be prohibitive for real-time applications.
Questions for authors
-
Can you provide a comparison with dedicated procedural planning methods (e.g., VidAssist, other search-based planning approaches) to establish whether JEPA-Pro is competitive with methods specifically designed for this task?
-
Can you analyze whether the learned state embeddings capture genuine world dynamics? For example, do they preserve object positions, velocities, or other physical properties?
-
Can you extend the planning horizon beyond T=4 steps? How does performance degrade with longer horizons, and what modifications would be needed to handle tasks requiring 10+ steps?
-
Can you analyze failure cases? What types of tasks does JEPA-Pro struggle with, and why?
-
Can you discuss the computational cost of test-time search? For beam-4 with T=4, the model evaluates 256 candidate sequences — is this practical for real-time applications?
-
Can you compare against VL-JEPA and other JEPA variants to establish whether the CTP-JEPA formulation provides advantages over existing approaches?
3 comments
Response to Reviewer VdfN: state probing, longer horizons (part 2)
W-b / Q2 (do the embeddings capture world state?). In high-level world modeling, we argue that the state should capture the consequence of a macro action, not the physical detail of its execution. After "pick up the mug," the state must reflect that the mug is now in hand and off the table; how fast it was grasped, with which hand, along which trajectory is deliberately abstracted away unless explicitly specified in the action. The estimator learns this abstraction because it is trained on procedural data where the same action is executed in vastly different scenes and styles, so what the embedding should capture is what the action changes. Poses, velocities and contact dynamics belong to the low-level controller of a hierarchical system, and our action vocabulary carries no low-level content for the state to track. We tested directly whether this symbolic state is decodable from the embeddings, probing ground-truth state on ALFRED (probes fit on unseen-train, evaluated on unseen-test; the probed embeddings are vision-only):
| Probe target | Random baseline (shuffled labels) | Linear, predicted state ŝ |
|---|---|---|
| Held object (48 classes) | 41.4 | 78.2 |
| Agent location (53 classes) | 13.8 | 74.7 |
The predicted states carry linearly decodable possession and location state. The residual errors come from partial observability, not from the transitions themselves. The possession transition itself is read nearly perfectly (0.2% error on empty-hand steps); the remaining error is fine-grained identity of the in-hand object among look-alike classes, peaking one step after the grasp, when the object is occluded in the hand. At that point the identity is no longer visible in the observation. Likewise, location error grows during extended close-up manipulation when the scene is out of view, and prediction restores exactly this: at the location-change step the probe errs 12.9% on the predicted state vs 26.0% when reading the current observation's embedding. We agree the embeddings represent only a partially observable state, and the revision states this scope alongside the probe table.
W-e / Q3 (horizon T=4 and "simulated-only"). T=4 is the standard published Procedural Planning task definition (Chang et al. [4]; Patel et al. [5]), which we adopt for comparability. Two of our benchmarks already probe longer horizons on real video: ENACT requires ordering up to 10 steps (JEPA-Pro is one of only five systems above chance at L=10 forward), and three of five benchmarks (ENACT, WorldPrediction, EgoPlan) are real egocentric video, not simulation; only the planning adaptation study is simulated. We additionally ran the requested extension: the unchanged planner (same heads, same beam search) rolled out to T ∈ {4, 6, 8} on both simulators, evaluated on mAcc on the fixed subpopulation of test windows with ≥8 future reference steps, so horizons are compared on identical queries:
| T | ALFRED | BEHAVIOR | refs containing a repeat (ALFRED / BEHAVIOR) |
|---|---|---|---|
| 4 | 24.8 | 13.1 | 8% / 27% |
| 6 | 23.0 | 11.5 | 30% / 51% |
| 8 | 21.5 | 9.9 | 65% / 73% |
Performance declines proportionally with horizon. Most of the loss at added depths comes from repeated actions: as the last column shows, long references increasingly re-execute earlier actions, which our planner is designed to avoid. For 10+ step tasks we see two requirements, which we discuss in the revision: action representations that distinguish re-executing an action in a new context from a degenerate repeat, and shorter-horizon replanning rather than deeper open-loop rollout. At 0.6–1.6 s per search (see Q5), replanning after every executed step is already practical and probably an easy lever for longer horizons.
Response to Reviewer VdfN: failure analysis, search cost, VL-JEPA comparison (part 3)
W-f / Q4 (failure analysis). We ran this analysis on the retained beam-4 predictions for procedural planning (ViT-L reproduction; ALFRED 1,106 / BEHAVIOR 8,075 windows); it is in the revised appendix. While long-horizon drift can be an issue, in the current setup one common failure case is propagation of early mistakes. When the planner fails to output the correct first step, it very rarely recovers in the following 2–4 steps of the same window. Grounding the very next step, not long-horizon drift, dominates; this is why we report nAcc alongside SR. Only ~26% of first-wrong windows recover any later position:
| ALFRED | BEHAVIOR-1K | |
|---|---|---|
| failed windows already wrong at position 1 (%) | 63 | 81 |
| accuracy at positions 2–4 when position 1 is wrong (%) | 21 | 11 |
| accuracy at positions 2–4 when position 1 is correct (%) | 74–90 | 15–34 |
Many exact-match "failures" are procedurally plausible plans. A recurring pattern is the correct procedure shifted by one step (pick → microwave → heat → navigate vs reference navigate → pick → microwave → heat), which exact match scores 0. A blinded VLM judge (Qwen3-VL-8B, given goal and executed history, with model and reference plans judged unmarked under the identical protocol):
| Plans rated structurally executable | ALFRED | BEHAVIOR-1K |
|---|---|---|
| our failed plans | 46.7% | 26.0% |
| hidden reference plans (judge calibration) | 73.3% | 51.7% |
Relative to the judge's own ceiling, one half to two thirds of exact-match failures are coherent continuations rather than incoherent plans; plans judged executable also overlap the reference more (mIoU 0.47 vs 0.32).
W-f / Q5 (search cost). The search does not enumerate 256 full sequences: beam search expands width × pool candidates per depth with shared prefixes, and all candidate expansions at each depth are scored in a single batched forward. Measured on one H200: a complete beam-4, four-step plan costs 0.55 s on ALFRED and 1.40 s on BEHAVIOR end to end (plus 0.07–0.18 s perception), within 5–10 GiB of GPU memory, faster than a single zero-shot Qwen3-VL-8B generation for the same query on the same GPU. On both datasets, step-level decisions occur at multi-second intervals, so test-time search at these widths is practical in real time. The search budget is also a deployment parameter: a cheap reactive proposal can serve as the first attempt, with full world-model search engaged only when the plan score signals uncertainty. We also measured how the cost scales with the action space (BEHAVIOR-1K):
| Pool size | beam 1 | beam 2 | beam 3 | beam 4 |
|---|---|---|---|---|
| 100 | 0.11 s | 0.20 s | 0.28 s | 0.37 s |
| 200 | 0.19 s | 0.34 s | 0.50 s | 0.65 s |
| 300 | 0.25 s | 0.45 s | 0.66 s | 0.86 s |
| 400 | 0.32 s | 0.60 s | 0.87 s | 1.14 s |
| 491 (full) | 0.39 s | 0.73 s | 1.06 s | 1.40 s |
Latency grows linearly in pool size and beam width. For scale: the average BEHAVIOR-1K step lasts 17.8 s (median 14.0 s), so even the most expensive configuration (beam 4 over the full 491-action pool) runs much faster than the time it takes to execute a single step. Test-time search is comfortably real-time at procedural timescales.
Q6 (comparison with VL-JEPA and JEPA variants). This comparison is in the paper for world modeling, and we have now extended it to EgoPlan-Bench with new results. It is the most controlled comparison available, because VL-JEPA is our initialization: every delta isolates what CTP training adds. The WorldPrediction comparison already reports VL-JEPA (BASE 63.9 / SFT 65.7) on WM against JEPA-Pro 71.8 under the same caption-free protocol (+6.1 over the strongest existing JEPA-family model). We emphasize what VL-JEPA structurally cannot do: it is a vision-text alignment model with no action-conditioned transition model, so it cannot predict future states; world-model scoring and procedural planning are undefined for it, and it can only be used reactively. We therefore evaluated VL-JEPA on EgoPlan-Bench in the two ways it can be used, under matched ViT-L scale and identical budgets:
| EgoPlan-Bench, controlled ViT-L comparison | Macro acc. |
|---|---|
| VL-JEPA frozen + fresh predictor adapted on top (same budget) | 27.8 |
| VL-JEPA frozen features + reactive probe (its only native mode) | 43.7 |
| CTP-JEPA frozen features + identical reactive probe | 50.1 |
| JEPA-Pro, world-model scoring (full) | 54.3 |
The results show CTP training is needed on both axes: reactively, CTP features beat the initialization by +6.4 under the identical probe, and an action-conditioned transition model cannot be recovered post hoc from the frozen alignment features. These numbers are now a table in the main text of the revision.
[4] Chang et al., "Procedure Planning in Instructional Videos," ECCV 2020.
[5] Patel et al., "Pretrained Language Models as Visual Planners for Human Assistance," ICCV 2023.
Response to Reviewer VdfN: contribution scope, learned cost and procedure-specific baselines (part 1)
We thank the reviewer for the detailed review. Before the point-by-point answers, we first address weakness a & d together and use that opportunity to clarify the scope of our work. We revised our submission regarding all the points mentioned by the reviewer in either the main body or the supplementary materials.
W-a, W-d, claim and scope. End-to-end latent world models are an established paradigm at the low level: LeWorldModel [1] and PLDM [2], as stated in Related Work, learn latent dynamics in fixed, controlled environments with low-level action spaces, and their value is not beating specialist policies on their benchmarks but showing that a learned dynamics model enables planning by search. Our paper carries this paradigm to the high level, with semantically abstract procedural steps, open-vocabulary text actions and real egocentric video, and trains it end to end, which to our knowledge no prior system had done. We want first to clarify that 1) the claim is an end-to-end-trained high-level world model that enables planning by search, in a noisier setting than fixed-camera manipulation and at a higher level of temporal abstraction; 2) VLMs are the baselines we report against because generating plans in token space toward a language goal is their natural ground, not because the goal is beating specialist models. Low-level world models are not judged that way either. We see high-level world modeling as one necessary building block toward enabling autonomous agents to act in the physical world, just as low-level world models are for current manipulation tasks.
The contribution is therefore not the invention of each block but making this joint training work at the high level, and we ran controlled ablations that bear directly on both weaknesses. On the combination claim: training a predictor on top of the frozen pretrained encoder (a DINO-WM [3] style recipe), under an identical adaptation budget, reaches 27.8 on EgoPlan-Bench vs 54.3 for our end-to-end model. On the cost: it is not raw cosine between pretrained embeddings but a learned metric with two trained heads projecting into a shared space, trained with a goal-contrastive loss and a temporal progress hinge. Removing the goal-contrastive term drops ALFRED beam-4 offline SR from 54.3 to 41.5, and removing the progress term drops BEHAVIOR from 6.3 to 4.3. We agree the submitted version did not separate the claim from the evidence clearly enough: these ablations, and several more in our answers to the questions below, should have been in the paper. The revision adds them and tightens the contribution statement accordingly.
W-c / Q1 (procedural-planning-specific baselines). The paper does compare against procedure-specific methods where benchmark protocols allow: VLWM-critic (the strongest published procedure-specific planner on WorldPrediction-PP, 45.4, which we discuss as the caption-based ceiling) and OEPP (36.8), alongside the VLM baselines. For procedural planning specifically, we implemented a VidAssist-style search framework as requested by the reviewer: VLM candidate proposal and VLM self-evaluation under beam search, refining the one-shot plan with a final visual re-verification, on the same Qwen3-VL backbone used for our zero-shot baselines, and ran it under our exact offline planning protocol:
The search framework does improve the VLM: it refines 38–44% of plans and gains on every metric on both datasets (ALFRED SR +1.5, mAcc +4.3; BEHAVIOR SR +0.6, mAcc +1.4), yet it closes only a small part of the gap to JEPA-Pro on the identical windows. The revised version includes this baseline in the planning section, with the full protocol and metrics in the appendix.
[1] Maes et al., "LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels," arXiv:2603.19312, 2026. [2] Sobal et al., "Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models," NeurIPS 2026. [3] Zhou et al., "DINO-WM: World Models on Pre-trained Visual Features Enable Zero-shot Planning," ICML 2025.