acceptodds
Under review as a conference paper at ICLR 2027

D-JEPA: A Decision-Aligned Latent World Model

18 views

Abstract

Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. We identify a decision-local prediction gap: among the few futures competing for execution, a candidate predicted closer to the goal can produce a worse realized outcome than an available alternative. We introduce D-JEPA, a decision-aligned latent world model that learns decision-relevant relations among candidate futures from executed outcomes. A bounded, permutation-equivariant operator jointly reasons over goal-relative predictive features and ordinal evidence, refining pretrained predictive geometry where action choices are most consequential. Restricted predictor adaptation and a shared ordinal interface extend this alignment across complementary predictive geometries. D-JEPA further realizes the learned decision structure in JEPA-compatible future representations, enabling deployment through native latent-distance planning. Evaluations across latent control, manipulation, pretrained action-producing models, physical robots and autonomous driving demonstrate improved action selection, including 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical robot tasks. These results establish decision-relevant relational structure as a direct bridge between predictive world modeling and effective control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent world models, joint-embedding predictive architectures, decision alignment, representation learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_KVBVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper works on latent world model which address the gap where predictive latent distance may not reliably reflect which candidate action will succeed when executed. It introduces a bounded, permutation equivariant relational alignment mechanism, and incorporates decision-relevant structure into future representations to support planning through native latent distances.

The main contributions are:

  1. identifying and quantifying the gap between predictive latent geometry and realized decision quality

2.proposing a shared relational alignment framework that can integrate complementary predictive representations

3.enabling decision-aligned preferences to be realized directly in JEPA-compatible future latent spaces

Exp results include an average success rate increase on four RoboTwin tasks from 61.72% to 76.76%, an increase on two physical PiPER tasks from 64.0% to 81.0%, and an increase in mean PDMS from 57.36 to 95.34 across seven selected driving scenes.

Critical strengths
  1. identify the limitation of latent prediction does not necessarly introduces good action
  2. propose bounded, permutation-equivariant relational alignment mechanism that learns preferences among candidate futures and incorporates decision-relevant structure into JEPA-compatible latent representations.
  3. it proves it method on several datasets including RoboTwin, PiPER, and automated driving
Critical weaknesses

Insufficient evidence regarding the dependence on execution-outcome supervision and candidate-set design. D-JEPA learns decision-relevant relations among candidate futures from execution outcomes. However, the practical effectiveness of this approach may depend on the quality and diversity of candidate actions, the availability and cost of outcome supervision, and whether the learned preferences remain reliable when the candidate distribution changes

Questions for authors

How sensitive is D-JEPA's performance to the quality, diversity, and number of candidate futures, as well as the amount of execution-outcome supervision?

What extent do the reported gains arise specifically from aligning latent representations with execution outcomes, rather than from additional predictor adaptation or training signals?

3 comments

Dear Reviewer KVBV,

Thank you for recognizing the problem, alignment mechanism and cross-domain evidence. We have conducted additional analyses to separately examine candidate availability, execution-supervision requirements and the contributions of the alignment components.

  1. More available candidates improve selection.

We evaluated nested subsets of the same 256 PushT confirmation starts, keeping model weights and thresholds fixed across 16 shared subset seeds. Ranks and relational outputs are recomputed within each subset.

CandidatesLeWM (%)TD-JEPA (%)D-JEPA relational (%)
880.4277.1081.96
1681.5277.4783.94
3282.1877.9586.28
6383.5976.9587.11

D-JEPA outperforms both native selectors at every candidate budget, improving from 81.96% to 87.11% as more alternatives become available, without retraining for each candidate count (Appendix D.1, p. 23; Table 16, p. 22).

  1. Execution-supervision requirements.

We additionally varied the number of fitting starts while keeping 64 calibration starts fixed, corresponding to 4,032 calibration candidate outcomes. Each fitting start supplies 63 executed candidate outcomes, with three training seeds evaluated under the same optimization and calibration protocol.

Fitting startsFitting labelsD-JEPA set alignment (%)
452,83585.29
1127,05683.85
22414,11287.50
44828,22486.33

All four budgets exceed the unchanged LeWM reference of 83.59%. Even with 45 fitting starts and the fixed calibration set, D-JEPA reaches 85.29%. The table reports all four prespecified fitting budgets; the full-budget configuration remains unchanged.

  1. Separating relational alignment from predictor adaptation.

The matched module comparison on the same 256 PushT confirmation starts gives:

ConfigurationSuccess (%)
Native TD-JEPA76.95
Predictive adaptation79.69
Relational alignment, fixed predictors87.11
Calibrated composition87.89

Relational alignment provides the largest individual improvement while keeping the predictive models fixed. Predictor adaptation contributes a separate improvement, and calibrated composition combines the two sources of evidence. Execution outcomes supervise relational alignment, while native predictive costs provide its rank inputs.

We further compared matched pointwise and set models using identical features, execution labels, losses and correction bounds. Pointwise scoring achieves 85.81%, compared with 86.33% for set alignment (set-minus-pointwise difference +0.52 pp; 95% CI [-1.17, 2.34]). A separate success-BCE scorer achieves 82.42%, with set alignment improving by 3.91 pp (95% CI [0.39, 7.55]).

The matched scoring controls examine candidate interaction and the learning objective under shared execution supervision, complementing the predictor-adaptation ablation above.

As an additional mechanism analysis, we fixed the predictors, trained weights, target tokens and all native ranks/base scores, changing only the latent descriptors of other candidates. The set model changes the target-pair score margin in all 64 calibration cases (median absolute margin change 0.0357), whereas the matched pointwise model has zero effect. This confirms that the learned set operator uses candidate context.

Section 6.3 and Table 3 summarize module contributions (pp. 9–10). Appendices C.1–C.2/Table 10 report matched scoring and context analysis (pp. 19–20); Appendix D.2/Tables 20–21 reports supervision-budget and candidate-content sensitivity (p. 24); Appendix A.8/Table 6 specifies the supervision protocol (p. 16).

Candidate quality and diversity at a fixed candidate count.

We additionally examined candidate composition while keeping the candidate count fixed at K=16. Each of the six proposal families was removed in turn, using unchanged selectors and outcome-independent subsets from the original 256-start sealed cohort, separate from the confirmation cohort. Candidate coverage measures whether a successful candidate is available, while diversity is measured by pairwise action-space RMS distance.

Removing small-noise proposals increases raw action-space diversity but reduces successful-candidate coverage and the performance of both calibrated fusion and learned selection. Other proposal-family removals have smaller effects. This indicates that geometric spread alone is not a sufficient measure of candidate quality: locally perturbed alternatives can provide useful executable candidates even when they contribute less to overall geometric diversity. We therefore examine candidate coverage alongside selection success rather than treating diversity alone as a proxy for candidate quality.

Changing the proposal distribution.

We also evaluated newly generated, reference-action-free CEM candidate pools with repeated replanning on 256 outcome-unfiltered PushT pairs. The evaluation includes the unchanged relational checkpoint and separately adapted set/pointwise heads trained on disjoint CEM fitting and calibration pools. The adapted set and pointwise models achieve 89.06% and 87.50% success, respectively (paired difference +1.56 pp; 95% CI [−0.20, +3.33]), while the original calibrated fusion reaches 88.28%. This provides an additional evaluation of both the existing selector and adaptation under a changed proposal distribution. The protocol, candidate coverage and closed-loop outcomes are reported in Appendix A.9/Table 7 (pp. 16–17).

AuthorsOpenReview ↗

A Sincere Note of Appreciation

Dear Reviewer KVBV,

We are genuinely grateful for your positive assessment of D-JEPA's problem formulation, alignment mechanism and evidence across tasks. Your questions about candidate quality, diversity and execution supervision have been particularly helpful in guiding our additional experiments and clarifications.

Given the different assessments across the reviews, we would sincerely appreciate it if you could reconsider the strength of your recommendation after reviewing our responses and the additional evidence. If these clarifications satisfactorily address your concerns, we would be grateful if you would consider a stronger recommendation. If the additional evidence also gives you greater confidence in the work, we would welcome a corresponding update to your confidence.

Thank you again for recognizing the value of the work and for engaging with us constructively.

Reviewer_xCnGOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a JEPA type approach to planning, notably addressing the well known problem that the latent distance may not be informative for the task. Unfortunately, the method proposed to remedy this state of affair is not presented in the form of well-structured text but as a bunch of crowded figures, flow charts and appendices. This may perhaps work for a presentation with running oral commentary, but not as a written paper in a major conference.

Critical strengths

The paper addresses an interesting problem and appears to give good results in Table 1.

Critical weaknesses

The paper is not written as a scientific paper and is essentially unreadable. It consists mostly of a successions of flow charts, tables and figures instead of readable text clearly identifying a problem, proposing a methodology to solve it, then presenting experiments and drawing conclusions. There may very well be good ideas in there, but the paper is unreadable as is.

Questions for authors

I strongly encourage the authors to rewrite their paper in conventional form.

8 comments

Dear Reviewer xCnG,

Thank you for your candid feedback on the paper's readability and for recognizing the interest of the problem and the quantitative results. Our original use of diagrams and tables was intended to make the method design and architecture more intuitive. However, we agree that their prominence sometimes interrupted the written explanation.

Following your suggestion, we have revised the main text, reducing it from 11 figures and 6 tables to 3 figures and 4 tables, while keeping it within the permitted 10-page limit. We made three focused changes:

Retain the essential evidence in the main text. Figures 1–3 preserve the problem overview, decision-local diagnosis and main method architecture (pp. 1, 3 and 4). Tables 1–4 present the core action-selection results, robotic execution, ablations and native-distance deployment (pp. 8–10). Supporting figures and condition-level tables have been moved to Appendices B, C, E and G, while their aggregate findings remain in the main text.

Present the method as a continuous explanation. Section 4 now connects the inputs, predictive descriptors, ordinal coordinates, learning objectives, predictor proposal, calibrated composition and representation realization through continuous prose (pp. 4–6). Experimental figures and tables no longer interrupt this discussion. We have also added concise explanations after the existing equations to clarify the role of each computation in action selection.

Reduce repetition and clarify the experimental setup. Section 5 distinguishes the evaluation populations and readout configurations (pp. 6–7). Section 6 presents the main findings alongside the four retained tables (pp. 7–10), with detailed breakdowns referenced in the appendix. The overview and main architecture remain in the main text, while supporting visualizations are organized alongside their corresponding explanations.

The conference permits and encourages updates to paper PDFs and supplementary materials during rebuttal. We have uploaded the revised PDF together with a blue-marked comparison highlighting the textual changes. We would be grateful if you could review the updated manuscript and reconsider its readability. Our following comment provides a more detailed guide to the revisions.

Thank you again for this constructive suggestion.

Thank you again for encouraging a clearer written presentation. To make the revisions easy to inspect, we detail below how the main text was reorganized from 11 figures and 6 tables to 3 figures and 4 tables. All locations refer to the revised anonymous PDF, and the blue-marked version highlights the textual changes.

  1. Give each retained display a distinct role.

Figure 1 provides the overall problem and method overview (p. 1); Figure 2 supports the decision-local diagnosis alongside Section 3 (p. 3); and Figure 3 presents the main method architecture alongside Section 4 (p. 4). Figures 1 and 3 retain their complete original artwork. The four main tables now serve the following roles:

Retained evidenceOriginal tableRevised location
PushT, Reacher and GranularTable 1Table 1, p. 8
RoboTwin and physical robotsTable 2Table 2, p. 8
Alignment mechanisms and predictive evidenceTable 5Table 3, p. 9
Native-distance deploymentTable 6Table 4, p. 10

The separate 32-start historical PushT diagnostic is now presented as Table 9 (p. 19), allowing the main action-selection table to focus on the core evaluation populations. All baseline rows and Granular metrics are retained. The ablation table explicitly distinguishes the 256-start confirmation population from the 128-start mechanism population, while the deployment table identifies the relational readout encoded by ordinal realization.

  1. Replace interruptions with a connected computational explanation.

Section 4 now opens by introducing the shared inputs and the roles of its three subsections. Section 4.1 (pp. 4–5) explains the goal-relative descriptor and ordinal coordinate, followed by the individual learning objectives and inference gate after Equations 1–2.

Section 4.2 (pp. 5–6) explains how adapted futures provide a complementary proposal through the fixed relational readout, followed by calibrated composition. It then explains how additional predictive geometries contribute complementary evidence.

Section 4.3 (p. 6) explains the displacement direction and magnitude bound after Equation 3, and how the radius specified in Equation 4 allows native goal distance to recover the aligned action.

These explanations can now be read consecutively, without intervening experimental tables or execution galleries.

  1. Preserve detailed evidence in the appendix while retaining the main findings in the paper.

The multi-geometry/composition and representation-lifting diagrams are now Figures 4–5 in Appendix B (p. 19). Execution sequences are presented in Appendices E and G (pp. 27, 31–32), while the component and distribution-shift plots are Figures 6–7 (pp. 20 and 22).

The original driving results table is now Table 24 in Appendix G.2 (p. 32), and the condition-level shape and appearance results are presented in Table 17 in Appendix C.5 (p. 22). Sections 6.1–6.2 retain the corresponding aggregate findings and direct references to these detailed results.

Section 5 also consolidates the evaluation populations, candidate interface and selection configurations, allowing readers to understand the experimental results without reconstructing the protocols from multiple separate displays.

The purpose of these changes is to let the written explanation carry the paper's main argument, with figures and tables supporting it at the relevant points. We appreciate your suggestion and the opportunity to make the presentation clearer and more accessible.

AuthorsOpenReview ↗

A Sincere Thank You and Request for Reconsideration

Dear Reviewer xCnG,

Thank you again for your feedback. We have carefully revised the manuscript to address your concerns about its presentation and readability, and sincerely hope the updated version is clearer.

We would be grateful if you could reconsider your original rating in light of these revisions. If there is anything we may have overlooked or any remaining issue, we would sincerely welcome your further feedback. We truly appreciate your guidance and your help in improving this work.

Thank you again for your time and consideration.

Reviewer_D57KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

D-JEPA addresses mismatches between predicted goal distance and executed outcomes by learning bounded, set-wise corrections over candidate futures. It combines complementary predictive models, restricted predictor adaptation, and calibrated selection, then encodes the learned ordering in terminal latent distances. Evaluations cover simulated control, RoboTwin, physical manipulation, driving, and selected shape and appearance changes.

Critical strengths
  1. The paper addresses a meaningful decision-level failure mode: accurate global prediction need not identify the best executable action. Shared candidate pools and bounded, permutation-equivariant corrections provide a coherent way to improve selection while controlling action availability.
    1. The evaluation includes substantial controls and statistical reporting. Matched candidates, disjoint fitting/calibration/test identities, paired bootstrap intervals, and module ablations support the reported selection gains. RoboTwin success improves from 61.72% to 76.76%, while paired physical-robot success improves from 64% to 81%.
Critical weaknesses
  1. Representation lifting establishes ranking compatibility rather than predictive validity. Equation 4 places terminal latents at radii determined by an already computed rank, making exact rank recovery true by construction. This demonstrates a useful planning interface, but not improved future-state prediction. Reacher’s transport diagnostic measures latent displacement while executed actions still use relational selection, leaving the representation-level contribution insufficiently validated.
  2. The incremental value of set-wise reasoning is not fully isolated. Execution-informed cost learning has precedents such as Traj-LeWM. Although the paper includes fusion and adaptation controls, the core PushT/Reacher comparisons lack a per-candidate scorer matched in inputs, outcome supervision, and correction constraints. The gains therefore do not clearly separate relational interaction from supervised reranking and complementary predictive evidence.
  3. The main PushT result uses a selectively constructed population. Appendix A.1 filters confirmation starts using candidate success/failure structure and small pretrained-score boundary gaps. This is a valid diagnostic setting, but the reported 87.89% does not establish performance on an unfiltered task distribution. Exact eligibility thresholds, retention rates, and evaluation with freshly generated candidates during closed-loop planning would clarify broader effectiveness.
  4. Application breadth does not establish low-cost generality. Several experiments use task-local fitting, and appearance tests reuse perturbation families seen during training. The RoboTwin grasping interface also introduces scene-conditioned geometry and motion-template proposals. These extensions are useful, but the cost of obtaining execution-labelled candidate sets and adapting the selector to new tasks remains unclear.
Questions for authors
  1. Beyond reproducing the learned order, do the modified latents remain consistent with encoded executed futures and temporal dynamics? What benefit does representation lifting provide beyond the relational readout?
  2. Does set attention outperform a per-candidate MLP or learned cost scorer with the same features, supervision, and correction constraints? How does the method differ from execution-informed trajectory-cost learning?
  3. What are the exact PushT eligibility thresholds and retention rate? Do the gains persist on unfiltered starts with newly generated candidates and repeated closed-loop replanning?
  4. What execution-label and calibration budgets are required per new task? Can you separate selector gains from task-specific proposal engineering and evaluate held-out perturbation families without refitting?
6 comments

Dear Reviewer D57K,

Thank you for recognizing the decision-level problem and the matched experimental controls. We have added further experiments to clarify the contributions of relational selection, predictive adaptation and future-representation realization.

  1. Representation realization and predictive consistency.

For PushT, exact ordinal realization encodes the learned decision in terminal goal-relative distances. On the same 256-start confirmation cohort, relational readout and realized native-distance selection both achieve 87.11% success, with median inference times of 35.03 and 34.02 ms/start, respectively (revised Table 4, p. 10). This verifies that the learned decision can be recovered through the native planning interface.

For Reacher, we additionally evaluated actual execution using native latent distance after temporal transport. This experiment uses 256 new, outcome-unfiltered episodes with newly executed reference-derived candidate pools. All 63 candidates are executed per episode, and models are locked before evaluation. Learned-direction transport and fixed-direction continuation share the same warm start, fitting/calibration data, seed, losses, 1,500 updates and selection opportunities; the learned variant adds a shared direction MLP.

Native-distance readoutSuccessMean physical goal cost
Original TD-JEPA67.58%0.04516
Previous fixed direction67.97%0.04513
Fixed-direction continuation68.36%0.04484
Learned-direction transport70.70%0.04348

Learned transport improves success by 3.13 pp over original TD-JEPA (paired 95% CI [0.78, 5.86]) and by 2.34 pp over matched continuation ([0.78, 4.30]). Against the latter, 17 action sequences change, with six gains and no losses.

We also measured predictive consistency. Five-step prediction MSE is 0.084995 for learned transport versus 0.084957 for matched continuation (+0.044% relative); temporal-increment MSE is 0.161059 versus 0.160981 (+0.049%). Thus, the execution improvement is accompanied by nearly unchanged prediction errors. This provides complementary evidence for the practical effect of temporal transport through native-distance selection.

Predictive adaptation provides another complementary representation-level contribution: in the original PushT ablation, it improves native-distance success from 76.95% to 79.69%.

  1. Matched independent scoring and trajectory-cost learning.

We added matched pointwise/set comparisons on PushT and Reacher using identical features, execution labels, losses, correction bounds and fitting/calibration opportunities. Both receive the same native-rank inputs; only the set model learns cross-candidate interactions.

TaskPointwiseSet interaction
PushT, 256 starts85.81 ± 1.58%86.33 ± 0.68%
Reacher, 128 starts90.36 ± 0.45%91.93 ± 1.19%

Values are mean ± sample SD across three seeds. For each start, paired differences are first averaged across seeds, followed by 10,000 bootstrap resamples of starts. The 95% intervals for set-minus-pointwise are [-1.17, 2.34] pp and [-1.82, 5.21] pp. Mean success favors set interaction in both tasks; both paired intervals include zero.

A separate success-BCE scorer achieves 82.42% on PushT, compared with 86.33% for set alignment (+3.91 pp; paired 95% CI [0.39, 7.55]). This provides an additional reference for learned execution-success scoring.

Regarding Traj-LeWM (Huang et al., 2026), which we have cited and discussed in Section 2, it learns a per-trajectory goal-conditioned cost using synthetic preferences and mined failures, combining trajectory-level evidence with endpoint distance. D-JEPA instead learns bounded relations over a complete candidate set using dense model-specific predictive features and cross-geometry ordinal evidence, then realizes the resulting decision in future geometry. The methods therefore differ in their learning units and decision interfaces. Our matched pointwise controls further examine the contribution of candidate interaction within D-JEPA.

These clarifications are reflected in Sections 4.2–4.3 and 6.4, Appendices C.1–C.2, and D.4 (Tables 10, 22 and 23).

Thank you again for asking about held-out perturbations, execution-supervision requirements and the contribution of task-specific proposal interfaces. We provide additional evaluations and clarify the scope of the reported transfer results.

  1. Held-out perturbation families without refitting.

We added two held-out appearance evaluations in which the respective perturbation family is excluded from both fitting and calibration. Each uses 50 fresh base identities. Clean observations and the other five appearance families supply fitting and calibration data; model and gate selection do not use the held-out condition (Appendix C.6, Table 19, p. 23).

Held-out familyNativeCalibrated fusionD-JEPA
Object colour30%32%48%
Blur72%82%82%

For unseen object colour, D-JEPA improves by 18 pp over native selection (paired 95% CI [2, 32]) and 16 pp over calibrated fusion ([2, 30]). For unseen blur, it improves over native selection and matches calibrated fusion. Neither held-out condition participates in model or gate selection. These experiments directly evaluate transfer to perturbation families excluded from task-local fitting.

  1. Execution-supervision and calibration budgets.

The recorded fitting and calibration requirements are:

ComponentFitting startsCalibration startsCandidates/start
PushT relational alignment4486463
PushT predictor adaptation38412863
Reacher matched modules38412863
Appearance alignment126 base identities63 base identities63/condition

The PushT relational and fusion fitting/calibration pools provide 32,256 retained candidate outcomes over 512 starts. The relational calibration identities are included within the predictor-adaptation calibration population, while the remaining identities contribute to relational fitting. The 256 confirmation identities are separate from these fitting and calibration populations.

We also evaluated fitting-data sensitivity while keeping 64 calibration starts fixed. Using 45, 112, 224 and 448 fitting starts, the three-seed mean set success rates are 85.29%, 83.85%, 87.50% and 86.33%, respectively (Appendix D.2, Table 20, p. 24). This provides an empirical view of how selection performance varies with the available execution-supervision budget.

  1. Separating selector gains from task-specific proposal engineering.

In the four-task RoboTwin comparison, all methods share the same pretrained VLA, observations, candidate identities and execution horizon:

MethodOverall success
Native VLA61.72%
Fixed correction66.80%
Scalar gate66.21%
D-JEPA76.76%

This comparison evaluates selection improvements under a shared proposal interface. The task-specific grasp proposal construction and execution interface are separately described in Appendix G.1. The main comparison therefore provides the relevant evidence for the contribution of candidate selection under matched proposals, while the appendix details the application-specific interface.

These additions clarify the supervision requirements, held-out evaluation protocol and role of task-specific proposals in the reported results.

Thank you for raising the questions about candidate-set construction and the applicability of the PushT results to ordinary planning. We have expanded the historical evaluation protocol and conducted an additional evaluation using fresh online proposals.

  1. Historical candidate construction and eligibility.

Each PushT start initially provides 256 sequences of 25 controls: one reference, 96 Gaussian reference perturbations, 31 five-block permutations, 64 smooth-random sequences and 64 uniform-random sequences. After excluding the reference from selection, 63 candidates are retained: the 16 lowest-cost non-reference candidates under TD-JEPA, plus 47 additional candidates selected round-robin across proposal families in predictive-cost order.

The confirmation eligibility rule requires both successful and unsuccessful candidates within the native top 16, with a minimum absolute cross-class score gap no greater than 0.00021289736. This threshold is the fixed 30th percentile of 185 finite gaps in the original 256-start reservoir.

We additionally examined the historical pre-filter population:

PopulationStartsSuccessful-candidate coverageNative TD successEligible
All saved scans12,14599.98%87.65%21.50%
After suite/reservoir exclusions10,99399.97%88.44%15.09%

Coverage means that at least one of the final 63 selectable candidates has a successful executed outcome. Native TD success means that the lowest-native-cost candidate succeeds. Both rates use all starts in the corresponding population as the denominator.

After the original exclusions, 1,659 starts satisfy the eligibility rule. Excluding the previous confirmation cohort leaves 1,403 eligible starts, from which 256 were selected using a fixed ordering. These identities are separate from fitting, calibration and previous development.

  1. Fresh online proposals and closed-loop replanning.

We conducted an additional PushT evaluation on 256 new start–goal pairs using upstream LeWM CEM proposals. CEM uses 300 samples over 30 iterations and 30 elites. The planner executes 25 controls before replanning, with a maximum of 50 controls. Goals are recorded states 25 controls later in the same episode; no future reference actions are provided to the planner.

The first 63 candidates from the final CEM iteration form the initial pool without filtering by predictive cost or execution outcome. Methods share the initial pool and use the same search budget after their trajectories diverge.

Matched set and pointwise heads are trained using separate 448-start fitting and 64-start calibration CEM pools, with identical features, execution labels, losses, correction bounds and three seeds. All models and gates are fixed before evaluation.

Online closed-loop evaluation, 256 pairsSuccess
LeWM86.72%
TD-JEPA83.59%
Original calibrated fusion88.28%
Unchanged D-JEPA relational checkpoint86.72%
Native CEM mean86.72%
CEM-adapted set, three-seed mean ± SD89.06 ± 1.17%
Matched pointwise, three-seed mean ± SD87.50 ± 1.56%

Initial pools contain at least one candidate succeeding within 25 controls in 247/256 cases (96.48%); terminal-success coverage is 243/256 (94.92%). Among the 247 initially covered cases, the adapted set selects an eventually successful candidate in 77.19% on average, compared with 75.71% for LeWM and 76.52% for original fusion.

Set achieves 89.06% closed-loop success versus 87.50% for pointwise scoring (paired difference +1.56 pp; 95% CI [-0.20, 3.33]), while original fusion reaches 88.28%. This experiment separates candidate availability, initial selection and closed-loop execution under newly generated proposals. All 256 episodes are separate from decision-layer fitting, calibration and previous evaluations, although their source dataset was available for predictive-model pretraining.

The revised manuscript reports candidate eligibility in Appendix A.1 (pp. 13–14), supervision scopes in Appendix A.8/Table 6 (p. 16), and online evaluation in Appendix A.9/Table 7 (pp. 16–17). Sections 5–6 summarize the corresponding experimental settings and findings.

Reviewer_c7cZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper makes the observation that a world model can rank most action candidates well and still pick the wrong one among the top few. The authors use outcomes from executed actions to train a reranker and report gains across simulation, physical robots, and driving. The results show that reranking can help (though it’s still unclear how much of the gain comes from comparing candidates jointly or how well it carries over to unfiltered starts with online proposals).

Critical strengths
  1. The paper identifies a concrete failure mode in latent-space planning. On PushT, LeWM’s average within-start rank correlation between predicted and realized goal costs drops from 0.90 across all 63 candidates to 0.11 among the four candidates it ranks highest (Figure 2 and Table 11).

  2. The evaluation goes beyond the core simulation benchmarks. The authors apply the approach to a pretrained VLA on RoboTwin, to two physical robot tasks, and to trajectory selection in driving. I found the physical robot result particularly useful. Success rises from 64% to 81% across 100 paired trials with the same underlying planner and candidate budget.

Critical weaknesses
  1. I'm unsure how much the PushT gain would carry over to ordinary planning. The 256-start confirmation set was chosen to include close calls between successful and failed candidates (Appendix A.1). That's a reasonable way to test reranking, and all methods get the same candidates. What I can't tell is how often these close calls occur on unfiltered starts, or how often an online planner would produce a successful candidate in the first place. The paper gives little detail on how the 63 stored sequences were generated. I had the same question about driving, where the evaluation covers seven difficult scenes and every scene has a candidate that achieves the maximum score (Table 3). I'd like to see an evaluation with online proposals on unfiltered starts that reports how often the pool contains a successful candidate and how often each method finds it.

  2. I'm not convinced that cross-candidate attention is what drives the gains. A simple test would be to score each candidate independently using the same inputs, execution labels, and training loss. Appendix G.2 describes an MLP control along these lines for driving. However, I couldn't find its results or an equivalent comparison for Pusht or Reacher. This matters because calibrated four-geometry fusion and the learned relational scorer both reach 99.22% on the 128-start ablation cohort (Table 5). The other components are harder to assess. The composition gate adds two net successes out of 256 on Pusht (Table 9), with thresholds selected on a 64-start calibration subset and an implementation that rounds the gate advantage to four decimal places to reproduce the released decision rule (Appendix A.2). I'd like to see how that gain varies with the threshold. Ordinal realization reproduces the learned ranking by construction, and the reported Reacher success uses the relational score to choose actions, leaving the effect of temporal transport on control unclear. The maximum learned displacement reported in Appendix D.2 is 0.0017 against a bound of 0.1. As presented, the results support learning to select candidates from execution outcomes. They don't yet support the decision-aligned world-model framing in the abstract and Section 4.3. To support that claim, I'd want to see a comparison with a learned success scorer or an action-chunk value function, along with evidence that the changes to the future embeddings improve executed decisions.

Questions for authors
  1. How were the 63 stored PushT candidates generated for each start? Before applying the confirmation-set filter, what fraction of starts contained a successful candidate, and how often did D-JEPA and the strongest baseline select one on those starts?

  2. Appendix G.2 describes an independent MLP control for driving. How does it compare with the set-attention model under the same training and calibration protocol? Do you have a matched comparison for PushT or Reacher using the same features, execution labels, and loss?

  3. For Reacher, did you evaluate executed success when actions were selected by native goal distance after temporal transport? If so, how does it compare with selection by the relational score alone?

6 comments

Dear Reviewer c7cZ,

Thank you for raising these questions. D-JEPA connects the discovery of a decision-local prediction gap with execution-informed relational alignment and the realization of learned decision structure in JEPA-compatible future representations for native-distance planning. We added native-distance execution tests (Comment 2), matched candidate-scoring controls, and composition sensitivity analysis to address the representation and selection contributions directly.

  1. Matched independent scoring.

The four-geometry ablation shows that calibrated fusion and relational alignment both reach 99.22% on the separate 128-start mechanism population (revised Table 3, p. 9). We added matched pointwise controls on PushT and Reacher. Both models use identical predictive features, execution labels, losses, correction bounds, optimization budgets and calibration opportunities. The pointwise model removes cross-candidate message passing while retaining the same native-rank inputs (Appendix C.1, Table 10, pp. 19–20).

EvaluationPointwiseSet interaction
PushT, 256 starts85.81 ± 1.58%86.33 ± 0.68%
Reacher, 128 starts90.36 ± 0.45%91.93 ± 1.19%

Values are mean ± sample SD across three training seeds. For each evaluation start, we first average the paired success difference across seeds, then compute a 95% percentile interval using 10,000 bootstrap resamples of starts. The intervals are [-1.17, 2.34] pp for PushT and [-1.82, 5.21] pp for Reacher. Set interaction yields higher mean success in both tasks; both paired intervals include zero.

Set Attention is the candidate-interaction mechanism used in our implementation. The matched Set and Pointwise models share the same execution-informed decision-alignment objective, so this comparison measures the incremental contribution of candidate interaction. If further architectural comparisons would be helpful, we would be happy to add experiments with graph neural networks (GNNs), pairwise comparators or other candidate-interaction mechanisms.

We also evaluated a learned-success alternative. A success-BCE head reaches 82.42 ± 1.03% on PushT, compared with 86.33% for set alignment. This compares execution-supervised scoring configurations with different learning objectives.

  1. Composition and threshold sensitivity.

On the 256-start PushT confirmation cohort, calibrated composition increases success from 223/256 to 225/256 relative to relational selection (+0.78 pp; paired 95% CI [-1.17, 2.73]). Composition provides a complementary refinement to relational selection.

We evaluated 41 threshold offsets over [-0.020, +0.020] around the original predictive-composition threshold of 0.00653. Success counts range from 223 to 226, while offsets of -0.001, 0 and +0.001 all retain 225 successes. We therefore retain the original calibrated threshold. Four-decimal rounding of the relational gate advantage preserves all 256 confirmation choices; rounding only the composition advantage or threshold also leaves its choices unchanged (Appendix C.3).

These additional comparisons clarify the respective contributions and operating behavior of independent scoring, relational alignment and calibrated composition.

Thank you for asking how representation changes affect executed decisions. Our added execution study tests action selection directly through native latent distance.

  1. PushT ordinal realization.

On the 256-start PushT confirmation cohort, ordinal realization and relational readout both achieve 87.11% success, with median inference times of 34.02 and 35.03 ms per start, respectively (revised Table 4, p. 10). Equation 4 encodes the learned ordering in terminal future-to-goal distances, allowing the planner to recover the same action choices through its native distance criterion. This verifies the compatibility of the learned decision structure with the native planning interface.

  1. Reacher temporal transport with native-distance execution.

In addition, we evaluated actual execution using native latent distance after temporal transport. This experiment uses 256 new, outcome-unfiltered episodes with newly executed reference-derived candidate pools. All 63 candidates are executed per episode, and models are locked before evaluation.

The learned-direction transport and fixed-direction continuation share the same warm start, fitting and calibration data, seed, losses, 1,500 updates and selection opportunities. The learned variant adds only a shared direction MLP.

Native-distance readoutSuccess
TD-JEPA67.58% (173/256)
Previous same-space fixed direction67.97%
Fixed-direction continuation, matched budget68.36% (175/256)
Learned-direction temporal transport73.44% (188/256)

Learned transport improves success by 5.86 pp over the original TD-JEPA readout (paired 95% CI [0.78, 5.86]) and by 5.08 pp over the matched fixed-direction continuation ([1.9, 8.2]). Against the latter, it changes 17 selected action sequences, with 15 gains and two losses. Mean physical goal cost decreases from 0.04484 to 0.04348.

On this same 256-episode population, direct relational selection achieves 91.41%, providing a reference for the complementary role of candidate-relation reasoning. The results show that learned temporal transport also improves executed decisions when selection is performed through the native latent-distance interface.

Predictive adaptation provides another complementary contribution: in the original PushT ablation, it improves native-distance success from 76.95% to 79.69%.

We have clarified these mechanisms in Sections 4.3 and 6.4 (pp. 6 and 10). The Reacher native-distance execution study is reported in Appendix D.4/Table 22 (pp. 24–26).

We also thank you for asking about candidate construction, successful-candidate availability and online planning. We provide additional protocol details and evaluations below.

  1. Historical candidates and confirmation-set eligibility.

Each PushT start initially provides 256 sequences of 25 controls: one reference, 96 Gaussian perturbations, 31 block permutations, 64 smooth-random and 64 uniform-random sequences. After excluding the reference from selection, 63 candidates are retained: the 16 lowest-cost non-reference candidates under TD-JEPA_dist and 47 further candidates selected round-robin across proposal families.

Eligibility requires both outcomes in the native top 16 and a minimum cross-class score gap ≤0.00021289736 (the 30th percentile of 185 finite gaps in the original reservoir).

We examined the historical pre-filter population:

PopulationStartsSuccessful-candidate coverageNative TD successEligible
All saved scans12,14599.98%87.65%21.50%
After suite/reservoir exclusions10,99399.97%88.44%15.09%

Coverage is the fraction with at least one successful candidate among the final 63; Native TD success is the fraction where the lowest-cost candidate succeeds. Both use all starts as denominator. After exclusions, 1,659 starts meet the rule; excluding the previous confirmation cohort leaves 1,403, from which 256 were selected by fixed ordering. These identities are separate from fitting, calibration and prior development. Full-archive D-JEPA scores were not retained, so we do not infer its archive-wide success rate from the filtered confirmation results.

  1. Fresh online CEM proposals and closed-loop evaluation.

We evaluated 256 new PushT start–goal pairs with LeWM CEM (300 samples, 30 iterations, 30 elites, 25-control horizon), replanning every 25 controls up to 50. Goals are recorded states 25 controls later in the same episode; the planner receives no future reference actions. Methods share the unfiltered initial pool of the first 63 final-iteration candidates and retain equal search budgets after trajectories diverge.

Matched set and pointwise heads use separate 448-start fitting and 64-start calibration pools, identical features, labels, losses, bounds and three seeds. Models and gates were fixed before evaluation.

Online closed-loop evaluation, 256 pairsSuccess
LeWM86.72%
TD-JEPA_dist83.59%
Original calibrated fusion88.28%
Unchanged D-JEPA relational checkpoint86.72%
Native CEM mean86.72%
CEM-adapted set, three-seed mean ± SD89.06 ± 1.17%
Matched pointwise, three-seed mean ± SD87.50 ± 1.56%

At least one candidate succeeds within 25 controls in 247/256 initial pools (96.48%); terminal-success coverage is 243/256 (94.92%). Among the 247 covered cases, the adapted set selects a successful initial sequence in 77.19% on average, versus 75.71% for LeWM and 76.52% for fusion. The set–pointwise closed-loop difference is +1.56 pp (95% CI [-0.20, 3.33]). These episodes are separate from decision-layer fitting, calibration and prior evaluations; their source dataset was available for predictive-model pretraining.

  1. Matched independent MLP control for driving.

We added a matched MLP comparison on 64 driving scenes, each with 32 shared trajectories and three seeds, using matched inputs, labels, training and calibration.

Driving readoutMean PDMS
Native Drive-JEPA97.3093
Pointwise MLP96.1180
Set interaction97.50

Set minus MLP is +1.3820 PDMS points (scene-paired 95% bootstrap CI [+0.3, +2.5]). These scenes are separate from fitting and calibration and differ from the seven scenes in revised Table 24; their source logs were used in historical diagnostics. Appendix G.2 reports this matched comparison (p. 33).

Candidate construction and eligibility are detailed in Appendix A.1 (pp. 13–14), online evaluation in Appendix A.9/Table 7 (pp. 16–17), and the driving control in Appendix G.2. These additions clarify the evaluation scope, candidate availability and decision-level contributions.

Reviewer_U3msOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. This paper explores what it calls a decision-local prediction gap. The idea is the following: Among the few futures competing for execution at each planning step, a candidate predicted closer to the goal in latent space can produce a worse realized outcome than an available alternative. D-JEPA addresses this by training a lightweight, permutation-equivariant operator that jointly reasons over goal-relative predictive features and ordinal evidence from executed outcomes, refining the pretrained predictive geometry where action choices are most consequential. The learned decision structure is written back into JEPA-compatible latent representations, enabling deployment through native latent-distance planning. The method is evaluated across a number of environments, specifically control, manipulation, pretrained action-producing models, physical robots and autonomous driving. They note 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical robot tasks.

Critical strengths
  1. The problem formulation is fairly rigorous. The decision-local prediction gap alone is interesting and significant. It is explained very well across the paper with prose, rigor and figures.

  2. The empirical validation is fairly broad. There are a number of environments that are reasonably different from each other that the generalization is fairly convincing.

Critical weaknesses
  1. Now, perhaps the only limitation is that the ablation suggests certain simple baselines are also pretty competitive. This is not a very big limitation per se, but it may impact the paper's analysis.
Questions for authors

I would consider more proper analysis of the various baselines that are presented in the paper. Some of them are doing fairly well. Could the authors place the true value of their contribution given how it compares with the baselines from an absolute metrics point of view?

2 comments

Dear Reviewer U3ms,

Thank you for recognizing the significance of the decision-local prediction gap and the breadth of our evaluation. D-JEPA connects this diagnosis to execution-informed relational alignment and realizes the learned decision structure in JEPA-compatible future representations for native-distance planning. The comparisons below quantify the resulting action-selection benefits over competitive alternatives.

  1. Absolute benefits over strong alternatives.
EvaluationComparatorComparator (%)D-JEPA (%)Gain (pp)
PushT confirmation, 256 startsCalibrated two-geometry fusion87.5087.89+0.39
Reacher, 128 startsLeWM86.7293.75+7.03
Held-out shapes, 300 startsCalibrated rank fusion44.6753.67+9.00
RoboTwin, 512 startsFixed correction66.8076.76+9.96
Physical robots, 100 paired trialsNative V-JEPA 2-AC planner64.0081.00+17.00

These comparisons use matched candidate availability within each task (revised Tables 1–2, p. 8; Table 8, p. 19; Table 17, p. 22). For PushT, we include the stronger calibrated two-geometry fusion (87.50%), rather than relying only on the four-geometry fusion result (85.94%). Paired comparisons against LeWM show gains of +4.30 pp on PushT (95% CI [0.39, 8.20]) and +7.03 pp on Reacher ([0.78, 14.06]) (Table 12, p. 21).

  1. Interpreting the results against competitive fusion.

On the separate 128-start four-geometry ablation population, fusion and relational alignment both reach 99.22%. On the 256-start confirmation cohort, composition improves over calibrated two-geometry fusion by 0.39 pp, with a paired 95% CI of [-2.34, 2.73]. Reacher, geometric transfer, RoboTwin and physical execution show the absolute benefits summarized above.

The matched module ablation further clarifies the contribution: native TD-JEPA, predictive adaptation, relational alignment with fixed predictors, and calibrated composition achieve 76.95%, 79.69%, 87.11% and 87.89%, respectively. Relational alignment provides the largest individual improvement, and composition adds a further 0.78 pp over relational alignment.

We also added a learned-success comparison. On the same 256 PushT confirmation starts, a success-BCE scorer achieves 82.42%, while bounded set alignment reaches 86.33%, a gain of 3.91 pp (paired 95% CI [0.39, 7.55]). Both use the same predictive features and execution labels, with different learning objectives. This provides an additional comparison against a straightforward execution-supervised scoring alternative.

  1. Additional Reacher evaluation.

We further evaluated the unchanged relational model on 256 new Reacher episode identities, selected before observing outcomes and without boundary-based filtering. All 63 candidates per start were newly executed. D-JEPA achieves 91.41%, compared with 82.42% for LeWM, an improvement of 8.98 pp (paired 95% CI [4.30, 13.67]). This provides additional evidence on a separate, outcome-unfiltered population, complementing the original 128-start evaluation.

Thank you for encouraging us to examine the absolute value over competitive alternatives. We have made this comparison more explicit in Sections 6.1 and 6.3 (pp. 7–10). The learned-success comparison is provided in Table 10 (p. 20), fusion comparisons in Appendix C.3 (pp. 20–21), and the additional Reacher evaluation in Table 22 (p. 26).

AuthorsOpenReview ↗

Our Sincere Appreciation

Dear Reviewer U3ms,

We sincerely appreciate your recognition of the significance of the decision-local prediction gap and the breadth of our evaluation. Your request to clarify the absolute benefits over competitive baselines has helped us further strengthen the experimental analysis and communicate the contribution more precisely.

Given the substantial differences in the assessments across reviewers, we would respectfully appreciate it if you could reconsider the strength of your recommendation after reviewing our responses and the revised evidence. If the additional comparisons and clarifications address your concerns, we would be grateful if you would consider a more positive recommendation. If they also strengthen your confidence in the work, we would welcome a corresponding update to your confidence assessment.

Thank you again for your time, thoughtful evaluation and constructive engagement with our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.