- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper works on latent world model which address the gap where predictive latent distance may not reliably reflect which candidate action will succeed when executed. It introduces a bounded, permutation equivariant relational alignment mechanism, and incorporates decision-relevant structure into future representations to support planning through native latent distances.
The main contributions are:
- identifying and quantifying the gap between predictive latent geometry and realized decision quality
2.proposing a shared relational alignment framework that can integrate complementary predictive representations
3.enabling decision-aligned preferences to be realized directly in JEPA-compatible future latent spaces
Exp results include an average success rate increase on four RoboTwin tasks from 61.72% to 76.76%, an increase on two physical PiPER tasks from 64.0% to 81.0%, and an increase in mean PDMS from 57.36 to 95.34 across seven selected driving scenes.
Critical strengths
- identify the limitation of latent prediction does not necessarly introduces good action
- propose bounded, permutation-equivariant relational alignment mechanism that learns preferences among candidate futures and incorporates decision-relevant structure into JEPA-compatible latent representations.
- it proves it method on several datasets including RoboTwin, PiPER, and automated driving
Critical weaknesses
Insufficient evidence regarding the dependence on execution-outcome supervision and candidate-set design. D-JEPA learns decision-relevant relations among candidate futures from execution outcomes. However, the practical effectiveness of this approach may depend on the quality and diversity of candidate actions, the availability and cost of outcome supervision, and whether the learned preferences remain reliable when the candidate distribution changes
Questions for authors
How sensitive is D-JEPA's performance to the quality, diversity, and number of candidate futures, as well as the amount of execution-outcome supervision?
What extent do the reported gains arise specifically from aligning latent representations with execution outcomes, rather than from additional predictor adaptation or training signals?
3 comments
Part 2
Candidate quality and diversity at a fixed candidate count.
We additionally examined candidate composition while keeping the candidate count fixed at K=16. Each of the six proposal families was removed in turn, using unchanged selectors and outcome-independent subsets from the original 256-start sealed cohort, separate from the confirmation cohort. Candidate coverage measures whether a successful candidate is available, while diversity is measured by pairwise action-space RMS distance.
Removing small-noise proposals increases raw action-space diversity but reduces successful-candidate coverage and the performance of both calibrated fusion and learned selection. Other proposal-family removals have smaller effects. This indicates that geometric spread alone is not a sufficient measure of candidate quality: locally perturbed alternatives can provide useful executable candidates even when they contribute less to overall geometric diversity. We therefore examine candidate coverage alongside selection success rather than treating diversity alone as a proxy for candidate quality.
Changing the proposal distribution.
We also evaluated newly generated, reference-action-free CEM candidate pools with repeated replanning on 256 outcome-unfiltered PushT pairs. The evaluation includes the unchanged relational checkpoint and separately adapted set/pointwise heads trained on disjoint CEM fitting and calibration pools. The adapted set and pointwise models achieve 89.06% and 87.50% success, respectively (paired difference +1.56 pp; 95% CI [−0.20, +3.33]), while the original calibrated fusion reaches 88.28%. This provides an additional evaluation of both the existing selector and adaptation under a changed proposal distribution. The protocol, candidate coverage and closed-loop outcomes are reported in Appendix A.9/Table 7 (pp. 16–17).
A Sincere Note of Appreciation
Dear Reviewer KVBV,
We are genuinely grateful for your positive assessment of D-JEPA's problem formulation, alignment mechanism and evidence across tasks. Your questions about candidate quality, diversity and execution supervision have been particularly helpful in guiding our additional experiments and clarifications.
Given the different assessments across the reviews, we would sincerely appreciate it if you could reconsider the strength of your recommendation after reviewing our responses and the additional evidence. If these clarifications satisfactorily address your concerns, we would be grateful if you would consider a stronger recommendation. If the additional evidence also gives you greater confidence in the work, we would welcome a corresponding update to your confidence.
Thank you again for recognizing the value of the work and for engaging with us constructively.
Dear Reviewer KVBV,
Thank you for recognizing the problem, alignment mechanism and cross-domain evidence. We have conducted additional analyses to separately examine candidate availability, execution-supervision requirements and the contributions of the alignment components.
We evaluated nested subsets of the same 256 PushT confirmation starts, keeping model weights and thresholds fixed across 16 shared subset seeds. Ranks and relational outputs are recomputed within each subset.
D-JEPA outperforms both native selectors at every candidate budget, improving from 81.96% to 87.11% as more alternatives become available, without retraining for each candidate count (Appendix D.1, p. 23; Table 16, p. 22).
We additionally varied the number of fitting starts while keeping 64 calibration starts fixed, corresponding to 4,032 calibration candidate outcomes. Each fitting start supplies 63 executed candidate outcomes, with three training seeds evaluated under the same optimization and calibration protocol.
All four budgets exceed the unchanged LeWM reference of 83.59%. Even with 45 fitting starts and the fixed calibration set, D-JEPA reaches 85.29%. The table reports all four prespecified fitting budgets; the full-budget configuration remains unchanged.
The matched module comparison on the same 256 PushT confirmation starts gives:
Relational alignment provides the largest individual improvement while keeping the predictive models fixed. Predictor adaptation contributes a separate improvement, and calibrated composition combines the two sources of evidence. Execution outcomes supervise relational alignment, while native predictive costs provide its rank inputs.
We further compared matched pointwise and set models using identical features, execution labels, losses and correction bounds. Pointwise scoring achieves 85.81%, compared with 86.33% for set alignment (set-minus-pointwise difference +0.52 pp; 95% CI [-1.17, 2.34]). A separate success-BCE scorer achieves 82.42%, with set alignment improving by 3.91 pp (95% CI [0.39, 7.55]).
The matched scoring controls examine candidate interaction and the learning objective under shared execution supervision, complementing the predictor-adaptation ablation above.
As an additional mechanism analysis, we fixed the predictors, trained weights, target tokens and all native ranks/base scores, changing only the latent descriptors of other candidates. The set model changes the target-pair score margin in all 64 calibration cases (median absolute margin change 0.0357), whereas the matched pointwise model has zero effect. This confirms that the learned set operator uses candidate context.
Section 6.3 and Table 3 summarize module contributions (pp. 9–10). Appendices C.1–C.2/Table 10 report matched scoring and context analysis (pp. 19–20); Appendix D.2/Tables 20–21 reports supervision-budget and candidate-content sensitivity (p. 24); Appendix A.8/Table 6 specifies the supervision protocol (p. 16).