- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses two correlated issues in the training and inference of LeWM: during training, the model learns only target-agnostic and local state transitions; during planning, the plan was selected by the distance between the predicted endpoint latent to the target latent, while ignore the process in between. The paper proposes to train a lightweight scorer jointly with the WM, the latent trajectory cost (LTC), which evaluates the complete latent trajectory and can therefore distinguish candidates with identical endpoints. LTC shares the encoder with the WM, and is trained by a pairwise preference loss. The negative training trajectories are from handcrafted construction and real failures during the training, with expert trajectories as positives. The training of LTC also shapes the encoder. At planning time, LTC is added to the endpoint distance to rank candidates. The experiments show superior performance over LeWM on Push-T, OGBench-Cube, Reacher, and Two-Room.
Critical strengths
This work is well-motivated, addressing a concrete limitation of reward-free latent MPC: only the endpoint is accounted in planning. The proposed scoring module is simple, and appears applicable to other JEPA-style WMs. Ablations are carefully designed: Table 4 separates the effect of trajectory supervision on the representation from the effect of LTC at planning time. Table 3 adds a matched (in capacity and supervision) control with paired confidence intervals. In addition, mining negative trajectories from the model's own failures and using them to train the scorer is a sensible design.
Critical weaknesses
The main argument of this work, that scoring the complete trajectory rather than only the endpoint drives the planning gains, is not yet sufficiently supported. Table 3 provides evidence that a LTC scorer performs better than endpoint scorer. However, Table 6 shows that the compared endpoint scorer is as weak as a simple euclidean cost (comparing rows 1,4 in Table 6, almost same success rate, the learned one is even a bit worse). Table 3 therefore shows that LTC outperforms an endpoint scorer that provides essentially no benefit. Recent work shows that learned endpoint costs can be highly effective (e.g., TRM [1]). A stronger endoint scorer baseline is needed to support the central claim.
[1] Li, Liangyu, et al. "Beyond Euclidean Proximity: Repairing Latent World Models with Horizon-Matched Trajectory Reachability Metrics." arXiv preprint arXiv:2605.22164, 2026.
Questions for authors
Could you strengthen the endpoint scorer baseline, for example with a TRM-style cost or the temporal cost of TD-JEPA [2]? I see these are just arXiv works, but they would provide a stronger test of whether path information adds value beyond a well-trained endpoint cost.
[2] Bai, Jiaxin, et al. "Temporal-distance jepa: Plan-aware representation learning for latent world model predictive control." arXiv preprint arXiv:2607.25337, 2026.
3 comments
Thank you for the additional experiments, which address my main concern. As the gains over the strongest endpoint cost are mostly pronounced on certain task (Reacher), I will keep my score of weak acceptance.
Thank you for taking the time to review our response and additional experiments, and for acknowledging that these experiments address your main concern.
Your suggestion to include stronger endpoint cost baselines was particularly valuable. It helped us more thoroughly validate the additional contribution of trajectory scoring beyond strong endpoint costs and better understand its effectiveness across different tasks.
Thank you again for your constructive feedback throughout the review process and for the time and effort you have devoted to evaluating our work.
TRM-style and TD-JEPA-style endpoint costs with and without LTC
Thank you for raising the question of endpoint baseline strength. In Table 3 of the original submission, we compared endpoint and trajectory scorers matched in capacity and trained on the same preference pairs. As you noted, Table 6 shows that adding our original preference-trained endpoint head to the Euclidean endpoint cost had little effect on closed-loop success. Following your suggestion, we evaluated TRM-style and TD-JEPA-style endpoint costs to test whether adding LTC improves closed-loop success when the learned endpoint cost is held fixed.
The TRM-style head learns an endpoint cost by predicting how many time steps separate two observations from the same episode. For the TD-JEPA-style cost, we used the authors’ released implementation of the directed distance head and temporal loss. Forward time gaps within a trajectory serve as training targets, and goals from other trajectories provide negative examples. In our experiments, both endpoint heads score a candidate using only its predicted terminal latent state and the goal latent state. TRM · TD-JEPA
For each new endpoint cost, we held the Traj-LeWM checkpoint and trained endpoint head fixed. On each task, we evaluated the endpoint cost alone and with LTC added on the same 128 start–goal queries. Both conditions used the same CEM random seeds and planning budget. We calibrated the LTC weight on the validation set and kept it fixed during testing. The table reports the number of successful closed-loop trials.
Used alone, the TD-JEPA-style endpoint cost yielded more successes than Euclidean distance on all four tasks. Its total of 417/512 matched that of Euclidean distance with LTC. Keeping this learned endpoint cost fixed and adding LTC increased the total to 435/512, a net gain of 18 successes (+3.52 percentage points). Adding LTC to the TRM-style endpoint cost increased the total from 382/512 to 417/512, a gain of 6.84 percentage points.
On Reacher, path scoring improved closed-loop success with all three endpoint costs. Adding LTC yielded 15, 13, and 12 additional successes with the Euclidean, TRM-style, and TD-JEPA-style endpoint costs, respectively. For the TD-JEPA-style cost, the remaining six additional successes came from Push-T and Cube, with three on each task. On Two-Room, the learned endpoint costs alone accounted for the full gain. Euclidean distance solved 113/128 queries. Both learned endpoint costs solved all 128 queries on their own and maintained that result when LTC was added.