- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses zero-shot, long-horizon, goal-conditioned visual planning with JEPA-style latent world models trained on offline, reward-free data. The authors argue that latents trained only for prediction give distances that do not track progress toward distant goals. They also argue that fixed-interval subgoals blend distinct task phases. TrekWM addresses this in two ways. A pairwise ranking loss shapes the world-model embedding so that latent distance preserves temporal order. A winner-take-all subgoal proposer places subgoals at real latents within a temporal window. Across 11 settings in five simulated environments, rank-correlation analysis and a 2×2 factorial results indicate that the two components are complementary, and extensive ablations support the design choices.
Critical strengths
- Well-motivated problem with measured hardness. The paper addresses zero-shot, long-horizon, goal-conditioned visual planning from offline, reward-free data. It shows empirically that latents trained only for prediction stop tracking progress toward distant goals: LeWM's rank correlation with time-to-go falls to about zero beyond short horizons (Table 19). It also explains why fixed-interval subgoals blend distinct task phases.
- Clear positioning and a testable hypothesis. The work is positioned between JEPA world models, cost and metric shaping, and hierarchical planning. The central hypothesis is stated clearly: a plannable latent needs distances that preserve temporal order, and subgoals that keep each planning segment within the range where those distances are reliable.
- Design choices are empirically justified, and the results support the hypothesis. Shaping the embedding itself outperforms a separate metric head (Table 12). The flexible-schedule objective places subgoals at real bottlenecks, with doorway localisation of 99.2 vs 59.8 for the fixed schedule (Table 4). Shaping raises rank correlation in every environment (Table 19). The 2×2 factorial (Table 10) shows the components are complementary on Cube-50: success is 67.3 with both, against 32.0 with subgoals alone and 0.7 with shaping alone.
- Thorough ablations and appropriate metrics. The ablations cover the regulariser and ranking loss, embedding vs head, fixed vs flexible schedule, number of subgoals, clock and episode budget, pointer-check interval, HWM search budget, HIQL-style subgoals and the subgoal chain on Temporal-JEPA. Success rate is the standard metric, and the rank-correlation analysis directly tests the geometric claim.
Critical weaknesses
-
Narrow scope and empirical-only hardness. "Long horizon" means at most 100 steps on filtered simulated tasks, and Cube episodes are truncated, so the scope is narrower than the framing suggests. The hardness argument is entirely empirical: there is no formal account of when or why distances learned for prediction stop tracking progress.
-
Incremental novelty. Value-guided JEPA and Temporal-Distance JEPA already shape the embedding itself, so the gap reduces to combining shaping with adaptive subgoals. Each component is also close to prior work:
- The ranking loss falls within the line of work that uses temporal supervision to shape representations, such as time-contrastive learning and VIP. The paper should spell out how ordinal ranking on the embedding differs from these methods.
- The subgoal objective, as the authors acknowledge, adapts Multiple Choice Learning and Explorative Modeling.
- The spectral hinge appears secondary and task-dependent (Table 11), and it performs poorly without the ranking loss on PushT.
-
Engineering contribution without theoretical grounding. The method trains only on forward pairs, treating goal distances as asymmetric, yet the planner uses symmetric Euclidean distance. The paper does not analyse this mismatch. Results only partly support the hypothesis. Shaping alone gives little or no gain on long manipulation tasks: direct-planning success falls from 11.3 to 8.7 on PushT-100 and rises only from 0.0 to 0.7 on Cube-50 (Table 10). Shaping also degrades world-model prediction (Table 18). On TwoRoom, one-step relative error rises from 0.03 to 1.53. On Reacher, position-probe error rises from 0.63 to 3.04 cm.
-
Insufficient statistical rigour and incomplete metrics.
- Results use a single training seed with no confidence intervals, so several wins fall within binomial noise of about ±4 points (e.g., Maze-far 34.7 vs 33.3).
- There is no sensitivity analysis for τ, the pair weights or the ranking gate.
- Path efficiency, success-vs-horizon curves for all environments, compute and planning latency, and significance tests are not reported.
- The SOTA claim needs qualifying.
- Weaker baseline configuration. HWM is evaluated in the authors' image-only shared setup with a reduced sampling budget, not its published configuration. The authors' own run of HWM's released code reports Diverse Maze results above TrekWM's (Section A.6). The protocols differ, so the superiority claim should be stated as holding within this shared setup.
- Missing comparisons. The paper cites but does not compare against closely related methods (ProWorld, Value-guided JEPA, TRM, Traj-LeWM, FF-JEPA).
- Unexplained result. Temporal-JEPA outperforms TrekWM on Reacher-50, and the paper does not explain why.
Questions for authors
Refer Weaknesses section
8 comments
Response to Reviewer AG9g (2/5): W2 & W3
W2. Ranking, subgoal chains, and prior work
Novelty. Our contribution is how latent shaping and adaptive subgoals work together. Subgoals break the task into shorter stages, and the planner uses the shaped distances to judge progress within each stage. On Cube-50, neither component is enough on its own. Shaping alone reaches 0.7% success and subgoals alone reach 32.0%. Together they reach 67.3% (Table 10). The choice of latent space still matters when we hold the subgoal planner fixed. On Cube-50, Temporal-Distance JEPA reaches 4.7% and the two VG-JEPA variants reach 20.7% and 36.7%, compared with 68.2% for TrekWM (Table 16 and Table R6 in our response to Reviewer hXpg).
Ranking. Our loss supervises which state is closer to a shared future goal in the embedding used by the predictor, proposer, and planner. Time-contrastive learning supervises temporal proximity within a window. VIP learns a discounted value used as a reward, and VG-JEPA fits a goal-conditioned value in the latent distance. Our loss constrains the order without fitting a value scale. Temporal-Distance JEPA predicts directed time-to-go with a separate head. With the same subgoal planning framework, TrekWM performs better on harder manipulation tasks, while results are closer on Maze (Tables 12 and 16 and Table R6 in our response to Reviewer hXpg).
Subgoals. Multiple Choice Learning and Explorative Modeling inspire our closest-target matching. Our generator extends this idea to sequences by producing an entire subgoal chain in one pass. This raises two questions specific to a chain: whether to match the chain as a whole or match each subgoal separately, and how to preserve progress toward the goal without fixing the spacing between subgoals. We match each subgoal independently to the closest observed state in its own time window in the planner's latent space. The windows move forward with the subgoal index, encouraging coarse temporal order while letting the generator choose where to place each waypoint. The learned chain places more waypoints at the TwoRoom doorway than a fixed schedule. The manipulation ablations favor learned subgoals and show that the same proposer benefits from TrekWM's latent space (Tables 3, 4, 10, 16, and 17).
Spectral hinge. As stated in the submission, the hinge is a supporting regularizer that prevents collapse. Its standalone performance is weak on PushT. Ranking also improves success with SIGReg, which can be used in place of the hinge (Table 11).
W3. Symmetry, shaping alone, and prediction error
Symmetry. We agree that the symmetric Euclidean cost cannot capture directional reachability, even though the ranking pairs are sampled forward in time. Modeling this asymmetry with a quasimetric is an important direction for future work. In TrekWM, local subgoals limit how far the planner has to rely on the symmetric distance at each stage.
Shaping and subgoals. Shaping alone yields small or negative gains on long manipulation tasks, which motivated our use of adaptive subgoals (Table 10). Ranking helps the planner judge progress, and subgoals let it plan over shorter segments. Shaping helps in some flat-planning settings, and the two components together perform better on harder tasks (Table 17, Table R10, and Table R8 in our response to Reviewer pibC).
Prediction error. Prediction error can rise without changing which plan looks best. On TwoRoom, much of the error comes from a nearly constant offset. With the same flat planner, TrekWM ranks plans better and succeeds more often despite higher five-step prediction error (Table R11). On Reacher, most position-probe error is already present before rollout. Table 18 shows how prediction error varies across tasks.
Table R11. Prediction accuracy and planning-cost geometry on TwoRoom. All models use the same flat planner. FVU measures 5-step prediction error (lower is better); rank columns report Spearman correlation.
| World model | FVU ↓ | Cost rank | Geodesic rank | Expert in top 10% (D = 50) | Expert in top 10% (D = 100) | Success (D = 50) |
|---|---|---|---|---|---|---|
| LeWM | 0.038 | 0.719 | 0.262 | 41.4 | 17.0 | 24.0 |
| LeWM, TRM training recipe | 0.056–0.059 | 0.770 | 0.511 | 56.7 | 30.8 | 48.7 |
| LeWM released by the TRM authors | 0.265–0.286 | 0.742 | 0.495 | 55.7 | 25.3 | 41.1 |
| TrekWM | 0.115 | 0.924 | 0.845 | 98.2 | 52.6 | 97.3 |
Response to Reviewer AG9g (3/5): W4 - part 1
W4. Uncertainty, sensitivity, and efficiency
Training seeds. Table R12 reports confidence intervals over three TrekWM training seeds and paired tests against LeWM. Due to limited compute, LeWM uses one training run. The gain intervals resample TrekWM seeds and evaluation tasks with LeWM fixed. The paired tests show gains over LeWM, including on Maze.
Table R12. Three-seed results with confidence intervals and significance tests (Weakness 4). Mean ± s.d. and 95% t-interval over three training seeds. Δ compares TrekWM with LeWM: its 95% CI comes from a bootstrap that resamples training seeds and evaluation tasks, and p from a paired t-test over the three runs. * indicates that all three seeds achieve 100% success. Baselines in this table are those evaluated in the submission. For PushT-50, the CI and p of Δ use the per-task outcomes of a LeWM rerun (35.3%).
| Task type | Task | TrekWM mean ± s.d. | TrekWM 95% CI | LeWM | Δ vs. LeWM [95% CI] | p | Best baseline | Δ vs. best baseline |
|---|---|---|---|---|---|---|---|---|
| Same traj. | Cube 25 | 83.6 ± 13.6 | [49.8, 100] | 26.7 | +56.9 [+41.1, +70.7] | 0.019 | DINO-WM: 62.7 | +20.9 |
| Same traj. | Cube 50 | 68.2 ± 2.7 | [61.5, 74.9] | 0.0 | +68.2 [+61.5, +74.7] | <0.001 | DINO-WM: 0.7 | +67.5 |
| Same traj. | PushT 50 | 82.4 ± 1.4 | [79.0, 85.9] | 36.0 | +46.4 [+38.2, +54.9] | <0.001 | HWM: 41.3 | +41.1 |
| Same traj. | PushT 100 | 67.1 ± 2.8 | [60.2, 74.0] | 11.3 | +55.8 [+47.3, +64.0] | <0.001 | HWM: 23.3 | +43.8 |
| Same traj. | TwoRoom 50 | 100.0 ± 0.0 | [100, 100] | 24.0 | +76.0 [+68.7, +83.0] | * | DINO-WM: 86.7 | +13.3 |
| Same traj. | TwoRoom 100 | 100.0 ± 0.0 | [100, 100] | 16.7 | +83.3 [+76.0, +90.0] | * | T-JEPA: 54.7 | +45.3 |
| Same traj. | Reacher 50 | 87.6 ± 4.2 | [77.0, 98.1] | 50.7 | +36.9 [+27.3, +46.4] | 0.004 | T-JEPA: 90.7 | −3.1 |
| Same traj. | Reacher 100 | 96.9 ± 2.7 | [90.2, 100] | 53.3 | +43.6 [+35.3, +52.0] | 0.001 | T-JEPA: 93.3 | +3.6 |
| Maze | near | 79.1 ± 1.4 | [75.7, 82.6] | 9.3 | +69.8 [+62.9, +76.2] | <0.001 | T-JEPA: 52.7 | +26.4 |
| Maze | mid | 54.9 ± 5.6 | [41.1, 68.7] | 6.0 | +48.9 [+39.6, +58.0] | 0.004 | T-JEPA: 28.7 | +26.2 |
| Maze | far | 30.0 ± 7.0 | [12.7, 47.3] | 3.3 | +26.7 [+17.8, +36.0] | 0.022 | T-JEPA: 14.7 | +15.3 |
| Cross traj. | Cube | 59.1 ± 2.1 | [53.8, 64.4] | 0.7 | +58.4 [+51.3, +65.3] | <0.001 | HWM: 2.7 | +56.4 |
| Cross traj. | PushT | 40.4 ± 2.7 | [33.8, 47.1] | 8.7 | +31.8 [+23.1, +40.2] | 0.002 | HWM: 8.7 | +31.8 |
| Cross traj. | TwoRoom | 99.6 ± 0.8 | [97.6, 100] | 6.7 | +92.9 [+88.7, +96.7] | <0.001 | T-JEPA: 44.7 | +54.9 |
| Cross traj. | Reacher | 94.7 ± 2.4 | [88.7, 100] | 50.0 | +44.7 [+36.7, +52.9] | <0.001 | T-JEPA: 80.7 | +14.0 |
Response to Reviewer AG9g (1/5): W1
Thank you for the detailed review. We will clarify the scope of the paper and include the analyses and comparisons below. And we will revise the paper accordingly.
W1. Horizon, task design, and planning geometry
Horizon. We use “long horizon” relative to the latent world-model baselines in these simulated tasks. Our main evaluation reaches 100 steps, compared with 25 in LeWM's protocol (Section 4 and Table 1). Table R10 extends the horizon sweep, and Table R12 reports the results on HWM's Maze distance bands.
Table R10. Success versus start–goal distance. TrekWM are means over three training seeds. “TrekWM − subgoals” uses the same models and plans directly to the goal. † indicates a significant improvement over “TrekWM − subgoals” after Holm correction.
Table R10(a). Cube
Table R10(b). PushT
Table R10(c). Reacher
Table R10(d). TwoRoom
Task filters and Cube truncation. Both filters use task geometry (Appendix A.1). For Cube-25, we remove windows where the block moves less than the 4 cm success radius. These account for 38% of the pool, and a random policy succeeds on 43% of the unfiltered tasks. In TwoRoom, we keep pairs in opposite rooms at least 100 units apart, so the agent has to go through the doorway. We cut Cube demonstrations after the first successful placement (about 70 frames) because they continue into a second attempt. Cube-50 still tests reaching, grasping, transporting, and placing the block (see Table R9 in our response to Reviewer pibC).
Why prediction alone leaves distances undetermined. Suppose g(f(ot),at)=f(ot+1) and the latent distribution is isotropic Gaussian. A radius-dependent rotation φ preserves that distribution. Replacing the encoder with f′=φ∘f and the predictor with g′(z,a)=φ(g(φ−1(z),a)) preserves perfect prediction and the distribution targeted by SIGReg. The rotation can change Euclidean distances and reorder candidate plans. Ranking directly constrains that order. For distant goals, TrekWM retains positive correlation with time-to-go as LeWM's correlation falls (Table 19).