acceptodds
Under review as a conference paper at ICLR 2027

JEPath: a world model for first-person navigation

Abstract

JEPA world models combined with test-time trajectory optimization have recently achieved strong results on tasks conditioned on a single goal image. However, most of these results assume fully observable environments, and their adaptation to first-person navigation raises specific challenges. In this study, we introduce PADoom, a benchmark built on ViZDoom where the same environments and trajectories can be observed either from a fully observable top-down view or from a first-person view. We show that perceptual aliasing, where different locations produce nearly identical observations, requires combining visual and spatial information, and that the spatial signal must avoid absolute position encoding to generalize across maps. Based on these findings, we propose JEPath, a new JEPA world model that jointly predicts future visual features and ego-motion from observations and actions. Unlike models that predict absolute position, JEPath predicts motion and integrates it along candidate paths. The planner scores paths by distance to the goal rather than visual similarity, resolving perceptual aliasing while still relying on vision to handle navigation-relevant scene structure such as obstacles. On PADoom, JEPath outperforms recent world models and goal-conditioned policies for first-person navigation. It also generalizes substantially better to unseen maps, where competing methods experience a pronounced performance drop. The benchmark and models will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Latent Planning, JEPA, Representation Learning, First-person Navigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CeJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the challenges of using JEPA-based world models for first-person navigation, including perceptual aliasing, obstacle avoidance, and generalization to unseen maps. The authors introduce PADoom, a benchmark that provides both first-person and top-down views of the same environments. They also propose JEPath, a world model that predicts future visual features and ego-motion. Instead of using visual similarity for planning, JEPath predicts future trajectories and selects actions based on their distance to the goal. Experiments show that JEPath outperforms existing world models and goal-conditioned policies on both seen and unseen maps.

Critical strengths
  1. The relative ego-motion formulation shows promising results. The paper shows that predicting relative ego-motion can be more effective than directly predicting absolute positions. The ablation results provide some evidence for this design choice.

  2. The proposed framework achieves competitive navigation performance. JEPath outperforms existing world models and goal-conditioned policies on both seen and unseen maps. In particular, its improvements over goal-conditioned policies that use the same spatial goal information suggest that the framework offers benefits beyond simply changing the goal representation.

Critical weaknesses

1. The task formulation limits the significance of addressing perceptual aliasing. The paper identifies perceptual aliasing as a major challenge in image-goal navigation. However, JEPath changes the setting to point-goal navigation, where the agent's current pose and goal coordinates are assumed to be known. I think this is essentially solving a different problem, rather than directly addressing the original challenge. In many real-world navigation tasks, such spatial information is not readily available. More importantly, perceptual aliasing is largely avoided by providing the relative spatial relationship between the agent and the goal, rather than being resolved by the proposed world model. This makes the claimed contribution to addressing perceptual aliasing less convincing.

2. The proposed planning objective does not fully address obstacle-aware navigation.

The paper argues that successful planning requires the cost function to reflect the geodesic distance to the goal. However, JEPath only minimizes the Euclidean distance between the predicted endpoint and the goal. Even with accurate motion predictions, this objective may fail when avoiding obstacles requires temporarily moving away from the goal. The method relies on searching for a complete detour within a limited planning horizon, which may not work for larger or more complex obstacles. I think this limitation is particularly important because obstacle avoidance is one of the main challenges the paper aims to address, yet the current design does not fully satisfy the planning conditions identified by the authors.

Questions for authors
  1. How does the method handle obstacles that require temporarily moving away from the goal, especially when the detour exceeds the planning horizon? Could the authors provide experiments on such challenging cases?
  2. The training trajectories are collected using A* and a path-following controller that tends to avoid collisions, while CEM may generate many actions that lead to obstacles. How does JEPath reliably predict motion for these potentially out-of-distribution actions?
  3. In the considered setting, actions are discrete and have fixed execution durations. I think it should be possible to estimate ego-motion directly from the action sequence, which may already be reasonably accurate when there are no collisions or obstacles. Could the authors provide a baseline that uses such action-based motion estimates to predict and evaluate candidate trajectories for navigation? This would help clarify how much the learned ego-motion prediction actually contributes to the performance.
  4. The evaluation mainly uses short navigation tasks, and the paper acknowledges that unseen-map episodes may be slightly easier than seen-map episodes. Could the authors provide difficulty-matched evaluations and more challenging long-horizon tasks to better support the generalization claims?
Reviewer_y7YcOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces JEPath, a JEPA-based world model designed for first-person navigation. The authors first present PADoom, a benchmark that enables a systematic investigation of the challenges faced by existing world models under partial observability, including perceptual aliasing, obstacle awareness, and spatial generalization.

To address these challenges, JEPath jointly predicts future visual representations and relative ego-motion. By integrating predicted ego-motion into candidate trajectories, the proposed method enables effective spatial planning based on goal coordinates.

Extensive experiments demonstrate that JEPath consistently outperforms existing world models and goal-conditioned policies on both seen and unseen maps, highlighting the effectiveness and generalization capabilities of the proposed approach.

Critical strengths

The paper addresses an important and meaningful problem in world-model-based navigation. The authors clearly identify the limitations of existing JEPA world models when applied to first-person navigation and provide insightful analyses of perceptual aliasing, unobserved obstacles, and absolute position encoding. These findings offer valuable insights for future research in world models and embodied navigation. The experimental results are impressive. JEPath consistently achieves substantial improvements over existing approaches across multiple evaluation settings. In particular, it achieves a 79.4% success rate on unseen hard maps, compared with 48.4% for the strongest baseline. These results provide compelling evidence of the method's effectiveness in the evaluated setting.

Critical weaknesses

Robustness to pose estimation noise. JEPath relies on the current agent pose during planning. Could the authors briefly discuss how sensitive the method might be to inaccuracies in pose estimation?

Long-horizon navigation. The paper demonstrates promising results across different planning horizons. Could the authors elaborate on the potential extensions of JEPath to substantially longer-horizon navigation tasks?

Computational efficiency. It would be helpful to provide additional information about the inference latency or computational cost of the planning process, particularly regarding its potential for real-time applications.

Questions for authors

Please follow the above questions.

Reviewer_j2zjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies goal-conditioned planning with JEPA-style latent world models (WMs) under first-person (egocentric) observation, a regime in which prior work (DINO-WM, NWM, TS) was evaluated almost exclusively under fully observable, exocentric views. The authors introduce PADoom, a ViZDoom-based benchmark of 100 procedurally generated maps (50 train / 50 held out) in which every scene is rendered in two synchronized views — first-person (PDoom) and top-down (ADoom) — so that observability is the only varying factor. Using three hand-built "pathological" maps plus controlled engine-copy rollouts, they demonstrate that (a) image-similarity planning costs break under perceptual aliasing, (b) planners fail on unobserved obstacles unless the model reads scene structure from vision, and (c) absolute-pose inputs/targets do not generalize to unvisited coordinates. They then propose JEPath: a frozen DINOv2 encoder plus a causal ViT predictor (114M trainable params) that jointly predicts the next latent and the body-frame ego-motion of each step; a CEM planner integrates the predicted motions from the (observed) current pose and scores candidate paths by final distance to a spatial goal (ω = 1). On PADoom, JEPath reaches 88.2/73.9% SR (seen, Normal/Hard) and loses nothing on unseen maps (89.7/79.4%), outperforming all baselines by 12–31 points SR.

Critical strengths
  1. Cleanly isolated scientific question. PADoom's paired PDoom/ADoom rendering of identical trajectories is a genuinely well-designed instrument: any performance delta between the two views is attributable to observability alone (Sec. 3.2). The 50/50 seen/unseen map split similarly separates world knowledge from map memorization. This is a useful contribution independent of the proposed model, and the benchmark + code release promise adds value.

  2. Evidence-driven diagnosis, not vibes. The three pathological cases (App. A, Fig. 6) are controlled experiments with an oracle engine copy standing in for the WM, so each failure is attributed to exactly one factor: image cost drops from 100% → 25% under aliasing while the spatial cost stays at 100% (A.1); deleting walls from the planner's copy drops success 96.7% → 26.7% with Fisher exact p = 1.5e-8 (A.2); an absolute-pose model scores 0/10 at unvisited coordinates while the relative-motion model scores 10/10 (A.3). Each claim in the design story is backed by a measurement.

  3. The core insight is real and simple. Predicting body-frame ego-motion (supervised by pose deltas) instead of absolute pose, and scoring integrated paths by position rather than latent similarity, is a minimal change that directly addresses all three diagnosed failures. The unseen-map results support it: JEPath is the only WM that does not degrade out of domain (Tab. 1; DINO-WM_prop drops 6–7 points, JEPA-WM_prop drops 12–16 points SR), and the absolute-pose-target ablation sits 10.8 points below the [CLS] readout (Tab. 5).

  4. Honest controls and ablations. Blind-GCIQL (black frames) quantifies what goal coordinates alone achieve and is used to show the unseen eval set is, if anything, slightly easier — an unusually candid calibration (Sec. 4.2). The ω sweep (Fig. 4, stable within 2.6 points over ω ∈ [2e-3, 1]), horizon/distance curves (Fig. 5), motion-head aggregation study (Tab. 5), and CEM-vs-MCTS comparison at matched rollout budget (Tab. 6) are all the right experiments. Notably, the main table reports ω = 1 rather than the best-scoring ω = 0.1 (75.9%), i.e., the authors did not cherry-pick the cost weight.

  5. Baselines are retrained on identical data. All baselines consume the same 29,250 episodes and are evaluated on the same pairs; NWM/RAE-NWM use official implementations adapted to the discrete action set (App. D). This is much stronger than quoting published numbers.

Critical weaknesses
  1. Baselines are not capacity/compute-matched, and the paper says so only in passing. App. D.2 states DINO-WM is trained "at its released size, which is why its predictor is a fifth of JEPath rather than a match for it," and TS's predictor is "an order of magnitude smaller than DINO-WM's" (with two thirds of TS's parameters in the projector/pooling head). The headline Tab. 1 gaps (e.g., Hard seen: JEPath 73.9 vs. DINO-WM_prop 37.6) therefore confound the design claim (pose input, spatial cost) with a ~5x predictor-capacity difference. The one capacity-matched comparison, JEPA-WM_prop (same predictor, pose fed back), is indeed the strongest WM baseline — which is consistent with capacity, not design, explaining part of the gap. At minimum, a DINO-WM_prop scaled to 114M predictor parameters is needed to close this confound. Relatedly, training steps differ (JEPath 117k at batch 64 vs. DINO-WM/TS 234k at batch 32) — roughly comparable in samples, but this should be stated explicitly rather than left to the reader to compute.

  2. The planner is re-anchored to ground-truth pose at every replan, which quietly removes the drift problem the paper emphasizes. Tab. 4 lists "Pose anchor: true pose, read at every replan," and with T_e = T_h = 10 the agent executes only 10 actions before being re-localized by oracle. The narrative (Sec. 3.3, "the model never has to know where the agent is") holds only because the planner always knows exactly where the agent is. The "unvisited coordinates" failure mode (App. A.3) is caused precisely by feeding back predicted poses — yet JEPath's closed-loop success never tests long-horizon integration without oracle re-anchoring, because 60 actions = 6 replans = 6 oracle corrections. An experiment varying T_e (or removing the true-pose anchor entirely, chaining predicted poses) would quantify how much of Tab. 1 depends on this oracle.

  3. No odometry-quality measurement at all. The entire method rests on the accuracy of the ego-motion head, yet no paper quantity reports per-step motion error or integrated drift over the T_h = 10 horizon — not even as a diagnostic. Given that Sec. 2 explicitly positions the motion head against action-conditioned visual odometry (DeepVO, TartanVO, DINO-VO), the absence of any VO-style error metric or VO baseline is a real hole: we cannot tell whether JEPath wins because its motion estimates are accurate or merely because their errors are small relative to the δ = 25 success radius at this horizon. Fig. 5b's degradation with target distance hints at compounding error but attributes it to "more training should reduce [it]" without evidence.

  4. The ADoom control is arguably confounded by encoder suitability, weakening the "cost of observability" framing. JEPath scores higher in first person than top-down (73.9 vs. 51.3 SR, Tab. 2). The explanation offered — the top-down agent is a few-pixel dot within one DINOv2 patch and its heading is invisible — amounts to admitting that DINOv2's frozen features are near-blind to the state variable in ADoom. The "cost of the first-person view" measured in Tab. 2 is thus entangled with "DINOv2 cannot read this top-down rendering," which is an encoder property, not an observability property. A resolution- or encoder-matched ADoom variant (or a learned encoder arm) would be needed to make the PDoom/ADoom gap purely about observability as Sec. 3.2 promises.

  5. NWM/RAE-NWM numbers are weak evidence. Both are single-seed ("single seeds sampled," Tab. 1 caption), run with reduced sampling (25 denoising steps, "kept low for compute," App. D.2), and — by construction, since they predict no pose — can only plan with the image cost that the paper itself shows is broken under aliasing. Their 5–20% SR cells therefore mostly re-confirm the paper's own thesis rather than compare methods, and without error bars even that is noisy. Either give them a spatial-cost variant (e.g., decode-and-localize) or drop them from the headline table.

  6. Scope limits are underplayed. All results are in ViZDoom with 19 discrete actions, frame skip 4, static maps, and ground-truth poses at training time (the ego-motion target m_t is computed from recorded poses, Eq. (3)) and at test time (goal position + true-pose anchor). The paper frames first-person navigation as "the simplest instance of this general case" (Sec. 1) and draws broad conclusions, but nothing tests continuous actions, dynamic scenes, noisy odometers, or pose estimation at train time — the exact conditions under which "predict ego-motion from vision" is known to be hard. A limitation paragraph would cost little; the conclusion's gesture at RL for longer horizons does not cover this.

  7. Minor inconsistency between the analysis and the main table. Fig. 4 shows ω = 0.1 peaks at 75.9% while ω = 1 (used in Tab. 1) yields ~73.9%; the text frames ω = 1 as the principled aliasing-proof choice, but since Tab. 5/E ablations are run on Hard-seen at ω = 1 while Fig. 4 shows the image term still helps there, the "aliasing cannot break condition (I)" argument (Sec. 3.3) and the empirical benefit of the image term (Sec. 4.3) are in mild tension that is never resolved.

Questions for authors

SEE WEAKNESS

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.