acceptodds
Under review as a conference paper at ICLR 2027

See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

13 views

Abstract

What visual representation should a navigation policy act on? A central idea of representation learning is that the task’s own error should shape the representation. Navigation policies built on vision foundation models largely set this idea aside: the encoders’ pretraining objectives decide much of what the vision-language backbone sees. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy’s own objectives out of the internal structure of vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On the VLNVerse fine-grained benchmark, RepresentNav achieves a success rate of 83.80%, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). Analysis shows where they come from. With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot’s own motion over the next second: the dream emerges as a compact latent world model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied navigation, navigation foundation model, representation learning, pretrained visual encoders, multi-level feature fusion, latent dynamics, world models, long-horizon planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RoEcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents RepresentNav, a navigation foundation model built on Qwen pretrained model that serves embodied navigation by waypoints prediction. The proposed methods includes mainly two parts, first a Representation Residuals, it is a three-layer transformer that mixes features from serveral depths of frozen vision foundation models through attention residuals, and passes per-view [CLS] and spatial tokens to the language model. Moreover, the paper proposes a four-step latent CoT component that trained as a JEPA to predict the [CLS] representations of future observations ahead. The paper reports results on five benchmarks and real-robot trials on two platforms, and authors` analyses attribute the gains to the constructed space and the latent prediction.

Critical strengths
  • The paper poses a clear and relevant question on which visual interface a VLM-based navigation policy should act on, and frames it well in terms of general-purpose versus purposive representations. The writing is clear and the narrative is coherent throughout.

  • The controlled comparison in Table 2(a) is informative. With the same three frozen encoders, Representation Residuals raise SR from 87.30 to 93.45 and lower CR from 28.70 to 18.05 relative to concatenation, and the dynamic encoder adds 3.65 SR through the constructed interface against 0.60 through concatenation. The result suggests that the interface to frozen encoders can matter as much as the choice of encoders, which is useful knowledge for the community, quite insightful.

  • The proposed model covers three families of navigation tasks with strong results across five benchmarks and it is deployed on a quadruped and a humanoid robots in real world.

  • The analysis goes beyond leaderboard numbers (regularizer and curriculum ablations, latent interventions, post-hoc readouts). The authors are also candid about the limits of several analyses.

Critical weaknesses

W1. The gain of the proposed RR is not attributed to its claimed source. Section 5.1 attributes the gain over concatenation to intermediate layers, but the comparison also changes the fusion module and the visual tokens given to the language model, and the token format of the concatenation baseline is not specified. The decisive control is missing: the same module restricted to the final layer of each encoder under the same token budget. Table 5 addresses capacity only. Concatenation is also the only fusion baseline, although the paper cites more expressive alternatives.

W2. The “latent world model” interpretation seems overstated. The latents are not conditioned on actions and are trained toward observations along expert trajectories, so the decoded ego-motion may simply reflect the plan. The readout comparison is also not like-for-like. Each encoder reference is a single globally pooled vector and no persistence baseline decodes the +0.4 s targets from the current frame, although the scene changes little within 0.4 s. That unsupervised latents already exceed the strongest encoder reference on occupancy (31.37 vs. 29.79 IoU) suggests that the decodable content comes mostly from the context rather than from prediction.

W3. The benchmark comparisons leave key factors uncontrolled. RepresentNav adds three frozen foundation models whose sizes are not reported, while all baseline numbers are copied from papers with different sensors and data. The VLNVerse margins (+20.05 SR over Qwen-RobotNav-8B and +36.32 over TANGO, a humanoid whole-body VLA) far exceed those elsewhere, but the camera setting, embodiment, and controller are not stated. Neither component is ablated on VLN or tracking, where the largest gains appear. Tracking collisions are also markedly higher than those of TrackVLA++ (11.89 vs. 4.71, monocular DT).

W4. The real-world claims rest on off-board computation and limited evidence. Inference runs on an off-board RTX 4090 server that receives the RGB streams and returns waypoints to the onboard controller, while the robot itself carries a Jetson Orin Nano 8GB (Figure 14). Each camera frame is encoded by three additional frozen encoders, one of them on a four-frame clip, and inference latency rises from 144.15 ms for the native model to 209.76 ms (Figure 14(c)). The paper reports neither the end-to-end latency with network transfer nor the control rate. It also does not examine communication delays or dropouts, although the abstract highlights deployment amid moving people. The real-robot evidence consists of 40 trials against a single baseline (Uni-NaVid), and the limitations paragraph (Section 6) does not discuss computational cost or onboard deployment.

Questions for authors
  1. Could the authors report RR restricted to the final layer of each encoder, with the same module and token budget? Please also specify the token format and token count of the concatenation baseline.

  2. Is the ego-motion decoded from the latents closer to the policy’s own predicted waypoints than to the ground-truth motion? Do depth and occupancy remain decodable from the +1.2 s and +1.6 s latents, where persistence is a weaker explanation?

  3. Which camera configuration, embodiment, and controller does RepresentNav use on VLNVerse, and were the baselines evaluated under the same settings and trained on VLNVerse data? What explains the 36-point gap to TANGO on Val-Unseen?

  4. Could the authors report the native-ViT and no-dream variants of Table 2 on VLNVerse and EVT-Bench? On EVT-Bench, what explains the higher collision rate relative to TrackVLA++, and does it reflect a trade-off with the tracking rate?

  5. What are the sizes of the three encoders and the total parameter count? Can the model run on the onboard Jetson Orin Nano 8GB, and what are the end-to-end latency with network transfer and the control rate in the real-world trials?

  6. How is CR defined in the point-goal ablations? Section 4.1 lists only SR and SPL for point-goal. If CR counts episodes with any collision, an SR of 95.00 with a CR of 13.45 implies that at least 8.45% of episodes succeed despite a collision.

7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We are delighted that the reviewer RoEc finds the question of which visual interface a VLM-based navigation policy should act on clear and relevant, considers the controlled comparison in Table 2(a) informative for the community, and appreciates that our analyses go beyond leaderboard numbers. We also thank the reviewer for the careful reading and precise feedback, which have helped us significantly strengthen the manuscript. We hope the following revisions and clarifications address the reviewer's remaining concerns. We have uploaded a revised PDF; table, figure, and appendix numbers below refer to the revised version.

1. Final-layer control and token format (Q1, W1)

We thank the reviewer for suggesting this decisive control.

Token format. In the concatenation baseline, the final-layer features of the three encoders are aligned to the same 16 × 16 grid by the same adapters, concatenated along channels, and read out in the same token format as Representation Residuals: each current view contributes [CLS] plus 64 fine tokens, and each historical view [CLS] plus eight coarse tokens (Appendix A.1). All interfaces therefore use the same token budget, and we have added the token format of every interface to Appendix A.1.

Final-layer control and other fusions. We train Representation Residuals on the final layer of each encoder, with the module and token budget unchanged. Since concatenation was our only fusion baseline, we also add three learned fusions: a same-size three-layer transformer on the concatenated final-layer features, cross-attention fusion over the full-depth feature maps of all encoders, and per-layer gating with the same parameters and FLOPs as Representation Residuals (one weight per feature map, shared across positions) (setting of Table 2(a): point-goal, no dream):

Model VersionEncoder layersTokens per current / historical viewFusion paramsSR (%)SPL (%)CR (%)
ConcatenationFinal1 + 64 / 1 + 8–87.3080.2028.70
Concatenation + 3-layer transformerFinal1 + 64 / 1 + 847.3M90.9582.4319.20
Cross-attention fusionFull depth1 + 64 / 1 + 859.6M90.7081.7822.65
Per-layer gatingFull depth1 + 64 / 1 + 847.0M91.4580.4319.20
Representation ResidualsFinal1 + 64 / 1 + 847.0M92.3083.4819.80
Representation Residuals (ours)Full depth1 + 64 / 1 + 847.0M93.4584.2818.05

We agree that Table 6 (Table 5 in the submission) addresses capacity only; this control addresses where the gain comes from. We find that, restricted to final-layer features, Representation Residuals reach 92.30% SR and 19.80% CR, compared with 90.95% SR and 19.20% CR for a transformer of the same size on the same final-layer inputs and token format. Both the final layers and the intermediate layers therefore contribute: of the 6.15 SR gain over concatenation (87.30%), learned fusion of the final layers accounts for 3.65 SR and the Representation Residuals operator for a further 1.35 SR, while access to the intermediate layers adds the remaining 1.15 SR and lowers CR from 19.80% to 18.05%. The intermediate layers help only through position-wise selection: over the same full-depth feature maps, per-layer gating, with matched parameters and FLOPs but one weight per feature map shared across positions, reaches 91.45%, and cross-attention fusion, with more parameters and computation, 90.70%, both below the final-layer Representation Residuals. Part of the improvement over concatenation comes from learned fusion itself, while at equal capacity Representation Residuals remain 2.00–2.50 SR higher, owing to position-wise, content-dependent selection across encoders and their layers. We have added this comparison and the breakdown above to Appendix B.2 (Table 7), summarized it in Section 5.1, and retained Figures 4 and 8 as descriptions of what Representation Residuals attend to.

AuthorsOpenReview ↗

Rebuttal - Part II

2. The world-model interpretation and the readouts (W2, Q2)

We thank the reviewer for this careful analysis and agree that "latent world model" overstates what the rollout is. The latents are not conditioned on actions, and all training trajectories are expert trajectories, so the rollout predicts the future that the policy is about to bring about. Unlike action-conditioned world models [1, 2, 3], it cannot be queried with alternative actions; it is closer to models that predict the future and read the action from it, such as DreamZero [4], which is likewise trained on demonstrations. This matches the intent of our design ("dream what I want"), and we have removed "latent world model" from the abstract, introduction, Figure 1, Section 5.3, and conclusion.

Ego motion and the plan. We compare the decoded ego motion with the ground truth and with the policy's own predicted waypoints on the same test observations (ADE over +0.2 to +1.2 s):

ComparisonADE (cm)
Decoded ego motion vs. ground truth4.45
Decoded ego motion vs. policy waypoints3.42

We find that the decoded ego motion is closer to the policy's own waypoints than to the ground truth. Since the planning head reads its waypoints after the rollout, this confirms the reviewer's reading: the decoded ego motion largely reflects the motion the policy intends to execute.

Persistence and longer horizons. We add a persistence baseline that decodes the future targets from the Representation Residuals [CLS] of the current frame with the same decoder. We decode each latent at its own horizon and, as a reference, decode the same targets from the Representation Residuals [CLS] computed on the real future frame, which differs from persistence only in the input frame (occupancy IoU % / depth MAE):

HorizonPersistence: RR [CLS] on current frameDream latentReference: RR [CLS] on real future
+0.4 s29.73 / 0.131734.81 / 0.091335.95 / 0.0752
+0.8 s29.51 / 0.135234.30 / 0.102836.66 / 0.0748
+1.2 s29.44 / 0.138434.38 / 0.111436.85 / 0.0756
+1.6 s29.42 / 0.141134.41 / 0.119037.31 / 0.0757

Depth and occupancy remain decodable at +1.2 s and +1.6 s. The dream latent exceeds persistence by 4.8–5.1 occupancy IoU at every horizon and stays within 1.1–2.9 IoU of the Representation Residuals [CLS] computed on the real future frame. Since the latents are not action-conditioned and the decoded ego motion largely reflects the policy's plan, we present the readouts as evidence of what the policy's representation space holds rather than of a world model. We have added these analyses to Appendix B.4 (Table 13).

[1] Recurrent world models facilitate policy evolution

[2] Navigation world models

[3] LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels

[4] World action models are zero-shot policies

AuthorsOpenReview ↗

Rebuttal - Part III

2. The world-model interpretation and the readouts (W2, Q2), continued

Context and prediction. We agree that much of the decodable content comes from the context: the rollout without supervision, which shares the latent's inputs, already reaches 31.37% occupancy IoU, above the strongest mean-pooled encoder reference (29.79%). With the same inputs, prediction supervision raises this to 34.81% (+3.44 IoU, +4.88 mIoU) and lowers depth MAE from 0.1068 to 0.0913 (Table 12). The four latents also distinguish future frames 0.4 s apart: successive predictions of the same frame agree more closely (cosine similarity 0.969) than predictions of frames 0.4 s apart (0.955), with similarity falling to 0.838 at 1.6 s apart (Appendix B.4). In closed loop, the policy depends on this predicted content: zeroing the latents raises CR from 13.45% to 20.05% on the full point-goal benchmark.

Like-for-like references. Table 12 already contains a learned single-vector summary of the real future frame, the Representation Residuals [CLS] (35.95% IoU). We further add attention-pooled encoder references, in which a learnable query pools the tokens of each encoder into one vector that is fed to the same decoder (+0.4 s; occupancy IoU % / depth MAE):

Model VersionMean-pooled (Table 12)Attention-pooled
Backbone ViT on real future25.47 / 0.162629.97 / 0.1014
DA3 on real future26.14 / 0.144731.03 / 0.0886
DINOv3 on real future25.85 / 0.152131.30 / 0.0932
openD4RT on real future29.79 / 0.102533.22 / 0.0876
RR [CLS] on real future35.95 / 0.0752–
RepresentNav (predicted, without the future frame)34.81 / 0.0913–

We find that attention pooling narrows the gap: openD4RT and DA3 reach lower depth error than the dream latent (0.0876 and 0.0886 vs. 0.0913), while the latent remains higher in occupancy (34.81% vs. at most 33.22%). We have therefore removed "more accurately than pretrained encoders applied to the real future frame" from the abstract, the introduction, and Section 5.3, compare instead with the learned Representation Residuals [CLS] reference, and added all readout analyses to Appendix B.4 (Tables 13–15).

3. Definition of CR (Q6)

We thank the reviewer for the careful check. CR is the percentage of episodes with at least one collision. Under the InternVLA-N1 protocol used in Table 2, a collision does not terminate the episode and success requires only reaching the goal, so an episode can succeed after a collision. The reviewer's inference is therefore correct: at least 8.45% of the episodes succeed despite a collision, which is why we report CR alongside SR in these ablations. In contrast, success on ABotN-PointBench requires collision-free arrival, under which RepresentNav reaches 97.5% SR (Table 1(c)). We have defined CR in Section 4.1, added the definition for each benchmark to Appendix A.4, and clarified there that "a quarter fewer collisions" refers to this episode-level rate (18.05% → 13.45%).

Reviewer_sfG2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RepresentNav, a navigation foundation model that learns task-adaptive visual representations from multiple frozen vision foundation models. Its Representation Residuals module integrates intermediate features from semantic, geometric, and dynamic encoders, allowing navigation objectives to shape a shared representation space. The model further introduces a JEPA-style latent chain-of-thought that predicts future representations before generating navigation waypoints. A two-stage attention curriculum is used to train the predictive representations and navigation policy. Experiments cover vision-language navigation, active visual tracking, and point-goal navigation, including VLNVerse, VLN-CE, EVT-Bench, and real-world robotic evaluations.

Critical strengths
  1. Strong performance across diverse navigation tasks.

The paper reports substantial improvements on several benchmarks, particularly VLNVerse, while also demonstrating competitive results on visual tracking and point-goal navigation. The evaluation across multiple tasks and real-world robotic platforms provides encouraging evidence of the framework's general applicability.

  1. Interesting integration of task-adaptive representations and predictive latent reasoning.

The combination of multi-layer, multi-encoder feature integration and JEPA-style future representation prediction is an interesting direction. The component ablations and post-hoc readout experiments provide useful evidence that the learned representations contain navigation-relevant geometric, semantic, and motion information.

Critical weaknesses
  1. Insufficient justification and isolation of the key methodological choices.

The paper combines Representation Residuals, JEPA-style latent prediction, and a two-stage attention curriculum, but several important design choices are not sufficiently validated. Representation Residuals are mainly compared with simple feature concatenation rather than stronger learned fusion methods. The JEPA objective updates both predictor and target representations without EMA or stop-gradient, yet alternative target-update strategies are not evaluated. Furthermore, the two-stage curriculum uses disjoint data halves, making it difficult to isolate the effects of attention masking, loss weighting, and data allocation. These missing controls weaken the evidence that the specific proposed designs are necessary for the reported improvements.

  1. Limited evidence supporting the claimed latent world-model capabilities.

A central claim is that the learned latent chain-of-thought forms a compact world model containing geometry, semantics, occupancy, and motion information. However, this claim is mainly supported by post-hoc decoding experiments. While these results are encouraging, the paper does not sufficiently establish how sensitive they are to decoder capacity and training protocols, or whether the decoded information is causally important for navigation decisions. More direct evaluations of prediction consistency, long-horizon accuracy, and the relationship between latent prediction quality and downstream navigation performance would strengthen this central claim.

Questions for authors

Q1. JEPA training stability.

How does the proposed VISReg-based objective compare with EMA-target or stop-gradient variants? Can the authors provide training stability curves or representation statistics to show whether the current formulation is more stable or effective?

Q2. Representation fusion.

Can the authors compare Representation Residuals with stronger learned fusion baselines, such as per-layer gating or cross-attention fusion, under matched parameter and computational budgets? This would help clarify the specific contribution of the proposed representation construction mechanism.

Q3. Two-stage curriculum.

Why are disjoint data halves used for the two training stages? Have the authors evaluated using the full dataset in both stages or changing only the attention masks while keeping loss weights fixed? Which component of the curriculum contributes most to the performance improvement?

7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We are delighted that the reviewer sfG2 finds our performance strong across diverse navigation tasks and considers the integration of task-adaptive representations and predictive latent reasoning an interesting direction. We also thank the reviewer for the detailed questions, which have helped us isolate the contribution of each design choice. We hope the following revisions and clarifications address the reviewer's remaining concerns. We have uploaded a revised PDF; table, figure, and appendix numbers below refer to the revised version.

1. JEPA training stability (Q1)

We thank the reviewer for the question. Our design follows LeWorldModel [1], which trains a JEPA end-to-end with a regularizer on the targets instead of the stop-gradient and exponential moving average (EMA) targets of BYOL [2] and I-JEPA [3]. Following the reviewer's suggestion, we train an I-JEPA-style alternative: the targets are computed by an EMA copy of Representation Residuals (τ increased linearly from 0.996 to 1.0, as in I-JEPA [3]) with stop-gradient and per-view LayerNorm, VISReg is removed, and the prediction loss and all other settings are unchanged (point-goal, full model):

Model VersionTarget updateRegularizerSR (%)SPL (%)CR (%)
RepresentNavJoint gradientVISReg95.0086.7013.45
Table 9Joint gradientNone92.9084.0916.40
Table 9Joint gradientSIGReg93.8584.3917.15
Table 9Joint gradientVICReg93.2583.9816.40
Stop-gradient + EMAStop-gradient, EMA target (τ: 0.996 → 1.0)None92.6084.1518.10

Training stability. We log statistics of the targets and predictions at every training step for all five variants; the full training curves over both stages (target std, prediction std, normalized prediction error, prediction loss, navigation loss, and gradient norm) are shown in Appendix Figure 9. The table reports medians over the last 200 steps; std is the per-dimension standard deviation across samples, and the normalized error is the prediction MSE divided by the target variance, where 1 corresponds to predicting the batch mean. Since each variant learns its own targets, this ratio detects collapse but does not compare prediction quality across variants:

Model VersionTarget stdPrediction stdPrediction error / target variance
Joint gradient + VISReg (ours)0.950.920.06
Joint gradient + SIGReg0.950.930.05
Joint gradient + VICReg0.950.950.03
Joint gradient, no regularizer0.00050.00094.4
Stop-gradient + EMA target0.340.260.43

With the same statistics logger, we further compare VISReg and the EMA target from the end of Stage I to the end of training; the corresponding training curves are shown in Appendix Figure 10:

Model VersionTarget effective rank (RankMe [4])Target mean pairwise cosinePrediction effective rank
Joint gradient + VISReg79 → 83−0.01 → −0.0179 → 82
Stop-gradient + EMA target79 → 470.50 → 0.8861 → 23

We find three distinct behaviors:

  • No regularizer. The jointly trained targets collapse within the first few hundred steps. The prediction loss falls to 1e-6, but the normalized error stays above 1, so the predictions carry no sample-specific information.
  • Stop-gradient + EMA. Training does not diverge, but the predictions start collapsed and recover only partially. When Stage II shifts the loss weight to the trajectory, the targets contract again (std 0.69 → 0.34, effective rank 79 → 47). On held-out samples, its predictions explain 63% of the target variance, compared with 96% for VISReg.
  • VISReg. The target and prediction std reach 0.95 within the first ~100 steps and remain there through both stages.

Joint training is therefore stabilized by regularizing the target distribution, without an EMA teacher; in our setting, the EMA teacher is less stable and yields lower navigation performance. If the reviewer considers it necessary, we would be happy to add a stop-gradient-only variant (with VISReg, without EMA) in the final version. We have added these results to Appendix B.3 (Tables 9 and 10).

[1] LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels

[2] Bootstrap your own latent: A new approach to self-supervised learning

[3] Self-supervised learning from images with a joint-embedding predictive architecture

[4] RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank

AuthorsOpenReview ↗

Rebuttal - Part II

2. Comparison with stronger fusion baselines (Q2)

We thank the reviewer for the suggestion. Under the setting of Table 2(a) (point-goal, no dream), we train three learned fusion baselines: per-layer gating (one softmax weight per feature map, shared across positions, with the same parameters and FLOPs as Representation Residuals), cross-attention fusion over the full-depth feature maps of all encoders, and a same-size three-layer transformer on the concatenated final-layer features. We also include Representation Residuals restricted to the final layer of each encoder. Parameters and FLOPs count the representation module only; the frozen encoders and the language model are identical:

Model VersionEncoder layersFusion paramsFLOPsSR (%)SPL (%)CR (%)
ConcatenationFinal––87.3080.2028.70
Concatenation + 3-layer transformerFinal47.3M0.80T90.9582.4319.20
Cross-attention fusionFull depth59.6M2.65T90.7081.7822.65
Per-layer gatingFull depth47.0M0.94T91.4580.4319.20
Representation ResidualsFinal47.0M0.80T92.3083.4819.80
Representation Residuals (ours)Full depth47.0M0.94T93.4584.2818.05

We find that all three learned fusions improve over concatenation but remain below Representation Residuals: cross-attention fusion, with 27% more parameters and 2.8× the FLOPs, reaches 90.70% SR, the same-size transformer 90.95%, and per-layer gating, matched in parameters and FLOPs, 91.45%. At matched capacity, Representation Residuals thus remain 2.00–2.50 SR higher, so their advantage does not come from additional capacity. The operator itself matters: with the same final-layer inputs and a matched budget (47.0M vs. 47.3M parameters, 0.80T FLOPs), Representation Residuals reach 92.30% SR, 1.35 points above the same-size transformer at a similar CR (19.80% vs. 19.20%). The intermediate layers add a further 1.15 SR and lower CR to 18.05%, but only through position-wise selection: per-layer gating and cross-attention over the same full-depth feature maps reach only 91.45% and 90.70%, both below the final-layer Representation Residuals. Part of the improvement over concatenation comes from learned fusion itself, while at equal capacity Representation Residuals remain 2.00–2.50 SR higher, owing to position-wise, content-dependent selection across encoders and their layers. We have added this comparison to Appendix B.2 (Table 7) and summarized it in Section 5.1.

3. Design of the two-stage curriculum (Q3)

We thank the reviewer for the question.

Disjoint data halves. We split the data so that every variant, including the single-stage baseline, sees the same samples the same number of times within the same training budget, and so that Stage II does not revisit samples already fitted in Stage I. Each half is drawn evenly from every source and task, and the split is by sample rather than by episode, so both stages cover all tasks and scenes (Appendix A.2). Following the reviewer's suggestion, we also train a version that uses all data in both stages, with twice the training steps (last row below).

Contribution of each component. Following the reviewer's suggestion, we decouple the attention mask from the loss weights (point-goal, full model; data, recipe, and evaluation as in the paper):

Model VersionAttention maskLoss weightsDataSR (%)SPL (%)CR (%)
RepresentNavRestricted → openStage I → Stage IIDisjoint halves95.0086.7013.45
Single-stage (Table 3)OpenFixedAll data92.4083.6815.15
Mask schedule onlyRestricted → openFixed (as single-stage)Disjoint halves92.5583.3819.10
Weight schedule onlyOpenStage I → Stage IIDisjoint halves92.6583.4417.85
All data in both stages (2× steps)Restricted → openStage I → Stage IIAll data in each stage94.5085.0214.45

We find that:

  • Neither schedule alone reproduces the gain. Switching only the attention mask reaches 92.55% SR, and switching only the loss weights reaches 92.65%, both on par with the single-stage model (92.40%); neither lowers CR (19.10% and 17.85% vs. 15.15%).
  • The gain requires both schedules together. Only their combination raises SR to 95.00% and lowers CR to 13.45%: in Stage I, the restricted mask makes the action position read the latents while the loss weights emphasize prediction. This is consistent with Table 4, where the two-stage model responds much more strongly to its latents than the single-stage model.
  • The data split is not a limiting factor. Training both stages on all data, at twice the cost, reaches 94.50% SR and 14.45% CR, without exceeding the disjoint halves (95.00% and 13.45%).

We have added these results to Appendix A.3 (Table 3).

AuthorsOpenReview ↗

Rebuttal - Part III

4. Evidence for the latent world-model claim (W2)

We agree with the reviewer that post-hoc decoding alone does not establish this claim. We have therefore removed the term "latent world model" from the abstract, the introduction, Figure 1, Section 5.3, and the conclusion, and provide the evaluations the reviewer suggested below.

Sensitivity to decoder capacity. We repeat the readouts with decoders of different capacity and compare the dream latent with the strongest encoder reference of Table 12, openD4RT applied to the real future frame (+0.4 s; occupancy IoU % / depth MAE):

DecoderopenD4RT on real futureRepresentNav
Linear29.39 / 0.109133.59 / 0.1030
Two-layer MLP29.99 / 0.101835.25 / 0.0940
Default (Table 12)29.79 / 0.102534.81 / 0.0913

Beyond capacity, every readout uses the same training budget and validation-based model selection for all representations, and the reported numbers are three-seed ensembles (Appendix B.4); the ordering holds under all three decoders.

Long-horizon accuracy. We decode each of the four latents at its own horizon and compare with a persistence baseline, which decodes the same future targets from the Representation Residuals [CLS] of the current frame with the same decoder, without the future frame and without a rollout. As a reference, we also decode the same targets from the Representation Residuals [CLS] computed on the real future frame (Table 12), so that the two differ only in the input frame (occupancy IoU % / depth MAE):

HorizonPersistence: RR [CLS] on current frameDream latentReference: RR [CLS] on real future
+0.4 s29.73 / 0.131734.81 / 0.091335.95 / 0.0752
+0.8 s29.51 / 0.135234.30 / 0.102836.66 / 0.0748
+1.2 s29.44 / 0.138434.38 / 0.111436.85 / 0.0756
+1.6 s29.42 / 0.141134.41 / 0.119037.31 / 0.0757

We find that the ordering of Table 12 holds across decoder capacities, and that depth and occupancy of the scene ahead remain decodable from the latents at all four horizons: the dream latent exceeds persistence by 4.8–5.1 occupancy IoU at every horizon and stays within 1.1–2.9 IoU of the Representation Residuals [CLS] computed on the real future frame. We therefore present the readouts as evidence of what the policy's representation space holds rather than of a world model, and base the role of the dream on its effect on navigation (Part IV). We have added these analyses to Appendix B.3 and B.4 (Tables 11, 13, and 14) and revised Section 5 accordingly.

Reviewer_DMSoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RepresentNav, a navigation foundation model that learns visual representations for robot navigation using Representation Residuals to combine features from different layers of different pretrained vision models. It then uses a latent chain-of-thought to predict future scene representations before it plans a path. The results show strong performance on several benchmarks and also on physical robots. However, most of the gains seem to come from use of a significantly larger, scaled-up model rather than key algorithmic novelty.

Critical strengths
  1. The paper addresses an important problem: fixed visual features do not always contain the information that robot navigation needs. The proposed method lets the navigation task control which visual features the model uses.
  2. The combination of task-specific representations and future prediction is very interesting.
  3. The experiments cover several navigation tasks, and the ablation studies help identify the effect of each component. The analysis of depth, semantics, occupancy, and motion also gives useful evidence about the information stored in the learned representations.
Critical weaknesses
  1. The paper does not clearly show that future predictions are used for planning. While latent prediction reduces collision rates and encodes spatial/motion info, these results do not establish that the planner actually uses predicted states to select better actions. In Stage II, the policy receives both current observations and predicted latents, so improvements could stem from auxiliary supervision or extra computation rather than future reasoning. Although there is open-loop trajectory shifts evaluation under intervention, a closed-loop intervention that removes or replaces predicted latents during navigation is important to establish their causal role.
  2. The learned representation is not compared with a capacity-matched alternative. Table 2 shows Representation Residuals outperform feature concatenation, but Representation Residuals also introduce trainable layers and cross-layer access. This comparison does not separate the benefit of selecting across encoder layers from the benefit of additional model capacity. A stronger baseline would use a similarly sized transformer over concatenated final-layer features. Without this comparison, evidence for the proposed representation mechanism remains incomplete.
  3. Higher tracking performance comes with substantially higher collision rates: RepresentNav improves tracking rates, but its collision rate is worse than existing methods. For a system operating around people and obstacles, the paper does not sufficiently explain the source of these collisions or whether tracking gains require aggressive, unsafe behavior.
  4. High computational cost and insufficient evaluation of simpler alternatives: Combining multiple vision encoders, a vision-language backbone, a representation module, and autoregressive latent prediction introduces heavy training and inference costs. Current ablations examine individual component contributions but do not establish whether the full architecture is necessary. It remains unclear whether simpler feature fusion, fewer encoder layers, or shorter rollouts could achieve comparable results at lower cost. A performance-versus-compute analysis would strengthen the paper.
  5. Suggestions (not critical weaknesses): (a) The authors could explain the role of the four future latent states with a simple navigation example. (b) It would also help to make the difference between a decodable representation and a representation that directly supports planning more explicit. The real-world section would benefit from a brief description of typical failure cases in addition to successful examples. (c) Finally, the paper could state the main trade-off between navigation performance, collision rate, and inference cost more directly.
Questions for authors

Please see weaknesses.

7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

We are delighted that the reviewer DMSo finds the problem we address important, considers the combination of task-specific representations and future prediction very interesting, and finds our ablations and readout analyses useful. We also thank the reviewer for the constructive feedback, which has helped us significantly strengthen the manuscript. We hope the following revisions and clarifications address the reviewer's remaining concerns. We have uploaded a revised PDF; table, figure, and appendix numbers below refer to the revised version.

1. Model scale and the source of the gains

We thank the reviewer for raising this point. We now report the parameter counts of RepresentNav and component ablations on VLN and tracking.

Model size. RepresentNav is built on Qwen3-VL-4B; the parameter counts of its components are listed below:

ComponentParametersTrained
Qwen3-VL-4B (LLM + ViT) and planning head4.44BLLM and planning head
DINOv3 ViT-S/1621.6MFrozen
Depth Anything 3 (Base)135MFrozen
OpenD4RT video encoder1.01BFrozen
Adapters, Representation Residuals, projectors47MYes
Total≈5.7B

With about 5.7B parameters, RepresentNav is smaller than Qwen-RobotNav-8B, yet exceeds it by 20.05 SR (fine-grained) and 12.41 SR (coarse-grained) on the VLNVerse Full Test, and by 8.5 SR and 3.0 TR on EVT-Bench single-target tracking with a single camera (Table 1). Table 2(a) further isolates the interface: with the backbone, training data, and three frozen encoders fixed, Representation Residuals (47M parameters) raise SR from 87.30% to 93.45% and lower CR from 28.70% to 18.05% relative to concatenation, and variants with 1 to 16 layers lie within 2.2 SR of one another (Table 6).

Gains on VLN and tracking. Following the training setting of the paper, we train the native-ViT variant (only the Qwen3-VL ViT, without external encoders or dream) and the no-dream variant with the same backbone, and evaluate them on VLNVerse (Val-Unseen, fine-grained) and EVT-Bench (single-target tracking, STT) under the protocols of the paper:

Model Version (four views)VLNVerse SR (%)SPL (%)OSR (%)NE ↓STT SR (%)TR (%)CR (%) ↓
Native ViT, no dream82.6780.4788.361.8487.786.42.23
Representation Residuals, no dream87.8885.0892.611.4388.893.92.66
RepresentNav89.2186.5092.731.4290.092.92.85
Model Version (single view)VLNVerse SR (%)SPL (%)OSR (%)NE ↓STT SR (%)TR (%)CR (%) ↓
Native ViT, no dream80.8579.2885.941.9573.982.06.47
Representation Residuals, no dream81.4579.1388.361.7286.593.04.53
RepresentNav83.1581.4986.911.8387.192.74.63

We find that the self-constructed space improves over the native interface on both benchmarks (+5.2 VLNVerse SR and +7.5 STT TR with four views; +0.6 SR and +11.0 TR with a single camera), and the dream further improves success in both settings (+1.3 VLNVerse SR and +1.2 STT SR with four views; +1.7 VLNVerse SR and +0.6 STT SR with a single camera). The gains of RepresentNav therefore stem from these two components, not from model scale alone. We have added the parameter counts to Appendix A.1 and these results to Appendix B.2 (Table 8).

AuthorsOpenReview ↗

Rebuttal - Part II

2. Whether the planner uses the predicted latents

This is a very good point. We agree that the open-loop shifts in Table 4 measure sensitivity rather than navigation performance, and now provide closed-loop evidence: on the full point-goal benchmark (4,000 trajectories), we intervene on the four predicted latents at every step, keeping images, history, and instructions unchanged:

Model VersionInference conditionSR (%)SPL (%)CR (%)
RepresentNavNormal95.0086.7013.45
RepresentNavZeroed latents93.5583.3420.05
RepresentNavLatents from another scene93.8084.5315.50
RepresentNavLatents masked from the action position89.1578.5628.75

We find that the policy depends on its latents in closed loop. Zeroing them raises CR from 13.45% to 20.05%, above even the no-dream model (18.05%, Table 2(a)), and masking them from the action position raises CR to 28.75% and lowers SR by 5.85 points. Latents from another scene lower SR to 93.80% and raise CR to 15.50%.

Regarding additional computation, Table 2(b) already contains this control. The w/o-supervision variant has the same architecture, rollout length, inference cost, and two-stage training as RepresentNav, and only omits the prediction loss and VISReg on the latents. At equal computation, the prediction objective raises SR from 93.35% to 95.00% and lowers CR from 15.90% to 13.45%:

Model VersionLatent rolloutTwo-stage curriculumPrediction loss + VISRegSR (%)SPL (%)CR (%)
No dream–––93.4584.2818.05
W/o supervision✓✓–93.3584.3415.90
RepresentNav✓✓✓95.0086.7013.45

Regarding auxiliary supervision, if the dream acted only as a training signal, intervening on the latents at test time would leave navigation unchanged; the closed-loop results above show that it does not. We have added these experiments to Appendix B.3 (Table 11) and revised Section 5.2 accordingly.

AuthorsOpenReview ↗

Rebuttal - Part III

3. Capacity-matched alternative

We thank the reviewer for the suggestion and train the suggested baseline directly: the final-layer features of the three encoders are concatenated and passed through a three-layer transformer of the same size as Representation Residuals before entering the language model. We also restrict Representation Residuals to the final layer of each encoder, with the module and token budget unchanged, and add two learned fusions over the full-depth feature maps of all encoders: cross-attention fusion, and per-layer gating with the same parameters and FLOPs as Representation Residuals (one softmax weight per feature map, shared across positions). All variants follow the setting of Table 2(a) (point-goal, no dream); parameters and FLOPs count the representation module only:

Model VersionEncoder layersFusion paramsFLOPsSR (%)SPL (%)CR (%)
ConcatenationFinal––87.3080.2028.70
Concatenation + 3-layer transformerFinal47.3M0.80T90.9582.4319.20
Cross-attention fusionFull depth59.6M2.65T90.7081.7822.65
Per-layer gatingFull depth47.0M0.94T91.4580.4319.20
Representation ResidualsFinal47.0M0.80T92.3083.4819.80
Representation Residuals (ours)Full depth47.0M0.94T93.4584.2818.05

We find that the same-size transformer on the concatenated final-layer features reaches 90.95% SR, and cross-attention fusion, with 27% more parameters and 2.8× the FLOPs, reaches 90.70%. At matched capacity, Representation Residuals thus remain 2.00–2.50 SR above learned fusions of the same size, so their advantage does not come from capacity, consistent with Table 6. With the same final-layer inputs and a matched budget (47.0M vs. 47.3M parameters, 0.80T FLOPs), Representation Residuals reach 92.30% SR, 1.35 points above the same-size transformer at a similar CR (19.80% vs. 19.20%), and selecting across the full depth of the encoders adds a further 1.15 SR and lowers CR to 18.05%. Per-layer gating, with the same parameters, FLOPs, and input feature maps as Representation Residuals but one weight per feature map shared across positions, reaches 91.45% SR, 2.00 below Representation Residuals and even below the final-layer variant (92.30%). Both the operator and the intermediate layers therefore contribute, and the intermediate layers help only through position-wise selection. Part of the improvement over concatenation comes from learned fusion itself, while at equal capacity Representation Residuals remain 2.00–2.50 SR higher, owing to position-wise, content-dependent selection across encoders and their layers. We have added this comparison to Appendix B.2 (Table 7) and summarized it in Section 5.1.

4. Collision rate in tracking

We thank the reviewer for pointing this out. In distracted tracking, RepresentNav collides more often than TrackVLA++ (11.89% vs. 4.71% with a single camera). We find that this gap stems from a mismatch between our tracking data and the collision definition of EVT-Bench. In our data collection, the robot is modeled as a 0.7 m × 0.5 m × 0.5 m box, and a collision means contact with this box. EVT-Bench instead records a collision whenever the center distance between the robot and the tracked target falls below 0.5 m, which ends the episode, and contacts with distractors or static obstacles do not count toward CR. The collected trajectories can therefore follow the target at a center distance below 0.5 m without contact, which the benchmark counts as a collision. Consistently, the collisions of the paper's model are recorded close to this threshold:

Center distance at the collision stepEpisodes (%, cumulative)
< 0.5 m (EVT-Bench definition)11.89
< 0.45 m5.10
< 0.4 m2.51
< 0.3 m0.50
< 0.2 m0

More than half of the collisions are recorded at a center distance between 0.45 m and 0.5 m, and none below 0.2 m. We therefore re-collect the tracking data under the benchmark's definition and retrain RepresentNav following the training setting of the paper (distracted tracking, single camera):

Model VersionTracking dataTR (%)SR (%)CR (%)
TrackVLA++–68.866.54.71
RepresentNav (paper)Original85.568.311.89
RepresentNavRe-collected under the EVT-Bench definition84.568.45.02

We find that, with the data aligned to the benchmark definition, CR falls from 11.89% to 5.02%, comparable to TrackVLA++ (4.71%), while TR remains 15.7 points higher (84.5% vs. 68.8%). The higher collision rate in the paper therefore mainly reflects the misaligned collision definition rather than aggressive tracking, and the tracking gain does not depend on it. We have added this analysis to Appendix B.5 (Table 17).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their valuable comments and suggestions. We appreciate that the reviewers recognized our work as addressing an important and clearly posed question (DMSo, RoEc), combining task-constructed representations with predictive latent reasoning in an interesting way (DMSo, sfG2), providing informative controlled analyses beyond benchmark numbers (DMSo, sfG2, RoEc), and demonstrating strong performance across diverse navigation tasks and on real robots (DMSo, sfG2, RoEc).

Core contributions

  • A self-constructed representation space. Instead of letting the encoders' pretraining objectives decide what the backbone sees, the policy's own objective constructs its visual representation through Representation Residuals, a compact three-layer transformer (47M parameters) over the full depth of semantic, geometric, and dynamic foundation models.
  • A dream in that space. Before every action, a latent chain-of-thought predicts the representation of the scene ahead, trained end-to-end with the policy as a JEPA.
  • State-of-the-art navigation. RepresentNav sets a new state of the art on VLNVerse (83.80% SR, +20.05), VLN-CE R2R, EVT-Bench (+10.1 TR with a single camera), and two point-goal benchmarks, transfers to real quadruped and humanoid robots, and uses less than half the training compute of Qwen-RobotNav and NavFoM.

Key findings

  • The interface matters as much as the encoders. With the same three encoders, the constructed space nearly doubles the gain of concatenation (SR 87.30% → 93.45%); the dynamic encoder adds 0.60 SR through concatenation but 3.65 through the constructed space, which draws 67% of its share from an intermediate layer.
  • Where to dream matters. Dreaming in the backbone ViT's space raises CR to 22.00%, above the no-dream model (18.05%), whereas dreaming in the constructed space lowers it to 13.45%.
  • One dreamed latent holds the scene ahead. Without occupancy supervision, the +0.4 s latent decodes 3D occupancy nearly as well as the representation of the real future frame (34.81% vs. 35.95% IoU), and the later latents exceed the current observation by about 5 IoU up to +1.6 s.

Summary of responses

We summarize our responses to the reviewers' comments as follows:

  • We isolate the source of the Representation Residuals gain. Beyond concatenation, the only baseline in the submission, per-layer gating matched in parameters and FLOPs, a same-size transformer, and cross-attention fusion all remain 2.0–2.75 SR below Representation Residuals, which still reach 92.30% when restricted to final-layer features (DMSo, sfG2, RoEc).
  • We provide closed-loop evidence that the policy uses its latents, beyond the open-loop shifts of the submission: zeroing them raises CR from 13.45% to 20.05%, the policy's own out-of-date latents of the same scene raise it to 18.95%, and episodes with less accurate predictions collide more often (r = 0.46) (DMSo, sfG2).
  • We provide persistence baselines at four horizons, decoder-capacity sweeps, attention-pooled encoder references, and the consistency of successive predictions of the same frame (cosine similarity 0.969) (sfG2, RoEc).
  • We provide ablations of the JEPA target update (stop-gradient with EMA targets reaches 92.60% vs. 95.00% SR) and of the two-stage curriculum, which needs both the attention mask and the loss-weight schedule (sfG2).
  • We provide component ablations on VLNVerse and EVT-Bench, an analysis of tracking collisions (CR 11.89% → 5.02% with data aligned to the benchmark definition), and the training cost, model size, latency, quantization (14.0 → 9.1 GB), real-robot deployment details, and CR definitions (DMSo, RoEc).

We appreciate the time invested by the reviewers, ACs, and SACs in helping us improve our paper. We are glad that our responses have addressed the main concerns of DMSo, who replied before the discussion ended. All of the above experiments and discussions are included in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.