- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents RepresentNav, a navigation foundation model built on Qwen pretrained model that serves embodied navigation by waypoints prediction. The proposed methods includes mainly two parts, first a Representation Residuals, it is a three-layer transformer that mixes features from serveral depths of frozen vision foundation models through attention residuals, and passes per-view [CLS] and spatial tokens to the language model. Moreover, the paper proposes a four-step latent CoT component that trained as a JEPA to predict the [CLS] representations of future observations ahead. The paper reports results on five benchmarks and real-robot trials on two platforms, and authors` analyses attribute the gains to the constructed space and the latent prediction.
Critical strengths
-
The paper poses a clear and relevant question on which visual interface a VLM-based navigation policy should act on, and frames it well in terms of general-purpose versus purposive representations. The writing is clear and the narrative is coherent throughout.
-
The controlled comparison in Table 2(a) is informative. With the same three frozen encoders, Representation Residuals raise SR from 87.30 to 93.45 and lower CR from 28.70 to 18.05 relative to concatenation, and the dynamic encoder adds 3.65 SR through the constructed interface against 0.60 through concatenation. The result suggests that the interface to frozen encoders can matter as much as the choice of encoders, which is useful knowledge for the community, quite insightful.
-
The proposed model covers three families of navigation tasks with strong results across five benchmarks and it is deployed on a quadruped and a humanoid robots in real world.
-
The analysis goes beyond leaderboard numbers (regularizer and curriculum ablations, latent interventions, post-hoc readouts). The authors are also candid about the limits of several analyses.
Critical weaknesses
W1. The gain of the proposed RR is not attributed to its claimed source. Section 5.1 attributes the gain over concatenation to intermediate layers, but the comparison also changes the fusion module and the visual tokens given to the language model, and the token format of the concatenation baseline is not specified. The decisive control is missing: the same module restricted to the final layer of each encoder under the same token budget. Table 5 addresses capacity only. Concatenation is also the only fusion baseline, although the paper cites more expressive alternatives.
W2. The “latent world model” interpretation seems overstated. The latents are not conditioned on actions and are trained toward observations along expert trajectories, so the decoded ego-motion may simply reflect the plan. The readout comparison is also not like-for-like. Each encoder reference is a single globally pooled vector and no persistence baseline decodes the +0.4 s targets from the current frame, although the scene changes little within 0.4 s. That unsupervised latents already exceed the strongest encoder reference on occupancy (31.37 vs. 29.79 IoU) suggests that the decodable content comes mostly from the context rather than from prediction.
W3. The benchmark comparisons leave key factors uncontrolled. RepresentNav adds three frozen foundation models whose sizes are not reported, while all baseline numbers are copied from papers with different sensors and data. The VLNVerse margins (+20.05 SR over Qwen-RobotNav-8B and +36.32 over TANGO, a humanoid whole-body VLA) far exceed those elsewhere, but the camera setting, embodiment, and controller are not stated. Neither component is ablated on VLN or tracking, where the largest gains appear. Tracking collisions are also markedly higher than those of TrackVLA++ (11.89 vs. 4.71, monocular DT).
W4. The real-world claims rest on off-board computation and limited evidence. Inference runs on an off-board RTX 4090 server that receives the RGB streams and returns waypoints to the onboard controller, while the robot itself carries a Jetson Orin Nano 8GB (Figure 14). Each camera frame is encoded by three additional frozen encoders, one of them on a four-frame clip, and inference latency rises from 144.15 ms for the native model to 209.76 ms (Figure 14(c)). The paper reports neither the end-to-end latency with network transfer nor the control rate. It also does not examine communication delays or dropouts, although the abstract highlights deployment amid moving people. The real-robot evidence consists of 40 trials against a single baseline (Uni-NaVid), and the limitations paragraph (Section 6) does not discuss computational cost or onboard deployment.
Questions for authors
-
Could the authors report RR restricted to the final layer of each encoder, with the same module and token budget? Please also specify the token format and token count of the concatenation baseline.
-
Is the ego-motion decoded from the latents closer to the policy’s own predicted waypoints than to the ground-truth motion? Do depth and occupancy remain decodable from the +1.2 s and +1.6 s latents, where persistence is a weaker explanation?
-
Which camera configuration, embodiment, and controller does RepresentNav use on VLNVerse, and were the baselines evaluated under the same settings and trained on VLNVerse data? What explains the 36-point gap to TANGO on Val-Unseen?
-
Could the authors report the native-ViT and no-dream variants of Table 2 on VLNVerse and EVT-Bench? On EVT-Bench, what explains the higher collision rate relative to TrackVLA++, and does it reflect a trade-off with the tracking rate?
-
What are the sizes of the three encoders and the total parameter count? Can the model run on the onboard Jetson Orin Nano 8GB, and what are the end-to-end latency with network transfer and the control rate in the real-world trials?
-
How is CR defined in the point-goal ablations? Section 4.1 lists only SR and SPL for point-goal. If CR counts episodes with any collision, an SR of 95.00 with a CR of 13.45 implies that at least 8.45% of episodes succeed despite a collision.
7 comments
Rebuttal - Part II
2. The world-model interpretation and the readouts (W2, Q2)
We thank the reviewer for this careful analysis and agree that "latent world model" overstates what the rollout is. The latents are not conditioned on actions, and all training trajectories are expert trajectories, so the rollout predicts the future that the policy is about to bring about. Unlike action-conditioned world models [1, 2, 3], it cannot be queried with alternative actions; it is closer to models that predict the future and read the action from it, such as DreamZero [4], which is likewise trained on demonstrations. This matches the intent of our design ("dream what I want"), and we have removed "latent world model" from the abstract, introduction, Figure 1, Section 5.3, and conclusion.
Ego motion and the plan. We compare the decoded ego motion with the ground truth and with the policy's own predicted waypoints on the same test observations (ADE over +0.2 to +1.2 s):
| Comparison | ADE (cm) |
|---|---|
| Decoded ego motion vs. ground truth | 4.45 |
| Decoded ego motion vs. policy waypoints | 3.42 |
We find that the decoded ego motion is closer to the policy's own waypoints than to the ground truth. Since the planning head reads its waypoints after the rollout, this confirms the reviewer's reading: the decoded ego motion largely reflects the motion the policy intends to execute.
Persistence and longer horizons. We add a persistence baseline that decodes the future targets from the Representation Residuals [CLS] of the current frame with the same decoder. We decode each latent at its own horizon and, as a reference, decode the same targets from the Representation Residuals [CLS] computed on the real future frame, which differs from persistence only in the input frame (occupancy IoU % / depth MAE):
| Horizon | Persistence: RR [CLS] on current frame | Dream latent | Reference: RR [CLS] on real future |
|---|---|---|---|
| +0.4 s | 29.73 / 0.1317 | 34.81 / 0.0913 | 35.95 / 0.0752 |
| +0.8 s | 29.51 / 0.1352 | 34.30 / 0.1028 | 36.66 / 0.0748 |
| +1.2 s | 29.44 / 0.1384 | 34.38 / 0.1114 | 36.85 / 0.0756 |
| +1.6 s | 29.42 / 0.1411 | 34.41 / 0.1190 | 37.31 / 0.0757 |
Depth and occupancy remain decodable at +1.2 s and +1.6 s. The dream latent exceeds persistence by 4.8–5.1 occupancy IoU at every horizon and stays within 1.1–2.9 IoU of the Representation Residuals [CLS] computed on the real future frame. Since the latents are not action-conditioned and the decoded ego motion largely reflects the policy's plan, we present the readouts as evidence of what the policy's representation space holds rather than of a world model. We have added these analyses to Appendix B.4 (Table 13).
[1] Recurrent world models facilitate policy evolution
[2] Navigation world models
[3] LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels
[4] World action models are zero-shot policies
Rebuttal - Part III
2. The world-model interpretation and the readouts (W2, Q2), continued
Context and prediction. We agree that much of the decodable content comes from the context: the rollout without supervision, which shares the latent's inputs, already reaches 31.37% occupancy IoU, above the strongest mean-pooled encoder reference (29.79%). With the same inputs, prediction supervision raises this to 34.81% (+3.44 IoU, +4.88 mIoU) and lowers depth MAE from 0.1068 to 0.0913 (Table 12). The four latents also distinguish future frames 0.4 s apart: successive predictions of the same frame agree more closely (cosine similarity 0.969) than predictions of frames 0.4 s apart (0.955), with similarity falling to 0.838 at 1.6 s apart (Appendix B.4). In closed loop, the policy depends on this predicted content: zeroing the latents raises CR from 13.45% to 20.05% on the full point-goal benchmark.
Like-for-like references. Table 12 already contains a learned single-vector summary of the real future frame, the Representation Residuals [CLS] (35.95% IoU). We further add attention-pooled encoder references, in which a learnable query pools the tokens of each encoder into one vector that is fed to the same decoder (+0.4 s; occupancy IoU % / depth MAE):
| Model Version | Mean-pooled (Table 12) | Attention-pooled |
|---|---|---|
| Backbone ViT on real future | 25.47 / 0.1626 | 29.97 / 0.1014 |
| DA3 on real future | 26.14 / 0.1447 | 31.03 / 0.0886 |
| DINOv3 on real future | 25.85 / 0.1521 | 31.30 / 0.0932 |
| openD4RT on real future | 29.79 / 0.1025 | 33.22 / 0.0876 |
| RR [CLS] on real future | 35.95 / 0.0752 | – |
| RepresentNav (predicted, without the future frame) | 34.81 / 0.0913 | – |
We find that attention pooling narrows the gap: openD4RT and DA3 reach lower depth error than the dream latent (0.0876 and 0.0886 vs. 0.0913), while the latent remains higher in occupancy (34.81% vs. at most 33.22%). We have therefore removed "more accurately than pretrained encoders applied to the real future frame" from the abstract, the introduction, and Section 5.3, compare instead with the learned Representation Residuals [CLS] reference, and added all readout analyses to Appendix B.4 (Tables 13–15).
3. Definition of CR (Q6)
We thank the reviewer for the careful check. CR is the percentage of episodes with at least one collision. Under the InternVLA-N1 protocol used in Table 2, a collision does not terminate the episode and success requires only reaching the goal, so an episode can succeed after a collision. The reviewer's inference is therefore correct: at least 8.45% of the episodes succeed despite a collision, which is why we report CR alongside SR in these ablations. In contrast, success on ABotN-PointBench requires collision-free arrival, under which RepresentNav reaches 97.5% SR (Table 1(c)). We have defined CR in Section 4.1, added the definition for each benchmark to Appendix A.4, and clarified there that "a quarter fewer collisions" refers to this episode-level rate (18.05% → 13.45%).
Rebuttal - Part I
We are delighted that the reviewer RoEc finds the question of which visual interface a VLM-based navigation policy should act on clear and relevant, considers the controlled comparison in Table 2(a) informative for the community, and appreciates that our analyses go beyond leaderboard numbers. We also thank the reviewer for the careful reading and precise feedback, which have helped us significantly strengthen the manuscript. We hope the following revisions and clarifications address the reviewer's remaining concerns. We have uploaded a revised PDF; table, figure, and appendix numbers below refer to the revised version.
1. Final-layer control and token format (Q1, W1)
We thank the reviewer for suggesting this decisive control.
Token format. In the concatenation baseline, the final-layer features of the three encoders are aligned to the same 16 × 16 grid by the same adapters, concatenated along channels, and read out in the same token format as Representation Residuals: each current view contributes [CLS] plus 64 fine tokens, and each historical view [CLS] plus eight coarse tokens (Appendix A.1). All interfaces therefore use the same token budget, and we have added the token format of every interface to Appendix A.1.
Final-layer control and other fusions. We train Representation Residuals on the final layer of each encoder, with the module and token budget unchanged. Since concatenation was our only fusion baseline, we also add three learned fusions: a same-size three-layer transformer on the concatenated final-layer features, cross-attention fusion over the full-depth feature maps of all encoders, and per-layer gating with the same parameters and FLOPs as Representation Residuals (one weight per feature map, shared across positions) (setting of Table 2(a): point-goal, no dream):
We agree that Table 6 (Table 5 in the submission) addresses capacity only; this control addresses where the gain comes from. We find that, restricted to final-layer features, Representation Residuals reach 92.30% SR and 19.80% CR, compared with 90.95% SR and 19.20% CR for a transformer of the same size on the same final-layer inputs and token format. Both the final layers and the intermediate layers therefore contribute: of the 6.15 SR gain over concatenation (87.30%), learned fusion of the final layers accounts for 3.65 SR and the Representation Residuals operator for a further 1.35 SR, while access to the intermediate layers adds the remaining 1.15 SR and lowers CR from 19.80% to 18.05%. The intermediate layers help only through position-wise selection: over the same full-depth feature maps, per-layer gating, with matched parameters and FLOPs but one weight per feature map shared across positions, reaches 91.45%, and cross-attention fusion, with more parameters and computation, 90.70%, both below the final-layer Representation Residuals. Part of the improvement over concatenation comes from learned fusion itself, while at equal capacity Representation Residuals remain 2.00–2.50 SR higher, owing to position-wise, content-dependent selection across encoders and their layers. We have added this comparison and the breakdown above to Appendix B.2 (Table 7), summarized it in Section 5.1, and retained Figures 4 and 8 as descriptions of what Representation Residuals attend to.