acceptodds
Under review as a conference paper at ICLR 2027

VGN: Value-Guided Navigation from RGB for Robot Motion Planning

Abstract

Visual navigation from monocular RGB observations remains challenging due to the need for reliable scene understanding and safe decision-making in diverse real-world environments. Existing modular approaches often depend on additional sensing or interaction-heavy pipelines for reliable traversability estimation, while end-to-end methods typically infer navigation-relevant structure indirectly through trajectory-centric supervision. Motivated by these observations, we propose Value-Guided Navigation (VGN), an RGB-based framework for learning dense traversability-aware and goal-conditioned planning representations for downstream navigation. To achieve this, we leverage vision foundation models together with classical planning algorithms to generate dense traversability and cost-to-go pseudo-labels directly from static RGB images. This supervision is generated without trajectory supervision or robot interaction and is used to train the student planning representation model. The learned representation is planner-agnostic and can be integrated with multiple downstream planning mechanisms. Experiments across multiple outdoor datasets, closed-loop simulation environments, and real-world deployment on a Unitree Go2 robot demonstrate effective RGB-based traversability estimation and local motion planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Learning representations for robotic perception and Planning, Visual Navigation, Goal-Conditioned Planning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_wFk2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

One value-guided Navigation is proposed to learn dense traversability-aware and goal-conditioned planning representations for downstream navigation. Extensive experiments across multiple outdoor datasets, closed-loop simulation environments, and real-world deployment on a Unitree Go2 robot demonstrate effective RGB-based traversability estimation and local motion planning

Critical strengths

The training strategy is interesting. Without relying on collected trajectory demonstration, VGC generates pseudo-labels from RGB images, which allows training from static image data with on relying on real‑robot interaction.

The representation is learned goal‑aware cost‑to‑go representations in image space for downstream planners.

The evaluations are extensive, such as open‑loop cross‑dataset generalization, simulation closed‑loop navigation

Critical weaknesses

VGN performs on 2D images, with not considerations of depth, 3D geometry. As such, It lacks explicit depth estimation, which may introduce ambiguous traversability for tall grass, uneven terrain, foreground occlusions.

The supervision pipeline relies on GroundingDINO + SAM for semantic segmentation, plus manually hand‑coded traversability prior ranges for each terrain category. The quality of pseudo‑labels is not accurate and degrade the performances and downstream generalization ability.

The ablation studies show modest marginal gains from cost‑to‑go. Specifically, Table 5 shows only moderate navigation improvements when adding learned cost‑to‑go on top of pure traversability.

Questions for authors

Check the detailed comments in Weaknesses part.

3 comments

W1: We agree that the absence of explicit depth and 3D geometric reasoning is a limitation of VGN, as already acknowledged in Section 5 of the original submission. Our focus is on learning navigation-oriented representations directly from monocular RGB images without explicit geometric reconstruction. We specifically discuss challenges involving ambiguous traversability in tall grass and uneven terrain, foreground occlusions, and nearby obstacles. Incorporating depth- and geometry-aware planning remains an important direction for future work.

W2: We acknowledge that errors in GroundedSAM predictions and manually assigned traversability priors may propagate to the student and affect generalization. To further assess prediction quality beyond teacher-generated pseudo-labels, we added an evaluation against traversability references derived from human-annotated semantic labels (revised paper Table 3). On RUGD, VGN achieves MAE 0.082 and IoU 0.663, compared with 0.085 and 0.664 for the teacher. On Cityscapes, VGN achieves MAE 0.056 and IoU 0.929, compared with 0.042 and 0.936 for the teacher.

Table 3: Traversability evaluation using semantic-label-derived GT.

DatasetMethodMAE ↓RMSE ↓IoU ↑Corr. ↑
RUGDTeacher0.0850.1760.6640.861
RUGDVGN0.0820.1690.6630.874
CityscapesTeacher0.0420.1290.9360.957
CityscapesVGN0.0560.1380.9290.952

The lower IoU on RUGD is primarily associated with mulch regions, where GroundedSAM often assigns higher-traversability terrain categories, resulting in elevated teacher and student predictions. Appendix B in the revised manuscript details the reference construction and analyzes this discrepancy. Although these references rely on semantic traversability assumptions rather than physical measurements, they provide complementary evidence beyond teacher–student agreement and the existing human pairwise evaluation (Table 2). We have also explicitly acknowledged the potential impact of pseudo-label errors and handcrafted priors on generalization in the revised manuscript’s Section 5.

Moreover, we have added an independent zero-shot traversability evaluation on the CaT dataset (Appendix B), using externally annotated navigability ground truth. Without any fine-tuning or adaptation, the RUGD-trained VGN achieves an IoU of 0.814 and F1-score of 0.897, compared with 0.706/0.828 for W-RIZZ and 0.806/0.893 for CHUNGUS, respectively. VGN also achieves the highest IoU, F1-score, and recall across all evaluated thresholds (0.3, 0.4, and 0.5). We additionally provide qualitative comparisons.

Table 12: Zero-shot traversability evaluation on the CaT validation set at different prediction thresholds. Best results within each threshold are highlighted in bold.

τMethodIoU ↑F1 ↑Prec. ↑Recall ↑
0.3VGN (RUGD)0.8140.8970.8960.899
W-RIZZ0.7060.8280.7710.893
CHUNGUS0.8060.8930.9450.846
0.4VGN (RUGD)0.8020.8900.9080.873
W-RIZZ0.6870.8150.8350.796
CHUNGUS0.7240.8400.9750.738
0.5VGN (RUGD)0.7750.8730.9280.824
W-RIZZ0.6080.7570.8980.653
CHUNGUS0.5480.7080.9870.552

W3: We agree with the reviewer that adding learned CTG to conventional graph search provides modest improvements, as shown in Table 5 (revised manuscript: Table 6). This is partly because conventional graph search already computes paths using the traversability map, limiting the additional benefit of an explicit cost-to-go representation. However, our objective extends beyond improving conventional graph search to learning an explicit, goal-conditioned planning representation. As demonstrated in our original submission, direct gradient rollout and gradient-guided sampling using learned CTG achieve 68% and 65% success in Forest, and 71% and 68% in City, respectively (original manuscript: Table 4; revised manuscript: Table 5). These results demonstrate that learned CTG can directly guide navigation without relying on conventional graph search. We further added a controlled comparison in Table 8, showing that learned CTG achieves comparable navigation performance to Dijkstra-derived CTG across different planning mechanisms. Beyond this demonstrated utility, its differentiable architecture offers potential for future joint optimization with compatible differentiable planning frameworks.

Area_Chair_V5B9OpenReview ↗

Response Reminder

Dear Reviewer,

As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.

We greatly appreciate your contribution.

The authors mainly address my comments and provide additional experiments to address my comments on the performance gains of the GroundingDINO + SAM. I maintain my previous rating.

Reviewer_bccAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VGN, a framework that learns two dense representations from monocular RGB images: a traversability map and a goal-conditioned cost-to-go (CTG) map in image space, used for local point-goal navigation.

The supervision comes from a teacher pipeline. GroundedSAM produces open-vocabulary segmentation, which is mapped to traversability pseudo-labels through a hand-designed category-to-cost table. CTG pseudo-labels are then generated by running Dijkstra on top of the traversability map. The student network consists of a frozen LoftUp encoder, a semantic distillation module, a traversability head, and a CTG head. The learned representation can be used with several image-space planners. Experiments cover cross-dataset evaluation, HDR evaluation with human pairwise labels, closed-loop navigation in the CMU simulator, and real-world deployment on a Unitree Go2.

Critical strengths

The problem setting is meaningful. Turning large-scale static images into dense planning supervision, without demonstrations or robot interaction, is a scalable idea. The same representation plugs into several downstream planners. Closed-loop experiments report confidence intervals, and the appendix gives detailed implementation notes.

Critical weaknesses

1. The need for a learned CTG is not shown. The CTG head appears to take only the predicted traversability map and goal information as input, so it is essentially approximating Dijkstra. Dijkstra is exact and cheap on the same input. The paper's own Table 5 and Table 6 show that the learned CTG gives no clear gain over "traversability + Dijkstra." On the Orin, the CTG head is actually slower than Dijkstra. The speedup of the system mainly comes from distilling away GroundedSAM, not from learning the CTG.

2. The SPL numbers look inconsistent across rows. In Tables 4 to 6, some rows have SPL higher than success rate, which only makes sense under "average over successful episodes only." Other rows (such as WVN) match the standard SPL definition. Mixing the two definitions makes the SPL comparisons hard to trust.

3. The evaluation mostly measures how well the student fits the teacher, and some claims are too strong. Most quantitative metrics use teacher pseudo-labels as the reference, so they reflect distillation fidelity rather than true navigation quality. Simple baselines are missing, such as Euclidean distance as CTG, or "student traversability + Dijkstra." Some "outperforms all baselines" claims fall inside overlapping confidence intervals, and HDR is best only at τ=0.5, and is clearly worse than CHUNGUS at τ=0.1 and 0.25.

4. The ablations are inconsistent and do not support the claim that each component helps. In Table 12, the same full model appears in two rows with different numbers. Most differences are at the third or fourth decimal place, and all results come from a single run with no variance. Several ablated variants are actually better than the final design, which contradicts the text.

Questions for authors
  1. How is SPL computed in each row of Tables 4 to 6? Could you report standard SPL for all methods?
  2. In Table 6, which traversability map is "Teacher CTG" computed on? Could you add "student traversability + online Dijkstra" as a baseline, including its runtime on the Orin?
  3. Does the CTG head use any visual features? What information can a learned CTG provide that Dijkstra cannot?
  4. Could you add simple CTG baselines, and evaluate against dataset semantic ground truth or NaviTrace human trajectories, independent of the teacher?
  5. Why do the two rows of the full model in Table 12 differ? Could you report mean and variance across random seeds?
8 comments

W2 + Q1: We thank the reviewer for highlighting the inconsistency in our SPL reporting. Upon re-examining the episode-level evaluation results, we identified that the original Tables 4–6 inadvertently mixed standard SPL, averaged over all episodes, with conditional SPL, averaged only over successful episodes. This inconsistency arose during result aggregation and reporting rather than from the underlying per-episode evaluation logic. We have recomputed and corrected the SPL values in the revised manuscript using the conventional standard SPL definition: SPL = (1/N) Σᵢ₌₁ᴺ [Sᵢ ℓᵢ / max(pᵢ, ℓᵢ)]. Here, Sᵢ indicates whether episode i was successful, ℓᵢ denotes the shortest-path distance, and pᵢ is the executed path length. Unsuccessful episodes contribute zero to SPL. In the revised manuscript, all main-text navigation tables consistently report standard SPL, computed using the same aggregation procedure across all methods and 100 episodes per environment. The corresponding 95% confidence intervals have also been updated. For clarity, we reproduce the corrected standard SPL results from the revised Table 5 below.

Table 5: Standard SPL (95% CI) across Forest and City.

ProtocolMethodForest SPL ↑City SPL ↑
Full StackiPlanner0.639 [0.548, 0.730]0.649 [0.558, 0.739]
ViPlanner0.546 [0.447, 0.644]0.699 [0.609, 0.789]
POVNav0.465 [0.381, 0.549]0.173 [0.109, 0.243]
VGN (Graph)0.682 [0.610, 0.751]0.659 [0.569, 0.746]
VGN (Graph w/o heuristic)0.655 [0.583, 0.722]0.572 [0.480, 0.661]
VGN (CTG-based graph)0.495 [0.400, 0.590]0.596 [0.499, 0.693]
VGN (Gradient)0.580 [0.498, 0.661]0.679 [0.591, 0.765]
VGN (Sampling)0.600 [0.509, 0.687]0.672 [0.581, 0.763]
Fixed Nav.WVN0.020 [0.000, 0.049]0.471 [0.377, 0.567]
W-RIZZ0.631 [0.537, 0.723]0.109 [0.050, 0.170]
CHUNGUS0.697 [0.607, 0.786]0.617 [0.522, 0.710]
VGN (RUGD)0.717 [0.628, 0.806]0.568 [0.469, 0.666]
VGN (Cityscapes)0.806 [0.727, 0.877]0.724 [0.637, 0.809]

Furthermore, we have clarified the distinction between standard SPL and conditional SPL (SPL_succ) in Appendix F.2. Conditional SPL measures path efficiency averaged only over successful episodes and is reported separately as a complementary metric. The corresponding conditional SPL results for the complete navigation stack and fixed-navigation-module comparisons are presented in Appendix Table 16, reproduced below.

Table 16: Conditional SPL (95% CI) for closed-loop navigation comparisons in Table 5.

ProtocolMethodForest SPL_succ ↑City SPL_succ ↑
Full StackiPlanner0.999 [0.997, 1.000]0.999 [0.996, 1.000]
ViPlanner0.993 [0.989, 0.997]0.999 [0.998, 1.000]
POVNav0.846 [0.813, 0.879]0.826 [0.768, 0.884]
VGN (Graph)0.853 [0.824, 0.882]0.969 [0.953, 0.986]
VGN (Graph w/o heuristic)0.818 [0.786, 0.850]0.922 [0.885, 0.959]
VGN (Gradient)0.853 [0.818, 0.888]0.957 [0.932, 0.981]
VGN (Sampling)0.924 [0.904, 0.944]0.989 [0.979, 0.998]
Fixed Nav.WVN0.977 [0.895, 1.000]0.961 [0.933, 0.989]
W-RIZZ0.986 [0.978, 0.994]0.989 [0.973, 1.000]
CHUNGUS0.996 [0.992, 0.999]0.979 [0.953, 1.000]
VGN (RUGD)0.996 [0.994, 0.999]0.996 [0.992, 1.000]
VGN (Cityscapes)0.995 [0.992, 0.998]0.992 [0.983, 1.000]

These corrections ensure consistent SPL reporting and enable direct comparisons across all evaluated methods. We appreciate the reviewer for identifying this issue, which has helped us improve the accuracy and clarity of our evaluation.

W1+ Q2: We thank the reviewer for these observations. We agree that learned CTG does not consistently outperform traversability + Dijkstra in navigation performance, as shown in our results. However, our objective is not to outperform exact graph search, but to amortize goal-conditioned cost-to-go computation, enabling dense CTG prediction without online Dijkstra search. Regarding the baseline, student traversability + online Dijkstra was already included in the original submission (original Table 6; revised Table 7). We have now explicitly clarified the traversability source for each CTG configuration. We also agree that the primary system-level speedup comes from replacing GroundedSAM rather than Dijkstra, as explicitly discussed following Table 7. To further address the runtime concern, we additionally tested online Dijkstra on the Jetson Orin AGX through the full ROS2 pipeline. At 28×28 resolution, Dijkstra achieves a processing rate comparable to learned CTG at 224×224, with both operating at approximately 1.4 Hz in this additional test. Thus, we do not claim that learned CTG is universally faster. We use learned CTG for amortized dense prediction, with its onboard runtime reported in Table 9.

W1+Q3: We thank the reviewer for this question. The CTG head does not directly use additional visual features; it takes the predicted traversability map and goal information as input. Therefore, we do not claim that the learned CTG contains additional scene information unavailable to Dijkstra operating on the same inputs. Rather, our objective is to learn a dense, goal-conditioned planning representation through feed-forward prediction. As demonstrated in revised Table 8, learned CTG achieves comparable navigation performance to Dijkstra-derived CTG across different planning mechanisms (i.e. CTG-based graph search without hard obstacle masking, direct gradient rollout), supporting the effectiveness of amortizing classical cost-to-go computation while preserving downstream planning utility.

Table 8: Effect of CTG source on graph search and gradient rollout.

CTGGraph Forest Succ.Graph Forest SPLGraph City Succ.Graph City SPLGrad. Forest Succ.Grad. Forest SPLGrad. City Succ.Grad. City SPL
Dijkstra55%0.52162%0.61572%0.64069%0.677
Learned52%0.49560%0.59668%0.58071%0.679

Furthermore, unlike conventional discrete Dijkstra computation, the learned CTG head is differentiable, offering the potential for future joint optimization with compatible differentiable planning frameworks.

W3+ Q4: We thank the reviewer for the constructive feedback. We agree that teacher-based metrics primarily measure distillation fidelity rather than navigation effectiveness. However, our evaluation also includes teacher-independent assessments, which we have further strengthened in the revised manuscript.

Independent traversability evaluation. In addition to the human-annotated HDR evaluation (Table 2), we have introduced a new evaluation using semantic ground-truth annotations from RUGD and Cityscapes (Table 3 and Appendix B). Traversability references are constructed using predefined terrain priors assigned to human-annotated semantic categories, independently of teacher predictions. Regarding HDR, we acknowledge the threshold-dependent results: VGN achieves the lowest disagreement at τ=0.5, whereas CHUNGUS performs better at τ=0.1 and τ=0.25. Our revised discussion reflects this distinction rather than claiming superiority across all thresholds.

Table 3: Traversability evaluation using semantic-label-derived GT.

DatasetMethodMAE ↓RMSE ↓IoU ↑Corr. ↑
RUGDTeacher0.0850.1760.6640.861
RUGDVGN0.0820.1690.6630.874
CityscapesTeacher0.0420.1290.9360.957
CityscapesVGN0.0560.1380.9290.952

Independent navigation evaluation. Our closed-loop experiments (Table 5) assess actual robot navigation rather than teacher agreement. Success requires reaching the goal within a 0.5 m Euclidean distance without collision, while SPL compares executed trajectory lengths against reference start-to-goal distances. We evaluate 100 randomized episodes in each of the Forest and City environments against established navigation baselines. These metrics assess navigation outcomes independently of teacher-generated labels.

CTG baselines. We would like to clarify that the original submission already included student traversability + Dijkstra CTG (original Table 6; revised Table 7), enabling direct comparison with learned CTG. Given this existing classical CTG comparison, we prioritized evaluating navigation performance rather than introducing an additional Euclidean-distance CTG baseline.

Human trajectories. VGN is designed to learn without human trajectory supervision. While NaviTrace trajectories could provide a complementary evaluation reference, agreement with a particular demonstrated path does not necessarily indicate successful navigation, since multiple collision-free routes may reach the goal. We therefore emphasize closed-loop success, collision avoidance, and path efficiency as direct measures of navigation effectiveness. We also recognize that overlapping confidence intervals limit conclusions about superiority. Accordingly, our revised discussion emphasizes competitive performance and environment-dependent trade-offs rather than statistically established improvements. Together, these evaluations provide complementary evidence of VGN's effectiveness beyond teacher–student agreement, while acknowledging the remaining limitations.

W4+ Q5: We thank the reviewer for carefully examining the ablation results and highlighting these concerns.

Regarding the inconsistent full-model results, the “W Distillation” and “LoftUp” rows correspond to the same full-model configuration. However, three metric values were incorrectly reported in the “LoftUp” row during manuscript preparation. We have corrected these values in revised Table 14 to match the full-model results reported in the “W Distillation” row. We apologize for this reporting error.

Regarding variance across random seeds, repeating all ablation configurations across multiple random seeds would require substantial additional computational resources and was not feasible within the revision timeframe, particularly given the long job queues on the university's high-performance computing systems used for producing existing results.

Regarding the effectiveness of individual components, we agree that some ablated variants achieve marginally better results on individual metrics. As discussed in the revised Appendix D, the full configuration provides competitive performance across traversability and planning metrics, including favorable traversability RMSE, Kendall correlation, and CTG correlation. We now explicitly acknowledge that alternative token refinement, injection, and projection configurations can achieve comparable or slightly better results on individual metrics, rather than suggesting that every component consistently improves all metrics. These revisions correct the reporting inconsistency and provide a more balanced interpretation of the ablation results and their limitations.

Reviewer_xFZvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VGN aims to tackle navigation with traversability-aware and goal-conditioned cost-to-go representations from RGB observations with direct planning in image space. They aim to avoid the need for large amounts of hand labelled data and collected/interaction trajectories with a “supervision-generation” pipeline that uses vision foundation models (GroundingDINO and SAM). The representations from the teacher module are distilled to a lightweight student module. These representations are then used to predict a cost-to-go representation in 2D image space which can be directly used for downstream planning. Through experiments in simulated environments, the authors show the effectiveness of the RGB image-based method for navigation without the need for trajectory-level data and other sensors. Real-robot demonstration was also conducted and could run on robot hardware at 1.40Hz.

Critical strengths
  1. The authors propose a sound method to circumvent the need for large scale data annotation, trajectory collection or robot interactions with a supervision generation pipeline. VGN also shows strong results in actual navigation task with a distilled student model based on the supervision generation pipeline as VGN can achieve good traversability representations and cost-to-go representation for high success rates in downstream navigation.
  2. Substantial experiments on simulated environments and real hardware. Simulated results convincingly show VGN’s ability to navigate with only RGB-images as an input, compared to baselines like POVNav and ViPlanner. They also show that the traversability estimator of VGN is useful in downstream planning when compared to other recent methods like W-RIZZ and CHUNGUS.
  3. Real robot demo is provided with run time analysis to verify the ability for the proposed VGN inference pipeline to be run on real robot hardware.
Critical weaknesses
  1. Cost calculations are based on 8 connected graph in image space with a propagation cost of d(i,j)denoting spatial cost between pixels (Section 3.2). This is clearly not representative of the real 3D environment yet there is no ablation, test, or justification of this design choice. A justification would be good if authors belief that the effect of the distortion of 3D environment in 2D image space would not be a major concern. It would be interesting to see if this d(i,j) could be derived from some learned or classical method of BEV projection from image space, but this would be future work and not needed for the rebuttal.
  2. Some clarity issues that are elaborated on in the questions:
  • There is some vagueness with regards to Table 1 traversability comparison. Is it comparing between VGN’s teacher and student? If that is the case this is only comparing distillation effectiveness and not the effectiveness of VGN’s ability to capture traversability information and generalise.
  • Where is the City environment from?
  • It is unclear from the main text and the appendix how the 3D point is converted to 2D map goals.
  • It is also unclear why the student is required, why couldn’t GroundingDINO and SAM be used directly to get both the traversability and cost-to-go representations. I gather that it is likely due to the additional runtime required by GroundingDINO and SAM but this is not explicitly stated in the abstract, introduction, method or conclusion or I might have missed it. However, there should be more clarity with regards to the need for a student. Furthermore, without the student, using GroundingDINO and SAM is not entirely novel.
Questions for authors
  1. Are traversability comparison metrics in Table 1 derived from comparing VGN student’s traversability prediction with a. the traversability derived from the dataset’s terrain labels or b. the traversability from VGN teacher’s traversability prediction with GroundingDINO and SAM? It is not exactly clear which it is from my reading, and they have slightly different relevance to the results. It would be best to add clarity about the comparison and also provide a comparison of VGN’s teacher and student predicted traversibility vs the actual hand labelled ground truth from the datasets since they do provide image-level terrain labels. This would bolster the strength of the results from VGN further.
  2. Where is the City environment from? From my understanding the CMU Autonomous Exploration Development Simulator does not have a City environment. If it is from some where else, please do cite the source.
  3. How is goal converted from 3D to image space? What if the goal is not in frame (behind etc.). Appendix D only mentions “the projection instead produces a locally observable image-space subgoal”. How is the subgoal generated then, with rules and heuristics? There should be some extra information substantiating image level trajectory planning in the related work section with V-STRONG (which is already cited) and other related work possibly ALTER (Chen et al. 2024 IROS 2024)
  4. Why is the student model required? Are there any experiments you can provide to explain the rationale of distillation like runtime etc.
7 comments

W1: We thank the reviewer for highlighting this important limitation. We agree that pixel-based transition costs in an 8-connected image-space graph do not accurately represent physical distances in 3D environments and may be affected by perspective and terrain distortions. Accordingly, we have explicitly acknowledged this limitation in Section 5 (Limitations) of the revised manuscript by adding: “Image-space transition costs do not capture true 3D distances and are subject to perspective and terrain distortions.” We also agree that incorporating geometry-aware costs or BEV-based projection is a valuable direction for future work.

W2.1+Q1 We thank the reviewer for highlighting this ambiguity. We agree that distinguishing teacher–student agreement from evaluation against independently annotated references is important. To clarify, Table 1 evaluates VGN's student predictions against GroundedSAM teacher-generated pseudo-labels, rather than the datasets' human-annotated terrain labels. We have explicitly clarified this in both the main text of Section 4 and the caption of Table 1 in the revised manuscript. We also agree that an evaluation against human annotations would strengthen the results. However, the datasets do not provide continuous, manually annotated traversability scores. Therefore, we constructed traversability references from the human-annotated semantic segmentation labels of RUGD and Cityscapes by assigning each terrain category the midpoint of its predefined traversability range (Appendix Table 10). We have added Table 3 to evaluate both the teacher and VGN student against these semantic-label-derived references. On Cityscapes, the teacher and student achieve IoUs of 0.936 and 0.929, respectively, while on RUGD they achieve 0.664 and 0.663. These results demonstrate strong agreement on Cityscapes, with comparatively lower agreement on RUGD.

Table 3: Traversability evaluation using semantic-label-derived GT.

DatasetMethodMAE ↓RMSE ↓IoU ↑Corr. ↑
RUGDTeacher0.0850.1760.6640.861
RUGDVGN0.0820.1690.6630.874
CityscapesTeacher0.0420.1290.9360.957
CityscapesVGN0.0560.1380.9290.952

We further investigate the RUGD discrepancy in Appendix B (Figures 10 and 11). The lower IoU is primarily associated with mulch regions, where GroundedSAM frequently assigns higher-traversability semantic categories, such as pavement, soil, or grass, resulting in elevated teacher and student predictions. Although mulch accounts for only 11.5% of evaluated pixels, it contributes 67.9% and 70.1% of the binary disagreements for the teacher and student, respectively. We believe this additional evaluation provides a more informative assessment beyond teacher–student agreement, while acknowledging that semantic-label-derived traversability references are not equivalent to directly measured physical traversability.

Moreover, we have added an independent zero-shot traversability evaluation on the CaT dataset (Appendix B), using externally annotated navigability ground truth. Without any fine-tuning or adaptation, the RUGD-trained VGN achieves an IoU of 0.814 and F1-score of 0.897, compared with 0.706/0.828 for W-RIZZ and 0.806/0.893 for CHUNGUS, respectively. VGN also achieves the highest IoU, F1-score, and recall across all evaluated thresholds (0.3, 0.4, and 0.5). We additionally provide qualitative comparisons.

Table 12: Zero-shot traversability evaluation on the CaT validation set at different prediction thresholds. Best results within each threshold are highlighted in bold.

τMethodIoU ↑F1 ↑Prec. ↑Recall ↑
0.3VGN (RUGD)0.8140.8970.8960.899
W-RIZZ0.7060.8280.7710.893
CHUNGUS0.8060.8930.9450.846
0.4VGN (RUGD)0.8020.8900.9080.873
W-RIZZ0.6870.8150.8350.796
CHUNGUS0.7240.8400.9750.738
0.5VGN (RUGD)0.7750.8730.9280.824
W-RIZZ0.6080.7570.8980.653
CHUNGUS0.5480.7080.9870.552

W2.2+ Q2: We thank the reviewer for pointing out this missing clarification. The reviewer is correct that the City environment is not part of the original CMU Autonomous Exploration Development Simulator. We obtained the City environment from the publicly available Gazebo Models and Worlds Collection (https://github.com/leonhartyao/gazebo_models_worlds_collection) and integrated it into the CMU simulator framework for closed-loop navigation evaluation. We have revised the experimental setup description in Section 4 (Closed-Loop Navigation Evaluation) to explicitly acknowledge the external environment and added the corresponding citation.

W2.3+Q3.1: We thank the reviewer for highlighting the missing details of goal projection and local subgoal selection. We agree that the original manuscript did not sufficiently explain how global goals are projected into image space or how out-of-frame goals are handled. To address this, we have clarified the procedure in Section 3.1 of the revised manuscript as follows: “During deployment, the image-space goal ḡt is obtained by transforming a global 3D point goal gt into the robot’s local frame using its pose from an external localization system, followed by a ground-plane projection using camera parameters. Goals outside the camera view are mapped to bounded image-space targets based on their relative direction, with detailed projection and target-selection rules provided in Appendix E.” Specifically, the global 3D goal is first transformed into robot-relative forward and lateral coordinates using the robot's current pose and heading. These coordinates are then projected into image space using the camera intrinsics and height under a locally planar ground assumption. When the goal lies in front of the robot but projects outside the image boundaries, its location is clipped to an interior image margin. For goals behind or approximately beside the robot, a lateral image-space target is selected according to the goal's relative direction, encouraging the robot to turn toward the destination. The resulting target is then checked against the predicted traversability map. If it falls within a non-traversable region, it is shifted to the nearest feasible pixel. If no feasible pixel exists, local planning is skipped for that frame. As the robot moves, the global goal is repeatedly reprojected using the updated robot pose, allowing the local target to adapt progressively as the robot navigates around obstacles. We have expanded Appendix E with the mathematical projection equations, explicit target-selection rules, and traversability-based correction procedure. Figure 15 further illustrates how the image-space targets are progressively updated during real-world navigation, including cases where the initial projection overlaps with obstacles.

Q3.2: We thank the reviewer for highlighting the need for a more comprehensive discussion of image-space planning. We agree that the original Related Work section did not sufficiently cover this direction. Accordingly, we have expanded Section 2.1 (Modular Visual Navigation) with the following discussion: “A related line of work explores planning directly in image space, from graph-based path search over image-derived costs (Otte et al., 2007) to depth-aware pixel-space A planning (Aich, 2025). POVNav (Pushp et al., 2025) further explores navigability-guided subgoal selection and visual path planning.” Regarding the suggested works, V-STRONG (Jung et al., 2024) projects human-driven trajectories into image space to generate self-supervised traversability supervision, rather than directly performing image-space path planning. We therefore retain V-STRONG in our discussion of VFM-based traversability estimation. Similarly, ALTER (Chen et al., 2024) focuses on online visual traversability learning using LiDAR-derived geometric costs as supervision, rather than directly planning trajectories in image space. We have incorporated ALTER into the semantic and geometric traversability discussion in Section 2.1, explicitly stating: “TerrainNet (Meng et al., 2023) and RoadRunner (Frey et al., 2024) jointly modeled semantic and geometric terrain understanding for off-road navigation, while ALTER (Chen et al., 2024) leveraged LiDAR-derived geometric costs to supervise online visual traversability learning.” These revisions strengthen the coverage of both image-space planning and traversability learning, while clarifying how VGN differs by learning goal-conditioned cost-to-go representations from RGB observations without robot trajectory supervision.

W2.4+Q4: We thank the reviewer for raising this concern. We agree that the motivation for student distillation was not sufficiently explicit in the original manuscript. Although GroundingDINO and SAM can provide traversability representations, followed by classical planning to compute cost-to-go (CTG), repeated teacher inference is computationally expensive. We have therefore clarified this motivation in the revised Introduction: “The student thus avoids repeated traversability estimation using the computationally expensive vision foundation model-based teacher, while amortizing goal-conditioned cost-to-go computation into a learned prediction.” The original submission already included a teacher–student ablation (revised manuscript: Table 7), providing evidence of both inference efficiency and downstream navigation performance. On an RTX 4090, student traversability inference is 13.1× faster than GroundedSAM (35.2 vs. 461.0 ms), reducing total pipeline latency from 473.8 to 42.7 ms. With the same Dijkstra-based CTG, student traversability also improves navigation success from 67% to 82% in Forest and 48% to 66% in City. Importantly, the student also learns to predict goal-conditioned CTG directly, amortizing classical planning computation into a feed-forward prediction without requiring online Dijkstra propagation. The novelty of VGN lies in learning traversability-aware and goal-conditioned planning representations from static RGB images without trajectory supervision or robot interaction, rather than directly applying GroundingDINO and SAM. By distilling navigation-oriented supervision generated through vision foundation models and classical planning, VGN learns reusable representations that support multiple downstream planning mechanisms. Moreover, the differentiable CTG head offers the potential for future joint optimization with compatible differentiable planning frameworks.

Reviewer_mVSjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VGN learns traversability-aware, goal-conditioned planning representations from static RGB images without collecting robot-interaction trajectories or expert demonstrations for student training. A GroundedSAM-based teacher produces terrain regions, assigns manually specified traversability priors, and uses Dijkstra propagation on the resulting image graph to generate dense cost-to-go (CTG) targets. A student with frozen DINOv2/LoftUp features learns traversability and CTG prediction, with region-level semantic alignment and teacher-token refinement during training. At deployment, external localization and a ground-plane projection convert a global point goal into an image-space subgoal. Predicted maps support graph search, direct gradient rollout, and gradient-guided sampling, followed by image-based tracking and pose-based near-goal control. Validation includes cross-dataset teacher-label agreement, independent human pairwise traversability judgments, closed-loop navigation in Forest and City simulation environments, and qualitative Go2 demonstrations with onboard runtime measurements.

Critical strengths

1 A meaningful route to reducing navigation data collection Generating traversability and goal-conditioned CTG targets from static images addresses a practical bottleneck: acquiring diverse robot trajectories and interaction labels. The method exposes explicit spatial representations and can use several downstream planning mechanisms. This is a coherent and potentially reusable contribution even without a new RL algorithm or a new search algorithm. 2 Validation extends beyond fitting the pseudo-label teacher The paper includes independent human pairwise traversability evaluation, closed-loop simulation with uncertainty intervals, and onboard Go2 demonstrations. The graph-search stack’s reported 80% Forest success versus 64% for the strongest evaluated baseline is encouraging. The authors also disclose the lack of explicit depth and embodiment modeling rather than presenting the image-space planner as a complete geometric solution.

Critical weaknesses

1 The CTG advantages should be stated more narrowly The controlled comparisons show scenario-dependent CTG benefits. In Table 5, learned CTG leaves Forest success unchanged at 80% and reduces reported path efficiency from 0.845 to 0.682, while improving City success from 64% to 68%. In Table 6, with student traversability fixed, classical CTG obtains 82%/66% success in Forest/City, versus 80%/68% for learned CTG. These results support a useful learned representation and planner compatibility, but not consistent navigation-quality superiority over classical CTG. This limits the strength of the CTG-specific claim rather than negating the full framework. Likewise, the 71% versus 70% City comparison should be described as comparable performance rather than convincing superiority. The CTG-only runtime improvement on RTX 4090 is 12.8 to 7.5 ms; most of the full teacher-to-student speedup comes from traversability prediction. Separating these sources of efficiency would make the contribution more precise. I do not require learned CTG to outperform classical computation on every metric: amortized prediction and compatibility with different planners can be useful contributions on their own. 2 Two reporting and implementation details warrant clarification The main text and Appendix F.2 define SPL as path efficiency averaged over successful episodes, rather than conventional success-weighted SPL over all episodes. This is a legitimate complementary measure if consistently applied and clearly named. Table 4 contains some baseline values close to success rate and several VGN values near one; this pattern raises a question about the denominator, but does not establish an error. Please confirm a common implementation across methods. Separately, Section 3.3 describes prediction from teacher-injected Fground during training and Fproj during inference. Clarifying the supervision applied to the teacher-free branch and confirming that all reported evaluation removes teacher inputs would close an important gap in the description. I regard these as targeted clarification requests, not demonstrated failures of the system.

Questions for authors
  1. Reporting clarification: Do all SPL entries in Tables 4–6 use the same successful-episode denominator and failure handling? Please confirm the shared implementation and distinguish conditional path efficiency from conventional all-episode success-weighted SPL. If the implementation is already consistent, clarifying the description would address this concern.
  2. Method clarification: How is prediction from teacher-injected Fground during training transferred to teacher-free Fproj at deployment? Are task losses also applied to Fproj, or is another transfer mechanism used? Please confirm that the reported offline and navigation evaluations use the teacher-free branch. An explanation of the implemented training path may suffice; I am not assuming that teacher information was used at evaluation.
  3. Claim clarification: Given the mixed results in Tables 5–6, do you view the main CTG contribution as amortized prediction and compatibility with several local planners, rather than consistent navigation-quality gains over classical CTG? Please distinguish CTG-only speedup from the traversability-distillation speedup. Additional paired analysis of the existing runs would be informative, but a new large-scale robot benchmark is not necessary for my recommendation.
4 comments

W2 + Q1: We thank the reviewer for highlighting the inconsistency in our SPL reporting. Upon re-examining the episode-level evaluation results, we identified that the original Tables 4–6 inadvertently mixed standard SPL, averaged over all episodes, with conditional SPL, averaged only over successful episodes. This inconsistency arose during result aggregation and reporting rather than from the underlying per-episode evaluation logic. We have recomputed and corrected the SPL values in the revised manuscript using the conventional standard SPL definition: SPL = (1/N) Σᵢ₌₁ᴺ [Sᵢ ℓᵢ / max(pᵢ, ℓᵢ)]. Here, Sᵢ indicates whether episode i was successful, ℓᵢ denotes the shortest-path distance, and pᵢ is the executed path length. Unsuccessful episodes contribute zero to SPL. In the revised manuscript, all main-text navigation tables consistently report standard SPL, computed using the same aggregation procedure across all methods and 100 episodes per environment. The corresponding 95% confidence intervals have also been updated. Furthermore, we have clarified the distinction between standard SPL and conditional SPL (SPL_succ) in Appendix F.2. The conditional SPL results for the complete navigation stack and fixed-navigation-module comparisons are now reported separately in Appendix Table 16, corresponding to the standard SPL results in revised Table 5. These corrections ensure consistent SPL reporting and enable direct comparisons across all evaluated methods.

W2 + Q2: We thank the reviewer for identifying this ambiguity in our methodology description. We acknowledge that the original wording incorrectly suggested that F_ground was used for traversability prediction during training. In the actual implementation, F_proj is used for task prediction during both training and inference, while F_ground is used exclusively for teacher-guided semantic alignment during training. Specifically, the traversability loss (L_trav) directly supervises predictions from F_proj, while the cost-to-go loss (L_ctg) propagates through the traversability prediction pathway to the same projected features. In parallel, the semantic alignment loss (L_sem), computed using F_ground, provides teacher-guided supervision through the shared projection module. Thus, F_proj learns both task-relevant and semantically grounded representations without relying on teacher-injected features for prediction. We have corrected and clarified this distinction in Section 3.3 and Appendix C.1 of the revised manuscript. The t-SNE visualization in Appendix Figure 12 further provides qualitative evidence of the learned feature reorganization. We also confirm that all reported offline and closed-loop navigation evaluations of the student model use the teacher-free prediction pathway, without teacher masks, tokens, or feature injection. For additional verification, the implementation is available in the anonymized source code linked in our submission. Specifically, trainer.py (train_step(), lines 468–476 and 531–559) shows that the traversability and cost-to-go losses supervise the prediction pathway originating from F_proj. The forward() function in student/condensation.py further shows how F_ground is constructed during training and how the teacher-free pathway operates during inference.

W1+ Q3: We thank the reviewer for this constructive assessment and agree that the primary contribution of learned CTG lies in amortizing classical cost-to-go computation into a feed-forward prediction and supporting different local planning mechanisms, rather than consistently outperforming classical CTG in navigation quality. We have revised the manuscript to better reflect this distinction. First, we have clarified the efficiency motivation in the Introduction by explicitly stating: “The student thus avoids repeated traversability estimation using the computationally expensive vision foundation model-based teacher, while amortizing goal-conditioned cost-to-go computation into a learned prediction.” Second, we have revised the experimental discussion in Section 4 to explicitly acknowledge the environment-dependent benefits of learned CTG. As shown in revised Table 6, introducing learned CTG maintains the same 80% success rate in Forest, with a modest SPL improvement from 0.668 to 0.682, while improving success from 64% to 68% and SPL from 0.614 to 0.659 in City. We now explicitly state: “These results highlight the environment-dependent benefits of learned CTG.” Furthermore, revised Table 7 acknowledges that, with student traversability fixed, Dijkstra CTG achieves 82% versus 80% success in Forest, whereas learned CTG achieves 68% versus 66% in City. As clarified in the revised discussion, these results highlight “environment-dependent trade-offs of learned CTG rather than consistent navigation improvements over classical CTG.” Third, we have clarified the sources of inference efficiency in Section 4 and revised Table 7. On the RTX 4090, CTG computation decreases from 12.8 ms using Dijkstra to 7.5 ms using learned prediction, while traversability estimation decreases from 461.0 ms to 35.2 ms through student distillation. We explicitly acknowledge: “Thus, most of the measured inference latency reduction arises from traversability distillation rather than the learned CTG head.” Fourth, we have revised the navigation performance discussion in Section 4 to avoid overstating the City results. Specifically, we now state: “In City, the direct gradient rollout achieves 71% success, comparable to ViPlanner (70%).” We further acknowledge that ViPlanner achieves the highest City SPL (0.699), while VGN's direct gradient rollout remains competitive with an SPL of 0.679. Finally, we have added a controlled comparison in Table 8, showing that learned CTG achieves comparable navigation performance to Dijkstra-derived CTG across different planning mechanisms (i.e. CTG-based graphs earc without hard obstacle masking, direct gradient rollout). These results support the effectiveness of amortizing classical cost-to-go computation into a learned representation while preserving downstream planning utility. Furthermore, the differentiable CTG head offers potential for future joint optimization with compatible differentiable planning frameworks

Table 8: Effect of CTG source on graph search and gradient rollout.

CTGGraph Forest Succ.Graph Forest SPLGraph City Succ.Graph City SPLGrad. Forest Succ.Grad. Forest SPLGrad. City Succ.Grad. City SPL
Dijkstra55%0.52162%0.61572%0.64069%0.677
Learned52%0.49560%0.59668%0.58071%0.679

Together, these revisions provide a more precise characterization of learned CTG as an amortized, goal-conditioned planning representation whose navigation benefits depend on the environment and downstream planner. We appreciate the reviewer's suggestions, which have helped us better align our claims with the experimental evidence.

Area_Chair_V5B9OpenReview ↗

Response Reminder

Dear Reviewer,

As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.

We greatly appreciate your contribution.

Reviewer_mVSjOpenReview ↗

Response to Author Rebuttal

Thank you for the response and clarifications. I have considered the additional information and will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.