acceptodds
Withdrawn from ICLR 2027

GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent vision-and-language navigation (VLN) systems increasingly adopt streaming Video-LLM policies that map egocentric RGB observations and instructions directly to low-level actions. Yet these policies inherit weak 3D geometric priors from 2D pretraining. Existing geometry-aware extensions charge a persistent inference-time price: depth sensors, 3D encoders, or per-step perception tool calls. We propose GeoScaffold, a geometric supervision framework that pays this price once, at training time, by internalizing geometry into the policy itself. It first learns a compact depth tokenizer on depth maps from the training trajectories and freezes it. It then fine-tunes the policy with a handful of learnable geometry query tokens, training their hidden states to reconstruct navigation-critical geometry such as depth, connectivity, and traversability. This supervision turns the query states into compact geometric latents for action decoding, and through the shared weights also internalizes geometry into the backbone's own representations. Like a scaffold, the tokenizer, target generators, and reconstruction heads are discarded after training, leaving the backbone and action interface unchanged. Extensive experiments show that GeoScaffold consistently outperforms leading vision-only navigators on continuous VLN benchmarks, offering a practical paradigm for lightweight edge deployment of spatially aware embodied navigation models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
navigation foundation models, vision-and-language navigation, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ihuYOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to add eight learnable "geometry query tokens" at the end of each user turn of a streaming Video-LLM navigation policy. During training, the last-layer hidden states of these tokens are decoded by small heads to reconstruct three targets for the current frame, and binary connectivity and traversability masks. Query tokens attend bidirectionally within their group. After training, the heads and tokenizer are discarded, so inference uses only RGB and text, with the eight tokens left in the prompt. On R2R-CE / RxR-CE Val-Unseen the 4B model reaches 64.8 / 59.6 SR, above 7B geometry-enhanced methods such as JanusVLN and GA-VLN, at 81 ms per frame versus 750–1,160 ms.

Critical strengths
  • The paper is clear and the goal is practical. The design is simple and backbone-agnostic.
  • Controlled comparisons are good: same backbone with/without GeoScaffold on three model families (Table 2), and a "query tokens without reconstruction" control showing the gain is not from extra tokens.
  • The ablations are informative: which hidden states to supervise (image tokens hurt badly, action tokens hurt mildly), discrete vs continuous vs pixel depth targets, attention mask, token budget, and blocking the latents at test time to show that action decoding reads them.
Critical weaknesses
  • The novelty is incremental. Auxiliary reconstruction through learnable query/perception tokens is well established (ROSS, 3DRS, Spatial Forcing, DreamVLA, Perception Tokens, LISA-style readouts). The new part is the specific target set for VLN.
  • The headline Table 1 comparison is confounded by backbone. GeoScaffold-4B uses Qwen3-VL, while most baselines use older 7B backbones. The same backbone trained with the full data recipe but without GeoScaffold is missing from Table 1.
  • The latency comparison in Fig. 4 contrasts a 4B RGB-only model with 7B models that run VGGT or BEV builders. Any RGB-only policy of the same size would have the same latency, so this is not evidence for GeoScaffold specifically.
  • All simulation numbers are single runs. Many ablation gaps are 1–2 SR on ~1.8K episodes, which is near the sampling noise of SR.
Questions for authors
  1. Full-data baseline. Please add "Qwen3-VL-4B w/o GeoScaffold" trained with the identical full-data recipe (ScaleVLN, DAgger, LLaVA-Video, ScanQA, MMC4) to Table 1. Without it, the 64.8 SR cannot be attributed to GeoScaffold rather than to the newer backbone.
  2. Comparison with training-only alternatives. Spatial Forcing (aligning hidden states to VGGT features during training only) and 3DRS also add no inference cost. How does GeoScaffold compare to them on the same backbone and data?
  3. Why does image-token supervision collapse to 43.2 SR? That is 5.6 points below the policy with no geometry at all. Was the loss weight or head re-tuned for that variant? A drop this large suggests an implementation or weighting issue rather than an inherent property.
1 comment

Thank you for your constructive review and for recognizing the controlled comparisons and ablations.


Q1: Is the novelty mainly the VLN target set? How does GeoScaffold compare with training-only alternatives such as Spatial Forcing and 3DRS?

A1: Spatial Forcing and 3DRS also use training-only supervision, so removing auxiliary modules at inference is a shared advantage. Our contribution combines navigation-specific targets with persistent geometric query memory, retaining spatial cues across viewpoint changes, rooms, and corridors for later action decisions.

  1. Historical geometric memory. GeoScaffold retains eight query states per frame throughout the episode, accessible to actions after RGB frames leave the visual window. We now report results for separately trained policies for each access scope, with fixed supervision targets. Full-history access improves SR/SPL by 4.5/5.6 points over current-turn access and 1.6/3.1 over sliding-window access (Table 10). These gains demonstrate the importance of geometric history for VLN: earlier spatial cues guide later route decisions across rooms and corridors.
Accessible geometry queriesNE ↓OS ↑SR ↑SPL ↑
Current-turn queries5.4257.250.845.4
Sliding-window queries5.1759.553.747.9
Full-history queries (Ours)5.0761.955.351.0
  1. Navigation targets. Depth provides distance cues, connectivity identifies route continuations, and traversability identifies walkable surfaces. The submitted target ablation shows that removing any target lowers SR from 55.3 to 51.1–53.1 and SPL from 51.0 to 47.1–49.2 (Table 3), supporting their complementary roles.

  2. VGGT feature alignment. Aligning the VLN model's final-layer image tokens with VGGT's final-layer features yielded only a modest SR gain. Retaining dense image states for direct access to geometric history would require many more cached tokens, so we chose compact query memory with eight queries per frame.


Q2: Please add a same-backbone baseline under the full-data recipe.

A2: We now report this comparison below. The results are also included in Appendix A, Table 7 (p. 14). The baseline uses Qwen3-VL-4B and the same full-data recipe, with geometry queries, reconstruction heads, and geometric losses removed. GeoScaffold improves SR/SPL by 6.4/9.2 points on R2R-CE and SR/SPL/nDTW by 3.6/6.5/3.2 points on RxR-CE.

Val-UnseenMethodNE ↓OS ↑SR ↑SPL ↑nDTW ↑
R2R-CEQwen3-VL-4B5.0164.358.451.3–
R2R-CE+ GeoScaffold4.1772.164.860.5–
RxR-CEQwen3-VL-4B6.13–56.045.763.1
RxR-CE+ GeoScaffold4.51–59.652.266.3

Q3: The latency comparison mixes 4B and 7B systems. Can the advantage be attributed to GeoScaffold?

A3: The comparison measures complete-system deployment cost. With Qwen2.5-VL-7B, GeoScaffold runs at 121 ms/frame on an RTX 4090, compared with 750 ms for GA-VLN and 1,160 ms for JanusVLN, showing low latency also at 7B. At inference, we remove the tokenizer and reconstruction heads. The retained queries add attention computation and KV storage.

SystemBackbone scaleRTX 4090 latency (ms/frame) ↓
GeoScaffoldQwen3-VL-4B81
GeoScaffoldQwen2.5-VL-7B121
GA-VLN7B750
JanusVLN7B1,160

Q4: Are the simulation results single runs?

A4: All simulation results are averages over three random seeds. We apologize for not stating this clearly.


Q5: Why does image-token supervision drop to 43.2 SR?

A5: The submitted ablation (Table 5) keeps the three targets, loss weights, head parameter count, and training hyperparameters unchanged. For depth reconstruction, it mean-pools each frame's final-layer image states into a 3,584-dimensional vector and predicts depth tokens using LayerNorm and MLP. Pooling compresses spatial information into one vector, and the image states support both navigation and depth reconstruction. Dedicated queries provide separate geometric readout positions. Supervising a subset of image tokens raises SR from 43.2 to 48.7, but remains below the 55.3 achieved with dedicated queries.

Reviewer_haDGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoScaffold, a training-time geometric supervision framework for VLN. It adds learnable geometry query tokens to a streaming Video-LLM policy and supervises their hidden states to reconstruct depth, connectivity, and traversability. The depth tokenizer and reconstruction heads are removed after training, so the deployed policy uses RGB observations and language instructions without a separate depth sensor or 3D encoder. The paper reports improvements on R2R-CE and RxR-CE, across several backbones, and in real-world robot evaluations.

Critical strengths
  1. The method incorporates geometric supervision during training while keeping the deployed navigation pipeline relatively lightweight. The reported latency and on-board robot results support the practical motivation.

  2. The paper evaluates multiple navigation benchmarks and model backbones, includes controlled comparisons with and without GeoScaffold, and provides ablations of the supervision targets, query states, and attention design.

Critical weaknesses
  1. The most significant concern is novelty. The authors insert learnable query tokens into the Transformer of a VLN model, where they participate in attention, and apply geometric supervision to the resulting latents, particularly depth supervision and semantic supervision derived from DINO and SAM. This idea seems to be a direct adaptation of DreamVLA [1] to VLN. The authors should explain the differences from DreamVLA more clearly to establish the novelty of this work.

  2. It would be helpful to provide visualizations showing that the geometry queries have indeed learned geometric information. For example, given the reconstruction loss during training, can the learned depth tokens predict plausible depth maps during inference in the robot evaluation? Such visual evidence would offer an intuitive demonstration of the proposed mechanism.

  3. In Table 7, why does performance decrease as the number of depth query tokens increases? Common knowledge might expect the benefit of this redundancy to saturate, rather than for performance to deteriorate.

[1] Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. NeurIPS2026.

Questions for authors

See Weaknesses.

1 comment

Thank you for your careful review and for recognizing the practical motivation and experimental coverage. We address each point below.


Q1: The approach looks like DreamVLA applied to VLN. What are the differences?

A1: Thank you for pointing out that the differences from DreamVLA were not sufficiently clear. We share the idea of learnable states with perceptual supervision. Our navigation-specific design concerns what those states represent and how actions access them over time. Compared with manipulation in a localized workspace, streaming VLN spans rooms and corridors with larger viewpoint changes and wider depth variation. Earlier spatial cues remain relevant after they leave view, motivating persistent geometric query memory throughout the route.

  1. Direct access to historical query states. DreamVLA uses observation history, but its released attention mask allows action tokens to directly access prediction queries only from the same timestep. GeoScaffold retains eight query states per frame for direct action access throughout the episode, including after the RGB frames leave the visual window, as detailed in Section 3.2 and Appendix C.3 (pp. 18–19). We now report the corresponding query-access results below, with a separately trained policy for each scope. Full-history access improves SR/SPL by 4.5/5.6 points over current-turn access and by 1.6/3.1 points over sliding-window access on R2R-CE Val-Unseen. These results are also included in Appendix A, Table 10 (p. 15).
Accessible geometry queriesNE ↓OS ↑SR ↑SPL ↑
Current-turn queries5.4257.250.845.4
Sliding-window queries5.1759.553.747.9
Full-history queries (Ours)5.0761.955.351.0
  1. Navigation-specific reconstruction targets. DreamVLA forecasts future dynamic regions, depth, and semantic features for manipulation. Our queries reconstruct current-view depth, passage-connectivity, and traversability, supporting distance judgment, route continuation, and local movement. The target ablation reported in the submission shows that removing any single target lowers SR from 55.3 to 51.1–53.1 and SPL from 51.0 to 47.1–49.2 (Table 3).

Q2: Please visualize the geometric information learned by the queries.

A2: Thank you for this helpful suggestion. We now include depth reconstruction visualizations from held-out R2R-CE Val-Unseen scenes in Appendix A, Figure 7 (p. 16). The examples show depth decoded from the query states alongside the RGB input, ground-truth depth, and the VQ-VAE reconstruction of ground-truth depth. The decoded depth is used only for visualization and is not fed back into action prediction. We also report depth reconstruction accuracy on 2,500 held-out frames (Table 12, p. 16): query-based reconstruction reaches 78.3% δ₁, compared with 91.7% for the VQ-VAE reference and 50.2% for a frozen baseline without GeoScaffold. The table below also reports connectivity/traversability IoUs of 70.6%/87.4%, compared with 34.2%/42.5% for the baseline.

Reconstruction sourceDepth δ₁ ↑ (%)Connectivity IoU ↑ (%)Traversability IoU ↑ (%)
Query states from RGB observations78.370.687.4
Baseline features (frozen backbone, matched depth head)50.234.242.5
Ground-truth depth through the VQ-VAE91.7––

Q3: Why do more depth queries reduce performance?

A3: Thank you for raising this question. The trend is not a uniform decline. SR rises from 52.7 to 55.3 when moving from two to four depth queries, stays at 55.0 with six, and drops to 53.4 with eight. One possible explanation is that extra query positions change attention allocation and optimization without adding useful information, so a compact query group acts as a useful constraint under a fixed training budget. A smaller query set may also limit disruption to the pretrained model's existing attention patterns.

Reviewer_vL8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GeoScaffold, a training-time geometric supervision framework for efficient vision-language navigation. The method introduces learnable geometry query tokens whose hidden states are supervised to reconstruct depth, connectivity, and traversability. A pretrained VQ-VAE provides discrete depth targets, while segmentation-based pseudo-labels supervise navigation-relevant geometric regions. All reconstruction modules are removed after training, allowing the policy to operate with RGB observations and language instructions alone. Experiments on R2R-CE and RxR-CE demonstrate consistent improvements across multiple VLM backbones, with additional evidence from real-world robotic deployment.

Critical strengths
  1. Practical and well-motivated design. The training-only geometric supervision strategy addresses an important limitation of geometry-enhanced navigation methods by avoiding additional perception modules at inference time.
  2. Strong empirical performance. GeoScaffold achieves competitive results on both R2R-CE and RxR-CE. More importantly, controlled comparisons demonstrate consistent success-rate improvements of 5.5–7.9 percentage points across three different VLM backbones.
  3. Promising real-world applicability. The experiments on a Unitree Go2 robot demonstrate improvements on geometry-demanding navigation instructions, and the reported on-device deployment supports the practical relevance of the framework.
Critical weaknesses
  1. Limited methodological novelty. The central approach combines established techniques, including auxiliary reconstruction objectives, VQ-VAE discretization, and learnable query tokens. Although their integration into streaming VLN is effective, the conceptual distinction from existing perception-supervised VLM/VLA methods, such as 3DRS and Spatial Forcing, remains relatively modest.
  2. Insufficient evidence for learned geometric representations. The paper interprets the query hidden states as compact geometric latents, but provides limited quantitative evaluation of their actual geometric accuracy. The reconstruction examples and attention ablations support the utility of these representations without establishing how faithfully they encode depth, connectivity, or traversability.
  3. Unclear robustness and statistical reliability. The paper does not report variation across training seeds or confidence intervals for the main benchmark results. Additionally, connectivity and traversability supervision relies on automatically generated segmentation masks, but their quality and robustness to annotation noise are not systematically evaluated.
Questions for authors
  1. Can the authors quantitatively evaluate the geometric information retained in the learned query states, for example through held-out depth reconstruction metrics, connectivity/traversability IoU, or frozen-feature linear probes? How do these representations compare with those learned by the baseline backbone?
  2. How much of the improvement comes specifically from the proposed query-token mechanism rather than auxiliary geometric supervision? Would a parameter-matched auxiliary prediction head operating on pooled visual features achieve comparable performance?
1 comment

Thank you for recognizing the practical motivation, controlled comparisons, and real-world evaluation.


Q1: What distinguishes GeoScaffold's reconstruction and query design from perception-supervised methods such as 3DRS and Spatial Forcing?

A1: Our contribution combines navigation-specific supervision with persistent geometric query memory. Streaming VLN spans rooms and corridors with larger viewpoint changes and wider depth variation. Later decisions depend on spatial cues that have left view.

  1. Historical memory. Each frame contributes eight geometry-supervised queries accessible to actions after its RGB frame leaves the visual window (Section 3.2 and Appendix C.3, pp. 18–19). We now report results for separately trained policies for three access scopes on R2R-CE Val-Unseen (Appendix A, Table 10, p. 15). Full-history access improves SR/SPL by 4.5/5.6 points over current-turn access and 1.6/3.1 over sliding-window access, supporting access to earlier geometry.
Accessible queriesNE ↓OS ↑SR ↑SPL ↑
Current-turn5.4257.250.845.4
Sliding-window5.1759.553.747.9
Full-history (Ours)5.0761.955.351.0
  1. Navigation targets. Depth supports distance judgment, connectivity identifies doors and corridor entrances, and traversability identifies walkable surfaces. The submitted target ablation shows that removing any target lowers SR from 55.3 to 51.1–53.1 and SPL from 51.0 to 47.1–49.2 (Table 3).

  2. VGGT alignment. Aligning final-layer image tokens with VGGT's final-layer features yielded only a modest SR gain. Retaining these dense states for direct access to geometric history would require many more cached tokens, so we use eight queries per frame.


Q2: Do queries encode geometry? Please provide reconstruction metrics and a baseline comparison.

A2: We now report reconstruction on 2,500 held-out frames from 11 R2R-CE Val-Unseen scenes. GeoScaffold uses its jointly trained depth head. The same-backbone baseline is frozen and uses a separately trained, parameter-matched depth head. Depth results and visualizations are now included in Appendix A, Table 12 and Figure 7 (p. 16). We also report both mask IoUs below.

Reconstruction sourceDepth δ₁ ↑ (%)Connectivity IoU ↑ (%)Traversability IoU ↑ (%)
Query states from RGB78.370.687.4
Frozen baseline features50.234.242.5
GT depth through VQ-VAE91.7––

Complementary attention analysis (Appendix D.1) covers 768 teacher-forced turns from training trajectories with identical inputs. Compared with the same-backbone policy trained without geometric supervision, GeoScaffold assigns more action-decoding attention to directional/geometric instruction words in 68% of turns and connective image regions in 74%, supporting increased attention to spatial cues.


Q3: Do gains come from queries or auxiliary supervision? Would a parameter-matched pooled-feature head work equally well?

A3: We clarify the existing pooled-feature ablation and summarize the controls.

  1. Pooled readout (Table 5). Its depth head reads per-frame mean-pooled final-layer image features, keeping targets, losses, and head parameter count unchanged. Dedicated queries provide separate geometric readout positions. Pooling may lose spatial detail, while depth supervision on image tokens may weaken their visual/language information.
Hidden-state sourceNE ↓OS ↑SR ↑SPL ↑
Pooled image states6.7850.643.238.5
Action states5.2358.451.847.9
Query tokens (Ours)5.0761.955.351.0
  1. Supervision and access (Tables 2, 5, 6). Unsupervised queries do not improve on the baseline. Blocking learned queries at test time reduces performance. Both geometric supervision and access to the states contribute.
ConfigurationSR ↑SPL ↑
Same backbone without GeoScaffold48.844.6
Queries without reconstruction47.743.3
Full GeoScaffold55.351.0
GeoScaffold, queries blocked at test52.847.7
  1. Real-world evidence (Appendix B, Figure 8, p. 17). Each Unitree Go2 environment has 20 geometry-demanding instructions, executed three times per policy. Gains exceed the 15.0-point SR gain on general instructions.
Geometry-demanding tasksBaseline SR (%)GeoScaffold SR (%)Gain (points)
Indoor51.771.7+20.0
Outdoor25.046.7+21.7

Q4: Are results reliable across seeds, and are generated masks accurate?

A4: All simulation results are averaged over three random seeds. We apologize for omitting this detail. We also report a manual audit of generated masks on 2,000 sampled images: 82% were judged correct with no errors identified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.