acceptodds
Withdrawn from ICLR 2027

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Embodied navigation requires agents to ground instructions or object goals in spatial observations and translate plans into successful execution. As multimodal large language models (MLLMs) become increasingly capable, they offer stronger support for navigation without task-specific training; however, improved semantic reasoning alone does not ensure that proposed actions remain consistent with spatial evidence, task progress, and execution outcomes. We introduce HarnessVLN, a zero-shot, training-free framework that unifies instruction-following and object-goal navigation through a shared Agent Harness. The Harness coordinates perception, memory, and execution tools through a unified interface, validating planner proposals for evidential support, geometric feasibility, and subgoal consistency before dispatch. It jointly manages hierarchical event memory and a persistent Spatiotemporal Graph to track task progress, preserve spatial evidence, and contextualize failures. Structured execution feedback updates this shared state, guiding subsequent planning, recovery, and termination. Across R2R, RxR, HM3D-v2, and HM3D-OVON, HarnessVLN achieves success rates of 59.6%, 51.4%, 76.0%, and 59.3%, respectively, outperforming prior training-free methods. Humanoid robot deployment further demonstrates its applicability to both navigation tasks in real-world environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agent, Vision-Language Navigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ah5nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a harness for inducing navigation behavior from large pretrained LLM/VLMs. They propose a graph based memory system and a specific set of tool calls to achieve this.

Critical strengths

Overall the results are competitive and on the relevant benchmarks. The reporeted ablations have are relevant. The inclusion of failure cases (Fig 10, 11) and the matched-model MSGNav comparison in Table 6 is informative. The quality of the writing is good.

Critical weaknesses

I see 3 main issues with this paper.

  1. Clarity: Many of the core mechanism do not seem to be clearly explained well. For example, argument validation, failure consistency etc. See questions below for explict sections which were unclear.
  2. Understanding relation to baselines. There's very little discussion in this paper about what separates this method from other Agentic Navigation frameworks. Those methods have their own form of memory updates and tooling which purport to give similar bennefits. This paper would bennefit from a more explicit discussion or ablation about why it's specific design choices are better than alternatives.
  3. Experimental reporting. In a setting like this where many baselines are competitive and margins are close, stastical rigor is important in the reporting. I recommend the methods from this paper [1] The gain over AgenticNav-GPT-5.5 on Rand100 is 3 SR, i.e. 3 episodes, from a single run, with a 12.6 drop in SPL. In Table 4, Graph adds 1 SR on OVON and Stop adds 2 SR. These are within the 95% confidence interval of a single 100-episode single-seed evaluation.

Also the results in table two seem to be improperly bolded. On R2R, HarnessVLN loses to AgenticNav and mini-swe-agent on Rand100 by quite a large margin. This somewhat undercuts strength of the results when SR is very close and SPL is apparently worse.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in Neural Information Processing Systems 34 (2021): 29304-29320.

Questions for authors

Questions:

"Before dispatching a tool call, the Harness validates its arguments, the provenance and recency of its supporting evidence, the geometric feasibility of the proposed target, and its consistency with the active subgoal." How is this done? How often are proposals rejected?

"Spatially compatible observations are merged into an existing place; otherwise, a new node is created." How is this done? Is it just the 0.75 m radius in Table 5? Does entity merging require a semantic match?

"where R, C, and A measure semantic relevance, recency, and spatial salience, respectively" Where do these functions come from? C's form is not given and A is never defined.

"A command is dispatched only if the target is supported by available evidence, geometrically reachable, and consistent with the active subgoal and applicable failure history." How is consistency with the failure history assessed? The appendix states penalties "are capped to avoid permanently excluding previously unsuccessful branches", which suggests a soft ranking term rather than a hard check.

"pending TODOs do not prevent a stop request." How is F_progress in Eq 4 enforced then?

"We request a sampling temperature of 0.7" What reasoning effort was used for GPT-5.5 and GPT-5.6-luna, and was this temperature actually applied?

3 comments
AuthorsOpenReview ↗

Response (1/2) — Harness Validation, Structured Memory, and Cross-Backbone Evaluation

We thank the reviewer for the insightful comments and provide our responses below; we have also revised the manuscript and uploaded an updated PDF with all changes highlighted in blue.

1. Diagnostics, Target Correction, and STOP Rejection (w1/q1)

Validation is not a uniform hard gate. Deterministic code checks tool registration, declared arguments, and side-effect labels, recording violations as diagnostics. Evidence IDs are episode-specific; recency mainly supports diagnostics, while cached STOP targets must be no more than 150 steps old. Depth projection checks traversability and adjusts invalid targets in 0.1 m increments. TODO rules permit one active subgoal and prevent rewriting completed items. Semantic consistency is assessed during ordinary planning and grounding; failure history affects retrieval ranking.

STOP verification combines an additional MLLM check of a fresh panorama with a geometric gate: 1.0 m for cached projected targets or 2.5 m for fresh bounding-box depth. It uses agent evidence without ground-truth goal access and returns rejection feedback to the planner.

GPT-5.5, Open-Nav R2R Rand100, 100 episodes, 500-step accounting window:

MeasureResult
Contract violations0
STOP rejections67/140 (47.9%); 32 episodes
Rejection breakdown42 MLLM vetoes; 25 geometric rejections among 98 MLLM approvals (25.5%)
Targets requiring pullback321/1,131 (28.4%); median distance 0.4 m
Target timeout abandonment487 events; 4.87/episode
Rejected TODO operations51 events; 0.51/episode

These mechanisms have separate counting denominators. Code: runtime, TODO, projection and STOP.

2. Unified Execution and Structured Memory (w2)

AgenticNav and Uni-LaViRA already provide navigation tools and memory. Our distinction is a configurable runtime coordinating planning, memory updates, execution feedback, and termination through shared interfaces across instruction-following navigation and ObjectNav. Persistent state remains outside individual model calls, and task-specific components can be replaced under the common protocol.

Event memory organizes task events and TODO progress; the graph links places, entities, observations, and failures. Local retrieval combines subgoal relevance, neighbor expansion, and bounded failure penalties, supporting evidence reuse and recovery without permanently excluding failed branches or adding an MLLM retrieval call.

Disabling Mem. or Graph retains the semantic map, visit records, and observation history. Table 4 therefore measures incremental benefits beyond these baseline capabilities: Mem. adds 7 percentage points of SR on both tasks, Graph adds 6 on R2R and 1 on OVON, and STOP validation adds 2 on both. The larger R2R graph gain is consistent with its need to connect route segments and subgoals. STOP validation helps confirm ambiguous ObjectNav targets but reduces OVON SPL by 1.2 percentage points, motivating task-specific configuration. These results support the evaluated designs, not superiority over every alternative. Code: map, navigation, graph, retrieval.

3. Additional Backbone Evidence and Statistical Limits (w3)

We added cumulative GPT-6-astra ablations on 100 episodes per dataset. SR improves by 10 percentage points on OVON and 12 on Open-Nav R2R. The OVON gain matches GPT-5.6-luna's improvement from 45% to 55% on the same subset.

ConfigurationOVON SROVON SPLR2R SRR2R SPLR2R nDTWR2R NE
Baseline53.036.465.047.055.34.19
+ Mem.61.038.770.048.856.94.00
+ Mem. + Graph60.042.376.055.563.83.23
Full (+ STOP validation)63.039.477.056.564.03.07

Overall benefits persist with the stronger model, although Graph trades a small OVON SR decrease for higher SPL. This supports cross-model applicability, not statistical significance. The experiments do not replace confidence intervals, repeated runs, or paired analysis, nor establish every component's necessity.

AuthorsOpenReview ↗

Response (2/2) — Path Efficiency, Graph Retrieval, STOP Verification, and Model Settings

4. Correcting Table Marking and Explaining SPL Costs (w3)

We acknowledge lower Rand100 SPL than AgenticNav and mini-swe-agent despite higher SR. We will bold the best value within each comparable setting and label our method separately.

We analyzed the full GPT-5.5 system's 58 successful trajectories. Distances are means:

Trajectory statisticResult
Actual path / shortest start-to-goal distance16.0 m / 8.4 m
Travel before first entering the 3 m success radius9.3 m
Travel after first entry6.7 m; about 42% of the path
Post-entry travel within the success radius5.7 m
Goal distance: first entry → final STOP2.9 m → 1.4 m

Substantial travel occurs after first reaching the success region. R2R scores the final position within a 3 m goal radius; runtime termination checks the grounded target against current observations. Continued approach can yield closer stops at greater path cost. These statistics do not fully explain the baseline SPL gap. Task-specific STOP settings can balance confirmation and efficiency.

5. Deterministic Rules for Place and Entity Merging (w1/q2)

Places merge with the nearest node within 0.75 m in horizontal world-coordinate plane, without semantic or heading matching; otherwise, a new node is created.

Entity rules, in order:

  1. Within 0.35 m, merge without requiring matching labels, tolerating description changes.
  2. Otherwise, within 1.0 m, require a category or alias match.
  3. If both observations lack coordinates, require the same place and a category or alias match.

Labels are trimmed and lowercased; no embeddings or extra MLLM calls are used. Code: association, thresholds.

6. Explicit Definitions of Local Retrieval Scores (w1/q3)

R, C, and A are hand-designed. Query and place text become lowercase token sets with stop words removed. For token w:

idf(w) = 1 + ln((1 + N) / (1 + df(w)))

N counts non-empty candidate place descriptions per episode; df(w) counts those containing w. Vectors assign idf(w) to present tokens and zero otherwise.

TermDefinition
RCosine similarity between the IDF-weighted query and place vectors, clipped to [0,1]
Cclip(t_last_seen / T, 0, 1); 1 when T <= 0
Amin(1, (associated entities + neighboring places) / 6)

T is the current graph turn; R is zero for empty token sets or zero denominators. A measures structural prominence, rather than image saliency or geometric proximity. Default score:

score = R + 0.3C + 0.3A - min(1, 0.5n)

n counts applicable failures for the current subgoal at the place. With no usable query or corpus text, retrieval falls back to recency and structural salience, retaining the penalty. No extra MLLM call is needed. Code: scoring.

7. Failure History Is Advisory Retrieval Guidance (w1/q4)

Failure history is a soft ranking term, not a pre-dispatch hard gate. The system counts still-applicable failures for the place and current subgoal, matched by trimmed, lowercased strings, and applies min(1, 0.5n). Records provide planning context without prohibiting actions and can be marked inapplicable when subgoals are completed or removed. Code: penalty, filtering and expiration.

8. Progress Informs STOP Proposals Before Verification (w1/q5)

The TODO ledger tracks a revisable plan. Pending items may become unnecessary once the goal is reached and do not automatically prevent STOP. Progress informs the ordinary decision model; final verification does not independently recheck the ledger or historical trajectory.

Eq. (4) describes the STOP decision chain:

Execute STOP = FprogressF_{\mathrm{progress}} ∧ FsemanticF_{\mathrm{semantic}} ∧ FgeometricF_{\mathrm{geometric}}.

The progress term is the model's STOP proposal, not a deterministic completion certificate. It is followed by semantic verification using fresh observations and a deterministic geometric gate, without an additional progress verifier. Code: planning and verification.

9. Requested Reasoning and Sampling Settings (q6)

The implementation defaults to reasoning_effort='low' for GPT-5.5 and GPT-5.6-luna. Planning, recovery, and STOP verification request temperature=0.7; grounding requests temperature=0. Unsupported reasoning-effort parameters may be removed during retries. Code establishes requested settings, not whether the provider applied temperature; we will distinguish requested and effective settings using request/retry logs. Code: API handling.

Dear Reviewer ah5n,

Thank you again for your thoughtful feedback. We would like to briefly summarize our responses to your main concerns:

  • Mechanism clarity: We clarified which validation steps provide diagnostics and which enforce execution constraints, along with the graph merging rules, retrieval scores, and STOP decision process. We also clarified that failure history guides retrieval through soft penalties and distinguished requested API settings from settings confirmed by provider logs.
  • Relation to existing frameworks: We explained the role of our configurable runtime and structured memory across both navigation tasks. We clarified what remains available when memory or graph components are disabled, so the ablations measure their incremental benefits beyond the baseline map and history.
  • Experimental evidence and reporting: We added GPT-6-astra comparisons and ablations, showing overall SR gains of 12 percentage points on R2R and 10 on OVON. We explicitly acknowledged that these results do not establish statistical significance or replace repeated evaluations. We also analyzed the SPL costs and acknowledged the table highlighting issue, specifying how it should be corrected.

When you have a chance, we would appreciate your thoughts on whether these clarifications and additional results resolve your concerns and warrant reconsidering your current assessment. If you have any remaining questions or concerns, please feel free to share them. We will do our best to address them.

Thank you for your time and consideration.

Reviewer_bboLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a training-free framework for instruction-following and object-goal navigation. A pretrained multimodal model proposes actions, while an external harness maintains task progress, execution events, and a persistent spatiotemporal graph. Actions and stopping requests are checked against spatial evidence, geometry, and task state. The contribution extends beyond prompt modification: it changes the information available to the model, the conditions under which actions are executed, and how execution feedback is retained.

Critical strengths
  1. Good system design. The framework addresses practical navigation failures through complementary mechanisms: event memory tracks progress and execution feedback, the ST Graph maintains spatial evidence, and stopping validation checks completion requests. These mechanisms form a coherent navigation framework beyond prompt modification alone.
  2. Improvements in matched-backbone ObjectNav comparisons. On the fixed 100-episode subsets in Appendix Table 6, HarnessVLN improves success over MSGNav from 62% to 76% on HM3D-v2 and from 37% to 55% on OVON using the same backbone, alongside improvements in SPL. These results provide useful evidence for the effectiveness of the harness.
  3. Useful evaluation and supporting analysis. The paper evaluates both instruction-following and object-goal navigation, reports full-benchmark results, and includes cumulative component ablations and additional backbone experiments. The qualitative examples cover both successful behavior and failure cases, including a premature stopping request that is rejected before the agent eventually succeeds.
Critical weaknesses
  1. Incremental gains and efficiency trade-offs. While the matched-backbone ObjectNav improvements are encouraging, the R2R comparison shows a modest SR increase from 55% to 58%, accompanied by a decrease in SPL from 48.4 to 35.8. The component gains also vary: the ST Graph adds six percentage points on R2R but only one on OVON, and stopping validation adds two points on each subset. On OVON, adding stopping validation also slightly reduces SPL, from 34.2 to 33.0. Since these comparisons use 100-episode subsets, further analysis of the cases helped or hindered by these mechanisms would clarify the practical significance of the gains and the trade-off between successful completion and navigation efficiency.

  2. The benefit of the harness as backbone capability changes is not established. Appendix Table 6 shows that the complete framework operates with different models, but does not provide corresponding baseline-versus-harness comparisons for the additional backbones. It therefore remains unclear whether stronger base models retain comparable benefits from the proposed components or reduce the need for them. This limits the conclusions that can be drawn about the framework’s value across models.

Questions for authors
  1. Why are different backbones used for the two navigation settings—GPT-5.5 for instruction-following navigation and GPT-5.6-luna for ObjectNav? What motivates this choice, and would using the same backbone across both settings affect the conclusions?
  2. As the base model becomes more capable, do the proposed harness components remain necessary and provide comparable gains? Appendix Table 6 evaluates the complete framework with different backbones, but does not show whether a stronger model reduces the benefit of event memory, the ST Graph, or stopping validation. A baseline-versus-full-harness comparison, together with selected component ablations on a stronger backbone, would help clarify this.
  3. Section 4.2 mentions invoking NavDP for stair traversal. How does the ST Graph represent connections between floors and support planning across them? How many evaluation episodes require moving between floors, and can the authors report performance on this subset separately?
3 comments
AuthorsOpenReview ↗

Response (1/2) — Navigation Harness Trade-offs Across Tasks and Backbones

We thank the reviewer for the insightful comments and provide our responses below; we have also revised the manuscript and uploaded an updated PDF with all changes highlighted in blue.

1. Task-Dependent Gains and the Cost of Goal Confirmation (w1)

Thank you for this comment. We first acknowledge the observed trade-off between higher SR and lower SPL. We then address it from two perspectives, drawing on our trajectory analysis and the design goals of the unified navigation harness.

Sources of the efficiency cost. We inspected the 58 successful trajectories of the full GPT-5.5 configuration:

Trajectory statisticResult
Mean actual path length / shortest start-to-goal distance16.0 m / 8.4 m
Travel before first entering the 3 m success radius9.3 m
Travel after first entering the success radius6.7 m, approximately 42% of the total path
Post-entry travel that remains within the success radius5.7 m
Distance to goal: first entry → final STOP2.9 m → 1.4 m
Diagnostic SPL after truncating successful trajectories at first entry51

These statistics suggest that continued approach, target reassignment, and repeated verification near the goal are major sources of the SPL loss. Even after satisfying the R2R distance criterion, the agent may continue searching for or aligning with the specific stopping location described in the instruction. This reflects a mismatch between runtime termination decisions and the evaluation criterion. R2R defines success by whether the final position is within 3 m of the goal, whereas our system verifies termination using current observations and the grounded semantic target. This can lead the agent to move further and stop closer to the goal, reducing path efficiency.

Unified harness and task adaptation. Our contribution is a unified harness for multiple navigation tasks. It organizes planning, memory, execution feedback, and termination verification through shared tool interfaces, while allowing components to be configured or replaced for each task. For tasks such as ObjectNav, where target categories or language descriptions may be ambiguous, more precise termination decisions are useful. STOP verification checks whether the observed object satisfies the target requirements, helping prevent premature termination based solely on proximity to a location. This additional verification can, however, reduce SPL.

2. Overall Harness Benefits Persist on a Stronger Backbone (w2/q2)

We added baseline-versus-full-harness comparisons across backbone capabilities and cumulative component ablations on GPT-6-astra.

On the same HM3D-OVON subset, GPT-5.6-luna improves from 45.0% to 55.0% SR, while GPT-6-astra improves from 53.0% to 63.0%—a 10-percentage-point gain for both. On Open-Nav R2R, GPT-6-astra improves from 65.0% to 77.0%, a 12-percentage-point gain.

The GPT-6-astra ablations below use 100 episodes per dataset.

ConfigurationOVON SR ↑OVON SPL ↑R2R SR ↑R2R SPL ↑R2R nDTW ↑R2R NE ↓
Baseline53.036.465.047.055.34.19
+ Mem.61.038.770.048.856.94.00
+ Mem. + Graph60.042.376.055.563.83.23
Full (+ STOP validation)63.039.477.056.564.03.07

The full harness retains substantial gains with the stronger backbone. Event memory and STOP validation increase SR on both datasets. The graph improves SR and SPL on R2R, while on OVON it improves SPL with a slight SR decrease. These results support continued, task-dependent benefits, but the cumulative ablation does not establish that every component is strictly necessary or that all metrics improve monotonically.

3. Backbone Selection Aligns Baselines Rather Than Reflecting Task Requirements (q1)

We use GPT-5.5 for instruction-following navigation and GPT-5.6-luna for ObjectNav primarily to match existing comparison settings and reduce backbone differences in method comparisons. This choice does not imply that the two tasks require different models. The harness protocol and tool organization remain unchanged.

Appendix Table 6 reports the complete framework’s ObjectNav performance with three backbones:

BackboneHM3D-v2 SR (%) ↑HM3D-OVON SR (%) ↑
GPT-5.6-luna76.055.0
GPT-6-astra86.063.0
Qwen3.8-flash72.051.0

These results demonstrate applicability across backbones while showing that absolute performance depends on model choice. The additional experiments above use GPT-6-astra across both R2R and ObjectNav and retain SR gains of 12 and 10 percentage points, respectively. Thus, using a common backbone can preserve the overall harness benefit, although component gains need not be identical across models or tasks.

AuthorsOpenReview ↗

Response (2/2)— Cross-Floor Navigation Design and Evaluation

4. Cross-Floor Navigation Uses Visual Planning and NavDP Execution (q3)

The ST Graph currently uses a 2D topology and does not provide explicit multi-floor graph search. Cross-floor navigation instead relies on coordination between the visual planner and navigation executor.

At each decision, the planner outputs stair ∈ {false, "up", "down"} alongside the action. The prompt instructs it to activate this flag when traversing stairs between floors, and grounding identifies the stair entrance. In the hybrid-controller configuration, an active flag switches execution from FMM to NavDP. NavDP generates trajectories from RGB-D observations and an agent-frame point goal, which are converted into discrete turning and forward actions. Execution returns to FMM when the flag is cleared.

For VLN, we classify episodes with an absolute start-to-goal height difference exceeding 1 m as cross-floor. For ObjectNav, an episode is classified as requiring a floor change when no valid target viewpoint exists on the starting floor.

We additionally report cross-floor results on 100-episode R2R and RxR subsets. The R2R results are obtained by offline truncation and re-scoring of the GPT-5.5 full-system trajectories.

Benchmark / backboneCross-floor / total episodesCross-floor SR / SPL / NESame-floor SR / SPL / NE
R2R — GPT-5.56 / 10066.7 / 47.1 / 3.3557.4 / 35.1 / 4.30
RxR — GPT-5.525 / 10048.0 / 31.7 / 5.5456.0 / 37.6 / 5.92

On R2R, four of six cross-floor episodes succeed, yielding 66.7% SR and 47.1% SPL. The overall results are 58.0% SR and 35.8% SPL, consistent with the main comparison. On RxR, cross-floor SR is 48.0%, compared with 56.0% for same-floor episodes, and the stair flag covers 20/25 cross-floor episodes (80.0%).

Under the stated ObjectNav criterion, none of the 1,000 HM3D-v2 or 3,000 HM3D-OVON validation episodes requires a floor change, since each has a valid target viewpoint on its starting floor.

Dear Reviewer bboL,

Thank you again for your thoughtful feedback. We would like to briefly summarize our responses to your main concerns:

  • Performance gains and efficiency: We analyzed the 58 successful R2R trajectories and explained how continued approach, target reassignment, and repeated goal verification contribute to higher SR but lower SPL. We also clarified why component gains differ across tasks and acknowledged the efficiency costs of additional goal confirmation.
  • Backbone selection and harness benefits: We clarified that the original backbone choices align with existing baseline settings. We added baseline-versus-full-harness comparisons and component ablations using GPT-6-astra across both tasks. The full harness improves SR by 12 percentage points on R2R and 10 points on HM3D-OVON, supporting continued benefits with a stronger backbone while showing that individual component gains remain task dependent.
  • Cross-floor navigation: We clarified that the ST Graph currently uses a 2D topology, while stair traversal relies on visual planning and NavDP execution. We also defined the cross-floor evaluation criteria and reported separate results on R2R and RxR subsets.

When you have a chance, we would appreciate your thoughts on whether these clarifications and additional results resolve your concerns and warrant reconsidering your current assessment. If you have any remaining questions or concerns, please feel free to share them. We will do our best to address them.

Thank you for your time and consideration.

Reviewer_3ZoWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HarnessVLN, a training-free framework for instruction-following and object-goal navigation. A shared agent harness coordinates a multimodal language model with perception, retrieval, grounding, navigation, and termination tools. Its main components are hierarchical event memory, a persistent spatiotemporal graph, and validation of proposed actions and stopping requests against spatial evidence and task progress.

Critical strengths
  1. Explicitly tracking task progress, observation provenance, and execution failures is a sensible way to reduce inconsistencies in long-horizon navigation. Treating stopping as a request that requires additional verification is particularly well motivated. The shared implementation across two navigation task families gives the system practical relevance.
  2. The paper reports results on the full evaluation splits of four benchmarks. The inclusion of failure cases and explicit discussion of the distinction between internal stop validation and benchmark success also improves the transparency of the evaluation.
Critical weaknesses
  1. Contribution. The contribution beyond closely related agentic navigation systems is not sufficiently isolated. The paper acknowledges that AgenticNav already exposes navigation capabilities through callable tools, while Uni-LaViRA combines unified navigation with progress memory and recovery. The proposed distinction therefore rests substantially on the organization of event memory, the persistent graph, and evidence-based validation. However, the experiments do not establish which of these design choices is necessary relative to simpler alternatives or the closest existing frameworks. The cumulative ablation in Table 4 is informative but does not isolate component interactions. On the 100-episode ObjectNav subset, adding the graph and stop validation produces gains of only a little improvement, respectively, without reported variability. This limits the evidence supporting the central methodological contribution.
  2. Efficiency. In Table 2, the GPT-5.5 comparison on R2R improves SR from 55.0 to 58.0 relative to AgenticNav, but reduces SPL from 48.4 to 35.8. This is a interesting trade-off. The paper also lacks episode-level inference cost and latency measurements, which matter for a framework involving repeated observation, retrieval, and verification. Uni-LaViRA is compared only on the R2R subset, despite its relevance to both task families. The extent of the framework’s advantage remains uncertain on computation tradeoff without clearer protocol alignment and broader comparison with this close baseline.
Questions for authors
  1. Clarification: What are the concrete implementations of proposal validation and the progress term in Eq (4)? Which checks are deterministic, and which require another MLLM judgment? In particular, how does the final progress check handle the Appendix 2.1 prompt that permits a stop request while TODO items remain pending?
  2. Clarification: For the matched-backbone R2R comparisons, are the episode IDs, observation access, low-level execution settings, and action budgets identical? How do the authors explain the substantial SPL decrease relative to AgenticNav?
  3. Additional evidence: Can the authors provide variability estimates or repeated evaluations for the component ablations, especially the small ObjectNav gains? A comparison between the full system and a version with each principal component individually removed would help establish which mechanisms drive the improvement.
  4. Additional evidence: Is a comparison with Uni-LaViRA available under matched backbones and evaluation conditions on ObjectNav or RxR? Please also report average model calls and execution time per episode, if available, so that the practical cost of the reported gains can be assessed.
  5. GOAT-Bench's instructions were generated by vision-language models, which may introduce ambiguous or inaccurate descriptions. A human-verified benchmark for this task, LangMap [1], is already publicly available. It would be beneficial to include results on this benchmark, along with the evaluation settings, to further strengthen the empirical validation of the proposed method. However, given the limited rebuttal period, I understand if conducting these additional experiments is not feasible.

[1] LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation. arXiv, 2026.

3 comments
AuthorsOpenReview ↗

Response (1/2) — Navigation Harness Design and Evaluation

We thank the reviewer for the insightful comments and provide our responses below; we have also revised the manuscript and uploaded an updated PDF with all changes highlighted in blue.

1. Methodological Contribution and Ablation Scope (w1/q3)

Ablation scope. “Mem.” and “Graph” toggle our hierarchical event memory and persistent spatiotemporal graph, respectively. With either component disabled, the baseline retains its semantic map, visit records, and observation history. AgenticNav and Uni-LaViRA likewise retain navigation experience through maps or history. With other settings fixed, Table 4 isolates the gains from adding these components to the baseline. We will clarify what each switch disables and what the baseline retains. Semantic map code · Navigation and retrieval code

Contribution. The framework separates the decision model from the harness-managed runtime, which exposes navigation tools and supports editable, updatable memory. Event memory organizes task events and execution feedback. The persistent graph links places, entities, observations, and failures for later retrieval.

Task-dependent gains. The graph improves SR by 6 percentage points on R2R and 1 point on HM3D-OVON. R2R requires linking route segments, landmarks, and subgoals, giving the agent more opportunities to reuse graph evidence. In ObjectNav, the baseline map and history already provide substantial context for reaching a single target. Event memory improves SR by 7 percentage points on both tasks, suggesting that both benefit from access to past events and execution feedback.

2. R2R Success Gains and Path Efficiency Costs (w2/q2)

We use Open-Nav’s public R2R-CE Rand100 subset, as do Open-Nav and AgenticNav, and evaluate Uni-LaViRA on the same episode IDs. We keep observation access, low-level execution settings, action budgets, and model selection consistent across comparisons.

We analyzed all 58 successful trajectories from the full GPT-5.5 configuration under a 500-step budget.

Trajectory statisticResult
Mean actual path length / shortest start-to-goal distance16.0 m / 8.4 m
Mean travel before first entering the 3 m success radius9.3 m
Mean travel after first entry6.7 m (~42% of total path length)
Mean post-entry travel within the radius5.7 m
Mean distance to goal at first entry → final STOP2.9 m → 1.4 m
Diagnostic SPL with successful paths truncated at first entry51

The substantial post-entry travel helps explain why SR rises while SPL falls. R2R scores success by whether the final position is within 3 m of the goal. Our runtime verifies the grounded target against current observations, so the agent may continue approaching, reassigning the target, or checking the instruction’s stopping location after first entering that radius.

The harness shares tool interfaces for planning, memory, execution feedback, and STOP verification across tasks, with components configurable or replaceable by task. In ObjectNav, where category labels or descriptions can be ambiguous, verifying that the observed object matches the request helps prevent premature stops based on proximity alone. Thus, this verification can add travel and lower SPL.

3. Progress Assessment and STOP Verification (q1)

CheckImplementationExtra MLLM call?
Tool-call contractsValidate registration, arguments, and side-effect labelsNo
TODOs and evidenceValidate TODO transitions and cited evidence IDsNo for validation itself
Navigation geometryProject depth, check traversability, adjust targetsNo
Subgoal consistencyUse task, TODOs, and observation contextNo
FprogressF_{\mathrm{progress}}Propose STOP from progress and observationsNo
FsemanticF_{\mathrm{semantic}}Recheck the proposed STOP against fresh views and the taskYes
FgeometricF_{\mathrm{geometric}}Check distance from depth or a valid cached targetNo

The TODO ledger tracks a revisable plan and supports recovery. Pending TODOs may become unnecessary once the goal is reached and can remain open when the model proposes STOP. Final verification relies on visual and geometric evidence and skips a separate review of the ledger and trajectory history.

We will clarify that Eq. (4) describes the STOP decision chain: the model proposes STOP based on progress, followed by semantic verification by the model and deterministic geometric validation by the runtime.

Execute STOP = FprogressF_{\mathrm{progress}} ∧ FsemanticF_{\mathrm{semantic}} ∧ FgeometricF_{\mathrm{geometric}}.

Here, progress is assessed by the decision model, semantic verification uses fresh observations, and geometric verification is performed by the runtime.

AuthorsOpenReview ↗

Response (2/2) — Runtime Costs and Additional Benchmark Results

4. Limited Call Growth with Higher Context and Runtime Costs (w2/q4)

Thank you for this suggestion. We report per-episode costs for the cumulative ablations on a fixed 100-episode subset of each dataset, using a 500-step budget. All costs are means; SR is reported as a percentage.

DatasetConfigurationSR ↑Calls/episodeInput / output tokensTime (s)
R2RBaseline43.030.5151k / 11.7k1209
R2R+ Mem.50.025.5186k / 9.8k1155
R2R+ Mem. + Graph56.028.0218k / 10.6k1253
R2RFull (+ STOP)58.030.8248k / 12.0k1462
HM3D-OVONBaseline45.019.864k / 5.6k422
HM3D-OVON+ Mem.52.019.9137k / 6.6k880
HM3D-OVON+ Mem. + Graph53.021.0147k / 6.8k860
HM3D-OVONFull (+ STOP)55.021.5157k / 7.2k966

The full system improves SR by 15 percentage points on R2R and 10 percentage points on HM3D-OVON, while model calls increase by only 1.0% and 8.6%, respectively. Most additional token consumption comes from input context, which grows to 1.64× and 2.45× the baseline. Mean execution time increases by approximately 21% on R2R and reaches 2.29× the baseline on HM3D-OVON. These results show improved task completion with limited call growth, but substantial context and runtime costs.

5. ObjectNav Comparison with Uni-LaViRA under Matched Models and Episodes (w2/q4)

Thank you for this suggestion. We additionally evaluated Uni-LaViRA on HM3D-v2 and HM3D-OVON using the same GPT-5.6-luna model and the same 100 episode IDs per dataset as HarnessVLN. Together with the GPT-5.5 comparison on the shared R2R Rand100 subset, the results are:

DatasetModelMethodSR ↑SPL ↑
R2R Rand100GPT-5.5Uni-LaViRA42.025.7
R2R Rand100GPT-5.5HarnessVLN58.035.8
HM3D-v2GPT-5.6-lunaUni-LaViRA58.018.4
HM3D-v2GPT-5.6-lunaHarnessVLN76.040.8
HM3D-OVONGPT-5.6-lunaUni-LaViRA39.018.2
HM3D-OVONGPT-5.6-lunaHarnessVLN55.033.0

Compared with Uni-LaViRA, HarnessVLN improves SR by 16.0, 18.0, and 16.0 percentage points and SPL by 10.1, 22.4, and 14.8 percentage points on R2R, HM3D-v2, and HM3D-OVON, respectively. These results extend the direct comparison with this closely related baseline to ObjectNav and support gains in task completion and success-weighted path efficiency under matched models and evaluation episodes.

6. LangMap Evaluation with Human-Verified Language Goals (q5)

Thank you for this suggestion. We added an evaluation on a LangMap single-goal subset spanning four levels of language-specified goals.

Following the official validity filters, we sampled 200 episodes from 9,314 episodes across 36 HM3D v0.2 validation scenes: 50 each for Scene, Room, Region, and Instance goals. Using seed 42, we sampled scenes in shuffled round-robin order within each level, covering all 36 scenes and limiting concentration in larger scenes. We extracted public PlaNaVid results for the same episode IDs and evaluated Uni-LaViRA, MSGNav, and HarnessVLN using GPT-5.6-luna.

Overall and goal-level results (SR / SPL, %)

MethodOverallSceneRoomRegionInstance
PlaNaVid-3B25.5 / 12.824.0 / 15.326.0 / 10.428.0 / 13.624.0 / 12.1
PlaNaVid-7B24.5 / 10.932.0 / 16.238.0 / 15.114.0 / 5.014.0 / 7.2
Uni-LaViRA17.5 / 7.820.0 / 8.924.0 / 10.516.0 / 6.810.0 / 5.1
MSGNav25.0 / 10.024.0 / 11.224.0 / 6.632.0 / 13.220.0 / 9.0
HarnessVLN40.5 / 24.242.0 / 25.740.0 / 23.640.0 / 25.040.0 / 22.4
PlaNaVid-7B (full-dataset reference)31.7 / 15.435.8 / 17.234.7 / 16.431.1 / 15.327.8 / 13.7

On the shared subset, HarnessVLN improves SR / SPL by 15.0 / 11.4 percentage points over PlaNaVid-3B, 23.0 / 16.4 points over Uni-LaViRA, and 15.5 / 14.2 points over MSGNav. It achieves the highest SR and SPL at all four goal levels, with SR ranging from 40.0% to 42.0%, and has lower measured latency than the two other GPT-5.6-luna methods (634 vs. 807 and 726 s/goal). These results support its applicability to LangMap’s human-verified hierarchical language goals on the evaluated subset.

Dear Reviewer 3ZoW,

Thank you again for your thoughtful feedback. We would like to briefly summarize our responses to your main concerns:

  • Contribution and validation: We clarified what the memory and graph ablations disable, what the baseline retains, and why their gains differ across tasks. We also explained the STOP decision process, distinguishing model-based progress and semantic judgments from deterministic geometric checks.
  • Comparison fairness and efficiency: We clarified the matched R2R evaluation settings and analyzed how continued verification after entering the goal radius contributes to higher SR but lower SPL. We also reported per-episode model calls, token usage, and execution time, acknowledging the additional costs.
  • Broader empirical evidence: We added ObjectNav comparisons with Uni-LaViRA using the same models and episodes, along with a 200-episode evaluation on the human-verified LangMap benchmark. These results show improvements in both SR and SPL over the evaluated baselines.

When you have a chance, we would appreciate your thoughts on whether these clarifications and additional results resolve your concerns and warrant reconsidering your current assessment. If you have any remaining questions or concerns, please feel free to share them. We will do our best to address them.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.