- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a harness for inducing navigation behavior from large pretrained LLM/VLMs. They propose a graph based memory system and a specific set of tool calls to achieve this.
Critical strengths
Overall the results are competitive and on the relevant benchmarks. The reporeted ablations have are relevant. The inclusion of failure cases (Fig 10, 11) and the matched-model MSGNav comparison in Table 6 is informative. The quality of the writing is good.
Critical weaknesses
I see 3 main issues with this paper.
- Clarity: Many of the core mechanism do not seem to be clearly explained well. For example, argument validation, failure consistency etc. See questions below for explict sections which were unclear.
- Understanding relation to baselines. There's very little discussion in this paper about what separates this method from other Agentic Navigation frameworks. Those methods have their own form of memory updates and tooling which purport to give similar bennefits. This paper would bennefit from a more explicit discussion or ablation about why it's specific design choices are better than alternatives.
- Experimental reporting. In a setting like this where many baselines are competitive and margins are close, stastical rigor is important in the reporting. I recommend the methods from this paper [1] The gain over AgenticNav-GPT-5.5 on Rand100 is 3 SR, i.e. 3 episodes, from a single run, with a 12.6 drop in SPL. In Table 4, Graph adds 1 SR on OVON and Stop adds 2 SR. These are within the 95% confidence interval of a single 100-episode single-seed evaluation.
Also the results in table two seem to be improperly bolded. On R2R, HarnessVLN loses to AgenticNav and mini-swe-agent on Rand100 by quite a large margin. This somewhat undercuts strength of the results when SR is very close and SPL is apparently worse.
[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in Neural Information Processing Systems 34 (2021): 29304-29320.
Questions for authors
Questions:
"Before dispatching a tool call, the Harness validates its arguments, the provenance and recency of its supporting evidence, the geometric feasibility of the proposed target, and its consistency with the active subgoal." How is this done? How often are proposals rejected?
"Spatially compatible observations are merged into an existing place; otherwise, a new node is created." How is this done? Is it just the 0.75 m radius in Table 5? Does entity merging require a semantic match?
"where R, C, and A measure semantic relevance, recency, and spatial salience, respectively" Where do these functions come from? C's form is not given and A is never defined.
"A command is dispatched only if the target is supported by available evidence, geometrically reachable, and consistent with the active subgoal and applicable failure history." How is consistency with the failure history assessed? The appendix states penalties "are capped to avoid permanently excluding previously unsuccessful branches", which suggests a soft ranking term rather than a hard check.
"pending TODOs do not prevent a stop request." How is F_progress in Eq 4 enforced then?
"We request a sampling temperature of 0.7" What reasoning effort was used for GPT-5.5 and GPT-5.6-luna, and was this temperature actually applied?
3 comments
Response (2/2) — Path Efficiency, Graph Retrieval, STOP Verification, and Model Settings
4. Correcting Table Marking and Explaining SPL Costs (w3)
We acknowledge lower Rand100 SPL than AgenticNav and mini-swe-agent despite higher SR. We will bold the best value within each comparable setting and label our method separately.
We analyzed the full GPT-5.5 system's 58 successful trajectories. Distances are means:
| Trajectory statistic | Result |
|---|---|
| Actual path / shortest start-to-goal distance | 16.0 m / 8.4 m |
| Travel before first entering the 3 m success radius | 9.3 m |
| Travel after first entry | 6.7 m; about 42% of the path |
| Post-entry travel within the success radius | 5.7 m |
| Goal distance: first entry → final STOP | 2.9 m → 1.4 m |
Substantial travel occurs after first reaching the success region. R2R scores the final position within a 3 m goal radius; runtime termination checks the grounded target against current observations. Continued approach can yield closer stops at greater path cost. These statistics do not fully explain the baseline SPL gap. Task-specific STOP settings can balance confirmation and efficiency.
5. Deterministic Rules for Place and Entity Merging (w1/q2)
Places merge with the nearest node within 0.75 m in horizontal world-coordinate plane, without semantic or heading matching; otherwise, a new node is created.
Entity rules, in order:
- Within 0.35 m, merge without requiring matching labels, tolerating description changes.
- Otherwise, within 1.0 m, require a category or alias match.
- If both observations lack coordinates, require the same place and a category or alias match.
Labels are trimmed and lowercased; no embeddings or extra MLLM calls are used. Code: association, thresholds.
6. Explicit Definitions of Local Retrieval Scores (w1/q3)
R, C, and A are hand-designed. Query and place text become lowercase token sets with stop words removed. For token w:
idf(w) = 1 + ln((1 + N) / (1 + df(w)))
N counts non-empty candidate place descriptions per episode; df(w) counts those containing w. Vectors assign idf(w) to present tokens and zero otherwise.
| Term | Definition |
|---|---|
| R | Cosine similarity between the IDF-weighted query and place vectors, clipped to [0,1] |
| C | clip(t_last_seen / T, 0, 1); 1 when T <= 0 |
| A | min(1, (associated entities + neighboring places) / 6) |
T is the current graph turn; R is zero for empty token sets or zero denominators. A measures structural prominence, rather than image saliency or geometric proximity. Default score:
score = R + 0.3C + 0.3A - min(1, 0.5n)
n counts applicable failures for the current subgoal at the place. With no usable query or corpus text, retrieval falls back to recency and structural salience, retaining the penalty. No extra MLLM call is needed. Code: scoring.
7. Failure History Is Advisory Retrieval Guidance (w1/q4)
Failure history is a soft ranking term, not a pre-dispatch hard gate. The system counts still-applicable failures for the place and current subgoal, matched by trimmed, lowercased strings, and applies min(1, 0.5n). Records provide planning context without prohibiting actions and can be marked inapplicable when subgoals are completed or removed. Code: penalty, filtering and expiration.
8. Progress Informs STOP Proposals Before Verification (w1/q5)
The TODO ledger tracks a revisable plan. Pending items may become unnecessary once the goal is reached and do not automatically prevent STOP. Progress informs the ordinary decision model; final verification does not independently recheck the ledger or historical trajectory.
Eq. (4) describes the STOP decision chain:
Execute STOP = ∧ ∧ .
The progress term is the model's STOP proposal, not a deterministic completion certificate. It is followed by semantic verification using fresh observations and a deterministic geometric gate, without an additional progress verifier. Code: planning and verification.
9. Requested Reasoning and Sampling Settings (q6)
The implementation defaults to reasoning_effort='low' for GPT-5.5 and GPT-5.6-luna. Planning, recovery, and STOP verification request temperature=0.7; grounding requests temperature=0. Unsupported reasoning-effort parameters may be removed during retries. Code establishes requested settings, not whether the provider applied temperature; we will distinguish requested and effective settings using request/retry logs. Code: API handling.
Dear Reviewer ah5n,
Thank you again for your thoughtful feedback. We would like to briefly summarize our responses to your main concerns:
- Mechanism clarity: We clarified which validation steps provide diagnostics and which enforce execution constraints, along with the graph merging rules, retrieval scores, and STOP decision process. We also clarified that failure history guides retrieval through soft penalties and distinguished requested API settings from settings confirmed by provider logs.
- Relation to existing frameworks: We explained the role of our configurable runtime and structured memory across both navigation tasks. We clarified what remains available when memory or graph components are disabled, so the ablations measure their incremental benefits beyond the baseline map and history.
- Experimental evidence and reporting: We added GPT-6-astra comparisons and ablations, showing overall SR gains of 12 percentage points on R2R and 10 on OVON. We explicitly acknowledged that these results do not establish statistical significance or replace repeated evaluations. We also analyzed the SPL costs and acknowledged the table highlighting issue, specifying how it should be corrected.
When you have a chance, we would appreciate your thoughts on whether these clarifications and additional results resolve your concerns and warrant reconsidering your current assessment. If you have any remaining questions or concerns, please feel free to share them. We will do our best to address them.
Thank you for your time and consideration.
Response (1/2) — Harness Validation, Structured Memory, and Cross-Backbone Evaluation
We thank the reviewer for the insightful comments and provide our responses below; we have also revised the manuscript and uploaded an updated PDF with all changes highlighted in blue.
1. Diagnostics, Target Correction, and STOP Rejection (w1/q1)
Validation is not a uniform hard gate. Deterministic code checks tool registration, declared arguments, and side-effect labels, recording violations as diagnostics. Evidence IDs are episode-specific; recency mainly supports diagnostics, while cached STOP targets must be no more than 150 steps old. Depth projection checks traversability and adjusts invalid targets in 0.1 m increments. TODO rules permit one active subgoal and prevent rewriting completed items. Semantic consistency is assessed during ordinary planning and grounding; failure history affects retrieval ranking.
STOP verification combines an additional MLLM check of a fresh panorama with a geometric gate: 1.0 m for cached projected targets or 2.5 m for fresh bounding-box depth. It uses agent evidence without ground-truth goal access and returns rejection feedback to the planner.
GPT-5.5, Open-Nav R2R Rand100, 100 episodes, 500-step accounting window:
These mechanisms have separate counting denominators. Code: runtime, TODO, projection and STOP.
2. Unified Execution and Structured Memory (w2)
AgenticNav and Uni-LaViRA already provide navigation tools and memory. Our distinction is a configurable runtime coordinating planning, memory updates, execution feedback, and termination through shared interfaces across instruction-following navigation and ObjectNav. Persistent state remains outside individual model calls, and task-specific components can be replaced under the common protocol.
Event memory organizes task events and TODO progress; the graph links places, entities, observations, and failures. Local retrieval combines subgoal relevance, neighbor expansion, and bounded failure penalties, supporting evidence reuse and recovery without permanently excluding failed branches or adding an MLLM retrieval call.
Disabling Mem. or Graph retains the semantic map, visit records, and observation history. Table 4 therefore measures incremental benefits beyond these baseline capabilities: Mem. adds 7 percentage points of SR on both tasks, Graph adds 6 on R2R and 1 on OVON, and STOP validation adds 2 on both. The larger R2R graph gain is consistent with its need to connect route segments and subgoals. STOP validation helps confirm ambiguous ObjectNav targets but reduces OVON SPL by 1.2 percentage points, motivating task-specific configuration. These results support the evaluated designs, not superiority over every alternative. Code: map, navigation, graph, retrieval.
3. Additional Backbone Evidence and Statistical Limits (w3)
We added cumulative GPT-6-astra ablations on 100 episodes per dataset. SR improves by 10 percentage points on OVON and 12 on Open-Nav R2R. The OVON gain matches GPT-5.6-luna's improvement from 45% to 55% on the same subset.
Overall benefits persist with the stronger model, although Graph trades a small OVON SR decrease for higher SPL. This supports cross-model applicability, not statistical significance. The experiments do not replace confidence intervals, repeated runs, or paired analysis, nor establish every component's necessity.