- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes NavHarness, a vision-language navigation framework comprising goal generation, completion verification, action execution, and memory compression agents, with verified subgoal completion defining history compression boundaries. Using GPT-6-Astra with compression disabled, it achieves 79.0%/82.5% SR on R2R-CE/RxR-CE, outperforming Codex CLI with the same backbone by 5.0/6.2 percentage points. The paper also provides comparisons across backbones, context consumption analyses, and real-world robot evaluations comprising 24 trials across eight routes.
Critical strengths
1. The framework connects goal execution, completion assessment, and memory updates through a task-motivated design.
Checking subgoal completion against observations while preserving unresolved history during goal revision directly addresses the distinction between executing an action and achieving its intended outcome. Compared with simply truncating history, this design explicitly distinguishes completed stages from unresolved states needed for recovery.
2. Matched-backbone and matched-interface comparisons show consistent framework-level gains.
With GPT-6-Astra and compression disabled, NavHarness improves SPL over Codex CLI by 4.1/4.7 percentage points on the two benchmarks, while R2R-CE SR improvements extend across three backbones.
Critical weaknesses
The lack of independent ablations for the core components leaves the necessity of the proposed coupling mechanism unproven. Moreover, memory compression offers limited benefits with the strongest backbone while incurring a substantial performance cost. Inconsistent evaluation protocols across methods further undermine the central claim of superiority over existing navigation models.
1. The experiments do not establish the paper’s central claim that the gains arise from the specific coupling of goals, verification, and compression.
Figure 4 compares framework variants against Codex CLI without separately removing goal generation or completion verification, and no comparison evaluates completion-triggered compression against periodic or length-triggered compression under matched budgets. Moreover, on RxR-CE with GPT-6-Astra, compression saves only 4.05% of total tokens while reducing SPL from 64.7 to 58.3, leaving the effectiveness of the memory mechanism as a major contribution insufficiently supported.
2. The claimed superiority over existing navigation models relies on comparisons across different evaluation cohorts.
NavHarness evaluates only 100 episodes per benchmark, whereas Table 1 includes results obtained under other protocols; Appendix B acknowledges that these results do not constitute a matched ranking, yet the main text directly claims improvements over methods such as Qwen-RobotNav. The real-world comparison likewise provides only baseline aggregates without establishing identical routes, trial counts, and success criteria, so the demonstrated advantage is primarily limited to the controlled comparison with Codex CLI.
Questions for authors
-
What are the individual contributions of goal generation and completion verification? Could you provide goal-only, verification-only, and combined variants under matched model-call or inference budgets to establish the benefit of the proposed coordination mechanism?
-
Why should compression be triggered specifically by verified goal completion? Under comparable compression frequencies or context budgets, does this strategy outperform periodic or length-triggered summarization, and what explains the 6.4-point SPL decrease for only 4.05% token savings on RxR-CE with GPT-6-Astra?
-
Can the main baselines be compared under matched evaluation conditions? Please provide representative strong baseline results on the same episodes and clarify whether the real-world baselines used the same eight routes, three trials per route, success thresholds, and human-intervention rules as NavHarness.
3 comments
Response to Reviewer TPfh (2/2)
W2 & Q3: The claimed superiority over existing navigation models relies on comparisons across different evaluation cohorts. NavHarness evaluates only 100 episodes per benchmark, whereas Table 1 includes results obtained under other protocols; Appendix B acknowledges that these results do not constitute a matched ranking, yet the main text directly claims improvements over methods such as Qwen-RobotNav. The real-world comparison likewise provides only baseline aggregates without establishing identical routes, trial counts, and success criteria, so the demonstrated advantage is primarily limited to the controlled comparison with Codex CLI. Can the main baselines be compared under matched evaluation conditions? Please provide representative strong baseline results on the same episodes and clarify whether the real-world baselines used the same eight routes, three trials per route, success thresholds, and human-intervention rules as NavHarness.
R3: (1) Full simulation evaluation. We added full Val-Unseen evaluations on R2R-CE and RxR-CE, allowing comparison with published navigation metrics of end-to-end policies such as OmniNav. On RxR-CE, NavHarness/GPT-6-Astra achieves 82.6% SR and 67.1 SPL, versus 73.6% SR and 62.0 SPL for OmniNav in Table 1 of the revised manuscript. NavHarness and Codex CLI use the same backbones and evaluation sets, with improvements across both backbones and benchmarks. Table R8 presents the full results (Table 4 in Appendix G of the revised manuscript).
(2) Identical real-world settings. NaVILA, AwareVLN, StreamVLN, JanusVLN, and NavHarness use exactly the same eight routes, three trials per route (24 trials), success thresholds, and human-intervention rules. Table R6 (refer to R3 to Reviewer HN8n) reports navigation performance and inference latency (Table 3 of the revised manuscript).
We added four routes with three trials each (12 real-world trials). NavHarness maintains a substantial performance advantage. Table R9 reports navigation performance and inference latency (Table 9 in Appendix J of the revised manuscript). Figures 11–14 in the revised manuscript visualize these additional cases.
Table R8. Full Val-Unseen evaluation on R2R-CE and RxR-CE (NE: m; other metrics: %).
Backbone Method R2R-CE RxR-CE NE ↓ OSR ↑ SR ↑ SPL ↑ NE ↓ SR ↑ SPL ↑ nDTW ↑ GPT-5.6-Sol Codex CLI 4.80 68.7 64.4 45.8 8.63 32.3 23.3 44.1 GPT-5.6-Sol NavHarness 4.60 70.6 68.7 48.9 7.42 39.5 28.7 49.8 GPT-6-Astra Codex CLI 3.17 78.8 75.7 62.9 2.96 77.3 61.6 75.1 GPT-6-Astra NavHarness 2.56 86.4 82.2 68.9 2.61 82.6 67.1 75.9 Table R9. Additional real-world evaluation and inference latency (4 routes × 3 trials; inference time: s/episode).
Method SR (%) ↑ NE (m) ↓ Infer. (s) ↓ NaVILA 25.00 3.78 117.6 AwareVLN 25.00 3.81 125.5 StreamVLN 33.33 3.71 91.3 JanusVLN 50.00 2.21 231.3 NavHarness 91.67 1.32 94.2
Response Reminder
Dear Reviewer,
As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.
We greatly appreciate your contribution.
Response to Reviewer TPfh (1/2)
Thank you for the thoughtful review and constructive suggestions. We have responded to your concerns as follows, and we have revised the manuscript accordingly. We would greatly appreciate it if you could consider giving us a higher rating.
W1 & Q1: The experiments do not establish the paper’s central claim that the gains arise from the specific coupling of goals, verification, and compression. Could you provide goal-only, verification-only, and combined variants under matched model-call or inference budgets to establish the benefit of the proposed coordination mechanism?
W1 & Q2: On RxR-CE with GPT-6-Astra, compression saves only 4.05% of total tokens. Why should compression be triggered specifically by verified goal completion? Under comparable compression frequencies or context budgets, does this strategy outperform periodic or length-triggered summarization, and what explains the 6.4-point SPL decrease for only 4.05% token savings on RxR-CE with GPT-6-Astra?