acceptodds
Under review as a conference paper at ICLR 2027

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Abstract

Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. The Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51 m navigation error across eight challenging routes evaluated three times each.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Agent, Process Monitor

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TPfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NavHarness, a vision-language navigation framework comprising goal generation, completion verification, action execution, and memory compression agents, with verified subgoal completion defining history compression boundaries. Using GPT-6-Astra with compression disabled, it achieves 79.0%/82.5% SR on R2R-CE/RxR-CE, outperforming Codex CLI with the same backbone by 5.0/6.2 percentage points. The paper also provides comparisons across backbones, context consumption analyses, and real-world robot evaluations comprising 24 trials across eight routes.

Critical strengths

1. The framework connects goal execution, completion assessment, and memory updates through a task-motivated design.

Checking subgoal completion against observations while preserving unresolved history during goal revision directly addresses the distinction between executing an action and achieving its intended outcome. Compared with simply truncating history, this design explicitly distinguishes completed stages from unresolved states needed for recovery.

2. Matched-backbone and matched-interface comparisons show consistent framework-level gains.

With GPT-6-Astra and compression disabled, NavHarness improves SPL over Codex CLI by 4.1/4.7 percentage points on the two benchmarks, while R2R-CE SR improvements extend across three backbones.

Critical weaknesses

The lack of independent ablations for the core components leaves the necessity of the proposed coupling mechanism unproven. Moreover, memory compression offers limited benefits with the strongest backbone while incurring a substantial performance cost. Inconsistent evaluation protocols across methods further undermine the central claim of superiority over existing navigation models.

1. The experiments do not establish the paper’s central claim that the gains arise from the specific coupling of goals, verification, and compression.

Figure 4 compares framework variants against Codex CLI without separately removing goal generation or completion verification, and no comparison evaluates completion-triggered compression against periodic or length-triggered compression under matched budgets. Moreover, on RxR-CE with GPT-6-Astra, compression saves only 4.05% of total tokens while reducing SPL from 64.7 to 58.3, leaving the effectiveness of the memory mechanism as a major contribution insufficiently supported.

2. The claimed superiority over existing navigation models relies on comparisons across different evaluation cohorts.

NavHarness evaluates only 100 episodes per benchmark, whereas Table 1 includes results obtained under other protocols; Appendix B acknowledges that these results do not constitute a matched ranking, yet the main text directly claims improvements over methods such as Qwen-RobotNav. The real-world comparison likewise provides only baseline aggregates without establishing identical routes, trial counts, and success criteria, so the demonstrated advantage is primarily limited to the controlled comparison with Codex CLI.

Questions for authors
  1. What are the individual contributions of goal generation and completion verification? Could you provide goal-only, verification-only, and combined variants under matched model-call or inference budgets to establish the benefit of the proposed coordination mechanism?

  2. Why should compression be triggered specifically by verified goal completion? Under comparable compression frequencies or context budgets, does this strategy outperform periodic or length-triggered summarization, and what explains the 6.4-point SPL decrease for only 4.05% token savings on RxR-CE with GPT-6-Astra?

  3. Can the main baselines be compared under matched evaluation conditions? Please provide representative strong baseline results on the same episodes and clarify whether the real-world baselines used the same eight routes, three trials per route, success thresholds, and human-intervention rules as NavHarness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TPfh (1/2)

Thank you for the thoughtful review and constructive suggestions. We have responded to your concerns as follows, and we have revised the manuscript accordingly. We would greatly appreciate it if you could consider giving us a higher rating.

W1 & Q1: The experiments do not establish the paper’s central claim that the gains arise from the specific coupling of goals, verification, and compression. Could you provide goal-only, verification-only, and combined variants under matched model-call or inference budgets to establish the benefit of the proposed coordination mechanism?

R1: We evaluated Goal-only, Verify-only, and their combination with GPT-6-Astra on each benchmark's fixed 100 episodes, with Memory disabled. Each component improves navigation, and their combination achieves the highest SR and SPL on both benchmarks, showing that goal adaptation and completion verification contribute complementary benefits. R2R-CE and RxR-CE results are reported in Table R1 (refer to R2 to Reviewer 7LNy) and in Table 5 in Appendix H of the revised manuscript. The compression comparison is reported in R2 below.

W1 & Q2: On RxR-CE with GPT-6-Astra, compression saves only 4.05% of total tokens. Why should compression be triggered specifically by verified goal completion? Under comparable compression frequencies or context budgets, does this strategy outperform periodic or length-triggered summarization, and what explains the 6.4-point SPL decrease for only 4.05% token savings on RxR-CE with GPT-6-Astra?

R2: (1) Verified completion provides a task-grounded boundary: the Memory Agent summarizes completed navigation stages while retaining observations for the current goal and unresolved cues. Fixed intervals or length thresholds depend on history size and cannot distinguish completed stages from ongoing goals. Table R7 compares Goal with periodic compression (Periodic), length-triggered compression (Length), and frame sampling, which retains initial/recent frames and selects historical frames by similarity. Compression strategies are compared across both backbones and benchmarks in Table 7 in Appendix I, together with detailed context statistics.

On GPT-6-Astra/RxR-CE, Goal achieves 81.0% SR and 58.3 SPL, exceeding Periodic, Length, and frame sampling on both metrics. Their total token usage differs from Goal by less than 7%, while their SR ranges from 48.0% to 67.0%. This comparison shows the advantage of using verified task progress to select compression boundaries. On R2R-CE, Goal achieves 77.0% SR and 65.6 SPL at 548.87k tokens, compared with 72.0% SR and 63.2 SPL at 546.09k tokens for Length.

(2) Token savings and SPL with GPT-6-Astra. Memory's benefit depends on the redundant history accumulated by the backbone. GPT-6-Astra follows instructions more accurately, with fewer execution steps, detours, and backtracks, so there is less history to compress. GPT-5.6-Sol more often takes incorrect turns and backtracks; the additional execution steps accumulate repeated observations and historical context, allowing Memory to remove substantially more repeated input. On RxR-CE, Goal reduces total tokens per episode relative to Off by 56.59% for GPT-5.6-Sol and 4.05% for GPT-6-Astra. These episode totals include both navigation and Memory calls.

On GPT-6-Astra/RxR-CE, Goal reduces images per decision from 27.26 to 13.62 and text from 105.92k to 38.50k characters. SR changes from 82.5% to 81.0%, SPL from 64.7 to 58.3, and inference latency from 142.5 to 145.3 s/episode. Together with Table R7, these results show how verified completion supports shorter contexts while retaining higher SR and SPL than the other compression strategies at comparable total token usage.

Table R7. Memory compression on R2R-CE and RxR-CE (GPT-6-Astra; tokens: k/episode).

NavHarnessR2R-CERxR-CE
StrategySR ↑SPL ↑Total ↓Total/Off (%) ↓SR ↑SPL ↑Total ↓Total/Off (%) ↓
Off79.066.6580.91—82.564.71538.84—
Periodic68.059.1605.72104.2750.036.71575.15102.36
Length72.063.2546.0994.0148.037.31494.9597.15
Frame sampling75.063.9458.3678.9067.051.51426.3292.69
Goal77.065.6548.8794.4881.058.31476.4595.95
AuthorsOpenReview ↗

Response to Reviewer TPfh (2/2)

W2 & Q3: The claimed superiority over existing navigation models relies on comparisons across different evaluation cohorts. NavHarness evaluates only 100 episodes per benchmark, whereas Table 1 includes results obtained under other protocols; Appendix B acknowledges that these results do not constitute a matched ranking, yet the main text directly claims improvements over methods such as Qwen-RobotNav. The real-world comparison likewise provides only baseline aggregates without establishing identical routes, trial counts, and success criteria, so the demonstrated advantage is primarily limited to the controlled comparison with Codex CLI. Can the main baselines be compared under matched evaluation conditions? Please provide representative strong baseline results on the same episodes and clarify whether the real-world baselines used the same eight routes, three trials per route, success thresholds, and human-intervention rules as NavHarness.

R3: (1) Full simulation evaluation. We added full Val-Unseen evaluations on R2R-CE and RxR-CE, allowing comparison with published navigation metrics of end-to-end policies such as OmniNav. On RxR-CE, NavHarness/GPT-6-Astra achieves 82.6% SR and 67.1 SPL, versus 73.6% SR and 62.0 SPL for OmniNav in Table 1 of the revised manuscript. NavHarness and Codex CLI use the same backbones and evaluation sets, with improvements across both backbones and benchmarks. Table R8 presents the full results (Table 4 in Appendix G of the revised manuscript).

(2) Identical real-world settings. NaVILA, AwareVLN, StreamVLN, JanusVLN, and NavHarness use exactly the same eight routes, three trials per route (24 trials), success thresholds, and human-intervention rules. Table R6 (refer to R3 to Reviewer HN8n) reports navigation performance and inference latency (Table 3 of the revised manuscript).

We added four routes with three trials each (12 real-world trials). NavHarness maintains a substantial performance advantage. Table R9 reports navigation performance and inference latency (Table 9 in Appendix J of the revised manuscript). Figures 11–14 in the revised manuscript visualize these additional cases.

Table R8. Full Val-Unseen evaluation on R2R-CE and RxR-CE (NE: m; other metrics: %).

BackboneMethodR2R-CERxR-CE
NE ↓OSR ↑SR ↑SPL ↑NE ↓SR ↑SPL ↑nDTW ↑
GPT-5.6-SolCodex CLI4.8068.764.445.88.6332.323.344.1
GPT-5.6-SolNavHarness4.6070.668.748.97.4239.528.749.8
GPT-6-AstraCodex CLI3.1778.875.762.92.9677.361.675.1
GPT-6-AstraNavHarness2.5686.482.268.92.6182.667.175.9

Table R9. Additional real-world evaluation and inference latency (4 routes × 3 trials; inference time: s/episode).

MethodSR (%) ↑NE (m) ↓Infer. (s) ↓
NaVILA25.003.78117.6
AwareVLN25.003.81125.5
StreamVLN33.333.7191.3
JanusVLN50.002.21231.3
NavHarness91.671.3294.2
Area_Chair_S2oLOpenReview ↗

Response Reminder

Dear Reviewer,

As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.

We greatly appreciate your contribution.

Reviewer_HN8nOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NavHarness, an agentic Vision-Language Navigation framework that combines adaptive goal generation, goal-specific verification, and progress-aware memory compression. The method is evaluated on R2R-CE, RxR-CE, and real-world robot navigation tasks, showing promising improvements over several baselines.

Critical strengths
  1. The framework provides a coherent way to connect adaptive goal generation, execution verification, and memory management, addressing important challenges in long-horizon navigation.
  2. NavHarness achieves strong results on R2R-CE and RxR-CE, with consistent improvements over the matched Codex CLI baselines across different model backbones.
  3. The real-world experiments on a quadruped robot provide useful evidence of the framework's practical applicability beyond simulation.
Critical weaknesses
  1. The paper lacks separate ablations for adaptive goal generation and verification. Since these components are evaluated together, it is difficult to determine how much each contributes to the overall performance improvement.
  2. The benefits of memory compression are not fully convincing. With GPT-6-Astra, the total token savings are only around 4–6%, while compression leads to performance degradation in some settings. The paper also lacks comparisons with simpler compression strategies, making it unclear whether verification-triggered compression is necessary.
  3. The evaluation is relatively limited in scale, with only 100 episodes per simulation benchmark and 24 real-world trials. The paper does not report confidence intervals or statistical significance, making it difficult to assess the robustness of the reported improvements.
Questions for authors
  1. Could the authors provide separate ablations for the Goal Agent and Verify Agent to better understand their individual contributions?
  2. How does the proposed verification-triggered memory compression compare with simpler approaches, such as fixed-interval or token-budget-based compression, under comparable computational budgets?
  3. Could the authors provide additional analysis of verification accuracy, inference latency, and statistical uncertainty to better assess the reliability and practical efficiency of the framework?

If the authors can address most of my concerns, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HN8n (1/2)

Thank you for the thoughtful review and constructive suggestions. We have responded to your concerns as follows, and we have revised the manuscript accordingly. We would greatly appreciate it if you could consider giving us a higher rating.

W1 & Q1: The paper lacks separate ablations for adaptive goal generation and verification. Since these components are evaluated together, it is difficult to determine how much each contributes to the overall performance improvement. Could the authors provide separate ablations for the Goal Agent and Verify Agent to better understand their individual contributions?

R1: We evaluated Goal-only, Verify-only, and their combination with GPT-6-Astra on each benchmark's fixed 100 episodes, with Memory disabled. Each component improves navigation, and their combination achieves the highest SR and SPL on both benchmarks, showing that goal adaptation and completion verification contribute complementary benefits. Table R1 (refer to R2 to Reviewer 7LNy) reports the R2R-CE and RxR-CE results, which also appear in Table 5 in Appendix H of the revised manuscript.

W2 & Q2: The benefits of memory compression are not fully convincing. With GPT-6-Astra, the total token savings are only around 4–6%, while compression leads to performance degradation in some settings. The paper also lacks comparisons with simpler compression strategies, making it unclear whether verification-triggered compression is necessary. How does the proposed verification-triggered memory compression compare with simpler approaches, such as fixed-interval or token-budget-based compression, under comparable computational budgets?

R2: (1) Memory's benefit depends on the redundant history accumulated by the backbone. GPT-6-Astra follows instructions more accurately, with fewer execution steps, detours, and backtracks, so there is less history to compress. GPT-5.6-Sol more often takes incorrect turns and backtracks; the additional execution steps accumulate repeated observations and historical context, allowing Memory to remove substantially more repeated input. On RxR-CE, Goal reduces total tokens per episode relative to Off by 56.59% for GPT-5.6-Sol and 4.05% for GPT-6-Astra. These episode totals include both navigation and Memory calls.

(2) We compare Goal with periodic compression (Periodic), length-triggered compression (Length), and frame sampling, which retains initial/recent frames and selects historical frames by similarity. Table R7 (refer to R2 to Reviewer TPfh) reports GPT-6-Astra results on R2R-CE and RxR-CE; compression strategies are compared across both backbones and benchmarks in Table 7 in Appendix I, together with detailed context statistics.

On GPT-6-Astra/RxR-CE, Goal achieves 81.0% SR and 58.3 SPL, exceeding Periodic, Length, and frame sampling on both metrics. Their total token usage differs from Goal by less than 7%, while their SR ranges from 48.0% to 67.0%. This comparison shows the advantage of using verified task progress to select compression boundaries. On R2R-CE, Goal achieves 77.0% SR and 65.6 SPL at 548.87k tokens, compared with 72.0% SR and 63.2 SPL at 546.09k tokens for Length.

AuthorsOpenReview ↗

Response to Reviewer HN8n (2/2)

W3 & Q3: The paper does not report confidence intervals or statistical significance. Could the authors provide additional analysis of verification accuracy, inference latency, and statistical uncertainty to better assess the reliability and practical efficiency of the framework?

R3: (1) Human verification. We manually evaluated 150 Verify decisions from 115 episodes. Agreement with human judgments is 84.67%; macro precision/recall/F1 are 84.67%/85.64%/84.37%. Precision for completed subgoals is 100.00%. Table R2 (refer to R2 to Reviewer 7LNy) reports the class-wise results (Table 6 in Appendix H of the revised manuscript).

(2) Variability and full evaluation. Table R4 presents mean ± standard deviation across two evaluation runs, each using the same fixed 100 episodes per benchmark (Table 1 of the revised manuscript). We added full R2R-CE and RxR-CE Val-Unseen evaluations, using the same backbones and episodes as Codex CLI. NavHarness/GPT-6-Astra achieves 82.6% SR and 67.1 SPL on RxR-CE. Gains persist across both backbones and benchmarks at this larger scale. Table R8 (refer to R3 to Reviewer TPfh) reports the full results (Table 4 in Appendix G of the revised manuscript).

(3) Real-world evaluation. We added four routes with three trials each (12 real-world trials). Table R9 (refer to R3 to Reviewer TPfh) reports navigation quality and inference latency on the additional routes (Table 9 in Appendix J of the revised manuscript). NavHarness maintains a substantial performance advantage. Figures 11–14 in the revised manuscript visualize these additional cases.

(4) Latency. Table R5 reports per-episode token consumption and inference latency (Table 2 of the revised manuscript). On RxR-CE, GPT-6-Astra takes 146.4/142.5/145.3 s/episode with Codex CLI (Native), NavHarness (Off), and NavHarness (Goal), respectively. GPT-5.6-Sol decreases from 386.7 s with Off to 241.6 s with Goal.

Table R6 reports navigation performance and inference latency on the original 24 real-world trials (Table 3 of the revised manuscript). During real-world deployment, the robot communicates with a server running the open-source baselines, while NavHarness accesses its backbone through a remote API. In the original real-world evaluation, NavHarness takes 99.3 s/episode, versus 121.1/150.4/100.7/243.7 s for NaVILA/AwareVLN/StreamVLN/JanusVLN. It achieves the highest SR, lowest NE, and lowest inference latency under this deployment setting.

Table R4. Navigation performance (mean ± standard deviation) over two evaluation runs (NE: m; other metrics: %).

BackboneMethodR2R-CERxR-CE
NE ↓OSR ↑SR ↑SPL ↑NE ↓SR ↑SPL ↑nDTW ↑
GPT-5.6-SolCodex CLI4.75 ± 0.5970.0 ± 3.063.0 ± 4.044.5 ± 3.59.03 ± 0.4928.0 ± 3.021.5 ± 2.541.2 ± 2.0
GPT-5.6-SolNavHarness4.60 ± 0.4269.5 ± 3.567.0 ± 2.047.9 ± 1.47.28 ± 0.6734.0 ± 5.025.0 ± 3.647.0 ± 3.5
GPT-6-AstraCodex CLI4.49 ± 0.5275.0 ± 4.074.0 ± 3.062.5 ± 2.13.14 ± 0.3676.0 ± 3.060.0 ± 2.671.9 ± 1.7
GPT-6-AstraNavHarness3.27 ± 0.2281.5 ± 2.579.0 ± 3.066.6 ± 2.72.42 ± 0.4582.5 ± 4.564.7 ± 3.273.8 ± 2.4

Table R5. Simulation token consumption and inference latency (tokens: k/episode; inference time: s/episode).

BackboneConfigurationR2R-CERxR-CE
TokensInfer.TokensInfer.
GPT-5.6-SolCodex CLI (Native)1410.31216.02027.81298.5
GPT-5.6-SolNavHarness (Off)1345.23197.34298.55386.7
GPT-5.6-SolNavHarness (Goal)861.82175.81865.92241.6
GPT-6-AstraCodex CLI (Native)689.81138.91149.50146.4
GPT-6-AstraNavHarness (Off)580.91126.51538.84142.5
GPT-6-AstraNavHarness (Goal)548.87120.81476.45145.3

Table R6. Original real-world evaluation and inference latency (8 routes × 3 trials; inference time: s/episode).

MethodSR (%) ↑NE (m) ↓Infer. (s) ↓
NaVILA20.833.91121.1
AwareVLN20.833.89150.4
StreamVLN25.003.78100.7
JanusVLN50.002.34243.7
NavHarness83.331.5199.3
Area_Chair_S2oLOpenReview ↗

Response Reminder

Dear Reviewer,

As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.

We greatly appreciate your contribution.

Thanks for your response. I will raise my score to 3.

Reviewer_7LNyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors presented NavHarness, an agentic harness that allows for visual language navigation in a continuous environment. The main contribution is the harness with 4 components: Goal, Verify, Memory, Visuomotor Agents. They evaluate this on R2R-CE and RxR-CE benchmarks and compare to many methods under Trained navigation models, Navigation workflows and other VLN agents (including latest harness and models) and is able to beat them using less tokens (compressed) compared to the other VLN agents.

Critical strengths

Very comprehensive baselines with many SOTA 2026 models and for the agents the latest agent harness + models. Gives a lot of credibility that their model can outperform these baselines.

Critical weaknesses

One slight weakness is the agent architecture is towards the simple side, but it's still able to outperform SOTA baselines. Another weakness is missing ablation studies for the different agent components and experiments showing the credibility of the auto- verification agent.

Questions for authors

In real world settings, could this be deployed or would the token consumption be prohibitive?

Future work: consider adding more tools in the navigation toolset (expand) and using self-improvement across episodes to improve the agent overall.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7LNy (1/2)

Thank you for the thoughtful review and constructive suggestions. We have responded to your concerns as follows, and we have revised the manuscript accordingly. We would greatly appreciate it if you could consider giving us a higher rating.

W1: One slight weakness is the agent architecture is towards the simple side, but it's still able to outperform SOTA baselines.

R1: NavHarness's agent architecture addresses two challenges in long-horizon navigation. (1) The Goal Agent turns long instructions into actionable subgoals, adapts them to observations, and uses the Verify Agent to assess completion. (2) The Memory Agent controls context growth by summarizing completed stages while retaining landmarks, spatial relations, and cues needed for unfinished navigation.

W2: Another weakness is missing ablation studies for the different agent components and experiments showing the credibility of the auto- verification agent.

R2: (1) We evaluated Goal-only, Verify-only, and their combination with GPT-6-Astra on each benchmark's fixed 100 episodes, with Memory disabled. Each component improves navigation, and their combination achieves the highest SR and SPL on both benchmarks, showing that goal adaptation and completion verification contribute complementary benefits. R2R-CE and RxR-CE results are reported in Table R1 and in Table 5 in Appendix H of the revised manuscript.

(2) We grouped Verify feedback by meaning into completed, not completed, and uncertain, and sampled 50 decisions from each group (150 decisions from 115 episodes). Human evaluation used the original instruction, current subgoal, verification question, all RGB images received by Verify, and execution history to assess completion, non-completion, or insufficient evidence. Agreement with human judgments is 84.67%, and precision for completed subgoals is 100.00% on these sampled decisions. Table R2 reports the class-wise metrics (Table 6 in Appendix H of the revised manuscript).

For example, an instruction requires two successive right turns after exiting, then stopping. The current subgoal is to reach a white partition and complete the first turn. Observations show the kitchen on the left, the partition on the right, and the corridor extending toward the next corner. Verify and the human evaluator both judge this subgoal complete while retaining the second turn and stop as unfinished, illustrating how verification grounds local progress in observations.

Table R1. Goal/Verify ablation on R2R-CE and RxR-CE (GPT-6-Astra; NE: m; other metrics: %).

MethodR2R-CERxR-CE
NE ↓OSR ↑SR ↑SPL ↑NE ↓SR ↑SPL ↑nDTW ↑
Baseline4.4975.074.062.53.1476.060.071.9
Goal-only4.1976.075.063.42.8880.062.772.8
Verify-only4.1178.076.062.83.0878.061.472.3
NavHarness3.2781.579.066.62.4282.564.773.8

Table R2. Human evaluation of Verify (150 sampled decisions; all metrics: %).

ClassPrecision (%)Recall (%)F1 (%)
Completed100.0083.3390.91
Not completed72.0094.7481.82
Uncertain82.0078.8580.39
Macro average84.6785.6484.37
AuthorsOpenReview ↗

Response to Reviewer 7LNy (2/2)

Q1: In real world settings, could this be deployed or would the token consumption be prohibitive?

R3: NavHarness has been deployed on a real robot. We selected R2R-CE and RxR-CE episodes with trajectory and instruction lengths similar to the real-world tasks and compared total tokens per episode. As shown in Table R3 (Table 8 in Appendix J of the revised manuscript), the real-world tasks consume 1023.97k tokens per episode, compared with 1094.59k for matched R2R-CE tasks and 1598.08k for matched RxR-CE tasks. Thus, real-world token consumption remains comparable to that of simulation tasks with similar trajectory and instruction lengths.

Table R3. Real-world and matched simulation task usage.

EnvironmentTrajectory (m)Instruction wordsTokens/ep (k)
Real world27.039.01023.97
R2R-CE (matched)27.832.91094.59
RxR-CE (matched)25.149.11598.08

Q2: Future work: consider adding more tools in the navigation toolset (expand) and using self-improvement across episodes to improve the agent overall.

R4: We agree that expanding the navigation toolset and enabling self-improvement across episodes are promising directions. We also aim to extend goal decomposition, verification, and memory to a unified framework for ground and aerial navigation.

Area_Chair_S2oLOpenReview ↗

Response Reminder

Dear Reviewer,

As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.

We greatly appreciate your contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.