acceptodds
Under review as a conference paper at ICLR 2027

ReDual-VLN: Recoverable Dual-System Vision-and-Language Navigation via Language-Conditioned Backtracking

Abstract

Vision-and-language navigation (VLN) aims to enable embodied agents to navigate unseen environments by following natural-language instructions. Long-horizon VLN is vulnerable to route deviations during local execution, which can disrupt the remaining task. Recovery requires selecting an appropriate recovery anchor and maintaining task context for subsequent navigation, placing additional reasoning and memory demands on the VLN model. We propose ReDual-VLN, a dual-system framework that separates high-level reasoning from local navigation execution. A VLM-based Route Supervisor performs route planning, progress tracking, and recovery reasoning, while Hierarchical Context Management (HCM) maintains semantic plans, execution history, and recovery anchors. A Backtracking-Enabled VLN (BE-VLN) then focuses on executing local navigation and recovery instructions. To equip BE-VLN with the required execution capabilities, we construct ReTrace-RxR with paired short-horizon navigation and backtracking data, enabling the model to follow local instructions, predict instruction termination, and perform language-conditioned backtracking. When a deviation occurs, the Route Supervisor selects a previously visited anchor on the intended route as the recovery target and instructs BE-VLN to backtrack to it. After verifying the return, the supervisor directs the executor to resume the remaining instructions, forming a closed loop of deviation detection, backtracking-based recovery, and task resumption. Systematic evaluations in simulation and the real world demonstrate that ReDual-VLN improves long-horizon navigation performance and substantially enhances recovery and task resumption following route deviations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-and-Language Navigation, Long-Horizon Navigation, Dual-System Architecture, Backtracking, Recovery.

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xiTmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes ReDual-VLN, a dual-system framework for long-horizon VLN with deviation recovery. The upper VLM is designed for route planning, context management, and recovery decisions, and the lower VLN executes local navigation and backtracking instructions. The authors also introduce ReTrace-RxR for training short-horizon navigation and backtracking. The results show improvements on both R2R-CE and RxR-CE.

Critical strengths

S1. The paper focuses on an important and practical problem in long-horizon VLN. The motivation is clear, and the framework is sound. The results look pretty good, especially the improvements brought by the recovery mechanism on both benchmarks.

S2. I also like the idea of constructing paired forward and backtracking data with instruction-specific termination labels. This idea seems useful for training the VLN executor, and might be able to be applied to other VLN methods.

Critical weaknesses

W1. I think the novelty of the proposed framework could be better clarified. Some very recent works, such as ReflectVLN, Route2Step, and Uni-LaViRA, have already explored similar ideas, including hierarchical reasoning, deviation correction, and backtracking. Although these works are also quite recent, I think it would be helpful if the authors could better discuss the differences and highlight the unique contributions of their approach.

W2. The recovery mechanism should have a more detailed evaluation. The overall improvement is clear, but it would be helpful to know how well each part works, such as deviation detection, anchor selection, backtracking, and task resumption. Also, the backtracking training data seems to mainly cover returning from the endpoint of a segment to its starting point. I am not sure how well this generalizes to actual navigation errors, especially when the agent deviates in the middle of an instruction. Some experimental results are not well explained. For example, in Table 2, BE-VLN achieves a much higher SR than SPAN-Nav, but the improvement in OSR is quite small. I was wondering whether the improvement mainly comes from better stopping predictions rather than better navigation. I think this deserves some further discussion.

Questions for authors

Q1. How does the proposed recovery mechanism compare with simpler approaches, such as replanning from the current position or returning to the last completed sub-instruction? I think this comparison would help clarify the actual benefit of the proposed recovery strategy.

Q2. Can the authors provide a more direct ablation study to show the independent contribution of HCM, while keeping the upper VLM and instruction decomposition unchanged?

Q3. How well does the backtracking model generalize to actual route deviations, especially when the agent deviates in the middle of an instruction or needs to return to an intermediate anchor rather than the starting point of a segment?

Q4. In Table 2, BE-VLN improves SR from 48.25% to 81.01% compared with SPAN-Nav, while OSR only increases from 83.16% to 84.11%. Could the authors clarify how much of this improvement comes from better instruction termination rather than navigation itself?

7 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses: Contributions and Recovery Evidence

Comment 1: Contributions and Recovery Evidence

Thank you. We make long-horizon navigation recoverable through learned local return execution, HCM-managed route context and verified completion of unfinished requirements. We address Q1-Q4 separately.

W1. Contributions and practical advantages

ReflectVLN and Route2Step advance hierarchical correction and progress tracking; Uni-LaViRA supports map-planned returns, replanning and pending-goal memory. Our contribution combines learned language-conditioned return execution, instruction-relative termination and verified completion of the interrupted objective.

We train the local capabilities required by recovery. A landmark passed on a full route can be a short instruction's endpoint. ReTrace-RxR aligns forward/return instructions with direction-specific observations, motion and terminal boundaries, making both commands executable through one policy. With the backbone, heads and update budget fixed, adding forward data raises short-horizon SR from 55.74% to 81.01%; adding return data raises backtracking SR from 47.80% to 63.38%, each against the full corpus lacking that subset (Appendix B.8).

We restore a usable route reference through language and RGB history. After a wrong turn, the current scene may no longer align with the remaining instruction. We select an identifiable visited anchor on the intended route, generate a return instruction from the actual traversal, including its off-route portion, and execute it with BE-VLN. Unlike a map-planned waypoint return, this uses the same language/RGB interface as forward execution. In the 18-entry paired comparison, we achieve 77.78% final-goal SR, versus 66.67% with fixed-boundary return and 44.44% with direct replanning, with the lowest mean actions and travel. This tests return and continuation from matched recovery states (Appendix C.6, Table 20).

We preserve the original obligation while restoring the route. HCM separates the unfinished objective from the temporary return command and links both to execution attempts, boundaries and visual evidence. We keep route-level history with the Supervisor, while BE-VLN uses its current instruction and 32-observation visual window. We verify landmark identity, topology and a navigable reentry region, then regenerate the original objective's remaining requirements. With VLM, cached decomposition, executor and recovery capability fixed, HCM raises SR from 68.75% to 80.00% on 80 tasks (Appendix B.7.2).

These components turn local return into long-horizon task recovery. With the same Supervisor, HCM, BE-VLN checkpoint, review settings and action budget, enabling recovery raises full-route SR from 69.6% to 76.7% on R2R-CE and 72.3% to 78.5% on RxR-CE (Table 1).

W2. Recovery stages, natural interruptions, and local completion

We evaluate recovery through completion of the interrupted requirements. We run 300 random RxR-CE val-unseen episodes once each and analyze all 45 natural attempts from 18 episodes. Independent manual inspection confirms that every intervention precedes genuine spatial/semantic completion of the active sub-instruction.

Recovery outcomeCountRate
Anchor return: exit geodesic distance <=3 m40/4588.9%
Task resumption initiated42/4593.3%
Error corrected and interrupted requirements completed35/4577.8%

We manually check post-return trajectories against the interrupted sub-instruction. Of the 35 completions, 32 meet the geometric criterion and three complete from a suitable reentry state beyond it (Appendix C.3). Thus, we distinguish geometric return, execution handoff and objective completion.

The ten failures have one primary attribution each: detection (1), anchor selection (2), return execution (0), verification (2), or post-return objective completion (5), including two budget-exhausted cases. This diagnosis identifies the importance of connecting a returned state to the unfinished requirements.

We recover from natural intermediate states using historical evidence. We use within-segment anchors in 44 attempts, obtaining 39 returns and 35 objective completions. For the six anchors outside BE-VLN's recent history, trajectory/landmark review confirms arrival at the intended anchor in 5/6; 3/6 satisfy the exit-pose criterion and 2/6 complete the interrupted objective. HCM supplies the historical anchor image to the Supervisor, which turns it into a language instruction for the recent-window executor (Appendix C.4, Tables 17-18).

We separate local-task learning from the benefit of selecting ITF. On the same short-horizon test with BE-VLN fixed, small-motion stopping achieves 76.13% SR and ITF achieves 81.01%. ITF adds 4.88 SR points; OSR is 83.59%/84.11%, narrowing the OSR-SR gap from 7.46 to 3.10 points. These controls connect task-specific motion/boundary training to reliable local completion (Appendix B.8-B.9; Q4).

AuthorsOpenReview ↗

Response to Questions: Recovery, HCM, and Local-Task Learning

Comment 2: Recovery, HCM, and Local-Task Learning

Thank you. We address Q1-Q4 below; W1-W2 are answered separately.

Q1. What do adaptive anchors add over simpler recovery strategies?

We improve final-goal completion from matched recovery states. We achieve 14/18 successes, versus 12/18 for fixed-boundary return and 8/18 for direct replanning. A valid within-step anchor can restore a route reference without returning to an earlier boundary.

From 27 natural events in a previously screened pool, we retain the earliest entry per episode, yielding 18. We restore pose, decomposition, progress and history, fixing VLM evidence, executor and remaining budget. Each strategy starts with one VLM call, including adaptive selection/return-instruction generation. Strategy rules apply to the unfinished entry step; later steps share a controller. Direct replanning permits return motion. Success requires final STOP at the original goal (<3 m).

StrategySR (%)NE (m)ActionsPath (m)
Direct replanning44.4411.17154.1123.05
Fixed-boundary return66.678.21150.2221.48
Adaptive anchor77.787.73142.9421.17

We average actions and travel over all entries, including return, continuation and failures. These results support restoring a route reference before continuing the task (Appendix C.6, Table 20).

Q2. What does HCM contribute with the Supervisor and decomposition fixed?

We isolate an 11.25-point SR gain from context organization. On the same 80 random RxR-CE val-unseen tasks, we fix the VLM, executor, cached decomposition, image-selection policy/budget, review triggers, recovery and Controller safeguards.

Both receive the original instruction and basic dispatch state (objective, command and return target), retaining full supervisory/recovery capabilities. Our Chronological History control reconstructs relations from ordered commands/reviews. With HCM, we add instruction-window, attempt, boundary and visual-reference associations linking temporary returns to original obligations. Appendix D provides both complete prompts.

MetricChronological HistoryHCM
SR (%)68.7580.00
SPL x10045.3655.27
OSR (%)75.0085.00

We also reduce cumulative review tokens by 6.85% and service time by 7.28%, averaged across all 80 tasks and excluding initial decomposition (Appendix B.7.2, Table 11).

Q3. Can BE-VLN handle intermediate interruptions and anchors?

Yes: we initiate recovery before instruction completion and return to online-selected intermediate anchors. We analyze all 45 natural attempts from 18 of 300 random RxR-CE val-unseen episodes; manual inspection verifies every entry precedes genuine completion of the active forward instruction.

Natural recovery subsetAnchor returnTask recovery
All mid-instruction attempts40/4535/45
Within-segment anchors39/4435/44
Anchors outside executor history3/62/6

Rows are nested. Anchor return uses exit geodesic distance <=3 m; we manually verify correction and completion of the interrupted requirements. For outside-history anchors, we confirm 5/6 arrivals from trajectories/landmarks, 3/6 meet the exit-pose criterion, and 2/6 complete the objective. Our Supervisor sees the historical image; BE-VLN follows its instruction with recent context (Appendix C.4, Tables 17-18).

Training demonstrations start at segment endpoints, but trajectory/ITF supervision covers intermediate states under the same return instruction. Online, we generate commands from the actual interruption and traversal, supporting these state-dependent returns (Appendix A.2.2).

Q4. How much of the SR gain comes from stopping versus navigation?

We learn effective local execution and gain a further 4.88 SR points from selecting ITF. ReTrace-RxR trains motion and terminal boundaries relative to each local instruction. Adding its forward subset, with other sources and ITF supervision retained, raises short-horizon SR from 55.74% to 81.01% (Appendix B.8, Table 12).

We then compare stopping criteria on the same short-horizon test, fixing BE-VLN weights, the trajectory predictor, input interface and budget. Both conditions retain the same training, including ITF supervision.

CriterionSR (%)OSR (%)SPL x100NE (m)nDTW x100
Small motion76.1383.5971.452.1377.01
ITF81.0184.1176.432.0080.22

With small-motion stopping, we already obtain 76.13% SR versus SPAN-Nav's 48.25%; selecting ITF raises SR to 81.01%. We narrow BE-VLN's OSR-SR gap from 7.46 to 3.10 points, converting more goal-region visits into completed instructions. B.8 tests motion/boundary training jointly; B.9 directly measures the additional inference-time stopping benefit with the trained executor fixed.

Thank you very much for the detailed response and additional experiments. I can see quite a lof of effort here. The HCM ablation, stopping analysis, and natural recovery results address several of my concerns.

My main concern is still how reliably the system decides when recovery is needed. The recovery analysis covers cases where recovery was triggered, but how often does the system miss a deviation or trigger recovery when it is not needed?

Moreover, the results for anchors outside the executor history are not yet so convincing. From the table, only 2/6 attempts achieved task recovery. The evidence for reliable recovery might not be strong enough. But I am open to discussion on this concern.

Additionally, the results in W1 and Q1 seem to be different. Is this a typo, or am I missing anything?

I am open to further discussion, and keep my original score for now.

Reviewer_9CasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReDual-VLN, a dual-system framework for recovering from route deviations in vision-and-language navigation. A VLM-based Route Supervisor manages instruction decomposition, progress tracking, recovery-anchor selection, and task resumption, while a specialized BE-VLN executor follows local navigation and backtracking instructions. The executor provides instruction-termination feedback for supervisory review. The authors also construct ReTrace-RxR with short-horizon navigation and reverse-traversal demonstrations. Experiments on R2R-CE and RxR-CE report improved task success, with additional evaluations of local navigation, backtracking, termination, and executor replacement.

Critical strengths
  1. The recovery protocol addresses an important distinction between returning to a location and resuming the correct task state. The framework separates the active execution command from the unfinished route objective. Returning to an anchor does not automatically advance the original task index; instead, the supervisor issues the remaining requirements from the recovered state. This is a sensible design for avoiding incorrect progress updates during recovery.
  2. The recovery-enabled versus recovery-disabled comparison provides meaningful system-level evidence. The two configurations share the executor checkpoint, supervisor, review settings, and action budget. Enabling recovery increases SR from 69.6% to 76.7% on R2R-CE and from 72.3% to 78.5% on RxR-CE, while also improving SPL (Table 1). This supports a benefit from the recovery procedure beyond hierarchical execution alone.
Critical weaknesses
  1. Essential system and dataset specifications are missing, limiting reproducibility and assessment of experimental validity. The paper does not identify the Route Supervisor’s model/version, provide its prompts, specify the ITF threshold or periodic-review interval, or explain how the complete visual history is represented within its context budget. ReTrace-RxR is used for both training and independently constructed local evaluations, but dataset sizes and train/test separation by scene and source trajectory are not reported. The training-data mixture and major optimization settings are also unspecified. These omissions affect the strength of the supervisor, opportunities for intervention, and independence of the local evaluations; they are consequential to interpreting the reported results.
  2. The experiments do not sufficiently isolate the proposed context-management and data contributions. The HCM comparison simultaneously introduces a high-level VLM, global history, instruction decomposition, progress assessment, and completion verification. It therefore establishes the value of this supervision package rather than the specific HCM design. Similarly, comparisons against different executors do not isolate the contribution of ReTrace-RxR without training ablations under a matched backbone and budget. The overall recovery gain is encouraging, but the paper does not characterize whether it comes from the emphasized difficult cases—mid-instruction deviations, intermediate recovery anchors, and anchors outside the executor’s recent history. This limits the evidence for the specific mechanisms beyond the complete system.
  3. Unclear correspondence between visualized predictions and executed actions. In the supplementary videos, the overlaid predicted actions appear inconsistent with the agent’s subsequent motion, making the qualitative results difficult to interpret.
Questions for authors
  1. What are the training and test sizes and scene/source-route splits of ReTrace-RxR? Are all forward/backtracking pairs and segments from the same source trajectory kept within one split? Please also specify the full-route and QA training sources.
  2. Which model and version implement the Route Supervisor? Please provide its prompts, review interval, ITF threshold, action budget, and treatment of long visual histories, together with the executor’s training configuration.
  3. Can you isolate HCM under the same supervisor and information budget, and provide matched training ablations for adding short-horizon and backtracking data? These comparisons would clarify which proposed components account for the gains.
5 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses: Implementation, Ablations, and Recovery Analysis

Thank you for the feedback. We address Q1–Q3 separately.

W1. System, data, and training specifications

Execution. We use gpt-5.6-sol as Route Supervisor and initialize BE-VLN from Qwen3-VL-8B. Supervisor receives the original instruction, decomposition, HCM state, active command, selected images, and ITF. BE-VLN takes the current instruction and recent four-view RGB observations, predicting eight waypoints and instruction-termination feedback (ITF). ITF estimates membership in the demonstrated terminal window; Supervisor verifies spatial and semantic completion.

During recovery, we preserve the original objective separately from the temporary return command. After verifying return, we issue the remaining requirements of that same objective. This reconnects local execution with the unfinished route rather than treating anchor arrival as task completion.

We trigger review at ITF >=0.9 or every 16 completed Actor navigation calls under a shared 350-environment-action budget. We select up to eight panoramas including current, plus at most one event panorama (maximum nine). BE-VLN uses the latest 32 observations (all if fewer), spatially average-pooling historical features while retaining current detail; we retain this window on RECOVER. We provide the full prompt in Appendix D, selection rules in A.1.1, and execution settings in B.2.

Joint training and reproducibility. We jointly train one BE-VLN on full-route, short-horizon forward, backtracking, and image/video QA examples in the same training mixture. Navigation examples share trajectory/ITF heads and supervise waypoints and instruction-relative terminal windows; QA examples train the same backbone through language modeling. We specify architecture and losses in Appendices A.3–A.4.3, audited local splits in A.2.3, sources in A.4.4, and checkpoint settings in A.4.5. These include AdamW, learning rate 1e-5, global batch 512, 12,729 updates, and trajectory/ITF/QA loss weights of 20:1:1. Our DAgger QA uses only R2R/RxR training splits.

W2. Component contributions and difficult recovery

Controlled evidence. We compare the same Supervisor with Chronological History or HCM to test HCM's contribution. In Chronological History, the VLM infers links from temporal records; HCM links objectives, commands, attempts, boundaries and images. Models, decomposition, image selection/budget, review rules and recovery stay fixed on 80 paired tasks. SR rises from 68.75% to 80.00%, SPL from 45.36 to 55.27; review tokens/time fall 6.85%/7.28%. Forward data F adds 25.27 short-SR points with backtracking B retained; B adds 15.58 return-SR points with F retained (B.7.2/B.8).

Natural recovery. Across 300 randomly selected RxR-CE val-unseen episodes, 18 contain 45 natural attempts. We include all outcomes and independently verify by manual inspection that every start precedes genuine sub-instruction completion. Recovery starts follow actual execution errors, without injected deviations or preset training endpoints.

SituationAttemptsAnchor returnTask recovered
Mid-instruction4540/4535/45
Intermediate anchor4439/4435/44
Outside recent history63/62/6

Rows are nested. We score anchor return by exit geodesic distance <=3 m. We manually inspect post-return trajectories for correction of the error and completion of the interrupted sub-instruction's remaining requirements. Appendices C.1 and C.3–C.4 define and report these outcomes.

For the six outside-history cases, our trajectory/landmark audit confirms arrival at the intended anchor in 5/6, while 3/6 satisfy the final exit-pose criterion. Two complete the objective; one reaches the anchor but takes the wrong continuation branch; two reach its neighborhood but exhaust the action budget; one accepts a wrong landmark. All six contribute to the 2/6 task-recovery result. We give the historical image to Supervisor, which converts it into a return instruction for BE-VLN's current/recent observations. Appendix C.4, Tables 17–18, details the cases.

From return to final-goal completion. In a separate paired study of 18 recovery entries, we match pose, progress, history, models, initial visual evidence, and remaining budget. Adaptive-anchor return completes the original goal in 14/18 cases, compared with 12/18 for fixed-boundary return and 8/18 for direct replanning, which also allows necessary return motion. We report this comparison in Appendix C.6, Table 20. It directly links restoring a historical route reference with completion of the original task.

W3. Corrected video overlays

We thank the reviewer for pointing out this inconsistency. We corrected the horizontal mirroring error in the supplementary trajectory overlays and resubmitted videos. This visualization error did not affect model predictions, executed actions, or quantitative results.

AuthorsOpenReview ↗

Response to Questions: Dataset, System Configuration, and Controlled Experiments

We address Q1–Q3 here and W1–W3 in a separate comment.

Q1. Data sizes, split inheritance, and training sources

We construct ReTrace-RxR from VLN-CE expert routes aligned through Landmark-RxR and re-render return observations. Appendices A.2.1–A.2.3 describe construction and the audited local splits.

Local subsetSegmentsSource routesScenes
Forward train69813604358
Backtracking train38011538957
Forward test15338122311
Backtracking test6458107811

We keep each source route's annotations, segments, and forward/return pairs in one split. Local train/test scenes and routes are disjoint. Every training return has a forward counterpart; unpaired forward examples remain. Direction subsets overlap within splits; counts are unique per row.

We list navigation and DAgger-QA sources in Appendix A.4.4, Table 5:

SourceRecords/examples
RxR-CE full-route navigation2017759
R2R-CE full-route navigation631244
ReTrace-RxR short forward1386900
ReTrace-RxR backtracking1610400
Our backtracking DAgger QA (R2R)9700
Our backtracking DAgger QA (RxR)21100

Navigation entries count prediction-time records; QA entries count question–answer examples. Full routes come from VLN-CE; our DAgger rollouts and QA use only R2R/RxR training splits.

Q2. Supervisor, visual history, and joint training

Execution protocol. We use Supervisor service ID gpt-5.6-sol and initialize BE-VLN from Qwen3-VL-8B. Supervisor receives the instruction, decomposition, HCM state, active command, images, and ITF, returning evidence, a CONTINUE/ADVANCE/RECOVER/STOP decision, the objective, and any new instruction. After verified return, ADVANCE resumes the same objective. We provide the full prompt in Appendix D and execution rules in A.1 and B.2.

We review at ITF >=0.9 or every 16 completed Actor calls, sharing 350 environment actions across forward execution and recovery. We use up to eight panoramas including current, plus one event panorama at most (maximum nine), prioritizing command/attempt/recovery/boundary references before sparse execution coverage. BE-VLN uses the latest 32 four-view RGB observations (all if fewer), retaining the window on RECOVER and spatially average-pooling history while preserving current features. We detail selection and encoding in Appendices A.1.1 and A.3.2.

Joint training. We jointly train one BE-VLN on full-route, short-horizon forward, backtracking, and image/video QA data. Navigation examples supervise shared trajectory/ITF heads with waypoints and instruction-relative terminal windows; QA examples train the same backbone through language modeling. We define this joint objective in Appendices A.3–A.4.3.

We report checkpoint-recorded optimization in Appendix A.4.5, Table 6:

SettingValue
Optimizer; learning rateAdamW; 1e-5
Schedule; warmupCosine; 3%
Per-device batch; accumulation; ranks4; 8; 16
Global batch512
Epochs; optimizer updates1; 12729
Precision; distributed optimizerBF16; ZeRO-1
Trajectory/ITF/QA loss weights20:1:1

Q3. Fixed-Supervisor HCM and matched training ablations

Context organization. We compare the same Supervisor with Chronological History (the VLM infers task links from temporal records) and HCM (explicit links among objectives, commands, attempts, boundaries and images). Both retain the active objective, command, completion checks and recovery. On 80 paired tasks, we fix the VLM and executor, cached decomposition, image selection/budget, review rules and Controller safeguards to test HCM's contribution (B.7.2, Table 11):

ContextSR (%)SPL x100Review tokensReview time (s)
Chronological History68.7545.36257133254.70
HCM80.0055.27239520236.15

We gain 11.25 SR points and 9.91 SPL, with 6.85%/7.28% lower review tokens/time. Per-episode cumulative means exclude initial decomposition.

Training contributions. Full denotes our complete corpus. Removing ReTrace-RxR forward F, backtracking B, or both retains all other sources and their sampling probabilities without renormalization. We match backbone, initialization, heads, losses, seed, and update budget, and evaluate the same local tests without Supervisor. Appendix B.8, Table 12, reports:

Training dataShort SR (%)Short SPLReturn SR (%)Return SPLTPS@45° (%)
Full w/o F & B52.8845.2335.6030.7518.01
Full w/o B78.7967.2647.8032.2937.67
Full w/o F55.7449.2052.2635.4172.92
Full81.0176.4363.3838.5477.23

SPL is multiplied by 100; TPS@45° measures headings within 45° among successful returns. Our full mixture performs best on all five metrics, connecting complementary local skills with unfinished-task recovery.

Dear Reviewer,

Thank you for your detailed and thoughtful review. We have now posted a full response to your comments. In particular, we would be grateful if you could kindly reconsider your evaluation of our manuscript based on our analysis and explanations. If there are any remaining questions or concerns, we would be glad to provide further clarification. We sincerely hope that our revisions have addressed your concerns and that the improvements we made will be favorably considered during your re-evaluation. Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_VpP2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReDual-VLN aims to address the challenges of local execution error accumulation during long-range navigation and the difficulty of recovery after route deviation. The system consists of two main modules, Route Supervisor and BE-VLN, which are responsible for high-level navigation and executing local navigation instructions, respectively. Additionally, the authors propose the HCM module, which maintains the semantic task plan, complete execution history, and historical recovery anchors at the upper level, while restricting the visual history window of the low-level navigation model. Overall, the core of ReDual-VLN lies in extending traditional instruction-following into a navigation framework equipped with historical context management, path backtracking, and task recovery capabilities.

Critical strengths

The dual-system architecture presented in this paper features a rational division of responsibilities, effectively separating global task management from local navigation execution. The upper-level Route Supervisor handles long-range task understanding, historical information management, and recovery decision-making, while the lower-level BE-VLN is responsible for local action generation and instruction termination feedback. This design prevents the low-level navigation model from undertaking full task reasoning simultaneously, providing it with only a limited recent visual history, which helps control the context scale of the local execution model. Furthermore, the paper systematically demonstrates navigation performance and analyzes the role of each module.

Critical weaknesses

I believe the paper still has several issues. If the following concerns are addressed, I would be willing to raise my score:

  1. The paper does not provide detailed descriptions regarding the model used, nor its training and inference pipelines (e.g., input prompts, frame sampling strategies, and historical image compression mechanisms).
  2. Details of core modules are lacking. For example, Deviation Detection does not have a clear criterion for determining deviation, nor does it provide algorithmic details of Recovery Anchor Selection, and it does not explain how candidate anchors are ranked and judged.
  3. While the paper claims that the proposed method alleviates the history modeling burden of the low-level model, this design introduces an additional high-level VLM inference process. However, the paper lacks experimental validation regarding efficiency advantages, such as inference latency and peak memory usage.
Questions for authors

In addition to the issues mentioned in the weaknesses section, I have a few questions:

  1. How does the HCM handle the continuously growing complete visual history?
  2. Can the BE-VLN generalize to backtracking tasks from arbitrary interruption locations?
  3. There are currently many methods for decomposing complex instructions(e.g., SkillNav, IDEAL-VLN). Could the authors further explain the advantages of ReDual-VLN compared with existing methods?
6 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses: Implementation Details, Recovery Decisions, and Inference Efficiency

Thank you for the feedback. We clarify how our Supervisor manages route objectives and recovery while BE-VLN learns and executes local navigation tasks. We address W1-W3 here and Q1-Q3 separately.

W1. Models, prompts, and training/inference

Supervisor. We use gpt-5.6-sol. Each review receives the original instruction, decomposition, HCM state, selected panoramas, and ITF score. The Supervisor verifies spatial/semantic requirements and selects CONTINUE, ADVANCE, RECOVER, or STOP. ADVANCE dispatches the next objective after completion or resumes the same objective after verified return; RECOVER issues/revises a return instruction.

We trigger review at ITF >= 0.9 or every 16 completed Actor navigation calls. Each review uses up to eight panoramas including the current view, plus at most one event-specific panorama: nine panoramas maximum. The complete review prompt is in Appendix D; image selection and execution settings are in A.1.1 and B.2.

BE-VLN. We initialize BE-VLN from Qwen3-VL-8B and supply the active instruction and latest 32 observations, including the current one, or all available observations if fewer. Each has four RGB views. We spatially average-pool historical features, retain current spatial detail, and concatenate features chronologically. We retain this recent window across instruction switches and recovery.

We train BE-VLN to predict eight waypoints and instruction-relative ITF. ITF estimates membership in the demonstration's terminal window and triggers review; the Supervisor verifies completion. Full-route, short-forward, and backtracking training share the backbone and trajectory/ITF heads. Navigation samples supervise both outputs; image/video QA uses language-modeling supervision. Architecture, losses, and optimization are in A.3-A.4.

W2. Deviation detection and anchor selection

We use HCM's task and temporal associations to select historical evidence, then apply the review-prompt rules to current and historical images.

Deviation. We initiate recovery when chronological evidence establishes a conflict between the executed route and pending objective that requires backtracking, such as entering an incorrect branch. Uncertain completion alone does not trigger recovery.

Anchor selection. We first require candidates to be visited, identifiable, supported by instruction/history as on the intended route, and suitable for resuming the objective. Among valid candidates, the VLM prompt prefers unambiguous intermediate landmarks with lower return effort along the actual traversal. A verified instruction-start boundary is the fallback when intermediate evidence is insufficient.

Return verification. We generate return instructions from the actual traversal, including its off-route portion. The Supervisor verifies landmark identity, topology, and a navigable reentry region; visibility or high ITF alone does not establish arrival. We then issue the same objective's remaining requirements, preserving its identity through return and resumption. (A.1 and D.)

W3. Efficient local prediction and supervisory review

We reduce history-processing costs at both levels: a recent window supports frequent local prediction, while HCM organizes evidence for route-level review. We evaluate their costs separately.

Executor. We compare history settings on the same 80 longest RxR val-unseen English guide episodes, with identical L40S GPUs, BF16, sensors, and action budget. CUDA-synchronized timing covers preprocessing, backbone, and heads; warm statistics exclude each service's first prediction.

MetricConventional HistoryHCM Local History
Mean/P95 prediction (ms)367.63/471.41236.26/253.95
Peak reserved memory (GiB)26.22117.680
Runtime peak allocated increment (GiB)0.79750.6543

We reduce mean/P95 latency by 35.7%/46.1% and peak reserved memory by 32.6%. Memory values are maxima across services; allocated increments subtract loaded-model baselines. (B.7.1.)

Supervisor. To test HCM's contribution, we compare the same Supervisor with Chronological History (the VLM infers task links from temporal records) and HCM (explicit links among objectives, commands, attempts, boundaries and images). Both retain the active objective, command, completion checks and recovery. On a separate 80-task random subset, we fix the VLM and executor, cached decomposition, image selection/budget, review rules and Controller safeguards.

MetricChronological HistoryHCM
SR (%)68.7580.00
Review calls14.82514.513
Prompt tokens257,133239,520
Review service time (s)254.70236.15

Cumulative review tokens/time fall 6.85%/7.28%. Per-episode means exclude initial decomposition (B.7.2). Mean recovery wall time, including all reviews, is 27.13 s (C.2).

AuthorsOpenReview ↗

Response to Questions: Visual History Management, Backtracking Generalization, and Recovery Benefits

Response to Questions: Visual History, Backtracking Generalization, and Recovery Benefits

We explain how our history management and learned local executor support recovery and completion of unfinished route objectives. W1-W3 are addressed separately.

Q1. How does HCM handle growing visual history?

We separate the complete observation archive from each Supervisor review's visual context. HCM associates observations with route objectives, command versions, execution attempts, and task boundaries to select evidence relevant to current execution and recovery.

We use up to eight panoramas including the current view, plus at most one event-specific panorama: nine panoramas maximum. We prioritize the command-issue observation, execution/recovery references, previous completed boundary, and current subtask's starting reference, then sparsely cover execution and recent boundaries. Images are deduplicated and time-ordered.

This retrieves relevant evidence even after it leaves BE-VLN's recent window. Attached panoramas provide visual evidence; unpictured references retain task/time metadata. We bound per-review visual input while retaining the growing archive and text records. (A.1.1.)

Q2. Can BE-VLN generalize to arbitrary interruption locations?

We support backtracking from the actual interruption state, including within unfinished sub-instructions. We apply trajectory and ITF supervision throughout each return demonstration, exposing BE-VLN to intermediate states under the same return objective even though demonstrations begin at segment endpoints.

At deployment, we select a visited route anchor and generate an instruction grounded in the actual interruption state and recorded traversal, including its off-route portion. BE-VLN executes it using current observations and retained recent history. (A.2.2, A.3.2-A.4.)

We evaluate 300 randomly selected RxR-CE val-unseen episodes. Eighteen episodes yield 45 natural recovery attempts, all retained, with no injected deviations or starts preset to training endpoints. Manual inspection confirms that all interventions precede genuine completion of the active forward sub-instruction. They occur after 7-63 local navigation actions, with a median of 25.

Online recovery outcomeResult
Anchor return: exit geodesic distance <= 3 m40/45 (88.9%)
Task recovery: error corrected and remaining sub-instruction requirements completed35/45 (77.8%)

We manually verify task recovery against the interrupted instruction's spatial/semantic requirements. These results demonstrate successful return and completion from naturally occurring mid-instruction states at varied interruption times. (C.1, C.3, C.5.)

Q3. Advantages over instruction-decomposition methods

SkillNav uses a VLM router to select skill experts; IDEAL-VLN uses semantic anchoring/action alignment and hierarchical correction during training. We combine learned local forward/return execution with instruction-relative termination feedback and recovery that preserves the unfinished objective.

We use HCM to distinguish the original objective from a temporary return command. Our Supervisor generates a return instruction from the actual traversal, verifies arrival at a visited route anchor, and expresses the remaining requirements from the recovered state. This restores an identifiable route reference for completing the interrupted task. (Section 3.2.)

Learned local execution. Table 2 tests the same BE-VLN checkpoint without Supervisor: 81.01%/63.38% SR on short-forward/backtracking tasks. B.8 (Table 12) isolates ReTrace-RxR's contribution: adding forward data raises short-horizon SR from 55.74% to 81.01%; adding backtracking data raises return SR from 47.80% to 63.38%, retaining all other training sources.

Recovery strategy. In a separate paired study (C.6), we test strategies from the same 18 recovery entries, fixing the pose, progress, history, models, initial evidence, and remaining action budget. Direct replanning can include necessary return motion. We measure final success at the original route goal, including return and continuation.

Continuation strategyFinal-task successes
Direct replanning8/18 (44.44%)
Fixed-boundary return12/18 (66.67%)
Adaptive-anchor return14/18 (77.78%)

These paired results support restoring a historical route reference, with adaptive anchors further improving final-task completion. (C.6, Table 20.)

Long-range navigation. Table 1 compares recovery enabled/disabled while retaining the same Supervisor, HCM, BE-VLN checkpoint, review settings, episodes, and action budget. Enabling recovery raises SR from 69.6% to 76.7% on R2R-CE and 72.3% to 78.5% on RxR-CE. Together, these experiments connect learned local capabilities to effective return, objective completion, and higher full-route success.

Dear Reviewer,

Thank you for your detailed and thoughtful review. We have now posted a full response to your comments. In particular, we would be grateful if you could kindly reconsider your evaluation of our manuscript based on our analysis and explanations. If there are any remaining questions or concerns, we would be glad to provide further clarification. We sincerely hope that our revisions have addressed your concerns and that the improvements we made will be favorably considered during your re-evaluation. Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_UULsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. ReDual-VLN addresses route deviations in long-horizon vision-and-language navigation by introducing an upper-level Route Supervisor and a lower-level BE-VLN. The upper level handles route planning, progress tracking, and recovery decisions, while the lower level executes local navigation and language-conditioned backtracking.
  2. The paper construct the ReTrace-RxR dataset to train short-horizon navigation, instruction-termination assessment, and backtracking capabilities for the lower-level model. And verify the improvement on long-range navigation and deviation recovery in both simulation and real environments.
Critical strengths
  1. ReDual-VLN proposes a dual-system arch in which the Route Supervisor handles global route understanding, task-progress tracking, and recovery decisions, while BE-VLN performs local visual execution and termination assessment. This avoids requiring a single VLN model to simultaneously manage long-term memory, route planning, and low-level action control.
  2. The method goes beyond simply detecting deviations. It also selects a historical anchor, generates a language conditioned backtracking instruction, verifies whether the agent has returned to the target location, and resumes the unfinished task, placing greater emphasis on task continuity than standard replanning or local correction methods.
  3. The proposed ReTrace-RxR dataset covers short-horizon navigation, instruction-level termination, and reverse backtracking along previously executed routes. By training a shared executor for these capabilities, it reduces the need to design separate policies for different stages of navigation.
Critical weaknesses
  1. In Ln 207, Ψreview\Psi_{review} is responsible for deviation detection, anchor selection, backtracking-instruction generation, and return verification, but the paper does not specify the model, prompt, thresholds, output format, or concrete algorithm used.
  2. Sec. 3.3.1 and Appendix A.2.2 construct only complete backtracking trajectories from a segment endpoint to its starting anchor. However, deployment allows recovery from an arbitrary intermediate point to an intermediate anchor, potentially including off-route portions. The paper does not demonstrate that this training data covers realistic mid-instruction recovery.
  3. Equ.2 provides the Supervisor with the full history Hj=(o0,…,otj)Hj=(o_0,…,o_tj), whereas Sec. 3.2.2 claims that HCM reduces the burden of long-context processing. In reality, BE-VLN’s context is bounded, the Supervisor still processes an ever-growing full visual history.
  4. Sec. 3.2 states that the Supervisor is invoked when ITF satisfies a review condition or when a periodic review is due, but it does not specify the ITF threshold or how to handle repeated backtracking failures, returning to an incorrect anchor, or repeated RECOVER decisions. Appendix A.1 still provides only a state description, without an executable state machine or termination policy.
Questions for authors
  1. The paper without the comparisons with current location-based replanning, goal-image navigation, or map-based localization methods. In complex environments, returning to historical anchors may result in longer paths and a higher likelihood of repeated errors compared to replanning.
  2. Backtracking inevitably increases path length, action counts, and supervisor invocation frequency, yet the paper fails to separately report the additional distance, time, inference overhead, and failure probability for each recovery. Looking solely at final SR/SPL makes it impossible to determine whether the benefits outweigh these costs.
  3. The paper primarily reports final navigation metrics such as SR, NE, and SPL, without separately evaluating deviation detection, anchor selection, backtracking success rate, return verification, and post-recovery task resumption. Consequently, it remains unclear whether failures stem from detecting deviations, finding anchors, or executing the backtracking itself.
  4. Sec. 3.2.1 only mentions that the supervisor verifies arrival at the anchor, but provides no criteria regarding visual matching, pose error, topological consistency, or language conditioning. Relying solely on a VLM for judgment may introduce vulnerability to high-level errors.
  5. The paper does not explicitly specify whether deviations were naturally produced by the model or artificially injected. Furthermore, the timing, distance, and difficulty distribution of the occurrences are equally important, if the tested deviation patterns resemble the training backtracking trajectories, recovery performance may be overestimated.
3 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses: Recovery implementation, interruption-state generalization, and HCM efficiency

Supervisor Protocol, Mid-Instruction Recovery, and HCM

Thank you for the feedback. We preserve the unfinished route objective during history-grounded return, verify arrival, and complete its remaining requirements from the recovered state.

W1. Supervisor implementation

We implement the Supervisor as a prompt-guided VLM using gpt-5.6-sol. Each review receives the original instruction, route decomposition, HCM state, selected visual evidence, and ITF. We return supporting observations, a CONTINUE/ADVANCE/RECOVER/STOP decision, the route-objective identifier, and a local instruction when needed. We trigger review at ITF ≥0.9 or every 16 completed Actor navigation calls.

We detect deviations by comparing observed spatial transitions with the unfinished objective. We select an identifiable visited anchor on the intended route that supports continuation, preferring unambiguous intermediate anchors with low return effort along the observed traversal. A verified instruction-start boundary is the fallback when intermediate evidence is insufficient. We generate return instructions from the actual traversal, verify landmark identity, topology, and a navigable reentry region, then express the same objective's remaining requirements. Appendix D provides our complete review prompt and Controller transition rules. (A.1, B.2.)

W2. Mid-instruction and intermediate-anchor recovery

Although return demonstrations begin at segment endpoints, we train a language- and history-conditioned policy with trajectory and ITF supervision throughout each return. Intermediate states are therefore supervised under the same anchor objective. At deployment, we select a visited anchor and generate a return command from the actual interruption state and traversal, including observed off-route portions. BE-VLN executes using current RGB and retained recent history. (A.2.2, A.3.2.)

We evaluate 300 randomly selected RxR-CE val-unseen episodes, each run once. Eighteen episodes yield 45 natural recovery attempts. Independent manual inspection confirms that all 45 interrupt genuinely unfinished forward instructions.

Recovery subsetAttemptsAnchor returnInterrupted objective completed
Mid-instruction4540/4535/45
Within-segment anchors4439/4435/44
Outside executor history63/62/6

We measure anchor return by exit geodesic distance ≤3 m and manually verify completion of the interrupted objective's remaining spatial and semantic requirements. These subsets are nested. Only one attempt returns to the overall route start; the other 44 use within-segment intermediate anchors. (C.1, C.3–C.4.)

For all six outside-history cases, HCM supplies the historical anchor image to the Supervisor, which translates it into a return instruction for BE-VLN. Trajectory and landmark review confirms arrival in 5/6 cases; 3/6 satisfy the exit-pose criterion and 2/6 complete the objective. Other cases involve landmark/continuation errors or budget exhaustion after arrival. These cases connect historical-reference retrieval beyond the 32-observation window to language-conditioned return and completion of the preserved objective. (C.4.)

W3. Visual-history input and efficiency

We retain the archive and link images to objectives, commands, attempts and boundaries. We prioritize current, command-issue and recovery images before sparse execution samples, using at most eight base panoramas plus one event panorama. Archive and text records grow with history (A.1.1).

Executor. With BE-VLN's 32-observation window and pooled history, mean warm prediction latency falls 367.63→236.26 ms and incremental peak allocated memory 0.7975→0.6543 GiB on 80 long-route episodes (B.7.1).

Supervisor. We compare Chronological History versus HCM within the same Supervisor to test HCM's contribution. With Chronological History, the VLM infers task links from temporal records; HCM supplies them explicitly. Both retain task state, completion checks and recovery. On a separate 80-task subset, we fix models, cached decomposition, visual selection/budget and review rules. Mean per-episode cumulative review tokens fall 257,133→239,520, service time 254.70→236.15 s, and SR rises 68.75→80.00% (B.7.2).

W4. Repeated recovery and termination

We share a 350-environment-action episode budget across forward execution and recovery. CONTINUE retains an effective command. RECOVER initiates return or revises an ineffective executed return leg, preserving the anchor and unfinished objective. We execute and review until verified arrival or budget exhaustion. After verified return, ADVANCE issues the same objective's remaining requirements without changing its index. We advance the index only after verifying objective completion and issue STOP after satisfying the final route requirements. This preserves the original task through repeated reviews and return-command revisions. (A.1, B.2, D.)

AuthorsOpenReview ↗

Response to Questions: Historical-Anchor Recovery: Comparisons, Costs, and Outcomes

Thank you. We connect history-grounded return to original-task completion through controlled continuation results, recovery costs, and decision audits.

Q1. Replanning, image-goal navigation, and map-based localization

We restore a recognizable route reference, then express the unfinished requirements. On 18 paired natural recovery entries, we match pose, progress, history, evidence, models, and remaining budget. Direct replanning permits necessary return motion; fixed return uses the preceding completed boundary; adaptive return selects a route-consistent historical anchor.

StrategyFinal-task SRActionsTravel
Direct replanning44.44%154.11123.053 m
Fixed-boundary return66.67%150.22221.478 m
Adaptive-anchor return77.78%142.94421.165 m

We evaluate final STOP at the original goal with geodesic error <3 m. Including return motion and failures, we increase final-task SR from 44.44% to 77.78% while reducing mean actions and travel in the matched comparison. (C.6.)

Map-based methods localize in an explicit map. We retrieve the observed anchor and access route from RGB history in a map-free setting. Image-goal navigation may require exploration when destination/access is unknown; our return uses a visited reference. The Supervisor converts historical evidence into language for BE-VLN to execute from current RGB and recent history, without an extra goal image or coordinates. (A.1, B.3, D.)

Q2. Per-recovery cost and failure probability

We measure all 45 natural attempts from initiating RECOVER through accepted return exit or termination. Cumulative Supervisor costs include initiation, intermediate reviews, and exit verification.

Mean per attemptValue
Return travel3.71 m
Execution actions29.84
Initial RECOVER response5.39 s
VLN inference and simulated execution4.27 s
Initial RECOVER tokens15,241
Supervisor calls3.07
Cumulative Supervisor service time18.45 s
Cumulative Supervisor tokens (total)48,904
Complete interval wall time27.13 s

We complete the interrupted objective in 35/45 attempts; 10/45 (22.2%) remain unrecovered. This analysis measures recovery overhead and local completion; C.6 separately tests final-task completion and efficiency under paired controls. (C.2–C.3, C.6.)

Q3. Recovery stages and deviation-detection audit

Across 45 attempts, we obtain 40 geometric anchor returns, initiate resumption in 42, and complete the remaining spatial/semantic requirements in 35, verified manually. We assign one primary cause per unrecovered attempt: detection 1, anchor selection 2, backtracking execution 0, verification 2, and objective completion 5. This separates local return performance from high-level completion decisions. (C.3.)

We audit 1,852 reviews from a separate random sample of 180 episodes, including 162 without backtracking. We find persistent missed deviations in 3/180 episodes (1.67%) and an unnecessary initiation in 1/180 (0.56%). This extends our evaluation beyond triggered recoveries to episodes where a needed intervention could be missed. (C.7.)

Q4. Return verification and high-level judgments

We combine BE-VLN's instruction-relative ITF with route-level visual verification. ITF supplies terminal-window feedback and triggers review, alongside periodic checks. We compare current views with the historical anchor and language requirements, checking landmark identity, topology, and a navigable reentry region. After verified arrival, we issue the same objective's remaining requirements and assess completion through subsequent execution.

In outside-history cases, we identify one mistaken landmark and one wrong continuation after correct arrival. We use the 3 m geometric criterion for retrospective evaluation. (A.1, A.4.2, C.3–C.4, D.)

Q5. Natural deviations and state coverage

We collect 45 attempts from our execution of 300 randomly selected RxR-CE val-unseen episodes, one run each, without injected deviations or preset endpoints. Manual inspection confirms interruption before instruction completion.

We intervene after 7–63 local actions (median 25). Expert-route distance has median 1.95 m and range 0.00296–10.95 m; straight-line anchor span has median 2.92 m and range 0.1143–8.96 m. The action count records execution before intervention; the distances describe reference-route proximity and anchor span.

We synthesize training returns from expert traversals and test naturally encountered online states, with scene- and source-route-disjoint ReTrace-RxR local splits. Our per-state return supervision supports recovery at varied interruption states and online-selected intermediate anchors. (A.2.2–A.2.3, C.1, C.4–C.5.)

Dear Reviewer,

Thank you for your detailed and thoughtful review. We have now posted a full response to your comments. In particular, we would be grateful if you could kindly reconsider your evaluation of our manuscript based on our analysis and explanations. If there are any remaining questions or concerns, we would be glad to provide further clarification. We sincerely hope that our revisions have addressed your concerns and that the improvements we made will be favorably considered during your re-evaluation. Thank you again for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: From Route Recovery to Task Completion

Dear Area Chair,

We sincerely thank you and the reviewers for your time and constructive feedback. Through the discussion, we have strengthened the experimental evidence and clarified the central contribution of ReDual-VLN: recovering an unfinished navigation task through learned return execution, task-linked history management, and verified resumption of the original objective. We would like to highlight how our results substantiate this contribution.

We first establish the local capabilities required for recovery. With ReTrace-RxR, we align forward and return instructions with direction-specific observations, motion, and instruction-relative terminal boundaries, enabling one executor to perform both tasks. Under matched backbone and training-budget settings, adding our forward data raises short-horizon SR from 55.74% to 81.01%, while adding our return data raises backtracking SR from 47.80% to 63.38% (Appendix B.8). Our fixed-checkpoint analysis further isolates a 4.88-point SR improvement from selecting ITF as the inference-time stopping criterion (Appendix B.9). These experiments substantiate the contribution of our local-task training and termination feedback.

We then connect these capabilities to task continuity through HCM. During recovery, we preserve the unfinished route objective separately from the temporary return command. After verifying arrival at a suitable anchor, we issue the remaining requirements of that same objective. To isolate HCM’s contribution, we fix the Supervisor, executor, decomposition, image-selection policy and budget, and review and recovery rules on 80 tasks. Replacing chronological history with HCM raises SR from 68.75% to 80.00%, while reducing cumulative review tokens and service time by 6.85% and 7.28%, respectively (Appendix B.7.2). This comparison demonstrates the value of explicitly linking objectives, commands, execution attempts, and historical evidence.

We evaluate this process on naturally occurring execution errors. Across 300 randomly selected RxR-CE val-unseen episodes, we analyze all 45 recovery attempts from 18 episodes, without injecting deviations or presetting interruption states. Every attempt begins before genuine completion of the active instruction, and 44 use intermediate anchors. We obtain 40/45 geometric anchor returns and 35/45 completions of the interrupted instruction’s remaining spatial and semantic requirements (Appendices C.1–C.4). To examine decisions beyond triggered recoveries, we additionally audit 1,852 reviews across 180 episodes, including 162 without backtracking. Persistent missed deviations affect three audited episodes, and unnecessary recovery initiation affects one (Appendix C.7). We also provide a detailed analysis of all six outside-history cases, distinguishing anchor arrival from subsequent objective completion.

Crucially, we connect recovery to completion of the original navigation goal. In a separate paired comparison of 18 recovery entries, we match the starting pose, progress, history, models, initial visual evidence, and remaining action budget. Our adaptive-anchor strategy completes the original goal in 14/18 cases, compared with 12/18 for fixed-boundary return and 8/18 for direct replanning, while requiring the lowest mean actions and travel when return motion and failures are included (Appendix C.6). At the full-route level, enabling recovery with the Supervisor, HCM, executor checkpoint, review settings, and action budget fixed raises SR from 69.6% to 76.7% on R2R-CE and from 72.3% to 78.5% on RxR-CE (Table 1).

We are encouraged by the substantive follow-up discussions with Reviewers xiTm, VpP2, and 9Cas. Reviewer xiTm confirmed that our new audit addresses their main concern about missed deviations and unnecessary recovery and raised their score from 2 to 3. We also appreciate that Reviewer VpP2 explicitly expressed a willingness to raise their score if their concerns were addressed and subsequently confirmed, after carefully reading our responses, that most concerns had been addressed, with only a few minor questions remaining. We further appreciate Reviewer 9Cas’s acknowledgment that our implementation clarifications and new HCM and training-data ablations address much of what they raised in their original review and improve the paper. We have incorporated the additional analyses and clarifications into the revised manuscript, together with the complete prompts, execution protocol, training specifications, and scene- and source-route-disjoint dataset splits.

We believe these controlled experiments make a strong case for our central contribution: connecting learned return execution and structured historical context to successful continuation and completion of the original navigation task. We respectfully ask for your favorable consideration of our paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.