acceptodds
Under review as a conference paper at ICLR 2027

RCPNav: Reversible Cruise–Precision Control with Keyframe Memory and Landmark Graphs for Aerial Vision-and-Language Navigation

Abstract

Aerial vision-and-language navigation requires a UAV to use onboard visual observations to locate a destination described in natural language, with applications in urban inspection and search. Reaching the destination requires efficient long-distance flight as well as accurate localization near the target. A one-way coarse-to-fine controller may switch prematurely to local search and cannot return to cruise even if the target remains distant. Long flights also make it difficult to retain useful past observations and reason about spatial relations among named landmarks. We propose RCPNav, which combines three components: a reversible cruise–precision controller that can return to cruise when the target is estimated to be farther away; selective keyframe memory that retrieves earlier views conditioned on the instruction; and a language-grounded landmark graph that organizes named places to support goal prediction. On all three evaluation splits of the refined CityNav benchmark, RCPNav achieves the highest success rate and success weighted by path length among the compared methods. Ablation experiments with separately retrained variants show that all three components improve navigation performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Aerial Vision-and-Language Navigation, Reversible Cruise–Precision Control, Historical Keyframe Memory, Language-Grounded Landmark Graph, Long-Horizon UAV Navigation, Geographic Semantic Map

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7cmqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes RCPNav for aerial vision-and-language navigation. The framework combines a reversible cruise–precision controller, selective keyframe memory, and a language-grounded landmark graph to support long-range navigation and near-target localization. Experiments on CityNav show improvements in success rate and path efficiency.

Critical strengths

RCPNav provides a coherent framework that combines current observations, historical visual evidence, and landmark relations.

The reversible controller addresses premature switching from long-range navigation to local search.

The method achieves strong results on CityNav, and the separately retrained ablations show that all three components contribute to performance.

Critical weaknesses

Limited methodological novelty. The proposed framework mainly combines heuristic mode switching, keyframe memory, landmark-graph encoding, and multimodal fusion. These components are largely standard, making the contribution primarily an engineering integration.

Insufficient justification of the design choices. The experiments do not compare the proposed components with simpler alternatives, so the necessity of the reversible controller, Top-2 keyframe selection, and landmark graph remains unclear.

Limited experimental generalization. The evaluation is restricted to the original and refined versions of CityNav, which are variants of the same benchmark. The ablations are conducted only on one validation split, without statistical variance or significance analysis.

Limited robustness analysis. The method relies heavily on progress estimation, manually selected switching thresholds, and annotated landmarks, but its sensitivity to inaccurate progress predictions, missing landmarks, or different threshold settings is not evaluated.

Questions for authors

See Weaknesses.

11 comments

We thank the reviewer for the detailed comments. We clarify how RCPNav differs from existing approaches and provide additional controlled experiments supporting our design choices.

W1. Methodological Novelty

RCPNav addresses when to switch between global navigation and local search during long-range aerial navigation. Underestimating the distance to the destination can trigger local search prematurely, and subsequent observations can change the estimated goal location. RCPNav allows the agent to resume cruise as its estimate changes and uses past visual observations and spatial relations among landmarks to refine its estimate of the destination.

Reversible cruise–precision control. HETT (Ding et al., 2026) predicts a coarse goal location and then refines local actions. HTNav (Fan et al., 2026) generates the next waypoint once the agent reaches its current subgoal. RCPNav continues to update its estimated distance to the final destination during precision control. If the updated estimate indicates that the destination is still far away, the agent can resume cruise from its current position, recovering from a premature switch to local search.

Historical memory. HETT aggregates past visual features in a spatial grid, and OpenFly-Agent (Gao et al., 2026) updates its keyframe memory in first-in, first-out order. RCPNav retains two keyframes based on the visibility of landmarks mentioned in the instruction. The agent can therefore use relevant visual evidence after those landmarks leave its current field of view.

Landmark graph. GeoNav (Xu et al., 2026) uses dynamic cognitive maps and hierarchical scene graphs, and LookasideVLN (Ning et al., 2026) models directional relationships between landmarks. RCPNav weights messages by the distance between landmarks and uses an instruction-conditioned readout to combine spatial cues from multiple landmarks when predicting the destination.

W2. Justification of Design Choices

We expanded Section 4.3 to compare simpler alternatives for all three components on refined CityNav val-unseen. Table 2 reports the controller and memory controls, and Table 3 reports the graph controls with and without auxiliary supervision.

(1) Controller and Historical Memory. Cruise only disables Precision; No return to Cruise retains both modes but prohibits the return transition; No history removes stored keyframes; and FIFO-2 (First-In, First-Out) replaces the oldest stored frame in a full two-frame cache rather than selecting frames by score.

ConfigurationNE↓SR↑OSR↑SPL↑
Cruise only49.318.034.115.9
No return to Cruise52.120.232.919.0
No history50.719.234.117.4
FIFO-252.419.831.819.2
RCPNav48.521.334.019.9

Compared with No return to Cruise, reversible control improves SR by 1.1 points and SPL by 0.9 points. Compared with FIFO-2, selective memory improves SR by 1.5 points, OSR by 2.2 points, and SPL by 0.7 points, while reducing NE by 3.9 m.

These results support both reversible mode switching and selective keyframe retention over simpler alternatives.

(2) Landmark Graph. Self-only replaces the adjacency matrix with A=IA = I, disabling cross-node message passing while retaining landmark inputs and instruction-conditioned readout. We compare it with the full graph under two auxiliary-loss settings.

Graph VariantNE↓SR↑OSR↑SPL↑
Full graph, λ=0.148.521.334.019.9
Self-only, λ=0.150.820.337.517.7
Full graph, λ=050.020.436.317.1
Self-only, λ=049.218.236.416.1
No landmark graph53.918.528.618.2

With auxiliary supervision, the full graph improves SR by 1.0 point and SPL by 2.2 points over Self-only. Without auxiliary supervision, it still improves SR by 2.2 points and SPL by 1.0 point.

The gains in SR and SPL hold under both settings. The effect on NE depends on the loss setting, as discussed in Section 4.3.

W3. Experimental Generalization

We selected CityNav because it directly evaluates the task RCPNav is designed for: locating language-described destinations using named landmarks and geographic information. Closely related methods, including HETT (Ding et al., 2026) and HTNav (Fan et al., 2026), report results on both original and refined CityNav, enabling comparisons under a common task setting.

Other aerial VLN benchmarks differ in their navigation objectives, instruction formats, and environments:

CityNav (Lee et al., 2025) provides goal descriptions grounded in nearby landmarks and geographic relationships. Its CityFlight simulator renders real-world 3D point clouds of Cambridge and Birmingham.

AerialVLN (Liu et al., 2023) uses human-written, step-by-step flight instructions aligned with trajectory segments in Unreal Engine 4 and AirSim environments.

OpenFly (Gao et al., 2026) provides automatically generated route instructions and trajectories across Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting environments.

UAV-Need-Help (Wang et al., 2025) evaluates target-oriented navigation with different levels of assistant guidance in continuous six-degree-of-freedom UAV simulation.

Section 4.1 describes both CityNav versions. We evaluate navigation performance in scenes unseen during training using CityNav’s val-unseen and test-unseen splits. In the revision, Table 1 adds GeoNav on original CityNav, and Tables 2 and 3 expand the ablations on refined CityNav val-unseen.

W4. Robustness to Progress Estimates, Switching Thresholds, and Landmark Annotations

We report additional sensitivity experiments below on refined CityNav val-unseen. Section 3.3 describes the default switching and STOP rules, Section 3.4 describes the two-frame memory, and Section 4.1 specifies the landmark annotation protocol.

(1) Switching and STOP Thresholds. RCPNav uses three Progress thresholds for entering Precision, returning to Cruise, and STOP confirmation. The default values are 0.80/0.70/0.90, with STOP requiring two consecutive predictions meeting its threshold. We vary these thresholds while keeping the other components unchanged.

Entry / Return / STOPNE↓SR↑OSR↑SPL↑
0.80 / 0.70 / 0.90 (Default)48.521.334.019.9
0.70 / 0.60 / 0.9052.819.632.618.0
0.90 / 0.80 / 0.9053.217.731.615.9
0.80 / 0.70 / 0.8552.420.130.718.9
0.80 / 0.70 / 0.9553.321.538.519.5

The default thresholds achieve the lowest NE (48.5 m) and highest SPL (19.9%). Lowering or raising the mode-switching thresholds reduces SR and SPL. Increasing the STOP threshold to 0.95 slightly improves SR and OSR but increases NE and reduces SPL. Overall, the default setting provides a favorable balance between final localization accuracy, navigation success, and path efficiency.

(2) Historical Memory Capacity. We further evaluate different numbers of retained historical keyframes KK, using the same keyframe selection and readout mechanisms.

Historical FramesNE↓SR↑OSR↑SPL↑
K=154.119.532.317.9
K=2 (Default)48.521.334.019.9
K=450.722.335.620.6

Increasing K from 1 to 2 improves all four metrics. Increasing K to 4 further improves SR by 1.0 percentage point, OSR by 1.6 points, and SPL by 0.7 points, although NE increases from 48.5 m to 50.7 m.

Moreover, K=4 doubles the maximum number of retained historical frames and memory tokens compared with K=2, increasing historical visual processing, storage, and attention computation. We therefore adopt K=2 as a compact performance–cost trade-off, achieving the lowest NE while maintaining competitive navigation success and path efficiency.

(3) Landmark Annotations. The original CityNav pipeline (Lee et al., 2025) uses GPT-3.5 to extract landmark names from navigation instructions and retrieves corresponding geographic regions from OpenStreetMap. RCPNav constructs its landmark graph using the named landmarks and geographic footprints provided by CityRefer.

HETT (Ding et al., 2026) manually refined the CityNav annotations by restoring missing landmark references and correcting erroneous associations. HTNav (Fan et al., 2026) also reports manual inspection and correction of missing annotations, extraction errors, and misspelled landmark names.

As described in Section 4.1, we train on HETT-refined CityNav, which corrects landmark annotations and excludes 311 episodes. For retained episodes, the instructions and demonstration trajectories are unchanged. Table 1 reports results on both versions.

Reviewer_yHvrOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RCPNav for aerial vision-and-language navigation. The main idea is to define two modes: cruise mode for long-range navigation and precision mode for local navigation. RCPNav makes this cruise-precision navigation reversible. Specifically, it allows the navigation agent to return to cruise mode when a previous switch to precision mode turns out to be premature. In the experiments, RCPNav achieves strong improvements over prior methods on both the original CityNav and the refined CityNav benchmarks, particularly in terms of SR and SPL.

Critical strengths
  1. The reversible cruise-precision control is an impressive and promising idea. It provides a simple way to recover from premature switching to local navigation. Ablation study shows that this component gives the largest performance gain among the proposed modules.

  2. The method achieves clear performance improvements over prior approaches. In particular, RCPNav obtains strong SR, SPL, and NE results across the evaluation splits, supporting the effectiveness of the proposed framework.

  3. The evaluation is comprehensive. Experiments are conducted on both the original CityNav benchmark and the refined version, which helps demonstrate that the improvements are consistent across different dataset settings.

Critical weaknesses
  1. More qualitative examples are needed. The current examples mainly demonstrate cruise-precision switching, but it would be useful to show how each module works in practice.

  2. The failure mode analysis is also limited. The paper would benefit from examples and statistics showing typical failure cases, such as incorrect progress estimation, and wrong mode switching.

Minor weakness: The symbol ⊮ may be an artifact generated by LLM-based writing. The authors should carefully check this notation and revise it to "1" if that is the intended symbol.

Questions for authors

Could the authors provide more qualitative examples and failure mode analysis?

1 comment

We thank the reviewer for the positive assessment and helpful suggestions. We provide additional qualitative examples and a systematic failure analysis.

W1. Qualitative Examples

We added two qualitative examples to Section 4.3 to show what each module contributes in a navigation episode. The captions include the instruction excerpts.

(1) Historical Memory (Figure 3). At t=19t = 19, the instruction-referenced landmark Bin Brook is no longer visible in the current observation but remains visible in the two retained historical frames from t=11t = 11 and t=12t = 12. This illustrates how selective memory preserves task-relevant visual evidence from earlier viewpoints.

(2) Landmark Graph (Figure 4). We mark the five landmarks mentioned in the instruction and compare trajectories for the full graph and Self-only. In this episode, the full graph ends closer to the target.

W2. Failure Analysis

We added Section 4.5 to analyze all 2,122 failed trajectories on refined CityNav val-unseen. Most failures occurred before the agent reached the goal region.

(1) Failure categories. Among these trajectories, 1,779 (83.8%) never entered the 20 m goal region, while 343 (16.2%) entered it but finished outside. Across all failures, 1,593 issued STOP outside the goal region, and 529 exhausted their action budget.

(2) Goal localization. For each trajectory, we take the median Euclidean goal-prediction error across decisions. The median of those trajectory-level errors was 53.8 m for failures and 20.4 m for successes.

(3) Mode switching and stopping. In 1,791 failed trajectories, the agent entered Precision more than 20 m from the true goal. Among the 343 failures that entered the goal region, 269 (78.4%) continued moving at the second of two consecutive decisions within it.

These results identify inaccurate goal localization, early mode transitions, and missed stopping opportunities as important failure patterns.

Minor Concern. Indicator Notation

We revised Eq. (9) in Section 4.1 to use 1[·] for both the terminal-success and oracle-success indicators.

Reviewer_eTKeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RCPNav addresses language-goal-driven UAV navigation. It targets two problems: entering local navigation too early makes it hard to recover long-range flight, and historical visual information and landmark relations are underused. The method uses ModernBERT, DINOv3, and a shared multimodal Transformer to fuse the instruction, the current image, the pose, a semantic map, a two-frame history memory, and a landmark graph. From these inputs, it predicts the global goal position, the local direction, and how close the UAV is to the goal. The controller uses these predictions to switch in both directions between a Cruise mode and a Precision mode. The history module picks keyframes based on turning and landmark visibility. The landmark graph supplies geographic context through distance-weighted message passing and language attention. On the test-unseen split of the corrected CityNav, the method reaches 29.98% SR and 28.04% SPL, outperforming the methods compared in the main table. Ablation models, each retrained separately, also support the overall contribution of the three components.

Critical strengths
  1. The history memory adds past visual evidence through a fixed number of tokens, which improves navigation while keeping temporal causality intact.
  2. The method achieves higher SR and SPL on all three evaluation splits of both the original and corrected CityNav, and the contribution of each of the three components is checked through separate retraining.
  3. Reversible mode switching gives the agent a way to recover after entering precision navigation too early, and removing it causes the largest performance drop among the three components.
Critical weaknesses
  1. The ablation does not isolate the contribution of allowing a return to Cruise from other factors. The count of returns also cannot show that the mechanism successfully corrected premature switches.

  2. New keyframes qualify for writing only when heading changes by 15°, which may skip important observations during straight flight. Current experiments do not verify that this filtering rule outperforms simple history caching.

  3. Landmark-graph edges explicitly encode only spatial proximity. The ablation does not separate the contribution of relational modeling from that of auxiliary landmark information and the additional supervision signal. The explanation that it improves spatial-relation reasoning is therefore not yet sufficiently supported.

  4. The compared methods do not include memory-related baselines.

Questions for authors

see weakness

5 comments

We thank the reviewer for the constructive comments. We provide additional controlled comparisons and analyses to clarify the contributions of reversible control, historical memory, and landmark-graph aggregation.

W1. Contribution of Reversible Control

Allowing the agent to return from Precision to Cruise improves navigation even when both modes are available. We expanded Section 4.3 and added two controls to Table 2. Cruise only disables Precision. No return to Cruise retains both modes and disables only the return transition. RCPNav allows both transitions.

The comparison uses refined CityNav val-unseen.

ControllerNE↓SR↑OSR↑SPL↑
Cruise only49.318.034.115.9
No return to Cruise52.120.232.919.0
RCPNav48.521.334.019.9

Compared with No return to Cruise, RCPNav improves SR by 1.1 percentage points, SPL by 0.9 points, and reduces NE by 3.6 m, demonstrating the benefit of reversibility beyond simply having two navigation modes.

Table 4 reports 72, 228, and 308 Precision-to-Cruise return events on refined CityNav val-seen, val-unseen, and test-unseen, respectively. Figure 5 shows an early entry into Precision followed by a return to Cruise and a second entry into Precision near the destination.

W2. The 15° Threshold and Historical Memory Selection

The 15° yaw-change threshold filters out candidates with similar viewing directions, which can carry overlapping evidence. Removing the threshold lowers SR from 21.3% to 19.5% and SPL from 19.9% to 18.3% on refined CityNav val-unseen.

We added a FIFO-2 baseline to Table 2 and expanded Section 4.3 to compare retention at the same two-frame capacity. FIFO-2 replaces the oldest frame when a new frame is written to a full cache. No history removes stored frames. The additional Without 15° threshold experiment reported here removes the yaw-change requirement for admitting a candidate and retains score-based selection.

All four configurations use refined CityNav val-unseen.

Memory StrategyNE↓SR↑OSR↑SPL↑
No history50.719.234.117.4
FIFO-252.419.831.819.2
Without 15° threshold54.719.530.818.3
RCPNav48.521.334.019.9

Removing the 15° threshold also increases NE by 6.2 m. Compared with FIFO-2, RCPNav improves SR, OSR, and SPL by 1.5, 2.2, and 0.7 percentage points, respectively, and reduces NE by 3.9 m.

W3. Landmark-Graph Aggregation and Auxiliary Supervision

The full graph improves SR and SPL with and without the auxiliary loss. We added Table 3 to report both comparisons in Section 4.3. Self-only sets the adjacency matrix to the identity (A = I), retaining landmark inputs and instruction-conditioned readout while disabling cross-node message passing.

The comparison uses refined CityNav val-unseen.

Graph VariantNE↓SR↑OSR↑SPL↑
Full graph, λ=0.148.521.334.019.9
Self-only, λ=0.150.820.337.517.7
Full graph, λ=050.020.436.317.1
Self-only, λ=049.218.236.416.1

With auxiliary supervision, the full graph improves SR by 1.0 point and SPL by 2.2 points over Self-only. Without auxiliary supervision, the improvements remain 2.2 and 1.0 points, respectively.

Section 4.3 also reports that the full graph has lower NE with auxiliary supervision (48.5 m vs. 50.8 m) and higher NE without it (50.0 m vs. 49.2 m).

W4. Comparison with History-Aware Navigation Methods

We independently reproduced GeoNav (Xu et al., 2026, Pattern Recognition) on original CityNav and added its results for all three splits to Table 1 in Section 4.2, alongside the existing HETT baseline.

The methods use different historical representations: HETT aggregates previous visual features into a Historical Grid Map; GeoNav maintains dynamically updated cognitive maps and hierarchical scene graphs; and RCPNav selectively retains two historical keyframes and extracts instruction-relevant visual evidence.

Results on original CityNav:

SplitMethodNE↓SR↑OSR↑SPL↑
Val-UnseenHETT62.117.525.114.5
GeoNav62.816.533.410.3
RCPNav60.619.231.317.8
Test-UnseenHETT72.923.039.317.0
GeoNav74.323.837.816.2
RCPNav70.625.037.123.4

RCPNav achieves the lowest NE and highest SR and SPL among these methods on both unseen splits. Compared with GeoNav, it improves SR by 2.7 and 1.2 points and SPL by 7.5 and 7.2 points on Val-Unseen and Test-Unseen, respectively.

RCPNav achieves these results with two selectively retained historical keyframes. GeoNav has higher OSR on both unseen splits, and HETT also has higher OSR on test-unseen.

Thanks for the additional experiments. A few concerns remain.

  1. Without the 15° threshold, score-based selection appears to underperform FIFO-2. Could you clarify whether the gain mainly comes from the yaw-diversity rule rather than the selection mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.