acceptodds
Under review as a conference paper at ICLR 2027

RefineFly: Failure-Aware Post-Training for Aerial Vision-Language Navigation

Abstract

Unmanned aerial vehicle vision-language navigation (UAV-VLN) requires agents to translate visual observations and language instructions into reliable flight actions in complex environments. Although recent end-to-end UAV vision-language-action (UAV-VLA) policies reduce reliance on separately designed perception, planning, and control modules, their behavior-cloning objectives provide limited supervision for errors arising during closed-loop execution. Reinforcement learning (RL) offers a promising solution, while limited interaction budgets, long-tailed scene distributions, and policy drift complicate sustained learning. To this end, we propose RefineFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Built on token-level proximal policy optimization (PPO), RefineFly maintains dynamic failure memory within a two-stage scene curriculum to sustain learning from unresolved tasks as training shifts from empirical to balanced scene sampling. Throughout this process, stage-specific reference regularization constrains policy deviation. Experiments on the TravelUAV benchmark demonstrate that RefineFly outperforms all comparison methods across all three splits, improving success rate over AerialVLA by 3.12 to 8.37 percentage points with a total rollout budget of about 30% of the training-set size. Moreover, ablations reveal that the benefits of sustained failure learning vary across evaluation splits, while the two-stage scene curriculum improves overall performance, highlighting the importance of training distributions when learning from failures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
UAV VLN, VLA, post-training, reinforcement learning, embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P1UmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RefineFly, an online reinforcement-learning post-training framework for end-to-end aerial vision-language-action policies. Starting from AerialVLA, it combines token-level PPO, a dynamic memory of unsuccessful task initializations, a two-stage curriculum shifting from empirical to balanced scene sampling, and stage-specific reference-policy KL regularization. Memory-selected tasks generate fresh trajectories using the current policy. Evaluation on TravelUAV reports success-rate improvements over AerialVLA of 8.37, 5.39, and 3.12 percentage points on Seen, Unseen Map, and Unseen Object splits, respectively. Ablations investigate the components, sampling schedules, and token-level versus sequence-level PPO.

Critical strengths

I liked the consistent improvements over AerialVLA across all three evaluation splits (Tables 1–3), including success rate, navigation error, oracle success rate, and SPL. The gains also extend to harder trajectories: success rate on Seen-Hard increases from 46.30% to 56.38%. These results support the value of online post-training for improving an existing end-to-end aerial policy while keeping the same action interface.

I also found the analysis of failure-focused training and generalization useful. The authors show that adding failure memory alone improves performance on Seen and Unseen Object but substantially hurts performance on Unseen Map. Appendix B provides paired task-level comparisons, and Table 5 includes a useful two-stage baseline that keeps the original scene distribution. Together, these experiments help explain how failure-focused training and scene sampling affect generalization.

Critical weaknesses

My main concern is that the contribution of failure memory is unclear in the full framework. In Table 4, adding memory to token-level PPO changes average success rate from 49.17% to 49.07% and reduces Unseen Map success rate from 42.90% to 31.21%. Adding KL regularization and the curriculum improves the results, but the paper does not test the same PPO, KL, and two-stage curriculum without failure memory. This comparison is needed to understand whether memory contributes to the final gains or whether they mainly come from regularization and scene rebalancing. A comparison with existing task-replay methods, such as Prioritized Level Replay, would also help clarify what the proposed memory lifecycle adds.

I would also like clearer evidence on training stability and whether the comparisons use the same budget. Tables 1–3 report three evaluation seeds, and the ablations use one evaluation seed, but variation across independent training runs is not reported. The baseline results are also taken from previous publications rather than evaluated under the same repeated-evaluation setup. Appendix A explains the approximate 31.25% rollout budget, including repeated executions, but it remains unclear whether this includes collecting the initial failures and whether all ablations use the same rollout and optimizer-update budgets. These details matter because the paper studies sampling strategies under a limited interaction budget, and some comparisons use different numbers of stages.

Minor comment: The performance claim in the abstract and conclusion could be more precise. RefineFly achieves the best SR and SPL across all three splits, but LongFly performs better in some NE and OSR comparisons. For example, on Unseen Object–Hard, LongFly achieves NE of 57.07 versus 62.34 and OSR of 74.16% versus 66.43%. It also achieves higher OSR on Seen-Full and Seen-Easy. Section 4.2 explicitly acknowledges the Unseen Object–Hard exceptions, and the abstract and conclusion could specify which metrics the overall performance claim refers to.

Questions for authors
  1. Could you report results for the full framework without failure memory, while keeping token-level PPO, both curriculum stages, the reference-policy schedule, and the same interaction and optimizer-update budgets? Also, how does the proposed memory compare with a simpler task-replay or task-prioritization baseline under the same setup? These comparisons would help me understand what failure memory adds.

  2. Could you clarify the total number of executed episodes, environment steps, and optimizer updates for each configuration in Tables 4–5, including their allocation across stages? Does the interaction budget include collecting failures to initialize memory? Also, how are reference-policy resets and learning-rate schedules handled when comparing one-stage and two-stage training?

  3. How many independent post-training runs were used for the reported results? Could you compare the full method and the version without failure memory across multiple training seeds, and evaluate the initial AerialVLA checkpoint using the same evaluation seeds and decoding settings? This would help determine whether the improvements are consistent across training runs.

2 comments
AuthorsOpenReview ↗

Response to Reviewer P1Um (Part 1/2)

We sincerely thank Reviewer P1Um for the thorough assessment and insightful comments. We provide budget logs and baseline re-evaluations below, and have added the memory and replay ablations, multi-seed runs, and metric clarifications to the revised manuscript.

Q1 (W1 & Q1): Failure-memory contribution, Unseen Map trade-off, and PLR

A1: Thank you for raising this question. We evaluated RefineFly w/o Memory under identical token-level PPO, two-stage curriculum, reference resets, and budgets, alongside Prioritized Level Replay (PLR) in Stage I under ID 3 settings (given limited rebuttal time), as shown below.

MethodMemoryRef. KLCurriculumSeenUnseen MapUnseen Obj.Avg.
AerialVLA---47.9637.5856.6047.38
Memory Only (ID 2)Ours××56.2131.2159.7849.07
PLR (Stage I)PLR✓×45.9135.4951.6744.36
RefineFly Stage I (ID 3)Ours✓×55.0137.4762.3251.60
RefineFly w/o Memory×✓✓48.1742.9055.3348.80
RefineFly (ID 4)Ours✓✓56.2844.8960.4153.86

Dynamic failure memory drives a major share of the post-training gain. Removing it lowers average SR by 5.06 pp to 48.80%, showing that reference KL and the scene curriculum alone yield limited improvement. Conversely, memory alone in ID 2 lifts Seen (+8.04 pp) and Unseen Object (+3.34 pp) on familiar maps while lowering Unseen Map by 11.69 pp to 31.21%. Since Seen and Unseen Object share training maps while Stage I concentrates 96.07% of data in 11 common scenes (vs. 3.93% across 8 rare scenes), unconstrained updates over-adapt to frequent maps and hurt zero-shot transfer. Integrating memory with reference KL and the curriculum curbs drift and improves cross-scene generalization in ID 4.

Furthermore, while PLR ranks a global buffer by advantage magnitude and staleness without retirement or retry limits, our memory runs fresh on-policy rollouts from failed initializations across active, solved, and dropped states within stage-wise scene quotas. Replacing it with PLR in Stage I lowers average SR by 7.24 pp below ID 3 (44.36%), owing to early-collision bias in step-averaged advantages, budget drain on unretired or intractable tasks, and priority concentration on frequent maps.

Q2 (W2 & Q2): Episodes, steps, updates, stage allocation, and LR schedules

A2: Thank you for pointing this out. Interaction details for Tables 4 and 5 are shown below.

MethodStagesEpochsEpisodesStepsUpdates
ID 1 (Token-PPO)11002,679159,449500
ID 2 (+ Memory)11002,789159,236500
ID 3 (+ Ref. KL)11002,744159,626500
Uniform Sampling11002,904159,401500
Original Distribution1 + 2100 + 1002,744 + 2,804159,626 + 159,568500 + 500
RefineFly (ID 4)1 + 2100 + 1002,744 + 2,050159,626 + 159,297500 + 500

Episode counts are not fixed in advance. Each stage fixes 100 epochs × 200 maximum steps × 8 parallel environments (up to 160k steps), and since episodes span epochs until termination, survival steps determine episode counts. ID 4 finishes fewer Stage II episodes owing to longer rare-scene routes and survival.

Below we summarize the stage allocation, initial memory, and cross-stage schedules.

  • Two-Stage Allocation. Both stages share identical per-stage step and update budgets. Specifically, the ~30% rollout ratio in Appendix A.2 refers to the per-stage execution budget, which we have clarified in the revised manuscript.
  • Initial Failure Memory. Initial failures come from a pre-training full-set evaluation of the frozen AerialVLA policy without updates, excluded from the online rollout budget.
  • Reference-Policy Reset. Each stage trains independently and saves its final checkpoint. Stage I uses AerialVLA weights as initial and reference policy, whereas Stage II loads the Stage I checkpoint as initial and new reference policy while reinitializing the value head.
  • Learning-Rate Schedule. Stepping once per epoch, single-stage runs use a 10-epoch linear warm-up (5e-5 actor, 1e-4 value head) and a 90-epoch cosine decay to a 0.1 minimum ratio (5e-6), while Stage II reinitializes the optimizer and scheduler for a second identical cycle.
AuthorsOpenReview ↗

Response to Reviewer P1Um (Part 2/2)

Q3 (W2 & Q3): Independent post-training runs and re-evaluating AerialVLA under identical settings

A3: Thank you for raising this question. Tables 1 to 3 in the main paper report three evaluation seeds (1, 42, and 12345) for one trained checkpoint, whereas ablations use evaluation seed 1. To test variation across independent post-training runs, we trained two additional RefineFly checkpoints (seeds 42 and 12345 alongside seed 0) and report their SR (%) under evaluation seed 1 reported below, and full test metrics and 200-epoch curves are in Appendix D.

MethodSeenUnseen MapUnseen Obj.Avg.
AerialVLA47.9637.5856.6047.38
Seed 0 (Main Paper)56.2844.8960.4153.86
Seed 4255.9240.1960.5752.23
Seed 1234558.2544.2662.8055.10
RefineFly (Mean ± Std)56.82 ± 1.0243.11 ± 2.0861.26 ± 1.0953.73 ± 1.18

Every training seed improves over AerialVLA on all three splits, yielding 53.73 ± 1.18% average SR. Given limited rebuttal time, RefineFly w/o Memory in Q1 was trained on seed 0 alone, where its 48.80% average SR remains 4.93 pp below this multi-seed mean. In Appendix D (Figure 7), Stage I rollout return converges to 2.5 to 3.0 within 30 epochs with small cross-seed spread, while environment return grows from ~1.5 to ~2.8 alongside local epoch-to-epoch fluctuations from finite-batch sampling across parallel environments under sparse event rewards. After rebalancing scene quotas at epoch 100, environment return briefly adjusts near ~2.0 under higher rare-scene allocation before rebounding to ~2.8 to 3.3, and Stage II rollout return recalibrates from ~0.9 to 2.5 to 3.0 within 20 epochs as the reinitialized value head adapts.

Furthermore, we re-evaluated the public AerialVLA checkpoint using the same 8×A100 AirSim runtime environment and evaluation seed (seed 1) as RefineFly alongside the official decoding settings of AerialVLA, as shown in the table below.

Baseline Evaluation ComparisonSeenUnseen MapUnseen Obj.Avg.
AerialVLA (Published)47.9637.5856.6047.38
AerialVLA (Re-evaluated on A100)44.0137.7951.6744.49
RefineFly (Mean ± Std)56.82 ± 1.0243.11 ± 2.0861.26 ± 1.0953.73 ± 1.18

In our 8×A100 runtime environment, AerialVLA achieves lower SR on Seen and Unseen Object than the results reported in the AerialVLA paper. After discussing this discrepancy with the AerialVLA authors, we reached a consensus that it arises from rendering and runtime differences between data-center compute GPUs (NVIDIA A100) and graphics GPUs (RTX 4090). All RefineFly post-training starts from this lower A100 baseline and improves average SR by +9.24 pp under the identical setup, while the main tables retain the published AerialVLA results to facilitate consistent comparison across prior and future work.

Q4 (W3): Precision of metric claims in the Abstract and Conclusion

A4: Thank you for pointing this out. We have revised the Abstract, Section 4.2 (Performance on Seen Environments), and Section 5 (Conclusions) to state explicitly that RefineFly achieves the highest success rate (SR) and path efficiency (SPL) across all three evaluation splits.

Finally, we provide anonymous RefineFly LoRA checkpoints at https://huggingface.co/refinefly/refinefly/tree/main for reproducible evaluation under the public AerialVLA codebase.

Reviewer_9ABZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RefineFly, a failure-aware reinforcement learning post-training framework for end-to-end UAV vision-language-action (VLA) policies. Built upon token-level PPO, the method introduces dynamic failure memory to revisit unsuccessful navigation tasks and a two-stage scene curriculum that transitions from empirical to balanced scene sampling. Stage-specific KL regularization is employed to mitigate policy drift during training. Experiments on the TravelUAV benchmark demonstrate improvements over existing methods across seen and unseen evaluation splits under a limited rollout budget.

Critical strengths
  • RefineFly consistently improves navigation performance over AerialVLA across all three evaluation splits, achieving success rate improvements. The results demonstrate the effectiveness of RL post-training for end-to-end UAV navigation.
  • The framework combines failure replay, scene curriculum, and reference regularization while preserving the original VLA action interface. The ablation studies provide useful insights into the contributions and trade-offs of these components.
Critical weaknesses
  • The proposed framework primarily combines established techniques, including token-level PPO, failure replay, curriculum learning, and KL regularization. Although effective, the approach appears to integrate existing strategies rather than introduce a fundamentally new learning mechanism. Moreover, the framework seems broadly applicable to other navigation tasks, with limited methodological specialization for UAV-VLN.
  • Although RefineFly improves upon the pretrained AerialVLA, it remains unclear whether similar improvements could be achieved by applying the proposed two-stage scene curriculum to conventional behavior cloning. Such a comparison would help clarify the benefits of RL post-training beyond scene distribution rebalancing.
  • The two stage curriculum introduces an abrupt shift from empirical to balanced scene sampling. Although final performance improves, the paper does not sufficiently examine whether this transition causes performance degradation or forgetting in scenes that were successfully learned during Stage I.
Questions for authors
  • What are the key methodological contributions of RefineFly beyond integrating existing RL techniques, particularly failure replay and curriculum learning?
  • How would AerialVLA perform if the proposed two-stage scene curriculum were applied to behavior cloning instead of RL post-training?
  • How does the transition from empirical to balanced scene sampling affect training stability? Could the authors provide training reward curves and per-scene success rates across both stages, particularly to assess whether well-performing scenes from Stage I experience performance degradation or catastrophic forgetting during Stage II?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9ABZ (Part 1/3)

We sincerely thank Reviewer 9ABZ for the constructive review and helpful questions. We report new BC curriculum baselines and per-scene analyses below, and have added the replay ablations and multi-seed stability results to the revised paper.

Q1 (W1 & Q1): Methodological contributions beyond combining RL techniques and specialization for UAV-VLN

A1: Thank you for raising this insightful question. RefineFly is designed for the specific constraints of closed-loop 3D UAV navigation, where token-level PPO, failure memory, and scene curriculum address different parts of post-training.

Compared with 2D ground VLN, 3D UAV navigation predicts coupled 3-DoF controls (forward, altitude, yaw) and an optional LAND token. It also involves costly simulation with rapid viewpoint changes, vertical obstacles, and failures such as collisions, drift, being stuck, or premature landing. Moreover, the training scenes are highly imbalanced, with 96.07% of samples from common scenes and 3.93% from rare scenes. Masked token-level PPO handles the coupled action tokens through per-token clipping (Appendix C), while failure memory and the two-stage curriculum jointly address failure revisitation and scene imbalance under a limited rollout budget.

Unlike hard-example sampling or score-based level replay (PLR), our memory stores failed task initializations and generates fresh rollouts with the current policy. Tasks transition across active, solved, and dropped states, retiring mastered tasks, capping persistent failures, and reactivating regressions within the scene quotas of each curriculum stage, coordinating lifecycle tracking directly with scene rebalancing.

We evaluated the full framework without failure memory and compared our memory with PLR under identical Stage-I settings and budget:

MethodMemoryRef. KLCurriculumSeenUnseen MapUnseen Obj.Avg.
AerialVLA---47.9637.5856.6047.38
Memory Only (ID 2)Ours××56.2131.2159.7849.07
PLR (Stage I)PLR✓×45.9135.4951.6744.36
RefineFly Stage I (ID 3)Ours✓×55.0137.4762.3251.60
RefineFly w/o Memory×✓✓48.1742.9055.3348.80
RefineFly (ID 4)Ours✓✓56.2844.8960.4153.86

It can be seen that removing failure memory from the full framework lowers average SR by 5.06 pp, while replacing our lifecycle with PLR lowers Stage-I average SR by 7.24 pp (to 44.36%). Score-based PLR struggles in closed-loop 3D flight since GAE magnitudes fluctuate with trajectory length on early crashes, persistent obstacle traps remain in the buffer without retirement or retry limits, and scene-agnostic ranking allocates most rollouts to majority scenes. Conversely, memory without regularization (ID 2) overfits training maps, whereas the curriculum and KL term without failure memory scarcely improve over ID 1 (Appendix B.4).

AuthorsOpenReview ↗

Response to Reviewer 9ABZ (Part 2/3)

Q2 (W2 & Q2): Benefit of RL post-training vs. applying the two-stage scene curriculum to behavior cloning

A2: Thank you for suggesting this baseline. Since AerialVLA is already trained via behavior cloning on the Stage-I empirical distribution, we continued fine-tuning it with behavior cloning under our Stage-II balanced curriculum and official AerialVLA settings, as shown below.

MethodStage-II Curr.SeenUnseen MapUnseen Obj.Avg.
AerialVLA×47.9637.5856.6047.38
AerialVLA + Stage-II BC✓34.5632.6740.8636.03
RefineFly✓56.2844.8960.4153.86

The results show that applying the Stage-II balanced curriculum to offline BC lowers average SR by 11.35 pp below AerialVLA.

Two factors can explain this gap. First, rare scenes contain very few expert trajectories (Appendix A.1), causing equalized offline BC quotas to train repeatedly on small fixed subsets and overfit rare scenes while degrading common ones. Second, offline BC supervises only expert states and provides no corrective signal when compounding errors lead to collisions, drift, or premature landing in closed-loop flight. RefineFly instead collects fresh rollouts on each revisit and learns from explicit failure penalties, while reference KL regularization toward the frozen Stage-I policy prevents balanced sampling from overwriting previously acquired navigation capabilities.

Q3 (W3 & Q3 - Part 1): Training stability across the Stage I-to-Stage II transition

A3: Thank you for raising this question on transition stability. We trained RefineFly across three independent seeds (0, 42, and 12345) and added both the 200-epoch return trajectories and multi-seed test results to Appendix D, with full-split SR (%) under evaluation seed 1 reported below.

MethodSeenUnseen MapUnseen Obj.Avg.
AerialVLA47.9637.5856.6047.38
Seed 0 (Main Paper)56.2844.8960.4153.86
Seed 4255.9240.1960.5752.23
Seed 1234558.2544.2662.8055.10
RefineFly (Mean ± Std)56.82 ± 1.0243.11 ± 2.0861.26 ± 1.0953.73 ± 1.18

The results demonstrate that RefineFly outperforms AerialVLA across all splits in every run, reaching 53.73 ± 1.18% mean SR with low variance across training seeds.

In Appendix D (Fig. 7), Stage-I rollout return climbs within 30 epochs and holds between 2.5 and 3.0 with narrow variance, accompanied by environment return rising overall from ~1.5 to ~2.8. Switching to equal scene quotas at epoch 100 causes a temporary decrease in environment return toward ~2.0 as rare scenes receive a larger rollout share, followed by sustained recovery to ~2.8 to 3.3 without policy collapse. At the same boundary, Stage-II rollout return starts near 0.9 as the value head is reinitialized to prevent Stage-I value estimates from biasing rebalanced updates, rising within 20 rollout epochs to 2.5 to 3.0 with tighter cross-seed concentration. Together with anchoring updates to the frozen Stage-I reference policy, this schedule supports stable adaptation across scene distributions.

AuthorsOpenReview ↗

Response to Reviewer 9ABZ (Part 3/3)

Q4 (W3 & Q3 - Part 2): Per-scene success rates across all three splits and catastrophic forgetting analysis

A4: To check directly whether scenes learned in Stage I suffer catastrophic forgetting during Stage II, we evaluated per-scene SR (%) after Stage I and Stage II across all three splits, as shown below.

SplitSceneTasksStage I SR (%)Stage II SR (%)Change (pp)
SeenNYCEnvironmentMegapa28160.1457.30-2.84
SeenTropicalIsland16578.7976.36-2.43
SeenCarla_Town0116563.0368.48+5.45
SeenNewYorkCity14520.0025.52+5.52
SeenCarla_Town10HD14143.2651.06+7.80
SeenModernCityMap12460.4853.23-7.25
SeenCarla_Town1511145.9550.45+4.50
SeenCarla_Town0410655.6649.06-6.60
SeenCarla_Town057350.6857.53+6.85
SeenCarla_Town076048.3366.67+18.34
SeenCarla_Town064776.6070.21-6.39
Unseen MapModularPark95837.4744.89+7.42
Unseen Obj.TropicalIsland26674.0669.92-4.14
Unseen Obj.NYCEnvironmentMegapa18956.6153.44-3.17
Unseen Obj.NewYorkCity7625.0025.000.00
Unseen Obj.Carla_Town153378.7954.55-24.24
Unseen Obj.Carla_Town052080.0085.00+5.00
Unseen Obj.Carla_Town062050.0080.00+30.00
Unseen Obj.Carla_Town041471.4371.430.00
Unseen Obj.Carla_Town071163.6481.82+18.18

Across all three splits, shifting to balanced sampling in Stage II does not cause catastrophic forgetting on scenes learned in Stage I. On Test Seen, six of the eleven maps improve further and lift overall Seen SR from 55.01% to 56.28%, while the largest and highest-scoring Stage-I maps (NYCEnvironmentMegapa and TropicalIsland) stay close to their Stage-I levels. As the sampling distribution shifts during training, a few seen maps show moderate declines, while the small Carla_Town15 subset on Unseen Object decreases by 8 tasks. This adjustment trades a small degree of scene-specific adaptation on Unseen Object for a 7.42 pp gain on Unseen Map.

Two aspects of the Stage-II design preserve previously acquired navigation capabilities during this transition. Specifically, balanced sampling still allocates sufficient rollout budget to each common scene, while stage-specific reference KL regularization constrains policy deviation from Stage-I behaviors.

To facilitate verification, our trained RefineFly LoRA weights are anonymously available at https://huggingface.co/refinefly/refinefly/tree/main and can be evaluated directly with the public AerialVLA code.

Reviewer_9RrxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RefineFly, a failure-aware reinforcement learning post-training framework for end-to-end UAV vision-language-action (VLA) policies. To address the limitations of behavior cloning in closed-loop navigation, the method combines token-level PPO with dynamic failure memory, a two-stage scene curriculum, and reference-policy KL regularization. The failure memory enables repeated learning from unresolved navigation tasks, while the curriculum adjusts scene sampling to improve coverage of underrepresented environments. Experiments on TravelUAV demonstrate consistent improvements over AerialVLA across seen environments, unseen maps, and unseen objects, highlighting the effectiveness of failure-aware post-training for aerial navigation.

Critical strengths
  1. The paper addresses an important challenge in UAV-VLA: improving navigation policies through feedback from their own execution failures. The proposed framework integrates dynamic failure memory with scene-aware curriculum learning and reference regularization. The overall design is well motivated and provides a practical approach to improving closed-loop navigation under limited interaction budgets.

  2. RefineFly achieves consistent improvements over AerialVLA across all three TravelUAV evaluation splits, with success rate gains ranging from 3.12 to 8.37 percentage points. The experimental evaluation includes different trajectory difficulties, component ablations, scene sampling comparisons, and additional analyses of failure memory and PPO variants. Particularly, the observed trade-offs between failure-focused learning and generalization provide useful insights into RL post-training for UAV-VLA policies.

Critical weaknesses
  1. The main components, including PPO, failure-oriented task sampling, curriculum learning, and KL regularization, are individually established techniques. Although their integration into UAV-VLA post-training is meaningful, the paper could better clarify the distinctive characteristics of dynamic failure memory compared with conventional hard-example sampling or experience replay, particularly regarding its memory lifecycle and interaction with scene-aware sampling.

  2. The results are encouraging, but the evaluation mainly focuses on a single benchmark and reports multiple evaluation seeds rather than clearly demonstrating variability across independent post-training runs. Additional discussion of training stability and the applicability of the framework to other UAV navigation settings would strengthen the conclusions.

Questions for authors
  1. What are the key differences between the proposed dynamic failure memory and conventional hard-example sampling or prioritized experience replay? Which aspects of its lifecycle management are particularly beneficial for UAV navigation?

  2. In Table 4, introducing failure memory improves Seen and Unseen Object performance but decreases Unseen Map performance. Could the authors further explain this trade-off and how the subsequent reference regularization and scene curriculum address it?

  3. Do the reported evaluation seeds correspond to independently post-trained policies or repeated evaluations of the same trained policy? Could the authors clarify the sensitivity of the results to post-training randomness?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Rrx (Part 1/2)

We sincerely thank Reviewer 9Rrx for the positive assessment and insightful questions. We have added failure-memory ablations, prioritized replay comparisons, and multi-seed post-training runs, and updated the revised manuscript with further clarifications.

Q1 (W1 & Q1): Dynamic failure memory vs. hard-example sampling and PER

A1: Thank you for highlighting this key aspect. Dynamic failure memory coordinates outcome-driven lifecycle management with scene-aware sampling distributions to sustain closed-loop failure learning while preserving generalization.

Compared with hard-example sampling or PER (with PLR as a concrete example), dynamic failure memory differs in rollout freshness, lifecycle tracking, and scene-distribution alignment. Specifically, PER replays off-policy transitions weighted by TD errors, whereas hard-example sampling and PLR rank tasks globally by difficulty or advantage magnitude without task retirement. In contrast, our memory records failed task initializations for fresh on-policy rollouts, tracks active, solved, and dropped states with reactivation, and samples unresolved tasks under the prescribed scene distribution.

These distinctions bring three clear advantages for closed-loop 3D UAV navigation under a limited rollout budget. First, fresh on-policy rollouts provide current feedback as the policy evolves. Second, retiring solved tasks saves rollout budget and dropping tasks after two failed retries avoids unsolvable 3D obstacle traps. Third, scene-aligned sampling balances focused refinement in common scenes with broader failure learning across rare scenes. We verify these benefits via a leave-one-out ablation removing failure memory and a Stage I PLR comparison under identical ID 3 settings and budget, as shown in the table below.

MethodMemoryRef. KLCurriculumSeenUnseen MapUnseen Obj.Avg.
AerialVLA---47.9637.5856.6047.38
PLR (Stage I)PLR✓×45.9135.4951.6744.36
RefineFly Stage I (ID 3)Ours✓×55.0137.4762.3251.60
RefineFly w/o Memory×✓✓48.1742.9055.3348.80
RefineFly (ID 4)Ours✓✓56.2844.8960.4153.86

Both failure memory and lifecycle-regulated scene sampling contribute directly to these gains. Without failure memory, average SR falls by 5.06 pp to 48.80%, indicating that KL constraints and scene rebalancing alone yield little improvement over token-level PPO. Substituting PLR for our memory in Stage I lowers average SR by 7.24 pp below ID 3 (to 44.36%), as step-averaged GAE priority tends to overweight brief early crashes during value adaptation, repeated selection without solved retirement or retry bounds expends rollouts on mastered or unsolvable tasks, and global ranking concentrates updates on dominant maps (Appendix B.4).

Q2: Unseen Map trade-off and mitigation via KL and curriculum

A2: Thank you for raising this trade-off. This divergence stems from shared map geometry across the Seen and Unseen Object splits compounded by training-scene imbalance. Specifically, both splits operate in familiar training maps, where fresh rollouts on unresolved failures refine obstacle avoidance and landing control to raise SR by 8.04 pp and 3.34 pp in ID 2. In contrast, the Stage I distribution skews heavily toward 11 common scenes at 96.07% over 8 rare scenes at 3.93%. Consequently, unconstrained updates on dominant maps cause scene-specific overfitting and policy drift on Unseen Map, lowering SR by 11.69 pp as over half of previously solved tasks regress.

Reference KL regularization and the two-stage scene curriculum jointly mitigate this trade-off by curbing policy drift and rebalancing scene distributions. First, anchoring token updates to the frozen stage-initial policy constrains deviation during repeated rollouts in ID 3, partially recovering unseen-map performance while rare scenes remain underrepresented. Subsequently, shifting to a balanced scene distribution in Stage II under the Stage I reference expands fresh and memory rollouts in rare scenes, enabling ID 4 to achieve the strongest Unseen Map and overall performance across Tables 4 and 5.

AuthorsOpenReview ↗

Response to Reviewer 9Rrx (Part 2/2)

Q3 (W2): Benchmark choice and applicability across UAV settings

A3: Thank you for this constructive comment on benchmark scope and applicability. We clarify that adopting TravelUAV reflects the simulation and policy requirements of closed-loop end-to-end UAV-VLA post-training. Specifically, closed-loop UAV-VLA RL requires 3D collision physics, continuous 3-DoF control with intrinsic landing, and an open VLA checkpoint, currently met publicly by TravelUAV with AerialVLA rather than discrete collision-free graphs (AerialVLN, AVDN), fixed primitives (OpenFly), or offline benchmarks (SpatialSky-Bench, UAVBench).

At the same time, RefineFly relies on no TravelUAV-specific planners, making the framework readily adaptable to other closed-loop aerial navigation settings while preserving its 3D flight specialization. Specifically, masked token-level PPO applies to any autoregressive VLA, flight-event lifecycles require only onboard sensors and task resets, and the two-stage curriculum requires only scene and task identifiers, allowing natural extension to other interactive 3D benchmarks such as UAV-ON.

Q4 (W2 & Q3): Evaluation seeds vs. independent post-training runs and stability

A4: Thank you for raising this question on post-training randomness. Tables 1 to 3 evaluate a single trained policy across three evaluation seeds (1, 42, and 12345).

To measure sensitivity across independent post-training runs, we trained two additional policies (seeds 42 and 12345) alongside seed 0 from the main paper and evaluated all three policies under evaluation seed 1, as shown below.

MethodSeenUnseen MapUnseen Obj.Avg.
AerialVLA47.9637.5856.6047.38
Seed 0 (Main Paper)56.2844.8960.4153.86
Seed 4255.9240.1960.5752.23
Seed 1234558.2544.2662.8055.10
RefineFly (Mean ± Std)56.82 ± 1.0243.11 ± 2.0861.26 ± 1.0953.73 ± 1.18

All three independently trained policies exceed AerialVLA across every split and metric including NE, OSR, and SPL (Appendix D, Table 10), reaching 53.73 ± 1.18% average SR (+6.35 pp).

The three-seed training curves in Appendix Figure 7 also confirm consistent optimization across both scene distributions. Stage I environment return trends upward overall from ~1.5 to ~2.8, adjusts briefly near ~2.0 when epoch 100 increases rare-scene sampling, and climbs back to ~2.8 to 3.3 without instability. Meanwhile, Stage II rollout return rapidly calibrates from ~0.9 to 2.5 to 3.0 within 20 epochs after value-head reinitialization.

Finally, we have anonymously released the LoRA weights of RefineFly at https://huggingface.co/refinefly/refinefly/tree/main for direct evaluation with the public AerialVLA codebase.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

Thank you very much for your constructive comments and insightful suggestions, which have helped us improve the quality of our original manuscript. We have provided detailed point-by-point discussions and updated our manuscript (marked in blue) based on the concerns of reviewers. Below is a summary of the work we have done during the rebuttal phase.

We have improved the clarity and precision of the revised manuscript. For example, we clarified the active/solved/dropped failure-memory lifecycle, refined the metric-specific claims on SR, SPL, and OSR in the Abstract, Section 4.2, and Conclusion, specified the per-stage rollout budget, schedule parameters, and seed setup, and anonymously released our trained LoRA weights to support reproducibility.

We have supplemented additional experiments and diagnostic analyses. Considering the simulation time required for online RL training, we focused our quantitative experiments on the following aspects: 1) removing failure memory from the full framework and comparing with Prioritized Level Replay (PLR) in Stage I, 2) running independent training seeds with return curves over 200 epochs and results by scene across stages, and 3) testing behavior cloning under the Stage II curriculum, evaluating AerialVLA in the same runtime environment, and reporting detailed interaction logs. We hope that these additional results address the concerns of reviewers.

We further discussed our motivations and methodological designs for closed-loop 3D UAV navigation, explaining how masked token-level PPO handles coupled 3-DoF control and landing tokens, how lifecycle-regulated failure memory coordinates fresh on-policy rollouts with stage-wise scene quotas, and how reference KL regularization and two-stage scene rebalancing jointly resolve the generalization trade-off on unseen maps.

We also clarified the scope and current limitations of our evaluation under costly 3D simulation and long-tailed scene distributions. Specifically, we explained why TravelUAV with AerialVLA meets the closed-loop collision and continuous-control requirements of this study, noted that our post-training components do not rely on benchmark-specific planners, and left validation on additional interactive benchmarks and real-world flights to future work.

Finally, we would like to thank the three anonymous reviewers once again for their thoughtful comments and helpful suggestions, which have helped us improve both the technical rigor and presentation of the manuscript. We also thank the Area Chair for their time and effort in overseeing the review process.

Sincerely,

Authors of Paper 23536

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.