- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RefineFly, an online reinforcement-learning post-training framework for end-to-end aerial vision-language-action policies. Starting from AerialVLA, it combines token-level PPO, a dynamic memory of unsuccessful task initializations, a two-stage curriculum shifting from empirical to balanced scene sampling, and stage-specific reference-policy KL regularization. Memory-selected tasks generate fresh trajectories using the current policy. Evaluation on TravelUAV reports success-rate improvements over AerialVLA of 8.37, 5.39, and 3.12 percentage points on Seen, Unseen Map, and Unseen Object splits, respectively. Ablations investigate the components, sampling schedules, and token-level versus sequence-level PPO.
Critical strengths
I liked the consistent improvements over AerialVLA across all three evaluation splits (Tables 1–3), including success rate, navigation error, oracle success rate, and SPL. The gains also extend to harder trajectories: success rate on Seen-Hard increases from 46.30% to 56.38%. These results support the value of online post-training for improving an existing end-to-end aerial policy while keeping the same action interface.
I also found the analysis of failure-focused training and generalization useful. The authors show that adding failure memory alone improves performance on Seen and Unseen Object but substantially hurts performance on Unseen Map. Appendix B provides paired task-level comparisons, and Table 5 includes a useful two-stage baseline that keeps the original scene distribution. Together, these experiments help explain how failure-focused training and scene sampling affect generalization.
Critical weaknesses
My main concern is that the contribution of failure memory is unclear in the full framework. In Table 4, adding memory to token-level PPO changes average success rate from 49.17% to 49.07% and reduces Unseen Map success rate from 42.90% to 31.21%. Adding KL regularization and the curriculum improves the results, but the paper does not test the same PPO, KL, and two-stage curriculum without failure memory. This comparison is needed to understand whether memory contributes to the final gains or whether they mainly come from regularization and scene rebalancing. A comparison with existing task-replay methods, such as Prioritized Level Replay, would also help clarify what the proposed memory lifecycle adds.
I would also like clearer evidence on training stability and whether the comparisons use the same budget. Tables 1–3 report three evaluation seeds, and the ablations use one evaluation seed, but variation across independent training runs is not reported. The baseline results are also taken from previous publications rather than evaluated under the same repeated-evaluation setup. Appendix A explains the approximate 31.25% rollout budget, including repeated executions, but it remains unclear whether this includes collecting the initial failures and whether all ablations use the same rollout and optimizer-update budgets. These details matter because the paper studies sampling strategies under a limited interaction budget, and some comparisons use different numbers of stages.
Minor comment: The performance claim in the abstract and conclusion could be more precise. RefineFly achieves the best SR and SPL across all three splits, but LongFly performs better in some NE and OSR comparisons. For example, on Unseen Object–Hard, LongFly achieves NE of 57.07 versus 62.34 and OSR of 74.16% versus 66.43%. It also achieves higher OSR on Seen-Full and Seen-Easy. Section 4.2 explicitly acknowledges the Unseen Object–Hard exceptions, and the abstract and conclusion could specify which metrics the overall performance claim refers to.
Questions for authors
-
Could you report results for the full framework without failure memory, while keeping token-level PPO, both curriculum stages, the reference-policy schedule, and the same interaction and optimizer-update budgets? Also, how does the proposed memory compare with a simpler task-replay or task-prioritization baseline under the same setup? These comparisons would help me understand what failure memory adds.
-
Could you clarify the total number of executed episodes, environment steps, and optimizer updates for each configuration in Tables 4–5, including their allocation across stages? Does the interaction budget include collecting failures to initialize memory? Also, how are reference-policy resets and learning-rate schedules handled when comparing one-stage and two-stage training?
-
How many independent post-training runs were used for the reported results? Could you compare the full method and the version without failure memory across multiple training seeds, and evaluate the initial AerialVLA checkpoint using the same evaluation seeds and decoding settings? This would help determine whether the improvements are consistent across training runs.
2 comments
Response to Reviewer P1Um (Part 2/2)
Q3 (W2 & Q3): Independent post-training runs and re-evaluating AerialVLA under identical settings
A3: Thank you for raising this question. Tables 1 to 3 in the main paper report three evaluation seeds (1, 42, and 12345) for one trained checkpoint, whereas ablations use evaluation seed 1. To test variation across independent post-training runs, we trained two additional RefineFly checkpoints (seeds 42 and 12345 alongside seed 0) and report their SR (%) under evaluation seed 1 reported below, and full test metrics and 200-epoch curves are in Appendix D.
| Method | Seen | Unseen Map | Unseen Obj. | Avg. |
|---|---|---|---|---|
| AerialVLA | 47.96 | 37.58 | 56.60 | 47.38 |
| Seed 0 (Main Paper) | 56.28 | 44.89 | 60.41 | 53.86 |
| Seed 42 | 55.92 | 40.19 | 60.57 | 52.23 |
| Seed 12345 | 58.25 | 44.26 | 62.80 | 55.10 |
| RefineFly (Mean ± Std) | 56.82 ± 1.02 | 43.11 ± 2.08 | 61.26 ± 1.09 | 53.73 ± 1.18 |
Every training seed improves over AerialVLA on all three splits, yielding 53.73 ± 1.18% average SR. Given limited rebuttal time, RefineFly w/o Memory in Q1 was trained on seed 0 alone, where its 48.80% average SR remains 4.93 pp below this multi-seed mean. In Appendix D (Figure 7), Stage I rollout return converges to 2.5 to 3.0 within 30 epochs with small cross-seed spread, while environment return grows from ~1.5 to ~2.8 alongside local epoch-to-epoch fluctuations from finite-batch sampling across parallel environments under sparse event rewards. After rebalancing scene quotas at epoch 100, environment return briefly adjusts near ~2.0 under higher rare-scene allocation before rebounding to ~2.8 to 3.3, and Stage II rollout return recalibrates from ~0.9 to 2.5 to 3.0 within 20 epochs as the reinitialized value head adapts.
Furthermore, we re-evaluated the public AerialVLA checkpoint using the same 8×A100 AirSim runtime environment and evaluation seed (seed 1) as RefineFly alongside the official decoding settings of AerialVLA, as shown in the table below.
| Baseline Evaluation Comparison | Seen | Unseen Map | Unseen Obj. | Avg. |
|---|---|---|---|---|
| AerialVLA (Published) | 47.96 | 37.58 | 56.60 | 47.38 |
| AerialVLA (Re-evaluated on A100) | 44.01 | 37.79 | 51.67 | 44.49 |
| RefineFly (Mean ± Std) | 56.82 ± 1.02 | 43.11 ± 2.08 | 61.26 ± 1.09 | 53.73 ± 1.18 |
In our 8×A100 runtime environment, AerialVLA achieves lower SR on Seen and Unseen Object than the results reported in the AerialVLA paper. After discussing this discrepancy with the AerialVLA authors, we reached a consensus that it arises from rendering and runtime differences between data-center compute GPUs (NVIDIA A100) and graphics GPUs (RTX 4090). All RefineFly post-training starts from this lower A100 baseline and improves average SR by +9.24 pp under the identical setup, while the main tables retain the published AerialVLA results to facilitate consistent comparison across prior and future work.
Q4 (W3): Precision of metric claims in the Abstract and Conclusion
A4: Thank you for pointing this out. We have revised the Abstract, Section 4.2 (Performance on Seen Environments), and Section 5 (Conclusions) to state explicitly that RefineFly achieves the highest success rate (SR) and path efficiency (SPL) across all three evaluation splits.
Finally, we provide anonymous RefineFly LoRA checkpoints at https://huggingface.co/refinefly/refinefly/tree/main for reproducible evaluation under the public AerialVLA codebase.
Response to Reviewer P1Um (Part 1/2)
We sincerely thank Reviewer P1Um for the thorough assessment and insightful comments. We provide budget logs and baseline re-evaluations below, and have added the memory and replay ablations, multi-seed runs, and metric clarifications to the revised manuscript.
Q1 (W1 & Q1): Failure-memory contribution, Unseen Map trade-off, and PLR
A1: Thank you for raising this question. We evaluated RefineFly w/o Memory under identical token-level PPO, two-stage curriculum, reference resets, and budgets, alongside Prioritized Level Replay (PLR) in Stage I under ID 3 settings (given limited rebuttal time), as shown below.
Dynamic failure memory drives a major share of the post-training gain. Removing it lowers average SR by 5.06 pp to 48.80%, showing that reference KL and the scene curriculum alone yield limited improvement. Conversely, memory alone in ID 2 lifts Seen (+8.04 pp) and Unseen Object (+3.34 pp) on familiar maps while lowering Unseen Map by 11.69 pp to 31.21%. Since Seen and Unseen Object share training maps while Stage I concentrates 96.07% of data in 11 common scenes (vs. 3.93% across 8 rare scenes), unconstrained updates over-adapt to frequent maps and hurt zero-shot transfer. Integrating memory with reference KL and the curriculum curbs drift and improves cross-scene generalization in ID 4.
Furthermore, while PLR ranks a global buffer by advantage magnitude and staleness without retirement or retry limits, our memory runs fresh on-policy rollouts from failed initializations across active, solved, and dropped states within stage-wise scene quotas. Replacing it with PLR in Stage I lowers average SR by 7.24 pp below ID 3 (44.36%), owing to early-collision bias in step-averaged advantages, budget drain on unretired or intractable tasks, and priority concentration on frequent maps.
Q2 (W2 & Q2): Episodes, steps, updates, stage allocation, and LR schedules
A2: Thank you for pointing this out. Interaction details for Tables 4 and 5 are shown below.
Episode counts are not fixed in advance. Each stage fixes 100 epochs × 200 maximum steps × 8 parallel environments (up to 160k steps), and since episodes span epochs until termination, survival steps determine episode counts. ID 4 finishes fewer Stage II episodes owing to longer rare-scene routes and survival.
Below we summarize the stage allocation, initial memory, and cross-stage schedules.
5e-5actor,1e-4value head) and a 90-epoch cosine decay to a 0.1 minimum ratio (5e-6), while Stage II reinitializes the optimizer and scheduler for a second identical cycle.