acceptodds
Withdrawn from ICLR 2027

Progress Is the Reward: UAV Policy Optimization via Progress Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-quality UAV control policies rely on expert demonstrations that are costly to collect and limited in scale. More importantly, supervised imitation provides only frame-level action supervision, leaving episode-level progress regulation unlearned and making long-horizon policies prone to inappropriate pacing and unstable stopping. Yet the temporal structure required for such regulation is already embedded in complete expert trajectories. We propose UAVProgressor-RL, a progress-aware reinforcement learning framework that learns task progress from limited expert trajectories and converts it into dense feedback for policy optimization. The history-conditioned progress model UAVProgressor estimates normalized task progress from visual history, the current observation, and the task instruction, and its predictions are used to construct rewards for progress improvement, schedule alignment, and completion-aware stopping. We further introduce Schedule-Aligned Displacement Error (SADE) to evaluate whether spatial execution remains aligned with task progress. Extensive experiments on UAVFlow and OpenUAV show that UAVProgressor-RL achieves the best overall performance across five policy backbones and retains its gains on unseen maps, successfully turning limited demonstrations into effective dense supervision. Our models and training code will be publicly available.

Primary area:
reinforcement learning
Keywords:
UAV ControlProgress, Reinforcement Learning, Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HbWhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UAVProgressor-RL, a progress-aware RL framework for long-horizon UAV control from limited demonstrations. It learns a visual progress predictor from expert trajectories and uses its predictions to construct rewards for task advancement, schedule alignment, and stable stopping. The authors introduce SADE to measure progress-aligned spatial execution. Experiments on UAVFlow and an OpenUAV unseen-map subset evaluate five policy backbones, data efficiency, alternative progress predictors, reward components, and RL optimizers. The method consistently improves SADE over the reported baselines.

Critical strengths
  1. The paper addresses an important limitation of frame-level imitation for long-horizon UAV control. By explicitly modeling task progress and terminal behavior, UAVProgressor-RL targets execution pacing and post-completion stability, which are not directly captured by standard action-supervision objectives.

  2. The empirical evaluation is broad and includes several targeted analyses. The method is evaluated across five policy backbones, UAVFlow and an OpenUAV unseen-map subset, different demonstration budgets, alternative progress predictors, temporal perturbations, reward-component ablations, and RL optimizers. These experiments provide useful evidence about the method’s behavior and the roles of its components.

Critical weaknesses
  1. The methodological contribution appears relatively simple and strongly tailored to UAV trajectory-following tasks. The proposed reward is a heuristic combination of progress improvement, schedule alignment, and completion-aware stopping. While this design is intuitive for the evaluated UAV scenarios, the paper does not sufficiently demonstrate that the same formulation is applicable to other control tasks or task structures.

  2. The progress-based reward relies on a potentially restrictive progress assumption. Progress supervision is derived from the normalized temporal index of a reference execution, and the schedule-alignment term assumes an approximately linear progress schedule. This may be problematic for tasks such as navigation, search, or exploration, where progress can be non-linear, involve pauses or backtracking, or remain ambiguous during early exploration. The paper does not evaluate whether the proposed reward remains effective in such settings.

  3. The main evaluation metric is closely coupled to the proposed reward design. SADE is introduced by the authors and directly measures synchronization with reference progress, delay, and post-completion motion, which are also optimized by the proposed rewards. The paper does not sufficiently report independent measures such as task success rate, goal-reaching accuracy, failure/collision rate, or control efficiency. Thus, it remains unclear whether the SADE improvements reflect genuinely better task completion and control quality.

Questions for authors
  1. Could you specify which tasks or instances correspond to the three columns in Figure 4(a), and explain the main reasons for the performance differences across the subplots?

  2. The progress target ϕt=t/(T−1)\phi_t=t/(T-1) and the schedule reference ct=clip(t/Tref,0,1)c_t=\mathrm{clip}(t/T_{\mathrm{ref}},0,1) impose a temporally linear reference progress. How would the method handle tasks whose progress is non-linear, includes pauses or backtracking, or is ambiguous during exploration, such as navigation or search? Have the authors considered or evaluated non-linear progress schedules? Please also clarify which parts of the current formulation would need to change for applications beyond trajectory-following UAV control, such as embodied manipulation or navigation.

Reviewer_C3VZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finetunes UAV navigation policies with RL and the reward comes from a learned progress model. The model is trained on expert flights and predicts how far along the task the drone is - frame index divided by episode length. It is then frozen and used in a PPO reward that pays for progress, penalizes falling behind the expert's pace and rewards stopping near the end. The authors also propose a new metric (SADE) which measures how far the drone is from where the expert was at the same point in time.

In simulation (UAVFlow) the method beats SFT and three other learned rewards on overall SADE for all five policies tested with the biggest gain on FastWAM. The paper also reports unseen OpenUAV maps, a data-size sweep, a comparison of progress predictors and ablations.

Critical strengths
  1. Simple idea that works across policies. Progress labels come for free from demonstrations, and the same reward model improves all five policies. If this holds on standard metrics, it is a cheap way to fine-tune navigation policies with RL.
  2. The progress model itself looks good. It is much more accurate than the other predictors (Table 2). Fig. 5 suggests it reacts to what the drone actually does rather than just counting time.
Critical weaknesses
  1. The main metric is too close to the reward, and standard metrics are missing. SADE uses the same expert time schedule as the reward, so the method is partly trained on what it is evaluated on. There is no success rate, and nDTW shows up only once, for one baseline, with a tiny gap. The intro talks about overshoot and collisions, yet there is no landing task and no collision rate. I can't tell whether the drone completes more tasks, or just learns to follow the expert's pace and stop in the right place. This is the main reason for my score.
  2. Baselines and statistics are unclear. The terminal reward is the most important component in the ablation, but it has no formula. The ORM baseline is not defined. RoboMeter and Robo-Dopamine seem to be the authors' reward with a different progress predictor, not the original methods. There are no seeds or standard deviations, while some margins are very small.
Questions for authors

Clarifications

  1. Can you report success rate and nDTW for all methods in Table 1 (and SR/SPL on OpenUAV)? If the method still wins on success rate, I would raise my score.
  2. What is the formula of the terminal reward, and what exactly is the ORM baseline? Are RoboMeter and Robo-Dopamine run with their original rewards, or with yours?
  3. How many RL seeds are behind Table 1, and how large is the variance?

Suggestions

  1. The reward already uses the expert episode length, so a simple baseline that uses the expert positions directly would show what the learned progress model adds.
  2. FastWAM shares the backbone with the progress model and gets by far the largest gain, while the ablations seem to use only FastWAM. Repeating them on another policy would help.
  3. Related work on progress as a reward (e.g. PROGRESSOR, Rank2Reward, GVL, and the progress monitor of Ma et al., ICLR 2019) should be discussed.
Reviewer_YBhiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a way to improve RL-based policy optimization for UAV trajectory tracking by measuring progress. A module called UAVProgressor is trained on full historical trajectories to monitor the progress of the UAV along the desired trajectory and provide dense progress reward with the goal of improving policy optimization.

The metric used for assessing the training performance of UAV trajectory tracking is defined in an appendix as SADE (Schedule-Aligned Displacement Error). A table of comparisons shows the tracking performance of the proposed learning scheme versus other baselines for various trajectory tracking tasks. The proposed scheme yields better performance than the baselines, except for two tasks over a long horizon.

Critical strengths

One positive aspect of the paper is that it tries to properly quantify trajectory tracking for UAVs which exhibits complexities. For instance, a small UAV may be tracking its reference trajectory well until a wind gust throws it back. One would expect that the UAV recovering by coming back on the right path, despite being late on its schedule, would in many cases still be an acceptable outcome. This paper attempts to measure these effects by combining in the same metric spatial excursions from the flight path with delay in execution which could be used for learning a control policy trading off spatial accuracy and time accuracy.

Critical weaknesses

The Schedule-Aligned Displacement Error (SADE) is the key metric used to compare results in the paper but it is unfortunately poorly defined in Appendix C where notation is not well-defined and the text is incomprehensible around line 1152. There is no guarantee that SADE would be accepted by the community as a performance metric, so it should be very clearly defined in the paper and thorough motivation should be provided for its use.

Questions for authors

It would greatly improve the manuscript if the VLA tasks were better described.

Table 1 is misleading with boldface quantities suggesting they are the best results when it`s not the case for two results of the proposed method on long-horizon tasks.

Do you have evidence that SADE could be adopted by the community as a metric of choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.