- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SUDA, a platform that runs Isaac Sim and the PX4 autopilot on one clock step that makes it easier to synchronize aligned commands, executed motion and camera frames at 20Hz across 4 drones per scene. Because a real autopilot lags in real life and usually overshoots path at times, the construct SUDA-UAV, a controller-executed dataset for dynamics- aware UAV policy learning (13.7k flights, 1.6M frames), where each step carries a 15D motion-response vector. The third and final part of their paper is the execution adapter - where the policy predicts this 15D response but only 7 channels get sent to PX4 and the other 8 act as auxiliary training targets. On a 200-task closed loop benchmark, training on executed instead of planned targets lifts mean success from 41.4% to 51.3% across eight policy families.
Critical strengths
- This is a very well documented test bed for closed loop UAV testing and all policies run through through a common PX4 mapping
- A wide, matched comparison with a consistent result - 8 existing model families, plus the authors’ own Qwen-based model, are each trained twice with the same settings: once on the drone’s actual motion and once on the planned motion. Success goes up in all nine of them
Critical weaknesses
- The dynamics-awareness claim isn’t isolated - where the QwenVLA reference baseline is trained on reference-derived state (D.1) but fed executed state in flight (F.3). With no run that is separating the targets from inputs, its plausible that the gain could come from this mismatch or a better target adapter fit rather than dynamics awareness. Training and testing also use the same simulated PX4 setup, with no real flights or changed dynamics, so nothing shows the benefit carries beyond this one controller setup.
- The benchmark doesn’t show it measures real control - where tasks are short (median reference 1 m, B.4), so hovering alone would hit OR@1m on about half of them, yet there’s no hover baseline. An undescribed GPT-5.5 Policy (E.1) also matches the SUDA-trained family mean (51.5% vs 51.3% SR)
Questions for authors
- Could you add a hover baseline and the recorded PX4 flights of the 200 references under the same metrics - where it shows what staying put scores vs. the recorded flights themselves?
- The QwenVLA reference baseline is trained on reference-derived state (D.1) but gets executed state in flight (F.3). Can you add a run with reference targets and executed-state input?
- What is the GPT-5.5 Policy (E.1), and was it trained on SUDA-UAV?
- Does the gain still hold when the dynamics change - i.e that’s where dynamics awareness should show up ?
- Each checkpoint is trained once (D.3). Can you add 2 or 3 seeds, at least for the smaller gains?
- Tables A8 and A9 give different FDE for the same checkpoints and even flip whether FDE improves on average - which one is correct?
3 comments
Response to Reviewer uLTN (2/2)
Both policies are evaluated on the same 200 tasks using the originally reported protocol. GPT-5.5 provides a zero-training reference for direct multimodal-model control and validates the platform’s closed-loop workflow. Matched comparisons within each trained policy family quantify the benefit of executed data, with a mean SR gain of 9.9 percentage points [7.2, 12.6] across eight families.
Q4 Changes in dynamics
Does the gain still hold when the dynamics change - i.e that’s where dynamics awareness should show up ?
A4. Thank you for suggesting a direct dynamics change. We hold the SFT weights fixed and increase vehicle mass by 20%, from 1.6186kg to 1.9423kg, on the supplementary backend. Both conditions complete the same 200 tasks with matched initialization, controller parameters, camera interfaces, and execution budgets. Using the original reported SR protocol, the mass change produces a +1.0 percentage-point SR difference, with close trajectory metrics. These results characterize the fixed policy’s sensitivity to the tested mass variation.
| Fixed-policy test | Tasks per condition | SR change | nDTW change | ATE change | FDE change |
|---|---|---|---|---|---|
| 20% mass increase | 200 | +1.0 pp | +0.0010 | −0.0024m | −0.0035m |
Only mass changes. Inertia, rotors, motors, PX4 parameters, initialization, random sequences, and the camera interface remain paired. SR changes from 54.0% to 55.5%, with close trajectory metrics. Supplementary Table S4 gives the settings. The result supports similar aggregate performance for the fixed policy under the tested mass change.
Supplementary Table S1 verifies platform dynamics through 12 matched target-frequency, speed, and control-mode conditions, each repeated three times. The SUDA−AirSim tracking-RMSE difference ranges from −1.7 to +15.3mm.
Q5 Training randomness
Each checkpoint is trained once (D.3). Can you add 2 or 3 seeds, at least for the smaller gains? A5. Thank you for suggesting independent training repeats. We added three PPO seeds with a common SFT initialization and the same budget of 511 updates and 4085 trajectories. One run has completed training and evaluation on all 200 tasks. The remaining two runs are in progress.
Under the original reported SR protocol, the completed repeat gains five successful tasks and loses two relative to its matched SFT parent, giving a net SR improvement of 1.5 percentage points on the supplementary backend.
| Completed additional repeat | Evaluation tasks | SR change | ATE change | FDE change |
|---|---|---|---|---|
| PPO over matched SFT parent | 200 | +1.5 pp | −0.0032m | −0.0384m |
The submitted paper reports the following paired-task statistics.
| Comparison | SR change (pp) | Paired-task 95% interval |
|---|---|---|
| Eight-family data adaptation | +9.9 | [7.2, 12.6] |
| SFT over T2A | +37.0 | [30.0, 44.0] |
| PPO over SFT | +2.0 | [−1.5, 5.5] |
| DepthAux over SFT | +0.5 | [−4.5, 5.5] |
These intervals quantify task variation for fixed checkpoints. The additional independent runs provide the training-repeat measurements, with the three-run mean and sample standard deviation to follow completion of the remaining evaluations.
Q6 FDE discrepancy
Tables A8 and A9 give different FDE for the same checkpoints and even flip whether FDE improves on average - which one is correct?
A6. Thank you for checking the numerical consistency. A8 contains an FDE transcription error introduced when duplicate appendix columns were removed and the table was reformatted. A9 agrees with the task-level records for the same 200 tasks. The correction is
| Statistic | Original A8 FDE w/o / w/ (m) | Correct FDE w/o / w/ (m) |
|---|---|---|
| VAMPO | 1.545 / 1.302 | 1.777 / 1.715 |
| Eight-family mean | 1.697 / 1.602 | 1.837 / 1.890 |
FDE retains the Euclidean distance from each final position to its reference endpoint, averaged equally over 200 tasks. Unrounded VAMPO values are 1.776837/1.714779m, and the eight-family means are 1.836977/1.890251m. Supplementary Table S7 lists all corrected family values. QwenVLA-style SFT reduces FDE from 3.200m to 1.403m, alongside the reported improvements in SR, nDTW, and ATE. Supplementary Table S7 provides the corrected family-level endpoint statistics.
Thanks for the response, but the reference baseline does no better than hovering and the input-mismatch run is still missing and hence i'll keep the score. Few questions :-
- Since the hover gets a 41% SR almost same as the ref baseline and the 8 family ref mean, what does the w/ vs w/o gain actually show?
- Your A2 table confirms the reference baseline trains on reference derived state but flies on executed state - could you still add the run with reference targets and executed state input?
- The 20% mass change barely moved the paths. Could you perhaps change something that actually alters the flown response and test both supervision types?
Response to Reviewer uLTN (1/2)
Response to Reviewer uLTN
We thank you for recognizing the common PX4 testbed and broad comparisons, and for your specific suggestions on calibration, inputs, and result consistency.
Q1 Hover and recorded flights
A1. Thank you for suggesting this direct benchmark calibration. We compare the source paths for the 200 tasks, actual PX4 hover with the same starting conditions and time budgets, and both learned policies using common metrics.
The sources contain 155 recorded executed paths and 45 synthetic translation references. The first two rows compare each source path with its own reference, calibrating the reference-path scores. The hover row uses measured PX4 states obtained while continuously sending hold targets. Its OR@1m and OR@3m are 59.0% and 82.0%.
To distinguish stationary sampling from spatial motion, arc-length nDTW resamples paths at fixed spatial intervals before applying the original DTW cost and exponential normalization. Ours improves over hover by 23.5 percentage points in SR and 0.186 in 0.1m arc-length nDTW. The policy ordering agrees at 0.1m and 0.2m spacing. SR follows the original reported evaluation protocol, while arc-length nDTW provides an additional spatial-path diagnostic.
Hover uses 50Hz physics updates and the nearest measured state at each 20Hz scoring boundary, with a maximum offset of 10ms. A separate matched 200-response replay yields position RMSE/ATE/FDE of 0.455/0.374/0.603m, quantifying re-execution through the seven-channel mapping. Supplementary Tables S2–S3 give settings and metric formulas.
Q2 Executed-state input held fixed
A2. Your question identifies the relationship between training-time motion states and online executed states. D.1 and F.3 specify the conditions below.
SUDA supervision constructs the current motion context from measured states and predicts their subsequent measured responses, aligning training data with the physical quantities observed during online execution. Both conditions share RGB, language, architecture, and training settings. Tables 3 and A11 report an SR improvement from 41.5% to 64.5%, an nDTW improvement from 0.336 to 0.628, and ATE/FDE reductions of 0.905/1.797m.
Q3 GPT-5.5 Policy
A3. Thank you for requesting this clarification. GPT-5.5 Policy directly calls the official multimodal API and receives zero task-specific training on SUDA-UAV. It evaluates a general multimodal model used directly as a flight policy and validates the platform's unified closed-loop execution and evaluation workflow.