- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an automated data construction pipeline with several methods including hybrid expert, behavior cloning and RL. A large-scale dataset is set up with 80K+ episode. Typical VLA methods are evaluated.
Critical strengths
- Large-scale dataset for aerial manipulation
- Automated data construction pipeline
Critical weaknesses
- The paper highlights the advantages of RL-generated demonstrations in terms of trajectory diversity and execution efficiency. However, the proportion of demonstrations generated by RL, BC, and the hybrid expert in the final dataset remains unclear. The authors should clarify the data composition and the contribution of RL-generated demonstrations.
- The dataset is substantially larger than existing aerial manipulation datasets, but the benefits of this increase are not sufficiently demonstrated. Experiments with different training data sizes would help clarify how dataset scale affects model.
- The paper presents complex tasks such as Pick–Navigate–Place and Open–Pick–Navigate–Place, but Table 1 mainly reports results for individual Pick and Place skills. Quantitative evaluation of complete long-horizon tasks, including success rates and failure analysis across stages, would better support the paper's claims.
Questions for authors
While the paper demonstrates improved simulation throughput, the scalability of the data generation and learning pipeline remains less clear. How does increasing the dataset size affect VLA performance and data collection efficiency?
6 comments
Response to Reviewer VoVq’s Weakness 3 (continued)
LongHorizonRearrangement: detailed results
Each method runs 50 episodes per task. Entries are success rate (%) ± binomial standard error; summary rows are unweighted means over the indicated tasks.
| Task | ACT | DP | -FAST | ||
|---|---|---|---|---|---|
| Open (Fridge) | 68.0 ± 6.6 | 76.0 ± 6.0 | 52.0 ± 7.1 | 68.0 ± 6.6 | 76.0 ± 6.0 |
| Close (Fridge) | 46.0 ± 7.0 | 44.0 ± 7.0 | 46.0 ± 7.0 | 74.0 ± 6.2 | 64.0 ± 6.8 |
| Open (Kitchen Counter) | 0.0 ± 0.0 | 68.0 ± 6.6 | 48.0 ± 7.1 | 82.0 ± 5.4 | 68.0 ± 6.6 |
| Close (Kitchen Counter) | 14.0 ± 4.9 | 54.0 ± 7.0 | 44.0 ± 7.0 | 62.0 ± 6.9 | 68.0 ± 6.6 |
| Open–Pick–Navigate–Place (Fridge) | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 2.0 ± 2.0 |
| Open–Pick–Navigate–Place (Kitchen Counter) | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 2.0 ± 2.0 | 4.0 ± 2.8 |
| Open–Place (Kitchen Counter) | 0.0 ± 0.0 | 18.0 ± 5.4 | 6.0 ± 3.4 | 98.0 ± 2.0 | 84.0 ± 5.2 |
| Open–Place (Apple, Kitchen Counter) | 0.0 ± 0.0 | 12.0 ± 4.6 | 2.0 ± 2.0 | 98.0 ± 2.0 | 60.0 ± 6.9 |
| Open–Place–Close (Apple, Kitchen Counter) | 0.0 ± 0.0 | 4.0 ± 2.8 | 0.0 ± 0.0 | 88.0 ± 4.6 | 40.0 ± 6.9 |
| Open–Place–Close (Kitchen Counter) | 0.0 ± 0.0 | 8.0 ± 3.8 | 0.0 ± 0.0 | 80.0 ± 5.7 | 32.0 ± 6.6 |
| Mean S.R.: Open/Close (4 tasks) | 32.0% | 60.5% | 47.5% | 71.5% | 69.0% |
| Mean S.R.: multi-stage (6 tasks) | 0.0% | 7.0% | 1.3% | 61.0% | 37.0% |
| Mean S.R.: all 10 tasks | 12.8% | 28.4% | 19.8% | 65.2% | 49.8% |
LongHorizonRearrangement: failure-stage analysis of Open–Pick–Navigate–Place
For each model and task, the 50 episodes are partitioned into terminal failure stages or complete success. Percentages use all 50 episodes, so each row sums to 100%; they are not conditional on reaching a stage.
| Task | Model | Open failure | Pick failure | Nav failure | Place failure | Success |
|---|---|---|---|---|---|---|
| Fridge | ACT | 8.0% | 64.0% | 24.0% | 4.0% | 0.0% |
| Fridge | DP | 8.0% | 72.0% | 20.0% | 0.0% | 0.0% |
| Fridge | -FAST | 32.0% | 60.0% | 8.0% | 0.0% | 0.0% |
| Fridge | 12.0% | 48.0% | 18.0% | 22.0% | 0.0% | |
| Fridge | 8.0% | 48.0% | 26.0% | 16.0% | 2.0% | |
| Kitchen-counter drawer | ACT | 100.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Kitchen-counter drawer | DP | 4.0% | 78.0% | 18.0% | 0.0% | 0.0% |
| Kitchen-counter drawer | -FAST | 98.0% | 2.0% | 0.0% | 0.0% | 0.0% |
| Kitchen-counter drawer | 2.0% | 48.0% | 30.0% | 18.0% | 2.0% | |
| Kitchen-counter drawer | 4.0% | 34.0% | 38.0% | 20.0% | 4.0% |
Pick accounts for 48-72% of fridge episodes. In the drawer task, ACT and -FAST fail at Open in 100% and 98%; DP fails at Pick in 78%. Nav plus Place account for 48% and 58% for and , respectively.
Results on PackageDelivery: single-parcel delivery
Each model runs 50 episodes per setting; both metrics use all 50. One policy continuously executes pickup, navigation, placement, and retreat, without resets or scripted handovers. It outputs velocity commands and a speed mode (cruise 4.0 m/s; manipulation 0.6 m/s). receives a language instruction with the destination door identifier; ACT/DP receive its one-hot encoding. No target coordinates are provided.
| Model | Short-range (truck): complete-task success (%) | Short-range (truck): pickup success (%) | Long-range (delivery depot): complete-task success (%) | Long-range (delivery depot): pickup success (%) |
|---|---|---|---|---|
| ACT | 0.0% | 30.0% | 0.0% | 32.0% |
| DP | 0.0% | 4.0% | 0.0% | 2.0% |
| 14.0% | 76.0% | 10.0% | 64.0% |
Results on PackageDelivery: 3-DoF arm parcel tasks
We evaluate each model on 30 distinct task scenarios: 15 single-parcel and 15 two-parcel tasks. Each model runs 30 Pick episodes and 45 Place episodes, with each scenario evaluated once for Pick and each parcel once for Place. Pick succeeds when all parcels assigned to the task are loaded into the onboard basket. Each Place episode evaluates one parcel after scripted loading and rule-based navigation to the destination. The denominators are 30 tasks for Pick and 45 parcels for Place.
| Model | Pick success (%) | Per-parcel placement success (%) | Task success (%) |
|---|---|---|---|
| ACT | 3.3% | 93.3% | 3.3% |
| DP | 0.0% | 0.0% | 0.0% |
| 36.7% | 77.8% | 23.3% |
Response to Reviewer VoVq’s Question 1
Effect on policy performance. Please see our response to W2, which compares five demonstration budgets and two data sources on PrepareGroceries and LongHorizonRearrangement.
For data generation, we collect demonstrations automatically across GPU-parallel environments, reusing frozen PPO manipulation policies together with rule-based navigation and stage coordination, without per-episode human teleoperation. Tables Q1-A and Q1-B provide the simulation/rendering throughput and configurations. Under our native RGB 240 Hz configuration, throughput increases from 11.5 samples/s with one environment to 2,132.1 samples/s with 3,904 parallel environments on a single RTX 5090.
Data collection efficiency. On a single RTX 5090 GPU, our pipeline collects and saves approximately 250 complete trajectories per hour, including policy inference, environment resets, and disk writing. The rate varies by task. Larger datasets are obtained by collecting additional episodes with the same frozen policies, without retraining the demonstration generator.
Table Q1-A. Simulation/rendering throughput and GPU memory usage. N is the number of parallel environments. Each cell reports samples/s / total GPU memory usage in GB, measured on the same RTX 5090. The columns marked with an asterisk (*) correspond to the configurations actually used by the two benchmarks: Ours RGB 240 Hz* and AIR-VLA 640×480*. An em dash indicates that no measurement is reported for that configuration and N. Configurations are listed in Table Q1-B.
| N | Ours RGB 60 Hz | Ours RGB 240 Hz* | Ours RGB-D 240 Hz | AIR-VLA 128×128 | AIR-VLA 640×480* |
|---|---|---|---|---|---|
| 1 | 29.2 / 3.29 | 11.5 / 3.29 | 11.4 / 3.39 | 23.0 / 7.42 | 22.4 / 7.78 |
| 2 | 55.0 / 3.30 | 21.5 / 3.30 | 21.4 / 3.39 | 44.7 / 7.54 | 42.2 / 8.27 |
| 4 | 109.8 / 3.30 | 42.6 / 3.30 | 42.2 / 3.39 | 85.6 / 7.59 | 75.2 / 8.71 |
| 8 | 200.9 / 3.44 | 78.9 / 3.44 | 78.4 / 3.53 | 164.1 / 7.75 | 127.0 / 10.30 |
| 16 | 369.8 / 3.51 | 147.3 / 3.51 | 145.9 / 3.62 | 301.3 / 8.04 | 195.9 / 12.91 |
| 32 | 685.1 / 3.87 | 291.0 / 3.87 | 291.1 / 3.98 | 523.4 / 8.29 | 266.3 / 20.33 |
| 48 | — | — | — | — | 309.4 / 26.01 |
| 64 | 1142.1 / 4.24 | 508.8 / 4.24 | 504.8 / 4.36 | 818.4 / 9.05 | 317.3 / 30.46 |
| 128 | 1664.2 / 4.87 | 809.4 / 4.87 | 803.8 / 5.01 | 1065.7 / 11.15 | — |
| 256 | 2212.0 / 6.45 | 1181.8 / 6.45 | 1177.9 / 6.65 | 1282.6 / 13.85 | — |
| 512 | 2462.2 / 8.08 | 1360.8 / 7.99 | 1357.8 / 8.25 | 1188.4 / 20.05 | — |
| 768 | — | — | — | 1265.7 / 25.51 | — |
| 1024 | 2795.8 / 12.16 | 1710.3 / 12.02 | 1706.9 / 12.42 | 1370.1 / 30.36 | — |
| 1152 | — | — | — | 1413.5 / 32.70 | — |
| 2048 | 3081.5 / 21.77 | 1961.5 / 21.56 | 1964.9 / 22.17 | — | — |
| 3904 | 3412.1 / 31.15 | 2132.1 / 31.51 | 2137.0 / 32.66 | — | — |
Table Q1-B. Settings for the five configurations in Table Q1-A.
| Setting | Ours RGB 60 Hz | Ours RGB 240 Hz* | Ours RGB-D 240 Hz | AIR-VLA 128×128 | AIR-VLA 640×480* |
|---|---|---|---|---|---|
| Simulator | ManiSkill/SAPIEN | ManiSkill/SAPIEN | ManiSkill/SAPIEN | Isaac Sim 6.1 | Isaac Sim 6.1 |
| Image resolution | 128×128 | 128×128 | 128×128 | 128×128 | 640×480 |
| Modality | RGB | RGB | RGB-D | RGB | RGB |
| Physics rate | 60 Hz | 240 Hz | 240 Hz | 60 Hz | 60 Hz |
| Control rate | 20 Hz | 20 Hz | 20 Hz | 20 Hz | 20 Hz |
Response to Reviewer VoVq’s Weaknesses 1–3
Response to VoVq-W1
Thank you for highlighting this ambiguity. All released episodes are collected using frozen PPO manipulation policies with rule-based navigation and task-stage coordination. Expert, BC, and PPO are compared as alternative generation strategies, but hybrid-expert and standalone BC rollouts are not mixed into the released dataset.
Contribution of RL-generated demonstrations. PPO supplies the learned manipulation policies used to generate the released demonstrations. It optimizes task rewards through environment interaction, while standalone BC imitates expert actions. Our trajectory analysis on TidyHouse and PrepareGroceries shows that PPO produces a broader range of successful approach trajectories than the hybrid expert and BC. Once trained, these policies are frozen and reused for automatic demonstration collection across randomized initial states. For quantitative evidence of downstream training benefits, please see our response to W2, which compares PPO-generated and hybrid-expert demonstrations across data budgets on PrepareGroceries and LongHorizonRearrangement.
We will correct the dataset-source description in Section 4.1, explicitly report this composition, and clarify the role of PPO in the data-collection pipeline.
Response to VoVq-W2
Thank you for requesting experiments at different training data sizes. We compare PPO-generated and hybrid-expert demonstrations at five budgets on two task suites.
Results on PrepareGroceries using ACT
Demos denotes the number of training demonstrations per skill, with the same budget used separately for Pick and Place. Success is the unweighted mean of the two skill success rates.
With PPO-generated demonstrations, ACT's mean success increases from 4.1% to 30.5%. From 100 to 1,000 demonstrations per skill, the gain is 5.4 percentage points, compared with 1.1 percentage points for hybrid-expert demonstrations.
Results on LongHorizonRearrangement using π0.5
We also train π0.5 with 1, 10, 100, 500, and 1,000 demonstrations per task from each source. Success is averaged over all ten LongHorizonRearrangement tasks: four Open/Close tasks, four Open–Place variants, and two Open–Pick–Navigate–Place tasks.
With PPO-generated demonstrations, mean success increases from 5.0% to 51.4%. From 100 to 1,000 demonstrations per task, it improves from 37.2% to 51.4% (a gain of 14.2 percentage points), while hybrid-expert results remain between 27.0% and 29.4%. At 1,000 demonstrations per task, PPO-generated demonstrations yield a 23.2-percentage-point advantage. This provides additional evidence of data-budget benefits on the evaluated LongHorizonRearrangement task suite.
Response to VoVq-W3
All six models use the same complete-task success criterion on TidyHouseLong. Under Full VLA, GR00T succeeds in 4/450 episodes (0.9%): 2% on Master Chef Can, 2% on Tuna Fish Can, 4% on Bowl, and 0% on other objects. The other five models achieve 0%.
TidyHouseLong: complete-task success across three inference modes
Each model runs 450 episodes per mode. Full VLA controls Pick, Nav, and Place. Rule-based Nav replaces only Nav. Hybrid expert Pick uses expert Pick, rule-based Nav, and learned Place. All starting episodes, including Pick/Nav failures, enter the denominator.
Pick dominates Full VLA failures: 450/450 for ACT, DP, π0, and π0.5; 448/450 for π0-FAST; and 433/450 for GR00T. Unreached stages cannot be judged reliable from zero failures.
Hybrid expert Pick