acceptodds
Under review as a conference paper at ICLR 2027

AeroManip-VLA: Scaling Vision-Language-Action Learning for Aerial Manipulation with RL-Generated Demonstrations

Abstract

Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities for general-purpose manipulation. However, extending Vision-Language-Action (VLA) models to aerial robots introduces distinct challenges due to the tight coupling between manipulation and flight, continuously changing observations, and safety-critical physical interactions. These challenges demand diverse training data and systematic policy evaluation, yet collecting demonstrations and evaluating policies directly on physical aerial platforms are costly, difficult to scale, and hard to repeat under controlled conditions. We present AeroManip-VLA, a scalable benchmark for aerial VLA data generation and policy evaluation. AeroManip-VLA provides a GPU-accelerated simulation framework with low-level payload-aware flight and manipulation control in massively parallel environments. Building on this framework, we combine reusable reinforcement learning policies with expert task rules to automatically generate demonstrations without human teleoperation across diverse objects, environments, and randomized initial conditions. The generated data include basic skills such as grasping and placing, as well as long-horizon tasks that require both navigation and manipulation. We further introduce automated event labeling and trajectory categorization to filter demonstrations. These mechanisms enable fine-grained analysis of task progress, behavioral outcomes, and safety-related failures. Finally, we evaluate a range of imitation learning and VLA baselines across different task settings, revealing their performance characteristics and failure modes. Together, \methodname enables scalable aerial manipulation data generation, structured trajectory analysis, and systematic VLA evaluation in simulation prior to real-world deployment.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language-Action, Aerial Manipulation, Reinforcement Learning, Automated Data Generation, Learning from Demonstrations, Robotic Datasets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VoVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an automated data construction pipeline with several methods including hybrid expert, behavior cloning and RL. A large-scale dataset is set up with 80K+ episode. Typical VLA methods are evaluated.

Critical strengths
  1. Large-scale dataset for aerial manipulation
  2. Automated data construction pipeline
Critical weaknesses
  1. The paper highlights the advantages of RL-generated demonstrations in terms of trajectory diversity and execution efficiency. However, the proportion of demonstrations generated by RL, BC, and the hybrid expert in the final dataset remains unclear. The authors should clarify the data composition and the contribution of RL-generated demonstrations.
  2. The dataset is substantially larger than existing aerial manipulation datasets, but the benefits of this increase are not sufficiently demonstrated. Experiments with different training data sizes would help clarify how dataset scale affects model.
  3. The paper presents complex tasks such as Pick–Navigate–Place and Open–Pick–Navigate–Place, but Table 1 mainly reports results for individual Pick and Place skills. Quantitative evaluation of complete long-horizon tasks, including success rates and failure analysis across stages, would better support the paper's claims.
Questions for authors

While the paper demonstrates improved simulation throughput, the scalability of the data generation and learning pipeline remains less clear. How does increasing the dataset size affect VLA performance and data collection efficiency?

6 comments
AuthorsOpenReview ↗

Response to Reviewer VoVq’s Weaknesses 1–3

Response to VoVq-W1

Thank you for highlighting this ambiguity. All released episodes are collected using frozen PPO manipulation policies with rule-based navigation and task-stage coordination. Expert, BC, and PPO are compared as alternative generation strategies, but hybrid-expert and standalone BC rollouts are not mixed into the released dataset.

Contribution of RL-generated demonstrations. PPO supplies the learned manipulation policies used to generate the released demonstrations. It optimizes task rewards through environment interaction, while standalone BC imitates expert actions. Our trajectory analysis on TidyHouse and PrepareGroceries shows that PPO produces a broader range of successful approach trajectories than the hybrid expert and BC. Once trained, these policies are frozen and reused for automatic demonstration collection across randomized initial states. For quantitative evidence of downstream training benefits, please see our response to W2, which compares PPO-generated and hybrid-expert demonstrations across data budgets on PrepareGroceries and LongHorizonRearrangement.

We will correct the dataset-source description in Section 4.1, explicitly report this composition, and clarify the role of PPO in the data-collection pipeline.

Response to VoVq-W2

Thank you for requesting experiments at different training data sizes. We compare PPO-generated and hybrid-expert demonstrations at five budgets on two task suites.

Results on PrepareGroceries using ACT

Demos denotes the number of training demonstrations per skill, with the same budget used separately for Pick and Place. Success is the unweighted mean of the two skill success rates.

Demos per skillPPO demonstrationsHybrid expert demonstrations
14.1%0.0%
1015.5%5.3%
10025.1%15.8%
50028.0%16.4%
100030.5%16.9%

With PPO-generated demonstrations, ACT's mean success increases from 4.1% to 30.5%. From 100 to 1,000 demonstrations per skill, the gain is 5.4 percentage points, compared with 1.1 percentage points for hybrid-expert demonstrations.

Results on LongHorizonRearrangement using π0.5\pi_{0.5}

We also train π0.5\pi_{0.5} with 1, 10, 100, 500, and 1,000 demonstrations per task from each source. Success is averaged over all ten LongHorizonRearrangement tasks: four Open/Close tasks, four Open–Place variants, and two Open–Pick–Navigate–Place tasks.

Demos per taskPPO demonstrationsHybrid expert demonstrations
15.0%0.0%
1025.0%8.2%
10037.2%27.0%
50042.4%29.4%
100051.4%28.2%

With PPO-generated demonstrations, mean success increases from 5.0% to 51.4%. From 100 to 1,000 demonstrations per task, it improves from 37.2% to 51.4% (a gain of 14.2 percentage points), while hybrid-expert results remain between 27.0% and 29.4%. At 1,000 demonstrations per task, PPO-generated demonstrations yield a 23.2-percentage-point advantage. This provides additional evidence of data-budget benefits on the evaluated LongHorizonRearrangement task suite.

Response to VoVq-W3

All six models use the same complete-task success criterion on TidyHouseLong. Under Full VLA, GR00T succeeds in 4/450 episodes (0.9%): 2% on Master Chef Can, 2% on Tuna Fish Can, 4% on Bowl, and 0% on other objects. The other five models achieve 0%.

TidyHouseLong: complete-task success across three inference modes

Each model runs 450 episodes per mode. Full VLA controls Pick, Nav, and Place. Rule-based Nav replaces only Nav. Hybrid expert Pick uses expert Pick, rule-based Nav, and learned Place. All starting episodes, including Pick/Nav failures, enter the denominator.

ModelFull VLARule-based NavHybrid expert Pick
ACT0.0%0.0%16.7%
DP0.0%0.4%19.6%
π0\pi_0-FAST0.0%0.0%14.2%
π0\pi_00.0%0.0%10.9%
π0.5\pi_{0.5}0.0%0.0%15.8%
GR00T N1.70.9%0.4%25.1%

Pick dominates Full VLA failures: 450/450 for ACT, DP, π0\pi_0, and π0.5\pi_{0.5}; 448/450 for π0\pi_0-FAST; and 433/450 for GR00T. Unreached stages cannot be judged reliable from zero failures.

Hybrid expert Pick

ModelPick failuresNav failuresPlace failuresSuccessesOverall S.R. (%)
ACT160281877516.7
DP159271768819.6
π0\pi_0-FAST160271996414.2
π0\pi_0160272144910.9
π0.5\pi_{0.5}160291907115.8
GR00T N1.71592914911325.1
AuthorsOpenReview ↗

Response to Reviewer VoVq’s Weakness 3 (continued)

LongHorizonRearrangement: detailed results

Each method runs 50 episodes per task. Entries are success rate (%) ± binomial standard error; summary rows are unweighted means over the indicated tasks.

TaskACTDPπ0\pi_0-FASTπ0\pi_0π0.5\pi_{0.5}
Open (Fridge)68.0 ± 6.676.0 ± 6.052.0 ± 7.168.0 ± 6.676.0 ± 6.0
Close (Fridge)46.0 ± 7.044.0 ± 7.046.0 ± 7.074.0 ± 6.264.0 ± 6.8
Open (Kitchen Counter)0.0 ± 0.068.0 ± 6.648.0 ± 7.182.0 ± 5.468.0 ± 6.6
Close (Kitchen Counter)14.0 ± 4.954.0 ± 7.044.0 ± 7.062.0 ± 6.968.0 ± 6.6
Open–Pick–Navigate–Place (Fridge)0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.0
Open–Pick–Navigate–Place (Kitchen Counter)0.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.04.0 ± 2.8
Open–Place (Kitchen Counter)0.0 ± 0.018.0 ± 5.46.0 ± 3.498.0 ± 2.084.0 ± 5.2
Open–Place (Apple, Kitchen Counter)0.0 ± 0.012.0 ± 4.62.0 ± 2.098.0 ± 2.060.0 ± 6.9
Open–Place–Close (Apple, Kitchen Counter)0.0 ± 0.04.0 ± 2.80.0 ± 0.088.0 ± 4.640.0 ± 6.9
Open–Place–Close (Kitchen Counter)0.0 ± 0.08.0 ± 3.80.0 ± 0.080.0 ± 5.732.0 ± 6.6
Mean S.R.: Open/Close (4 tasks)32.0%60.5%47.5%71.5%69.0%
Mean S.R.: multi-stage (6 tasks)0.0%7.0%1.3%61.0%37.0%
Mean S.R.: all 10 tasks12.8%28.4%19.8%65.2%49.8%

LongHorizonRearrangement: failure-stage analysis of Open–Pick–Navigate–Place

For each model and task, the 50 episodes are partitioned into terminal failure stages or complete success. Percentages use all 50 episodes, so each row sums to 100%; they are not conditional on reaching a stage.

TaskModelOpen failurePick failureNav failurePlace failureSuccess
FridgeACT8.0%64.0%24.0%4.0%0.0%
FridgeDP8.0%72.0%20.0%0.0%0.0%
Fridgeπ0\pi_0-FAST32.0%60.0%8.0%0.0%0.0%
Fridgeπ0\pi_012.0%48.0%18.0%22.0%0.0%
Fridgeπ0.5\pi_{0.5}8.0%48.0%26.0%16.0%2.0%
Kitchen-counter drawerACT100.0%0.0%0.0%0.0%0.0%
Kitchen-counter drawerDP4.0%78.0%18.0%0.0%0.0%
Kitchen-counter drawerπ0\pi_0-FAST98.0%2.0%0.0%0.0%0.0%
Kitchen-counter drawerπ0\pi_02.0%48.0%30.0%18.0%2.0%
Kitchen-counter drawerπ0.5\pi_{0.5}4.0%34.0%38.0%20.0%4.0%

Pick accounts for 48-72% of fridge episodes. In the drawer task, ACT and π0\pi_0-FAST fail at Open in 100% and 98%; DP fails at Pick in 78%. Nav plus Place account for 48% and 58% for π0\pi_0 and π0.5\pi_{0.5}, respectively.

Results on PackageDelivery: single-parcel delivery

Each model runs 50 episodes per setting; both metrics use all 50. One policy continuously executes pickup, navigation, placement, and retreat, without resets or scripted handovers. It outputs velocity commands and a speed mode (cruise 4.0 m/s; manipulation 0.6 m/s). π0.5\pi_{0.5} receives a language instruction with the destination door identifier; ACT/DP receive its one-hot encoding. No target coordinates are provided.

ModelShort-range (truck): complete-task success (%)Short-range (truck): pickup success (%)Long-range (delivery depot): complete-task success (%)Long-range (delivery depot): pickup success (%)
ACT0.0%30.0%0.0%32.0%
DP0.0%4.0%0.0%2.0%
π0.5\pi_{0.5}14.0%76.0%10.0%64.0%

Results on PackageDelivery: 3-DoF arm parcel tasks

We evaluate each model on 30 distinct task scenarios: 15 single-parcel and 15 two-parcel tasks. Each model runs 30 Pick episodes and 45 Place episodes, with each scenario evaluated once for Pick and each parcel once for Place. Pick succeeds when all parcels assigned to the task are loaded into the onboard basket. Each Place episode evaluates one parcel after scripted loading and rule-based navigation to the destination. The denominators are 30 tasks for Pick and 45 parcels for Place.

ModelPick success (%)Per-parcel placement success (%)Task success (%)
ACT3.3%93.3%3.3%
DP0.0%0.0%0.0%
π0.5\pi_{0.5}36.7%77.8%23.3%
AuthorsOpenReview ↗

Response to Reviewer VoVq’s Question 1

Effect on policy performance. Please see our response to W2, which compares five demonstration budgets and two data sources on PrepareGroceries and LongHorizonRearrangement.

For data generation, we collect demonstrations automatically across GPU-parallel environments, reusing frozen PPO manipulation policies together with rule-based navigation and stage coordination, without per-episode human teleoperation. Tables Q1-A and Q1-B provide the simulation/rendering throughput and configurations. Under our native RGB 240 Hz configuration, throughput increases from 11.5 samples/s with one environment to 2,132.1 samples/s with 3,904 parallel environments on a single RTX 5090.

Data collection efficiency. On a single RTX 5090 GPU, our pipeline collects and saves approximately 250 complete trajectories per hour, including policy inference, environment resets, and disk writing. The rate varies by task. Larger datasets are obtained by collecting additional episodes with the same frozen policies, without retraining the demonstration generator.

Table Q1-A. Simulation/rendering throughput and GPU memory usage. N is the number of parallel environments. Each cell reports samples/s / total GPU memory usage in GB, measured on the same RTX 5090. The columns marked with an asterisk (*) correspond to the configurations actually used by the two benchmarks: Ours RGB 240 Hz* and AIR-VLA 640×480*. An em dash indicates that no measurement is reported for that configuration and N. Configurations are listed in Table Q1-B.

NOurs RGB 60 HzOurs RGB 240 Hz*Ours RGB-D 240 HzAIR-VLA 128×128AIR-VLA 640×480*
129.2 / 3.2911.5 / 3.2911.4 / 3.3923.0 / 7.4222.4 / 7.78
255.0 / 3.3021.5 / 3.3021.4 / 3.3944.7 / 7.5442.2 / 8.27
4109.8 / 3.3042.6 / 3.3042.2 / 3.3985.6 / 7.5975.2 / 8.71
8200.9 / 3.4478.9 / 3.4478.4 / 3.53164.1 / 7.75127.0 / 10.30
16369.8 / 3.51147.3 / 3.51145.9 / 3.62301.3 / 8.04195.9 / 12.91
32685.1 / 3.87291.0 / 3.87291.1 / 3.98523.4 / 8.29266.3 / 20.33
48————309.4 / 26.01
641142.1 / 4.24508.8 / 4.24504.8 / 4.36818.4 / 9.05317.3 / 30.46
1281664.2 / 4.87809.4 / 4.87803.8 / 5.011065.7 / 11.15—
2562212.0 / 6.451181.8 / 6.451177.9 / 6.651282.6 / 13.85—
5122462.2 / 8.081360.8 / 7.991357.8 / 8.251188.4 / 20.05—
768———1265.7 / 25.51—
10242795.8 / 12.161710.3 / 12.021706.9 / 12.421370.1 / 30.36—
1152———1413.5 / 32.70—
20483081.5 / 21.771961.5 / 21.561964.9 / 22.17——
39043412.1 / 31.152132.1 / 31.512137.0 / 32.66——

Table Q1-B. Settings for the five configurations in Table Q1-A.

SettingOurs RGB 60 HzOurs RGB 240 Hz*Ours RGB-D 240 HzAIR-VLA 128×128AIR-VLA 640×480*
SimulatorManiSkill/SAPIENManiSkill/SAPIENManiSkill/SAPIENIsaac Sim 6.1Isaac Sim 6.1
Image resolution128×128128×128128×128128×128640×480
ModalityRGBRGBRGB-DRGBRGB
Physics rate60 Hz240 Hz240 Hz60 Hz60 Hz
Control rate20 Hz20 Hz20 Hz20 Hz20 Hz
Reviewer_RhDCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents AeroManip-VLA, a scalable benchmark for aerial manipulation with vision-language-action (VLA) models. Its core idea is to replace the human teleoperation used by prior work with an automated demonstration-generation pipeline driven by a "privileged hybrid expert + reinforcement learning (PPO) policy." The system contributions are: (i) GPU-accelerated massively parallel simulation built on ManiSkill3/SAPIEN; (ii) teleoperation-free automated demonstration generation; and (iii) an event-based trajectory annotation and outcome-categorization scheme that supports fine-grained failure analysis.

Critical strengths

The research motivation is meaningful, and this paper provides a dedicated benchmark for aerial VLA. Aerial manipulation previously lacked a systematic and reproducible evaluation setup. AeroManip-VLA offers a unified platform for aerial VLA data generation and policy evaluation, which is a useful direction.

** The paper substantially scales up the data compared with prior work.** By combining a privileged hybrid expert with reinforcement-learning policies for automated generation, this work scales the dataset to over 80K demonstrations without any teleoperation, which is a substantial advance in data volume.

The experimental analysis validates the effectiveness of PPO and shows that it finds faster solution routes. This paper uses trajectory diversity, spatial coverage, velocity-direction entropy, and execution time to show that, although PPO is less training-efficient than BC, it discovers successful paths that are largely absent from expert and BC data, and these paths execute faster, which indicates that PPO yields higher-quality, near-optimal demonstration data.

Critical weaknesses

This paper does not report the data composition across task types, and gives little discussion of extending the benchmark toward aerial VLN. The benchmark does include long-horizon tasks, but I am interested in how much of the 80K+ dataset these long-horizon or long- distance tasks actually account for. The main text provides no distribution of data by task type or skill, and reports only the aggregate volume. Moreover, the core advantage of an aerial platform lies in large-space, long-distance 3D mobility. If the Nav skill and the long-horizon tasks could be further extended into a joint navigation-and-manipulation evaluation over large-scale scenes driven by language instructions, it would be far more aligned with the practical value of aerial operation than the current tabletop-style pick-and-place tasks.

The paper evaluates too few methods, with results from only four existing baselines. At present the downstream evaluation includes only ACT, Diffusion Policy, π₀, and π₀.₅, and focuses on the Pick/Place skills. I would like to see the authors expand it to include more recent VLA and visuomotor policies, as well as methods adapted to the aerial setting, so that the benchmark's evaluation value and its usefulness as a reference for future work are more fully demonstrated.

Questions for authors

Q1. How exactly is the PPO reward designed, and is it sparse or dense? The main text states only that reward functions are defined according to task progress and success criteria, together with penalties for undesirable actions and contacts, without giving the reward terms or weights. Please clarify whether the reward is sparse (given only upon success) or dense (with staged shaping for approach, alignment, and grasp), and how each stage is defined.

Q2. What is the composition of the released dataset, and the task-completion time statistics per task? Section 4.1 states that the released dataset draws primarily on the hybrid expert, with BC and PPO studied as data-generation strategies for comparison. On this basis, I would like to know two things: first, the data composition of the final 80K+ dataset broken down by task type or skill (short-horizon Pick/Place versus long-horizon Pick-Nav-Place, Open-Pick-Nav-Place, and so on); and second, beyond the TidyHouse and PrepareGroceries results already reported in Figure 4, the average completion time or trajectory length of the expert policy on the remaining task types.

Q3. Which configuration actually generated the released dataset? The paper is inconsistent in how it describes the data source: the abstract and Figure 1 call the data RL-generated expert trajectories, Section 4.1 instead states that the released dataset draws on the hybrid expert, while Figures 2-4 treat Expert, BC, and PPO as three parallel and mutually compared data-generation strategies. Please clarify which configuration produced the final 80K+ dataset, whether the frozen learned components embedded in the hybrid expert were trained by BC or by PPO, and if PPO was used, whether it is the same policy as the PPO treated as an independent comparison in Figures 2-4.

9 comments
AuthorsOpenReview ↗

Response to Reviewer RhDC’s Weakness 1

Response to RhDC-W1-1

Thank you for highlighting task composition and language-guided aerial operation. We classify tasks containing Nav as long-horizon, long-distance tasks:

DatasetEpisodes without NavEpisodes with NavTotal
PrepareGroceries16,000016,000
TidyHouse18,0009,00027,000
LongHorizonRearrangement8,0002,00010,000
PackageDelivery040,00040,000
Total (current inventory)42,00051,00093,000

PackageDelivery task composition. Each setting covers Cracker Box, Sugar Box, Pudding Box, Gelatin Box, and Parcel Box, with 1,000 episodes per task-object combination (40 combinations in total).

OriginDestination scopeOperationCollection platformObjectsEpisodes
TruckSingle destinationDelivery1-DoF arm55,000
TruckSingle destinationCollection1-DoF arm55,000
TruckMultiple destinationsDelivery1-DoF arm55,000
TruckMultiple destinationsCollection1-DoF arm55,000
Delivery depot / warehouseSingle destinationDelivery3-DoF arm55,000
Delivery depot / warehouseSingle destinationCollection3-DoF arm55,000
Delivery depot / warehouseMultiple destinationsDelivery3-DoF arm55,000
Delivery depot / warehouseMultiple destinationsCollection3-DoF arm55,000
Total40,000

Nav-containing tasks account for 51,000 / 93,000 = 54.8% of all episodes. PrepareGroceries contains Pick and Place. TidyHouse contains Pick, Place, and long tasks across nine objects; all 9,000 long-task episodes include Nav. LongHorizonRearrangement comprises 4,000 Open/Close episodes, 4,000 Open-Place variants, and 2,000 Open-Pick-Navigate-Place episodes. Counts include successful, failed, and timed-out episodes, classified by assigned task.

Response to RhDC-W1-2

Results on PackageDelivery: single-parcel delivery

We extend evaluation to outdoor truck and delivery-depot settings. Each model runs 50 episodes per setting; complete-task and pickup success both use all 50 episodes. The same policy continuously controls pickup, flight, placement, and retreat, without resets or scripted handovers. It outputs velocity commands and a speed mode (cruise 4.0 m/s; manipulation 0.6 m/s), with no rule-based navigator. π0.5\pi_{0.5} receives a language instruction containing the destination door identifier; ACT/DP receive its one-hot encoding. No target coordinates are provided.

ModelShort-range (truck): complete-task success (%)Short-range (truck): pickup success (%)Long-range (delivery depot): complete-task success (%)Long-range (delivery depot): pickup success (%)
ACT0.0%30.0%0.0%32.0%
DP0.0%4.0%0.0%2.0%
π0.5\pi_{0.5}14.0%76.0%10.0%64.0%

Results on PackageDelivery: 3-DoF arm parcel tasks

Navigation is rule-based; learned Pick and Place are evaluated independently. Before each Place episode, a scripted expert completes loading and navigation to the handover point. Each segment receives a language instruction with parcel label, cargo slot, and address for π\pi-family policies; ACT/DP receive segment type and a one-hot cargo-slot-to-address mapping.

Each model runs 30 Pick episodes for 30 distinct scenarios (15 single-parcel, 15 two-parcel) and 45 Place episodes, one per parcel, without repeated trials. Pick succeeds when all assigned parcels enter the onboard basket. Pick success uses 30 tasks; placement success uses 45 parcels. Composite task success pairs each task's Pick result with all its one or two Place results, requiring all to succeed (30-task denominator); it is not measured in a continuous rollout.

ModelPick success (%)Per-parcel placement success (%)Composite task success (%)
π0.5\pi_{0.5}36.777.823.3
π0\pi_0-FAST6.762.23.3
ACT3.393.33.3
π0\pi_00.020.00.0
DP0.00.00.0

The single-parcel setting already combines language-specified destinations with continuous navigation and manipulation. For a modular extension, the intermediate Nav stage could be handled by an aerial VLN model that navigates to the language-specified interaction location, while manipulation policies handle the preceding and subsequent object interactions. We will add the task breakdown, results, and distinct evaluation protocols to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer RhDC’s Weakness 2

Response to RhDC-W2

We add π0\pi_0-FAST and GR00T N1.7 to the four existing baselines. The table reports unweighted mean success across 16 PrepareGroceries and 18 TidyHouse Pick/Place-object combinations.

ModelPrepareGroceries Mean S.R. (%)TidyHouse Mean S.R. (%)
ACT29.527.8
DP33.927.3
π0\pi_0-FAST42.025.7
π0\pi_037.526.7
π0.5\pi_{0.5}44.625.2
GR00T N1.744.530.9

Adaptation to the aerial embodiment. Following the policy-controller decomposition in Tucker et al. [1], we adapt input/output interfaces without changing VLA architectures. For the 1-DoF PrepareGroceries/TidyHouse platform, inputs are three RGB views (front, downward, gripper), language, and a 19-D state: gripper joint positions/velocities, base position, 6-D rotation, and linear/angular velocities. Policies predict 16-step chunks of world-frame commands [vx,vy,vz,ψ˙,g][v_x,v_y,v_z,\dot{\psi},g] (velocity, yaw rate, gripper); a low-level flight controller tracks the flight commands. π0\pi_0-FAST is fine-tuned from its released base checkpoint with FAST-tokenized chunks and a larger token budget. GR00T N1.7 is fine-tuned as a new embodiment, adds 3-D end-effector position to the state (22-D), and treats the same command outputs as absolute, non-end-effector actions.

[1] Tucker et al. π\pi, But Make It Fly: Physics-Guided Transfer of VLA Models to Aerial Manipulation. CoRL 2026.

LongHorizonRearrangement: five-model evaluation

Each model runs 50 episodes/task (500 total): four Open/Close, four Open-Place variants, and two Open-Pick-Navigate-Place tasks. Entries show success (%) with binomial standard error; summaries are unweighted task means.

TaskACTDPπ0\pi_0-FASTπ0\pi_0π0.5\pi_{0.5}
Open (Fridge)68.0 ± 6.676.0 ± 6.052.0 ± 7.168.0 ± 6.676.0 ± 6.0
Close (Fridge)46.0 ± 7.044.0 ± 7.046.0 ± 7.074.0 ± 6.264.0 ± 6.8
Open (Kitchen Counter)0.0 ± 0.068.0 ± 6.648.0 ± 7.182.0 ± 5.468.0 ± 6.6
Close (Kitchen Counter)14.0 ± 4.954.0 ± 7.044.0 ± 7.062.0 ± 6.968.0 ± 6.6
Open–Pick–Navigate–Place (Fridge)0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.0
Open–Pick–Navigate–Place (Kitchen Counter)0.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.04.0 ± 2.8
Open–Place (Kitchen Counter)0.0 ± 0.018.0 ± 5.46.0 ± 3.498.0 ± 2.084.0 ± 5.2
Open–Place (Apple, Kitchen Counter)0.0 ± 0.012.0 ± 4.62.0 ± 2.098.0 ± 2.060.0 ± 6.9
Open–Place–Close (Apple, Kitchen Counter)0.0 ± 0.04.0 ± 2.80.0 ± 0.088.0 ± 4.640.0 ± 6.9
Open–Place–Close (Kitchen Counter)0.0 ± 0.08.0 ± 3.80.0 ± 0.080.0 ± 5.732.0 ± 6.6
Mean S.R.: Open/Close (4 tasks)32.0%60.5%47.5%71.5%69.0%
Mean S.R.: multi-stage (6 tasks)0.0%7.0%1.3%61.0%37.0%
Mean S.R.: all 10 tasks12.8%28.4%19.8%65.2%49.8%

TidyHouseLong: fully learned Pick-Navigate-Place

All six models use the same complete-task success criterion. GR00T succeeds in 4/450 episodes (0.9%): 2% on Master Chef Can, 2% on Tuna Fish Can, 4% on Bowl, and 0% on other objects. The other five models achieve 0%.

TidyHouseLong: Hybrid expert Pick

The hybrid expert controls Pick, a rule-based navigator controls Nav, and the evaluated policy controls Place. All starting episodes count, including Pick/Nav failures. Entries show success (%) with binomial standard error. Overall S.R. is the unweighted nine-object mean; best results are bold.

ModelMaster Chef CanCracker BoxSugar BoxTomato Soup CanTuna Fish CanPudding BoxGelatin BoxPotted Meat CanBowlOverall S.R. (%)
ACT16.0 ± 5.218.0 ± 5.426.0 ± 6.212.0 ± 4.620.0 ± 5.734.0 ± 6.724.0 ± 6.00.0 ± 0.00.0 ± 0.016.7
DP16.0 ± 5.222.0 ± 5.916.0 ± 5.24.0 ± 2.820.0 ± 5.746.0 ± 7.032.0 ± 6.616.0 ± 5.24.0 ± 2.819.6
π0\pi_0-FAST22.0 ± 5.98.0 ± 3.812.0 ± 4.64.0 ± 2.822.0 ± 5.930.0 ± 6.522.0 ± 5.96.0 ± 3.42.0 ± 2.014.2
π0\pi_018.0 ± 5.410.0 ± 4.216.0 ± 5.26.0 ± 3.414.0 ± 4.914.0 ± 4.912.0 ± 4.62.0 ± 2.06.0 ± 3.410.9
π0.5\pi_{0.5}22.0 ± 5.914.0 ± 4.922.0 ± 5.96.0 ± 3.414.0 ± 4.936.0 ± 6.818.0 ± 5.44.0 ± 2.86.0 ± 3.415.8
GR00T N1.756.0 ± 7.020.0 ± 5.720.0 ± 5.728.0 ± 6.316.0 ± 5.230.0 ± 6.518.0 ± 5.432.0 ± 6.66.0 ± 3.425.1
AuthorsOpenReview ↗

Response to Reviewer RhDC’s Question 1: Pick

Response to RhDC-Q1

Thank you for requesting these details. All four manipulation skills use dense, state-dependent reward shaping, supplemented by terminal success and failure rewards. Pick and Place use state-dependent reward terms, including latched grasp/release flags; Open/Close uses a latched flag to switch between two groups of shaping terms.

rtenv=rtshape+b×1[successt]−2×1[failuret].r_t^{\mathrm{env}}=r_t^{\mathrm{shape}}+b\times\mathbb{1}[\mathrm{success}_t]-2\times\mathbb{1}[\mathrm{failure}_t].

Here, tt is the control-step index, rtenvr_t^{\mathrm{env}} is the environment reward, and rtshaper_t^{\mathrm{shape}} is the sum of the shaping terms. 1[C]\mathbb{1}[C] equals 1 when condition CC holds and 0 otherwise. Thus, success adds a bonus of bb, while failure subtracts 2 from the shaping reward.

The tables below give the complete weighted shaping terms and their activation conditions. For compactness, fk(x)=1−tanh⁡(kx)f_k(x)=1-\tanh(kx); indicator flags take values in {0,1}\{0,1\}. Symbols are defined separately for each skill. Terminal bonuses and termination conditions are summarized at the end.

1. Pick: approach, align, grasp, lift, and stabilize

The shaping reward is the sum of the following rows. These terms can be active simultaneously.

Reward componentWeighted contributionActivation / purpose
Approach0.1f2(dh)+0.1f2(dg)+0.1f10(dg)0.1f_2(d_h)+0.1f_2(d_g)+0.1f_{10}(d_g)Always active; coarse and fine approach shaping.
Yaw alignment0.05 near(1−held)f3(∣Δψ∣)0.05\,\mathrm{near}(1-\mathrm{held})f_3(\lvert\Delta\psi\rvert)Soft proximity weighting; disabled after the first grasp.
Grasp0.2g0.2gActive while the object is grasped.
Lift0.2g clip⁡(Δzobj/hlift,0,1)0.2g\,\operatorname{clip}(\Delta z_{\mathrm{obj}}/h_{\mathrm{lift}},0,1)Rewards lifting while maintaining the grasp.
Stable hold0.15gmin⁡(nstable/10,1)0.15g\min(n_{\mathrm{stable}}/10,1)Rewards consecutive stable grasp-and-lift steps.
Object displacement penalty−0.2(1−g)tanh⁡(10∥Δpobjxy∥)-0.2(1-g)\tanh(10\lVert\Delta p_{\mathrm{obj}}^{xy}\rVert)Penalizes horizontal object displacement without a grasp.
Action penalty−0.015∥at∥2-\frac{0.01}{5}\lVert a_t\rVert^2Always active.
Yaw penalty−0.03(1−near⋅need)ψ˙cmd2-0.03(1-\mathrm{near}\cdot\mathrm{need})\dot\psi_{\mathrm{cmd}}^2Reduced near the target when alignment is needed.

Gate definitions. gg indicates a current grasp; held\mathrm{held} records whether a grasp has ever occurred in this episode. The proximity gate is continuous:

\mathrm{need}=\mathbb{1}[\lvert\Delta\psi\rvert>0.05\ \wedge\ \mathrm{held}=0].$$ Here $\sigma$ is the logistic sigmoid; $d_h,d_g$ are vehicle distances to the pre-grasp hover and grasp targets (m), and $\Delta\psi$ is grasp-yaw error (rad). $\Delta z_{\mathrm{obj}}$ is lift height relative to the object's initial height, $h_{\mathrm{lift}}$ its object-specific threshold, and $n_{\mathrm{stable}}$ the consecutive stable grasp-and-lift count. $\Delta p_{\mathrm{obj}}^{xy}$ is object displacement from its initial horizontal position. $a_t$ is the normalized five-dimensional action, with yaw component $\dot\psi_{\mathrm{cmd}}$. $\operatorname{clip}$ truncates to the specified lower and upper bounds; $\lVert\cdot\rVert$ denotes the Euclidean norm, $\lvert\cdot\rvert$ absolute value, and $\wedge$ logical AND. **Execution constraint.** Independently of the soft reward gate, the policy's yaw command is executed only when $d_h<0.3$ m, $\lvert\Delta\psi\rvert>0.05$ rad, and no grasp has yet occurred; otherwise the executed yaw command is zero. Terminal rewards and termination conditions for all skills are given in Part 5.
Reviewer_WhBWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AeroManip-VLA is a simulation benchmark for vision-language-action (VLA) learning on aerial manipulators. The authors argue that flight and manipulation are tightly coupled, viewpoints change continuously, and contact changes the vehicle’s payload, so teleoperated data and real-robot evaluation do not scale. They build a ManiSkill3 simulator with ReplicaCAD scenes, an X500-class quadrotor (a fixed 1-DoF gripper or a 3-DoF arm), three onboard RGB-D views, and a 5-D velocity/yaw/gripper action interface. A payload-aware cascaded flight controller runs in GPU-parallel environments. Demonstrations are produced without human teleoperation by composing a privileged hybrid expert (geometric planning, rules, and scripted stages, with access to object poses and masses) with behavior cloning and PPO skill policies. The task suite is TidyHouse, PrepareGroceries, PackageDelivery, and LongHorizonRearrangement, built from Pick, Place, Open, Close, and Nav. They also define an event taxonomy for progress, failures, and safety. The paper claims more than 80K trajectories.

Critical strengths

Payload-aware flight control is tested as a mechanism, not only named as a module. This is the strongest part of the paper and the reason I do not recommend a clear rejection. The authors derive the integrator state needed to hover with a payload, ξ⋆z = g mobj/m0, and then match it with per-class lift-off time, tilt-rate spectra, and failure counts. For aerial manipulation, a mass step at grasp enters the attitude loop directly. This analysis shows that the simulator is not simply attaching the object rigidly to the airframe.

Critical weaknesses

As a VLA scaling benchmark, the experiments do not cover the problem the paper defines. Four task suites, five skills, language instructions, and long-horizon Pick–Nav–Place and Open–Pick–Nav–Place are the paper’s selling point. Quantitative learned-policy results stop at short-horizon Pick/Place on PrepareGroceries. Figure 5’s long-horizon sequences are qualitative. There is no performance-versus-data curve, so scaling is not measured. AIR-VLA already evaluates long-horizon aerial VLAs on 3,000 teleoperated demonstrations. AM-Bench already provides 12 tasks, scripted experts, ACT, Diffusion Policy, π0, and π0.5, aerodynamic disturbances, and hardware validation. What this paper should add is large-scale demonstration generation without teleoperation, plus trajectory-level diagnosis. The main text does not show that these demonstrations improve long-horizon VLA performance, and it does not report the composition of the 80K trajectories (task, platform, expert versus PPO, success filter, splits). Appendix A.6 also states that links are rigid and that flexible fingers, motor lag, propeller wash, and ground effect are absent, which are factors AM-Bench already includes. The benchmark is less aerial-specific than the closest concurrent work, and the VLA evaluation is narrower than the claims. This is the main reason for weak rejection. The demonstration generator is not reproducible, and the dataset depends on it. Section 3.3.1 says the teacher uses object poses, collision geometry, and task targets, and that it is used only to generate successful demonstrations, but it does not specify the stages, when RL is called, when scripts take over, or whether failures enter the 80K. Figure 1 says “RL-generated expert trajectories.” Section 4.1 recommends releasing hybrid-expert data, with BC as a cheap distillation. Appendix A.3 says RL policies generate the data. These describe different distributions. PPO trajectories are also more diverse and faster (3.8 s versus 15.8 s on TidyHouse). When the expert is scripted to fly conservatively, that gap alone does not show that PPO found a better strategy family. A benchmark whose core artifact has an unclear generator and unclear privileged inputs is hard for the community to adopt as a standard dataset.

Questions for authors

Answers to the following would materially change the assessment. Sim-to-real is already stated as future work, so it is not requested here. Clarification that determines whether the main results can be trusted. How do the 80K trajectories break down by task, skill, platform (1-DoF gripper or 3-DoF arm), and generator (privileged expert, BC, or PPO)? Are failures kept according to the appendix S/F/T labels, or are only successes kept? Were π0 and π0.5 fine-tuned on this data or evaluated zero-shot? What were the fine-tuning steps, action dimension, and number of evaluation episodes? Why do the standard errors for π0 Pick match about 30 episodes, while the other cells match about 50? Clarification that determines whether the “cross-task VLA benchmark” claim holds. Please report end-to-end success for the same baselines on Pick–Nav–Place and Open–Pick–Nav–Place. Clarification of how much privileged information contributes to Table 1. Do the evaluated ACT, Diffusion Policy, π0, and π0.5 policies all use simulator-provided object-mass feedforward? Please add their success rates under the nominal controller. The appendix already shows the expert falling from 92% to 41% on heavy objects, so it matters how much of the VLA number comes from this compensation. Whether the AIR-VLA comparison is valid. Please remeasure samples per second at matched image resolution, camera count, and physics rate. The current 5.4× cannot support a claim about parallel simulation capacity. Citation. The Related Work description of LodeStar (arXiv:2204.07876) does not match that paper. Please correct it and name the demonstration-generation paper that was intended. This affects confidence that the related work was checked.

17 comments
AuthorsOpenReview ↗

Response to Reviewer WhBW’s Weakness 1

As a VLA scaling benchmark, the experiments do not cover the problem the paper defines.

Thank you for pointing out the limited quantitative coverage of our evaluation. We provide additional results on LongHorizonRearrangement, TidyHouse, and PackageDelivery, expand the PrepareGroceries comparison to six baselines, and add both fully learned and hybrid-assisted TidyHouseLong evaluations.

Results on LongHorizonRearrangement

We evaluate five baselines on ten LongHorizonRearrangement tasks covering Open/Close, Open–Place variants, and Open–Pick–Navigate–Place.

Each method is evaluated over 50 episodes per task (500 in total). Task entries report success rate (%) ± binomial standard error. Summary rows report the unweighted mean success rate over the indicated tasks. The six multi-stage tasks comprise four Open–Place variants and two Open–Pick–Navigate–Place tasks.

For Open–Pick–Navigate–Place, the learned policy controls all stages, including navigation. Success requires completing the entire task in a continuous rollout, without stage resets or scripted takeovers.

TaskACTDPπ0\pi_0-FASTπ0\pi_0π0.5\pi_{0.5}
Open (Fridge)68.0 ± 6.676.0 ± 6.052.0 ± 7.168.0 ± 6.676.0 ± 6.0
Close (Fridge)46.0 ± 7.044.0 ± 7.046.0 ± 7.074.0 ± 6.264.0 ± 6.8
Open (Kitchen Counter)0.0 ± 0.068.0 ± 6.648.0 ± 7.182.0 ± 5.468.0 ± 6.6
Close (Kitchen Counter)14.0 ± 4.954.0 ± 7.044.0 ± 7.062.0 ± 6.968.0 ± 6.6
Open–Pick–Navigate–Place (Fridge)0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.0
Open–Pick–Navigate–Place (Kitchen Counter)0.0 ± 0.00.0 ± 0.00.0 ± 0.02.0 ± 2.04.0 ± 2.8
Open–Place (Kitchen Counter)0.0 ± 0.018.0 ± 5.46.0 ± 3.498.0 ± 2.084.0 ± 5.2
Open–Place (Apple, Kitchen Counter)0.0 ± 0.012.0 ± 4.62.0 ± 2.098.0 ± 2.060.0 ± 6.9
Open–Place–Close (Apple, Kitchen Counter)0.0 ± 0.04.0 ± 2.80.0 ± 0.088.0 ± 4.640.0 ± 6.9
Open–Place–Close (Kitchen Counter)0.0 ± 0.08.0 ± 3.80.0 ± 0.080.0 ± 5.732.0 ± 6.6
Mean S.R.: Open/Close (4 tasks)32.0%60.5%47.5%71.5%69.0%
Mean S.R.: multi-stage (6 tasks)0.0%7.0%1.3%61.0%37.0%
Mean S.R.: all 10 tasks12.8%28.4%19.8%65.2%49.8%

Across all ten tasks, π0\pi_0 achieves 65.2% mean success, followed by π0.5\pi_{0.5} (49.8%), DP (28.4%), π0\pi_0-FAST (19.8%), and ACT (12.8%). Performance varies substantially by task composition. On Open–Pick–Navigate–Place, π0\pi_0 achieves 0% and 2% in the fridge and kitchen-counter settings, while π0.5\pi_{0.5} achieves 2% and 4%; ACT, DP, and π0\pi_0-FAST achieve 0% in both settings. These complete navigation-and-manipulation chains remain challenging, despite higher success on several shorter compositions.

Results on PrepareGroceries and TidyHouse

We evaluate six baselines on Pick and Place across eight PrepareGroceries objects and nine TidyHouse objects. Mean S.R. is the unweighted average over the 16 and 18 skill–object combinations, respectively.

ModelPrepareGroceries Mean S.R. (%)TidyHouse Mean S.R. (%)
ACT29.527.8
DP33.927.3
π0\pi_0-FAST42.025.7
π0\pi_037.526.7
π0.5\pi_{0.5}44.625.2
GR00T N1.744.530.9

The added π0\pi_0-FAST and GR00T N1.7 results broaden model coverage on PrepareGroceries, while TidyHouse extends the quantitative evaluation to another task suite. The table above reports individual Pick/Place skill performance.

TidyHouseLong: fully learned Pick–Navigate–Place

All six models use the same full-task success criterion for fully learned Pick–Navigate–Place on TidyHouseLong. Each model is evaluated on 50 episodes per object, totaling 450 episodes. ACT, Diffusion Policy, π0\pi_0-FAST, π0\pi_0, and π0.5\pi_{0.5} all achieve 0.0% mean success across the nine objects. GR00T achieves 0.9% mean success: 2.0% on Master Chef Can, 2.0% on Tuna Fish Can, and 4.0% on Bowl, with zero success on the remaining objects. GR00T succeeds in 4 of 450 episodes.

AuthorsOpenReview ↗

Response to Reviewer WhBW’s Weakness 1 (continued)

TidyHouseLong: Hybrid expert Pick

We additionally evaluate six policies in a TidyHouseLong pipeline where the hybrid expert controls Pick, a rule-based navigator controls Navigate, and the evaluated policy controls Place. Success rates include all episodes started from the task initial state, including failures during Pick and Nav. Each object column reports success rate (%) with binomial standard error; Overall S.R. is the unweighted mean across the nine objects. Best results in each column are bold.

ModelMaster Chef CanCracker BoxSugar BoxTomato Soup CanTuna Fish CanPudding BoxGelatin BoxPotted Meat CanBowlOverall S.R. (%)
ACT16.0 ± 5.218.0 ± 5.426.0 ± 6.212.0 ± 4.620.0 ± 5.734.0 ± 6.724.0 ± 6.00.0 ± 0.00.0 ± 0.016.7
DP16.0 ± 5.222.0 ± 5.916.0 ± 5.24.0 ± 2.820.0 ± 5.746.0 ± 7.032.0 ± 6.616.0 ± 5.24.0 ± 2.819.6
π0\pi_0-FAST22.0 ± 5.98.0 ± 3.812.0 ± 4.64.0 ± 2.822.0 ± 5.930.0 ± 6.522.0 ± 5.96.0 ± 3.42.0 ± 2.014.2
π0\pi_018.0 ± 5.410.0 ± 4.216.0 ± 5.26.0 ± 3.414.0 ± 4.914.0 ± 4.912.0 ± 4.62.0 ± 2.06.0 ± 3.410.9
π0.5\pi_{0.5}22.0 ± 5.914.0 ± 4.922.0 ± 5.96.0 ± 3.414.0 ± 4.936.0 ± 6.818.0 ± 5.44.0 ± 2.86.0 ± 3.415.8
GR00T N1.756.0 ± 7.020.0 ± 5.720.0 ± 5.728.0 ± 6.316.0 ± 5.230.0 ± 6.518.0 ± 5.432.0 ± 6.66.0 ± 3.425.1

Results on PackageDelivery: single-parcel delivery

We evaluate ACT, DP, and π0.5\pi_{0.5} on single-parcel delivery in the short-range truck setting and the long-range delivery-depot setting, with 50 episodes per model in each setting. Complete-task success and pickup success are reported separately, both as percentages of all 50 episodes.

Single-parcel evaluation protocol. The same policy continuously executes pickup, flight, placement, and retreat, without stage resets or scripted handovers. Navigation is policy-controlled: the policy outputs velocity commands and a speed mode (4.0 m/s for cruising; 0.6 m/s for manipulation), with no rule-based navigator. For the evaluated π0.5\pi_{0.5} policy, the language instruction includes the destination door identifier; ACT and DP receive a one-hot encoding of that identifier. None of these policies receives target coordinates in the single-parcel evaluation.

ModelShort-range (truck): complete-task success (%)Short-range (truck): pickup success (%)Long-range (delivery depot): complete-task success (%)Long-range (delivery depot): pickup success (%)
ACT0.0%30.0%0.0%32.0%
DP0.0%4.0%0.0%2.0%
π0.5\pi_{0.5}14.0%76.0%10.0%64.0%

Results on PackageDelivery: 3-DoF arm parcel tasks

Navigation is rule-based; learned Pick and Place are evaluated independently. Before each Place episode, a scripted expert completes loading and navigation to the handover point. Each segment receives a language instruction with parcel label, cargo slot, and address for π\pi-family policies; ACT/DP receive segment type and a one-hot cargo-slot-to-address mapping.

Each model runs 30 Pick episodes for 30 distinct scenarios (15 single-parcel, 15 two-parcel) and 45 Place episodes, one per parcel, without repeated trials. Pick succeeds when all assigned parcels enter the onboard basket. Pick success uses 30 tasks; placement success uses 45 parcels. Composite task success pairs each task's Pick result with all its one or two Place results, requiring all to succeed (30-task denominator); it is not measured in a continuous rollout.

ModelPick success (%)Per-parcel placement success (%)Composite task success (%)
π0.5\pi_{0.5}36.777.823.3
π0\pi_0-FAST6.762.23.3
ACT3.393.33.3
π0\pi_00.020.00.0
DP0.00.00.0

The complete-task results and failure-stage analysis are further detailed in our response to Q6.

AuthorsOpenReview ↗

Response to Reviewer WhBW’s Weakness 2

There is no performance-versus-data curve, so scaling is not measured. AIR-VLA already evaluates long-horizon aerial VLAs on 3,000 teleoperated demonstrations.

Thank you for highlighting the need to clarify our contribution and quantify the benefits of increasing demonstration data.

Contribution relative to AIR-VLA and AM-Bench. AIR-VLA (arXiv:2601.21602) provides 3,000 teleoperated demonstrations and evaluates long-horizon aerial tasks. AM-Bench (arXiv:2609.00641) studies physical interaction, embodiment, policy–control interfaces, and disturbances; it also includes scripted data generation, data-scaling experiments, and diagnostic metrics. Our benchmark emphasizes household rearrangement and outdoor package-delivery workflows, including explicit Pick–Navigate–Place and Open–Pick–Navigate–Place compositions. Within this scope, our contribution connects automated demonstration generation, downstream data utility, and stage-resolved failure analysis.

Generation without teleoperation. Our pipeline combines frozen PPO manipulation policies with rule-based navigation and stage coordination. It produces a 93K-episode corpus across four task suites, retaining successful, failed, and timed-out episodes. The experiments below compare both demonstration budget and generation source.

Performance versus data. We use 1, 10, 100, 500, and 1,000 demonstrations per skill or task from PPO policies or the hybrid expert. Both demonstration-budget studies train on successful trajectories only, for both PPO and hybrid-expert data. The 1,000-episode inventory per task setting in W3 describes the released dataset; successful demonstrations for the scaling experiments are drawn from a larger training pool.

PrepareGroceries — ACT. Each budget applies separately to Pick and Place; reported success is their unweighted mean.

Demos per skillPPO demonstrationsHybrid expert demonstrations
14.1%0.0%
1015.5%5.3%
10025.1%15.8%
50028.0%16.4%
100030.5%16.9%

LongHorizonRearrangement — π0.5\pi_{0.5}. Success is averaged over all ten tasks: four Open/Close tasks, four Open–Place variants, and two Open–Pick–Navigate–Place tasks.

Demos per taskPPO demonstrationsHybrid expert demonstrations
15.0%0.0%
1025.0%8.2%
10037.2%27.0%
50042.4%29.4%
100051.4%28.2%

In both studies, success with PPO-generated data continues to improve from 100 to 1,000 demonstrations. On PrepareGroceries, the gain is 5.4 percentage points with PPO-generated data versus 1.1 with hybrid-expert data; on LongHorizonRearrangement, the corresponding gains are 14.2 and 1.2 points. At the largest budget, PPO-generated data yields gains of 13.6 and 23.2 percentage points, respectively, over hybrid-expert data.

Trajectory-level diagnosis. We analyze where complete-task execution fails. In Open–Pick–Navigate–Place from the fridge, Pick failures account for 48–72% of all episodes across five baselines. In the drawer setting, ACT and π0\pi_0-FAST fail at Open in 100% and 98% of episodes, respectively. For π0\pi_0 and π0.5\pi_{0.5}, Open failures fall to 2% and 4%, while Nav and Place failures together account for 48% and 58%.

On fully learned TidyHouseLong, 96.2–100% of episodes terminate at Pick. With Hybrid expert Pick and rule-based navigation, overall task success reaches 10.9–25.1% across the six models, counting all episodes, including failures in the assisted stages. These results identify different bottlenecks across tasks and models, providing more actionable diagnosis than aggregate success rates alone. Detailed stage-wise statistics are provided in Q6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.