acceptodds
Withdrawn from ICLR 2027

AirPlayground: Benchmarking Aerial Navigation across Diverse Reconstructed Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large-scale aerial resources provide a foundation for learning navigation across diverse environments. However, existing resources remain limited in providing (1) densely sampled flight demonstrations for learning different policy formulations and (2) broad environment coverage for standardized closed-loop evaluation. To address these limitations, we introduce AirPlayground, a unified benchmark for training and evaluating aerial navigation policies across diverse environments. We first construct 94 photorealistic 3D Gaussian Splatting (3DGS) environments spanning indoor, urban outdoor, and vegetation settings, with aligned collision geometry. Building on these environments, we develop a scene-adaptive pipeline for automated expert flight collection, yielding 94K demonstrations and 46.5M synchronized observation–state–action samples at 50 Hz. We further provide an extensible API that supports training different policy formulations on this shared corpus and evaluating them under unified protocols in the corresponding environments. We benchmark seven baselines spanning four policy paradigms across three navigation settings in seen and unseen environments, and further examine scaling behavior with ACT. Scaling ACT's training corpus from 5 to 70 scenes raises median success rates on seen and unseen environments from 30.6% and 16.8% to 41.2% and 29.4%, respectively. Through this shared training and evaluation framework, AirPlayground supports analysis of how offline prediction errors relate to closed-loop outcomes. Our analyses highlight the roles of prediction errors, execution protocols, and scene interactions in closed-loop performance.

Primary area:
datasets and benchmarks
Keywords:
aerial navigation, embodied AI, 3D Gaussian splatting, closed-loop evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SAbuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AirPlayground, a benchmark for training and evaluating aerial navigation policies in reconstructed environments. It contains 94 3D Gaussian Splatting scenes, 94,000 expert trajectories, and 46.5 million synchronized samples. The benchmark provides reconstructed appearance, collision geometry, automated trajectory collection, and a common interface for different policy types. Seven baselines are evaluated on seen and unseen scenes, followed by studies of data scaling, execution horizons, and the relationship between offline prediction errors and closed-loop performance.

Critical strengths
  • AirPlayground provides broad scene coverage and a large number of dense flight demonstrations. Pairing demonstrations with executable reconstructed environments could be useful for studying closed-loop aerial navigation.
  • The paper documents the data-collection and evaluation pipelines in detail. The unified interface supports several policy output formats, and the execution-horizon experiments show that deployment settings can substantially affect navigation performance.
Critical weaknesses
  • Success, collision, task generation, and expert planning all rely on geometry produced from reconstructed point clouds through several heuristic and scene-specific processing steps. However, the paper does not measure the accuracy of the resulting obstacle boundaries or free space against independent ground truth. The reported image-quality metrics evaluate rendering rather than collision geometry and cover only a subset of the scenes. This is consequential because inaccurate geometry would directly change both the demonstrations and the reported closed-loop outcomes, making it unclear whether the benchmark measures navigation ability or artifacts of its reconstruction pipeline.

  • The evaluated policies differ in pretraining, optimization budget, adapters, output representations, and execution mechanisms, so their relative performance cannot be attributed cleanly to policy formulation. More importantly, the offline-to-closed-loop correlations are computed from only seven aggregated policy-level observations, without uncertainty estimates. The benchmark also considers only fixed-height planar velocity control. These limitations make the claims about policy comparison, deployment diagnosis, and aerial-navigation generalization substantially stronger than the available evidence.

Questions for authors
  • How was the accuracy of the reconstructed collision geometry evaluated? Can the authors compare obstacle boundaries and free space against independent measurements from the physical environments?
  • How sensitive are success and collision rates to point-cloud density, height filtering, component removal, voxel resolution, and the manually selected inflation radius?
  • The navigable polygon and several geometry parameters are defined separately for each scene. How much manual work was required, and how was annotation consistency maintained across the 94 scenes?
  • The paper states that 69.1% of the demonstrations contain virtual box obstacles. Are these obstacles visible in the rendered RGB observations and included in collision checking? If not, how can an image-based policy respond to them? Only successful expert trajectories are retained in the main dataset. Does this result in insufficient coverage of recovery states and off-trajectory behavior?
  • AirSim360[1] already provides closed-loop UAV simulation, automated trajectory generation, navigation-oriented data, and aligned multimodal observations. How does AirPlayground differ from AirSim360 beyond using 3DGS reconstructions and increasing the number of scenes and trajectories? Why is AirSim360 not discussed in the related work or included in the resource comparison table? The relevant capabilities of AirSim360 are described in its paper and official project.
  • Has any policy trained in AirPlayground been deployed in the corresponding physical environment? If not, the paper should clarify that it evaluates navigation inside reconstructed simulators rather than real-world transfer.
  • The current contribution is dominated by dataset construction and system integration. The paper provides limited new insight into learning representations, training objectives, or generalization. I would like the authors to clarify what machine learning question AirPlayground enables or answers beyond providing a larger aerial simulation resource.

[1] Ge X, Pan Y, Zhang Y, et al. Airsim360: A panoramic simulation platform within drone view[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 26931-26940.

Reviewer_q2dFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AirPlayground, a benchmark for aerial navigation that combines reconstructed real-world environments, dense expert demonstrations, and standardized closed-loop evaluation. The benchmark contains 94 executable 3D Gaussian Splatting environments, 94K flight trajectories, and 46.5M synchronized RGB-state-action samples. The authors provide a unified API that supports multiple navigation policy paradigms and evaluate seven baselines across different navigation settings. Beyond benchmarking policy performance, the paper studies the gap between offline prediction metrics and closed-loop navigation outcomes, showing that execution protocols and local scene geometry strongly influence deployment performance.

Critical strengths
  1. The scale and structure of AirPlayground are valuable. Combining 94 executable 3DGS environments with 94K dense flight trajectories and synchronized RGB-state-action supervision creates a resource that supports both imitation learning and closed-loop evaluation.

  2. The unified cross-policy API is a strong design choice. It allows policies with different output spaces and execution mechanisms to be compared under the same environments without forcing them into an unnatural common representation.

  3. The benchmark analysis goes beyond reporting navigation scores. The offline-to-closed-loop study reveals that common prediction errors such as FDE do not reliably explain deployment failures, and identifies execution horizon and local geometry as important factors.

Critical weaknesses
  1. The benchmark contribution is substantial, but the paper does not fully demonstrate how AirPlayground changes the conclusions of existing aerial navigation research. Most experiments evaluate existing policies on the new benchmark, while stronger evidence would come from showing that training or developing methods with AirPlayground leads to improvements that cannot be obtained from existing resources.

  2. The validity of the expert demonstrations deserves more discussion. The dataset contains only successful trajectories generated by A*-based planning followed by waypoint tracking, which provides scalable supervision but may produce a narrow behavior distribution. The benchmark would be more useful if it quantified how the collected demonstrations differ from alternative expert strategies or real UAV flight behavior.

  3. Several conclusions from the offline-to-closed-loop analysis are based on correlations across seven policies. For example, the relationship between offline errors and collision outcomes may depend heavily on the selected architectures and training procedures. A larger policy population or controlled modifications of individual policies would make these diagnostic conclusions more convincing.

  4. The benchmark currently evaluates policies mainly in reconstructed static environments. Although the 3DGS scenes provide realistic appearance and aligned geometry, the paper does not study whether policies trained on AirPlayground transfer to physical UAVs or how reconstruction artifacts affect navigation decisions.

  5. The benchmark supports multiple goal settings, but the main comparison focuses heavily on PointGoal navigation. Since aerial navigation often involves semantic targets, language instructions, and long-horizon exploration, broader evaluation of language-conditioned or open-ended navigation would better match the motivation of a general aerial navigation benchmark.

Questions for authors

Pls see weakness

Reviewer_YBYwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a dataset for visual aerial navigation in complex indoor and outdoor environments. The dataset is generated with a 3D visual capture setup worn by a person, who then collects data by walking around the scenes of interest. Trajectory are generated by selecting start and goal at random in the scene (potentially adding a few invisible obstacles to add more curves to the drone's motion). The dataset is then used to train imitation learning policies, which are evaluated both in open loop and closed loop.

Critical strengths

The key strength of this work is, in my opinion, the visual diversity and realism of the collected scenes. Prior dataset use similar techniques for simulation (3DGS), but they fly in open spaces far away from obstacles. This dataset instead has 94 scenes (even if it is unclear how different they are, see for example Fig. 9), where the platform flies in close proximity to obstacles and under the canopy. This level of visual realism could potentially be useful, and it is something I have not seen before in drone datasets (but present in other datasets for visual navigation of ground robots).

Critical weaknesses

The main weakness of this work is the lack of justification of key design decisions. This dataset is explicitly aimed at research in the visual drone navigation space, but there are several limitations that will potentially block its adoption:

  1. It is unclear which drone is used for generating the velocity commands. Since the dynamics is underactuated, the mass and inertia of the drone have a significant impact on the control commands, in particular at such high frequency. It is unclear how and why a policy trained on this data would be useful beyond the specific drone characteristics used to generate the dataset. Prior work (e.g., OpenFly) overcomes this problem by providing waypoints, which are then passed to a position controller. Since this work claims the level of commands to be one of the key contributions compared to prior work (See Table 1), this choice should be better justified.
  2. It is unclear why the data frequency should be so high (50Hz). According to the statistics provided in Sec. 3.5, the average speed is around 2m/s. Therefore, a new image is collected, on average, every 4cm of center of mass displacement. Why is such density necessary, if all obstacles are static and the drone moves so slowly? The difference between images is going to be very small, particularly when the obstacles are further than a couple of meters away.
  3. The approach used for generating trajectories is suboptimal: all trajectories are planar, so a policy trained on this dataset would not be able to perform altitude control. In addition, the randomly placed invisible boxes (LL 214-215) would bias the drone towards zig-zag motions in regions that visually have no obstacle. If the boxes are instead visible (the manuscript is not very clear in this regard), how are they made photorealistic?

Overall, these are critical weaknesses that would block researchers to use this dataset.

Questions for authors
  1. Section 3.4 reads like the description of a data loader. I am not sure it is adding much.
  2. 1K trajectories per scene seems like a lot. It would be helpful to clarify if so many are needed.
  3. Clarify what are the differences between scenes (potentially adding images). The scenes in Fig. 9 look all alike.
  4. Real-world demonstration of policies learned with the proposed dataset would be a great addition, adding validity to the main contribution.
  5. Having as an additional modality a language description of the task would also be very beneficial for policy training. Right now the dataset only allows image-goal navigation methods to be investigated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.