- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AirPlayground, a benchmark for training and evaluating aerial navigation policies in reconstructed environments. It contains 94 3D Gaussian Splatting scenes, 94,000 expert trajectories, and 46.5 million synchronized samples. The benchmark provides reconstructed appearance, collision geometry, automated trajectory collection, and a common interface for different policy types. Seven baselines are evaluated on seen and unseen scenes, followed by studies of data scaling, execution horizons, and the relationship between offline prediction errors and closed-loop performance.
Critical strengths
- AirPlayground provides broad scene coverage and a large number of dense flight demonstrations. Pairing demonstrations with executable reconstructed environments could be useful for studying closed-loop aerial navigation.
- The paper documents the data-collection and evaluation pipelines in detail. The unified interface supports several policy output formats, and the execution-horizon experiments show that deployment settings can substantially affect navigation performance.
Critical weaknesses
-
Success, collision, task generation, and expert planning all rely on geometry produced from reconstructed point clouds through several heuristic and scene-specific processing steps. However, the paper does not measure the accuracy of the resulting obstacle boundaries or free space against independent ground truth. The reported image-quality metrics evaluate rendering rather than collision geometry and cover only a subset of the scenes. This is consequential because inaccurate geometry would directly change both the demonstrations and the reported closed-loop outcomes, making it unclear whether the benchmark measures navigation ability or artifacts of its reconstruction pipeline.
-
The evaluated policies differ in pretraining, optimization budget, adapters, output representations, and execution mechanisms, so their relative performance cannot be attributed cleanly to policy formulation. More importantly, the offline-to-closed-loop correlations are computed from only seven aggregated policy-level observations, without uncertainty estimates. The benchmark also considers only fixed-height planar velocity control. These limitations make the claims about policy comparison, deployment diagnosis, and aerial-navigation generalization substantially stronger than the available evidence.
Questions for authors
- How was the accuracy of the reconstructed collision geometry evaluated? Can the authors compare obstacle boundaries and free space against independent measurements from the physical environments?
- How sensitive are success and collision rates to point-cloud density, height filtering, component removal, voxel resolution, and the manually selected inflation radius?
- The navigable polygon and several geometry parameters are defined separately for each scene. How much manual work was required, and how was annotation consistency maintained across the 94 scenes?
- The paper states that 69.1% of the demonstrations contain virtual box obstacles. Are these obstacles visible in the rendered RGB observations and included in collision checking? If not, how can an image-based policy respond to them? Only successful expert trajectories are retained in the main dataset. Does this result in insufficient coverage of recovery states and off-trajectory behavior?
- AirSim360[1] already provides closed-loop UAV simulation, automated trajectory generation, navigation-oriented data, and aligned multimodal observations. How does AirPlayground differ from AirSim360 beyond using 3DGS reconstructions and increasing the number of scenes and trajectories? Why is AirSim360 not discussed in the related work or included in the resource comparison table? The relevant capabilities of AirSim360 are described in its paper and official project.
- Has any policy trained in AirPlayground been deployed in the corresponding physical environment? If not, the paper should clarify that it evaluates navigation inside reconstructed simulators rather than real-world transfer.
- The current contribution is dominated by dataset construction and system integration. The paper provides limited new insight into learning representations, training objectives, or generalization. I would like the authors to clarify what machine learning question AirPlayground enables or answers beyond providing a larger aerial simulation resource.
[1] Ge X, Pan Y, Zhang Y, et al. Airsim360: A panoramic simulation platform within drone view[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 26931-26940.