acceptodds
Withdrawn from ICLR 2027

PAIMON: An Aerial World Tracking Model Bridging Simulation to Reality

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling high-quality supervision for aerial target tracking is challenging, as real-world flights rarely provide synchronized geometric and action supervision. Although simulation offers such supervision at scale, the appearance gap hinders sim-to-real transfer. We present PAIMON, a dual-expert MoT world action model that decouples action-relevant geometric learning from appearance adaptation. PAIMON translates simulated RGB observations toward real-world appearance with a structure-preserving image editor, while action-conditioned future-depth prediction provides geometric supervision during training. At deployment, PAIMON directly predicts actions from current observations, visual and ego-motion histories, and target instructions, without explicit video rollout. We further introduce the Aerial Tracking Platform (ATP) for rule-based expert data collection and closed-loop evaluation with high-fidelity 3D Gaussian-splatting scenes. PAIMON achieves the highest average closed-loop success and tracking rates among the evaluated baselines. Real-world flight tests further demonstrate transfer of simulation-learned tracking behavior to RGB-based closed-loop tracking under varying flight conditions. Code and data will be released upon publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Aerial Tracking, World Action Model, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1oSJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PAIMON learns language-conditioned aerial target tracking from simulation only. An action expert that outputs a short camera trajectory is trained together with a video expert that predicts future depth, joined by an asymmetric attention mask, and the video branch is dropped at inference. Simulated frames are edited toward real appearance with an image-editing model. On the authors' ATP platform PAIMON obtains the best average success against five baselines, and a small quadrotor test is reported.

Critical strengths
  • Future depth is only a training signal, so inference needs no video rollout and the planning cost stays low. The separation is clean and the idea is easy to test.
  • The evaluation includes environments outside the training sources, and the appendix reports honestly where PAIMON loses to baselines, which makes the main results more credible.
Critical weaknesses
  • The sim-to-real claim is not matched by the evidence. The paper treats the sim-to-real gap mainly as an appearance gap, while the expert trajectories come from rules for reachability, visibility and collision. It is unclear how they match real quadrotor dynamics such as acceleration limits, actuator latency and disturbances, or how the simulator's camera geometry is matched to the cropped real images. The real-world test, a few episodes in a motion-capture lab with no quantitative results or baseline, shows that PAIMON can be deployed, not that it bridges simulation and reality as the title claims.
  • Neither design component is validated on its own. ACTION-ONLY removes the video expert, the depth task, the cross-expert interaction and the extra capacity together, so it shows that the branch helps, not that action-conditioned future depth is the key factor. For appearance transfer, FID is measured against generic image datasets rather than real UAV footage, and structure preservation is shown only qualitatively, although editing starts close to pure noise and the paper notes that starting from pure noise changes the person's facing direction and count. Since depth, boxes and actions are copied unchanged to the edited frames, such drift becomes label noise. PAIMON-Raw is also better on the real-derived Juzizhoutou scene, and its Table 4 average is miscomputed.
Questions for authors
  • For the real flights, how well was the target tracked, what failures occurred, and how are the camera intrinsics and policy outputs matched to the real vehicle?
  • Can you isolate action-conditioned future depth, for example by predicting future RGB or current depth instead, or by detaching the auxiliary loss from the action branch?
  • Is there a quantitative check that editing preserves the target's box, mask and depth edges?
  • Since only a short prefix is executed at 2 Hz, how large are the open-loop errors over that prefix, given the large long-horizon yaw errors in Table 3?
Reviewer_kyrNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PAIMON, a dual-expert world-action model for aerial target tracking. The model consists of an action expert and a video expert. The action expert uses anchor-centered diffusion to predict future camera trajectories, while the video expert predicts future depth conditioned on the current RGB observation, candidate actions, and ego-motion. An asymmetric attention mask is used so that future depth provides geometric supervision during training but is removed at deployment, enabling RGB-only closed-loop action prediction. The paper also introduces the Aerial Tracking Platform (ATP), which functions as a benchmark-like data-collection and closed-loop evaluation suite. ATP uses HM3D and Habitat-GS to generate large-scale data containing RGB observations, depth, camera poses, target boxes, language instructions, and expert actions, and unifies evaluation across four environments. The authors apply FLUX-based image editing to reduce the appearance gap between simulation and real images. PAIMON is evaluated using open-loop trajectory errors, closed-loop success and tracking rates, ablations, inference-cost measurements, and a real-world flight demonstration. According to Table 2, PAIMON obtains the best average SR/TR among the evaluated baselines (0.82/0.86). The paper also includes a small indoor flight demonstration consisting of 10 episodes, in which the authors report continuous target tracking.

Critical strengths

1.Important and practically relevant problem. Aerial target tracking simultaneously involves visual recognition, camera motion, occlusion recovery, and closed-loop control. The paper addresses the challenging problem of transferring simulation-learned tracking behavior to the real world. 2.Interesting separation between training-time geometric supervision and deployment-time RGB-only inference. The use of future depth during training, together with an asymmetric attention mask that removes future-depth dependence at deployment, is a clear and potentially generalizable design idea. 3.Broad closed-loop evaluation. The paper evaluates HM3D, Habitat-GS, GYM-unrealCV, and HQ-GS, and reports not only SR and TR but also visibility, survival, occlusion, jerk, and kinematic-violation metrics. This is more informative than reporting only open-loop ADE/FDE. 4.Relatively detailed implementation description. The appendix provides architectural specifications, token counts, optimization settings, training steps, diffusion schedules, evaluation definitions, and real-world deployment details, which should help future reproduction. 5.ATP is a useful benchmark-like infrastructure contribution. Although the paper calls ATP a platform rather than only a benchmark, it combines rule-based expert data collection, aligned multimodal simulation data, multiple closed-loop environments, standardized SR/TR metrics, and auxiliary diagnostics. This makes the work useful beyond the proposed PAIMON model and provides a reusable basis for evaluating aerial tracking policies. 6.The paper already includes useful, though incomplete, ablations. PAIMON-Raw provides evidence about the contribution of appearance transfer, and PAIMON-5B examines whether simply increasing model capacity is sufficient. These experiments do not fully isolate future-depth supervision, but they partially address alternative explanations and should be credited in the assessment.

Critical weaknesses

1.The evaluated baselines are all transplanted from other task domains, so the SOTA claim does not cover task-native tracking methods. The five compared baselines (SINGER, ACT, π0, π0.5, WALL-OSS) originate in aerial vision-language navigation, bimanual manipulation, and generalist manipulation/embodied policies; none was originally designed for aerial target tracking. Although retraining all methods on the same ATP data makes the comparison internally consistent, it makes the comparison home-field: the environments, supervision formats (depth, anchor modes), and evaluation protocol are all built around PAIMON's training structure. Meanwhile, methods designed for this task family — TrackVLA, UAV-TrackVLA, YOPO/YOPOv2, RSPT, D-VAT — are discussed in related work but never compared, and the paper even inherits TrackVLA's official HM3D scene partition (703 training / 101 validation scenes) and its STT/DT/AT episode structure without evaluating against it. The claim "highest average success and tracking rates among the evaluated baselines" is therefore technically accurate but cannot be read as state-of-the-art among aerial tracking methods. At minimum, TrackVLA (whose data partition the paper directly builds on) should be retrained and compared under the same protocol, or the authors should explain the concrete obstacle that prevents this. 2.The ACTION-ONLY ablation does not isolate the independent contribution of future-depth supervision. PAIMON-Raw and PAIMON-5B usefully test appearance transfer and one larger-capacity configuration. However, ACTION-ONLY removes the video branch, future-depth loss, cross-expert interaction, and substantial capacity together; its average SR/TR is 0.68/0.75 versus 0.82/0.86 for PAIMON (Table 4). It therefore supports the full package, but does not isolate the depth objective. Moreover, the asymmetric mask limits forward visibility, not necessarily backward gradients: because depth queries read action tokens, the depth loss can still backpropagate through action representations, as the paper itself notes. A loss-only ablation and a stop-gradient/depth-to-action-blocking ablation would distinguish the auxiliary objective from cross-expert coupling. The PAIMON-5B result should be limited to the tested training configuration, not generalized to model scaling overall. 3.Appearance transfer is not directly audited for task-label preservation, and the FID reference domains are indirect. Positive FID gains against ImageNet, Landscape, COCO, Pascal VOC, and NYU Depth V2, together with qualitative examples, show appearance change but do not verify preservation of target location, identity cues, depth correspondence, or action labels. These reference domains are not specific to aerial tracking. This is relevant when editing changes clothing used by DT/AT instructions. A concise audit using target-box IoU/location changes, RGB-depth consistency, and instruction-target matching, with the sampling and human-check protocol, would better support the structure-preservation claim. FID should be treated as appearance-distribution evidence, not direct evidence of tracking transfer. 4.Per-baseline protocol details and real-world logging are too coarse to verify the comparison and feasibility claims. The paper states that methods share training data/evaluation settings and that FLUX editing is applied to closed-loop observations for all methods except PAIMON-Raw; thus, I do not assume the comparison is unfair. For reproducibility, however, a compact per-baseline summary of initialization, training budget, input modalities, action interface, and replanning is needed, and the paper should clarify whether HQ-GS inputs are raw or edited and whether online editor latency is counted. On the real-world side, the 10 indoor episodes (about 15 m per flight; target speed about 1.3 m/s), using motion capture and remote RTX 5090 inference over a dedicated LAN at approximately 2 Hz, support a controlled feasibility claim rather than broad real-world robustness; a full real-world baseline suite is not essential, but per-episode success, temporary losses, distance violations, or safety interventions, together with the controller/safety logic mapping predicted trajectories to velocity commands, should be reported.

Questions for authors

Questions 1 and 2 below are the two central issues for this paper to reach the acceptance line; Questions 3-5 would substantially strengthen the evidence and clarify the scope of the claims. 1.The paper inherits TrackVLA's HM3D scene partition and STT/DT/AT episode structure, yet TrackVLA and other task-native tracking methods (e.g., UAV-TrackVLA, YOPO, D-VAT) do not appear in the main comparison. What specifically prevents retraining TrackVLA on ATP under the same 12-step trajectory protocol? If the obstacle is architectural (e.g., output interface mismatch) or computational, the authors should state it; otherwise, the absence of the closest task-native baseline substantially weakens the SOTA claim. 2.What are the parameter, compute, and input-token differences between PAIMON and ACTION-ONLY? As the most useful extension, the authors could add (i) a variant retaining the video branch but removing only the future-depth loss and (ii) a variant retaining the branch/loss but stopping gradients or otherwise blocking the depth-to-action information path. Since the asymmetric mask controls forward token visibility but does not necessarily block reverse-mode gradients, these experiments would clarify whether the benefit comes specifically from future-depth supervision or from the broader dual-expert setup. 3.What quantitative checks establish that image editing preserves target localization, pose/identity cues, depth correspondence, and instruction-target consistency? Could the authors report target-box IoU, center/area changes, depth-consistency error, and instruction-target matching accuracy, and describe how edited samples were selected and manually checked? 4.Could the authors provide a per-baseline protocol table covering initialization, training budget/tuning, input modalities, action interface, and replanning implementation, and explicitly state whether each evaluation environment (including HQ-GS) is raw or FLUX-edited? Please also clarify whether editor latency is included in the reported evaluation/deployment cost. This request is intended to make the existing high-level fairness statement reproducible, not to assume that the comparison is necessarily unfair. 5.Could the authors report per-episode outcomes of the 10 real-world flights, explicit success/failure criteria, temporary target losses or distance violations, safety interventions, and control failures, and clarify the controller, clipping, and safety logic that convert the predicted camera trajectory into the reported forward-, yaw-, and vertical-velocity commands?

Reviewer_Ujc9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This objective of this work is simulation to real world transfer of vision-based policies for human tracking in aerial vehicles. The key technical contribution consists in jointly training two transformers: one for future depth prediction and the other for action prediction. The two transformers share information via cross-attention. The action predictor is trained with behavioral cloning data generated in a simulator. To enable transfer, simulated frames are transformed into photorealistic frames using off-the-shelf tools.

The results mainly focus on sim-to-sim transfer, while the sim-to-real results consist of a single experiment and are only qualitative.

Critical strengths
  1. The idea of training a depth model to extract useful representations for sim-to-real is interesting and sound. I have not seen this idea being applied before for the problem of visual sim-to-real. All the other part of the architecture and training procedure are quite standard (e.g., making images look realistic to favour transfer).
  2. The presented method generalizes very well to simulators unseen during training (e.g., great results on GYM-unrealCV). It also achieves good results on hq-gs dataset which contains scenes reconstructed from real world captures.
Critical weaknesses
  1. The work is not put in sufficient context with respect to existing work. There are off-the-shelf commercial systems (e.g., Skydio) that already have this capabilities. From the research perspective, a few works have demonstrated high-speed human tracking with aerial vehicles (see for example "YOPOv2-Tracker: An End-to-End Agile Tracking and Navigation Framework from Perception to Action"). The most similar approach, "TrackVLA: Embodied Visual Tracking in the Wild", which has a very similar interface and architecture, is not compared against. Overall, baselines are selected to be mainly other VLAs which are designed for manipulation (with the exception of SINGER).

  2. The main claim of the paper is on sim-to-real transfer, however, quantitative evaluation is only on simulator based photorealistic environments which makes it hard to assess the paper claims. The current real world results are very limited and only qualitative. Being the central claim sim-to-real, extensive real-world experiments are necessary to justify the validity of the approach.

Questions for authors
  1. Could you clarify your positioning with respect to prior work, both with discussions and quantitative experiments?
  2. Could you add quantitative real-world experiments to justify the contribution? A demonstration video would be of great help.
  3. Could you clarify th practical value of ATP as a unified benchmark for aerial tracking? How does it differentiate to existing benchmarks such as EVT Bench and Habitat-GS? Will it be public?
  4. Why spending so much text on how to define the asymmetric attention mask? I believe that section can be shortened significantly, as the design is sound but quite standard for learning representations (see for instance "AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation").
  5. How are the baselines (e.g., pi0) trained? Are they trained from scratch or fine tuned? I could not find any detail
  6. PAIMON exhibits pretty high jerk compared to SINGER, the only navigation specific baseline. Why is that?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.