DeTrack: A Benchmark and Altitude-Aware Dual World Model for Drone-embodied Tracking

Abstract—Aerial object tracking has wide applications in public safety, emergency rescue, wildlife monitoring, etc. However, existing aerial object tracking is mainly limited to the passive tracking paradigm with 2D video sequences pre-recorded in fixed camera locations or along pre-defined flight paths, lacking drone-embodied scene perception, egocentric interaction, and movement control in dynamic 3D scenes. In this paper, we define a novel drone-embodied tracking (DeTrack) task and build a large-scale benchmark with evaluation metrics for this task. It refers to tracking targets in interactive 3D scenes with online drone-egocentric observation and active control in a closed loop. The DeTrack benchmark differs from conventional passive aerial tracking in five characteristics: (1) drone-embodied closed-loop tracking, (2) target tracking coupled with obstacle avoidance, (3) spatio-temporally composite occlusion in 3D scenes, (4) altitude mediated contradiction between visibility and safety, (5) extensive and extreme scale variation. Besides, the drone-embodied tracking faces intrinsic altitude-mediated contradiction between visibility and safety: the view field is wide while the target detail is weak at high flight altitude, and the target detail is sufficient while the obstacles are dense at low flight altitude. Therefore, we further propose a novel altitude-aware dual worlds (AaDWorlds) for drone-embodied tracking. It consists of an altitude-aware perception (AaP) module and dual world models (DWM). Specifically, the altitude-aware perception module adaptively encodes, couples, and decodes altitude-aware representations. The dual world models imagine future states under both high- and low altitude regimes. Eventually, the imagined future states (from DWM) and pseudo egocentric observations (from AaP) at both high and low altitudes effectively complement the original drone egocentric observation, simultaneously facilitating the view field, target detail, and obstacle avoidance during drone-embodied tracking.