Uni-Fly360: Learning Omnidirectional Obstacle Avoidance Within End-to-End Flight

Reliable autonomous flight is essential for unmanned aerial vehicles (UAVs), yet many tasks demand omnidirectional obstacle avoidance: when the flight direction deviates from a task-specific heading, obstacles may approach from arbitrary directions. This requirement exposes the limited coverage of forward-facing sensors and motivates panoramic perception and control. Existing approaches, however, either suffer from restricted visual coverage or lack designs tailored to panoramic imagery. We propose Uni-Fly360, an end-to-end framework for omnidirectional obstacle avoidance from panoramic image sequences. It adopts optical flow as an intermediate representation that captures motion cues while reducing dependence on domain-specific appearance, thereby mitigating the sim-to-real gap. To disentangle ego-motion from object motion in optical flow, we introduce a differentiable spherical motion reasoning module that exploits antipodal consistency and full-sphere constraints to suppress rotation, estimate the ego-translation direction, and construct local motion residuals. Its differentiable formulation integrates spherical geometric reasoning into the joint optimization of perception and control. Experiments on three representative panoramic tasks show that Uni-Fly360 achieves state-of-the-art performance and transfers well to real-world videos. Code and models will be released.