acceptodds
Under review as a conference paper at ICLR 2027

Uni-Fly360: Learning Omnidirectional Obstacle Avoidance Within End-to-End Flight

Abstract

Reliable autonomous flight is essential for unmanned aerial vehicles (UAVs), yet many tasks demand omnidirectional obstacle avoidance: when the flight direction deviates from a task-specific heading, obstacles may approach from arbitrary directions. This requirement exposes the limited coverage of forward-facing sensors and motivates panoramic perception and control. Existing approaches, however, either suffer from restricted visual coverage or lack designs tailored to panoramic imagery. We propose Uni-Fly360, an end-to-end framework for omnidirectional obstacle avoidance from panoramic image sequences. It adopts optical flow as an intermediate representation that captures motion cues while reducing dependence on domain-specific appearance, thereby mitigating the sim-to-real gap. To disentangle ego-motion from object motion in optical flow, we introduce a differentiable spherical motion reasoning module that exploits antipodal consistency and full-sphere constraints to suppress rotation, estimate the ego-translation direction, and construct local motion residuals. Its differentiable formulation integrates spherical geometric reasoning into the joint optimization of perception and control. Experiments on three representative panoramic tasks show that Uni-Fly360 achieves state-of-the-art performance and transfers well to real-world videos. Code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omnidirectional Obstacle Avoidance, End-to-End Learning, Sim-to-Real Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uRGJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Uni-Fly360, an end-to-end framework for UAV omnidirectional obstacle avoidance using panoramic images. It uses optical flow as intermediate representation to reduce sim-to-real gap. A differentiable spherical motion reasoning module disentangles ego-motion from object motion. Perception and policy are jointly optimized in a differentiable simulator. Evaluated on 3 tasks in sim and tested on real videos.

Critical strengths
  1. Joint Optimization via Differentiable Simulation: Unifying the perception (optical flow) and control modules in an end-to-end differentiable pipeline is a strong contribution. The ablation study (Table 2) convincingly demonstrates that joint optimization significantly improves performance compared to frozen perception, validating the core motivation of the paper.

  2. Spherical Motion Reasoning (SMR): The geometric design of the SMR module is elegant. By leveraging antipodal constraints to cancel rotation and estimating the global translation axis, it provides physically meaningful features to the policy without requiring heavy neural networks. The ablation confirms its effectiveness when combined with joint training.

Critical weaknesses
  1. Lack of Real-World Flight Validation: The paper claims strong "sim-to-real transfer," but only evaluates the model by feeding pre-recorded real-world videos and visualizing the output acceleration vectors (Fig. 4). Without deploying the policy on a physical UAV, it is impossible to verify if the predicted commands actually result in safe closed-loop obstacle avoidance in reality. This significantly weakens the practical claim of the paper.

  2. Degraded Performance After Collisions: As shown in Table 1, while Uni-Fly360 achieves higher Success Rates (SR), its Collision Time (CT) is notably worse than the baseline Fly360 in Hovering Maintenance (1.86s vs 1.19s) and Fixed-Trajectory Filming (2.8s vs 0.27s). The authors acknowledge that optical flow becomes unstable after collisions. For a safety-critical task like obstacle avoidance, prolonged collision duration is a significant drawback that needs better justification or mitigation.

Questions for authors
  1. Are there any plans or fundamental blockers to deploying Uni-Fly360 on a real physical drone for closed-loop flight tests? If computational cost is the issue, what is the current inference FPS of the entire pipeline?

  2. The SMR module relies on the assumption that the background is predominantly static to estimate ego-translation. How robust is the system in environments heavily populated with dynamic/moving obstacles?

  3. Regarding the high Collision Time (CT) in Tasks 1 and 3 compared to Fly360: could the auxiliary loss weights be tuned to recover faster from collisions, or is this an inherent limitation of using optical flow instead of depth?

2 comments
AuthorsOpenReview ↗

some explanation and additional experinments

Response to Question1

Thanks. Since submitting the paper, we have conducted extensive real-world flight experiments to validate our method’s closed-loop control performance in real environments. The experimental results and UAV platform specifications are presented below.

Table. Hardware and deployment configuration of the real-world UAV platform.

ItemSpecification
UAV platformCustom quadrotor
Takeoff mass628 g, including battery and payload
Flight controllerself-developed FC
Onboard computing8-core CPU at 2.0 GHz; 16 GB RAM; peak AI throughput: 15 TOPS
Panoramic sensingTwo fisheye cameras with automatic panoramic stitching
Visual inputERP images, 224 × 448 pixels (H×WH \times W)
State estimationOnboard estimation of position, velocity, and attitude
Control interfaceThree-dimensional body-frame velocity commands
Model complexity and runtime7.05 M parameters; 47 FPS; 21.13 ms latency, measured on a desktop NVIDIA RTX 3090 GPU

We evaluated our system on two real-world tasks:

  1. Hovering obstacle avoidance, where obstacles approach from the front, rear, and sides, requiring the UAV to evade them and recover stable hovering.
  2. Continuous pursuit evasion, where a person continuously pursues the UAV and the UAV performs evasive maneuvers.

The success rates are reported in table below.

VariantABCDHovering Obstacle Avoidance: SR ↑Continuous Pursuit Evasion: SR ↑
Depth (Frozen)✗✓✗✗7/105/10
Panoramic flow (Frozen)✓✓✗✗3/101/10
Panoramic flow (Joint)✓✓✗✓6/104/10
Panoramic flow + SMR (Frozen)✓✓✓✗4/102/10
Uni-Fly360 (ours) (Joint)✓✓✓✓10/108/10

Response to Question2

Thank you for raising this concern. To assess robustness to moving obstacles, we re-evaluated the compared variants separately in static-obstacle and dynamic-obstacle scenarios. Table below reports success rate (SR) and collision time (CT) for both settings.

VariantABCDStatic Obstacles: SR ↑Static Obstacles: CT (s) ↓Dynamic Obstacles: SR ↑Dynamic Obstacles: CT (s) ↓
Depth (Frozen)✗✓✗✗38/400.2823/403.88
Uni-Fly360 (ours) (Joint)✓✓✓✓39/400.1535/402.96

Response to Critical Question2:

Thank you for highlighting this issue. We agree that the higher CT in Tasks 1 and 3 is a limitation of our method. We adopted the CT protocol from Fly360, in which episodes continue after collisions. CT therefore measures cumulative contact duration and reflects post-collision recovery and trajectory stability.

After analyzing the results, we found that although CT is averaged over all episodes, its nonzero contributions come exclusively from episodes containing collisions. In Fixed-Trajectory Filming, both methods have only three such episodes out of 40, making CT sensitive to individual prolonged contacts. This limits the available observations for assessing recovery, while the longer contact duration remains a concern.

To further investigate this issue, we reproduced several observed failure scenarios and conducted extensive repeated evaluations in these settings. As shown in Table below, Uni-Fly360 achieves comparable CT (1.21 vs. 1.28 s) while maintaining a substantially higher success rate (63% vs. 38%).

Table. Comparison on reproduced failure scenarios.

MethodFailure Scenario Reproduction: SR ↑Failure Scenario Reproduction: CT (s) ↓
Fly36038/1001.28
Uni-Fly360 (ours)63/1001.21
Reviewer_uRGJOpenReview ↗

Response to Authors

Thank you for the comprehensive rebuttal. The additional real-world flight experiments (10/10 and 8/10 success rates) and failure scenario analysis adequately address my concerns regarding closed-loop validation and collision recovery. The clarifications on the CT metric and dynamic obstacle performance are satisfactory.

My rating remains unchanged: 3 (Weak Acceptance).

Reviewer_UE8HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors proposed Uni-Fly360 to achieve the end-to-end training for omnidirectional obstacle avoidance from panorama sequences. It replace the common depth-based intermediate representation with the opatical flow, since it could povide more motion cues reducing the dependence on domain-specific appearance. To better achieve the disentanglement between object motion and ego-motion on the panoramas, authors propose a hand-craft differentiable spherical motion reasoning (SMR) module. Experiments demonstrate improved success rates on three simulated flight tasks, while qualitative results on recorded panoramic videos provide preliminary evidence of transfer to real-world imagery.

Critical strengths

The questions are practical and the methods are well motivated. The movement direction of the drone may be different from the shooting orientation, so panoramic perception is indeed valuable; using optical flow as an intermediate representation also helps reduce the strategy's reliance on texture and appearance.

The combination of geometric constraints and task-driven learning is relatively complete. SMR organizes panoramic optical flow into global translation structure and local motion inconsistency information, and allows control loss to inversely update the perception module.

There are substantial gains in simulation results and component ablation. Compared with Fly360, the hovering success rate increases from 28/40 to 38/40, and the target following increases from 33/40 to 40/40. Table 2 also does verify the role of joint training and SMR in the complete system.

Critical weaknesses

a) The real-world evaluation provides limited evidence of closed-loop transfer. The experiments of real-world qualitative analysis are only based on offline videos and visulaize the predicted acceleration commands; therefore, these commands are not executed on a UAV. Consequently, the experiments do not establish whether the policy produces stable and collision-free real-world flight. Moreover, the camera orientation is kept fixed during recording, so the robustness to camera rotation (the motivation for SMR) is not tested.

b) The benefit of SMR need clearer validation. Although Tab.2 shows improvements from SMR under joint training, it does not clarify how much each component and the additional motion supervision contribute, such as combine the optical flow only with Q_{ego} or D_{dyn}. Besides, since the policy already receives the UAV’s velocity, comparing SMR with the translation-axis directly derived from this information and SMR with the translation-axis from the optical flow estimation would also help establish the value of choosing the optical flow as the intermediate representation.

c) More explanation for the results of Collision Time. From the Tab.1, the CT results shows the optical flow will become unstable when facing the collision. Especially, for the Fixed-Trajectory Filming, the SR result is same with Fly360 but the CT result is much worse. For the obstacle avoidance system, it is a key limitation and more analysis is required.

Minor

Is the cover figure finished by AI tool? I checked the statement and founded "the preparation of illustrative icons". But I encourage the authors could plot it by self, because the current one dose not emphasize your contribution clearly and it is a little ugly.

Questions for authors

See weakness. Since the rating system is different from the previous, the current rate is based on my concerns. And I encourage the authors to answer my concerns. If so, I will consider to improve my score.

6 comments
AuthorsOpenReview ↗

some explanation and additional experinments

Response to Critical Weakness (a):

Thank you for highlighting this issue. We fully agree that real-video evaluation is insufficient to establish safe closed-loop flight on a physical UAV. Since submitting the paper, we have conducted extensive real-world flight experiments to validate our method’s closed-loop control performance in real environments. The experimental results and UAV platform specifications are presented below.

We evaluated our system on two real-world tasks:

  1. Hovering obstacle avoidance, where obstacles approach from the front, rear, and sides, requiring the UAV to evade them and recover stable hovering.
  2. Continuous pursuit evasion, where a person continuously pursues the UAV and the UAV performs evasive maneuvers.

The success rates are reported in the table.

Hardware and deployment configuration of the real-world UAV platform.

ItemSpecification
UAV platformCustom quadrotor
Takeoff mass628 g, including battery and payload
Flight controllerself-developed FC
Onboard computing8-core CPU at 2.0 GHz; 16 GB RAM; peak AI throughput: 15 TOPS
Panoramic sensingTwo fisheye cameras with automatic panoramic stitching
Visual inputERP images, 224 × 448 pixels (H×WH \times W)
State estimationOnboard estimation of position, velocity, and attitude
Control interfaceThree-dimensional body-frame velocity commands
Model complexity and runtime7.05 M parameters; 47 FPS; 21.13 ms latency, measured on a desktop NVIDIA RTX 3090 GPU

**Real-world success rates across two obstacle avoidance tasks.

VariantABCDHovering Obstacle Avoidance: SR ↑Continuous Pursuit Evasion: SR ↑
Depth (Frozen)✗✓✗✗7/105/10
Panoramic flow (Frozen)✓✓✗✗3/101/10
Panoramic flow (Joint)✓✓✗✓6/104/10
Panoramic flow + SMR (Frozen)✓✓✓✗4/102/10
Uni-Fly360 (ours) (Joint)✓✓✓✓10/108/10

Response to Critical Weakness (b1):

Thank you for pointing this out. We acknowledge this concern and have conducted additional ablation experiments specifically on the SMR module. The results are presented in the table below.

All variants use panoramic flow. F/J: frozen/joint training. Each cell: SR ↑; CT (s) ↓. Global translation-axis reasoning (GTR). Local motion residual (LMR).

VariantTrainingHoveringFollowingFilming
GTRF20/40; 3.4727/40; 4.7630/40; 5.35
LMRF21/40; 3.5226/40; 4.4430/40; 5.29
SMRF21/40; 3.3028/40; 4.2129/40; 5.70
GTRJ32/40; 3.1635/40; 1.8132/40; 3.62
LMRJ33/40; 3.2434/40; 1.5634/40; 3.74
Uni-Fly360 (SMR)J38/40; 1.8640/40; 0.0037/40; 2.80

Response to Critical Weakness (b2):

Thank you for this suggestion. Camera-frame velocity provides a translation axis, while SMR estimates this axis from panoramic flow to compute local motion residuals. This differentiable process allows navigation gradients to update the flow estimator through global spherical constraints.

The ablation below compares both axis sources under frozen perception and joint training. With frozen perception, performance is similar, with a slight advantage for the velocity-derived axis. Under joint training, the flow-derived axis performs better, supporting the benefit of navigation feedback through global motion geometry, alongside auxiliary geometric supervision using the velocity-derived direction as a reference.

All variants use SMR. F/J: frozen/joint training. Each cell: SR ↑; CT (s) ↓.

Axis sourceTrainingHoveringFollowingFilming
VelocityF23/40; 2.9631/40; 3.8331/40; 5.12
FlowF21/40; 3.3028/40; 4.2129/40; 5.70
VelocityJ23/40; 2.9031/40; 3.8632/40; 5.28
Flow (Uni-Fly360)J38/40; 1.8640/40; 0.0037/40; 2.80

Response to Critical Weakness (c):

We follow Fly360’s CT protocol, in which episodes continue after collisions and CT measures cumulative contact duration. In Fixed-Trajectory Filming, both methods have only three failed episodes, making the average CT sensitive to individual prolonged contacts. To investigate this, we reproduced observed failure scenarios and conducted 100 trials per method. As shown in Table below, Uni-Fly360 achieves comparable CT (1.21 vs. 1.28 s) while maintaining a substantially higher success rate (63% vs. 38%).

Table. Comparison on reproduced failure scenarios.

MethodFailure Scenario Reproduction: SR ↑Failure Scenario Reproduction: CT (s) ↓
Fly36038/1001.28
Uni-Fly360 (ours)63/1001.21

Response to Minor Weakness:

Thanks! The cover figure was manually drawn, but AI-rendered. We will revise it next vision.

For minor weakness,we have plotted the cover figure mannually and revised it in the rebuttal revision PDF.

Thanks for your additional experiments. There are several questions: a) Did you fix the camera orientation during recording? If possbile, could you please provide some recordings of your real-world experiments? And the reported 47 FPS was measured on a desktop RTX 3090. What latency and control update rate did you actually get on the UAV? Could you also explain how the policy’s acceleration output was converted into the velocity commands used in the real flights?

b) Thanks for your additional ablation experiments on the SMR. Under the joint training, did both variants use the same auxiliary supervision? When using velocity to calculate Q_ego, the optical flow network can't be optimized. To check whether the gain comes from navigation feedback through the axis estimate, a useful control would be to keep the flow-based axis and the same losses, but block only that gradient path.

c) Sorry for possbile misunderstanding the Response to Critical Weakness (c). The reruns in the three failure scenarios are helpful and show a higher success rate with similar overall collision time. For the new results, if CT is averaged over all trials as in the paper, the average CT among failed trials is about 3.27 s for Uni-Fly360 (100 * 1.21 / 3) and 2.06 s for Fly360. Could you explain why Uni-Fly360 spends more time in contact when it fails? Is the difference driven by a few unusually long contacts, or is it common across failed runs?

Reviewer_HcGrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Uni-Fly360 jointly trains a panoramic optical-flow estimator and recurrent flight controller through differentiable dynamics. Its spherical motion reasoning module combines antipodal flow constraints into a soft, sign-invariant translation-axis estimate and supplies local motion residuals to the controller. The contribution is the task-driven integration of these geometric cues with panoramic perception and control. Evaluation consists of three simulated avoidance tasks, component ablations, and qualitative policy outputs on recorded real-world panoramas.

Critical strengths
  1. The interaction between geometry and joint training is supported by a useful ablation. In Table 2, adding SMR to jointly trained panoramic flow raises collision-free episodes from 31/40 to 38/40 for hovering and from 34/40 to 40/40 for target following. Adding it to frozen perception changes outcomes by at most one episode. This is evidence for adapting perception through the geometric module, beyond simply supplying panoramic coverage or a larger collection of features.

  2. The motion representation has a coherent geometric basis. Antipodal averaging removes first-order rotational flow, and the accumulated plane constraints identify the translation axis under a predominantly static scene. The matrix-exponential descriptor avoids the axis-sign ambiguity. Appendix A.2 appropriately limits the cancellation to small motions and explicitly states that the residual also responds to rotation and estimation error. The contribution builds on the cited antipodal geometry of Lim and Barnes (2008); its value here is the differentiable integration with navigation training.

Critical weaknesses
  1. The reported “success” omits the task objectives, and the safety comparison has a substantial tradeoff. Equation (10) defines success solely as zero collision time. It does not require maintaining the hover location, following the target at its prescribed offset, completing the filming trajectory, or preserving the desired heading. Consequently, Tables 1–2 do not establish that better collision avoidance is achieved at comparable task execution. The training losses in Appendix A.3 encourage reference tracking, but do not measure it at evaluation. Moreover, filming has identical success counts to Fly360, 37/40, with collision time increasing from 0.27 to 2.80 seconds. The paper acknowledges this deterioration and attributes it to post-collision flow instability; that explanation does not remove the tradeoff. Reporting tracking/trajectory completion alongside collision-free success is necessary to support the claimed improvement on these tasks.

  2. The real-world analysis establishes plausibility more clearly than the claimed transfer advantage. The abstract and contribution statement explicitly describe transfer to videos, and I assess the experiment on that basis. However, Section 4.3 and Figure 4 infer “correct and timely” avoidance and strong transfer from acceleration arrows on recorded images, without a quantitative correctness criterion or a real-image comparison to the frozen-perception variants. Camera orientation is held fixed, so the experiment also leaves the real-image rotation case motivating SMR untested. These examples show that the network produces plausible outputs on real panoramas; they do not determine whether joint optimization preserves or improves usable motion reasoning across the domain gap. They also provide no closed-loop evidence, since predicted commands do not determine subsequent observations. A controlled real-video comparison with known camera motion and obstacle approach would strengthen the representation-transfer claim; stable flight transfer would require closed-loop evidence.

Questions for authors
  1. For the same 40 episodes per task, what are the position/offset, trajectory-progress, and heading errors for Uni-Fly360 and Fly360? Does the improvement persist when success requires both collision avoidance and satisfying the specified task?
  2. What criterion establishes the correctness and timing of the real-video responses? On the same real sequences, how do joint training and SMR change motion estimates or avoidance responses relative to frozen panoramic flow, particularly with camera rotation? This would test the claimed representation-transfer benefit without equating video replay with deployed flight.
4 comments
AuthorsOpenReview ↗

some explanation and additional experinments

Response to Critical Weakness1 and Question1:

Thanks! We agree with this concern.

To address this, we have inspected the waypoint trajectories in Airsim and re-ran the evaluation using a stricter criterion that requires both collision-free operation throughout the 120-second episode and fulfillment of the corresponding task objectives. Following the task definitions in Sec. 4.1, these requirements are:

Hovering Maintenance:

Evade approaching obstacles while remaining within a prescribed neighborhood of the designated hover position, and recover stable hovering after each avoidance maneuver. We evaluate position error relative to the hover point throughout the episode, together with deviation from the desired target-facing heading.

Dynamic Target Following:

Track the moving target at the prescribed relative offset while maintaining a target-facing heading. We evaluate offset error and deviation from the reference waypoint trajectory defined by this offset, together with trajectory progress and heading error.

Fixed-Trajectory Filming:

Follow and complete the prescribed filming trajectory while keeping the camera oriented toward the target. We evaluate deviation from the reference waypoint trajectory, progress toward completing the waypoint sequence, and target-facing heading error.

An episode is counted as successful only when it satisfies both the collision-avoidance requirement and the corresponding task requirements. The results under this stricter evaluation criterion are presented in the table below.

VariantABCDHovering Maintenance: SR ↑Hovering Maintenance: CT (s) ↓Dynamic Target Following: SR ↑Dynamic Target Following: CT (s) ↓Fixed-Trajectory Filming: SR ↑Fixed-Trajectory Filming: CT (s) ↓
Depth (Frozen)✗✓✗✗29/401.2332/401.3637/400.69
Panoramic flow (Frozen)✓✓✗✗23/403.327/404.4230/405.84
Panoramic flow (Joint)✓✓✗✓32/402.734/403.6335/404.89
Panoramic flow + SMR (Frozen)✓✓✓✗22/403.6928/404.2130/405.66
Uni-Fly360 (ours) (Joint)✓✓✓✓36/401.240/400.0037/400.43

To further investigate this issue (mentioned traddoff), we reproduced several observed failure scenarios and conducted extensive repeated evaluations in these settings. As shown in table below , Uni-Fly360 achieves a CT comparable to Fly360 while maintaining a substantially higher success rate.

Table. Comparison on reproduced failure scenarios.

MethodFailure Scenario Reproduction: SR ↑Failure Scenario Reproduction: CT (s) ↓
Fly36038/1001.28
Uni-Fly360 (ours)63/1001.21

Response to Critical Weakness2 and Question2:

Thank you for highlighting this issue. We fully agree that real-video evaluation provides evidence of whether the learned motion representations transfer to real imagery, but is insufficient to establish safe closed-loop flight on a physical UAV. Since submitting the paper, we have conducted extensive real-world flight experiments to validate our method’s closed-loop control performance in real environments. The experimental results and UAV platform specifications are presented below.

Table. Hardware and deployment configuration of the real-world UAV platform.

ItemSpecification
UAV platformCustom quadrotor
Takeoff mass628 g, including battery and payload
Flight controllerself-developed FC
Onboard computing8-core CPU at 2.0 GHz; 16 GB RAM; peak AI throughput: 15 TOPS
Panoramic sensingTwo fisheye cameras with automatic panoramic stitching
Visual inputERP images, 224 × 448 pixels (H×WH \times W)
State estimationOnboard estimation of position, velocity, and attitude
Control interfaceThree-dimensional body-frame velocity commands
Model complexity and runtime7.05 M parameters; 47 FPS; 21.13 ms latency, measured on a desktop NVIDIA RTX 3090 GPU

We evaluated our system on two real-world tasks:

  1. Hovering obstacle avoidance, where obstacles approach from the front, rear, and sides, requiring the UAV to evade them and recover stable hovering.
  2. Continuous pursuit evasion, where a person continuously pursues the UAV and the UAV performs evasive maneuvers.

The success rates are reported in table below.

VariantABCDHovering Obstacle Avoidance: SR ↑Continuous Pursuit Evasion: SR ↑
Depth (Frozen)✗✓✗✗7/105/10
Panoramic flow (Frozen)✓✓✗✗3/101/10
Panoramic flow (Joint)✓✓✗✓6/104/10
Panoramic flow + SMR (Frozen)✓✓✓✗4/102/10
Uni-Fly360 (ours) (Joint)✓✓✓✓10/108/10

Thank you for the task evaluation and real-world experiments. These are useful additions. I have two remaining questions:

  1. Please specify the quantitative thresholds used to determine task success and report the position/offset errors, trajectory progress, and heading errors for Uni-Fly360 and Fly360. Please also clarify whether the revised evaluation uses the same episodes as the original results.
  2. Did the real-world flight experiments include camera rotation, particularly changes in yaw? If so, could you briefly describe the tested conditions? This would help clarify the method's performance under rotation.
AuthorsOpenReview ↗

We report relevant experimental data and give some explanation.

  1. Thank you for the question. In our simulation experiments, yaw was fixed directly; therefore, heading error was not used as a task evaluation metric. In addition to requiring collision-free flight, we applied the following task-specific criteria to determine success:
TaskTask-success criterion
Hovering MaintenanceThe position error relative to the target hovering point remains at or below 1.5 m throughout the episode.
Dynamic Target FollowingThe UAV reaches the endpoint and passes within 1 m of each predefined waypoint.
Fixed-Trajectory FilmingThe UAV reaches the endpoint and passes within 1 m of each predefined waypoint.

For Hovering Maintenance, the task is to avoid obstacles while maintaining the UAV's position within the specified region around the target point.

For Dynamic Target Following and Fixed-Trajectory Filming, the task setup permits the UAV to deviate from a fixed path when avoiding obstacles. We therefore specify a sparse set of waypoints rather than requiring the UAV to follow a single predefined trajectory. We report:

  • Endpoint position error: the average distance to the endpoint in successful episodes.
  • Waypoint offset error: for each waypoint, we calculate the minimum distance between the UAV and that waypoint over the episode. We average these distances across waypoints within each successful episode, and then average the resulting values across successful episodes.
  • Trajectory progress: the average percentage of the route completed in failed episodes.
TaskMethodEndpoint position error in successful episodes (m) ↓Waypoint offset error in successful episodes (m) ↓Trajectory progress in failed episodes (%) ↑
Dynamic Target FollowingFly3600.450.5243.71
Dynamic Target FollowingUni-Fly3600.430.5662.4
Fixed-Trajectory FilmingFly3600.720.3688.2
Fixed-Trajectory FilmingUni-Fly3600.720.2993.1

For the revised evaluation, we reran the experiments using the same experimental settings as the original evaluation, rather than recomputing the metrics from the original episode logs.

2.We apologize for the ambiguity in our use of “fixed yaw.” The real-world flights included camera rotation relative to the world as the UAV changed its attitude, while the camera remained fixed relative to the UAV body.

Our panoramic camera consists of two fisheye cameras whose images are stitched into a panoramic view. It was rigidly mounted on the UAV. Here, “fixed yaw” refers to a fixed desired heading for the UAV in the world frame. The UAV's attitude was not locked, and its actual yaw, roll, and pitch varied during flight. As we did not use IMU measurements to perform horizon correction on the panoramic observations, the input image sequences retained the effects of these attitude changes. The experiments therefore included camera rotation during flight.

Thus, the observed optical flow contains contributions from both camera translation and rotation. Translational flow carries scene-dependent geometric cues useful for navigation. SMR exploits spherical motion constraints to reduce the influence of rotational flow when estimating the global translation axis, supporting navigation under the attitude changes encountered during flight.

To examine the contribution of SMR under these real-world flight conditions, we compared the following variants:

VariantABCDHovering Obstacle Avoidance: SR ↑Continuous Pursuit Evasion: SR ↑
Depth (Frozen)✗✓✗✗7/105/10
Panoramic flow (Frozen)✓✓✗✗3/101/10
Panoramic flow (Joint)✓✓✗✓6/104/10
Panoramic flow + SMR (Frozen)✓✓✓✗4/102/10
Uni-Fly360 (ours) (Joint)✓✓✓✓10/108/10

A: optical-flow representation; B: panoramic observations; C: spherical motion reasoning; D: joint perception--control optimization. SR is the number of successful trials out of 10.

Under joint training, adding SMR increases the number of successful trials from 6/10 to 10/10 on hovering obstacle avoidance and from 4/10 to 8/10 on continuous pursuit evasion. These results support the contribution of SMR to closed-loop performance under the attitude changes encountered in these flights.

Thank you for your response and clarifications. My concerns have been addressed, and I have updated my rating from 2 to 3. Please incorporate the additional explanations and experimental results into the final version of the paper, ensuring that they are clearly presented and well organized.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.