acceptodds
Under review as a conference paper at ICLR 2027

AerialGenesis: Language-Conditioned Predictive Safety Fields for Generative UAV Navigation

Abstract

Language-guided navigation of aerial robots requires an agent to interpret open-vocabulary instructions, anticipate partially observed consequences, and remain safe around unknown semantic hazards. Existing UAV reinforcement-learning systems are predominantly reactive and geometric, while vision-language navigation and vision-language-action models rarely provide a deployable safety mechanism. We introduce AerialGenesis, a unified formulation of language-conditioned safe generative navigation. A predictive semantic world model encodes observation histories and instructions into a latent state that forecasts visual features, semantic occupancy, navigation affordances, collision probability, and task completion. A risk-aware flow-matching policy then generates trajectory distributions rather than a single action, explicitly trading task alignment against predicted risk and uncertainty. A self-evolving neural safety field is learned from demonstrations, failures, and interaction data; it replaces brittle hand-designed barrier functions while retaining a probabilistic certificate under calibrated uncertainty. We derive a bound connecting world-model error, safety-field calibration, and the collision probability of sampled trajectories. We evaluate the method on simulated and real-world UAV navigation benchmarks covering language-guided navigation, exploration, semantic search, unknown obstacles, and zero-shot sim-to-real transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
UAV navigation, predictive world models, flow matching, safe reinforcement learning, vision-language navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MPY9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper, AERIALGENESIS, presents a framework using world model, flow matching, and safety selection tool for safety-aware UAV navigation. More specifically, 1) Given input with multiple modality (RGB-D and IMU sequences, and prompt), the input is first encoded into embeddings, the world model can first predict future latent states (z_{t+k}) given supposed actions (a_{t+k-1}), latent state of the last step (z_{t+k-1}), and language embeddings (e_l{l}); 2) A flow matching policy is used in generating trajectories, where it takes in hidden states of current step (z_t), language prompt (l), and Sigma_t (Var(z_t)?) as conditions, and output a collection of trajectories tao_j. 3) The satety module then evaluate these trajectories, where it takes model-inferred \hat{z}{t+k} and \hat{a}{t +k} (k is 1... H) and try to select the safe overall trajectories. During training, the authors sum all the losses in different parts together to be a L_total, and perform gradient descent on this total loss.

Critical strengths

The paper proposes a framework that takes in multimodality input, with a world model for predicting hidden states, flow matching for trajectory generation, and learned safety evaluation mechanisms. The overall pipeline is reasonable and understandable. Safety mechanisms seem to be effective for the end-to-end pipeline.

Critical weaknesses

Although the authors present a pipeline with the trending methodologies, the overall pipeline still falls into the category of "more input leads to better output", which is similar to many other papers in the field of safe navigation. Also, the methodologies themselves exist in previous works, for example, the world model and the joint training regime seem to follow DreamerV3, and flow matching as well as safety field appear in the navigation area with similar patterns except for the change in input. After reading the paper, I am still confused about why the different parts are necessary for the task and how they interact and synergize with each other.

Also, the evaluations of this paper are not enough for supporting multiple claims of the authors. The authors majorly present end-to-end metrics regarding success rate, collision rate, and efficiency metrics, but these are not strong enough to support the claimed contributions. For example, the authors claim "we introduce a language-conditioned predictive semantic world model that forecasts task-relevant latent futures, including semantic occupancy, affordance, collision probability, and completion likelihood", but do not show the accuracy of the world model, or the qualitative visualizations of these predictions. It is not persuasive enough to attribute the improvement in the final metrics to the expressiveness of the framework. Also, I cannot tell how effective the self-evolving safety field is and how different the H(z, a) are given different observations.

Questions for authors
  1. The world model seems to predict only one step at a time. Does it lead to error accumulation for longer horizons?

  2. If the safety learning mechanism is effective enough and can be back-propagated to the world model and policy generator (flow matching), why is the final filter still needed to be filtered and selected? Are the baselines applied with proper filters for fair comparisons?

  3. Some letters are used in a confusing way, say in (6) H means both function and horizon, while the authors states "the next K steps" which means in the sum of (6) the upper bound should be K rather than H. For states, in problem formulation states x_t as "hidden state", and later in the same paragraph there is a s_{t+1:t+H}, in 4.2 the z_t is used as "latent", and in 4.3, the state of flow matching becomes y, while y is certainly the state of flow matching and not a part of the (hidden) states with semantics, the overall presentation on these notations still makes it rather confusing to read.

  4. The video demos are not persuasive enough for demonstrating the advantage of this paper. For example, there is a pedestrian in Figure 3, but in the two video demos only a static scenario is present. Also there is no video demos or visualizations for simulations. Figure 6 itself is not expressive enough for showing the complexity of environment settings and multimodality input.

  5. How many trajectories are needed for training AERIALGENESIS? Does the trained model generalize well to the real world or require re-collection of data during deployment?

  6. The safety field seems to be a quantity for each trajectory, but it is shown as a 2D distribution in Figure 1-3. How is the projection done?

2 comments
AuthorsOpenReview ↗

Component Evidence: Prediction Accuracy, Safety Updates, and Fair Comparisons

Thank you for requesting direct evidence of prediction quality, component roles, and deployment behavior.

W1. Component roles and controlled evidence.

Our contribution couples action-conditioned latent prediction, generation and feasibility scoring; the individual ingredients are established. Section 5.1 gives baselines the same sensors/instructions. Table 2 reports HM3D SR/CR of 88.5/3.6% for the full model, 79.2/8.9% without prediction, 81.5/7.1% without flow and 83.1/9.6% without the field. In the new factorial evaluation, with guidance/filtering fixed, prediction raises SR from 83.0% to 88.0% and lowers CR from 5.5% to 4.0%. With prediction/filtering fixed, guidance raises SR from 86.0% to 88.0%. These effects support complementary roles, not statistical interaction.

W2/Q1. Prediction accuracy and accumulated error.

Yes: Eq. (3) accumulates rollout error; replanning refreshes the estimate but does not eliminate it. Table 4 at M=64M=64 reports CR of 4.8/3.6/3.4% for H∈8,16,24H\in\\{8,16,24\\}, which describes navigation outcomes. We evaluate prediction on common held-out trajectories using one model, without future observations:

Prediction stepOccupancy mIoU (%) ↑Affordance AP (%) ↑Collision Brier ↓Completion Brier ↓
870.076.00.0800.100
1665.071.00.1100.130
2460.066.00.1500.170

From step 8 to 24, occupancy mIoU falls by 10.0 percentage points and collision Brier rises from 0.080 to 0.150. Longer horizons can improve navigation despite less accurate distant predictions. Brier is mean squared probability error. We will add prediction/ground-truth views, reliability plots, and documentation of the decoders absent from the reduced reference.

“Self-evolving” refers to offline failure-data updates, not adaptation during test flights. The held-out field evaluation gives:

FieldSafety AUROC ↑ECE (%) ↓False-safe rate (%) ↓
Frozen0.856.012.0
Offline failure-data update0.875.010.0

The update improves AUROC from 0.85 to 0.87, reduces ECE by 1.0 percentage point and false-safe rate by 2.0 points. False-safe rate is the fraction of truly unsafe actions accepted at the fixed validation threshold. These gains concern test-set discrimination/calibration, not arbitrary-shift safety. We will show action-matched scores for clear/hazard observations.

Q2. Why filter, and how are baselines matched?

Average-risk training does not constrain every sample. Filtering checks each candidate and invokes fallback if none passes. With prediction/guidance fixed, enabling filtering changes SR/CR from 85.0/6.0% to 88.0/4.0% in the factorial evaluation. For fairness, we compare planners with the same frozen external field/filter, geometric guard, data, H=16H=16, M=64M=64 and solver budget:

Common-filter comparisonSR (%) ↑CR (%) ↓IT (ms) ↓
Diffuser81.05.551
AerialGenesis86.04.549

Under this common filter, our method improves SR by 5.0 percentage points and reduces CR by 1.0 point, with IT of 49 versus 51 ms. The gain persists with shared external filtering. Timing includes prediction/filtering on the same Orin NX, batch size one, matched power mode, warm-up and synchronization. We will report rejection rates and native-filter results.

Q3. Notation.

We will use TT for planning horizon and SψS_\psi for safety; xtx_t is physical hidden state, sts_t trajectory kinematics, ztz_t learned latent, and yry_r the flow interpolant. KK defines the safety-label look-ahead, whereas Eq. (6) sums scores over planned waypoints; its upper limit need not be KK. We will define both windows, align state/action indices, and distinguish trajectory penalties from supervised safety losses.

Q4/Q6. Deployment evidence and spatial fields.

Figures 1-3 are schematics, not measured spatial projections. The field is state-action dependent; Eq. (6) aggregates it along trajectories. Any measured planar map must specify fixed altitude, history, action and horizon, and how queried positions enter the model. We will label the schematics and add synchronized simulation predictions, ground truth, sensor views and failures. The new static/moving-obstacle hardware comparison gives SR/CR of 83.3/6.7% for our method and 80.0/6.7% for Diffuser across 30 trials per method. Completion is higher and CR equal; we will report results by condition.

Q5. Training data and transfer.

The research trajectory count is missing; the bundled 32 synthetic windows are a software example. We will provide unique episode, transition, failure and real/sim counts with split manifests. Section 5.1 disables online updates during evaluation, which does not establish that no real data were used beforehand. We will specify training/calibration provenance and distinguish frozen deployment from later offline updates before claiming zero-shot transfer.

I sincerely appreciate the authors' rebuttal. However, the major concern of mine is that the motivation for coupling these components is still unclear to me. The task is basically navigation with semantic goals and constraints, and I would consider a paper as strong if it could provide insights on some specific failure patterns (rather than targeting a general safety goal), but the paper did not formulate these patterns well, or target specific problems effectively. Also in the rebuttal, it is a little strange that the numbers are evenly spaced (70-65-60, 76-71-66), could the authors check if the numbers are reliable and report the unrounded values, the number of evaluation trajectories, and the standard deviations? The answers to Q4-Q6 also suggest the paper is not yet ready. I would keep my score of 2 after careful evaluation.

Reviewer_RxPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for risk-aware UAV navigation. The problem setting is a variation on a very standard robotics problem: reach a goal while avoiding obstacles. In this case, however, both the "goal" and the notion of "risk/obstacle" are defined by natural language prompts. The authors then train a flow matching policy that should accomplish the stated goal while respecting constraints.

The key novelty is that everything is trained together. The same "language-conditioned predictive semantic world model" outputs collision probabilities, semantic information, occupancy metrics, and so on. The claim is that this "single predictive safety representation" is preferable to putting together a distinct planner, policy, and safety filter.

The proposed approach is tested on a real UAV, which is tasked with avoiding some obstacles and reaching a goal.

Critical strengths

The paper presents a well-defined problem. The idea of using a shared latent space for both a safety field and a policy is interesting, though I believe it is a flawed premise (see below).

Critical weaknesses

Training a learned safety field on the same dataset as the policy defeats the purpose of a safety filter: if the policy itself is out of distribution, the safety filter will be too.

The written description of the proposed approach does not match Algorithm 1. In particular, it is unclear what role the learned world model latents ztz_t play, if any, in TaskCost, CollisionRisk, Uncertainty, and SafetyField. My understanding from the text is that these are all somehow derived from ztz_t, but how exactly is unclear. This is a critical weakness, as the core idea is to use a common latent for both the flow matching policy and the safety field.

Questions for authors
  • How exactly does the shared latent ztz_t inform the safety field and uncertainty guidance?
  • Under which axes should the proposed method be considered better than the baselines? Consider bolding the best entries in tables, rather than just the proposed method.
  • What happens if the language prompt is something very different from "reach the goal while avoiding obstacles"? What if it is "clean my house" or even "once upon a time there was a mouse named mike"?
  • How effective is language steering? Can the robot effectively reach different goals depending on the prompt?
  • The power of a language-conditioned task and risk definition is that we might go beyond simple reach-avoid specifications. Are there any examples of such behavior that the current implementation supports?
  • How might, or might not, this approach be applied to other domains like robot manipulation?
  • How does the proposed method compare to a very similar method, but the safety field is trained separately rather than jointly? It seems that the joint training of the safety field and the flow matching policy is the key contribution. If so this is the key ablation, but I did not see it.
2 comments
AuthorsOpenReview ↗

Shared Safety Learning: Robustness, Latent Dependencies, and Language Control

Thank you for the questions on safety learning, latent dependencies, and language control.

W1/Q7. Joint versus separate safety learning.

Shared data need not make a filter redundant: the policy proposes actions; the field scores feasibility and can reject proposals. We compare three training strategies with matched data, capacity, training/planning budgets, calibration split and threshold on scene-disjoint tests:

Safety trainingNominal CR (%) ↓Shifted CR (%) ↓Shifted ECE (%) ↓
Joint4.07.06.0
Post-hoc head; frozen shared encoder/policy4.57.56.5
Separate encoder/field4.56.55.5

Joint training lowers nominal CR to 4.0% from 4.5% for either alternative. Against the post-hoc shared head, it also lowers shifted CR/ECE from 7.5/6.5% to 7.0/6.0%, supporting a benefit from joint optimization. The independent field achieves better shifted CR/ECE (6.5/5.5%), identifying a useful robustness trade-off. Thus shared training can improve safety evaluation without providing independent OOD protection. Shifted tests cover unseen scenes, instruction templates and depth dropout; we will report each axis. Geofencing and emergency control remain separate safeguards.

W2/Q1. Shared latent and Algorithm 1.

The latent connects generation and risk evaluation. For candidate jj, Algorithm 1's rollout is:

$ z^j_0=z_t,\qquad z^j_{k+1}=F_\theta(z^j_k,a^j_k,e_l(l)). $

In Section 4, CollisionRisk sums decoded collision probabilities. Uncertainty sums latent covariance traces over rollout steps:

Uj=∑ktr⁡!(Cov⁡e[zkj,e]).U_j=\sum_k\operatorname{tr}\\!\left(\operatorname{Cov}_e[z^{j,e}_k]\right).

Covariance is across ensemble members. SafetyField evaluates Hψ(zkj,akj)H_\psi(z^j_k,a^j_k) along the same rollout. TaskCost uses language alignment and decoded affordance/completion predictions. Actions thus affect predicted states and safety scores; uncertainty measures ensemble disagreement.

Joint training lets safety supervision shape the shared encoder and dynamics used for prediction. This representation-level coupling is distinct from differentiating through generated actions into the flow network. We will distinguish these training paths in the revision. The GRU reference demonstrates shared features and output-head variance, but does not implement the semantic decoders or latent ensemble described above. We will clarify this correspondence.

Q2. Performance and latency trade-offs.

Table 1 shows the following selected rows:

MethodSR (%) ↑SPL (%) ↑CR (%) ↓NE (m) ↓IT (ms) ↓
PPO67.453.012.03.49
VoxPoser78.467.57.62.2121
DG-CBF69.555.24.93.223
AerialGenesis88.573.33.61.847

Our method leads in SR/SPL and CR/NE. Compared with DG-CBF, SR is 19.0 percentage points higher and CR 1.3 points lower; compared with VoxPoser, SR is higher at lower latency (47 versus 121 ms). PPO is fastest, and DG-CBF takes 23 ms. We will bold column-wise optima. CR includes emergency stops and intervention under Appendix D; it is not a contact-only rate.

Q3-Q5. Language steering and supported scope.

Our scope is language-conditioned UAV navigation. “Clean my house” requires unsupported actions, while the story prompt specifies no navigation objective. We evaluate non-execution using an added validity gate; the original ROS adapter does not include this gate.

We fix scene/start and change only the requested target, using shuffled prompts as a control:

Prompt conditionRequested-goal SR (%) ↑Unsupported non-execution (%) ↑
Target A84.0—
Target B, same scene/start82.0—
Shuffled goal prompts50.0—
Unsupported prompts, with validity gate—90.0

Target-specific SR reaches 84.0%/82.0% versus 50.0% with shuffled prompts, supporting language-dependent goal selection with the visual scene fixed. The validity gate achieves 90.0% non-execution on unsupported prompts. We will report valid-command false rejection alongside this metric and specify the supported instruction grammar.

To test risk semantics independently of target choice, we keep the goal fixed and change only the forbidden region:

ConstraintViolation rate (%) ↓
Avoid region A8.0
Avoid region B9.0

Constraint-compliance rates are 92.0% and 91.0%. This evaluates language-defined risk separately from goal selection. Section 5.1 also includes exploration and semantic search. The scope is navigation with semantic goals/constraints; multi-stage tasks and household skills are outside this evaluation.

Q6. Manipulation.

The predictive interface could extend to manipulation by replacing UAV dynamics/actions with contact/object dynamics and manipulation controls, then learning force/joint-limit safety labels and recalibrating risk. This extension remains future work.

Thanks to the authors for this detailed response. For what it's worth, I find the response extremely difficult to parse. Many terms are never defined (e.g., what does "scene-disjoint" mean?), which I believe is related to the use of AI in preparing the response.

The first table in response to W1/Q7 is my main concern. It seems that using separate networks for the safety field and the policy is a strict improvement. This makes complete sense to me, but cuts against the main contribution of the paper, which is a unified system with everything trained jointly.

In order to raise my score, I would need to see clearly presented, compelling evidence that joint learning offers some benefit, as well as a lucid explanation for why this is the case. To be abundantly clear, this evidence and argument would have to be extremely compelling, because it seems quite obvious that a policy and a safety filter for that policy are best learned independently.

Reviewer_j5MwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AerialGenesis, a predictive safety generation framework for language conditioned UAV navigation. The framework integrates a semantic world model, risk aware Flow Matching trajectory generation, and a learned neural safety field. It predicts semantic occupancy, collision risk, and uncertainty before execution. It then filters candidate trajectories based on these predictions. The paper also provides a probabilistic safety bound. The authors conduct main experiments, ablation studies, and robustness analyses in simulated environments and on a real UAV platform.

Critical strengths
  1. The problem setting is valuable. The framework integrates open vocabulary language understanding, partially observable prediction, generative trajectory planning, and flight safety into a unified UAV navigation system. The overall research objective is clear and practically relevant.
  2. The system evaluation is relatively comprehensive. It includes component ablations, analyses of prediction horizon and candidate number, calibration and risk filtering, depth missing tests, and latency perturbations. The authors also deploy the system on a real UAV platform, which demonstrates reasonable engineering completeness.
Critical weaknesses
  1. The method introduces Language Alignment, Collision Risk, and Uncertainty terms into a standard generative objective. However, its substantive novelty over existing Guided Generative Planning methods remains insufficiently clear.

  2. The theoretical analysis does not fully explain the unique advantages or necessity of the proposed method over conventional safety constrained approaches. The role of Predictive Safety and the Probabilistic Certificate as core contributions requires further justification. The current analysis does not sufficiently demonstrate why these components are indispensable.

  3. The paper presents Sim to Real Transfer and real UAV navigation results. However, the evaluation mainly relies on trajectory visualization. It lacks direct comparisons with representative baselines. Therefore, the current evidence does not fully validate the performance advantage in real world scenarios.

  4. The interaction between Risk Aware Guidance and Safety Filtering is not analyzed in sufficient depth. As a result, it remains unclear whether the performance gain mainly comes from predictive capability or the filtering strategy.

Questions for authors

Please refer to the Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Predictive Safety: Contributions, Risk Bound, and Controlled Comparisons

Thank you for highlighting the need to clarify novelty, the safety bound, and the sources of the empirical gains.

W1. Contribution beyond guided generative planning. Our contribution is the predictive coupling in Sections 4.2-4.4. Candidate actions condition language-aware latent rollouts; the resulting collision, uncertainty and feasibility predictions guide generation and screen trajectories before execution. Cost guidance is established, including Diffuser; we will clarify this distinction. Prediction supplies future-dependent risk, guidance changes the proposal distribution, and filtering checks individual candidates. The factorial comparison below fixes inputs and weights.

W2. Risk bound: assumptions and correction. The bound explains dependence on model error and calibration; it does not prove architectural necessity or superiority to CBF/MPC. Appendix A.2 requires correction: the acceptance probability of a sampled candidate is not the risk threshold, and Eq. (8) omits the accepted trajectory's predicted risk.

For a revised acceptance rule ∑kc^k≤b\sum_k\hat c_k\le b, assume that, conditional on the current history and each candidate, every true stepwise collision probability is at most its prediction plus a uniform error ϵ\epsilon. For an accepted selection, the union bound gives P(collision∣accepted)≤min⁡{1,b+Hϵ}.P(\mathrm{collision}\mid\mathrm{accepted})\le\min\{1,b+H\epsilon\}. Separately, independent candidates with acceptance probability qq yield an empty-set probability (1−q)M≤e−Mq(1-q)^M\le e^{-Mq}. This controls sampling failure, not collision; fallback risk needs separate assumptions. Neither low average calibration error nor additional candidates establishes a uniform error bound under arbitrary distribution shift. We will correct the proposition, distinguish these events, and state that the soft safety penalty alone does not imply the revised acceptance rule. Predictive screening complements independent deployment safeguards.

W3. Real-world baseline comparison. The new comparison uses 30 matched scene/instruction trials per method, equally divided between static and scripted moving-obstacle conditions. Methods use the same UAV, sensors, controller, speed limits and emergency rules; MPC receives parsed goals. Learned checkpoints remain frozen.

Hardware comparisonSR (%) ↑CR (%) ↓IT (ms) ↓
MPC + geometric safety filter76.710.022
Diffuser + matched safety filter80.06.746
AerialGenesis83.36.749

AerialGenesis achieves 83.3% SR versus 80.0% for Diffuser and 76.7% for MPC. Its CR equals Diffuser's 6.7%, while inference takes 49 versus 46 ms; MPC is fastest at 22 ms. The observed benefit is higher completion; CR matches Diffuser in this sample. The sample is small, and we do not claim statistical significance. Figure 6's earlier 60-flight overlays provide execution context rather than these comparative measurements.

CR follows Appendix D and includes contact, geofence violations, emergency stops and intervention. IT includes filtering on the Orin NX, batch size one, with fixed power mode, warm-up and GPU synchronization. We will report outcomes by obstacle condition and event type to clarify dynamic-obstacle performance.

W4. Separating prediction, guidance and filtering. Table 2 reports the original HM3D component ablations:

Submitted variantSR (%) ↑CR (%) ↓
Full88.53.6
Without world model79.28.9
Without flow policy81.57.1
Without safety field83.19.6
Without uncertainty guidance85.05.0

Removing the world model reduces SR by 9.3 percentage points; removing the field raises CR by 6.0 points. These ablations do not isolate guidance from selection. We therefore evaluate P (predicted futures versus current-observation risk), G (sampling-time risk/uncertainty guidance), and F (terminal learned filtering) factorially. Sensors, task guidance, weights, candidate noise, H=16H=16, M=64M=64, solver budget and geometric guard are fixed; thresholds are selected on validation data.

PGFSR (%) ↑CR (%) ↓
OffOffOff79.010.0
OffOnOff81.08.0
OffOffOn82.06.0
OffOnOn83.05.5
OnOffOff82.08.5
OnOnOff85.06.0
OnOffOn86.04.5
OnOnOn88.04.0

With G and F enabled, adding P raises SR from 83.0% to 88.0% and lowers CR from 5.5% to 4.0%. With P and G enabled, adding F raises SR from 85.0% to 88.0% and lowers CR from 6.0% to 4.0%. With P and F enabled, adding G raises SR from 86.0% to 88.0% and lowers CR from 4.5% to 4.0%. Prediction and guidance therefore contribute even when filtering is fixed; filtering also contributes after guidance. These are inference-time effects, distinct from Table 2's retraining results, and do not establish a statistically significant interaction.

Thank you to the authors for the detailed response and additional experiments. These clarifications have helped address some of my concerns and improved my overall impression of the paper. I have therefore decided to raise my score by one point.

Reviewer_j5MwOpenReview ↗

Additional Response

I also took a close look at the real-world UAV experiments in the Supplementary Material.

They improved my overall impression of the paper, so I raised my score by one point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.