acceptodds
Withdrawn from ICLR 2027

The World Outside, the Motion Within: Holistic Spatial Intelligence for UAVs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial intelligence for unmanned aerial vehicles (UAVs) relies on the tight coupling of external environmental perception and internal ego-motion awareness. However, existing models struggle to preserve spatial knowledge across viewpoint changes and to disentangle ego-motion from object motion while capturing how the UAV's motion shapes future observations. To address these challenges, we introduce EnvSelf-UAV, a unified framework that couples external spatial perception with internal ego-motion understanding. Specifically, a Dynamic Environment Spatial Memory (DESM) module maintains spatially grounded memory by updating regional appearance and geometry across video frames. An Ego-Motion Dynamics Modeling (EMDM) module infers UAV motion from shared geometric changes across matched regions and uses a latent world model to predict future environmental representations conditioned on this motion. A Question-Guided Memory Injection (QGMI) mechanism then selectively integrates environmental, motion, and predictive evidence into a language decoder for spatial reasoning. Extensive evaluations across three key benchmarks demonstrate that EnvSelf-UAV achieves superior performance, while downstream navigation tasks further validate the effectiveness of the proposed architecture.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Intelligence, UAV, Latent World Models, Vision–Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XVnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EnvSelf-UAV, a framework for UAV spatial reasoning built on Qwen3.5. DESM maintains K learnable memory cells that are updated across frames with gated appearance and box updates. EMDM matches memory regions across adjacent frames, fits a 2D affine transform to estimate a motion state, and rolls out a latent world model to predict future environment features. QGMI injects environment, motion and predicted memories into the decoder through gated cross-attention. Results are reported on SIS-Bench, zero-shot SpatialUAV and OpenUAV-QA with 4B and 9B backbones.s

Critical strengths
  • The decomposition into environment memory, ego-motion and future prediction is clear, and the method section is easy to follow.
  • The gains are large on SIS-Bench (76.9 vs 61.1 for the 4B backbone and 66.9 for the SFT baseline), and the model also transfers to SpatialUAV without training on it.
  • Training data, hyperparameters and module settings are reported in Appendix B.3.
Critical weaknesses
  1. The text layer of the PDF figures contains notes that are not visible in the rendered figures. Figure 3 contains "Other models: illustrative estimates" and "SIS-Motion: supplied result", and Figure 4 contains "Open markers: illustrative estimates". The authors should confirm whether every plotted value in Figures 3 and 4 is a measured result. If some values are estimates, the OpenUAV-QA comparison and the sensitivity analysis cannot support the claims in Sections 4.2 and 4.4.
  2. The ego-motion module fits a 2D affine transform to the centers of 16 memory boxes. This cannot separate ego-motion from independently moving objects, and it does not model parallax or altitude change, although both are central claims of the paper. Appendix A.2 also states that the motion state is not a metric pose. No experiment isolates scenes with moving objects, and camera transformation on SpatialUAV stays at 1.8 to 2.6.
  3. Several component gains in Table 3 are small: QGMI adds 0.3 points on SIS-Bench, and DESM lowers path planning from 35.2 to 30.4. All results come from a single seed (seed 42), so it is unclear which differences exceed run-to-run variance.
  4. Most baselines in Tables 1 and 2 are zero-shot general models, while EnvSelf-UAV is fine-tuned on SIS-Motion-54K. The fair comparisons are the SFT baseline and SIS-Motion. Memory-based or 3D-aware video models, such as VLM-3R, which is cited, are not compared. The navigation evaluation on OpenUAV-QA is multiple-choice question answering, not closed-loop flight, and the margin over SIS-Motion is 2.3 points on a truncated axis.
  5. The paper does not report the added parameters, training cost or inference latency of the three modules.
Questions for authors
  1. Are all values in Figures 3 and 4 measured? Please provide the full numbers for every model and every K and H setting.
  2. Can the motion state be evaluated directly against ground-truth camera motion, and how does EMDM behave when large objects move independently?
  3. Please report mean and standard deviation over several training seeds for Table 3.
  4. How does EnvSelf-UAV compare with a 3D-aware video baseline such as VLM-3R fine-tuned on the same data?
Reviewer_GKmNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EnvSelf-UAV, a framework for UAV spatial reasoning that integrates environmental memory, ego-motion understanding, and future representation prediction. Its Dynamic Environment Spatial Memory maintains regional appearance and geometric information across frames. Ego-Motion Dynamics Modeling estimates motion representations from common geometric changes among matched regions and conditions a latent dynamics model to predict future environmental features. Question-Guided Memory Injection incorporates these complementary representations into a language decoder. The authors evaluate the framework on SIS-Bench, SpatialUAV, and OpenUAV-QA, reporting improved spatial and motion-related question answering, cross-benchmark transfer, and navigation-related decision accuracy. Progressive ablations examine the contributions of the proposed components.

Critical strengths

1.The paper addresses a meaningful problem with a coherent approach. Maintaining environmental knowledge under viewpoint changes and distinguishing ego-motion from object motion are important challenges for UAV visual understanding. Connecting persistent regional memory, motion inference, and future representation prediction is a reasonable approach to these challenges. This alignment between the problem and the architecture makes the framework potentially useful for UAV spatial reasoning.

2.The reported results provide encouraging evidence of practical utility. In Table 3, the complete framework improves over the task-finetuned baseline from 66.9 to 76.9 on SIS-Bench and from 44.1 to 49.8 on SpatialUAV. The cross-benchmark results and progressive ablations suggest benefits beyond task finetuning alone. These improvements support the framework’s empirical value, although their precise sources require further clarification.

Critical weaknesses

1.The contribution of the proposed architecture is insufficiently separated from additional training and established alternatives. The method includes AerialVG image pretraining with spatial supervision, but it is unclear whether the SFT baseline receives equivalent data, supervision, and training budget. Consequently, the substantial improvement attributed to DESM may partly reflect additional training. Furthermore, the progressive ablations do not establish whether standard recurrent memory and motion-feature fusion would achieve comparable performance under matched conditions. This matters because the paper’s contribution depends on demonstrating the value of its particular integration of largely established components.

2.The experiments provide limited direct validation of the mechanisms underlying the central motivation. Table 3 retains aggregate scores but reports only four SIS-Bench subtask results, omitting Action Prediction and spatial/spatiotemporal consistency despite their relevance to future prediction and persistent spatial understanding. Reporting a subset is reasonable, but the selected breakdown leaves important claims insufficiently tested. Moreover, the contribution of EMDM’s geometric pathway relative to its global-feature fallback is unclear, and LWM is not evaluated through held-out prediction accuracy or a comparison without motion conditioning. Thus, the reported improvements do not sufficiently establish whether the intended mechanisms address the stated difficulties, or whether benefits arise mainly from additional supervision and model capacity.

Questions for authors

1.Clarification of training controls and performance attribution. Does the Table 3 SFT baseline receive the same AerialVG image pretraining and spatial supervision as the complete method? Please specify the training differences between variants. Under matched training conditions, what evidence distinguishes the benefit of DESM from that of a standard recurrent or query-based memory?

2.Clarification of the geometric motion pathway. What fraction of evaluated transitions uses valid region-based affine estimation, and what fraction invokes the global-feature fallback? How does performance change when geometric consistency weighting is removed or the fallback pathway is used throughout? This would help determine whether the proposed geometric mechanism contributes materially to the gains.

3.Task-level evidence connecting the ablations to the motivation. Could the authors report the Action Prediction results with and without LWM, and the spatial/spatiotemporal consistency results with and without temporal DESM updates? These tasks would more directly assess the capabilities motivating the respective modules. For LWM, does removing motion conditioning degrade held-out future prediction accuracy and downstream performance? This would help establish whether its benefit depends on modeling motion-conditioned environmental evolution.

My current assessment is tentative. I would consider raising my score if the authors can clarify the fairness of the training comparisons and provide convincing evidence that the proposed memory, geometric motion inference, and motion-conditioned prediction mechanisms contribute to the reported gains. Targeted results addressing the questions above would be particularly helpful.

Reviewer_NPiwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EnvSelf-UAV, which improves the spatial understanding and action reasoning of UAV vision–language models by jointly modeling the external environment and the vehicle's own motion. The framework consists of three components. Dynamic Environment Spatial Memory (DESM) maintains memory cells augmented with 2D region geometry and updates the environmental representation across frames. Ego-Motion Dynamics Modeling (EMDM) builds a motion representation through region matching and consistency weighting with respect to the common image-plane motion, and uses a latent world model (LWM) to predict future environmental features. Question-Guided Memory Injection (QGMI) retrieves information from the environmental, motion, and predicted memories conditioned on the question and the decoding context, and injects it into the language decoder through a gated residual connection.

The model is trained in two stages, spatial image pretraining and video adaptation, using image instruction data converted from AerialVG and SIS-Motion-54K, respectively. The method is evaluated at two scales, Qwen3.5-4B and Qwen3.5-9B, achieving 76.9% and 79.9% accuracy on SIS-Bench and overall scores of 49.8 and 53.9 on SpatialUAV, respectively, and it reports 94.5% accuracy on offline navigation QA on OpenUAV-QA. The authors further provide component ablations, analyses of memory capacity and prediction horizon, and both success and failure cases.

However, one issue needs to be clarified first: the source files of Figures 3 and 4 contain embedded notes that label some of the values as "illustrative estimates." Until the authors clarify this, I cannot treat the OpenUAV-QA comparison or the parameter analysis as supporting evidence (see Weakness 1). In addition, the training configurations of the ablation variants and the model-selection procedure for the zero-shot evaluation need to be specified, and the specific benefits of geometric consistency and motion-conditioned prediction are currently supported only indirectly by downstream QA performance. Providing these clarifications, together with a small number of targeted mechanism-level controls, would substantially strengthen the paper.

The author needs to provide evidence and relevant explanations to explain these invisible annotations, otherwise I will not raise my score.

Critical strengths
  1. The problem setting has clear domain value, and the method is coherently motivated.

UAV video understanding must simultaneously cope with continuously changing viewpoints, historical environmental information, and visual changes caused jointly by camera motion and the motion of scene objects. The paper addresses these issues by organizing environmental memory, motion understanding, and future environment prediction into a unified framework with clear information dependencies among its components. Using historical regional evidence to support motion inference, and then feeding motion and predictive information into language reasoning, is a design well suited to the task.

  1. The full method yields clear performance gains and outperforms the task-adapted SFT baseline.

On SIS-Bench, the 4B model improves from 61.1% (backbone) to 76.9%, still 10.0 points above the SFT baseline (66.9%). On SpatialUAV, the full model reaches 49.8 compared with 44.1 for SFT, a gain of 5.7 points. The 9B version improves further on both benchmarks and achieves the highest overall score among the non-proprietary models listed in Table 2. These results provide fairly strong support for the effectiveness of the full training recipe combined with the architecture.

  1. The ablations cover the main components and provide a useful first-level attribution.

Table 3 separates the contributions of image-based spatial memory, temporal adaptation, ego-motion inference, future prediction, and QGMI. Extending image-only DESM with temporal adaptation raises the SIS-Bench overall score from 71.6% to 74.7%, and adding the LWM on top of motion inference raises path planning from 35.7 to 42.1. These results indicate that the main components contribute to downstream performance and help readers understand the role of each part of the framework.

  1. Robustness and training-information boundaries are handled carefully.

EMDM includes mutual-match filtering, robust consensus fitting, geometric-consistency weighting, and a fallback to global features when fitting fails, with the fallback branch sampled during training. The authors explicitly distinguish their non-metric motion representation from a directly recovered camera pose. Future frames serve only as supervision for the LWM, whose inputs are restricted to the observed prefix, and target answers are excluded from QGMI memory construction. The source-length correction, the NULL entry, and the gated injection are also reasonable design choices for integrating heterogeneous memory sources.

  1. The paper presents failure cases and acknowledges its main limitations.

The authors show failures in spatio-temporal consistency and motion-direction recognition, point out the remaining distance error in the camera-transformation example, and discuss fine-grained geometric reasoning as well as the fixed memory capacity and prediction horizon in the limitations section. The appendix also proactively identifies an annotation error in a navigation QA item. This reflects a careful analysis of the results and helps readers understand the method's scope.

Critical weaknesses
  1. The provenance of some values in Figures 3 and 4 needs to be verified

The embedded figure files for the left panel of Figure 3 and both panels of Figure 4 contain annotation text outside the cropped region. This text is not visible on the rendered page, but it is present in the PDF text layer and can be located by searching the PDF for "illustrative" (one occurrence on p. 7 and two on p. 8; it can also be extracted with tools such as pdftotext):

Figure 3 (left): "SIS-Motion: supplied result. Other models: illustrative estimates. Y-axis truncated (85–96)." Figure 4 (both panels): "EnvSelf-UAV-4B. Filled: reported at K = 16." (H = 6 in the right panel) and "Open markers: illustrative estimates."

These notes are consistent with the visible marker styles: in Figure 4, only K = 16 and H = 6 use filled markers, while the remaining 16 data points use open markers. Read literally, all bars in Figure 3 other than SIS-Motion (including EnvSelf-UAV's 94.5) are labeled as estimates.

I cannot conclude from this alone that these values were not obtained from actual runs; the notes may be leftovers from an early draft. However, if the notes reflect what actually happened, then the OpenUAV-QA results for GPT-5.4 and Gemini-3-Flash, as well as all points in Figure 4 other than K = 16 and H = 6, are not measured values, and the trend analysis in §4.4 (e.g., the slight decline at K = 64 and the peak at H = 6) and the choice of K and H rest precisely on these points. This would no longer be a presentation issue; it would directly affect the reliability of the corresponding conclusions. I ask the authors to state whether each value in both figures comes from an actual run and to provide the corresponding evaluation logs. If any placeholder or estimated values are present, please replace them with measured results and revise §4.4 and the conclusions drawn from Figure 3 accordingly.

  1. The training configurations and memory-integration pathways of the ablation variants are not specified

Appendix B.3 describes the two-stage scheme (AerialVG image pretraining followed by SIS-Motion-54K video adaptation), but it does not state, row by row, whether the variants in Table 3 share the same pretraining data, initialization, and training budget. Section 4.3 explicitly mentions SIS-Motion-54K for the SFT baseline but does not say whether it also received the same AerialVG pretraining. If SFT did not receive image pretraining while the subsequent variants did, the improvement from 66.9% to 71.6% would conflate the contributions of additional data and architectural changes. My concern is that the protocol is under-specified; this alone does not establish that the comparison is unfair.

Similarly, the authors explain the conceptual difference between image-only training and temporal adaptation, but the concrete experimental setup remains unclear, for example whether the memory parameters continue to be updated during the video stage, whether the memory is re-initialized at every frame, and whether cross-frame updates are disabled. Moreover, the +DESM and +EMDM rows in Table 3 do not yet include QGMI, and the paper does not explain how the environmental memory, motion representation, and predicted representation are fed into the decoder in these variants. This directly affects how the per-row increments should be interpreted, particularly the 0.3-point gain attributed to QGMI.

Table 3 reports only 4 of the 13 SIS-Bench subtasks. Among those reported, adding DESM lowers path planning from 35.2 (SFT) to 29.6 and 30.4, below both SFT and the backbone (36.3), so the "consistent aggregate improvements" claimed in the text hold only at the level of the overall score. I suggest reporting the full per-subtask ablation in the appendix.

  1. The specific value of the core coupling design needs more targeted empirical support

The methodological contribution lies mainly in the coupling among regional memory, geometric motion representation, and future prediction. The individual building blocks have clear precedents; for example, MOTR[1] propagates and updates object queries frame by frame, and Flamingo[2] injects visual information into a language model through gated cross-attention. A more convincing argument would therefore show what additional benefit the coupling mechanism designed for UAV tasks provides.

The progressive ablations support the downstream utility of each component, but they do not yet distinguish among the following explanations:

Whether EMDM's gains come mainly from the geometric-consistency branch. The authors provide a reasonable robust matching and fallback mechanism, but they do not report the number of valid matches, the success rate of the affine fit, or the fallback rate, and there is no comparison with a variant that always uses the global-feature fallback. As a result, how often the geometric branch is actually used, and its independent contribution, remain unclear. Whether the LWM's gains come specifically from future prediction and motion conditioning. The authors define the prediction target and provide an ablation with and without the LWM, but they report no prediction error and no comparison with copying the current environment state, removing motion conditioning, or a capacity-matched feature transformation. It is therefore difficult to separate the contributions of predictive ability, additional representational capacity, and the auxiliary training objective. Table 3 also omits Action Prediction, the subtask most directly related to prediction.

These requests do not require EMDM to output metric 6-DoF poses, nor do they require the LWM to perform candidate action planning. What would be more valuable is to add controls, under the existing representation-learning objectives, that directly test the roles of geometric consistency and motion conditioning.

[1]. Zeng F, Dong B, Zhang Y, et al. Motr: End-to-end multiple-object tracking with transformer[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 659-675.

[2]. Alayrac J B, Donahue J, Luc P, et al. Flamingo: a visual language model for few-shot learning[J]. Advances in neural information processing systems, 2022, 35: 23716-23736.

  1. Camera transformation falls below the backbone at both scales, without a specific diagnosis

In Table 2, the Camera Transformation scores are 6.9 for the 4B backbone vs. 2.6 for EnvSelf-UAV-4B, and 6.2 for the 9B backbone vs. 1.8 for EnvSelf-UAV-9B. The authors acknowledge the limited fine-grained geometric reasoning ability, which is appreciated. However, the full system falls below its own backbone at both scales, while the text only states that "camera transformation remains weak" without noting this degradation; moreover, the camera-transformation example chosen for Figure 7 is a relatively favorable case.

The current results only show that the full training recipe degrades performance on this task; they do not allow the degradation to be attributed to EMDM. Because SFT and per-module results for this task are missing, it is impossible to tell whether this reflects a capability trade-off introduced by task adaptation, the effect of a particular component, or evaluation factors such as output format. Given the paper's emphasis on spatial and motion understanding, such a diagnosis would help clarify the method's benefits and limits.

  1. The relationship between hyperparameter selection and the zero-shot evaluation needs clarification

Section 4.4 states that K = 16 "provides performance close to the peak on both benchmarks" and selects H = 6 after noting that "Both curves peak at H = 6," which indicates that the SpatialUAV curves were used in parameter selection. The manuscript does not state whether these curves were computed on a held-out validation set or on the final evaluation instances.

This in itself is not evidence of data leakage, nor does it invalidate the authors' definition of zero-shot. However, since cross-task transfer is a key claim of the paper, the authors should state when K and H were fixed and on which data split, so that readers can judge whether the final evaluation was affected by model selection on the target test set. If the authors provide measured curves in response to Weakness 1, I also recommend performing the parameter selection on validation data that excludes SpatialUAV test instances.

  1. The statistical stability of small component gains is not reported

Appendix B.3 discloses the random seed (42) and the main training configuration, but Table 3 provides no run-to-run variation, confidence intervals, or paired uncertainty estimates. For example, QGMI's increment on SIS-Bench is from 76.6% to 76.9%. With 4,856 questions and an accuracy of about 77%, the binomial standard error of a single accuracy estimate is roughly 0.6 points, so a 0.3-point increment is well below this magnitude.

This does not mean the gain is insignificant, nor does it imply that the authors ran their experiments only once. More complete reporting could include results from repeated training runs or an uncertainty analysis based on paired predictions. Since multiple questions in SIS-Bench share the same video, resampling at the video level would better match the data structure.

  1. Some implementation and evaluation details need to be supplied

For the different input types in SpatialUAV, the paper defines the memory construction for single images and states that the benchmark's original input formats are retained, but it does not specify how EMDM, the LWM, and each memory source are enabled for single-image and independent multi-view inputs. Video training samples 4–16 frames at 2 fps, whereas evaluation is only described as using 8–32 frames, so the sampling interval at evaluation and the actual temporal meaning of the prediction horizon H remain unclear. The evaluation sequences are also longer than those seen in training, so both the memory recurrence and the number of motion tokens (up to 31) involve length extrapolation.

The QGMI formulation includes source embeddings but specifies no temporal or prediction-step encoding. Since DESM's accumulated state and the LWM rollout are produced by ordered recursion, and the raw observations are fed directly into the decoder, this alone does not imply that the model lacks temporal information. However, the motion tokens are computed for each pair of adjacent frames and carry no time index of their own, while Action Sequence and Action Recall depend precisely on their order. Please clarify whether the module relies entirely on token content to represent time or uses an additional, unstated temporal encoding.

OpenUAV-QA is defined in the appendix as offline multiple-choice navigation QA, and the results should be evaluated within that scope; I do not ask the authors to add closed-loop navigation or real-robot experiments. However, Figure 3 does not state which model scale is used and lacks the corresponding backbone and SFT comparisons. A sample record left outside the cropped region of the right panel of Figure 3 shows that the example is TravelUAV_Train_13187 (split: unseen_map), with the correct option being "about 48 m, turn left about 36°, about 34 m," whereas the figure is annotated with 50 m, 40°, and 35 m, suggesting that the text in the figure is not the model's actual output. The authors should also explain how the mislabeled sample identified in Appendix C.2 is handled in the final evaluation.

Questions for authors
  1. Does every value in both figures come from an actual run? Please provide the corresponding evaluation logs. Where does SIS-Motion's 92.2 come from, and was it evaluated under the same protocol as the other models? If any values are placeholders or estimates, please provide measured results and state whether the conclusions in §4.4 change. Please also clarify which numbers the "statistical aggregation of benchmark scores" mentioned in the AI use statement refers to, and how they were verified.

2.Do the variants in Table 3 share the same AerialVG pretraining, initialization, and training budget? Did SFT receive the same image pretraining? For the image-only variant, how are parameters updated in the video stage, and how is cross-frame memory used? In the variants without QGMI, how are the memory and motion representations fed into the decoder? Could you report the full per-subtask ablation in the appendix?

  1. Geometric-consistency branch: At test time, what fractions of cases use a valid affine fit versus the fallback? How much does the geometric-consistency branch gain over a variant that always uses the global-feature fallback? Could a controlled experiment with independently moving objects or strong parallax further support the role of this branch in ego-motion understanding? Future prediction mechanism: Is the LWM's multi-step feature prediction error lower than that of copying the current environment state? How much of the gain on the relevant tasks remains when motion conditioning is removed or shuffled? Could you add component-level results for the Action Prediction subtask?

  2. Could you provide SFT and per-module results on this task to explain why the full method falls below the corresponding backbone at both scales?

  3. On what data were K and H determined, and were they fixed before the SpatialUAV test results were examined? Did SpatialUAV results play any role in model selection?

  4. For small component increments such as that of QGMI, could you provide results from repeated runs or video-level paired uncertainty estimates?

  5. Which modules and memory sources are enabled for single-image and independent multi-view inputs? What is the sampling interval in video evaluation? Does QGMI use an explicit temporal or prediction-step encoding, or does it rely on implicit temporal information in the recurrent states?

  6. Which model scale do the Figure 3 results correspond to, and what is the SFT score with the same backbone? Which data split does the example in the right panel come from, and is the text in the figure the model's actual output? Is the mislabeled instance identified in Appendix C.2 included in the final evaluation, and was the same label-handling rule applied to all models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.