acceptodds
Under review as a conference paper at ICLR 2027

Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation

Abstract

Vision-Language Models have achieved strong progress in ground-view visual understanding, yet they remain brittle in high-altitude Unmanned Aerial Vehicle scenes, where objects are tiny and densely packed, textures are repetitive, and top-down orientations are ambiguous. We introduce UAVReason, a large-scale UAV-native dataset and evaluation suite for studying unified aerial reasoning and generation under this nadir-view domain shift. UAVReason aligns RGB imagery, depth maps, semantic segmentation masks, captions, and question-answer pairs within a consistent aerial domain. It contains 23.6K captioned frames, 273K VQA pairs including 68.2K two-frame temporal questions, and 188.8K cross-modal generation samples across RGB, depth, and segmentation modalities. We further adapt UAVReason-Bagel as a unified understanding-and-generation baseline that jointly optimizes language reasoning and dense visual generation objectives. Experiments show that general-purpose VLMs and off-the-shelf unified generators struggle with UAV-native grounding, while UAVReason-Bagel substantially improves over its pretrained counterpart, increasing VQA-1F F1 from 0.394 to 0.711, VQA-2F F1 from 0.427 to 0.822, and heading-aware VQA F1 from 0.798 to 0.973. For generation, it improves segmentation mIoU to 0.143 and reduces KID from 0.078 to 0.048 for depth-segmentation-text-conditioned RGB synthesis. More importantly, our ablations reveal a bidirectional synergy between synthesis and reasoning. Dense generation objectives improve temporal semantic consistency, while language-level reasoning regularizes sparse-condition image synthesis. These results suggest that unified reasoning and generation provide effective geometry-aware structural priors for physically grounded aerial intelligence. All data, code, and evaluation tools will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
UAV, spatial reasoning, temporal reasoning, VQA, image generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LwqzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UAVReason, built on UAVScenes, which brings together RGB imagery, depth, semantic segmentation, single- and two-frame question answering, and conditional generation for overhead UAV images. It contains approximately 23,600 frames, 273,000 question–answer pairs, and 188,800 generation examples. The authors also develop a BAGEL-based baseline for joint understanding and generation. Through comparisons with pretrained models, single-task training, and other models, they examine adaptation to aerial imagery and the interaction between understanding and generation. The main contributions are the dataset and the unified evaluation setup.

Critical strengths
  1. The unified dataset provides a useful setting for studying understanding and generation together. The same real aerial images come with language, depth, and segmentation supervision, allowing these tasks to be studied under consistent data conditions. Deriving numerical answers through explicit geometric calculations and splitting the data by complete flight sequences are both sensible choices. They help reduce errors in numerical answers and data leakage between neighboring frames, respectively.

  2. The experiments show clear gains from domain adaptation and some benefits from combining tasks. Compared with pretrained BAGEL, two-frame QA F1 increases from 0.427 to 0.822, and segmentation mIoU rises from 0.033 to 0.143. Adding understanding tasks to generation-only training also reduces KID for RGB generation conditioned on depth and text from 0.052 to 0.036. These results show that the dataset supports adaptation to aerial imagery and that language supervision can help image generation in this particular setting.

Critical weaknesses
  1. Joint training does not consistently improve reasoning, and the performance drops need a better explanation. In Table 2, compared with reasoning-only training, joint training improves the LLM-J score only for standard two-frame QA, from 0.571 to 0.600. The score drops from 0.454 to 0.397 for single-frame QA and from 0.770 to 0.657 for two-frame QA with heading information. F1 also decreases across all three QA settings. Since the declines also appear in the semantic scores, limitations of lexical-overlap metrics cannot be the main explanation. The results show benefits on specific tasks alongside performance trade-offs, and the broader claim that understanding and generation benefit each other needs to be qualified.

  2. Some aspects of the geometric and temporal evaluation need clarification. RGB→Depth is explicitly included in the benchmark, but neither the main paper nor the appendix reports quantitative depth prediction results. Appendix B also defines tolerance-based accuracy for numerical questions, yet the corresponding results are not reported.

Questions for authors
  1. In Table 2, joint training improves the semantic score for standard two-frame QA but lowers it for single-frame QA and two-frame QA with heading information. Are the gains concentrated in particular types of temporal questions? How do the authors explain this difference?

  2. What are the correct question, reference answer, and model prediction for the right-hand panel of Figure 6? Is the discrepancy limited to the figure, or does it also affect the evaluation data and scoring?

  3. What are the quantitative results for RGB→Depth and the tolerance-based accuracy defined in Appendix B? Do they show that joint training improves the prediction of geometric quantities such as distance?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer Lwqz

We appreciate the detailed questions on joint training, geometric evaluation, and Figure 6. Below, we explain the original ablations and report the completed depth, numerical, and temporal-category evaluations.

Q1: The central claim of "bidirectional synergy" remains insufficiently supported.

A1: Thank you for highlighting that lexical-overlap metrics cannot explain the reasoning declines. Table 2 shows mixed reasoning outcomes: joint training raises standard two-frame LLM-J from 0.571 to 0.600, while single-frame LLM-J decreases from 0.454 to 0.397 and heading-aware LLM-J from 0.770 to 0.657. F1 also declines across all three QA settings. We add the supplementary paired comparison and its confidence interval, which includes zero, to Appendix F.6 (lines 1378–1393).

 

Understanding supervision provides broader benefits to conditional RGB synthesis: Table 3 shows higher CLIP in all three settings and lower KID in two, including depth-and-text KID from 0.052 to 0.036. DINO results vary, and segmentation is unchanged. Generation-heavy sampling and loss weighting (Section 4) may contribute to competition with reasoning objectives. This explanation remains a hypothesis. We revise the cross-task claim to asymmetric cross-task transfer in the abstract (lines 27–31), the contribution statement in Section 1 (lines 104–107), the conclusion in Section 6 (lines 487–491), and the discussions of Tables 2 and 3 in Sections 5.1–5.2 (lines 455–461 and 470–479). This emphasizes the synthesis benefits and reasoning trade-offs. Q3 and Q2 provide the supplementary capability analyses.

Q2: 1. Missing quantitative RGB-to-Depth evaluation.

2. Missing tolerance-based accuracy results for numerical VQA.

A2: Thank you for requesting these quantitative evaluations. We report supplementary experiments for RGB-to-depth prediction and tolerance-based numerical VQA, using fixed evaluation samples and common scoring procedures for both models.

 

1. RGB-to-depth. We evaluate 200 test frames, with 50 from each test sequence, against synchronized LiDAR axial depth on finite, positive reference pixels. We decode all models’ predictions using a fixed 120 m scale, corresponding to the verified near-white, far-black target encoding of the UAVReason-BAGEL checkpoint:

where ĝ is the predicted grayscale value and d̂ is depth in metres. No scale is fitted to the evaluation references.

ModelAbsRel ↓RMSE (m) ↓δ₁ ↑
Pretrained BAGEL0.482343.160.2643
UAVReason-BAGEL0.443737.580.2807

Under this common decoding protocol, UAVReason-BAGEL achieves lower AbsRel and RMSE and higher δ₁ than pretrained BAGEL, providing quantitative evidence of improved depth prediction following aerial-domain adaptation.

 

2. Numerical VQA. Source checking retains 34 eligible scalar questions from 80 candidates, excluding 36 identity/color questions and 10 target/reference cases under a common eligibility rule. Both models use the same answer extractor, and unparseable responses count as incorrect. We instantiate the tolerance criterion in Appendix B as

We report the relative-only setting (ε = 0) on all 34 questions and on the 30 questions with length references in metres. On the same 30 length questions, we additionally report a 0.5 m absolute-tolerance floor. Brackets give nominal item-level 95% Wilson intervals.

Model5%: all 345%: 30 length0.5 m floor: 30 length
Pretrained BAGEL5/34 [6.4–30.1%]4/30 [5.3–29.7%]4/30 [5.3–29.7%]
UAVReason-BAGEL13/34 [23.9–55.0%]12/30 [24.6–57.7%]19/30 [45.5–78.1%]

Numerical accuracy increases from 5/34 to 13/34 under the relative-only criterion and from 4/30 to 19/30 with the 0.5 m tolerance floor. Under the relative-only criterion, pretrained BAGEL and UAVReason-BAGEL answer 4/17 and 13/17 zero-reference questions correctly, respectively. Their nonzero-reference counts are 1/17 and 0/17. With the 0.5 m floor, the corresponding length-question counts are 4/30 and 19/30, including 0/15 and 5/15 with nonzero references. This separates zero-reference recognition from nonzero magnitude estimation.

 

Together, these supplementary evaluations quantify improvements over pretrained BAGEL in depth prediction and numerical VQA. The joint-versus-single-task evidence is discussed separately in Q1. We revise Appendix B to define both numerical tolerance settings explicitly (lines 749–753). We add the depth protocol, encoding, decoding, and results to the “RGB-to-depth” part of Appendix F.4 (Eq. 2 and Table 12, lines 1279–1301), and the numerical results and zero/nonzero-reference analysis to Appendix F.5 (Eq. 3 and Table 14, lines 1332–1357).

AuthorsOpenReview ↗

Responses to Reviewer Lwqz

Q3: Are the gains concentrated in particular types of temporal questions? How do the authors explain this difference?

A3: Thank you for asking about the temporal reasoning gains and the performance differences across tasks. We add a paired category-level comparison of UAVReason-BAGEL and its reasoning-only variant on the same 40 standard two-frame questions, using the supplementary Qwen3-VL-8B judge. The table reports UAVReason-BAGEL’s semantic-correct counts and nominal item-level 95% Wilson intervals. The reasoning-only counts are given below. We add the category-level comparison to Appendix F.6 (Table 15, lines 1362–1376), followed by the paired comparison and statistical uncertainty analysis (lines 1378–1393).

Temporal categorySemantic correct / total95% Wilson interval
Identity8/956.5–98.0%
Distance change7/945.3–93.7%
Relation change7/1232.0–80.7%
Approach or separation5/1023.7–76.3%
Total27/4052.0–79.9%

On these same questions, the reasoning-only variant scores 7/9 for identity, 6/9 for distance change, 6/12 for relation change, and 4/10 for approach or separation. UAVReason-BAGEL therefore has one additional semantically correct answer in each category. On this 40-question subset, the four-answer difference is spread across categories rather than concentrated in one. At the task level, the original ablations show the asymmetric pattern discussed in Q1 (A1): a higher standard two-frame semantic score coexists with declines on other reasoning metrics. The generation-heavy allocation of supervision may contribute to these trade-offs.

Q4: Figure 6 inconsistency and its impact on evaluation reliability.

A4: Thank you for identifying this inconsistency. We apologize for the confusing presentation. In the original version of Figure 6, the titles of the middle and right panels are inadvertently swapped, and the question from the middle panel is mistakenly duplicated in the right panel. The middle panel corresponds to heading-aware two-frame VQA, while the right panel illustrates standard two-frame VQA. We correct the middle/right panel titles, the duplicated right-panel question, and the inconsistent reference-answer wording in Figure 6 on page 17. The corrected right panel asks whether the white sedan approaches or separates from the traffic barrier. The reference answer indicates separation, with a distance increase of 0.631 m, while our model predicts the correct separation trend with an estimated increase of 0.461 m. We verify that these errors are confined to figure preparation and do not affect the underlying evaluation data, scoring, or reported quantitative results.

Q5: Missing geometric evaluation results and evidence of joint training benefits.

A5: For dense geometric prediction, the original single-task comparison shows equal segmentation mIoU for joint and generation-only training (0.143, Table 3). The supplementary depth and numerical results in Q2 (A2) improve over pretrained BAGEL, measuring aerial-domain adaptation. They do not isolate a joint-versus-single-task effect. The supported cross-task benefit is in conditional RGB synthesis, as detailed in Q1 (A1). We add the depth results and protocol to Appendix F.4 (Eq. 2 and Table 12, lines 1279–1301), the segmentation references to the same appendix (Table 13, lines 1294–1327), and the numerical VQA results and protocol to Appendix F.5 (Eq. 3 and Table 14, lines 1332–1357).

Reviewer_zjj5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UAVReason, a benchmark for evaluating visual reasoning and image generation from UAV imagery. Built upon UAVScenes, the benchmark combines RGB images, depth maps, semantic segmentation masks, captions, and question-answer pairs to support spatial reasoning, temporal reasoning, and cross-modal image generation. The authors also adapt the existing BAGEL framework to the UAV domain, jointly training language reasoning and visual generation tasks to investigate whether these capabilities can benefit each other. Overall, the paper provides a useful benchmark and demonstrates the benefits of domain-specific training for aerial scene understanding and generation. However, further evidence is needed to support the claimed mutual benefits between reasoning and generation, as well as the reliability of the automatically generated annotations and the generalization of the model to unseen environments.

Critical strengths
  1. The paper addresses an important research problem in UAV visual understanding. It specifically focuses on challenges associated with aerial perspectives, which are difficult for existing general-purpose vision-language models.

  2. The proposed UAVReason benchmark integrates visual reasoning and cross-modal generation within a unified evaluation framework. It supports multiple tasks using aligned RGB images, depth maps, segmentation masks, and language annotations, providing a useful resource for studying aerial scene understanding.

  3. The dataset construction includes a geometry-based annotation pipeline that combines depth information, 3D object representations, and programmatic answer generation. This approach provides structured supervision for spatial and temporal reasoning and reduces reliance on manually written answers.

Critical weaknesses
  1. One of the paper's central claims is that reasoning and generation can mutually benefit each other. However, Table 2 shows that the reasoning-only model achieves higher F1 scores across all three VQA settings and outperforms the joint model on several other metrics. Although joint training improves the LLM-based evaluation score for two-frame VQA, the overall results are mixed. These findings do not fully support the claimed mutual benefits, and it remains unclear under what conditions joint training is more effective than training the two tasks separately.

  2. UAVReason-Bagel primarily adapts the existing BAGEL architecture to aerial imagery using established reasoning and generation objectives. Although the unified benchmark and domain-specific training represent useful contributions, the methodological novelty appears limited. In particular, it is unclear what new technical capabilities the proposed approach introduces beyond applying an existing unified architecture to a UAV-specific dataset.

  3. The dataset is constructed from images collected across 20 maps, but the separation between training and testing data is not sufficiently clear. In particular, it is unclear whether the split is based on maps, flight trajectories, or individual frames. If closely related frames or scenes appear in both sets, the reported performance may not accurately reflect generalization to unseen environments. This raises concerns about the reliability of the evaluation results for new UAV scenes.

  4. The benchmark relies on an automated pipeline to generate object annotations, spatial relationships, and question-answer pairs. Although the answers are computed using geometric information, their accuracy still depends on the quality of the underlying segmentation, object identification, and instance refinement. Errors in these stages may propagate into the generated annotations, particularly for small or closely spaced objects. The lack of sufficient quantitative validation makes it difficult to assess the reliability of the benchmark.

  5. Although UAVReason-Bagel improves segmentation accuracy over the pretrained model, its absolute performance remains relatively low, with an mIoU of 0.143 in the main experiments and 0.194 in the resolution ablation. These results suggest that the model still has difficulty capturing detailed semantic and spatial information from aerial images. Without comparisons against established UAV segmentation methods, it is difficult to assess how competitive the proposed approach is for dense visual understanding.

Questions for authors
  1. Table 2 shows that the reasoning-only model outperforms the joint model on several key reasoning metrics. Could the authors explain these results and clarify what evidence supports the claimed mutual benefits between reasoning and generation?

  2. How are the training and testing data separated across the 20 UAV maps and flight trajectories? Are there any measures taken to prevent closely related frames or scenes from appearing in both sets?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer zjj5

We appreciate the reviewer’s recognition of the aerial task suite. We clarify its contribution and evaluation scope, and separate original results from the new comparisons.

Q1: The central claim of "bidirectional synergy" remains insufficiently supported.

A1: Thank you for asking us to reconcile the claim with the single-task comparisons. The ablations show asymmetric cross-task transfer. Understanding supervision benefits conditional RGB synthesis: Table 3 reports higher CLIP scores in all three conditioning settings and lower KID in two, with one tie. Depth-and-text KID decreases from 0.052 to 0.036. DINO results depend on conditioning, and segmentation mIoU remains unchanged.

 

For reasoning, Table 2 reports a higher standard two-frame LLM-J score under joint training (0.600 versus 0.571), while reasoning-only training leads VQA F1, caption CIDEr, and single-frame and heading-aware LLM-J. We retain the 0.029 increase as a descriptive result. We add a supplementary paired comparison to Appendix F.6 (lines 1378–1393), reporting a confidence interval that includes zero. We revise the cross-task claim to asymmetric cross-task transfer in the abstract (lines 27–31), the contribution statement in Section 1 (lines 104–107), the conclusion in Section 6 (lines 487–491), and the discussions of Tables 2 and 3 in Sections 5.1–5.2 (lines 455–461 and 470–479). The empirical contribution is to identify where shared supervision helps and where it introduces trade-offs. Q6 discusses a possible explanation.

Q2: The methodological novelty appears limited.

A2: The framework adds concrete evaluation capabilities to the source data: metric spatial questions with references computed from 3D geometry, temporal questions tied to object identities across frames, and cross-modal prediction and synthesis on the same aerial observations. Section 3.1 describes the geometric construction, and Section 3.2 defines the aligned reasoning and generation tasks. UAVScenes supplies the sensing resources and semantic annotations, and BAGEL supplies the unified architecture. UAVReason connects these resources through geometry-grounded language tasks and a common evaluation protocol. UAVReason-BAGEL in Section 4 provides the reference implementation. The task-specific ablations in Q1 and the segmentation comparison in Q5 demonstrate how the benchmark measures cross-task transfer and dense grounding.

Q3: The separation between training and testing data is not sufficiently clear.

A3: Thank you for requesting a precise split definition. We clarify in Section 3.2 (lines 255–260) that the data comprise 20 flight sequences, rather than 20 geographically distinct maps, and specify the sequence-level split. The reported split contains 19,901 training frames and 3,699 test frames from 16 training sequences and four test sequences, AMvalley03, HKairport01, HKairport02, and HKisland03. Our manifest check confirms these memberships.

 

UAVScenes records repeated flights over four geographic regions, including runs at different cruising speeds (Wang et al., ICCV 2025, Table S8). Retaining continuous sequences supports temporal questions and paired geometric supervision. We separate complete flights, while nearby views can occur across repeated visits. On the available audit manifest, file-byte and decoded-RGB checks find no exact cross-split duplicates. 3,694/3,699 test frames lie within 5 m horizontally of a training frame, using time-aligned GNSS positions. We clarify the held-out-flight setting within shared regions in Section 3.2 (lines 258–260), and add sequence membership and the duplicate/proximity checks to Appendix F.2 (Table 10, lines 1237–1255).

Split or auditObserved result
Training / test sequences16 / 4
Exact cross-split duplicates (file-byte and decoded-RGB checks)0
Test camera positions within 5 m horizontally of training3,694 / 3,699
AuthorsOpenReview ↗

Responses to Reviewer zjj5

Q4: The lack of sufficient quantitative validation makes it difficult to assess the reliability of the benchmark.

A4: We thank the reviewer for highlighting how errors in object identification and instance refinement can propagate into spatial and temporal annotations. To address this concern, we extend the verification in Section 3.1 with quantitative geometric checks and an independent human audit of object grounding and reference correctness.

 

1. Geometric and numerical consistency. We check the arithmetic of stored distance changes and compare reference depth with synchronized LiDAR.

ValidationScopeResult
Distance-change arithmetic41,494 recordsAll consistent with stored endpoints
Reference depth vs. synchronized LiDAR200 framesMedian per-frame relative discrepancy: 0.877%

These checks assess the numerical stage of annotation generation. To examine whether those quantities and descriptions are associated with the correct visual objects, we additionally conduct the following human audit.

 

2. Object grounding and reference correctness. Three assessors who are not involved in dataset construction independently review 100 fixed examples, with 25 from each of the four language tasks. Model identities and LLM judge scores are concealed. Object correspondence is assessed on the 75 VQA examples, and reference correctness is assessed on all 100 examples, including captions.

AssessmentPassErrorUnverifiable
Object correspondence (75 examples)78.2%11.6%10.2%
Reference correctness (100 examples)69.0%20.0%11.0%

Percentages aggregate the three assessors’ ratings, corresponding to 225 object-correspondence judgments and 300 reference-correctness judgments. “Error” denotes a negative assessment. “Unverifiable” denotes insufficient evidence to decide. The object-correspondence assessment directly examines the instance-association stage raised by the reviewer, while reference verification examines the resulting answers and descriptions. Item-level judgments identify potential object-association errors, reference inconsistencies, and evidence-limited cases for checking against source geometry, instance records, and reference annotations.

 

Together, these evaluations examine both the numerical consistency of generated annotations and their grounding in the intended objects. We add the audit protocol and aggregate results to Appendix F.1 (Tables 8–9, lines 1192–1225), the geometric checks and source-review procedure to the same appendix (lines 1227–1233), and a summary under “Human Quality Audit” in Section 3.1 (lines 211–215).

Q5: Low segmentation mIoU and lack of comparisons with established UAV segmentation methods.

A5: UAVReason-BAGEL improves segmentation mIoU from 0.033 to 0.143 over pretrained BAGEL while supporting reasoning, dense prediction, and conditional synthesis in one model. We add established segmentation references on 400 fixed test frames with a common 256 × 256 scoring grid.

 

For shared road/car transfer, UAVReason-BAGEL obtains 0.1011 mIoU, close to Cityscapes-trained SegFormer-B0 (0.1016) and above UAVid-trained BiMSA and PIDNet-S/M (0.0228–0.0292). These complete external checkpoints receive no UAVReason adaptation. Our model is UAVReason-adapted. After adaptation on UAVReason, SegFormer-B0 and UNetFormer obtain 18-class mIoU of 0.4970 and 0.4554, versus 0.1506 for UAVReason-BAGEL. This supplies the requested specialist reference alongside the unified model’s broader task coverage.

 

These comparisons complement the original adaptation gains with specialist reference points for evaluating unified aerial models. We add the segmentation results and protocols to the “RGB-to-segmentation” part of Appendix F.4 (Table 13, lines 1294–1327), and update Section 5.2 to identify these shared-class transfer and target-domain specialist comparisons (lines 477–479). Specialized segmentation models achieve higher 18-class mIoU, highlighting the remaining gap in dense prediction. Our contribution is a unified multitask benchmark and baseline rather than a segmentation-specific state-of-the-art model.

AuthorsOpenReview ↗

Responses to Reviewer zjj5

Q6: Explain Table 2 results and clarify what evidence supports the claimed mutual benefits between reasoning and generation?

A6: Thank you. Q1 (A1) summarizes the asymmetric transfer and supporting results. Section 4 uses 2:1 generation-to-reasoning sampling and loss weighting. This allocation may contribute to competition with language objectives. We add it as a possible explanation in the Section 5.1 discussion of Table 2 (lines 459–461), because its causal effect is not isolated.

Q7: How are training and test sequences separated? How is cross-split scene overlap prevented?

A7: As clarified in Q3 (A3), we split UAVReason by complete flight sequences (16 training and 4 testing), rather than individual frames. File-byte and decoded-RGB checks find no exact cross-split duplicates. Since UAVScenes contains repeated flights over shared regions, nearby viewpoints may occur across splits. We therefore define the evaluation as held-out-flight generalization rather than geographically disjoint generalization. The split details and overlap checks are provided in Section 3.2 (lines 258–260) and Appendix F.2 (Table 10, lines 1237–1255).

AuthorsOpenReview ↗

Responses to Reviewer zjj5

Follow-up on Q4: The lack of sufficient quantitative validation makes it difficult to assess the reliability of the benchmark.

A4: We evaluated how excluding or repairing flagged items affected model scores across all four tasks. We checked all 100 examples from the original audit and 60 additional examples, for a total of 40 per task. We compared each reference with the images and the available masks, bounding boxes, or endpoint poses. The evidence supported 134 of the 160 original references. We flagged 24 items with specific reference issues and two for which the evidence was inconclusive. The source evidence alone determined which items were flagged, independently of model predictions and scores.

 

1. Protocol A — Strict exclusion

We excluded all 26 flagged items and scored the five models using the original questions, references, and predictions for the remaining 134. Each table reports the strongest comparator for each task.

TaskItemsStrongest comparatorComparator scoreUAVReason-BAGELLead
Single-frame F129Pretrained BAGEL0.39940.81530.4159
Two-frame F130Pretrained BAGEL0.44090.83070.3898
Heading F137Pretrained BAGEL0.79870.97490.1762
Caption CIDEr38GPT-6.1 Terra0.52001.78471.2647

Single-frame and two-frame questions accounted for 21 of the 26 flagged items. With all 21 excluded, UAVReason-BAGEL led pretrained BAGEL by 0.4159 and 0.3898 F1 on these tasks, respectively. These margins are independent of how the flagged items were repaired.

 

2. Protocol B — Full repair

We retained all 160 items and applied all 26 documented revisions. Fifteen changed only the reference, nine revised or replaced both the question and its reference, and two converted visual heading questions into endpoint-pose diagnostics. After editing, we checked each revised item against its source evidence. Using the same model checkpoints, we regenerated predictions for the 11 items whose inputs changed and reused cached predictions for the other 149.

TaskItemsStrongest comparatorComparator scoreUAVReason-BAGELLead
Single-frame F140Pretrained BAGEL0.39810.68960.2915
Two-frame F140Pretrained BAGEL0.42800.69740.2694
Heading F1, image-only38Pretrained BAGEL0.77770.94920.1715
Caption CIDEr40GPT-6.1 Terra0.51341.81401.3006

The heading row covers the 38 image-only items. The other two involved displacements too small for reliable visual direction judgments. Including these two endpoint-pose diagnostics gave 0.9031 F1 for UAVReason-BAGEL and 0.7420 for pretrained BAGEL across all 40 heading items.

 

After repair, UAVReason-BAGEL’s single-frame F1 fell from 0.7797 to 0.6896 and its two-frame F1 from 0.7976 to 0.6974. Both caption repairs corrected location errors. For example, one reference placed a boat and its wake in the upper left of the image, and the corrected reference placed them in the upper right. Across all 40 captions, UAVReason-BAGEL’s CIDEr fell from 1.8373 to 1.8140. After repair, UAVReason-BAGEL led the strongest comparator by 0.2915 F1 on single-frame questions and 0.2694 F1 on two-frame questions. It ranked first among the five models in all four task comparisons under each protocol.

Reviewer_azMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UAVReason, a unified benchmark for spatio-temporal reasoning and cross-modal generation in UAV scenes. Built upon UAVScenes, the benchmark contains 23.6K frames, 273K VQA pairs, and 188.8K cross-modal generation samples, covering diverse understanding and generation tasks. Based on this benchmark, the authors further develop UAVReason-Bagel by jointly fine-tuning a pretrained BAGEL model on reasoning and generation objectives. Experiments demonstrate substantial improvements over the pretrained model and several existing baselines, suggesting the potential benefits of domain-specific supervision for aerial scene understanding and generation.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem. UAV imagery introduces unique challenges for vision-language models, including small objects, dense scenes, and ambiguous spatial orientations.

  2. The proposed benchmark integrates diverse tasks, including single-frame reasoning, temporal reasoning, dense perception, and conditional image generation. Such a unified benchmark could provide a useful testbed for studying multimodal models in aerial scenarios.

Critical weaknesses
  1. Lack of human evaluation for benchmark reliability. The benchmark relies heavily on automatically generated annotations, including questions, answers, and captions. Although the authors introduce programmatic verification to improve annotation consistency, it remains unclear whether these annotations are sufficiently reliable and aligned with human understanding. In particular, I would like to see a human evaluation covering three aspects: (1) the validity and clarity of generated questions, (2) the correctness of the corresponding answers, and (3) the agreement between LLM-based evaluation and human judgments. Such an evaluation would provide stronger evidence for the reliability of the proposed benchmark.

  2. Limited evaluation of frontier models. The experiments mainly compare against existing open-source models, while stronger frontier models, such as GPT, Claude, and Gemini, are not included. Given the rapid progress of multimodal foundation models, evaluating these models on the reasoning and captioning tasks would provide a more comprehensive understanding of the benchmark's difficulty and help establish stronger reference baselines.

  3. Figure readability could be improved. Several figures are presented as rasterized images with relatively small text and annotations, making some details difficult to see. I suggest using vector graphics for diagrams and textual elements where possible, and improving the readability of the qualitative comparisons.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer azMT

We thank the reviewer for the concrete suggestions. We report the independent human audit, the added frontier-model comparisons, and the revised readability improvements below.

Q1: Lack of human evaluation for benchmark reliability.

A1: We thank the reviewer for specifying the three aspects that human evaluation should address. We report a blinded, task-stratified human audit covering question validity, reference correctness, and agreement with LLM-based evaluation. Three assessors who are not involved in dataset construction independently review the same 100 fixed examples, comprising 25 each from single-frame VQA, standard two-frame VQA, heading-aware VQA, and captioning. They use a common rubric, and heading-aware examples include per-frame orientation cues. For response evaluation, each example includes one anonymized answer, with 50 from pretrained BAGEL and 50 from our UAVReason-BAGEL model. Model identities and LLM judge scores are concealed.

 

1. Question validity and reference correctness. The audit assesses question validity and clarity, reference-answer correctness, and whether the questions refer to the intended visual objects.

AssessmentPassErrorUnverifiable
Question validity (75 examples)84.0%13.8%2.2%
Reference correctness (100 examples)69.0%20.0%11.0%
Object correspondence (75 examples)78.2%11.6%10.2%

Percentages aggregate three assessors’ ratings, giving 225 judgments for each VQA-specific criterion and 300 for reference correctness. They summarize human assessments of the audited examples rather than adjudicated dataset-wide accuracy. Negative judgments and cases with insufficient evidence are recorded separately for source verification.

 

2. Agreement between human and LLM evaluation. We compare each assessor’s judgments with those of the supplementary Qwen3-VL-8B judge on identical model responses and supplied references.

CriterionMean human–LLM agreementMean Cohen’s κ
Semantic correctness90.2%0.799
Strict correctness87.5%0.750

Agreement and Cohen’s κ are calculated separately for each assessor–judge pair and then averaged across the three assessors. Semantic agreement ranges from 89.3% to 90.7%. One VQA response is unscorable for one assessor, resulting in denominators of 74, 75, and 75. For the 25 captions, the mean absolute human–judge score difference, averaged across assessors, is 0.61 points on a 1–10 scale. These results support the consistency of the supplementary evaluator with human scoring under the supplied references. Reference correctness is assessed separately in the first table.

 

3. Complementary geometric verification. All 41,494 audited distance changes are arithmetically consistent with their stored endpoints. On 200 frames, reference depth shows a median per-frame relative discrepancy of 0.877% against synchronized LiDAR.

 

The completed audit directly addresses all three requested aspects and adds object-grounding checks to the original programmatic verification. We add the audit protocol and aggregate results to Appendix F.1 (Tables 8–9, lines 1192–1225), the geometric checks and source-review procedure to the same appendix (lines 1227–1233), and a summary under “Human Quality Audit” in Section 3.1 (lines 211–215). Before the public release, we plan to manually review flagged VQA and caption items against the source evidence, correct confirmed annotation errors, and exclude items that remain ambiguous or unverifiable from the released benchmark.

AuthorsOpenReview ↗

Responses to Reviewer azMT

Q2: Limited evaluation of frontier models.

A2: Thank you for recommending stronger reference models. We add Qwen3-VL-32B, gpt-6.1-terra, gemini-3.8-flash, and Claude Sonnet 5.5 on a fixed 160-example supplementary subset, with 40 examples per task. The table compares these models with pretrained BAGEL and UAVReason-BAGEL on that subset. This includes all three requested frontier-model families and extends the benchmark’s reference coverage. We add these results and evaluation settings to Appendix F.3 (Table 11, lines 1259–1275) and a pointer to this comparison in the Section 5.1 discussion (lines 459–461).

ModelSingle-frame F1Two-frame F1Heading F1Caption CIDEr
Pretrained BAGEL0.3770.4510.7960.448
Qwen3-VL-32B0.3260.3690.0900.607
gpt-6.1-terra0.2250.3730.0890.506
gemini-3.8-flash0.2050.3110.2600.421
Claude Sonnet 5.50.2060.2450.3060.350
UAVReason-BAGEL0.7800.8010.9751.837

UAVReason-BAGEL achieves the highest F1 and CIDEr scores on this fixed subset. All added baselines use the same fixed examples, image order, prompts, and F1/CIDEr scoring scripts. VQA F1 is computed directly from normalized answer text and measures lexical overlap with the reference. The added baselines are evaluated zero-shot, while UAVReason-BAGEL is adapted to the benchmark.

Q3: Figure readability could be improved.

A3: Thank you for these presentation suggestions. We review the figures throughout the manuscript and improve their readability and visual presentation, including increasing the font sizes and refining the layouts in Figures 2 and 7. We also check that figure text and annotations are legible at normal PDF viewing scale.

Thanks for the retry. From the response to Q1, it seems that the current version of dataset is still noisy and the error rate is high. Based on this, I tend to remain my original score. I hope the author can improve the quality of the data later.

By the way, the images can be presented with PDF, whose text can be copied and the figures are scalable.

Reviewer_fr48OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UAVReason, a benchmark for jointly evaluating vision-language reasoning and cross-modal generation in nadir-view UAV scenes. Built on UAVScenes, it aligns RGB images, depth maps, segmentation masks, and language annotations, comprising 23,600 image frames, 273,000 QA pairs, and 188,800 generation samples. Its core contributions include a unified dataset and evaluation protocol, a QA construction pipeline incorporating explicit 3D geometric computation, and UAVReason-Bagel, a BAGEL-based baseline jointly trained on understanding and generation tasks.
Evaluation is conducted using a flight-sequence-level split (holding out four complete sequences for testing) and compares the method against general-purpose VLMs, remote-sensing models, and unified generation models. Compared with pretrained BAGEL, single-frame and two-frame VQA F1 scores increase from 0.394/0.427 to 0.711/0.822, respectively; segmentation mIoU rises from 0.033 to 0.143; and the KID for RGB synthesis conditioned on depth, segmentation, and text drops from 0.078 to 0.048. Ablations show that understanding supervision benefits certain conditional generation tasks, while generation supervision improves the semantic judge score for standard two-frame VQA. However, reasoning-only training still outperforms joint training on several understanding metrics. Consequently, the evidence for bidirectional synergy remains task- and metric-dependent, rather than demonstrating universal benefits across all tasks.

Critical strengths
  1. Unified, geometrically grounded UAV benchmark. UAVReason aligns spatio-temporal VQA, depth, segmentation, and conditional generation on real aerial data, deriving verifiable numerical and relational answers directly from geometry. This provides a solid, consistent testbed for cross-task evaluation.

  2. Clear empirical insights into domain adaptation and joint training. Fine-tuning over pretrained BAGEL brings clear gains in QA, segmentation, and conditional generation. Furthermore, single-task ablations reveal concrete cross-task dynamics—understanding supervision aids specific generation tasks, while generation supervision boosts semantic scores on two-frame VQA—highlighting realistic trade-offs rather than overstated universal synergy.

Critical weaknesses

The central claim of "bidirectional synergy" remains insufficiently supported. Compared with reasoning-only training, joint training underperforms on the majority of VQA and captioning metrics. The primary evidence that generation benefits reasoning relies on a minor increase in the semantic judge score for standard two-frame VQA—rising from 0.571 to 0.600—reported without any statistical uncertainty. Similarly, the advantages of understanding supervision for generation depend heavily on the specific conditioning setting and evaluation metric. Consequently, the empirical evidence points to limited task complementarity rather than robustly validating the paper's core claim.

Questions for authors

How was data quality validated? Was an independent manual audit conducted?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer fr48

We thank the reviewer for recognizing the value of UAVReason as a unified, geometrically grounded benchmark and for highlighting the distinction between domain adaptation and cross-task transfer.

Q1: The central claim of "bidirectional synergy" remains insufficiently supported.

A1: Thank you for highlighting the scope of the cross-task claim. We agree that the ablations are better characterized as asymmetric cross-task transfer. In Table 3, adding understanding supervision improves CLIP scores in all three conditional RGB-synthesis settings and lowers KID in two, with a tie in the third. Depth-and-text KID decreases from 0.052 to 0.036. DINO results vary by conditioning setting, and segmentation mIoU remains unchanged.

 

To quantify uncertainty, we add a paired comparison of UAVReason-BAGEL and its reasoning-only variant, using the supplementary Qwen3-VL-8B judge. On 40 fixed two-frame questions, the semantic-correct counts are 27/40 and 23/40, respectively. On an expanded 256-question sample, 251 pairs have valid judge scores. Their mean three-level LLM-J scores are 0.542 and 0.524: a difference of +1.79 percentage points (frame-pair bootstrap 95% CI [−3.00, +6.69], 10,000 resamples).

 

We add this paired analysis and its statistical uncertainty to Appendix F.6 (lines 1378–1393), explicitly separating it from the original Table 2 results (0.571 to 0.600, 235B judge). We revise the cross-task claim to asymmetric cross-task transfer in the abstract (lines 27–31), the contribution statement in Section 1 (lines 104–107), the conclusion in Section 6 (lines 487–491), and the discussions of Tables 2 and 3 in Sections 5.1–5.2 (lines 455–461 and 470–479). The benchmark’s contribution is to make these benefits and trade-offs measurable under a common geometric setting. Improvements over pretrained BAGEL separately demonstrate aerial-domain adaptation.

Q2: How was data quality validated? Was an independent manual audit conducted?

A2: We thank the reviewer for emphasizing the importance of independent validation for automatically generated annotations. Yes. We conduct an independent, blinded human audit of 100 fixed examples, comprising 25 each from single-frame VQA, standard two-frame VQA, heading-aware VQA, and captioning. Three assessors who are not involved in dataset construction independently review the same examples using a common rubric. Model identities and LLM judge scores are concealed.

 

Human annotation audit. The assessors evaluate question validity and clarity, reference correctness, and correspondence between the questions and the intended visual objects.

AssessmentPassErrorUnverifiable
Question validity (75 examples)84.0%13.8%2.2%
Reference correctness (100 examples)69.0%20.0%11.0%
Object correspondence (75 examples)78.2%11.6%10.2%

Percentages aggregate the three assessors’ ratings: 225 judgments for each VQA-specific criterion and 300 for reference correctness. “Error” denotes an assessor’s negative judgment, while “Unverifiable” denotes insufficient evidence to decide. These outcomes are recorded separately at the item level for source verification.

 

Human–LLM agreement. On the same model responses and supplied references, semantic agreement between the assessors and the supplementary Qwen3-VL-8B judge averages 90.2%, with a mean Cohen’s κ of 0.799. This provides quantitative support for the consistency of the supplementary scoring procedure.

 

Geometric verification. Complementing the human audit and the construction checks in Section 3.1, all 41,494 audited distance changes are arithmetically consistent with their stored endpoints. On 200 frames, reference depth shows a median per-frame relative discrepancy of 0.877% against synchronized LiDAR.

 

Together, these checks examine annotation quality, geometric consistency, and evaluation agreement. We add the audit protocol and aggregate results to Appendix F.1 (Tables 8–9, lines 1192–1225), the geometric checks and source-review procedure to the same appendix (lines 1227–1233), and a summary under “Human Quality Audit” in Section 3.1 (lines 211–215).

AuthorsOpenReview ↗

Responses to Reviewer fr48

Follow-up on Q2: How was data quality validated? Was an independent manual audit conducted?

A2: Building on the independent audit reported above, we checked the references against the source evidence and evaluated how the flagged items affected the model comparison. We checked all 100 examples from the original audit and 60 additional examples, for a total of 40 per task. We compared each reference with the images and the available masks, bounding boxes, or endpoint poses. The evidence supported 134 of the 160 original references. We flagged 24 items with specific reference issues and two for which the evidence was inconclusive. The source evidence alone determined which items were flagged, independently of model predictions and scores.

 

1. Protocol A — Strict exclusion

We excluded all 26 flagged items and scored the five models using the original questions, references, and predictions for the remaining 134. Each table reports the strongest comparator for each task.

TaskItemsStrongest comparatorComparator scoreUAVReason-BAGELLead
Single-frame F129Pretrained BAGEL0.39940.81530.4159
Two-frame F130Pretrained BAGEL0.44090.83070.3898
Heading F137Pretrained BAGEL0.79870.97490.1762
Caption CIDEr38GPT-6.1 Terra0.52001.78471.2647

Single-frame and two-frame questions accounted for 21 of the 26 flagged items. With all 21 excluded, UAVReason-BAGEL led pretrained BAGEL by 0.4159 and 0.3898 F1 on these tasks, respectively. These margins are independent of how the flagged items were repaired.

 

2. Protocol B — Full repair

We retained all 160 items and applied all 26 documented revisions. Fifteen changed only the reference, nine revised or replaced both the question and its reference, and two converted visual heading questions into endpoint-pose diagnostics. After editing, we checked each revised item against its source evidence. Using the same model checkpoints, we regenerated predictions for the 11 items whose inputs changed and reused cached predictions for the other 149.

TaskItemsStrongest comparatorComparator scoreUAVReason-BAGELLead
Single-frame F140Pretrained BAGEL0.39810.68960.2915
Two-frame F140Pretrained BAGEL0.42800.69740.2694
Heading F1, image-only38Pretrained BAGEL0.77770.94920.1715
Caption CIDEr40GPT-6.1 Terra0.51341.81401.3006

The heading row covers the 38 image-only items. The other two involved displacements too small for reliable visual direction judgments. Including these two endpoint-pose diagnostics gave 0.9031 F1 for UAVReason-BAGEL and 0.7420 for pretrained BAGEL across all 40 heading items.

 

After repair, UAVReason-BAGEL’s single-frame F1 fell from 0.7797 to 0.6896 and its two-frame F1 from 0.7976 to 0.6974. Both caption repairs corrected location errors. For example, one reference placed a boat and its wake in the upper left of the image, and the corrected reference placed them in the upper right. Across all 40 captions, UAVReason-BAGEL’s CIDEr fell from 1.8373 to 1.8140. UAVReason-BAGEL ranked first among the five models on all four task comparisons in each table.

Comments on the paper

2 comments

Dear Reviewer fr48, zjj5, and Lwqz,

As the rebuttal discussion period is drawing to a close, we kindly encourage you to review the authors' responses to your previous concerns and provide any further feedback or clarification as appropriate.

Thank you for your time and cooperation.

Best regards,

Your Area Chairs

AuthorsOpenReview ↗

Response to All Reviewers: Summary of Revisions

We are grateful to all reviewers for their constructive feedback. The questions and weaknesses raised in the reviews have improved the paper substantially.

We address every comment from each reviewer separately below. The revised version of the paper has been uploaded, and it mainly includes the following:

  • Annotation audit. Three independent assessors review 100 examples. Their semantic judgments on VQA responses agree with the supplementary LLM judge 90.2% of the time on average. We extend the audit to 160 examples, check each reference against source evidence, and flag 26 items. Repairing all 26 lowers single-frame and two-frame F1. UAVReason-BAGEL ranks first on all four metrics on the audited subset under both exclusion and repair.
  • Cross-task transfer. We narrow our claim to asymmetric transfer. Understanding supervision improves CLIP scores in all three RGB synthesis settings. Reasoning results vary by task, and the 95% confidence interval for the paired two-frame score difference includes zero. We also report results by temporal question category.
  • Additional baselines. We evaluate GPT, Claude, Gemini, and Qwen3-VL on a fixed set of 160 examples using common prompts and scoring. We also compare UAVReason-BAGEL with specialist segmentation models on 400 test frames.
  • Geometric evaluation. We verify the arithmetic consistency of 41,494 distance changes and evaluate depth prediction against synchronized LiDAR on 200 frames. Depth RMSE decreases from 43.16 m for pretrained BAGEL to 37.58 m for UAVReason-BAGEL. We also report tolerance-based accuracy on 34 numerical VQA questions.
  • Evaluation scope and figures. We verify the split of 16 training flights and four held-out flights, find no exact duplicate frames across splits, and document geographic overlap between flights. We improve figure readability and correct the labels and example in Figure 6. The correction leaves all reported scores unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.