- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces UAVReason, built on UAVScenes, which brings together RGB imagery, depth, semantic segmentation, single- and two-frame question answering, and conditional generation for overhead UAV images. It contains approximately 23,600 frames, 273,000 question–answer pairs, and 188,800 generation examples. The authors also develop a BAGEL-based baseline for joint understanding and generation. Through comparisons with pretrained models, single-task training, and other models, they examine adaptation to aerial imagery and the interaction between understanding and generation. The main contributions are the dataset and the unified evaluation setup.
Critical strengths
-
The unified dataset provides a useful setting for studying understanding and generation together. The same real aerial images come with language, depth, and segmentation supervision, allowing these tasks to be studied under consistent data conditions. Deriving numerical answers through explicit geometric calculations and splitting the data by complete flight sequences are both sensible choices. They help reduce errors in numerical answers and data leakage between neighboring frames, respectively.
-
The experiments show clear gains from domain adaptation and some benefits from combining tasks. Compared with pretrained BAGEL, two-frame QA F1 increases from 0.427 to 0.822, and segmentation mIoU rises from 0.033 to 0.143. Adding understanding tasks to generation-only training also reduces KID for RGB generation conditioned on depth and text from 0.052 to 0.036. These results show that the dataset supports adaptation to aerial imagery and that language supervision can help image generation in this particular setting.
Critical weaknesses
-
Joint training does not consistently improve reasoning, and the performance drops need a better explanation. In Table 2, compared with reasoning-only training, joint training improves the LLM-J score only for standard two-frame QA, from 0.571 to 0.600. The score drops from 0.454 to 0.397 for single-frame QA and from 0.770 to 0.657 for two-frame QA with heading information. F1 also decreases across all three QA settings. Since the declines also appear in the semantic scores, limitations of lexical-overlap metrics cannot be the main explanation. The results show benefits on specific tasks alongside performance trade-offs, and the broader claim that understanding and generation benefit each other needs to be qualified.
-
Some aspects of the geometric and temporal evaluation need clarification. RGB→Depth is explicitly included in the benchmark, but neither the main paper nor the appendix reports quantitative depth prediction results. Appendix B also defines tolerance-based accuracy for numerical questions, yet the corresponding results are not reported.
Questions for authors
-
In Table 2, joint training improves the semantic score for standard two-frame QA but lowers it for single-frame QA and two-frame QA with heading information. Are the gains concentrated in particular types of temporal questions? How do the authors explain this difference?
-
What are the correct question, reference answer, and model prediction for the right-hand panel of Figure 6? Is the discrepancy limited to the figure, or does it also affect the evaluation data and scoring?
-
What are the quantitative results for RGB→Depth and the tolerance-based accuracy defined in Appendix B? Do they show that joint training improves the prediction of geometric quantities such as distance?
2 comments
Responses to Reviewer Lwqz
Q3: Are the gains concentrated in particular types of temporal questions? How do the authors explain this difference?
A3: Thank you for asking about the temporal reasoning gains and the performance differences across tasks. We add a paired category-level comparison of UAVReason-BAGEL and its reasoning-only variant on the same 40 standard two-frame questions, using the supplementary Qwen3-VL-8B judge. The table reports UAVReason-BAGEL’s semantic-correct counts and nominal item-level 95% Wilson intervals. The reasoning-only counts are given below. We add the category-level comparison to Appendix F.6 (Table 15, lines 1362–1376), followed by the paired comparison and statistical uncertainty analysis (lines 1378–1393).
Temporal category Semantic correct / total 95% Wilson interval Identity 8/9 56.5–98.0% Distance change 7/9 45.3–93.7% Relation change 7/12 32.0–80.7% Approach or separation 5/10 23.7–76.3% Total 27/40 52.0–79.9% On these same questions, the reasoning-only variant scores 7/9 for identity, 6/9 for distance change, 6/12 for relation change, and 4/10 for approach or separation. UAVReason-BAGEL therefore has one additional semantically correct answer in each category. On this 40-question subset, the four-answer difference is spread across categories rather than concentrated in one. At the task level, the original ablations show the asymmetric pattern discussed in Q1 (A1): a higher standard two-frame semantic score coexists with declines on other reasoning metrics. The generation-heavy allocation of supervision may contribute to these trade-offs.
Q4: Figure 6 inconsistency and its impact on evaluation reliability.
A4: Thank you for identifying this inconsistency. We apologize for the confusing presentation. In the original version of Figure 6, the titles of the middle and right panels are inadvertently swapped, and the question from the middle panel is mistakenly duplicated in the right panel. The middle panel corresponds to heading-aware two-frame VQA, while the right panel illustrates standard two-frame VQA. We correct the middle/right panel titles, the duplicated right-panel question, and the inconsistent reference-answer wording in Figure 6 on page 17. The corrected right panel asks whether the white sedan approaches or separates from the traffic barrier. The reference answer indicates separation, with a distance increase of 0.631 m, while our model predicts the correct separation trend with an estimated increase of 0.461 m. We verify that these errors are confined to figure preparation and do not affect the underlying evaluation data, scoring, or reported quantitative results.
Q5: Missing geometric evaluation results and evidence of joint training benefits.
A5: For dense geometric prediction, the original single-task comparison shows equal segmentation mIoU for joint and generation-only training (0.143, Table 3). The supplementary depth and numerical results in Q2 (A2) improve over pretrained BAGEL, measuring aerial-domain adaptation. They do not isolate a joint-versus-single-task effect. The supported cross-task benefit is in conditional RGB synthesis, as detailed in Q1 (A1). We add the depth results and protocol to Appendix F.4 (Eq. 2 and Table 12, lines 1279–1301), the segmentation references to the same appendix (Table 13, lines 1294–1327), and the numerical VQA results and protocol to Appendix F.5 (Eq. 3 and Table 14, lines 1332–1357).
Responses to Reviewer Lwqz
We appreciate the detailed questions on joint training, geometric evaluation, and Figure 6. Below, we explain the original ablations and report the completed depth, numerical, and temporal-category evaluations.
Q1: The central claim of "bidirectional synergy" remains insufficiently supported.
Q2: 1. Missing quantitative RGB-to-Depth evaluation.
2. Missing tolerance-based accuracy results for numerical VQA.