acceptodds
Under review as a conference paper at ICLR 2027

3D-VL-Fly: A Multi-task 3D Vision–Language Benchmark for Aerial Post-Disaster Damage Assessment

25 views

Abstract

Recent advances in multimodal large language models have enabled sophisticated visual understanding and reasoning for disaster response. Emerging benchmarks such as DisasterM3 have begun to extend multimodal learning to post-disaster scenarios through tasks such as visual question answering and damage assessment. However, these benchmarks predominantly rely on 2D imagery, limiting the evaluation of multimodal models' ability to reason over the three-dimensional geometric structure of disaster environments. Important evidence, including collapsed roof structures, debris extent, building-level spatial relationships, and partially occluded structures, are difficult to fully characterize from a single 2D view. In this study, we introduce 3D-VL-Fly, the first 3D vision–language benchmark specifically designed for post-disaster damage assessment. Built from 3D point-cloud scenes generated from aerial imagery collected following Hurricane Ian (2022), the benchmark provides building-level instance segmentation in addition to semantic segmentation of roads and trees, attributes per building independently, and question–answer pairs spanning eight question families with spatially grounded building references. Beyond visual question answering, 3D-VL-Fly incorporates both referring segmentation, which evaluates a model's ability to localize language-referred buildings or regions, and reasoning segmentation, which evaluates whether a model can identify the 3D evidence relevant to answering a reasoning-oriented question. This unified formulation enables the evaluation of multimodal models not only on what they answer, but also on where the relevant evidence is located in a 3D disaster scene. We evaluate a range of pretrained 3D and 2D vision–language models and encoders, and fine-tune a representative 3D vision–language model. Released models remain below a simple majority baseline on VQA and achieve low segmentation accuracy, indicating substantial room for improvement.

Primary area:
datasets and benchmarks
Keywords:
Post-disaster damage assessment, 3D point clouds, Visual question answering, Reasoning segmentation, Vision-language models, UAV remote sensing.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hyCaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces 3D-VL-Fly, a 3D vision-language benchmark for post-disaster damage assessment. Built from aerial reconstructions following Hurricane Ian, it contains 64 scenes and 308 buildings with instance masks, attributes, and language annotations. The benchmark supports VQA, language-guided segmentation, attribute recognition, and instance segmentation, with evaluations of pretrained models and a fine-tuning experiment.

Critical strengths
  1. Extending 3D vision-language evaluation to real disaster scenes provides a useful resource for studying building damage and outdoor spatial understanding.
  2. Majority baselines and separate damage-attribute results help distinguish general building recognition from damage-assessment capability.
Critical weaknesses
  1. Annotators access high-resolution photographs that point-cloud models do not receive. The paper acknowledges that some details may be less observable, but does not assess which labels can be reliably inferred from point clouds alone. Low scores may therefore reflect missing input evidence as well as limited model capability.
  2. The paper introduces both referring and reasoning segmentation, but the task description mainly lists directional and attribute-based references, and results are split only by direction. Please clarify how reasoning queries are distinguished from referring expressions and report their performance separately.
Questions for authors
  1. The dataset contains 64 scenes, but experiments use only 49. Why were the remaining 15 not included, and do they differ from the evaluated subset in damage severity or reconstruction quality?
  2. How are building-to-building and building-to-road distances defined: between centers, nearest boundaries, or otherwise? Is reconstruction scale error considered when generating distance-based answers?
Reviewer_bBR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

3D-VL-Fly is a benchmark that asks vision-language models about 3D point clouds of neighborhoods hit by Hurricane Ian in 2022, taken from the 3D-AeroRelief dataset. The authors separate 308 buildings by hand across 64 scenes, have three annotators label eight properties per building such as damage level, roof type and debris, estimate north for each scene, and automatically generate questions plus prompts that ask a model to pick out described buildings. Testing uses 16 held-out scenes with 85 buildings. No released model, reading point clouds or rendered images, beats always giving the most common training answer for each question type (best 37.28% against 44.88%), buildings are picked out poorly (below 9 mIoU), image features predict building properties better than point-cloud features, and fine-tuning one 3D model reaches 41.52% on questions and 17.32 mIoU.

Critical strengths
  1. The evaluation is unusually honest. It uses held-out areas, majority baselines for VQA and attributes, scene-level bootstrap intervals, and a per-attribute table the authors turn against their own headline number. That care is what exposes the problems below, and it is why I would like to see a revision.
  2. The negative result is useful. Released 3D models transfer poorly to aerial point clouds (1.46% and 23.39% on non-directional VQA for Grounded 3D-LLM and 3D-LLaVA), and point-cloud encoders sit at the majority level on roof pitch, where an image encoder reaches 73.81%.
Critical weaknesses
  1. The benchmark cannot yet tell model failure from questions that priors answer or the input cannot. A constant answer per question type (44.88%) beats every model on VQA, including the fine-tuned one (41.52%). Stories is constant in the test set, and the three damage attributes sit at or below the majority level, e.g. damage level 63.53% against 61.18%, two buildings out of 85. Labels were partly made from photos the 3D models never see, and the north and metric scale behind directional and distance questions come with no error estimate. An oracle answering from the ground-truth labels, a person answering from the point cloud alone, and a text-only model fine-tuned on the questions would separate label errors, unobservable questions and priors. The claim that 3D adds evidence beyond 2D is also untested, and the existing comparisons mostly favor 2D, except on directional VQA.
  2. Scale and positioning. 64 scenes from one storm and 85 test buildings is pilot size, and one building moves attribute accuracy by 1.2 points. The closest prior benchmarks, CityRefer (Miyanishi et al., NeurIPS 2023), which adds referring expressions to SensatUrban, and City-3DQA (Sun et al., ACM MM 2024), which does question answering on city-scale point clouds, are neither cited nor compared.
Questions for authors
  1. What is the angular error of the north estimate on the check frames in Appendix A?
  2. 3D-AeroRelief sets scale by assuming a single-story house is about 3 m tall. Is that the scale behind the 5 m distance bins, and how accurate is it?
  3. Why are the 15 scenes outside Areas 3 to 8 left out?
  4. What was annotator agreement per attribute before voting? A three-way split cannot happen for binary attributes, so the under 1% figure says little.
  5. What does a mask of every building in the scene score on segmentation? With about five buildings per test scene it could be near 20 IoU, above every row of Table 2.
  6. Please report the fine-tuned model on directional questions, over several seeds, next to a text-only model trained on the same data.
Reviewer_6ewgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces 3D-VL-Fly, a multi-task 3D vision–language benchmark for post-disaster damage assessment. It extends the existing 3D-AeroRelief data with rich annotations,  supporting scene-level visual question answering, language-guided building segmentation, building-attribute recognition, and class-agnostic building instance segmentation. 17 pretrained models were tested, together with a fine-tuning experiment. The results showi that there is still substantial room for improvement.

Critical strengths

(1) Existing disaster VQA benchmarks are predominantly based on 2D images, while important post-disaster evidence can be difficult to characterize from a single view. 3D-VL-Fly brings language-grounded reasoning and segmentation to real UAV-reconstructed disaster scenes rather than indoor point-cloud environments, introducing a practically relevant setting.

(2) The same building instances and attributes support scene-level VQA, referring and reasoning segmentation, attribute recognition, and instance segmentation, potentially evaluating both whether a model produces the correct answer and whether it localizes the corresponding evidence.

Critical weaknesses

(1) The dataset contains 64 scenes and 308 buildings from one hurricane event in Florida, while the quantitative test set contains only 16 scenes and 85 buildings from two held-out areas. The dataset is potentially useful as an initial benchmark, but the current scope makes it difficult to determine whether models are learning generalizable disaster reasoning or patterns specific to this site and acquisition campaign.

(2) Native 3D models receive differently sampled and processed point clouds, while 2D MLLMs receive three orthographic renderings with metric scene dimensions. The evaluated architectures, pretraining data, input representations, and inference interfaces therefore differ substantially. The paper states that these results should not be interpreted as a single ranking across modalities, but this also means that the experiments cannot demonstrate that 3D input improves damage reasoning relative to appropriately matched 2D or multi-view input. A controlled comparison using the same scenes, comparable model capacity and training supervision, and matched information content would be needed to isolate the contribution of the 3D representation.

Questions for authors

(1) Appendix A only states that the authors combined annotations, directions, and computed distances to generate the questions. Could the authors explain whether the questions are generated using deterministic templates, an LLM, manual writing, or a combination of these methods? At the same time, please also report the number of questions per family, the answer distributions, the rate of ambiguous or invalid questions, or any human evaluation of question quality. This way, it will be easier to clarify how much of the benchmark measures scene understanding rather than exploitation of answer priors and recurring linguistic structures.

(2) What are the train, validation, and test counts and answer distributions for each of the eight question families? Are macro scores computed over all eight families in both directional and non-directional subsets, or only over families present in each subset?

(3) Were the manually separated building instance masks reviewed by a second annotator? How were attached structures, neighboring roofs, vegetation-covered boundaries, and incomplete reconstructions handled?

(4) Areas 3–6 are used for training and validation, while Areas 7–8 form the test set. Are these areas geographically separated and collected in different flights, or could adjacent areas share buildings, neighborhoods, reconstruction artifacts, or acquisition conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.