acceptodds
Under review as a conference paper at ICLR 2027

ReVA: A New Scene-Centric Dataset for Remote Sensing Video Question Answering

12 views

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable advances in remote sensing. However, existing remote sensing multimodal reasoning benchmarks exhibit two critical limitations: they rely on (i) template-driven questions, which causes repetitive questions; and (ii) static image inputs that fail to capture the inherent temporal nature of drone/UAV videos. This leaves systematic evaluation of remote sensing video reasoning largely unexplored. To address this gap, we introduce ReVA, a new dataset for remote sensing video question answering, designed to assess spatiotemporal, scene-centric, and reasoning-oriented capabilities of MLLMs. ReVA comprises 2,438 drone videos spanning 18 cities worldwide (580K frames) and 22K high-quality question–answer pairs across 11 challenging QA tasks. We develop a semi-automatic annotation pipeline that leverages Text LLMs and MLLMs for question-answer generation with human verification. We evaluate 23 proprietary and open-source Video LLMs on ReVA, exposing fundamental limitations of current models. These findings position ReVA as a critical benchmark for advancing MLLMs toward better remote sensing video understanding and temporal reasoning capabilities for real-world deployments.

Primary area:
datasets and benchmarks
Keywords:
Scene-Centric Dataset, Remote Sensing Video Understanding, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RTgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper primarily focuses on constructing a remote sensing video question-answering dataset. Motivated by the limitations of existing benchmarks, particularly their reliance on template-based questions and lack of temporal information, the authors introduce a new VideoQA dataset based on UAV remote sensing videos.

Critical strengths
  1. The paper analyzes the limitations of static remote sensing VQA and natural-domain VideoQA benchmarks, providing a reasonable motivation for introducing a remote sensing video dataset.
  2. The authors have made substantial efforts in dataset construction and annotation, establishing a comprehensive pipeline that includes video filtering, caption generation, question generation, answer generation, and human verification.
Critical weaknesses
  1. Although the paper introduces a novel data format, it does not sufficiently develop an evaluation protocol tailored to the characteristics of the proposed dataset.
  2. It remains unclear whether temporal ordering and cross-frame relational reasoning are essential for answering the questions. For instance, even without understanding the temporal order of events, a model might significantly outperform single-frame baselines simply by recognizing more objects, roads, and scene details from the 32 sampled frames. Therefore, it may be necessary to distinguish between visual state changes and temporal relational reasoning to better assess the intended capabilities.
Questions for authors
  1. Section 3.1 (page 4) states that all videos are trimmed to approximately 15 seconds, while the appendix (page 14) indicates that 32 frames are uniformly sampled from each video. Although this setting is reasonable for conventional short-video question answering, the paper repeatedly emphasizes long-range temporal reasoning and long-range dependencies. How do the authors reconcile this relatively short temporal window with their claims regarding long-range temporal understanding?
Reviewer_bHnCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReVA, a scene-centric remote sensing video question answering dataset containing 2,438 UAV videos from 18 cities and about 22K multiple-choice QA pairs across 11 tasks. The QA pairs are generated by an MLLM through a five-stage semi-automatic pipeline and then verified by humans. The authors evaluate 23 video MLLMs on ReVA and further propose ReMoSense, a motion-aware model that encodes camera ego-motion and object dynamics.

Critical strengths
  1. Remote sensing video QA, especially temporal and causal reasoning, remains underexplored, so the dataset is a useful resource for the community.

  2. The construction is careful in several respects, including two-round human verification, an inter-reviewer agreement study, and splitting at the video-sequence level to avoid scene leakage.

Critical weaknesses
  1. The positioning of the paper is unclear. The abstract presents a dataset only, whereas the introduction lists ReMoSense as a main contribution, so it is hard to tell whether the paper should be judged as a dataset paper or a method paper.

  2. The annotation claim is overstated. The QA pairs are generated by a single MLLM and only verified by humans, yet Table 1 labels them as manually annotated.

  3. Evaluation of frontier models is missing. The "proprietary" entries are agent frameworks rather than GPT or Gemini themselves, and Qwen-VL models are not evaluated even though the data are generated by Qwen3-VL, leaving possible same-source bias unexamined.

  4. The difficulty of the benchmark is not convincingly established. Causal reasoning is already near saturation (94–98%, and 72.7% from the first frame alone), no human performance is reported, and the 15-second, 640×360 clips limit the claimed long-range temporal reasoning.

  5. Closely related work is overlooked. A2Seek (NeurIPS 2025) already provides reasoning-centric aerial video understanding, and CameraBench (NeurIPS 2025) targets camera-motion understanding, which is central to ReMoSense; neither is cited or compared.

Questions for authors
  1. Is the main contribution the dataset or ReMoSense? If it is the dataset, ReMoSense would be better presented as a baseline.

  2. How do recent frontier models, such as the GPT-6 series, Gemini, and Qwen3.8, perform on ReVA in a zero-shot setting?

  3. What is the human accuracy on ReVA, and how large is the gap between humans and the best model?

  4. The appendix includes an MLLM-based QA verification and refinement prompt, which is not part of the five-stage pipeline in the main text. Could the authors clarify where this step is applied?

  5. Single-frame results are reported with a model trained only on videos. Would a strong image MLLM given a single frame perform much better?

  6. In addition, 14 of the 18 cities come from VisDrone, whose footage is captured below 50 meters. To what extent can such low-altitude, near-ground views be regarded as remote sensing data?

Reviewer_tqfZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors introduce ReVA which is a video question answering benchmark for drone based videos. It contains 2,438 15-second clips where 1,280 are newly collected rural US videos plus clips from ERA, an has VisDrone/UAVDT videos too. It also has about 22K multiple-choice QA pairs that covers 11 different tasks in four groups containing factual perception, temporal understanding, spatial/viewpoint reasoning and causal reasoning. The QA pairs come from a semi-automatic pipeline where Qwen3-VL is used to generate captions, questions and answers, that is then validated by humans. The paper evaluate 23 video LLMs and also propose ReMoSense that is a framework and adds two kinds of motion tokens to a video LLM. The first is a global correspondence token that is built from a cost volume between consecutive frames, and the second is object motion tokens that are updated recurrently over time. It scored 80.04% after fine-tuning on ReVA, which is about 3.7 points above the best baseline.

Critical strengths

This is a well-written, well-presented, and well-structured paper. The flow of paper is well-coherent throughout and as a reader, the story goes on progressivly while reading it. Here, the authors tried to address a real gap as most remote sensing VQA datasets use still images and templated questions. One of the other challenge discussed is the drone video layout as it do requires temporal and viewpoint reasoning, that is essential to understand the context very well. The newly collected videos by authors add rural and higher-altitude scenes that existing UAV datasets do lack as discussed. The evaluation scope of the paper also covers many recent video LLMs and includes a training split, which makes the dataset useful for both benchmarking and adaptation. The authors also proposed a framework along with the ReVA dataset which makes the paper more stronger. Moreover, the results in Table 1 has a good gain for overall scores.

Critical weaknesses

The paper is well-writtend and well-structured. However, there are some issues that can be considered to make this work strong. Following are some of the weaknesses of the paper:

  • The ReMoSense has a basic design problem. The global correspondence token applies an MLP to each similarity value independently and then pools over both spatial dimensions. The resulting token therefore depends only on the set of similarity values, not on where they occur in the frame. This means it cannot distinguish a left pan from a right pan, or forward from backward motion. The object motion tokens closely resemble existing recurrent memory and query-propagation designs, which limits their novelty. Furthermore, the ablation also does not separate the benefit of motion modeling from simply adding extra learnable tokens. Can the authors provide some further details to this point?
  • There is also no evidence that the tokens actually capture camera motion, which the authors could check against the telemetry from their own DJI flights.
  • The proposed method is evaluated only on ReVA, so its generality is unknown. I suggest authors to provide some further evaluation of RemoSense so as to make the generalization aspect more stronger for this paper.
Questions for authors
  • Why does fine-tuning on ReVA hurt some baselines, and were all models given the same tuning effort?
  • How can the pooled cost volume encode the direction of motion? Can the authors show that the global token predicts camera motion too, for example against drone telemetry, and add a control that uses the same number of extra tokens without motion modeling?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-Up on Reviewer Feedback

Dear Reviewer,

As the discussion period is approaching its end, we would like to thank you for your thoughtful feedback. We hope our responses and additional analyses have adequately addressed your concerns.

Please let us know if any questions remain or if further clarification would be helpful. We would be happy to provide additional details.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.