acceptodds
Under review as a conference paper at ICLR 2027

ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has empowered Unmanned Aerial Vehicle (UAV) with exceptional capabilities in spatial reasoning, semantic understanding, and complex decision-making, making them inherently suited for UAV Search and Rescue (SAR). However, existing UAV SAR research is dominated by traditional vision and path-planning methods and lacks a comprehensive and unified benchmark for embodied agents. To bridge this gap, we first propose the novel task of Embodied Search and Rescue (ESAR), which requires aerial agents to autonomously explore complex environments, identify rescue clues, and reason about victim locations to execute informed decision-making. Additionally, we present ESARBench, the first comprehensive benchmark designed to evaluate MLLM-driven UAV agents in highly realistic SAR scenarios. Leveraging Unreal Engine 5 and AirSim, we construct four high-fidelity, large-scale open environments mapped directly from real-world Geographic Information System (GIS) data to ensure photorealistic landscapes. To rigorously simulate actual rescue operations, our benchmark incorporates dynamic variables including weather conditions, time of day, and stochastic clue placement. Furthermore, we create a dataset of 600 tasks modeled after real-world rescue cases and propose a robust set of evaluation metrics. We evaluate diverse baselines, ranging from traditional heuristics to advanced ground and aerial MLLM-based ObjectNav agents. Experimental results highlight the challenges in ESAR, revealing critical bottlenecks in spatial memory, aerial adaptation, and the trade-off between search efficiency and flight safety. We hope ESARBench serves as a valuable resource to advance research on Embodied Search and Rescue domain.

Primary area:
datasets and benchmarks
Keywords:
UAV, Search and Rescue, Embodied Navigation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_4ch7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalizes a new task: Embodied Search and Rescue (ESAR), presents a simulator (UAV-ESAR Simulator) to evaluate solution algorithms on the task, and proposes a benchmark (ESARBench) on the ESAR task that uses the simulator.

The ESAR task is a search and rescue task where an autonomous agent, embodied by a UAV has to localize one or several victims in an outdoor environment. The agent is initially given mission details such as the weather, start position, time of day, the number of missing victims and a description of where the victims were last seen. In addition to this information, the agent is expected to reason about visual clues (e.g. equipment like tents or backpacks), discovered during the mission to guide it towards the victim(s).

The UAV-ESAR Simulator is based on Airsim-Colosseum which is a flight simulator developed in Unreal Engine 5. ESAR-missions can be deployed in 4 different maps/worlds that are constructed from real-world GIS-data. The UAV dynamics and sensors are the same as in the base Airsim-Colosseum implementation, sensors include: IMU, GPS, LiDAR and RGB-D camera images. Weather can be adjusted in the simulator.

ESARBench leverages the UAV-ESAR Simulator to generate ESAR tasks by “sampling” scenarios from the simulator. A task consists of a map, a number of victims, an initial textual description of the mission, weather, time-of-day and where the visual clues should be located. The autonomous agent is evaluated on the ESAR task according to four proposed metrics.

The authors state that the main contributions are that they formalize the problem of embodied search and rescue, and present a benchmark for which the community can now evaluate their algorithms on. The authors evaluate nine methods on ESARBench, 7 of these are methods from previous works that attempt to solve the same or adjacent tasks.

Critical strengths
  • The authors evaluate many baselines on their proposed benchmark.
  • The idea of evaluating a method’s ability to reason about and take advantage of “clues” that it encounters seems reasonable in a search-and-rescue task.
  • The maps are appropriate for the task.
Critical weaknesses
  • Evaluation metrics. Four evaluation metrics are proposed in the paper, but there are some concerns with most of them. For SR, when applying the Hungarian algorithm, is there not a risk of attributing the model to be better than it actually is? Say the model makes thousands of predictions, i.e., N >> M (where most are false positives), wouldn’t this yield a high SR?. For TSR, there is no discussion on how to set T_{\max}, this surely varies depending on the environment (maybe hours on K2 / Snowy Peak, while it could be days for Dapeng / Coast, using your maps as an example). The CDS metric is not properly defined in the paper. The metric depends on C_{total} which is not introduced, and C_{exact}/CRR, whose definition is unclear. One option is to interpret CRR/C_{exact} to be the size of the intersection between the set of all predicted clues and “ground truth clues” over the size of the ground truth clues, for all clues encountered during a mission. This seems strange as the temporal aspect of when a clue is discovered should be important. If the agent identifies a false positive clue at time t, that then later appears at time t’ > t, then this would increase the CRR. It would improve the paper overall if you could give an example of what a clue is, i.e. what is in the sets M⋅\mathcal{M}_{\cdot}? The RS metric seems to be a combination of all previous metrics weighted a bit arbitrary. It also includes an indicator variable IsafeI_{safe} which is equal to one if the agent “safely completes the mission”. Safety isn’t defined anywhere and it could mean many things. Safety could mean simply that the UAV doesn’t crash during the mission, it could also mean that it waits for harsh weather conditions to clear before starting the mission and similar behavior. From the definitions of SR and TSR (eq. (4)-(5)) it seems like the range of the metrics is [0,1], but this is not the case in Table 2.

  • Results and Discussion. The results in Table 2 and 3 are difficult to interpret without any variance/std deviation. Some results seem a bit strange, many baselines perform worse than Random across metrics and benchmark difficulties. Furthermore, looking at the results for e.g. “Pure-MLLM” on the “Extreme” difficulty, it scores 0.00 on SR, TSR and CDS, but the RS is 7.08; given the definition of RS, I don’t see how that could be possible. Minor note: I suppose you have multiplied SR (and all other metrics) by 100 in the tables? Figure 5: Why is the crash rate so large? The crash rate is beyond 20% for every baseline. Figure 4 is difficult to interpret. In (c), showing that there are 600+ tents doesn’t provide much insight without knowing the total amount of missions/scenarios in the benchmark. Fig. 4d shows that in most scenarios, the goal is less than 200m from the start?

  • Clarity. Section 3.1 could be improved, it uses vague terms to describe variables “internal state, S_t”, “historical context H_t” or doesn’t describe them at all (A_{t+1}). Section 3.3 could provide more detail about how tasks are generated, for instance how clues are placed. This seems especially important as if clues such as clothes are scattered around the environment independent of the victims locations, the clues are uninformative. Section 3.3.2 starts to describe different “difficulty tiers” of ESARBench. The scoring seems a bit arbitrary, in particular for weather, time-of-day and strong-clue-assistance. Would it be possible to have a more data-driven approach to this scoring? The baseline methods seem to score about the same on “Hard” and “Extreme” on most metrics.

  • Novelty. Table 1 lists previous benchmarks related to “aerial robotics” and section 2.2 briefly discusses these. However, I still don’t think it’s clear what differentiates your work from those. The table outlines the underlying Unreal Engine version (4 or 5), the map scale (this is ambiguous, what does small/medium/large mean?) the number of different weather settings (doesn’t this simply depend on UE version + software extensions?) and real data (there is no discussion about the benefit of this). Apart from the software version of UE and real data, it would be good to highlight the novelty of the simulator and benchmark more and how the changes that you have made are relevant for the task.

  • Structure and language. Authors may want to improve the presentation of the paper.

Questions for authors
  • What are S_t, H_t, A_t and M_t exactly?
  • Is C_pred one more output from \pi ?
  • What are the sets \mathcal{M} in eq. (3)?
  • Why is CRR a good metric?
  • Why not look at other common metrics such as false positive clues that agents “discover” as well?
  • What is C_{total} in eq. (6)?
  • What is temporal efficiency (E_t) in eq. (7)?
  • How is I_{safe} defined?
  • What is the reasoning behind the weighting in eq. (7)?
  • What is the variance of the numbers in Table 2 & 3? Is any of the methods outperforming Random with some level of statistical significance?
  • What is the difference between Simple/Medium/Hard/Extreme ESARBench?
  • Is there an apparent correlation between high CDS and high SR/TSR? Is it beneficial for a model to discover the clues?
Reviewer_XcxcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ESARBench, a benchmark for embodied UAV search and rescue. The authors construct four large-scale simulation environments using real-world geographic data, Unreal Engine 5, and AirSim. The benchmark includes 600 tasks with different terrains, weather conditions, and rescue scenarios. It evaluates UAV agents on victim localization, clue discovery, search efficiency, and flight safety. The authors evaluate several traditional exploration methods and MLLM-based agents. The results show that existing methods still face significant challenges in complex UAV search and rescue tasks.

Critical strengths
  1. The evaluation metrics require clearer definitions and explanations. Although the paper introduces several evaluation metrics, some important details are not sufficiently specified. For example, the success rate uses a distance threshold to determine whether a victim is correctly localized, but the exact value of this threshold is not clearly provided. The handling of incorrect location reports also needs clarification. These details are important because they can directly affect the evaluation results and the reproducibility of the benchmark.
  2. Insufficient failure analysis and inconsistencies in the manuscript. The experiments show that existing methods perform poorly on the proposed benchmark. However, the paper provides limited analysis of why these methods fail. It remains unclear whether the main difficulties come from visual perception, semantic reasoning, navigation, or search planning. A more detailed failure analysis would help explain the challenges introduced by the benchmark. In addition, there are inconsistencies between the main paper and the supplementary material. For example, Figure 3 on page 6 describes the K2 environment as 3 km × 3 km, while Appendix C describes it as 2 km × 2 km. Such inconsistencies should be corrected, especially since environment scale is important for evaluating search tasks.
Critical weaknesses
  1. The evaluation metrics require clearer definitions and explanations. Although the paper introduces several evaluation metrics, some important details are not sufficiently specified. For example, the success rate uses a distance threshold to determine whether a victim is correctly localized, but the exact value of this threshold is not clearly provided. The handling of incorrect location reports also needs clarification. These details are important because they can directly affect the evaluation results and the reproducibility of the benchmark.
  2. Insufficient failure analysis and inconsistencies in the manuscript. The experiments show that existing methods perform poorly on the proposed benchmark. However, the paper provides limited analysis of why these methods fail. It remains unclear whether the main difficulties come from visual perception, semantic reasoning, navigation, or search planning. A more detailed failure analysis would help explain the challenges introduced by the benchmark. In addition, there are inconsistencies between the main paper and the supplementary material. For example, Figure 3 on page 6 describes the K2 environment as 3 km × 3 km, while Appendix C describes it as 2 km × 2 km. Such inconsistencies should be corrected, especially since environment scale is important for evaluating search tasks.
Questions for authors

Clarification Questions

  1. Evaluation metrics: What is the exact distance threshold used to determine successful victim localization? How was this threshold selected? How are incorrect or multiple location reports handled?
  2. Environment consistency: Figure 3 in the main paper and Appendix C report different sizes for the K2 environment (3 km × 3 km versus 2 km × 2 km). Which value is correct? Please check the consistency of the environment configurations throughout the paper and supplementary material. Additional Experimental Evidence
  3. Failure analysis: Could the authors provide a more detailed analysis of the failures of different baseline methods? In particular, it would be helpful to distinguish failures caused by perception, semantic reasoning, navigation, and search planning. This would better explain the main challenges of ESARBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.