acceptodds
Under review as a conference paper at ICLR 2027

LifelineQA: An Evidence-Grounded Benchmark for Situational-Awareness Question Answering in Emergency Operations

Abstract

Disasters disrupt the lifeline systems that communities depend on, including safety and security, food, water, shelter, health services, energy, communications, transportation, and hazardous-material management. During response operations, decision makers need timely and reliable answers about what is happening, which lifelines are affected, where conditions are changing, and how evidence from evolving operational reports should be interpreted. Yet existing question-answering (QA) benchmarks rarely capture this setting, where information is time-sensitive, distributed across reports, and tied to domain-specific lifeline frameworks. We introduce LifelineQA, an expert-curated, evidence-grounded benchmark for lifeline-centered QA in emergency operations. LifelineQA includes three complementary question types: situational questions about incident status, threats, and response resources; lifeline-specific questions about risks, interruptions, and impacts affecting Federal Emergency Management Agency (FEMA) Community Lifelines; and complex questions requiring synthesis across reports, evidence units, or dates, including spatiotemporal reasoning. Each answerable question is linked to verifiable source evidence. We evaluate ten large language models as QA systems across five evidence conditions to assess correctness, evidence support, abstention, and temporal robustness. Results show that models answer accurately when given relevant evidence, but retrieving that evidence and citing sufficient support remain distinct barriers. LifelineQA establishes a new evaluation setting for reliable, evidence-grounded QA in high-stakes emergency operations.

Primary area:
datasets and benchmarks
Keywords:
question answering, large language models, datasets and benchmarks, evidence-grounded evaluation, emergency management, disaster response

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1QoiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LifelineQA, a benchmark with 800 items for evidence-grounded question answering over Arizona wildfire incident reports. It contains situational, lifeline-specific, and complex questions, with expert-reviewed answers and supporting evidences. 10 LLMs are evaluated under five evidence conditions to study answer correctness, evidence support, abstention, and temporal robustness.

Critical strengths

The expert review is a substantive part of this work. Each candidate was independently reviewed by two reviewers, followed by specialist adjudication, and 476 items were revised before finalization. This is more useful than simply releasing LLM generated questions without checking them. Linking answers to specific report fields also provides a concrete basis for inspecting their support.

Critical weaknesses
  • Some reliability metrics have problems that might affect their interpretation. Appendix A2.2 defines supported-claim precision as entailed / (entailed + contradicted), excluding neutral claims. Therefore, an answer containing one supported claim and many supported, non-contradicted claims could still received perfect support precision, if I understand this correctly. Similarly, over-abstention is defined using benchmark-level answer-ability, altho models are instructed to answer only from the supplied evidence. Under retrieval, refusing because the required evidence was not retrieved can then be counted as over-abstention. This can measure lost end2end answer coverage, but does not, by itself, establish inappropriate refusal by the reader.
  • The retrieval results do not sufficiently separate benchmark difficulty from baseline limitations. The questions explicitly identify the incident and relevant dates, and retrieval chunks already contain report-date and field metadata. But the evaluated retrievers are BM25, a dense retriever, and their hybrid, without a reported date-filtered comparison. The hybrid's Recall score establishes a limitation of that setup, but a simple baseline that first selects the requested reporting dates would help show how much difficulty remains.
  • The stated selection ILP does not enforce its intended coverage objective. Appendix A1.2 maximizes document and incident coverage, but the constraints allow a document to be marked as selected even when no question is drawn from it.
Questions for authors

Q1: does the implementation exclude neutral claims from supported-claim precision, as the formula states? How do the conclusions change when neutral claims count as unsupported and abstention is separately assessed against the evidence actually supplied?

Q2: how much does retrieval improve when reports are first filtered by the dates explicitly requested in the question, suing the same evidence budget? Which question types remain difficult after this control?

Q3: does the implemented ILP contain a constraint linking each marked document to at least one selected question?

Reviewer_rdH8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivation: Existing benchmarks rarely test, at the same time, official incident reports, successive versions of the same record, precise temporal context, and complete evidence citation.

Proposed approach: The paper introduces LifelineQA, an expert-curated, evidence-grounded benchmark for lifeline-centered question answering in emergency operations. Its novelty lies mainly in this new evaluation setting. The benchmark is modest in size (800 QA pairs) and covers only wildfire incidents in Arizona.

Contributions: The authors claim three contributions. (1) An expert-reviewed QA benchmark over evolving official incident reports (ICS-209) that connects emergency management information needs to QA evaluation, with questions organized by FEMA Community Lifelines. (2) Each answerable item is linked to structured, field-level evidence and operational context, enabling verification. (3) An evaluation protocol that tests LLMs under five evidence conditions (closed-book, gold evidence, retrieval, full incident history, and gold evidence with temporal distractors), separating prior knowledge, reading, retrieval, contextual grounding, and answer verification.

Conclusions: The authors argue that the main bottleneck of current systems is not generating answers but finding the right evidence, gathering all of the evidence an answer requires, grounding the answer in the correct reporting date for the same incident, and explicitly citing evidence sufficient to support the answer. When evidence from adjacent reporting periods is added, citations are more likely to draw on adjacent-date versions, while the answers themselves are largely unaffected.

Critical strengths
  1. Rigorous dataset construction. The five evidence conditions are well motivated and reveal distinct test scenarios. Every answerable item is linked to field-level evidence, and evaluating answer correctness separately from evidence sufficiency is sound. Each item was independently judged by two reviewers on six dimensions.
Critical weaknesses
  1. Narrow scope, limited external validity and generalizability. The benchmark covers a single hazard (wildfire), a single state (Arizona), and a single report template (ICS-209).

As a QA benchmark over official incident reports it has practical value, but the Conclusion's claim that LifelineQA "establishes a foundation for evaluating emergency QA across hazards, regions, and operational contexts" is not yet supported by the evidence.

The authors deliberately chose official reports over social media, on the grounds that official reports are the system of record that guides response decisions. This choice is reasonable, but it also means the benchmark sidesteps the hardest part of real emergency information environments: early information often comes from social media, news, sensors, and local announcements, and is noisy, unverified, mutually conflicting, and of uneven source credibility. Official incident reports are highly structured, and the models read semi-structured text labeled with field identifiers. A dataset intended for disaster early warning would need to account for data cleanliness and should not be limited to a single hazard type. Report structures and affected lifelines differ across hazards, so conclusions drawn from wildfires may not transfer.

If the main goal is to study the information retrieval and synthesis abilities of LLMs, the benchmark needs controlled comparisons across structured, semi-structured, and unstructured sources, as well as control variables such as source conflict, missing information, and differences in source credibility.

  1. Limited novelty at the machine learning level. Situational, temporal, comparative, causal, multi-hop, unanswerable, and evidence-grounded QA have all been studied extensively. The contribution is mainly an innovation in data organization rather than a new learning task, modeling method, or evaluation principle.

  2. No qualitative analysis of model errors. The paper reports many aggregate metrics but shows almost no concrete examples of model errors. A qualitative analysis is missing, for example of cases where a model omits part of the required evidence, or retrieves the right evidence but misinterprets it.

  3. (minor) Overlap between the question generator and the judge. Candidate QA pairs were generated by GPT-5.6 Sol, and GPT-5.6 Sol is also one of the two judges that grade model answers. This does not directly amount to self-grading, but it may couple question style with grading preferences. Expert review ensures factual quality but does not necessarily remove biases in the question distribution.

  4. Issues with the dataset itself. (1) Coverage of complex reasoning is overstated. The paper describes nine complex categories, but according to Figure 3, causal accounts for 0.1%, multi-hop 0.2%, cascading 0.2%, and interdependency 0.4% of the benchmark, about 7 questions in total. (2) The benchmark is small: only 800 questions, a single hazard, and a single state, and all examples shown appear to be 2025 incidents. The number of 800 itself is fine, it is just the diversity. (3) There are only 11 unanswerable questions, and in the gold evidence condition the unanswerable items are given an empty context, so every model has a false-answer rate of 0. The abstention analysis is therefore largely uninformative.

Questions for authors

N/A

Reviewer_h6xoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces LifelineQA, a question-answering benchmark built from official ICS-209 reports on Arizona wildfires. It contains 800 expert-reviewed items covering incident facts, community lifelines and questions that need information from several fields or report dates and every answerable item has annotated supporting evidence. Ten LLMs are tested under five evidence conditions, including gold evidence, retrieval, the full incident history and evidence from competing report dates. Models perform well with gold evidence but much worse with retrieved evidence.

Critical strengths

Each candidate was independently reviewed by two reviewers and checked by a third specialist. More than half were revised before the dataset was finalized with substantial agreement on answer correctness and evidence support. The evaluation also separates correct answers from answers backed by sufficient citations and comparing gold evidence, retrieval, and the full incident history shows where models lose accuracy when they have to find the evidence themselves.

Critical weaknesses

I think the dated reports and evidence annotations are useful, but it is not clear how much of the retrieval difficulty comes from changing report versions and how much comes from the retrieval setup. Section 4.3 uses an untuned hybrid of BM25 and bge-small that reaches 40.2% Recall@10, so part of the large gap in Table 3 may come from the retriever rather than the task. A stronger retriever or reranker would help establish whether the observed difficulty persists beyond the chosen baseline.

The selection procedure in Section 3.2.2 is intended to include questions from a range of reports and lifeline categories, but the paper does not report how many incidents and reports the final set covers or whether a few long-running incidents account for most of it. Abstention is tested on only 11 unanswerable items, which the authors acknowledge, but this is too few for reliable comparisons. It is also unclear whether the released record will include the non-gold fields needed to reproduce the retrieval and full-incident results.

Questions for authors
  1. Have you evaluated a stronger retriever or reranker? If so, does the gap between gold evidence and retrieval remain?
  2. How many incidents and reports does the final set cover, and do a few incidents account for most of the questions?
  3. Will the released record include the non-gold fields and dated report content needed to reproduce all evidence conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.