- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LifelineQA, a benchmark with 800 items for evidence-grounded question answering over Arizona wildfire incident reports. It contains situational, lifeline-specific, and complex questions, with expert-reviewed answers and supporting evidences. 10 LLMs are evaluated under five evidence conditions to study answer correctness, evidence support, abstention, and temporal robustness.
Critical strengths
The expert review is a substantive part of this work. Each candidate was independently reviewed by two reviewers, followed by specialist adjudication, and 476 items were revised before finalization. This is more useful than simply releasing LLM generated questions without checking them. Linking answers to specific report fields also provides a concrete basis for inspecting their support.
Critical weaknesses
- Some reliability metrics have problems that might affect their interpretation. Appendix A2.2 defines supported-claim precision as entailed / (entailed + contradicted), excluding neutral claims. Therefore, an answer containing one supported claim and many supported, non-contradicted claims could still received perfect support precision, if I understand this correctly. Similarly, over-abstention is defined using benchmark-level answer-ability, altho models are instructed to answer only from the supplied evidence. Under retrieval, refusing because the required evidence was not retrieved can then be counted as over-abstention. This can measure lost end2end answer coverage, but does not, by itself, establish inappropriate refusal by the reader.
- The retrieval results do not sufficiently separate benchmark difficulty from baseline limitations. The questions explicitly identify the incident and relevant dates, and retrieval chunks already contain report-date and field metadata. But the evaluated retrievers are BM25, a dense retriever, and their hybrid, without a reported date-filtered comparison. The hybrid's Recall score establishes a limitation of that setup, but a simple baseline that first selects the requested reporting dates would help show how much difficulty remains.
- The stated selection ILP does not enforce its intended coverage objective. Appendix A1.2 maximizes document and incident coverage, but the constraints allow a document to be marked as selected even when no question is drawn from it.
Questions for authors
Q1: does the implementation exclude neutral claims from supported-claim precision, as the formula states? How do the conclusions change when neutral claims count as unsupported and abstention is separately assessed against the evidence actually supplied?
Q2: how much does retrieval improve when reports are first filtered by the dates explicitly requested in the question, suing the same evidence budget? Which question types remain difficult after this control?
Q3: does the implemented ILP contain a constraint linking each marked document to at least one selected question?