- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-hop QA agents that read the right paragraphs but still answer wrong, and argues that retry-based correction (an agentic port of SSR, and Reflexion) is costly and often unnecessary.
The proposed EIR runs one retrieval trajectory where each step updates a requirement list and an evidence-citing conclusion list and picks the next paragraph, while the verbatim sentences go into a separate store.
A fresh final call then answers from only the question and the stored sentences, without the lists or the provisional answer. The authors call this the evidence-inference boundary.
Experiments use 1,000-question subsets of HotpotQA, 2Wiki and MuSiQue in the distractor setting with Claude Haiku 4.5 and GPT-4.1 Mini, plus a corpus-search extension.
EIR beats Direct, Agentic SSR and oracle-stopped Reflexion on answer F1 at about 4.85 calls per question, and an error analysis separates incomplete-evidence errors from inference errors.
The evaluation is careful, but the core design, answering with a separate reader over everything collected, was already proposed and ablated by IRCoT, and the paper's own controls suggest that's where the gain comes from.
Critical strengths
-
Careful, paired and transparent evaluation. All methods run on the same questions with paired bootstrap CIs, exact McNemar tests and Holm correction, costs are reported next to accuracy, and negative results (the non-significant Haiku MuSiQue case, the corpus-setting loss) are reported openly. This matters because the measurements themselves can be trusted, even where I disagree with the interpretation.
-
A useful diagnostic that separates retrieval failures from inference failures. Supplying the missing gold paragraphs recovers about 27% of incomplete-evidence errors, while re-answering on the same evidence recovers about 1%. It shows that retrieval completeness, not retrying, is the main lever, and others can reuse the decomposition.
Critical weaknesses
-
The core idea isn't new, and the closest prior work isn't cited. IRCoT (Trivedi et al., ACL 2023) already answers with a separate reader over all collected paragraphs on the same three benchmarks, and reports that taking the answer from the retrieval-time reasoning works worse. Checklist-style gap tracking is in FAIR-RAG (cited in one line), and PRISM, ReSP and Search-o1 also separate evidence collection from answering. Related work is one paragraph, and the only competitors are the authors' own port of a math self-refine method and 2023 Reflexion. This matters because the 'new agentic paradigm' framing doesn't hold up, and beating weak retry baselines doesn't show EIR beats existing retrieve-then-read agents.
-
The claimed mechanism is never isolated, and the baselines look handicapped. Direct's action prompt seems to hold only the latest paragraph, it reads only about 2 to 3 paragraphs on average, and it gets a smaller output budget and stricter parsing. The paper's own controls point away from the boundary: a single Full Pool call over all provided paragraphs nearly ties EIR on average at about a quarter of the cost, and the list-feedback control keeps most of EIR's gain over Direct. No variant gives the final call the lists or the provisional answer, which is the ablation the thesis needs. If the gain is just 'let the answerer see all the text', the result is expected. The abstract also overclaims, e.g. it counts the non-significant +1.1 over Reflexion as an improvement.
Questions for authors
- In the main distractor evaluation, what exactly is in the action prompts of Direct, Agentic SSR and Reflexion: only the latest paragraph, or all previously read paragraphs? Do they use the same output limits and the same lenient action extraction as EIR?