- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper introduces ResearchSpec a prompting procedure that implements explicit state tracking of task conditions (with fields for necessity, resolved status, source evidence). Assistants use this state tracking to generate a response and to assess support. Authors compare ResearchSpec to pure LLM, two step reasoning + response, and ClarifyAgent on MiP, FaithEval, ClarifyMT, and SciConvBench. These experiments show improvements over direct generation and other analyses examine answer/abstain behaviour, source links. Overall ResearchSpec seems to improve accuracy at higher inference cost.
Critical strengths
ResearchSpec maintains explicit accounts of when to answer/clarify with high granularity annotation of information gaps. Evaluation shows gains over direct generation across 4 benchmarks.
Critical weaknesses
Added novelty/value over existing structured clarification methods is unclear. The spec includes additional tracking fields such as necessity, evidence, authorization, previous assessments, but it's not necessarily clear from experiments which components drive gains. On ClarifyMT-Bench, ClarifyAgent seems more accurate and efficient. Moreover the 20 sample budget focused comparisons seems underpowered as well, though acknowledged.
Questions for authors
- Do Pure LLM and two-stage baselines receive explicit, comparable instructions to identify missing information/abstain/clarify?
- Is there additional correctness/usefulness metrics for ClarifyMT?