- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CLARPAIR, a benchmark for evaluating whether LLMs appropriately answer, clarify, or abstain when faced with underspecified or otherwise defective requests. It constructs minimal pairs of defective and repaired requests across five categories and evaluates nine model configurations. Of 162 generated pairs, 103 remain in the strict evaluation set after verification and auditing. The paper reports that models generally handle defective requests well but struggle with contradictory requests and questions requiring unavailable information.
Critical strengths
The paper addresses an important practical problem: distinguishing when an assistant should ask for clarification rather than make unsupported assumptions. Evaluating both defective and answerable requests is sensible, and the analysis of scoring sensitivity and benchmark validity provides some useful observations.
Critical weaknesses
- The novelty and overall contribution are limited.
The central idea of evaluating clarification decisions using paired defective and answerable requests is intuitive and closely related to existing work on clarification-seeking, selective answering, and counterfactual evaluation. The paper itself discusses closely related benchmarks, including CLAMBER, QuestBench, and Su & Cardie (2026).
The main methodological contribution is the construction of a relatively small synthetic benchmark and a more detailed scoring rubric. While useful, it is unclear that these constitute a sufficiently substantial advance over existing evaluation approaches. The paper does not propose a new method for improving clarification behavior or provide much new understanding of how models make these decisions.
- The benchmark is small, and its validity and generalizability remain concerns.
The original 162 pairs are reduced to only 103 in the strict evaluation set. Some categories are particularly small, such as contradictory requests (16 original pairs). Moreover, not all pairs represent clean minimal interventions. For example, transforming an unknowable personal fact into a general-knowledge question changes more than simply resolving missing information.
The entire benchmark is synthetically generated and primarily LLM-verified and LLM-judged. Although the authors perform additional audits, the final multi-field scoring rubric is evaluated using only 48 author-audited items, without independent blinded validation. The authors also acknowledge that some of these items informed the scoring rules.
Consequently, it is difficult to assess whether the reported performance differences are robust or generalize to realistic user interactions. The reported findings are mostly descriptive and do not substantially advance our understanding beyond identifying several familiar failure categories.
Questions for authors
-
What is the key scientific insight provided by CLARPAIR that cannot be obtained from existing clarification and selective-answering benchmarks, particularly those already evaluating both ambiguous and disambiguated requests?
-
How robust are the results under a larger, independently human-validated benchmark, especially for categories with very few examples?
-
To what extent do the synthetic minimal pairs capture realistic clarification decisions, where ambiguity and missing information are often context-dependent?
3 comments
Dear Reviewers,
The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!
Best, AC
Thank you for the response. I’m raising my score from 1 to 2 because the comparison using the same judge shows that a coarse rubric can misclassify useful conditional answers as guesses. This gives the paper a clearer contribution than I credited in my initial review.
My main concern remains that the final rubric was audited on a small set of responses that also informed its design. Independent validation on a fresh sample would help establish how broadly the finding holds.
We thank the reviewer for pushing on the novelty question. The delta over existing clarification benchmarks is threefold, and each part is measured rather than asserted:
1. Joint pair metric on a same-denominator four-cell decomposition. Prior paired designs (CLAMBER, QuestBench, Knowing-but-Not-Showing) score the two sides separately or coarsely. We report n11/n10/n01/n00 on the same strict-core pairs, plus the conditional rate P(flawed handled | twin correct) = 88–96% and a missing-data sensitivity range per model. This is what makes the dominant failure cell visible: n01 — twin solved, defect ignored — pooling to 68 items concentrated in contradictory (28) and missing-var (26). Marginal-only metrics cannot see this.
2. The scoring contract is itself a measured variable. On identical responses, the coarse single-label rubric and the multi-field contract produce materially different conclusions; we decompose the shift into contract vs. judge vs. coverage. Concretely, of the 213 flawed-side responses that v1 labeled
answered, v2 finds true committed guesses on only 114 — 99 (47%) were legitimate conditional help the single label could not express. And under a fixed judge on the same items, the rubric contrast alone is 7/20 vs. 2/20. No prior clarification benchmark quantifies how its own label contract moves its headline numbers; here it changes the qualitative claim (the v1 "up to 47% fabricated" figure on unknowable items collapses to ≤11%).3. Answerability partitioning. Twins that silently require live information are flagged by a stricter, different-family verifier and scored in a separate slice rather than mislabeled as over-clarification — the 41-item environment-dependent slice is reported alongside, not dropped. This confound plausibly affects other LLM-drafted answerability benchmarks, which is itself a finding; the flagging stage transfers to any benchmark whose "answerable" items may secretly require tools.
On validation depth. We cannot offer independent blinded human validation we did not perform. What we offer instead is a bounded sensitivity analysis plus full release: under the audit-corrected labels, exactly 2 of the 27 headline cells move (twin-correct 89.5→90.4% and joint 81.5→82.6% on one row); judge-vs.-audit-corrected outcome agreement over all 2,750 responses is 99.96%; and the label noise the audit did find (field disputes on 9/48 items) changes a derived outcome in exactly one case. Every label, flag, and override is released, so any group can re-score the benchmark under their own contract — which is itself part of the contribution.