acceptodds
Under review as a conference paper at ICLR 2027

Ask or Guess? A Minimal-Pair Benchmark for Clarification-Seeking in LLMs

Abstract

Deployed assistants receive requests that are underspecified, ambiguous, premised on false facts, contradictory, or unknowable. For many of these, the reliable re- sponse is not a refusal but a question. Abstention benchmarks measure whether models decline to answer; interactive benchmarks measure whether they can clar- ify across multi-turn dialogue. Harder to compare is the single-turn decision boundary itself — whether a model answers, asks, or declines on a defective re- quest, whether it over-clarifies on requests that were answerable to begin with, and what even counts as a correct response on each side. We introduce CLARPAIR, a benchmark of 162 paired requests spanning five defect types: each defective request is paired with a twin that differs only by the missing piece, so answer- ing, clarifying, and abstaining can be scored jointly without confounding task difficulty. Pairs are LLM-drafted, verifier-filtered, and re-verified by a stricter, different-family verifier whose answerability test flags 40 twins that silently re- quire live information — a slice we score separately rather than mislabel as over- clarification. We score responses under an explicit behavior contract — naming the defect, asking back, correcting a premise, and conditional help all count as ap- propriate handling; only committing to an unsupported specific answer counts as a guess. Evaluating nine model configurations across five families and two access regimes, we find defective-request handling at 87–94% and correct answers on 90– 97% of verified-answerable twins, yet joint pair success — both sides right on the same item — reaches only 80–90%. Failures concentrate where the contract bites: on contradictory requests models silently resolve the conflict instead of surfacing it (as low as 17% appropriate), and on requests that only tools could answer, text-only API models commit unsupported answers at 39–56% (62–68% for tool-equipped agents, some plausibly legitimate retrievals). Twin-side errors decompose into rare over-clarification (2–8%) and a newly visible class of wrong answers (0–8%). Un- der a coarser single-label rubric the same responses looked substantially worse, which is itself a finding: the scoring contract materially changes measured conclu- sions. We release the benchmark, the three-stage verification pipeline, all behavior labels, and the scoring harness.

Primary area:
datasets and benchmarks
Keywords:
clarification seeking, abstention, hallucination, benchmark, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZNTJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CLARPAIR, a benchmark for evaluating whether LLMs appropriately answer, clarify, or abstain when faced with underspecified or otherwise defective requests. It constructs minimal pairs of defective and repaired requests across five categories and evaluates nine model configurations. Of 162 generated pairs, 103 remain in the strict evaluation set after verification and auditing. The paper reports that models generally handle defective requests well but struggle with contradictory requests and questions requiring unavailable information.

Critical strengths

The paper addresses an important practical problem: distinguishing when an assistant should ask for clarification rather than make unsupported assumptions. Evaluating both defective and answerable requests is sensible, and the analysis of scoring sensitivity and benchmark validity provides some useful observations.

Critical weaknesses
  1. The novelty and overall contribution are limited.

The central idea of evaluating clarification decisions using paired defective and answerable requests is intuitive and closely related to existing work on clarification-seeking, selective answering, and counterfactual evaluation. The paper itself discusses closely related benchmarks, including CLAMBER, QuestBench, and Su & Cardie (2026).

The main methodological contribution is the construction of a relatively small synthetic benchmark and a more detailed scoring rubric. While useful, it is unclear that these constitute a sufficiently substantial advance over existing evaluation approaches. The paper does not propose a new method for improving clarification behavior or provide much new understanding of how models make these decisions.

  1. The benchmark is small, and its validity and generalizability remain concerns.

The original 162 pairs are reduced to only 103 in the strict evaluation set. Some categories are particularly small, such as contradictory requests (16 original pairs). Moreover, not all pairs represent clean minimal interventions. For example, transforming an unknowable personal fact into a general-knowledge question changes more than simply resolving missing information.

The entire benchmark is synthetically generated and primarily LLM-verified and LLM-judged. Although the authors perform additional audits, the final multi-field scoring rubric is evaluated using only 48 author-audited items, without independent blinded validation. The authors also acknowledge that some of these items informed the scoring rules.

Consequently, it is difficult to assess whether the reported performance differences are robust or generalize to realistic user interactions. The reported findings are mostly descriptive and do not substantially advance our understanding beyond identifying several familiar failure categories.

Questions for authors
  • What is the key scientific insight provided by CLARPAIR that cannot be obtained from existing clarification and selective-answering benchmarks, particularly those already evaluating both ambiguous and disambiguated requests?

  • How robust are the results under a larger, independently human-validated benchmark, especially for categories with very few examples?

  • To what extent do the synthetic minimal pairs capture realistic clarification decisions, where ambiguity and missing information are often context-dependent?

3 comments

We thank the reviewer for pushing on the novelty question. The delta over existing clarification benchmarks is threefold, and each part is measured rather than asserted:

1. Joint pair metric on a same-denominator four-cell decomposition. Prior paired designs (CLAMBER, QuestBench, Knowing-but-Not-Showing) score the two sides separately or coarsely. We report n11/n10/n01/n00 on the same strict-core pairs, plus the conditional rate P(flawed handled | twin correct) = 88–96% and a missing-data sensitivity range per model. This is what makes the dominant failure cell visible: n01 — twin solved, defect ignored — pooling to 68 items concentrated in contradictory (28) and missing-var (26). Marginal-only metrics cannot see this.

2. The scoring contract is itself a measured variable. On identical responses, the coarse single-label rubric and the multi-field contract produce materially different conclusions; we decompose the shift into contract vs. judge vs. coverage. Concretely, of the 213 flawed-side responses that v1 labeled answered, v2 finds true committed guesses on only 114 — 99 (47%) were legitimate conditional help the single label could not express. And under a fixed judge on the same items, the rubric contrast alone is 7/20 vs. 2/20. No prior clarification benchmark quantifies how its own label contract moves its headline numbers; here it changes the qualitative claim (the v1 "up to 47% fabricated" figure on unknowable items collapses to ≤11%).

3. Answerability partitioning. Twins that silently require live information are flagged by a stricter, different-family verifier and scored in a separate slice rather than mislabeled as over-clarification — the 41-item environment-dependent slice is reported alongside, not dropped. This confound plausibly affects other LLM-drafted answerability benchmarks, which is itself a finding; the flagging stage transfers to any benchmark whose "answerable" items may secretly require tools.

On validation depth. We cannot offer independent blinded human validation we did not perform. What we offer instead is a bounded sensitivity analysis plus full release: under the audit-corrected labels, exactly 2 of the 27 headline cells move (twin-correct 89.5→90.4% and joint 81.5→82.6% on one row); judge-vs.-audit-corrected outcome agreement over all 2,750 responses is 99.96%; and the label noise the audit did find (field disputes on 9/48 items) changes a derived outcome in exactly one case. Every label, flag, and override is released, so any group can re-score the benchmark under their own contract — which is itself part of the contribution.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Thank you for the response. I’m raising my score from 1 to 2 because the comparison using the same judge shows that a coarse rubric can misclassify useful conditional answers as guesses. This gives the paper a clearer contribution than I credited in my initial review.

My main concern remains that the final rubric was audited on a small set of responses that also informed its design. Independent validation on a fresh sample would help establish how broadly the finding holds.

Reviewer_UN4AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I do not believe this submission meets the bar for the ICLR Datasets & Benchmarks track in its current form. This paper should be desk-rejected! The paper has a very poor presentation, it is not acceptable that a benchmark paper has zero figures and only one table for experiments. All reported findings derive from a single run of nine configurations under one neutral system prompt with one sample per item. The model selection leans on lightweight variants, lacks frontier flagship models, and overlaps substantially with the drafting, verification, and judging pipeline (four evaluated models also serve as drafter, verifier, or judge), which undermines confidence in the results. I would encourage the authors to substantially expand the experimental section and carefully reorganize the presentation before resubmission.

Critical strengths

Please see the summary.

Critical weaknesses

Please see the summary.

Questions for authors

Please see the summary.

3 comments

We thank the reviewer for the direct criticisms; several are fair and we address each in turn.

Presentation. Fair — the experiments section is table-dense. In revision we will add a figure (per-category joint-success heatmap and/or the four-cell joint decomposition as a graphic) and move secondary cells to the appendix.

Single run / one sample per item. True, and disclosed in Limitations; decoding settings are in Appendix A. ClarPair is a measurement study of a fixed contract, not a capability ranking, so we mitigate single-sample noise structurally rather than by claiming per-item response stability: every cell is reported as k/n with Wilson intervals, and label noise is bounded by two further judge passes — a different-family cross-judge on a 1,500-item subset (90.3% agreement) and a re-judge of all 324 responses of the self-conflicted model (86.4%). We agree a multi-seed response pass would strengthen the study and list it as future work.

Judge/evaluated-model overlap. Disclosed — the v2 judge (DeepSeek-V4-Pro) is itself an evaluated model and its row is marked † in Table 1. The paper decomposes the judge contribution explicitly rather than asserting neutrality: v1-vs-v2 rubric on identical responses, a same-judge cross-rubric contrast (7/20 vs 2/20 on the same items), and a judge-effect direction check under v1 (self-judge 3/20 vs. different-family cross-judge 7/20). This is the strongest available evidence short of a fully disjoint judge, which we list as future work.

Coverage / scale. Nine configurations is modest but deliberate. The raw artifacts actually contain 13 model configurations; the four OpenRouter rows (17–56 judged responses each) were excluded under a pre-stated inclusion rule for coverage fragmentation, not for their scores. The contribution is the measurement design — minimal pairs, the joint metric, the contract sensitivity analysis, and the live-info confound — which does not stand or fall on adding more leaderboard rows. Expanding the evaluated set is future work, not a precondition.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

The author said everything in the future work, and do not do any modification for the current version.

NEVER EDIT THE PDF

I highly recommend that authors submit the future paper in the next round of submissions.

Reviewer_oLF4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CLARPAIR, a benchmark of 162 paired requests covering five types of defective input. Each request has a repaired twin, with the goal of measuring whether an assistant handles the defect appropriately while still answering when enough information is available. Verification and auditing yield a strict core of 103 pairs. Across nine model configurations, the authors report 87–94% appropriate handling, 90–97% accuracy on answerable twins, and 80–90% joint success. The paper also examines how answerability assumptions and scoring rules affect these conclusions, using multiple LLM judges and human audits.

Critical strengths

I think the paired evaluation is the most useful part of this work. An assistant can avoid unsupported answers by asking about everything, but that would hardly make it a good assistant! Having closely related defective and answerable requests gives us a way to evaluate both sides of this problem. In particular, I find joint success a useful diagnostic, since the model needs to handle the defective request and solve its repaired counterpart.

I also found the discussion of scoring quite informative. A response that explains the missing information and gives a conditional formula can be perfectly reasonable, and grouping it with unsupported guesses would give us a misleading picture. I appreciate that the authors examined these distinctions and openly reported problems discovered in their own benchmark. In my opinion, this analysis provides useful lessons beyond this particular dataset.

Critical weaknesses

My first concern is about what the strict-core minimality criterion actually guarantees. In Table 1, changing “How many championships has he won?” into a question about Messi’s FIFA World Cup championships seems to resolve both the person and the scope of the statistic. Similarly, asking about a grandmother’s private ingredient and asking for common pie thickeners seem to be somewhat different tasks. I appreciate that the authors explicitly acknowledge the weaker minimality of unknowable pairs, but could they help me understand why these examples still belong to the strict core? More generally, do the main conclusions depend on pairs that change the requested task or answer scope? I may be interpreting minimality too strictly here, but since controlled pairing is central to the contribution, I would be very happy if the authors could help resolve this concern.

My other concern is about validation of the final scoring procedure. My understanding is that the newer multi-field rubric was audited on 48 selected responses by the authors, without blinding to model identity, and that this audit also informed the rules and overrides. The larger audits primarily concern the earlier rubric. Given how much the scoring changes the conclusions, do we have independent evidence that the final appropriate/inappropriate decisions are reliable? In particular, could the authors clarify agreement on the final outcomes, rather than only individual fields, and how sensitive the headline results are to disputed labels? I may have missed relevant evidence in the supplementary material, and would appreciate being pointed to it.

Questions for authors

I would appreciate the authors’ responses to the questions and concerns raised in the main review above.

2 comments

We thank the reviewer for the careful reading and the two well-posed concerns. Both are answerable with analyses we have now run end-to-end on the released labels (the script regenerating every number below is included in the supplementary material).

Q1. Do the conclusions depend on pairs whose twin changes task/answer scope (strict-core minimality)?

Two parts to the answer.

(a) What the pairing actually controls. The minimal-pair design controls for topic and surface form, not for identical answer scope — and we say so explicitly in the Table 1 caption and §3. For unknowable pairs the twin must ask a general-knowledge question precisely because the defect is that the requested fact is private; no repair can keep "my grandmother's secret ingredient" while remaining answerable. The measured quantity is therefore not "same answer, different phrasing" but "does the model notice which side of the answerability boundary it is on" — which is exactly the claim the joint metric makes.

On the specific examples: the Messi pair (p033, ambiguous_ref) resolves referent and scope — but that is what a repair is; the pair still shares topic (football championships) and differs only in whether the needed referent is recoverable. The grandmother-pie pair (p127, unknowable) asks about common thickeners — a different sub-question than the private recipe, yes; the diagnostic is whether the model treats "unknowable as stated" vs. "answerable general question" differently. Both IDs are in the released item list. The per-category strict-core breakdown (missing_var 52, false_premise 17, ambiguous_ref 16, unknowable 10, contradictory 8) is produced by the released scoring script; we will also print it in the appendix at revision.

(b) Quantified sensitivity — the headline numbers do not depend on these pairs. We recomputed every reported metric with whole categories removed from scoring (unknowable is 10/103 strict-core pairs and is the family with the structural scope change). Ranges over the nine configurations:

scored sliceflawed appropriatetwin correctjoint
full strict core87.4–93.5%90.4–96.6%80.2–90.1%
− unknowable86.5–93.3%89.4–96.1%80.5–90.1%
− unknowable − ambiguous_ref82.0–90.9%89.4–100%*79.7–96.2%*

* upper bound is the partial-coverage Doubao-2.0-Lite row (n=32 twins, 26 joint pairs); among the eight full-coverage models the ranges are 89.4–96.4% (twin) and 79.7–89.3% (joint).

On the first cut every model moves ≤1.7pp on every metric. On the second cut, every full-coverage model still shows the same qualitative pattern: contradictory remains the weakest cell for all eight, joint remains below both marginals for all eight, and the dominant discordant cell remains n01 (twin correct, flawed side mishandled), now concentrated entirely in contradictory + missing-var. The ordering is essentially preserved — SWE-2's lead and the API cluster are unchanged.

We also stress that the pipeline quarantines rather than hides these cases: 5 knowledge-defeatable "defects" were removed from flawed scoring, 40 twins requiring live information were moved to a separately-reported environment-dependent slice, and p075t's residual ambiguity was moved to contested. None of these removals is post-hoc on the outcome — the flags are part of the released manifest.

Q2. Validation of the final scoring procedure.

We agree the audit is a development audit — author-labeled, unblinded, and the audited items informed rule development; §4.2 says exactly this and we do not claim it as independent validation. What we can add is the honest bound on what that caveat costs, i.e., outcome-level agreement and headline sensitivity to the disputed labels:

  • Outcome agreement on the 48 audited items: 47/48 derived outcomes are identical between judge labels and audit-corrected labels; the single flip (p109t twin, wrong→correct) is itself the confirmed mislabel the audit caught.
  • Headline sensitivity: judge-only vs. judge+overrides differs in exactly 2 of the 27 headline cells — both on the Doubao-2.0-Code-nothink row (twin-correct 89.5→90.4%, joint 81.5→82.6%). All other cells are bit-identical.
  • Whole-corpus bound: across all 2,750 judged responses, judge-vs- audit-corrected outcome agreement is 2,749/2,750 = 99.96%.
  • Field-level agreement (already reported): 333/336 = 99.1% confirmed, 326/336 = 97.0% counting the seven boundary disputes as errors; item-level all-fields-correct 39/48 = 81.3% — i.e., field disputes are real but almost never change a derived outcome.

So the label noise the reviewer is worried about exists, but it cannot move any conclusion in the paper. A blinded, uniformly random audit on frozen labels remains future work and is listed as such in Limitations.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.