acceptodds
Under review as a conference paper at ICLR 2027

ClarifyCodeBench: Evaluating LLMs on Clarifying Underspecified Requirements for Code Generation

Abstract

Large language models (LLMs) have emerged as powerful programming assistants. However, the efficacy of code generation is fundamentally constrained by the quality of input requirements, which, in real-world software development, are frequently underspecified: they omit information needed to determine the intended behavior. Although LLMs excel at one-shot code synthesis, their ability to proactively clarify user intent remains underexplored despite its importance to robust software engineering. Existing benchmarks largely overlook this interactive bottleneck, using perfectly specified prompts that do not reflect the iterative nature of requirement elicitation. To bridge this gap, we introduce ClarifyCodeBench, a novel interactive benchmark specifically designed to evaluate LLMs' capability in resolving requirement underspecification. Constructed from real world programming tasks, ClarifyCodeBench features high-quality manual annotations, including 11 unique underspecification types, associated clarification questions, and corresponding ground-truth answers. Furthermore, we formalize two rigorous metrics to assess the interaction quality: Key Question Rate (KQR), which measures the fraction of key questions that a model asks about, and Turn-discounted Key Question Rate (TKQR), which also penalizes inefficient questioning. We conduct a systematic evaluation of four current LLMs using ClarifyCodeBench. Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to strong requirement clarification efficacy; 2) The Reasoning Paradox: While increasing computation in reasoning models enhances code correctness, it reduces clarification coverage; 3) The Multi-underspecification Ceiling: LLMs' clarification performance degrades sharply as the density of underspecification increases, revealing a significant bottleneck in handling complex, real-world specifications. Our work emphasizes the necessity for future research to move from static synthesis to interactive requirement elicitation.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Code Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RNpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether coding agents ask for missing information before writing code. The authors build ClarifyCodeBench by deleting one to three details from 80 LiveCodeBench problems, giving 419 underspecified variants. Each omission has a key question, a reference answer, and one of 11 underspecification types. Agents may ask one question per turn. An LLM judge returns the reference answer for matched questions and a neutral reply otherwise. Across six standalone configurations, Claude Code and ClarifyGPT ask more when information is missing but cover few omissions. Coverage falls as omissions increase. Enabling reasoning raises pass@1 but lowers coverage.

Critical strengths
  • The topic is well motivated, and the paper constructs a useful benchmark for future study.
  • The evaluation framework is comprehensive. It separates whether to ask, what to ask, how early to ask, and the effect on correctness, revealing which stage of clarification fails.
  • They covered a wide range of recent models, including reasoning and non-reasoning models.
  • The 11 underspecification types, grounded in requirements engineering standards, allow analysis by the kind of missing information.
Critical weaknesses
  • The novelty is incremental. ClarEval already measures intent coverage and dialogue efficiency. The main additions are per-omission types and a varying number of omissions.
  • The paper does not show that the deleted details are needed to solve the tasks. Its validation only checks whether the reference answer changes the model's output, not whether the answer is needed to pass the tests. The validation result is also not reported. Without clarification, pass@1 is only 4–15 points below that with all answers given.
  • The underspecification is artificial. Annotators create it by deleting details from competitive programming problems. The paper never checks whether these omissions resemble those in naturally underspecified requirements.
  • The LLM judge is validated only against other LLMs, with no human validation. Its agreement on assumption matching is lower (κ 0.67–0.72).
Questions for authors
  • Two PhD students annotate each task and resolve disagreements through discussion, but no agreement is reported. How often did they agree before discussion on the deletions, the key questions and reference answers, and the underspecification types?
1 comment

Thank you for recognizing that our topic is "well motivated" and our evaluation framework is "comprehensive."

Q. Annotator agreement.

Both annotators independently annotated all 560 candidate tasks and agreed on 247 of them (44.1%) before discussion. Disagreements mainly concerned whether the edited problem statement still read naturally and whether the deletion introduced unintended underspecification elsewhere in the task. After discussing the remaining 313 candidates, we retained a further 172 and excluded 141, yielding 419 tasks in total.

W1. Novelty relative to ClarEval.

ClarifyCodeBench evaluates two aspects of effective clarification:

1. Knowing when to ask. We compare fully specified and underspecified versions of the same task to measure whether agents ask when a requirement is missing and refrain from asking when the requirements are complete.

2. Prioritizing key questions. ClarEval allows agents to ask multiple questions in one turn, so unrelated questions bundled with a key question incur no additional turn cost. We instruct agents to ask one question per turn. Under this protocol, each unrelated question asked before a key question delays it by one turn and lowers its contribution to TKQR.

Controlled comparisons. We construct variants of the same task by deleting specific requirements while keeping the rest of the prompt and the original tests fixed. This lets us compare how often agents ask about the same detail when it is missing alone or alongside other omissions, and measure how restoring missing information affects pass@1 on the same tests.

W2. The paper does not show that the deleted details are needed to solve the tasks.

We evaluate clarification separately from code generation. Agents often proceed directly to code by guessing a missing detail. When the guess matches the intended behavior, the code can pass the original tests even though the agent never clarified the requirement. Prior exposure to public LiveCodeBench problems could also narrow the observed pass@1 gap.

We constructed ClarifyCodeBench-CF, a set of 88 counterfactual tasks. For each task, we kept the underspecified prompt unchanged, so exactly the same detail was missing. We then wrote a different plausible answer to the corresponding clarification question and new tests for the behavior specified by that answer.

GPT-5.6-sol on ClarifyCodeBench-CF

ConditionPass@1
Direct15.9%
Interactive39.8%
Oracle97.7%

Clarification behavior in the Interactive condition

PrecisionRecallF1KQRTKQR
0.9200.2610.4070.2610.261

Asking for clarification improved pass@1 by 23.9 percentage points, and providing all missing information raised it to 97.7%. Because the underspecified prompts stayed the same, the large gap in pass@1 between direct generation at 15.9% and the oracle condition at 97.7% shows that the agent needs the missing details to reliably implement the user's intended behavior.

W3. Do the omissions in ClarifyCodeBench resemble those in naturally underspecified requirements?

Naturalness was an explicit annotation criterion. The annotators discussed whether each deletion left a request that a user might plausibly make and excluded variants they judged unnatural.

Two additional PhD students reviewed a random sample of 50 retained prompts and agreed that 98% read naturally. This check supports the naturalness of the resulting prompts.

W4. The LLM judge is validated only against other LLMs, with no human validation.

We manually reviewed all 368 unmatched questions and 100 of the 482 matched questions, finding 91% and 96% agreement with GPT-4o, respectively.

Overall

Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.

Reviewer_VjjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is interested in the problem of coding agents asking clarifying questions when the coding task is known to be underspecified. For this, the authors prepare a small benchmark of 80 competitive problems that have been manually edited to remove 1, 2, or 3 important pieces of information required to solve the task. Then coding agents can be evaluated on their ability to ask the correct clarifying questions, to ask them early in the interaction, and to not ask other unnecessary questions. The paper finds that current systems do not perform well in this task.

Critical strengths

The paper has a clear goal, well formulated and clearly useful. The design of the benchmark is simple, with manual edits to create underspecified scenarios central to its evaluation. The finding that models are not asking many questions, and not benefiting a lot from asked questions, is interesting. Since the paper shows that pure coding strength does not strongly correlate with good ability to clarify questions this could set specific target for improving future coding agents.

Critical weaknesses

I feel that the experimental analysis of the paper is a bit chaotic and not well-structured. Restructuring it around more coherent RQs and clear metrics would benefit the paper.

In particular for discrimination, why not measure the number of extra questions in underspecified scenarios? It seems a bit surprising to count the number of scenarios where the model asks at least one question in the control and underspecified group (H and F in the paper) and not count the number of questions? Intuitively, one cares about how many unnecessary questions the model will ask in both underspecified and fully specified scenarios. The setup already implicitly assumes that every question asked that is within the set K is a "hit" and every question asked that is not within the set K is a "miss". While this could be debated, this choice would imply precision / recall / F1 as the most natural metrics. Yet, these are not the standard metrics being reported, except, surprisingly, for Table 4. Also, table 4 is particularly surprising because it evaluates Claude Code variants with different metrics than the previous 6 settings. Why not the same metrics for all systems? Then Claude Code variants disappear from the rest of the experimental analysis.

More importantly for the longer term viability of the benchmark: It is not clear that problems from competitive programming with removed information are a good example of use cases where clarifying questions are important. Typical scenarios of software engineering where coding agents are used would involve a much larger design space where ambiguity is inherent and many possible design choices exist. There would never be a "fully specified scenario". Competitive coding has this very specific style which models know to be fully specified, which would prompt them to answer directly instead of asking clarifying questions, making the problems observed in the paper overestimates. Furthermore, the problems considered here can easily have leaked into the training / post-training of various current and future models, making it unclear whether models are evaluated on their ability to ask questions or on their ability to remember what the missing information was.

"Two PhD students independently annotate every task and cross-check the edits, questions, and reference answers, resolving disagreements through discussion." It would be interesting and important to comment on the level of disagreement here and the amount of effort it took.

Minor:

  • In Figure 1, it seems the color code is not consistent. For example, the Selectivity / Completeness / Efficiency is not related to the Ask / Assume / Code separation.
  • For LLM judges' evaluation in Section 6.2, I think the most important thing to measure is not agreement between different judges, but a small-scale manual meta-evaluation to tell us how often the LLM judges are wrong.
  • The paper lacks a proper grounding in a larger set of previous works that study interactive coding agents, e.g., (Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks. Rontogiannis et al), (IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering. Li et al), (KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models. Yu et al), just to mention a few.
Questions for authors
  • See questions above about the choice of metrics to measure the discrimination
  • Why not report results with Claude Code for all experiments?
  • Can you give more information about the disagreement between humans creating the edits?
2 comments

Thank you for highlighting our clear research goal, which you describe as "well formulated and clearly useful," and the simplicity of our benchmark design. As you note, our finding that strong coding ability does not reliably translate into effective clarification points to a concrete target for improving future coding agents.

Q1. The choice of metrics to measure the discrimination.

Thank you for your suggestion. We will reorganize the experimental analysis and report the same metrics in the revised paper.

For the table 3 in our paper, we report recall, precision, and F1 on underspecified tasks. We also report the number of unnecessary questions and the total number of questions asked on both underspecified and fully specified tasks.

AgentReasoningRecallPrecisionF1Unnecessary / Total on underspecified tasksUnnecessary / Total on fully specified tasks
Claude Opus 4.8On0.0770.7260.13920/730/0
Claude Opus 4.8Off0.2190.4940.303158/31222/22
GPT-5.5On0.0580.7550.10813/530/0
GPT-5.5Off0.0970.7050.17128/950/0
Gemini 3.1 ProOn0.1780.5490.269101/2247/7
GPT-5 miniOn0.0620.4840.11048/9312/12
Overall0.1150.5670.192368/85041/41

Q2. Why not report results with Claude Code for all experiments?

Our main experiments evaluate agents without harness. We selected Claude Code as a strong coding harness to test whether the low clarification coverage we observed also persists when the model and harness operate together.

Q3. Can you give more information about the disagreement between humans creating the edits?

Both annotators independently annotated all 560 candidate tasks and agreed on 247 of them (44.1%) before discussion. Disagreements mainly concerned whether the edited problem statement still read naturally and whether the deletion introduced unintended underspecification elsewhere in the task. After discussing the remaining 313 candidates, we retained a further 172 and excluded 141, yielding 419 tasks in total. Independent annotation and joint review took approximately 140 person-hours combined.

W. Whether competitive programming tasks reflect real-world clarification needs and whether data leakage could let models recall the missing information.

We agree that real-world software development often leaves important design choices open, making it valuable for coding agents to know when to ask for clarification. As a first step, ClarifyCodeBench uses competitive programming tasks so we can isolate specific missing requirements and test whether the code satisfies them. In future work, we plan to explore real-world development tasks, where it is harder to control which requirements are missing and to evaluate how well agents clarify them.

Because ClarifyCodeBench is built from public LiveCodeBench problems, agents may have seen the original statements during training. An agent that recalls the missing detail can achieve a high pass@1 on the original tests without asking for clarification. We therefore constructed ClarifyCodeBench-CF, a set of 88 counterfactual tasks, to measure how much clarification improves code correctness. For each task, we keep the underspecified prompt unchanged, provide a different plausible answer when the agent asks, and evaluate its code against new tests that reflect that answer.

GPT-5.6-sol on ClarifyCodeBench-CF

ConditionPass@1
Direct15.9%
Interactive39.8%
Oracle97.7%

Clarification behavior in the Interactive condition

PrecisionRecallF1KQRTKQR
0.9200.2610.4070.2610.261

ClarifyCodeBench-CF further highlights the value of clarification. Allowing GPT-5.6-sol to ask questions raised pass@1 from 15.9% to 39.8%, still far below the Oracle result of 97.7%.

Minor W.

Thank you for these suggestions. We will revise the color coding in Figure 1 to distinguish evaluation metrics from agent actions and discuss the cited studies in the related work section.

We manually reviewed all 368 unmatched questions and 100 of the 482 matched questions, finding 91% and 96% agreement with GPT-4o, respectively.

Overall

Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.

Thank you for the detailed and substantive rebuttal. I appreciate the additional experiments and clarifications.

The revised evaluation metrics seem more natural, and the manual validation of the LLM judges is a bit reassuring. The additional information about the annotation process is useful, although the relatively low initial agreement (44.1%) still shows the difficulty of constructing these tasks and raises some questions about the reliability of the resulting benchmark.

My main remaining concern is the representativeness of the benchmark and what conclusions can be drawn from it. While competitive programming provides a convenient controlled setting, clarification in real-world software engineering often involves resolving genuinely open design choices rather than recovering specific missing requirements. This matters because of the amount of related work targetting similar problem, a new benchmark should bring better matching with real-world scenarios.

Overall, I will maintain my score.

Reviewer_nrxCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ClarifyCodeBench, an interactive benchmark for evaluating whether coding agents identify and clarify underspecified requirements before generating code. The benchmark is constructed by deleting information from competitive programming problems and annotating the resulting omissions with clarification questions, reference answers, and underspecification types. The evaluation measures clarification selectivity, coverage, and question timing, alongside functional correctness under direct, interactive, and oracle conditions. The results suggest that strong coding performance does not necessarily translate into effective clarification.

Critical strengths
  1. The paper addresses an important practical limitation of coding agents: generating correct code requires understanding the user’s intended behavior, which may not be fully specified in the initial request.

  2. The annotated omissions and interactive comparisons provide a useful framework for examining clarification behavior separately from code-generation ability.

Critical weaknesses
  1. In Section 4.1, questions receive informative answers only when they match the predefined annotations; other questions receive a neutral reply. This may penalize reasonable clarification questions outside the annotation list. Section 6.2 reports agreement between LLM judges, but agreement does not establish that their decisions are correct. Human checks of rejected questions would help address this concern.

  2. Section 5.3 and Table 3 show that agents are less likely to identify all missing details when more details are removed. However, finding all three missing details is naturally harder than finding one. Table 5 provides additional evidence about question counts, but does not directly show whether each missing detail becomes harder to identify. Comparing the same missing detail alone and alongside other omissions would better support this conclusion.

Questions for authors

How does the answer-returning protocol handle useful questions outside the annotation set, or a single question that resolves multiple omissions? Has a sample of matching decisions, including rejected questions, been independently evaluated by humans?

2 comments

Thank you for recognizing that our work "addresses an important practical limitation of coding agents." We also appreciate your observation that our annotated omissions and interactive comparisons "provide a useful framework" for examining clarification behavior separately from code-generation ability.

W1 & Q How are questions outside the annotation set or covering multiple omissions handled, and have humans reviewed rejections?

1. Review of unmatched questions. We manually reviewed all 368 unmatched questions. Of these, 5.4% asked for useful information outside the annotation set, and 1.4% addressed multiple annotated omissions in a single question. These cases were too infrequent to change our main conclusion

2. Human validation. In addition to reviewing all unmatched questions, we manually checked 100 of the 482 questions that GPT-4o marked as matched. Human reviewers agreed with GPT-4o in 96% of these cases. Together, these checks support the robustness of our conclusion.

W2. Whether the same missing detail is harder to identify alongside other omissions than on its own.

Thank you for suggesting this comparison. For each missing detail in a task with two or three omissions, we matched it to the version of the same task where only that detail was missing. Using the original GPT-4o judge, we measured how often the agent asked about that detail. Rates are percentages, and differences are in percentage points.

AgentAloneWith 1 other omissionDifferenceAloneWith 2 other omissionsDifference
Opus 4.8 with reasoning5.68.3+2.75.27.2+2.0
Opus 4.8 without reasoning26.021.0-5.026.819.0-7.8
GPT-5.5 with reasoning4.44.7+0.34.68.5+3.9
GPT-5.5 without reasoning10.110.4+0.39.27.2-2.0
Gemini 3.1 Pro18.918.0-0.917.613.7-3.9
GPT-5 mini6.56.2-0.35.94.6-1.3
Mean11.911.4-0.511.510.0-1.5

Across the six configurations, agents asked about the same missing detail only 0.5 percentage points less often on average with one additional omission and 1.5 points less often with two. This suggests that an individual missing detail does not become substantially harder to identify as more details are removed.

Overall

Thank you again for your thoughtful review. We hope the additional discussion address your remaining concerns. We would be grateful for your continued support.

Thanks for your detailed response. My concerns are largely addressed; I will maintain my positive evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.