- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether coding agents ask for missing information before writing code. The authors build ClarifyCodeBench by deleting one to three details from 80 LiveCodeBench problems, giving 419 underspecified variants. Each omission has a key question, a reference answer, and one of 11 underspecification types. Agents may ask one question per turn. An LLM judge returns the reference answer for matched questions and a neutral reply otherwise. Across six standalone configurations, Claude Code and ClarifyGPT ask more when information is missing but cover few omissions. Coverage falls as omissions increase. Enabling reasoning raises pass@1 but lowers coverage.
Critical strengths
- The topic is well motivated, and the paper constructs a useful benchmark for future study.
- The evaluation framework is comprehensive. It separates whether to ask, what to ask, how early to ask, and the effect on correctness, revealing which stage of clarification fails.
- They covered a wide range of recent models, including reasoning and non-reasoning models.
- The 11 underspecification types, grounded in requirements engineering standards, allow analysis by the kind of missing information.
Critical weaknesses
- The novelty is incremental. ClarEval already measures intent coverage and dialogue efficiency. The main additions are per-omission types and a varying number of omissions.
- The paper does not show that the deleted details are needed to solve the tasks. Its validation only checks whether the reference answer changes the model's output, not whether the answer is needed to pass the tests. The validation result is also not reported. Without clarification, pass@1 is only 4–15 points below that with all answers given.
- The underspecification is artificial. Annotators create it by deleting details from competitive programming problems. The paper never checks whether these omissions resemble those in naturally underspecified requirements.
- The LLM judge is validated only against other LLMs, with no human validation. Its agreement on assumption matching is lower (κ 0.67–0.72).
Questions for authors
- Two PhD students annotate each task and resolve disagreements through discussion, but no agreement is reported. How often did they agree before discussion on the deletions, the key questions and reference answers, and the underspecification types?
Thank you for recognizing that our topic is "well motivated" and our evaluation framework is "comprehensive."
Q. Annotator agreement.
Both annotators independently annotated all 560 candidate tasks and agreed on 247 of them (44.1%) before discussion. Disagreements mainly concerned whether the edited problem statement still read naturally and whether the deletion introduced unintended underspecification elsewhere in the task. After discussing the remaining 313 candidates, we retained a further 172 and excluded 141, yielding 419 tasks in total.
W1. Novelty relative to ClarEval.
ClarifyCodeBench evaluates two aspects of effective clarification:
1. Knowing when to ask. We compare fully specified and underspecified versions of the same task to measure whether agents ask when a requirement is missing and refrain from asking when the requirements are complete.
2. Prioritizing key questions. ClarEval allows agents to ask multiple questions in one turn, so unrelated questions bundled with a key question incur no additional turn cost. We instruct agents to ask one question per turn. Under this protocol, each unrelated question asked before a key question delays it by one turn and lowers its contribution to TKQR.
Controlled comparisons. We construct variants of the same task by deleting specific requirements while keeping the rest of the prompt and the original tests fixed. This lets us compare how often agents ask about the same detail when it is missing alone or alongside other omissions, and measure how restoring missing information affects pass@1 on the same tests.
W2. The paper does not show that the deleted details are needed to solve the tasks.
We evaluate clarification separately from code generation. Agents often proceed directly to code by guessing a missing detail. When the guess matches the intended behavior, the code can pass the original tests even though the agent never clarified the requirement. Prior exposure to public LiveCodeBench problems could also narrow the observed pass@1 gap.
We constructed ClarifyCodeBench-CF, a set of 88 counterfactual tasks. For each task, we kept the underspecified prompt unchanged, so exactly the same detail was missing. We then wrote a different plausible answer to the corresponding clarification question and new tests for the behavior specified by that answer.
GPT-5.6-sol on ClarifyCodeBench-CF
Clarification behavior in the Interactive condition
Asking for clarification improved pass@1 by 23.9 percentage points, and providing all missing information raised it to 97.7%. Because the underspecified prompts stayed the same, the large gap in pass@1 between direct generation at 15.9% and the oracle condition at 97.7% shows that the agent needs the missing details to reliably implement the user's intended behavior.
W3. Do the omissions in ClarifyCodeBench resemble those in naturally underspecified requirements?
Naturalness was an explicit annotation criterion. The annotators discussed whether each deletion left a request that a user might plausibly make and excluded variants they judged unnatural.
Two additional PhD students reviewed a random sample of 50 retained prompts and agreed that 98% read naturally. This check supports the naturalness of the resulting prompts.
W4. The LLM judge is validated only against other LLMs, with no human validation.
We manually reviewed all 368 unmatched questions and 100 of the 482 matched questions, finding 91% and 96% agreement with GPT-4o, respectively.
Overall
Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.