- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SWE-Clar evaluates requirement clarification and subsequent software implementation on 100 locally adapted tasks derived from Upwork commissions with at least US$4,000 in total payments. A coding agent first interacts with a simulated client whose replies are reviewed by a supervisor. The frozen clarification context then initializes independent implementation and requirement-answering branches, preventing assessment questions from guiding code generation. The benchmark includes 512 requirement units linked to questions and acceptance tests. Nine models achieve task pass rates of 39–67%. For two models, no-interaction and complete-requirement controls show substantial benefits from clarification and a remaining information gap. Requirement-level analyses also show that answering correctly and implementing correctly are related but distinct outcomes.
Critical strengths
-
Freezing a shared clarification context and branching before the assessment questions are introduced addresses a concrete evaluation-contamination problem. Linking 512 requirement questions to corresponding acceptance checks makes answer–implementation disagreements interpretable. This is a useful methodological contribution for studying failures that final task success alone cannot explain.
-
Clarification raises implementation scores from 6% to 49% and from 5% to 39% for the two ablated models, with paired task-bootstrap intervals excluding zer. Complete requirements provide a further comparison. Appendix A reports two industry-background reviewers, passing reference and behaviorally equivalent implementations for all 100 tasks, and rejection of one targeted incorrect implementation per task. Appendix D also compares continued clarification. These checks provide direct support for the validity of the task scores and the claimed benefit of clarification.
Critical weaknesses
1.GPT-5.6 Luna serves as client, supervisor, and answer judge, but sensitivity to these model choices is not evaluated. Appendix B.3 mentions agreement with human judgments without reporting the sample size, sampling procedure, or label-level agreement. Appendix E shows a net benefit from supervision, but among the 561 revised replies, 2.9% change from correct to incorrect and 7.5% remain incorrect. Thus, roughly 10.4% of this selected subset still has incorrect final replies; this is not an error-rate estimate for all 4,418 replies. The lack of a representative audit leaves uncertainty about how often candidates are misinformed and how robust the secondary clarification scores are. These issues matter for attributing failures to the coding agent and for interpreting comparisons across models.
2.SWE-Clar gives the client complete, fixed requirements, and Section 2.2 and Appendices F.2–F.3 prescribe which information may be disclosed in response to each question. In particular, broad requests for the full specification trigger a request to narrow the topic, while focused questions can elicit complete workflow interfaces. This is a useful controlled setting, but the paper does not establish whether this response policy reflects actual client behavior or whether model comparisons remain stable under other plausible policies. It may favor particular questioning strategies, including those effective at recovering the benchmark’s hidden delivery contracts. Moreover, fixed-requirement clarification captures only part of requirements elicitation, where needs can be refined or negotiated during interaction. Real commission provenance and the human checks in Appendix A support task relevance and consistency; they do not directly validate the simulated interaction. This limits interpretation of the scores as evidence about agents’ ability to clarify requirements with real clients.
Questions for authors
1.What are the sample size, selection procedure, and per-label human agreement for requirement-answer grading? What is the final client-reply error rate on a representative sample including replies accepted without revision, and are those errors concentrated in particular tasks or candidate models? A small independent regrading exercise would help establish whether the main comparisons are robust.
2.What evidence from source conversations, requirements-elicitation research, or expert assessment of sampled dialogues supports the prescribed information-disclosure policy? On a small representative subset, do the main conclusions remain stable under alternative plausible client policies, such as providing a broader requirements overview when asked? Please clarify which aspects of real requirements elicitation the fixed-requirement setting is intended to capture and which remain outside its scope.
5 comments
Response to Reviewer h8Wa (2/2)
(Continued; "GR n" = point n of our general response.)
W2 / Q2: Realism of the client response policy
The client response policy follows how clients answer in requirements interviews and in our source projects.
(a) Evidence for the policy
Requirements-elicitation research, our source conversations and human ratings of sampled replies all support the policy (GR 7).
- Research finds that clients cannot state complete requirements upfront. In requirements-engineering studies, much of what clients know is implicit and comes out in answer to the analyst's questions (Goguen & Linde, 1993; Zowghi & Coulin, 2005; Ferrari et al., 2016), and vague questions get vague answers (Bano et al., 2019). Simulated users in τ-bench (Yao et al., 2025) and Ambig-SWE (Vijayvargiya et al., 2026) likewise hold the full specification and answer only what is asked.
- In our source conversations, requests for all requirements at once are rare and get brief replies. We labeled, with model assistance, the freelancers' questions in the full chat records behind our tasks. Freelancers sent 2,323 question messages in 80 projects, and only 21 asked for all requirements at once. Clients answered these briefly (at most 1,072 characters, often asking back), and in 17 of the 21 cases the freelancer then asked specific questions.
- Annotators find the client agent's replies realistic. They rated 300 client-agent replies 4.3/5 for realism as freelance client communication (none below 3).
(b) Does the policy favor strategies that recover the hidden delivery contracts?
No, knowing the full interface earns almost nothing. We added a new interface-only experiment in which agents get the complete delivery interface but no requirements and cannot ask questions. Implementation reaches only 8% (Luna) and 6% (Flash), against 49% and 39% under the standard protocol. The scores come from finding the business requirements, not from recovering the interface.
(c) Other plausible policies
The comparison between Luna and Flash remains stable in every other setting we ran. With a Qwen-3.8 Max client agent that revealed more than the policy allows, Luna still beats Flash (77% vs. 64%; GR 6). In the interface-only experiment above, where agents receive only the delivery interface and cannot ask questions, Luna again leads (8% vs. 6%). With complete requirements handed over at the start, the furthest a broader overview can go, Luna still leads (75% vs. 71%).
(d) What the fixed-requirement setting captures
The setting captures requirements that were already negotiated in the real project, and negotiation through proposals during clarification. The fixed requirements are not invented: they are what the client and the freelancer settled in the real conversation, and requirements left unsettled there are not included. Negotiation also remains part of clarification: the agent can propose a design, and the client agent confirms or corrects it, just as 46% of the real freelancers' question messages proposed a design for the client to confirm. What the setting leaves out is a client who changes goals after delivery, which calls for new acceptance criteria rather than better clarification.
Thank you again for the thoughtful questions; we will add these experiments, audits and analyses to the next version of the paper, and we are happy to discuss further.
Follow-up on our response
Dear Reviewer h8Wa,
Thank you again for your careful review. We have posted our response in this thread, including the representative audits of the client agent's replies and of the judge agent that you asked for, and the evidence for the client response policy. As the discussion period is closing soon, we would be grateful to hear whether it addresses your concerns.
Best regards, The Authors
Response to Reviewer h8Wa (1/2)
Thank you for the careful review, and for recognizing a useful methodological contribution for failures that task success alone cannot explain, and that the task scores and the benefit of clarification are supported by evidence. "GR n" refers to point n of our general response.
W1 / Q1: Sensitivity to Luna's three roles, and representative audits
The results do not depend on using Luna as client, supervisor and judge agents; in the two audits you asked for, the client agent is accurate and the judge agent largely agrees with humans. We gave each role to a model from another family and ran both audits.
(a) Sensitivity to the model choices
The comparisons hold with another model family in each role.
(b) Judge audit: sample size, selection and per-label agreement
Humans largely agree with the judge agent. Two annotators labeled 300 answers covering all 100 tasks (three per task), balanced across the nine models and the three judge labels. Both labeled without seeing the judge agent's results (Cohen's κ = 0.81 between them), and a third person gave the final label for the 37 items on which they disagreed. Humans agree with the judge agent on 74.7% (κ = 0.62). Per label, they confirm 82% of its Correct, 75% of its Partial and 66% of its Incorrect labels. In 93.3% of answers, humans and the judge agent agree or differ by only one level (Correct vs. Partial, or Partial vs. Incorrect), so they rarely disagree on whether an answer is right or wrong; they differ mainly in judging partially correct answers. Rescoring with human labels keeps the ranking of all nine models (GR 5).
(c) Client-reply error rate on a representative sample
The audit finds no factual errors, so none can concentrate in particular tasks or models. Annotators audited 300 client-agent replies sampled from all 100 tasks and nine models: 200 the supervisor agent approved at first review (the replies accepted without revision that you ask about) and 100 it approved after revision. None contains a factual error, so with 95% confidence at most 1.8% of all replies approved at first review are wrong. None is incomplete, and none reveals requirements the question did not ask about. The 561 replies on which the supervisor agent requested revision are not concentrated in a few tasks or models either: they occur in 97 of the 100 tasks and with all nine models, and the task with the most holds only 3.0% of the 561.
(d) The 10.4% in Appendix E
As you note, the 10.4% is not the overall error rate; our representative audit shows the overall factual error rate is low. You note that about 10.4% of the 561 revised replies remain incorrect, and that this is not an error rate for all 4,418 replies. We agree, so we estimated the overall rate from the representative audit above, which samples both groups of replies: none of the 200 replies approved at first review and none of the 100 revised replies contains a factual error. With 95% confidence, at most 1.8% of the 3,857 replies approved at first review and at most 3.6% of the 561 revised replies are factually wrong. The 10.4% already refers to the final approved replies, after all revision rounds; it is higher because it counts a reply as still wrong if it has any of three problems: a factual error, an incomplete answer, or revealing requirements beyond what the question asked. The Appendix E labels were also produced with model assistance and verified manually only where needed, whereas the audit above was done by human annotators, who found none of the three problems in the 100 revised replies they checked; on these replies the model-assisted labels are the stricter of the two. By factual errors alone, only 8 of the 561 (1.4%) remain, and revision turned 46.0% of these replies from incorrect to correct and only 2.9% from correct to incorrect.
These checks support attributing failures to the coding agent rather than to the client agent's replies, and the model ranking holds under human labels and under a judge agent from another family.