acceptodds
Under review as a conference paper at ICLR 2027

SWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?

Abstract

Greenfield software development in freelance settings requires coding agents to turn incomplete client requests into working software. This makes requirement clarification essential, yet clients may be unable to provide continuous feedback over long development cycles, and receiving their answers does not guarantee correct understanding of complex requirements. These challenges motivate two design principles: clarification should precede code implementation, and its effectiveness should be assessed through explicit tests of requirement understanding. Guided by these principles, we introduce SWE-Clar, the first benchmark grounded in real, high-value client commissions to evaluate requirement clarification and code implementation in this setting. It comprises 100 tasks based on real software development jobs on Upwork, each priced above US$4,000. Each task begins with multi-round interaction between a coding agent and a client agent, followed by separate runs for code implementation with acceptance testing and clarification assessment. For the latter, we construct 512 questions paired with reference answers from real client–freelancer interactions, covering both explicitly stated requirements and those inferred from the conversational context. Both runs begin from the same clarification context but remain isolated, preventing assessment questions from guiding code implementation. Experiments with nine frontier models show that even the strongest, GPT-6 Astra, achieves a task pass rate of only 67%, underscoring the challenge of independent software delivery in this setting. Code is included in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
Coding agents, requirements clarification, software engineering, interactive evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h8WaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Clar evaluates requirement clarification and subsequent software implementation on 100 locally adapted tasks derived from Upwork commissions with at least US$4,000 in total payments. A coding agent first interacts with a simulated client whose replies are reviewed by a supervisor. The frozen clarification context then initializes independent implementation and requirement-answering branches, preventing assessment questions from guiding code generation. The benchmark includes 512 requirement units linked to questions and acceptance tests. Nine models achieve task pass rates of 39–67%. For two models, no-interaction and complete-requirement controls show substantial benefits from clarification and a remaining information gap. Requirement-level analyses also show that answering correctly and implementing correctly are related but distinct outcomes.

Critical strengths
  1. Freezing a shared clarification context and branching before the assessment questions are introduced addresses a concrete evaluation-contamination problem. Linking 512 requirement questions to corresponding acceptance checks makes answer–implementation disagreements interpretable. This is a useful methodological contribution for studying failures that final task success alone cannot explain.

  2. Clarification raises implementation scores from 6% to 49% and from 5% to 39% for the two ablated models, with paired task-bootstrap intervals excluding zer. Complete requirements provide a further comparison. Appendix A reports two industry-background reviewers, passing reference and behaviorally equivalent implementations for all 100 tasks, and rejection of one targeted incorrect implementation per task. Appendix D also compares continued clarification. These checks provide direct support for the validity of the task scores and the claimed benefit of clarification.

Critical weaknesses

1.GPT-5.6 Luna serves as client, supervisor, and answer judge, but sensitivity to these model choices is not evaluated. Appendix B.3 mentions agreement with human judgments without reporting the sample size, sampling procedure, or label-level agreement. Appendix E shows a net benefit from supervision, but among the 561 revised replies, 2.9% change from correct to incorrect and 7.5% remain incorrect. Thus, roughly 10.4% of this selected subset still has incorrect final replies; this is not an error-rate estimate for all 4,418 replies. The lack of a representative audit leaves uncertainty about how often candidates are misinformed and how robust the secondary clarification scores are. These issues matter for attributing failures to the coding agent and for interpreting comparisons across models.

2.SWE-Clar gives the client complete, fixed requirements, and Section 2.2 and Appendices F.2–F.3 prescribe which information may be disclosed in response to each question. In particular, broad requests for the full specification trigger a request to narrow the topic, while focused questions can elicit complete workflow interfaces. This is a useful controlled setting, but the paper does not establish whether this response policy reflects actual client behavior or whether model comparisons remain stable under other plausible policies. It may favor particular questioning strategies, including those effective at recovering the benchmark’s hidden delivery contracts. Moreover, fixed-requirement clarification captures only part of requirements elicitation, where needs can be refined or negotiated during interaction. Real commission provenance and the human checks in Appendix A support task relevance and consistency; they do not directly validate the simulated interaction. This limits interpretation of the scores as evidence about agents’ ability to clarify requirements with real clients.

Questions for authors

1.What are the sample size, selection procedure, and per-label human agreement for requirement-answer grading? What is the final client-reply error rate on a representative sample including replies accepted without revision, and are those errors concentrated in particular tasks or candidate models? A small independent regrading exercise would help establish whether the main comparisons are robust.

2.What evidence from source conversations, requirements-elicitation research, or expert assessment of sampled dialogues supports the prescribed information-disclosure policy? On a small representative subset, do the main conclusions remain stable under alternative plausible client policies, such as providing a broader requirements overview when asked? Please clarify which aspects of real requirements elicitation the fixed-requirement setting is intended to capture and which remain outside its scope.

5 comments
AuthorsOpenReview ↗

Response to Reviewer h8Wa (1/2)

Thank you for the careful review, and for recognizing a useful methodological contribution for failures that task success alone cannot explain, and that the task scores and the benefit of clarification are supported by evidence. "GR n" refers to point n of our general response.

W1 / Q1: Sensitivity to Luna's three roles, and representative audits

The results do not depend on using Luna as client, supervisor and judge agents; in the two audits you asked for, the client agent is accurate and the judge agent largely agrees with humans. We gave each role to a model from another family and ran both audits.

(a) Sensitivity to the model choices

The comparisons hold with another model family in each role.

  • A judge agent from another family keeps the ranking. As the independent regrading you suggested, we regraded all 4,608 answers with GLM-5.3 Flash. The ranking barely changes (Spearman ρ = 0.967; only two near-tied pairs swap), and neither judge agent favors its own model: Luna is last on clarification under the Luna judge agent, and Flash drops from 8th to 9th when judged by its own model (GR 2).
  • Client and supervisor agents from another family keep the order of Luna and Flash. We reran Luna and Flash on all 100 tasks with Qwen-3.8 Max in both roles. Luna still beats Flash (77% vs. 64%); absolute scores rise because the Qwen client agent revealed test values against the policy (GR 6).
  • Filling the three roles gives Luna no advantage. Luna ranks near the bottom on implementation. Implementation is scored only by executable tests, and the judge agent never sees the agent's code or its test results.

(b) Judge audit: sample size, selection and per-label agreement

Humans largely agree with the judge agent. Two annotators labeled 300 answers covering all 100 tasks (three per task), balanced across the nine models and the three judge labels. Both labeled without seeing the judge agent's results (Cohen's κ = 0.81 between them), and a third person gave the final label for the 37 items on which they disagreed. Humans agree with the judge agent on 74.7% (κ = 0.62). Per label, they confirm 82% of its Correct, 75% of its Partial and 66% of its Incorrect labels. In 93.3% of answers, humans and the judge agent agree or differ by only one level (Correct vs. Partial, or Partial vs. Incorrect), so they rarely disagree on whether an answer is right or wrong; they differ mainly in judging partially correct answers. Rescoring with human labels keeps the ranking of all nine models (GR 5).

(c) Client-reply error rate on a representative sample

The audit finds no factual errors, so none can concentrate in particular tasks or models. Annotators audited 300 client-agent replies sampled from all 100 tasks and nine models: 200 the supervisor agent approved at first review (the replies accepted without revision that you ask about) and 100 it approved after revision. None contains a factual error, so with 95% confidence at most 1.8% of all replies approved at first review are wrong. None is incomplete, and none reveals requirements the question did not ask about. The 561 replies on which the supervisor agent requested revision are not concentrated in a few tasks or models either: they occur in 97 of the 100 tasks and with all nine models, and the task with the most holds only 3.0% of the 561.

(d) The 10.4% in Appendix E

As you note, the 10.4% is not the overall error rate; our representative audit shows the overall factual error rate is low. You note that about 10.4% of the 561 revised replies remain incorrect, and that this is not an error rate for all 4,418 replies. We agree, so we estimated the overall rate from the representative audit above, which samples both groups of replies: none of the 200 replies approved at first review and none of the 100 revised replies contains a factual error. With 95% confidence, at most 1.8% of the 3,857 replies approved at first review and at most 3.6% of the 561 revised replies are factually wrong. The 10.4% already refers to the final approved replies, after all revision rounds; it is higher because it counts a reply as still wrong if it has any of three problems: a factual error, an incomplete answer, or revealing requirements beyond what the question asked. The Appendix E labels were also produced with model assistance and verified manually only where needed, whereas the audit above was done by human annotators, who found none of the three problems in the 100 revised replies they checked; on these replies the model-assisted labels are the stricter of the two. By factual errors alone, only 8 of the 561 (1.4%) remain, and revision turned 46.0% of these replies from incorrect to correct and only 2.9% from correct to incorrect.

These checks support attributing failures to the coding agent rather than to the client agent's replies, and the model ranking holds under human labels and under a judge agent from another family.

AuthorsOpenReview ↗

Response to Reviewer h8Wa (2/2)

(Continued; "GR n" = point n of our general response.)

W2 / Q2: Realism of the client response policy

The client response policy follows how clients answer in requirements interviews and in our source projects.

(a) Evidence for the policy

Requirements-elicitation research, our source conversations and human ratings of sampled replies all support the policy (GR 7).

  • Research finds that clients cannot state complete requirements upfront. In requirements-engineering studies, much of what clients know is implicit and comes out in answer to the analyst's questions (Goguen & Linde, 1993; Zowghi & Coulin, 2005; Ferrari et al., 2016), and vague questions get vague answers (Bano et al., 2019). Simulated users in τ-bench (Yao et al., 2025) and Ambig-SWE (Vijayvargiya et al., 2026) likewise hold the full specification and answer only what is asked.
  • In our source conversations, requests for all requirements at once are rare and get brief replies. We labeled, with model assistance, the freelancers' questions in the full chat records behind our tasks. Freelancers sent 2,323 question messages in 80 projects, and only 21 asked for all requirements at once. Clients answered these briefly (at most 1,072 characters, often asking back), and in 17 of the 21 cases the freelancer then asked specific questions.
  • Annotators find the client agent's replies realistic. They rated 300 client-agent replies 4.3/5 for realism as freelance client communication (none below 3).

(b) Does the policy favor strategies that recover the hidden delivery contracts?

No, knowing the full interface earns almost nothing. We added a new interface-only experiment in which agents get the complete delivery interface but no requirements and cannot ask questions. Implementation reaches only 8% (Luna) and 6% (Flash), against 49% and 39% under the standard protocol. The scores come from finding the business requirements, not from recovering the interface.

(c) Other plausible policies

The comparison between Luna and Flash remains stable in every other setting we ran. With a Qwen-3.8 Max client agent that revealed more than the policy allows, Luna still beats Flash (77% vs. 64%; GR 6). In the interface-only experiment above, where agents receive only the delivery interface and cannot ask questions, Luna again leads (8% vs. 6%). With complete requirements handed over at the start, the furthest a broader overview can go, Luna still leads (75% vs. 71%).

(d) What the fixed-requirement setting captures

The setting captures requirements that were already negotiated in the real project, and negotiation through proposals during clarification. The fixed requirements are not invented: they are what the client and the freelancer settled in the real conversation, and requirements left unsettled there are not included. Negotiation also remains part of clarification: the agent can propose a design, and the client agent confirms or corrects it, just as 46% of the real freelancers' question messages proposed a design for the client to confirm. What the setting leaves out is a client who changes goals after delivery, which calls for new acceptance criteria rather than better clarification.

Thank you again for the thoughtful questions; we will add these experiments, audits and analyses to the next version of the paper, and we are happy to discuss further.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer h8Wa,

Thank you again for your careful review. We have posted our response in this thread, including the representative audits of the client agent's replies and of the judge agent that you asked for, and the evidence for the client response policy. As the discussion period is closing soon, we would be grateful to hear whether it addresses your concerns.

Best regards, The Authors

Reviewer_q75ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work focuses on underspecified coding problems where the coding agent should interact with the user to clarify the requirements. For this, the authors collect real tasks from Upwork and real exchanges that they parse into requirements. Coding agents are then evaluated on their ability to produce code that passes the task and to explain the requirements. The paper shows that current frontier models do not saturate the task, showing the clarification step is still unsolved.

Critical strengths

The paper focuses on an important and practical problem. It is particularly interesting that the benchmark is rooted in real tasks and real exchanges with clients. This has the advantage of evaluating coding agents in a setup close to their practical deployment use-cases. The finding that the clarification step is not solved, despite being already kind of known, is useful, and the benchmark could set a new target for coding agents. The authors also shared downloadable code and data.

Critical weaknesses

The main weakness that I see is that the paper is not sufficiently technically precise.

This becomes especially problematic for the description of the dataset construction and the evaluation metrics, where the paper does not describe what is actually being done but remains very vague and high-level. Appendix A does not provide technical details that are sufficient. For example: "GPT-6 Astra and human contributors jointly develop the acceptance tests and reference implementations." But that does not explain what was being done, what was the outcome, what does it look like, and so on.

In general, the paper does not describe how the tasks from Upwork are converted into the units necessary for the benchmark and even less discusses the stability and performance of the elements involved in this processing pipeline. This is unfortunate because it makes it difficult to assess the scientific validity of the work. For example, Figure 4 shows a line plot of requirements revealed as conversation goes on; unfortunately, we cannot know what the requirements are, how they are obtained, or what is the error rate of the process that converts discussions into RUs.

Finally, the analyses are fairly minimal and superficial. One would be curious about a more detailed error analysis. What types of requirements are missed by which agent? Which type of questions do agents ask and which type do they systematically fail to ask? What types of requirements can agents typically not implement and not explain? ...

Questions for authors
  • Can you provide a more detailed description of the pipeline that produces the benchmark and the RUs (and can you provide error estimates for these steps)?
  • What types of requirements are missed by which agent? Which type of questions do agents ask and which type do they systematically fail to ask? What types of requirements can agents typically not implement and not explain?
6 comments
AuthorsOpenReview ↗

Response to Reviewer q75R (1/2)

Thank you for the review, and for noting that the paper addresses an important, practical problem, that grounding in real tasks and client exchanges brings evaluation close to real deployment, and that the benchmark could set a new target for coding agents. We agree that our description of construction and scoring was too high-level; below we make each step concrete, give error estimates, and add the error analysis you asked for. "GR n" refers to point n of our general response.

W1 / W2 / Q1: How each task is built

Each task is built from a real project in six steps, and is accepted only after executable checks and human cross-checking.

  1. Selecting projects. From 122 Upwork projects with at least eight milestones, US$4,000 in payments and 20 messages, we kept the 100 that run locally without the client's accounts and can be checked by tests.
  2. Reading the source. Astra reads each project's initial request and full conversation and lists the requirements that affect delivery, marking which were stated and which follow from context.
  3. Rebuilding the task. We keep the business goal, replace private data, accounts and deployments with synthetic records and local services, and write a short initial request, a starter project and a runtime environment.
  4. Writing requirement units (RUs). Each task's requirements are grouped into 3–7 RUs, 512 in total: 209 core RUs (the task's main behavior) and 303 refinement RUs (the detailed business rules). Each RU has a specification, an assessment question, a delivery interface and a test.
  5. Writing tests and three implementations. In the step you quote as "jointly develop", Astra produced only first drafts of each task's acceptance tests and three implementations: a reference, a behaviorally equivalent one, and a targeted incorrect one that breaks one RU. Humans then reviewed, ran and refined them into the final versions. A task was accepted only when the reference and equivalent implementations passed every test and the incorrect one failed; otherwise humans fixed the test or implementation and reran all three.
  6. Cross-checking. Two checkers with industry experience checked each task for consistency among requirements, questions and tests.

W1: Evaluation metrics

Both scores follow fixed rules and start from the same frozen conversation; they are defined in Section 2.3, with the grading rubric in Appendix B. In short: when clarification ends, the conversation is frozen and two branches start from it. One writes the code: a task is completed only if all its tests pass, and the implementation score is the share of tasks completed. The other answers each RU's assessment question: the judge agent grades each answer as correct, partial or incorrect (1, 0.5 or 0), and the clarification score averages these within each task and then across tasks. The assessment questions never reach the implementation branch.

W2 / Q1: Error estimates

We measured the error of each step with blinded human audits and executable checks.

  • 96.3% of RUs faithfully reflect the source requirements. Four blinded annotators checked all 512 RUs against the source requests and conversations; 3.7% reflect them partly and none contradicts the source (GR 4).
  • On all 100 tasks, the tests pass the reference and equivalent implementations and fail the targeted incorrect one.
  • Client-agent replies contain no factual error in the audit. Annotators audited 300 sampled replies and found none, so with 95% confidence at most 1.8% of all replies approved at first review are wrong (GR 5).
  • Humans largely agree with the judge agent. Two annotators labeled 300 graded answers without seeing the judge agent's results (Cohen's κ = 0.81 between them). They agree with it on 74.7% (κ = 0.62), and on 93.3% of answers they agree or differ by only one level. Rescoring with human labels keeps the ranking of all nine models, and regrading with a judge agent from another model family barely changes it (Spearman ρ = 0.967; GR 2, GR 5).

W1: What the outcome looks like

Here is one RU, followed from the source conversation to its test. In task SWECLAR-001, the client accepted the freelancer's suggestion to price listings from comparable sales. This became RU-003, a refinement RU hidden at the start: it fixes which sales count as comparable, how they are ordered, how the price is adjusted and how the result is rounded. Its delivery interface is POST /api/listings/<id>/analysis, and its assessment question asks for the pricing rules. Its test sends eleven sales chosen to hit the boundary cases and checks the selected IDs, the midpoint price and the confidence exactly. The human audit confirmed that it comes from a freelancer proposal the client accepted and faithfully reflects the source requirement. With no interaction, Luna and Flash both fail its test; after clarification, all nine models pass.

AuthorsOpenReview ↗

Response to Reviewer q75R (2/2)

(Continued; "GR n" = point n of our general response.)

W2: Your Figure 4 example

Figure 4 plots the requirements a client stated or confirmed in the real conversation, each at the message where it first appears, and 96.3% of the RUs built from these conversations faithfully reflect the source. Part 1/2 covers the conversion pipeline and its error estimates; here we answer the three questions you ask about Figure 4.

What the requirements are: the requirements a client stated or confirmed in the real conversation.

How they are obtained: a model marks each requirement and the message where it first appears, and every cited message is checked against the original conversation. When a requirement changes, we keep the client's final confirmed version, and unconfirmed proposals are excluded (Appendix A.2).

Error rate of the process that converts discussions into RUs: the source-to-RU audit in Part 1/2 finds that 96.3% of RUs faithfully reflect the source requirements and none contradicts the source (GR 4).

W3 / Q2: Error analysis

Agents mainly fail because they do not ask about refinement RUs (the detailed business rules). We extended the error analysis along your three questions.

(a) Which requirements are missed, and by which agent

Agents miss mainly refinement RUs, and weaker agents miss more of them. Under the standard protocol, Astra passes 93.8% of core RUs and 85.1% of refinement RUs, while Flash passes 88.0% and 68.3%. Across the nine models, 65–74% of failed tasks fail only on refinement RUs, and only 5–14% fail only on core RUs. Knowing the interface does not close this gap: in our new interface-only experiment, agents get the complete delivery interface but no requirements and cannot ask questions, and still pass only 27–31% of refinement RUs and complete 6–8% of tasks. Per requirement under the standard protocol, combining each answer with its test result:

AstraFlash
Answered correctly, test passes71.3%49.4%
Answered incompletely or wrongly, test fails3.5%15.4%

The share answered correctly but failing the test is similar across models (6.1–7.8%).

(b) Which questions agents ask, and which they fail to ask

Agents ask mostly about behavior and interfaces, and rarely about scope, priorities or environment. We classified by topic all 16,634 individual questions in the agents' 4,418 clarification messages:

  • 68% concern behavior: business rules, boundaries, state and workflow, error handling, and permissions and data (11–15% each). 27% concern interfaces, and under 0.1% ask for the full specification. The mix is similar across models (66–72% behavior).
  • Agents rarely ask about scope and priorities (2.8% of questions, vs. 15% for real freelancers) or about environment and tech stack (1.6% vs. 5%).
  • What differs between models is how much they ask. The number of questions per task correlates with the implementation score at ρ = 0.80, and the average number of clarification rounds, from 2.48 for Flash to 8.12 for Opus, at ρ = 0.96. There are exceptions: Astra uses fewer rounds than Opus yet scores higher, and Fable matches Opus's clarification score with about 60% of its rounds.
  • Once a requirement is fully disclosed, only 2.8–6.4% of answers are incorrect (Appendix C.4). So weaker agents fail mainly because they do not ask about refinement RUs, not because they misunderstand them.

(c) Which requirements agents can neither implement nor explain

Four themes recur among the hardest requirements: transactions and replay, interfaces with asynchronous state, numerical boundaries, and data consistency. One requirement defeats all nine models: the rules for alarm states and role boundaries in task SWECLAR-055 (RU-005), where every model answers incorrectly and fails the test.

SWE-Clar thus shows not only whether an agent fails but where: in asking, in understanding, or in implementing. Thank you again for pushing us to make the construction concrete; we will add the construction details, the error estimates, and these analyses with their full data to the next version of the paper, and we are glad to clarify anything further.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer q75R,

Thank you again for your review. We have posted our response in this thread, with the step-by-step construction pipeline, error estimates for these steps, and the more detailed error analysis you asked for. As the discussion period is closing soon, we would be grateful to hear whether it addresses your concerns.

Best regards, The Authors

Reviewer_krtEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SWE-Clar evaluates coding agents on 100 greenfield tasks drawn from Upwork jobs paying at least US$4,000. Each agent first clarifies the request with a simulated client, GPT-5.6 Luna, whose replies are reviewed by a supervisor. The conversation is then frozen and supplied to two independent branches: an implementation run evaluated by acceptance tests, with task credit only when every test passes, and a QA run covering 512 requirement questions evaluated by an LLM judge. GPT-6 Astra assisted with constructing requirements, tests and reference implementations, with human cross-validation. Across nine models, implementation scores range from 39% to 67% and correlate with clarification scores at r = 0.89. In the information-access comparison, GPT-5.6 Luna scores 6%, 49% and 75% under no interaction, standard clarification and complete requirements, respectively; GLM-5.3 Flash scores 5%, 39% and 71%.

Critical strengths
  1. The separation between implementation and requirement QA is the strongest part of the design. Asking the QA questions within the coding run would expose requirements before implementation. Freezing the conversation and branching into independent runs avoids that leakage while yielding two measurements of the same requirements. Their disagreement in Fig. 9 is informative in its own right. This design could be useful in other interactive benchmarks and is the principal contribution supporting my positive assessment.

  2. The benchmark remains challenging: tasks are lengthy, success requires passing every test, and even the best model reaches only 67%. The release includes tasks, the harness and prompts. However, the incorrect-implementation check in App. A tests only one requirement per task, leaving the sensitivity of the tests for most requirements unclear.

Critical weaknesses
  1. The benchmark's connection to real client clarification needs stronger validation. The tasks originate in paid commissions, but agents work toward a constructed specification and interact with an LLM client governed by a detailed response policy. The following uncertainties affect how broadly the results can be interpreted: (1) GPT-6 Astra helps derive requirements, including ones inferred from context. Appendix A describes human consistency checks among client requirements, questions and tests, but does not report an independent, quantified audit of fidelity to the original client requests. Such an audit would help separate coherent benchmark construction from faithful recovery of client intent. (2) Disclosure policies P2/P3 constrain what agents can learn, and P5 withholds the project README. Appendix C.3 appropriately describes disclosure performance as conditional on the current client response policy. GPT-5.6 Luna with complete requirements scores 75%, compared with 67% for GPT-6 Astra after clarification. This illustrates the importance of information access, although it does not by itself establish how rankings would change under another client policy. A policy-sensitivity study would strengthen the interpretation. (3) Delivery interfaces are predetermined, must be requested, and also form part of the QA grading criteria (Sec. 2.4, App. B.3). Consequently, the 5 to 6% no-interaction result may largely reflect missing interface information rather than missing business requirements. A breakdown of failures would help distinguish these explanations. (4) GPT-5.6 Luna serves as client, supervisor and judge, and is one of the two coding models used in the information-access and timing comparisons. GPT-6 Astra assisted with benchmark construction and achieves the highest implementation score. These shared model roles motivate cross-family robustness checks, but do not establish bias. Appendix B.3 reports manual agreement with the judge without sample counts or an agreement statistic. Appendix E reports that 10.4% of the 561 replies selected for revision were incorrect after revision. All 4,418 questions received supervisor review, but the final correctness of the other 3,857 approved replies is not separately quantified.

  2. The evidence does not support the full scope of several conclusions. (1) Table 2 reports task-bootstrap confidence intervals, with implementation-score intervals roughly ±10 points, but does not report variability across independent runs. Overlap alone does not establish that pairwise differences are insignificant. Paired task-level comparisons and repeated runs would clarify which parts of the ranking are reliable. (2) The information-access ablations and the interleaved comparison in Appendix D cover GPT-5.6 Luna and GLM-5.3 Flash. Extending them to a leading model would help determine whether its remaining failures are primarily due to incomplete clarification or implementation difficulty. (3) Clarification scores remain below 100% even with complete requirements: 93.1% and 96.2%. These shortfalls could reflect answer-generation errors, judge errors, or both; they are not a direct estimate of judge noise. A manually adjudicated breakdown would strengthen the interpretation of the 2.8%–6.4% Incorrect answers after full disclosure in Appendix C.

Questions for authors
  1. Can you provide human-calibrated evidence that the extracted requirements match the original conversations and that the simulated client's responses resemble real client behavior?

  2. What proportions of requirements and failed tasks relate to delivery interfaces? How does a no-interaction agent perform when the interface alone is supplied?

  3. How many judge labels were manually checked, and what was the agreement with human graders? Can you distinguish answer-generation errors from judge errors under complete requirements, and assess robustness with clients and judges from another model family?

  4. Could you report no-interaction and full-requirements results for GPT-6 Astra or Claude Opus 5?

  5. For at least a subset of models, what variability appears across repeated runs, and which Table 2 differences are statistically significant?

5 comments
AuthorsOpenReview ↗

Response to Reviewer krtE (1/3)

Thank you for the careful review, and for noting that the frozen-context design yields two measurements of the same requirements that could serve other interactive benchmarks, and that the benchmark remains challenging. We answer each point below in your order; "GR n" refers to point n of our general response.

W1(1) / Q1: Fidelity to the source projects

A blinded human audit finds the requirements faithful to the source projects and the client agent's replies realistic. We agree a quantified fidelity audit was missing, so we ran one: four blinded annotators checked all 512 requirement units (RUs) against the source conversations. 96.3% faithfully reflect the source requirements (99.0% of core RUs, each task's main behavior) and none contradicts the source (GR 4). On the second half of Q1, annotators rated 300 client-agent replies 4.3/5 for realism (none below 3), and none reveals requirements the question did not ask about (GR 5).

W1(2): Sensitivity to the client response policy

The client response policy is what lets the benchmark test whether an agent finds missing requirements, and it matches how real clients answer. Under this policy, a focused question gets a complete answer on its topic. A request for the full specification does not, because if the client agent handed over everything, there would be nothing left for the agent to find. Real clients answer the same way (GR 7), and such requests are also rare: under 0.1% of the agents' 16,634 questions and 0.8% of real freelancers'. On how rankings change under another client policy: the ranking of Luna and Flash is the same in all four settings we ran. Luna is ahead with no interaction (6% vs. 5%), under the standard protocol (49% vs. 39%), with a Qwen-3.8 Max client agent that disclosed more than the policy allows (77% vs. 64%; GR 6), and with complete requirements (75% vs. 71%).

W1(2) / W1(3) / Q2: Missing interface information vs. missing requirements

Low scores with no interaction are not caused by missing interface details. The coding-branch prompt (P5, Appendix F) withholds the project README, which contains the full delivery interface, so an agent learns an interface only by asking the client agent. To test whether missing interface information explains the low scores with no interaction, we added a new interface-only experiment: agents get this README but no requirements, and cannot ask questions.

No interactionInterface-onlyStandard protocol
Luna implementation score6%8%49%
Flash implementation score5%6%39%

(a) Interface alone

The interface alone barely helps. Implementation reaches only 8% (Luna) and 6% (Flash), against 6% and 5% with no interaction and 49% and 39% under the standard protocol (+41 and +33 points). The interface lifts the pass rate of core RUs from about 60% to 80%, but refinement RUs (the detailed business rules) stay low (22% → 31% for Luna, 20% → 27% for Flash). Agents gain the specific business requirements from clarification; the interface is only the entry point for the acceptance tests.

(b) Share of requirements

Every requirement involves an interface, but knowing it is not enough. Every RU is tested through a delivery interface. 83% of refinement RUs share an interface with a core RU, yet even when that core RU passes, the refinement RU passes only 24–27% of the time with no interaction, against 72–80% after clarification.

(c) Share of failed tasks

About half of the failures with no interaction, and most after clarification, lie in business rules alone. With no interaction, about half of the failed tasks fail a core RU (Luna 51/94, Flash 49/95). After clarification only about a third do (Luna 17/53, Flash 19/63); the other 68% and 70% fail only on refinement RUs, that is, on business rules the interface does not reveal.

AuthorsOpenReview ↗

Response to Reviewer krtE (2/3)

(Continued; "GR n" = point n of our general response.)

W1(4) / Q3: Luna and Astra in supporting roles

Using Luna and Astra in supporting roles does not bias the results. We ran the cross-family checks and human audits you suggested.

  • Luna as client, supervisor and judge agents gives Luna no advantage. Luna itself ranks near the bottom on implementation and last on clarification. Implementation is scored only by executable tests, and the judge agent never sees the agent's code or its test results.
  • A judge agent from another family keeps the ranking. We regraded all 4,608 answers with GLM-5.3 Flash: the two rankings correlate at Spearman ρ = 0.967, and Flash drops from 8th to 9th when its own model is the judge agent (GR 2).
  • Client and supervisor agents from another family keep the order of Luna and Flash. We reran Luna and Flash on all 100 tasks with Qwen-3.8 Max in both roles: Luna still beats Flash (77% vs. 64%); absolute scores rise because the Qwen client agent revealed test values against the policy (GR 6).
  • Astra's construction role gives Astra no edge. The tests accept any correct implementation: a behaviorally equivalent implementation written differently from the reference passes on 100/100 tasks, while targeted incorrect implementations are rejected. Two checkers with industry experience also cross-checked requirements, questions and tests.
  • Humans largely agree with the judge agent. To report the sample size and agreement that Appendix B.3 lacks, we had two annotators label 300 answers covering all 100 tasks, balanced across the nine models and the three judge labels. Both labeled without seeing the judge agent's results (Cohen's κ = 0.81 between them), and a third person gave the final label for the 37 items on which they disagreed. Humans agree with the judge agent on 74.7% (κ = 0.62), confirming 82% of its Correct, 75% of its Partial and 66% of its Incorrect labels. Rescoring with human labels keeps the ranking of all nine models (GR 5).
  • Approved client-agent replies contain no factual errors. For the 3,857 replies approved without revision, which Appendix E leaves unquantified, a blinded audit of 200 found none, so with 95% confidence at most 1.8% of them are factually wrong (GR 5). The 10.4% in Appendix E is not the error rate of client-agent replies: it covers only the 561 replies on which the supervisor agent had already requested revision, and it counts a final approved reply as still wrong if it has any of three problems: a factual error, an incomplete answer, or revealing requirements beyond what the question asked. Only 8 of these replies (1.4%) still contain a factual error. So even replies that the supervisor agent first flagged rarely keep a factual error after revision.
AuthorsOpenReview ↗

Response to Reviewer krtE (3/3)

(Continued; "GR n" = point n of our general response.)

W2(1) / Q5: Statistical tests and repeated runs

Of the 36 pairs formed by the nine models, 10 differ significantly on implementation and 28 on clarification; run-to-run SDs are 1–3 points. As you suggested, we ran task-paired tests on all 36 model pairs (exact McNemar for implementation, Wilcoxon signed-rank for clarification; p < 0.05) and two extra runs of Luna and Flash. In the paired tests:

  • Implementation finds a significant difference in 10 pairs, each a higher-ranked model against one of the bottom three (Kimi, Luna, Flash). It shows clear tiers but cannot separate models within a tier, because their implementation scores on the benchmark are close to begin with.
  • Clarification separates 28 pairs: it grades each of a task's 3–7 requirements separately, so it is finer-grained and better at telling models apart.

Across three runs, Luna's implementation score is 49%, 44% and 46%, and Flash's 39%, 35% and 40% (means and SDs in GR 3). SDs of 1–3 points are far below the gap between these two models, and Luna's implementation score is above Flash's in every run.

W2(2) / Q4: Controls for the strongest model

For the strongest model, most remaining failures come from incomplete clarification, and some from implementation difficulty. We ran both controls for Astra. It completes 5%, 67% and 84% of tasks with no interaction, the standard protocol and complete requirements (clarification 48.2, 87.5 and 99.5); both gains are significant (+62 points, 95% CI 53–71; +17, CI 7–27). 23 of its 33 standard-protocol failures pass with complete requirements, while it still fails 16 tasks at a clarification score of 99.5. SWE-Clar thus separates the two sources of failure and leaves clear headroom even for the strongest model.

W2(3) / Q3: Lost points with complete requirements

With complete requirements, almost all lost points come from the agents' own errors, not from the judge agent. You note that even with complete requirements, Luna and Flash score only 93.1 and 96.2. To tell whether these lost points come from wrong answers or wrong grading, two annotators checked every answer that missed full credit under complete requirements (97 answers from Luna, Flash and Astra) against the full requirements. 92 were genuinely wrong and only 5 were misgraded by the judge agent. The lost points, including the 2.8–6.4% of fully disclosed requirements answered incorrectly in Appendix C.4, therefore reflect real misunderstandings by the agents. The judge agent does give near-full credit when answers are right: Astra scores 99.5.

Test sensitivity (your note under Strengths)

Every requirement's test fails without a real implementation and passes with any correct one. All 512 RU tests fail on the starter project (0% pass) and pass on the behaviorally equivalent implementation, so no test can be passed without implementing its requirement. In addition, for one requirement per task we wrote a targeted incorrect implementation that breaks only that requirement, and the tests rejected it on 100/100 tasks.

Thank you again for the constructive suggestions, which led directly to these experiments; we will add all of the new experiments and analyses to the next version of the paper. We are glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognizing that SWE-Clar addresses an important, practical problem in a setting close to real deployment, grounded in real tasks and client exchanges, and could set a new target for coding agents (q75R); that it remains challenging (krtE); that its frozen-context design yields two measurements of the same requirements, explains failures that task success alone cannot, and could be reused in other interactive benchmarks (krtE, h8Wa); and that the task scores and the benefit of clarification are supported by evidence (h8Wa). We address the shared concerns with new experiments on all 100 tasks and blinded human audits:

1. The strongest model fails mainly because clarification is incomplete, but implementation is also hard (krtE, h8Wa). Astra completes 5% of tasks with no interaction, 67% under the standard protocol and 84% with complete requirements (both increases significant). 23 of its 33 standard-protocol failures (70%) pass with complete requirements, yet at a clarification score of 99.5 it still fails 16 tasks. SWE-Clar thus separates the two sources of failure and leaves clear headroom even for the strongest model.

2. The ranking does not depend on a GPT judge agent (krtE, h8Wa). Regrading all 4,608 answers with GLM-5.3 Flash barely changes the ranking (Spearman ρ = 0.967); only two near-tied pairs swap (Fable/Opus, Flash/Luna). The new judge agent is more lenient (58% of Partial labels become Correct; 1.5% of Correct labels are downgraded). Neither judge agent favors its own model: Luna is last under the GPT judge agent, and Flash drops from 8th to 9th when judged by its own model.

3. Results are stable across runs (krtE). Over three runs (mean ± SD), implementation is 46.3 ± 2.5 (Luna) and 38.0 ± 2.6 (Flash), and clarification 70.7 ± 1.0 and 72.1 ± 1.4; Luna's implementation score is above Flash's in every run.

4. The requirements are faithful to the source projects (krtE, h8Wa, q75R). Four blinded annotators checked all 512 requirement units (RUs) against the initial requests and full client–freelancer conversations. 96.3% faithfully reflect the source requirements (99.0% of core RUs, each task's main behavior) and none contradicts the source; 92.4% trace to what the project stated or implied, and 7.6% are conventions we added to make the requirements testable, none conflicting with the source.

5. The client agent is accurate, and the judge agent largely agrees with humans (krtE, h8Wa, q75R). Annotators audited 300 client-agent replies sampled from all 100 tasks and nine models: 200 the supervisor agent approved at first review and 100 it approved after revision. None contains a factual error, so with 95% confidence at most 1.8% of all replies approved at first review are wrong. None reveals requirements the question did not ask about, and realism averages 4.3/5. For the judge agent, two annotators labeled 300 answers graded by the Luna judge agent (Cohen's κ = 0.81 between them); they agree with it on 74.7% (κ = 0.62), far more than with the GLM judge agent (54.7%, κ = 0.27), and rescoring with human labels keeps the ranking of all nine models.

6. With a client agent from another model family, the two models keep their order (krtE, h8Wa). With Qwen-3.8 Max as client and supervisor agents, Luna still beats Flash (77% vs. 64%). Both score much higher than with the Luna client agent, because the Qwen client agent did not follow the policy: in 39 tasks it revealed values that exist only in the private acceptance tests, and its supervisor agent requested revision on only about 3% of replies (vs. 13–21%). Luna's implementation score matches its complete-requirements score with the Luna client agent (77% vs. 75%), while its clarification score stays lower (82.0 vs. 93.1). The client model's instruction following is therefore key to making the benchmark faithfully simulate the real setting, which is why we use Luna as the client agent.

7. The current client response policy matches how real clients answer (krtE, h8Wa). Requirements-engineering studies find that clients cannot state complete requirements upfront and reveal them in answer to specific questions (Goguen & Linde, 1993; Zowghi & Coulin, 2005; Bano et al., 2019). Our source chat records agree: the 21 freelancer requests for all requirements got brief replies, followed by specific questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.