- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors' idea is to create a formally defined process to test coding language models' ability and efficiency in fulfilling requirements. They propose testing the agents' capabilities in three ways; first, handing the agents the full requirements list; second, adding requirements iteratively without bug reports from outside (they call this evo); and finally, adding the requirements with an llm's description of the bugs that occurred (they call this progressive plus correction). They evaluate the methods with different metrics measuring correctness and the number of requests in the case of llm-assisted debugging. They conclude that, based on their observations, frontier models' correctness improves when bug reports are fed in.
Critical strengths
The authors describe a structured process to evaluate models and add various additional information in the appendix for further reading and understanding. The authors provide a new coding benchmark that was screened against existing public solutions and is therefore less likely to be already reward-hacked by the market competitors. The authors used great effort to produce a readable and well-told story that does not read as completely automatically created by language models.
Critical weaknesses
The authors don't cite any relevant software engineering reference that formalizes requirements engineering and requirements evolution, which have been studied for many years. The benchmark's validity is never evaluated. It is unclear how it was ensured that the requirements were correct and correctly ordered, and that the tests were also correctly implemented and tested the actual requirements. It could easily be that the 9 tasks that were not solved by any of the 12 models were ill-posed. This was never discussed. It is also unclear to what extent the coding tasks were part of the training data, since the screening only excludes tasks with an existing public full solution. It is further unclear how the benchmark would be prevented from becoming meaningless after publishing the results. It is not clear what the main contribution is, because they showed that their iterative development benchmark runs, but not that any of its parts are valid. Not valid in a formal way, by implementing pipelines that are well understood, nor by actual results that show repeatability. The final pipeline's evaluation seems as engineered as the results seem trivial; that models’ performance improves when bug reports are fed in is expected, even when these are LLM-generated.
Questions for authors
- How was it verified that the contract reflects the intent behind the underlying issues and not merely the LLM authors' interpretation? Is there a measure for this, and are there inter-rater agreement values for the human review?
- Have the 9 tasks that no model solves been reviewed by experts for solvability and unambiguous specification? How many valid model-generated solutions are rejected by the tests?
- Is there a human baseline, even a small sample, showing the developers’ ability to solve the tasks from the user messages alone?
- Which established theory or standard of requirements engineering and quality assurance (e.g. requirements engineering, V&V, regression testing) does the process model build on? How do requirement group, requirement state, and evolution/completion feedback map onto known concepts?
- The model’s description shows only monotonically growing requirements (Q_i ⊂ Q_{i+1}). How would an impossible or contradictory state be detected and reported separately from model failure?
- Is the main finding’s observation (Full < Evo < Evo+Comp) confounded by compute budget (mean 6.9M / 12.1M / 17.8M tokens)? Is there a Full run with an equal budget?
- Hy3, DeepSeek V4 Flash/Pro, and GLM 5.3 ran over all candidates and influenced task selection and correction, and Hy4-Preview is both a feedback generator and an evaluated model. How was it examined whether this biases the rankings?
- How will the authors ensure the benchmark’s protection against contamination and over-optimization after publication (held-out split, updates)? The screening only excludes tasks with a known public full solution.
13 comments
Response to Reviewer o2fe (2/7): W2 / Q1–Q3
W2 / Q1–Q3: Validity of the requirements, their order, the tests, and the unsolved tasks
Validity is never evaluated: are the requirements correct and correctly ordered, do the tests check them, and could the nine tasks no model solves be ill-posed (W2)? How is it verified that the contract reflects the issues' intent rather than the LLM authors' interpretation, and what is the inter-rater agreement (Q1)? Were the nine tasks expert-reviewed for solvability, and how many valid solutions do the tests reject (Q2)? Is there a human baseline for solving the tasks from the user messages alone (Q3)?
Validity was evaluated for every task, by human review, by external anchoring, and by evaluation outcomes.
Fidelity to issue intent (Q1). Two human reviewers independently reviewed every task: each checked the contract against its source issues, replayed every dialogue turn at the reference checkpoint, ran the project and tests, and verified that each test assertion follows from the required public behavior (Appendix B.3.10). Before this review, two construction checks independent of the contract's author ground the contract in the issues. First, each task is anchored in at least two complete public issues at an exact repository checkpoint; their text, user motivation, and relevant interfaces are preserved and linked from all later task components, and an independent reviewer confirms that they support the proposed target and its scope (Appendix B.3.1). Second, an independent Contract Reviewer checks that the contract follows the sources, respects state dependencies, is observable through public interfaces, and leaves implementation choices open (Appendix B.3.5).
Measure and inter-rater agreement (Q1). Agreement is a condition of acceptance: a task was accepted only if both reviewers independently approved it, and an objection from either returned it to the responsible construction stage, which reran every affected downstream check. Disagreements were therefore resolved by revising the task rather than by majority, and every released task carries the approval of both reviewers. The revision states this protocol in Appendix B.3.10.
Order and tests (W2). Each check must fail on the preceding reference checkpoint and pass on every later one (Eq. 6, Appendix B.3.9), so each requirement enters at its intended state and remains satisfiable together with all earlier ones. Tests and references are built from the same frozen contract by isolated roles: the Test author works without access to reference code, and the Reference author without access to tests, fixtures, assertions, or test results (Appendices B.3.6 and B.3.8). Their artifacts must agree at every state of every task, so two independent interpretations of each requirement coincide executably.
Rejection of valid solutions (Q2). Contract-valid alternative implementations test whether the checks accept different valid designs, and targeted faulty variants test whether they reject specific defects (Appendix B.3.9). Before freezing, reviewers examined the complete trajectories of four trial models on all 123 candidates for task defects exposed by implementations other than the reference; confirmed defects were corrected and revalidated before the final 61 tasks were accepted (Appendix C.1). After evaluation, all 263 final failures under Evo+Comp, including all 108 on the nine unsolved tasks, were reviewed against the implementation and execution evidence (Appendix F.2). Each was attributed to a specific agent-side error, such as a behavior or API mismatch, a build or runtime break, code not reaching the user-facing path, or inconsistent shared state (Tables 27–28). No failure was attributed to a test rejecting a contract-valid implementation.
Response to Reviewer o2fe (3/7): W2 / Q1–Q3 (continued)
W2 / Q1–Q3 (continued)
The nine unsolved tasks (W2, Q2). The submitted paper analyzes them in Section 4.4 (Figure 7) and Appendix F.3 (Table 29). Like every task, each passed the two-reviewer review for source fidelity, solvability, and unambiguous public behavior, and each has reference checkpoints that pass every state together with the repository baseline (Appendix B.3.9). We examined all 36 runs on each task (12 models × 3 workflows; 324 runs in total):
| Task (repository) | States K | Max final NodeCredit (Evo+Comp) | Models with partial final credit (Evo+Comp) |
|---|---|---|---|
| NextChat | 6 | 0.83 | 10 |
| ECharts | 2 | 0.50 | 5 |
| Excalidraw | 6 | 0.67 | 11 |
| micro | 6 | 0.83 | 11 |
| Nuxt | 6 | 0.00 | 0 |
| rtk | 6 | 0.83 | 7 |
| bat | 5 | 0.80 | 10 |
| Slidev | 5 | 0.80 | 6 |
| Solid | 6 | 0.50 | 7 |
Max final NodeCredit: the largest fraction of final requirement groups satisfied by any of the 12 Evo+Comp runs (Eq. 8). Models with partial final credit: how many of the 12 models satisfy at least one final requirement group in their Evo+Comp run.
- Models make substantial progress on every task. Every task except Nuxt has runs satisfying 50–83% of its final requirement groups, and 5 to 11 of the 12 models satisfy part of each. On Nuxt, a GPT-5.6 Sol run passes states 1–5 of 6 before the final-stage change regresses earlier behavior.
- The failures reflect a coherent difficulty profile. Inconsistent state across components accounts for 32.4% of final failures on these tasks, versus 2.6% on solved tasks (Section 4.4, Figure 7); on Nuxt, all 12 final failures are attributed to inconsistent shared state or regression of earlier behavior.
- The headline results hold without these tasks. Without the nine tasks, the pooled workflow gains are +13.3 and +14.6, both with repository-clustered intervals above zero (Appendix G.4).
The user messages are sufficient (Q3). The question behind a human baseline is whether the user messages contain the information needed to complete each task. Three checks address it directly:
- The messages describe the behavior that the tests check. Human acceptance simulates the evaluation turn by turn: the reviewer inspects the dialogue shown to the agent, runs the project and tests at the corresponding reference checkpoint, and follows the public observation points to confirm that the reference behavior matches the user semantics the messages express; reviewers also verify that each test assertion follows from the required public behavior (Appendix B.3.10).
- Missing information is found and corrected before freezing. Reviewers examined the runs of the four trial models on all candidates against the frozen task semantics (see Q2), and repeated runs of two of them flag unstable outcomes, which can indicate an underspecified requirement (Appendix C.1). Confirmed defects were returned to the responsible construction stage and revised.
- Agents complete most tasks from the messages alone. 52 of 61 tasks are completed by at least one model (under Evo+Comp, 14 by all models and 38 by some; Appendix F.12, Table 40). Agents never see the tests or the reference implementation, and dialogue review ensures that the messages contain no hidden-test wording, reference information, or future requirement (Appendix B.3.5), so these successes rely only on user-level information.
Response to Reviewer o2fe (1/7): W1 / Q4
We thank the reviewer for the detailed critique and for recognizing the structured evaluation process and the screening against existing public solutions. We answer related weaknesses and questions together. All results below are new unless a paper location is given. Revisions in the updated PDF are highlighted in blue.
W1 / Q4: Grounding in requirements engineering
The submitted paper grounds the preservation checks in regression testing (Section 2.1; [1]), and we agree that requirements engineering deserves explicit treatment. The formulation maps directly onto established concepts:
In these terms, FeatureLoop operationalizes requirements evolution as a sequence of baselines. Each baseline is validated by executable acceptance tests, regression tests protect earlier baselines, and user acceptance feedback refers only to the current baseline.
Revision. Section 2.1 now relates requirement groups, requirement states, and evolution feedback to acceptance criteria, requirements baselines, and requirements change. The new Section 5 (Related Work) covers requirements volatility, software evolution, V&V, and regression testing.