acceptodds
Under review as a conference paper at ICLR 2027

FeatureLoop: Benchmarking Coding Agents under Evolving Requirements

Abstract

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Primary area:
datasets and benchmarks
Keywords:
benchmark, coding agents, software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o2feOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors' idea is to create a formally defined process to test coding language models' ability and efficiency in fulfilling requirements. They propose testing the agents' capabilities in three ways; first, handing the agents the full requirements list; second, adding requirements iteratively without bug reports from outside (they call this evo); and finally, adding the requirements with an llm's description of the bugs that occurred (they call this progressive plus correction). They evaluate the methods with different metrics measuring correctness and the number of requests in the case of llm-assisted debugging. They conclude that, based on their observations, frontier models' correctness improves when bug reports are fed in.

Critical strengths

The authors describe a structured process to evaluate models and add various additional information in the appendix for further reading and understanding. The authors provide a new coding benchmark that was screened against existing public solutions and is therefore less likely to be already reward-hacked by the market competitors. The authors used great effort to produce a readable and well-told story that does not read as completely automatically created by language models.

Critical weaknesses

The authors don't cite any relevant software engineering reference that formalizes requirements engineering and requirements evolution, which have been studied for many years. The benchmark's validity is never evaluated. It is unclear how it was ensured that the requirements were correct and correctly ordered, and that the tests were also correctly implemented and tested the actual requirements. It could easily be that the 9 tasks that were not solved by any of the 12 models were ill-posed. This was never discussed. It is also unclear to what extent the coding tasks were part of the training data, since the screening only excludes tasks with an existing public full solution. It is further unclear how the benchmark would be prevented from becoming meaningless after publishing the results. It is not clear what the main contribution is, because they showed that their iterative development benchmark runs, but not that any of its parts are valid. Not valid in a formal way, by implementing pipelines that are well understood, nor by actual results that show repeatability. The final pipeline's evaluation seems as engineered as the results seem trivial; that models’ performance improves when bug reports are fed in is expected, even when these are LLM-generated.

Questions for authors
  1. How was it verified that the contract reflects the intent behind the underlying issues and not merely the LLM authors' interpretation? Is there a measure for this, and are there inter-rater agreement values for the human review?
  2. Have the 9 tasks that no model solves been reviewed by experts for solvability and unambiguous specification? How many valid model-generated solutions are rejected by the tests?
  3. Is there a human baseline, even a small sample, showing the developers’ ability to solve the tasks from the user messages alone?
  4. Which established theory or standard of requirements engineering and quality assurance (e.g. requirements engineering, V&V, regression testing) does the process model build on? How do requirement group, requirement state, and evolution/completion feedback map onto known concepts?
  5. The model’s description shows only monotonically growing requirements (Q_i ⊂ Q_{i+1}). How would an impossible or contradictory state be detected and reported separately from model failure?
  6. Is the main finding’s observation (Full < Evo < Evo+Comp) confounded by compute budget (mean 6.9M / 12.1M / 17.8M tokens)? Is there a Full run with an equal budget?
  7. Hy3, DeepSeek V4 Flash/Pro, and GLM 5.3 ran over all candidates and influenced task selection and correction, and Hy4-Preview is both a feedback generator and an evaluated model. How was it examined whether this biases the rankings?
  8. How will the authors ensure the benchmark’s protection against contamination and over-optimization after publication (held-out split, updates)? The screening only excludes tasks with a known public full solution.
13 comments
AuthorsOpenReview ↗

Response to Reviewer o2fe (1/7): W1 / Q4

We thank the reviewer for the detailed critique and for recognizing the structured evaluation process and the screening against existing public solutions. We answer related weaknesses and questions together. All results below are new unless a paper location is given. Revisions in the updated PDF are highlighted in blue.

W1 / Q4: Grounding in requirements engineering

No work formalizing requirements engineering and requirements evolution is cited (W1). Which established theory or standard (e.g., requirements engineering, V&V, regression testing) does the process model build on, and how do requirement groups, states, and evolution/completion feedback map onto known concepts (Q4)?

The submitted paper grounds the preservation checks in regression testing (Section 2.1; [1]), and we agree that requirements engineering deserves explicit treatment. The formulation maps directly onto established concepts:

FeatureLoop conceptEstablished conceptReference
Requirement group with its acceptance checksVerifiable requirement, whose satisfaction is decided by testISO/IEC/IEEE 29148 [2]
Requirement state Q_iRequirements baseline in incremental developmentBoehm [3]; Larman & Basili [4]
Evolution feedback (adds or clarifies requirements)Requirements change under volatilityHarker et al. [5]; Zowghi & Nurmuliani [6]; Lehman [7]
Acceptance checks of the current stateValidation against the current baselineBoehm [3]
Retained checks of earlier groupsRegression testingYoo & Harman [1]
Completion feedback (user-observed behavior that does not yet meet the current baseline)User acceptance feedback in iterative developmentLarman & Basili [4]
Staged protocol in a persistent workspaceIterative and incremental developmentLarman & Basili [4]
Cumulative-state validation of tests against referencesRequirements–test alignmentBjarnason et al. [8]

In these terms, FeatureLoop operationalizes requirements evolution as a sequence of baselines. Each baseline is validated by executable acceptance tests, regression tests protect earlier baselines, and user acceptance feedback refers only to the current baseline.

Revision. Section 2.1 now relates requirement groups, requirement states, and evolution feedback to acceptance criteria, requirements baselines, and requirements change. The new Section 5 (Related Work) covers requirements volatility, software evolution, V&V, and regression testing.

AuthorsOpenReview ↗

Response to Reviewer o2fe (2/7): W2 / Q1–Q3

W2 / Q1–Q3: Validity of the requirements, their order, the tests, and the unsolved tasks

Validity is never evaluated: are the requirements correct and correctly ordered, do the tests check them, and could the nine tasks no model solves be ill-posed (W2)? How is it verified that the contract reflects the issues' intent rather than the LLM authors' interpretation, and what is the inter-rater agreement (Q1)? Were the nine tasks expert-reviewed for solvability, and how many valid solutions do the tests reject (Q2)? Is there a human baseline for solving the tasks from the user messages alone (Q3)?

Validity was evaluated for every task, by human review, by external anchoring, and by evaluation outcomes.

Fidelity to issue intent (Q1). Two human reviewers independently reviewed every task: each checked the contract against its source issues, replayed every dialogue turn at the reference checkpoint, ran the project and tests, and verified that each test assertion follows from the required public behavior (Appendix B.3.10). Before this review, two construction checks independent of the contract's author ground the contract in the issues. First, each task is anchored in at least two complete public issues at an exact repository checkpoint; their text, user motivation, and relevant interfaces are preserved and linked from all later task components, and an independent reviewer confirms that they support the proposed target and its scope (Appendix B.3.1). Second, an independent Contract Reviewer checks that the contract follows the sources, respects state dependencies, is observable through public interfaces, and leaves implementation choices open (Appendix B.3.5).

Measure and inter-rater agreement (Q1). Agreement is a condition of acceptance: a task was accepted only if both reviewers independently approved it, and an objection from either returned it to the responsible construction stage, which reran every affected downstream check. Disagreements were therefore resolved by revising the task rather than by majority, and every released task carries the approval of both reviewers. The revision states this protocol in Appendix B.3.10.

Order and tests (W2). Each check must fail on the preceding reference checkpoint and pass on every later one (Eq. 6, Appendix B.3.9), so each requirement enters at its intended state and remains satisfiable together with all earlier ones. Tests and references are built from the same frozen contract by isolated roles: the Test author works without access to reference code, and the Reference author without access to tests, fixtures, assertions, or test results (Appendices B.3.6 and B.3.8). Their artifacts must agree at every state of every task, so two independent interpretations of each requirement coincide executably.

Rejection of valid solutions (Q2). Contract-valid alternative implementations test whether the checks accept different valid designs, and targeted faulty variants test whether they reject specific defects (Appendix B.3.9). Before freezing, reviewers examined the complete trajectories of four trial models on all 123 candidates for task defects exposed by implementations other than the reference; confirmed defects were corrected and revalidated before the final 61 tasks were accepted (Appendix C.1). After evaluation, all 263 final failures under Evo+Comp, including all 108 on the nine unsolved tasks, were reviewed against the implementation and execution evidence (Appendix F.2). Each was attributed to a specific agent-side error, such as a behavior or API mismatch, a build or runtime break, code not reaching the user-facing path, or inconsistent shared state (Tables 27–28). No failure was attributed to a test rejecting a contract-valid implementation.

AuthorsOpenReview ↗

Response to Reviewer o2fe (3/7): W2 / Q1–Q3 (continued)

W2 / Q1–Q3 (continued)

The nine unsolved tasks (W2, Q2). The submitted paper analyzes them in Section 4.4 (Figure 7) and Appendix F.3 (Table 29). Like every task, each passed the two-reviewer review for source fidelity, solvability, and unambiguous public behavior, and each has reference checkpoints that pass every state together with the repository baseline (Appendix B.3.9). We examined all 36 runs on each task (12 models × 3 workflows; 324 runs in total):

Task (repository)States KMax final NodeCredit (Evo+Comp)Models with partial final credit (Evo+Comp)
NextChat60.8310
ECharts20.505
Excalidraw60.6711
micro60.8311
Nuxt60.000
rtk60.837
bat50.8010
Slidev50.806
Solid60.507

Max final NodeCredit: the largest fraction of final requirement groups satisfied by any of the 12 Evo+Comp runs (Eq. 8). Models with partial final credit: how many of the 12 models satisfy at least one final requirement group in their Evo+Comp run.

  • Models make substantial progress on every task. Every task except Nuxt has runs satisfying 50–83% of its final requirement groups, and 5 to 11 of the 12 models satisfy part of each. On Nuxt, a GPT-5.6 Sol run passes states 1–5 of 6 before the final-stage change regresses earlier behavior.
  • The failures reflect a coherent difficulty profile. Inconsistent state across components accounts for 32.4% of final failures on these tasks, versus 2.6% on solved tasks (Section 4.4, Figure 7); on Nuxt, all 12 final failures are attributed to inconsistent shared state or regression of earlier behavior.
  • The headline results hold without these tasks. Without the nine tasks, the pooled workflow gains are +13.3 and +14.6, both with repository-clustered intervals above zero (Appendix G.4).

The user messages are sufficient (Q3). The question behind a human baseline is whether the user messages contain the information needed to complete each task. Three checks address it directly:

  • The messages describe the behavior that the tests check. Human acceptance simulates the evaluation turn by turn: the reviewer inspects the dialogue shown to the agent, runs the project and tests at the corresponding reference checkpoint, and follows the public observation points to confirm that the reference behavior matches the user semantics the messages express; reviewers also verify that each test assertion follows from the required public behavior (Appendix B.3.10).
  • Missing information is found and corrected before freezing. Reviewers examined the runs of the four trial models on all candidates against the frozen task semantics (see Q2), and repeated runs of two of them flag unstable outcomes, which can indicate an underspecified requirement (Appendix C.1). Confirmed defects were returned to the responsible construction stage and revised.
  • Agents complete most tasks from the messages alone. 52 of 61 tasks are completed by at least one model (under Evo+Comp, 14 by all models and 38 by some; Appendix F.12, Table 40). Agents never see the tests or the reference implementation, and dialogue review ensures that the messages contain no hidden-test wording, reference information, or future requirement (Appendix B.3.5), so these successes rely only on user-level information.
Reviewer_6mbeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provide a benchmark named FeatureLoop. This benchmark distinguishes evolution feedback, which adds or clarifies requirements, from completion feedback, which reports observed failures, and compares complete-upfront, progressive, and progressive-plus-correction workflows. Across 12 models, the best model completes 72.1% of tasks under the correction-enabled workflow, while only 15.4% of failed states recover within two revisions. The paper further analyzes incomplete repairs, regressions, verification paths, model-specific failure profiles, and reported token use.

Critical strengths
  1. The benchmark formulation captures an important gap between final patch correctness and the actual development process. Persistent workspaces, state-conditioned requirement disclosure, corrective reports, and cumulative checks make it possible to distinguish local uptake, complete repair, and preservation; this produces substantially more useful evidence than another terminal pass-rate leaderboard and is the main reason I lean positive.

  2. The construction process is unusually careful for a repository benchmark. Issue-grounded goals, separate test and reference workspaces, cross-state validation, baseline checks, historical screening, human review before and after trial-agent runs, and versioned artifacts collectively provide credible evidence that tasks are feasible and behaviorally meaningful across a reasonably broad set of software forms.

Critical weaknesses
  1. The model-level evidence is underpowered relative to the granularity of the conclusions. There is only one run per model–task–workflow, 61 tasks are clustered in 34 repositories, 33 tasks have only two states, and the per-model confidence intervals overlap heavily; provider-default sampling also leaves run-to-run variance unmeasured. The clustered pooled intervals support an aggregate workflow effect, but rankings, recovery/retention profiles, and token-efficiency comparisons should be treated as descriptive until replicated, especially because reported tokens are not necessarily comparable across providers.

  2. The paper does not provide code or data to rigorously validate the effectiveness of the proposed mechanism, and the writing is somewhat rough.

Questions for authors

Suggestion: Provide the code and update the figures in the paper.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6mbe (1/3): W1 (1)–(3)

We thank the reviewer for the positive assessment of the formulation and construction, and for the careful reading of the statistical evidence. We agree with the distinction the reviewer draws between the pooled workflow effect and the model-level comparisons, and have revised the paper accordingly. All results below are new unless a paper location is given. Revisions in the updated PDF are highlighted in blue.

W1: Strength of the model-level evidence

With one run per cell, 61 tasks in 34 repositories, 33 two-state tasks, overlapping per-model intervals, and unmeasured variance under provider-default sampling, the model-level evidence is underpowered. The clustered intervals support the pooled workflow effect, but rankings, recovery/retention profiles, and token-efficiency comparisons should be treated as descriptive until replicated, especially as reported tokens are not necessarily comparable across providers.

The repository-clustered intervals support the pooled workflow effect, and the evaluation logs now provide model-level evidence for each point below: repeated runs measure single-run variation (1), each model's workflow gain has its own repository-clustered interval above zero (4), three profile contrasts pass Fisher's exact tests, and the FDAC ranking holds across alternative designs (6). As the reviewer suggests, Sections 4.3 and 4.5 and the Limitations paragraph present the model-level comparisons as observations of the recorded runs.

(1) "Only one run per model–task–workflow"

To measure run-to-run variation, we use the repeated runs from task review (Appendix C.1): Hy3 and DeepSeek V4 Flash ran each of their 183 cells at least three times, and up to five times where outcomes disagreed (1,451 valid runs). Each draw of one run per cell gives one possible alternative single-run result; over 10,000 draws:

  • Single-run variation is much smaller than the workflow gains. The single-run SD is 2.4–3.3 points for a completion rate and 3.7–4.4 points for a workflow gain, against gains of 9–18 points.
  • The workflow gains hold whichever run is drawn. Both gains are positive in at least 98.7% of draws for both models, and every 95% interval excludes zero.
  • The reported values are representative of repeated runs. Every reported Table 1 value lies inside its resampled 95% interval.

The paper's workflow conclusions therefore hold under single-run variation. The two models sit at opposite ends of the ranking (tied second and last under Evo+Comp), so the measurement covers both strong and weak performance; the full tables are in our response to Eyon, Q2.

(2) "61 tasks clustered in 34 repositories"

Tasks from the same repository are not independent, so all pooled intervals in the paper resample whole repositories (Appendix E.9); those of Full → Evo (+11.3) and Evo → Evo+Comp (+12.4) lie above zero (Appendix F.8). To check whether any single repository drives the effect, we repeated the analysis 34 times, each time removing all tasks of one repository:

GainAll 61 tasks (Appendix F.8)Range over the 34 leave-one-repository-out analyses
Full → Evo+11.3+8.5 to +12.1
Evo → Evo+Comp+12.4+11.2 to +12.9

The pooled gains hold after removing any single repository, and the correction gain varies by less than two points.

(3) "33 tasks have only two states"

Most trajectory evidence comes from multi-stage development: the 28 tasks with three to eight states contain 141 of the 207 requirement states and 113 of the 146 possible evolution transitions (Table 10). Workflow gains by group (repository-clustered bootstrap, 10,000 draws):

TasksTasks / repositoriesFullEvoEvo+CompFull → EvoEvo → Evo+CompFull → Evo+Comp
Two states33 / 2148.062.677.3+14.6 (100%)+14.6 (100%)+29.3 (100%)
Three to eight states28 / 2231.238.748.5+7.4 (97.0%)+9.8 (100%)+17.3 (100%)

Parentheses give the share of draws with a positive gain. Absolute completion is lower on longer tasks, since a run must satisfy more cumulative states. Even so, every workflow gain is positive in both groups, and positive in nearly all draws.

AuthorsOpenReview ↗

Response to Reviewer 6mbe (2/3): W1 (4)–(6)

W1 (continued)

(4) "Per-model confidence intervals overlap heavily"

Overlapping intervals between models do not affect the comparison within each model, which is paired on the same tasks. The paper's repository-clustered bootstrap (Appendix E.9), applied to each model separately (5,000 draws):

ModelFull (Table 1)Evo+Comp (Table 1)Full → Evo+Comp gain95% intervalDraws with gain > 0
GPT-6 Astra59.072.1+13.1[+1.9, +23.9]98.3%
GPT-5.6 Sol52.570.5+18.0[+6.2, +32.1]99.8%
Kimi K339.370.5+31.1[+19.4, +43.1]100%
Hy4-Preview47.568.9+21.3[+8.8, +33.8]100%
DeepSeek V4 Flash37.770.5+32.8[+18.3, +48.4]100%
DeepSeek V4.1 Flash42.670.5+27.9[+15.8, +40.0]100%
GLM 5.344.365.6+21.3[+10.2, +31.5]100%
Qwen 3.8 Flash39.357.4+18.0[+6.7, +30.0]99.9%
GLM 5.3 Flash36.162.3+26.2[+14.0, +39.0]100%
Grok 4.627.963.9+36.1[+19.7, +53.3]100%
DeepSeek V4 Pro36.160.7+24.6[+10.7, +37.0]100%
Hy321.336.1+14.8[+6.7, +24.1]100%

All 12 models have a 95% interval above zero, so the Full → Evo+Comp effect holds for each model individually, not only in the pooled data. Between models, completion rates that differ by one or two tasks are read as ties, as Section 4.1 does for the four models at 43 successful deliveries.

(5) "Provider-default sampling leaves run-to-run variance unmeasured"

Provider-default sampling is a deliberate choice (Table 18): coding agents are deployed with their providers' default settings, and a fixed non-default temperature would evaluate a configuration that users do not run. This makes run-to-run variance important to quantify, and the repeated runs in point (1) measure it: under the default settings, the single-run SD of the completion rate is 2.4–3.3 points for both a strong and a weak model, and every workflow gain remains positive under resampling.

(6) "Rankings, recovery/retention profiles, and token-efficiency comparisons should be treated as descriptive until replicated"

The revision presents these comparisons as observations of the recorded runs, each with the following support.

Rankings. Table 1 orders models by FDAC for presentation, and Section 4.1 treats equal completion counts as ties. The FDAC sensitivity analysis (Eyon, Q1) shows that this ordering is robust to the discount weights: across 25 designs, the top three models and the last model are unchanged.

Recovery and retention profiles. Each rate in Section 4.3 is reported with its counts over the failures or requirements that the model actually encountered (Appendix F.10). Appendix G.2 adds two-sided Fisher's exact tests on these counts for three contrasts, all significant: retention of DeepSeek V4.1 Flash versus V4 Flash (61/62 vs. 52/62, p = 0.009), and Grok 4.6's retention (33/46 vs. 536/596 for the other 11 models, p = 0.0009) and Recover@2 (18/73 vs. 95/660, p = 0.026).

Token efficiency. Section 4.5 now refers to reported tokens per successful delivery; point (7) explains how these counts are recorded and what they measure.

AuthorsOpenReview ↗

Response to Reviewer 6mbe (3/3): W1 (7), W2 / Suggestion

W1 (continued)

(7) "Reported tokens are not necessarily comparable across providers"

We agree that reported tokens are not a common unit across providers, and the revision makes this explicit. For every model call, the harness records the input and output token counts that the provider returns in its API response and accumulates them over the run (Appendix D.2: "Agent Tokens accumulates reported input and output tokens over every call in the run"); no tokens are re-estimated or re-tokenized by the evaluator. We use reported usage because it is the quantity by which each model is metered and billed, that is, the resource a developer actually spends; a common tokenizer would give a shared unit, but not the quantity users pay for. The two kinds of comparison are affected differently:

  • Within-model comparisons use a single tokenizer. The workflow-level findings, for example that correction increases tokens per run by 47.2% (Section 4.5) and the matched-pair analysis under o2fe Q6, compare runs of the same model.
  • The cross-model contrast is a comparison of reported usage. GLM 5.3 reports 7.1 times as many tokens per successful delivery as Grok 4.6 at a similar completion rate (Section 4.5; Table 42: 42.29M vs. 5.97M), a gap well beyond typical differences in tokenizer granularity. Section 4.5 now presents it as a difference in reported usage.

Revision. The leave-one-repository-out, task-depth, per-model, and profile-significance analyses are in Appendices G.1 and G.2 (Tables 45–48), and Table 44 summarizes the workflow gains under every check. Section 4.1 states that both gains persist under these checks, and Section 3 states the source of the token counts.

W2 / Suggestion: Code, data, writing, and figures

No code or data are provided to validate the proposed mechanism, and the writing is somewhat rough. Suggestion: provide the code and update the figures.

The complete benchmark and evaluation artifacts are at https://anonymous.4open.science/r/featureloop-DE66:

  • all 61 tasks, with their contracts, dialogues, tests, starting repositories, and reference implementations;
  • the evaluation code, including the three protocols and scoring;
  • all run logs of the evaluation, including the repeated runs of Appendix C.1.

We have improved the writing and reorganized the main text as described in our response to Eyon (W3). On the figures, the breakdown of local versus complete success is now Figure 5 in Section 4.2. Figures 6–9 are redrawn at print size with a consistent color scheme: Figures 6 and 9 give each model its own marker and a direct label, Figure 7 orders the failure diagnoses by their share on tasks that no model completes, and Figure 9 adds the pooled token use of each workflow (left). In Section 2, Figures 2 and 3 are now the full protocol and construction diagrams (formerly Figures 15 and 10): Figure 2 shows disclosure checks and both feedback paths, Figure 3 marks the admission step from history screening to reference construction, and Figure 4(b) shows stage-level ranges as interquartile bars. Figures 1–3 and the two case diagrams (Figures 10 and 15) now share one panel layout. Removing the forced page breaks also eliminates the blank space they left in Sections 2 and 4.

We thank the reviewer again for the careful reading of the statistical evidence, and we would be glad to clarify any remaining points.

Dear Reviewer 6mbe,

Thank you again for your time and careful reading of our statistical evidence. As the discussion period closes on November 18 (AoE), we would like to check whether our response has addressed your concerns. It adds repeated-run, per-model, leave-one-repository-out, and significance analyses for the model-level evidence, and releases the complete code and data. We would be happy to clarify anything further.

Best regards,

The Authors

Reviewer_EyonOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating coding agents under evolving requirements in a persistent repository workspace. It distinguishes feedback that introduces new requirements from feedback that reports failures of existing requirements, and uses cumulative executable checks to assess implementation, repair, and preservation of earlier functionality. Experiments compare complete specifications upfront, progressive requirements, and progressive requirements with corrective feedback. The results show higher final completion rates with progressive interaction, while revealing limited recovery of failed stages and regressions in previously working functionality.

Critical strengths

1.The benchmark addresses a useful evaluation setting: coding agents must incorporate new requirements, respond to failure reports, and preserve earlier functionality in the same repository.

2.The task construction includes several safeguards: separate test and reference implementation authoring, cross-stage validation, human review, and trial agent runs.

Critical weaknesses

1.FDAC conflates necessary requirement disclosure with avoidable corrective effort. It gives full credit to pre-disclosure completion but only half credit to immediate completion after disclosure, even though 85 of 147 later requirement groups require previously unavailable information. The discount factors and the rule that one unresolved intermediate stage forces the entire task score to zero also lack sufficient validation.

2.The workflow comparisons do not adequately quantify run-to-run variability. Appendix F.8 reports that 57 of the 119 model–task pairs that change from failure under Evo to success under Evo+Comp receive no corrective feedback in the successful run. This highlights the importance of variation between independently executed trajectories. Each model–task–protocol combination is evaluated only once, while the repository-clustered bootstrap measures sensitivity to the repository sample rather than repeated execution. These results do not invalidate the aggregate workflow gains, but repeated runs are needed to establish the stability of model-specific gains and conclusions about responsiveness to correction.

  1. The allocation of content between the main paper and appendix obscures the central contributions. Several main-text pages contain substantial unused space, while essential material—including the full FDAC definition, detailed recovery results, and the breakdown distinguishing local repair from cumulative completion—is deferred to the appendix. Meanwhile, a lengthy AI use statement occupies space on page 9. The concern is not that the paper must fill the page limit, but that the available space is not used to present the evidence needed to assess its main claims. The main text should prioritize the core evaluation design and decisive experimental results so that the contribution can be understood and assessed without extensive navigation through the appendix.
Questions for authors

1.How do FDAC scores and model rankings change if immediate completion after disclosure receives full credit for requirements that depend on previously unavailable information? Are the rankings robust to different discount factors and to relaxing the rule that one unresolved intermediate stage makes the entire task score zero?

2.Appendix F.8 reports that 57 of the 119 failure-to-success transitions occur without corrective feedback in the successful run. Do you have repeated-run results that quantify execution variability and establish whether the model-specific workflow gains remain stable? If not, could you provide such results for a representative subset of models and tasks?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Eyon (1/3): W1 / Q1

We thank the reviewer for the precise and constructive questions. Both can be answered from the existing evaluation data, and both analyses support the paper's conclusions. All results below are new unless a paper location is given. Revisions in the updated PDF are highlighted in blue.

W1 / Q1: FDAC and necessary disclosure; robustness to the discount factors and the zero rule

FDAC gives only half credit to completion immediately after disclosure, although 85 of 147 later requirement groups require previously unavailable information, and one unresolved intermediate stage forces the task score to zero. How do scores and rankings change under full credit for such groups, other discount factors, or a relaxed zero rule?

FDAC measures development from the user's side. Whether a disclosure was avoidable or necessary, the user still had to make another request before the requirement was met, so the default weights (Eq. 11) credit trajectories that need fewer user interventions. To test whether other treatments of disclosure, the zero rule, or the discount factors would change the conclusions, we recomputed FDAC from the state-level trajectories of all 732 Evo+Comp runs, varying each design choice in turn:

  • Disclosure credit: full credit for immediate completion after disclosure for the 85 later-information groups (need-foreseeability N=0 or discovery depth D=4, Table 15).
  • Zero rule: an arithmetic instead of a geometric mean, or unresolved states floored at 0.05 instead of 0.
  • Combined: both relaxations at once.
  • FDAC-soft (Eq. 13).
  • Discount factors: the after-disclosure factor {0.25, 0.5, 0.75, 1} × the per-revision factor {0.2, 0.4, 0.6, 0.8, 1}, 19 settings besides the original (0.5, 0.4).

Agreement with the original ranking:

VariantKendall's τ vs. original FDACMax rank shift
Full credit after disclosure for the 85 later-information groups0.912
Arithmetic mean (no zero rule)0.912
Unresolved states floored at 0.050.912
Both relaxations combined0.852
FDAC-soft (Eq. 13)0.823
19-point grid over both factors0.77 / 0.88 / 0.94 (min / median / max)≤ 3

Per-model scores (rank in parentheses):

ModelOriginal FDACLater-info full creditArithmetic meanBoth relaxationsRank range over all 25 variants
GPT-6 Astra46.5 (1)62.3 (1)49.9 (1)64.0 (1)1–2
GPT-5.6 Sol45.4 (2)59.5 (2)49.8 (2)62.7 (2)1–3
Kimi K344.2 (3)58.1 (3)47.4 (3)60.6 (3)2–3
Hy4-Preview42.7 (4)56.0 (4)46.5 (4)58.6 (5)4–5
DeepSeek V4 Flash41.2 (5)53.7 (6)46.0 (6)58.2 (6)5–7
DeepSeek V4.1 Flash40.8 (6)54.1 (5)46.1 (5)58.7 (4)4–7
GLM 5.340.5 (7)53.6 (7)44.1 (7)57.1 (7)5–7
Qwen 3.8 Flash37.3 (8)49.9 (8)39.9 (10)51.5 (10)8–11
GLM 5.3 Flash37.2 (9)48.7 (11)41.7 (8)52.7 (9)8–11
Grok 4.637.1 (10)49.4 (9)40.8 (9)53.1 (8)8–11
DeepSeek V4 Pro36.7 (11)49.0 (10)39.8 (11)51.2 (11)8–11
Hy321.2 (12)27.8 (12)23.3 (12)28.9 (12)12

Findings.

  • The tiered ranking is invariant. In all 25 variants, GPT-6 Astra, GPT-5.6 Sol, and Kimi K3 are the top three and Hy3 is last; ranks 4–7 and 8–11 are always held by the same models, and no model moves more than three positions.
  • The ranking holds without the zero rule and with full disclosure credit. Replacing the geometric mean or flooring unresolved states gives τ = 0.91, and so does full credit after disclosure, which raises scores across models while largely preserving their order.

Revision. Appendix E.3 now contains these tables next to the FDAC definition (Tables 22–23), including the disclosure-aware variant, and Section 3 reports their range where FDAC is defined (τ = 0.77–0.94 across 24 alternative designs).

AuthorsOpenReview ↗

Response to Reviewer Eyon (2/3): W2 / Q2

W2 / Q2: Run-to-run variability and model-specific workflow gains

Each model–task–protocol combination is run once, the repository-clustered bootstrap does not measure repeated execution, and 57 of the 119 failure-to-success transitions in Appendix F.8 occur without corrective feedback. Do repeated runs establish whether model-specific gains are stable, at least for a representative subset?

Yes. Such repeated runs already exist from task review (Appendix C.1), where repeated execution finds tasks with unstable outcomes so that they can be examined for defects such as an underspecified requirement or a nondeterministic check; a task with such a defect was revised or discarded before freezing. Two trial models, Hy3 and DeepSeek V4 Flash, ran every task under every workflow at least three times. A cell whose runs agreed on every requirement group was settled; otherwise it was extended to five runs and settled by a strict majority, and cells still without a majority were reviewed with their trajectories. Each of their 183 cells on the final 61 tasks therefore has at least three valid runs (742 and 709, respectively). Every counted run uses task content, checks, protocol, and harness identical to the frozen release, so the repeats are exchangeable executions of the same cell and measure run-to-run variability directly.

These two models form a representative subset. They sit at opposite ends of the ranking (DeepSeek V4 Flash is tied second under Evo+Comp; Hy3 is last), so the measurement covers both strong and weak performance. Bringing the other ten models to the same depth would add roughly 5,400 runs and about USD 100K, because each run is a complete multi-stage development in a real repository: one pass of the 12 models comprises 2,196 runs with about 26.9B reported tokens and 2,460 agent-hours (on average about 12.2M tokens and 67 minutes per run) and costs about USD 36K in API fees.

Procedure. For each model and workflow, we compare the reported leaderboard value (Table 1, the latest valid run of each cell) with the distribution obtained by drawing one valid run per cell 10,000 times (seed 20261106), summarized by its mean, its standard deviation (the single-run SD), and its 95% interval.

Completion rate by workflow (Table 9):

ModelWorkflowRuns per cellReported (Table 1)Resampled meanSingle-run SD95% interval
DeepSeek V4 FlashFull4.0037.738.63.0[32.8, 44.3]
Evo3.9857.452.73.3[45.9, 59.0]
Evo+Comp3.6470.570.42.7[65.6, 75.4]
Hy3Full4.1621.322.12.8[16.4, 27.9]
Evo3.9832.831.22.4[26.2, 36.1]
Evo+Comp4.0236.141.03.3[34.4, 47.5]

Workflow gains (percentage points, Table 9):

ModelGainReported (Table 1)Resampled meanSingle-run SD95% intervalDraws with gain > 0
DeepSeek V4 FlashFull → Evo+19.7+14.14.4[+4.9, +23.0]99.9%
Evo → Evo+Comp+13.1+17.74.3[+9.8, +26.2]100.0%
Hy3Full → Evo+11.5+9.03.7[+1.6, +16.4]98.7%
Evo → Evo+Comp+3.3+9.84.1[+1.6, +18.0]98.7%

Findings.

  • Model-specific workflow gains are stable under repeated execution. For both models, both gains are positive in at least 98.7% of draws, and every 95% interval excludes zero.
  • The leaderboard values are representative. Every reported value lies inside its resampled 95% interval.
  • Single-run variability is small relative to the workflow effects. The single-run SD is 2.4–3.3 points per workflow and 3.7–4.4 points per gain, so gains of 9–18 points are two to four times their own SD.
AuthorsOpenReview ↗

Response to Reviewer Eyon (3/3): W2 / Q2 (continued), W3

W2 / Q2 (continued)

Every model shows a positive workflow effect. To extend the evidence to all 12 models, we apply the paper's repository-clustered bootstrap (Appendix E.9) to each model's reported runs: the Full → Evo+Comp gain is positive for every model, each with a 95% interval above zero (table in our response to 6mbe, W1).

The 57 rescues in Appendix F.8. Appendix F.8 separates paired outcome changes, which compare the final outcomes of two independently executed workflows, from within-run recovery, in which a failed state becomes correct after completion feedback in the same trajectory. The paper's claims about correction rest on the second: of the 119 rescued pairs, 54 contain an observed same-state recovery after feedback, and all Recover@1/Recover@2 counts (Table 38) are within-run. The 57 pairs without a completion message are runs in which no failure was observed, so no feedback was triggered; their outcome change compares two separate executions, precisely the variation quantified above. Under resampling of the repeated executions, the correction gain remains positive in 100% and 98.7% of draws for the two models, so the workflow effect is not an artifact of comparing single trajectories.

Further evidence from the submitted paper. Several model-specific changes far exceed the measured single-run variability: Kimi K3 rises from 39.3% under Full to 70.5% under Evo+Comp (Table 1), and Grok 4.6 gains 26.2 points from correction, with 16 newly completed tasks and none lost (Table 35).

Revision. Appendix C.1 now describes the repeated runs and the variation they measure (Table 9), and Appendix F.8 refers to it. Appendix D.1 states that each reported cell is its latest valid run, and Appendix G.1 (Table 47) gives the per-model gains for all 12 models.

W3: Allocation of content between the main text and appendix

Main-text pages leave unused space while essential material (the full FDAC definition, detailed recovery results, and the breakdown of local repair versus cumulative completion) is deferred to the appendix, and a lengthy AI-use statement occupies page 9.

We have reorganized the main text so that the core evaluation design and decisive results can be assessed without the appendix:

  1. Full FDAC definition in Section 3. The state factors and task aggregation, previously only in Appendix E.3 (Eqs. 11–12), are now stated in Section 3.
  2. Recovery results and the local-versus-cumulative breakdown in Section 4.2 (Figure 5), previously only in Table 24 and Appendices F.7 and F.10:
    • of 927 first responses to requests that introduce previously unavailable information, 77.3% implement the requested behavior; in 8.2% of all 927, the new behavior passes while an earlier requirement fails;
    • 113 of 733 failed states (15.4%) recover within two revisions, 95 of them after the first;
    • 19 of 91 successful individual repairs (20.9%) still leave another requirement unmet;
    • 73 of 642 previously passing requirement groups (11.4%) later fail.
  3. Robustness stated with the findings it supports. Section 4.1 states that both workflow gains persist under repository removal, model exclusions, and repeated runs, and that every model gains from Full to Evo+Comp; Section 4.3 reports the significance of the profile contrasts; and Section 4.5 shows that the workflow ordering holds at matched token use. Appendices C.1, E.3, and G give every analysis, with Table 44 summarizing both gains under every check.
  4. A shorter AI-use statement, reduced to three sentences among the unnumbered statements after the main text.
  5. Space used for evidence. Removing the forced page breaks frees the space that now holds the FDAC definition, Figure 5, the recovery and preservation counts, and the new Related Work section (Section 5).

We hope these analyses answer the reviewer's questions on FDAC, run-to-run variability, and the allocation of content, and we would be glad to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive and detailed feedback, and are encouraged that they found the evaluation setting useful (Eyon), the formulation to capture "an important gap between final patch correctness and the actual development process" and the construction "unusually careful" (6mbe), and the benchmark well structured and screened against existing public solutions (o2fe).

The reviews raise five central questions, each answered with new evidence from the evaluation logs:

1. Are the workflow conclusions stable under repeated execution and across the task mixture? Yes.

  • During task review, two of the trial models, Hy3 and DeepSeek V4 Flash, ran every task under every workflow at least three times, up to five where outcomes disagreed (1,451 valid runs over their 183 cells each). Drawing one run per cell 10,000 times, both workflow gains are positive in at least 98.7% of draws for both models.
  • Removing any one of the 34 repositories leaves both pooled gains between +8.5 and +12.9 points, and both gains are positive on two-state tasks and on tasks with three to eight states.
  • All 12 models gain from Full to Evo+Comp, each with a repository-clustered 95% interval above zero.

(Details: Eyon Q2; 6mbe W1.)

2. Are FDAC rankings robust to its weights? Yes. Across 24 alternative FDAC designs (full credit for requirements that need later information, no zero rule, a grid over both discount factors), Kendall's τ with the original ranking is 0.77–0.94; the top three and the last model never change, and no model moves more than three positions. (Details: Eyon Q1.)

3. Are the rankings and pooled findings free of bias from the task-selection and feedback-selection models? Yes. The four task-selection models span the leaderboard rather than clustering at the top (Hy3 is last in every workflow). Hy4-Preview ranks third under Full (no feedback) but sixth by completion rate under Evo+Comp, where it selects the feedback; because it only selects predefined observation points, it cannot add information. Excluding all five models, both pooled gains remain positive (+11.5 and +12.9) with repository-clustered 95% intervals above zero. (Details: o2fe Q7.)

4. Are the model-level profile contrasts statistically significant? Yes. Recovery and retention are counted for all 12 models (Table 38), and three profile contrasts in Section 4.3 are significant under two-sided Fisher's exact tests (p ≤ 0.026): DeepSeek V4.1 Flash retains more than V4 Flash, and Grok 4.6 recovers more but retains less than the other 11 models. (Details: 6mbe W1.)

5. Are the tasks valid and solvable? Yes. Tests and references are built by isolated roles yet agree at every state of every task, and contract-valid alternative implementations probe acceptance of other valid designs. 52 of 61 tasks are completed by at least one model from the user messages alone, and all 263 final failures are traced to specific agent-side errors, none to a test rejecting a contract-valid implementation. On eight of the remaining nine tasks, some runs end with 50–83% of the final requirement groups met; on the ninth, a run passes five of its six states before the last stage breaks them. (Details: o2fe W2 / Q1–Q3.)

Revised PDF. Numbering follows the submitted version, except that Scope and Limitations and Conclusion are now Sections 6–7, the full protocol and construction diagrams (Figures 15 and 10) replace Figures 2 and 3, Figures 5–9, 16, and 18–27 are now 6–10, 15, and 16–25, and Tables 9–20 and 21–40 are now 10–21 and 24–43; numbers below refer to the revised PDF, in which revisions are highlighted in blue. The main changes are:

  1. Robustness. Section 4.1 states that both gains persist under repository removal, model exclusions, and repeated runs. Appendix C.1 reports the repeated runs (Table 9), Appendix E.3 the FDAC sensitivity (Tables 22–23), and the new Appendix G the remaining analyses (task mixture and per-model gains, profile significance, selection and feedback models, unsolved tasks, compute), summarized in Table 44.
  2. Main text. Section 3 contains the full FDAC definition, states that no token or time budget is imposed, and defines Agent Tokens as provider-reported usage. Section 4.2 adds Figure 5 (local vs. complete-state success) and reports recovery and preservation with counts; Section 4.5 shows that the workflow ordering holds at matched token use.
  3. Requirements engineering. Section 2.1 maps the formulation to acceptance criteria, requirements baselines, and requirements change; the new Section 5 (Related Work) draws on requirements engineering; Section 6 states the cumulative scope.
  4. Appendices and statements. Appendix B.7 adds post-release maintenance, D.1 states which run each cell reports, and D.2 spells out how completion messages are composed.
  5. Artifacts. Tasks, evaluation code, and run logs: https://anonymous.4open.science/r/featureloop-DE66

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.