acceptodds
Under review as a conference paper at ICLR 2027

NL2FullStack: Evaluating Data Modeling and Full-Stack Application Generation from Natural Language

Abstract

Large language model (LLM) coding agents can build complete database-backed applications from written briefs, but existing benchmarks grade user-visible workflows or the resemblance of a generated schema to a single reference, so backend business logic and data integrity are overlooked. We present NL2FullStack, a benchmark for data modeling and full-stack application generation from natural language. An agent receives a business brief and a binding engineering contract, then must ship a bootable application with a backend that implements the required workflows and a database that enforces the specified rules. Grading is deterministic and uses no LLM judge: the harness replays hidden, seeded workloads over the application's declared routes to test business logic, probes the database directly to verify integrity constraints, and checks that the interface displays backend rows. Across 89 tasks with five attempts each, Claude Opus 4.8 passes 62.9% of attempts, ahead of GPT-5.6 Sol (48.8%), Kimi K3 (44.5%), DeepSeek V4 Pro (29.2%), and Qwen3.8-Max (27.2%). The top four models pass the database tier on 87–96% of attempts, so nearly all the gap between models comes from the backend tier, where pass rates fall from 68% to 38%. For every model, the dominant failure is rejecting valid input under constraints absent from the contract. To mitigate contamination from public benchmarks, we release a nine-task public set with the grader, agent harness, and every graded attempt, and hold out eighty tasks as a private evaluation set.

Primary area:
datasets and benchmarks
Keywords:
benchmark, coding agents, full-stack application generation, data modeling, database schema, data integrity, business logic, deterministic evaluation, LLM agents, software engineering, text-to-application, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CLycOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NL2FullStack introduces 89 tasks for building database-backed applications from business briefs and engineering contracts. Its grader combines direct database tests, HTTP workloads checked against a hidden reference implementation and interface checks. The paper compares five models and analyzes why their applications fail.

Critical strengths

The direct SQL tests check whether the database enforces constraints when writes bypass application code. This is a useful addition to the interface, API and database snapshots examined by FullStack-Bench [1]. The manifest lets applications use different schemas. Tests with correct, no-op and near-miss implementations check whether the grader catches the intended defects. This targets applications that appear to work while allowing invalid data.

[1] FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translation. https://arxiv.org/abs/2602.03798v1

Critical weaknesses
  1. The paper gives conflicting accounts of the grader used for the reported results. Section 5 says grading problems were corrected before evaluation. Appendix J says month and timestamp formats were undisclosed and the workloads remain as shipped. It also describes the reference implementation selling a cemetery plot when the brief forbids it, and about 47 uniqueness failures where the database enforces the key but the probe misses it. These may concern an earlier version, but the paper does not say so. If they remain in the final results, the failure analysis mixes model errors with disagreements between the tests and the specification. This particularly affects the conclusion that models mostly fail by rejecting valid inputs.
  2. The reliability results do not follow the stated five-attempt protocol. Table 9 contains 385 Opus builds, with three to five attempts per task. These counts correctly reproduce the 62.9 percent mean, and the 30 always-solved tasks fit Table 3's definition of success on all observed attempts. However, Table 4 labels 34 percent as pass^5, although only 11 tasks show five successes in five attempts. Counting tasks solved at least once across varying numbers of attempts also does not directly establish pass@5. These metrics need an explanation of how tasks with fewer attempts were handled.
Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

One frozen grader; complete five-attempt protocol

Thank you.

W1. Conflicting grader accounts. The appendix passages you quote described an earlier grading pass and should not have sat beside Section 5's account. The revision has one account, in Section 5 and Appendix L, and every reported attempt was graded once under that frozen grader:

(a) Month and timestamp forms: disclosed and handled by a retry that never retries a write meant to be refused (it fires on 49 of 2,225 attempts). (b) Gold that contradicts its brief: the cemetery-plot sale and six others were rewritten to respect the brief, with each probe's tested property kept. (c) Clock-dependent gold: three tasks fixed so the outcome no longer depends on the grading date.

On the remaining uniqueness-probe blind spot, an upper bound from the released records credits every attempt whose only failure is a Tier-A uniqueness loss. No model moves more than 7 points and the ranking is unchanged (Opus 67.6 to 68.5, Sol 55.5 to 62.5, Kimi 53.3 to 57.3, Qwen 45.6 to 47.2, DeepSeek 33.5 to 37.3).

We also replayed all 2,225 applications under nine further grader changes (409 + error envelope on refusals, ok:true on reads, missing fields not equal to null, complete money assignment, ordered reads, timestamps to the second, bipartite interface matching, database failures voiding a verdict). None changed a verdict.

The over-strictness conclusion is unaffected. Refusing a valid write is still the most common first failure for four of five models (32 to 40% of their losing attempts), and over-strict Tier-B failures outnumber over-permissive ones by 15:1 to 106:1 per model.

W2. Five-attempt protocol. The submission held only 385 of Opus's 445 attempts because the run was still being completed. Every model now has exactly 5 attempts on all 89 tasks. pass^5 is the fraction of tasks with 5/5 (Opus: 31 tasks, 35%) and pass@5 the fraction with at least one pass (Opus: 76/89), so no task has fewer attempts and neither metric needs special handling.

Reviewer_wbFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

NL2FullStack evaluates whether LLM coding agents can generate running, database-backed applications that implement the business rules and data-integrity requirements specified in natural-language briefs and engineering contracts. The benchmark comprises 89 tasks and uses deterministic, manifest-based grading of database enforcement, HTTP business logic, and interface wiring; experiments evaluate five models through repeated builds, with an additional comparison between OpenHands and OpenCode. Claude Opus 4.8 achieves the highest reported pass rate of 62.9%.

Critical strengths
  1. The three-tier grading harness provides a useful setting for evaluating underlying data requirements. Combining database-enforcement probes, HTTP workload replay, and interface-wiring checks extends evaluation beyond schema similarity or user-visible workflows alone. The manifest-based protocol and deterministic grading are practical contributions to evaluating agent-built, database-backed applications.

  2. The experimental study provides broad model coverage and examines variation across builds and coding-agent harnesses. The additional OpenCode comparison and explicit investigation of grading artifacts strengthen the empirical study.

Critical weaknesses
  1. The validation of the business brief and engineering contract is insufficient to establish their completeness and consistency with the gold implementation. Although the paper introduces a control trio, adversarial review, and contract-derivability lint, it does not adequately explain how these mechanisms validate the agent-visible requirements. Appendix J reports both undisclosed input formats and a gold action that appears to contradict a stated business rule (p. 21). Consequently, some reported “over-strictness” may reflect incomplete or inconsistent instructions rather than a failure to implement the specified requirements.

  2. The real-world representativeness of the author-created tasks is insufficiently substantiated. The ethics statement specifies that every business setting, brief, contract, and gold implementation was authored by the authors, with workload values generated from seeds. However, the paper does not document their grounding in actual application requirements, production incidents, or independent domain-expert validation. Coverage across eight sectors and seventeen failure families therefore provides limited evidence that the selected rules and task difficulties represent practical database-backed application development.

Questions for authors
  1. Could the authors reconcile the replication counts and numerical results? Table 9 lists 385 Opus attempts rather than the stated 445, and its royalty_netting results differ from Appendix F. Please clarify the authoritative records and their use in the pass rate, bands, and pass^5.

  2. How does the OpenHands blocking-output issue affect the reported results? Appendix H reports substantial incidence and lost runtime, while the optional drain timeout remains disabled. A corrected-runtime comparison, together with actual step/token consumption and the missing Opus cost, would help assess its impact on self-testing and refinement.

  3. Could the authors provide the implementation and adjudication criteria for the contract-derivability lint and adversarial review? Please clarify which reported runs passed the lint and how the two-author verification distinguished model errors from specification ambiguity or grader artifact. Also, how can this manual check strategy populates to large volume evaluation?

  4. How would correcting the acknowledged probe limitations affect the scores and failure signatures? Appendix J identifies approximately 47 uniqueness losses where enforcement exists but is not recognized, and credits a rejection probe as enforced when its setup write is refused. Can the authors further distinguish successful enforcement from unexercised probes and report the effect of correcting these cases.

2 comments
AuthorsOpenReview ↗

Records reconciled; grader artifacts measured; scope of the tasks

Thank you for the detailed questions.

Q1. Replication counts.The submission captured an incomplete Opus run (385 of 445). Every model now has 5 attempts on all 89 tasks. Pass rate, bands (never, sometimes, always over c=0..5) and pass^5 are computed from these complete records, and royalty_netting reads 10/25 everywhere.

W1. Brief and contract consistency. The two defects you cite (undisclosed input formats, and a gold action that contradicts its brief) are fixed in the revision: (a) Seven brief-contradicting golds (including the cemetery-plot sale) and three clock-dependent golds were rewritten with each probe's tested property kept, and all attempts were graded under the fixed grader. (b) The month and timestamp forms are disclosed and retried, and a write the workload expects to be refused is never retried, so the retry cannot turn a required refusal into an acceptance.

To measure what residual grader behavior could contribute, we replayed all 2,225 stored applications under nine grader changes (general response, item 4). None changed a verdict. Enforcing exact category labels changes 18, of which 17 are wording the brief does not fix. The over-strictness finding therefore does not depend on these behaviors.

Q2. OpenHands background-output blocking. Appendix H now reports the census: (a) Share of attempts blocked: Opus 83%, Sol 87%, Kimi 25%, Qwen 94%, DeepSeek 98%. (b) Pass rates of blocked vs unblocked attempts differ by at most 9 points for four models (Opus 66% vs 75%). (c) Qwen's gap is wider (44% vs 69%, over only 26 unblocked attempts), and we flag it as a possible cost. (d) Blocked attempts sit idle after building (median 147 minutes for Opus), and only 3 attempts reached the wall while still working. The released harness exposes the drain timeout so the cost can be measured directly. Estimated cost is now reported for every model, including Opus ($3,187).

Q3. Lint and adversarial review. We state plainly that no contract-derivability lint gated this corpus (Section 5), and we recommend one. The adversarial review examined each failing pilot build against the probe it failed and classified it in one of three ways: (a) Model error: the brief or contract states the rule the application broke. (b) Specification gap: the contract does not fix the value or behavior the probe requires. (c) Grader artifact: an application correct by the contract still fails. Only the last two led to fixes, and each fix had to keep the task's control trio intact (oracle passes, no-op and near-miss fail) without rescuing any genuine failure. On scale: the review works per artifact class, not per attempt. After the pilot, the cost is one deterministic replay of the stored applications. We used exactly that replay for the robustness check above (2,225 applications in roughly ten minutes on ephemeral sandboxes).

Q4. Probe limitations. The uniqueness probe can miss keys enforced as triggers, partial or EXCLUDE constraints. From the released records we computed an upper bound on the effect: we pass every attempt whose only failing tier is Tier A and whose first failure is a uniqueness loss (77 attempts). No model moves more than 7 points and the ranking is unchanged (reported → upper bound): Opus 4.8 67.6% → 68.5%; GPT-5.6 Sol 55.5% → 62.5%; Kimi K3 53.3% → 57.3%; Qwen3.8-Max 45.6% → 47.2%; DeepSeek V4 Pro 33.5% → 37.3%.

The other limitation, a rejection probe whose setup write was already refused being credited as enforced, can only undercount over-permissive failures. The revision says so and reports false-accept rates as lower bounds. W2. Representativeness. The rules are standard data-integrity requirements (referential integrity, business-key uniqueness, immutability after sign-off, effective dating, derived versus stored values) stated in the plain business language of 89 different settings. We do not claim the tasks are drawn from production incidents or validated by independent domain experts, and the revision states this limitation explicitly.

Thank you for correcting the identified issues related to W1. My remaining concern is whether there is corpus-level evidence, preferably quantitative, that the quality-control process improves brief–contract–gold consistency beyond correcting individual observed failures. Since the previous version contained brief–contract–gold inconsistencies, what checks have been performed on the other instances to determine whether similar problems remain, including those not discussed in the paper? Could the authors provide before-and-after audit results or other quantitative evidence showing how the corrections improved benchmark quality?

Regarding W2, thank you for explicitly acknowledging the limitation of author-created tasks. Concrete comparisons with documented real-world business requirements would help clarify which characteristics are preserved, simplified, or omitted. Could the authors provide such comparisons and, where feasible, quantitative evidence of to characterize the gap between the benchmark synthesized tasks and practical application development?

These are my remaining problems, the others are solved. Thank the authors for providing detailed explanation.

Reviewer_HPMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

NL2FullStack evaluates whether coding agents can build database-backed applications that faithfully implement business rules stated in natural-language briefs. Across 89 tasks, its deterministic grader checks database-enforced invariants with direct SQL, replays seeded workflows against a hidden reference over HTTP, and verifies that the interface displays backend data. The paper evaluates five models, reports a leading task-averaged pass rate of 62.9% for Claude Opus 4.8, and identifies rejection of contract-valid inputs as a frequent failure.

Critical strengths
  1. The motivation and problem definition are valuable. Sections 1 and 2 explain why successful user workflows need not imply database-enforced integrity, while schema similarity can penalize a correct alternative design. This is a meaningful target for evaluating full-stack coding agents.
  2. Section 3 proposes a concrete three-tier protocol that tests database constraints, HTTP business logic, and interface wiring separately. Its randomized workloads and control-trio validation are promising ways to assess complete applications without requiring a reference schema.
Critical weaknesses

The paper's central weakness is that its reported experimental data are internally inconsistent in ways that affect its main conclusions. The authors must explain these discrepancies and provide a reconciled account of the results.

  1. The repeated-run results do not match the per-task ledger. Sections 3.1 and 4.1 state that every model makes five attempts on each of 89 tasks, but Table 9 records only 385 Opus attempts rather than 445. Table 4 nevertheless reports 34% for pass^5, defined as passing all five attempts, although Table 9 shows only 11 of 89 tasks with an observed 5/5 result, and the other tasks counted as always solved include 3/3 and 4/4 results. The missing attempts and inconsistent definition directly affect the claimed reliability of the leading model and comparisons that depend on five retries.
  2. The benchmark-difficulty and worked-example results also conflict. Figure 2c and Table 6 give different pass rates for all 17 failure families: for example, Derived, not stored is 20% in Figure 2c but 39% in Table 6. This leaves the category-level difficulty and diagnostic claims without a consistent numerical basis. Section 3.3 and Appendix F report 3 passes in 25 attempts and 22 failures for royalty netting, whereas Table 9 reports 8 passes in 24 attempts and 16 failures. The paper's explanation that 11 of 22 failures stopped at an input-format restriction cannot be reconciled with its per-task results. These are consequential conflicts in the evidence for the paper's conclusions, not merely presentation issues.
Questions for authors
  1. Why does Table 9 contain only 385 Opus attempts? Please account for the missing 60 and give the corrected definitions and values of the headline pass rate and pass^5.
  2. Which results are authoritative for the 17 failure families and royalty netting? Please reconcile Figure 2c with Table 6 and Section 3.3 and Appendix F with Table 9 using the frozen attempt records.
1 comment
AuthorsOpenReview ↗

Reconciled records

Thank you for checking the ledger against the aggregates. Both inconsistencies were real. The submission was made while the Opus 4.8 run was incomplete, and our tables were computed at different points.

Q1. 385 Opus attempts; pass^5. The per-task table held only the Opus attempts finished at submission time (3 to 5 per task, 385 in total). The remaining 60 have since been run under the same configuration, so every model now has exactly 5 attempts on all 89 tasks (445 each). With complete records the definitions are unambiguous: (a) pass rate: mean over tasks of passes/5 (b) pass@5: tasks with at least one pass (c) pass^5: tasks with all five passing For Opus 4.8 these are 67.6% [60.2, 74.8], 76/89 and 35% (31 tasks at 5/5), the last of which can be counted directly from the per-task table (Appendix I). The submitted pass^5 was computed as "all observed attempts pass" over tasks with 3 to 5 attempts, which, as you note, is not pass^5. That figure no longer appears.

Q2. Failure families and royalty_netting. All family-level numbers now come from the frozen per-attempt records through one script: Figure 2c, the family table in Appendix E and the text of Section 3.3 report the same pooled pass rate for each of the 17 families (for example, Derived, not stored: 48% in both). For royalty_netting, Section 3.3, Appendix F and the per-task table all report 10 passes in 25 attempts. Of the 15 losing attempts, 7 refused a valid lease-term write (5 refused the second term record that changes a term, 2 refused the first) and 8 returned a wrong figure on a read. The submitted "11 of 22 failures stopped at an input format" came from an earlier grading pass in which month and timestamp arguments were sent in forms the contract never stated. That dialect is now handled by a disclosed retry, which fires on 49 of the 2,225 attempts (Appendix J), so it no longer drives the worked example. We agree these were consequential, not presentational. The revision regenerates every figure and table from the released per-attempt records, so a reader can recompute each number.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers. All three identified the same central problem, internally inconsistent experimental records, and they were right. At the time of submission, evaluations were still running, so its per-task ledger held 385 of the 445 Opus attempts while several aggregates and prose passages had been computed at different points. We have uploaded a revision in which every number, table and figure is regenerated by one script from one frozen set of per-attempt records. What changed: (1) Complete protocol. Every model now has exactly 5 attempts on each of the 89 tasks (445 per model, 2,225 in total), each with a 4-hour wall. (2) Corrected headline. Opus 4.8 is 67.6% [60.2, 74.8], then Sol 55.5%, Kimi 53.3%, Qwen 45.6% and DeepSeek 33.5%. pass^5 is now well-defined (all five attempts pass), and so is pass@5 (at least one of five). (3) Single grader account. Section 5 and Appendix L now describe one frozen grader. Every gold workload that contradicted its own brief (7 tasks, including the cemetery-plot sale) or depended on the grading clock (3 tasks) was rewritten with each probe's tested property kept, and all attempts were graded once under the frozen grader. The undisclosed month and timestamp forms are now handled by a disclosed retry (Appendix J). (4) Grader robustness. After the evaluation we replayed all 2,225 stored applications in fresh sandboxes under nine further grader changes: refusals must carry the protocol's 409 + error envelope, reads must carry ok:true, missing fields no longer equal nulls, money is matched by complete assignment, the four ordered reads are compared in order, timestamps are compared to the second, interface rows are matched by maximum bipartite matching, and database connection or reset failures void a verdict instead of scoring. None of these changed a verdict. Enforcing exact category labels would change 18 of 2,225 verdicts: 17 use wording the brief never fixes ("canceled" for "cancelled") and 1 is a genuine mislabel. The model ranking is unchanged under every variant. Point-by-point replies follow under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.