- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NL2FullStack introduces 89 tasks for building database-backed applications from business briefs and engineering contracts. Its grader combines direct database tests, HTTP workloads checked against a hidden reference implementation and interface checks. The paper compares five models and analyzes why their applications fail.
Critical strengths
The direct SQL tests check whether the database enforces constraints when writes bypass application code. This is a useful addition to the interface, API and database snapshots examined by FullStack-Bench [1]. The manifest lets applications use different schemas. Tests with correct, no-op and near-miss implementations check whether the grader catches the intended defects. This targets applications that appear to work while allowing invalid data.
[1] FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translation. https://arxiv.org/abs/2602.03798v1
Critical weaknesses
- The paper gives conflicting accounts of the grader used for the reported results. Section 5 says grading problems were corrected before evaluation. Appendix J says month and timestamp formats were undisclosed and the workloads remain as shipped. It also describes the reference implementation selling a cemetery plot when the brief forbids it, and about 47 uniqueness failures where the database enforces the key but the probe misses it. These may concern an earlier version, but the paper does not say so. If they remain in the final results, the failure analysis mixes model errors with disagreements between the tests and the specification. This particularly affects the conclusion that models mostly fail by rejecting valid inputs.
- The reliability results do not follow the stated five-attempt protocol. Table 9 contains 385 Opus builds, with three to five attempts per task. These counts correctly reproduce the 62.9 percent mean, and the 30 always-solved tasks fit Table 3's definition of success on all observed attempts. However, Table 4 labels 34 percent as pass^5, although only 11 tasks show five successes in five attempts. Counting tasks solved at least once across varying numbers of attempts also does not directly establish pass@5. These metrics need an explanation of how tasks with fewer attempts were handled.
Questions for authors
See weakness
One frozen grader; complete five-attempt protocol
Thank you.
W1. Conflicting grader accounts. The appendix passages you quote described an earlier grading pass and should not have sat beside Section 5's account. The revision has one account, in Section 5 and Appendix L, and every reported attempt was graded once under that frozen grader:
(a) Month and timestamp forms: disclosed and handled by a retry that never retries a write meant to be refused (it fires on 49 of 2,225 attempts). (b) Gold that contradicts its brief: the cemetery-plot sale and six others were rewritten to respect the brief, with each probe's tested property kept. (c) Clock-dependent gold: three tasks fixed so the outcome no longer depends on the grading date.
On the remaining uniqueness-probe blind spot, an upper bound from the released records credits every attempt whose only failure is a Tier-A uniqueness loss. No model moves more than 7 points and the ranking is unchanged (Opus 67.6 to 68.5, Sol 55.5 to 62.5, Kimi 53.3 to 57.3, Qwen 45.6 to 47.2, DeepSeek 33.5 to 37.3).
We also replayed all 2,225 applications under nine further grader changes (409 + error envelope on refusals, ok:true on reads, missing fields not equal to null, complete money assignment, ordered reads, timestamps to the second, bipartite interface matching, database failures voiding a verdict). None changed a verdict.
The over-strictness conclusion is unaffected. Refusing a valid write is still the most common first failure for four of five models (32 to 40% of their losing attempts), and over-strict Tier-B failures outnumber over-permissive ones by 15:1 to 106:1 per model.
W2. Five-attempt protocol. The submission held only 385 of Opus's 445 attempts because the run was still being completed. Every model now has exactly 5 attempts on all 89 tasks. pass^5 is the fraction of tasks with 5/5 (Opus: 31 tasks, 35%) and pass@5 the fraction with at least one pass (Opus: 76/89), so no task has fewer attempts and neither metric needs special handling.