- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether increasing programming-specification complexity produces a locatable breakpoint in LLM code-generation correctness. It constructs 100 problem families, each containing a base task and five cumulative requirement additions, yielding 600 prompts. Function signatures and test inputs remain fixed within a family, while expected outputs change with the requirements. Four LLM judges score the prompts using a frozen six-dimensional complexity rubric. Sixteen models generate five responses per prompt, producing 48,000 draws.
The primary analysis fits family-adjusted linear and segmented relationships between judge-rated complexity and graded test correctness, using a cluster wild bootstrap, multiplicity correction, and additional localization and direction criteria. No model satisfies the full breakpoint criterion; one rejects linearity without a sufficiently localized change. All models have lower average correctness at the highest rung. A separately labeled post-hoc analysis shows that output truncation accounts for much of the observed decline for six models, while explicitly acknowledging that correctness among completed programs is subject to selection. The contributions are the controlled prompt families, the registered analysis pipeline, and the resulting empirical characterization.
Critical strengths
- The dataset lets researchers add requirements step by step and compare performance on the same base problem.
- The analysis separates answers cut short by token limits from incorrect programs, helping explain why performance declines.
Critical weaknesses
- The method misses most sudden drops in performance, detecting only about 11 out of 100 real drops even in its best simulated case (Table 4). Its finding of no sudden drop therefore gives little evidence about whether models actually break down as tasks become more complex, leaving the paper's main question unanswered. Please improve the experimental design until it reliably finds known drops in simulations, then repeat the model evaluation.
- The paper relies on complexity ratings that need stronger checks, since adding a requirement lowers the score in 77 of the 100 problem families (Appendix E). If these ratings put problems in the wrong order, a sudden performance drop could be hidden, so the paper cannot tell how much its result depends on the ratings. Please have independent judges check the ratings and show whether the results change when problems are ordered by the number of added requirements.
Questions for authors
- How often does the revised design find sudden drops deliberately built into simulated data?
- How do the results change when using independently checked complexity ratings or the number of added requirements?
2 comments
Thanks for your response, I have increased my overall score from 2 to 3.
Thank you for the review. Both weaknesses are fair, and they became the main jobs of the revision.
"The method misses most sudden drops ... Please improve the experimental design until it reliably finds known drops in simulations, then repeat the model evaluation."
We agree that the calibration is weak for moderate breaks, and the paper says so (Appendix G, Table 5, which was Table 4). We did not redesign the test and run it again. A revised procedure could be explored on these data, but a confirmatory test of it needs fresh draws, and more support at the top needs a new corpus. That is a new study. What a revision can do is say what this design and these data can and cannot see (Section 4, Appendix K, Table 15, Figure 3).
So the honest reading is narrower than "no break". At that middle split a pure step larger than 18.4 points is ruled out for every model. Near the top of the ladder the design locates a break poorly for several models. Appendix K now ends with what the next design must fix before it is run: longer ladders so the top has support, an output limit that does not end the answers, rung as a registered axis with disputed prompts sent to several raters, and a power target met in simulation first.
"The paper relies on complexity ratings that need stronger checks ... Please have independent judges check the ratings and show whether the results change when problems are ordered by the number of added requirements."
Where a break would sit along C depends on the size of each step, where agreement is weakest, and on levels across families, where only one outside rater agrees with the judges.