acceptodds
Under review as a conference paper at ICLR 2027

Complexity Ladder: A Controlled Test of Specification Complexity in Code Generation

Abstract

A previous study, ComplexityKink, found on a scraped corpus of about 5,000 prompts that the structural complexity of a programming prompt predicts whether a language model solves it, and that the relationship has a breakpoint. An observational corpus cannot say whether complexity itself did that, because prompts that are complex in the wild differ in many other ways at once. This study controls the prompt instead of sampling it. We take one hundred base problems, ninety-eight of them contest problems taken as written, and produce five further versions of each by adding one requirement at a time, holding the base task, the function signature and the test harness fixed: 600 prompts, each rated on the same frozen rubric by four judges. Sixteen models generate every prompt five times, 48,000 draws in all. For each model separately, a pre-registered and calibrated procedure tests whether family-adjusted correctness departs from a single straight line along the rated complexity composite and, where it does, locates the change and reports its direction and size. No model meets the registered criterion for a breakdown: one, GPT-5.6 Sol, rejects the straight line but its departure cannot be located on the grid, and the other fifteen do not reject at this design's precision. Every model is nonetheless lower at rung 5 than at rung 0, by 2.5 to 40.8 points of graded correctness. A descriptive split, chosen after the result, shows that for six models most of that decline arrives as answers cut off at the output limit, which score zero, while correctness conditional on a program running changes little, and that for the others the programs that run get worse. These results do not establish that breakpoints are absent, and they do not explain the earlier observational finding; they show what a controlled ladder, read with a calibrated procedure, can and cannot see.

Primary area:
datasets and benchmarks
Keywords:
code generation, specification complexity, large language models, benchmark design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_n2u2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether increasing programming-specification complexity produces a locatable breakpoint in LLM code-generation correctness. It constructs 100 problem families, each containing a base task and five cumulative requirement additions, yielding 600 prompts. Function signatures and test inputs remain fixed within a family, while expected outputs change with the requirements. Four LLM judges score the prompts using a frozen six-dimensional complexity rubric. Sixteen models generate five responses per prompt, producing 48,000 draws.

The primary analysis fits family-adjusted linear and segmented relationships between judge-rated complexity and graded test correctness, using a cluster wild bootstrap, multiplicity correction, and additional localization and direction criteria. No model satisfies the full breakpoint criterion; one rejects linearity without a sufficiently localized change. All models have lower average correctness at the highest rung. A separately labeled post-hoc analysis shows that output truncation accounts for much of the observed decline for six models, while explicitly acknowledging that correctness among completed programs is subject to selection. The contributions are the controlled prompt families, the registered analysis pipeline, and the resulting empirical characterization.

Critical strengths
  • The dataset lets researchers add requirements step by step and compare performance on the same base problem.
  • The analysis separates answers cut short by token limits from incorrect programs, helping explain why performance declines.
Critical weaknesses
  • The method misses most sudden drops in performance, detecting only about 11 out of 100 real drops even in its best simulated case (Table 4). Its finding of no sudden drop therefore gives little evidence about whether models actually break down as tasks become more complex, leaving the paper's main question unanswered. Please improve the experimental design until it reliably finds known drops in simulations, then repeat the model evaluation.
  • The paper relies on complexity ratings that need stronger checks, since adding a requirement lowers the score in 77 of the 100 problem families (Appendix E). If these ratings put problems in the wrong order, a sudden performance drop could be hidden, so the paper cannot tell how much its result depends on the ratings. Please have independent judges check the ratings and show whether the results change when problems are ordered by the number of added requirements.
Questions for authors
  1. How often does the revised design find sudden drops deliberately built into simulated data?
  2. How do the results change when using independently checked complexity ratings or the number of added requirements?
2 comments

Thank you for the review. Both weaknesses are fair, and they became the main jobs of the revision.

"The method misses most sudden drops ... Please improve the experimental design until it reliably finds known drops in simulations, then repeat the model evaluation."

We agree that the calibration is weak for moderate breaks, and the paper says so (Appendix G, Table 5, which was Table 4). We did not redesign the test and run it again. A revised procedure could be explored on these data, but a confirmatory test of it needs fresh draws, and more support at the top needs a new corpus. That is a new study. What a revision can do is say what this design and these data can and cannot see (Section 4, Appendix K, Table 15, Figure 3).

  • What the data rule out. Under a straight line with a pure downward step, the registered bands (nominal 97.5 percent) bound the step at each supported split, for all sixteen models at once. At the middle split (C from 13.5 to 13.75), a step larger than 2.6 to 18.4 points, depending on the model, is ruled out. Anywhere on the grid the limits run from about 15 to 55 points, weakest at the top.
  • How often planted drops are found (your question 1). We planted steps into these data and ran the full registered decision at the calibration's budget. The rates are conditional on these data. With each model's rejection taken at α/16, the decision is estimated to return H1 for a step planted at the middle split 80 percent of the time at about 8 to 27 points under one generator, and about 12 to 26 under a second that zeroes draws above the split, where the two Mistral models did not reach 80 percent at the largest size tried. Under the additive null, H1 fires in none of 1,000 datasets. At the top of the grid the design is much weaker. Under the first generator, H1 stays under 80 percent for eight models at every size we tried, up to 30 points. Under the second, the straight line is rejected but the change mostly cannot be located there.

So the honest reading is narrower than "no break". At that middle split a pure step larger than 18.4 points is ruled out for every model. Near the top of the ladder the design locates a break poorly for several models. Appendix K now ends with what the next design must fix before it is run: longer ladders so the top has support, an output limit that does not end the answers, rung as a registered axis with disputed prompts sent to several raters, and a power target met in simulation first.

"The paper relies on complexity ratings that need stronger checks ... Please have independent judges check the ratings and show whether the results change when problems are ordered by the number of added requirements."

  • Ordered by rung (your question 2). We refit the registered procedure with rung in place of C (Appendix I, Table 8). No model meets H1. Sol no longer rejects (p = 0.28). Mistral Large 3 and Gemma 4 31B do, Gemma within Monte Carlo error of its threshold, and neither gets a direction. With six rungs, only three gaps can be estimated, and the first added requirement's is not one of them.
  • Each judge alone. No judge's own composite gives a located downward break for any model. These grids were not calibrated.
  • Each dimension alone. GPT-5.6 Terra on branching and MiniMax M3 on data structures meet every H1 condition, but on grids of five and four candidates, where their location sets keep every one and nothing is calibrated, so we do not call them breaks.
  • Human ratings (Appendix E, Table 4). Four people rated prompts on the frozen rubric, with no scores, tests, family or rung labels shown. Two are authors. One rated 204 prompts, 34 whole families, and agrees with the judge mean at ICC(2,1) 0.40, near the low end of the judges' own 0.36 to 0.73. The other, on 48 of them, agrees less (0.23). Two are not authors, both work in software, and they used a page and scoring rule fixed before any rating. One rated 96 prompts: ICC 0.41 and Spearman 0.77, 4.4 points higher. On the same 16 families, that rater and the first author agree with the judges at Spearman 0.73 and 0.76 after removing each family's mean, 0.04 and 0.25 on the size of each step, and 0.89 and 0.46 across families (0.27 on the first author's 34). The other rated 48 and agrees little: ICC 0.10 and Spearman 0.46, 5.1 points lower, and 0.17, -0.15 and 0.32 on those three measures. The highest ICC between any two of the four is 0.18. These ratings do not validate the index, and we did not refit the analysis on them.
  • The 77 families. Of the 500 rung steps, 354 go up, 35 stay flat and 111 go down, with a mean drop of 0.72 points against a mean step of 0.84 (Appendix E). The analysis uses C as measured, and each family's intercept absorbs its level.

Where a break would sit along C depends on the size of each step, where agreement is weakest, and on levels across families, where only one outside rater agrees with the judges.

Thanks for your response, I have increased my overall score from 2 to 3.

Reviewer_V5MKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper attempts to study that, in coding tasks, whether or not there is a breakpoint for a catastrophic performance collapse with the increase of specification complexity. Different from sampling from real-world tasks of various level of difficulty, the authors select 100 basic tasks and gradually add 5 additional requirements to each of the task, resulting in 600 tasks across 6 complexity levels. 4 LLM judges are deployed for scoring with a fixed complexity rubric composed of 6 dimensions, which in total evaluate 16 different LLMs with 5 attempts per task per model, accumulating 48,000 generations. From the results and a preregistered breakpoint detection criterion, it turns out that the breakpoint is non-existent in the settings of the paper, and all models perform better at basic-level tasks compared with the highest complexity rung. The authors also find out that the performance degradation is partly due to response cutoff when the output token limit is hit.

Critical strengths

(1) The scale of the experiment. The authors run extensive experiments with 16 LLMs on 600 tasks (augmented from 100 base tasks), showing decent amount of work and efforts.

(2) Interesting topic and honest framing. Whether or not there is a breakpoint for catastrophic performance degradation at a certain complexity level is an interesting and meaningful area of study. The authors are honest when reporting the results and point out most of the limitations.

Critical weaknesses

(1) Novelty of results. Though sound and honest, all the results seem natural and intuitive to me, and I struggle to find out what's the takeaway or insights from the paper. The only innovative part, i.e. the introduction of a method to detect the breakpoint, is a negative one where the breakpoint isn't detected. Moreover, for the breakpoint detection criterion itself, the statistical power is low as it introduces too many false negatives as acknowledged by the authors in Page 4 and Page 5. As a result, I don't find the paper merits acceptance at ICLR level.

(2) Complexity ladder classification remains questionable. As shown by the authors, the complexity is not monotonic for different ladders and the correlation of complexity grading by different models is as low as 0.51 for Spearman. It raises concern about robust and convincing classification of different complexity ladders. Also, the complexity is also dependent on the base tasks, and it doesn't make a lot of sense to me when all base tasks are automatically deemed to be of the same difficulty.

(3) Soundness issue on result reporting. Cutoff due to token limit seems to be an error introduced during API call and it's fully artificial from the authors. It's unfair to let the models take the blame for not completing a task when the authors set a certain token limit and the response of the model gets cut off because of that.

Questions for authors

I don't have any particular questions for the authors, but it would be helpful if the authors could provide a rebuttal targeting the weaknesses identified by me, especially regarding the novelty of the results (Weakness 1). Also it would be helpful to address Weakness 2 and 3.

2 comments

Thank you.

Weakness 1, novelty. We think three results are new.

First, a controlled test of an observational claim. ComplexityKink reported breakpoints along this same composite on a scraped corpus, where complexity travels with domain, author and base problem. Within controlled families, where the base statement, signature and test inputs are held fixed, the same instrument finds no locatable break in any of 16 models. The studies also differ in panel and protocol, so this is a contrast, not an explanation. The revision says what that null is worth: at the middle split (C from 13.5 to 13.75), the registered bands (nominal 97.5 percent) rule out a pure downward step on a straight line larger than 2.6 to 18.4 points, depending on the model. In a sensitivity analysis on these data, with rejection taken at α/16 and 3,839 replicates, the decision is estimated to return H1 for a step planted at that split 80 percent of the time at about 8 to 27 points. That is not the power of a new study (Section 4, Appendix K, Figure 3). Near the top of the ladder it locates a break poorly for several models.

Second, what the decline at the top rung is made of. For six models it is mostly answers cut off at the output limit, and the programs that do arrive, a selected subset, score within about 7 points of rung 0. A later exploratory sample requested twice the limit. In it GPT-5.6 Terra, GPT-5.6 Luna and Qwen3.8 Max score 4.2, 1.5 and 0.2 points below their rung 0, and Kimi K2.7 Code, DeepSeek V4-Pro and Qwen3.8 27B close 44 to 76 percent of the gap, but only 13 to 65 percent on all-pass (Tables 1 and 26). A fixed output limit can make a rising cost of answering look like lost ability.

Third, what the wrong programs get wrong (Appendix L). Of the 8,078 wrong programs whose rerun reproduced, 7 percent match the specification with requirements dropped on every test they answered. In 50 sampled near misses, read by language-model agents, the most common code is an error in combining requirements.

On power, the calibration is weak for moderate breaks, and we report the whole table (Appendix G, Table 5).

Weakness 2, the ladder.

  • C is measured, not assumed monotone. Of the 500 rung steps, 354 go up, 35 stay flat and 111 go down, with a mean drop of 0.72 points against a mean step of 0.84. Mean correctness on the shipped suites falls from rung 0 to rung 5 within families for every model, with one interval that includes zero (Appendix I, Table 7).
  • Base tasks are not treated as equally difficult. Each family has its own intercept, which absorbs the level of its base problem, so the slope rests on differences within families. Where a split falls along C still depends on each family's rated level, where only one of four raters agrees well with the judges. The base problems were chosen to spread difficulty across bands.
  • No single judge and no ordering by rung gives a break either. On each judge's own composite, and ordered by rung, no model meets H1 (Appendix I). The rung grid has only three splits, and neither kind of grid was calibrated. Two of the authors and two people who are not authors also rated 204, 48, 96 and 48 prompts with scores and family and rung labels hidden (Appendix E, Table 4). In that order, their ICCs against the judge mean are 0.40, 0.23, 0.41 and 0.10, where the judges' own run from 0.36 to 0.73 on the largest sample. Their Spearman correlations with the judges are 0.76, 0.44, 0.73 and 0.17 after removing each family's mean, and 0.23, -0.01, 0.04 and -0.15 on the size of each step. The outside raters score 4.4 points higher and 5.1 lower than the judges. The highest ICC between any two of the four is 0.18, so these checks do not validate the index. The between-judge correlation, 0.51, is modest.

Weakness 3, the output limit. We agree that the limit is ours, and the revision removes the wording that put the blame on the model. The score measures the deployed model under a stated limit. The limit was set high, 65,536 tokens for most models, to avoid manufacturing an effect, and it is the same at every rung. Tables 1, 20 and 22 report the share cut off at it separately from wrong programs. Because the limit is a choice, we collected a later sample requesting 131,072 tokens. In it the share cut off at rung 5 falls from 21 to 43 percent to 0 to 5 percent, with the scores given above (Table 1, Appendix J). The rerun is exploratory and a later sample of rung 5 only, so it cannot separate the limit from changes between the runs, and the registered result stands as it was. Two exploratory scenarios ran the registered procedure with every cut-off draw scored as full marks, and with those draws left out and each emptied cell filled from the same family's nearest rung. Sol no longer rejects under either, and no model meets H1 (Appendix I). As long as the limit binds, a decline with complexity is partly a statement about the limit (Section 5).

Thank you for your response. The rebuttal makes sense, but it is still hard to identify enough contributions that merit an ICLR publication. The paper mainly reports a negative result on its own protocol and as acknowledged by the authors, the output limit issue makes the evaluation even more ambiguous and noisy. I am not convinced that the community would have meaningful takeaways from this manuscript, and thus I retain my score.

Reviewer_WDPoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether code generation performance exhibits a clear breakpoint as specification complexity increases. The authors construct controlled complexity ladders and evaluate 16 language models using a pre-registered statistical analysis. The results show that although performance generally declines with increasing complexity, no reliable breakpoint is identified. The study also highlights different patterns of performance degradation, particularly output truncation and incorrect generated programs.

Critical strengths
  1. The paper revisits the performance breakpoints reported in previous observational studies and investigates whether similar patterns can be observed under more controlled conditions. This is an meaningful research question.
  2. Complexity Ladder provides a systematic way to study performance changes by progressively adding requirements to the same programming problem while keeping the base task and test inputs fixed. This design helps reduce variations between different problems.
  3. The experiments cover 16 language models and 48,000 code generations, providing a broad comparison of model behaviors. The paper also reports negative findings rather than forcing a positive conclusion, which adds value to the study.
Critical weaknesses
  1. The paper uses 98 programming problems from LiveCodeBench, covering contest dates from June 2023 to March 2025. However, the potential impact of data contamination is not sufficiently discussed. Since the evaluated models have different training data and knowledge cutoff dates, some models may have seen the original problems during training, while others may not. Although the paper introduces additional requirements to construct different complexity levels, familiarity with the original problems may still affect how models handle these requirements. More importantly, such differences may influence not only the absolute correctness but also the observed performance trends as complexity increases. I suggest that the authors clarify the dataset version and provide additional analyses based on problem release dates and model training cutoff dates, where available, to examine whether the main findings remain consistent on problems that are less likely to have appeared in training data.
  2. The complexity measure relies entirely on LLM judgments, but it is unclear whether these scores reflect the actual difficulty of implementing the requirements. Adding requirements does not necessarily increase implementation complexity, and different requirements may introduce very different levels of difficulty. I suggest validating the scores against human judgments and changes in reference implementations, as well as examining the sensitivity of the results to different complexity dimensions and LLM judges.
  3. The evaluation uses only 10–20 small test cases per problem, which may not sufficiently assess the correctness of generated programs. Moreover, several advanced models achieve nearly perfect correctness even at the highest complexity level, suggesting that the tasks may not be challenging enough to reveal their capability limits.
  4. The current analysis mainly separates output truncation from correctness degradation among executed programs, but provides limited insight into why models produce incorrect code as complexity increases. For example, are the errors caused by missing requirements, incorrect algorithms, or difficulties in combining multiple requirements? I suggest conducting a finer-grained error analysis and examining how different error types relate to the six complexity dimensions. This would help explain which aspects of specification complexity contribute to the observed performance degradation.
Questions for authors
  1. Could the authors clarify whether potential data contamination affects the observed performance trends? How well do the LLM-based complexity scores align with human judgments of implementation difficulty, and are the conclusions robust to different complexity measures?

  2. Would the findings remain consistent with more comprehensive test suites and harder problems, especially for models showing near-ceiling performance?

  3. are the errors caused by missing requirements, incorrect algorithms, or difficulties in combining multiple requirements or other reasons? And, how does this happen? Could the authors provide a finer-grained error analysis to explain how different complexity dimensions relate to specific code generation failures?

1 comment

Thank you for the review.

W1, contamination and the dataset version. The version is now stated (Appendix D): release v6 of the LiveCodeBench code-generation dataset at commit 0fe84c3912ea. The registered split at the median contest date is now reported (Section 4, Appendix I, Tables 6 and 7), between 2024-09-07 and 2024-09-14 with 49 families on each side. Neither half gives a located break. For twelve models rung-0 correctness differs by less than 3 points between the halves, and every model's slope is lower on the older half. Mistral Large 3 is the main exception at rung 0: it starts 18 points higher on the older half. The date-by-rung interaction is positive for all sixteen models, and its interval excludes zero for seven. So the decline is steeper on the older problems, not flatter. But the judged complexity also rises more on the older half, 4.6 points against 3.9, and the split cannot separate exposure from drift in contest style, so this is not evidence about exposure. We have not run the comparison against each model's training cutoff, because we did not assemble sourced cutoffs.

W2, the complexity measure.

  • Human judgments (Appendix E, Table 4). Two of the authors and two people who are not authors rated prompts on the frozen rubric, with no scores, tests, family or rung labels shown, on 204, 48, 96 and 48 prompts. In that order, ICC(2,1) against the judge mean is 0.40, 0.23, 0.41 and 0.10, where each judge agrees with the other three at 0.36 to 0.73 on the largest sample. After removing each family's mean, Spearman is 0.76, 0.44, 0.73 and 0.17, and on how much each requirement adds it is 0.23, -0.01, 0.04 and -0.15. The outside raters, under a rule fixed before any rating, agree at Spearman 0.77 and 0.46 overall and score 4.4 points higher and 5.1 lower. This checks agreement on the structural rubric, not how hard a requirement is to implement.
  • Dimensions and judges (Appendix I, Table 8). On single dimensions (graded correctness), GPT-5.6 Terra (branching) and MiniMax M3 (data structures) meet every H1 condition, but their location sets keep every candidate, five and four, so we do not call them breaks. On each judge's own composite, no model gives a located downward break. Both H1 results are a steeper slope, and the other data-structures rejections get no label (Table 29).
  • Reference implementations. We have not analysed how much each reference changes per requirement.

W3, tests and the ceiling. The registered held-out suites and all-pass are now reported (Table 6). The held-out suites cover 98 families, twelve inputs in almost all, disjoint from the shipped cases. The H1 decision is the same on every endpoint. Sol is the only model whose test rejects, its departure cannot be located, and no model meets H1. The size of the decline is not the same. On the held-out suites Llama-3.3-70B falls 1.0 point against 10.4, and Mistral Medium 3.5 is 5.5 points higher at rung 5 than at rung 0. The two suites rarely disagree about a passing program: of the 33,400 draws that ran on both and pass every shipped test, 199 fail a held-out test. The held-out suites come from the same generators, so they are alternate suites rather than independent validation (Appendix D). Correctness here means correctness on these suites, not at contest-scale inputs. On the ceiling, three models stay near the top across the whole ladder, and harder base problems would need a new corpus. The ten models in the middle lose 13.5 to 40.8 points, so the ladder has room to fall for most of the panel.

W4, why the programs are wrong (Section 4, Appendix L, Table 16). We reran the 8,080 wrong programs at rungs 1 to 5 against the reference with each subset of requirements left out. 8,078 reproduced, and the shares are of those.

  • 7 percent match the specification with requirements dropped, on every test they answered.
  • 2.4 percent match different reduced specifications on different tests, a share rising from 0.9 percent at rung 1 to 4.6 at rung 5.
  • 85 percent are wrong in ways no reduced specification explains.

Language-model agents, working to our instructions, read 50 sampled near misses. The codes are 16 errors in combining requirements, 12 of one requirement coded wrong, 12 of the base task wrong and 10 misread requirements. All ten misreadings are in two families, one of them a second ambiguity found this way. On the dimensions, matches to the specification with a requirement removed are most frequent for requirements that raised branching: 15.0 per 1,000 executed draws with retained records that carry them, against 6.1 to 9.6 for the others. But the size of the judged rise barely predicts such a match (Spearman 0.06 to 0.14). Most wrong programs have no such match. Why they fail is still open; a dropped requirement combined with another error is not ruled out. A match does not show that a program dropped the requirement.

Reviewer_PJ7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Complexity Ladder, a controlled benchmark investigating whether code-generation performance exhibits identifiable breakdown points as specification complexity increases. It constructs 600 prompts from 100 base problems with five incremental requirements and evaluates 16 models over 48,000 generations. A pre-registered statistical procedure is used to detect model-specific breakpoints.

No model satisfies the registered breakpoint criterion, although all models exhibit lower correctness at the highest complexity level. The analysis further suggests that output truncation, rather than incorrect executable programs, accounts for much of the degradation in several models.

Critical strengths
  1. Controlled experimental design. The incremental specification ladders, fixed base problems, and extensive model evaluations provide a more systematic setting than observational prompt collections. Pre-registration and family-clustered statistical inference further strengthen methodological transparency.

  2. Informative failure analysis. The distinction between output truncation and incorrect executable programs reveals an important evaluation issue. The authors also transparently report the detection procedure's limitations.

Critical weaknesses
  1. Insufficient statistical power for the central research question. Table 4 (Appendix G) shows that the full breakpoint detection procedure identifies only 0–10.6% of simulated true breakpoints. Moreover, candidate splits are restricted to complexity scores between 10.25 and 16.50, excluding the upper complexity region where several models degrade most sharply. Consequently, detecting no localized breakpoint across 16 models provides limited evidence regarding whether such breakdowns exist.

  2. Incomplete evaluation of correctness and robustness. The primary metric uses only 10–20 small, selected test cases per problem, while the pre-registered held-out and all-pass evaluations are not reported. Output truncation also exceeds 30% at the highest rung for several models, making performance degradation strongly dependent on the generation budget. Additionally, Appendix H acknowledges 65 provider failures incorrectly scored as zero for GPT-5.6 Sol. These issues weaken the reliability and interpretation of the main empirical findings.

Questions for authors
  1. Given the extremely low detection power reported in Table 4, what meaningful breakpoint magnitudes can the current study exclude? Can the authors provide a better-powered analysis or extend the complexity range to cover the observed upper-end performance degradation?

  2. Can the authors report the pre-registered held-out and all-pass results, together with robustness analyses addressing output truncation and the incorrectly classified provider failures? Would the main conclusions remain unchanged under these alternative evaluations?

1 comment

Thank you for the review. The revision adds endpoint checks, sensitivity analyses and a later sample at twice the requested output limit, for six models at rung 5 and not Sol. It also says which parts of your requests remain undone.

W1 and question 1: power, and what the study can exclude. We agree that the calibration is weak (Appendix G, Table 5, which was Table 4). The revision adds what these data rule out and what the design would have found on them (Section 4, Appendix K, Table 15, Figure 3).

  • Excluded magnitudes. Under a straight line with a pure downward step, the registered bands give nominal 97.5 percent simultaneous bounds on the step at each supported split. At the middle split (C from 13.5 to 13.75), a step larger than 2.6 to 18.4 points, depending on the model, is ruled out. For a step at an unknown place on the grid, the limits run from about 15 to 55 points, weakest at the top.
  • Planted changes in these data. The registered decision, with each model's rejection at α/16, is estimated to return H1 for a step planted at the middle split 80 percent of the time at about 8 to 27 points under an additive generator, or 12 to 26 under one that zeroes draws above the split, where the two Mistral models did not reach 80 percent at the largest size tried. Under the additive null, H1 fires in none of 1,000 datasets.
  • The top of the range. The grid ends at 16.5 because no split above it meets the support policy registered before any outcome. 68 prompts exceed it, and above 18 there are 15 prompts in 10 families. At the top split, six models have fitted gaps of about 17 to 22 points down, but their simultaneous intervals include zero, so the data cannot tell. A large planted break between 17.5 and 17.75 is called by some model in most datasets, at the wrong place.
  • More power, a wider range. We did not change the registered analysis after seeing the data. Extending the range needs ladders with more rungs, which is a new corpus, and Section 5 says so.

W2 and question 2: endpoints, truncation and Sol.

  • Held-out and all-pass are reported (Appendix I, Table 6). On all-pass, and on the held-out suites scored both as graded and as all-pass, Sol is the only model whose test rejects (p from 0.0009 to 0.0013). Its location set holds 20 or 21 of the 25 candidates, and no model meets H1. The 34 held-out scores lost with 36 records are given as two scenarios, all 0 and all 1, and both give the same decision. The secondary refits that were run also have all-pass versions (Tables 11 to 14), and none supports H1 on the composite axis. Sol's all-pass estimates and bands are in Table 30. The truncation exclusion as registered is still missing.
  • Truncation. The registered rerun with truncated draws excluded cannot be run as written. Excluding them empties 544 family-by-rung cells, and the procedure has no rule for a cell with no measurement. What we ran instead is the exploratory rerun requesting 131,072 tokens, where two routes stopped at 128,000 (Tables 1 and 26, Appendix J). It measures rung 5 at a higher limit in a later sample, which is a different question from the exclusion. The share cut off at rung 5 falls from 21 to 43 percent to 0 to 5 percent. In graded correctness three models then score about 4.2, 1.5 and 0.2 points below their rung 0, and the other three close 44 to 76 percent of the gap. On all-pass the three are 3.8 to 4.6 points below, and the others close 13 to 65 percent. Two scenarios on the other side of the registered scoring can also be run. With every cut-off draw scored as full marks, no model meets H1, and Sol no longer rejects (p = 0.048). With those draws left out and each emptied cell given the value of the same family's nearest rung, a fill rule of our own, Sol's p is 0.38, gpt-oss-120b's is the only rejection and is not located, and no model meets H1. So Sol's rejection survives neither.
  • Sol's failed replies (Appendix H). The count is 66, not 65. Giving all 68 unparseable Sol replies full marks gives Sol p = 0.00028. Giving full marks to only the 66 provider failures gives 0.00020, and leaving those 66 out of their cells gives 0.00024. Each retains 18 of 25 candidates, against 20 and p = 0.00084 in the registered result. One GPT-5.6 Luna reply ended in the same failure event and was also scored zero. Treating it the same way changes no search or H1 decision. Sol still rejects and still fails to localize, and no other model's decision changes. Its direction does change: the downward label at its best split is lost, because one of the two gap intervals reaches zero (Table 29).

No endpoint or failure-treatment refit that we ran supports H1 on the composite axis. The registered truncation exclusion, a wider ladder and a redesigned evaluation remain undone. What changes is how the top of the ladder reads: for three models, most of the decline there is gone in the later sample at the higher requested limit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revision

Thank you to all four reviewers. The revised PDF is uploaded. The main text now ends on page 10, as ICLR allows in a revision. The primary result is unchanged. It comes from one run made before the reviews, which later runs only replay, and everything added after it is labelled registered or exploratory. What changed:

  1. Most of the registered secondary analyses (Section 4, Appendix I, Tables 6 to 14). The refits use the confirmatory decision code on the same admitted draws. On all-pass and on the held-out suites, Sol is again the only model whose test rejects, its departure again cannot be located, and no model meets H1. The same holds without the ambiguous family and without the four filter-blocked draws. Split at the median contest date, neither half gives a located break. Against the registered cubic, no model's best split is distinguished after Holm. On single dimensions two models meet every H1 condition on graded correctness, but their location sets keep every candidate, so we call neither a break. Those that were run are repeated on all-pass, with the same H1 decision on the composite axis. Appendix I lists what is still missing.

  2. Rung 5 at twice the requested output limit (Section 4, Table 1, Appendix J). This is exploratory. For the six models with the most answers cut off, we sent every rung-5 prompt five more times requesting a limit of 131,072 tokens instead of 65,536 (the two GPT-5.6 routes ended answers at 128,000). The share cut off falls from 21 to 43 percent to 0 to 5 percent. In graded correctness, GPT-5.6 Terra, GPT-5.6 Luna and Qwen3.8 Max then score 4.2, 1.5 and 0.2 points below their earlier rung 0, and the other three close 44 to 76 percent of the gap. On all-pass the first three are 3.8 to 4.6 points below, and the others close 13 to 65 percent. It is a later sample of rung 5 only, so it cannot separate the limit from changes between the runs.

  3. Cut-off draws at full marks, and left out (Section 4, Appendix I). Both are exploratory. The registered exclusion of cut-off draws cannot be run as written, because it empties 544 family-by-rung cells, but two scenarios can. With every cut-off draw scored as full marks, Sol no longer rejects (p = 0.048), gpt-oss-120b rejects without localizing, and no model meets H1. With those draws left out and each emptied cell filled from the same family's nearest rung, a fill of our own, Sol's p is 0.38 and the rest reads the same. Sol's was the only rejection in the confirmatory analysis, and it survives neither scenario.

  4. Human ratings (Appendix E, Table 4). Two of the authors and two people who are not authors rated 204, 48, 96 and 48 prompts on the frozen rubric, with no scores, tests, family or rung labels shown. In that order, ICC(2,1) against the judge mean is 0.40, 0.23, 0.41 and 0.10; each judge agrees with the other three at 0.36 to 0.73 on the largest sample. The outside raters, under a rule fixed before any rating, agree at Spearman 0.77 and 0.46, scoring 4.4 points higher and 5.1 lower. One author and one outside rater rank a family's rungs much as the judges do, the other two less, and none agrees well on step sizes. The highest ICC between a pair is 0.18.

  5. What the data rule out (Section 4, Appendix K, Table 15, Figure 3). Under a straight line with a pure downward step, the registered simultaneous bands (nominal 97.5 percent) rule out a step larger than 2.6 to 18.4 points, depending on the model, at the middle split (C from 13.5 to 13.75), and larger than about 15 to 55 points anywhere on the grid. Planting steps into these data, the decision, with each model's rejection taken at α/16, is estimated to return H1 for a step planted at that split 80 percent of the time at about 8 to 27 points. These are sensitivities on these data, not the power of a fresh study. At the top of the grid the design is much weaker.

  6. What the wrong programs got wrong (Section 4, Appendix L, Table 16). We reran the 8,080 wrong programs at rungs 1 to 5 against the reference with every subset of requirements removed, and had language-model agents read 50 near misses.

  7. Sol's failed replies (Section 4, Appendix H). Giving all 68 unparseable Sol replies full marks leaves the search and H1 decisions unchanged. So does giving full marks to only the provider failures, 66 of Sol's and one of Luna's, or leaving them out. Sol still rejects and still fails to localize, but loses its downward label at the best split (Table 29).

  8. Smaller additions. The LiveCodeBench version, Monte Carlo intervals for every search and profile p-value and band critical value, a second specification ambiguity, the 36 admitted draws whose run records were lost after the analysis, counts of draws without a completed test run by status, model, rung and complexity (Table 22), and the delivery table relabelled as final ledger attempt indices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.