acceptodds
Under review as a conference paper at ICLR 2027

LLMs Can Follow Instructions, But Not Many at Once: Multiplicative Collapse in Compositional Instruction Following

Abstract

Large language models follow a single explicit instruction reliably, but deployed prompts rarely contain just one. We ask what happens as instructions accumulate: how fast the chance of satisfying all of them declines, what sets the rate, and whether effort at inference time can slow it. We introduce Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that varies the number of simultaneous constraints from 1 to 12 and checks every constraint with a deterministic verifier, with no LLM judge at any stage. Across 19 models, 36 constraint types, and 472,019 checks, per-constraint pass rates decline gently while joint success collapses: at , models pass 45% of individual constraints but satisfy all eight on only 9.6% of prompts. No pair of constraints interferes: failures of different constraints are nearly uncorrelated, so the risk of failure compounds across constraints, and no choice or arrangement of constraints avoids the collapse. The per-constraint rates themselves follow a clear hierarchy: constraints that must be tracked across the whole output lose their single-constraint accuracy about twice as fast as lexical ones, and constraints settled by a single decision degrade least. When constraints cannot all be met, models keep the ones that hold up best under load. Planning before generation does not delay the collapse; self-correction and best-of-5 sampling delay it by one or two constraints. Only four of the 19 models satisfy every constraint on a majority of prompts at , and 12 lose that majority at three constraints or fewer. We will release all verifiers, probes, model outputs, and code for reproducibility.

Primary area:
datasets and benchmarks
Keywords:
instruction following, compositional generalization, language-model agents, evaluation, constraint satisfaction, deterministic verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T8ZoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether LLMs are able to produce responses that satisfy multiple constraints at the same time. The particular constraints are described only very generally in the paper but they appear to be things like: include a particular word, all words need to be unique, use markdown, include the word count, etc. The authors show that as the number of constraint increases, LLMs are unable to satisfy all of them at the same time, and increasingly make mistakes. For the stronger models, many of the failures are near misses (only one constraint was not satisfied).

Critical strengths

The dataset, while very narrow in scope (see below), is carefully constructed and the experiments are thorough. There are some interesting empirical tidbits (no interference between constraints; strong models can satisfy many constraints at the same time and only start to break down when more than 6 or 7 constraints need to be satisfied together).

Critical weaknesses
  1. Abuse of page limit: oddly enough there is no table in the paper that lists the constraints. They are described quite vaguely in the main paper and then there's a link to two tables, neither of which appear to be actually in the paper (this probably means they're in the appendix, which I am not able to review because I have a lot of papers in my batch). Likewise, there is no discussion of limitations in the paper, but there is a link to the appendix. In my view, limitations are an important part of the discussion section of a paper, especially in a paper like this one that proposes a dataset that seems quite similar though perhaps somewhat better than previous datasets.

  2. The particular operationalization of instruction following through formal constraints (counting words, writing acrostics, etc) is quite limited and has been included in a number of datasets already, so I wonder if there is substantial novelty or value in creating yet another dataset (albeit one that's harder than previous ones). I understand of course that more interesting or realistic constraints (in the sense that they might have more applications in the real world) pose verification challenges compared to easy-to-verify formal constraints, but perhaps our models are now strong enough that we can verify "loose" constraint satisfaction more effectively.

Questions for authors

Not a question but a minor grammatical comment - "interfere" cannot be used intransitively, so it should be "do constraints interference with each other", not "do constraints interfere".

1 comment
AuthorsOpenReview ↗

Response to Reviewer T8Zo

We thank the reviewer for their careful reading and constructive comments, which have helped us clarify several points and improve the paper.

"... oddly enough there is no table in the paper that lists the constraints. [...] Likewise, there is no discussion of limitations in the paper, but there is a link to the appendix."

We agree that both belong in the main paper, and have added them. Section §3.1 now includes a table listing all 36 constraint types, grouped by dimension, with a one-line description of what each verifier checks. The full specifications remain in Table 7. We have also added a limitations paragraph at the end of §6, covering the points previously discussed in Section A: CSE examines one task family and one way of combining constraints, where all k requirements must be satisfied simultaneously; it includes only constraints that can be checked by deterministic verifiers; and results at k = 9 to 12 are based on 134 to 253 probes each.

"The particular operationalization of instruction following through formal constraints (counting words, writing acrostics, etc) is quite limited [...] perhaps our models are now strong enough that we can verify 'loose' constraint satisfaction more effectively."

We agree that formal constraints capture only a limited part of instruction following. We chose them because we need to verify responses reliably as the number of constraints increases. Since the paper measures how performance changes with k, we need to ensure that errors in verification do not also increase with k. An LLM judge would have to track all k constraints to assess a response, which is precisely the ability we are testing. FollowBench, for example, uses an LLM judge and evaluates up to five constraints (§1).

Even with deterministic verifiers, we found that careful auditing was necessary. An LLM review of near-miss failures identified four parsing issues, including one that incorrectly marked 1,051 responses as failures for R3 (§3.3). Extending the study to looser constraints using an LLM judge would be valuable, and our audited verdicts could provide a reference for calibrating such a judge. We have added this possibility to the limitations.

Our new analyses also show how these constraints interact in settings closer to practical prompting. For example, combining a structured output requirement, such as a JSON object, ranked list, or table, with restrictions on sentences or words can lower the pass rate of the latter by up to 28 pp (our reply to Reviewer evAu, Part 1).

"Not a question but a minor grammatical comment - 'interfere' cannot be used intransitively, so it should be 'do constraints interference with each other', not 'do constraints interfere'."

Thank you for pointing this out. We have made the wording more explicit: "Do constraints interfere with each other?"

Reviewer_PYFFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LLMs perform when asked to satisfy multiple instructions simultaneously. The authors introduce Constraint Saturation Evaluation (CSE), a benchmark containing 36 types of constraints that can be checked with deterministic verifiers. They evaluate 19 models with up to 12 constraints per prompt and find that the probability of satisfying all constraints drops rapidly as more constraints are added. The paper also examines correlations between constraint failures, differences across constraint types, and whether additional inference-time computation can improve performance.

Critical strengths
  1. The experiments cover a relatively large number of models and constraint types. Using deterministic verifiers also makes the evaluation more reliable than relying on LLM judges.

  2. The analysis of different constraint types is interesting. In particular, the finding that constraints requiring continuous tracking throughout generation degrade faster than simpler lexical constraints may be useful for understanding instruction-following failures.

Critical weaknesses
  1. I am not convinced by the paper's conclusion that constraint failures are largely independent. The authors mainly support this claim using weak pairwise correlations, but this does not rule out more complex interactions between constraints. Also, the independence-based model predicts only 3.3% joint success at k=8, compared with the observed 9.6%. I think the evidence is insufficient to conclude that constraint interference is negligible or that constraint selection cannot help.

  2. I have concerns about the benchmark construction. The authors exclude 389 probes considered impossible, but these are not formally verified to be unsatisfiable. Some of them can actually be solved by other models. Since this filtering depends partly on model performance, it may introduce bias into the evaluation. I would also like to see more analysis of how rejection sampling affects the distribution of constraint combinations at larger k.

  3. The novelty of the main findings seems somewhat limited. The observation that joint success decreases rapidly when more constraints are added is expected, given that each constraint has a nonzero failure probability. While the paper provides extensive empirical results, I do not think it establishes a sufficiently strong explanation of the underlying mechanism. The presentation could also be improved, as the paper includes many secondary analyses that make the main argument difficult to follow.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Part 1 of Response to Reviewer PYFF

We appreciate the reviewer's thoughtful feedback and the questions they raised. We have addressed these points through additional analyses and revisions to the paper.

"I am not convinced by the paper's conclusion that constraint failures are largely independent. [...] Also, the independence-based model predicts only 3.3% joint success at k=8, compared with the observed 9.6%. I think the evidence is insufficient to conclude that constraint interference is negligible or that constraint selection cannot help."

We agree and have narrowed the claim accordingly. Weak pairwise correlations do not tell us whether one constraint's presence lowers another's pass rate, so we now test this directly. Within each model and k, we compare each constraint type's pass rate on probes that contain another type with its pass rate on probes that do not (1,202 ordered pairs, all 19 models). We find that specific pairs do interfere. In 52 pairs, the presence of the second constraint lowers the first one's pass rate by more than 10 pp, with drops as large as 30.5 pp. Each of the ten strongest pairs shows a drop in at least 17 of the 19 models. Many of these cases involve one constraint that requires a particular output structure, such as a JSON object or ranked list, and another that restricts the sentences or words in that output. However, these large effects are not typical: for 995 of the 1,202 pairs, the change is below 5 pp. Part 1 of our reply to Reviewer evAu presents the full analysis and explains why the original φ summary missed these interactions.

The gap at k=8 is also real. When we account for shared probe difficulty, estimated using the other 18 models, predicted joint success rises from 3.3% to 5.4%, explaining about a third of the gap. The same analysis explains half of the gap at k=5 and 6% at k=12 (Part 3 of our reply to Reviewer evAu). The product therefore captures the decline in joint success as k increases, but it does not accurately predict the level of success at high k.

Changes to the paper. We replaced the abstract's "No pair of constraints interferes [...]" and §6's "choosing or ordering constraints cannot help", as detailed in our reply to Reviewer evAu. §4.3 now reports both analyses.

"The authors exclude 389 probes considered impossible, but these are not formally verified to be unsatisfiable. Some of them can actually be solved by other models. Since this filtering depends partly on model performance, it may introduce bias into the evaluation."

Including the 389 inferred probes as satisfiable lowers every model's sCSR, but leaves the rankings unchanged. We agree that these probes are hard rather than proven impossible. §4.6 reports that Claude 5 Opus and GPT-5.6 Sol solve six of these probes (seven probe-model pairs). Since the exclusion partly depends on model outcomes, we recomputed Table 1 with all 389 probes counted as satisfiable:

Table R1. sCSR in %; Δ in pp, computed before rounding; k* is the breakdown point. Bold marks the two breakdown points that change.

ModelsCSRsCSR, inferred includedΔk*k*, inferred includedRankRank, inferred included
GPT-6 Astra64.358.7−5.67711
GPT-5.563.157.6−5.57622
GPT-5.6 Sol56.351.5−4.86633
Claude 5 Opus52.347.9−4.46644
Claude 4.7 Opus46.242.2−4.15455
Gemini 3.1 Pro39.235.8−3.44466
Kimi K336.933.7−3.24477
Claude 4.6 Opus31.528.7−2.73388
Gemini 3.1 Flash-Lite24.122.0−2.13399
GPT-5.4 Pro23.221.2−2.0331010
DeepSeek V4 Pro22.420.5−2.0331111
GPT-5.218.817.1−1.6221212
Claude 4.5 Sonnet17.315.8−1.5221313
Llama 4 Maverick15.013.7−1.3221414
Qwen 3 235B Inst.12.611.5−1.1221515
Llama 3.1 405B12.411.3−1.1221616
Llama 3.1 70B11.610.6−1.0221717
Grok 4.111.310.3−1.0221818
Kimi K2.67.26.6−0.6111919
  • Every model's sCSR decreases, by between 0.6 pp (Kimi K2.6) and 5.6 pp (GPT-6 Astra), but no model changes rank.
  • Only two models change breakdown point: GPT-5.5 (7 → 6) and Claude 4.7 Opus (5 → 4).
  • Averaged across models, sCSR decreases from 9.6% to 8.5% at k=8 and from 2.7% to 1.8% at k=12. Since 360 of the 389 probes have k ≥ 6, including them makes the decline slightly steeper.

Excluding these probes therefore raises the reported sCSR at high k, but does not change the model rankings or the overall pattern of collapse.

AuthorsOpenReview ↗

Part 2 of Response to Reviewer PYFF

"I would also like to see more analysis of how rejection sampling affects the distribution of constraint combinations at larger k."

Rejection sampling changes the mix of constraint types at large k, but we find no corresponding increase in average probe difficulty. At k = 10 to 12, individual constraint types appear between 0.26 times (R2, transitive ordering) and 1.65 times (L6, vowel-consonant ratio) as often as they would under uniform sampling. Six types appear at less than half their uniform share, while two appear at more than 1.5 times. At k = 3 to 5, the distribution is closer to uniform, with every type appearing between 0.74 and 1.27 times its uniform share.

We also examined whether this shift makes the probes at larger k systematically harder. Section P shows no trend with k in the average difficulty of the constraints included in each probe. Moreover, probes composed of easier and harder constraint types converge to nearly the same sCSR by k=8 (9.9% versus 9.3%). The two types that most often lower other constraints' pass rates, R1 and R2, actually become less common at large k, appearing at 0.68 and 0.26 times their uniform share.

Changes to the paper. Section U now includes Table 1 recomputed with the inferred probes treated as satisfiable, and Section P reports the frequency of each constraint type at every k.


"The observation that joint success decreases rapidly when more constraints are added is expected, given that each constraint has a nonzero failure probability. While the paper provides extensive empirical results, I do not think it establishes a sufficiently strong explanation of the underlying mechanism."

We agree that joint success is expected to decline as more constraints are added. Our contribution is to measure how quickly it declines, what determines that rate, and where the product approximation breaks down.

  • The rate varies substantially across models and cannot be predicted from single-constraint accuracy alone. The per-constraint decay factor r ranges from 0.721 (DeepSeek V4 Pro) to 0.988 (Claude 5 Opus; §4.2). For example, Kimi K3 ranks fourth among 19 models at k=1 (94.3%), but falls to 17th on mCSR (§6). Benchmarks that test only one to three constraints cannot capture these differences in decay.
  • The type of constraint also helps explain the rate. Constraints that require tracking structure across the entire output are harder to maintain as more requirements are added. Structural constraints lose their single-constraint accuracy 1.97 times as fast as lexical constraints (95% CI 1.81 to 2.15; §4.4).
  • Our new analyses in Part 1 provide further evidence about the mechanisms behind these results. We find that specific pairs of constraints interfere, often when one requires a particular output structure and another restricts the sentences or words within it. We also find that shared probe difficulty explains between a third and a half of the gap between observed joint success and the product prediction at k = 5 to 8.
Reviewer_evAuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how LLM instruction following performance decreases as constraints increases, and whether failures combine independently or interfere with proposed CSE, a procedurally generated benchmark with 36 constraint types and deterministic verifiers. One of the main observation is that per-constraint accuracy declines gently while joint success collapses.

Critical strengths
  1. A rigorous benchmark for measuring degradation as constraints accumulate: No LLM judge, constraint count up to k=12, and 19 models are covered.
  2. A clear presentation of settings and analysis.
Critical weaknesses
  1. The claims about interference and constraint ordering are stronger than the evidence supports.
    φ (§4.3) measures association between outcomes when constraints co-occur; it does not measure whether one constraint’s presence lowers another’s pass rate. For example, A could pass 90% of the time with C and 60% with B while pairwise φ remains near zero. Likewise, randomized presentation order and a mean |ρ| of 0.011 (§4.5) show little average monotonic position association, but do not exclude non-monotonic or constraint-specific ordering effects. These analyses therefore do not establish that selecting or ordering constraints cannot help. This limitation directly affects the paper’s central distinction between independence and interference.

  2. The explanation for joint success remains incomplete. Weak pairwise correlations do not establish joint independence. Observed joint success substantially exceeds the product prediction, particularly for the strongest models at k=8 (33.7% versus 12.2%); at k=12, the overall figures are 2.7% versus 0.1%. The authors acknowledge this discrepancy and treat the product as an empirical lower bound. However, they do not quantify whether the proposed probe-level difficulty explanation accounts for the gap. The product baseline is informative, but the evidence does not fully establish the dependence structure underlying joint success.

Questions for authors
  1. Could the near-zero average position correlation conceal non-monotonic or constraint-specific effects? Would comparing alternative orderings of the same probes, particularly hard-to-easy versus easy-to-hard, recover the ordering effects reported by Zeng et al. (2025)?

  2. Beyond the GPT-6 Astra control, how stable are model rankings and breakdown points when evaluated on the same probes at common output-token budgets? The reported improvement in DeepSeek V4 Pro from k*=3 to 5 makes this an important comparison.

3 comments
AuthorsOpenReview ↗

Part 1 of Response to Reviewer evAu

We thank the reviewer for their careful reading and constructive comments, which have helped us clarify several points and improve the paper.

"φ (§4.3) measures association between outcomes when constraints co-occur; it does not measure whether one constraint's presence lowers another's pass rate. [...] These analyses therefore do not establish that selecting or ordering constraints cannot help."

You are correct: we tested this, and find that specific pairs of constraints do interfere. Our original φ analysis only considers probes containing both constraints, so it cannot tell us whether adding one constraint changes the other's pass rate. To answer that question, we compare, for every ordered pair of constraint types (A, B), A's pass rate when B is present with its pass rate when B is absent. We find that B's presence changes A's pass rate in 628 of the 1,202 pairs, lowering it in 249 and raising it in 379. As a check, we shuffle pass and fail outcomes within each model, k, and constraint type. Under this shuffle, only 0.1 pairs reach significance on average. The larger effects are also unlikely to arise by chance: in 52 pairs, B lowers A's pass rate by more than 10 percentage points, whereas none does after shuffling.

A (pass rate lowered)B (present)Δ pass rate of A [95% CI]Models with a drop
L6 vowel-consonant ratioW5 consonant cluster density−30.5 pp [−34.6, −26.4]19 of 19
S2 exact sentence countR2 transitive ordering−28.1 pp [−34.3, −21.9]19 of 19
S2 exact sentence countR1 scene graph−27.1 pp [−31.0, −23.4]18 of 19
W1 no word repetitionR1 scene graph−23.7 pp [−27.6, −19.8]19 of 19
M2 self-countingN2 no digits−22.0 pp [−23.7, −20.3]17 of 19

The intervals are based on 1,000 bootstrap resamples of probes. Many of the largest drops occur when one constraint requires a particular output structure or content, such as a JSON object, ranked list, table, or numbers, while another restricts the words or sentences in that same output. Satisfying the first can therefore make the second harder, or even impossible, to satisfy. For example, R3 requires JSON with repeated keys "pet" and "color", while W1 forbids repeating words (Section A).

At the same time, these large effects are not typical. For 995 of the 1,202 pairs, the change in pass rate is smaller than 5 pp, and the median change is +0.7 pp. The decay reported in §4.1 and §4.2 therefore still holds.

Why the φ summary missed these pairs. In our original analysis, we set φ to 0 whenever it is undefined within a model × k cell, which happens when one of the constraints always passes or always fails. Strong interference can produce exactly these cases. With this convention, the 19-model analysis reproduces the published summary (mean +0.085, minimum −0.028). For M2 and N2, however, 97.6% of observations fall into cells where φ is undefined. If we exclude those cells instead of treating them as zero, the minimum φ is −0.68. The original summary therefore obscured some of the strongest interactions.

Changes to the paper.

  • Abstract: We replaced "No pair of constraints interferes: failures of different constraints are nearly uncorrelated, so the risk of failure compounds across constraints, and no choice or arrangement of constraints avoids the collapse." with "Most pairs of constraints change each other's pass rate by less than 5 percentage points, so the risk of failure compounds across constraints. Specific pairs do interfere: a constraint that fixes part of the output, such as a JSON object or a ranked list, lowers the pass rate of constraints checked on sentences or words by up to 28 percentage points. Choosing constraints can therefore help, whereas their order has no detectable average effect."
  • §4.3: We retitled this section "Do constraints interfere with each other?" and added the direct pairwise test. Section N reports all pairs and explains how undefined cells were handled in the φ summary.
  • §6: We replaced "Failures barely depend on which constraints appear together, so choosing or ordering constraints cannot help." with "Avoiding specific pairs of constraints raises constraint compliance, whereas the order of the constraints has no detectable average effect."
AuthorsOpenReview ↗

Part 2 of Response to Reviewer evAu

"Could the near-zero average position correlation conceal non-monotonic or constraint-specific effects? Would comparing alternative orderings of the same probes, particularly hard-to-easy versus easy-to-hard, recover the ordering effects reported by Zeng et al. (2025)?"

The average effect masks differences across constraints: changing their order helps some and hurts others, but these effects largely cancel out. Reordering constraints within otherwise identical probes does not detectably change overall constraint compliance. We test constraint order in three ways.

Position effects for individual constraint types. We first examine whether a constraint is easier to satisfy depending on where it appears in the prompt. For each of the 36 types, we compare its pass rate when listed first, in the middle, or last, separately within each model and k (k ≥ 3). We find position effects for 17 of the 36 types, with differences as large as 9.2 percentage points. The direction varies across constraints. For example, W2 (word length range) passes 9.2 pp more often when listed first rather than last, while R2 passes 7.9 pp more often when listed last. Five of the 17 types do better at either end than in the middle. These differences largely cancel when we combine all constraint types: the average difference between first and last position is only +1.0 pp (95% CI +0.4 to +1.6).

Easy-to-hard versus hard-to-easy order in the existing data. The composer randomly shuffles the constraints in each probe, so some probes happen to list easier constraints before harder ones, while others do the reverse. We measure difficulty using each constraint type's single-constraint pass rate, pooled across models. Within each model and k, we compare the third of probes whose order is closest to easy-to-hard with the third closest to hard-to-easy. We find no detectable difference in either sCSR (+0.3 pp, 95% CI −1.1 to +1.5) or mCSR (+0.0 pp, 95% CI −1.1 to +1.0).

The same probes in three orders. We also test whether explicitly sorting the constraints produces the improvement reported by Zeng et al. (2025). We evaluate three models on 150 satisfiable probes (50 each at k = 6, 8, and 10), presenting each probe in three versions: its original order, easy to hard, and hard to easy. Everything else in the prompt stays the same. We use the composer's own function to rebuild the prompts and verify that it reproduces all 150 original prompts exactly.

sCSR (%) ↑OriginalEasy → hardHard → easyEasy→hard minus hard→easy [95% CI]
GPT-5.5 (148 probes)25.720.926.4−5.4 pp [−11.5, +1.4]
Claude 4.7 Opus (150)27.326.728.0−1.3 pp [−9.3, +6.7]
Gemini 3.1 Pro (148)8.19.510.1−0.7 pp [−4.7, +3.4]
All three (446)20.419.121.5−2.5 pp [−6.3, +1.3]

We compare the different orders on the same probes, with confidence intervals from 2,000 bootstrap resamples. None of the differences reaches p < 0.05 under McNemar's test (smallest p = 0.15, for GPT-5.5). The two sorted orders also differ by no more than 2.0 pp in mCSR for any model.

Claude 4.7 Opus and Gemini 3.1 Pro use the same settings as in the paper. Their original-order results are consistent with those reported there on the same probes (sCSR 26.4% versus 26.4%, and 8.0% versus 10.0%). GPT-5.5 required one adjustment: at its default reasoning effort, it exhausted the entire 4,096-token budget on reasoning, so we used low effort, as we do for GPT-5.6 Sol and Claude 5 Opus in the paper. For this reason, we compare GPT-5.5's results across orders within this experiment rather than directly against the main paper.

Together, these tests show that position affects individual constraints, sometimes substantially, but the effects differ in direction and largely cancel out. Neither the random ordering in the existing data nor explicitly sorting the same probes from easy to hard or hard to easy produces a detectable change in the rate at which models satisfy all constraints.

Changes to the paper. In §4.5, we replaced "a constraint's position in the prompt barely predicts whether it passes (mean |ρ|=0.011; Section O)" with "position changes the pass rate of 17 of the 36 constraint types by up to 9.2 pp, in different directions that cancel on average, and sorting the same probes from easy to hard or from hard to easy does not detectably change sCSR (Section O)". Section O now reports all three analyses.

AuthorsOpenReview ↗

Part 3 of Response to Reviewer evAu

"However, they do not quantify whether the proposed probe-level difficulty explanation accounts for the gap."

Shared probe difficulty explains about half of the gap at k=5 and a third at k=8. For each model, we estimate how difficult a probe is by measuring how often the other 18 models satisfy its constraints. We then fit a logistic regression for each model, with a separate intercept for each constraint type at each k and one slope for probe difficulty. The intercepts preserve each type's average pass rate at each k, as used in the original product. We then multiply these probe-adjusted pass rates to obtain a new prediction.

sCSR (%), 19 modelsk=2k=5k=8k=10k=12
Observed57.621.29.65.12.7
Product of per-constraint pass rates (paper)55.815.13.30.80.1
Product given probe difficulty57.518.25.41.70.2
Share of the gap explained94%50%34%21%6%

For the five models with k* ≥ 5, the prediction at k=8 increases from 12.2% to 19.9%, compared with 33.7% observed, explaining 36% of the gap. Because difficulty measured using other models is only an approximation of how difficult a probe is for the model being evaluated, these shares are lower bounds. At k ≥ 10, most of the gap remains unexplained.

Changes to the paper. In §5, we replaced "This would also explain why joint success sits above the product at high k." with "Shared probe difficulty, measured on the other models, accounts for 50% of the gap between joint success and the product at k=5, 34% at k=8, and 6% at k=12 (Section P)." Section P reports the full table for every k and both groups of models.

"Beyond the GPT-6 Astra control, how stable are model rankings and breakdown points when evaluated on the same probes at common output-token budgets?"

At a common budget of 4,096 tokens, all 18 comparable models retain their Table 1 ranking. Fifteen models in Table 1 already run at 4,096 tokens. Of the four evaluated at 16,384 tokens, three also have 4,096-token runs on the same probes; Claude 4.7 Opus does not. Counting probes without a scored response as failures, we obtain:

ModelsCSR (%), 4,096 tokensk*, 4,096 tokenssCSR (%), Table 1k*, Table 1
GPT-6 Astra63.8764.37
GPT-5.563.1763.17
GPT-5.6 Sol60.2756.36
Claude 5 Opus50.8652.36

The other 14 models also retain their Table 1 order. Only GPT-5.6 Sol's breakdown point changes, increasing from 6 to 7. Claude 5 Opus produces a scored response on only 3,090 of the 4,527 probes at 4,096 tokens, which is why we report its results at 16,384 tokens in the paper. DeepSeek V4 Pro's improvement from k*=3 to 5 comes from the planning control, which uses a smaller subset of 50 probes per k. We do not have 16,384-token runs of the 4,096-token models on the full probe set, so we use 4,096 tokens for the common-budget comparison.

Changes to the paper. We added this table to the appendix on model notes, alongside the existing GPT-5.6 Sol budget comparison in Section A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.