acceptodds
Under review as a conference paper at ICLR 2027

Beyond Benchmark Accuracy: Stress-Testing Reasoning Claims in Large Language Models

Abstract

Large language models, and reasoning models in particular, score highly on mathematical, logical and coding benchmarks, and those scores are increasingly read as evidence of general reasoning ability. Accuracy on a fixed item set does not establish that the underlying competence survives controlled variation in problem depth, surface form or evaluation condition. We stress-test reasoning claims along four dimensions — complexity generalization, perturbation robustness, process faithfulness and familiarity–novelty sensitivity — and add a cross-dimensional analysis of how the first two interact. Fourteen contemporary configurations over ten model checkpoints, including four reasoning-disabled counterparts of listed checkpoints, are evaluated on two generated task families with a depth ladder reaching , on five benchmark reference points scored as integer item counts, and under transformations separated by whether they preserve the correct answer. Benchmark accuracy and stress behaviour order the roster similarly overall (Spearman ) and come apart where a benchmark discriminates least: configurations within 5\ percentage points of one another on MATH-500 differ by up to 23.8\ points in their answer-preserving robustness gap, and the two orderings disagree on 8 of 91 configuration pairs. The incremental information is local to near-ties rather than a general decoupling, and it is smaller among distinct checkpoints than across all configurations. The perturbation penalty compounds with depth rather than adding to it — the fitted interaction is negative for every configuration, and for the strongest the gap widens from 0.8 points at to 11.3 at . The direction holds under depth entered as a factor, as a spline and segmented at the changepoint, under matched input length, and under inference clustered on seed templates rather than items: 184 of 192 intervals lie below zero, the exceptions being the two strongest configurations under the widest specifications. Threshold-like degradation is identified for ten configurations with the changepoint tracking capability, right-censored for the two strongest and unidentifiable for the two smallest, so a single ladder cannot serve the whole capability range. These are behavioural results about generalization under stated conditions, not findings about internal computation. Within that scope they indicate that benchmark accuracy and robust generalization should be reported as distinct empirical properties, and that a robustness estimate obtained at one difficulty level understates fragility at greater depth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning Models, Reasoning Evaluation, Robust Generalization, Stress Testing, Chain-of-Thought, Benchmark Robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6B8JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes a multidimensional framework for stress-testing reasoning capabilities in large language models beyond standard benchmark accuracy. It evaluates complexity generalization, perturbation robustness, process faithfulness, and familiarity–novelty sensitivity, with particular emphasis on the interaction between reasoning depth and perturbations. Experiments on 14 model configurations across two synthetic task families and several established benchmarks reveal that robustness differences can be substantial even among models with similar benchmark accuracy, and that perturbation sensitivity generally increases with reasoning depth.

Critical strengths
  1. Systematic evaluation of reasoning robustness. The paper provides a structured evaluation framework covering multiple aspects of reasoning behavior. The distinction between answer-preserving and answer-changing transformations is particularly useful for avoiding inappropriate robustness comparisons.

  2. Careful experimental design and statistical analysis. The experiments cover diverse model configurations, multiple reasoning depths, and perturbation types. The authors additionally examine alternative statistical specifications, template-level dependencies, inference-budget effects, and run-to-run variability, which strengthens the reliability of the empirical findings.

Critical weaknesses
  1. Limited novelty relative to existing reasoning robustness evaluations. The individual evaluation dimensions, including complexity generalization, perturbation robustness, rationale faithfulness, and novelty sensitivity, have been extensively studied. The main methodological contribution is combining these dimensions and modeling the depth–perturbation interaction. However, it remains unclear whether this integration provides sufficiently new insights beyond existing robustness and complexity studies.

  2. Limited generalizability beyond synthetic tasks. The central depth–perturbation interaction is primarily established on two generated task families, Synth-Comp and Synth-Logic. Although these tasks enable controlled evaluation, they may not capture the diversity of reasoning processes in realistic mathematical, coding, or agentic settings. The paper therefore provides limited evidence that the observed interaction generalizes to real-world reasoning tasks.

  3. Limited additional discriminative value over benchmark accuracy. The reported ranking correlation between benchmark accuracy and robustness is already high (Spearman ρ = 0.92). When considering only distinct model checkpoints, it increases to 0.98, with only 2 of 45 model pairs exhibiting ranking inversions. This suggests that the proposed evaluation framework provides relatively limited additional discrimination between independent models, weakening its practical significance as an alternative to standard benchmark accuracy.

Questions for authors

Q1. Can the authors validate the depth–perturbation interaction on more realistic reasoning tasks beyond the two synthetic families?

Q2. What additional insights does the proposed framework provide beyond existing complexity and perturbation evaluations when these are applied separately?

Q3. Given the high ranking agreement with benchmark accuracy, in which practical scenarios would the proposed framework meaningfully change model selection decisions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6B8J (novelty, external validity, discriminative value)

The contribution is incremental, external validity is unestablished, and the additional discrimination over benchmark accuracy is local rather than general. The revised PDF adds three analyses from the released data: a descriptive near-tie noise comparison, a depth-transfer analysis and a smooth-model comparison, in Table 2 (§5.1) and the Rebuttal Analyses appendix.

1. Novelty (Q2)

Neither axis is new alone; we claim no priority (§2 closest-prior-work table). The revised Introduction states three findings: (i) where deep-level accuracy stays above the floor bound, a shallow-depth estimate understates the deep penalty by a median factor of 9.6; (ii) robustness separates configurations a saturated benchmark cannot: all 38 near-tie pairs show a gap difference above a descriptive noise band; (iii) one ladder cannot cover the capability range, and a smooth-model comparison shows n∗n^* is the best single break under this ladder, not a discrete mechanism. The answer-preserving / answer-changing split remains the enabling distinction.

2. Generalizability (Q1)

The interaction is examined only on Synth-Comp and Synth-Logic; the five established benchmarks are accuracy reference points, not perturbation tests. External validity is unestablished. A real-task layer needs items with an identifiable depth ladder, answer-preserving transformations with controlled edit counts and a new evaluation run. It remains a necessary next step. Revised Limitations states this explicitly and the Introduction points to it.

3. Discriminative value (W3, Q3)

The MATH-500-based agreement is ρ=0.916\rho = 0.916 (8/91 inversions); FrontierMath gives the same (8/91) and, with one configuration per checkpoint, ρ=0.976\rho = 0.976 (2/45) when the reasoning-enabled member is kept (App. O) but ρ=0.770\rho = 0.770 (8/45) with the reasoning-disabled member. The appendix table reports both.

Table 2 adds a descriptive noise comparison: in all 38 pairs within 5 pp on MATH-500 the gap difference exceeds the descriptive run-to-run noise band (median 7.8 pp, median z=28.1z = 28.1; the denominator is a standard error of the mean difference, and the 38 pairs are not independent tests). The four reasoning ON/OFF contrasts all show a smaller gap for the enabled member, but only two have weight identity established and none is a causal estimate of a reasoning effect.

A concrete case shows diverging evaluation evidence. Gemini 3.7 Flash and Gemini 3.1 Pro are 0.4 pp apart on MATH-500 (94.6 vs 94.2), yet the ordering reverses on FrontierMath (23.4 vs 28.6), robustness gap (13.0 vs 10.0 pp) and normalised complexity AUC (0.860 vs 0.940). The 0.4 pp margin is small and may be affected by run-to-run variability. Excluding saturated configurations and those near the accuracy floor, a shallow estimate understates the deep penalty by a median 9.6× at the main cut-off; the understatement stays substantial across all three cut-offs (6.71×–9.63×), while rank agreement is more sensitive to the exclusion rule (0.39, 0.90, 1.00).

We have not demonstrated a prospective model-selection reversal or a better deployment outcome. What is measured is an error: for workloads deeper than the level at which robustness was estimated, a single-depth figure systematically understates fragility. This boundary is in §5.1, §6 and the appendix.

4. Presentation

The abstract is one paragraph; the Introduction's scope paragraph is condensed; repeated qualifications were removed; the contribution list and Discussion opening state the three findings.

Thank you for the authors' response and for addressing some of my concerns. However, my Q1 remains unresolved. In particular, the central depth–perturbation interaction is still evaluated only on Synth-Comp and Synth-Logic, without validation on more realistic reasoning tasks. Although the authors have acknowledged this limitation and clarified the scope of their claims, the lack of evaluation beyond these two synthetic task families remains a significant limitation. Therefore, I will maintain my original rating, and improve the presentation score.

Reviewer_Wkm2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, they tried to check the common question about the reasoning ability of LLMs in a way that is accurate enough for it, or if the problem conditions change, models that have a high accuracy still have difficulties. They do this with a stress-testing approach with four different dimensions and then evaluate 14 model configurations from 10 different checkpoints with two synthetic tasks with different levels of complexity and five benchmarks. And also make a distinction between when perturbations keep the correct answer and when they change the answer but keep the reasoning procedure. They show that accuracy and robustness are highly correlated, but if two models have similar accuracy, they can have different robustness, and they also show that the effect of perturbation is stronger when problem depth is higher, and different models show different points where their accuracy starts to reduce rapidly.

Critical strengths

Their extensive experiments and generalizability are very solid. Their design that they do not rely on model accuracy changes under perturbation and expand it to examine how these effects interact with problem complexity is solid. Also in 5.4 the way that they show that the effect of perturbations is not necessarily constant across different difficulty levels is also solid. Also they way they address that when it is not the case but accuracy floor make weaker models be more robust at higher difficulty levels with logit scale interaction is solid. They also consider testing different specifications of the depth variable, use a length control, and account for dependencies between examples generated from the same template. They also consider not evaluating answer preserving and answer changing perturbation with the same consistency metric and make distinction between them in 3.

Critical weaknesses

It is not convincing that robustness captures something different from benchmark accuracy. Although in 5.1 they show accuracy and robustness is highly correlated, they also mentioned that the difference becomes more pronounced when models have similar accuracy but it is not clear. For example in table1 they compare models on math500 that are very close to saturation for the strongest model but they also report frontier math that has a more accuracy variance. And the top four models on math500, their frontier math ranking follow their robustness ranking too but it is not the case for math 500. So the difference might be because of the benchmark they use that is saturated instead of what they claim that robustness is a different measure. They also mention that correlation increases to 0.98 when only one configuration per checkpoint is considered.

It is not convincing that perturbation has the same effective difficulty or strength in different depth levels. Although in 5.4 they show that perturbation becomes more damaging as the problem depth increases, but the interpretation of the interaction between complexity and perturbation is not convincing. For example, when the number of steps increases changes in premise reordering, variable renaming or reformulation for models become more difficult to process so this negative interaction might come from the way that perturbation changes with problem complexity instead of coming from an interaction between reasoning depth and robustness. Although they consider some useful controls such as different depth specifications, class-specific effects, and input-length matching, but these can not establish that the extent of the perturbation is comparable across different depth levels. The results show this pattern statistically, but for more convincing findings it is good to better understand that it is related to increasing reasoning depth or to changes in the difficulty of the perturbation itself.

On the small side note the last paragraph of introduction i think it is not necessary to come there and it could be added in a limitation section.

On another small side of note, make the abstract as all in one paragraph is much better than having multiple paragraphs in the abstract.

Questions for authors

In 5.1, can you compare the robustness rankings directly with frontiermath or other unsaturated benchmarks, instead of with math500? Because it is useful to know that robustness differences are still informative after accounting for performance on other benchmarks.

In 5.4, can you make it clear that are the perturbations controlled to have comparable strength across different depth levels? For example, if the number of changes by a perturbation is fixed when the reasoning depth is increasing does the negative interaction stay?

In 5.2, as you use a two segment model and compare it with a single linear model for identifying the points that performance change, can you clarify that why can not a smooth non linear decline explain the same thing without a distinct point that it changes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Wkm2 (saturation, perturbation dose, changepoint comparison)

Q1 and Q3 receive additional analyses (§5.1, §5.2, §5.4, Table 2, Rebuttal Analyses appendix). Q2 remains unresolved at the level of a fixed-edit control; we report the partial decomposition the existing data support.

1. Q1: benchmark saturation (completed)

Using the released benchmark, paired-outcome and roster data, with §5.1's gap definition (unweighted mean Category-A risk difference over five classes, both families, n=7n=7), and retaining the reasoning-enabled member of each checkpoint (App. O):

ScopeReferenceρ\rhoInv.Max gap diff (≤5 pp)
14 configurationsMATH-5000.9168/9123.8 pp (38 pairs)
14 configurationsFrontierMath0.9168/9118.4 pp (24 pairs)
10 checkpoints, ON-retainingMATH-5000.9762/4512.9 pp (24 pairs)
10 checkpoints, ON-retainingFrontierMath0.9762/4514.7 pp (13 pairs)

Median |gap diff| is 7.8 pp (IQR 4.7–12.5) on MATH-500, 6.4 pp (3.1–11.0) on FrontierMath; distributional summaries, the OFF-only subset and sensitivity are in the appendix. Retaining the reasoning-disabled member instead gives ρ=0.770\rho = 0.770 with 8/45, so the reduced-set agreement is selection-sensitive. §5.1 now labels 23.8 pp as a MATH-500-based quantity for this roster and keeps the claim local to near-ties. An unsaturated reference point changes neither global agreement nor inversions, only the size of the local separation.

2. Q2: perturbation strength (partial)

The seven existing specifications change how depth enters the model or the input length; none holds transformed-element count constant as nn grows. The released data contain no per-item edit count and no fixed-edit arm, so that question cannot be answered from them. A fixed-edit evaluation needs new items and a new run; it remains future work.

What the data support is a bounded partial decomposition, now in §5.4 and the appendix:

  • The interaction is negative in all 140 per-class fits, with every upper interval limit below zero (per-class counts are in the appendix table).
  • The four non-insertion classes (A1, A2, A3, A5) alter text without adding distractor content: 112/112 negative; the class with unchanged input length (premise reordering, 0% logit-scale length share) is 28/28 negative.
  • The largest mean logit-scale length share is 30.7%.

§5.4 now says the interaction describes the increasing penalty under this protocol, not a depth-specific effect; Limitations states the confound. Edit counts are not held constant across depths, so these are descriptive bounds on one driver of perturbation dose.

3. Q3: smooth competitors (completed)

Five candidates (single line, two-segment, quadratic, four-parameter logistic, three-degree spline) were fitted on ten aggregated depth levels per model–family curve. The objective is a design-effect-adjusted binomial pseudo-likelihood (effective-sample-size approximation): trials are scaled by the measured design effect, successes are set from observed accuracy and rounded, so effective counts are generally non-integer. AICc counts ten depth levels and the grid-searched break as a parameter; stability uses leave-one-depth-out (LODO) prediction.

Among 20 identified curves, two-segment wins 12 by AICc and 8 by LODO. The 4 censored curves show no consistent two-segment preference (linear 2, logistic 2 by AICc), while the 4 saturated curves more often select smooth alternatives (quadratic 3, spline 1). The comparisons are qualitatively consistent with the regime distinction. This is a candidate-set sensitivity analysis, not independent validation. LODO uses the same seed templates across depth levels, so it measures fit across depth points, not cross-template generalisation; the design-effect deflation addresses within-level clustering only. The appendix states both limits. ΔAIC\Delta\mathrm{AIC} compares two segments with a single line; n∗n^* is the best single break under this ladder and candidate set, not a discrete mechanism (§5.2).

4. Presentation

Both requests are implemented: the abstract is one paragraph and the Introduction's closing paragraph is one scope statement.

Thank you for the detailed response and the additional analyses. I appreciate the comparison with FrontierMath and the more precise framing of the additional information provided by robustness. The comparison with smooth alternatives also helps clarify how the reported changepoints should be interpreted. My main remaining concern is that perturbation strength is not held constant across depth levels. The analyses by perturbation class and the input-length controls are useful, but they cannot distinguish the effect of increasing reasoning depth from an increase in the number or difficulty of the changes introduced. I appreciate that the revised paper explicitly acknowledges this limitation and narrows its interpretation accordingly. These additions strengthen the paper, but this remaining limitation keeps my assessment unchanged. I will maintain my original score.

Reviewer_24yYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper stress-tests 14 LLM configurations (10 checkpoints, four run with reasoning both on and off) along four axes: accuracy (with problem depth), on two synthetic task families (arithmetic word problems and propositional logic), robustness to perturbations that preserve the answer versus ones that change it, sensitivity to edits of the model’s written reasoning and sensitivity to unfamiliar problem templates.

Critical strengths
  • It cleanly separates perturbations that keep the answer the same from ones that change it, and scores each appropriately.
  • It is unusually transparent: integer counts are given, the confounds in the reasoning-on/off pairs are flagged, code is released, and scope limits are stated.
  • A very extensive set of experiments
Critical weaknesses
  • The paper is extremely hard to read. I had to read it twice and certain sentences multiple times. I have a feeling the whole thing is written by AI, but I could obviously be wrong. Just cleaning up the writing would be doing a massive favor to this paper.

  • The divergence claim is weak. With distinct checkpoints only, ρ = 0.98 and just 2 of 45 pairs invert. Inversions among near-ties are exactly what noise in the benchmark ranking would produce.

  • Narrow scope and incremental novelty. The paper uses two synthetic families, and the depth ladder is too short for the frontier models.

  • The core contribution is crossing two axes that GSM-Symbolic, MATH-Perturb and Illusion of Thinking (https://machinelearning.apple.com/research/illusion-of-thinking) already study separately.

  • There's quite a lot of numbers in this paper (which is fine) but some of them did not match up. So I put it through an LLM to check for inconsistencies and this is what came out:

  • The smallest gap is “5.0” in Secs. 5.1/6 but 5.3 in Table 1. The frontier span is “5.0–11.7” in the text vs. 5.3–13.0 in Table 1.

  • The headline “0.8 → 11.3 pts” matches no table: Table 17 gives 0.8 → 19.3, Table 9 gives 2.1 → 11.7, and Fig. 3 gives 4.0 → 22.9.

  • Template correlation is 0.090 (design effect 3.16/3.61) in App. S and I, but 0.024 (design effect 1.65) in Table 22. The clustered detectable-effect ranges disagree accordingly.

  • The abstract says 184/192 intervals are below zero. App. S says all 192 are, and App. J says every identified interval is.

  • App. J’s description of Table 2 doesn’t match it: “within 1%” vs. 1.10× and 1.06×.

  • Sec. 3’s model includes a template random effect, but Table 9’s caption describes a model without one.

  • Tables 13–16 describe p-value columns that aren’t present.

  • App. W says only four configurations were re-run; App. E and Table 7 say all 14.

Questions for authors
3 comments
AuthorsOpenReview ↗

Response to Reviewer 24yY (numerical consistency, presentation, informativeness)

Thank you for the numerical check. All eight items are corrected in the revised PDF. The underlying evaluation data and primary outcome tables are unchanged; the precision-analysis text now reports the template-simulation-based clustered MDES of 3.1–7.0 pp (Table 23). The direction of the primary findings is unchanged.

1. Presentation and the divergence claim

We shortened long sentences, merged the abstract into one paragraph, and cut the scope statements down to one. The contribution list and Discussion opening now state three quantitative findings. The one-per-checkpoint result is selection-sensitive: retaining the reasoning-enabled member gives ρ=0.976\rho = 0.976 with 2/45 inversions, retaining the reasoning-disabled member gives ρ=0.770\rho = 0.770 with 8/45. Abstract, §5.1, conclusion and Table 2 now say so.

The inversion count alone is weak evidence; the claim concerns gap magnitude. Table 2 adds a descriptive noise comparison: for each of the 38 pairs within 5 pp on MATH-500, z=∣Δˉa−Δˉb∣/sa2/na+sb2/nbz = |\bar\Delta_a-\bar\Delta_b|/\sqrt{s_a^2/n_a+s_b^2/n_b}, with n=10n = 10 runs per configuration. The denominator is a standard error of the mean difference, not the run-to-run SD. All 38 ratios exceed the descriptive noise band (median gap difference 7.8 pp, median z=28.1z = 28.1); the pairs share configurations and are not 38 independent tests. FrontierMath leaves the global correlation and inversion count unchanged.

2. The eight numerical items

  1. Smallest gap / frontier span. Table 1 and App. N give 5.3 pp (95% CI 3.8–7.0) and 5.3–13.0 pp; 5.0/11.7 were 12-configuration leftovers. §5.1 and §6 corrected.
  2. Headline 0.8→11.3. Four estimands, each correct within its own definition: 0.8→11.3 is the observed five-class mean for GPT-5.6 Sol (reasoning: high) on Synth-Comp (abstract, §5.4); 0.8→19.3 is A4-only at n=3/n=12 (Table 18); 2.1→11.7 is the depth-bin summary in Table 10 (Δ\Delta low 2.1 at n=3–5, Δ\Delta high 11.7 at n=10–12, same model); 4.0→22.9 is the fitted frontier-family mean in the Level-3 thumbnail of Figure 3 (recomputed 4.00/22.93). The abstract labels the observed value; §5.4 states Figure 1a's fitted curves are a different estimand.
  3. ICC / MDES. 0.090 is the latent-logit construction parameter; 0.0244/0.0225 is measured observed-scale ICC (deff 1.586/1.652). Clustered MDES is 3.1–7.0 pp (2.4–5.5 pp independent) from the template simulation in Table 23. App. S rewritten; App. I qualified; the obsolete description deleted.
  4. 184/192. 184 of 192 identified intervals lie below zero; 8 reach zero, all on the two strongest configurations under the widest specifications. App. S and App. J corrected.
  5. App. J ratios. Now 1.10×, 1.06×, 1.11×, 1.00×, 0.67× and 0.88×, matching Table 2.
  6. §3 model. One source-item random intercept; template dependence enters at inference (App. D). The template term was removed from the equation so §3 and App. H agree.
  7. p-value columns. Tables 14–17 captions described a permutation p and an independence mid-p the tables never print; both are in the released paired-outcome data. Captions now point there.
  8. App. W. All 14 configurations are evaluated five times; the four are the subset without a settable temperature, and the remainder is ten, not eight.

3. Informativeness

The revised Introduction states three findings: a shallow-depth estimate understates the deep penalty by a median factor of 9.6 where the accuracy floor does not bind; one ladder cannot cover the capability range, and a smooth-model comparison shows n∗n^* is the best single break under this ladder, not a discrete mechanism; and the interaction survives adjustment but is weaker (184/192, not all 192), holding in all 140 per-class fits including the zero-length-change class.

Are any of the changes in the manuscript highlighted in a different color? It's hard to know what has changed if not.

Yes — all changes in the revised main PDF are now marked in blue. We have uploaded that highlighted version as the revised PDF. The blue text covers the eight numerical corrections (§5.1, §6, App. I/J/S/W and the paired-table captions), the rewritten contribution list and Limitations, the new three-panel table in §5.1 with its companion appendix, and the added statistical definitions. Section 2 of our original response lists each corrected value together with the table or figure where it now appears, so the changes can be checked one by one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.