- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes a multidimensional framework for stress-testing reasoning capabilities in large language models beyond standard benchmark accuracy. It evaluates complexity generalization, perturbation robustness, process faithfulness, and familiarity–novelty sensitivity, with particular emphasis on the interaction between reasoning depth and perturbations. Experiments on 14 model configurations across two synthetic task families and several established benchmarks reveal that robustness differences can be substantial even among models with similar benchmark accuracy, and that perturbation sensitivity generally increases with reasoning depth.
Critical strengths
-
Systematic evaluation of reasoning robustness. The paper provides a structured evaluation framework covering multiple aspects of reasoning behavior. The distinction between answer-preserving and answer-changing transformations is particularly useful for avoiding inappropriate robustness comparisons.
-
Careful experimental design and statistical analysis. The experiments cover diverse model configurations, multiple reasoning depths, and perturbation types. The authors additionally examine alternative statistical specifications, template-level dependencies, inference-budget effects, and run-to-run variability, which strengthens the reliability of the empirical findings.
Critical weaknesses
-
Limited novelty relative to existing reasoning robustness evaluations. The individual evaluation dimensions, including complexity generalization, perturbation robustness, rationale faithfulness, and novelty sensitivity, have been extensively studied. The main methodological contribution is combining these dimensions and modeling the depth–perturbation interaction. However, it remains unclear whether this integration provides sufficiently new insights beyond existing robustness and complexity studies.
-
Limited generalizability beyond synthetic tasks. The central depth–perturbation interaction is primarily established on two generated task families, Synth-Comp and Synth-Logic. Although these tasks enable controlled evaluation, they may not capture the diversity of reasoning processes in realistic mathematical, coding, or agentic settings. The paper therefore provides limited evidence that the observed interaction generalizes to real-world reasoning tasks.
-
Limited additional discriminative value over benchmark accuracy. The reported ranking correlation between benchmark accuracy and robustness is already high (Spearman ρ = 0.92). When considering only distinct model checkpoints, it increases to 0.98, with only 2 of 45 model pairs exhibiting ranking inversions. This suggests that the proposed evaluation framework provides relatively limited additional discrimination between independent models, weakening its practical significance as an alternative to standard benchmark accuracy.
Questions for authors
Q1. Can the authors validate the depth–perturbation interaction on more realistic reasoning tasks beyond the two synthetic families?
Q2. What additional insights does the proposed framework provide beyond existing complexity and perturbation evaluations when these are applied separately?
Q3. Given the high ranking agreement with benchmark accuracy, in which practical scenarios would the proposed framework meaningfully change model selection decisions?
2 comments
Thank you for the authors' response and for addressing some of my concerns. However, my Q1 remains unresolved. In particular, the central depth–perturbation interaction is still evaluated only on Synth-Comp and Synth-Logic, without validation on more realistic reasoning tasks. Although the authors have acknowledged this limitation and clarified the scope of their claims, the lack of evaluation beyond these two synthetic task families remains a significant limitation. Therefore, I will maintain my original rating, and improve the presentation score.
Response to Reviewer 6B8J (novelty, external validity, discriminative value)
The contribution is incremental, external validity is unestablished, and the additional discrimination over benchmark accuracy is local rather than general. The revised PDF adds three analyses from the released data: a descriptive near-tie noise comparison, a depth-transfer analysis and a smooth-model comparison, in Table 2 (§5.1) and the Rebuttal Analyses appendix.
1. Novelty (Q2)
Neither axis is new alone; we claim no priority (§2 closest-prior-work table). The revised Introduction states three findings: (i) where deep-level accuracy stays above the floor bound, a shallow-depth estimate understates the deep penalty by a median factor of 9.6; (ii) robustness separates configurations a saturated benchmark cannot: all 38 near-tie pairs show a gap difference above a descriptive noise band; (iii) one ladder cannot cover the capability range, and a smooth-model comparison shows n∗ is the best single break under this ladder, not a discrete mechanism. The answer-preserving / answer-changing split remains the enabling distinction.
2. Generalizability (Q1)
The interaction is examined only on Synth-Comp and Synth-Logic; the five established benchmarks are accuracy reference points, not perturbation tests. External validity is unestablished. A real-task layer needs items with an identifiable depth ladder, answer-preserving transformations with controlled edit counts and a new evaluation run. It remains a necessary next step. Revised Limitations states this explicitly and the Introduction points to it.
3. Discriminative value (W3, Q3)
The MATH-500-based agreement is ρ=0.916 (8/91 inversions); FrontierMath gives the same (8/91) and, with one configuration per checkpoint, ρ=0.976 (2/45) when the reasoning-enabled member is kept (App. O) but ρ=0.770 (8/45) with the reasoning-disabled member. The appendix table reports both.
Table 2 adds a descriptive noise comparison: in all 38 pairs within 5 pp on MATH-500 the gap difference exceeds the descriptive run-to-run noise band (median 7.8 pp, median z=28.1; the denominator is a standard error of the mean difference, and the 38 pairs are not independent tests). The four reasoning ON/OFF contrasts all show a smaller gap for the enabled member, but only two have weight identity established and none is a causal estimate of a reasoning effect.
A concrete case shows diverging evaluation evidence. Gemini 3.7 Flash and Gemini 3.1 Pro are 0.4 pp apart on MATH-500 (94.6 vs 94.2), yet the ordering reverses on FrontierMath (23.4 vs 28.6), robustness gap (13.0 vs 10.0 pp) and normalised complexity AUC (0.860 vs 0.940). The 0.4 pp margin is small and may be affected by run-to-run variability. Excluding saturated configurations and those near the accuracy floor, a shallow estimate understates the deep penalty by a median 9.6× at the main cut-off; the understatement stays substantial across all three cut-offs (6.71×–9.63×), while rank agreement is more sensitive to the exclusion rule (0.39, 0.90, 1.00).
We have not demonstrated a prospective model-selection reversal or a better deployment outcome. What is measured is an error: for workloads deeper than the level at which robustness was estimated, a single-depth figure systematically understates fragility. This boundary is in §5.1, §6 and the appendix.
4. Presentation
The abstract is one paragraph; the Introduction's scope paragraph is condensed; repeated qualifications were removed; the contribution list and Discussion opening state the three findings.