acceptodds
Under review as a conference paper at ICLR 2027

Robust Reasoning Benchmark

Abstract

While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving ability remains sensitive to con- text and textual formatting. We introduce the Robust Reasoning Benchmark (RRB), a pipeline of 13 deterministic textual perturbations applied to AIME 2024 and AIME 2025. Evaluating 8 state-of-the-art models, we find that frontier models are largely resilient, with the notable exception of Claude 4.6 Opus, which categori- cally refuses many transformed prompts, while open-weights reasoning models degrade by up to 53% on average and up to 100% on individual perturbations. By prompting models to solve several unperturbed problems in sequence, we identify Intra-Query Attention Dilution: open-weights models from 7B to 120B parameters lose accuracy on later problems. At matched preceding token length, the same mod- els perform better when earlier reasoning is replaced with neutral text. Per-layer attention traces show persistent attention mass on the earlier problems.

Primary area:
datasets and benchmarks
Keywords:
reasoning, AI, ML, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N9kUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Robust Reasoning Benchmark comprising 13 transformations of AIME 2024/2025 problems. It evaluates whether eight LLMs retain mathematical accuracy when the problem's textual representation changes without changing its mathematical content. Its second contribution looks at Intra-Query Attention Dilution. Accuracy decreases when open-weight reasoning models solve multiple independent problems within one query. The authors evaluate reconstruction separately from solving, compare prior reasoning against neutral context of matched token length and inspect attention allocation across three open-weight models.

Critical strengths
  1. A controlled and reproducible robustness benchmark. The transformations have deterministic inverses which allows exact verification that the original mathematical content is preserved. Evaluation across eight models with repeated trajectories and problem-level bootstrap intervals makes the empirical comparisons informative.

  2. The sequential-reasoning experiment shows an interesting failure mode. In Table 2 (page 8) replacing previous reasoning with equal-length neutral text improves target accuracy by 2.9–23.9 percentage points. This goes beyond simply demonstrating that longer inputs reduce performance.

Critical weaknesses
  1. The central attention-dilution experiment has an important confound.

This is the main reason I'd lean toward rejection. The passive-context control matches preceding token length but does not match the position at which the target problem initially appears.

  • In the sequential condition, the target is already present in the initial user prompt.
  • In the passive condition, the target is placed immediately after the neutral passage. So, the observed advantage could partly arise from target recency or retrieval differences, rather than interference from self-generated reasoning. The authors acknowledge this limitation on page 8. It nevertheless affects the interpretation of one of the paper's central contributions.
  1. Limited separation of decoding failures and mathematical reasoning failures.

The paper provides useful decode-only experiments in Section 5.2 which partially address this concern. However, for transformations such as Rail Fence, symbol interleaving and vertical snake encoding reconstructing the original problem is itself a challenging task. So, low solution accuracy may primarily reflect decoding failures rather than a degradation in mathematical reasoning. While the authors compare reconstruction and solving accuracy these measurements come from separate trajectories. Therefore, it remains unclear how frequently models fail to solve a problem despite successfully recovering its original statement. A paired decode-and-solve evaluation or an oracle-decoding control would help distinguish these failure modes and clarify what the benchmark reveals about mathematical reasoning robustness specifically.

Questions for authors

1- Can the authors repeat the passive-context comparison while matching both the preceding token length and the target's position in the context varying only whether the preceding content contains prior mathematical reasoning or neutral text? Does the accuracy gap persist under this control? This would help determine whether the observed degradation is attributable to interference from prior reasoning rather than target recency.

2- Can the authors evaluate decoding and solving within the same trajectories and report solution accuracy conditional on successful reconstruction? An oracle-decoding control would also help quantify how much of the observed degradation stems from decoding difficulty versus downstream mathematical reasoning. This distinction seems particularly important for character-level and spatial transformations where reconstruction itself frequently fails.

Reviewer_vKLgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The authors apply 13 deterministic, invertible text transformations to the 60 problems of AIME 2024 and AIME 2025. The transformations fall into four groups:

    • semantic and lexical: Not Not, Opposites, Wrappers
    • interleaving with a second problem: by line, word or character
    • reversals: of sentences, of words, of characters within words
    • 2D layouts: Snake Horizontal, Snake Vertical, Rectangle Perimeter, Rail Fence For every transformation except Not Not, the prompt states the decoding rule. A programmatic inverse checks every transformed problem.

    Eight models are evaluated. Five are open-weights reasoning models: Qwen3-30B-A3B-Thinking, OpenReasoning-Nemotron 7B and 32B, DSR1-Llama-70B and gpt-oss-120b, with 16 samples per problem. Three are APIs: GPT-5.4, Gemini 3.1 Pro and Claude 4.6 Opus, with 8 samples per problem. The token limit is 32K for all models.

    • GPT-5.4 and Gemini lose about 3 and 7 points on average.
    • The open models lose between about 35 and 53 points. On Interleave (Symbol), Rail Fence and Snake (Vertical) they score 0 to 11%.
    • Opus loses 52 points, which the paper attributes to refusals.

    A decode-only control, run on Qwen3 only, asks the model to reconstruct the statement without solving it. On the four coarsest transformations, reconstruction succeeds 3.1 to 28.5 points more often than solving does. On the character-level and grid transformations, reconstruction itself fails.

  2. Within Intra-Query Attention Dilution, ehre no transformation is applied. Models solve 1 to 4 AIME problems in one prompt, and only the last problem is scored. Accuracy on the last problem falls for every open model; for example, Nemotron-7B goes from 81.9 to 57.9 on AIME 2024.

    • In a passive-context control, the earlier problems and their solutions are replaced by neutral prose of the same token length. This recovers 2.9 to 23.9 points, and five of six intervals exclude zero.
    • Per-layer attention in three models shows that target tokens send 10.4 to 16.5% of their attention to the earlier problems and their reasoning.
    • The paper concludes that context boundaries between completed reasoning steps, for example sub-agents, are worth testing.
Critical strengths
  1. The pertrubation transformations are deterministic, every transformed input is checked with an inverse, and the decoding rule is given in the prompt. So failures cannot come from corrupted inputs or an unstated task.
  2. The decode-only control, with its stub rate and CER threshold, separates two kinds of failure. On the four coarse transformations, Qwen3 reconstructs 73 to 96% of statements but solves only 57 to 80%. On Rail Fence, Snake (Vertical) and Interleave (Symbol), verbatim recovery is 0.0 to 1.5%. So the very large drops at character granularity are decoding failures, and the moderate drops at coarse granularity are not.
  3. The matched-length control shows that the multi-problem decline is not only a matter of context length. Earlier problems and their solutions are replaced with neutral prose of the same preceding token length. This recovers +10.2 to +23.9 points in five of six model and dataset cells. The decline appears in three models and both datasets. It is not caused by truncation, which stays below 2% except for DSR1 at 6.0%. Models are more and more often asked to solve several subtasks in one long trajectory, so a test that separates context content from context length is useful.
Critical weaknesses
  • The benchmark mostly measures character-level decoding. The decode-only control shows that the largest drops come from failed decoding, a known tokenization limit (McCoy et al., 2023).
  • The benchmark has little range: GPT-5.4 is near ceiling, the open models are near zero on several transformations, and each dataset has only 30 problems.
  • The passive-context control does not isolate the effect of earlier reasoning. Besides content, it changes where the target sits, how many problems the model must solve, and whether the prefix is user input or the model's own output. The paper does not report the target's reasoning length per condition, so a compute-allocation explanation cannot be ruled out.
  • The effect is presented as specific to open-weights models, but Opus 4.6 drops from about 99% and 98% to 85% and 79% with four problems, more than gpt-oss-120b.
  • The Opus refusal claim is not documented. There is no detection method, no count and no example, and refusals share a plot segment with failed answers. Results also differ between AIME 2024 and 2025 under the same transformation (Opposites 93% against 56% correct, Wrappers 98% against 67%), which points to run or API effects.
Questions for authors

Q1: In the passive-context control, the target problem sits right before the model answers. In the multi-problem prompt, it sits at the start, far from where the model answers. Can you rerun the control with the target in the same position in both, for example by repeating the target after the earlier solutions? This would show whether the accuracy gap comes from the earlier reasoning or only from the target's position.

Q2: How long is the target's reasoning at Base, at d=1,2,3d = 1, 2, 3, and in the passive arm? Measure it in tokens, from the start of the target to its boxed answer. Does accuracy still fall with dd when you compare samples with similar target-reasoning length?

Q3: How do you identify an Opus refusal: from the API stop reason, a text pattern, or manual reading? Please report, for each cell, refusals separately from wrong or missing answers, show some examples, and state when each dataset was run. Why do Opposites and Wrappers differ so much between AIME 2024 and AIME 2025?

Reviewer_CrN8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Robust Reasoning Benchmark, which evaluates mathematical problem solving under reversible text transformations and in prompts containing multiple problems. The transformations include semantic substitutions, text reordering, and character encodings. The evaluation finds substantial differences in model sensitivity to these transformations and declining accuracy in the sequential setting. Further analyses reveal decoding failures and lower accuracy with prior solutions than with neutral context.

Critical strengths
  • The paper addresses an important issue in model evaluation: whether high mathematical benchmark scores translate into reliable performance under changes in input format and task context.
  • The empirical evaluation is broad across transformation types and model families. It compares open-weight and proprietary reasoning models under a common protocol, providing a systematic assessment of their robustness.
Critical weaknesses
  • The decode-then-solve setup lacks a clear motivation. Artificial encodings add string manipulation demands to mathematical problem solving, while their relevance to practical reasoning remains unclear. The paper does not adequately explain which capability this composite task measures or why it provides a useful measure of reasoning robustness.
  • The transformation and multi-problem settings introduce different sources of difficulty, and their connection is not sufficiently explained. Previous studies, including Batch Prompting [1] and MPE [2], have already examined performance degradation with increasing batch size. The paper does not establish what new insight the combined evaluation provides.
  • The scores do not distinguish decoding failures from mathematical reasoning failures. The reconstruction tests use separate runs, so they cannot establish how often a model solves a problem incorrectly after recovering it correctly. This limits the benchmark's diagnostic value.
  • The evaluation is confined to competition mathematics. This scope limits the evidence for broader claims about reasoning robustness across tasks.

[1] Batch Prompting: Efficient Inference with Large Language Model APIs

[2] Evaluating LLMs with Multiple Problems at once

Questions for authors
  • Can the authors evaluate decoding and solving within the same trajectory to clarify how much of the accuracy loss comes from each stage?
  • How does the difficulty of the preceding problems affect accuracy on the final problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.