- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the Robust Reasoning Benchmark comprising 13 transformations of AIME 2024/2025 problems. It evaluates whether eight LLMs retain mathematical accuracy when the problem's textual representation changes without changing its mathematical content. Its second contribution looks at Intra-Query Attention Dilution. Accuracy decreases when open-weight reasoning models solve multiple independent problems within one query. The authors evaluate reconstruction separately from solving, compare prior reasoning against neutral context of matched token length and inspect attention allocation across three open-weight models.
Critical strengths
-
A controlled and reproducible robustness benchmark. The transformations have deterministic inverses which allows exact verification that the original mathematical content is preserved. Evaluation across eight models with repeated trajectories and problem-level bootstrap intervals makes the empirical comparisons informative.
-
The sequential-reasoning experiment shows an interesting failure mode. In Table 2 (page 8) replacing previous reasoning with equal-length neutral text improves target accuracy by 2.9–23.9 percentage points. This goes beyond simply demonstrating that longer inputs reduce performance.
Critical weaknesses
- The central attention-dilution experiment has an important confound.
This is the main reason I'd lean toward rejection. The passive-context control matches preceding token length but does not match the position at which the target problem initially appears.
- In the sequential condition, the target is already present in the initial user prompt.
- In the passive condition, the target is placed immediately after the neutral passage. So, the observed advantage could partly arise from target recency or retrieval differences, rather than interference from self-generated reasoning. The authors acknowledge this limitation on page 8. It nevertheless affects the interpretation of one of the paper's central contributions.
- Limited separation of decoding failures and mathematical reasoning failures.
The paper provides useful decode-only experiments in Section 5.2 which partially address this concern. However, for transformations such as Rail Fence, symbol interleaving and vertical snake encoding reconstructing the original problem is itself a challenging task. So, low solution accuracy may primarily reflect decoding failures rather than a degradation in mathematical reasoning. While the authors compare reconstruction and solving accuracy these measurements come from separate trajectories. Therefore, it remains unclear how frequently models fail to solve a problem despite successfully recovering its original statement. A paired decode-and-solve evaluation or an oracle-decoding control would help distinguish these failure modes and clarify what the benchmark reveals about mathematical reasoning robustness specifically.
Questions for authors
1- Can the authors repeat the passive-context comparison while matching both the preceding token length and the target's position in the context varying only whether the preceding content contains prior mathematical reasoning or neutral text? Does the accuracy gap persist under this control? This would help determine whether the observed degradation is attributable to interference from prior reasoning rather than target recency.
2- Can the authors evaluate decoding and solving within the same trajectories and report solution accuracy conditional on successful reconstruction? An oracle-decoding control would also help quantify how much of the observed degradation stems from decoding difficulty versus downstream mathematical reasoning. This distinction seems particularly important for character-level and spatial transformations where reconstruction itself frequently fails.