- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors base their paper on a very simple question: If the phrasing of a math problem is changed, such as a different name, an irrelevant sentence added, etc., will language models solve the original problem correctly? They go on to predict from the original problem statement if a rewrite will break an LLM's ability to solve the original problem.
To answer this in a way presentable to ICLR, they have added a lot of dense causal framing and notation to bury the simple and interesting idea.
Two new datasets are introduced in which an LLM is used to rewrite the original problem statements, producing corrupted twins from two standard benchmark datasets.
Overall, the contributions are limited in the context of advancement in frontier model technology, and the experiments are not robust enough to show any new (or exciting/surprising/insightful) results.
Critical strengths
The premise is good. I like the exploration on whether small rewrites can break the math reasoning abilities of small language models.
Critical weaknesses
Hardness is controlled only through a proxy such as reasoning steps, level of competition, and not measured relative to the model that is used for solving. The metrics are all defined to give some reportable metric without any sense of why that metric is useful (CCD)
The problem statement is about fragility with language models, so just doing runs with small models, without showing performance or transferability to larger models as solvers is not useful.
Questions for authors
What is the main insight for practitioners from your results? Is your result statistically robust and transferable to large models?
4 comments
"[...] just doing runs with small models, without showing performance or transferability to larger models as solvers is not useful."
"Overall, the contributions are limited in the context of advancement in frontier model technology, and the experiments are not robust enough [...]."
"Is your result statistically robust and transferable to large models?"
The paper does not use only small solvers. Table 4 measures CCD with gpt-4o-mini, Qwen3.7-plus and DeepSeek-v4-pro and reports the full train-on-one, evaluate-on-another matrix. Transfer is partial, as the paper states, but positive in every cell, and on GSM-IC-Adv a predictor trained on gpt-4o-mini beats the in-solver one for Qwen3.7-plus (0.355 vs. 0.299) and DeepSeek-v4-pro (0.411 vs. 0.389).
We now add Qwen2.5-72B-Instruct and normalize every score by the ceiling. Frail-π is trained on gpt-4o-mini labels and scored against each larger solver's own measured CCD on the same held-out problems; for Qwen2.5-72B we also train on that solver's own labels (5 seeds). Agreement is the noise-corrected correlation of each solver's CCD with gpt-4o-mini's.
| Solver | Solvable problems | Mean CCD | Fragile share (CCD > 0.05) | CCD agreement with gpt-4o-mini | Frail-π trained on gpt-4o-mini | Frail-π trained on own labels |
|---|---|---|---|---|---|---|
| gpt-4o-mini | 1,241 | 0.126 | 42% | — | 0.61 | (same) |
| Qwen3.7-plus | 1,293 | 0.035 | 16% | 0.59 | 0.41 (declared complexity 0.19) | — |
| Qwen2.5-72B-Instruct | 1,262 | 0.066 | 27% | 0.68 | 0.46 (declared complexity 0.15) | 0.53 (declared complexity 0.16; best baseline 0.35) |
Fragility is partly shared across solvers and partly solver-specific, which is why CCD is defined per solver and transfer is tested rather than assumed. Even so, a predictor trained only on gpt-4o-mini ranks the larger solvers' fragility at more than two and three times the level of declared complexity, and trained on the 72B model's own labels it beats every baseline on all 5 seeds. On frontier relevance, larger solvers are more robust, not robust: 16% of the problems Qwen3.7-plus solves are still fragile and 7% break on at least one rewrite in five (27% and 13% for Qwen2.5-72B). As frontier models near the accuracy ceiling, accuracy alone cannot separate robust from fragile successes, so this matters more with scale.
On statistical robustness, the paper already reports two suites, ceiling-normalized Spearman, an identity-twin noise control, three twin generators (Figure 3, Table 3) and three solvers (Table 4), with Frail-π ahead of every baseline on both suites over five seeds (Table 2).
"What is the main insight for practitioners from your results?"
Correct answers are not equally trustworthy: 40% of the problems gpt-4o-mini always solves are fragile. Difficulty does not reveal which ones, since no hardness measure exceeds 0.39 of the ceiling. Larger models reduce this fragility but do not remove it (16% to 27% fragile). And the fragile items can be found cheaply: from the original problem text alone, with no rewrite run, Frail-π reaches 0.61 of the ceiling (Table 2), which makes robustness audits and curated robustness test sets affordable.
We hope this addresses your concerns and ask you to reconsider the soundness and overall scores.
Clarifications help, but not enough
I went through the authors' responses to all reviews. The main concern about practical insights is not sufficiently resolved. The responses to the other reviewers also indicate that the main results are not statistically significant (e.g., per rewrite, objective gains). I did not understand the explanation for why causal framing was chosen. The rebuttal text is verbose, and I couldn't understand it even after reading it multiple times; sorry. Thanks for clarifying how the twin is produced, but that does not make the contribution acceptable. The bigger models added now need to be reviewed and synthesized with everything else, and the paper should be rewritten, I think. As such, I maintain my original scores.
Thank you for the follow-up, and sorry the rebuttal was long.
We want to respectfully push back on your point about significance of the results One correction as our main results are significant. Frail-π beats every baseline on all 10 seeds (0.61 vs. at most 0.38, p = 0.002). The only non-significant number is a small extra gain (+0.02) from one training choice, and none of our claims rests on it.
You asked whether the results hold for larger models, so we ran that test. They do. Larger models are more robust, but 16–27% of the problems they solve are still fragile, and our predictor trained on gpt-4o-mini still ranks their fragility far better than difficulty does. We’d really appreciate it if this were taken into account now, since it answers your original question.
Why causal framing, simply: a rewrite can only break a problem the model already solves, so we only count those problems. If you average over all problems, nothing can predict the result.
The practical takeaway: many correct answers are fragile (about 40%), difficulty doesn’t tell you which ones, and we can find them from the problem text alone.
Thank you for the review however we respectfully disagree with several conclusions that result already in the paper address, and add new runs where you ask for more.
We respectfully disagree with this characterisation, which attributes a motive rather than identifying a flaw. The causal framing was not added to make the work presentable and it does not bury the idea: it is what defines the quantity we measure and makes it measurable, so removing it would remove results, not decoration. It selects the target: the natural first choice, the average effect of a rewrite over all problems, nets breaks against repairs and no method ranks it (best ρ_S = 0.162, Appendix C.3), while CCD on the solvable stratum is rankable (Table 2). It identifies complexity as an effect modifier rather than a confounder, which is why CRD residualizes complexity instead of controlling for it, and why Table 3 can show the result is orthogonal to the declared proxy. And its paired design makes each problem its own control, so no confounding adjustment is needed, with identity twins measuring the 1.1% re-solving floor against which every break rate is read. Where we agree is presentation: Sections 1 to 3 introduce the notation before the idea. We will edit that to lead with the simple idea and a worked example and move the potential-outcomes notation and principal stratification to the appendix, while the results this framing produces stay in the main text.
To clarify where a language model enters the pipeline: it is never used to write a twin. Every twin is produced by deterministic, seeded code from three fixed operator families: renaming entities (consistent substitution from a fixed gender-neutral pool), reordering sentences (a hash-seeded shuffle of the setup, question sentence last), and inserting a setup sentence copied verbatim from a different GSM8K problem. The perturbation text is therefore reproducible from the released code and the answer key is preserved by construction. A language model is used in two places, both of them downstream of the perturbation: as the No-Op judge in the audit, and as the solver whose outcomes define the labels. Before any label is measured every pair passes an LLM No-Op adjudicator, renaming and reordering checks, and deterministic collision filters, and any contradiction verdict or entity, gold-answer or multi-question collision drops the pair (Table 12), we will add this clarification and rephrase LLM authored twins sentence accordingly to reflect this and happy to share the anonymized dataset if is more convincing for you to check that.
CCD is solver-relative by construction: which problems count and which rewrites break them come from this solver's own outcomes (Definition 2), and Table 2 already includes the solver's measured base pass rate, which ranks CCD negatively. The declared proxy enters the estimates only through CRD, where it must be external to avoid circularity (Section 4). We have now added every hardness measure we can compute, four of them measured on the solver itself and one across three solvers, since this is the form of the question that matters; the full table is in our response to Reviewer xUAv. The strongest single measure, the solver's answer entropy on the base problem, reaches 0.39 of the ceiling, and all seven combined reach 0.36, while Frail-π reaches 0.61 and keeps 0.47 after all of them are partialled out. On the 1,045 problems gpt-4o-mini answers correctly in every base sample, across all five runs and a separate batch, its correctness-based hardness measures are constant, yet 21% still break on at least one rewrite in five.
CCD is useful because it answers what accuracy cannot: if the model solved this problem, will it still solve it when phrased differently? gpt-4o-mini solves 1,212 GSM-IC-Adv problems in every run, and accuracy treats them as equally solid, yet 40% break on more than 5% of harmless rewrites. This supports three uses. Robustness audits can flag fragile correct answers before deployment without solving every rewrite. Benchmark curation can select the problems most likely to break: Frail-π ranks CCD at 0.61 of the ceiling against 0.21 for declared complexity (Table 2), and at a 5% budget it doubles random recall of fragile items, matching an oracle (Appendix D.3). Model diagnosis can use CRD to find problems more fragile than their difficulty predicts.
Continued in next comment ....