acceptodds
Under review as a conference paper at ICLR 2027

Fragility is not Reducible to Hardness: Conditional Causal Difficulty for LLMs

Abstract

Large language models show large, seemingly arbitrary performance drops when a task is rewritten in ways that leave its requirements untouched. We recast task difficulty as a causal quantity: the treatment effect of meaning-preserving perturbations on a model. A meaning-preserving rewrite can only be observed to hurt a task the model can already solve, so we place the estimand on the principal stratum where the degradation occurs: the conditional causal difficulty (CCD), the probability that a randomly drawn twin breaks a solvable task. Because harder tasks tend to be more phrasing-sensitive, complexity enters as an effect modifier rather than a confounder, and we residualize it out to obtain complexity-residualized difficulty (CRD); zero for a task exactly as fragile as its complexity predicts, positive for anomalously fragile ones. Current benchmarks do not supply the data CCD requires, so we introduce GSM-IC-Adv and MATH-Adv, augmenting GSM-IC and MATH with twins built from perturbation families, raising fragility on GSM-IC-Adv by over the original GSM-IC () while keeping the answer key unchanged. Our predictor learns the per-perturbation break probability and reaches of the suite reliability ceiling on GSM-IC-Adv and on MATH-Adv, against / for a surface-feature baseline and / for supervised baselines such as VAP3, and doubles random recall of fragile items under a curation budget. Additionally, cross-fitted residualization leaves CRD uncorrelated with the declared proxy in practice.

Primary area:
foundation or frontier models, including LLMs
Keywords:
conditional causal difficulty, benchmark, meaning-preserving perturbations, task difficulty prediction, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9TV2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors base their paper on a very simple question: If the phrasing of a math problem is changed, such as a different name, an irrelevant sentence added, etc., will language models solve the original problem correctly? They go on to predict from the original problem statement if a rewrite will break an LLM's ability to solve the original problem.

To answer this in a way presentable to ICLR, they have added a lot of dense causal framing and notation to bury the simple and interesting idea.

Two new datasets are introduced in which an LLM is used to rewrite the original problem statements, producing corrupted twins from two standard benchmark datasets.

Overall, the contributions are limited in the context of advancement in frontier model technology, and the experiments are not robust enough to show any new (or exciting/surprising/insightful) results.

Critical strengths

The premise is good. I like the exploration on whether small rewrites can break the math reasoning abilities of small language models.

Critical weaknesses

Hardness is controlled only through a proxy such as reasoning steps, level of competition, and not measured relative to the model that is used for solving. The metrics are all defined to give some reportable metric without any sense of why that metric is useful (CCD)

The problem statement is about fragility with language models, so just doing runs with small models, without showing performance or transferability to larger models as solvers is not useful.

Questions for authors

What is the main insight for practitioners from your results? Is your result statistically robust and transferable to large models?

4 comments

Thank you for the review however we respectfully disagree with several conclusions that result already in the paper address, and add new runs where you ask for more.

"To answer this in a way presentable to ICLR, they have added a lot of dense causal framing and notation to bury the simple and interesting idea."

We respectfully disagree with this characterisation, which attributes a motive rather than identifying a flaw. The causal framing was not added to make the work presentable and it does not bury the idea: it is what defines the quantity we measure and makes it measurable, so removing it would remove results, not decoration. It selects the target: the natural first choice, the average effect of a rewrite over all problems, nets breaks against repairs and no method ranks it (best ρ_S = 0.162, Appendix C.3), while CCD on the solvable stratum is rankable (Table 2). It identifies complexity as an effect modifier rather than a confounder, which is why CRD residualizes complexity instead of controlling for it, and why Table 3 can show the result is orthogonal to the declared proxy. And its paired design makes each problem its own control, so no confounding adjustment is needed, with identity twins measuring the 1.1% re-solving floor against which every break rate is read. Where we agree is presentation: Sections 1 to 3 introduce the notation before the idea. We will edit that to lead with the simple idea and a worked example and move the potential-outcomes notation and principal stratification to the appendix, while the results this framing produces stay in the main text.

"[...] an LLM is used to rewrite the original problem statements, producing corrupted twins [...]."

To clarify where a language model enters the pipeline: it is never used to write a twin. Every twin is produced by deterministic, seeded code from three fixed operator families: renaming entities (consistent substitution from a fixed gender-neutral pool), reordering sentences (a hash-seeded shuffle of the setup, question sentence last), and inserting a setup sentence copied verbatim from a different GSM8K problem. The perturbation text is therefore reproducible from the released code and the answer key is preserved by construction. A language model is used in two places, both of them downstream of the perturbation: as the No-Op judge in the audit, and as the solver whose outcomes define the labels. Before any label is measured every pair passes an LLM No-Op adjudicator, renaming and reordering checks, and deterministic collision filters, and any contradiction verdict or entity, gold-answer or multi-question collision drops the pair (Table 12), we will add this clarification and rephrase LLM authored twins sentence accordingly to reflect this and happy to share the anonymized dataset if is more convincing for you to check that.

"Hardness is controlled only through a proxy [...] and not measured relative to the model that is used for solving. The metrics are all defined to give some reportable metric without any sense of why that metric is useful (CCD)"

CCD is solver-relative by construction: which problems count and which rewrites break them come from this solver's own outcomes (Definition 2), and Table 2 already includes the solver's measured base pass rate, which ranks CCD negatively. The declared proxy enters the estimates only through CRD, where it must be external to avoid circularity (Section 4). We have now added every hardness measure we can compute, four of them measured on the solver itself and one across three solvers, since this is the form of the question that matters; the full table is in our response to Reviewer xUAv. The strongest single measure, the solver's answer entropy on the base problem, reaches 0.39 of the ceiling, and all seven combined reach 0.36, while Frail-π reaches 0.61 and keeps 0.47 after all of them are partialled out. On the 1,045 problems gpt-4o-mini answers correctly in every base sample, across all five runs and a separate batch, its correctness-based hardness measures are constant, yet 21% still break on at least one rewrite in five.

CCD is useful because it answers what accuracy cannot: if the model solved this problem, will it still solve it when phrased differently? gpt-4o-mini solves 1,212 GSM-IC-Adv problems in every run, and accuracy treats them as equally solid, yet 40% break on more than 5% of harmless rewrites. This supports three uses. Robustness audits can flag fragile correct answers before deployment without solving every rewrite. Benchmark curation can select the problems most likely to break: Frail-π ranks CCD at 0.61 of the ceiling against 0.21 for declared complexity (Table 2), and at a 5% budget it doubles random recall of fragile items, matching an oracle (Appendix D.3). Model diagnosis can use CRD to find problems more fragile than their difficulty predicts.

Continued in next comment ....

"[...] just doing runs with small models, without showing performance or transferability to larger models as solvers is not useful."

"Overall, the contributions are limited in the context of advancement in frontier model technology, and the experiments are not robust enough [...]."

"Is your result statistically robust and transferable to large models?"

The paper does not use only small solvers. Table 4 measures CCD with gpt-4o-mini, Qwen3.7-plus and DeepSeek-v4-pro and reports the full train-on-one, evaluate-on-another matrix. Transfer is partial, as the paper states, but positive in every cell, and on GSM-IC-Adv a predictor trained on gpt-4o-mini beats the in-solver one for Qwen3.7-plus (0.355 vs. 0.299) and DeepSeek-v4-pro (0.411 vs. 0.389).

We now add Qwen2.5-72B-Instruct and normalize every score by the ceiling. Frail-π is trained on gpt-4o-mini labels and scored against each larger solver's own measured CCD on the same held-out problems; for Qwen2.5-72B we also train on that solver's own labels (5 seeds). Agreement is the noise-corrected correlation of each solver's CCD with gpt-4o-mini's.

SolverSolvable problemsMean CCDFragile share (CCD > 0.05)CCD agreement with gpt-4o-miniFrail-π trained on gpt-4o-miniFrail-π trained on own labels
gpt-4o-mini1,2410.12642%—0.61(same)
Qwen3.7-plus1,2930.03516%0.590.41 (declared complexity 0.19)—
Qwen2.5-72B-Instruct1,2620.06627%0.680.46 (declared complexity 0.15)0.53 (declared complexity 0.16; best baseline 0.35)

Fragility is partly shared across solvers and partly solver-specific, which is why CCD is defined per solver and transfer is tested rather than assumed. Even so, a predictor trained only on gpt-4o-mini ranks the larger solvers' fragility at more than two and three times the level of declared complexity, and trained on the 72B model's own labels it beats every baseline on all 5 seeds. On frontier relevance, larger solvers are more robust, not robust: 16% of the problems Qwen3.7-plus solves are still fragile and 7% break on at least one rewrite in five (27% and 13% for Qwen2.5-72B). As frontier models near the accuracy ceiling, accuracy alone cannot separate robust from fragile successes, so this matters more with scale.

On statistical robustness, the paper already reports two suites, ceiling-normalized Spearman, an identity-twin noise control, three twin generators (Figure 3, Table 3) and three solvers (Table 4), with Frail-π ahead of every baseline on both suites over five seeds (Table 2).

"What is the main insight for practitioners from your results?"

Correct answers are not equally trustworthy: 40% of the problems gpt-4o-mini always solves are fragile. Difficulty does not reveal which ones, since no hardness measure exceeds 0.39 of the ceiling. Larger models reduce this fragility but do not remove it (16% to 27% fragile). And the fragile items can be found cheaply: from the original problem text alone, with no rewrite run, Frail-π reaches 0.61 of the ceiling (Table 2), which makes robustness audits and curated robustness test sets affordable.

We hope this addresses your concerns and ask you to reconsider the soundness and overall scores.

Reviewer_9TV2OpenReview ↗

Clarifications help, but not enough

I went through the authors' responses to all reviews. The main concern about practical insights is not sufficiently resolved. The responses to the other reviewers also indicate that the main results are not statistically significant (e.g., per rewrite, objective gains). I did not understand the explanation for why causal framing was chosen. The rebuttal text is verbose, and I couldn't understand it even after reading it multiple times; sorry. Thanks for clarifying how the twin is produced, but that does not make the contribution acceptable. The bigger models added now need to be reviewed and synthesized with everything else, and the paper should be rewritten, I think. As such, I maintain my original scores.

Thank you for the follow-up, and sorry the rebuttal was long.

We want to respectfully push back on your point about significance of the results One correction as our main results are significant. Frail-π beats every baseline on all 10 seeds (0.61 vs. at most 0.38, p = 0.002). The only non-significant number is a small extra gain (+0.02) from one training choice, and none of our claims rests on it.

You asked whether the results hold for larger models, so we ran that test. They do. Larger models are more robust, but 16–27% of the problems they solve are still fragile, and our predictor trained on gpt-4o-mini still ranks their fragility far better than difficulty does. We’d really appreciate it if this were taken into account now, since it answers your original question.

Why causal framing, simply: a rewrite can only break a problem the model already solves, so we only count those problems. If you average over all problems, nothing can predict the result.

The practical takeaway: many correct answers are fragile (about 40%), difficulty doesn’t tell you which ones, and we can find them from the problem text alone.

Reviewer_xUAvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces two measures of LLM fragility under "meaning-preserving rewrites". CCD measures the rewrite failure rate for problems the solver LLM initially answers correctly. CRD subtracts the expected CCD at a given complexity level. The authors propose Frail-π\pi to predict CCD from a problem and derive CRD through residualization. Experiments on two constructed math perturbation sets show that the predictor ranks problem fragility better than the evaluated baselines.

Critical strengths

Frail-π\pi provides a practical way to predict CCD from the original problem without running the solver LLM on many rewritten versions during inference. It achieves higher ranking correlations than baselines on both math perturbation sets, showing promise for identifying fragile problems.

Critical weaknesses
  1. The CCD definition and the reported evaluation seem inconsistent. Definition 2 indicates that CCD is conditioned on Y(J)=1Y(J)=1. However, Table 2 reports negative correlations between Y(J)Y(J) and CCD. Since CCD only considers tasks with Y(J)=1Y(J)=1, the correlation should be undefined. A clarification of whether the evaluation includes initially incorrect problems or uses a different definition is necessary.
  2. The main conceptual contribution is unclear. CCD and CRD are reasonable measures, but the paper does not clearly show what the causal framework adds beyond the definitions. The experiments mainly show that Frail-π\pi can predict the proposed scores, but what do these scores reveal about LLM fragility? In addition, how can we understand the title "Fragility is not Reducible to Hardness" through the proposed measures and experiments? To me, the experiments support a prediction method more clearly than the story of differentiating fragility from hardness, not to mention that the definition of "hardness" comes from the dataset itself or reasoning steps, which can only partially reflect the intrinsic "hardness" of a problem.
Questions for authors
  1. Clarification of Table 2. What score is used in the "Measured base correctness Y(J)Y(J)" row, and which problems are included in this evaluation? If all evaluated problems satisfy Y(J)=1Y(J)=1, as required by Definition 2, this score should be constant and its Spearman correlation with CCD should be undefined. Please explain how the reported correlations were obtained.
  2. Clarification of the main contribution. Beyond predicting the proposed scores, what new insight does the framework provide about LLM fragility? In particular, which results support the claim that fragility is not reducible to hardness, given that the complexity measures are proxies and CRD’s orthogonality to these proxies?
  3. Clarification of the overall approach. Could the authors explain in plain human language how CCD, CRD, and Frail-π\pi fit together to answer the main research question? The abstract and introduction introduce many terms before making this connection clear, and the later sections do not resolve it. This makes it difficult to assess the core contribution and how the experiments support it.
4 comments

Thank you for the questions. Your central soundness concern turns on how Y(J) = 1 is measured, which we clarify first.

"The CCD definition and the reported evaluation seem inconsistent. [...] Table 2 reports negative correlations between Ȳ(J) and CCD. Since CCD only considers tasks with Y(J)=1, the correlation should be undefined. [...]"

"Clarification of Table 2. What score is used in the 'Measured base correctness Ȳ(J)' row, and which problems are included in this evaluation? [...] Please explain how the reported correlations were obtained."

There is no inconsistency. Y(J) = 1 is not observed directly; we observe how often the solver solves the base. Every run that builds twins also re-solves the base, so Ȳ(J) is the fraction of correct base samples over all runs. As Section 5 states, a problem enters the solvable stratum when Ȳ(J) ≥ 0.5, "the operational form of Y(J)=1 in every result below", and a break counts only in runs where the base was solved in that same run. Table 2 includes only these problems, as Definition 2 requires, so no initially incorrect problem enters and no other definition is used. The score in that row is Ȳ(J) itself, which ranges from 0.5 to 1 on the stratum, so its Spearman correlation with CCD is defined. The row also makes the paper's point: on GSM-IC-Adv, 1,212 of 1,241 stratum problems have Ȳ(J) = 1 and are equally easy by the solver's own measure, yet 40% of them break on more than 5% of harmless rewrites, against about 3% expected from re-solving noise. The revision places this definition next to Definition 2.

"[...] the paper does not clearly show what the causal framework adds beyond the definitions. [...] what do these scores reveal about LLM fragility? In addition, how can we understand the title "Fragility is not Reducible to Hardness" [...]? To me, the experiments support a prediction method more clearly than the story of differentiating fragility from hardness, not to mention that the definition of "hardness" comes from the dataset itself or reasoning steps [...]."

"[...] what new insight does the framework provide about LLM fragility? In particular, which results support the claim that fragility is not reducible to hardness, given that the complexity measures are proxies [...]?"

The framework does three things the definitions alone do not. It selects the target: the average effect of a rewrite over all problems nets breaks against repairs and no method ranks it (best ρ_S = 0.162, Appendix C.3), while CCD on the solvable stratum is rankable (Table 2). It makes the target solver-relative, which is why Table 4 tests transfer across solvers. And it shows complexity is an effect modifier, to be residualized rather than controlled for, which gives CRD (Table 3).

We agree a declared proxy captures hardness only in part, as Proposition 1 itself states that CRD is orthogonal only to the declared proxy. We disagree, however, that the experiments support only a prediction method, because the title's claim does not rest on the proxy or on CRD but on hardness measured on the solver. The paper already reports one such measure, the solver's own base pass rate, which ranks CCD negatively (Table 2). In new runs we computed every hardness measure we can, oriented so that harder is larger, including difficulty across three solvers as a measure not tied to one solver:

Hardness measure (harder = larger)Spearman / ceiling
Gold-solution step count (GSM8K)0.17
Declared complexity score0.21
Base correctness over five runs (solver-measured)0.09
Pass rate over a separate batch of base samples (solver-measured)0.37
Answer entropy over that batch (solver-measured)0.39
Mean solution length over that batch (solver-measured)0.18
Difficulty across gpt-4o-mini, Qwen3.7-plus, Qwen2.5-72B0.13
All of the above, one linear model fit on training folds0.36 ± 0.03
Frail-π, raw / after partialling out all of the above0.61 ± 0.05 / 0.47 ± 0.07

Continued in next comment ....

If fragility were reducible to hardness, the combined predictor would approach the ceiling, Frail-π would collapse once hardness is controlled, and CCD would be nearly constant within a hardness band. None of this holds. The combined predictor reaches 0.36, Frail-π keeps 0.47 after all seven measures are partialled out and within every band of hardness CCD still varies and remains predictable:

Problems, grouped by combined hardnessShare breaking on ≥ 1 rewrite in 5Frail-π, Spearman / ceiling
Easiest tercile16%0.47
Middle tercile22%0.61
Hardest tercile40%0.58
Solved in every base sample, across all five runs and the separate batch (1,045 problems; base correctness, pass rate and answer entropy constant)21%0.54

The last row is the most direct test: the solver never fails on these problems, so its correctness-based hardness measures do not vary at all, yet one in five is still fragile and Frail-π still ranks them. So the scores show that fragility is common among correct answers, is not explained by hardness, and can be predicted before any rewrite is run, with Frail-π ahead of every baseline in the same re-run on all 10 seeds (0.61 against at most 0.38, Wilcoxon p = 0.002). It also transfers across solvers: trained only on gpt-4o-mini, Frail-π ranks the CCD of Qwen3.7-plus at 0.41 and of Qwen2.5-72B at 0.46 of the ceiling.

"Could the authors explain in plain human language how CCD, CRD, and Frail-π fit together to answer the main research question? [...]"

A model solves a math problem. We rewrite it so it asks the same thing with the same answer, and check whether the model still solves it. The question is whether such failures are just a symptom of hard problems. CCD measures them: the share of harmless rewrites that break a problem the model solves. Comparing CCD with hardness measured on the same solver answers the question, since hardness explains little of CCD; that is the title. CRD keeps the part of CCD that declared difficulty does not predict, flagging unexpectedly fragile problems. Frail-π predicts CCD from the original problem without running any rewrite, which makes the measure usable at scale. GSM-IC-Adv and MATH-ADV supply the data, since existing suites either change the answer or break too few problems to rank (Table 1). The revised introduction opens with this paragraph.

We would be grateful if you would reconsider the soundness and overall scores in light of these clarifications.

Thank you for the clarifications and the new experiments.

The explanation of the solvability criterion and the measured base correctness resolves my concern about the correlation in Table 2. The new comparison with solver-based hardness measures also helps. The analysis of the problems that the solver answers correctly in every base sample gives direct support to the claim that the measured hardness indicators explain only part of fragility.

My remaining concern is the extent of the contribution. Predicting fragility from the original problem has practical value, and it can help to prioritize robustness testing. I understand that the causal formulation helps specify the evaluation target. However, its additional conceptual contribution beyond defining conditional rewrite failure and residualizing it against a complexity measure remains unclear to me. In particular, CRD’s orthogonality follows largely from its construction. The new ablation in the response to Reviewer eDFm shows that most of the predictive gain comes from encoder fine-tuning. The per-rewrite objective adds only about 0.02 of the ceiling, and this gain is not statistically significant (p = 0.08 and 0.38). The paper lists this objective as a main contribution, and the ablation gives it little support.

The response reduces my concerns about the measurement and strengthens the empirical evidence. I am still not convinced that the conceptual and methodological contributions are large enough for acceptance. The paper also needs to separate more clearly the properties that hold by definition from the findings that come from experiments. For now, I keep my rating of 2. I remain open to further discussion.

Thank you for the follow-up, and we’re glad the Table 2 concern is cleared up.

The per-rewrite objective gain was never the point of Contribution 2. The point is that you can predict which problems are fragile from the text alone, before running any rewrite. That holds in every cell of the ablation (the weakest is 0.49, VAP3 is 0.38), and Frail-π beats every baseline on all 10 seeds (p = 0.002). We use the per-rewrite form because it gives CCD, the dose-response parameters and CRD from one model. A problem-level model can’t give you those. So judging the whole contribution by this 0.02 doesn’t seem fair to us.

On orthogonality: we agree it comes from the construction, and we’ll say so clearly. But our main claim doesn’t depend on it. We tested hardness measured on the solver itself, and none of those measures was used to build CCD. All of them together reach only 0.36, and one in five problems the solver never fails is still fragile. The construction doesn’t force that result; it comes from the data.

We also think the two new benchmarks are part of the contribution, since no existing suite could be used for this. Since the soundness concern is resolved, we’d kindly ask you to at least reconsider the soundness score. We’ll also add a table that separates what holds by definition from what we found in experiments.

Reviewer_eDFmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that conventional notions of task difficulty fail to capture an important source of LLM unreliability where a problem may be solvable in its original wording yet become unsolvable after a meaning-preserving rewrite. The authors formalize this behavior through conditional causal difficulty (CCD), defined as the probability that a randomly drawn perturbation breaks a task that the model originally solves. The authors formalize difficulty as perturbation-induced degradation, a predictor that estimates this quantity and benchmark datasets with sufficient variation in task fragility.

Critical strengths
  1. The paper discusses a meaningful limitation of standard LLM evaluation where accuracy does not necessarily reflct performance on semantically equivalent formulations and provide a useful conceptual distinction between unconditional causal difficulty and CCD.
  2. The authors are honest about what the causal design does and doesn’t contribute with reasonable evidences of per-perturbation supervision from encoder fine-tuning, evaluates multiple random seeds, and reports reliability target ceiling.
  3. The draft have structural collision filters, independent LLM adjudication, and a human-reviewed sample which are important because a perturbation that changes the actual problem is not a valid example.
Critical weaknesses
  1. The draft is too dense and very difficult to follow. Too much jargons and inline equations which make it almost unreadable. It requires a lot of work.
  2. Do the main CCD rankings remain stable under deterministic decoding or a different number of solver samples?
  3. The ablations help establish that the complete recipe works, but the paper's main empirical gain is difficult to align one key methodological component.
  4. A problem may be fragile to distractor insertion but robust to sentence reordering. Aggregating these outcomes into a single score can obscure such distinctions.
Questions for authors

See above.

1 comment

Thank you for the review and for recognising the distinction between unconditional and conditional causal difficulty. We answer each point below, and both technical questions with new runs.

"The draft is too dense and very difficult to follow. Too much jargons and inline equations [...]."

We agree. The paper focused on stating the formalism before the idea. We will edit it to open with one worked example in plain words and moves some of the equations to the appendix along with the potential-outcomes notation and the principal-stratification discussion.

"Do the main CCD rankings remain stable under deterministic decoding or a different number of solver samples?"

The paper's labels use temperature 0.2 with five samples per prompt that scored on the modal answer (Section 5). To address your concern further, we tested the decoding in the deterministic way on the GSM-IC-Adv with 10 samples per prompt and majority vote at three temperatures. Agreement is the noise-corrected Spearman correlation of CCD and the last column scores Frail-π.

Labels used to measure CCDCCD agreement with labelsOverlap of solvable set (Jaccard)Frail-π
Temperature 0, 1 sample (reference)——0.611 ± 0.044
10 samples, temperature 0.2, majority vote0.970.970.606 ± 0.047
10 samples, temperature 0.5, majority vote0.970.980.602 ± 0.047
10 samples, temperature 0.7, majority vote0.970.980.597 ± 0.052

As you can see, neither decoding nor sample count changes which problems are solvable or which are fragile and the paper's setting (temperature 0.2) lies inside the range tested. The paper also measures how much of a break could be pure sampling noise with the same solver and decoding settings, and counts how often the model now fails even though nothing was changed (Appendix D.2, Table 9). On GSM-IC-Adv this happens in only 1.1% of re-solves, whereas real rewrites at the highest dose (m = 9) break solved problems 12.6% of the time on average, more than ten times as often. The breaks we measure therefore come from the rewrites, not from randomness in decoding.

"[...] the paper's main empirical gain is difficult to align one key methodological component."

We showed in Section 5.1 attributes the gain in two steps. With the objective fixed, a fine-tuned encoder beats every frozen representation on both suites (Table 6), and with TF-IDF+SVD features fixed, the per-rewrite objective (the paper's per-observation objective) adds 0.078 on GSM-IC-Adv and nothing on MATH-ADV (Tables 2 and 6). You are right that one cell was missing and we added a fine-tuned encoder trained on problem-level rates with the full 2×2 with identical code, head, splits and 10 seeds with varying only these two factors.

Encoder (Spearman / ceiling)Problem-level objectivePer-rewrite objective
Frozen0.488 ± 0.0570.503 ± 0.048
Fine-tuned0.594 ± 0.0600.611 ± 0.044

As you can see here, the gain comes from fine-tuning the encoder, which adds 0.11 of the ceiling under either objective (10/10 and 9/10 seeds, p = 0.002 and 0.004), like Section 5.1 already concludes (“fine-tuning accounts for much of the lift”). The per-rewrite objective adds a further 0.02 or so under either encoder, consistent in sign (7/10 seeds) but not significant (p = 0.08 and 0.38), and the revision states it that way. Its role is not this accuracy gain: it lets one model output CCD, the dose-response parameters and CRD together, which a problem-level model cannot. All four cells stay well above the strongest baseline in the same re-run (VAP3, 0.38), so the main claim, that fragility is predictable from the problem text alone, does not depend on the objective.

"A problem may be fragile to distractor insertion but robust to sentence reordering. Aggregating these outcomes into a single score can obscure such distinctions."

We agree the families carry different information and the measure already separates them. Definition 2 takes the distribution over operators as given, so restricting it to one family gives that family's CCD with the same definition, estimator and predictor, from the same labels at no extra cost. The pooled score is the CCD of the declared mix which is what a robustness audit over that mix needs. The revision reports per-family CCD next to the pooled score. On GSM-IC-Adv, Frail-π trained on the pooled labels still ranks each family:

FamilyFrail-π, Spearman / ceiling
Injection0.573 ± 0.060
Reordering0.438 ± 0.071
Renaming0.349 ± 0.223

We release the per-family labels with GSM-IC-Adv, so any subset of families can be scored without new measurement. We hope the clarifications and new runs address your concerns, and would be grateful if you would reconsider your rating.

Comments on the paper

1 comment

Dear reviewers, please review the authors’ rebuttal and provide your responses as soon as possible. Your timely feedback is important for moving the discussion forward. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.