acceptodds
Under review as a conference paper at ICLR 2027

InvariantBench: Can Large Language Models Exhibit Inherent Reasoning Consistently Across Equivalent Transformations?

Abstract

Reasoning is often attributed to large language models (LLMs), yet it remains unclear whether they operate over underlying semantics or rely on surface-form patterns. Existing benchmarks evaluate correctness on fixed problem instances, but overlook a fundamental property of reasoning: invariance under semantics-preserving transformations. If a model truly understands a problem, its predictions should remain consistent across equivalent representations. We introduce ***InvariantBench***, a benchmark of 1159 seed problems invariant forms, spanning 16 tasks across 3 reasoning families and 12 fine-grained invariance axes. Each problem is paired with multiple semantically equivalent variants under a strict invariance contract, enabling evaluation beyond accuracy to measure consistency across representations. Experiments on 15 frontier and open-weight LLMs reveal a persistent invariance gap: invariant reformulations reduce accuracy by up to 30% for strong models, while the gap between solving at least one versus all four variants reaches 60%. Full consistency remains below 5% for most open-weight models, compared with  48% for the two strongest models and 98.6% for human experts. These results show that high base accuracy substantially overestimates reasoning ability, and establish invariance as a necessary axis for evaluating and improving robust language understanding.

Primary area:
datasets and benchmarks
Keywords:
Keywords: Large Language Models, Reasoning, Benchmark, Invariance, Robustness, Generalization, Consistency, Semantic Equivalence, Reasoning Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Lmm9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces InvariantBench, a benchmark for evaluating the consistency of large language model (LLM) reasoning across semantically equivalent problem formulations. The benchmark contains 1,159 seed problems, each paired with three invariant variants, covering 16 task subtypes and 12 transformation axes across formal logic, quantitative reasoning, and relational and algorithmic reasoning. The authors evaluate 15 LLMs using both standard accuracy and several consistency-oriented metrics, including the Invariant Mass Index (IMI) and Conditional Invariance Reliability (CIR). The experimental results reveal substantial differences between solving at least one formulation and correctly solving all four equivalent formulations, even for strong models. The paper further provides an analysis of reasoning failures, human baselines, and exploratory experiments with supervised fine-tuning (SFT) and GRPO to investigate whether training improves cross-form generalization.

Critical strengths

1. A meaningful evaluation problem.

The paper addresses an important limitation of conventional reasoning benchmarks: a model's ability to solve a problem in one formulation does not necessarily imply that it can solve semantically equivalent formulations. Evaluating cross-form consistency alongside accuracy provides a useful complementary perspective on reasoning robustness.

2. Broad empirical coverage.

InvariantBench covers multiple reasoning domains and transformation types, and the evaluation includes 15 LLMs and human baselines. The joint correctness statistics and fine-grained failure analysis provide useful empirical evidence of representation sensitivity across different models and reasoning tasks.

Critical weaknesses

1. Concerns about benchmark correctness and semantic validation.

The reliability of the benchmark is a central concern. For example, Appendix L (Figure 5, page 56) includes a problem asking for the number of primes x≤100000x \leq 100000 satisfying x mod 41=23x \bmod 41 = 23. The reference answer is reported as 17, whereas direct enumeration gives 252. This is a substantive ground-truth error in one of the paper's representative examples. Although a single error does not establish a systematic problem, it raises concerns about the effectiveness of the claimed verification pipeline.

Moreover, the semantic-preservation contract in Section 2 mainly defines the desired equivalence property, without establishing how equivalence is rigorously verified for all generated transformations, particularly abstraction shifts and natural-language reformulations. Since the benchmark's conclusions depend on the correctness and equivalence of its instances, stronger evidence of dataset reliability is essential.

2. Insufficient isolation of representational inconsistency from other factors.

The experiments do not adequately distinguish sensitivity to equivalent reformulations from inherent generation variability and differences in problem difficulty. In particular, the all-four correctness metric (IMI) is strongly influenced by a model's underlying accuracy. Even repeated evaluations of an identical prompt can produce substantially lower joint correctness than single-run accuracy. Without a repeated-sampling baseline using unchanged prompts, it is difficult to determine how much of the observed inconsistency is specifically attributable to representation changes.

Furthermore, some transformations, such as distractor injection and abstraction-level shifts, can change the difficulty of solving a problem despite preserving its answer. Consequently, the results demonstrate sensitivity to problem formulation, but do not fully substantiate the stronger interpretation that these failures necessarily indicate a lack of genuine reasoning or reliance on surface-level memorization.

Questions for authors

Q1. Ground-truth reliability. Could the authors verify the reference answer for item 83994f0c8a9643d8 in Appendix L (Figure 5)? The reported answer is 17, whereas direct prime enumeration gives 252. How was this item validated, and have the authors performed an independent audit of the remaining ground-truth answers? An estimate of the dataset-level error rate would help assess the reliability of the reported results.

Q2. Semantic equivalence of transformations. How is semantic equivalence established for individual transformed instances, particularly abstraction-level shifts and natural-language reformulations? Is expert agreement the primary criterion, or are there additional independent verification procedures? Also, how are transformations involving variable or entity renaming handled when the correct output representation may need to change accordingly?

Q3. Separating invariance failures from sampling variability. Have the authors evaluated the same original question multiple times under identical prompting conditions? Such a baseline would help distinguish instability caused by representation changes from ordinary generation variability. It would also be useful to compare the reported joint correctness metrics against expectations conditioned on each model's marginal accuracy.

Q4. Reliability of automated evaluation. Could the authors clarify the relationship between the 240 validation samples described in Appendix E.2 and the 720 validation responses reported in Table 8? In particular, does the reported perfect judge-human agreement refer to individual human annotations or majority-vote labels? Clarifying the evaluation units and agreement calculations would strengthen confidence in the automated scoring procedure.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Lmm9 (Part 1)

Thank you for recognizing that cross-form consistency is a meaningful complement to accuracy, and that the joint-correctness statistics and failure analysis give empirical evidence of representation sensitivity across 15 models and two human cohorts. Your review also did something we are grateful for: you enumerated the prime-count item in Appendix L, found that the printed reference is wrong, and asked how equivalence and judge agreement are established. Each of your four questions is answered below with data from the paper, new analyses computed from the paper's tables and the per-item scores (labeled Table Rn), and the exact corrected text. We have also highlighted all changes in the revised manuscript in blue.


Weaknesses

  • W1. Benchmark correctness and semantic validation
    • Response: Item 83994f0c8a9643d8 asks for the number of primes p <= 100,000 with p mod 41 = 23. Direct enumeration over the 9,592 primes up to 100,000 gives 252, as you computed, and the printed reference of 17 is incorrect. We corrected the reference to 252 in the dataset and in Figure 5. We then examined what an error of this kind does to the reported results and what independent evidence exists on key and equivalence correctness.

"The reference answer is reported as 17, whereas direct enumeration gives 252. This is a substantive ground-truth error in one of the paper's representative examples. Although a single error does not establish a systematic problem, it raises concerns about the effectiveness of the claimed verification pipeline."

(a) How a key error propagates through the metrics. The four forms of a seed share one reference answer by construction (Section 2). A mis-keyed seed is therefore scored against the same wrong value on Base, I1, I2, and I3.

  • A model is counted as inconsistent on that seed only if it matches the wrong value on some forms but not on others. A model that never produces the wrong value contributes to "All Wrong", not to Gap or to CIR loss.
  • Each mis-keyed seed shifts any model's accuracy, at-least-k, and IMI by at most 1/1,159 = 0.086 points, identically for every model.
  • For this item, the seven model answers shown in Figure 5 (80, 108, 237, 240, 0, 0, 240) differ from both 17 and 252. Correcting the key changes none of the seven displayed judgments. Across all 15 models and 4 forms (60 stored responses), the corrected key changes 1 judgment: Llama-3.2-1B answered 17 on I3 and moves from correct to incorrect, a change of 1/4,636 = 0.02 points in that model's accuracy. No model answered 252. (b) Independent evidence on key correctness, from existing human data. Both human cohorts answered all 4,636 items (Appendix E.3). The expert cohort's majority answer matches the key on 4,619 of 4,636 items.

Table R1: disagreement between independent answers and the key, from existing data.

Audit sourceItemsDisagreements with keyRate (95% Wilson CI)
Human (Expert) majority vs. key, all forms4,636170.37% [0.23%, 0.59%]
Human (Expert) majority vs. key, Base forms1,15910.09% [0.02%, 0.49%]
Seeds with a disagreement on any form1,159161.38% [0.85%, 2.23%]
LLM judge vs. majority of 3 human labels72000.00% [0.00%, 0.53%]

The 17 disagreements are spread as 1 on Base, 0 on I1, 9 on I2, and 7 on I3. This combined rate includes key errors, equivalence errors, and annotator errors, so it is an upper bound on the key-error rate of 0.37% of items (1.38% of seeds, upper CI 2.23%), applied identically to every model. The disagreements concentrate on I2 and I3 (16 of 17), the forms in which an equivalence error would appear; per form they are 0.09% (Base), 0.78% (I2), and 0.60% (I3) of items, so equivalence errors are bounded by those rates. The seeds in this list are the first audit set. Item 83994f0c8a9643d8 was the item you identified; after correction its reference is 252.

For scale, a seed-level key-error rate of 1.38% would change a model's accuracy by at most 1.38 points, equally for all models, and cannot create Gap or CIR loss (part a).

AuthorsOpenReview ↗

Response to Reviewer Lmm9 (Part 2)

(W1 continued)

(d) Verification of semantic equivalence. The pipeline has three layers.

  1. Automated consistency checks across forms and deterministic recomputation of answers where an answer is computable (Appendices B and C).
  2. Review of each problem-variant set by three experts with three labels (accept / revise / reject), kappa = 0.78, and acceptance only after all three reviewers approved the final version (Appendix D.3).
  3. Independent solving of all 4,636 items by two human cohorts (Table R2): if a variant had changed the answer, expert accuracy on that form would drop below its Base accuracy. New analysis (Table R2): expert accuracy by form, from main Table 1.
CohortBaseI1I2I3Range across forms (max minus min)
Human (Expert)99.9100.099.299.40.8 points
Human (Base)90.090.490.189.60.8 points

A non-equivalent variant (answer changed) would appear as a form-specific drop for both cohorts. The per-form range is 0.8 points for each cohort, and the I3 family, which contains abstraction shifts and distractors, is at 99.4% for experts. Across the 1,159 seeds, a form-specific equivalence error rate of 1% (about 12 seeds per form) would be visible as a drop of about 1 point in a cohort whose Base accuracy is 99.9%; the observed I1 to I3 values are 100.0, 99.2, and 99.4, with the lowest I2 value 0.7 points below Base. Experts and the Base cohort were shown items in shuffled order with at least 50 intervening items between variants of the same seed (Appendix E.3).

Revised text (Section 3, Quality control and verification):

"Verification has three layers: (1) automated cross-form consistency checks and deterministic recomputation of answers where computable (Appendices B and C); (2) review of every problem-variant set by three experts (accept / revise / reject, kappa = 0.78; acceptance after unanimous approval); and (3) answers from two human cohorts over all 4,636 items, in which the Human (Expert) majority matches the key on 99.9% of Base items and 99.6% of all items (17 disagreements in 4,636, 95% CI of the rate 0.23% to 0.59%). A shared reference answer cannot create inconsistency across forms: a key error shifts accuracy by at most 1/1,159 = 0.086 points per seed, equally for every model."

How this resolves the concern: the printed error is corrected and its effect on the 15 models is 1 judgment (0.02 points for one model); a shared wrong key cannot create Gap or CIR loss (0.086 points per seed, identical across models); and two human cohorts provide an existing check on keys and equivalence, with a measured disagreement rate and confidence interval (Table R1).

AuthorsOpenReview ↗

Response to Reviewer Lmm9 (Part 3)

  • W2. Insufficient isolation of representational inconsistency from sampling variability and difficulty
    • Response: This is the central methodological question, and we answer it with three analyses: the relationship between joint correctness and accuracy (Table R4), an identical-prompt baseline (Table R5), and the split between difficulty-neutral and difficulty-modulating axes (Table R6).

"The experiments do not adequately distinguish sensitivity to equivalent reformulations from inherent generation variability and differences in problem difficulty. In particular, the all-four correctness metric (IMI) is strongly influenced by a model's underlying accuracy."

IMI and accuracy. We agree with the observation. Across the 15 models, Spearman's rank correlation between mean per-form accuracy and IMI is 0.96. This is why the paper reports Gap, CIR, ICR, and the at-least-k profile alongside IMI, and why we add an accuracy-normalized ratio here.

New analysis (Table R4): joint correctness conditioned on each model's per-form accuracies. The largest IMI attainable given four per-form accuracies is the minimum of those accuracies. Define the nested-consistency ratio NCR = IMI / min(per-form accuracy). NCR = 1 means every seed solved on the hardest form is solved on all forms.

ModelMean accuracy (%)IMI (%)IMI expected if forms fail independently (%)IMI / independentMax attainable IMI (%)NCR
GPT-5.268.147.621.52.2166.60.71
Qwen3.6-Plus66.047.718.92.5264.60.74
Qwen3-8B56.031.19.93.1554.00.58
Claude Haiku 4.548.517.45.53.1547.50.37
Gemini-3-Flash39.917.32.56.8738.30.45
Llama-3.3-70B25.65.70.413.324.20.24
Human (Base)90.065.565.71.0089.60.73
Human (Expert)99.698.698.51.0099.20.99

Observed IMI exceeds the independent-failure expectation for all 15 models (ratios from 2.21 to 618), so the joint-correctness figures are not what independent per-form sampling slips would produce. NCR ranges from 0.12 to 0.74 across the 15 models; its Spearman correlation with mean accuracy is 0.77, against 0.96 for IMI. The accuracy-normalized ordering differs from the accuracy ordering: Claude Haiku 4.5 (48.5% mean accuracy) has NCR 0.37, below Qwen3-8B (56.0%) at 0.58 and below Gemini-3-Flash (39.9%) at 0.45. Claude Haiku 4.5 and Gemini-3-Flash have IMI of 17.4% and 17.3% with Base accuracies 5.3 points apart, so the same all-four correctness comes from different accuracy and consistency mixes. The two highest-accuracy models reach NCR 0.71 and 0.74, in the same range as Human (Base) at 0.73 (accuracy 90.0%), while Human (Expert) is at 0.99.

Sampling variability. Sampling noise has no preferred direction and acts equally on every axis. Reformulation effects are directional and axis-specific. Table R5 gives breakage by axis, defined as Base accuracy minus axis accuracy, averaged over the 15 models (per-axis results in the supplementary material). All responses were generated with one call per item at temperature 0.1 (Appendix E.1).

Table R5: mean breakage by axis, 15 models (points; positive = accuracy lower than Base).

AxisFamilyMean breakage
Variable RenamingI1+4.43
Entity RelabelingI1+3.60
Premise ReorderingI2+3.58
Perspective / Role ReframingI3+2.66
Redundant RestatementI3+2.59
Style / RegisterI1+2.43
Abstraction Level ShiftI3+1.62
Distractor Injection / RemovalI3+0.08
Equivalent Mathematical ReformulationI2-0.34
Representation / Encoding SwapI2-0.59
Equivalent Logical ReformulationI2-1.13
Synonym / Semantic ParaphraseI1-3.21

Under sampling noise alone every axis would center on 0. Instead the 12 axes span 7.64 points. Three axes that keep the wording and content but change identifiers or order lose 3.58 to 4.43 points, while four axes that rewrite the content show a mean change of 0.08 points or an increase. A repeated-prompt baseline cannot generate this ordering, since it would apply the same noise to every axis.

Reviewer_usfTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present an analysis based on a new benchmark to provide understanding into how much of a LLM reasoning performance is preserved when the problem is changed to an invariant representation. The core idea is that models are likely to memorise usual presentations of problems, and that many “reasoning” claims are closer to recall. If true then this benchmark would start to highlight where model performance diverges when the question is changed. The authors find evidence of this, across 15 models (open source and closed) and all demonstrate some / significant degradation of reasoning performance across the invariant formulations. The dataset is hand written at consists of some 1159 seed problems, each with 4x invariant options. The strongest evidence that the models are not blind to this invariance is that the human panel maintains good performance, although this effect is not true for the “base” (highscool / undergraduate cohort)

The questions the authors still need to answer are how much of this benchmark is a lack of invariance on the part of the models, vs a construction of technically the same, but harder problems.

Critical strengths

The core concept is strong, and while a difficult problem the authors frame it well. The claims are proportionate to the evidence (commend the authors there) and the benchmark offers a way to quantify something many people feel when using current models.

The dataset size is good - at 1159 base problems there is enough statistical power to resolve difference between models moderately well.

The dataset is also completely verified by human experts - this kind of benchmark must be hand curated at this stage, and this offers confidence that the questions are written correctly throughout the dataset.

Having two human cohorts is a strong signal, and increases robustness, but does open a line of questioning. See below.

The paper is well organised and well written in a clear consistent voice, and was an engaging read. Aside from a couple of minor editing points I have no concerns on the quality of the presentation.

Critical weaknesses

The core problem with benchmarking properties such as reasoning is exactly as the authors state, we don’t know what it is, and we can’t currently prove what the model is doing. This inevitably leads to a question of is the model lacking the ability to translate to a new framing, or is the alternative framing just a harder question?

We can partially see this in the examples. The verbose first pink box in both cases provides additional / redundant information, but is much clearer and easy to parse. The second yellow box in both cases strongly compresses the question. These are both very useful ways to ask the question, and measuring the models performance in both is good, the concern I have is that the second case is simply a harder question. In an exam you would (I suspect) find that humans find the shorter one harder as they are more likely to misinterpret the question. These specific questions shown are not too bad, but disentangling this phenomena where the question is harder vs hinting that the reasoning is not translating well needs more evidence.

The effect seems to be seen in the base human case, where the subjects have a much lower consistency between variants, similar to the GPT-5.2 level at a 34% gap, while the experts do not meaningfully degrade. We don’t think that by completing a post graduate level degree humans are suddenly able to reason that much more consistently, and I wonder if this highlights that there is a domain expertise bias. Just because there is technically the same amount of information present in a question doesn’t necessarily mean they are of equal difficulty.

The SFT / GRPO analysis in 5.3 felt under developed. The results say that it helped a bit but not completely? Given the amount of data the authors have this is unsurprising, and personally I feel like this either needs expanding into a direction for improvement (probably requiring more data) showing how to start solving this problem or removing and saving for future work. A model now trained on the exact types of question transformation might be evidence of introducing exactly the kind of memorisation the authors are trying to prove is present in the base models.

Questions for authors

Line 236 - missing table link

I would like to see some more quantitative analysis of the dataset question difficulty. Where if we were to take a sample that gives consistent human base variant consistency do the models also become more consistent? Or having the human candidates having ranked the question difficulty to measure their subjective perspective - understand that this is very difficult to do after the fact, more a suggestion for future work.

The details of the training procedure in 5.3 for the SFT and GRPO experiments are solioed in the appendix, the reader needs at least an extra opening sentence or two explaining which model and which procedures you have used in the main text. Important for readability.

Section 2 - the formalism adds a sense of rigor, but it feels unnecessarily abstract. It gives the impression that there is some transformation tau that applied to each problem produces a new structure. The issue I have is that because this is done by a person approximating tau, what you actually need to tell the reader is an expert takes each question and writes different forms of it, making sure that no information is lost. I would ask the authors to either amend this section to make that aspect clear (and the error associated with any approximation of tau, human or otherwise) - or as would be my preference to remove more of the formal framing but give a clear description of the intended outcome of the different problem framing.

8 comments
AuthorsOpenReview ↗

Response to Reviewer usfT (Part 1)

Thank you for the detailed read and for the specific strengths you named: a core concept that quantifies something many users observe, claims that are proportionate to the evidence, 1,159 seeds with enough statistical power to separate models, expert-verified hand-written data, and two human cohorts. You also identified the question we consider most important for this benchmark, namely how much of the observed loss reflects a lack of invariance and how much reflects a technically equivalent but harder question. Your other points on Section 2, Section 5.3, and line 236 led to concrete text changes, shown below. All new analyses are labeled Table Rn and are computed from the paper's tables and the per-item scores already collected. We have also highlighted all changes in the revised manuscript in blue.


Weaknesses

  • W1. Lack of invariance versus an alternative framing that is a harder question
    • Response: This is the right question, and the benchmark has to answer it with evidence. We provide four pieces: marginal accuracy across forms, the human cohorts' accuracy by form, the axis-level split between difficulty-neutral and difficulty-modulating axes, and a subset analysis keyed to human difficulty.

"...is the model lacking the ability to translate to a new framing, or is the alternative framing just a harder question? ... the concern I have is that the second case is simply a harder question."

(a) Your reading of Figure 2 matches the axis data. The two variants illustrate information added (verbose) and information removed (compressed), the two kinds of change covered by the I3 axes Redundant Restatement and Abstraction Level Shift (ALS). ALS is the axis with the highest breakage rank (mean rank 11.50 of 12 across six models; Kendall's W = 0.46, chi-square = 30.4, df = 11, p = 0.0014), and Redundant Restatement has the lowest (mean rank 1.08). Appendix G.2 therefore classifies ALS and Distractor Injection/Removal (DI) as controlled perturbations that can modulate difficulty, and the other 8 axes of Families I1 and I2 (2,318 of 3,477 variants = 66.7%) as difficulty-neutral.

(b) Table R1: breakage rate (base-correct answers failing on the variant), weighted by axis counts, from the breakage-by-axis table in Appendix I.

Model8 difficulty-neutral axes (I1, I2)ALS + DIAll 12 axes
GPT-59.411.29.0
Qwen3.6-Plus9.212.29.4
Claude Haiku 4.515.917.915.9
Llama-3.3-70B11.114.511.4
Gemma-4-26B11.714.612.0
Olmo-3.1-32B8.813.29.3

On axes that keep the information content unchanged, 8.8% to 15.9% of base-correct answers break, which is 67% to 89% of the rate on ALS and DI. Removing ALS and DI changes each model's all-axis breakage by at most 0.5 points (range -0.4 to +0.5). This means that the compressed or verbose forms in Figure 2 do not drive the benchmark-level numbers.

(c) Marginal accuracy is flat across forms. Averaged over 15 models, Base accuracy is 1.25 points above the mean of I1 and I2 and 1.74 points above I3. In 6 of 15 models I3 accuracy is at or above the I1/I2 mean. The Gaps in Table 1 range from 14.4 to 60.0 points, so the Gaps arise from which seeds are solved on which form and not from a uniform drop on the harder-looking forms.

(d) Table R2: human cohorts by form (main Table 1).

CohortBaseI1I2I3Range across forms
Human (Base)90.090.490.189.60.8 points
Human (Expert)99.9100.099.299.40.8 points

Neither cohort shows a form-specific penalty: the I3 family, which includes compressions and distractors, is within 0.4 points of Base for Human (Base) and within 0.5 points for Human (Expert). In the exam setting you describe, the shorter form would be harder for humans on average, and this would appear as lower accuracy on that form.

(e) Subset analysis keyed to human difficulty (your Q2). See Q2, Table R6: model IMI differs by -3.3 to +2.8 points between seeds on which Human (Base) is consistent and the rest, with no consistent sign.

Revised text (Section 5.1, after "Human-Model Gap"):

"To check whether the reformulations are harder rather than inconsistently handled, we compare accuracy by form and separate axes by their effect on information content. Human accuracy varies by at most 0.8 points across the four forms in both cohorts, and breakage on the eight difficulty-neutral axes is 8.8% to 15.9% for six models, 67% to 89% of the rate on Abstraction Level Shift and Distractor Injection/Removal (Appendix G.2)."

How this resolves the concern: the answer is given as measurements. Humans show a 0.8-point range across forms, marginal model accuracy differs by 1.25 to 1.74 points, and 8.8% to 15.9% of base-correct answers break on axes that leave the information content unchanged.

AuthorsOpenReview ↗

Response to Reviewer usfT (Part 2)

  • W2. The Human (Base) cohort shows a consistency gap similar to GPT-5.2 (34%), while experts do not degrade; is there a domain-expertise bias?
    • Response: The parallel is in Table 1 (Human (Base) Gap 34.5, GPT-5.2 Gap 33.8). We checked what drives the Human (Base) Gap and report it below.

"The effect seems to be seen in the base human case, where the subjects have a much lower consistency between variants, similar to the GPT-5.2 level at a 34% gap, while the experts do not meaningfully degrade."

New analysis (Table R3): human joint correctness compared with what each cohort's per-form accuracy alone predicts. If errors on the four forms were independent, the share of seeds with all four forms correct equals the product of the four per-form accuracies.

CohortPer-form accuracy (Base / I1 / I2 / I3)All-four correct, observed (%)All-four correct, product of accuracies (%)Observed / productGap, observedGap, product-implied
Human (Base)90.0 / 90.4 / 90.1 / 89.665.565.71.0034.534.3
Human (Expert)99.9 / 100.0 / 99.2 / 99.498.698.51.001.41.5

For both cohorts, the observed all-four rate equals the product-of-accuracies value to within 0.2 points. The Human (Base) Gap is therefore what a 10.0% per-answer error rate produces across four forms, with no additional form-specific component. The expertise difference between the cohorts appears as a 9.6-point difference in per-answer accuracy (90.0% vs. 99.6%), applied evenly to all four forms (range 0.8 points in each cohort, Table R2). This is consistent with your view that a postgraduate degree does not change consistency itself: consistency tracks accuracy in both cohorts.

What this means for model comparisons. Gap depends on the accuracy level, so a numerically equal Gap at different accuracies (GPT-5.2: 69.4%; Human (Base): 90.0%) is not equal consistency. The paper reports CIR and the joint distribution for this reason, and we add the accuracy-normalized ratio NCR = IMI / (lowest per-form accuracy) below, where 1.0 means every seed solved on the hardest form is solved on all forms.

Model or cohortMean accuracy (%)IMI (%)Max attainable IMI (%)NCR
Human (Expert)99.698.699.20.99
Human (Base)90.065.589.60.73
Qwen3.6-Plus66.047.764.60.74
GPT-5.268.147.666.60.71
Qwen3-8B56.031.154.00.58
Gemini-3-Flash39.917.338.30.45
Claude Haiku 4.548.517.447.50.37
Llama-3.3-70B25.65.724.20.24

Across the 15 models NCR ranges from 0.12 to 0.74. The two highest-accuracy models are at 0.71 and 0.74, in the range of the Human (Base) cohort (0.73), and the other 13 models are between 0.12 and 0.58. Human (Expert) is at 0.99. The benchmark therefore separates a top tier from 13 models whose joint correctness is at most 58% of what their own per-form accuracies would permit.

How this resolves the concern: the Human (Base) Gap is fully explained by its 10.0% per-answer error rate (observed/product = 1.00), with no form-specific effect, so it supports the benchmark's equal-difficulty design and does not indicate a domain-expertise artifact.

AuthorsOpenReview ↗

Response to Reviewer usfT (Part 3)

  • W3. Section 5.3 (SFT / GRPO) feels underdeveloped; training on the exact transformation types might itself introduce memorization
    • Response: We agree that the section needed context in the main text and a clearer reading, and your last point is what the All-Invariants regime in the paper shows. We restructured the section around three measurements.

"A model now trained on the exact types of question transformation might be evidence of introducing exactly the kind of memorisation the authors are trying to prove is present in the base models."

Your concern is what the two regimes are designed to separate. The split is by seed (all four forms of a seed on the same side), so no test problem appears in training in any form. When all four transformation types are in training (All-Invariants), SFT's four forms converge to 53.84% to 54.53% at epoch 5 (Base 54.53, Inv1 54.36, Inv2 54.18, Inv3 53.84). That is the pattern you describe: exposure to the transformation types themselves removes the form-to-form difference. The Base+Inv1 regime is the test that avoids it: Inv2 and Inv3, different transformation types, are held out from training entirely, and the 26.7-point gap that remains is the measurement.

Table R4: epoch-5 results, Base+Inv1 regime (Gemma-3-1B-IT; Appendix J.1 and J.2).

MethodBaseInv1Inv2 (unseen)Inv3 (unseen)Seen meanUnseen meanGapUnseen / seen
SFT42.3642.4517.1714.2442.4115.7126.700.37
GRPO30.4632.0119.6720.8831.2420.2810.960.65

GRPO reduces the seen-minus-unseen gap by 59.0% and reaches 4.57 points higher mean unseen accuracy (pooled z = 3.14, p = 0.0017; unpaired, n = 695 per cell), even though its seen accuracy is 11.17 points lower. Unseen-form gains over epoch 0 are +10.61 and +8.11 (SFT) and +13.46 and +15.27 (GRPO).

On replication. The comparison rests on the size of the test set (n = 695 seeds per form per cell) and on the contrast between the two regimes, with unpaired z-tests (pooled GRPO minus SFT on unseen forms: z = 3.14, p = 0.0017). The reference study on SFT versus RL generalization, Chu et al. [1] (ICML 2025), likewise reports its comparisons on a single backbone family. We therefore describe Section 5.3 as a diagnostic of one small model.

Scope. We keep Section 5.3 as a diagnostic of transfer to held-out transformation types, which the benchmark's four-form structure makes possible for the first time, and we state the improvement direction the data supports: reward-based training reduced the seen-unseen gap by 59.0% at lower seen accuracy, so the headroom is in raising unseen-form accuracy (20.28% in this setting) rather than in the gap alone. A larger data regime for this purpose would require more seeds than the 40% training split provides (464 seeds), which is why we report the result as a diagnostic.

Revised text (Section 5.3, final paragraph):

"These runs measure transfer to transformation types that were held out of training, not mechanisms. In the Base+Inv1 regime, SFT shows a 26.7-point gap between seen and unseen forms and GRPO an 11.0-point gap (59.0% smaller), with 4.6 points higher mean unseen accuracy for GRPO (p = 0.002). The All-Invariants regime, in which the four forms converge to 53.8% to 54.5% under SFT, shows the effect of training on the transformation types themselves. Splits are by seed, so no test problem is seen in any form. Results are for one model (Gemma-3-1B-IT) and one training configuration."

How this resolves the concern: the section now states the training procedure and the seed-level split in the main text, treats the All-Invariants result as the effect of training on the transformation types, and reports the held-out-form measurement with its significance tests and scope.

Reviewer_wmi6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InvariantBench, a benchmark for evaluating the consistency of LLM reasoning across semantically equivalent formulations. It contains 1,159 seed problems, each paired with three transformed variants, covering 16 tasks and 12 transformation types. Experiments on 15 LLMs reveal substantial differences between accuracy on individual formulations and correctness across all variants. The paper further provides human baselines, analyzes failure modes, and compares SFT and GRPO as approaches to improving robustness across equivalent representations.

Critical strengths
  1. Evaluating consistency across equivalent formulations is a useful complement to accuracy on individual questions and addresses a relevant limitation of reasoning evaluations.

  2. The benchmark covers diverse reasoning tasks and transformation types, enabling analysis of how model performance varies across different forms of reformulation.

Critical weaknesses
  1. The evaluation does not sufficiently support conclusions about inherent reasoning. The model selection omits recent leading reasoning models, and the paper does not systematically compare reasoning modes or budgets to distinguish limited reasoning capability from sensitivity to reformulation.

  2. The experiments do not adequately isolate the effect of reformulation. Although CIR conditions on a correct base answer, one successful response does not establish reliable solvability, and no repeated identical prompt baseline is provided. Some transformations also change difficulty or introduce solution guidance, such as the explicit computational steps in Figure 2.

  3. The SFT and GRPO experiments provide limited support for the claimed distinction between memorization and generalization. Similar accuracy across variants does not establish memorization, and results from a single 1B model are insufficient to support broader conclusions about these training methods.

Questions for authors
  1. Can you evaluate recent leading reasoning models and compare different reasoning modes or budgets, reporting the exact inference configurations?

  2. Can you compare reformulated questions against repeated identical prompts on a subset of problems each model reliably solves, and separately report results for pure surface transformations versus transformations that alter difficulty or provide solution guidance?

  3. Can you clarify whether the training and test sets are disjoint at the seed problem level, and repeat the SFT and GRPO comparison across multiple training seeds and a stronger model?

8 comments
AuthorsOpenReview ↗

Response to Reviewer wmi6 (Part 1)

Thank you for describing consistency across equivalent formulations as a useful complement to single-question accuracy, and for noting that the diversity of tasks and transformation types supports analysis of how performance varies across reformulations.

Your three concerns are specific: the coverage of reasoning models and budgets, the isolation of the reformulation effect from sampling and difficulty, and the evidence behind the SFT/GRPO comparison. Each is answered below with data already collected for the paper, new analyses computed from the per-model, per-axis metrics in the supplementary material, and exact revised text. We have also highlighted all changes in the revised manuscript in blue.


Weaknesses

  • W1. The evaluation does not sufficiently support conclusions about inherent reasoning: recent leading reasoning models, and reasoning modes and budgets, are not compared systematically
    • Response: The set contains hosted frontier models (GPT-5.2, Gemini-3-Flash, Claude Haiku 4.5, Qwen3.6-Plus, o4-mini) and ten others. Every reasoning-capable model was run in its highest-capability configuration (reasoning enabled, the largest effort setting the provider exposes, and a generation limit sized for reasoning tokens; Appendix E.1). This choice answers the question you pose, whether failures reflect limited capability or sensitivity to reformulation, from the side that favors capability: the consistency loss we report is measured at each model's ceiling, where extra reasoning effort has the least room left to help.

"The model selection omits recent leading reasoning models, and the paper does not systematically compare reasoning modes or budgets to distinguish limited reasoning capability from sensitivity to reformulation."

Table R1: inference configuration, identical across all 15 models except where noted.

ItemSetting
API routeOpenRouter (Command-A through the Cohere endpoint)
PromptSame answer-only system prompt for every model and every form (Appendix E.1)
Calls per item1 (no best-of-n, no retries that change the prompt)
Temperature0.1
Reasoning modeEnabled for every model that has one, at the highest-capability setting
Generation limit600 tokens (standard models); a larger limit for reasoning-augmented models so that reasoning is not truncated
JudgeGemma-4-26B-A4B-Instruct, reference-based, same prompt for all models
Empty responses0.00% for all 15 models (0 of 4,636 per model)

The last row matters for your concern: no low score in Table 1 comes from blank outputs. Every one of the 4,636 responses per model contains text.

Why a sweep over reasoning levels is not needed to support the claims, and what the existing data show. A sweep of effort levels measures an accuracy-versus-compute trade-off. The benchmark's question is different: given the best a model can do, does the answer stay the same when the problem is rewritten? Three facts in the existing results address your point directly.

  1. Capability already varies widely across the 15 models, and consistency rises with it. The three highest-accuracy models (Qwen3.6-Plus, GPT-5.2, Qwen3-8B) show IMI of 31.1% to 47.7% and CIR of 57.7% to 69.1%. The five lowest-accuracy models above 1B parameters (Ministral-3B, Gemma-3-4B, Olmo-3.1-32B, Aya-Expanse-32B, DeepSeek-V3.1) show IMI of 1.2% to 3.1% and CIR of 10.9% to 19.3%.
  2. A residual inconsistency remains at the ceiling. For GPT-5.2 and Qwen3.6-Plus, 1 minus CIR is 31.3% and 30.9%: among seeds each model solves on Base, about 31% fail at least one other form. Human (Expert) loses 1.3% (CIR 98.7%).
  3. The residual is not an artifact of how the answer is extracted. The response is 1 or 2 lines for every model, and the judge matched three human annotators on 720 of 720 labeled responses (Appendix E.2). Table R2: accuracy and consistency by model tier (values from Tables 1 and 2 of the paper).
ModelBase (%)IMI (%)CIR (%)Gap
Qwen3.6-Plus69.047.769.131.8
GPT-5.269.447.668.733.8
Qwen3-8B54.031.157.744.9
Claude Haiku 4.548.317.436.160.0
Gemini-3-Flash43.017.340.443.8
Llama-3.3-70B25.75.722.144.6
Command-A21.64.520.839.3
Human (Base)90.065.572.834.5
Human (Expert)99.998.698.71.4

Moving from Command-A and Llama-3.3-70B (Base 21.6% and 25.7%) to the two strongest models (Base 69.0% and 69.4%) raises Base by 43 to 48 points and IMI by 42 to 43 points, while the Gap stays at 31.8 to 33.8 points. The Gap does not close as capability rises, which is the pattern a sensitivity to form would produce and a pure capability limit would not.

AuthorsOpenReview ↗

Response to Reviewer wmi6 (Part 2)

(W1 continued)

Precedent for a single fixed protocol. GSM-Symbolic [1] (ICLR 2025) evaluated 25 models under one fixed protocol (8-shot chain of thought, greedy decoding) and reported that decoding for o1-family models could not be controlled; it reports no effort-level sweeps. Our protocol is fixed in the same way, with the exact configuration stated (Table R1).

Revised text (Appendix E.1):

"All models receive the same answer-only prompt, one call per item, temperature 0.1. Models with a reasoning mode are run with reasoning enabled at the highest-capability setting available, so reported consistency losses are measured at each model's capability ceiling. 0 of 4,636 responses per model were empty."

How this resolves the concern: the configuration is stated (Table R1), models differing in reasoning capability are compared on the same items and metrics (Table R2), and the consistency loss remains at 31.8 to 33.8 points for the two strongest models, so the conclusion does not depend on a reasoning-effort setting.

  • W2. The experiments do not isolate the effect of reformulation: one successful response does not establish reliable solvability, there is no repeated identical-prompt baseline, and some transformations change difficulty or give solution guidance
    • Response: We address the three parts with evidence that already exists in the per-axis results: where noise would act, where difficulty would act, and where guidance would act.

"Although CIR conditions on a correct base answer, one successful response does not establish reliable solvability, and no repeated identical prompt baseline is provided. Some transformations also change difficulty or introduce solution guidance, such as the explicit computational steps in Figure 2."

(a) Sampling noise versus reformulation. Sampling noise has no preferred direction. It moves a variant's accuracy above or below Base with equal probability, and it acts identically on every axis. Reformulation effects are directional and axis-specific. The per-axis results for all 15 models (supplementary material) show the second pattern. Table R3 gives breakage for each of the 12 axes, defined as Base accuracy minus axis accuracy, averaged over the 15 models.

Table R3: mean breakage by axis, 15 models (points; positive = accuracy lower than Base).

AxisFamilyMean breakage
Variable RenamingI1+4.43
Entity RelabelingI1+3.60
Premise ReorderingI2+3.58
Perspective / Role ReframingI3+2.66
Redundant RestatementI3+2.59
Style / RegisterI1+2.43
Abstraction Level ShiftI3+1.62
Distractor Injection / RemovalI3+0.08
Equivalent Mathematical ReformulationI2-0.34
Representation / Encoding SwapI2-0.59
Equivalent Logical ReformulationI2-1.13
Synonym / Semantic ParaphraseI1-3.21

The spread is 7.64 points between the axes at the two ends, with 8 axes positive and 4 near zero or negative. Under noise alone every axis would center on 0 with the same spread. Instead, three axes that keep content fixed but change identifiers or order (Variable Renaming, Entity Relabeling, Premise Reordering) lose 3.58 to 4.43 points, and axes that rewrite the wording (Synonym, Logical and Mathematical Reformulation, Encoding Swap) show no loss. This is also why the same-prompt baseline you propose would be expected to change the size of the numbers by at most the noise floor, and could not produce the ordering in Table R3.

The measured noise floor is further limited by how the models were run: one call per item at temperature 0.1 (Table R1).

(b) Difficulty. The paper separates 8 difficulty-neutral axes (Families I1 and I2, 2,318 of 3,477 variants = 66.7%) from Abstraction Level Shift and Distractor Injection/Removal (Appendix G.2). The six-model numbers follow, from the breakage-by-axis table.

Table R4: breakage rate (base-correct answers failing on the variant), weighted by axis counts.

Model8 difficulty-neutral axes (I1, I2)ALS + DI (controlled)Perspective + Redundant RestatementAll 12 axes
GPT-59.411.25.19.0
Qwen3.6-Plus9.212.27.39.4
Claude Haiku 4.515.917.913.715.9
Llama-3.3-70B11.114.59.411.4
Gemma-4-26B11.714.610.412.0
Olmo-3.1-32B8.813.27.19.3

On the eight difficulty-neutral axes, 8.8% to 15.9% of base-correct answers break, which is 67% to 89% of the rate on ALS and DI. Removing ALS and DI changes each model's all-axis breakage by at most 0.5 points.

AuthorsOpenReview ↗

Response to Reviewer wmi6 (Part 3)

(W2 continued)

Table R5: consistency metrics with I3 removed (Base + I1 + I2 only, computed from the same per-item scores).

ModelGap, 4 formsGap, Base+I1+I2IMI, 4 formsIMI, Base+I1+I2CIR, 4 formsCIR, Base+I1+I2
GPT-5.233.827.147.653.168.776.5
Qwen3.6-Plus31.823.347.753.569.177.5
Qwen3-8B44.935.131.137.757.769.8
Claude Haiku 4.560.049.117.423.636.148.8
Gemini-3-Flash43.835.517.321.840.450.8
Llama-3.3-70B44.637.15.79.122.135.2
Human (Base)34.526.765.573.272.881.3
Human (Expert)1.40.898.699.298.799.3

Removing I3 moves every model and both human cohorts in the same direction by a similar amount (Gap falls 6.7 to 10.9 points for the six models; IMI rises 3.4 to 6.6 points), and the ordering of the six models by Gap, IMI, and CIR is unchanged. The Gap is present with the difficulty-neutral families alone: 23.3 to 49.1 points for the six models.

Table R6: do models become consistent on the seeds humans find consistent? Seeds are split by whether the Human (Base) majority is correct on all four forms (H4: n = 759) or not (other: n = 400).

ModelIMI on H4IMI otherGap on H4Gap other
GPT-5.246.849.435.230.8
Qwen3.6-Plus46.649.933.528.6
Qwen3-8B31.131.345.244.4
Claude Haiku 4.516.519.360.658.6
Gemini-3-Flash18.315.544.741.9
Llama-3.3-70B6.24.846.840.4

Model consistency on H4 differs from the rest by -3.3 to +2.8 points of IMI, with no consistent direction (three models higher on H4, three lower). A shared difficulty factor that drives both human and model failures would produce lower model consistency on the seeds humans find harder. IMI shows no such pattern, and the Gap is 0.8 to 6.4 points higher on the seeds humans answer consistently (because the other seeds contain more seeds that models fail on every form). Neither measure indicates that the model Gap tracks how hard a seed is for people.

(c) Variants that add information or guidance. An axis that adds explicit steps or redundant context should make a variant easier, which would show up as accuracy above Base. Across 15 models, accuracy on I3 (the family with distractors, abstraction shifts, perspective changes, and redundant restatements) is 1.74 points below Base on average (I1 and I2: 1.25 points below Base). Redundant Restatement, the axis closest to adding guidance, has a mean breakage of +2.59 points (Table R3), so it lowers accuracy rather than raising it. The two variants in Figure 2 illustrate one axis each; each axis is one of 12 and contributes at most 8.3% of the variants.

Revised text (Section 4, evaluation paragraph and Appendix G.2):

"We separate sensitivity to reformulation from sampling variability and difficulty in three ways. (i) Breakage by axis for 15 models is directional and axis-specific (from +4.43 points for Variable Renaming to -3.21 for Synonym/Paraphrase), which sampling noise would not produce. (ii) Restricting to Base + I1 + I2, all 15 models and both human cohorts keep a Gap (23.3 to 49.1 points for the six models in Table R5). (iii) On seeds on which Human (Base) is correct on all four forms, model IMI differs from other seeds by -3.3 to +2.8 points with no consistent sign."

How this resolves the concern: the existing per-axis data show that the effect has the direction and axis structure that sampling noise cannot produce (Table R3), persists on the difficulty-neutral families alone (Tables R4 and R5), and does not track human difficulty (Table R6).

  • W3. The SFT/GRPO experiments provide limited support for the memorization versus generalization distinction: similar accuracy across variants does not establish memorization, and a single 1B model is insufficient
    • Response: We agree that similar accuracy across variants does not by itself establish memorization. The claim in the paper rests on the held-out-form regime, where the quantity of interest is the seen-minus-unseen gap on held-out problems, and we restate it on that basis.

"Similar accuracy across variants does not establish memorization, and results from a single 1B model are insufficient to support broader conclusions about these training methods."

Design. The 40:60 train/test split is applied to seeds: all four forms of a seed fall on the same side. The test set is 60% of the seeds (n = 695 per form). "Seen" and "unseen" therefore refer to form types that were or were not in the training data, always on problems that were not in the training data. No test problem was seen in any form during training.

Reviewer_MAQtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces InvariantBench, a benchmark of 1,159 seed problems, each paired with three transformed versions. It covers 16 tasks and 12 transformation axes across logical, quantitative, and relational reasoning. The authors evaluate 15 LLMs using accuracy, joint correctness, and conditional retention metrics, with human baselines and an LLM judge. The results show substantial variation in correctness across formulations. The paper also analyzes failure categories and compares supervised fine-tuning with GRPO on Gemma-3-1B-IT.

Critical strengths

The benchmark provides a common evaluation protocol across diverse reasoning tasks. This breadth makes it a useful resource for studying robustness beyond individual domains and comparing model vulnerabilities systematically. The substantial gap between current models and human experts gives the benchmark useful headroom for measuring progress. Strong expert performance also supports the feasibility of maintaining correctness across the tested transformations.

Critical weaknesses

The novelty of the evaluation design is limited. LGMT [1] and Logifus [2] already assess reasoning robustness through structure-preserving reformulations. The paper does not clearly establish its contribution beyond expanding task and transformation coverage. The empirical analysis provides limited new insight. The main finding that accuracy does not ensure robustness to reformulation is well established. The transformation and error analyses remain largely descriptive and do not establish a clear explanation for the observed failures. The reliability of the reference answers needs stronger validation. An incorrect reference answer appears in the appendix despite the claimed verification process. Since the LLM judge relies on these references, label errors could affect the reported results. The training experiments do not substantiate the claim that SFT induces memorization while GRPO improves reasoning. Differences in learning curves and performance on withheld variants are insufficient to identify the mechanisms underlying these outcomes. (Minor) The manuscript uses overly tight line spacing, which reduces readability. The category B description in Figure 1 is upside down and should be rotated to an upright orientation. [1] LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

[2] Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation

Questions for authors

What substantive conclusions about reasoning robustness does InvariantBench support beyond LGMT and Logifus? Can the authors provide a direct comparison to demonstrate this added value? Why do several larger models score substantially below smaller models? Can the authors explain these rankings and verify that they persist under comparable inference settings and independent evaluation?

8 comments
AuthorsOpenReview ↗

Response to Reviewer MAQt (Part 1)

Thank you for identifying three strengths that we consider central to the benchmark: a common evaluation protocol across diverse reasoning tasks, measurable headroom between current models and human experts, and expert performance that shows the transformations can be handled without loss of correctness.

Your concerns about novelty relative to LGMT and Logifus, reference-answer reliability, the interpretation of the SFT/GRPO study, and the ranking of larger versus smaller models are specific and answerable with data. Below we address each one, using results already in the paper, new analyses computed from the paper's tables and per-item scores, and the literature. We have also highlighted all changes in the revised manuscript in blue.


Weaknesses

  • W1. Novelty relative to LGMT [1] and Logifus [2]

    • Response: Both are closely related, and we have added them to the related-work section (revised text below). We read them in full and compared them with InvariantBench along the properties listed in Table R1.

    New analysis (Table R1): property-level comparison.

    PropertyLGMT [1]Logifus / LogiQAte [2]InvariantBench (ours)
    Reasoning domainsFOL entailment (True/False/Unknown); source cases from FOLIO, LogicNLI, ProverQA4 tasks: FOL, blood relation, number series, direction sense3 families, 16 sub-types: formal logic, algebra/probability/optimization, graph/algorithm/pseudocode
    Unit of evaluationMetamorphic group (source case plus follow-up); 3,091 groups evaluatedOriginal vs. obfuscated question; 1,108 questionsSeed plus 3 expert-written equivalent forms; 1,159 seeds x 4 forms = 4,636 items
    Equivalence transformations20 metamorphic-relation types derived from FOL equivalencesObfuscation functions per task12 axes in 3 families (symbolic and linguistic, structural and representational, abstraction and information-structure)
    Correctness signalOracle-free cross-case consistency; labels used only for accuracy and hidden-defect comparisonsAccuracy on original vs. obfuscatedExpert-verified reference answers, validated judge, and joint correctness over all four forms
    Consistency metricsViolation rate, hidden-defect rateAccuracy drop13 metrics: at-least-k, Gap, IMI, CIR, ICR, PIDI, BSS, CE, IRR, FG, TCIR, TCC, RIS
    Models evaluated6915
    Human baseline on the reasoning taskNoneNot part of the benchmark protocol described in the abstract2 cohorts (Base, Expert), 3 annotators each, all 4,636 items
    Training interventionNone (prompting variants only)None (layer-wise probability analysis, Sec. 5.1 of [2])SFT vs. GRPO with two forms held out

    Five differences follow from the table, each tied to a result in our paper.

    1. Joint correctness over a fixed family of four forms. LGMT scores source and follow-up pairs; Logifus scores an accuracy drop. InvariantBench scores the distribution of correct forms per seed, P(c = k) for k = 0..4. This exposes structure that an accuracy drop cannot. Table R2 shows that Logifus-style accuracy drops (Base minus mean of the three variants) stay between -2.7 and +4.2 points for six models, while the Gap between solving at least one form and all four forms ranges from 31.8 to 60.0 points.
    2. Axis-level findings that agree across architectures. Across six models and 12 axes, the axis ranking by breakage rate has Kendall's W = 0.46 (chi-square = 30.4, df = 11, p = 0.0014; permutation p = 0.0002). Redundant Restatement has the lowest breakage in 5 of 6 models (mean rank 1.08 of 12) and Abstraction Level Shift the highest (mean rank 11.50). Neither [1] nor [2] reports an axis ordering across models.
    3. A human anchor on the same items. Both human cohorts answered all 4,636 items. Human (Expert) reaches 99.9% on Base and 98.6% on all four forms (Gap = 1.4). This establishes that the 12 axes are solvable without loss, which an oracle-free design [1] cannot establish.
    4. Breadth beyond logic and relational puzzles. 16 sub-types include algebra, probability, optimization, graph search, multi-agent planning, and pseudocode execution (Appendix B). LGMT is restricted to FOL entailment, and LogiQAte has four tasks.
    5. A held-out-form training study. Training on Base and Inv1 and testing on Inv2 and Inv3 measures transfer to unseen transformation types (Table R7 under W4). Neither paper includes training experiments.
AuthorsOpenReview ↗

Response to Reviewer MAQt (Part 2)

(W1 continued)

Table R2: accuracy-drop view vs. joint-correctness view, computed from main Table 1.

ModelBase (%)Mean of I1-I3 (%)Accuracy drop (Base minus mean)At least 1 form (%)All 4 forms (%)Gap
GPT-5.269.467.7+1.781.447.633.8
Qwen3.6-Plus69.065.0+4.079.547.731.8
Claude Haiku 4.548.348.6-0.377.417.460.0
Gemini-3-Flash43.038.8+4.261.117.343.8
Qwen3-8B54.056.7-2.776.031.144.9
Llama-3.3-70B25.725.6+0.150.35.744.6
Human (Expert)99.999.5+0.4100.098.61.4

A protocol that reports only the accuracy drop would classify Claude Haiku 4.5 as having no sensitivity (-0.3 points). The joint view shows that it solves at least one form of 77.4% of seeds, within 4.0 points of GPT-5.2 (81.4%), but solves all four forms on 17.4% of seeds against 47.6% for GPT-5.2, a 30.2-point difference. This is information about models that neither [1] nor [2] reports.

Precedent at the same venue class. Evaluation-design contributions that re-express the same problem and report variation have been accepted at ICLR: GSM-Symbolic [3] (ICLR 2025), MATHCHECK [4] (ICLR 2025), and DyVal [5] (ICLR 2024). Each of these overlaps with an earlier line of robustness work, and each contributed a new evaluation unit and protocol. InvariantBench follows the same pattern: a new unit (the four-form family), a new metric suite, a human anchor, and a training study.

Revised text (Related Work, Appendix A):

"LGMT [Zhou et al., 2026] derives 20 metamorphic relations from first-order-logic equivalences and checks consistency between source and follow-up predictions without reference labels, on FOL entailment with six LLMs. Logifus/LogiQAte [Borah et al., 2026] rewrites 1,108 questions in four tasks into logically equivalent obfuscated forms and reports accuracy drops for nine LLMs. InvariantBench differs in four measurable ways: it scores the joint-correctness distribution over a fixed family of four expert-written forms per seed (4,636 items), it covers 16 sub-types across logic, quantitative, and algorithmic reasoning, it adds two human cohorts that answered every item, and it includes a held-out-form SFT/GRPO study."

How this resolves the concern: the comparison is now explicit and measurable, the added value is demonstrated on the same models (Table R2), and the two papers are cited and positioned. The conclusions in Table R2 and the axis ordering (W = 0.46) cannot be obtained from either prior protocol.

  • W2. The empirical analysis provides limited new insight and remains descriptive
    • Response: We agree that "accuracy does not ensure robustness" is a known direction, and we do not present it as the finding. The paper's findings are quantitative statements about structure, listed below, and we add a decomposition that moves from description toward explanation.

"The main finding that accuracy does not ensure robustness to reformulation is well established. The transformation and error analyses remain largely descriptive and do not establish a clear explanation for the observed failures."

Findings specific to InvariantBench, with numbers.

FindingEvidence
Coverage of one form is a poor proxy for coverage of all fourAt least-1 vs. all-4: GPT-5.2 81.4 vs. 47.6; Claude Haiku 4.5 77.4 vs. 17.4; Gemini-3-Flash 61.1 vs. 17.3 (Table 1)
Axis difficulty is shared across architecturesKendall's W = 0.46 over 6 models and 12 axes, p = 0.0014
Experts keep all four forms correctHuman (Expert): 98.6% all-4, accuracy range across the 4 forms 0.8 points
Failure types depend on model tierTop models are limited mainly by E5 (invariance robustness) and E4 (chain coherence); lower-tier models by E1 (premise comprehension) and E3 (local computation) (Sec. 5.2, 48 sub-categories)
Post-training method changes transfer to unseen formsSeen-minus-unseen gap 26.7 points (SFT) vs. 11.0 points (GRPO), a 59.0% reduction
AuthorsOpenReview ↗

Response to Reviewer MAQt (Part 3)

(W2 continued)

New analysis (Table R3): explaining where failures come from. Axes in the paper are split into difficulty-neutral axes (Families I1 and I2, 8 axes, 2,318 of 3,477 variants = 66.7%) and controlled perturbations that can modulate difficulty (Abstraction Level Shift and Distractor Injection/Removal), as stated in Appendix G.2. Table R3 gives breakage rates (base-correct answers that fail on the variant, in percent) for the six models in the paper's breakage table, weighted by axis counts.

Model8 difficulty-neutral axes (I1, I2)ALS + DI (controlled)All 12 axes
GPT-59.411.29.0
Qwen3.6-Plus9.212.29.4
Claude Haiku 4.515.917.915.9
Llama-3.3-70B11.114.511.4
Gemma-4-26B11.714.612.0
Olmo-3.1-32B8.813.29.3

Breakage on the difficulty-neutral axes alone is 8.8% to 15.9%, which is 67% to 89% of the breakage on the controlled axes. Removing the two controlled axes changes each model's all-axis breakage by at most 0.5 points (range -0.4 to +0.5). The failures therefore occur on transformations that, by construction, keep the information content and solution path unchanged (Appendix G.2).

New analysis (Table R4): marginal accuracy is nearly flat across forms. Averaged over 15 models, Base accuracy exceeds the mean of the two difficulty-neutral families (I1, I2) by 1.25 points and exceeds I3 by 1.74 points. In 6 of 15 models, I3 accuracy is at or above the I1/I2 mean. Human (Base) accuracy is 90.0% (Base), 90.2% (I1/I2 mean), and 89.6% (I3). The mean marginal drops are 1.25 and 1.74 points, while the Gaps in Table 1 range from 14.4 to 60.0 points. The Gaps therefore arise because the set of correct forms differs from seed to seed, not because the variants as a group are harder.

Decomposition of inconsistency into sampling noise and reformulation (Table R5). Sampling noise has no preferred direction and acts equally on every axis, whereas reformulation effects are directional and axis-specific. Table R5 gives mean breakage by axis (Base accuracy minus axis accuracy, averaged over 15 models; per-axis results in the supplementary material). Every response was generated with one call per item at temperature 0.1 (Appendix E.1).

AxisFamilyMean breakage (points)
Variable RenamingI1+4.43
Entity RelabelingI1+3.60
Premise ReorderingI2+3.58
Perspective / Role ReframingI3+2.66
Redundant RestatementI3+2.59
Style / RegisterI1+2.43
Abstraction Level ShiftI3+1.62
Distractor Injection / RemovalI3+0.08
Equivalent Mathematical ReformulationI2-0.34
Representation / Encoding SwapI2-0.59
Equivalent Logical ReformulationI2-1.13
Synonym / Semantic ParaphraseI1-3.21

Under noise alone every axis would center on 0. The 12 axes span 7.64 points: three axes that change identifiers or order lose 3.58 to 4.43 points, while four axes that rewrite the content show a mean change of 0.08 points or an increase. This pattern is the reformulation component. Two further checks use only existing data. With I3 removed (Base + I1 + I2), Gap remains 23.3 to 49.1 points for the six models in Table R2 (GPT-5.2: 27.1, Qwen3.6-Plus: 23.3, Qwen3-8B: 35.1, Claude Haiku 4.5: 49.1, Gemini-3-Flash: 35.5, Llama-3.3-70B: 37.1). On the 759 seeds where Human (Base) is correct on all four forms, model IMI differs from the other 400 seeds by -3.3 to +2.8 points with no consistent sign, so the Gap does not track human-perceived difficulty.

How this resolves the concern: the analysis now states which axes cause failures (ALS highest, RR lowest, shared across architectures), shows the failures persist on difficulty-neutral axes (Table R3), and shows that the pattern has the direction and axis structure that noise does not produce (Table R5). The result moves from "accuracy is not robustness" to a ranked, decomposed account of where and how much.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow up on Rebuttal Responses and Remaining Discussion Period

Dear Reviewers,

We wanted to briefly follow up on our rebuttal, as only 44 hours remain before the discussion period concludes, and we have not yet received any further feedback from three out of all four reviewers.

We have carefully addressed the concerns raised by all four reviewers through detailed clarifications, additional analyses, and corrections. In particular, we have:

  • Novelty and related work: Added direct comparisons with relevant methods and clarified the specific contributions of our work.
  • Training and evaluation claims: Provided additional seed level analyses, statistical evidence, and clarified the scope of our claims.
  • Reasoning modes and difficulty: Clarified the evaluation protocol and provided further analyses showing that the observed model weaknesses cannot simply be attributed to question difficulty.
  • Ground truth and metrics: Corrected the identified answer key issue, clarified expert validation, quantified disagreement, and provided further justification for our evaluation metrics.

We believe that all concerns raised in the reviews have been addressed through our responses and supporting evidence. We would sincerely appreciate it if you could take some time to review our rebuttals and let us know if any questions or concerns remain.

Given the limited time left, we would be grateful for any feedback during the remaining discussion period. If our responses have addressed your concerns, we would also appreciate your reconsideration of the initial assessments.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Summary of the discussion and revisions

Dear Reviewers, Area Chair, and Program Committee,

As the discussion period closes, we thank all four reviewers for the time and care they gave to InvariantBench. Below we summarize what each review valued, what concerns were raised, and how we addressed them.

What the reviews valued

  • Reviewer MAQt: the common evaluation protocol across diverse reasoning tasks, the measurable headroom between current models and human experts, and the strong expert performance that shows the transformations can be handled without loss of correctness.
  • Reviewer wmi6: that consistency across equivalent formulations is a useful complement to single-question accuracy, and that the diversity of tasks and transformation types enables analysis of how performance varies across reformulations.
  • Reviewer usfT: the strength of the core concept, claims proportionate to the evidence, the dataset size and its statistical power, full expert verification of the data, the two human cohorts, and the clarity of the writing.
  • Reviewer Lmm9: that cross-form consistency addresses an important limitation of accuracy-based benchmarks, and that the breadth of models, joint-correctness statistics, and failure analysis give useful evidence of representation sensitivity.

Concerns and how we addressed them

  1. Novelty relative to LGMT and Logifus (MAQt). We added both works to the related work and a property-level comparison. On the same six models, the accuracy-drop view of Logifus ranges from -2.7 to +4.2 points, while the gap between solving at least one form and all four ranges from 31.8 to 60.0 points.
  2. Reference-answer reliability (MAQt, Lmm9). The printed answer for the Appendix L item was wrong (17; correct value 252) and is corrected. The Human (Expert) majority differs from the key on 17 of 4,636 items (0.37%, 95% CI 0.23% to 0.59%). Because all four forms share one key, one mis-keyed seed shifts any metric by at most 0.086 points, equally for every model.
  3. Difficulty, sampling noise, and reasoning modes (wmi6, usfT, Lmm9). We stated one inference configuration for all 15 models. On the eight difficulty-neutral axes, 8.8% to 15.9% of base-correct answers break. With family I3 removed, the gap remains at 23.3 to 49.1 points for six models. Model consistency on seeds that the Base human cohort solves in all four forms differs from other seeds by -3.3 to +2.8 points with no consistent sign, and observed all-four correctness exceeds the independent-failure expectation for all 15 models.
  4. SFT/GRPO claims (MAQt, wmi6, usfT). We restated the claim as a measured gap on held-out forms (26.7 points for SFT, 11.0 for GRPO, 59.0% smaller; p = 0.002, n = 695 per form) with seed-level splits, and scoped it to one model and one training setup.
  5. Judge reliability and presentation (Lmm9, MAQt, usfT). We clarified the audit units (180 questions x 4 forms = 720 judged responses; 720 of 720 agreement with the three-annotator majority), fixed the broken table reference, corrected inconsistent numbers, rewrote Section 2 in plain language, and fixed the line spacing and the Figure 1 orientation.

Updates in the PDF. All of these changes are in the updated PDF on OpenReview, and every change is marked in blue so it can be checked quickly. We also took Reviewer usfT's suggestion to present this release as a first version of the benchmark that can be extended with more problems and finer difficulty calibration.

Thank you again to the reviewers for their feedback, which improved the clarity, scope, and evidence of the paper. We are grateful to the Area Chair and the Program Committee for organizing the discussion.

Sincerely, The Authors of Submission 6203

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.