- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces InvariantBench, a benchmark for evaluating the consistency of large language model (LLM) reasoning across semantically equivalent problem formulations. The benchmark contains 1,159 seed problems, each paired with three invariant variants, covering 16 task subtypes and 12 transformation axes across formal logic, quantitative reasoning, and relational and algorithmic reasoning. The authors evaluate 15 LLMs using both standard accuracy and several consistency-oriented metrics, including the Invariant Mass Index (IMI) and Conditional Invariance Reliability (CIR). The experimental results reveal substantial differences between solving at least one formulation and correctly solving all four equivalent formulations, even for strong models. The paper further provides an analysis of reasoning failures, human baselines, and exploratory experiments with supervised fine-tuning (SFT) and GRPO to investigate whether training improves cross-form generalization.
Critical strengths
1. A meaningful evaluation problem.
The paper addresses an important limitation of conventional reasoning benchmarks: a model's ability to solve a problem in one formulation does not necessarily imply that it can solve semantically equivalent formulations. Evaluating cross-form consistency alongside accuracy provides a useful complementary perspective on reasoning robustness.
2. Broad empirical coverage.
InvariantBench covers multiple reasoning domains and transformation types, and the evaluation includes 15 LLMs and human baselines. The joint correctness statistics and fine-grained failure analysis provide useful empirical evidence of representation sensitivity across different models and reasoning tasks.
Critical weaknesses
1. Concerns about benchmark correctness and semantic validation.
The reliability of the benchmark is a central concern. For example, Appendix L (Figure 5, page 56) includes a problem asking for the number of primes satisfying . The reference answer is reported as 17, whereas direct enumeration gives 252. This is a substantive ground-truth error in one of the paper's representative examples. Although a single error does not establish a systematic problem, it raises concerns about the effectiveness of the claimed verification pipeline.
Moreover, the semantic-preservation contract in Section 2 mainly defines the desired equivalence property, without establishing how equivalence is rigorously verified for all generated transformations, particularly abstraction shifts and natural-language reformulations. Since the benchmark's conclusions depend on the correctness and equivalence of its instances, stronger evidence of dataset reliability is essential.
2. Insufficient isolation of representational inconsistency from other factors.
The experiments do not adequately distinguish sensitivity to equivalent reformulations from inherent generation variability and differences in problem difficulty. In particular, the all-four correctness metric (IMI) is strongly influenced by a model's underlying accuracy. Even repeated evaluations of an identical prompt can produce substantially lower joint correctness than single-run accuracy. Without a repeated-sampling baseline using unchanged prompts, it is difficult to determine how much of the observed inconsistency is specifically attributable to representation changes.
Furthermore, some transformations, such as distractor injection and abstraction-level shifts, can change the difficulty of solving a problem despite preserving its answer. Consequently, the results demonstrate sensitivity to problem formulation, but do not fully substantiate the stronger interpretation that these failures necessarily indicate a lack of genuine reasoning or reliance on surface-level memorization.
Questions for authors
Q1. Ground-truth reliability.
Could the authors verify the reference answer for item 83994f0c8a9643d8 in Appendix L (Figure 5)? The reported answer is 17, whereas direct prime enumeration gives 252. How was this item validated, and have the authors performed an independent audit of the remaining ground-truth answers? An estimate of the dataset-level error rate would help assess the reliability of the reported results.
Q2. Semantic equivalence of transformations. How is semantic equivalence established for individual transformed instances, particularly abstraction-level shifts and natural-language reformulations? Is expert agreement the primary criterion, or are there additional independent verification procedures? Also, how are transformations involving variable or entity renaming handled when the correct output representation may need to change accordingly?
Q3. Separating invariance failures from sampling variability. Have the authors evaluated the same original question multiple times under identical prompting conditions? Such a baseline would help distinguish instability caused by representation changes from ordinary generation variability. It would also be useful to compare the reported joint correctness metrics against expectations conditioned on each model's marginal accuracy.
Q4. Reliability of automated evaluation. Could the authors clarify the relationship between the 240 validation samples described in Appendix E.2 and the 720 validation responses reported in Table 8? In particular, does the reported perfect judge-human agreement refer to individual human annotations or majority-vote labels? Clarifying the evaluation units and agreement calculations would strengthen confidence in the automated scoring procedure.
9 comments
Response to Reviewer Lmm9 (Part 2)
(W1 continued)
(d) Verification of semantic equivalence. The pipeline has three layers.
- Automated consistency checks across forms and deterministic recomputation of answers where an answer is computable (Appendices B and C).
- Review of each problem-variant set by three experts with three labels (accept / revise / reject), kappa = 0.78, and acceptance only after all three reviewers approved the final version (Appendix D.3).
- Independent solving of all 4,636 items by two human cohorts (Table R2): if a variant had changed the answer, expert accuracy on that form would drop below its Base accuracy. New analysis (Table R2): expert accuracy by form, from main Table 1.
| Cohort | Base | I1 | I2 | I3 | Range across forms (max minus min) |
|---|---|---|---|---|---|
| Human (Expert) | 99.9 | 100.0 | 99.2 | 99.4 | 0.8 points |
| Human (Base) | 90.0 | 90.4 | 90.1 | 89.6 | 0.8 points |
A non-equivalent variant (answer changed) would appear as a form-specific drop for both cohorts. The per-form range is 0.8 points for each cohort, and the I3 family, which contains abstraction shifts and distractors, is at 99.4% for experts. Across the 1,159 seeds, a form-specific equivalence error rate of 1% (about 12 seeds per form) would be visible as a drop of about 1 point in a cohort whose Base accuracy is 99.9%; the observed I1 to I3 values are 100.0, 99.2, and 99.4, with the lowest I2 value 0.7 points below Base. Experts and the Base cohort were shown items in shuffled order with at least 50 intervening items between variants of the same seed (Appendix E.3).
Revised text (Section 3, Quality control and verification):
"Verification has three layers: (1) automated cross-form consistency checks and deterministic recomputation of answers where computable (Appendices B and C); (2) review of every problem-variant set by three experts (accept / revise / reject, kappa = 0.78; acceptance after unanimous approval); and (3) answers from two human cohorts over all 4,636 items, in which the Human (Expert) majority matches the key on 99.9% of Base items and 99.6% of all items (17 disagreements in 4,636, 95% CI of the rate 0.23% to 0.59%). A shared reference answer cannot create inconsistency across forms: a key error shifts accuracy by at most 1/1,159 = 0.086 points per seed, equally for every model."
How this resolves the concern: the printed error is corrected and its effect on the 15 models is 1 judgment (0.02 points for one model); a shared wrong key cannot create Gap or CIR loss (0.086 points per seed, identical across models); and two human cohorts provide an existing check on keys and equivalence, with a measured disagreement rate and confidence interval (Table R1).
Response to Reviewer Lmm9 (Part 3)
- W2. Insufficient isolation of representational inconsistency from sampling variability and difficulty
- Response: This is the central methodological question, and we answer it with three analyses: the relationship between joint correctness and accuracy (Table R4), an identical-prompt baseline (Table R5), and the split between difficulty-neutral and difficulty-modulating axes (Table R6).
"The experiments do not adequately distinguish sensitivity to equivalent reformulations from inherent generation variability and differences in problem difficulty. In particular, the all-four correctness metric (IMI) is strongly influenced by a model's underlying accuracy."
IMI and accuracy. We agree with the observation. Across the 15 models, Spearman's rank correlation between mean per-form accuracy and IMI is 0.96. This is why the paper reports Gap, CIR, ICR, and the at-least-k profile alongside IMI, and why we add an accuracy-normalized ratio here.
New analysis (Table R4): joint correctness conditioned on each model's per-form accuracies. The largest IMI attainable given four per-form accuracies is the minimum of those accuracies. Define the nested-consistency ratio NCR = IMI / min(per-form accuracy). NCR = 1 means every seed solved on the hardest form is solved on all forms.
| Model | Mean accuracy (%) | IMI (%) | IMI expected if forms fail independently (%) | IMI / independent | Max attainable IMI (%) | NCR |
|---|---|---|---|---|---|---|
| GPT-5.2 | 68.1 | 47.6 | 21.5 | 2.21 | 66.6 | 0.71 |
| Qwen3.6-Plus | 66.0 | 47.7 | 18.9 | 2.52 | 64.6 | 0.74 |
| Qwen3-8B | 56.0 | 31.1 | 9.9 | 3.15 | 54.0 | 0.58 |
| Claude Haiku 4.5 | 48.5 | 17.4 | 5.5 | 3.15 | 47.5 | 0.37 |
| Gemini-3-Flash | 39.9 | 17.3 | 2.5 | 6.87 | 38.3 | 0.45 |
| Llama-3.3-70B | 25.6 | 5.7 | 0.4 | 13.3 | 24.2 | 0.24 |
| Human (Base) | 90.0 | 65.5 | 65.7 | 1.00 | 89.6 | 0.73 |
| Human (Expert) | 99.6 | 98.6 | 98.5 | 1.00 | 99.2 | 0.99 |
Observed IMI exceeds the independent-failure expectation for all 15 models (ratios from 2.21 to 618), so the joint-correctness figures are not what independent per-form sampling slips would produce. NCR ranges from 0.12 to 0.74 across the 15 models; its Spearman correlation with mean accuracy is 0.77, against 0.96 for IMI. The accuracy-normalized ordering differs from the accuracy ordering: Claude Haiku 4.5 (48.5% mean accuracy) has NCR 0.37, below Qwen3-8B (56.0%) at 0.58 and below Gemini-3-Flash (39.9%) at 0.45. Claude Haiku 4.5 and Gemini-3-Flash have IMI of 17.4% and 17.3% with Base accuracies 5.3 points apart, so the same all-four correctness comes from different accuracy and consistency mixes. The two highest-accuracy models reach NCR 0.71 and 0.74, in the same range as Human (Base) at 0.73 (accuracy 90.0%), while Human (Expert) is at 0.99.
Sampling variability. Sampling noise has no preferred direction and acts equally on every axis. Reformulation effects are directional and axis-specific. Table R5 gives breakage by axis, defined as Base accuracy minus axis accuracy, averaged over the 15 models (per-axis results in the supplementary material). All responses were generated with one call per item at temperature 0.1 (Appendix E.1).
Table R5: mean breakage by axis, 15 models (points; positive = accuracy lower than Base).
| Axis | Family | Mean breakage |
|---|---|---|
| Variable Renaming | I1 | +4.43 |
| Entity Relabeling | I1 | +3.60 |
| Premise Reordering | I2 | +3.58 |
| Perspective / Role Reframing | I3 | +2.66 |
| Redundant Restatement | I3 | +2.59 |
| Style / Register | I1 | +2.43 |
| Abstraction Level Shift | I3 | +1.62 |
| Distractor Injection / Removal | I3 | +0.08 |
| Equivalent Mathematical Reformulation | I2 | -0.34 |
| Representation / Encoding Swap | I2 | -0.59 |
| Equivalent Logical Reformulation | I2 | -1.13 |
| Synonym / Semantic Paraphrase | I1 | -3.21 |
Under sampling noise alone every axis would center on 0. Instead the 12 axes span 7.64 points. Three axes that keep the wording and content but change identifiers or order lose 3.58 to 4.43 points, while four axes that rewrite the content show a mean change of 0.08 points or an increase. A repeated-prompt baseline cannot generate this ordering, since it would apply the same noise to every axis.
Response to Reviewer Lmm9 (Part 1)
Thank you for recognizing that cross-form consistency is a meaningful complement to accuracy, and that the joint-correctness statistics and failure analysis give empirical evidence of representation sensitivity across 15 models and two human cohorts. Your review also did something we are grateful for: you enumerated the prime-count item in Appendix L, found that the printed reference is wrong, and asked how equivalence and judge agreement are established. Each of your four questions is answered below with data from the paper, new analyses computed from the paper's tables and the per-item scores (labeled Table Rn), and the exact corrected text. We have also highlighted all changes in the revised manuscript in blue.
Weaknesses
83994f0c8a9643d8asks for the number of primes p <= 100,000 with p mod 41 = 23. Direct enumeration over the 9,592 primes up to 100,000 gives 252, as you computed, and the printed reference of 17 is incorrect. We corrected the reference to 252 in the dataset and in Figure 5. We then examined what an error of this kind does to the reported results and what independent evidence exists on key and equivalence correctness.(a) How a key error propagates through the metrics. The four forms of a seed share one reference answer by construction (Section 2). A mis-keyed seed is therefore scored against the same wrong value on Base, I1, I2, and I3.
Table R1: disagreement between independent answers and the key, from existing data.
The 17 disagreements are spread as 1 on Base, 0 on I1, 9 on I2, and 7 on I3. This combined rate includes key errors, equivalence errors, and annotator errors, so it is an upper bound on the key-error rate of 0.37% of items (1.38% of seeds, upper CI 2.23%), applied identically to every model. The disagreements concentrate on I2 and I3 (16 of 17), the forms in which an equivalence error would appear; per form they are 0.09% (Base), 0.78% (I2), and 0.60% (I3) of items, so equivalence errors are bounded by those rates. The seeds in this list are the first audit set. Item
83994f0c8a9643d8was the item you identified; after correction its reference is 252.For scale, a seed-level key-error rate of 1.38% would change a model's accuracy by at most 1.38 points, equally for all models, and cannot create Gap or CIR loss (part a).