acceptodds
Under review as a conference paper at ICLR 2027

Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

Abstract

Language model benchmarks assign each question a single outcome from one fixed wording, even though the same underlying request can be expressed in many equivalent ways. We argue that the quantity of interest should therefore not be correctness on one wording, but the *accessibility* of an inquiry, which we define as the probability that a model succeeds across meaning-preserving formulations. The distribution of accessibility across inquiries reveals a structure that accuracy and existing best-, worst-, or average-case prompt scores collapse: how broadly a model’s capabilities extend across inquiries and how dependably those capabilities can be elicited. Two models can have the same average performance while one succeeds across more inquiries unreliably and the other succeeds across fewer inquiries but more consistently, leading to opposite preferences depending on whether the task allows multiple retries until one success or when repeated consistent success is required. Across six benchmarks and more than twenty models, this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model looks better depends on whether one success across five formulations is enough or all five must succeed, quantities we call Access@5 and Reliable@5, respectively. For some pairs the two measures diverge significantly, by up to 10 points in opposite directions. Accessibility also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries in ways aggregate accuracy obscures, sometimes broadening the set of inquiries a model can succeed on while making success on them less repeatable. Accessibility turns formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes succeed from what users can actually depend on.

Primary area:
datasets and benchmarks
Keywords:
LLM, LLM Evaluation, LLM Reliability, LLM Capabilities

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P5mqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most existing LLM benchmark evaluations adopt only a single fixed wording for each question and compare models using a single accuracy score. Nevertheless, one semantic inquiry can have multiple semantically‑equivalent formulations, and models exhibit substantial performance discrepancies across these rephrased variants. This paper introduces the concept of accessibility: the probability that a model answers correctly over all semantically‑equivalent rephrasings of the underlying inquiry. An accessibility profile is constructed based on the accessibility distribution across all inquiries. Two key derived metrics are further proposed: Access@k, which measures success in at least one out of k formulations (suited for deployment scenarios where retries are permitted), and Reliable@k, which requires correct answers across all k formulations (for settings demanding reliable outputs on every invocation). A corresponding theoretical proposition is presented: given two models with equal mean accessibility, crossing reliability curves can lead to completely contradictory model rankings produced by Access@k and Reliable@k.

Critical strengths

It points out the inherent flaw of single‑query evaluation in traditional benchmarks: accuracy obtained from fixed wording is merely one sample point from the model’s capability distribution, and cannot fully characterize the model’s true capability for the underlying inquiry. This perspective offers strong insights for the LLM‑evaluation community.

Controlled comparative experiments are designed to demonstrate that correctness fluctuations induced by equivalent rephrasings are substantially larger than output fluctuations caused by temperature sampling. This shows that accessibility captures prompt‑formulation effects rather than mere generative randomness.

Critical weaknesses

All formulations are generated by GPT‑4.1‑mini. Though semantic validation is performed, there exists a distribution shift relative to real‑world user‑authored question rephrasings. Accessibility is therefore an estimate over a synthetic rephrasing set and may not equate to model behaviour under genuine user inputs.

The paper mixes syntactic rephrasing, sentence‑structure transformation and stylistic rewriting without conducting corresponding ablation studies. It remains unclear which category of rephrasing most readily degrades accessibility; consequently, it cannot pinpoint whether model fragility primarily stems from syntactic alterations or tonal‑style variations.

There is no comparison of how accessibility and the fraction of brittle‑sample change across base models, SFT‑aligned models and RLHF‑aligned models. No experimental evidence is provided regarding whether alignment raises or lowers Reliable@k.

Questions for authors

The formulations in the paper are model‑synthesized rephrasings. To what extent would the findings for the accessibility profile, Access@k and Reliable@k change if a dataset of real human‑user rephrasings were adopted?

The paper proposes a low‑cost approximation scheme: using two rephrasings for half of the inquiries. For different tasks, how should one strike the optimal trade‑off between the number of rephrasings and the number of inquiries? Are there practical reference thresholds that can be followed?

2 comments
AuthorsOpenReview ↗

Author response (1/2): The same meaning-preserving formulations for every model give a paired, conservative comparison, and sensitivity appears under both rewrite types

We thank the reviewer for the constructive and detailed review, and for recognizing that single-wording accuracy captures only one point of a model's capability and that our controlled comparison separates formulation effects from sampling noise. We answer each point below. The new analyses are in Appendices J and M of the revised paper.

1. Meaning-preserving formulations isolate the effect of wording, and real user phrasings could only add brittleness

Our goal is to measure how models respond to different formulations of the same inquiry. While synthetic paraphrases may not reflect the frequency of real user phrasings, they allow us to study this variation under controlled conditions. Three design choices make the comparisons sound:

  • The same inquiry for every model. All models answer the same inquiries and the same formulations, so differences between models never come from different test items.
  • Semantic equivalence. Every kept formulation passes LLM equivalence judges (two must agree on GSM8K, MATH-500, MMLU, and BFCL) and, except on BFCL, bidirectional NLI. Annotators confirm that at least 96.0% of the analysed formulations preserve the original meaning (Table 1).
  • Real variation. Formulations are generated under 15 structural and stylistic cues or with batched prompts that require different wordings, and near-verbatim rewrites are removed by a lexical-overlap cap (on BFCL, the most diverse accepted paraphrases are selected). They do not come from a single generator: on MMLU, 55% were generated by Gemma-3 27B and the rest by GPT-4.1-mini.

Our central findings do not require the formulations to match real-user frequencies. The experiments establish that similar average accuracy can hide different accessibility profiles, that Access@k and Reliable@k can rank models oppositely, and that model changes can redistribute capability across inquiries without large changes in accuracy. These effects arise even under careful, human-validated rewrites, while real user phrasing varies more widely (typos, added context, different vocabulary).

The formulations do not simply make the benchmarks harder, and formulation sensitivity runs in both directions. Accuracy and mean accessibility rank models almost identically (Kendall's $ \tau $ 0.90–0.99). Yet individual answers change both ways: on TruthfulQA, GSM8K, MATH-500, and BFCL, 14–36% of the formulations of canonically failed inquiries are answered correctly (median over models), while other inquiries solved in their canonical wording fail under alternative wordings. Changing the wording redistributes success across inquiries rather than only lowering it.

The exact accessibility values and operational metrics depend on the formulation distribution. We acknowledge this in Section 6, and the framework applies unchanged to any formulation distribution, including one weighted by real-user frequency. Our brittleness estimates are therefore a reproducible lower bound: the formulation sets are fixed and released, and over any larger set of valid formulations, including real user phrasings, the brittle share can only be higher and the reliable share only lower.

2. The requested ablation shows formulation sensitivity under both structural and stylistic rewrites

We conducted the requested ablation and added it as Appendix M. We compared structural and stylistic rewrites within each model–inquiry pair, controlling for inquiry difficulty.

BenchmarkStructural successStylistic successDifference [95% CI]
SimpleQA5.4%5.3%+0.1 [−0.1, +0.3]
TruthfulQA43.6%43.9%−0.3 [−1.2, +0.6]
GSM8K84.4%84.7%−0.3 [−0.9, +0.4]
MATH-50070.8%70.4%+0.4 [−0.6, +1.5]
MMLU68.2%69.2%−0.9 [−1.5, −0.4]

Neither category consistently degrades performance more. On SimpleQA, TruthfulQA, GSM8K, and MATH-500 the differences are within 0.4 percentage points and every confidence interval includes zero. On MMLU, structural rewrites are slightly less successful (−0.9 points), yet the brittle share measured with two formulations is higher under stylistic rewrites (16.4% vs. 14.2%). On the other benchmarks the brittle shares are similar across the two categories (BFCL formulations carry no cue). Among 73 individual cue–benchmark comparisons, only five differ significantly after Bonferroni correction, each by at most 1.7 points, in both directions and across both categories.

These results suggest that neither structural nor stylistic rewriting consistently explains greater formulation sensitivity, which occurs under both types. Since cues were assigned during generation rather than randomized, we interpret these as observed associations, not causal effects.

AuthorsOpenReview ↗

Author response (2/2): Our framework applies directly to alignment stages as a further application, and the best budget split depends on the target metric

3. Alignment studies are another application of our framework

Our framework measures how model capabilities vary across inquiries and equivalent formulations. We use it to compare models, distinguish wording effects from decoding variability, and study model changes. Evaluating specific training methods is not the paper's objective.

Section 5 already applies this framework to 119 model changes across scaling, native thinking, chain-of-thought prompting, and prompt optimization. These experiments reveal gains and regressions that aggregate accuracy can hide. For example, enabling native thinking in Qwen3 changes MMLU accuracy by only −3.0 points, while 25% of inquiries become more accessible and 32% less accessible, and the reliable share drops by 10.9 points.

Comparing base, SFT, and RLHF models is a valuable use of the same framework. Access@k, Reliable@k, and the brittle share can reveal whether alignment improves reliability, broadens capability, or redistributes success across inquiries. A controlled comparison would require matched checkpoints derived from the same base model and careful control of differences in prompting and format-following. We have added this direction to the Discussion as future work.

4. Allocating a fixed budget: the best split depends on the target metric

The best allocation depends on what we want to estimate. We extended Appendix J with a practical sampling analysis and evaluated the recommendations through simulations.

1. Mean accessibility. With a fixed budget of BB model calls, evaluating KK formulations per inquiry leaves n=B/Kn=B/K inquiries. Under independent sampling of inquiries and formulations, the variance of estimated mean accessibility is

\frac{K\operatorname{Var}_q(A_q)+ \mathbb E_q[A_q(1-A_q)]}{B}.$$ Thus, for estimating the mean alone, **covering more inquiries is generally preferable to evaluating more formulations of fewer inquiries**. **2. Access@k and Reliable@k.** These require at least $k$ formulations per inquiry. Choosing $K=k$ is the cheapest design that identifies the target quantity and maximizes inquiry coverage at a fixed budget. It is the most efficient design. **3. Practical budget thresholds.** Our simulations suggest the following guidance for estimating fixed-$K$ metrics within ±5 percentage points in at least 95% of simulated subsamples: - **Benchmarks with 600–902 inquiries:** $K=2$, evaluating half the inquiries at the cost of one conventional benchmark run, works well for SimpleQA, GSM8K, MMLU, and BFCL. - **TruthfulQA (471 inquiries):** This setting is borderline. Mean accessibility is estimated reliably, but Access@2 and Reliable@2 meet the target for only about half the models. - **MATH-500 (106 retained inquiries):** The same budget is generally insufficient for reliable estimates across models. Evaluating the full inquiry set is preferable. - **Larger $K$:** At a fixed budget of model calls, evaluating more formulations per inquiry means evaluating fewer inquiries, so precision worsens. We also derive an approximate sample-size rule for inquiry-level proportions: $$n_{\mathrm{req}}\approx \frac{z^2p(1-p)/\epsilon^2} {1+z^2p(1-p)/(\epsilon^2N)},$$ where $p$ is the expected proportion, $\epsilon$ the desired margin of error, and $N$ the benchmark size. This uses a finite-population correction for sampling inquiries, and the indicator of each sampled inquiry already includes the random choice of its formulations. In our simulations, the planning rule correctly predicts whether the target precision is met in **94.7% of tested settings**. **4. Full accessibility profiles.** Estimating the empirical reliable, latent, and brittle shares over the complete formulation pool is different from estimating Access@2 or Reliable@2. Two formulations cannot recover these full-pool endpoint shares. We therefore recommend evaluating all available formulations for a sampled subset of inquiries when these are the target quantities (on our benchmarks, 211–267 inquiries suffice for ±5 points, 0.9–3.8 times the cost of one conventional run). **Practical recommendation:** First choose the target metric. Use the minimum number of formulations needed as an initial budget allocation, prioritize inquiry coverage, and add formulations only when the full profile is the target. The appendix provides task-specific results rather than assuming one allocation is optimal for every benchmark.
Reviewer_otKjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Accessibility Profiles, a new framework for evaluating LLM capabilities beyond traditional benchmark accuracy. Instead of testing each question with a single wording, it measures how consistently a model answers correctly across multiple semantically equivalent formulations. The framework captures both capability breadth (how many questions a model can potentially answer) and reliability (how consistently it answers them correctly). Experiments across six benchmarks and over 20 models show that models with similar accuracy can differ substantially in reliability. The paper also demonstrates that prompt variations affect performance more than decoding randomness, and that model scaling, reasoning, and prompt optimization can redistribute capabilities in ways that traditional accuracy metrics fail to capture.

Critical strengths
  1. More comprehensive comparison of models with similar accuracy

  2. Reveals hidden regressions during model updates

  3. Provides deployment-specific evaluation metrics

  4. Addresses an important research problem with a more comprehensive framework

Critical weaknesses
  1. Latent Share may be substantially inflated by random guessing

    • The paper defines Latent Share as the proportion of inquiries answered correctly under at least one formulation. However, a single correct answer may result from random guessing rather than genuine capability.
    • For example, MMLU has four answer choices, giving a random-guessing accuracy of 25%. The paper reports a median of seven formulations per inquiry and a median Latent Share of 87.2%.
    • Under independent random guessing, the probability of obtaining at least one correct answer across seven attempts is:

    1 − (1 − 0.25)⁷ = 86.65%

    • This theoretical random baseline is remarkably close to the reported 87.2%. Although this does not imply that the evaluated models are guessing randomly, it demonstrates that at least one correct response is insufficient evidence of latent capability.
  2. Access@k has limited practical utility without a verification mechanism

Access@k measures the probability of obtaining at least one correct answer across multiple attempts, but does not account for whether the correct answer can be identified. Even if one out of five responses is correct, the result provides little practical benefit if the system cannot determine which response is correct. Therefore, Access@k primarily measures potential accessibility rather than directly usable performance, unless an external verifier or reliable answer-selection mechanism is available.

  1. Perfect performance on a limited number of formulations does not establish true reliability
    • The paper classifies an inquiry as Reliable when all tested formulations are answered correctly, although typically only around 7–10 formulations are evaluated per inquiry. Answering 7 out of 7 formulations correctly does not imply a true success probability of 100%. Additional formulations may reveal failures in previously Reliable inquiries.
    • Consequently, the reported Reliable Share may overestimate true reliability and is sensitive to both the number and distribution of tested formulations.
Questions for authors

This paper has several serious evaluation issues:

  1. Incorrect grading affects the analyzed data. For two GSM8K questions, the released grader rejects the correct answers 6 and 41 while accepting the incorrect answers 4 and 5, respectively. The corresponding correct model responses are labeled incorrect and included in the final analysis cohort. This raises concerns about the reliability of the reported metrics and model comparisons.

  2. The implemented metric does not match the theoretical definition. The paper’s proof assumes that the canonical question is included among the evaluated formulations, but the implementation excludes it from the accessibility calculation. 820 records violate the claimed per-question containment relationship. Although all 122 aggregate inequalities still hold, the stated proof does not justify the implemented metric.

  3. The human-validation statistics are inconsistent with the stated counting procedure. With sample sizes of 460, 232, and 380, the reported percentages of 96.9%, 98.0%, and 97.7%, respectively, cannot result from rounded integer success counts under an unweighted binary calculation. The authors need to provide the annotations and clarify the aggregation method.

4 comments
AuthorsOpenReview ↗

Author response (1/3): Guessing does not explain the findings, and Access@k rests on the premise of every benchmark

We thank the reviewer for the careful review and for checking our code and data in detail. We address the six points in the order of the review. Each answer points to the revised paper, and the updated supplement includes scripts that let the reviewer verify every claim directly.

1. Random guessing does not explain the MMLU results, and the findings hold after chance correction

This point applies primarily to MMLU, the only multiple-choice benchmark among our six datasets. SimpleQA, TruthfulQA, GSM8K, and MATH-500 require free-form answers, and BFCL evaluates structured tool calls. None of them has a comparable uniform four-choice guessing baseline.

The observed MMLU results differ substantially from independent random guessing. Responses are generated with greedy decoding, so each verdict is a deterministic function of the wording, not an independent random draw. Although the reported latent share is close to the reviewer's guessing baseline, the similarity does not extend to the rest of the accessibility profile (medians over 20 models):

  • Among inquiries answered incorrectly under the canonical wording, independent guessing predicts at least one correct paraphrase in 86.7% of cases, compared with 57.8% observed.
  • Guessing predicts 86.6% brittle inquiries, compared with 37.0% observed.
  • The probability of answering all seven formulations correctly by guessing is only 0.006%, compared with an observed reliable share of 46.1%. Reliable@5 under guessing is also only 0.1%.

We further tested whether our findings depend on chance-level successes (new Appendix O). On MMLU, we applied two alternative measures: a chance-adjusted accessibility score, and a stricter criterion requiring at least 5 of 7 correct answers.

MetricReportedChance-adjustedAbove-chance criterion
Latent share87.3%81.5%67.0%
Brittle share37.0%30.2%16.0%
Reliable share46.1%46.1%46.1%

The chance-adjusted score is s=max⁡(0,(A^q−0.25)/0.75)s=\max(0,(\hat A_q-0.25)/0.75) under a knows-or-guesses model. The stricter criterion corresponds to a one-sided binomial test at p<0.05p<0.05 against uniform guessing. Under both alternatives, substantial formulation sensitivity remains.

The model-selection findings also remain. After chance adjustment, LLaMA-3 8B is 8.7 points higher than LLaMA-3.1 8B in Reliable@5 ([5.0, 12.5]) and 6.5 points lower in Access@5 ([−10.3, −2.4]), so the two models still receive opposite rankings. The same phenomenon occurs on BFCL, where the four-choice guessing argument does not apply: GPT-5.4-Mini leads Qwen-3 1.7B by 9.6 points in Reliable@5 ([5.5, 14.0]) but trails it by 6.8 points in Access@5 ([−10.8, −2.8]).

Finally, our definitions are explicitly behavioral, not claims about internal knowledge. Section 3.3 defines latent as success on at least one tested formulation. Our framework measures observable accessibility and reliability, and the additional analyses show that its main findings cannot be explained by the uniform-guessing baseline.

2. Access@k rests on the same premise as every benchmark metric, and Section 6 discusses when it applies

Access@k is an evaluation quantity, defined like every benchmark metric: against reference answers. Knowing how to tell a correct answer from an incorrect one is the premise of benchmarking itself, and single-attempt accuracy and pass@k rest on it as well. Whether a deployed system can identify which of its attempts is correct is a separate, practical question, and our paper addresses it directly. Section 6 ("Deployment regimes and pipeline design") states:

When failures are cheap to detect and retry (a verifier, a test suite, or a human checks the output), the relevant quantity is Access@k; when failures go undetected or outputs feed later components automatically, it is Reliable@k.

Thus, we do not assume that Access@k is directly achievable in every deployment. The appropriate measure depends on the application and on whether successful attempts can be identified.

Access@k is also informative without a perfect verifier. For a system selecting one of kk responses, random selection yields expected accuracy μ\mu, an oracle achieves Access@k, and any practical selector (majority vote, a reward model, a user's judgment) falls in between. The gap Access@k−μ\mathrm{Access@}k-\mu is the maximum possible improvement over random selection.

The key contribution is that these requirements can favor different models. Proposition 1 characterizes this theoretically, and RQ1 shows that 50% (59/118) of near-tied model pairs have opposite Access@5 and Reliable@5 rankings. The availability of verification therefore determines how Access@k translates into deployment performance, not whether measuring it is useful.

AuthorsOpenReview ↗

Author response (2/3): Reliability is measured empirically on the same validated formulations for every model, and the grading corrections change no conclusion

3. Reliability is an empirical measurement on the same validated and varied formulations for every model, and more formulations can only reveal more brittleness

Our paper treats reliability labels as empirical, behavioral measurements over the tested formulations, not guarantees across all possible wordings. We do not claim that answering all tested formulations correctly establishes perfect reliability. Section 3.2 states that "DqD_q is generally unknown, so we approximate it with a uniformly weighted set of validated formulations," and Section 3.3 states:

These labels are empirical: reliable means success on every formulation we tested, and absent means failure on every formulation we tested, not necessarily across the entire formulation space.

The Limitations section adds:

With more formulations, an inquiry can only move from reliable or absent to brittle, so the reported brittle shares are lower bounds with respect to the formulation distribution.

Testing more formulations can only reveal additional brittleness, not remove it. When new valid formulations are added for the same inquiries, Rm+1≤RmR_{m+1}\le R_m, Lm+1≥LmL_{m+1}\ge L_m, and Bm+1≥BmB_{m+1}\ge B_m, where RR, LL, and B=L−RB=L-R are the empirical reliable, latent, and brittle shares. These follow exactly from the definitions.

Our main model comparisons do not require any inquiry to be perfectly reliable. They use Access@k and Reliable@k over a fixed number of attempts k≤mk\le m. For cc successes among mm tested formulations, (ck)/(mk)\binom{c}{k}/\binom{m}{k} is exactly the probability that kk formulations drawn uniformly without replacement all succeed, and under independent sampling from the target distribution it is also unbiased for AqkA_q^k (likewise for Access@k). All models are evaluated on the same validated and varied formulations of the same inquiries, so the comparisons are direct and controlled. Our findings therefore measure observable differences that single-wording accuracy cannot capture, without requiring universal reliability.

Grading, canonical wording, and human validation. We checked all three points against our code and data, corrected the identified errors, and recomputed the affected analyses. The framework, including its definitions, estimators, and theoretical results, is unchanged, and the central conclusions hold.

4. The grading corrections change at most 1.2% of verdicts and no conclusion

Both examples identified by the reviewer are valid. Our GSM8K grader passed the full worked solution to the answer checker rather than the final answer after ####, which misread the reference on 4 of 1,319 items (3 in our analysis set). We corrected this with the final-answer rule of lm-evaluation-harness. We also audited the broader grading pipeline and corrected two further extraction issues: boxed answers containing words or units (e.g., \boxed{41 points}) and case-insensitive letter matching on MMLU.

Scope and verification. The corrections change 1,807/157,458 GSM8K verdicts (1.15%), 36/19,593 MATH-500 verdicts (0.18%), and 146/101,440 MMLU verdicts (0.14%). SimpleQA, TruthfulQA, and BFCL are unaffected. All but one math change are incorrect-to-correct. The exception is a response whose final answer is 28 (days) for a question asking for weeks (reference 4), and like lm-evaluation-harness we compare the final number in the requested unit. On MMLU, 145 of 146 changes are incorrect-to-correct. We manually inspected 66 changed math verdicts (65 genuine) and all 146 changed MMLU verdicts (all genuine).

To independently assess the corrected grader (new Appendix N), we regraded 1,500 randomly sampled verdicts per math benchmark with a separate model and inspected every disagreement. Grading accuracy is 99.7% on GSM8K (error 0.33%, 95% CI [0.14, 0.78]) and 99.4% on MATH-500 (error 0.60%, [0.32, 1.14]).

Impact on conclusions. No conclusion changes, and the revised manuscript lists every number that moves (Appendix P). For example, Access@5 and Reliable@5 order 59 of 118 near-tied pairs oppositely (previously 62), and GSM8K benchmark medians move by about 0.3 points. To check whether remaining grading errors could change our findings, we deliberately corrupted the grades and reran the full analysis in two ways (Appendix N):

  • Random errors: we flipped 1%, 2%, or 5% of all verdicts at random, 2 to 15 times the measured error rate.
  • Errors like the original bug: for the three models the bug affected most, we marked 5%, 10%, or 20% of their correct answers as incorrect.

Across 24 runs, Access@5 and Reliable@5 still order 47–59% of near-tied pairs oppositely, large reversals with both gaps significant remain (3–11, against 6 in the paper), and R≤Acc≤LR\le\mathrm{Acc}\le L holds for all 122 model–benchmark pairs. verify/check_q1_grading.py in the supplement regrades every changed response.

AuthorsOpenReview ↗

Author response (3/3): Holding out the canonical wording is conservative and reaches the same conclusions, and human validation confirms the formulations

5. Holding out the canonical wording is the conservative choice, and both designs reach the same conclusions

The estimator is implemented as intended. One sentence describing it was wrong. Section 5.1 states that "the canonical formulation is among the evaluated formulations" and uses this to argue that every inquiry satisfies R≤Acc≤LR\le\mathrm{Acc}\le L. Our implementation, by design, does not include the canonical wording, so that sentence is incorrect. We have corrected it.

No definition, estimator, or proof depends on the canonical wording. Accessibility is defined over meaning-preserving formulations (Eq. 1) and estimated as the mean success over the validated alternative formulations (Eq. 2). The proofs (Appendix A) concern the distribution of accessibility across inquiries and never use the canonical wording. The containment argument in Section 5.1 was a remark, not part of any proof.

Why we hold the canonical wording out. The canonical outcome is what a standard benchmark reports, and RQ1 compares it with accessibility estimated from the other wordings. Counting it on both sides would make that comparison partly circular. The canonical wording is also one fixed, benchmark-specific phrasing, with no reason to treat it as representative.

What the 820 cases are. The reviewer counted 820 model–inquiry records where per-inquiry containment fails. After the grading correction there are 818: in 396 the canonical wording succeeds but every alternative fails, and in 422 the canonical wording fails but every alternative succeeds. These are not errors. They are the cases where the benchmark's own wording behaves differently from every other wording, which is exactly what the paper measures. As the reviewer notes, R≤Acc≤LR\le\mathrm{Acc}\le L still holds for all 122 model–benchmark pairs. We now present it as an observed result.

With the canonical wording included (new Appendix L): violations drop from 818 to 0, benchmark medians move by at most 2.5 points, the brittle share rises on every benchmark (so our held-out numbers are the conservative ones), and the model comparisons are unchanged: 59 of 117 near-tied pairs are ordered oppositely (59 of 118 held out), with 6 large significant reversals under both designs. We also state the hold-out in Section 3.2. verify/check_q2_canonical.py reproduces these counts for the submitted and revised data.

6. 96.0–97.8% of the formulations in the final analysis are human-equivalent

The issue was in how we reported the numbers, not in the human annotations. Five annotators labeled 1,252 pairs: SimpleQA (460), TruthfulQA (232), GSM8K (380), and MATH-500 (180). We used these labels to compare equivalence judges and select rules that prioritize high semantic precision, even at the cost of rejecting some valid paraphrases.

Where the original numbers came from:

  • SimpleQA (96.9%) and TruthfulQA (98.0%) combined the human rejection rate with the fraction of answers to human-rejected paraphrases that passed the answer-grading filter: 1−(16/460)(270/304)1-(16/460)(270/304) and 1−(7/232)(88/133)1-(7/232)(88/133). This does not directly measure the precision of accepted paraphrases.
  • GSM8K (97.7%) came from an earlier equivalence-judge comparison, in which 172 of 176 accepted paraphrases were human-equivalent.

We recomputed the results directly from the human labels and the formulations used in our experiments (revised Table 1):

BenchmarkPassing equivalence checksIncluded in final analysis
SimpleQA444/460 = 96.5%261/272 = 96.0%
TruthfulQA187/192 = 97.4%174/179 = 97.2%
GSM8K142/146 = 97.3%134/137 = 97.8%
MATH-500119/123 = 96.7%110/113 = 97.3%

Why the final sets are smaller. To compare models fairly, we keep only formulations that every model answered with a gradable response, and only inquiries with at least five such formulations, so all models are compared on the same inquiries and formulations. This excludes some valid paraphrases.

96.0–97.8% of the annotated formulations in our final analysis are human-equivalent. This directly checks the semantic validity of the formulations our results are computed on. Because the annotations were also used to select the validation rules, these values describe the annotated sample rather than an independent held-out set. The supplement provides the annotations, every pipeline decision for each pair, the final formulation sets, the exact prompts, and verify/check_q3_human_validation.py, which recomputes this table. Appendix B now documents the formulation protocol of each benchmark.

Thank authors for the detailed rebuttal. You have addressed all my concerns.

Reviewer_K3TVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates large language models on the original wording and several paraphrases of each benchmark question. The authors estimate each question's accessibility as the fraction of tested wordings that the model answers correctly. They use these results to distinguish questions a model can sometimes answer from those it answers correctly across all tested wordings. They compare the probability of at least one correct answer across k independently sampled wordings of the same question (Access@k) with the probability that all k answers are correct (Reliable@k), averaged over questions. Experiments on six benchmarks with more than twenty models show that models with similar mean accessibility can rank differently under these measures. Experiments on model size, reasoning, and prompt optimization show that gains on some questions and losses on others can be hidden by a small change in average accuracy.

Critical strengths

(+) I found the comparisons between models with similar mean accessibility useful. In particular, the reversals between Access@5 and Reliable@5 show how the preferred model can change depending on whether one correct answer is enough or all five answers must be correct.

(+) The experiments cover six benchmarks and more than twenty models. The comparison with repeated sampling of the same prompt is especially helpful, since it tests whether the differences come from rephrasing the question. Human checks test whether the paraphrases preserve meaning, and the experiments on model changes show how gains on some questions can be offset by losses on others.

Critical weaknesses

(-) The paraphrase construction procedure substantially overlaps with the 2026 paper by Faghih and colleagues (https://arxiv.org/abs/2607.22554). Both use GPT-4.1-mini, the same prompts for structural and stylistic paraphrasing, a threshold of 0.8 for word overlap, and a BART-MNLI entailment threshold of 0.75 in both directions. The models used to check semantic equivalence differ from the earlier paper's main setup. The earlier paper is cited in Related Work, but Appendix B should explain which steps follow that work and which differ.

(-) My main concern is the limited contribution beyond that work. Faghih and colleagues already report the share of questions answered correctly under at least one tested wording (A_any) or every tested wording (A_strict). If computed on the same wordings and scored responses, these correspond to the empirical latent and reliable shares used here. The reliability curve and metrics for a fixed number of attempts add detail to this distinction. Proposition 1 relates curve shapes to model rankings. The additional experiments compare wording changes with repeated sampling and examine the effects of model size, reasoning, and prompt optimization. In my view, these additions are not substantial enough to justify acceptance.

Questions for authors
  1. What are the main contributions beyond Faghih and colleagues? Please clarify which findings are new and whether any paraphrases, model responses, or human annotations were reused.

  2. Section 3.2 approximates the distribution of wordings with a validated set. Under what sampling assumptions are the estimators in Section 3.3 unbiased for the probabilities in Equations 4 and 5? Please clarify how the generation procedure and inclusion of the original wording satisfy those assumptions.

2 comments
AuthorsOpenReview ↗

Author response (1/2): This paper asks a new question and makes new contributions beyond Faghih et al. (2026)

We thank the reviewer for the careful comparison with Faghih et al. (2026) and for noting the value of the Access@5/Reliable@5 comparisons and of the repeated-sampling control. Below we state what is new, what is reused, and the assumptions behind our estimators.

1. This paper asks a new question and adds new theory, measures, and findings

Faghih et al. (2026) established that correctness varies across equivalent formulations and introduced AanyA_{\text{any}} and AstrictA_{\text{strict}}, corresponding to our empirical latent and reliable shares. We do not claim these endpoint measures or the underlying paraphrase-generation procedure as new.

The main difference is the question we address. The earlier work showed that benchmark accuracy can hide inconsistency across formulations. Our paper goes further: even knowing a model's average, best-case, and worst-case performance is not enough to understand how its capability is distributed across inquiries, or which model is better for a given application.

Our contributions address this gap in five ways:

1. A richer view of model capability. We treat each inquiry's accessibility AqA_q, its probability of success across equivalent formulations, as the quantity of interest. Accessibility profiles and reliability curves reveal how success is distributed across inquiries, including those that are neither always correct nor always incorrect. This captures information that average, latent, and reliable shares alone cannot reveal.

2. New theory and operational measures. We derive Access@k and Reliable@k from the accessibility distribution and establish Proposition 1, which characterizes when models must receive opposite rankings under these objectives. These measures connect evaluation to different practical requirements: whether at least one successful attempt is enough or every attempt must succeed.

The earlier summary metrics cannot determine these preferences. Consider two models with accessibility profiles:

  • Model A: (0,0.2,0.8,1)(0, 0.2, 0.8, 1)
  • Model B: (0,0.5,0.5,1)(0, 0.5, 0.5, 1)

Both have identical mean accessibility (0.50), latent share (0.75), and reliable share (0.25). Yet A has higher Reliable@2 (0.420 vs. 0.375), while B has higher Access@2 (0.625 vs. 0.580). The earlier measures cannot distinguish these models, but their accessibility profiles reveal different choices for different requirements.

3. New findings about model comparison and deployment. We show that stable aggregate rankings can be misleading. Although accuracy and mean accessibility rank models similarly overall, approximately half of near-tied model pairs reverse their ordering between Access@5 and Reliable@5. These ranking reversals and their implications for model selection, also highlighted as useful by the reviewer, were not established in the earlier work.

These distinctions matter for deployment. For example, two BFCL models with nearly equal mean accessibility require 1.92 versus 1.50 attempts per solvable inquiry (about 28% more calls), while the model requiring more attempts reaches more inquiries. Our framework therefore informs model selection and the trade-off between capability coverage, reliability, and retry costs.

4. A practical way to evaluate model changes. We study 119 changes involving scaling, reasoning, and prompt optimization. Rather than reporting only net accuracy changes, we identify which inquiries become more accessible, which become less accessible, and how reliable capability changes. These analyses reveal improvements and regressions that can cancel out in aggregate scores. Section 6 also provides a practical procedure: report per-inquiry gains and losses and compare regressions against a run-to-run baseline from the unchanged model.

5. Broader evaluation, controlled experiments, and practical feasibility. The earlier study evaluated four benchmarks and 13 models. We extend this to six benchmarks and more than 20 models, spanning additional model families and sizes, general-knowledge evaluation (MMLU), and tool calling (BFCL). We also extend and revalidate the formulation pools.

Our controlled experiments distinguish formulation sensitivity from decoding variability, finding larger formulation-driven differences in 11 of 12 settings. Furthermore, on four of six benchmarks, evaluating two formulations on half the inquiries estimates mean accessibility, Access@2, and Reliable@2 within five percentage points of their full-data values for every tested model in at least 95% of subsamples, at the cost of one conventional benchmark run.

AuthorsOpenReview ↗

Author response (2/2): Reused resources, estimators, and canonical wording

1 (continued). Reused, extended, and new resources

The formulation pools for SimpleQA, TruthfulQA, GSM8K, and MATH-500 originate from Faghih et al. (2026), as do the human-equivalence annotations and responses for models shared between the studies. The current evaluation builds on and extends these resources:

  • Inherited: GPT-4.1-mini generation, structural and stylistic prompts, lexical filtering, and NLI checks. SimpleQA retains its original GPT-4.1-mini equivalence judge, whose accepted formulations are 96.5% human-equivalent (444/460), and 96.0% of those in the final analysis (Table 1). Throughout, we choose equivalence rules for high semantic precision, even at the cost of rejecting some valid paraphrases.
  • Extended and revalidated: TruthfulQA, GSM8K, and MATH-500 include additional formulations (for GSM8K and MATH-500, partly from a new batched generation prompt). TruthfulQA is judged by GPT-5.4-mini, and GSM8K and MATH-500 require agreement between GPT-5 and GPT-5.4-mini.
  • New: MMLU and BFCL introduce new formulation pools, including a tool-calling-specific construction procedure for BFCL. Responses from additional evaluated models are newly generated.

We added a benchmark-level provenance table to Appendix B distinguishing inherited, extended, and new components.

In short, the earlier work established that wording can expose a reliability problem. The present paper shows why even its best-, worst-, and average-case measures are insufficient, develops a framework to characterize the missing information, and provides concrete ways to compare, select, and update models using it.



2. Sampling assumptions and estimators

The key distinction is between measuring accessibility over our validated formulation sets and estimating it for an unknown distribution of user phrasings.

For the evaluated formulation pool, our combinatorial estimators exactly compute Access@k and Reliable@k when kk distinct formulations are sampled uniformly without replacement. This is the quantity Section 3.2 defines: accessibility over a uniformly weighted set of validated formulations.

For the underlying distribution DqD_q, these estimators are unbiased for Equations 4–5 when the evaluated formulations are independent samples from DqD_q. Our generated and filtered formulations provide controlled, semantically validated variation, but are not necessarily representative of real-world user phrasing frequencies. We therefore distinguish the exact finite-pool quantities from their interpretation under an unknown real-user distribution.

Two further properties help interpret these measurements. First, our finite-pool estimates require no assumptions about how real users phrase questions. Second, as the set of tested formulations expands, the empirical brittle share can only increase and the reliable share can only decrease. Thus, these quantities have a clear interpretation relative to the tested formulation set, independently of real-user sampling assumptions.

Canonical wording. To clarify the description in Section 5.1, our evaluation holds out the canonical wording by design. This allows standard benchmark correctness to be compared independently against accessibility estimated from alternative formulations. The submitted text stated that the canonical wording was included. We have corrected that description to match the actual evaluation procedure.

We also repeated RQ1 with the canonical wording included (Appendix L). Benchmark medians change by at most 2.5 percentage points, and the central conclusions remain unchanged: Access@5 and Reliable@5 rank approximately half of near-tied model pairs oppositely under either design (59/118 with the canonical wording held out, 59/117 with it included). The main findings are therefore robust to this choice. Additional details are provided in our response to Reviewer otKj.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.