- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Most existing LLM benchmark evaluations adopt only a single fixed wording for each question and compare models using a single accuracy score. Nevertheless, one semantic inquiry can have multiple semantically‑equivalent formulations, and models exhibit substantial performance discrepancies across these rephrased variants. This paper introduces the concept of accessibility: the probability that a model answers correctly over all semantically‑equivalent rephrasings of the underlying inquiry. An accessibility profile is constructed based on the accessibility distribution across all inquiries. Two key derived metrics are further proposed: Access@k, which measures success in at least one out of k formulations (suited for deployment scenarios where retries are permitted), and Reliable@k, which requires correct answers across all k formulations (for settings demanding reliable outputs on every invocation). A corresponding theoretical proposition is presented: given two models with equal mean accessibility, crossing reliability curves can lead to completely contradictory model rankings produced by Access@k and Reliable@k.
Critical strengths
It points out the inherent flaw of single‑query evaluation in traditional benchmarks: accuracy obtained from fixed wording is merely one sample point from the model’s capability distribution, and cannot fully characterize the model’s true capability for the underlying inquiry. This perspective offers strong insights for the LLM‑evaluation community.
Controlled comparative experiments are designed to demonstrate that correctness fluctuations induced by equivalent rephrasings are substantially larger than output fluctuations caused by temperature sampling. This shows that accessibility captures prompt‑formulation effects rather than mere generative randomness.
Critical weaknesses
All formulations are generated by GPT‑4.1‑mini. Though semantic validation is performed, there exists a distribution shift relative to real‑world user‑authored question rephrasings. Accessibility is therefore an estimate over a synthetic rephrasing set and may not equate to model behaviour under genuine user inputs.
The paper mixes syntactic rephrasing, sentence‑structure transformation and stylistic rewriting without conducting corresponding ablation studies. It remains unclear which category of rephrasing most readily degrades accessibility; consequently, it cannot pinpoint whether model fragility primarily stems from syntactic alterations or tonal‑style variations.
There is no comparison of how accessibility and the fraction of brittle‑sample change across base models, SFT‑aligned models and RLHF‑aligned models. No experimental evidence is provided regarding whether alignment raises or lowers Reliable@k.
Questions for authors
The formulations in the paper are model‑synthesized rephrasings. To what extent would the findings for the accessibility profile, Access@k and Reliable@k change if a dataset of real human‑user rephrasings were adopted?
The paper proposes a low‑cost approximation scheme: using two rephrasings for half of the inquiries. For different tasks, how should one strike the optimal trade‑off between the number of rephrasings and the number of inquiries? Are there practical reference thresholds that can be followed?
2 comments
Author response (2/2): Our framework applies directly to alignment stages as a further application, and the best budget split depends on the target metric
3. Alignment studies are another application of our framework
Our framework measures how model capabilities vary across inquiries and equivalent formulations. We use it to compare models, distinguish wording effects from decoding variability, and study model changes. Evaluating specific training methods is not the paper's objective.
Section 5 already applies this framework to 119 model changes across scaling, native thinking, chain-of-thought prompting, and prompt optimization. These experiments reveal gains and regressions that aggregate accuracy can hide. For example, enabling native thinking in Qwen3 changes MMLU accuracy by only −3.0 points, while 25% of inquiries become more accessible and 32% less accessible, and the reliable share drops by 10.9 points.
Comparing base, SFT, and RLHF models is a valuable use of the same framework. Access@k, Reliable@k, and the brittle share can reveal whether alignment improves reliability, broadens capability, or redistributes success across inquiries. A controlled comparison would require matched checkpoints derived from the same base model and careful control of differences in prompting and format-following. We have added this direction to the Discussion as future work.
4. Allocating a fixed budget: the best split depends on the target metric
The best allocation depends on what we want to estimate. We extended Appendix J with a practical sampling analysis and evaluated the recommendations through simulations.
1. Mean accessibility. With a fixed budget of model calls, evaluating formulations per inquiry leaves inquiries. Under independent sampling of inquiries and formulations, the variance of estimated mean accessibility is
\frac{K\operatorname{Var}_q(A_q)+ \mathbb E_q[A_q(1-A_q)]}{B}.$$ Thus, for estimating the mean alone, **covering more inquiries is generally preferable to evaluating more formulations of fewer inquiries**. **2. Access@k and Reliable@k.** These require at least $k$ formulations per inquiry. Choosing $K=k$ is the cheapest design that identifies the target quantity and maximizes inquiry coverage at a fixed budget. It is the most efficient design. **3. Practical budget thresholds.** Our simulations suggest the following guidance for estimating fixed-$K$ metrics within ±5 percentage points in at least 95% of simulated subsamples: - **Benchmarks with 600–902 inquiries:** $K=2$, evaluating half the inquiries at the cost of one conventional benchmark run, works well for SimpleQA, GSM8K, MMLU, and BFCL. - **TruthfulQA (471 inquiries):** This setting is borderline. Mean accessibility is estimated reliably, but Access@2 and Reliable@2 meet the target for only about half the models. - **MATH-500 (106 retained inquiries):** The same budget is generally insufficient for reliable estimates across models. Evaluating the full inquiry set is preferable. - **Larger $K$:** At a fixed budget of model calls, evaluating more formulations per inquiry means evaluating fewer inquiries, so precision worsens. We also derive an approximate sample-size rule for inquiry-level proportions: $$n_{\mathrm{req}}\approx \frac{z^2p(1-p)/\epsilon^2} {1+z^2p(1-p)/(\epsilon^2N)},$$ where $p$ is the expected proportion, $\epsilon$ the desired margin of error, and $N$ the benchmark size. This uses a finite-population correction for sampling inquiries, and the indicator of each sampled inquiry already includes the random choice of its formulations. In our simulations, the planning rule correctly predicts whether the target precision is met in **94.7% of tested settings**. **4. Full accessibility profiles.** Estimating the empirical reliable, latent, and brittle shares over the complete formulation pool is different from estimating Access@2 or Reliable@2. Two formulations cannot recover these full-pool endpoint shares. We therefore recommend evaluating all available formulations for a sampled subset of inquiries when these are the target quantities (on our benchmarks, 211–267 inquiries suffice for ±5 points, 0.9–3.8 times the cost of one conventional run). **Practical recommendation:** First choose the target metric. Use the minimum number of formulations needed as an initial budget allocation, prioritize inquiry coverage, and add formulations only when the full profile is the target. The appendix provides task-specific results rather than assuming one allocation is optimal for every benchmark.
Author response (1/2): The same meaning-preserving formulations for every model give a paired, conservative comparison, and sensitivity appears under both rewrite types
We thank the reviewer for the constructive and detailed review, and for recognizing that single-wording accuracy captures only one point of a model's capability and that our controlled comparison separates formulation effects from sampling noise. We answer each point below. The new analyses are in Appendices J and M of the revised paper.
1. Meaning-preserving formulations isolate the effect of wording, and real user phrasings could only add brittleness
Our goal is to measure how models respond to different formulations of the same inquiry. While synthetic paraphrases may not reflect the frequency of real user phrasings, they allow us to study this variation under controlled conditions. Three design choices make the comparisons sound:
Our central findings do not require the formulations to match real-user frequencies. The experiments establish that similar average accuracy can hide different accessibility profiles, that Access@k and Reliable@k can rank models oppositely, and that model changes can redistribute capability across inquiries without large changes in accuracy. These effects arise even under careful, human-validated rewrites, while real user phrasing varies more widely (typos, added context, different vocabulary).
The formulations do not simply make the benchmarks harder, and formulation sensitivity runs in both directions. Accuracy and mean accessibility rank models almost identically (Kendall's $ \tau $ 0.90–0.99). Yet individual answers change both ways: on TruthfulQA, GSM8K, MATH-500, and BFCL, 14–36% of the formulations of canonically failed inquiries are answered correctly (median over models), while other inquiries solved in their canonical wording fail under alternative wordings. Changing the wording redistributes success across inquiries rather than only lowering it.
The exact accessibility values and operational metrics depend on the formulation distribution. We acknowledge this in Section 6, and the framework applies unchanged to any formulation distribution, including one weighted by real-user frequency. Our brittleness estimates are therefore a reproducible lower bound: the formulation sets are fixed and released, and over any larger set of valid formulations, including real user phrasings, the brittle share can only be higher and the reliable share only lower.
2. The requested ablation shows formulation sensitivity under both structural and stylistic rewrites
We conducted the requested ablation and added it as Appendix M. We compared structural and stylistic rewrites within each model–inquiry pair, controlling for inquiry difficulty.
Neither category consistently degrades performance more. On SimpleQA, TruthfulQA, GSM8K, and MATH-500 the differences are within 0.4 percentage points and every confidence interval includes zero. On MMLU, structural rewrites are slightly less successful (−0.9 points), yet the brittle share measured with two formulations is higher under stylistic rewrites (16.4% vs. 14.2%). On the other benchmarks the brittle shares are similar across the two categories (BFCL formulations carry no cue). Among 73 individual cue–benchmark comparisons, only five differ significantly after Bonferroni correction, each by at most 1.7 points, in both directions and across both categories.
These results suggest that neither structural nor stylistic rewriting consistently explains greater formulation sensitivity, which occurs under both types. Since cues were assigned during generation rather than randomized, we interpret these as observed associations, not causal effects.