acceptodds
Withdrawn from ICLR 2027

Smaller Than the Noise: A Measurement-Theoretic Audit of Evaluation Protocols for LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Option-order sensitivity complicates multiple-choice evaluation of large language models (LLMs), but its magnitude must be interpreted relative to repeated-generation variability and measurement precision. We evaluate six open-weight LLMs on 25 text-based items from the International Cognitive Ability Resource under eight cyclic option rotations, with two greedy and five stochastic administrations per rotation, yielding 8,400 item responses. We estimate domain-specific abilities using a between-item multidimensional item response theory model with fixed parameters reconstructed from human response summaries and standardized factor loadings. A mixed-effects analysis separates shared rotation effects, model-specific rotation effects, and residual variation across draws. Under stochastic decoding (), the estimated order component has an SD of 0.08–0.10 ability units, compared with 0.30–0.33 across draws. Under greedy decoding, 99.7% of scoreable repeated item-response pairs are identical, yet the order component has an SD of 0.28–0.30. Thus, repeatability under a fixed presentation does not imply robustness to option order. Across both regimes, these pooled components remain below half the mean information-based conditional standard error of measurement. These comparisons are conditional on the reconstructed response model and do not establish equivalence between models. Our findings show how reporting order sensitivity alongside repeated-generation variability and test information can clarify what short evaluations support.

Primary area:
datasets and benchmarks
Keywords:
Option-Order Effects in LLM Evaluation: Sampling Variability and Measurement Precision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aYPXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates option-order sensitivity in multiple-choice LLM evaluation relative to repeated-generation variability and test measurement precision. Using six open-weight models and 25 ICAR items, it combines human-derived IRT calibration with mixed-effects variance decomposition. The results suggest that option order dominates run-to-run variation under greedy decoding, whereas sampling variability dominates at temperature 0.7; both are smaller than the estimated conditional standard error of measurement (CSEM).

Critical strengths
  1. Meaningful research question. The paper addresses an important issue in LLM evaluation by distinguishing option-order sensitivity from repeated-generation variability and measurement uncertainty. This perspective helps clarify the interpretation and reliability of benchmark results.
  2. Systematic experimental design. The study employs controlled option rotations and repeated evaluations under both greedy and stochastic decoding. The combination of IRT-based scoring and mixed-effects analysis provides a structured framework for quantifying different sources of evaluation variability.
Critical weaknesses
  1. Uncertain validity of the measurement reference. The central conclusions rely on CSEM computed from item parameters reconstructed using human response data, without validating their measurement invariance for LLMs. The short subscales, near-ceiling performance, and prior-dependent ability estimates further weaken the interpretation of CSEM as a reliable reference for LLM evaluation.

  2. Insufficient evidence for the practical importance of order effects. Order-induced variation and CSEM measure different sources of uncertainty; a smaller pooled order SD does not establish that order effects are negligible for individual models or model comparisons. The analysis lacks direct evidence on ranking stability and the distribution of model- and item-specific sensitivities.

  3. Questionable criterion for model distinguishability. The minimum detectable difference is defined using the mean CSEM and an independent, equal-error approximation for two single administrations, whereas the reported model comparisons involve repeated evaluations of the same items. Consequently, the conclusion that most model pairs cannot be distinguished depends strongly on a particular, insufficiently justified threshold.

  4. Limited generalizability and novelty. The experiments cover only 25 items, six quantized models with thinking disabled, and eight cyclic rotations under a constrained output protocol. The method primarily combines established IRT and statistical techniques, without demonstrating that the proposed analysis improves practical decisions about model ranking, repeated evaluation, or order debiasing.

Questions for authors

Please see weaknesses.

Reviewer_m566OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how answer-option order affects LLM evaluation results and compares this effect with variability across repeated generations and the measurement uncertainty of the test itself. The authors evaluate six open-weight models on 25 text-based ICAR items using eight cyclic option rotations, with two repetitions at temperature 0 and five at temperature 0.7. All models are quantized and run with thinking disabled, generating an explanation followed by the selected option text under a JSON schema. Using fixed IRT item parameters reconstructed from human data, the authors convert model responses into ability estimates and compute the conditional standard error of measurement (CSEM). Under the settings studied, option-order variation in estimated ability is smaller than the CSEM; under stochastic decoding, variation across repeated generations also exceeds option-order variation.

Critical strengths

1. Considering multiple sources of evaluation uncertainty has practical value. The paper goes beyond reporting score changes after option reordering by comparing them with variability across repeated generations and the measurement precision of the test. This helps contextualize what the observed changes mean for model comparison.

2. The experimental setup is reasonably explicit and includes repeated measurements. The eight cyclic rotations place the correct answer in each of the eight positions once. Both decoding conditions include repeated generations, and the analysis distinguishes rotation effects shared across models from model-specific effects. The paper also explicitly discusses the short subscales, ceiling effects, and the unverified assumption of measurement invariance between humans and LLMs.

Critical weaknesses

1. The validity of the central measurement-error reference has not been sufficiently established. The authors fix item difficulties, discriminations, and the correlation between domains reconstructed from human data, then use them directly to estimate LLM abilities and CSEM. This relies on measurement invariance between humans and LLMs, but the paper does not provide sufficient model-fit checks or validation of this transfer. This approach is understandable as an exploratory, fixed scoring procedure. However, interpreting the resulting CSEM as the precision of LLM ability measurement, and using it to judge the magnitude of order effects and model differences, requires stronger evidence. The authors acknowledge this limitation, but acknowledgment alone does not resolve the concern because the central conclusions depend on this reference.

2. The small number of items and their limited suitability for some models constrain the interpretation of the main comparisons. The experiment uses only 25 items, with 16 and 9 items in the two subscales, and some models approach ceiling performance even with thinking disabled. The authors acknowledge that the items are poorly targeted at the strongest models, that CSEM is large, and that ability estimates near the ceiling depend on the prior. Consequently, the finding that option-order variation is smaller than measurement error may partly reflect the limited resolving power of this particular test. This result can characterize the limitations of the test, but it does not establish that option-order effects would have a similar relative magnitude in longer evaluations with better-targeted item difficulties.

3. The inference configurations are restrictive, and the robustness and generalizability of the main conclusions need stronger support. All models use 4-bit quantization, have thinking disabled, and produce constrained option-text outputs. The authors explain the decision to disable thinking partly through pilot observations of models exhausting the generation budget. However, the 2048-token generation limit is an experimental choice; exhausting this budget does not by itself justify excluding thinking mode. The paper provides neither a thinking-mode comparison with a larger generation budget nor evidence that the main findings are stable across quantization precisions. These choices do not invalidate the experiments, but they restrict the evidence to a particular inference configuration. Given the paper's aim of discussing uncertainty in LLM evaluation, its relevance to other common evaluation settings needs to be established more clearly.

Questions for authors

1. Applicability of human IRT parameters to LLMs. Could the authors provide model-fit checks to assess whether item-level correctness probabilities predicted using the human parameters agree with observed LLM responses, and discuss possible differential item functioning? Are the main conclusions stable under changes to the scoring method or parameter assumptions? An analysis of order and sampling variability using raw accuracy would also provide a useful complementary reference.

2. Item targeting and measurement error. Given that some models approach ceiling performance, could the authors provide additional validation on harder items better matched to model abilities, or a relevant sensitivity analysis? It would be helpful to distinguish how much of the comparison reflects small order effects and how much reflects a large CSEM resulting from short subscales and ceiling effects.

3. Thinking mode and generation budget. Please explain the rationale for the 2048-token generation limit. Could the authors increase the budget and include a thinking-mode comparison for at least some models that support it, to assess whether the findings concerning order effects, sampling variability, and their relative magnitudes still hold?

4. Quantization settings. All six models use 4-bit quantization. Could the authors validate the main trends using higher-precision versions of at least some models, or further delimit the model configurations to which the conclusions apply?

5. Where option order changes in the evaluation pipeline. The appendix indicates that changing the option order in the prompt also changes the option enumeration order in the output JSON schema. Please clarify whether the reported option-order effects include the joint effects of both changes, and whether their contributions could be separated by holding one order fixed. In addition, the eight cyclic rotations balance absolute positions but provide limited coverage of other relative arrangements. Please clarify the scope of the conclusions supported by this design.

6. External validation using closed-source API models. The current experiments cover only locally run, quantized models with thinking disabled. Additional validation using representative closed-source API models, together with items of appropriately matched difficulty, would help assess whether the main findings generalize to stronger models and practical evaluation settings.

7. Presentation of the experimental and analytical pipeline. A concise workflow diagram or a concrete item-level example connecting option rotation, repeated generation, IRT ability estimation, and comparison with CSEM would improve readability. The aim would be to clarify how these quantities relate to one another and how the experimental results support the final conclusions.

Reviewer_wYWpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

First, I noted that there is a title mismatch. The title on the ICLR site is different from the title in the PDF file. I think the title in the PDF file is more informative. The study evaluates the option-order effects in LLM evaluation, by evaluating 6 open-weight models on 25 items. A two-step mixed effect analysis was performed on the 8,400 item responses. In step I, an EAP estimates of model’s capability is obtained using fixed item parameters. In step II, a linear fixed effects model is fitted on the EAP estimates by considering the fixed effects of model and rotation and a random intercept for a model-by-rotation cell. Their main finding is that option-order effects dominate under greedy decoding, whereas sampling variability dominates under stochastic decoding.

Critical strengths

This paper does not introduce a new IRT model or statistical estimation method. I think the main contribution is applying existing psychometric concepts to contextualize LLM evaluation variability. Aside from the two-stage estimation, I found the model in Equation 5 to be thoughtful, in that it can distinguish between (1) option-order variability by using the fixed rotation effect, (2) model-specific option-order effects by using random intercept, and (3) sampling variability (i.e., residuals) to capture changes in responses across repeated generations

Critical weaknesses

My main concerns of the modeling approach are as follows

  1. In Equation 5, the dependent variable, theta, is obtained from EAP assuming item parameters are fixed. First, EAP as a Bayesian estimator has an inherent shrinkage effect. Second, the linear mixed effects model does not consider that the dependent variable is latent and hence, the model ignores the estimation error in theta. A better approach would be an integrated model rather than the two-stage approach. Considering that a model’s performance will be correlated under different option-order designs, the two-stage treats each model’s performance under each rotation condition as independent, thereby it is less statistically efficient.
  2. It was claimed that both the standard deviation of the fixed effects of rotation effect, and the total standard deviation are reported on the theta scale, and hence they are compared to the conditional standard error of measurement (CSEM), such as in table 1. However, CSEM measures the conditional precision of the IRT instrument under its assumed model, whereas the other variance estimates (or standard deviations) are based on the variability in estimated EAPs attributable to option rotations, repeated generations, or both. In other words, the comparison is really observed variability vs. instrument “true” variability, and I don’t think such a direct comparison is meaningful. A better reference is the overall variability of EAP estimates.
  3. I found the motivation and interpretation of the item-level dispersion analysis in Section 2.7 are unclear. Although the authors interpret Pearson dispersion ratios greater than one as evidence of excess variability across stochastic draws, the analysis uses Bernoulli responses and a logistic regression containing only main effects. Such ratios may reflect unmodeled interactions or mean-model misspecification rather than stochastic generation variability.
  4. I think using the fixed item parameters calibrated from human responses is fine. However, the empirical analysis does not establish whether the fixed human-calibrated IRT model fits the LLM responses. Some model data fit checking would be helpful here. 5.Although complementing statistical significance results with MDD is great, and it leads to one finding in the paper that statistically significant differences between LLMs do not necessarily mean the benchmark has sufficient measurement precision to distinguish their abilities. However, this conclusion depends on the definition of MDD, which seems to be quite wide in the paper, and that is why the results based on significance testing and MDD differ a lot. Further, since the primary topic of the paper is option-order sensitivity, not comparing LLM abilities, so I found exploration along this direction somewhat distracting.
Questions for authors

Why to choose ICAR for the study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.