- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates option-order sensitivity in multiple-choice LLM evaluation relative to repeated-generation variability and test measurement precision. Using six open-weight models and 25 ICAR items, it combines human-derived IRT calibration with mixed-effects variance decomposition. The results suggest that option order dominates run-to-run variation under greedy decoding, whereas sampling variability dominates at temperature 0.7; both are smaller than the estimated conditional standard error of measurement (CSEM).
Critical strengths
- Meaningful research question. The paper addresses an important issue in LLM evaluation by distinguishing option-order sensitivity from repeated-generation variability and measurement uncertainty. This perspective helps clarify the interpretation and reliability of benchmark results.
- Systematic experimental design. The study employs controlled option rotations and repeated evaluations under both greedy and stochastic decoding. The combination of IRT-based scoring and mixed-effects analysis provides a structured framework for quantifying different sources of evaluation variability.
Critical weaknesses
-
Uncertain validity of the measurement reference. The central conclusions rely on CSEM computed from item parameters reconstructed using human response data, without validating their measurement invariance for LLMs. The short subscales, near-ceiling performance, and prior-dependent ability estimates further weaken the interpretation of CSEM as a reliable reference for LLM evaluation.
-
Insufficient evidence for the practical importance of order effects. Order-induced variation and CSEM measure different sources of uncertainty; a smaller pooled order SD does not establish that order effects are negligible for individual models or model comparisons. The analysis lacks direct evidence on ranking stability and the distribution of model- and item-specific sensitivities.
-
Questionable criterion for model distinguishability. The minimum detectable difference is defined using the mean CSEM and an independent, equal-error approximation for two single administrations, whereas the reported model comparisons involve repeated evaluations of the same items. Consequently, the conclusion that most model pairs cannot be distinguished depends strongly on a particular, insufficiently justified threshold.
-
Limited generalizability and novelty. The experiments cover only 25 items, six quantized models with thinking disabled, and eight cyclic rotations under a constrained output protocol. The method primarily combines established IRT and statistical techniques, without demonstrating that the proposed analysis improves practical decisions about model ranking, repeated evaluation, or order debiasing.
Questions for authors
Please see weaknesses.