- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that evaluating LLMs by average accuracy overlooks both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. To address this, the paper models the "model × item" correctness matrix with 2PL-IRT and propose cBMM, which adopts the MMGN scheme to estimate the coefficients of the score matrix.
Critical strengths
- This paper focuses on the problem of evaluating a large number of models on a large number of items. I believe this is a real and important challenge that current LLM evaluation practice faces.
- The paper provides some theoretical guarantees and a fairly complete set of experiments.
Critical weaknesses
-
Regarding the claim in the proof of Theorem 1 that the rank ordering of is uniquely determined: if we shift by , then to keep the score matrix unchanged, must become . More generally, is invariant under with . Consequently, if the entries of are not all equal, I think 's relative ordering may change.
-
Each (model , item ) pair has only one observation, so the randomness of the model may not be removable from a single observation.
-
Several entries in the MATH-500 column of Table A.2 are nearly identical to those in the BBH column, so these numbers need to be double-checked.
-
[1,2] are also IRT-based approaches to efficient evaluation and benchmarking, so a comparison with it would be helpful.
[1] tinyBenchmarks: evaluating LLMs with fewer examples.
[2] Fluid Language Model Benchmarking.
Questions for authors
-
Regarding the claim in the proof of Theorem 1 that the rank ordering of is uniquely determined: if we shift by , then to keep the score matrix unchanged, must become . More generally, is invariant under with . Consequently, if the entries of are not all equal, wouldn't the relative ordering of change? Does this issue affect the main conclusions of the paper?
-
Each (model , item ) pair has only one observation. Can the randomness of the model be removed with only a single observation?
-
Several entries in the MATH-500 column of Table A.2 are nearly identical to those in the BBH column. Could the authors double-check these numbers?
8 comments
2. [W2,Q2] Single observation per pair (cont.)
This reduces exactly to the original binary update when ; the remaining block updates are unchanged.
Experiments on single vs. repeated generations. We set up (i) synthetic data with , , , and compare estimated parameters with their known generating values and evaluate predictions on five fresh draws per pair. (ii) LiveCodeBench with 34 models, 713 items, and 10 generations per pair [1]. In each replicate, we randomly divide the generations into a fitting pool of five and a held-out set of five. We fit cBMM using generations from the fitting pool and evaluate predictions on the held-out generations. We also assess parameter-ranking agreement with a separate cBMM fit on the five held-out generations.
As shown in Table R3-1, on synthetic data, the single-draw fit recovers ability and difficulty rankings with Spearman correlations of at least 0.960 and 0.987, respectively. Increasing the number of fitting draws improves parameter recovery, particularly for discrimination, whose correlation rises from approximately 0.87–0.88 at to 0.99 at . On LiveCodeBench, the single-generation fit predicts separate held-out generations with log-loss 0.273 and AUC 0.957; using five generations improves these to 0.251 and 0.961. Agreement with the reference fit remains high for ability rankings, while discrimination-ranking agreement increases more substantially, from 0.773 to 0.903. Thus, repeated draws improve probability prediction and item-parameter precision even when ability rankings are already stable.
Table R3-1. Single versus repeated fitting draws, reported as mean standard deviation over 10 replicates. is the number of fitting draws per model–item pair. (a) Synthetic data: Spearman correlations with the true parameters and predictive metrics on five fresh draws per pair. (b) LiveCodeBench: parameter-ranking correlations with a separate fit on five held-out generations, and predictive metrics evaluated on those generations.
| Spearman | Spearman | Spearman | Log-loss | AUC | |||
|---|---|---|---|---|---|---|---|
| (a) Synthetic data | |||||||
| 1000 | 100 | 1 | |||||
| 1000 | 100 | 4 | |||||
| 1000 | 100 | 16 | |||||
| 1000 | 200 | 1 | |||||
| 1000 | 200 | 4 | |||||
| 1000 | 200 | 16 | |||||
| (b) LiveCodeBench | |||||||
| 34 | 713 | 1 | |||||
| 34 | 713 | 2 | |||||
| 34 | 713 | 5 |
In short, the randomness of each model is modeled through , whose parameters are estimated by pooling information across models and items, and repeated sampling is an optional way to improve precision, not a requirement. We have clarified the probabilistic interpretation in Sections 1 and 2 of the revision, and the extension and experiments are given in Appendix A.5.
References
[1] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=chfJJYC3iL
3. [W3,Q3] Table A.2 entries
Thank you for checking the table so carefully. We re-checked every MATH-500 and BBH entry of Table A.2 of the original submission (Table A.4 of the revision) against the stored outputs of the original runs, and the values are correct. In addition, we have rerun these experiments with the exactly zero-mean initialization of in the revised algorithm, and in the updated table the MATH-500 and BBH columns are now clearly different.
4. [W4] tinyBenchmarks and Fluid Benchmarking
Thank you. We now compare cBMM with tinyBenchmarks [1] and Fluid Benchmarking [2] pipelines, together with other IRT-based approaches for efficient evaluation and benchmarking (as requested by Reviewer C7h9): PSN-IRT [3], ATLAS [4], and amortized model-based evaluation (reeval) [5] on five suites. Table R3-2 (Table A.19 of the revision) provides an overview of method characteristics and performance comparisons with cBMM. Compared to tinyBenchmarks and Fluid Benchmarking, replacing only their original py-irt calibration by cBMM makes calibration 3–171 times faster, lowers the error of tinyBenchmarks on four of the five suites, and leaves the results of Fluid Benchmarking essentially unchanged.
Table R3-2. Overview of the compared methods. The right columns summarize the results of cBMM relative to each method from Tables R3-3 and R3-4: held-out AUC for the direct estimators, and the MAE of the estimated full-benchmark accuracy for the pipelines. Speed-up is the ratio of the method's fitting or calibration time to that of cBMM. NN: neural networks; MFI: maximum Fisher information; SVI: stochastic variational inference.
| Method | Role | IRT model | Calibration algorithm | Item selection | AUC (cBMM vs. method) | MAE (cBMM vs. method) | Speed-up (cBMM vs. method) |
|---|---|---|---|---|---|---|---|
| cBMM | Direct / calibrator | 2PL | Block MM | – | – | – | – |
| PSN-IRT | Direct | 4PL | Minibatch Adam for NN | – | Higher or equal | – | 9–137$\times$ |
| reeval | Direct | Rasch | L-BFGS | – | Higher | – | 6–35$\times$ |
| ATLAS | Pipeline | 2PL | mirt (EM) | Adaptive MFI | – | Comparable | 7–1,397$\times$ |
| reeval CAT | Pipeline | Rasch | L-BFGS | Adaptive MFI | – | Lower | 8–31$\times$ |
| tinyBenchmarks | Pipeline | Multidimensional 2PL | py-irt (SVI) | Static anchor items | – | Lower | 5–171$\times$ |
| Fluid | Pipeline | 2PL with hierarchical priors | py-irt (SVI) | Adaptive MFI | – | Comparable | 3–16$\times$ |
Comparison setup. We added discussion of these methods in Appendix A.1 and experimental details in Appendix A.17. We distinguish direct estimation from downstream pipelines since these methods address different stages of the IRT-based evaluation protocol. For reeval, we use the non-amortized Rasch calibration variant; this comparison does not evaluate its text-embedding-based amortized variant. The experiments use the same type of CPU node and the runtime comparisons apply.
Direct estimators: PSN-IRT and reeval. We randomly hide 10% of the responses, fit on the remaining entries, and evaluate predictions on the hidden entries over 10 replicates. cBMM has the lowest log-loss on all five suites and the highest or tied-highest AUC on every suite, while running 9–137 times faster than PSN-IRT (Table R3-3).
References
[1] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34303–34326, 2024. https://proceedings.mlr.press/v235/maia-polo24a.html
[2] Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, and Noah A. Smith. Fluid language model benchmarking. In Second Conference on Language Modeling, 2025. https://openreview.net/forum?id=mxcCg9YRqj
[3] Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, et al. Lost in benchmarks? rethinking large language model benchmarking with item response theory. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence, 2026. https://doi.org/10.1609/aaai.v40i41.40814
[4] Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V Chawla. Adaptive testing for LLM evaluation: A psychometric alternative to static benchmarks. In Forty-third International Conference on Machine Learning, 2026. https://openreview.net/forum?id=ItJEC1Mk0V
[5] Sang T. Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo. Reliable and efficient amortized model-based evaluation. In Forty-second International Conference on Machine Learning, 2025. https://openreview.net/forum?id=HDbWrsgkB9
We thank the reviewer for the careful reading. Below we address each point.
1. [W1,Q1] Ordering of b
Thank you for pointing this out; Reviewer snKp raises the same point. We fix this by adding a reference scale through the location constraint 1⊤θ=0, under which b is identified exactly. We have revised Theorem 1 and Algorithm 1, accordingly. The change is mild: cBMM preserves the reference scale during updates, the calibration of zero-mean initialization of θ is all we need. We have re-run the affected experiments using an exact zero-mean initialization for θ rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.
The reference scale removes ambiguity. The likelihood is invariant under (θ,a,b)↦(cθ+d1,a/c,b−cda) with c>0. The constraint 1⊤θ=0 fixes the location shift, d=0: abilities are measured relative to the average model in the pool, b=N1X⊤1 is identified exactly, and −bj is the difficulty of item j for a model of average ability. θ and a remain identified up to a common scale c>0, which changes neither their orderings nor CV$(\hat{\mathbf a})$. The constraint loses nothing, because centering any solution leaves X, and hence the likelihood, unchanged. The revised Theorem 1 is stated in Section 3.3, followed by a discussion of why the location constraint is needed and how it identifies b exactly; the proof is in Appendix A.2.
Calibrating zero-mean initialization is all we need. The cBMM updates preserve the mean of θ exactly (Proposition 1 in Appendix A.2), so a zero-mean initialization keeps every iterate, and the limit, on the constraint set. We have revised Theorem 1 accordingly, and re-run the experiments using an exact zero-mean initialization for θ rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.
Empirical check. Across 50 random initializations with centered abilities, the estimated intercept rankings have Spearman correlations of at least 0.98 on IFEval, MuSR, MATH, GPQA, BBH, and MMLU-Pro (Appendix A.12). These results support the reproducibility of the estimates under the specified reference convention.
2. [W2,Q2] Single observation per pair
Thank you for raising this. We would like to first clarify that cBMM models correctness probabilities under IRT-based evaluation protocol; it does not remove generation randomness. Each yij is one Bernoulli draw from the response probability Φ(xij), and the targets of estimation are the parameters that generate this probability. A single draw cannot reveal Φ(xij) of one model-item pair on its own, and our framework does not try to infer it entry by entry. Instead, information is pooled across all models and items, and our new experiments on synthetic data and real repeated generations show that the probabilities estimated from single observation already capture generation variability, while repeated draws mainly sharpen the discriminations.
Why single observation suffices. In our model, Φ(xij) is the probability that model i answers item j correctly under the IRT-based evaluation protocol. Each observed correctness outcome is modeled as a Bernoulli draw from this probability. A single observation cannot identify an individual model–item probability in isolation. Instead, the structure xij=ajθi+bj pools information across the response matrix: each ability θi is informed by responses across items, and each item-parameter pair (aj,bj) is informed by responses across models. The following experiments based on repeated-generation extension then provide empirical evidence on whether this shared structure adequately predicts repeated outcomes.
Repeated-generation extension. When K generations are available per pair, we replace binary correctness by the empirical success rate γij. With the same K for every pair, the resulting objective is the binomial negative log-likelihood divided by K. Only the surrogate target in line 6 of Algorithm 1 changes:
Y←X+4σ(Γ−Φ(X)),Γ=(γij).