acceptodds
Under review as a conference paper at ICLR 2027

An Interpretable and Scalable Framework for Evaluating Large Language Models

Abstract

Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization–minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and provide interpretable insights into item difficulty and discrimination, informing more principled benchmark design than conventional average-accuracy evaluation.

Primary area:
interpretability and explainable AI
Keywords:
LLM Evaluation, Binary Observations, Item Response Theory, Majorization-Minimization, Scalability, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PWcHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that evaluating LLMs by average accuracy overlooks both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. To address this, the paper models the "model × item" correctness matrix with 2PL-IRT and propose cBMM, which adopts the MMGN scheme to estimate the coefficients of the score matrix.

Critical strengths
  1. This paper focuses on the problem of evaluating a large number of models on a large number of items. I believe this is a real and important challenge that current LLM evaluation practice faces.
  2. The paper provides some theoretical guarantees and a fairly complete set of experiments.
Critical weaknesses
  1. Regarding the claim in the proof of Theorem 1 that the rank ordering of (θ,a,b)(\boldsymbol{\theta}, \mathbf{a}, \mathbf{b}) is uniquely determined: if we shift θ\boldsymbol{\theta} by dd, then to keep the score matrix X=θa⊤+1b⊤X = \boldsymbol{\theta}\mathbf{a}^\top + \mathbf{1}\mathbf{b}^\top unchanged, b\mathbf{b} must become b−dca\mathbf{b} - \frac{d}{c}\mathbf{a}. More generally, XX is invariant under (θ,a,b)↦(cθ+d1, a/c, b−dca)(\boldsymbol{\theta}, \mathbf{a}, \mathbf{b}) \mapsto \left(c\boldsymbol{\theta} + d\mathbf{1},\ \mathbf{a}/c,\ \mathbf{b} - \frac{d}{c}\mathbf{a}\right) with c>0c > 0. Consequently, if the entries of a\mathbf{a} are not all equal, I think b\mathbf{b}'s relative ordering may change.

  2. Each (model ii, item jj) pair has only one observation, so the randomness of the model may not be removable from a single observation.

  3. Several entries in the MATH-500 column of Table A.2 are nearly identical to those in the BBH column, so these numbers need to be double-checked.

  4. [1,2] are also IRT-based approaches to efficient evaluation and benchmarking, so a comparison with it would be helpful.

[1] tinyBenchmarks: evaluating LLMs with fewer examples.

[2] Fluid Language Model Benchmarking.

Questions for authors
  1. Regarding the claim in the proof of Theorem 1 that the rank ordering of (θ,a,b)(\boldsymbol{\theta}, \mathbf{a}, \mathbf{b}) is uniquely determined: if we shift θ\boldsymbol{\theta} by dd, then to keep the score matrix X=θa⊤+1b⊤X = \boldsymbol{\theta}\mathbf{a}^\top + \mathbf{1}\mathbf{b}^\top unchanged, b\mathbf{b} must become b−dca\mathbf{b} - \frac{d}{c}\mathbf{a}. More generally, XX is invariant under (θ,a,b)↦(cθ+d1, a/c, b−dca)(\boldsymbol{\theta}, \mathbf{a}, \mathbf{b}) \mapsto \left(c\boldsymbol{\theta} + d\mathbf{1},\ \mathbf{a}/c,\ \mathbf{b} - \frac{d}{c}\mathbf{a}\right) with c>0c > 0. Consequently, if the entries of a\mathbf{a} are not all equal, wouldn't the relative ordering of b\mathbf{b} change? Does this issue affect the main conclusions of the paper?

  2. Each (model ii, item jj) pair has only one observation. Can the randomness of the model be removed with only a single observation?

  3. Several entries in the MATH-500 column of Table A.2 are nearly identical to those in the BBH column. Could the authors double-check these numbers?

8 comments

We thank the reviewer for the careful reading. Below we address each point.

1. [W1,Q1] Ordering of b\mathbf b

Thank you for pointing this out; Reviewer snKp raises the same point. We fix this by adding a reference scale through the location constraint 1⊤θ=0\mathbf 1^\top\boldsymbol\theta=0, under which b\mathbf b is identified exactly. We have revised Theorem 1 and Algorithm 1, accordingly. The change is mild: cBMM preserves the reference scale during updates, the calibration of zero-mean initialization of θ\boldsymbol\theta is all we need. We have re-run the affected experiments using an exact zero-mean initialization for θ\boldsymbol{\theta} rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.

The reference scale removes ambiguity. The likelihood is invariant under (θ,a,b)↦(cθ+d1,a/c,b−dca)(\boldsymbol\theta,\mathbf a,\mathbf b)\mapsto(c\boldsymbol\theta+d\mathbf 1,\\ \mathbf a/c,\\ \mathbf b-\tfrac{d}{c}\mathbf a) with c>0c>0. The constraint 1⊤θ=0\mathbf 1^\top\boldsymbol\theta=0 fixes the location shift, d=0d=0: abilities are measured relative to the average model in the pool, b=1NX⊤1\mathbf b=\frac1N\mathbf X^\top\mathbf 1 is identified exactly, and −bj-b_j is the difficulty of item jj for a model of average ability. θ\boldsymbol\theta and a\mathbf a remain identified up to a common scale c>0c>0, which changes neither their orderings nor CV$(\hat{\mathbf a})$. The constraint loses nothing, because centering any solution leaves X\mathbf X, and hence the likelihood, unchanged. The revised Theorem 1 is stated in Section 3.3, followed by a discussion of why the location constraint is needed and how it identifies b\mathbf b exactly; the proof is in Appendix A.2.

Calibrating zero-mean initialization is all we need. The cBMM updates preserve the mean of θ\boldsymbol\theta exactly (Proposition 1 in Appendix A.2), so a zero-mean initialization keeps every iterate, and the limit, on the constraint set. We have revised Theorem 1 accordingly, and re-run the experiments using an exact zero-mean initialization for θ\boldsymbol{\theta} rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.

Empirical check. Across 50 random initializations with centered abilities, the estimated intercept rankings have Spearman correlations of at least 0.98 on IFEval, MuSR, MATH, GPQA, BBH, and MMLU-Pro (Appendix A.12). These results support the reproducibility of the estimates under the specified reference convention.


2. [W2,Q2] Single observation per pair

Thank you for raising this. We would like to first clarify that cBMM models correctness probabilities under IRT-based evaluation protocol; it does not remove generation randomness. Each yijy_{ij} is one Bernoulli draw from the response probability Φ(xij)\Phi(x_{ij}), and the targets of estimation are the parameters that generate this probability. A single draw cannot reveal Φ(xij)\Phi(x_{ij}) of one model-item pair on its own, and our framework does not try to infer it entry by entry. Instead, information is pooled across all models and items, and our new experiments on synthetic data and real repeated generations show that the probabilities estimated from single observation already capture generation variability, while repeated draws mainly sharpen the discriminations.

Why single observation suffices. In our model, Φ(xij)\Phi(x_{ij}) is the probability that model ii answers item jj correctly under the IRT-based evaluation protocol. Each observed correctness outcome is modeled as a Bernoulli draw from this probability. A single observation cannot identify an individual model–item probability in isolation. Instead, the structure xij=ajθi+bjx_{ij}=a_j\theta_i+b_j pools information across the response matrix: each ability θi\theta_i is informed by responses across items, and each item-parameter pair (aj,bj)(a_j,b_j) is informed by responses across models. The following experiments based on repeated-generation extension then provide empirical evidence on whether this shared structure adequately predicts repeated outcomes.

Repeated-generation extension. When KK generations are available per pair, we replace binary correctness by the empirical success rate γij\gamma_{ij}. With the same KK for every pair, the resulting objective is the binomial negative log-likelihood divided by KK. Only the surrogate target in line 6 of Algorithm 1 changes:

Y~←X~+4σ(Γ−Φ(X~)),Γ=(γij).\widetilde{\mathbf Y} \leftarrow \widetilde{\mathbf X} +4\sigma\bigl(\mathbf\Gamma-\Phi(\widetilde{\mathbf X})\bigr), \qquad \mathbf\Gamma=(\gamma_{ij}).

2. [W2,Q2] Single observation per pair (cont.)

This reduces exactly to the original binary update when K=1K=1; the remaining block updates are unchanged.

Experiments on single vs. repeated generations. We set up (i) synthetic data with N=1000N=1000, J∈100,200J\in\\{100,200\\}, K∈1,4,16K\in\\{1,4,16\\}, and compare estimated parameters with their known generating values and evaluate predictions on five fresh draws per pair. (ii) LiveCodeBench with 34 models, 713 items, and 10 generations per pair [1]. In each replicate, we randomly divide the generations into a fitting pool of five and a held-out set of five. We fit cBMM using K∈1,2,5K\in\\{1,2,5\\} generations from the fitting pool and evaluate predictions on the held-out generations. We also assess parameter-ranking agreement with a separate cBMM fit on the five held-out generations.

As shown in Table R3-1, on synthetic data, the single-draw fit recovers ability and difficulty rankings with Spearman correlations of at least 0.960 and 0.987, respectively. Increasing the number of fitting draws improves parameter recovery, particularly for discrimination, whose correlation rises from approximately 0.87–0.88 at K=1K=1 to 0.99 at K=16K=16. On LiveCodeBench, the single-generation fit predicts separate held-out generations with log-loss 0.273 and AUC 0.957; using five generations improves these to 0.251 and 0.961. Agreement with the reference fit remains high for ability rankings, while discrimination-ranking agreement increases more substantially, from 0.773 to 0.903. Thus, repeated draws improve probability prediction and item-parameter precision even when ability rankings are already stable.

Table R3-1. Single versus repeated fitting draws, reported as mean ±\pm standard deviation over 10 replicates. KK is the number of fitting draws per model–item pair. (a) Synthetic data: Spearman correlations with the true parameters and predictive metrics on five fresh draws per pair. (b) LiveCodeBench: parameter-ranking correlations with a separate fit on five held-out generations, and predictive metrics evaluated on those generations.

NNJJKKSpearman θ^\hat{\boldsymbol\theta}Spearman a^\hat{\mathbf a}Spearman b^\hat{\mathbf b}Log-loss ↓\downarrowAUC ↑\uparrow
(a) Synthetic data
100010010.960±0.0020.960\pm0.0020.870±0.0210.870\pm0.0210.987±0.0050.987\pm0.0050.620±0.0040.620\pm0.0040.714±0.0050.714\pm0.005
100010040.989±0.0010.989\pm0.0010.964±0.0060.964\pm0.0060.996±0.0010.996\pm0.0010.615±0.0040.615\pm0.0040.719±0.0050.719\pm0.005
1000100160.997±0.0000.997\pm0.0000.989±0.0010.989\pm0.0010.999±0.0000.999\pm0.0000.614±0.0040.614\pm0.0040.721±0.0050.721\pm0.005
100020010.979±0.0010.979\pm0.0010.883±0.0170.883\pm0.0170.989±0.0020.989\pm0.0020.618±0.0030.618\pm0.0030.716±0.0040.716\pm0.004
100020040.995±0.0000.995\pm0.0000.967±0.0030.967\pm0.0030.997±0.0000.997\pm0.0000.615±0.0030.615\pm0.0030.719±0.0040.719\pm0.004
1000200160.999±0.0000.999\pm0.0000.990±0.0010.990\pm0.0010.999±0.0000.999\pm0.0000.614±0.0030.614\pm0.0030.720±0.0040.720\pm0.004
(b) LiveCodeBench
3471310.996±0.0020.996\pm0.0020.773±0.0110.773\pm0.0110.980±0.0010.980\pm0.0010.273±0.0020.273\pm0.0020.957±0.0000.957\pm0.000
3471320.998±0.0010.998\pm0.0010.848±0.0070.848\pm0.0070.987±0.0010.987\pm0.0010.258±0.0010.258\pm0.0010.960±0.0000.960\pm0.000
3471350.998±0.0010.998\pm0.0010.903±0.0060.903\pm0.0060.992±0.0010.992\pm0.0010.251±0.0010.251\pm0.0010.961±0.0000.961\pm0.000

In short, the randomness of each model is modeled through Φ(xij)\Phi(x_{ij}), whose parameters are estimated by pooling information across models and items, and repeated sampling is an optional way to improve precision, not a requirement. We have clarified the probabilistic interpretation in Sections 1 and 2 of the revision, and the extension and experiments are given in Appendix A.5.

References

[1] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=chfJJYC3iL


3. [W3,Q3] Table A.2 entries

Thank you for checking the table so carefully. We re-checked every MATH-500 and BBH entry of Table A.2 of the original submission (Table A.4 of the revision) against the stored outputs of the original runs, and the values are correct. In addition, we have rerun these experiments with the exactly zero-mean initialization of θ\boldsymbol\theta in the revised algorithm, and in the updated table the MATH-500 and BBH columns are now clearly different.

4. [W4] tinyBenchmarks and Fluid Benchmarking

Thank you. We now compare cBMM with tinyBenchmarks [1] and Fluid Benchmarking [2] pipelines, together with other IRT-based approaches for efficient evaluation and benchmarking (as requested by Reviewer C7h9): PSN-IRT [3], ATLAS [4], and amortized model-based evaluation (reeval) [5] on five suites. Table R3-2 (Table A.19 of the revision) provides an overview of method characteristics and performance comparisons with cBMM. Compared to tinyBenchmarks and Fluid Benchmarking, replacing only their original py-irt calibration by cBMM makes calibration 3–171 times faster, lowers the error of tinyBenchmarks on four of the five suites, and leaves the results of Fluid Benchmarking essentially unchanged.

Table R3-2. Overview of the compared methods. The right columns summarize the results of cBMM relative to each method from Tables R3-3 and R3-4: held-out AUC for the direct estimators, and the MAE of the estimated full-benchmark accuracy for the pipelines. Speed-up is the ratio of the method's fitting or calibration time to that of cBMM. NN: neural networks; MFI: maximum Fisher information; SVI: stochastic variational inference.

MethodRoleIRT modelCalibration algorithmItem selectionAUC (cBMM vs. method)MAE (cBMM vs. method)Speed-up (cBMM vs. method)
cBMMDirect / calibrator2PLBlock MM––––
PSN-IRTDirect4PLMinibatch Adam for NN–Higher or equal–9–137$\times$
reevalDirectRaschL-BFGS–Higher–6–35$\times$
ATLASPipeline2PLmirt (EM)Adaptive MFI–Comparable7–1,397$\times$
reeval CATPipelineRaschL-BFGSAdaptive MFI–Lower8–31$\times$
tinyBenchmarksPipelineMultidimensional 2PLpy-irt (SVI)Static anchor items–Lower5–171$\times$
FluidPipeline2PL with hierarchical priorspy-irt (SVI)Adaptive MFI–Comparable3–16$\times$

Comparison setup. We added discussion of these methods in Appendix A.1 and experimental details in Appendix A.17. We distinguish direct estimation from downstream pipelines since these methods address different stages of the IRT-based evaluation protocol. For reeval, we use the non-amortized Rasch calibration variant; this comparison does not evaluate its text-embedding-based amortized variant. The experiments use the same type of CPU node and the runtime comparisons apply.

Direct estimators: PSN-IRT and reeval. We randomly hide 10% of the responses, fit on the remaining entries, and evaluate predictions on the hidden entries over 10 replicates. cBMM has the lowest log-loss on all five suites and the highest or tied-highest AUC on every suite, while running 9–137 times faster than PSN-IRT (Table R3-3).

References

[1] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 34303–34326, 2024. https://proceedings.mlr.press/v235/maia-polo24a.html

[2] Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, and Noah A. Smith. Fluid language model benchmarking. In Second Conference on Language Modeling, 2025. https://openreview.net/forum?id=mxcCg9YRqj

[3] Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, et al. Lost in benchmarks? rethinking large language model benchmarking with item response theory. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence, 2026. https://doi.org/10.1609/aaai.v40i41.40814

[4] Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V Chawla. Adaptive testing for LLM evaluation: A psychometric alternative to static benchmarks. In Forty-third International Conference on Machine Learning, 2026. https://openreview.net/forum?id=ItJEC1Mk0V

[5] Sang T. Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo. Reliable and efficient amortized model-based evaluation. In Forty-second International Conference on Machine Learning, 2025. https://openreview.net/forum?id=HDbWrsgkB9

Reviewer_snKpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a faster way to fit item response models for LLM evaluation. Starting from a matrix of correct and incorrect answers, it estimates model ability together with item discrimination and easiness. The proposed solver, cBMM, replaces the logistic objective with a quadratic surrogate and alternates between constrained least-squares updates for the parameter blocks. The authors report faster fitting than existing IRT implementations and general-purpose optimizers, with broadly similar recovered rankings. They also study how the estimates relate to human difficulty labels and how ability-based rankings differ from average accuracy. Synthetic experiments, initialization checks, bootstrap analyses, and warm-start experiments examine the method's behavior under different evaluation settings.

Critical strengths
  • The solver addresses a practical bottleneck in fitting IRT models to large response matrices. Its updates are simple, and the reported runtime improvements are accompanied by close agreement with general-purpose optimizers in the recovered ability rankings.

  • The empirical study goes beyond a single runtime comparison. It includes different missingness patterns, repeated initializations, bootstrap uncertainty for rankings, and workflows for adding new models.

Critical weaknesses
  • The interpretation of item difficulty would benefit from a clearer choice of reference scale. Shifting model ability and adjusting the item intercepts can leave the predicted probabilities unchanged while changing the intercept ordering. Explaining how this ambiguity is handled would make the identifiability result and the difficulty analysis easier to assess.

  • The treatment of missing responses could be described more clearly. The objective is defined over observed entries, but it is not obvious how this mask is reflected in all of the block updates. Providing the exact updates used in the experiments would help connect the algorithm description, convergence analysis, and missing-data results.

Questions for authors

Could you clarify how the ability scale is fixed and how missing responses are handled in the block updates?

3 comments

We thank the reviewer for the positive and precise comments. Both points improve the clarity of the paper, and the revision addresses them as follows.

1. [W1] Reference scale for item difficulty

Thank you for this suggestion, which Reviewer PWcH also raises. We agree that interpreting item intercepts requires an explicit ability origin. We therefore introduce the location constraint 1⊤θ=0\mathbf 1^\top\boldsymbol\theta=0, under which b\mathbf b is identified exactly. We have revised Theorem 1 and Algorithm 1, accordingly. The change is mild: cBMM preserves the reference scale during updates, the calibration of zero-mean initialization of θ\boldsymbol\theta is all we need. We have re-run the affected experiments using an exact zero-mean initialization for θ\boldsymbol{\theta} rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.

The reference scale removes ambiguity. The likelihood is invariant under (θ,a,b)↦(cθ+d1,a/c,b−dca)(\boldsymbol\theta,\mathbf a,\mathbf b)\mapsto(c\boldsymbol\theta+d\mathbf 1,\\ \mathbf a/c,\\ \mathbf b-\tfrac{d}{c}\mathbf a) with c>0c>0. The constraint 1⊤θ=0\mathbf 1^\top\boldsymbol\theta=0 fixes the location shift, d=0d=0: abilities are measured relative to the average model in the pool, b=1NX⊤1\mathbf b=\frac1N\mathbf X^\top\mathbf 1 is identified exactly, and −bj-b_j is the difficulty of item jj for a model of average ability. θ\boldsymbol\theta and a\mathbf a remain identified up to a common scale c>0c>0, which changes neither their orderings nor CV$(\hat{\mathbf a})$. The constraint loses nothing, because centering any solution leaves X\mathbf X, and hence the likelihood, unchanged. The revised Theorem 1 is stated in Section 3.3, followed by a discussion of why the location constraint is needed and how it identifies b\mathbf b exactly; the proof is in Appendix A.2.

Calibrating zero-mean initialization is all we need. The cBMM updates preserve the mean of θ\boldsymbol\theta exactly (Proposition 1 in Appendix A.2), so a zero-mean initialization keeps every iterate, and the limit, on the constraint set. We have revised Theorem 1 accordingly, and re-run the experiments using an exact zero-mean initialization for θ\boldsymbol{\theta} rather than the previous standard normal initialization. As expected, all conclusions of the paper remain unchanged, as the empirical mean under a standard normal initialization naturally concentrates close to zero.

Empirical check. Across 50 random initializations with centered abilities, the estimated intercept rankings have Spearman correlations of at least 0.98 on IFEval, MuSR, MATH, GPQA, BBH, and MMLU-Pro (Appendix A.12). These results support the reproducibility of the estimates under the specified reference convention.


2. [W2] Treatment of missing responses

We agree that the observed-entry objective should be accompanied by explicit masked updates. We now write out the masked updates in full in Appendix A.4, and clarify that the formulas printed in lines 7–9 of Algorithm 1 apply to complete data. The masked updates minimize the observed-entry surrogate block by block and reduce to the printed formulas when all responses are observed. We also specify the centering step used to fix the ability origin and the non-degeneracy conditions required for the masked sufficient-descent argument.

Masked block updates. Let Ωi=j:(i,j)∈Ω\Omega_i=\\{j:(i,j)\in\Omega\\} and Ωj=i:(i,j)∈Ω\Omega^j=\\{i:(i,j)\in\Omega\\} denote the observed items for model ii and the observed models for item jj, respectively. At iteration tt, we form the surrogate target Y~(t)\widetilde{\mathbf Y}^{(t)} as in line 6 of Algorithm 1. For the unidimensional model, the observed-entry surrogate is, up to an additive constant,

Gt(θ,a,b)=L2∑(i,j)∈Ω(ajθi+bj−y~ij(t))2,L=14σ2.G_t(\boldsymbol\theta,\mathbf a,\mathbf b) = \frac{L}{2}\sum_{(i,j)\in\Omega} \left(a_j\theta_i+b_j-\widetilde y_{ij}^{(t)}\right)^2, \qquad L=\frac{1}{4\sigma^2}.

Minimizing this surrogate in the order a→b→θ\mathbf a\rightarrow\mathbf b\rightarrow\boldsymbol\theta gives the following intermediate updates:

a_j^+ =\max\left\\{0, \frac{\sum_{i\in\Omega^j}\theta_i^{(t)} \bigl(\widetilde y_{ij}^{(t)}-b_j^{(t)}\bigr)} {\sum_{i\in\Omega^j}(\theta_i^{(t)})^2} \right\\}, \qquad b_j^+ =\frac{1}{|\Omega^j|} \sum_{i\in\Omega^j} \left(\widetilde y_{ij}^{(t)}-a_j^+\theta_i^{(t)}\right), \qquad \theta_i^+ =\frac{\sum_{j\in\Omega_i}a_j^+ \bigl(\widetilde y_{ij}^{(t)}-b_j^+\bigr)} {\sum_{j\in\Omega_i}(a_j^+)^2}.

The aja_j update is a scalar nonnegative least-squares solution; the other two updates are ordinary least-squares solutions. The update reduces to complete data via Ωi=1,…,J\Omega_i=\\{1,\ldots,J\\} and Ωj=1,…,N\Omega^j=\\{1,\ldots,N\\}.

2. [W2] Treatment of missing responses (cont.)

Reference-scale centering. The masked updates may not preserve the ability mean. After every block-update cycle, we impose the location constraint via

m=1N∑iθi+,θ(t+1)=θ+−m1,a(t+1)=a+,b(t+1)=b++ma+.m=\frac{1}{N}\sum_i\theta_i^+,\qquad \boldsymbol\theta^{(t+1)}=\boldsymbol\theta^+-m\mathbf1,\qquad \mathbf a^{(t+1)}=\mathbf a^+,\qquad \mathbf b^{(t+1)}=\mathbf b^++m\mathbf a^+.

This transformation leaves every score aj+θi++bj+a_j^+\theta_i^++b_j^+ unchanged, and therefore preserves both the response probabilities and the observed-data loss.

Convergence with missing responses. The convergence analysis in Appendix A.3 is now stated for the observed-entry loss and covers exactly masked updates; Theorem 2 is its complete-data special case. Only the non-degeneracy assumption changes: (H2) is specified in its observed-entry form,

min⁡j∑i∈Ωj(θi(t))2≥κ,min⁡j∣Ωj∣≥κ,min⁡i∑j∈Ωi(aj(t+1))2≥κ,\textstyle\min_j\sum_{i\in\Omega^j}(\theta_i^{(t)})^2\geq\kappa,\qquad \min_j|\Omega^j|\geq\kappa,\qquad \min_i\sum_{j\in\Omega_i}(a_j^{(t+1)})^2\geq\kappa,

which bounds the denominators of the three masked updates away from zero. Each block subproblem is then LκL\kappa-strongly convex, giving sufficient descent of the observed-data loss FF: F(W(t))−F(W(t+1))≥(Lκ/2),dt2F(\mathbf W^{(t)})-F(\mathbf W^{(t+1)})\geq(L\kappa/2)\\,d_t^2, where dtd_t is the parameter change of the block updates. Centring leaves FF and a\mathbf a unchanged and moves the iterate by at most a constant multiple of dtd_t, so both this bound and the stationarity residual bound, under which the KKT residual at W(t+1)\mathbf W^{(t+1)} is at most BdtBd_t, carry over (Lemmas 1–2). With the KL property of the loss, the masked iterates therefore have finite length and converge to a KKT point, and the conditional rates also hold.


3. [Q1] Ability scale and missing responses

Please see our responses to W1 and W2.


We thank the reviewer again for the positive and precise comments, which clarified both the identifiability result and the treatment of missing data: under the location constraint b\mathbf b is identified exactly, and cBMM attains this with only a zero-mean initialization, leaving all conclusions of the paper unchanged; and with the masked updates written out in full, the same updates now underlie the algorithm, the released code and the convergence guarantee.

We hope these clarifications will merit an increased score and confidence.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer snKp:

Thank you again for your time and effort in reviewing our paper. As the discussion period closes on November 18, we would like to check whether our response has addressed your concerns. In brief, following your comments, we have clarified both the identifiability result and the treatment of missing data: under the location constraint 1⊤θ=0\mathbf{1}^\top\boldsymbol{\theta}=0, the item intercepts are identified exactly (revised Theorem 1 in Section 3.3, proof in Appendix A.2), cBMM attains this with only a zero-mean initialization (Proposition 1 in Appendix A.2), and all conclusions of the paper remain unchanged; and with the masked updates now written out in full (Appendix A.4), the algorithm, the released code, and the convergence guarantee (Appendix A.3) all rely on the same updates.

We hope these clarifications address your concerns regarding the reference scale for item difficulty and the handling of missing responses in the block updates. As the discussion deadline approaches, please let us know if you have any remaining questions or points of clarification. We would be glad to respond promptly.

Best regards,

The Authors

Reviewer_C7h9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes cBMM, an efficient framework for LLM evaluation based on Item Response Theory (IRT). It uses the majorization-minimization principle to reformulate parameter estimation as constrained matrix factorization, enabling efficient estimation of model abilities and item characteristics. The authors provide theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world benchmarks demonstrate substantial computational speedups while maintaining competitive estimation accuracy and providing interpretable insights into model abilities and item properties.

Critical strengths
  1. Computational efficiency and scalability. The proposed cBMM algorithm achieves 41–86× speedups over the fastest baselines across six real-world benchmarks, making IRT-based evaluation more practical at scale.

  2. Theoretical foundations. The paper provides theoretical analyses of parameter identifiability and algorithmic convergence, strengthening the mathematical foundation of the proposed approach.

  3. Comprehensive empirical validation. Experiments on both synthetic and real-world datasets demonstrate competitive estimation accuracy and meaningful interpretability of model abilities, item difficulty, and discrimination.

Critical weaknesses
  1. Limited diversity of evaluation benchmarks. The evaluation focuses on MATH-500 and six benchmarks from Open LLM Leaderboard v2. It remains unclear whether the proposed framework generalizes to other important evaluation settings, such as coding, agentic tasks, multimodal understanding.

  2. Limited evaluation on recent model generations. The large-scale experiments rely on Open LLM Leaderboard v2 results collected between June and December 2024. The absence of more recent reasoning-oriented models limits the evidence that the proposed framework remains informative when evaluating increasingly capable models on saturated or highly challenging benchmarks.

  3. Lack of independent validation of evaluation quality. The real-world experiments demonstrate that cBMM can produce rankings different from average accuracy, but do not establish which rankings better reflect model capabilities. Although synthetic experiments support improved ability recovery, independent real-world validation is needed to substantiate the claimed benefits of IRT-based evaluation beyond computational efficiency.

  4. Limited out-of-sample validation. Most real-world analyses estimate and evaluate model abilities using the same response matrix. Although Appendix A.8.4 studies leave-one-model-out scoring and warm-start efficiency, it does not establish whether calibrated item parameters can reliably predict the performance of genuinely unseen model families or substantially stronger future models.

  5. Insufficient validation of output stochasticity modeling. The paper motivates its approach by emphasizing the stochastic nature of LLM outputs, yet the real-world experiments primarily use binary correctness observations without repeated generations under controlled decoding settings. Consequently, it remains unclear whether the estimated response probabilities accurately capture actual generation variability rather than merely fitting differences across models and items.

  6. Potential gaps in the convergence proof. Appendix A.3 appears to conclude that the objective has a KL exponent of 1/2 from sub-analyticity, although this property alone does not establish the claimed exponent. Moreover, the relative-error argument seems to use optimality conditions for previously updated blocks as if they held at the final iterate of the entire block-update cycle. These steps require further justification and may affect the validity of the claimed convergence guarantees.

  7. Insufficient comparison with recent LLM-specific evaluation methods. The experiments primarily compare cBMM against traditional IRT implementations and general-purpose optimization algorithms, while omitting recent LLM-specific evaluation frameworks such as PSN-IRT [1], ATLAS [2], and amortized model-based evaluation [3]. In particular, PSN-IRT directly addresses model ability estimation and item characterization, making it a highly relevant baseline. Without comparisons against these methods, it is difficult to assess the practical advantages of cBMM beyond computational efficiency.

[1] Zhou, H., et al. (2026). Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2026), 40(41), 35085–35093.

[2] Li, P., et al. (2026). Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306, 70246–70269.

[3] Truong, S. T., et al. (2025). Reliable and Efficient Amortized Model-based Evaluation. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, 60238–60265.

Questions for authors

see weakness

10 comments

We thank the reviewer for the detailed and constructive review. Below we address each point.

1. [W1] Limited diversity of evaluation benchmarks

We thank the reviewer for this suggestion. We first clarify that cBMM operates on a binary response matrix and requires no domain-specific inputs, making it applicable to any benchmark with per-item pass/fail outcomes. To broaden the empirical evaluation, we added LiveCodeBench for code generation and SWE-bench Verified for agentic software engineering. On both suites, cBMM improves held-out response prediction over an average accuracy predictor, and item parameters calibrated on weaker models transfer to stronger models. These results extend the empirical evidence beyond the original seven suites to coding and agentic evaluation.

Code generation: LiveCodeBench. LiveCodeBench [1] consists of 34 models on 713 competitive-programming problems, graded pass/fail by hidden test cases; its structure and data collection are described in the new Appendix A.14 and Appendix A.18. Table R1-1 (Table A.13 of the revision) shows that cBMM reaches held-out AUC 0.944 against 0.710 for accuracy, and that item parameters calibrated on the weaker 80% of models score the strongest 20% with AUC 0.799 against 0.753. The ability rankings are identical across 10 random initializations, with Spearman correlation 1.000.

Agentic software engineering: SWE-bench Verified. SWE-bench Verified [2, 3] contains 180 systems submitted between October 2023 and September 2026, evaluated on 500 instances. Its structure and data collection are described in Appendix A.14 and Appendix A.18. As shown in Table R1-1 (Table A.13 of the revision), cBMM achieves a held-out AUC of 0.943 versus 0.713 for average accuracy. Item parameters calibrated on the weaker 80% of systems transfer to the strongest 20%, yielding an AUC of 0.917 versus 0.540. The estimated parameter rankings are also stable across random initializations, with Spearman correlations of at least 0.998.

Table R1-1. Results on the added benchmark suites. Held-out prediction uses 10% randomly hidden responses; results are mean ±\pm standard deviation over 10 replicates. Strength extrapolation calibrates item parameters on the bottom 80% of models or systems by average accuracy and evaluates the top 20%.

SuitecBMM (Held-out AUC ↑\uparrow)Acc. (Held-out AUC ↑\uparrow)cBMM (Extrapolation AUC ↑\uparrow)Acc. (Extrapolation AUC ↑\uparrow)
LiveCodeBench.944±.003\mathbf{.944}\pm.003.710±.010.710\pm.010.799.753
SWE-bench V..943±.002\mathbf{.943}\pm.002.713±.006.713\pm.006.917.540

References

[1] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=chfJJYC3iL

[2] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world Github issues? In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=VTF8yNQM66

[3] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, et al. Introducing SWE-bench verified, 2024. https://openai.com/index/introducing-swe-bench-verified/


2. [W2] Limited evaluation on recent model generations

We address this concern in two ways. First, MATH-500 already includes reasoning models released through September 2025, and the revision adds a challenging AMC/AIME suite containing predominantly 2025 reasoning models, together with SWE-bench Verified systems submitted through September 2026 (Response to W1). On the added suites, cBMM improves held-out response prediction and supports scoring stronger models or systems using item parameters calibrated on weaker ones. Second, new experiments show that cBMM remains stable and outperforms average accuracy on saturated or highly challenging benchmarks.

Coverage of recent reasoning models. Although the six Open LLM Leaderboard suites use responses collected in 2024, MATH-500 includes numerous recent reasoning models, as listed in Table A.22 of Appendix A.18. Examples include the DeepSeek-R1-Distill family, DeepSeek-R1-0528-Qwen3-8B, Qwen3-4B-Thinking-2507, Qwen3-30B-A3B-Thinking-2507, QwQ-32B, gpt-oss-20b, and the AceReason-Nemotron and OpenReasoning-Nemotron series. Figure 2 shows that cBMM distinguishes abilities among these models, with a 2025 reasoning model ranked highest in each size tier.

2. [W2] Limited evaluation on recent model generations (cont.)

Predictive performance on a challenging AMC/AIME suite. We added 100 problems from AMC 2023, AIME 2024, and AIME 2025, evaluated on 128 model configurations, most of which are reasoning models released in 2025 (Appendix A.14 and Appendix A.18). As shown in Table R1-2 (Table A.13 of the revision), with 10% of responses hidden at random, cBMM achieves a held-out AUC of 0.935 versus 0.790 for an average accuracy predictor. When item parameters are calibrated on the weaker 80% of configurations and used to evaluate the strongest 20%, cBMM achieves an AUC of 0.860 versus 0.558. These results support the usefulness of cBMM for evaluating recent reasoning models on challenging problems.

Table R1-2. Results on the AMC/AIME benchmark suite. Held-out prediction uses 10% randomly hidden responses; results are mean ±\pm standard deviation over 10 replicates. Strength extrapolation calibrates item parameters on the bottom 80% of configurations and evaluates the top 20%.

SuitecBMM (Held-out AUC ↑\uparrow)Acc. (Held-out AUC ↑\uparrow)cBMM (Extrapolation AUC ↑\uparrow)Acc. (Extrapolation AUC ↑\uparrow)
AMC/AIME.935±.006\mathbf{.935}\pm.006.790±.010.790\pm.010.860.558

Evaluation on saturated and challenging benchmarks. We simulate benchmarks composed of regular, highly saturated, and highly challenging items, where the ground-truth average success probability lies in [0.2,0.8][0.2, 0.8], [0.95,0.99][0.95, 0.99], and [0.01,0.05][0.01, 0.05] (Appendix A.9.2). As shown in Table R1-3 (Table A.3 of the revision), cBMM ranks the models with higher Spearman correlations with the ground-truth than accuracy in every setting. The gain appears largest under saturation: with highly saturated items, Spearman correlation with the ground-truth ability is 0.838 against 0.791 among the top-20% models, since cBMM weights the few remaining informative items by their discrimination. Very difficult items hurt less, as they still separate the strongest models. Saturation thus reduces the available information but does not destabilize cBMM.

Table R1-3. Simulated regular, highly saturated and highly challenging benchmarks with different levels of mean accuracy. Reported Spearman correlation with the ground-truth ability, mean ±\pm standard deviation over 10 replicates.

BenchmarkMean acc.cBMM (All models)Acc. (All models)cBMM (Top-20% models)Acc. (Top-20% models)
Regular.499±.006.499\pm.006.999±.000\mathbf{.999}\pm.000.998±.000.998\pm.000.974±.004\mathbf{.974}\pm.004.961±.008.961\pm.008
Highly saturated.923±.001.923\pm.001.991±.001\mathbf{.991}\pm.001.985±.002.985\pm.002.838±.019\mathbf{.838}\pm.019.791±.021.791\pm.021
Highly challenging.077±.001.077\pm.001.991±.001\mathbf{.991}\pm.001.985±.002.985\pm.002.968±.004\mathbf{.968}\pm.004.953±.004.953\pm.004

3. [W3] Lack of independent validation of evaluation quality

We agree that different rankings alone do not establish better evaluation quality. We therefore assess three complementary aspects: prediction of unseen responses, reproducibility of model rankings, and agreement across related benchmarks. cBMM improves held-out prediction across all seven suites, produces substantially more reproducible rankings on MuSR and GPQA with high heterogeneity, and shows stronger agreement between independently fitted mathematics benchmarks than average accuracy. Together, these analyses provide evidence for its evaluation quality beyond computational efficiency. We have reported these results in Appendix A.15.

Prediction of unseen responses. We randomly hide 10% of the responses, fit on the remaining entries, and evaluate predictions on the hidden entries over 10 replicates. As shown in Table R1-4 (Table A.14 of the revision), cBMM achieves lower mean log-loss and higher mean AUC than average accuracy on every suite. For example, its held-out AUC is 0.869 versus 0.538 on MuSR and 0.761 versus 0.539 on GPQA. These results show that modeling model–item response probabilities improves prediction of unseen outcomes beyond using model-level average accuracy.

Table R1-4. Held-out response prediction with 10% of entries hidden at random. Results are mean ±\pm standard deviation over 10 replicates. CV is the coefficient of variation of estimated item discrimination from the full fit.

SuiteCV$(\hat{\mathbf a})$cBMM (Log-loss ↓\downarrow)Acc. (Log-loss ↓\downarrow)cBMM (AUC ↑\uparrow)Acc. (AUC ↑\uparrow)
MATH-5000.44.368±.008\mathbf{.368}\pm.008.505±.006.505\pm.006.912±.003\mathbf{.912}\pm.003.821±.005.821\pm.005
IFEval0.48.424±.002\mathbf{.424}\pm.002.584±.001.584\pm.001.882±.001\mathbf{.882}\pm.001.749±.002.749\pm.002
MuSR1.33.436±.001\mathbf{.436}\pm.001.673±.000.673\pm.000.869±.001\mathbf{.869}\pm.001.538±.001.538\pm.001

3. [W3] Lack of independent validation of evaluation quality (cont.)

Table R1-4 (cont.)

SuiteCV$(\hat{\mathbf a})$cBMM (Log-loss ↓\downarrow)Acc. (Log-loss ↓\downarrow)cBMM (AUC ↑\uparrow)Acc. (AUC ↑\uparrow)
GPQA1.13.507±.001\mathbf{.507}\pm.001.605±.001.605\pm.001.761±.001\mathbf{.761}\pm.001.539±.001.539\pm.001
MATH0.58.156±.001\mathbf{.156}\pm.001.230±.001.230\pm.001.930±.001\mathbf{.930}\pm.001.800±.001.800\pm.001
BBH0.95.480±.000\mathbf{.480}\pm.000.670±.000.670\pm.000.848±.000\mathbf{.848}\pm.000.617±.000.617\pm.000
MMLU-Pro0.95.380±.000\mathbf{.380}\pm.000.593±.000.593\pm.000.885±.000\mathbf{.885}\pm.000.651±.000.651\pm.000

Reproducibility of model rankings. Predictive improvements alone do not establish better rankings. We therefore randomly divide each benchmark's items into two disjoint halves, estimate model rankings independently on each half, and measure their agreement. Table R1-5 (Table A.15 of the revision) shows similar reliability for cBMM and average accuracy on suites with relatively homogeneous discriminations. The advantage is substantially larger on MuSR and GPQA, where the rankings differ most from average accuracy. Among the top 20% of models, cBMM achieves Spearman correlations of 0.799 versus 0.250 on MuSR and 0.867 versus 0.521 on GPQA. Rankings by accuracy on these suites are therefore largely not reproducible across item subsets, whereas cBMM-based ability rankings are.

Table R1-5. Split-half ranking reliability: Spearman correlations between rankings fitted on two disjoint random halves of the items, reported as mean ±\pm standard deviation over splits. CV is the coefficient of variation of estimated item discrimination from the full fit.

SuiteCV$(\hat{\mathbf a})$splitscBMM (All models)Acc. (All models)cBMM (Top-20% models)Acc. (Top-20% models)
MATH-5000.44500.989±0.0020.989\pm0.0020.990±0.002\mathbf{0.990}\pm0.0020.784±0.068\mathbf{0.784}\pm0.0680.765±0.0690.765\pm0.069
IFEval0.48200.990±0.001\mathbf{0.990}\pm0.0010.988±0.0020.988\pm0.0020.872±0.020\mathbf{0.872}\pm0.0200.866±0.0210.866\pm0.021
MuSR1.33200.925±0.039\mathbf{0.925}\pm0.0390.774±0.0340.774\pm0.0340.799±0.067\mathbf{0.799}\pm0.0670.250±0.0670.250\pm0.067
GPQA1.13200.956±0.012\mathbf{0.956}\pm0.0120.745±0.0390.745\pm0.0390.867±0.032\mathbf{0.867}\pm0.0320.521±0.0760.521\pm0.076
MATH0.58200.988±0.001\mathbf{0.988}\pm0.0010.987±0.0010.987\pm0.0010.955±0.008\mathbf{0.955}\pm0.0080.949±0.0070.949\pm0.007
BBH0.95100.996±0.001\mathbf{0.996}\pm0.0010.992±0.0020.992\pm0.0020.981±0.007\mathbf{0.981}\pm0.0070.973±0.0050.973\pm0.005
MMLU-Pro0.95100.999±0.000\mathbf{0.999}\pm0.0000.997±0.0000.997\pm0.0000.981±0.0070.981\pm0.0070.986±0.003\mathbf{0.986}\pm0.003

Agreement across related benchmarks. Reliability establishes ranking stability, but not validity. We therefore examine whether the rankings agree across benchmarks targeting related capabilities. We fit MATH-500 and AMC/AIME separately and compare their rankings of 127 shared model configurations. As shown in Table R1-6 (Table A.16 of the revision), the cross-benchmark Spearman correlation is 0.918 for cBMM versus 0.901 for average accuracy. The difference is 0.017, with a paired-bootstrap 95% confidence interval of [0.006, 0.031]. This supports stronger cross-benchmark consistency, although neither benchmark supplies ground-truth ability.

Table R1-6. Cross-benchmark ranking agreement between MATH-500 and AMC/AIME on shared model configurations. The confidence interval for the correlation difference is obtained from paired-bootstrap model resamples.

cBMM (Spearman correlation)Acc. (Spearman correlation)Δρ\Delta\rhoBootstrap 95% CI
0.9180.901+0.017+0.017[0.006,,0.031][0.006,\\,0.031]

4. [W4] Limited out-of-sample validation

We address this concern with three out-of-sample protocols: holding out entire developers, extrapolating from weaker to stronger models, and evaluating models released after a calibration cutoff. In every protocol, the held-out models are excluded from item calibration; their abilities are estimated using one half of the items, and their responses on the other half are predicted with the item parameters fixed. cBMM achieves higher AUC than average accuracy in every setting examined. The fixed-item ability rankings also agree closely with those from a full joint refit, supporting the stability of ranking without recalibrating on the held-out models. Details are provided in Appendix A.16 (Tables A.17 and A.18).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for their valuable comments and insightful suggestions. We are glad that reviewers find the research problem a real and important challenge for current LLM evaluation practice (PWcH), and that cBMM addresses a practical bottleneck in fitting IRT models to large response matrices (snKp). We are also encouraged that reviewers appreciate the computational efficiency and scalability of cBMM, with 41–86× speedups over the fastest baselines (C7h9) and simple updates whose recovered ability rankings closely agree with general-purpose optimizers (snKp), as well as its theoretical guarantees for identifiability and convergence (C7h9, PWcH). Moreover, reviewers find the experiments comprehensive (C7h9, PWcH), going beyond a single runtime comparison (snKp), with meaningful interpretability of model abilities, item difficulty, and discrimination (C7h9).

In the following responses, we address each reviewer comment and concern point by point. We summarize the changes we have made below.

  • Added the reference scale of the item intercepts through the location constraint 1⊤θ=0\mathbf 1^\top\boldsymbol\theta=0, and stated explicitly in Theorem 1 (Section 3.3, Appendix A.2) and algorithm initializations. [snKp, PWcH]
  • Clarified the role of the KL exponent and the block-wise optimality conditions in Theorem 2 (Section 3.4, Appendix A.3), with the linear rate stated under the corresponding exponent condition. [C7h9]
  • Clarified the masked block updates for algorithm under missing responses in Appendix A.4. [snKp]
  • Clarified the probabilistic interpretation in Sections 1 and 2. [C7h9, PWcH]
  • Added the LiveCodeBench, SWE-bench Verified, and AMC/AIME suites in Appendix A.14 and Appendix A.18. [C7h9]
  • Added simulations of saturated and highly challenging benchmarks in Appendix A.9.2. [C7h9]
  • Added held-out prediction, ranking reproducibility, and cross-benchmark transferability analyses in Appendix A.15. [C7h9]
  • Added out-of-sample validation on unseen developers, stronger models, and later-released models in Appendix A.16. [C7h9]
  • Added comparisons with PSN-IRT, ATLAS, reeval, tinyBenchmarks, and Fluid Benchmarking in Appendix A.17, with discussion in Appendix A.1. [C7h9, PWcH]

These additions further substantiate the versatility and empirical advantages of cBMM:

  • Domain-agnostic. cBMM takes binary responses as input and applies to arbitrary evaluation domains, as validated on code generation, agentic software engineering, and competition mathematics with recent reasoning models.
  • Better evaluation, not just faster. It predicts held-out responses better than average accuracy, with more reproducible rankings.
  • Transferable calibration. Item parameters calibrated once carry over to unseen developers, stronger models, and later-released models.
  • Complementary to LLM-specific methods. It attains the lowest held-out log-loss among direct estimators (PSN-IRT, reeval), and can be readily incorporated into recent IRT-based efficient-evaluation pipelines (ATLAS, reeval CAT, tinyBenchmarks, Fluid Benchmarking) as a more accurate and substantially faster calibrator.

All changes are highlighted in blue in the revised manuscript. We hope our responses address your concerns and would be happy to provide further clarification on any concerns you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.