acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Item Response Theory for LLM Evaluation with Item-Marginalized Maximum Likelihood

14 views

Abstract

Item Response Theory (IRT) is increasingly used to rank language models, analyze benchmarks, and reduce evaluation cost. Yet conventional IRT estimation was developed for a response matrix geometry with many examinees and comparatively few items. Modern LLM evaluation often reverses this regime, comparing a small number of models on hundreds or thousands of benchmark items. We study the consequences of this mismatch and develop item-marginalized IRT estimation as an alternative formulation which treats evaluated model abilities as fixed inferential targets and benchmark items as random effects. Across simulations calibrated to six benchmarks, we find a systematic geometry-dependent crossover. Conventional person-marginalized IRT performs better when models are abundant, whereas item marginalization becomes increasingly advantageous as the response matrix becomes item rich. With only 12 models, item marginalization reduces average Kendall ranking loss across six benchmarks by 56% relative to person marginalization. This advantage translates directly to efficient LLM evaluation. With only 12–40 calibration models, item-marginalized item selection reduces held-out ranking loss by 18%, enabling IRT-based benchmark distillation from only a few dozen fully evaluated models rather than requiring large historical response matrices. On real sparse response matrices, item marginalization also consistently improves ranking recovery over conventional IRT. Mechanism and robustness experiments attribute these gains to improved population generalization of item characteristics and show that the crossover persists across alternative IRT models and strongly non-Gaussian latent populations. These results suggest that the direction of marginalization should reflect the geometry and generalization target of the evaluation, making IRT better aligned with modern LLM evaluation.

Primary area:
datasets and benchmarks
Keywords:
LLM Evaluation, Item Response Theory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bRLXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an item-marginalized IRT approach to better accommodate LLM evaluation, where the number of models is limited but the number of benchmark items is large. The authors compare the proposed method with conventional person-marginalized IRT through simulations and real response data, demonstrating improvements in model ranking and benchmark item selection.

Critical strengths

The motivation is very clear and well justified. I am somewhat surprised that this perspective has not been explored earlier in IRT, perhaps because similar application scenarios did not exist before. I find the idea interesting and potentially useful.

Critical weaknesses
  1. Lack of theoretical justification.

The paper does not provide sufficient theoretical results. I expect that much of the required theory should be relatively standard, but I think it is necessary to include it to make the contribution sufficiently substantial.

  1. Practical relevance and experimental validation.

I have some concerns about whether the formulation sufficiently reflects real LLM evaluation. Why is (Y) restricted to binary responses? I initially expected an item to correspond to an entire benchmark, in which case the observation would be a benchmark score rather than 0/1. Moreover, different benchmarks have different score scales. I feel that these practical considerations have not been sufficiently explored. Since the method is motivated by a real application, it should closely reflect that application; otherwise, the formulation risks becoming a toy model.

More importantly, does IM significantly outperform simple random item selection followed by averaging accuracy in real applications? I do not think the paper provides sufficient evidence for this, and this is my main concern.

Real LLM benchmarks also have substantial structure. Model performance in mathematics, coding, knowledge, and reasoning can be quite different, and may not be adequately captured by a unidimensional ability model and a common item distribution.

Minor writing issue: There are also some AI-style expressions. For example, terms such as "calibrated" and "calibration" appear repeatedly, but their precise meaning is not always clear from the main text.

Questions for authors
  1. Can the authors provide theoretical guarantees for the proposed estimator, particularly in the regime with a fixed number of models and an increasing number of items?

  2. Could the authors clarify the choice of binary responses at the individual-question level? How would the formulation accommodate benchmark-level scores with different scales?

  3. Can the authors provide stronger real-data evidence that IM significantly outperforms simple random item selection and raw accuracy averaging? This is the question that would most affect my assessment.

  4. How sensitive is the proposed method to the unidimensional IRT assumption, given the substantial heterogeneity across different types of LLM tasks?

2 comments

Dear Reviewer bRLX,

thank you for your valuable feedback. Since the main weaknesses are also reflected in the questions for authors, we directly address the four questions below.

Response to Question 1

Under correct model specification and standard regularity conditions, we can establish ranking consistency of IM for fixed MM and increasing JJ. Since items are independently sampled, the IM marginal likelihood is a sum of JJ independent terms. By the law of large numbers and a KL-divergence argument, the estimated response distribution converges to the true distribution. In the 2PL model, marginal response probabilities are strictly increasing in model ability because discrimination is positive. Therefore, the estimated model ranking converges to the true ranking when abilities are distinct.

Response to Question 2

Following previous IRT applications in LLM evaluation, such as MetaBench and tinyBenchmarks, we use binary responses at the individual question level, where 1 indicates a correct answer and 0 an incorrect answer. We fit IRT separately for each benchmark, so different benchmark score scales do not affect our estimation. The general IM formulation can also accommodate other response types by using an appropriate likelihood, which we consider valuable future work.

Response to Question 3

Experiment 3 directly uses real response matrices from the OpenLLM Leaderboard. IM generally outperforms raw accuracy in 21 of 30 conditions and 2 improvements remain statistically significant after Holm correction.

To further address this concern, we conducted an additional real-data experiment on full-bank accuracy prediction. For each benchmark, we fitted PM and IM using the responses of 12 LLMs to all benchmark items. We then predicted full benchmark accuracy for 256 held-out models using only their responses to the same randomly sampled subsets of 12, 24, or 40 items.

Mean RMSE across six benchmarks (percentage points, lower is better):

Queried itemsRandom averagingPMIM
1212.25710.2088.211
248.4748.3926.164
406.5427.3054.980

IM achieves lower RMSE than raw accuracy averaging in all 18 benchmark–budget conditions, with all 18 improvements remaining statistically significant after Holm correction. These results provide additional real-data evidence for the benefit of IM in reconstructing full-bank accuracy from limited responses.

We would also like to emphasize that our main aim is to propose a suitable IRT estimation method for LLM evaluation, rather than a complete benchmark construction pipeline such as MetaBench, which also recognizes raw accuracy as a strong baseline. IRT has broader applications in LLM evaluation, including annotation error detection, adaptive testing, and reliability analysis, where accurate ability and item parameter estimation is important. Based on our theory and experiments, we believe IM has potential benefits for these applications.

Response to Question 4

To address this concern, we generated two-domain Gaussian IRT data with varying task-ability correlations (ρ\rho), fitted unidimensional PM and IM with 50 replications per condition, and evaluated parameter recovery.

The following table shows a representative setting (M=24,J=2000M=24, J=2000), with PM / IM results averaged over 50 replications.

Task correlation ρ\rhoAbility Kendall's τ\tauDifficulty Pearson's rrLog-discrimination Pearson's rr
1.00.952 / 0.9800.871 / 0.9380.674 / 0.768
0.90.946 / 0.9750.862 / 0.9360.639 / 0.748
0.70.933 / 0.9650.843 / 0.9330.585 / 0.710
0.50.908 / 0.9490.821 / 0.9300.539 / 0.673
0.00.807 / 0.8810.755 / 0.9200.414 / 0.541

Stronger multidimensionality reduces recovery accuracy for both methods, while IM retains its advantage in this item-rich setting. This suggests that the relative advantage of IM can persist even when the unidimensional assumption is violated.

Thanks for the response. I'll maintain my original rating.

Reviewer_qC3VOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers item-respose theory (IRT) as a tool for jointly assessing language model performance on a benchmark and the difficulty (and in some cases discrimination ability) of each benchmark item. IRT was developed for assessing human abilities in settings with many evaluees (humans) and relatively few test items, and the authors note (following recent work by Jiang et al.) that the practical uses of IRT for language models face the opposite: relatively few evaluees (models) and many test items. They propose to treat evaluee parameters as fixed and item parameters as random, as opposed to the reverse as done in standard IRT. This is called item-marginalized IRT (as opposed to person-marginalized IRT). This has been done before, but, the authors claim, not for language model evaluation.

The experiments consider three questions. (1) In simulation, how does shifting the balance between more evaluees/fewer items and fewer evaluees/more items change the effectiveness of the two IRT variants? The experiments show a clear trend where the ratio between evaluees and items predicts the accuracy of the approach consistently in the hypothesized direction. Analysis suggests that item marginalization “reduces the influence of poorly identified, sample-specific item effects on model ability estimation.” (2) With real benchmark data, does item-marginalized IRT give more efficient benchmark construction when selecting items (using full item set benchmark rankings as a reference)? Experiments demonstrate that it does, consistently, across benchmarks. (3) From a "cold start," can item-marginalized IRT achieve more accurate rankings with few models and items than person-marginalized IRT? The experiments show that it can.

The work is important as it supports the consideration of IRT architecture as an element of design when applying these models to modern language model development, and shows a way toward reducing the cost of benchmark construction (by reducing the data required) and other IRT applications (e.g., experimentation to support model hyperparameter and architecture choices).

Critical strengths

The motivation for this change to item-marginalized IRT is well argued.

The technical explanation of the approach is very clear.

The experiments are very well designed to answer the research questions; a raw averaging baseline is included for the cold start experiments, providing an honest disclosure that this simple approach wins over IRT when budgets are very small.

The conclusions are well scoped, with extensive sensitivity analysis in the appendices and a clear statement of when the new method is likely preferable and when the conventional one is.

Limitations are clearly discussed.

Critical weaknesses

The paper is remarkably clear and well-constructed; though the methodological contribution is focused and may be of niche interest, I see no compelling reason not to publish the work.

Questions for authors

None

1 comment

Dear Reviewer qC3V,

many thanks for your positive and encouraging feedback. We hope our work can make a useful contribution to future research in LLM evaluation and AI measurement science.

Reviewer_oBCMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines the regular LLM-evaluation setting, where few models answer many items. Standard IRT (person-marginalized; PM) treats model abilities as random and item parameters as fixed. The authors flip this approach (item-marginalized, IM). Experiments on six MetaBench benchmarks and on real MetaBench data show that: 1) PM is better when there are more models, and IM is better when there are more items.2) Efficient benchmarking: IM-based selection works better when only 12–40 calibration models are available. 3) Cold-start evaluation: IM ranks models better than PM.

Critical strengths
  1. The paper is well motivated, presenting an interesting problem.
  2. The method is clear and elegant and is supported by well-designed experiments.
  3. Broad robustness and ablation studies.
  4. The applications for efficient evaluation are a good additional benefit of the method, showcasing how it can be used.
Critical weaknesses
  1. Missing baseline: The biggest problem. In the PM setup, item parameters are fixed effects with no prior, so each one is estimated from only 12–24 responses. IM puts a population distribution over items, which shrinks extreme estimates toward the middle. This can be interpreted as "shrinking item parameters helps when each item has little data," not anything specific to which axis is marginalized.
  2. Small absolute gains, presented as large relative ones. "56% reduction in ranking loss" is about +0.027 to +0.04 in Kendall's τ, measured on 12 models where τ is already about 0.95.
  3. The cold-start claim is stronger in the abstract than the data supports. IM beats raw accuracy in only 2 of 30 conditions after correction.
  4. The efficient evaluation section lacks comparisons. No comparison with other methods, tiny benchmarks, or others. If it complements, show the aggregate improvements.
Questions for authors

The paper states that the crossover holds for every IRT model, but it seems that the 3PL/U4PL item-parameter results don't support that. Is it necessary to discuss this further or to narrow the findings?

4 comments
~Han_Jiang2OpenReview ↗

Thanks for for building on our work & A question about the sources of IM's performance gains

Hi, thank you for all the effort you've put into this work!

I think Reviewer oBCM raises an interesting point about whether the gains come primarily from regularizing item parameters rather than reversing the direction of marginalization.

I also noticed that the performance gap between IM and PM narrows under VI in App. D, which makes me curious about how much the improvement depends on the estimation method, or whether a similar crossover would still occur with other estimation approaches, especially those that already regularize item parameters.

Dear Reviewer oBCM,

thank you for your valuable feedback. Our responses to the weaknesses and question are below.

Response to Weakness 1

To address this concern, we added an L2-regularized PM baseline that retains the original person-marginal likelihood while penalizing item parameters.

ℓPM−L2=ℓPM−λ2∑j=1J(bj2+αj2).\ell_{\mathrm{PM-L2}} = \ell_{\mathrm{PM}} - \frac{\lambda}{2} \sum_{j=1}^{J} (b_j^2 + \alpha_j^2).

We evaluated four regularization strengths (λ∈0.01,0.1,1,10\lambda\in\\{0.01,0.1,1,10\\}). Weak regularization slightly improves PM performance. However, IM still achieves higher Kendall's τ\tau than the best regularized PM in both item-rich settings:

MethodλM=12, J=4000M=24, J=2000
PM—0.9400.930
PM-L20.010.9410.933
PM-L20.10.9400.935
PM-L210.8950.906
PM-L2100.9000.918
IM—0.9740.961

These results suggest that the advantage of IM cannot be explained solely by shrinkage of item parameters. While regularization can improve PM estimation, IM additionally accounts for item-parameter uncertainty through marginalization.

Response to Weakness 2

We agree that the absolute gains in Kendall's τ\tau are relatively small. However, the improvements of 0.0335 for 12 models and 0.0313 for 24 models correspond to approximately 1.1 and 4.3 fewer incorrectly ranked model pairs, respectively, which can still be meaningful for model comparison. We will clarify the absolute gains alongside the relative reductions in ranking loss.

Response to Weakness 3

We would like to clarify that our abstract states that in cold start experiment IM improves ranking recovery over conventional IRT (PM), rather than over raw accuracy. In Experiment 3, IM outperforms PM in all 30 conditions, with 24 significant differences after Holm correction. We agree that raw accuracy remains a strong baseline, and IM outperforms it in 21 of 30 conditions, with 2 significant differences after correction.

Response to Weakness 4

We agree that a comparison with tinyBenchmarks would be valuable. However, tinyBenchmarks relies on full-bank response data from a relatively large number of previously evaluated LLMs to select anchor items and estimate IRT parameters. For example, its OpenLLM Leaderboard experiments use approximately 296 calibration models. In contrast, our efficient evaluation experiments specifically investigate the regime with only 12–40 models. Our primary objective is to propose a suitable IRT estimation method for LLM evaluation, and we see developing a corresponding benchmark compression pipeline based on our method as an interesting direction for future work.

Response to Question

Under 3PL and U4PL, the crossover remains clear for model ability ranking and item difficulty, while the results for discrimination and asymptote parameters are less consistent. We already mention in the Discussion that PM can be preferable when discrimination is the primary inferential target. We thank the reviewer for pointing this out and will carefully review the manuscript to avoid overgeneralizing our findings.

Dear Han,

thank you for your comment and for your work “Can We Trust Item Response Theory for AI Evaluation?”, which is an important motivation for our study. Congratulations on its acceptance at NeurIPS 2026!

Regarding the regularization concern raised by Reviewer oBCM, we have addressed it above. Our additional experiments suggest that IM's advantage cannot be explained solely by shrinkage of item parameters.

As discussed in our paper, we focus on MML as a classical and widely used IRT estimation method and additionally evaluate VI as a robustness check. I noticed that you also compared MCMC and PSN in your paper. MCMC provides a Bayesian alternative but can be computationally demanding at benchmark scale, as also demonstrated in your study. PSN follows a neural estimation approach rather than the marginal-likelihood framework studied here.

Thank you for your response. They addressed my concerns. If there were 3.5, I would raise my score to 3.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their constructive feedback and discussions. During the rebuttal, we provided additional evidence to address the concerns.

  1. Regularization (Reviewer oBCM): We introduced an L2-regularized PM baseline. While regularization can improve PM, IM consistently achieves better model ranking in item-rich settings, suggesting that its advantage cannot be explained solely by parameter shrinkage.

  2. Practical relevance (Reviewer bRLX): We conducted an additional real-data experiment involving 256 held-out LLMs across six benchmarks. IM outperforms random averaging in all 18 benchmark–budget conditions for full-bank accuracy prediction, with all improvements statistically significant after Holm correction.

We also clarified the practical significance of ranking improvements, cold-start results, the use of binary responses, and the scope of our findings. Furthermore, we provided a theoretical argument for ranking consistency and additional sensitivity analyses under the unidimensional IRT assumption.

Regarding reviewer feedback, Reviewer qC3V gave a clear acceptance (4). Reviewer oBCM indicated that they would raise their rating from weak acceptance (3) to 3.5 if the option were available. Reviewer bRLX maintained their original weak acceptance rating (3).

Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.