- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an item-marginalized IRT approach to better accommodate LLM evaluation, where the number of models is limited but the number of benchmark items is large. The authors compare the proposed method with conventional person-marginalized IRT through simulations and real response data, demonstrating improvements in model ranking and benchmark item selection.
Critical strengths
The motivation is very clear and well justified. I am somewhat surprised that this perspective has not been explored earlier in IRT, perhaps because similar application scenarios did not exist before. I find the idea interesting and potentially useful.
Critical weaknesses
- Lack of theoretical justification.
The paper does not provide sufficient theoretical results. I expect that much of the required theory should be relatively standard, but I think it is necessary to include it to make the contribution sufficiently substantial.
- Practical relevance and experimental validation.
I have some concerns about whether the formulation sufficiently reflects real LLM evaluation. Why is (Y) restricted to binary responses? I initially expected an item to correspond to an entire benchmark, in which case the observation would be a benchmark score rather than 0/1. Moreover, different benchmarks have different score scales. I feel that these practical considerations have not been sufficiently explored. Since the method is motivated by a real application, it should closely reflect that application; otherwise, the formulation risks becoming a toy model.
More importantly, does IM significantly outperform simple random item selection followed by averaging accuracy in real applications? I do not think the paper provides sufficient evidence for this, and this is my main concern.
Real LLM benchmarks also have substantial structure. Model performance in mathematics, coding, knowledge, and reasoning can be quite different, and may not be adequately captured by a unidimensional ability model and a common item distribution.
Minor writing issue: There are also some AI-style expressions. For example, terms such as "calibrated" and "calibration" appear repeatedly, but their precise meaning is not always clear from the main text.
Questions for authors
-
Can the authors provide theoretical guarantees for the proposed estimator, particularly in the regime with a fixed number of models and an increasing number of items?
-
Could the authors clarify the choice of binary responses at the individual-question level? How would the formulation accommodate benchmark-level scores with different scales?
-
Can the authors provide stronger real-data evidence that IM significantly outperforms simple random item selection and raw accuracy averaging? This is the question that would most affect my assessment.
-
How sensitive is the proposed method to the unidimensional IRT assumption, given the substantial heterogeneity across different types of LLM tasks?
2 comments
Thanks for the response. I'll maintain my original rating.
Dear Reviewer bRLX,
thank you for your valuable feedback. Since the main weaknesses are also reflected in the questions for authors, we directly address the four questions below.
Response to Question 1
Under correct model specification and standard regularity conditions, we can establish ranking consistency of IM for fixed M and increasing J. Since items are independently sampled, the IM marginal likelihood is a sum of J independent terms. By the law of large numbers and a KL-divergence argument, the estimated response distribution converges to the true distribution. In the 2PL model, marginal response probabilities are strictly increasing in model ability because discrimination is positive. Therefore, the estimated model ranking converges to the true ranking when abilities are distinct.
Response to Question 2
Following previous IRT applications in LLM evaluation, such as MetaBench and tinyBenchmarks, we use binary responses at the individual question level, where 1 indicates a correct answer and 0 an incorrect answer. We fit IRT separately for each benchmark, so different benchmark score scales do not affect our estimation. The general IM formulation can also accommodate other response types by using an appropriate likelihood, which we consider valuable future work.
Response to Question 3
Experiment 3 directly uses real response matrices from the OpenLLM Leaderboard. IM generally outperforms raw accuracy in 21 of 30 conditions and 2 improvements remain statistically significant after Holm correction.
To further address this concern, we conducted an additional real-data experiment on full-bank accuracy prediction. For each benchmark, we fitted PM and IM using the responses of 12 LLMs to all benchmark items. We then predicted full benchmark accuracy for 256 held-out models using only their responses to the same randomly sampled subsets of 12, 24, or 40 items.
Mean RMSE across six benchmarks (percentage points, lower is better):
IM achieves lower RMSE than raw accuracy averaging in all 18 benchmark–budget conditions, with all 18 improvements remaining statistically significant after Holm correction. These results provide additional real-data evidence for the benefit of IM in reconstructing full-bank accuracy from limited responses.
We would also like to emphasize that our main aim is to propose a suitable IRT estimation method for LLM evaluation, rather than a complete benchmark construction pipeline such as MetaBench, which also recognizes raw accuracy as a strong baseline. IRT has broader applications in LLM evaluation, including annotation error detection, adaptive testing, and reliability analysis, where accurate ability and item parameter estimation is important. Based on our theory and experiments, we believe IM has potential benefits for these applications.
Response to Question 4
To address this concern, we generated two-domain Gaussian IRT data with varying task-ability correlations (ρ), fitted unidimensional PM and IM with 50 replications per condition, and evaluated parameter recovery.
The following table shows a representative setting (M=24,J=2000), with PM / IM results averaged over 50 replications.
Stronger multidimensionality reduces recovery accuracy for both methods, while IM retains its advantage in this item-rich setting. This suggests that the relative advantage of IM can persist even when the unidimensional assumption is violated.