acceptodds
Under review as a conference paper at ICLR 2027

Diagnostic Fine-Grained Ability Profiling for Large Language Model Evaluation

Abstract

Evaluating large language models (LLMs) is a prerequisite for characterizing their ability boundaries and for guiding subsequent model improvement and real-world deployment. Current practices, however, rely predominantly on aggregate benchmark scores, which conceal the heterogeneity of underlying abilities and exhibit inconsistent rankings across evaluation settings. To address this limitation, we propose a diagnostic framework that estimates multidimensional ability profiles directly from test item level responses. This framework dynamically constructs a structured Q-matrix that explicitly represents the associations between test items and the fine-grained abilities they require. It then integrates this mapping with multidimensional Item Response Theory (mIRT) to jointly estimate an interpretable ability profile for each model, rather than collapsing distinct abilities into a single aggregate score. Instantiated with a 35-dimensional mathematical ability taxonomy, this approach demonstrates strong criterion validity (ability scores aligning with observed accuracy on associated items), high cross-benchmark reliability (consistent model rankings across benchmark pairs), and accurate prediction on held-out items. Furthermore, it generalizes to evaluation in physics, chemistry, and computer science, preserving high validity and reliability under varying item distributions. Overall, our framework replaces the single aggregate score with a multidimensional ability profile, providing a principled basis for more reliable, ability-aware evaluation in support of model improvement and deployment.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
model evaluation, fine-grained ability, evaluation pipeline, item response theory, model performance prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bvfSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors explore the use of multidimensional IRT (item response theory) as an evaluation paradigm that is more stable in ranking models and more interpretable. They use a taxonomy created by a human-in-the-loop process where an LLM is prompted to create candidate item level ability categories which are then arranged hierarchically and then the final taxonomy is annotated on benchmark items by a prompted LM as well. These item-ability annotations provide a so-called "Q matrix" which which is used to mask items that are not relevant to a given ability when fitting the IRT model. They adapt the NeuralCD framework (Wang et al., 2023) which essentially uses a 2 parameter IRT set up for each ability category to build a vector x of all the abilities which is input to an MLP and trained against the correct or incorrect item level responses. To assess their approach the authors provide a correlation between estimated ability scores and the model accuracies on the items used to fit the ability estimates, a correlation between abilities on different benchmarks when fitting on each benchmark separately, and one held out evaluation where they predict unseen item responses based solely on the LLM assigned ability categories and the fitted model ability estimates. In this third metric they compare against Adele, another multidimensional IRT baseline and find that their method gets better AUC on predictions. They then conduct the same set of experiments on 3 additional domains but do not compare against Adele on those.

Critical strengths
  1. The paper engages with a very real problem of model ranking instability across benchmarks, and I believe IRT is a very promising avenue to approach this problem.
  2. They demonstrate that a straight forward mIRT method can get good correlation across benchmarks within a domain and can predict held out items based on what abilities they are judged as requiring.
  3. They provide the prompts used for their LM-assisted taxonomy creation process which could help implement this kind of approach over new domains.
Critical weaknesses
  1. The paper only compares to one baseline (Adele) in one experiment even though many other methods from previous work would be appropriate baselines. It's also not clear why the authors compare to the Adele only in the math domain but not in the other 3 domains (I believe it should be possible since Adele is not domain specific). Relatedly the authors do not describe how exactly they make unseen item predictions with Adele. They simply say they use "an identical procedure" to the one they use on their approach but it's not clear to me how that would work with Adele.
  2. They make factually wrong claims about related work: at line 459 they describe a list of citations including Rodriguez et al. (2021) as "predominantly model a single latent trait and do not estimate multidimensional ability profiles". But Rodriguez explicitly explores multidimensional IRT with their "IRT-vec". The paper also does not cite other significant mIRT work like earlier work like tinyBenchmarks (Maia Polo et al., 2024). The only mIRT work that they engage with is Adele and they claim (line 070) "it still operates on a fixed, predefined ability taxonomy, treating model abilities as immutable dimensions rather than dynamic, context-dependent constructs jointly shaped by benchmark designs and task contexts." It seems a stretch to say that this distinguishes Adele from their work since the paper's method also creates a fixed taxonomy through a human-in-the-loop taxonomy building process, and nothing prevents the Adele approach from similarly being used with different domain specific taxonomies produced by a manual process.
  3. The methods in the paper are only partially defined in the body of the paper. For instance section 2.4 which defines the paper's IRT model, just defines how x is built and then says it "is the input to the subsequent MLP layer" but does not describe how that MLP is optimized or what its targets are. This information is only available through inspection of the anonymous code repo and appendix.
Questions for authors

How exactly did you make predictions with Adele for your results in figure 5? And is there a reason you could not include Adele results on your 3 additional domains?

Reviewer_ZF4sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper focusses on getting more insights out of benchmarks. It does so by combining multidimensional IRT with an LLM-derived taxonomy via a Q matrix. The authors show that their approach behaves sensibly on a range of benchmarks from maths over physics to chemistry.

I believe that the authors want to point out the combination of mIRT, Q-matrix and taxonomy as their main contribution - but I question how novel this is. I also do not see a demonstration of the impact of this methodology. The paper seems to mostly show that the method behaves in a sensible way.

Critical strengths
  • There is certainly a need to go beyond simple accuracy scores and to understand better what benchmarks are measuring. Approaches inspired by psychometrics can be a promising way to do so.
  • Using a taxonomy to do this analysis can be a meaningful way to do this analysis in a more rigorous way
Critical weaknesses
  • I realize it is a very annoying point for the authors to hear, but I do not fully see what the work adds beyond what is currently already out there. By now, IRT is very widely used, also in multidimensional form. It also has been coupled to LLMs before:
  • An obvious other paper that should probably be discussed/put into context are the "general scales" published in Nature https://arxiv.org/abs/2503.06378
  • CAIMIRA is methodologically close except for the learned skills in the Q matrix that are used in this work
  • and one can find many one - even just for multidimensional IRT one could perhaps highlight tinybenchmarks
  • The latter also links to my other main concern with the paper: It does not really clarify what we gain/learn with this method. Are there any predictions we can make using the insight gained with this benchmark? In tinybenchmarks, for instance, one could make benchmarks smaller/cheaper. Can the analysis method from this work here make some predictions one could test? For instance some skill loading one could then intervene one? This would be some added value. And it would be even more valuable if this were something one couldn't estimate with other existing methods
Questions for authors
  • What do you see as the main differentiation from other works that do mIRT and link it to LLM-driven analysis? What does your method enable that the existing methods cannot do? Where might your method be worse?
  • Is there a transferable insight about current benchmarking practice/a not obvious actionable prediction that you can make with your method but with none of the existing ones (e.g. https://arxiv.org/abs/2603.14676 )
1 comment

Dear authors,

thanks for putting in the time to reply to my comments!

I am afraid that I won't be able to address my score unless those two points are really addressed

W5: The latter also links to my other main concern with the paper: It does not really clarify what we gain/learn with this method. Are there any predictions we can make using the insight gained with this benchmark? In tinybenchmarks, for instance, one could make benchmarks smaller/cheaper. Can the analysis method from this work here make some predictions one could test? For instance some skill loading one could then intervene one? This would be some added value. And it would be even more valuable if this were something one couldn't estimate with other existing methods Q2: Is there a transferable insight about current benchmarking practice/a not obvious actionable prediction that you can make with your method but with none of the existing ones (e.g. https://arxiv.org/abs/2603.14676 )

For me, this would be addressed by concrete experimental findings: Concrete measurements of speedups or insights I could not gain with any other method.

To me, the current framework reads like one more other approach to analyze multiple benchmarks in a more cohesive way -- but beyond the others telling me it is better, I do not see clear experimental data showing me that it really is better. (This also shows in the discussion here: Much of the discussion has been around what previous work did - not not discussing concrete experimental findings in this paper)

Reviewer_PZybOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

First off, the figures are hard to read or parse (Figure 3 and later) and results are poorly written / presented.

Motivation: Benchmarks' scores reveal aggregate performance but doesn't help understand specific abilities and how models perform. The authors decompose the evals into capabilities and performance per capability and analyze the models, without changing how they are evaluated.

Method summary: Authors are inspired by the bloom framework to create axes for each model response. The broad axes unifying the axes are domain knowledge, cognitive processes that are adapted for the LLMs. First, the authors create a model-item response matrix (i.e. forward pass on all models on all the rows of the benchmark and have a binary 1/0 indicating pass or fail, based on the benchmark's own evaluator). Then, each item is decomposed into abilities that denote if that row requires a specific ability (the matrix is QQ). The axes of QQ is determined by "standard mathematical curricula" and an LLM is used to map each item to the specific abilities. Ultimately, the authors learn AA , IdiffI_{diff} , and IdiscI_{disc} that tell the ability of the model, how difficult each item was and unsure what the second term is (how unique that item is for each ability ?).

Unclear from the paper what the loss function is from the text. My guess is the neural network predicts what the final outcome is.

The authors primarily use math benchmarks and analyze 41 LLMs - the benchmarks are used for both evaluating this framework and ability to predict cross-benchmark performance. The LLM used to assign items to abilities is not mentioned in the main text. R1: The authors show strong correlations in 27 dimensions across two benchmarks (strong being greater than 0.7 spearman correlation) and weaker dimensions are due to the lack of rows evaluating these dimensions. R2: In cross-benchmark prediction, authors report 14 dimensions having moderate correlation (greater than 0.5) (the strong seem sparse). Finally, the authors look at other domains (Chemistry, Physics, CS) and show the axes pipeline works well for other domains.

Critical strengths
  1. Clear motivation and use of the proposed method.
  2. The math results are promising
  3. Comprehensive analyses across multiple models (which is appreciated)
  4. The human rater consistency (from the appendix) is assuring.
Critical weaknesses
  1. Discussing confounders - if the LLMs performs well, is it consistent (self-consistency), if it only performs well at certain temperatures and it doesn't, then are you evaluating the model or the (model, temperature) pair? Similarly for test-time compute / reasoning, depending on how evaluators set sampling strategies, fixed reasoning tokens is not a true representation of the performance, so abilities get muddled. Handling such scenarios requires nuance and at least some analysis on such models (Qwen3 and more).
  2. Results - Section 3 figures and writing is hard to read and understand. I recommend clean formatting, use bullets to separate the scenario, the result and implication.
  3. Creating the axes for Q - how easy or hard is it to create such dimensions ? (areas like health / law, even experts might disagree - what are good rules / guidelines the authors suggest in such cases)
  4. Limitations / handling process rewards (where intermediate steps might be correct) should also be mentioned, since having verifiable binary answers is the key to getting clear signals (presumably under LLM as judge the noise increases dramatically) - while I agree that these are pitfalls shared by existing benchmarks too, multiple seeds and iterations help reduce it a little but unclear how this framework can be setup there.
Questions for authors

The weaknesses are also questions on how / where the authors talk about this and implications of this work.

1 comment

Thank you for addressing my points, but I hope the authors agree that this is a significant revision and is not fit for acceptance. Nuanced discussions, recommendations (especially for people who'd benefit from this would be people doing evals around tricky areas like health and law and more); the analyses should also have discussions on consistency, confidence intervals and more on what can be predicted and is it pass@1, pass@k, so on. I hope this review clarified how the paper in its current form is not immediately valuable to me as a reader and good luck with submitting to future venues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.