acceptodds
Under review as a conference paper at ICLR 2027

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

10 views

Abstract

Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. These profiles also reveal how benchmarks share skills and how instruction-tuned models differ from their base models. We also demonstrate the utility of SkillEval in two downstream tasks: (1) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM, and (2) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance. Code is provided as supplementary material to enable reviewer reproduction; the Q-matrix, skill taxonomy, trained NCDM parameters, and per-LLM mastery profiles will be released publicly upon publication.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, cognitive diagnosis, skill profiling, model routing, interpretability, psychometrics, item response theory, benchmark analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PnzQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the SkillEval dataset, including the responses of 3,811 LLMs on 9,523 data samples from five benchmarks. Besides, the paper illustrates two usages of the SkillEval dataset: (1) Computerized Adaptive Testing and (2) LLM routing. (1) Computerized Adaptive Testing helps evaluate an unseen LLM using a trained CDM with several evaluated data samples. (2) LLM routing assigns each data sample to a capable but efficient LLM, reducing inference cost.

Critical strengths

(1) The proposed dataset is extensive, which includes results of 3,811 LLMs on 9,523 data samples. (2) The experiment is rigorous. SkillEval is evaluated through a comprehensive set of experiments, compared against IRT, IRTNet baselines. (3) The paper reads smoothly and is well written.

Critical weaknesses

(1) The idea of the paper looks similar to a series of papers. All of them use the cognitive diagnostic model for evaluating LLMs. I did not see the comparison of the SkillEval with these related papers.

General scales unlock AI evaluation with expanatory and predictive power. Nature 2026. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory. ACL 2025.

The main difference may be that the paper uses the NCDM as the cognitive diagnostic model and utilizes GPT-4o as the skill generator. In my opinion, while the NCDM has been a very popular cognitive diagnostic model, the usage of the NCDM cannot serve as the novelty of the paper.

(2) Table 3 illustrates that the SkillEval is a method, outperforming other methods. I am very confused that whether other methods are trained on SkillEval dataset. Maybe NCDM is more appropriate in the Table 3 comparison.

(3) The LLM-generated skills are difficult to understand. For example, in the Figure 5 (b), what are the meaning of "possible storage options based on context" and "work done in variable force scenarios"? The whole evaluation relies heavily on the LLM-generated skill quality.

Questions for authors

See weaknesses. And other questions: (1) The paper claims the evaluations of 3,811 LLMs. What are the 3,811 LLMs? And how are they evaluated? How many GPU hours are consumed?

(2) Are the skills generated from GPT-4o easy to understand? Can GPT-4o be replaced by other open-sourced LLMs without performance decreasement?

5 comments
AuthorsOpenReview ↗

Response to Reviewer PnzQ (1/3)

Thank you for these questions. Your summary describes SkillEval as a dataset, so we first clarify that it is a diagnostic framework. It estimates a profile over 100 named skills for each of 3,811 LLMs, and each skill is linked to the individual questions that test it. The response records come from RouterEval, which collects the per-question results of the Open LLM Leaderboard, and we ran no LLM inference. Our claims cover new questions from the benchmarks and subtasks seen in training, and new LLMs.

Held-out AUC pools all pairs of LLMs and held-out questions, as in Table 2. Within-question AUC is the AUC across LLMs for one held-out question, averaged over questions.

W1. Relationship to the general-scales study and IRT-Router.

We agree that NCDM itself is not the contribution, and the revision will add the comparison that was missing. It will also correct the Related Work appendix, which overlooks NIRT-Router, the NCDM-based variant of IRT-Router with named abilities. The approaches differ in where the ability dimensions come from. A latent ability, as in IRTNet and MIRT-Router, predicts well, but its dimensions have no names and can rotate from one training run to the next. Predefined ability lists, as in the general-scales study and NIRT-Router, have names, but they are fixed before any question is seen. SkillEval induces 100 skills from the questions of five benchmarks and links each dimension to a fixed set of questions through the resulting Q-matrix. Each dimension therefore has a name that can be checked against its questions and a meaning that stays fixed across training runs. Text-conditioned question parameters extend the same skills to questions that the model has never seen.

Ability dimensionsHow a question receives themLLMs profiled
General-scales study18 demand scales, fixed in advancea level on each scale, assigned with rubrics15
NIRT-Router25 named abilities, fixed in advanceassigned by an LLM to clusters of queries20
MIRT-Router and IRTNetunnamed latent dimensionsno named dimensions are assigned20 and 112
SkillEval100 skills induced from the questions of five benchmarksskill phrases proposed by an LLM for each question, then clustered3,811

Our experiments support these properties. Per-skill estimates correlate at 0.96 across training runs, whereas the dimensions of IRTNet correlate at 0.00, and at most 0.43 after alignment. Restricting the dimensions to named skills does not lower accuracy in ranking LLMs on unseen questions. In a rerun with the taxonomy built from the training questions alone, within-question AUC is 66.06 for SkillEval and 65.66 for IRTNet, which has about 39 times as many parameters.

The general-scales study also allows an external check. It publishes demand levels, assigned by GPT-4o with its rubrics, for 5,431 MMLU-Pro questions. For each LLM with MMLU-Pro responses on the leaderboard and for each scale, we computed how accuracy changes between demanding and other questions within subject categories. We then asked whether SkillEval profiles, learned on our five benchmarks in three training runs with the taxonomy of the paper, predict these changes for held-out LLMs beyond their overall MMLU-Pro accuracy. Among 526 LLMs, with merges and near-identical models removed, the profiles add 0.057 to R-squared, with a 95% interval from 0.035 to 0.077. The general-scales study rates what each question demands, and SkillEval estimates from responses what each LLM can do, so the two frameworks complement each other. We report this as a link between them, not as evidence of transfer to a new benchmark.

AuthorsOpenReview ↗

Response to Reviewer PnzQ (2/3)

W2. Table 3 and NCDM.

Yes, every method in Table 3 is trained on the same data. All are fitted on the responses of the 3,811 LLMs to the 7,618 training questions and evaluated on the same 1,905 held-out questions. Random routing involves no training, and the strongest single LLM is chosen by its accuracy on the training questions. Under these conditions, SkillEval routes best among the diagnostic models, with an Acc@1 of 65.46 over three training runs against 63.08 for IRTNet and 61.57 for the strongest single LLM.

SkillEval is itself an NCDM, which is why it can appear in Table 3. A standard NCDM learns a free parameter vector for every question, so it cannot score a question that it has not seen in training. SkillEval computes these parameters from the question text, which lets it score held-out questions on the same named skills. As you suggest, we ran the standard NCDM with the same Q-matrix where it can be evaluated, in the in-distribution setting of Table 2. There it reaches an AUC of 85.24, against 81.73 for SkillEval and 86.16 for IRT 2PL, since free question parameters fit the questions seen in training most closely. We will add this row to Table 2.

W3. Interpretability of the skills.

The skills are readable, and their names match what they measure. Three annotators judged 780 question-skill pairs covering all 100 skills under their descriptive names and accepted 83.6% by majority vote, with a mean pairwise Cohen's κ of 0.715. The names also predict how LLMs change. Chat tuning should strengthen instruction following, and in 109 pairs of a pretrained model and its chat-tuned version, the instruction-following skills rise by 0.81 standard deviations relative to the other skills, while further fine-tuning of an already chat-tuned model leaves them unchanged. We defined this corrected control after seeing a first reading.

The two labels that you quote are short labels from Figure 5, and the two cases differ. Each short label is the most frequent phrase of its cluster, and for these two skills it misleads. The first skill consists almost entirely of MuSR object-placement questions, which ask where a person would look for an object after the movements described in a story. Its descriptive name, "locate required items for activities", together with a representative question, makes its meaning clear. The second is a small mixed cluster dominated by BBH causal-judgement questions, so its assignments need correction as well as its name, and we will document that correction separately from the taxonomy used in the experiments. The revision uses the descriptive names in Figure 5 and in the Skill Cluster Examples appendix.

You are right that the evaluation relies on the skills, and our experiments show how. Replacing HAC with K-Means, alone or together with a different phrase embedder, gives most held-out questions a different skill set, yet held-out AUC changes by at most 0.29 points, with 95% intervals that include zero in these single-seed runs. Moving every question's skills to another question of similar difficulty, by contrast, removes SkillEval's advantage in ranking LLMs on a question over their overall strength in all five shuffles. Prediction therefore depends on which skills each question receives, which annotators checked, and not on how the phrases are grouped.

AuthorsOpenReview ↗

Response to Reviewer PnzQ (3/3)

Q1. The 3,811 LLMs, their evaluation and the compute.

SkillEval reuses public evaluation records, so profiling the 3,811 LLMs required no LLM inference. The 3,811 LLMs are the open-weight models whose per-question results on the five benchmarks RouterEval collects from the Open LLM Leaderboard. They span pretrained, fine-tuned and chat models as well as merges, and their identifiers are listed in the supplementary material. The leaderboard evaluates every model with the same evaluation harness and records whether each answer is correct. BBH, GPQA and MuSR are scored by comparing the likelihoods of the answer options, MATH by exact match of a generated answer, and IFEval by rule checks on the generated output. On our side, training the main model takes about three hours on one RTX 2080 Ti, as the Training Details appendix reports.

Q2. The extraction model.

Yes, the skills are easy to understand, even though they come from the smaller GPT-4o-mini. Three annotators accepted 83.6% of the 780 sampled question-skill assignments, and in Table 1 an LLM judge rates the clarity of the skill descriptions close to that of the expert-written Common Core standards. We expect a stronger extractor to improve the labels further.

We have not yet run an open-weight extractor. The LLM only labels the skills of each question, and the labels are grouped with embeddings, so any model that follows the labeling prompt in the Q-Matrix Construction Details appendix can take its place. A different extractor would also change the taxonomy, and our results change little when the taxonomy changes. Replacing the clustering method and the phrase embedder gives most held-out questions a different skill set, yet held-out AUC moves by at most 0.29 points.

Reviewer_5zexOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SkillEval, a framework combining LLM-based skill extraction, text clustering, and neural cognitive diagnosis to estimate interpretable skill profiles from LLM response records. Its main contributions are constructing named ability dimensions across benchmarks and applying these profiles to model routing, adaptive evaluation of new models, and analyses of benchmark coverage and instruction-tuning effects.

The framework is evaluated using responses from 3,811 LLMs to 9,523 items across five benchmarks, primarily with 100 skill dimensions. The experiments show that the profiles are stable across retraining and that query-specific model selection improves accuracy and the trade-off between accuracy and estimated computational cost. For new models, 20 adaptively selected questions yield benchmark-score prediction errors comparable to those obtained with 100 randomly selected questions. However, held-out-item prediction is weaker than IRTNet, and cross-benchmark transfer exhibits substantial limitations.

Critical strengths
  1. Scale of the empirical exercise. The dataset assembled — 3,811 LLMs × 9,523 items — is large for this literature (~34× IRTNet's model count), and the response matrices are released as supplementary material. This is a useful infrastructure contribution regardless of the method built on top of it.
  2. Honest reporting. The paper voluntarily reports results that weaken its own claims: below-random transfer on IFEval (Appendix J), the transductive evaluation design (Appendix P), and the non-causal nature of the co-mastery graph. I appreciate this and it increases my confidence that the reported numbers are not cherry-picked.
  3. Cross-seed stability of the profiles. With the Q-matrix held fixed, per-dimension correlations across retraining reach 0.96, versus ≈0 for IRTNet. This is a real advantage of anchoring dimensions to named skills. However, it is largely a property of the fixed taxonomy rather than of the estimation procedure, and interpretability-stability does not by itself produce predictive or practical gains.
Critical weaknesses
  1. The contribution is an incremental assembly of existing components. Each building block is taken from prior work essentially unchanged: NCDM (Wang et al., 2020), LLM-based skill extraction (Didolkar et al., 2024), text-conditioned item parameters (Gor et al., 2024; Cui & Sachan, 2025), and CD-CAT for adaptive selection (Cheng, 2009). The paper's contribution is the integration and the large-scale empirical study, not a methodological advance. The framing in §1–2 suggests a new evaluation paradigm, but what is delivered is closer to a well-executed engineering report on combining known techniques. I do not think this clears the novelty bar for ICLR on its own; it would need stronger empirical findings to compensate, and those findings have significant caveats.
  2. Item-level prediction on unseen items is worse than the baseline. In the OOD setting, SkillEval achieves AUC 72.15 / Acc 67.62 versus IRTNet's 75.35 / 69.38 (Table 2). The abstract claims profiles that "more accurately predict model behavior on unseen items," but this holds only at the benchmark-aggregate level (r = 0.989), where IRTNet is comparable anyway (r = 0.993). The skill-based inductive bias that provides interpretability simultaneously limits predictive flexibility, so the paper's central claim needs to be substantially qualified.
  3. The framework is item-bank-bound, which inverts its value proposition. The cross-benchmark transfer experiment (Appendix J) shows that with MuSR and IFEval held out, aggregate diagnostic fit collapses to AUC 0.469 — below chance — and IFEval routing falls below the random baseline (0.410 vs 0.437). The implication is that SkillEval works only within domains already covered by existing benchmarks, where it reorganizes information that practitioners already have; for genuinely new capability domains — exactly where a diagnostic evaluation tool would be most valuable — it provides nothing, and one must fall back to building a targeted benchmark by hand. The authors acknowledge this, but it fundamentally limits the paper's stated goal of "better exploiting benchmark data."
  4. The routing gain is marginal relative to the pipeline complexity. Acc@1 improves from 61.57% (strongest single LLM) to 65.67%, about 4 points, at the cost of an extraction–clustering–NCDM training pipeline. The K-sweep (Table 7) shows Acc@1 at K=100 (0.657) is below K=500 (0.676), so the gain is also configuration-sensitive. For many deployment settings this margin will not justify the operational overhead; the cost–accuracy curves are more persuasive, but they rest on a fixed model zoo of mostly small open models, and it is unclear the effect persists when a single frontier model dominates all candidates — the regime the field is moving toward.
Questions for authors
  1. Can you rerun the key experiments (Table 2 item-level prediction, Table 3 routing) in a fully inductive setting, where the Q-matrix, taxonomy, clustering choice, and K are all fixed using training items only? If performance degrades substantially, how should readers recalibrate the claims in the abstract?
  2. How stable are the taxonomy and downstream metrics if you (a) swap the extraction LLM, (b) swap the sentence embedder, or (c) swap HAC for K-Means — jointly rather than one factor at a time? Appendix D shows HAC was chosen by downstream AUC, which makes the single-pipeline results hard to interpret as evidence about skills rather than about this pipeline.
  3. Your candidate pool is dominated by small open models, and the routing gains rely on the existence of "weak-beats-strong" inversions (38% of items). Do you have evidence this inversion rate, and hence the routing headroom, persists when the candidate set includes current frontier models? If the strongest model's accuracy ceiling rises, does the 4-point Acc@1 gain shrink?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5zex (1/4)

Thank you for the detailed and accurate review, and for proposing the inductive rerun and the joint pipeline swaps. The inductive rerun shows that the taxonomy does not need the held-out questions, and the swaps show that prediction does not depend on one clustering pipeline. A new experiment with held-out subtasks shows where transfer stops. Our claims therefore cover new questions from the benchmarks and subtasks seen in training, and new LLMs.

Held-out AUC pools all pairs of LLMs and held-out questions, as in Table 2. Within-question AUC is the AUC across LLMs for one held-out question, averaged over questions.

W1. Contribution beyond existing components.

We agree that each component comes from prior work, and the revision will remove the framing as a new evaluation paradigm from the abstract and the Introduction. The combination gives each ability dimension a property that none of the components gives on its own. A benchmark score says how often an LLM succeeds, and a latent ability in IRTNet predicts well, but its dimensions have no names and can rotate from one training run to the next. SkillEval links each dimension to a fixed set of questions through a Q-matrix induced from the questions of five benchmarks. Each dimension therefore has a name that can be checked against its questions, and annotators accepted 83.6% of 780 sampled question-skill pairs. The text-conditioned question parameters extend the same named dimensions to questions that the model has never seen.

Each dimension also keeps its meaning across training runs. Per-skill estimates correlate at 0.96 between runs, whereas the dimensions of IRTNet correlate at 0.00, and at most 0.43 after alignment. You are right that this stability comes from the fixed taxonomy, and that is the purpose of the design. The IRTNet comparison shows that fitting the same responses does not produce it, since IRTNet is trained on the same data with more capacity. Stability is not a predictive gain, but it lets a reader compare one LLM's "instruction following" with another's across runs and across benchmarks. The contribution is the Q-matrix that links each dimension to its questions, so that the profiles of 3,811 LLMs are stated in 100 named skills.

W2. Item-level prediction compared with IRTNet.

Most of the gap to IRTNet comes from how question difficulty is estimated and not from the named skills, and we tested this in two parts. Held-out AUC pools two comparisons. Within a question, a model must rank the LLMs, which is what a skill profile describes. Across questions, it must also know which questions are harder, which depends on the network that estimates each question's parameters from its text. On the first comparison, SkillEval matches IRTNet. In the rerun of Q1, within-question AUC is 66.06 for SkillEval and 65.66 for IRTNet, with overlapping 95% intervals over seeds, although IRTNet has about 39 times as many parameters.

IRTNet leads on the second comparison, and most of that lead can be closed without changing the skills. We replaced SkillEval's question-side network with that of IRTNet and kept the named and monotone skill profiles unchanged. With the submitted taxonomy, pooled AUC on the development questions of Q1 rises from 74.21 to 75.97 over three seeds, against 76.40 for IRTNet, and the modified model is ahead of the original in every seed. This closes about 80% of the gap while every dimension keeps its name. Within-question AUC stays level with IRTNet, at 67.34 against 66.89.

We will correct the abstract accordingly. Its claim holds against a single average score per LLM, with a held-out AUC of 72.15 against 60.54. It does not hold against IRTNet, which reaches 75.35 on the split of Table 2, or against a table of each LLM's training accuracy on each subtask, which reaches 75.31 without a model. The revised abstract will name its comparator and report both results, and Table 2 will include the subtask table.

AuthorsOpenReview ↗

Response to Reviewer 5zex (2/4)

W3. Scope beyond the item bank.

SkillEval is designed for public leaderboards, where a fixed set of benchmarks meets a growing number of new LLMs, 3,811 in our data. It handles both kinds of new input in this setting. Its taxonomy can be built without the held-out questions, with a held-out AUC within 0.35 points of the submitted taxonomy. Twenty adaptively selected answers from a new LLM predict its benchmark scores with an error of 0.061, which random selection needs one hundred answers to reach. SkillEval does reorganize benchmarks that practitioners already have, and in doing so it describes every LLM on the same named skills across five benchmarks, which a benchmark score does not.

We tested transfer beyond the item bank directly, and it holds for neither SkillEval nor IRTNet. A new experiment extends the cross-benchmark result that you cite to held-out subtasks. Eight folds hold out all 38 subtasks, four or five at a time, and each fold rebuilds the taxonomy from its training questions without any response to a held-out subtask. On the unseen subtasks, SkillEval ranks the LLMs less well than a table of each LLM's training accuracy on the same benchmark, by 2.05 points of within-question AUC. The 95% interval runs from −3.02 to −1.08, and SkillEval is lower in seven of the eight folds, a negative result under the reading rule fixed before the runs. IRTNet also stays below this table, by 0.34 points over the folds. Without responses to a task, neither model predicts behavior on it better than benchmark-level accuracy.

For a new capability domain, we accept the consequence that you draw. SkillEval, like every model we compared, needs questions from that domain and responses to them. Once those responses exist, the domain becomes part of the item bank. With IFEval responses in training, for example, SkillEval routes the held-out IFEval questions as well as the strongest single LLM, at 82.0% against 81.0%, while without them its routing on IFEval falls below random selection. The revision will state this scope in the abstract and the main text, as the Cross-Benchmark Transfer appendix already does.

W4. Size and robustness of the routing gain.

SkillEval routes best among the diagnostic models in Table 3, with an Acc@1 of 65.46 over three training runs against 63.08 for IRTNet and 61.57 for the strongest single LLM. You read the K sweep correctly, and K = 100 is not the best value, since Acc@1 is highest at K = 500 and held-out AUC at K = 400. So the submitted configuration does not favor SkillEval. Across K, Acc@1 stays between 64.04 and 67.61 in one run per K, above the strongest single LLM at every K. Selecting K on the development questions of the Q1 rerun, without any held-out result, chooses K = 300, whose Acc@1 on the held-out questions is 3.3 points above that of K = 100. The gain also survives without the weaker candidates, and with the 10 strongest LLMs in the pool, SkillEval still gains 3.2 points over the best of them.

The cost-accuracy result, which you find more persuasive, also holds across seeds. Averaged over five training seeds, SkillEval reaches the accuracy of the strongest single LLM at 38% of its inference cost. The pipeline is built once per item bank, and routing reuses the same profiles without further training.

AuthorsOpenReview ↗

Response to Reviewer 5zex (3/4)

Q1. Fully inductive rerun of item-level prediction and routing.

We ran it, and every choice that you list can be made without the held-out questions and without a measurable loss. The taxonomy and the question-skill assignments now come from the training questions only. HAC with K = 100 clusters the skill phrases of the 7,618 training questions, and a phrase seen only in held-out questions joins its nearest training cluster. The model is fitted on 6,856 of these questions and the checkpoint is selected on the other 762, the development questions, so the 1,905 held-out questions serve only for evaluation. Each model has three seeds under the recipe of the five-seed runs, with a control on the submitted taxonomy.

ModelHeld-out AUCWithin-question AUCAcc@1
SkillEval, training-only taxonomy72.8966.0665.34, range 63.63 to 66.61
SkillEval, submitted taxonomy73.2466.1666.44
IRTNet, d = 1,02475.1665.6662.59, range 62.20 to 62.78
Subtask accuracy table, no model75.2367.4870.60
Strongest single LLM61.94

The revision will report this table.

Removing the held-out questions from the taxonomy does not lower performance. Held-out AUC differs between the two taxonomies by 0.35 points, with a 95% interval over questions from −0.13 to 0.80, and the intervals over seeds also include zero. The Acc@1 difference of 1.1 points lies within the range of 3.0 points across seeds, and SkillEval routes better than IRTNet and the strongest single LLM in all three seeds.

Fixing the clustering method and K on training questions, as you also ask, does not change this result. K-Means in place of HAC leaves held-out AUC unchanged. Selecting K on the development questions chooses K = 300, whose held-out AUC does not differ from that of K = 100 beyond seed noise and whose Acc@1 is 3.3 points higher. Since performance does not degrade under any of these choices, the inductive setting itself calls for no recalibration of the abstract.

Q2. Stability under joint swaps of the pipeline.

Prediction does not depend on the submitted pipeline. We swapped the clustering method alone and together with the phrase embedder, so that most held-out questions received a different skill set, and the results held. Both arms use K-Means with K = 100, on the submitted phrase embeddings or on phrase embeddings from bge-large-en-v1.5. All three runs use the split of Table 2 and one seed under the recipe of the five-seed runs, so their values are not comparable with those of Q1. We did not vary the extraction LLM. The closest check on extraction is in the Q-Matrix Construction Details appendix, where adding reference solutions to the extractor's input changes which skills are named, while a blind judge prefers neither set of labels.

MeasureSubmitted taxonomy, same seedK-MeansK-Means, bge-large-en-v1.5
Held-out questions keeping their submitted skill set100%15.4%11.9%
Held-out AUC, change with 95% interval72.1572.43, +0.29, from −0.55 to 1.1372.25, +0.10, from −0.83 to 0.90
Within-question AUC65.5166.2166.78
Acc@162.3365.5866.53

Replacing most skill sets does not cost accuracy. Held-out AUC changes by at most 0.29 points, with intervals that include zero, and within-question AUC and Acc@1 do not fall. The revision will replace the HAC selection statement of the Q-Matrix Construction Details appendix with this comparison and will state the unvaried extraction LLM as a limitation.

You also ask whether these results show something about the skills or only about this pipeline, and the evidence points to the skills. We moved every question's skills to another question of similar difficulty, so that each question keeps about the same difficulty while its skills change, and retrained the model. With the original skills, SkillEval ranks the LLMs on a held-out question better than a ranking by their overall strength alone, by 0.82 to 1.09 points of within-question AUC over five seeds. After the move, this advantage disappears in all five runs. So the results barely move when the pipeline changes, but they lose this advantage when each question loses its own skills.

AuthorsOpenReview ↗

Response to Reviewer 5zex (4/4)

Q3. Inversions, routing headroom and frontier models.

The routing gain does not rely on the many small models in the pool. In the rerun of Q1, we kept the 10, 50, 100 or 500 LLMs with the highest training accuracy, which share one strongest single LLM. SkillEval gains 3.2, 2.2, 2.4 and 3.6 points of Acc@1 over that LLM, against 3.4 with all 3,811 candidates, and the gain is separated from zero in at least two of three seeds at every pool size under the rule fixed before the analysis. IRTNet gains 0.6 to 1.4 points, with intervals that include zero at every pool size.

A router can gain only on questions that the strongest LLM fails, so a stronger frontier model leaves less room, and we expect the gain in Acc@1 to shrink with it. The RouterEval records cover open-weight LLMs of the Open LLM Leaderboard, so a pool with one dominant frontier model lies outside what they can test. The revision will state this scope in the routing section.

Reviewer_vrAmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillEval, a framework that combines LLM-based skill extraction with neural cognitive diagnosis models to learn interpretable skill-level ability profiles of LLMs. Experiments across 3,811 LLMs and five benchmarks demonstrate its applications in performance prediction, model routing, and adaptive testing.

Critical strengths
  1. The paper addresses an important limitation of traditional benchmark evaluation by providing fine-grained, interpretable skill-level ability profiles for LLMs.

  2. The paper conducts extensive experiments across thousands of LLMs and multiple benchmarks, demonstrating promising results in model routing and adaptive testing.

Critical weaknesses
  1. The core components, including LLM-based skill extraction, clustering, and cognitive diagnosis models, have been explored in prior work (e.g., Didolkar et al., 2024; FinCDM, 2025). The methodological contributions beyond integrating existing techniques remain unclear.

  2. The automatically constructed skill taxonomy has inconsistent granularity and potentially inaccurate assignments. For example, a domain-finding problem is assigned to the "Quadratic Formula" cluster despite not requiring that formula.

Questions for authors

Appendix K shows that the partial correlation drops to only 0.021 after controlling for subtask-level accuracy. What additional value do the learned skill profiles provide over simpler baselines?

Cross-benchmark transfer shows substantial performance degradation, particularly on IFEval. How well does the framework generalize to unseen benchmarks or skills?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vrAm (1/2)

Thank you for the careful reading. SkillEval estimates a profile over 100 named skills for each of 3,811 LLMs. The skills are induced from benchmark questions and linked to individual questions through a Q-matrix.

Below, held-out AUC pools responses across all LLMs and the 1,905 held-out questions, as in Table 2. Within-question AUC measures how well predictions rank LLMs on each question, averaged across questions.

W1. Components from prior work and the methodological contribution.

SkillEval's contribution is to connect skill discovery and cognitive diagnosis within a shared representation, and to establish what this representation supports across 3,811 LLMs, including adaptive evaluation. We build on existing extraction and cognitive diagnosis methods.

The two works you cite each provide one part of this combination, skill discovery or cognitive diagnosis, and neither provides both. Didolkar et al. use GPT-4 to give each GSM8K and MATH training question one skill label and use the labels to choose in-context examples. They report per-skill accuracy for this prompting method but do not estimate skill profiles of LLMs. FinCDM estimates skill profiles for 30 LLMs with cognitive diagnosis, on questions that domain experts wrote in the style of the CPA exam and tagged with 70 financial concepts. SkillEval does both. It induces skills from the questions of five benchmarks without expert annotation and estimates profiles for 3,811 LLMs on them. It also scores unseen questions on the same skills from their text.

It joins them through the Q-matrix, which links each skill to an explicit set of questions. Readers can therefore inspect what a skill represents, compare LLMs on the same skills and examine whether those estimates reproduce across retraining. In addition, question parameters computed from the question text extend prediction to unseen questions within the covered domains, and the profiles guide which questions to ask a new LLM.

Our experiments test each of these properties. First, human evaluation supports the question-skill assignments. Second, per-skill estimates correlate at 0.96 across retraining with the taxonomy fixed. The unnamed dimensions of IRTNet correlate at 0.00, and at most 0.43 after alignment. Third, restricting the model to 100 named skills does not lower its accuracy in ranking LLMs on unseen questions. In a rerun with the taxonomy built from the training questions alone, SkillEval reaches a within-question AUC of 66.06 and IRTNet 65.66, although IRTNet has unconstrained dimensions and about 39 times as many parameters. Finally, the same diagnostic model supports efficient evaluation of new LLMs, since 20 adaptively selected responses achieve a benchmark-score prediction error comparable to 100 randomly selected responses.

We will make this contribution explicit and add a comparison with Didolkar et al. and FinCDM that separates the inherited components from our design and findings.

W2. Skill granularity and assignment quality.

You correctly identify a mismatch, because the domain-finding problem does not require the quadratic formula. The displayed label is the most frequent extracted phrase in its cluster, which is too narrow for a cluster of several hundred questions, most of them algebra problems. We will replace these labels with the descriptive skill names used in human evaluation, and the version released upon publication will show example questions for each skill.

With the descriptive names, three annotators evaluated 780 question-skill pairs covering all 100 skills. They accepted 83.6% by majority vote, with a mean pairwise Cohen's κ of 0.715. The majority accepted fewer than half of the sampled assignments for 5 of the 100 skills, and we will correct their assignments in the version released upon publication.

In Table 1, an LLM judge also rates our skills as more uniform in granularity than the Common Core standards for mathematics, an expert-defined skill set. Where granularity does vary, it has little effect on prediction. Varying the number of skills from 50 to 500 keeps held-out AUC between 71.6 and 72.1. Replacing HAC with K-Means, alone or with a different phrase embedder, changes held-out AUC by less than 0.3 points, and the 95% intervals include zero in both single-seed comparisons.

Prediction does depend on which skills each question receives. To test this, we moved each question's skills to another question of similar difficulty, so that each question keeps about the same difficulty while its skills change. With the original skills, SkillEval ranks the LLMs on each question better than a ranking by their overall strength alone, by about one point of within-question AUC in all five training seeds. When the skills are moved, this gain disappears in all five runs.

AuthorsOpenReview ↗

Response to Reviewer vrAm (2/2)

Q1. What do the profiles add beyond simpler baselines?

The profiles' additional value is a shared skill representation that links capabilities across benchmarks. For predicting responses within a benchmark, they add little beyond subtask accuracy.

The partial correlations show this. The skill estimates add information beyond an LLM's five benchmark scores. With those scores controlled for, they still correlate at 0.135 with held-out accuracy on the same skill, while shuffled skills give about zero. Beyond its 38 subtask scores, they add almost nothing, and the correlation falls to the 0.021 that you cite. In line with this, a table of each LLM's training accuracy per subtask predicts held-out responses better than SkillEval, with a held-out AUC of 75.31 against 72.15.

The subtask table, however, works inside one benchmark at a time, because each of its 38 subtasks belongs to a single benchmark. Of the 100 skills, 70 contain questions from at least two benchmarks, so SkillEval describes every LLM on the same skills across benchmarks. Human evaluation and retraining show that these skills are readable and stable, and they let us compare LLMs in common terms. For example, 14 skills differ significantly between base and instruction-tuned models, and 11 of them, mostly instruction-following skills, are higher after tuning. We will report the subtask table beside the learned models and separate the evidence for prediction from the evidence for capability analysis and adaptive evaluation.

Q2. Generalization to unseen benchmarks or skills.

SkillEval addresses a common leaderboard setting in which established benchmarks are used to evaluate new LLMs. Within these benchmarks, it predicts responses to unseen questions from their text and estimates a new LLM's benchmark scores from a small set of answers. A new benchmark is harder, because SkillEval can only measure skills on which it has seen responses.

The IFEval result that you cite comes from missing training data. In that experiment, SkillEval is trained on MATH, BBH and GPQA and then routes questions from MuSR and IFEval, which it has never seen. MuSR needs mostly the same skills as BBH, so SkillEval has already learned from the BBH answers which LLMs are good at them. Routing on MuSR matches the strongest single LLM. IFEval tests instruction following, and 22.9% of its questions need a skill that no training question has. SkillEval has never seen an LLM answer a question that needs these skills, and routing on IFEval falls below random selection.

The IFEval failure disappears once its responses are part of training. SkillEval then routes the held-out IFEval questions as well as the strongest single LLM, at 82.0% against 81.0%. Because assigning skills needs no responses, we can count the questions of a new benchmark that need a skill no training question has before any LLM answers them, which warns of part of such a failure in advance.

No learned model we tested predicts unseen tasks better than benchmark accuracy does. When entire subtasks are held out across eight folds, SkillEval ranks the LLMs on their questions less well than a table of each LLM's accuracy on the same benchmark, by 2.05 points of within-question AUC. The 95% confidence interval runs from −3.02 to −1.08, and IRTNet also falls below this table.

Covering a new benchmark therefore needs responses to its questions, as for every model we compared. Its questions go through the same skill extraction, and each new skill phrase joins the closest existing skill. If the benchmark tests something the 100 skills do not cover, new skills are added instead. Retraining then takes about three hours on one GPU. The Cross-Benchmark Transfer appendix already states that deployment to new skill families requires expanding the training mixture rather than relying on zero-shot transfer. The revision will move this statement and the cross-benchmark results into the main text, and will state there that our claims hold within the trained skill structure.

Comments on the paper

2 comments

Dear Reviewers,

Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards

AC

AuthorsOpenReview ↗

General response to the area chair and reviewers

We thank the reviewers for their careful reading. All three note the scale of the study, with responses of 3,811 LLMs to 9,523 questions. Reviewer vrAm writes that skill-level profiles address "an important limitation of traditional benchmark evaluation," and PnzQ calls the experiments "rigorous." Reviewer 5zex points to the "cross-seed stability of the profiles."

Positioning. We agree that each component of SkillEval comes from prior work. The closest methods differ from SkillEval in where their ability dimensions come from. Latent abilities, as in IRTNet, predict well but have no names. Abilities in the general-scales study and NIRT-Router have names but are predefined rather than induced from the benchmark questions. SkillEval induces its skills from the questions and links each skill to its questions through the Q-matrix. Each dimension therefore has a name that can be checked against its questions and a meaning that stays fixed across training runs.

Ability dimensionsHow a question receives themLLMs profiled
General-scales study18 predefined demand scalesa level on each scale, assigned with rubrics15
NIRT-Router25 predefined named abilitiesassigned by an LLM to clusters of queries20
MIRT-Router and IRTNetunnamed latent dimensionsno named dimensions are assigned20 and 112
SkillEval100 skills induced from the questions of five benchmarksskill phrases proposed by an LLM for each question, then clustered3,811

Interpretability and prediction. SkillEval is built to make LLM evaluation interpretable. Every dimension of a profile is a named skill with its own set of questions, and the profiles are stable across retraining: per-skill estimates correlate at 0.96 between runs, against 0.00 for IRTNet. Interpretability costs some predictive accuracy, and most of the gap comes from how question difficulty is estimated. Replacing our question-side network with that of IRTNet closes about 80% of the pooled-AUC gap while every skill keeps its name, yet it raises within-question AUC by only 0.10 points against 1.75 for pooled AUC. Pooled AUC compares responses across easy and hard questions, so a model can raise it by estimating difficulty better without estimating the LLMs' profiles any better. We therefore also check within-question AUC, which compares LLMs on the same question, and on it SkillEval matches IRTNet (66.06 against 65.66), a model with about 39 times as many parameters.

SkillEval's predictive accuracy remains below that of the per-subtask accuracy baseline, and the revision states this explicitly. Its contribution is a shared representation of named skills, which shows how models differ beyond their aggregate scores. Between base and instruction-tuned models, 14 skills differ significantly: 11 are higher after tuning and 3 are lower. Skills from the same benchmark can move in opposite directions, which a benchmark average hides. In a separate paired analysis (reply to PnzQ, W3), the instruction-following skills rise after chat tuning, as their names predict. The profiles also support adaptive evaluation of new LLMs and cost-aware routing.

Other concerns

  • Taxonomy built without the held-out questions (5zex Q1). We rebuilt the taxonomy from the training questions alone. Held-out AUC stays within 0.35 points of the submitted taxonomy, with an interval that includes zero. SkillEval still routes better than IRTNet and the strongest single LLM in all three seeds.
  • Dependence on the pipeline (5zex Q2). We replaced the clustering method, alone and together with the phrase embedder. Most held-out questions then receive a different skill set, and held-out AUC changes by at most 0.29 points.
  • Information in the skills (vrAm W2, 5zex Q2). SkillEval uses the skills of a question to predict which LLMs answer it correctly. With them, it ranks LLMs on held-out questions better than their overall strength does, in all five training seeds. A control that gives each question the skills of another question of similar difficulty does not show this gain, which therefore comes from the skills of each question.
  • Robustness of the routing gain (5zex W4, Q3). Over five training seeds, SkillEval reaches the accuracy of the strongest single LLM at 38% of its cost. When only the 10 strongest LLMs compete, it still gains 3.2 points of Acc@1 over the best of them.

Revisions to the PDF. Changes are marked in blue. The abstract and the Introduction drop the framing as a new evaluation paradigm, name the comparator of the prediction claim and state the scope. Table 2 adds the subtask table and the standard NCDM. The appendices report these experiments, and the related work adds the comparison in the table. Each reply gives the full results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.