- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces the SkillEval dataset, including the responses of 3,811 LLMs on 9,523 data samples from five benchmarks. Besides, the paper illustrates two usages of the SkillEval dataset: (1) Computerized Adaptive Testing and (2) LLM routing. (1) Computerized Adaptive Testing helps evaluate an unseen LLM using a trained CDM with several evaluated data samples. (2) LLM routing assigns each data sample to a capable but efficient LLM, reducing inference cost.
Critical strengths
(1) The proposed dataset is extensive, which includes results of 3,811 LLMs on 9,523 data samples. (2) The experiment is rigorous. SkillEval is evaluated through a comprehensive set of experiments, compared against IRT, IRTNet baselines. (3) The paper reads smoothly and is well written.
Critical weaknesses
(1) The idea of the paper looks similar to a series of papers. All of them use the cognitive diagnostic model for evaluating LLMs. I did not see the comparison of the SkillEval with these related papers.
General scales unlock AI evaluation with expanatory and predictive power. Nature 2026. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory. ACL 2025.
The main difference may be that the paper uses the NCDM as the cognitive diagnostic model and utilizes GPT-4o as the skill generator. In my opinion, while the NCDM has been a very popular cognitive diagnostic model, the usage of the NCDM cannot serve as the novelty of the paper.
(2) Table 3 illustrates that the SkillEval is a method, outperforming other methods. I am very confused that whether other methods are trained on SkillEval dataset. Maybe NCDM is more appropriate in the Table 3 comparison.
(3) The LLM-generated skills are difficult to understand. For example, in the Figure 5 (b), what are the meaning of "possible storage options based on context" and "work done in variable force scenarios"? The whole evaluation relies heavily on the LLM-generated skill quality.
Questions for authors
See weaknesses. And other questions: (1) The paper claims the evaluations of 3,811 LLMs. What are the 3,811 LLMs? And how are they evaluated? How many GPU hours are consumed?
(2) Are the skills generated from GPT-4o easy to understand? Can GPT-4o be replaced by other open-sourced LLMs without performance decreasement?
5 comments
Response to Reviewer PnzQ (2/3)
W2. Table 3 and NCDM.
Yes, every method in Table 3 is trained on the same data. All are fitted on the responses of the 3,811 LLMs to the 7,618 training questions and evaluated on the same 1,905 held-out questions. Random routing involves no training, and the strongest single LLM is chosen by its accuracy on the training questions. Under these conditions, SkillEval routes best among the diagnostic models, with an Acc@1 of 65.46 over three training runs against 63.08 for IRTNet and 61.57 for the strongest single LLM.
SkillEval is itself an NCDM, which is why it can appear in Table 3. A standard NCDM learns a free parameter vector for every question, so it cannot score a question that it has not seen in training. SkillEval computes these parameters from the question text, which lets it score held-out questions on the same named skills. As you suggest, we ran the standard NCDM with the same Q-matrix where it can be evaluated, in the in-distribution setting of Table 2. There it reaches an AUC of 85.24, against 81.73 for SkillEval and 86.16 for IRT 2PL, since free question parameters fit the questions seen in training most closely. We will add this row to Table 2.
W3. Interpretability of the skills.
The skills are readable, and their names match what they measure. Three annotators judged 780 question-skill pairs covering all 100 skills under their descriptive names and accepted 83.6% by majority vote, with a mean pairwise Cohen's κ of 0.715. The names also predict how LLMs change. Chat tuning should strengthen instruction following, and in 109 pairs of a pretrained model and its chat-tuned version, the instruction-following skills rise by 0.81 standard deviations relative to the other skills, while further fine-tuning of an already chat-tuned model leaves them unchanged. We defined this corrected control after seeing a first reading.
The two labels that you quote are short labels from Figure 5, and the two cases differ. Each short label is the most frequent phrase of its cluster, and for these two skills it misleads. The first skill consists almost entirely of MuSR object-placement questions, which ask where a person would look for an object after the movements described in a story. Its descriptive name, "locate required items for activities", together with a representative question, makes its meaning clear. The second is a small mixed cluster dominated by BBH causal-judgement questions, so its assignments need correction as well as its name, and we will document that correction separately from the taxonomy used in the experiments. The revision uses the descriptive names in Figure 5 and in the Skill Cluster Examples appendix.
You are right that the evaluation relies on the skills, and our experiments show how. Replacing HAC with K-Means, alone or together with a different phrase embedder, gives most held-out questions a different skill set, yet held-out AUC changes by at most 0.29 points, with 95% intervals that include zero in these single-seed runs. Moving every question's skills to another question of similar difficulty, by contrast, removes SkillEval's advantage in ranking LLMs on a question over their overall strength in all five shuffles. Prediction therefore depends on which skills each question receives, which annotators checked, and not on how the phrases are grouped.
Response to Reviewer PnzQ (3/3)
Q1. The 3,811 LLMs, their evaluation and the compute.
SkillEval reuses public evaluation records, so profiling the 3,811 LLMs required no LLM inference. The 3,811 LLMs are the open-weight models whose per-question results on the five benchmarks RouterEval collects from the Open LLM Leaderboard. They span pretrained, fine-tuned and chat models as well as merges, and their identifiers are listed in the supplementary material. The leaderboard evaluates every model with the same evaluation harness and records whether each answer is correct. BBH, GPQA and MuSR are scored by comparing the likelihoods of the answer options, MATH by exact match of a generated answer, and IFEval by rule checks on the generated output. On our side, training the main model takes about three hours on one RTX 2080 Ti, as the Training Details appendix reports.
Q2. The extraction model.
Yes, the skills are easy to understand, even though they come from the smaller GPT-4o-mini. Three annotators accepted 83.6% of the 780 sampled question-skill assignments, and in Table 1 an LLM judge rates the clarity of the skill descriptions close to that of the expert-written Common Core standards. We expect a stronger extractor to improve the labels further.
We have not yet run an open-weight extractor. The LLM only labels the skills of each question, and the labels are grouped with embeddings, so any model that follows the labeling prompt in the Q-Matrix Construction Details appendix can take its place. A different extractor would also change the taxonomy, and our results change little when the taxonomy changes. Replacing the clustering method and the phrase embedder gives most held-out questions a different skill set, yet held-out AUC moves by at most 0.29 points.
Response to Reviewer PnzQ (1/3)
Thank you for these questions. Your summary describes SkillEval as a dataset, so we first clarify that it is a diagnostic framework. It estimates a profile over 100 named skills for each of 3,811 LLMs, and each skill is linked to the individual questions that test it. The response records come from RouterEval, which collects the per-question results of the Open LLM Leaderboard, and we ran no LLM inference. Our claims cover new questions from the benchmarks and subtasks seen in training, and new LLMs.
Held-out AUC pools all pairs of LLMs and held-out questions, as in Table 2. Within-question AUC is the AUC across LLMs for one held-out question, averaged over questions.
W1. Relationship to the general-scales study and IRT-Router.
We agree that NCDM itself is not the contribution, and the revision will add the comparison that was missing. It will also correct the Related Work appendix, which overlooks NIRT-Router, the NCDM-based variant of IRT-Router with named abilities. The approaches differ in where the ability dimensions come from. A latent ability, as in IRTNet and MIRT-Router, predicts well, but its dimensions have no names and can rotate from one training run to the next. Predefined ability lists, as in the general-scales study and NIRT-Router, have names, but they are fixed before any question is seen. SkillEval induces 100 skills from the questions of five benchmarks and links each dimension to a fixed set of questions through the resulting Q-matrix. Each dimension therefore has a name that can be checked against its questions and a meaning that stays fixed across training runs. Text-conditioned question parameters extend the same skills to questions that the model has never seen.
Our experiments support these properties. Per-skill estimates correlate at 0.96 across training runs, whereas the dimensions of IRTNet correlate at 0.00, and at most 0.43 after alignment. Restricting the dimensions to named skills does not lower accuracy in ranking LLMs on unseen questions. In a rerun with the taxonomy built from the training questions alone, within-question AUC is 66.06 for SkillEval and 65.66 for IRTNet, which has about 39 times as many parameters.
The general-scales study also allows an external check. It publishes demand levels, assigned by GPT-4o with its rubrics, for 5,431 MMLU-Pro questions. For each LLM with MMLU-Pro responses on the leaderboard and for each scale, we computed how accuracy changes between demanding and other questions within subject categories. We then asked whether SkillEval profiles, learned on our five benchmarks in three training runs with the taxonomy of the paper, predict these changes for held-out LLMs beyond their overall MMLU-Pro accuracy. Among 526 LLMs, with merges and near-identical models removed, the profiles add 0.057 to R-squared, with a 95% interval from 0.035 to 0.077. The general-scales study rates what each question demands, and SkillEval estimates from responses what each LLM can do, so the two frameworks complement each other. We report this as a link between them, not as evidence of transfer to a new benchmark.