- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to measure whether a benchmark actually measures what it claims, i.e. its construct validity. The authors use a two-dimensional Multidimensional Item Response Theory (MIRT) model, which separates benchmarks on two dimensions, namely general reasoning capability and safety alignment. This model is fit and evaluated on the responses of 100 LLMs on 6 reasoning datasets and 10 safety datasets, with empirical evidence showing that the one learned dimension is correlated with safety and one with reasoning. In terms of predicting LLMs' performance, the fit MIRT model outperforms all other baselines, such as include single dimensional IRT models, category-specific models, average-performance-based estimators and so on. The method is also able to identify datasets that do not have construct validity, in the sense that an LLMs performance on them does not correlate with the dimension learned for the intended capability.
Critical strengths
Novelty: The paper provides a new method to audit the reliability of a benchmark in terms of measuring the construct it claims to measure. The demonstrate the effectiveness of the approach on widely used datasets.
Soundness: Most methodological choices made in the paper are well motivated and technically sound.
Critical weaknesses
Potentially circular evaluation: the reliability of the method rests on the assumption that most benchmarks used to fit the MIRT model are at least good representations of the intended construct, there is no other external validation of this fact. Even though the authors use multiple ways to show the validity (e.g. in lines 311-348), validations such as an expert evaluation or correlation with already established metrics of reliability for both reasoning and safety alignment.
Generalizability: The paper demonstrates an effective partitioning of general reasoning capability and safety alignment, which are very commonly studied and relatively better understood abilities. There is no indication of whether this approach can isolate further, more subtle dimensions that affect LLM behavior such as sycophancy or biases of different kinds. For this approach to be generalizable, there should be a way to extend it to dimensions which are harder to quantify, ideally through a model with a higher dimension space to account for those or at least by testing the two-dimensional model on other capabilities.
Questions for authors
-
How do you address the risk that the learned dimensions do not capture the intended constructs if fit on datasets with low construct validity?
-
What are the challenges associated with applying this framework to other subjects of LLM benchmarks (e.g. sycophancy, bias, etc.), and how do you think that will change the results?
4 comments
W2 and Q2: BenchMIRT’s methods are generalizable to additional dimensions and capabilities
There is no indication of whether this approach can isolate further, more subtle dimensions that affect LLM behavior such as sycophancy or biases of different kinds. For this approach to be generalizable, there should be a way to extend it to dimensions which are harder to quantify, ideally through a model with a higher dimension space to account for those or at least by testing the two-dimensional model on other capabilities.
In Appendix J (referenced on line 273), we investigate a three and four dimensional MIRT model, and find that it does not offer significant gains over the two dimensional model.
In Appendix D (referenced on line 356), we test the generalization of this method to measure math and instruction following abilities. We fit a new MIRT function (in one through four dimensions) to the MMLU-pro math subset, the Math benchmark, and IFEval, and find three distinct capabilities – one for IFEval (possibly measuring instruction following), one capturing both MATH and MMLU-Pro (mathematical reasoning), and one capturing an additional MMLU-Pro ability (potentially a multiple choice formatting ability).
In Appendix C (referenced on line 356), we additionally test the generalization of BenchMIRT (in one through four dimensions) when fit on on just the safety benchmarks, just the safety benchmarks without BBQ/WMDP, and just the general reasoning benchmarks. We find that the full split of safety benchmarks reproduces the split found in the paper, with WMDP and BBQ separated to a separate latent factor, with Toxigen and XSTest loading on both. The safety benchmarks without WMDP and BBQ when run with BenchMIRT separate out Toxigen and XSTest, indicating that the model is very stably reproducing a similar split. Interestingly, there does not appear to be a clear split between jailbreak safety evals and more traditional safety evals – DoAnythingNow and Harmbench have very similar scores, for example. The reasoning results appear to split out Math.
As a new experiment, we ran the same accuracy testing from Section 4.1 and Table 1 on the subsets from Appendix C and Appendix D, and found similar accuracies to the full BenchMIRT method. We will add those results to Appendix C and Appendix D and add a discussion about generalization to the Conclusion.
We also note that BBQ, as a bias benchmark, shows the subtleties that BenchMIRT can uncover – it was more strongly correlated with the capability factor than the safety one.
Furthermore, this analysis suggests that BenchMIRT does not require strong construct validity in a benchmark to separate out underlying factors, as these were exploratory analyses. It does indicate that the capability factors may not map to some more subtle LLM benchmark subjects, as the similarities between the HarmBench and Do Anything Now benchmark show that it does not distinguish between jailbreak and more standard benchmarks, but that contains important information about the latent model capabilities themselves – if the same capability enables performance on both benchmarks then BenchMIRT can not separate them. More subtle constructs may be more challenging to apply BenchMIRT to, as BenchMIRT can only separate model abilities that are distinct.
I have read the authors' responses and I feel my fundamental concerns have not been addressed.
The authors have in fact done a fair amount of ablations to examine whether the dimensions learned on the MIRT model they fit on the datasets they selected indeed separates "safety" and "general reasoning". However, my concerns are more about the general approach.
If I want to use this approach on a selection of datasets for which the construct validity is unknown, I cannot draw any conclusions about the dimensions learned in the MIRT model (and any subsequent evaluations of construct validity from it), because the model might be affected by datasets that aren't strong in terms of their construct validity to begin with. For me to be able to trust the model, I need to know whether the datasets being selected to fit it on are sound in terms of construct validity. Since I require this external validation (say, from experts), it defeats the purpose of this approach to a large extent. The results from Appendix C seem to be inconclusive in addressing this.
Additionally, there doesn't seem to be enough evidence that (i) this model would work for separating a more complex combination of dimensions, or (ii) on dimensions that are different in nature from reasoning or safety (e.g. bias, sycophancy, hallucination, tool-use, etc.). The experiments the authors point to
In light of the above, I shall maintain my current rating.
We thank you for your response, and appreciate that you found our ablations on the BenchMIRT dimensions to be convincing. In response to your additional concerns, we would like to highlight a few points
If I want to use this approach on a selection of datasets for which the construct validity is unknown, I cannot draw any conclusions about the dimensions learned in the MIRT model
BenchMIRT allows for, and in fact depends on, the inclusion of multiple benchmarks around a concept to measure a single construct. That makes it robust to single benchmarks without construct validity (ie, though Toxigen is heavily saturated and does not strongly measure an ability, BenchMIRT still successfully uncovers a safety and general reasoning capability). The inclusion of multiple benchmarks and the exploratory nature of BenchMIRT helps to uncover 1) the number of capabilities represented and 2) what they represent. While the dimensions require human interpretation (as we did in this case for safety and general reasoning or math and instruction following), they enable further item, subset, and benchmark level conclusions without further expert validation.
(i) this model would work for separating a more complex combination of dimensions
We discuss this limitation in the analysis of the BenchMIRT run on the safety benchmarks -- BenchMIRT for example fails to separate HarmBench and Do Anything Now because the LLM models rely on the same capabilities to succeed. Safety and General Reasoning abilities have substantial research on the impact of their entanglement, and we believe that the benchmark, subset, and item-level analysis, with filtering and ranking implications, that BenchMIRT enables on these sixteen benchmarks represents a significant contribution.
(ii) on dimensions that are different in nature from reasoning or safety (e.g. bias, sycophancy, hallucination, tool-use, etc.)
Appendix C and Appendix D address separate combinations of benchmarks, and highlight that BenchMIRT is capable of separating, for example, Math and Instruction Following as capabilities. Due to compute constraints and the larger impact/interest in the nature of the capabilities, we focus on Safety and General Reasoning as the core abilities, and offer the ablations as evidence that it can generalize and be useful in additional applications, not as proof that it generalizes to all possible capabilities of interest. Additionally, we do include BBQ as a bias eval.
Thank you again for your time in reviewing.
We thank you for your review and appreciate that you found our method novel and technically sound. Please find our responses to your questions below:
TLDR:
W1 and Q1: The reliability of BenchMIRT factors is validated with external model and classifier behavior in addition to the benchmarks
We evaluate the reliability and stability of the factors in seven ways. Crucially, only the first method of validation requires construct validity in the benchmarks themselves. Qwen3 Guard provides a separately, independently verified to be reliable measurement. In addition, WMDP and BBQ loading on an unexpected factor, while Toxigen relies strongly on neither and XSTest appears split shows that BenchMIRT is able to fit even when several benchmarks do not have construct validity.
While concepts like safety are difficult to fully verify, we offer independent benchmark, model, and safety classifier confirmations.