acceptodds
Under review as a conference paper at ICLR 2027

BenchMIRT: Disentangling Safety and General Capabilities in LLM Evaluation

Abstract

LLM benchmarks are designed to evaluate models along specific constructs; however, they often have confounders (e.g., item complexity, formatting requirements) that muddle the abilities that they actually measure. In this work, we leverage psychometrics theories to determine the latent factors that LLM benchmarks measure, focusing specifically on safety and general reasoning abilities, which are often seen as in tension. Applying Multidimensional Item Response Theory (MIRT) to LLM benchmarks, we introduce \modelname, a two-dimensional MIRT model fit on 100 LLMs' outputs across sixteen safety and general reasoning benchmarks. We find that our two-dimensional model consistently uncovers distinct safety and reasoning latent factors without any conditioning, and that these factors can facilitate accurate predictions of LLM scores. We further demonstrate that the latent safety and general reasoning item parameters (i.e., difficulty and discrimination) can be used to audit benchmarks, surfacing different ability requirements between subsets, isolating possible item errors, and showing that some benchmarks are dependent on a different ability than they are known to measure (e.g., BBQ). Our work showcases the utility of psychometrics models to audit benchmarks towards more construct validity in LLM evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety, reasoning, evaluation, item response theory, benchmark analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8GjrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to measure whether a benchmark actually measures what it claims, i.e. its construct validity. The authors use a two-dimensional Multidimensional Item Response Theory (MIRT) model, which separates benchmarks on two dimensions, namely general reasoning capability and safety alignment. This model is fit and evaluated on the responses of 100 LLMs on 6 reasoning datasets and 10 safety datasets, with empirical evidence showing that the one learned dimension is correlated with safety and one with reasoning. In terms of predicting LLMs' performance, the fit MIRT model outperforms all other baselines, such as include single dimensional IRT models, category-specific models, average-performance-based estimators and so on. The method is also able to identify datasets that do not have construct validity, in the sense that an LLMs performance on them does not correlate with the dimension learned for the intended capability.

Critical strengths

Novelty: The paper provides a new method to audit the reliability of a benchmark in terms of measuring the construct it claims to measure. The demonstrate the effectiveness of the approach on widely used datasets.

Soundness: Most methodological choices made in the paper are well motivated and technically sound.

Critical weaknesses

Potentially circular evaluation: the reliability of the method rests on the assumption that most benchmarks used to fit the MIRT model are at least good representations of the intended construct, there is no other external validation of this fact. Even though the authors use multiple ways to show the validity (e.g. in lines 311-348), validations such as an expert evaluation or correlation with already established metrics of reliability for both reasoning and safety alignment.

Generalizability: The paper demonstrates an effective partitioning of general reasoning capability and safety alignment, which are very commonly studied and relatively better understood abilities. There is no indication of whether this approach can isolate further, more subtle dimensions that affect LLM behavior such as sycophancy or biases of different kinds. For this approach to be generalizable, there should be a way to extend it to dimensions which are harder to quantify, ideally through a model with a higher dimension space to account for those or at least by testing the two-dimensional model on other capabilities.

Questions for authors
  1. How do you address the risk that the learned dimensions do not capture the intended constructs if fit on datasets with low construct validity?

  2. What are the challenges associated with applying this framework to other subjects of LLM benchmarks (e.g. sycophancy, bias, etc.), and how do you think that will change the results?

4 comments

We thank you for your review and appreciate that you found our method novel and technically sound. Please find our responses to your questions below:

TLDR:

  • W1 and Q1: We externally validate the safety and general reasoning factors with model and classifier behavior in six ways, with only one method requiring construct validity of the benchmarks
  • W2 and Q2: BenchMIRT can be extended to additional capabilities such as Math and Instruction following, and we validate with three and four dimensional models

W1 and Q1: The reliability of BenchMIRT factors is validated with external model and classifier behavior in addition to the benchmarks

the reliability of the method rests on the assumption that most benchmarks used to fit the MIRT model are at least good representations of the intended construct, there is no other external validation of this fact.

We evaluate the reliability and stability of the factors in seven ways. Crucially, only the first method of validation requires construct validity in the benchmarks themselves. Qwen3 Guard provides a separately, independently verified to be reliable measurement. In addition, WMDP and BBQ loading on an unexpected factor, while Toxigen relies strongly on neither and XSTest appears split shows that BenchMIRT is able to fit even when several benchmarks do not have construct validity.

  • Benchmark-level scores
    • We compare the ability scores to the benchmark scores, and find strong correlations between our "safety" dimension and the safety benchmarks, and our general capabilities dimension and the capability benchmarks (Figure 2, lines 278-299).
  • Verifying the safety dimension with abliterated model pairs
    • We compare three LLMs with a version that has been abliterated, or deliberately modified to have no safety capability. We find that the abliterated versions significantly underperform on the items with the highest safety discrimination (Figure 3, lines 339-353). We also have now added a comparison between abliterated versions on the general reasoning dimension, and find that the paired t-tests across abliterated pairs where Dimension 1 discrimination > 2 result in scores of 4.98 (p approx 0, d = 0.05), 4.94 (p approx 0, d = 0.05), and 6.41 (p approx 0, d = 0.07), showing a significantly smaller gap that the safety dimension. We will add this analysis to Section 4.2. Both the existing and the new analyses indicate that the safety dimension indeed captures safety abilities in LLMs.
  • Verifying the general reasoning dimension with model size pairs
    • We compare five model pairs where we have different size models in the same family across the most discriminative safety and general capability items. While the larger models outperform the smaller ones generally, they do so more consistently on the general reasoning dimension and no smaller models outperform their larger partner on the general reasoning dimension while one (Llama 3.1) does on the safety dimension. (Appendix G, Figures 21 and 22). This strongly suggests that the general reasoning dimension in BenchMIRT captures general reasoning abilities in LLMs.
  • Verifying the safety dimension with Qwen3 Guard safety classifier
    • We classify all prompts as Safe, Controversial, or Unsafe according to the Qwen3 Guard safety classifier, finding Pearson r = -0.77 between that score and Dimension 0 discrimination and Pearson r = -0.22 with Dimension 1 discrimination, showing a strong positive correlation between unsafe prompts and more discriminative safety items, and a slight negative correlation between unsafe prompts and more discriminative general reasoning items (Section 4.2 lines 339-348, Appendix K).
  • Confirming the stability of the dimensions across repeated MIRT runs
    • We rerun the BenchMIRT fit in a ten fold cross validation across the LLM-item pairs, and find a strong stability in the item scores (Appendix E, referenced on line 285)
  • Confirming the stability of the dimensions in a held out cross validation where we train on 80 LLMs at a time
    • We rerun the BenchMIRT fit in a five fold cross validation across the LLM models, and find a strong stability in the item scores on all folds with diversity in scores. (Appendix E, referenced on line 285)
  • New: we compare the safety classifications of the responses from the Llama 3.1 8B model and its abliterated partner to the Dimension 0 and 1 discriminations. This shows that the items with the strongest safety discrimination are indeed the ones the abliterated model generates harmful content in response to.
Dimension 0Dimension 1
abliterated−0.76-0.30
original−0.39−0.23

While concepts like safety are difficult to fully verify, we offer independent benchmark, model, and safety classifier confirmations.

W2 and Q2: BenchMIRT’s methods are generalizable to additional dimensions and capabilities

There is no indication of whether this approach can isolate further, more subtle dimensions that affect LLM behavior such as sycophancy or biases of different kinds. For this approach to be generalizable, there should be a way to extend it to dimensions which are harder to quantify, ideally through a model with a higher dimension space to account for those or at least by testing the two-dimensional model on other capabilities.

In Appendix J (referenced on line 273), we investigate a three and four dimensional MIRT model, and find that it does not offer significant gains over the two dimensional model.

In Appendix D (referenced on line 356), we test the generalization of this method to measure math and instruction following abilities. We fit a new MIRT function (in one through four dimensions) to the MMLU-pro math subset, the Math benchmark, and IFEval, and find three distinct capabilities – one for IFEval (possibly measuring instruction following), one capturing both MATH and MMLU-Pro (mathematical reasoning), and one capturing an additional MMLU-Pro ability (potentially a multiple choice formatting ability).

In Appendix C (referenced on line 356), we additionally test the generalization of BenchMIRT (in one through four dimensions) when fit on on just the safety benchmarks, just the safety benchmarks without BBQ/WMDP, and just the general reasoning benchmarks. We find that the full split of safety benchmarks reproduces the split found in the paper, with WMDP and BBQ separated to a separate latent factor, with Toxigen and XSTest loading on both. The safety benchmarks without WMDP and BBQ when run with BenchMIRT separate out Toxigen and XSTest, indicating that the model is very stably reproducing a similar split. Interestingly, there does not appear to be a clear split between jailbreak safety evals and more traditional safety evals – DoAnythingNow and Harmbench have very similar scores, for example. The reasoning results appear to split out Math.

As a new experiment, we ran the same accuracy testing from Section 4.1 and Table 1 on the subsets from Appendix C and Appendix D, and found similar accuracies to the full BenchMIRT method. We will add those results to Appendix C and Appendix D and add a discussion about generalization to the Conclusion.

We also note that BBQ, as a bias benchmark, shows the subtleties that BenchMIRT can uncover – it was more strongly correlated with the capability factor than the safety one.

Furthermore, this analysis suggests that BenchMIRT does not require strong construct validity in a benchmark to separate out underlying factors, as these were exploratory analyses. It does indicate that the capability factors may not map to some more subtle LLM benchmark subjects, as the similarities between the HarmBench and Do Anything Now benchmark show that it does not distinguish between jailbreak and more standard benchmarks, but that contains important information about the latent model capabilities themselves – if the same capability enables performance on both benchmarks then BenchMIRT can not separate them. More subtle constructs may be more challenging to apply BenchMIRT to, as BenchMIRT can only separate model abilities that are distinct.

I have read the authors' responses and I feel my fundamental concerns have not been addressed.

The authors have in fact done a fair amount of ablations to examine whether the dimensions learned on the MIRT model they fit on the datasets they selected indeed separates "safety" and "general reasoning". However, my concerns are more about the general approach.

If I want to use this approach on a selection of datasets for which the construct validity is unknown, I cannot draw any conclusions about the dimensions learned in the MIRT model (and any subsequent evaluations of construct validity from it), because the model might be affected by datasets that aren't strong in terms of their construct validity to begin with. For me to be able to trust the model, I need to know whether the datasets being selected to fit it on are sound in terms of construct validity. Since I require this external validation (say, from experts), it defeats the purpose of this approach to a large extent. The results from Appendix C seem to be inconclusive in addressing this.

Additionally, there doesn't seem to be enough evidence that (i) this model would work for separating a more complex combination of dimensions, or (ii) on dimensions that are different in nature from reasoning or safety (e.g. bias, sycophancy, hallucination, tool-use, etc.). The experiments the authors point to

In light of the above, I shall maintain my current rating.

We thank you for your response, and appreciate that you found our ablations on the BenchMIRT dimensions to be convincing. In response to your additional concerns, we would like to highlight a few points

If I want to use this approach on a selection of datasets for which the construct validity is unknown, I cannot draw any conclusions about the dimensions learned in the MIRT model

BenchMIRT allows for, and in fact depends on, the inclusion of multiple benchmarks around a concept to measure a single construct. That makes it robust to single benchmarks without construct validity (ie, though Toxigen is heavily saturated and does not strongly measure an ability, BenchMIRT still successfully uncovers a safety and general reasoning capability). The inclusion of multiple benchmarks and the exploratory nature of BenchMIRT helps to uncover 1) the number of capabilities represented and 2) what they represent. While the dimensions require human interpretation (as we did in this case for safety and general reasoning or math and instruction following), they enable further item, subset, and benchmark level conclusions without further expert validation.

(i) this model would work for separating a more complex combination of dimensions

We discuss this limitation in the analysis of the BenchMIRT run on the safety benchmarks -- BenchMIRT for example fails to separate HarmBench and Do Anything Now because the LLM models rely on the same capabilities to succeed. Safety and General Reasoning abilities have substantial research on the impact of their entanglement, and we believe that the benchmark, subset, and item-level analysis, with filtering and ranking implications, that BenchMIRT enables on these sixteen benchmarks represents a significant contribution.

(ii) on dimensions that are different in nature from reasoning or safety (e.g. bias, sycophancy, hallucination, tool-use, etc.)

Appendix C and Appendix D address separate combinations of benchmarks, and highlight that BenchMIRT is capable of separating, for example, Math and Instruction Following as capabilities. Due to compute constraints and the larger impact/interest in the nature of the capabilities, we focus on Safety and General Reasoning as the core abilities, and offer the ablations as evidence that it can generalize and be useful in additional applications, not as proof that it generalizes to all possible capabilities of interest. Additionally, we do include BBQ as a bias eval.

Thank you again for your time in reviewing.

Reviewer_gYSrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BenchMIRT which applies the Multidimensional Item Response Theory (MIRT) to analyze safety and general reasoning benchmarks in LLMs. The authors collect a lot of LLM responses from safety & general reasoning benchmarks and use the data to train BenchMIRT. Then, BenchMIRT is used to analyze benchmark items, the two-dimensional model is mainly used in this paper. The results show that BenchMIRT predicts the LLM responses to benchmark items better than several baselines such as using single-dimensional IRT. Also, BenchMIRT is used to analyze existing benchmarks and analyze some noisy items.

Critical strengths

Wonderful presentation, crisp writing, easy to understand, and can feel that a lot of time was contributed to this paper. Also the topic of applying MIRT in validating existing benchmark items makes sense and especially the topic of safety-reasoning tradeoff is timely.

Extensive experiments: A lot of experiments were done to support the main claims of this paper. For example, not only using two dimensions but three and four dimensions were used to support generalizability of the method. This gives a feeling that the findings of the paper are reliable. This is similar to other experiments in this paper.

Critical weaknesses

(Major) Can we really say that the two dimensions really represent safety and general reasoning? (If addressed properly, I am willing to increase my score.)

While the paper shows correlation scores it does not directly prove that each dimension corresponds to safety and general reasoning. Also, it’s a minor weakness but it feels kind of too obvious that the two dimensions are safety and general reasoning because the benchmarks analyzed are actually evaluating these two concepts. A more interesting question would be whether we can really say that BenchMIRT fitted to a particular set of datasets and models could be reliably used for new models and new benchmarks.

In the end let’s say we use BenchMIRT to analyze datasets, I personally think a more realistic thing to do is people manually looking at each item and mapping to predefined dimensions or filtering out noisy items. In this paper BenchMIRT was used to find out potentially problematic items in existing benchmarks, but we do not know how accurately it identifies these items and how much human effort is needed to validate them. For a more practical use the accuracy or the usefulness in finding problematic items should be mentioned. Since a lot of LLM responses should be gathered for BenchMIRT I personally believe this is important.

Questions for authors

Why would someone use BenchMIRT rather than just manually inspecting each item in a benchmark?

Future work: What happens when we have items but we actually don’t know what it measures? I think this is an exciting research question that could be explored as an extension of BenchMIRT.

Nice work :)

4 comments

We appreciate your thorough review and the fact that you find our experiments extensive and reliable. In order to address some of your questions, please see below:

TLDR:

  • W1: The two dimensions really represent safety and general reasoning, as supported by seven analyses across benchmark, model, and outside classifier verification.
  • W2: While the benchmarks were based on the fact that they evaluate these concepts, it is not obvious either that there were two capabilities or that those concepts would be in fact safety and general reasoning.
  • W2 and Q2: BenchMIRT can be reliably used for new models and new benchmarks, even if we don’t know what the latent factors are
  • W3 and Q1: BenchMIRT is more reliable and efficient than people manually labelling items, and allows for exploratory analysis.

W1: We offer seven analyses that confirm that the two dimensions really represent safety and general reasoning capabilities

Can we really say that the two dimensions really represent safety and general reasoning? (If addressed properly, I am willing to increase my score.) We would like to clarify that we verify the mapping of the safety and general reasoning dimensions and their stability in seven ways, which consistently suggest our interpretation is correct.

  • Benchmark-level scores
    • We compare the ability scores to the benchmark scores, and find strong correlations between our "safety" dimension and the safety benchmarks, and our general capabilities dimension and the capability benchmarks (Figure 2, lines 278-299).
  • Verifying the safety dimension with abliterated model pairs
    • We compare three LLMs with a version that has been abliterated, or deliberately modified to have no safety capability. We find that the abliterated versions significantly underperform on the items with the highest safety discrimination (Figure 3, lines 339-353). We also have now added a comparison between abliterated versions on the general reasoning dimension, and find that the paired t-tests across abliterated pairs where Dimension 1 discrimination > 2 result in scores of 4.98 (p approx 0, d = 0.05), 4.94 (p approx 0, d = 0.05), and 6.41 (p approx 0, d = 0.07), showing a significantly smaller gap that the safety dimension. We will add this analysis to Section 4.2. Both the existing and the new analyses indicate that the safety dimension indeed captures safety abilities in LLMs.
  • Verifying the general reasoning dimension with model size pairs
    • We compare five model pairs where we have different size models in the same family across the most discriminative safety and general capability items. While the larger models outperform the smaller ones generally, they do so more consistently on the general reasoning dimension and no smaller models outperform their larger partner on the general reasoning dimension while one (Llama 3.1) does on the safety dimension. (Appendix G, Figures 21 and 22). This strongly suggests that the general reasoning dimension in BenchMIRT captures general reasoning abilities in LLMs.
  • Verifying the safety dimension with Qwen3 Guard safety classifier
    • We classify all prompts as Safe, Controversial, or Unsafe according to the Qwen3 Guard safety classifier, finding Pearson r = -0.77 between that score and Dimension 0 discrimination and Pearson r = -0.22 with Dimension 1 discrimination, showing a strong positive correlation between unsafe prompts and more discriminative safety items, and a slight negative correlation between unsafe prompts and more discriminative general reasoning items (Section 4.2 lines 339-348, Appendix K).
  • Confirming the stability of the dimensions across repeated MIRT runs
    • We rerun the BenchMIRT fit in a ten fold cross validation across the LLM-item pairs, and find a strong stability in the item scores (Appendix E, referenced on line 285)
  • Confirming the stability of the dimensions in a held out cross validation where we train on 80 LLMs at a time
    • We rerun the BenchMIRT fit in a five fold cross validation across the LLM models, and find a strong stability in the item scores on all folds with diversity in scores. (Appendix E, referenced on line 285)
  • New: we compare the safety classifications of the responses from the Llama 3.1 8B model and its abliterated partner to the Dimension 0 and 1 discriminations. This shows that the items with the strongest safety discrimination are indeed the ones the abliterated model generates harmful content in response to.
Dimension 0Dimension 1
abliterated−0.76-0.30
original−0.39−0.23

While concepts like safety are difficult to fully verify, we offer independent benchmark, model, and safety classifier confirmations.

W2: The two-factor solution is not too obvious

it feels kind of too obvious that the two dimensions are safety and general reasoning because the benchmarks analyzed are actually evaluating these two concepts.

While a two-factor solution was hypothesized, it is not obvious that a safety and general reasoning capability would indeed emerge. These are broad and varied categories: safety includes tasks ranging from jailbreak resistance to identifying offensive text, while general reasoning includes math and instruction following, which are quite distinct. Selecting benchmarks from these two groups does not establish that performance within each group reflects a single latent factor, or that the factors would align with these groups at all.

Prior work [1] has suggested that the format of a question may significantly impact a model’s ability to answer, raising the possibility of a capability split between multiple-choice questions and open-ended responses. A two-factor solution instead of a single factor or three or higher was also not a trivial emergence. SafetyWashing [2] finds that safety benchmarks may be highly correlated with general ability benchmarks, suggesting a single capability factor underlying all of the benchmarks. There was also the possibility that the set of general reasoning benchmarks or set of safety benchmarks could be better modeled by their low-level sub-categories (consider jailbreaking vs vanilla prompts or general knowledge vs mathematical ability). We further examined this by fitting three- and four-dimensional MIRT models. As Appendix J demonstrates (referenced on line 273), three and four dimensional MIRT models do not offer significantly better accuracy. This supports a two-factor representation for the benchmarks and models studied, rather than a claim that safety and general reasoning are the only possible capabilities. We recognize that this distinction may not have come through clearly in the paper and will make it more explicit.

W2 and Q2: BenchMIRT can be reliably used for new models and new benchmarks

A more interesting question would be whether we can really say that BenchMIRT fitted to a particular set of datasets and models could be reliably used for new models and new benchmarks.

What happens when we have items but we actually don’t know what it measures? I think this is an exciting research question that could be explored as an extension of BenchMIRT.

In Appendix D (referenced on line 356), we test the generalization of this method to measure math and instruction following abilities. We fit a new MIRT function to the MMLU-pro math subset, the Math benchmark, and IFEval, and find three distinct capabilities – one for IFEval (possibly measuring instruction following), one capturing both MATH and MMLU-Pro (mathematical reasoning), and one capturing an additional MMLU-Pro ability (potentially a multiple choice formatting ability).

In Appendix C (referenced on line 356), we additionally test the generalization of BenchMIRT when fit on on just the safety benchmarks, just the safety benchmarks without BBQ/WMDP, and just the general reasoning benchmarks. We find that the full split of safety benchmarks reproduces the split found in the paper, with WMDP and BBQ separated to a separate latent factor, with Toxigen and XSTest loading on both. The safety benchmarks without WMDP and BBQ when run with BenchMIRT separate out Toxigen and XSTest, indicating that the model is very stably reproducing a similar split. Interestingly, there does not appear to be a clear split between jailbreak safety evals and more traditional safety evals – DoAnythingNow and Harmbench have very similar scores, for example. The reasoning results appear to split out Math.

As a new experiment, we ran the same accuracy testing from Section 4.1 and Table 1 on the subsets from Appendix C and Appendix D, and found similar accuracies to the full BenchMIRT method. We will add those results to Appendix C and Appendix D and add a discussion about generalization to the Conclusion.

In addition, this shows that BenchMIRT may be directly applicable to the future work you suggest to evaluate ability splits when we are not sure what benchmarks are measuring. Because BenchMIRT does not label any items, and is flexible to the number of dimensions, it is a method of exploratory analysis. It can be used as is to probe how many abilities stand out in the first place, as well as examine each resulting capability factor.

[1] Ivan Yee Lee, Loris D’Antoni, and Taylor Berg-Kirkpatrick. The format tax. arXiv preprint arXiv:2604.03616, 2026.

[2] Richard Ren, Steven Basart, Adam Khoja, Alexander Pan, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Gabriel Mukobi, Ryan H Kim, et al. Safetywashing: Do ai safety benchmarks actually measure safety progress? Advances in Neural Information Processing Systems, 37: 68559–68594, 2024

W3 and Q1: BenchMIRT is more reliable and efficient than manual inspecting and labeling

I personally think a more realistic thing to do is people manually looking at each item and mapping to predefined dimensions or filtering out noisy items.

Why would someone use BenchMIRT rather than just manually inspecting each item in a benchmark?

BenchMIRT has four main advantages over manual labeling. First, it is much more efficient. Manually examining just flagged items is substantially more efficient than examining the full set.

Second, BenchMIRT is exploratory by nature, which means you do not need to know how many factors or what they are in order to analyze a dataset. Human annotations of these datasets might catch incorrectly labeled items, or clear conflations of themes, but would not be able to address the question of how many underlying factors or how related they are.

Third, BenchMIRT allows for an automated filtering of the benchmarks. Appendix B, referenced on line 465, shows that by using item difficulty and discrimination, it is possible to filter benchmarks to 50% or 10% of their original size and not only maintain the correlation to the desired capability but in fact increase it. This demonstrates the overall accuracy of the item labels.

Finally, BenchMIRT relies on the responses of LLMs instead of the content of the prompt. A human annotation would note that the copyright, contextual, and standard subsets of HarmBench cover substantially different content, but it is the responses of the LLMs that allow BenchMIRT to note that the capability needed for the copyright subset aligns less closely with the safety factor. A human annotation might catch clearly out of place or incorrectly labeled items, but it can not catch these latent capability factors.

We will add a discussion of the specific applicability of BenchMIRT to Section 4.4.

Thank you for the rebuttal. My initial concerns have been resolved, especially regarding whether the two dimensions really represent safety and general reasoning.

I have raised my score from 2 to 3.

Please make sure the rebuttal contents are incorporated into the paper.

Thank you.

Reviewer_qphhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-dimensional, two-parameter MIRT model fit to item-level responses. The two latent dimensions are not specified in advance, instead the authors interpret them post hoc as "safety" and "general reasoning." The paper audits benchmarks and concludes that BBQ and WMDP depend mainly on general capability rather than safety, that subsets of XSTest and HarmBench measure different abilities, and that some individual items appear to be mislabeled.

Critical strengths

The authors ran the same 100 models on ten safety benchmarks to match the existing capability data, which is pretty extesnive.

Critical weaknesses

My biggest concern is the limited impact of the idea, and the lack of claritiy in the two-dimensional split. The paper applies a standard two-dimensional 2PL MIRT model to a response matrix, so the main contribution is data collection and model fitting. The paper does not show what further research BenchMIRT enables beyond descriptive audit plots. For example, it does not show that the factors change a conclusion about specific models or training methods. Also, the two constructs found depend on the chosen benchmarks. With six capability and ten safety benchmarks selected a priori, a two-factor solution aligned with these two groups is largely built into the design. Moreover, the mapping of each dimension to "safety" and "general reasoning" is weakly justified. The benchmark-level conclusions overlap with other prior work like SafetyWashing. In its current form, the contribution is not significant enough for ICLR.

Questions for authors

Some clarification questions: Where does an always-refuse policy fall on Dimension 0? How does Dimension-0 ability correlate with refusal rates on benign prompts (the XSTest and WildJailbreak benign subsets)? Can you show one downstream use in which BenchMIRT changes a conclusion or a decision? eg., how a safety-training method or a model ranking is judged, or cheaper evaluation of held-out models with the filtered items.

6 comments

We thank you for your detailed review and feedback, and appreciate that you found our model selection to be extensive. Please find below a set of answers to your questions.

TLDR;

  • W1: We clarify that BenchMIRT’s main contribution is a benchmark auditing method that is generalizable to additional capabilities, including reasoning and safety subsets and a Math/Instruction Following ability split.
  • W2 and W3: We confirm that safety and capability mapping to our two dimensions is valid and correct with seven different analyses, and that the two-factor solution is non trivial.
  • W4: We address the uniqueness of BenchMIRT’s conclusions compared to earlier work.
  • Q1: We also conduct additional experiments on the relationship between refusal and the safety capability.
  • Q2: We show the generalizability of the filtering experiments to held out models. Further research enabled by BenchMIRT could include filtering methodologies to make more efficient and aligned benchmarks on a range of capabilities.

W1: BenchMIRT’s main contribution is a benchmark auditing method, not just data collection and model fitting.

The paper applies a standard two-dimensional 2PL MIRT model to a response matrix, so the main contribution is data collection and model fitting.

We would like to clarify that the main contributions of this paper include 1) a validation that MIRT is capable of discovering and defining the measurement of LLM model capabilities across benchmarks and thus a method for auditing benchmarks more broadly, 2) benchmark, subset, and item level construct validation across all 16 benchmarks, including filtering methods.

The goal of this audit is to understand what benchmarks and leaderboards actually measure and whether their scores support the comparisons we use them for. Auditing at the benchmark, subset, and item levels allows us to examine whether a benchmark measures a consistent construct, where its subsets differ, and which items align with the intended construct.

This can change conclusions about benchmarks and how they should be used, rather than only providing descriptive plots. For example, when different subsets of a benchmark such as HarmBench measure different constructs, averaging them can hide meaningful differences between models and make the resulting ranking less representative of the capability we want to evaluate. For practitioners, the audit helps determine which benchmarks or subsets to use, when to report subset scores separately, and which items to filter. These decisions matter when comparing models or training methods: a higher average score does not necessarily mean better performance on the intended capability.

We acknowledge that the current writing may not have made the implications of this auditing method clear enough, and we will revise the paper to address this.

W2: The two-factor solution is non trivial

With six capability and ten safety benchmarks selected a priori, a two-factor solution aligned with these two groups is largely built into the design.

While a two-factor solution was hypothesized, it is not trivial that a safety and general reasoning capability would indeed emerge. These are broad and varied categories: safety includes tasks ranging from jailbreak resistance to identifying offensive text, while general reasoning includes math and instruction following, which are quite distinct. Selecting benchmarks from these two groups does not establish that performance within each group reflects a single latent factor, or that the factors would align with these groups at all.

Prior work [1] has suggested that the format of a question may impact a model’s ability to answer, raising the possibility of a capability split between multiple-choice questions and open-ended responses. A two-factor solution instead of a single factor or three or higher was also not a trivial emergence. SafetyWashing [2] finds that safety benchmarks may be highly correlated with general ability benchmarks, suggesting a single capability factor underlying all of the benchmarks. There was also the possibility that the set of general reasoning benchmarks or set of safety benchmarks could be better modeled by their low-level sub-categories (consider jailbreaking vs vanilla prompts or general knowledge vs mathematical ability). We further examined this by fitting three- and four-dimensional MIRT models. As Appendix J demonstrates (referenced on line 273), three and four dimensional MIRT models do not offer significantly better accuracy, and allowing additional dimensions still yields only 2 interpretable capability factors. This supports a two-factor representation for the benchmarks and models studied, rather than a claim that safety and general reasoning are the only possible capabilities. We recognize that this distinction may not have come through clearly in the paper and will make it more explicit.

[1] Ivan Yee Lee, Loris D’Antoni, and Taylor Berg-Kirkpatrick. The format tax. arXiv preprint arXiv:2604.03616, 2026.

[2] Richard Ren, Steven Basart, Adam Khoja, Alexander Pan, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Gabriel Mukobi, Ryan H Kim, et al. Safetywashing: Do ai safety benchmarks actually measure safety progress? Advances in Neural Information Processing Systems, 37: 68559–68594, 2024

W3: The mapping of safety and general reasoning dimensions is independently justified with multiple analyses

the mapping of each dimension to "safety" and "general reasoning" is weakly justified.

We would like to clarify that we verify the mapping of the safety and general reasoning dimensions and their stability in seven ways, which consistently suggest our interpretation is correct.

  • Benchmark-level scores
    • We compare the ability scores to the benchmark scores, and find strong correlations between our "safety" dimension and the safety benchmarks, and our general capabilities dimension and the capability benchmarks (Figure 2, lines 278-299).
  • Verifying the safety dimension with abliterated model pairs
    • We compare three LLMs with a version that has been abliterated, or deliberately modified to have no safety capability. We find that the abliterated versions significantly underperform on the items with the highest safety discrimination (Figure 3, lines 339-353). We also have now added a comparison between abliterated versions on the general reasoning dimension, and find that the paired t-tests across abliterated pairs where Dimension 1 discrimination > 2 result in scores of 4.98 (p approx 0, d = 0.05), 4.94 (p approx 0, d = 0.05), and 6.41 (p approx 0, d = 0.07), showing a significantly smaller gap that the safety dimension. We will add this analysis to Section 4.2. Both the existing and the new analyses indicate that the safety dimension indeed captures safety abilities in LLMs.
  • Verifying the general reasoning dimension with model size pairs
    • We compare five model pairs where we have different size models in the same family across the most discriminative safety and general capability items. While the larger models outperform the smaller ones generally, they do so more consistently on the general reasoning dimension and no smaller models outperform their larger partner on the general reasoning dimension while one (Llama 3.1) does on the safety dimension. (Appendix G, Figures 21 and 22). This strongly suggests that the general reasoning dimension in BenchMIRT captures general reasoning abilities in LLMs.
  • Verifying the safety dimension with Qwen3 Guard safety classifier
    • We classify all prompts as Safe, Controversial, or Unsafe according to the Qwen3 Guard safety classifier, finding Pearson r = -0.77 between that score and Dimension 0 discrimination and Pearson r = -0.22 with Dimension 1 discrimination, showing a strong positive correlation between unsafe prompts and more discriminative safety items, and a slight negative correlation between unsafe prompts and more discriminative general reasoning items (Section 4.2 lines 339-348, Appendix K).
  • Confirming the stability of the dimensions across repeated MIRT runs
    • We rerun the BenchMIRT fit in a ten fold cross validation across the LLM-item pairs, and find a strong stability in the item scores (Appendix E, referenced on line 285)
  • Confirming the stability of the dimensions in a held out cross validation where we train on 80 LLMs at a time
    • We rerun the BenchMIRT fit in a five fold cross validation across the LLM models, and find a strong stability in the item scores on all folds with diversity in scores. (Appendix E, referenced on line 285)
  • New: we compare the safety classifications of the responses from the Llama 3.1 8B model and its abliterated partner to the Dimension 0 and 1 discriminations. This shows that the items with the strongest safety discrimination are indeed the ones the abliterated model generates harmful content in response to.
Dimension 0Dimension 1
abliterated−0.76-0.30
original−0.39−0.23

While concepts like safety are difficult to fully verify, we offer independent benchmark, model, and safety classifier confirmations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.