acceptodds
Withdrawn from ICLR 2027

CFI: Capability Fisher Information for Benchmark Diagnosis in LLM Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In current practice, LLM benchmarks are often evaluated by the reliability of the model rankings they produce, yet stable rankings do not necessarily imply sufficient and balanced measurement of the capabilities they are intended to assess. We propose Capability Fisher Information (CFI), a measurement-theoretic framework for diagnosing benchmark structure through the distribution of information in a benchmark-specific capability space. CFI represents each item as an information contribution characterized by a capability direction and strength, aggregates these contributions into a benchmark-level information operator, and supports three complementary diagnostics: Coverage, Uniformity, and Redundancy. Together, they expose capability blind spots, information imbalance, and repeated probing that overall accuracy or item counts cannot reveal. Across text-only and multimodal benchmarks, CFI reveals structural deficiencies overlooked by conventional statistics, while controlled structural perturbations and held-out model experiments validate the effectiveness of its diagnostics.

Primary area:
datasets and benchmarks
Keywords:
LLM Benchmarks, Benchmark Diagnostics, Capability Fisher Information, Evaluation Fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_feyXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper seeks to use a metric with mathematical similarities to Fisher Information to provide meaningful benchmark-specific diagnostics. They do this by reframing the typical item-discrimination/uncertainty estimated from multidimensional 2PL IRT settings as strength and direction mapped into categories/clusters using encoder-based textual embeddings. The categories/clusters are either accepted from the benchmark (MMLU-Pro) or they are determined by clustering, the capability space is projected to be approximately orthonormal to support dimensional capabilities. Instead of fitting a 2PL model, the paper introduces two operational strength regimes (uniform, population-referential), which help define the CFI metric. From it, they construct three diagnostics: coverage (an E-optimality criterion), uniformity (a version of Mauchly’s sphericity statistic, a type of spectral balance arising from geometric/arithmetic mean ratio), and redundancy (item-wise average maximum pairwise semantic/directional overlap). With a train/test panel of 10 open models each, they calculate these diagnostics for 7 benchmarks, and for three benchmarks with predefined categories they show category-level radar plots. They report that when they manipulate one benchmark's (MMLU) structures along the quantities used for the diagnostics, the diagnostics change accordingly (in an expected-by-construction design). They report modest categorical mean rank correlations for MMLU pro for Coverage and Uniformity with weaker associations for Redundancy. They show that Gram-whitening the category prototype text embeddings results in the best correlations for the former two.

Critical strengths

(1) Important question and shared vocabulary

WHY: The paper asks an important question (does a benchmark measure what it claims to measure) and responds with vocabulary from measurement theory and from optimal design statistics. These insights have the potential to yield meaningful diagnostics for the shifting landscape of LLM benchmarks, and connect to tools and methods that could help.

Critical weaknesses

(1) The paper does not provide evidence that the indicators measured are meaningful

WHY: After the setup (4.1) in the experiment results sections, section 4.2 is descriptive, 4.3 results are expected-by-construction (which is fine, but do not contribute to paper's claims); 4.4 reports no Cis on a very small sample, while being pooled across different strategies and structures (with no statistical disambiguation e.g. within-strategy correlations), making the relationships claimed tenuous; 4.5 demonstrates within-framework prototype-embedding stability (but does not provide evidence to support that the associations are meaningful and generalize across benchmarks).

Without providing empirical evidence for their appropriateness, the metrics created and the calculations rely heavily on sematic representations and either pre-existing labels (which would need to have evidence to show that such pre-existing labels are justified, as other research has shown scant evidence to support this idea https://openreview.net/forum?id=QLSH4qoj77, https://openreview.net/forum?id=Vr4YScS9DT, https://arxiv.org/pdf/2511.04703) or clustering. It is not clear that the semantic representations and pre-defined categories are reflective of capability spaces.

The paper never compares against trivial statistics that capture the same structure: the smallest per-category item count, the entropy or effective number of categories, the log-determinant or k-center coverage of raw embeddings, or category-stratified sampling as a selector. If simple label-count statistics explain the same variance, CFI adds little beyond "balance your categories", which may or may not even be needed or desirable, depending on the instrument's intended construct/use. Within-strategy correlations and partial correlations controlling for these baselines are needed to show that CFI captures something non-trivial.

(2) The language and claims extend far beyond the analyses

WHY: The CFI metric is not directly statistically related to Fisher information, despite the name--other than sharing an algebraic construction form (a sum of weighted outer products), 2PL Fisher Information uses item discrimination to determine how much precision can be expected for a latent ability, whereas this study's measure is based on a semantic direction and is closer to a weighted covariance matrix. A very clear test to provide evidence as to whether the algebraic similarity is capturing the claimed quantities of interest would be to compare CFI quantities with actual 2PL discrimination/information values, which is not done in the paper. Having a mismatch between the title and the mathematics is a symptom of a larger mismatch between language and experiments. Recommend a new name for the CFI metric that better matches the relationship it is capturing, rather than the construction that inspired it.

The abstract says CFI "reveals structural deficiencies overlooked by conventional statistics", but I cannot find evidence to support that in the paper. The paper does not have, for example, demonstration or evidence that rankings in specified categories are less stable, less predictive of performance on independent tasks/benchmarks for the same skill, or improved by adding items as we would expect. Using clustering to define groups and then measuring association of clusters as groups has a circular logic: the cluster-defined "capability space" is defined with the same embeddings, so Coverage then measures cluster balance.

The paper wants to offer a diagnostic framework, but a diagnositc framework needs evidence that its diagnoses are correct and lead to better benchmarks. Without that, the practical contribution is hard to judge.

Questions for authors

I initially had selected "clear rejection", but I thought of questions that could change my view:

(1) Can you demonstrate the relationship to actual Fisher information? Using a large response matrix (e.g., Open LLM Leaderboard results on MMLU-Pro), can you fit a multidimensional 2PL model and report (i) the alignment between estimated discrimination directions a^i\hat a_i and your viv_i, and (ii) the correlation between item and subset Fisher information under the fitted model and the corresponding CFI quantities? This would show whether CFI is a valid approximation/substitution or only an analogy.

(2) Can you demonstrate the effectiveness of the semantic embeddings approach as discriminative on benchmarks that have semantically similar items with different capability profiles? Something that would change my doubt here would be to use the large response matrix from Open LLM Leaderboard results on the MuSR benchmark across the three pre-defined domains of MuSR. Again, can you fit CFI statistics and IRt statistics across these larger datasets and show that the embeddings capture quantities that are capability-information-relevant? This would help to show whether the categorical and semantic information carries sufficient signal to be used as shown, and would be an initial point of entry to justify claims related to construct validity based on semantics/categories.

For the above two data examples, no additional LLM evaluations would need to be run, as the data already exists, hopefully making the calculation reasonable for rebuttals (https://huggingface.co/open-llm-leaderboard and I think the extracted response data is available in some of the papers linked in the weaknesses section). The latter, using the MuSR dataset, could help resolve some of my concerns about the statistical difference between Fisher Information and the CFI metrics based on prototype embeddings (and the appropriateness of embeddings for capturing measured constructs--for example, Gram whitening may help with collinearity, but that doesn't mean that the dimensions represent valid psychometric capabilities--it might just be topical similarity or something).

(3) how sensitive are the diagnostics to the number of clusters K and the dimension? How do you avoid Coverage simply reflecting cluster balance?

(4) For SEED-Bench, why do you think item counts very uneven while qcq_c is nearly even? Does this mean the whitened prototypes are poorly separated? Can you show a case where a low-qcq_c category independently is associated with poorer measurement (e.g., less stable rankings, lower test–retest reliability, IRT estimates, etc.)?

(5) Can you report correlations within each sampling strategy (e.g., among the 50 random subsets), with bootstrap CIs, for Coverage, Uniformity, and Redundancy, and the same for some baselines like smallest category count, category entropy, k-center radius, etc. Does CFI retain predictive value after controlling for these (e.g., partial correlations, incremental R2R^2, etc.)? Can you compare these correlations to a random baseline, e.g., if items are assigned to random category labels?

Reviewer_SaxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for diagnosing how a benchmark distributes measurement information across its intended capabilities, motivited by the rank-one Fisher information of a multidimensional 2PL IRT model. Empirically, the paper reports that Coverage and Uniformity correlate with how well 200 MMLU-Pro subsets preserve the rankings and scores of 10 held-out models. It also checks robustness to the choice of text encoder and basis calibration.

Critical strengths

An interesting, IRT-inspired way to look at benchmark structure. Also, the analysis was quite extensive.

Critical weaknesses

My main concern is the practical utility of CFI not demonstrated beyond qualitative insight. Most of the evidence is descriptive: CFI is applied to existing benchmarks and the resulting profiles are reported. The missing piece is the evidence showing CFI yields a meaningful benefit, e.g., using it for importance sampling or efficient evaluation.

Also, although the connection to Fisher information is largely motivational, the operational CFI replaces the response-based discrimination direction with a semantic embedding direction, and the strength ignores item discrimination, so it can be large even for items that do not separate models. The validation does not fully close this gap, in my opinion.

Questions for authors

What concrete decision or downstream task does CFI improve that existing tools cannot?

How can we be confident that Coverage, Uniformity, and Redundancy capture capability-level measurement structure rather than properties of the embedding space and of the benchmark's own taxonomy?

Reviewer_EacEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that model rankings and benchmark accuracy do not imply that the benchmark is capable of measuring the intended capabilities. To remedy this it proposes a framework called capability Fisher information. The work defines the concept of capability space, which is specific to a benchmark, and uses semantic embeddings of the samples in the benchmarks and predefined capability categories. These encoded capability contributions are then aggregated into a positive semi-definite matrix. This matrix is then used to define three metrics - Coverage, uniformity, and redundancy. The coverage is effectively the smallest eigenvalue of the Fisher information matrix, which roughly corresponds to the least supported capability. Uniformity is calculated by the ratio of the geometric and arithmetic means of eigenvalues and is intended to measure how evenly the information in the benchmark is distributed across all the measured capabilities. Redundancy measures pairwise similarity between capabilities. The work evaluates these capability-encoded metrics across several text and a few multimodal benchmarks.

Critical strengths

Building a theory of relevant benchmarking capabilities centered around information, measure, and measurement theory is an important problem. The key idea of understanding whether, LLM benchmarks are capable of adequately measuring the capabilities with, hopefully, good sample efficiency is also important. This work attempts to introduce a relatively simple framework based on capability "directions and strengths". The experimental range is adequate, and there is an interesting discussion about how "informativeness of an evaluation item” changes with the population of models being evaluated.

Critical weaknesses

While the problem is relevant and interesting. I think the central concepts and assumptions are insufficiently defined and motivated Concepts like global capability space and benchmark-specific capability subspace are introduced in the paper without sufficiently precise descriptions which establish exactly what constitutes a capability and how such a space can be meaningfully constructed or mathematically represented. In this paper, it seems to ,mean approximating “capabilities” by semantic embeddings and categories. But semantic similarity between questions does not necessarily imply similarity in the abilities needed to answer them. The relationship between the embedding space, as discussed in the paper, and the concrete capabilities concrete is not very clear to me and doesn't seem to be convincingly validated or theoretically established in the paper. In addition, although the paper uses measurement theory it does not clearly specify the underlying measurement model or valid. Neither is the Fisher information formulation fully convincing. It is unclear whether the response variance across various models is truly equivalent to the sensitivity of some specific latent capability. It seems that an item may have non-trivial response variance while providing little to no information about the intended capability. I'm unsure whether the proposed matrix estimates are truly Fisher information, or perhaps a weighted second moment matrix of the given semantic embeddings. An orthogonality idea is mentioned early on in the introduction, but is poorly motivated or explained. Although Gram whitening is used to construct an approximately orthogonal capability direction space, it is grounded in an unproven assumption: that meaningful capabilities are truly orthogonal or independent of one another (particularly in a given LLM evaluation, where different tasks like reasoning, mathematical understanding, comprehension, or any other specific domain knowledge can overlap substantially). I think there is no adequate justification as to why the transform-orthogonal geometry transformation is an appropriate one for evaluating benchmark quality. Also this does not account for the fact that orthogonality, approximate or not, is not guaranteed if the prototype matrix is rank deficient or poorly conditioned.

I also have a few minor concerns in terms of methodological novelty. While the problem itself is quite interesting and relevant, the mathematical tools used and the way they are used in the paper are combinations of existing ideas. For example, the smallest eigenvalue of an information matrix is a well-known criterion, while the determinant-to-trace formulation is another well-known measure for spectral balance. Using pairwise similarity for redundancies is also fairly natural and the derivation of FI is standard. The novelty of the work seems to be applying and combining these ideas in a semantic embedding space of benchmarks. I think the authors need to do more to establish that their formulation offers a more intuitive and mathematically grounded approach than other simpler approaches based on embedding diversity and categorical balance, spectral properties or existing IRT methods.

Another drawback of the work is that the important capabilities space is made using the benchmarks on items/categories, which seems to go against the earlier-built intuition in the paper that it attempts to figure out whether the benchmarks themselves lack crucial categories. It is unclear that a framework like this will be able to detect capabilities that are completely absent from the benchmark, because they may not be represented in the constructed space at all. Moreover, the three metrics introduced are fairly abstract and it is not clear under what conditions they can be interpreted as indicators of benchmark quality.

There are other issues, e.g., , unavailable capabilities will weaken the coverage metric. Duplicating items would simply increase the smallest eigenvalue. It is unclear whether the uniformity metric is a desirable diagnostic for every relevant benchmark, since different benchmarks may choose to focus on specific or rare capabilities, especially in OOD scenarios.

In summary, the work has interesting ideas and potential but at present lacks sufficient clarity.

Questions for authors

What exactly constitutes a capability in your framework? Why is orthogonality a desirable property of the capability space? Ideally, please provide theoretical or empirical justification. Why does using semantic embedding vectors make more sense than the ordinary discriminant vectors in Fisher information derivation? What happens to the coverage metric when entire capabilities are missing from the benchmark? Are the proposed metrics significantly different from existing approaches? Does uniform information allocation necessarily indicate a better benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.