acceptodds
Under review as a conference paper at ICLR 2027

From Flat Scores to Capability Profiles: A Capability-Cognition Framework for Generative AI Benchmarking

Abstract

Scalar benchmark scores obscure which capabilities a benchmark samples and at what cognitive depth. We introduce Capability-Cognition Profiling (CCP), which represents a benchmark as a distribution over domain content and cognitive operations , and characterises it through five profile descriptors, content breadth, cognitive breadth, cognitive depth, cognitive evenness, and profile overlap, plus two corpus-contribution measures (exclusive and Shapley coverage contribution). CCP does not rank benchmarks by quality; it makes explicit the capability distribution over which benchmark scores are computed. We instantiate CCP on world culture by constructing a multilingual consensus taxonomy (1,291 leaves, 15 depth levels; Chu–Liu–Edmonds arborescence over Wikipedia language editions) and mapping 29 major cultural benchmarks into the resulting capability space via a two-model LLM jury with an open-weight arbiter and independent human validation at multiple resolutions. Across seven contemporary models on seven benchmarks (9,800 evaluations), joint profile distance is significantly associated with pairwise rank disagreement under an exact benchmark-level permutation test (, exact ; leave-one-benchmark-out mean ), while content-only and cognition-only distances do not individually reach significance (); the joint's advantage over each marginal is itself not statistically established (). Composition standardisation to a common target produces one nominal leaderboard change, although uncertainty is substantial. The formalism and the graph-consensus taxonomy recipe are both domain-general; only the Wikipedia instantiation is culture-specific.

Primary area:
datasets and benchmarks
Keywords:
Generative AI Evaluation, Capability-Centric Benchmarking, Cognitive Depth, Consensus Tree Reconstruction, Cultural Intelligence, Benchmark Coverage

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z1oKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper derives some indicators of benchmarks using cultural domains inferred from wikipedia categories and a cognitive depth derived from Bloom's taxonomy of educational objectives.

Critical strengths

It's important to extract profiles of what benchmarks measure. In terms of culture, this may be less known and relevant.

The language asymmetries (page 6) may be an interesting finding, but this is buried in an overly complicated paper for what it brings.

Critical weaknesses

Poor connection with closely related work: Cognitive profile of benchmarks has been happening in the past few years. The authors use a terminology that is alien to the standard terminology of AI evaluation (partly borrowed from cognitive science, and psychometrics), about constructs, latent factors, cognitive capabilities. It ignores existing profiling methods, https://arxiv.org/pdf/2510.01232, most notably https://www.nature.com/articles/s41586-026-10303-2. This one is based more on knowledge and culture (but the above paper also covers the Bloom taxonomy in the appendix) than core capabilities, but it also tries to identify what each benchmark measures (in dimensions, and in levels), as in eq. 1. The related work is a concatenation of references without insight. On page 9 there's more in green than black! There's a bit of everything, put together, without much structure.

Lack of justification in cognitive science or cultural studies. There's this use of Bloom's taxonomy borrowed from education, without further justification. The definitions in section 3 are not given any justification. It also relies too soon to appendices, so it's hard to follow or see what each of these concepts mean, and how they relate to concepts in cognitive science or AI evaluation. The references are mostly about statistical methods or corrections, but not really about the concepts.

Motivation and usefulness: The empirical results of predicting rank disagreement do not clarify what this would be useful for and what problem it solves (that hasn't been addressed before). The RQ are expressed in terms of the own definitions, and are inward-looking rather than answering questions that matter. We see benchmarks in a map of two dimensions, but this could have been done with other much simpler approaches. There's no comparison with any other method, not only in possible results but in advantages or disadvantages.

*** Other comments that may be useful, I didn't know where to put them *** Avoid acronyms like "GenAI". Despite focusing on "cultural" benchmarks, the title doesn't reflect it.

Questions for authors
  • Could you use some other levels beyond Bloom's taxonomy. Why is it good for cultural dimensions?
  • Why did you choose the current benchmarks?
  • Is the choice of language arbitrary or there are confounders in benhmarks/languages?
Reviewer_scvtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Capability–Cognition Profiling (CCP), a framework for characterizing benchmarks as distributions over content domains and Bloom's cognitive levels rather than relying solely on aggregate scores. The authors construct a multilingual cultural taxonomy from Wikipedia, annotate 29 cultural benchmarks, and evaluate seven models on seven benchmarks. They find a moderate correlation between joint profile distance and model-ranking disagreement and explore how composition standardization affects leaderboards.

Critical strengths
  1. The paper addresses a relevant problem in LLM evaluation: aggregate scores can obscure substantial differences in benchmark composition.

  2. The multilingual taxonomy construction and annotation of 29 cultural benchmarks represent a considerable data collection and engineering effort.

  3. The paper provides several validation and sensitivity analyses, including human annotation, permutation testing, and leave-one-benchmark-out analysis.

Critical weaknesses
  1. Insufficient comparison with closely related work.

The main motivation and several methodological components substantially overlap with existing research. Huber and Niklaus (COLING 2025) and Zhang et al. (EMNLP 2025) have already applied Bloom's Taxonomy to analyze cognitive coverage across LLM benchmarks. Notably, Zhang et al. also examine both cognitive and knowledge dimensions and report similar findings regarding the dominance of lower-level cognitive tasks. Furthermore, BenchHub (Kim et al., 2025/2026) already supports composition-aware benchmark analysis, including fine-grained coverage analysis and model comparisons under different evaluation compositions.

These closely related works are not adequately discussed. The paper's characterization of prior Bloom-based evaluations as primarily operating within individual benchmarks is also incomplete. Consequently, it remains unclear what substantial methodological or empirical advances CCP provides beyond existing approaches.

  1. The main empirical results do not establish the advantage of the proposed joint representation.

The authors report that joint content–cognition profile distance correlates with ranking disagreement (r = 0.458, p = 0.024). However, content-only and cognition-only distances achieve relatively similar correlations (r = 0.347 and 0.375), and the joint representation does not significantly outperform either baseline (both p > 0.23).

Therefore, the experiments do not establish that combining the two dimensions provides meaningful additional predictive value. Moreover, the analysis involves only seven benchmarks, limiting the strength and generalizability of this central result.

  1. The composition-standardization experiment provides little evidence for meaningful leaderboard changes.

The authors motivate CCP partly through the consequences of benchmark composition for model rankings. However, the seven evaluated benchmarks share only one common L2 × Bloom cell, requiring standardization at the much coarser L1 level. The resulting single nominal ranking reversal has a confidence interval that includes zero.

Thus, the experiment does not convincingly establish that CCP-based standardization produces reliable or practically meaningful changes in model comparison. This is especially concerning given that similar composition-dependent ranking effects have already been explored in prior work.

  1. Uncertainty in the capability annotations is insufficiently accounted for.

The framework relies heavily on accurate content and Bloom-level annotations, yet agreement with independent human annotations is only 63.2% at culture L2 and 58.9% for Bloom levels (weighted κ = 0.44). The Bloom-controlled rewriting experiment provides some supporting evidence, but increased difficulty or linguistic complexity could also explain the observed accuracy decline.

It remains unclear how annotation errors and taxonomy choices affect the reported profile distances, coverage statistics, and ranking correlations. Without a robustness analysis propagating this uncertainty into the main results, the reliability of the conclusions is difficult to assess.

Questions for authors
  1. Incremental value: Given that the joint profile does not significantly outperform either marginal representation, what evidence supports the necessity of the joint content–cognition framework? Would the authors consider evaluating more benchmarks and comparing against simpler composition-based predictors?

  2. Annotation reliability: How sensitive are the main results to annotation errors, particularly in Bloom levels? Would the correlation between profile distance and ranking disagreement remain stable under human-validated labels or alternative annotation pipelines?

  3. Leaderboard standardization: Could the authors clarify the reported CulturalBench ranking reversal? Table 8 reports a positive reweighted difference for "Claude vs. Llama" while stating that positive differences favor the first model, which appears inconsistent with the claimed reversal. Also, how do the 385–1,316 common-support item counts in Table 6 relate to the 200-item-per-benchmark model evaluation?

Reviewer_hybfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper breaks down cultural benchmarks using two axes: cultural content and the cognitive operation required to answer a question. Then, it builds a shared taxonomy from multilingual Wikipedia, labels questions from 29 cultural benchmarks, and finds that benchmarks with different profiles tend to produce different (multilingual) model rankings.

Critical strengths

I do agree with their take, that "cultural knowledge" benchmark is very diverse and does not always measure the same thing. A better taxonomy and deeper analysis is needed. Therefore, their shared analysis across 29 benchmarks could be a useful resource. The authors include human validation, taxonomy ablations, and appropriately acknowledge uncertainty in their main results.

Critical weaknesses

The taxonomy design follows Wikipedia, but these categories cover very different amounts of knowledge, but each counts equally. Several categories also overlap, making the assigned label ambiguous. Eg: The arts, music, dance, cinematography are on the same level. Similarly with Comedy, Humour as 2 separate categories. The implication can be that a narrower benchmark could look broader. Likewise, similar questions could appear to cover different things. Now that designing a taxonomy itself is definitely not an easy task, so I do acknowledge that.

The correlation between profile distance and ranking disagreement is moderate and based on only 7 dataset. The joint content–cognition profile does not significantly outperform either axis alone. There could be many factors, such as language, question format, and difficulty that can also explain ranking differences, but these factors are not controlled.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.