- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper derives some indicators of benchmarks using cultural domains inferred from wikipedia categories and a cognitive depth derived from Bloom's taxonomy of educational objectives.
Critical strengths
It's important to extract profiles of what benchmarks measure. In terms of culture, this may be less known and relevant.
The language asymmetries (page 6) may be an interesting finding, but this is buried in an overly complicated paper for what it brings.
Critical weaknesses
Poor connection with closely related work: Cognitive profile of benchmarks has been happening in the past few years. The authors use a terminology that is alien to the standard terminology of AI evaluation (partly borrowed from cognitive science, and psychometrics), about constructs, latent factors, cognitive capabilities. It ignores existing profiling methods, https://arxiv.org/pdf/2510.01232, most notably https://www.nature.com/articles/s41586-026-10303-2. This one is based more on knowledge and culture (but the above paper also covers the Bloom taxonomy in the appendix) than core capabilities, but it also tries to identify what each benchmark measures (in dimensions, and in levels), as in eq. 1. The related work is a concatenation of references without insight. On page 9 there's more in green than black! There's a bit of everything, put together, without much structure.
Lack of justification in cognitive science or cultural studies. There's this use of Bloom's taxonomy borrowed from education, without further justification. The definitions in section 3 are not given any justification. It also relies too soon to appendices, so it's hard to follow or see what each of these concepts mean, and how they relate to concepts in cognitive science or AI evaluation. The references are mostly about statistical methods or corrections, but not really about the concepts.
Motivation and usefulness: The empirical results of predicting rank disagreement do not clarify what this would be useful for and what problem it solves (that hasn't been addressed before). The RQ are expressed in terms of the own definitions, and are inward-looking rather than answering questions that matter. We see benchmarks in a map of two dimensions, but this could have been done with other much simpler approaches. There's no comparison with any other method, not only in possible results but in advantages or disadvantages.
*** Other comments that may be useful, I didn't know where to put them *** Avoid acronyms like "GenAI". Despite focusing on "cultural" benchmarks, the title doesn't reflect it.
Questions for authors
- Could you use some other levels beyond Bloom's taxonomy. Why is it good for cultural dimensions?
- Why did you choose the current benchmarks?
- Is the choice of language arbitrary or there are confounders in benhmarks/languages?