- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes how MLLMs organize a set of cognitive tasks, comparing the resulting taxonomy against that produced by human scientists. An analysis of the use of these taxonomies to guide model performance suggested that using the AI-derived taxonomies results in a small performance boost.
Critical strengths
This paper is a creative synthesis of ideas from cognitive science and AI, seeing how allowing the AI to act as a scientist might improve on the way that human scientists organize a domain. The comparison of representations alone would have been interesting but not necessarily very exciting, so the introduction of interventions based on the taxonomies and the demonstration that the AI systems produce a more useful taxonomy for this purpose is a nice way of showing why these ideas matter. The followup analyses that explore variants on the systems are nice in fleshing out these results.
Critical weaknesses
The paper is generally clearly presented, but I think the key argument about why anyone should care about these taxonomies needs to come much sooner in the paper. The idea that these taxonomies can be useful for designing effective interventions is a nice one and the investigation needs to be strongly motivated in those terms sooner.
The quantitative analysis of the results using measures such as Jaccard and ARI makes sense, but the paper is lacking statistical significance tests for the comparison of the results. At present we can see that there are overall patterns of performance and increments in accuracy that result from using the new taxonomy, but these results are difficult to interpret in the absence of the relevant statistical significance tests. This is a particular issue since the differences are quite small.
Questions for authors
None
2 comments
Response to Reviewer 8mi5 (2/2)
[W2] Statistical significance
Thank you for suggesting statistical significance tests. We report three runs of the main comparison in Table 3 for Qwen3-VL, and paired significance tests for the single-run comparisons in Table 3.
-
Model-derived clusters outperform human categories in three runs. Due to the time limit, we only finish repeating the comparison between model-derived clusters and human cognitive categories on the test set using Qwen3-VL. Across the three runs, model-derived clusters outperform human cognitive categories, as shown below.
Model Human categories Model clusters p-value Qwen3-VL 71.80 ± 0.48 75.23 ± 0.10 < 0.01 -
Model-derived clusters significantly outperform human cognitive categories on the test set. Here we report paired significance tests comparing model-cluster and human-category intervention performance in Table 3, together with 95% confidence intervals. As shown in the table below, masks based on model-derived clusters significantly outperform those based on human cognitive categories on four of the five models.
Model Model clusters - Human categories (pp), 95% CI p-value Qwen3-VL +3.08 [1.18, 5.02] < 0.01 Qwen3.5 +2.39 [0.99, 3.77] < 0.01 Gemma 3 +1.98 [0.35, 3.62] < 0.05 Gemma 4 +1.55 [−0.16, 3.22] > 0.05 GLM-4.6V +2.38 [0.83, 3.93] < 0.01 -
Our method significantly outperforms recent L2S on two additional models. We extend the comparison with related work in Table 4 to Qwen3.5 and GLM-4.6V and use paired significance tests to assess the differences in test accuracy. As shown in the table below, our method significantly outperforms recent L2S on both additional models.
Model Ours − L2S (pp), 95% CI p-value Qwen3.5 +5.15 [3.61, 6.73] < 0.01 GLM-4.6V +2.70 [1.05, 4.37] < 0.01
We are also extending the three-run experiments and comparisons with prior methods to more models.
Response to Reviewer 8mi5 (1/2)
We sincerely appreciate your constructive comments. We detail our response below point by point. Please kindly let us know whether you have any further concerns.
[W1] Motivation and significance of the taxonomies
Thank you for this suggestion. We will revise the opening of the Introduction to explain why these taxonomies matter.
Cluster-level evaluation can reveal where models struggle and guide targeted interventions.
Cluster-level performance can identify which abilities a model struggles with at different granularities and guide test-time interventions. For example, our cluster-based intervention method outperforms human category-based interventions, and the recent intervention method L2S by 2.70-5.15 percentage points, as shown in Tables 3 and the response to [W2], respectively.
CoreCognition organizes tasks by human cognitive categories, which are not fully aligned with model abilities. Model-derived clusters could guide the design of benchmarks whose evaluation dimensions better reflect different model abilities, revealing differences in model performance that scores based on human cognitive categories may obscure.