- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper analyzes the evaluation scores of a large number of models and argues that model capabilities cannot be summarized by a single "general intelligence," and popular assessments such as mathematics and reasoning may not necessarily represent overall intelligence.
Critical strengths
The research question is worth discussing. It examines a common assumption: whether improved math and reasoning scores truly translate to improved general abilities.
Critical weaknesses
This paper is extremely difficult to understand, and its experiments are unreliable, making its conclusions unconvincing. Firstly, the data is extremely sparse. Only about 1.8% of the original score matrix has available data, with many relationships relying on algorithmic estimation. Covering a large number of assessments does not equate to reliably estimating the relationships between them. There are data processing issues affecting the core results. The lack of a unified "higher is better" and "lower is better" metric orientation may make assessments measuring similar abilities appear far apart in the graph, weakening the argument that "similar tasks do not cluster." Finally, the conclusions exceed the scope of the evidence. The paper analyzes the correlation between scores without verifying ability transfer through training intervention; therefore, it is insufficient to conclude that a unified ability structure is unfeasible or that training around general abilities lacks promise.
Questions for authors
See weaknesses.
We thank the reviewer for their feedback. We agree that the data is very sparse (only 1.8% of the matrix is observed). Below, we explain how we changed the imputation and fixed other issues.
Metric orientation
We have fixed the metric orientation: all 27 benchmarks where lower is better are now sign-flipped so that higher always means better, and all imputations and factor analyses in this revision use the corrected data. This changes ωh by at most 0.02 in every solution. The fix mainly affects semantic labels that contain flipped benchmarks. For example, 3 of the 4 benchmarks labeled
likelihood_probeare lower-is-better, where this label now clusters (p=0.001) but did not before the fix (p=0.41). Overall, the number of labels that cluster barely changes (13 to 14 of 39 for SoftImpute on the S table), and depending on the imputer 16–49% of labels cluster, so our conclusion is unchanged.Imputation Trustworthiness
The key improvement is that imputation is now run sequentially, imputing a higher-density, smaller subset to prefill the next lower-density and bigger subset. For the 0.1 table, we first impute the 0.5 subset, use it to prefill the 0.4 subset, and continue down to 0.1.
Using this sequential imputation, we changed our evaluation strategy for real data. We score each run with a weighted R2 (wR2): the held-out R2 of each step, averaged with weights equal to the number of new columns that step adds. This method has precedent in sequential least-squares imputation [1] and denoising-autoencoder imputation [2]. Overall, our new pipeline improves the trustworthiness of our imputations, as shown in both simulated and real data performance below.
Simulated MNAR Data Performance
Please refer to our reply to reviewer rXDF, where we describe our simulation imputation and performance.
Real Data Performance
Below is how our imputation methods perform on our real data. As shown, we achieved considerably better evaluation than the non-sequential step in our original draft. This allows us to increase the R2 gate to 0.4 (the table only shows results with wR2≥0.4).
Overclaiming Issues
We agree that observational evidence does not constitute experimental evidence. Due to poor wording, we may not have communicated this clearly in the original draft. We have restricted Sections 5.1–5.3 to descriptive claims, and statements about the effects of targeted training are now framed as hypotheses. A more conclusive claim would require empirical training experiments. A direct test would train models on one domain and measure the change on other domains, as in human cognitive-training studies.
The closest experimental evidence we know of is Meng et al. (2026), cited in Section 5, who find that input-similarity measures identify the best transfer source for at most 2 of 8 target tasks, and that semantic content does not predict transfer well, suggesting that transfer does not follow semantic similarity. This finding aligns with our results, but a more thorough experiment would be needed to test this directly.
Paper Revision
Since replies are limited in length, we are revising the manuscript to include all of these changes and will upload it before the discussion period ends, so that reviewers can take these into account.
References
[1] X. Zhang, X. Song, H. Wang, and H. Zhang. Sequential local least squares imputation estimating missing value of microarray data. Computers in Biology and Medicine, 38(10):1112–1120, 2008.
[2] Q. Ma, W.-C. Lee, T.-Y. Fu, Y. Gu, and G. Yu. MIDIA: Exploring denoising autoencoders for missing data imputation. Data Mining and Knowledge Discovery, 34(6):1859–1897, 2020.