- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper revives the C-test with an LLM as the reference machine. Items are sequences built from world-knowledge retrievals and transformations; difficulty is the conditional Levin complexity of a natural-language description of the rule, costed by a frozen 14B model. A composer generates items, a filter removes those with a cheap rival pattern, and the reference machine certifies an item by running the description. 71 items are administered to twelve LLMs and 51 certified ones carry the difficulty analysis. Difficulty correlates with accuracy (Spearman -0.79) and the induced ability correlates with Epoch's ECI. For the strongest examinee, items above about 190 bits form what the authors call the supra-intelligence window: the 14B machine certifies them and frontier models mostly fail them.
Critical strengths
The idea is good and the problem is real: benchmarks saturate, leak, and cannot grade items nobody has solved. Defining difficulty by construction with an LLM as the reference machine is, to my knowledge, new; the paper positions itself against the original C-test and SuperARC. The experiments are complete for a first demonstration, and the result that items costed and certified on a 4-bit 14B model have a difficulty that predicts the failure of much stronger models is worth having.
Critical weaknesses
-
The supra-intelligence window is the headline result, but the main text gives the reader very little to judge it with. Table 4 gives its bounds and says the fitted curve falls below one third there. How many items are in the window, how often the frontier models actually solve them, and with what uncertainty, are all left to Appendix F, where the authors themselves write that the window "should be taken with care" and report an interval that does not exclude a success rate above one third. That belongs in Section 7. There is also an asymmetry the paper should be clearer about: certification means the reference machine can run a description it is handed, while the examinee has to find the rule. The authors treat this as the design, which is fine, but "a 14B model certifies what frontier models cannot solve" then compares two different tasks, and the window should be described accordingly.
-
What the test measures. The items are sequence completion built from retrievals and transformations, which the paper maps onto crystallised and fluid intelligence, and the hard items are mostly the transformation-heavy ones (Figure 2). The only external check is the correlation with ECI over ten models, all of which sit below the window. That supports the scale in the range where models already succeed; at the top, where the paper puts its value, nothing shows that failing these items tracks any other capability. The claim to "quantify the pace of AI progress" is ahead of the evidence.
-
Two papers missing from Section 2: Huang et al. (arxiv.org/abs/2404.09937), on a model's compression of external text predicting its benchmark scores, and Wan et al. (arxiv.org/abs/2505.15784), on LLMs as computable approximations to Solomonoff induction. The second bears on reading the reference machine as an inductive device.
Questions for authors
-
Please bring the window's item count, the frontier models' success rate on those items and its interval from Appendix F into Section 7, and add intervals for the abilities in Figure 3.
-
For the items in the window, how do the frontier models fail: a different rule that also fits the prefix, the intended rule executed wrongly, or something else?
New PDF version
We have uploaded a new version of the PDF file. It adds the two references suggested by Reviewer fNUj, discussed in Appendix L and in the extended related work (Appendix S). As before, all changes are in blue.