acceptodds
Under review as a conference paper at ICLR 2027

One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AI

Abstract

Frontier-model leaderboards now rank systems on economic benchmarks, and those rankings inform what organisations buy and what regulators scrutinise. Whether such benchmarks measure a capability distinct from general test-taking is a question of construct validity that a structural test and a predictive test can answer in opposite ways. We show that they do on a hash-pinned snapshot of a frontier leaderboard with 421 model configurations across twelve benchmarks, four of them economic, of which 103 configurations carry all three sparsely scored economic benchmarks and 96 carry all twelve; four hypotheses and their thresholds were fixed before analysis, and every deviation from the plan is reported. The first factor of a three-factor extraction carries 74.5% of common variance and tracks release date (R^2 = 0.505), and date adjustment lowers its share by 14.9 points. Under the dimensionality rule fixed in advance the economic benchmarks form no factor of their own. A leave-one-benchmark-out test with factors re-estimated inside every fold nevertheless finds that a multi-factor representation predicts held-out economic scores better than a single general index (pooled Delta-MSE 0.037, 95% bootstrap interval [0.019, 0.055]) under the linear learners that fit best, an advantage that reverses for tree learners. Under the linear learners the same representation also predicts the eight other benchmarks better, so the battery carries predictive structure that one index misses and the economic benchmarks share it without forming a distinct factor. Construct validity should therefore be assessed by predictive tests alongside structural ones. We give a two-test protocol for benchmark builders and release the pinned data, the analysis plan and the code.

Primary area:
datasets and benchmarks
Keywords:
Construct Validity, Benchmark Validity, Predictive Validity, Factor Analysis, Leaderboard Audit, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_h6UDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

LLM benchmark scores obviously strongly correlate with release date. This paper conducts a very careful statistical analysis of LLM benchmark scores, with the main focus of isolating whether economic benchmarks like GDPVal and Terminal-Bench pick up on additional "economic capability" signal beyond overall capabilities. They answer this both with a factor analysis and with predictive validity tests.

Critical strengths

Many robustness checks are conducted.

Critical weaknesses
  1. The research question is not interesting. It is already well-accepted in the community that benchmark scores improve with release date, and it is also obvious that there is secondary signal in domain-specific benchmarks. Moreover there are no methodological contributions or interesting experiments. The paper just takes existing LLM benchmark data and applies a wide battery of known techniques, mostly things related to factor analysis and leave-one-out prediction power, to answer its questions.

  2. The writing of the paper is obscure and could use some more careful human attention.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer h6UD

Thank you for the review. We agree that benchmark scores rising with release date is expected, and that domain-specific benchmarks are likely to carry secondary signal. The paper does not claim either as a discovery, and it introduces no new estimator. Its contribution is narrower, an audit with thresholds fixed in advance. It measures how large these effects are on a current frontier leaderboard, finds that the release-date reduction misses its pre-specified threshold on the primary grid, and shows that whether economic benchmarks count as distinct depends on which test and which retention rule a reader applies. Leaderboards now publish economic columns as separate rankings, so we think the size of these effects is worth stating with intervals, though we accept that this is a modest contribution. On the writing, several reviewers found the paper hard to follow, and we will shorten the abstract, add background on the factor model, and cut methodological detail from the main text.

AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for this response and with running analyses we specified, as set out in our shared response.

Reviewer_XzuPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at leaderboard scores as a psychometric problem. By looking at the scores, it asks whether economic benchmarks measure something distinct from general test-taking. First, they perform a factor analysis to test whether the economic benchmarks form a factor on their own (structural test). Then, they ask if a multi-factor representation of the other benchmarks predicts a held-out economic score (predictive test). Overall, they find one dominant factor carrying 74,5% of the common variance that tracks release date. Using their pre-specified parallel analysis rule, the economic benchmarks do not form a distinct factor. Yet a three-factor representation beats the single index at predicting held-out economic scores under ridge and elastic net, and the advantage reverses for tree learners. From this evidence, the authors conclude that the structural and predictive tests disagree and argue that claims should say what they test on. They also propose a protocol for benchmark builders.

Critical strengths
  • Whether economic benchmarks add anything beyond a general index is an interesting and important question.
  • Clear methodology and transparent experimental design support reproducibility.
Critical weaknesses
  • The predictive gain comes from a three-factor model the structural analysis rejected, and the same gain appears for non-economic benchmarks. It seems to say little about economic benchmarks specifically.
  • Three factors are compared against a single index.
  • The results only hold for ridge and elastic net, while the effect is reversed for tree learners. Also, the tree-learner differences (−0.23 and −0.18) are far larger than the ridge gain (+0.037). While I appreciate the transparency, the paper does not adequately explain this observation.
  • The proposed two-test protocol is never really tested/validated.
  • The presentation of the paper is often hard to follow. For example, the abstract is already full of numbers and methodological details. With 96 configurations (only 89 distinct base models), parallel analysis has little power.
Questions for authors
  • Could you elaborate more on the observed reversal effect of tree-based learners?
  • What happens if you withhold the entire economic block for the predictive tests at once?
  • Why does the first factor only rung predict very poorly, but the mean index gets R^2 0.77?
  • A strong positive manifold appears with one capability and with several capabilities that rise together. How does your paper distinguish these?
1 comment
AuthorsOpenReview ↗

Response to Reviewer XzuP

Thank you for the updated review, which raises the same core issues as BTPG and P4zQ. We answer them in comments (A) to (C) of our shared response and add specifics here.

W1 (the gain comes from the three-factor model and is not economic-specific). We agree. The gain appears on the non-economic targets too, and comment (B) restates the finding accordingly. The block-withheld test you ask about in Q2 is the direct check.

W2 (three factors against one index). The comparison asks whether any representation beyond one index adds out-of-sample information, so the extra parameters are the object of the test. Two features guard against an easy win. Ridge selects its penalty by nested cross-validation, and the gain holds at every k from 2 to 5 (check C7). The tree learners also show that more inputs can lose.

W3 and Q1 (tree learners). Comment (C) gives the train and test errors behind the reversal. The tree learners fit the factor inputs more closely in training and predict worse out of sample, the pattern of overfitting with about 77 training rows per fold. With whole configurations resampled, both reversals have intervals that exclude zero, so they are not noise.

W4 (the protocol is not validated). We agree. The two-test protocol is a proposal drawn from this one battery, and we will present it as such, or move it to the discussion.

W5 (presentation and power). We will shorten the abstract and move most numbers into the body. On power, parallel analysis at n = 96 requires a second eigenvalue of 1.57 to retain a factor, against 0.91 observed. Small samples make such rules conservative, which is part of why the structural and predictive tests can diverge, and the revised limitations will say so.

Q2 (economic block withheld). With all four economic benchmarks withheld, three fixed factors still beat the mean index, at +0.024 [+0.001, +0.045] with Terminal-Bench Hard and +0.030 [+0.005, +0.054] without it. Under the pre-specified retention rule, which keeps one factor, the gain is +0.019 [+0.009, +0.030] with Terminal-Bench Hard and −0.021 [−0.040, −0.004] without it. The evidence that non-economic benchmarks carry economic-specific structure is therefore weak and depends on these choices. Our reply to Reviewer BTPG (Q3) gives the details.

Q3 (rung iii). Rung (iii) is mislabelled. It is a partialled oblique factor score, not a single general factor, which is why it predicts poorly. Our reply to Reviewer BTPG (Q4) gives the derivation and the fix.

Q4 (positive manifold). The paper cannot distinguish these cases from scores across models, and comment (A) states how we will scope the claims.

AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for this response and with running analyses we specified, as set out in our shared response.

Reviewer_P4zQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper takes a psychometric approach to evaluating AI benchmarks by asking whether "economic" benchmarks measure a new capability or if general model capability is the main factor influencing benchmark performance. They analyze a snapshot of benchmark data from the Artificial Analysis leaderboard, with 96 AI models on 12 benchmarks (4 of them economic and 8 academic). They found that the 4 economic benchmarks do not form their own factor distinct from general model capabilities. They found that the the model's score on one economic benchmark is slightly predictive of performance on the other 3, but only for linear models.

Critical strengths

S1. The authors fix (most of) their analysis plan upfront and report any deviations from the plan.

S2. The question itself is important, given how much weight leaderboards and benchmarks carry.

Critical weaknesses

My main critique is that the results are modest overall with results that are overstated in the paper, so I'm skeptical that there is any novel finding here that generalizes.

W1. The predictive result depends on the learner, and the learner was chosen after the fact. The predictive finding only held if the predictive test was linear, whereas random forest models had the opposite effect, with this reversal being about 5x higher than the gain in the linear case. Except for the ridge learner, the tests have no confidence intervals reported.

W2. The distinction between "economic" benchmarks and "academic" benchmarks seems arbitrary. For example, terminal bench is in the "economic" category but terminal bench hard is in "academic". It's unclear why the 4 economic tests are considered "economic."

W3. Clarity in writing. The paper assumes background knowledge in psychometrics (eg factors, etc) that most ICLR readers won't have. You should have included this necessary background.

W4. Many of the results seem like analysis artifacts. For example in the structural test, under an alternative test reported in the appendix (BIC), multiple factors are retained, which resolves the discrepancy that's the central claim of the paper.

Questions for authors

Can you compute confidence intervals for the predictive tests that didn't include them?

How did you select 4 of the 12 benchmarks as "economic" in nature? What is the justification that these would potentially require different capabilities than the academic benchmarks?

Which 96 models (using which settings) did you use for the analysis? I didn't find this in the main text.

How do you adjust for the selection bias that Artificial Analysis chooses which benchmarks to run, and your dataset is based on the 96 models that ran the 12 benchmarks in your set?

1 comment
AuthorsOpenReview ↗

Response to Reviewer P4zQ

Thank you for the specific questions. We take them in order.

W1 and Q1 (learner choice and intervals). The plan fixed no learner for the H4 contrast, and Appendix J lists this among the departures. The paper reports ridge as the lowest-error learner and gives the contrast under all four learners in check C8. Comment (C) of our shared response explains the tree-learner reversal with train and test errors, which show the tree learners overfitting the factor inputs at this sample size. Comment (C) now gives intervals for all four learners, each resampling whole configurations, and every one excludes zero. We will state H4 as holding under the linear learners everywhere it appears, as the abstract already does.

W2 and Q2 (the economic grouping). The four economic benchmarks were named in hypothesis H3 before analysis, and Appendix F gives the criterion. They score multi-step, tool-using or human-judged completion of work-like tasks, while the others score single-response accuracy. We agree the boundary is weakest at Terminal-Bench. Version 2.1 sits in the economic block as agentic task completion, and Terminal-Bench Hard sits with scientific coding, a placement a reader could reasonably reverse. The block-withheld analysis is run with and without Terminal-Bench Hard to show how much the result depends on that choice. With the economic block withheld, the gain under three fixed factors is +0.024 [+0.001, +0.045] with Terminal-Bench Hard and +0.030 [+0.005, +0.054] without it. Under parallel analysis it turns negative without Terminal-Bench Hard, at −0.021 [−0.040, −0.004]. Our reply to Reviewer BTPG (Q3) gives all four cells.

W3 (background). We agree. We will add a short primer on the factor model, loadings, common variance, rotation and parallel analysis, with one worked example.

W4 (BIC). We agree, and this sharpens the claim. The BIC prefers more than one factor on this matrix. Appendix D reports k = 4, and a fresh refit during the discussion gives k = 3, so we will report the computation in full. At k = 3 the economic block separates on date-adjusted scores (Section 5.3), so a BIC-based retention rule would likely reverse the H3 verdict, and we will report the loading pattern at whichever k the verified computation selects. On the predictive side, check C7 shows the gain at every k from 2 to 5 with intervals excluding zero (+0.058 at k = 4). Under the BIC, then, the two tests agree that more than one dimension carries information, and the disagreement lies between retention rules as much as between test types. Comment (B) restates the finding at that size.

Q3 (which 96 models). They come from 21 developers and reduce to 89 base models. The full list is in a separate comment in this thread, and the next version adds it as an appendix table.

Q4 (selection by the leaderboard operator). We cannot correct this selection, since the operator decides which configurations run the sparse economic suites and no record of that decision exists. The paper restricts every claim to the 96 configurations on which the economic benchmarks were run. In the next version we will add a descriptive comparison of these 96 with the other 325 retained configurations on release date, developer, reasoning mode and scores on the dense benchmarks, so readers can see who is in the sample.

AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for this response and with running analyses we specified, as set out in our shared response.

Reviewer_BTPGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper examines whether 12 AI benchmarks, including 4 economic benchmarks, measure distinct capabilities or primarily reflect general model performance. Using factor analysis, the authors identify one dominant factor and find no distinct economic factor after controlling for model release date. However, they find that using multiple factors to predict a model's score on a held-out benchmark, based on its scores on the other benchmarks, performs better than using a single general performance index. The authors interpret this as a disagreement between structural and predictive tests of benchmark validity and argue that both approaches are needed to assess whether benchmarks capture distinct capabilities.

Critical strengths

Main Strengths

  1. Relevant topic: What do benchmarks capture? Are they informative on various dimensions or are they measuring the same capability?
  2. Transparency: The paper is clear about methodology, failed hypothesis criteria, exploratory analyses, reproducibility, and limitations
  3. Appropriate robustness analysis: Deduplication, logit transformed benchmark scores, and other checks are all helpful.
Critical weaknesses

Main Weaknesses

  1. I am concerned that the authors cannot distinguish between (i) benchmarks measuring the same underlying capability and (ii) benchmarks measuring distinct capabilities that are highly correlated across models. For example, models may consistently perform well on some benchmarks and poorly on others, while their relative rankings remain similar. Both scenarios could produce one dominant factor, even if the benchmarks assess fundamentally different capabilities. While the authors aim to answer “One Capability or Many?”, their analysis primarily characterizes correlations in benchmark performance rather than identifying the underlying capabilities. I therefore find the conclusions regarding construct validity too strong, which substantially limits the paper’s contribution.

  2. I am not yet convinced that the structural and predictive tests provide conflicting evidence about benchmark validity. H3 examines whether economic benchmarks form a distinct factor after controlling for release date, whereas H4 tests whether multiple factors predict held-out benchmark scores better than a simple average. These findings are not necessarily contradictory, as better prediction does not necessarily imply a distinct economic capability. Moreover, H3 adjusts for release date while H4 does not (i.e., the variation analyzed differs). Overall, I question whether the results establish a meaningful disagreement (which the authors treat as a core finding).

Questions for authors

Questions for Authors

  1. Can the authors address my concerns under “main weaknesses” above?
  2. Could the authors report the models/configurations that entered the analysis?
  3. The predictive analysis uses other economic benchmarks to predict each held-out economic benchmark. While the authors acknowledge this limitation, I think predicting economic benchmark scores using only non-economic benchmarks would be particularly informative given the paper's focus on the distinctiveness of economic capabilities. I would encourage the authors to include this analysis.
  4. Why does the first-factor predictor performs so poorly despite it accounts for 74.5% of common variance (RMSE 0.950 versus 0.474 for the simple average)?
1 comment
AuthorsOpenReview ↗

Response to Reviewer BTPG

Thank you for a close reading and for questions we can answer with numbers.

W1 (one capability or several correlated ones). We agree with the point, and comment (A) of our shared response sets out how we will rewrite the claims. Our analysis characterises the covariance of scores across models. It cannot tell one capability from several that move together, and the revised paper will say so in the introduction.

W2 (whether the tests conflict, and the release-date mismatch). You are right that H3 is tested on date-adjusted scores while H4 uses raw scores, so the two tests analyse different variation. We reran H4 with every predictor and the target residualised on release date inside each training fold, applying the training-fold coefficients to the test fold. Under ridge the pooled ΔMSE is +0.033 with a 95% interval of [+0.013, +0.051], against +0.037 on raw scores, so the pooled gain survives date adjustment. Only two of the four targets exclude zero individually (Terminal-Bench v2.1 and τ³-Banking), so the per-target majority that H4 also requires is not met on date-adjusted scores, and we will report it that way. On whether the results conflict, we accept the narrower reading in comment (B). Better prediction does not imply a distinct economic capability, and the paper should claim only that a retention rule and an out-of-sample test answer differently about the same battery.

Q2 (configurations). The 96 configurations come from 21 developers and reduce to 89 base models under the deduplication rule of Appendix L. The full list is in a separate comment in this thread, and the next version adds it as an appendix table with developer, release date and the leaderboard's own variant label.

Q3 (non-economic predictors only). We agree this is the more informative test for distinctiveness. We ran it under ridge, with the eight non-economic benchmarks as the only predictors, once with Terminal-Bench Hard in the pool and once without it. With the three-factor representation fixed, the gain shrinks from +0.037 to +0.024 [+0.001, +0.045] with Terminal-Bench Hard and is +0.030 [+0.005, +0.054] without it. Under the pre-specified retention rule, parallel analysis selects one factor in every fold, and that representation gives +0.019 [+0.009, +0.030] with Terminal-Bench Hard and −0.021 [−0.040, −0.004] without it. Per target, only τ³-Banking excludes zero under three fixed factors. The non-economic benchmarks therefore predict economic scores beyond their mean only weakly, and whether they do at all depends on the retention rule and on Terminal-Bench Hard, the non-economic benchmark closest to the economic block. We will report all four cells.

Q4 (why rung iii predicts poorly). Rung (iii) is mislabelled in the paper. It takes column one of the oblimin-rotated three-factor solution, scored with the weights R⁻¹Λ. For correlated oblique factors (factor correlations 0.67 and 0.72), these pattern-form scores equal Φ⁻¹ times the regression scores, so each column is partialled against the other two. Column one therefore carries what is specific to that factor once the variance it shares with the others is removed. That shared variance is what the mean index captures, so this feature discards most of the predictive signal by construction. The 74.5% figure describes the unrotated loadings, a different quantity. We will define the rung precisely in Section 4.2 and add an unrotated one-factor score as the single-factor predictor it was meant to be.

AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for this response and with running analyses we specified, as set out in our shared response.

Comments on the paper

3 comments
Area_Chair_sJx7OpenReview ↗

Halfway through the discussion period

Dear reviewers,

Thank you for your continued participation in the review process. The authors have responded to the reviews. When you get a chance, please acknowledge their response and, where appropriate, please respond to their comments.

Thanks, AC

AuthorsOpenReview ↗

Response to all reviewers: shared points and one correction

We thank the four reviewers and the Area Chair. Several points recur across the reviews, so we answer them once here and refer to them from the individual replies.

(A) What the analysis can and cannot show. Reviewers BTPG and XzuP note that a dominant first factor arises both when one capability drives every benchmark and when several distinct capabilities rise together across models. We agree, and scores across models cannot separate these two cases. Our analyses characterise the covariance of benchmark scores across model configurations, and every use of "capability" in the paper should be read as a dimension of that covariance. We will rewrite the introduction and discussion to say this plainly, and to drop wording that implies we identify the capabilities underneath the scores. Separating the two cases needs item-level data, interventions on models, or an external criterion, and we will name these as the designs that could.

(B) Whether the two tests disagree. BTPG, P4zQ and XzuP each argue that the disagreement depends on analytic choices. We accept the narrower reading this implies. The structural verdict depends on the retention rule, since parallel analysis keeps one factor while the BIC prefers more than one. The predictive gain is also shared beyond the economic block, averaging 0.048 on the eight non-economic targets under ridge (positive on seven) against 0.037 on the economic four. What the data support is narrower than a conflict between validity paradigms. A pre-specified retention rule and an out-of-sample test return different answers about the same battery, and the economic benchmarks share structure beyond one index without forming a factor of their own. We will state the finding at that size.

(C) Learner dependence. Under the plan, no learner was fixed for the H4 contrast, and the paper reports the result under all four learners (check C8). Ridge and elastic net favour the multi-factor representation on all four economic targets. Random forest and gradient boosting reverse it. The tree learners fit worse on both rungs (rung ii test RMSE 0.489 and 0.525, against 0.474 for ridge) and degrade with factor inputs. Random-forest train RMSE falls from 0.356 to 0.308 between rungs ii and iv while test RMSE rises from 0.489 to 0.684. Gradient boosting moves from 0.213 to 0.172 in train and from 0.525 to 0.675 in test. With about 77 training rows per fold, this is the signature of overfitting the richer input, so we read the reversal as a property of the learners at this sample size. With whole configurations resampled, the pooled economic ΔMSE is +0.037 for ridge and +0.036 for elastic net, against −0.231 for random forest and −0.184 for gradient boosting, and every interval excludes zero, at [+0.016, +0.056], [+0.016, +0.054], [−0.324, −0.151] and [−0.267, −0.107] respectively. The direction of the result therefore depends on the learner family, and we will state H4 as holding under the linear learners wherever it appears.

(D) A correction found during the discussion. Appendix D states that the H4 bootstrap resamples models. The archived code instead resamples the 384 pooled (target, configuration) rows, which treats one configuration's four economic targets as independent draws and can narrow the interval. Resampling the 96 configurations, with all four economic targets of each carried together, gives a pooled ΔMSE of +0.037 with a 95% interval of [+0.016, +0.056], against the reported [+0.019, +0.055]. The interval still excludes zero, three of the four targets still exclude it individually, and τ²-Bench still does not, at [−0.012, +0.082]. Replaying the archived row-level scheme gives an upper bound of 0.056, not the printed 0.055, which is Monte Carlo error at B = 2000. We will report the configuration-level interval as primary, correct Appendix D, and add the resampling unit to the departures in Appendix J.

(E) Analyses added for this discussion. Three further analyses answer specific questions and are reported in the replies to the reviewers who asked for them. They test H4 with the economic block withheld from the predictors, test H4 on date-adjusted scores, and list the 96 configurations.

We will not upload a revised PDF in this window. Every change listed above will enter the next version, and the released code recomputes each new number.

AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for these responses and with running analyses we specified on the released code and pinned data. We reviewed every sentence, each states our position, every number was checked against the archived outputs, and we take full responsibility for the content.

AuthorsOpenReview ↗

The 96 complete-case configurations (requested by BTPG and P4zQ)

We list the 96 configurations that carry all twelve benchmarks, grouped by developer and ordered by release date within each developer. Parenthetical variant labels are the leaderboard's own. They come from 21 developers and reduce to 89 base models under the deduplication rule of Appendix L.

OpenAI (16): o3-mini (high); GPT-4.1 mini; GPT-4.1 nano; gpt-oss-120b (high); gpt-oss-20b (high); GPT-5 (high); GPT-5 mini (high); GPT-5.1 (high); GPT-5.4 (xhigh); GPT-5.4 mini (xhigh); GPT-5.4 nano (xhigh); GPT-5.5 (Non-reasoning); GPT-5.5 (high); GPT-5.5 (low); GPT-5.5 (medium); GPT-5.5 (xhigh).

Alibaba (15): Qwen3 14B (Reasoning); Qwen3 32B (Reasoning); Qwen3 8B (Reasoning); Qwen3 235B A22B 2507 (Reasoning); Qwen3 30B A3B 2507 (Reasoning); Qwen3 Next 80B A3B (Reasoning); Qwen3.5 397B A17B (Reasoning); Qwen3.5 122B A10B (Reasoning); Qwen3.5 9B (Reasoning); Qwen3.6 Plus; Qwen3.6 35B A3B (Reasoning); Qwen3.6 27B (Non-reasoning); Qwen3.6 27B (Reasoning); Qwen3.7 Max; Qwen3.7 Plus.

Mistral (13): Mistral Small 3.1; Mistral Small 3.2; Mistral Medium 3.1; Magistral Small 1.2; Magistral Medium 1.2; Ministral 3 14B; Ministral 3 3B; Ministral 3 8B; Mistral Large 3; Devstral 2; Devstral Small 2; Mistral Small 4 (Reasoning); Mistral Medium 3.5.

Anthropic (8): Claude 3.5 Haiku; Claude 4 Sonnet (Reasoning); Claude 4.5 Sonnet (Reasoning); Claude 4.5 Haiku (Reasoning); Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort); Claude Opus 4.7 (Adaptive Reasoning, Max Effort); Claude Opus 4.8 (Adaptive Reasoning, Max Effort); Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback).

Google (8): Gemma 3 12B Instruct; Gemma 3 27B Instruct; Gemini 2.5 Pro; Gemini 3.1 Pro Preview; Gemini 3.1 Flash-Lite; Gemma 4 26B A4B (Reasoning); Gemma 4 31B (Reasoning); Gemini 3.5 Flash (high).

DeepSeek (5): DeepSeek V3 (Dec '24); DeepSeek R1 (Jan '25); DeepSeek V3 0324; DeepSeek V4 Flash (Reasoning, Max Effort); DeepSeek V4 Pro (Reasoning, Max Effort).

Meta (5): Llama 3.1 Instruct 8B; Llama 3.3 Instruct 70B; Llama 4 Maverick; Llama 4 Scout; Muse Spark.

Amazon (4): Nova 2.0 Lite (high); Nova 2.0 Pro Preview (Non-reasoning); Nova 2.0 Pro Preview (low); Nova 2.0 Pro Preview (medium).

NVIDIA (3): NVIDIA Nemotron 3 Nano 30B A3B (Reasoning); NVIDIA Nemotron 3 Super 120B A12B (Reasoning); Nemotron 3 Ultra 550B A55B (Reasoning).

Z AI (3): GLM-4.7 (Reasoning); GLM-5.1 (Reasoning); GLM-5.2 (max).

IBM (2): Granite 4.1 30B; Granite 4.1 3B.

Kimi (2): Kimi K2.6; Kimi K2.7 Code.

MiniMax (2): MiniMax-M2.7; MiniMax-M3.

xAI (2): Grok 4.3 (Non-reasoning); Grok 4.3 (high).

Xiaomi (2): MiMo-V2-Flash (Non-reasoning); MiMo-V2.5-Pro.

Cohere (1): Command A+.

InclusionAI (1): Ring-2.6-1T.

MBZUAI Institute of Foundation Models (1): K2 Think V2.

OpenBMB (1): MiniCPM-V 4.6 1.3B.

StepFun (1): Step 3.7 Flash.

Upstage (1): Solar Pro 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.