- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses how to assess LLM capabilities after benchmarks saturate. The authors rely on how an LLM reached the solution instead of simply whether it reached there or not, which attempts to extract more knowledge of model capabilities beyond the final solution. Their method achieves 5.3x less prediction error compared to constant prediction.
Critical strengths
The general capability of assessing LLM performance using an LLM judge from a previous generation is interesting. Additionally, the paper shows that solved tasks still contain capability predictive information beyond accuracy. The evaluation covers a large spectrum of models and the authors provide a lot of analysis including pointing out the failures which shows a comprehensive analysis.
Critical weaknesses
Saturation is mostly solved through the numerous benchmarks that perform pairwise LLM evaluation models (i.e. arena benchmarks), even in a future where models exponentially exceed human capabilities in all metrics, arena benchmarks that have verifiable sufficiently hard rewards that are not based on human feedback or any subjective evaluation exist. Many tasks are easy create and validate than to solve thus humans could still evaluate models even in that future (e.g. generate lean proof of 10D navier-stokes or decide if a given finite set of turing programs halt under a fixed given budget). Additionally, some saturated benchmarks still can achieve high scores in Benchmark Agreement Testing which indicates that the baseline in the paper could be considered weak.
I would like to see more results justifying the prediction power of this method. Figure 3 relies on IRT (which has its own assumptions) for the target prediction. However, the authors could compare predictions against average capabilities of the models on unsaturated benchmarks. If only rank is considered it's possible to compare the predicted ordering of the target models against the ordering obtained by averaging abilities. I would generally want to see more evaluations. A current assumption of the protocol is that reliable estimates are available for the previous semester’s models to not be saturated yet (for the calibration ).
Another major concern is that evaluation on chain-of-thought is an AI safety risk as it incentivises obfuscation and puts pressure on LLMs to hide true intentions just to maximize an LLM judges evaluation.
Questions for authors
How sensitive are the results to the use of IRT prediction targets? Could the authors evaluate rank correlations and multiple other baselines to justify the protocol with more rigorous evaluation.
Is it possible that the LLM judge recognizes the family of the model from its transcript and predicts that the capabilities of next semester’s LLM is equal to the previous generation family plus a constant. For example, what would a baseline that uses model release dates and family information like historical capabilities and other possible useful historical info.
7 comments
> Regarding potential additional baseline protocols
We have expanded our comparisons to incorporate new baseline measures in Appendix E: Comparisons against non-transcript based baselines.
First, we have added an accuracy baseline using Benchmark Agreement Testing (BAT) on all four domains in the main semesters (2025-II, 2026-I). For each domain, we take accuracy on the easiest benchmark (the benchmark with highest average accuracy: ARC-Challenge, MMLU-Pro maths, AgentBench-OS, InterCode), and test how well accuracy rankings agree with the hardest benchmark of the same domain (HLE-Science, HLE-Math*, SWE-bench mini, CyBench; Figure 35).
*We exclude APEX 2025, which only has 12 items on which 13/24 models score 0%.
As expected, accuracy-based ranking retains predictive signal when the input benchmark has not saturated. In Coding and Cybersecurity, where the easiest benchmarks still show substantial variation in accuracy, rank correlations with harder benchmarks are 0.30 and 0.45 among target models. However, in Science and Mathematics, where the easiest benchmarks are close to saturation as in our defined problem setting, these correlations are indistinguishable from zero (0.00 and −0.09). The table below compares BAT with pairwise judging.
| Domain (easiest → hardest benchmark) | Accuracy on easiest benchmark, mean (range) | BAT (accuracy), τ [95% CI] | Pairwise judging, τ [95% CI] | Difference [95% CI] |
|---|---|---|---|---|
| Science (ARC-Challenge → HLE Science) | 98% (92–100) | 0.00 [−0.50, 0.50] | 0.12 [−0.41, 0.66] | 0.12 [−0.52, 0.77] |
| Maths (MMLU-Pro → HLE Math) | 94% (84–98) | −0.09 [−0.56, 0.38] | 0.38 [−0.12, 0.87] | 0.46 [0.15, 0.78] |
| Coding (AgentBench-OS → SWE-bench mini) | 79% (40–95) | 0.30 [−0.10, 0.71] | 0.39 [−0.10, 0.88] | 0.09 [−0.29, 0.47] |
| Cyber (InterCode → CyBench) | 86% (50–92) | 0.45 [0.12, 0.78] | 0.50 [0.21, 0.79] | 0.05 [−0.39, 0.50] |
| All domains | 0.17 [−0.07, 0.40] | 0.35 [0.04, 0.65] | 0.18 [0.01, 0.35] |
Further, BAT performance drops considerably as we restrict the item pool to simulate saturation as in our problem setting. We repeat the above comparison keeping only items on which every model exceeded a minimum X% accuracy, sweeping X from 70% (approaching saturation) to 100% (fully saturated). To keep the number of items constant across these levels, we resample (200 draws) N items at each level, where N is the number of items available at 100% saturation. The ‘All items’ column therefore differs slightly from the table above. BAT's rank agreement drops consistently (until definitionally reaching τ=0 at full saturation), whereas pairwise judging remains reliable at τ=0.31 -- 0.35 (table below; Figure 36). The same pattern holds for predicting IRT-derived θ* in the setting of our main analysis, where accuracy-based prediction consistently gives higher squared error than pairwise judging (Figure 37).
| Minimum accuracy of every model | All items | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|
| BAT (accuracy) | 0.15 | 0.08 | 0.02 | 0.05 | 0.00 |
| Pairwise judging | 0.35 | 0.35 | 0.34 | 0.35 | 0.31 |
| Difference (pairwise − BAT) [95% CI] | 0.21 [−0.00, 0.42] | 0.26 [0.05, 0.48] | 0.32 [0.06, 0.58] | 0.30 [0.03, 0.56] | 0.31 [0.01, 0.62] |
We reiterate that the underlying assumptions regarding these experiments are distinct from our primary setting, which intentionally restricts the protocol to correctly solved transcripts to isolate the information available beyond answer correctness.
Second, we added three baselines that use historical model capabilities to predict the capabilities of future models. The release-date trend reduces pooled MSE from 1.37 to 0.55, while the family-adjusted trend and the forecast based on the most recent same-family model achieve 0.63 and 0.84, respectively. Pairwise judging achieves a pooled MSE of 0.26, 53% lower than the strongest historical baseline. Only the hypothetical oracle achieves a lower pooled MSE (0.22) by assigning each model the measured mean capability for its target semester and domain.
In addition to these results, the reviewer also raised the possibility that the judge recognizes model families from their transcripts. We have thus conducted additional analysis on potential biases in the LLM judges to address this concern.
We fit a hierarchical logistic regression to the pairwise decisions, modelling each choice as a function of transcript position, the capability difference between the two models, the length difference between the two transcripts, and whether one candidate comes from the judge's own developer family. The same-family effect is minimal in every domain. For recent judges it is 0.03 log-odds per standard deviation (95% CI [−0.06, 0.12]), against 0.68 [0.57, 0.80] for the capability difference. Adding it to a model with position, capability and length raises explained deviance only from 18.9% to 19.6%. Judging a target with a judge from its own family does not reduce prediction error; replacing these judges with the next-ranked judge from another family raises error by a similar amount (+0.30 on the 10 affected cases) as changing judges for targets with no family overlap (+0.20). The seven targets whose developer families were absent from the reference set are predicted as accurately as the rest (MSE 0.25 vs 0.27).
We have added Appendix E to report the historical forecasts and retrospective oracle (Appendix E.1, Table 18) and the accuracy-based comparisons at different levels of saturation (Appendix E.2, Figures 35-37). We have also added the family holdout and the analysis of judges’ preferences for their own developer family in Appendix D.4.
We thank the reviewer for the detailed feedback and for recognising the breadth and depth of the evaluation. We address the remaining concerns below.
> Regarding the relationship to existing evaluation approaches
We would like to clarify our work addresses a different but complementary question to constructing harder benchmarks or maintaining dynamic evaluations: whether or not we extract useful capability information from existing benchmarks even after their accuracy scores cease to distinguish models.
Thus the focus is to evaluate the evaluation protocols themselves: we test whether a protocol can predict measured capability on harder, held-out tasks from saturated benchmarks alone. In our work, arena-style pairwise comparison would be one such protocol, which relies on distinguishing models through human preferences. Our pairwise judging protocol is within the realm of arena-style approaches, but uses LLM judges in place of human raters. It is within the scope of our existing results that tell us that using LLMs for these comparisons can be informative.
Similarly, as discussed in Section 1 of our paper, developing more challenging and verifiable tasks is valuable but difficult and costly, especially when evaluations must cover a broad range of real-world capabilities where verifiable tasks are often difficult to construct. Our results show that correctly solved transcripts from existing tasks also retain information that predicts performance on harder, held-out evaluations, without requiring additional task construction. We believe these approaches to be complementary, as new benchmarks can provide direct capability measurements, while transcript-based methods can extract additional information from existing evaluation data between benchmark updates.