acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Model Capabilities After Benchmark Saturation

Abstract

Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated. We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items. We evaluate representative contemporary approaches for extracting information from model transcripts: pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. We find that saturated transcripts retain capability information beyond benchmark accuracy. Amongst our evaluated protocols, pairwise judging performs best, reducing prediction error by compared to a baseline using only saturated benchmark scores. This capability-relevant information is distributed across transcript sections, but concentrated in specific channels of information such as reasoning and tool-calls. However, the predictive power of protocols does not always generalise across distribution shifts – for example by underestimating the capabilities of the first reasoning models introduced in early 2025. Saturated benchmarks thus retain capability-predicting signal in their task transcripts, but extending the lifetime of existing benchmarks requires methods designed to extract these signals robustly.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
evaluation, benchmark, saturation, scalable, oversight, llm, safety, cyber, judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DgF3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses how to assess LLM capabilities after benchmarks saturate. The authors rely on how an LLM reached the solution instead of simply whether it reached there or not, which attempts to extract more knowledge of model capabilities beyond the final solution. Their method achieves 5.3x less prediction error compared to constant prediction.

Critical strengths

The general capability of assessing LLM performance using an LLM judge from a previous generation is interesting. Additionally, the paper shows that solved tasks still contain capability predictive information beyond accuracy. The evaluation covers a large spectrum of models and the authors provide a lot of analysis including pointing out the failures which shows a comprehensive analysis.

Critical weaknesses

Saturation is mostly solved through the numerous benchmarks that perform pairwise LLM evaluation models (i.e. arena benchmarks), even in a future where models exponentially exceed human capabilities in all metrics, arena benchmarks that have verifiable sufficiently hard rewards that are not based on human feedback or any subjective evaluation exist. Many tasks are easy create and validate than to solve thus humans could still evaluate models even in that future (e.g. generate lean proof of 10D navier-stokes or decide if a given finite set of turing programs halt under a fixed given budget). Additionally, some saturated benchmarks still can achieve high scores in Benchmark Agreement Testing which indicates that the baseline in the paper could be considered weak.

I would like to see more results justifying the prediction power of this method. Figure 3 relies on IRT (which has its own assumptions) for the target prediction. However, the authors could compare predictions against average capabilities of the models on unsaturated benchmarks. If only rank is considered it's possible to compare the predicted ordering of the target models against the ordering obtained by averaging abilities. I would generally want to see more evaluations. A current assumption of the protocol is that reliable estimates are available for the previous semester’s models to not be saturated yet (for the calibration θref\theta_{ref}).

Another major concern is that evaluation on chain-of-thought is an AI safety risk as it incentivises obfuscation and puts pressure on LLMs to hide true intentions just to maximize an LLM judges evaluation.

Questions for authors

How sensitive are the results to the use of IRT prediction targets? Could the authors evaluate rank correlations and multiple other baselines to justify the protocol with more rigorous evaluation.

Is it possible that the LLM judge recognizes the family of the model from its transcript and predicts that the capabilities of next semester’s LLM is equal to the previous generation family plus a constant. For example, what would a baseline that uses model release dates and family information like historical capabilities and other possible useful historical info.

7 comments

We thank the reviewer for the detailed feedback and for recognising the breadth and depth of the evaluation. We address the remaining concerns below.

> Regarding the relationship to existing evaluation approaches

We would like to clarify our work addresses a different but complementary question to constructing harder benchmarks or maintaining dynamic evaluations: whether or not we extract useful capability information from existing benchmarks even after their accuracy scores cease to distinguish models.

Thus the focus is to evaluate the evaluation protocols themselves: we test whether a protocol can predict measured capability on harder, held-out tasks from saturated benchmarks alone. In our work, arena-style pairwise comparison would be one such protocol, which relies on distinguishing models through human preferences. Our pairwise judging protocol is within the realm of arena-style approaches, but uses LLM judges in place of human raters. It is within the scope of our existing results that tell us that using LLMs for these comparisons can be informative.

Similarly, as discussed in Section 1 of our paper, developing more challenging and verifiable tasks is valuable but difficult and costly, especially when evaluations must cover a broad range of real-world capabilities where verifiable tasks are often difficult to construct. Our results show that correctly solved transcripts from existing tasks also retain information that predicts performance on harder, held-out evaluations, without requiring additional task construction. We believe these approaches to be complementary, as new benchmarks can provide direct capability measurements, while transcript-based methods can extract additional information from existing evaluation data between benchmark updates.

> Regarding potential additional baseline protocols

We have expanded our comparisons to incorporate new baseline measures in Appendix E: Comparisons against non-transcript based baselines.

First, we have added an accuracy baseline using Benchmark Agreement Testing (BAT) on all four domains in the main semesters (2025-II, 2026-I). For each domain, we take accuracy on the easiest benchmark (the benchmark with highest average accuracy: ARC-Challenge, MMLU-Pro maths, AgentBench-OS, InterCode), and test how well accuracy rankings agree with the hardest benchmark of the same domain (HLE-Science, HLE-Math*, SWE-bench mini, CyBench; Figure 35).
*We exclude APEX 2025, which only has 12 items on which 13/24 models score 0%.

As expected, accuracy-based ranking retains predictive signal when the input benchmark has not saturated. In Coding and Cybersecurity, where the easiest benchmarks still show substantial variation in accuracy, rank correlations with harder benchmarks are 0.30 and 0.45 among target models. However, in Science and Mathematics, where the easiest benchmarks are close to saturation as in our defined problem setting, these correlations are indistinguishable from zero (0.00 and −0.09). The table below compares BAT with pairwise judging.

Domain (easiest → hardest benchmark)Accuracy on easiest benchmark, mean (range)BAT (accuracy), τ [95% CI]Pairwise judging, τ [95% CI]Difference [95% CI]
Science (ARC-Challenge → HLE Science)98% (92–100)0.00 [−0.50, 0.50]0.12 [−0.41, 0.66]0.12 [−0.52, 0.77]
Maths (MMLU-Pro → HLE Math)94% (84–98)−0.09 [−0.56, 0.38]0.38 [−0.12, 0.87]0.46 [0.15, 0.78]
Coding (AgentBench-OS → SWE-bench mini)79% (40–95)0.30 [−0.10, 0.71]0.39 [−0.10, 0.88]0.09 [−0.29, 0.47]
Cyber (InterCode → CyBench)86% (50–92)0.45 [0.12, 0.78]0.50 [0.21, 0.79]0.05 [−0.39, 0.50]
All domains0.17 [−0.07, 0.40]0.35 [0.04, 0.65]0.18 [0.01, 0.35]

Further, BAT performance drops considerably as we restrict the item pool to simulate saturation as in our problem setting. We repeat the above comparison keeping only items on which every model exceeded a minimum X% accuracy, sweeping X from 70% (approaching saturation) to 100% (fully saturated). To keep the number of items constant across these levels, we resample (200 draws) N items at each level, where N is the number of items available at 100% saturation. The ‘All items’ column therefore differs slightly from the table above. BAT's rank agreement drops consistently (until definitionally reaching τ=0 at full saturation), whereas pairwise judging remains reliable at τ=0.31 -- 0.35 (table below; Figure 36). The same pattern holds for predicting IRT-derived θ* in the setting of our main analysis, where accuracy-based prediction consistently gives higher squared error than pairwise judging (Figure 37).

Minimum accuracy of every modelAll items70%80%90%100%
BAT (accuracy)0.150.080.020.050.00
Pairwise judging0.350.350.340.350.31
Difference (pairwise − BAT) [95% CI]0.21 [−0.00, 0.42]0.26 [0.05, 0.48]0.32 [0.06, 0.58]0.30 [0.03, 0.56]0.31 [0.01, 0.62]

We reiterate that the underlying assumptions regarding these experiments are distinct from our primary setting, which intentionally restricts the protocol to correctly solved transcripts to isolate the information available beyond answer correctness.

Second, we added three baselines that use historical model capabilities to predict the capabilities of future models. The release-date trend reduces pooled MSE from 1.37 to 0.55, while the family-adjusted trend and the forecast based on the most recent same-family model achieve 0.63 and 0.84, respectively. Pairwise judging achieves a pooled MSE of 0.26, 53% lower than the strongest historical baseline. Only the hypothetical oracle achieves a lower pooled MSE (0.22) by assigning each model the measured mean capability for its target semester and domain.

In addition to these results, the reviewer also raised the possibility that the judge recognizes model families from their transcripts. We have thus conducted additional analysis on potential biases in the LLM judges to address this concern.

We fit a hierarchical logistic regression to the pairwise decisions, modelling each choice as a function of transcript position, the capability difference between the two models, the length difference between the two transcripts, and whether one candidate comes from the judge's own developer family. The same-family effect is minimal in every domain. For recent judges it is 0.03 log-odds per standard deviation (95% CI [−0.06, 0.12]), against 0.68 [0.57, 0.80] for the capability difference. Adding it to a model with position, capability and length raises explained deviance only from 18.9% to 19.6%. Judging a target with a judge from its own family does not reduce prediction error; replacing these judges with the next-ranked judge from another family raises error by a similar amount (+0.30 on the 10 affected cases) as changing judges for targets with no family overlap (+0.20). The seven targets whose developer families were absent from the reference set are predicted as accurately as the rest (MSE 0.25 vs 0.27).

We have added Appendix E to report the historical forecasts and retrospective oracle (Appendix E.1, Table 18) and the accuracy-based comparisons at different levels of saturation (Appendix E.2, Figures 35-37). We have also added the family holdout and the analysis of judges’ preferences for their own developer family in Appendix D.4.

Reviewer_SNd5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary
  • The paper looks at a very interesting problem which is whether we can still use benchmarks for evaluation after they get saturated. This is interesting as benchmarks getting more saturated over time which is relevant (due to many issues like Goodharting or leakage into the training data). They show different ways in which they can serve as meaningful evals of the growing capabilities of models.
  • As models grow to be more capable, traditional beliefs suggest that benchmarks cease to be useful which makes evaluation to be really tricky and this paper shows a different perspective showing that we can still use those stale benchmarks and extract meaningful information of the model that is more reflective of their capabilities like the transcripts of the model while solving the task or we can compare answers or transcripts from 2 different models instead of independently scoring.
  • This is super useful as producing harder benchmarks require a lot of efforts or expert human validation etc and it becomes harder as we want to test more advanced capabilities.
Critical strengths
  • Deals with an important problem that is well motivated.
  • Shows that transcripts of the model and how their answers on easier task can help predict how they will perform on harder tasks
  • We can gain some of the prediction accuracy of the target model on a harder benchmark based on how their answers or transcripts compare with the reference model on the easier benchmark showing that there is still value on the table that is not used on easier benchmarks and shifting more focus towards having evals that use more of the generated answers of our current models as evaluation.
  • They show that using reasoning or tools basically carry important signals on prediction accuracies on the harder benchmarks for some domains.
  • They show that we can explain capabilities gap on harder benchmarks easier across domains using the pairwise judging approach.
Critical weaknesses
  • For Figure 6, we could actually be expecting the capabilities gap to widen more so if we look at the fraction of variance unexplained by the dataset, this should increase with model capabilities. Right now because the prediction errors is measured in terms of the baseline as in relative performance on easier benchmark, it does not tell us how much can we explain in terms of the capability diff on a harder benchmark.
  • The paper does not explain the limits of these methods, for example, what happens when the target model is chosen from a time period that is far above the reference pool of models.
  • It would be great to have some measure of uncertainty in the benchmark scores to complement predictive signal.
Questions for authors
  • Define what portion of the predictive accuracy of the model on the harder benchmark can we actually explain (not relative to the weaker baseline presented)
  • Clearly define the limits of predictive accuracy based on stale benchmarks, we want to know when do we need a new benchmark with a new task compared to trying to squeeze more juice out of a stale one.
  • The model selection limits how broadly the results can be generalised. The main experiments use eight target models per period and primarily include open-weight reasoning models with accessible reasoning traces. Several models also belong to the same families, so the number of models may overstate the diversity of behaviours being evaluated. Although the authors test six closed-source models separately, that evaluation covers coding only. It therefore remains unclear whether the approach transfers reliably across model families and frontier models in other domains. Holding out entire model families would provide a stronger test of this generalisation.
6 comments

We thank the reviewer for the encouraging feedback and recognizing the potential of saturated transcripts to predict performance on harder tasks after benchmark saturation. We address the remaining concerns below.

> Regarding predictive capabilities of protocols

We thank the reviewer for this point. We thus additionally report how well each protocol ranks the target models within each period. This measure doesn’t depend on any baseline value or on how far the period moves. We compute rank correlations between predicted and measured capability over the targets of each period.

ProtocolSquared errorSpearman ρKendall τ
Baseline1.37 [0.69, 2.06]0.000.00
Deterministic features0.86 [0.32, 1.41]0.21 [−0.31, 0.72]0.15 [−0.24, 0.54]
Embedding1.01 [0.36, 1.66]0.34 [−0.10, 0.79]0.29 [−0.07, 0.66]
Absolute scoring1.01 [0.39, 1.64]0.28 [−0.13, 0.69]0.21 [−0.12, 0.54]
Pairwise judge0.26* [0.14, 0.38]0.51* [0.10, 0.92]0.38* [0.07, 0.69]

Pairwise judging achieves both the lowest squared error and the highest within-semester rank correlation. This suggests that its performance is not solely due to predicting the overall capability level of the target cohort, but also reflects information about relative differences between target models. We have added within-semester rank agreement in Appendix D.17 (Table 13 and Figure 32).

We also test whether prediction error increases as target capability exceeds that of the strongest reference model (Figure 8). Across all four domains, we find no statistically significant increase in pairwise judging error with capability distance (slopes from −0.12 to 0.10, all p ≥ .31). In contrast, the baseline exhibits substantially larger positive slopes (0.96–2.64, p ≤ .024).

The one clear exception is early 2025 in maths, when reasoning models first appear (Section 4.3, Figure 7). There the relationship between transcript behaviour and hard-item capability shifts, and the judge consistently underestimates the new model class.

> Regarding actionability of current results

Appendix D.14 (Figure 29) directly examines this question by progressively increasing the temporal distance between reference and target models.

For example, in Science, the MSE of pairwise judging increases from 0.25 with late-2025 reference models to 0.47 with early-2025 reference models, and reaches 10.39 when predicting early-2026 target models using references from 2023.

Our results suggest that the limitation of the currently tested protocols is related to the distribution shift between reference and target populations, as older and newer targets at the same distance do not differ significantly (p ≥ .52), while target models from two semesters away have higher error than target models from one semester away (0.59 vs. 0.14, p = .024).

Evaluation developers can apply a similar analysis methodology to periodically validate predictions against newly available capability measurements and monitor whether prediction error remains within an acceptable tolerance. Once the protocol no longer meets that tolerance, recalibrating the reference population or introducing new evaluation tasks becomes necessary.

> Regarding extended generalization of results

We additionally conduct a held-out model-family analysis, in which, for each target model, we remove every reference model from the same developer family from calibration and replace any same-family judge with the best-ranked candidate judge from a different family. We then recompute predictions from the existing judge verdicts.

Pairwise judging achieves an MSE of 0.43, compared with 0.26 under the original reference population; the baseline protocol scores 1.37. The judge-switch control produces a comparable increase in error even when there is no family overlap (0.27 to 0.45), indicating that the observed increase cannot be attributed solely to removing same-family information. Removing any single family from the evaluation set leaves pairwise MSE between 0.23 and 0.30, and the seven targets from families absent from the reference set are predicted as accurately as the rest (0.25 vs. 0.27).

We additionally note that, as the reviewer mentioned, our existing Section 4.3 (Figure 9) already evaluates six closed-source frontier models with hidden reasoning traces, where the pairwise judging protocol outperforms the baseline for all six. We have added the per-target family holdout and the judge-switch control in Appendix D.4 and updated Section 4.1 to point to them.

Reviewer_xoDCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As state-of-the-art models increasingly achieve near-perfect scores on existing benchmarks, it becomes difficult to distinguish between their capabilities. This paper investigates whether model transcripts from correctly solved tasks can provide additional information about model capabilities beyond final-answer accuracy, without the need to develop new and harder benchmarks. To investigate this, the authors use transcripts from saturated benchmark questions to predict model capabilities measured on harder, unsaturated questions. They evaluate several approaches for extracting information from these transcripts and find that pairwise judging achieves the lowest prediction error. Their experiments also examine where capability-related information is located within transcripts and how well these approaches generalize across different models and time periods.

Critical strengths
  • The paper addresses an important problem of how to distinguish between model capabilities when existing benchmarks become saturated and newer models achieve almost identical scores.
  • I like how the authors approach this problem by investigating whether we can still extract useful information about model capabilities from correctly solved questions on saturated benchmarks.
  • I appreciate how the experiments are organized around clear research questions, with each experiment building on the previous findings. The authors first compare different methods for predicting capability, then investigate which parts of the transcripts contribute to these predictions, and finally examine how well the methods generalize across different model generations and domains. I also find the additional experiments on different saturation thresholds and closed-source models useful.
  • I like that the authors also investigate cases where their approach does not work well, particularly when predicting the capabilities of newer reasoning models. This helps provide a better understanding of when transcript-based evaluation can be useful and where it may fail.
Critical weaknesses
  • The paper evaluates the proposed methods based on how well they predict IRT scores on unsaturated questions. While IRT is a useful measure that accounts for item difficulty, discrimination, and guessing, it is still based on answer correctness. I think model capability also involves aspects such as reasoning quality, robustness, and problem-solving efficiency, which may not be reflected in IRT scores. Since the paper's conclusions are framed around evaluating model capabilities, I would like to see more evidence that predicting IRT scores also reflects these broader aspects of capability, rather than only benchmark performance.
  • The proposed method relies on an LLM judge making an overall comparison between two correctly solved transcripts. I wonder whether judges are influenced by factors such as response length or writing style rather than the actual quality of reasoning. I think it would be useful to investigate these biases more systematically and compare the current approach against a rubric-based judge that separately evaluates aspects such as reasoning correctness, quality, and efficiency before making an overall judgment.
  • The baseline uses the average capability of reference models from the previous semester, which may underestimate newer models as capabilities improve over time. This could make the reported improvements appear larger.
Questions for authors
  1. I wonder whether the authors have considered a baseline that accounts for historical trends in model capabilities. Would the improvement from pairwise judging still be as significant compared to this baseline?
  2. I wonder whether the judges tend to prefer longer or more detailed responses, even when it is not necessarily better. Have the authors considered a more structured judging approach that evaluates different aspects of each response before making the final comparison?
5 comments

We thank the reviewer for their thoughtful feedback and for recognizing the importance of extracting capability-relevant information from saturated benchmarks. We address the remaining concerns below.

> Regarding evidence of broader capability measurement

Our primary evaluation measures the ability to predict performance on harder, unsaturated tasks, using IRT to place results from different benchmarks on a common scale. We test whether the findings depend on this choice of measurement in two ways.

First, we test whether our findings hold for measures of capability other than our IRT fit (including broader, external capability indices such as Epoch Capabilities Index (ECI) and Artificial Analysis Intelligence Index (AAII) [1,2]). This demonstrates that our results are not sensitive to the specific IRT-derived target we use. Second, we examine which properties of a transcript the pairwise judge predictions might rely on (in the second part of the section below: “Regarding potential LLM judge biases”).

Pairwise judging had the lowest error amongst the protocols regardless of the specific target (see table below). This suggests that the capability predictions are not specific to the IRT-derived metric we use, and correlate with more general capabilities aggregated by ECI and AAII. We report full results in Appendix D.18 (Tables 14 and 15).

Our prediction targets measure task-solving performance, while the protocols receive only transcripts from correctly solved saturated items. This design tests whether differences in how models solve the same tasks provide information about their performance on harder tasks, beyond the final-answer correctness available from those transcripts. Consistent with this, removing models’ reasoning from transcripts increased prediction errors (see Section 4.2). In the section below, we also characterise determinants of judge verdicts – which suggests that pairwise judging aligns with the correctness and quality of reasoning.

ProtocolSWE-bench accuracy targetECI TargetAAII TargetIRT Target
Baseline2.47 (27.1%)45.5 (6.7)76.1 (8.7)2.14 (1.46)
Deterministic features2.00 (20.1%)55.5 (7.4)71.2 (8.4)0.68 (0.82)
Embedding2.08 (20.8%)39.7 (6.3)61.8 (7.9)1.91 (1.38)
Absolute scoring1.91 (25.5%)40.6 (6.4)69.1 (8.3)1.90 (1.38)
Pairwise judge0.24 (7.4%)19.3 (4.4)59.7 (7.7)0.34 (0.58)

Values are mean squared errors, as in Tables 14 and 15. Parentheses give each error on an interpretable scale: mean absolute error in % accuracy for SWE-bench, and root mean squared error in index points for ECI and AAII and in θ for IRT.

[1] Epoch AI, Epoch Capabilities Index (ECI) documentation: https://epoch.ai/data/eci-documentation

[2] Artificial Analysis, Intelligence Benchmarking Methodology (Intelligence Index v4.3): https://artificialanalysis.ai/methodology/intelligence-benchmarking

> Regarding potential LLM judge biases

We have additionally analysed the relationship between pairwise judges' choices and superficial properties of transcripts such as length and prose style (such as use of pronouns, confidence, rate of contrastive terms), as requested by the reviewer.

The low performance of the ‘deterministic features’ protocol (Mean MSE=0.86) demonstrates that these superficial features are poor predictors of capability – and thus unlikely to fully account for the accuracy of judges (MSE=0.26).

To test whether these properties nevertheless influence judge decisions, we additionally quantify the contributions of transcript length, writing style, presentation order, and measured capability to pairwise verdicts (Appendix D.6, Table 10 and Figure 18).

For each of the eight pairwise judge models used in the main analysis, we estimate drivers of their verdicts by fitting a logistic regression using transcript length, word choices (e.g. hedging terms and first-person pronouns) and model capability – each entered as the difference between the first and second transcript shown – along with a constant preference for the first transcript (position bias). The full model predicts 69.9% of held-out verdicts (chance level: 50%). This is almost as high as the agreement rate exhibited by a judge when shown the same pair of transcripts in the opposite order (70.1%).

To measure how each of these predictors contributed to judge verdicts, we then removed each predictor in turn and refit regression models on this reduced subset (see table below).

Removed predictorDrop in accuracy (%)
Position bias (preference for the first transcript)3.3 [0.6, 5.9]
Capability difference2.8 [1.9, 3.6]
Transcript length1.8 [0.9, 2.6]
Writing style1.3 [0.7, 2.0]

Removing any of the predictors significantly reduced prediction of judge verdicts (ps<.05; paired t-tests, N=8 judges) – indicating that while judges are biased by features such as transcript length, position, and writing style, the capability difference between compared models contributes unique information to judgements that isn’t captured by these superficial features.

Across domains, only capability was consistently important for predicting judge verdicts. While length and writing style biased judges strongly in Science and Maths, we did not observe this in Coding and Cyber (Appendix D.6, Figure 19).

Additionally, we asked judges to rate each transcript separately on features such as correctness and quality of reasoning, and efficiency, as well as two presentation-related properties (clarity and detail of writing). Overall, correctness and quality correlated best with the verdicts of pairwise judges (with the transcript receiving higher scores in these features being picked as the more capable model on 92% and 84% of pairwise judgements, respectively). Full prompts and results are now included in Appendix D.6 (Table 11 and Figure 20).

FeatureJudge picks higher-rated transcript in this feature
Correctness (How valid and supportive of the solution each step is)92%
Quality (Depth of understanding and rigour)84%
Efficiency (Solution is reached without unnecessary steps or detours)57%
Clarity (How easy the response is to follow)72%
Detail (How much explanation and elaboration the response gives)73%

> Regarding adding a historical forecasting baseline

We have additionally evaluated four baselines in response to the reviewer’s suggestion. Three use capability estimates from models released before the target semester to forecast target-model capabilities from release dates and, in two cases, model-family history. The fourth uses the measured mean capability of the target population as a retrospective oracle.

ProtocolPooled MSE
Previous-semester reference mean1.37
Release-date trend0.55
Family-adjusted release-date trend0.63
Most recent same-family model, adjusted for elapsed time0.84
Pairwise judging0.26
Target-semester mean oracle0.22

The release-date trend reduces pooled MSE from 1.37 to 0.55. Pairwise judging achieves an MSE of 0.26, outperforming all three historical forecasting baselines, including the strongest release-date trend by 53%. The retrospective target-semester oracle achieves 0.22 but requires access to target capabilities unavailable at prediction time.

We have added the three historical forecasts and the retrospective target-semester mean oracle to Appendix E.1 (Table 18). We have updated Sections 3.2 and 4.1 to point to these comparisons.

We hope these clarifications and the additional experiments address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chairs

We thank all three reviewers for their thoughtful feedback and constructive suggestions.

We are grateful that two of the three reviewers (xoDC and SNd5) gave initial ratings leaning acceptance, recognizing the importance of benchmark saturation and the value of extracting capability-relevant information from correctly solved task transcripts.

  • Reviewer xoDC highlighted the importance of the research question, the clear progression of experiments, the breadth of evaluation across models and domains, and the analyses of saturation thresholds, closed-source models, and failure cases.
  • Reviewer SNd5 emphasized the value of recovering predictive information from saturated benchmarks, the findings on reasoning and tool-use signals, and the potential to extend the usefulness of existing evaluations without requiring new, harder tasks.
  • Reviewer DgF3 also recognized the interesting approach of using previous-generation LLM judges to assess newer models, the extensive evaluation across model families, and the comprehensive analysis of both successful predictions and failure cases.

We have addressed the reviewers' remaining concerns through additional experiments, analyses, and manuscript revisions:

  • For Reviewer xoDC, we have evaluated three alternative capability targets, including held-out SWE-bench accuracy and the Epoch and Artificial Analysis capability indices; conducted additional analyses of judge biases related to transcript length, writing style, presentation order, and reasoning quality; and introduced historical forecasting baselines. Pairwise judging achieves the lowest error across all evaluated targets and reduces MSE by 53% relative to the strongest historical forecasting baseline (0.55 to 0.26). These results strengthen the evidence that our findings are not specific to the IRT-based target, that judge predictions capture information beyond superficial transcript properties, and that the improvement persists against stronger baselines.

  • For Reviewer SNd5, we have added within-semester rank correlations, held-out model-family experiments and judge-switch controls, and highlighted analyses of prediction errors across increasingly distant reference populations. We have also converted capability predictions into predicted benchmark scores and reported error-tolerance curves. Pairwise judging achieves the strongest within-semester rank agreement, retains its advantage after removing same-family references, and predicts held-out benchmark scores with an average error of 9.9 percentage points versus 22.3 for the baseline. These results demonstrate that pairwise judging captures information about differences between target models, clarify its generalization across model families, and characterize the practical accuracy and limitations of the approach.

  • For Reviewer DgF3, we have clarified how transcript-based capability prediction complements arena-style comparisons and dynamic benchmarks, and conducted additional accuracy-based comparisons inspired by Benchmark Agreement Testing across increasing levels of saturation. We find that accuracy-based rankings lose their predictive signal as benchmarks saturate, whereas pairwise judging retains positive rank agreement. Alongside the new historical forecasting baselines, model-family analyses, alternative prediction targets, and rank-correlation results, these experiments directly address concerns about the strength of our baseline, sensitivity to IRT, and whether the observed predictive signal can be explained by model-family information alone.

We have incorporated these additional experiments and analyses into the revised manuscript, with changes highlighted in blue, and provided detailed explanations in our individual reviewer responses.

We hope these revisions address the reviewers' remaining concerns and would be grateful if the reviewers would reconsider their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.