acceptodds
Under review as a conference paper at ICLR 2027

Life After Benchmark Saturation: A Case Study of CORE-Bench

Abstract

When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version. We show that this approach privileges accuracy and misses the opportunity to study other key dimensions of agent performance: construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift in duration from human-agent collaboration. We use CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate that measuring agents along these dimensions yields meaningful insights into agent performance even after accuracy saturates. First, we surface threats to construct validity in CORE-Bench Hard that are difficult to anticipate with less capable agents. We introduce an improved benchmark, CORE-Bench v1.1, and an out-of-distribution task suite, CORE-Bench Extended. Second, we find that despite accuracy saturation, CORE-Bench v1.1 remains useful for measuring efficiency, reliability, model performance, and scaffold performance. Finally, we conduct a small-scale randomized study to measure uplift in duration from human-agent collaboration on real-world computational reproducibility tasks. We find a statistically significant speedup by about a factor of two — likely underestimated due to one-fifth of human-only reproductions reaching the time limit — and describe various other findings. Together, our contributions present a more rigorous alternative to the dominant accuracy-centric evaluation paradigm.

Primary area:
datasets and benchmarks
Keywords:
AI agent evaluation, benchmark saturation, construct validity, out-of-distribution generalization, multi-metric evaluation, computational reproducibility, human-agent collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VDPGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Authors show that "retire-and-replace" approach might not be the best way to treat saturated benchmarks. Model evaluations on those kind of benchmarks can still contain valuable insights such as model's efficiency, reliability, exploitation of shortcuts vs real task mastery. They argue that some task-level threats such as false positives or reward-hacking can only be revealed once the benchmark is saturated. Moreover, better benchmark performance might be attributed to better adaptation to this benchmark's specific scaffold, prompts, format etc. Authors also perform the study on the uplift in duration of agent-human collaboration in research paper reproduction tasks.

Critical strengths

Interesting view on benchmark saturation. Old benchmarks can still be useful to measure capabilities beyond just accuracy.

Critical weaknesses
  • No error bars. Especially considering the small size of the benchmarks - 39 and 19 tasks. So for these tasks (tables 1 and 3) 10pp might be well within the std of the scores.
  • The second part (the uplift) study is vaguely related to the first part (benchmark might have tasks with threats that might make saturated performance misleading and saturated benchmarks are still useful).
Questions for authors
  • Why only when benchmark is saturated one can find tasks with threats? One can imagine a set of hard tasks that are not fully solved by an agent due to their nature and at the same time have flaws such as over reliance on scaffolding, incorrect verifier etc.
  • How were the reproductions evaluated in human-agent reproduction uplift?
  • Was the goal of the uplift study to understand whether agents improve paper reproduction performance of humans or that the agents can reproduce papers autonomously? How is it related to the benchmark saturation problem?
2 comments

Thank you for providing feedback on the error bars and uplift study. We appreciate that you found our submission to be an interesting view on benchmark saturation. We have revised the paper in response to feedback with all text changes in blue.

1. Benchmark saturation motivation

Before accuracy saturation, identifying all potential threats to benchmark validity is difficult and often infeasible across all tasks. For example, in a paper with a large codebase, manually investigating hundreds of data files and scripts to identify potential shortcuts an agent could exploit may be impractical. Running highly capable agents on these tasks, however, can help surface such shortcuts by revealing the various avenues these agents were able to find to exploit them.

2. Human-agent reproduction uplift evaluation

The human-agent uplift study was evaluated in two main ways. First, we estimated uplift in duration using the fixed effects model in Section A.5.8. Second, we created a questionnaire, provided in Section A.5.6, to understand where agents were useful, what blockers human-agent collaborative runs encountered and resolved compared to human-only runs, where agents struggled, and where agents required human intervention. We provide qualitative observations in Section 4 and Section A.5.10. In addition, we conducted a post-hoc correctness audit of all 50 reproductions in Section A.5.11, which showed human-only runs had lower correctness rates, but the differences were not statistically significant.

3. Goal of the uplift study

The goal of the uplift study was to understand whether agents are useful for human computational reproducibility workflows. In Section 4, we state the following:

Computational reproducibility is grounded in scientific workflows where humans interpret, validate, and build on results. Once top-performing agents converge at near-ceiling accuracy, the question shifts from whether agents can complete a task to whether they provide value alongside humans.

The uplift study is related to benchmark saturation because when agents are obtaining low accuracy, the question is still whether agents can solve the task at all. However, once accuracy reaches saturation, this is used as evidence to suggest that the agent is ready for deployment. However, high benchmark accuracy may not always translate to utility when deployed alongside humans. We give a few examples of why this could be the case in Section 4:

High benchmark accuracy may not cleanly translate to uplift in duration: agent failures may be more time-consuming for a human (or the agent itself) to resolve than human failures or agents may take more time to effectively respond to human redirection.

Thus, Section 4 asks whether an agent's near-ceiling benchmark performance (saturation) actually translates to real-world usefulness when deployed alongside humans in the same domain. Saturation therefore makes measuring uplift in task duration important to investigate whether benchmark success reflects real-world value when the task is collaborative between humans and agents.

4. Error bars

We agree that the analyses in Section 3 should have error bars. In Tables 1 and 3, we show the accuracy values with 95% Wilson CI bounds. We have updated the paper to add error bars to the consistency plots (Figure 1) based on the standard errors added in Table 25.


Once again, we appreciate your time and attention to detail. Our work has already improved thanks to your feedback. We hope that our rebuttal has addressed your concerns and remain available for further discussion throughout the rebuttal process.

Thank you for addressing my concerns, I will revise my score.

Reviewer_GB2wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what can still be learned from an agent benchmark after leading configurations reach near-ceiling accuracy, using CORE-Bench Hard as a case study. Through automated and manual trajectory auditing, the authors identify task errors and exploitable shortcuts, construct the corrected 39-task CORE-Bench v1.1, and introduce a 19-task Extended suite with a different disciplinary composition. They then examine repeated-run reliability, confidence, token and monetary costs, and interactions between models and agent scaffolds. Finally, a blocked randomized study involving five researchers, 20 papers, and 50 reproduction sessions estimates that manual sessions take 2.11 times as long as agent-assisted sessions, with a 95% confidence interval of [1.62, 2.75]. The contribution is an integrated empirical case study showing that benchmark maintenance and additional evaluation dimensions can provide useful evidence after aggregate accuracy becomes less discriminative.

Critical strengths
  1. The benchmark audit provides concrete, consequential evidence for the paper’s central thesis. The authors identify 15 tasks with task-level errors and 20 tasks with shortcut opportunities, with four overlapping tasks, in the original 45-task suite. The examples distinguish incorrect grading, underspecified questions, unavailable resources, and pre-existing answer artifacts, and the manuscript documents corresponding repairs (Tables 8–11). The corrected benchmark still produces near-ceiling scores, while subsequent analyses reveal meaningful differences in cost and behavior. This makes the argument for continued benchmark use empirically substantive and provides a useful model for maintaining agent evaluations.

  2. The randomized human study connects benchmark evaluation to an observable workflow benefit. The study balances conditions within researchers and across papers, randomizes task order, and estimates duration differences using paper and researcher fixed effects with CR2 standard errors. The retrospective audit of both output and method correctness further strengthens the evidence, although its uncertainty and non-blinded nature limit stronger quality claims. This component materially supports acceptance because it examines whether agent capability translates into value in an actual reproduction workflow.

Critical weaknesses
  1. The confidence experiment does not cleanly measure the probabilistic calibration used to support the underconfidence claim. Section 3.1 contrasts a 93% empirical pass rate with 32.1% mean reported confidence and interprets the gap as severe underconfidence. However, Listing 1 explicitly anchors confidence ranges to execution quality: significant execution errors or partial outputs correspond to low scores, whereas clean execution and verified extraction correspond to high scores. This mixes process quality, verification, and final-answer correctness. An agent can recover from several intermediate errors and still submit a correct answer; the prompt can nevertheless encourage a low score. Consequently, the association between bash errors and confidence may partly reflect the elicitation instructions. This materially weakens the broader conclusion about agents’ ability to assess final correctness. A prompt that explicitly elicits the probability of final-answer correctness, without error-based score bands, would directly test the interpretation.

  2. The model–scaffold comparisons support configuration-level observations more strongly than isolated component attribution. Section 3.3 is framed as decoupling models and scaffolds, but several relevant factors change together. CORE-Agent receives a five-hour timeout, while the other frameworks receive 45 minutes; retry mechanisms and reasoning settings also differ (Table 1; Appendix A.4). In the reliability experiments, GPT-5.1 uses a different Codex CLI version from the other models, and Appendix A.4.1 documents substantial version sensitivity. These differences are disclosed, but they limit statements attributing behavior specifically to the model or scaffold. Likewise, the higher success rate of direct fixes relative to rewrites is observational: difficult failures may themselves induce rewriting. The configuration comparisons remain useful, but component-level and strategy-level conclusions need matched controls or narrower interpretation.

Questions for authors
  1. Confidence construct and validation. What exact event should the reported confidence quantify: final-answer correctness under the evaluator, successful code execution, or a broader notion of reproduction validity? Does the large underconfidence gap persist under a prompt that directly requests the probability of final-answer correctness without execution-quality score bands? Please also explain how execution_summary is generated, since its contents could influence this result.

  2. Statistical support for the discrimination claim. Section 3.1 states that no agent appears to outperform random guessing in distinguishing correct from incorrect runs. However, Figure 1(d) shows an AUROC of approximately 0.64 for GPT-5.1, with a displayed interval that appears to lie above 0.5. What do these intervals represent, and what inference supports the blanket statement? Reporting success/failure counts and AUROC uncertainty that accounts for repeated observations of the same tasks would materially clarify this conclusion.

  3. Attribution to models and scaffolds. Which of the main behavioral differences remain when reasoning settings, time budgets, retry policies, and relevant scaffold versions are aligned? A representative matched comparison would be useful. Where such isolation is unavailable, can the conclusions explicitly distinguish observations about complete agent configurations from effects attributed to individual components? The direct-fix versus rewrite result should similarly be interpreted in light of strategy selection

2 comments

Thank you for taking the time to provide feedback on the reliability metrics and model-scaffold analysis. We are glad that you found the benchmark audit and uplift study practically useful. We have revised the paper in response to feedback with all text changes in blue.

1. Confidence construct

Thank you for your feedback on the confidence metric. The current prompt leading to the underconfidence gap directly requests a single number reflecting the agent’s confidence of final answer correctness, rather than requesting a score band. The score band is intended to guide the agent in mapping its trajectory to a confidence score. Despite this, agents were still underconfident:

Please assess your confidence in the correctness of your submitted answers.

Please provide a confidence score from 0 to 100

We agree that execution_summary may influence the agent’s confidence score. We have added an explanation of execution_summary to the paper in the caption of Listing 1:

The execution_summary includes agent messages in the format Agent: <full text>. Shell commands are summarized to Command (exit=N): <command>, with commands limited to 200 characters, followed by Output: <output>, limited to 300 characters. Other OpenCode tools are represented as Tool <name> (<status>). The model's reasoning and file edits are excluded from the summary.

2. Attribution to models and scaffolds.

We agree that a matched comparison across scaffold settings would be useful for the analysis. We re-ran all tasks that had timed out by increasing the harness-level timeout for timed out runs to up to 10 hours. We have updated Section 1 (accuracy Tables 1 and 3), Section 3.1 (reliability), and Section 3.2 (efficiency). Extending and standardizing timeouts did not change our main results. For the camera-ready version, we will update the model-scaffold analysis with these results. We have also updated Section A.3 with additional details on the timeouts used.

We have added a Limitations sub-section to Section 3 explaining the differences in retry behavior:

Unlike Codex CLI, Claude Code and OpenCode, we did not use any scaffold-level retries for CORE-Agent because Hugging Face smolagents does not expose a directly comparable retry mechanism. In contrast, we evaluated the other scaffolds with three retries at both the harness and scaffold levels. Retry behavior was not fully standardized across scaffolds, which may affect comparisons.

3. Statistical support for the discrimination claim

Section 3.1 states agents frequently do not exhibit discrimination better than random chance and are not able to distinguish successes from failures. We agree that these claims do not fully capture the AUROC of 0.64 for GPT-5.1. After matching harness-level timeouts across agents, the AUROC of 0.64 for GPT-5.1 with Codex CLI decreased to 0.53. We have updated the paper to state that agents exhibit discrimination around random chance, rather than that they do not exhibit discrimination better than random chance. We have also added success/failure counts and AUROC uncertainty to Table 25.


Once again, we appreciate your time and attention to detail. Our work has already improved thanks to your feedback. We hope that our rebuttal has addressed your concerns and remain available for further discussion throughout the rebuttal process.

Thank the authors for the detailed reply. Please include the rebuttal content in the fianl version of the paper. I will maintain my score.

Reviewer_9Z9cOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether a benchmark can still be useful once leading agents reach near-perfect accuracy. Using CORE-Bench as a case study, the authors audit agent trajectories to identify shortcuts and grading errors, release a revised benchmark and an additional task suite spanning different research fields, and compare agents on reliability, cost, and the effects of model and scaffold choices. They find that agents with similar accuracy can differ substantially in cost and how they approach or fail tasks.

Critical strengths

S1: The paper addresses a practical question: what can we still learn from a benchmark once leading agents achieve near-perfect accuracy? The CORE-Bench case study shows that similar accuracy scores can conceal meaningful differences in cost, consistency, and failure modes.

S2: The benchmark audit provides concrete improvements to CORE-Bench. The authors identify and address shortcuts and grading errors through automated log analysis and manual inspection, making the revised benchmark a useful resource for future evaluations.

Critical weaknesses

W1: This paper targets on a interesting direction. However, its connection to “accuracy saturation” is insufficiently established. Since prior work already advocates evaluating efficiency, reliability, and practical utility, the contribution of this paper seems insignificant.

W2: The claim that saturation reveals benchmark validity threats lacks a systematic comparison. Section 2 argue that weaker agents often fail during dependency installation, whereas stronger agents reach later stages and expose shortcuts or grading errors. This is plausible, but the evidence primarily consists of examples instead of systematic comparison.

W3: The contribution is primarily an empirical evaluation and benchmark-maintenance study, with limited methodological novelty. Its significance therefore depends on the reliability, generalizability, and novelty of the resulting findings.

Questions for authors
  1. What does accuracy saturation add to the motivation for this study? Prior work already argues for evaluating efficiency, reliability, and practical utility. Which findings here depend on studying a saturated benchmark, and which would also apply before saturation?
  2. Can you provide a systematic comparison of the validity threats uncovered by weaker and stronger agents? For example, how many shortcuts or grading errors were revealed only by stronger agents, and is this explained by their reaching later stages of execution?
  3. Given the paper’s primarily empirical contribution, which findings do you consider new beyond prior work on multidimensional agent evaluation? What evidence suggests that these findings would extend beyond CORE-Bench and computational reproducibility tasks?
1 comment

Thank you for your review. We appreciate the reviewer considering our question practical and the updated CORE-Bench a useful resource.

1. Accuracy saturation as a motivation for the study

We agree that non-accuracy metrics are important both before and after saturation. However, despite prior work (cited in the Introduction) that introduces metrics for evaluating agents beyond accuracy, the dominant paradigm in the AI evaluation field today still focuses solely on headline accuracy. Many widely used agent benchmarks still primarily report only accuracy on one-dimensional leaderboards. If the community moves directly to harder successors without first recognizing the value of non-accuracy metrics on saturated benchmarks, they will continue to prioritize measuring only accuracy on the harder benchmarks as well. This pattern appears in the examples we highlighted in the introduction. To show that metrics beyond accuracy are important in general, we must first show that accuracy saturation doesn’t imply that the entire benchmark is no longer useful and no other dimensions remain informative. This is a key contribution of our paper.

2. Comparison of validity threats by weaker and stronger agents

The original CORE-Bench paper (Siegel et al., 2025) conducts an analysis of failure modes on CORE-Bench:

Agents often did not finish resolving dependency version issues before hitting the cost limit, getting stuck attempting to install the same library multiple times (Section D.3.3). However, as we showed in Figure 7, increasing the cost limit did not significantly increase accuracy because even with more time, they could not resolve the issue.

This showed that weaker agents are unable to progress past attempting even the first step of code reproduction and fail on the tasks, which we explain in the caption of Table 10. On the other hand, shortcuts and grading errors are easier to find when more capable agents today are able to find shortcuts to exploit or finish solving a task correctly but are still graded as incorrect.

3. Novelty compared to prior work

We agree that prior work has advocated for evaluating efficiency, reliability, and practical utility in principle. Despite this, the field still defaults to measuring accuracy alone. We show that despite accuracy scores that are no longer able to distinguish between agents, other metrics still can; in other words, metrics beyond accuracy are useful even when accuracy isn’t, establishing their importance beyond accuracy. Although our results are a case study of one benchmark, we believe the findings do generalize beyond CORE-Bench and computational reproducibility. First, the benchmark validity threats we uncovered have also been found in widely used benchmarks across domains, such as FrontierMath and TerminalBench 4. Second, none of the non-accuracy metrics are specific to CORE-Bench, so the multidimensional agent evaluation framework generalizes to other domains.


Once again, we appreciate your time and attention to detail. We hope that our rebuttal has addressed your concerns and remain available for further discussion throughout the rebuttal process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.