acceptodds
Under review as a conference paper at ICLR 2027

GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

11 views

Abstract

Agent self-evolution updates an agent's persistent state from prior experience and reuses it to solve related tasks more effectively. Evaluating self-evolution is difficult: existing benchmarks provide limited coverage of economically valuable task domains, do not always design training and test tasks such that test-time gains can be attributed to training experience, and remain vulnerable to data contamination. We present GDPevo, an evolution-native benchmark grounded in GDP-related enterprise workflows, together with the fully automated data pipeline that generates it. Its core mechanism, rule hybridization, decomposes each enterprise workflow into atomic business rules, distributes subsets of these rules across training tasks, and recombines them in held-out test tasks so that test-time gains are attributable. GDPevo spans CRM, ERP, finance, healthcare, legal, and data-centric workflows. Its V1 release contains 120 tasks in 12 groups, with five training and five held-out test tasks per group. Full automation enables the pipeline to expand the suite to 240 tasks in 24 groups (V2) within two days, providing a practical response to contamination. Using GDPevo, we evaluate four agents, each comprising a harness and a model, under four supervision types. Self-evolution consistently improves held-out accuracy by up to 16.44 percentage points. But the best evolved agents remain far below the fully informed oracle ceiling of 91.6%, indicating that the self-evolution ability of current agents remains far from fully realized.

Primary area:
datasets and benchmarks
Keywords:
Agent, Self-Evolution, Recursive Self-Improvement, Benchmark, GDP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7uvGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether agents can consolidate experience accumulated from previous tasks into persistent state (e.g., memory, skills, prompts, harness code) and reuse it on subsequent related tasks. To address three limitations of existing benchmarks — insufficient coverage of economically valuable enterprise-level tasks, train–test splits inherited from random partitions that make post-evolution gains non-attributable, and the vulnerability of static public task sets to data contamination — the authors propose GDPevo, an evolution-native benchmark together with a highly automated construction pipeline. Its core mechanism, rule hybridization, decomposes enterprise workflows into atomic business rules that do not exist in the model's world knowledge, distributes subsets of these rules across 5 training tasks, and recombines them in 5 held-out test tasks, so that test-time gains can be attributed to learning and compositionally applying the rules. The benchmark is constructed by agents through three stages — seed scenario discovery, task group generation, and calibration with quality filtering — covering CRM, ERP, finance, healthcare, legal, and data analysis, and is equipped with deterministic code-based graders while treating cost as a first-class metric. The experiments evaluate four agents under four supervision types in a skill-based evolution setting, and find that: any supervision brings gains of 2.59–16.44pp, with fewshot being the strongest while also reducing test-time cost, yet the best result remains far below the 91.6% human–AI collaboration reference; fewshot overfits in cross-domain transfer much like supervised fine-tuning, whereas reflect is more robust much like reinforcement learning; and a minimal skill creator already matches the official ones, indicating that the magnitude of evolution is mainly determined by the model's own intelligence.

Critical strengths

The writing of this paper is polished, and the authors define the problem well. Existing self-evolution work has not addressed evaluation on real or enterprise-level tasks; the authors designed GDPevo to systematically measure models' self-evolution capability on enterprise-level tasks. GDPevo designs tasks starting from enterprise-level work, decomposes them into various rules, and then uses Codex to synthesize the tasks. It finds that models' self-evolution capability on enterprise-level tasks is insufficient. Some of the findings are also interesting — for example, the counter-intuitive finding that a lower starting point does not imply more evolution headroom: DeepSeek has the lowest base accuracy (43.58%) yet also the smallest gain (+5.21), while Opus has the highest base accuracy yet the largest gain.

Critical weaknesses

W1: In the method section, the authors state: "Second, when the agent first learns from the 5 training tasks and evolves before solving the test tasks—here we use fewshot, the evolution method that currently performs best (Section 4)—the test score must improve by about 0.1–0.3; this rejects task groups whose hidden rules either cannot be learned from training or are too easy to learn." That is, valid samples are filtered by whether the fewshot strategy works. The conclusion of RQ1 — that few-shot is the most effective supervision method — is therefore clearly biased.

W2: To my understanding, training and testing share the same API system/database, so it is uncertain whether the actual gains come from adaptation to, or exploration of, the environment rather than from learning the rules. The authors may need to isolate adaptation to / exploration of the environment.

W3: Lack of human validation. All hidden rules are created by agents, using public benchmarks only as inspiration seeds, whereas real unwritten enterprise rules are considerably messier. A small-scale rating of task realism by human practitioners would be very important, or at least this gap should be discussed.

W4: Several works are referred to by name but not cited; please fix these issues, e.g., line 202: ALFWorld, WebShop, HotpotQA, HumanEval. This is a weakness.

Questions for authors

The authors mainly used Codex + GPT-5.5 to synthesize the data. What is the rationale for choosing this configuration? Have any other configurations been tried, such as Claude + Claude Code?

1 comment

We thank Reviewer 7uvG for the positive assessment of our problem formulation and benchmark design.

1. Calibration Criterion and Benchmark Positioning (W1)

We agree with your observation that any calibration filter inevitably introduces some bias, and as a benchmark paper rather than a method paper, we used SFT-like fewshot to verify task validity.

  1. Constructing any benchmark requires difficulty calibration and filtering, and filtering requires choosing a specific method, which once chosen inevitably introduces some bias. To verify that the 5 training tasks actually contain the hidden rules needed for the 5 test tasks, we used the SFT-like fewshot setting (training inputs + gold answers), which provides the most complete supervision signal to directly confirm task learnability.
  2. Crucially, this is a benchmark paper rather than a method paper—our goal is not to prove that one specific evolution method is superior to another (for example, that SFT-like methods are superior to RL-like methods), but to demonstrate that GDPevo is an effective benchmark on which diverse self-evolution methods achieve clear gains (indeed, the uncalibrated RL-like reflect-3 and unsupervised self modes also achieve +4.42 to +9.37 pp gains across all models), while still leaving significant headroom.

2. Adapting to APIs/Databases as Part of Agent Evolution — The FDE Analogy (W2)

In real enterprise deployments, adapting to internal APIs and databases is itself an integral part of agent self-evolution, akin to a Forward Deployed Engineer (FDE) helping a general agent without business context mature into an evolved agent.

  1. Just as a financial firm deploying a general agent without business context relies on FDEs to help it adapt to proprietary internal docs and APIs so that it matures into an evolved agent, familiarizing with the internal environment and mastering business rules together constitute the complete agent evolution process.
  2. A concrete example of such self-evolution through environment adaptation in our benchmark is the self mode: even without answers or feedback, self gains +4.42 to +6.42 pp by autonomously exploring and adapting to the internal APIs and databases, while reflect-3 and fewshot deliver an additional +8.72 to +12.02 pp jump by further mastering the deeper implicit business rules.

3. Human Validation Trade-off and Two-Sided Safeguards (W3)

We strongly agree with your perspective: building an auto-generated benchmark involves an inherent trade-off with manual curation, which we addressed by using validated real-world seed scenarios at the input stage and conducting dedicated human inspection in the last stage.

  1. Because our goal is an auto-generated, contamination-resistant benchmark, we grounded Stage 1 input seeds directly in 15 human-verified enterprise benchmarks (e.g., GDPval, SOP-Bench, JobBench in Section 3.2 and Table 6) rather than inventing scenarios from scratch.
  2. In the end (as noted in the paper and Appendix D), each task produces bilingual rule documentation (notes/notes.md), and human operators inspected the synthesized rules to verify that they match realistic business environments. By putting human effort at the two ends (the beginning and the end), we fully automated the data generation procedure in between.

4. Missing Citations (W4)

Thank you for catching this oversight, and we have already fixed and added all four missing citations at line 202 in our newly uploaded PDF.

  • We have added formal citations for ALFWorld (Shridhar et al., 2021), WebShop (Yao et al., 2022), HotpotQA (Yang et al., 2018), and HumanEval (Chen et al., 2021) at line 202 (Section 2.2.2) of our newly uploaded PDF.

5. Choosing Codex + GPT-5.5 for Data Synthesis and Claude Code + Opus-4.8 Concurrency Limits (Q1)

We selected the most SOTA models (GPT-5.5 and Opus-4.8) to ensure benchmark quality, but strict concurrency limits on Claude Code + Opus-4.8 prevented bulk data production, so we used Claude Code + Opus-4.8 for evaluation and Codex + GPT-5.5 for data synthesis.

  1. Because benchmark quality depends directly on model capability, we initially selected the strongest SOTA models (GPT-5.5 and Opus-4.8).
  2. When using Claude Code + Opus-4.8, we encountered strict concurrency limits and operational bottlenecks that could not sustain heavy parallel data production (10 builders, 6 reviewers, and 30-container calibration runs per group), so we used Claude Code + Opus-4.8 for test evaluation and Codex + GPT-5.5 for data production. At the same time, while we agree that model capability has a significant influence on benchmark quality, according to our initial observations during small-scale trials, GPT-5.5 and Opus-4.8 lead to benchmarks of similar quality.
Reviewer_AEVcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces GDPevo, a benchmark for evaluating agent self-evolution on enterprise tasks. It distributes business rules across training tasks and recombines them in held-out tests to assess learning and transfer.

Critical strengths
  1. The benchmark addresses a relevant problem with an explicit train-test design.
  2. Deterministic scoring and cost reporting make the evaluation interpretable.
  3. The naive baseline and cross-domain experiments provide useful insights.
Critical weaknesses
  1. Calibrating tasks based on fewshot improvements may favor this supervision condition and influence subsequent comparisons.
  2. The whole methodology is mostly descriptive, with limited formalization and some of the contents read like AI generated.
  3. Cross-domain transfer uses only one task group per domain, limiting the generalizability of the findings.
  4. Results in Figure 3 and Table 1 do not report uncertainty for the key comparisons. Clearer statistical reporting would help assess whether small differences between transfer conditions or skill creators are reliable.
Questions for authors

Cross-domain results (Fig. 3) use only one group per domain and report no uncertainty. How stable are the fewshot-overfitting vs. reflect-transfer patterns, and would they hold with more groups and error bars?

3 comments

We thank Reviewer AEVc for the insightful comments.

1. Calibration Bias and Benchmark Positioning (W1)

We agree with your observation that any calibration filter inevitably introduces some bias, and as a benchmark paper rather than a method paper, we used SFT-like fewshot to verify task validity.

  1. Constructing any benchmark requires difficulty calibration and filtering, and filtering requires choosing a specific method, which once chosen inevitably introduces some bias. To verify that the 5 training tasks actually contain the hidden rules needed for the 5 test tasks, we used the SFT-like fewshot setting (training inputs + gold answers), which provides the most complete supervision signal to directly confirm task learnability.
  2. Crucially, this is a benchmark paper rather than a method paper—our goal is not to prove that one specific evolution method is superior to another (for example, that SFT-like methods are superior to RL-like methods), but to demonstrate that GDPevo is an effective benchmark on which diverse self-evolution methods achieve clear gains (indeed, the uncalibrated RL-like reflect-3 and unsupervised self modes also achieve +4.42 to +9.37 pp gains across all models), while still leaving significant headroom.

2. Mathematical Formalization and AI-Assisted Polishing (W2)

We completely agree with your point and have added formal equations in our newly uploaded PDF (also attached in Point 3, feel free to skip for now), while our text is self-explanatory and human-controlled in all logic and content.

  1. Adding equations improves rigor and we have added them in Sections 3.3 and 4.1 of our newly uploaded PDF, while our original text was already self-explanatory and non-misleading (matching your "3: Clear and easy to follow" Presentation score).
  2. We candidly used AI to polish wording, but the logic and content of the entire paper were strictly controlled by the human authors.
  3. (Feel free to skip the equations for now) Given a group's rule set RR, training tasks (i∈1,…,5i\in\\{1,\dots,5\\}) and test tasks (j∈1,…,5j\in\\{1,\dots,5\\}) satisfy Rjte⊆⋃i=15Ritr=RR^{\text{te}}_j \subseteq \bigcup_{i=1}^5 R^{\text{tr}}_i = R (Ritr⊊R,Rjte≠RitrR^{\text{tr}}_i \subsetneq R, R^{\text{te}}_j \neq R^{\text{tr}}_i), ensuring all test rules are learnable from training via recombination.

3. Expanding Cross-Domain Transfer Across All 6 Domains (W3 & Q1)

We fully agree with your suggestion and have expanded the cross-domain transfer experiment in Figure 3 to cover all 6 domains, and we have directly updated Figure 3 (Section 4.3) in our newly uploaded PDF with the complete 3×33\times 3 transfer matrices and ± STD uncertainty metrics.

  1. Specifically, our expanded evaluation consists of two 3×33\times 3 cross-domain transfer settings covering all 6 domains—Domain Group 1 (CRM, ERP, and Finance) and Domain Group 2 (Healthcare, Legal, and Data Analysis)—and within each 3×33\times 3 setting we independently sampled task groups twice (Sample 1 and Sample 2), yielding four 3×33\times 3 matrices across 12 task groups in total (tg02/06/10 and tg01/07/09 for Group 1, and tg15/20/24 and tg13/19/22 for Group 2).
  2. Across all four 3×33\times 3 matrices, all three core findings of Section 4.3 (Figure 3) hold consistently: in-domain (diagonal) evolution is consistently positive and dominant, off-diagonal fewshot overfits the source domain with negative transfer, and reflect-3 exhibits milder overfitting and generalizes better across domains than fewshot in every matrix. We attach the full tables in the follow-up comment below, and because reading multi-group tables in a comment box can be less convenient, we recommend viewing the updated Figure 3 (Section 4.3) in our newly uploaded PDF version.

4. Error Bars and Uncertainty Metrics for Figure 3 and Table 1 (W4 & Q1)

We fully adopt your suggestion and have added ± STD for all comparisons in Figure 3 and Table 1 in our newly uploaded PDF, and we also report the Table 1 ± SE and pp-values below.

CreatorGPT-5.5 / Codex ACC ± STDΔ\Delta vs. base ± SE (pp)DS-V4-Pro / Codex ACC ± STDΔ\Delta vs. base ± SE (pp)
base49.66 ± 1.72%—42.48 ± 1.21%—
CC62.15 ± 1.21%+12.49 ± 2.06 (3.6e-6)46.74 ± 1.31%+4.26 ± 1.80 (0.027)
Codex62.19 ± 0.98%+12.53 ± 2.04 (2.9e-6)47.05 ± 2.25%+4.57 ± 2.37 (0.066)
DeepAgents62.69 ± 1.76%+13.03 ± 2.20 (4.9e-6)47.75 ± 0.94%+5.27 ± 2.27 (0.029)
OpenCode60.79 ± 2.40%+11.13 ± 1.67 (8.2e-7)47.84 ± 1.91%+5.36 ± 2.03 (0.015)
Naive65.12 ± 1.66%+15.46 ± 1.94 (4.6e-8)48.01 ± 1.38%+5.54 ± 2.02 (0.012)

Follow-up to Point 3 & Point 4 (W3, W4 & Q1): Complete 3×33\times 3 Cross-Domain Transfer Tables (with ± STD)

Below are the three additional 3×33\times 3 cross-domain transfer tables for Domain Group 1 (CRM / ERP / Finance) and Domain Group 2 (Healthcare / Legal / Data Analysis). Because reading multi-group 3×33\times 3 tables here in a comment box may be less convenient, we recommend viewing the updated Figure 3 (Section 4.3) in our newly uploaded PDF version, where all four 3×33\times 3 matrices (with ± STD) are presented as a much cleaner heatmap.

1. Domain Group 1 (CRM / ERP / Finance) — Sample 2 (tg01, tg07, tg09):

Source Domain \ Target DomainEvolution ModeTarget: CRM (tg01) (base: 43.22 ± 1.77%)Target: ERP (tg07) (base: 34.23 ± 3.36%)Target: Finance (tg09) (base: 59.85 ± 3.62%)
Source: CRM (tg01)fewshot Δ\Delta (ACC ± STD)+20.49 pp (63.71 ± 1.11%)-1.67 pp (32.56 ± 1.49%)-2.76 pp (57.09 ± 3.37%)
Source: ERP (tg07)fewshot Δ\Delta (ACC ± STD)-1.25 pp (41.97 ± 1.77%)+23.83 pp (58.06 ± 1.58%)-1.24 pp (58.61 ± 2.08%)
Source: Finance (tg09)fewshot Δ\Delta (ACC ± STD)+4.28 pp (47.50 ± 5.10%)+1.67 pp (35.90 ± 2.06%)+8.12 pp (67.97 ± 6.26%)
Source: CRM (tg01)reflect-3 Δ\Delta (ACC ± STD)+26.73 pp (69.95 ± 9.23%)+0.00 pp (34.23 ± 3.36%)-0.85 pp (59.00 ± 2.18%)
Source: ERP (tg07)reflect-3 Δ\Delta (ACC ± STD)+5.71 pp (48.92 ± 5.80%)+15.87 pp (50.10 ± 5.71%)-0.12 pp (59.73 ± 2.36%)
Source: Finance (tg09)reflect-3 Δ\Delta (ACC ± STD)-1.07 pp (42.15 ± 2.02%)-1.67 pp (32.56 ± 1.49%)+13.21 pp (73.06 ± 6.22%)

2. Domain Group 2 (Healthcare / Legal / Data Analysis) — Sample 1 (tg15, tg20, tg24):

Source Domain \ Target DomainEvolution ModeTarget: Healthcare (tg15) (base: 67.65 ± 1.77%)Target: Legal (tg20) (base: 59.75 ± 2.05%)Target: Data Analysis (tg24) (base: 64.77 ± 3.69%)
Source: Healthcare (tg15)fewshot Δ\Delta (ACC ± STD)+1.30 pp (68.95 ± 1.65%)+1.84 pp (61.60 ± 2.36%)-2.61 pp (62.16 ± 0.00%)
Source: Legal (tg20)fewshot Δ\Delta (ACC ± STD)-0.42 pp (67.23 ± 2.36%)+0.73 pp (60.48 ± 1.69%)-3.16 pp (61.61 ± 0.79%)
Source: Data Analysis (tg24)fewshot Δ\Delta (ACC ± STD)+0.42 pp (68.06 ± 3.12%)-0.99 pp (58.76 ± 2.00%)+16.83 pp (81.60 ± 2.64%)
Source: Healthcare (tg15)reflect-3 Δ\Delta (ACC ± STD)+0.00 pp (67.65 ± 1.77%)+1.17 pp (60.92 ± 1.99%)+1.31 pp (66.09 ± 5.55%)
Source: Legal (tg20)reflect-3 Δ\Delta (ACC ± STD)-0.83 pp (66.81 ± 2.12%)+2.43 pp (62.18 ± 0.66%)-4.73 pp (60.04 ± 3.00%)
Source: Data Analysis (tg24)reflect-3 Δ\Delta (ACC ± STD)-1.67 pp (65.98 ± 2.12%)-0.08 pp (59.67 ± 2.72%)+7.30 pp (72.07 ± 10.58%)

3. Domain Group 2 (Healthcare / Legal / Data Analysis) — Sample 2 (tg13, tg19, tg22):

Source Domain \ Target DomainEvolution ModeTarget: Healthcare (tg13) (base: 40.11 ± 1.17%)Target: Legal (tg19) (base: 47.33 ± 1.18%)Target: Data Analysis (tg22) (base: 63.81 ± 5.61%)
Source: Healthcare (tg13)fewshot Δ\Delta (ACC ± STD)+19.63 pp (59.74 ± 4.61%)+1.57 pp (48.91 ± 1.05%)-0.25 pp (63.56 ± 7.66%)
Source: Legal (tg19)fewshot Δ\Delta (ACC ± STD)-0.89 pp (39.22 ± 1.05%)+6.09 pp (53.42 ± 0.72%)-8.64 pp (55.18 ± 3.94%)
Source: Data Analysis (tg22)fewshot Δ\Delta (ACC ± STD)+0.21 pp (40.32 ± 1.30%)-0.93 pp (46.41 ± 1.05%)+17.13 pp (80.94 ± 6.25%)
Source: Healthcare (tg13)reflect-3 Δ\Delta (ACC ± STD)+9.86 pp (49.96 ± 7.27%)+0.83 pp (48.17 ± 2.04%)+3.08 pp (66.89 ± 3.98%)
Source: Legal (tg19)reflect-3 Δ\Delta (ACC ± STD)+1.10 pp (41.21 ± 0.09%)+2.79 pp (50.12 ± 2.86%)-1.09 pp (62.72 ± 8.60%)
Source: Data Analysis (tg22)reflect-3 Δ\Delta (ACC ± STD)+0.00 pp (40.11 ± 1.17%)+1.48 pp (48.81 ± 2.23%)+8.71 pp (72.52 ± 4.80%)

After considering the other reviewers' comments and the authors' rebuttal, the main concerns have been partially addressed. However, I have decided to retain my original score.

Reviewer_NnM4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A benchmark for agent self-evolution on 240 real business tasks in 24 groups. Two design choices stand out: each task's rubric is compiled into executable checks rather than judged by an LLM, and tasks are split into "learn a new skill" versus "fix an existing wrong skill." Reported gains range from +2.59 to +16.44 pp, with 91.6% human–grader agreement.

Critical strengths

Code-based grading addresses the noisy problem of LLM-judge-based agent evaluation, which is the most transferable part of the work.

The "fix an existing skill" axis fills a genuine gap — a wrong prior rule actively interferes, and prior suites rarely test this.

Critical weaknesses

The item-selection thresholds were calibrated with the very system being evaluated. The three filters (40–60% zero-shot, 0.1–0.3 few-shot lift, <0.8 post-evolution) were measured with Codex + GPT-5.5 xhigh, which is also a subject. Retained items are therefore, by construction, ones that agent fails before and passes after evolution. No rejection statistics are reported — how many items each filter dropped, and what gains look like on the dropped pool — so I cannot separate "GDPevo measures self-evolution" from "GDPevo was filtered to show self-evolution."

There is not a single significance test, and no Limitations section. With ±5–9 pp noise, a +2.59 pp gain is indistinguishable from zero.

The tables also disagree: Table 1 base 49.66/42.48 versus Table 2 and Figure 2 at 49.37/43.58, and Table 2's few-shot 64.51 matches no row of Table 1.

Questions for authors

How many items did each filter drop, and what is the gain on the dropped pool?

In the 8.4% of human–grader disagreements, does the grader systematically over- or under-award?

1 comment

We thank Reviewer NnM4 for the constructive feedback.

1. Agent Isolation and Multi-Round Revision Statistics (W1 & Q1)

Although task construction (Stage 2), review/calibration (Stage 3), and evaluation all use Codex + GPT-5.5, they are executed by different agent instances initialized with a clean context—it is never the same agent both generating/solving tasks and grading/reviewing them—and we report the full statistics and performance of the rejected pool and per-round passes in the table below.

  1. Although both stages use Codex + GPT-5.5, each stage launches a distinct, freshly initialized agent instance with a clean context in an isolated container, so different agents perform building, solving, and reviewing independently, and it is never the same agent both solving/constructing a task and grading/reviewing it.
  2. To prevent over-editing from distorting task logic, candidate groups undergo at most 4 revision rounds, and groups still failing after Round 4 enter the rejected pool and are discarded:
Stage / OutcomeGroupsProportionMean base ACCMean fewshot ACCMean Lift
Round 1 Pass516.7%51.42%65.78%+14.36 pp
Round 2 Pass1033.3%49.35%64.62%+15.27 pp
Round 3 Pass516.7%48.86%64.50%+15.64 pp
Round 4 Pass413.3%48.15%64.60%+16.45 pp
Dropped (> Round 4)620.0%64.85%69.72%+4.87 pp

2. Whole-Benchmark 3-Run Variance, Significance Tests, and Limitations Section (W2)

We fully agree with your suggestion and have added a dedicated Limitations section in our newly uploaded PDF and report the significance tests below, and across three full benchmark runs (Run 1/2/3), the whole-benchmark standard deviation is only ±0.12 pp to ±1.38 pp, confirming that the evolution gains are highly stable.

  1. While the ICLR 2027 guidelines do not require a separate Limitations section, we agree with your suggestion and have added a dedicated Limitations section after the AI Use Statement in our newly uploaded PDF.
  2. The ±5–9 pp in Table 2 only measured within-task variation on individual problems. To measure whole-benchmark stability, the proper calculation is to run the entire benchmark three times and compute the standard deviation across the three resulting benchmark-wide mean scores (Run 1, Run 2, Run 3), which is only ±0.12 pp to ±1.38 pp (even for DS-V4-Pro self, Run 1/2/3 means are 46.25% / 45.99% / 46.26%, all above base at 42.90% / 42.48% / 45.36%):
Model / Harnessbase (Mean ± STD)self (Mean ± STD, pp-val)reflect-3 (Mean ± STD, pp-val)fewshot (Mean ± STD, pp-val)
GPT-5.5 / Codex49.37% ± 0.14 pp55.80% ± 0.83 pp (2.2×10−52.2\times 10^{-5})57.82% ± 0.50 pp (3.0×10−83.0\times 10^{-8})64.51% ± 1.09 pp (6.1×10−156.1\times 10^{-15})
Opus-4.8 / CC50.63% ± 1.05 pp55.05% ± 0.75 pp (5.3×10−35.3\times 10^{-3})59.27% ± 0.48 pp (2.1×10−62.1\times 10^{-6})67.07% ± 0.81 pp (4.1×10−124.1\times 10^{-12})
GLM-5.2 / CC46.12% ± 1.38 pp50.96% ± 0.64 pp (2.9×10−42.9\times 10^{-4})55.49% ± 0.90 pp (8.0×10−88.0\times 10^{-8})60.09% ± 1.00 pp (1.1×10−111.1\times 10^{-11})
DS-V4-Pro / CC43.58% ± 1.27 pp46.17% ± 0.12 pp (0.1150.115)47.15% ± 1.09 pp (2.7×10−22.7\times 10^{-2})48.79% ± 1.06 pp (2.7×10−32.7\times 10^{-3})

3. Numerical Differences Between Table 1 and Table 2 (W3)

Table 1 and Table 2 are two separate experimental runs, and minor differences on matching setups reflect normal run-to-run stochastic variance rather than any experimental error.

  • Table 1 (skill-creator ablation) and Table 2 (main evaluation) were conducted as two separate experiments, so where a row in Table 2 shares the exact same setup as a row in Table 1, it is an independent run of that setup, and the minor score differences simply reflect normal run-to-run stochastic variance. We have clarified this in the Table 1 caption of our newly uploaded PDF.

4. Clarifying the 91.6% Oracle Ceiling (Q2)

91.6% is not "human–grader agreement," but the Human + Codex Oracle Ceiling on the test tasks (Section 4.2 and Appendix D).

  1. This number measures the highest score achievable when a human operator and an LLM (Codex + GPT-5.5) collaborate with full access to all supervision signals (all training rules, rubrics, and gold answers). Reaching 91.6% demonstrates two things: first, our benchmark is well-posed and solvable above 90% rather than an unsolvable benchmark, and second, the strongest current agent reaches at most 67.07% after self-evolution, showing that current agent self-evolution capabilities remain limited—which is why we call 91.6% the "Oracle Ceiling."
  2. Our deterministic code graders achieve 100.0% agreement on all reference answers, and the remaining 8.4% reflects human–agent execution slips on complex long-horizon tasks rather than grader error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.