- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether agents can consolidate experience accumulated from previous tasks into persistent state (e.g., memory, skills, prompts, harness code) and reuse it on subsequent related tasks. To address three limitations of existing benchmarks — insufficient coverage of economically valuable enterprise-level tasks, train–test splits inherited from random partitions that make post-evolution gains non-attributable, and the vulnerability of static public task sets to data contamination — the authors propose GDPevo, an evolution-native benchmark together with a highly automated construction pipeline. Its core mechanism, rule hybridization, decomposes enterprise workflows into atomic business rules that do not exist in the model's world knowledge, distributes subsets of these rules across 5 training tasks, and recombines them in 5 held-out test tasks, so that test-time gains can be attributed to learning and compositionally applying the rules. The benchmark is constructed by agents through three stages — seed scenario discovery, task group generation, and calibration with quality filtering — covering CRM, ERP, finance, healthcare, legal, and data analysis, and is equipped with deterministic code-based graders while treating cost as a first-class metric. The experiments evaluate four agents under four supervision types in a skill-based evolution setting, and find that: any supervision brings gains of 2.59–16.44pp, with fewshot being the strongest while also reducing test-time cost, yet the best result remains far below the 91.6% human–AI collaboration reference; fewshot overfits in cross-domain transfer much like supervised fine-tuning, whereas reflect is more robust much like reinforcement learning; and a minimal skill creator already matches the official ones, indicating that the magnitude of evolution is mainly determined by the model's own intelligence.
Critical strengths
The writing of this paper is polished, and the authors define the problem well. Existing self-evolution work has not addressed evaluation on real or enterprise-level tasks; the authors designed GDPevo to systematically measure models' self-evolution capability on enterprise-level tasks. GDPevo designs tasks starting from enterprise-level work, decomposes them into various rules, and then uses Codex to synthesize the tasks. It finds that models' self-evolution capability on enterprise-level tasks is insufficient. Some of the findings are also interesting — for example, the counter-intuitive finding that a lower starting point does not imply more evolution headroom: DeepSeek has the lowest base accuracy (43.58%) yet also the smallest gain (+5.21), while Opus has the highest base accuracy yet the largest gain.
Critical weaknesses
W1: In the method section, the authors state: "Second, when the agent first learns from the 5 training tasks and evolves before solving the test tasks—here we use fewshot, the evolution method that currently performs best (Section 4)—the test score must improve by about 0.1–0.3; this rejects task groups whose hidden rules either cannot be learned from training or are too easy to learn." That is, valid samples are filtered by whether the fewshot strategy works. The conclusion of RQ1 — that few-shot is the most effective supervision method — is therefore clearly biased.
W2: To my understanding, training and testing share the same API system/database, so it is uncertain whether the actual gains come from adaptation to, or exploration of, the environment rather than from learning the rules. The authors may need to isolate adaptation to / exploration of the environment.
W3: Lack of human validation. All hidden rules are created by agents, using public benchmarks only as inspiration seeds, whereas real unwritten enterprise rules are considerably messier. A small-scale rating of task realism by human practitioners would be very important, or at least this gap should be discussed.
W4: Several works are referred to by name but not cited; please fix these issues, e.g., line 202: ALFWorld, WebShop, HotpotQA, HumanEval. This is a weakness.
Questions for authors
The authors mainly used Codex + GPT-5.5 to synthesize the data. What is the rationale for choosing this configuration? Have any other configurations been tried, such as Claude + Claude Code?
We thank Reviewer 7uvG for the positive assessment of our problem formulation and benchmark design.
1. Calibration Criterion and Benchmark Positioning (W1)
We agree with your observation that any calibration filter inevitably introduces some bias, and as a benchmark paper rather than a method paper, we used SFT-like
fewshotto verify task validity.fewshotsetting (training inputs + gold answers), which provides the most complete supervision signal to directly confirm task learnability.reflect-3and unsupervisedselfmodes also achieve+4.42to+9.37 ppgains across all models), while still leaving significant headroom.2. Adapting to APIs/Databases as Part of Agent Evolution — The FDE Analogy (W2)
In real enterprise deployments, adapting to internal APIs and databases is itself an integral part of agent self-evolution, akin to a Forward Deployed Engineer (FDE) helping a general agent without business context mature into an evolved agent.
selfmode: even without answers or feedback,selfgains+4.42to+6.42 ppby autonomously exploring and adapting to the internal APIs and databases, whilereflect-3andfewshotdeliver an additional+8.72to+12.02 ppjump by further mastering the deeper implicit business rules.3. Human Validation Trade-off and Two-Sided Safeguards (W3)
We strongly agree with your perspective: building an auto-generated benchmark involves an inherent trade-off with manual curation, which we addressed by using validated real-world seed scenarios at the input stage and conducting dedicated human inspection in the last stage.
notes/notes.md), and human operators inspected the synthesized rules to verify that they match realistic business environments. By putting human effort at the two ends (the beginning and the end), we fully automated the data generation procedure in between.4. Missing Citations (W4)
Thank you for catching this oversight, and we have already fixed and added all four missing citations at line 202 in our newly uploaded PDF.
5. Choosing
Codex + GPT-5.5for Data Synthesis andClaude Code + Opus-4.8Concurrency Limits (Q1)We selected the most SOTA models (
GPT-5.5andOpus-4.8) to ensure benchmark quality, but strict concurrency limits onClaude Code + Opus-4.8prevented bulk data production, so we usedClaude Code + Opus-4.8for evaluation andCodex + GPT-5.5for data synthesis.GPT-5.5andOpus-4.8).Claude Code + Opus-4.8, we encountered strict concurrency limits and operational bottlenecks that could not sustain heavy parallel data production (10 builders, 6 reviewers, and 30-container calibration runs per group), so we usedClaude Code + Opus-4.8for test evaluation andCodex + GPT-5.5for data production. At the same time, while we agree that model capability has a significant influence on benchmark quality, according to our initial observations during small-scale trials,GPT-5.5andOpus-4.8lead to benchmarks of similar quality.