acceptodds
Under review as a conference paper at ICLR 2027

Asset or Expense? Keeping the Books of Self-Evolving Agents

Abstract

Self-evolving agents persist experience as artifacts such as playbooks, memories, or skill libraries, and are evaluated by the accuracy gained while the task stream runs. We ask what the deployer owns when evolution stops. We introduce a ledger that tags every LLM call by phase, content-addresses the artifact before measuring it, and fills four accounts: frozen held-out value, build cost, per-task carry cost, and the lifetime of the backbone the artifact depends on. A recognition gate, fixed before the closed-model runs, refuses to compute depreciation for an artifact whose value is not shown to be positive. Instantiated inside the AgentStream suite with its methods, tasks, and prompts unchanged (54 runs, 8 backbones), no artifact passed the gate; one was significantly harmful after multiple-comparison correction (GPT-5.5's own ACE playbook, −24 points on Tau2-telecom) and two more before correction. On the column with the largest reported online gain (GPT-5.4, Tau2-telecom), both the online gain we reproduce (+0.04) and the frozen value (+0.02, 95% CI [−0.06, +0.11]) are indistinguishable from zero, and on that model a length- and vocabulary-matched, syntax-destroyed copy of the playbook performs comparably. The costs, by contrast, are certain: even with 96% of prompt tokens served from cache, carrying the playbook raises per-task execution cost by 48% and overtakes the one-off build cost after about 67 tasks; online rewriting adds about one cache miss per task. Against the alternative of upgrading to the successor model, the artifact would need at least 7.7 points of value to break even. By these books every artifact we measured is an expense, and we state the four measurable conditions under which one would be an asset. We release the ledger, the gate, and a dataset of dated snapshot lifetimes (12–21 months for 2025 releases).

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-evolving agents, evaluation of LLM agents, inference cost

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DAtNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Self-evolving agents write what they learn into a persistent artifact, such as a playbook or memory store. This paper asks what the deployer actually owns once evolution stops. It proposes a bookkeeping protocol: a ledger records the cost of every model call, the artifact is frozen, and it is then judged by its value on held-out tasks, its build cost, its carry cost (the extra per-task cost of keeping it in the prompt) and its lifetime until the backbone model is retired. A recognition gate refuses to count the artifact as an asset unless its value is clearly positive. The protocol runs inside the AgentStream suite, mostly with ACE playbooks, on open-weight models and on GPT-5.4 and GPT-5.5 in Tau2-telecom, with a shuffled playbook as a placebo. No artifact passes the gate, and GPT-5.5's own playbook is harmful. The paper concludes that every measured artifact is an expense.

Critical strengths
  • The question is well posed and under-studied. Freezing the artifact and testing it on disjoint held-out tasks is the right counterfactual, and placing the online gain next to the frozen value on the same model shows why it matters.
  • The instrumentation is careful and auditable. Untagged calls leave a poison record that invalidates the run, and the ledger reconciles closely with the provider's bill.
  • Cache share by call position gives a clear account of churn (a cache miss caused by rewriting the artifact), and the snapshot-lifetime dataset is tied to a usable pay-back rule.
  • The paper shows that a token-count projection overstated carry several times over because it could not see caching. This is a useful lesson for anyone pricing agents.
Critical weaknesses
  • The abstract and the first sentence of the conclusion say every measured artifact is an expense. That holds only under the paper's convention that a refused artifact is booked as an expense. The body is more careful: it says the value account is "bounded rather than measured" and that refusal is not evidence of absence. That qualification does not reach the abstract. Most rounds sit on columns where AgentStream reports no prior gain. At these sample sizes the gate only recognizes large effects, so refusal there was close to certain.
  • The paper's no-asset GPT-5.4 baseline solves far more Tau2-telecom build tasks than AgentStream reports. The paper discloses this but cannot attribute it. Section 6 then reads the missing gain as the expected regression of a winner's-curse estimate. Regression to the mean predicts a smaller gain, not a large shift in the no-asset arm. The claim that the gap shifts both arms equally is also untested, since headroom can interact with the treatment.
  • Analysis rules are inconsistent or unstated. GPT-5.4 crash sessions are scored as failures, while a stalled GPT-5.5 session is excluded, with no rationale given. One rule should be used, with the other as a sensitivity check. The paper does not say whether the corrected or uncorrected bound governs the gate, and the Holm family omits the transplant, placebo and online-gain contrasts.
Questions for authors
  • What differs between your Tau2-telecom no-asset setup and AgentStream's? Does their exact configuration recover their baseline and reported gain?
  • Do independent GPT-5.5 builds reproduce the harm and its MMS concentration? Would any GPT-5.4 build pass the gate?
  • Is there a configuration, such as a hand-written playbook, where the gate passes and depreciation can be reported with an interval?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DAtN (Part 1/4)

Thank you for the close reading. Each of your points led to a new experiment or a correction. We give a summary first and then the details, in the order of your review. Run labels are explained where they first appear.

Summary

  • W1, the headline claim depends on a convention. The abstract now states the convention in the same sentence as the claim, and the claim no longer rests on it: four of the six closed-model artifacts are harmful, with intervals below zero.
  • W2 and Q1, the unexplained baseline. It is explained and tested. AgentStream's paper states reasoning effort medium, and its released code runs at none. Run as released, their code gives 0.44, close to the 0.50 in their paper. Under that setting the frozen playbook is harmful.
  • W3, the analysis rules. One session rule is used throughout, the bound that governs the gate is stated, and the Holm family is completed.
  • Q2, independent builds. The GPT-5.5 harm and its MMS concentration reproduce in three of three independent builds. Neither GPT-5.4 build passes the gate.
  • Q3, a configuration that passes. There is one. It is exploratory, and its limits are stated.

W1. The abstract and conclusion say every measured artifact is an expense

The abstract and the conclusion now state the convention together with the claim. The closing sentence of the abstract reads: "By these books none of the artifacts we measured is recognised as an asset, and under the gate's convention each is therefore booked as an expense: where the evidence is decisive it shows harm, and elsewhere it is inconclusive at our sample sizes." The qualification you cite from the body is now part of it.

Each artifact is reported with one of three outcomes: recognised (95% interval above zero), harmful (interval below zero), or inconclusive (interval contains zero).

As you note, refusal was close to certain in the open-weight rounds, which have one replicate on 50 tasks and sit on columns where no prior gain was reported. The revision reports them for the cost accounts and the protocol only. They are not used as evidence about value.

The value claim now rests on the two closed models, where six artifacts as shipped were measured.

  • Two are inconclusive: the two independent GPT-5.4 builds at reasoning effort medium, +0.02 [−0.05, +0.10] and +0.04 [−0.05, +0.12].
  • Three are harmful on GPT-5.5: three independent builds give −0.22 [−0.36, −0.08], −0.30 [−0.44, −0.16] and −0.18 [−0.32, −0.04], and pooled −0.23 [−0.34, −0.12].
  • One is harmful on GPT-5.4 at the reasoning setting of AgentStream's released code: −0.17 [−0.29, −0.05]. This is the column with the largest reported online gain (see W2 and Q1).

For the four harmful artifacts the outcome does not depend on the convention, because the interval itself lies below zero. We regard the replication across three independent builds as the main evidence of harm. For completeness, after Holm correction over all 46 contrasts the second GPT-5.5 build remains significant (adjusted p = 0.007) and the other three do not (0.081, 0.48 and 0.14).

AuthorsOpenReview ↗

Response to Reviewer DAtN (Part 2/4)

W2 and Q1. The baseline gap, the regression argument, headroom, and whether AgentStream's configuration recovers their baseline and gain

Thank you for raising this point. Your question led us to check our configuration against the code released with AgentStream, item by item, and that check identified the cause.

The cause is reasoning effort. Reasoning effort is the API setting that controls how many hidden reasoning tokens the model spends before it writes its answer. At none it answers directly, and at medium it reasons first. AgentStream's paper states that the model is "GPT-5.4-medium" (their Section 4), and we ran GPT-5.4 at reasoning effort medium to match their paper.

After your review we reran their baseline with the released code, as released. The only change was to disable the simulator-side cache, without which the released configuration does not run in our environment. The baseline solves 0.44 of the 50 build tasks (95% CI [0.30, 0.58]; 0.45 under their rule of dropping sessions without a result). This is close to the 0.50 in their paper. At medium the same agent solves 0.82 [0.70, 0.92], and the paired difference between the two settings is −0.38 [−0.52, −0.24].

In reproducing their baseline we found that the released code does not run GPT-5.4 at medium. It runs it at none: the released scripts leave the reasoning-effort argument unset, and our ledger records 0 reasoning tokens per task in that run, against about 2,400 per task at medium. The gap between our baseline and theirs therefore comes from an inconsistency between AgentStream's paper and its code. It does not come from an error in our runs: we followed the setting stated in their paper.

The full comparison, with concrete values:

  • Reasoning effort, agent calls. AgentStream's paper: medium. AgentStream's released code: none (the scripts leave the argument unset). Our submission: medium.
  • Reasoning effort, reflector and curator calls. Released code: none. Ours: none (same code path).
  • Temperature, agent calls. Released code: 1.0. Ours: not sent (provider default).
  • Temperature, reflector and curator. Released code: 0.0. Ours: 0.0.
  • Agent model id. Released code: alias gpt-5.4. Ours: gpt-5.4-2026-03-05. The alias resolved to this snapshot on every response.
  • User simulator. Released code: GPT-5.4, temperature 0.0. Ours: the same, dated snapshot.
  • Benchmark, tasks, trials. Released code: tau2-bench v0.1.3, 50 tasks, 1 trial. Ours: the same version, the same 50 tasks, 1 trial.
  • Step limits. Released code: 200 steps, 10 errors. Ours: the same.
  • Session without a result. Released code: dropped from the score. Ours: scored 0.
  • Client response cache. Released code: on. Ours: off.
  • API endpoint. Released code: chat completions. Ours: the same.

The inconsistency is confined to this one baseline. The GPT-5.4 Tau2-telecom column is the only AgentStream result that our paper replicates; GPT-5.5 and the open-weight models do not appear in their tables. We also recomputed the numbers of the submission's result tables (Tables 1, 3 and 8) from the archived ledgers. 32 of the 34 quantities reproduce exactly, and the other two trace to one rounded intermediate value in the upgrade-price calculation, which the revision corrects.

AuthorsOpenReview ↗

Response to Reviewer DAtN (Part 3/4)

W2 and Q1 (continued)

Does their configuration recover their baseline? Yes: 0.44 on their 50 tasks, against the 0.50 in their paper. The level is also stable across runs. The same setting gives 0.39 and 0.45 in our two no-artifact runs on the held-out tasks.

Does it recover their reported gain? We ran our full protocol at reasoning effort none and temperature 1.0 (label C1n), and checked both settings on each of the 4,067 requests.

  • Online. ACE evolving over the 50 build tasks solves 0.34, against 0.44 for the vanilla agent: −0.10 [−0.30, +0.10]. The online gain cannot be determined at this number of tasks, and this is one build. ACE solves 6 of the first 10 tasks, 5 of the next 15 and 6 of the last 25, while the playbook grows from 0 to 112 entries.
  • Frozen. On the 64 held-out tasks, with two replicates per arm, success is 0.25 with the frozen playbook and 0.42 without: −0.17 [−0.29, −0.05] (harmful; Holm-adjusted p = 0.14). The two replicates with the playbook agree on 60 of 64 tasks.

This is not a failure of the run: no session reached a step limit, no response was truncated, and every session with a result ended in the same way with and without the playbook.

The regression argument is removed. As you point out, regression of a selected estimate predicts a smaller gain and cannot produce a higher no-artifact arm. The difference in the no-artifact arm is a difference in setting.

Headroom. We also removed the sentence that the gap "shifts both arms of every paired contrast equally", which we had not tested. The frozen value is now measured at two levels of headroom on GPT-5.4. With the no-artifact arm at 0.42 (reasoning effort none) the playbook is harmful. With it at 0.82 to 0.87 (medium) two builds are inconclusive. The two settings differ in more than headroom, so this is not a clean test of the interaction. Within the medium setting we split the held-out tasks by whether the no-artifact run solved them in one replicate and estimated the effect on the other replicate: for C1 it is +0.02 [−0.07, +0.11] on solved tasks and +0.05 [−0.18, +0.29] on failed tasks. Whether the effect depends on headroom cannot be determined at this number of tasks.

What this means for the claims of the paper. They are unchanged, and the evidence for them is stronger, for three reasons. First, the submission measured the column with the largest reported gain under one reasoning setting. It is now measured under both, including the one that reproduces AgentStream's baseline, and no artifact is recognised under either. Second, under the setting of their code the result is a measured negative value and not a failed gate, so it does not depend on the convention you questioned in W1. Third, the ledger confirmed the inconsistency directly, because it records reasoning tokens for every call: 0 under their code and about 2,400 per task at medium. This is the kind of audit the protocol is meant to make possible.

Changes in the paper. The abstract reports both settings in place of "the online gain we reproduce". Section 5 gives the exact settings in place of "as in AgentStream". Finding 1 is rewritten around the two settings. The regression paragraph in Section 6 and the "shifts both arms equally" sentence are removed. The comparison above, with file and line references to the released code, is in Appendix H.2.

W3. Analysis rules are inconsistent or unstated

One session rule is now used everywhere, as you asked: a session without a valid result scores 0, for both models. Dropping such sessions is reported as a sensitivity check in Appendix H.3. The rule changes these numbers:

  • GPT-5.5 own playbook (C2). Submission: −0.24 [−0.39, −0.12], n = 49. Single rule: −0.22 [−0.36, −0.08], n = 50.
  • GPT-5.5 success without an artifact. Submission: 0.918. Single rule: 0.90.
  • GPT-5.4 playbook transplanted to GPT-5.5. Submission: 0.00 [−0.08, +0.08]. Single rule: +0.02 [−0.06, +0.12].

No gate outcome changes under either rule.

The gate uses the uncorrected two-sided 95% bound of the round. Holm-adjusted p-values are reported next to it and do not govern the gate. Section 2 now says this explicitly.

The Holm family in the submission omitted the transplant, the placebo contrast and the online-gain contrast. We now report two families and list every member in Appendix H.4.

  • Family A (11 members) is the eight rounds of the submission plus those three contrasts. In it only C2 is significant, with adjusted p = 0.029.
  • Family B (46 members) adds every contrast run during the discussion period. In it C2 is at 0.081. We do not call any discussion-period contrast pre-specified.

The revision uses 20,000 bootstrap resamples throughout, where the submission mixed 2,000 and 20,000. The construction is described in our response to Reviewer mM8x (Q2).

AuthorsOpenReview ↗

Response to Reviewer DAtN (Part 4/4)

Q2. Do independent GPT-5.5 builds reproduce the harm and its MMS concentration? Would any GPT-5.4 build pass the gate?

The harm reproduces on GPT-5.5. We rebuilt the ACE playbook from scratch twice more (C2b, C2c; C2 is the build in the submission). Each build was frozen and scored on the same 50 held-out tasks, 25 of which are MMS tasks, against a no-artifact run from the same month.

  • C2: 53 entries, with 0.68, without 0.90, difference −0.22 [−0.36, −0.08], tasks lost / gained 13 / 2, lost tasks that are MMS 13 of 13.
  • C2b: 53 entries, with 0.62, without 0.92, difference −0.30 [−0.44, −0.16], tasks lost / gained 17 / 2, lost tasks that are MMS 13 of 17.
  • C2c: 49 entries, with 0.74, without 0.92, difference −0.18 [−0.32, −0.04], tasks lost / gained 12 / 3, lost tasks that are MMS 12 of 12.

The MMS concentration reproduces: 38 of the 42 lost tasks are MMS tasks. One detail of the submission does not reproduce in every build: the lost conversations are longer with the playbook in C2 and C2c (median 113 against 87.5 messages, and 104 against 91) and shorter in C2b (48 against 73).

Neither GPT-5.4 build as shipped passes the gate. We rebuilt the playbook once more on GPT-5.4 at medium (C1b; C1 is the build in the submission). On 64 held-out tasks with two replicates per arm, C1 gives +0.02 [−0.05, +0.10] and C1b gives +0.04 [−0.05, +0.12].

One point of disclosure. Against the no-artifact runs archived in September, C1b would give +0.09 [+0.01, +0.16] and would pass. We reran the no-artifact arm in November, in the same month as C1b, and against that run C1b is inconclusive. We use the same-month run as the reference because a drift of the no-artifact level between the two months (+0.05 [−0.05, +0.14]) cannot be excluded. Both numbers are in Appendix H.1.

The harm goes with what GPT-5.5 writes for itself. We gave GPT-5.5 the two playbooks built by GPT-5.4. On the same 50 tasks it scores 0.92 and 0.96 with them, against 0.68, 0.62 and 0.74 with its own. The difference between the two groups is +0.26 [+0.16, +0.36] (Holm-adjusted p < 0.005). The GPT-5.4 playbooks are about twice as long, so length does not explain the harm. Our claim is about this difference. We do not claim that the GPT-5.4 playbooks are harmless on GPT-5.5, and we do not offer a mechanism.

Q3. Is there a configuration where the gate passes and depreciation can be reported with an interval?

There is one configuration that passes the gate, and it is exploratory. Section 6 of the submission noted that 76 of the 106 entries of the C1 playbook were never marked helpful by ACE's own curator. Keeping only the 30 entries that were marked helpful gives +0.09 [+0.03, +0.16] on GPT-5.4 at medium (two replicates), which the gate recognises. The rule uses counters from the build phase only.

The paper states its limits next to the result. We tried the rule after the main results were known. It passes on one playbook of three: the same rule gives −0.01 [−0.11, +0.09] on C1b and −0.28 [−0.42, −0.14] on C2. It is not separable from a word-shuffled placebo of the same pruned playbook (+0.04 [−0.02, +0.12]), and it does not survive Holm correction (adjusted p = 0.11). One control is consistent with the reading that the removed entries matter: on their own they give −0.16 [−0.27, −0.06], and the pruned playbook scores 0.26 higher than they do (95% CI [0.16, 0.37]; adjusted p < 0.005).

Depreciation cannot be estimated for it. Transplanted to GPT-5.5, the pruned playbook gives +0.06 [−0.02, +0.14]. GPT-5.5 already solves 0.92 of these tasks without any artifact, so the largest transplant value that could be observed is +0.08. Depreciation is a ratio of two noisy differences under a ceiling, and at 50 tasks its interval covers every bucket from retained to toxic. We report the two differences and the ceiling, and no point estimate.

Presentation

The table of gate outcomes for all rounds moves from the appendix into Section 3. Section 2 gains a table of symbols for the four accounts. Observation 2 is split into shorter paragraphs. The exploratory results are reported briefly in the text, labelled as exploratory, with the full tables in Appendix H.5. Neither the abstract nor the conclusion relies on them.

On soundness

Your soundness concerns were the convention behind the headline claim (W1), the unexplained baseline and the regression argument (W2), and the analysis rules (W3). The headline claim now states its convention and no longer rests on it. The baseline difference is traced to an inconsistency between AgentStream's paper and its code, and the protocol is run under both settings. One rule is used throughout. If anything remains open, we would be glad to address it.

Reviewer_pqPKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether artifacts produced by self-evolving agents (e.g., playbooks, memories) are still useful after evolution stops. The authors evaluate the value and cost of these artifacts once they are frozen. In the tested settings, none of the frozen artifacts shows significant positive value; keeping and using the artifacts adds cost.

Critical strengths
  • The paper asks an interesting and useful question that is different from the usual evaluation of self-evolving agents. Specifically, it asks whether an artifact that helps during evolution remains useful once it is frozen.

  • The approach (separating build and execution calls, freezing and hashing the resulting artifact, disabling further evolution, and evaluating the artifact on held-out tasks) allows a more informative comparison of performance with and without the frozen artifact.

  • The paper measures the ongoing cost of keeping and using an artifact, which is often left out of evaluation. For example, the GPT-5.4 playbook increases execution cost by 48% and the cost of carrying it exceeds its build cost after about 67 tasks.

Critical weaknesses
  • The recognition gate may be too strict. If the lower confidence bound on an artifact's value is not positive, the artifact is treated as an expense. But with only 50–64 tasks, the experiment can detect only fairly large gains. So failing the gate may simply mean that the artifact's value is uncertain.

  • Except for the main GPT-5.4 experiment, each artifact-building round has one replicate and each backbone is used for a single build. An apparently harmful playbook could therefore just come from an unlucky build.

Questions for authors

Would the conclusions change if artifacts that fail the recognition gate were treated as inconclusive rather than as expenses?

Do the results remain the same when the artifacts are rebuilt multiple times?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pqPK (Part 1/2)

Thank you for the review. We answer your two weaknesses and two questions in turn. Run labels are explained where they first appear.

W1 and Q1. Is the gate too strict, and would the conclusions change if artifacts that fail it were treated as inconclusive?

Your concern. With 50 to 64 tasks the experiment can detect only fairly large gains, so failing the gate may only mean that the value is uncertain.

What the gate can see. A failed gate at our sample sizes does mean that the value is uncertain. With one replicate on 50 tasks the 95% half-width of the difference is about 0.14. With two replicates per arm on 64 tasks it is about 0.08. A gain smaller than that cannot be recognised by our experiment.

What we changed. The revision separates three outcomes for each artifact:

  • recognised: the 95% interval of its value lies above zero;
  • harmful: the interval lies below zero;
  • inconclusive: the interval contains zero.

Do the conclusions change? The cost findings do not depend on the gate and are unchanged. For value, the closed-model evidence now consists of six artifacts as shipped. Here "with" and "without" are success rates on held-out tasks with and without the frozen playbook.

  • GPT-5.4, build in the submission (C1): with 0.84, without 0.82, difference +0.02 [−0.05, +0.10], outcome inconclusive.
  • GPT-5.4, independent rebuild (C1b): with 0.91, without 0.87, difference +0.04 [−0.05, +0.12], outcome inconclusive.
  • GPT-5.4 at the reasoning setting of AgentStream's released code (C1n): with 0.25, without 0.42, difference −0.17 [−0.29, −0.05], outcome harmful.
  • GPT-5.5, build in the submission (C2): with 0.68, without 0.90, difference −0.22 [−0.36, −0.08], outcome harmful.
  • GPT-5.5, independent rebuild (C2b): with 0.62, without 0.92, difference −0.30 [−0.44, −0.16], outcome harmful.
  • GPT-5.5, independent rebuild (C2c): with 0.74, without 0.92, difference −0.18 [−0.32, −0.04], outcome harmful.

No artifact is recognised. Two are inconclusive. For these two we cannot determine at this number of tasks whether the playbook has a small positive value, and the paper now says so. Four are harmful. For these four the conclusion does not rest on how the gate treats uncertainty, because the interval itself lies below zero.

So the statement that no artifact was recognised stands. Under the gate's accounting convention an artifact that is not recognised is still booked as an expense, and the abstract now states this convention in the same sentence as the claim. What the three outcomes add is the distinction you ask for: an inconclusive artifact is reported as inconclusive and not as evidence of no benefit. The closing sentence of the abstract is quoted in the General Response.

After Holm correction over all 46 contrasts, the adjusted p-values of the four harmful artifacts are 0.14 (C1n), 0.081 (C2), 0.007 (C2b) and 0.48 (C2c). The three GPT-5.5 builds pooled give −0.23 [−0.34, −0.12].

Changes in the paper. Section 2 defines the three outcomes. The abstract, Finding 1 and the conclusion use them. The limitations state the smallest effect our sample sizes can recognise.

AuthorsOpenReview ↗

Response to Reviewer pqPK (Part 2/2)

W2 and Q2. Do the results remain the same when the artifacts are rebuilt?

Your concern. Apart from the main GPT-5.4 experiment, each round had one replicate and each backbone a single build, so a harmful playbook could come from an unlucky build.

What we did. We rebuilt the ACE playbook from scratch twice more on GPT-5.5 (C2b and C2c in the list above) and once more on GPT-5.4 (C1b). Each build ran the same 50 build tasks, was frozen, and was scored on the same held-out tasks against a no-artifact run from the same month. We also ran the whole protocol once more on GPT-5.4 at the reasoning setting of AgentStream's released code (C1n; reasoning effort none, where the submission used medium). The two GPT-5.4 rebuilds have two replicates for each arm. Each GPT-5.5 build has one replicate, and the GPT-5.5 no-artifact run now has two.

Result. The results remain the same. On GPT-5.5 all three independent builds are harmful, with intervals below zero in each. On GPT-5.4 at reasoning effort medium both builds are inconclusive. The list above gives the numbers.

One further result that bears on your concern. We also gave GPT-5.5 the playbooks that GPT-5.4 had built. On the same 50 tasks, GPT-5.5 scores 0.26 higher with the two playbooks built by GPT-5.4 (C1 and C1b) than with the three it built for itself (95% CI [0.16, 0.36]; Holm-adjusted p < 0.005). The harm therefore goes with the content that GPT-5.5 writes for itself and not with carrying a block of text of that size. We do not offer a mechanism. Further details can be found in our response to Reviewer DAtN (Q2).

Limits. All builds processed the 50 build tasks in the same order, so the spread between builds excludes variation due to task order. Three builds per model is also a small number: the spread between the GPT-5.5 builds (SD 0.06) is about the size of the sampling error of one build (about 0.07).

Changes in the paper. The abstract describes the GPT-5.5 harm as observed in three independent builds. The table of builds is in Appendix H.1, and the limitation about a single build per backbone is updated.

I appreciate the author's response, and their efforts in addressing my questions. Distinguishing positive, harmful, and inconclusive outcomes helps frame the paper's contributions more clearly. The additional experimental results provide further insight. I maintain my positive score.

Reviewer_mM8xOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an accounting protocol for persistent artifacts produced by self-evolving agents. It separates frozen held-out value, build cost, recurring carry cost, and backbone lifetime through phase-tagged call ledgers and content-addressed snapshots. Across eight experimental rounds, mainly using ACE with one completed ReasoningBank round, none of the sampled artifacts passes the conservative asset-recognition gate. GPT-5.5’s own ACE playbook significantly reduces performance after Holm correction. The closed-model experiments additionally quantify cache-aware deployment costs and illustrate break-even economics relative to upgrading the backbone. These findings characterize the sampled artifacts and configurations, rather than establishing that persistent memory is generally ineffective.

Critical strengths
  1. A useful, operational evaluation protocol. Separating a frozen artifact from ongoing evolution addresses an important ambiguity in stream-level evaluations. Disjoint build/held-out tasks, content-addressed snapshots, disabled evolution during evaluation, and explicit call attribution make the evaluation object auditable (§2; Appendix C). This reusable protocol is the strongest contribution.
  2. Concrete cache-aware deployment measurements. The paper shows why token counts alone can misrepresent financial overhead: despite 96% cached prompt tokens, carrying the GPT-5.4 playbook adds approximately $0.056 per task, or 48% of baseline execution cost (Table 1; §5). The separation of build, carry, and rewriting costs provides useful information for deployment decisions.
Critical weaknesses
  1. Uncertainty across independently built artifacts is not characterized. Each backbone uses a single build, and most rounds have only one evaluation replicate (§8; Table 3). Task-level intervals therefore do not capture variability across build trajectories. The harmful GPT-5.5 result is compelling for this particular playbook, but does not establish typical harm from the method. Conclusions should remain explicitly conditional on the sampled artifacts; failing the recognition gate should not be presented as evidence of no benefit.
  2. The motivating replication operates at an unexplained different baseline. The reproduced GPT-5.4 no-asset baseline is 74%, compared with AgentStream’s reported 50%, and the online gain is approximately four rather than thirteen percentage points (§5, Finding 1; Table 4). This does not invalidate the within-study paired contrast, but changes the operating point and weakens an interpretation that the earlier gain was simply overestimated. A configuration/task-population reconciliation, or more limited replication claims, is needed.
Questions for authors

These are clarification requests that could materially affect the assessment:

  1. What task identities, versions, simulator settings, or aggregation differences explain the 74% versus 50% baseline? Are the reproduced and cited online comparisons otherwise directly comparable?
  2. How are the bootstrap two-sided p-values constructed and Holm-corrected? How are C1’s two replicates clustered, and should intervals after its post-result 14-task extension be interpreted as exploratory?
  3. Is the upgrade-price calculation restricted to the same tasks for both models? Table 1 reports GPT-5.4 over 64 tasks and GPT-5.5 over 49. Please provide a common-task comparison and uncertainty for the approximately $0.73 per additional success underlying the 7.7-point break-even illustration.
4 comments
AuthorsOpenReview ↗

Response to Reviewer mM8x (Part 1/4)

Thank you for the detailed review and for the specific clarification requests. We answer in the order of your points. Run labels are explained where they first appear.

W1. Uncertainty across independently built artifacts

Your concern. Each backbone had one build, so the task-level intervals do not reflect variation between build trajectories, and the harmful GPT-5.5 result could be specific to one playbook.

What we did. We rebuilt the ACE playbook from scratch twice more on GPT-5.5 (labels C2b and C2c; C2 is the build in the submission) and once more on GPT-5.4 (label C1b; C1 is the build in the submission). Each build ran the same 50 build tasks, was frozen, and was scored on the same held-out tasks against a no-artifact run from the same month.

Result on GPT-5.5 (50 held-out tasks; success with the playbook minus success without it):

  • C2 (submission): with 0.68, without 0.90, difference −0.22 [−0.36, −0.08], Holm-adjusted p 0.081.
  • C2b: with 0.62, without 0.92, difference −0.30 [−0.44, −0.16], Holm-adjusted p 0.007.
  • C2c: with 0.74, without 0.92, difference −0.18 [−0.32, −0.04], Holm-adjusted p 0.48.

All three intervals lie below zero. Pooling the three builds against the mean of the two no-artifact runs gives −0.23 [−0.34, −0.12]. The Holm column corrects over all 46 contrasts of the submission and the discussion period together. Under that correction C2b remains significant and C2 and C2c do not. In the smaller family of 11 contrasts built around the rounds of the submission (family A, defined under Q2), C2 is at 0.029.

The spread between the three builds (SD 0.06) is about the same size as the sampling error of a single build (bootstrap SE about 0.07). At this number of tasks we cannot separate build-to-build variation from task sampling noise.

Result on GPT-5.4 (64 held-out tasks, two replicates per arm): C1 gives +0.02 [−0.05, +0.10] and C1b gives +0.04 [−0.05, +0.12]. Both are inconclusive. Neither build is recognised by the gate.

A limit we state in the paper. All five builds processed the 50 build tasks in the same order. The spread between builds therefore excludes variation due to task order and should be read as a lower bound.

Changes in the paper. We followed your suggestion on wording. Conclusions are stated as conditional on the artifacts we sampled. Failing the gate is no longer presented as evidence of no benefit: each artifact is reported as recognised, inconclusive, or harmful, and the convention behind the word "expense" is stated in the same sentence as the claim. The abstract now describes the GPT-5.5 harm as observed in three independent builds, and the limitation that "a harmful playbook could be one unlucky trajectory" is updated accordingly. The full table of builds, including the task-category breakdown of the losses, is in Appendix H.1. Further details can be found in our response to Reviewer DAtN (Q2).

AuthorsOpenReview ↗

Response to Reviewer mM8x (Part 2/4)

W2 and Q1. The 74% versus 50% baseline

Thank you for raising this point. Your question led us to check our configuration against the code released with AgentStream, item by item, and that check identified the cause.

Your concern. Our no-artifact baseline is 74% where AgentStream reports 50%, and our online gain is about four points where they report thirteen. You note that this does not affect the paired contrast within our study, but it changes the operating point and weakens the reading that the earlier gain was overestimated. You ask which differences explain the gap.

What explains it. The cause is reasoning effort. Reasoning effort is the API setting that controls how many hidden reasoning tokens the model spends before it writes its answer. At none it answers directly, and at medium it reasons first. AgentStream's paper states that the model is "GPT-5.4-medium" (their Section 4), and we ran GPT-5.4 at reasoning effort medium to match their paper.

After your review we reran their baseline with the released code, as released. The only change was to disable the simulator-side cache, without which the released configuration does not run in our environment. The baseline solves 0.44 of the 50 build tasks (95% CI [0.30, 0.58]), which is close to the 0.50 in their paper.

In reproducing their baseline we found that the released code does not run GPT-5.4 at medium. It runs it at none: the released scripts leave the reasoning-effort argument unset, and our ledger records 0 reasoning tokens per task in that run, against about 2,400 per task at medium. The gap between our baseline and theirs therefore comes from an inconsistency between AgentStream's paper and its code. It does not come from an error in our runs: we followed the setting stated in their paper.

On the items you list: the task identities are the same 50 tasks, the benchmark is tau2-bench v0.1.3 in both, and the user simulator is GPT-5.4 at temperature 0 in both. Aggregation differs in one respect. Their script drops a session that ends without a result and ours scores it 0. This changes the number below by 0.01.

What we measured (same 50 build tasks, no artifact):

  • Ours, reasoning effort medium, ACE scaffold with an empty playbook (submission): 0.74 [0.62, 0.86].
  • Ours, reasoning effort medium, vanilla tool-calling agent: 0.82 [0.70, 0.92].
  • AgentStream's released code, as released (reasoning effort none, temperature 1.0), vanilla agent: 0.44 [0.30, 0.58]; 0.45 under their aggregation.
  • AgentStream's paper: 0.50.

On paired tasks the released code scores 0.38 lower than our medium setting (95% CI [0.24, 0.52]).

AuthorsOpenReview ↗

Response to Reviewer mM8x (Part 3/4)

W2 and Q1 (continued)

The inconsistency is confined to this one baseline. We now report GPT-5.4 under both reasoning settings, for the baseline and for our full protocol. The GPT-5.4 Tau2-telecom column is the only AgentStream result that our paper replicates; GPT-5.5 and the open-weight models do not appear in their tables. We checked the remaining configuration items against the released code (model, tasks, benchmark version, simulator, step limits, temperatures, and the reflector and curator calls), and they agree, apart from the differences we list here and in our response to Reviewer DAtN (W2 and Q1). We also recomputed the numbers of the submission's result tables (Tables 1, 3 and 8) from the archived ledgers. 32 of the 34 quantities reproduce exactly, and the other two trace to one rounded intermediate value in the upgrade-price calculation, which the revision corrects (see Q3).

Are the online comparisons comparable? At medium they are not, for the reason you give. At reasoning effort none we therefore ran ACE on the same 50 build tasks (label C1n). Its online success is 0.34 against 0.44 for the vanilla agent, a difference of −0.10 [−0.30, +0.10]. The online gain cannot be determined at this number of tasks, and this is a single build. At medium the two builds give +0.04 [−0.08, +0.16] (C1) and +0.16 [+0.02, +0.30] (C1b, built in November and compared with the September run) against the empty-playbook scaffold, and −0.04 [−0.18, +0.10] and +0.08 [−0.06, +0.22] against the vanilla agent.

Three differences from their code remain in the C1n run. We use the dated snapshot where they use the alias; the alias resolved to the same snapshot on every response. We score sessions without a result as 0. And the client cache is disabled, as in every run of our protocol.

The frozen value at reasoning effort none. We froze the C1n playbook and scored the 64 held-out tasks twice with it and twice without it. Success is 0.25 with the playbook and 0.42 without, a difference of −0.17 [−0.29, −0.05] (harmful by the gate; Holm-adjusted p = 0.14).

On the interpretation. We have withdrawn it. Section 6 of the submission read our missing gain as the regression of an estimate selected for its size. Now that the difference is traced to a setting, that reading is not supported, and the paragraph is removed. We make no claim that the earlier gain was overestimated.

Changes in the paper. The replication claim is narrowed as you suggest. The abstract no longer says "the online gain we reproduce" and instead reports both settings. The opening of Section 5 gives the exact settings in place of "as in AgentStream". Finding 1 is rewritten around the two settings. The table of measured accounts gains a row for reasoning effort none, and the table of outcomes for all rounds, now in Section 3, lists the new rounds. The limitation about the 74% baseline is replaced. The item-by-item configuration comparison is in Appendix H.2. Further details can be found in our response to Reviewer DAtN (W2 and Q1).

AuthorsOpenReview ↗

Response to Reviewer mM8x (Part 4/4)

Q2. Bootstrap p-values, Holm correction, clustering of C1's replicates, and the 14-task extension

Construction. The unit of resampling is the task. For each contrast we first compute one difference per task: the mean score over the replicates with the artifact minus the mean score over the replicates without it. We then resample tasks with replacement 20,000 times and take the percentile interval. C1's two replicates are therefore averaged within each task before resampling. They never count as independent tasks.

p-values. The two-sided p-value comes from the same resamples: twice the smaller of the share of resampled means at or below zero and the share at or above zero, with a floor of 0.0001.

Holm correction. We use the Holm step-down procedure and report two families, both listed member by member in Appendix H.4.

  • Family A has 11 members: the eight rounds of the submission plus the three contrasts that Reviewer DAtN noted were missing (the transplant of C1 to GPT-5.5, C1 against its placebo, and C1's online gain). In this family only C2 is significant (adjusted p = 0.029).
  • Family B has 46 members: family A plus every contrast added during the discussion period. In this family C2 is at 0.081 and the second GPT-5.5 build, C2b, is at 0.007.

The gate itself uses the uncorrected 95% bound of the round. Holm-adjusted values are reported next to it and do not govern the gate. Section 2 now says this explicitly.

One correction. The submission used 2,000 resamples for some rows and 20,000 for others. The revision uses 20,000 throughout. This moves C1's interval from [−0.06, +0.11] to [−0.05, +0.10].

The 14-task extension. Yes, it should be read as exploratory. We decided on the extension after seeing the 50-task result, so the 64-task interval for C1 is not a pre-planned analysis. We report both. On the first 50 tasks C1 gives +0.05 [−0.04, +0.14], and on all 64 tasks it gives +0.02 [−0.05, +0.10]. The outcome is inconclusive either way. The paper now labels the 64-task interval for C1 in this way.

The GPT-5.4 runs added during the discussion period all used the 64 tasks, and that set was fixed before they were run. For those runs Appendix H.1 also gives the result on the first 50 tasks. For example, C1b gives +0.05 [−0.05, +0.15] on 50 tasks, and C1n gives −0.21 [−0.34, −0.08].

Q3. Is the upgrade price computed on the same tasks, and how uncertain is it?

In the submission it was not. The GPT-5.4 side used one replicate on its 50 tasks and the GPT-5.5 side used 49 tasks.

Common-task comparison. 49 held-out tasks have a session for both models. On these tasks, with no artifact:

  • GPT-5.4 (mean of two replicates): success 0.796, execute cost per task USD 0.112.
  • GPT-5.5: success 0.918, execute cost per task USD 0.188.

The upgrade price is p = USD 0.62 per additional solved task, with a 95% interval of [USD 0.36, USD 1.96]. The interval is wide because the denominator is a 12-point difference in success measured on 49 tasks. In 94 of the 20,000 resamples that difference is zero or negative and the price is undefined; the interval is taken over the remaining resamples.

Effect on the break-even illustration. With p = USD 0.62 and a carry cost of USD 0.056 per task, the playbook breaks even at a value of 0.090, about 9 points where the submission said 7.7. Using the two ends of the price interval, the break-even value ranges from about 3 to about 15 points. For C1 the margin at the point estimate is USD −0.041 per task. At the upper end of C1's value interval the margin is USD +0.007 per task.

The reading of the submission is unchanged: the measured value of C1 (+0.02) is well below the break-even value. The wide interval on the price means the calculation is an illustration and not a precise threshold.

Changes in the paper. Finding 6 and the table of derived quantities now use the common-task price and report its interval. The abstract replaces "at least 7.7 points" with "about 9 points" and labels the figure as illustrative. While recomputing this table we also found that the printed price of USD 0.73 had used a rounded intermediate value; the appendix records this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful reviews. All three identify the protocol (a phase-tagged ledger, a frozen content-addressed artifact, held-out value, build and carry cost, backbone lifetime) and the cost measurements as the contribution of the paper. Their questions about the value results overlap on three points. We summarise here how we handled each one and answer every point in the individual responses.

  1. Uncertainty across independently built artifacts (mM8x W1; pqPK W2 and Q2; DAtN Q2). The submission had a single build per backbone. During the discussion period we rebuilt the artifact from scratch on both closed models and evaluated each new build with the same protocol. The harm on GPT-5.5 and the absence of a recognised artifact on GPT-5.4 both hold across the independent builds. Further details can be found in our response to Reviewer DAtN (Q2).

  2. The baseline difference from AgentStream (mM8x W2 and Q1; DAtN W2 and Q1). We identified the cause. AgentStream's paper states that GPT-5.4 was run at reasoning effort medium, and we ran it at medium accordingly. The code released with AgentStream leaves the reasoning-effort argument unset, so it runs GPT-5.4 at reasoning effort none. We confirmed this by rerunning their baseline with the released code, which gives a baseline close to the one in their paper. The difference therefore comes from an inconsistency between AgentStream's paper and its code, and not from our runs. We now report GPT-5.4 under both reasoning settings, for the baseline and for our full protocol. The inconsistency is confined to this one baseline: the GPT-5.4 Tau2-telecom column is the only AgentStream result that our paper replicates. Further details can be found in our response to Reviewer DAtN (W2 and Q1).

  3. What failing the gate means (pqPK W1 and Q1; DAtN W1; mM8x W1). The revision reports three outcomes for each artifact: recognised, inconclusive, or harmful. The accounting convention behind the word "expense" is now stated in the same sentence as the claim. The closing sentence of the abstract reads: "By these books none of the artifacts we measured is recognised as an asset, and under the gate's convention each is therefore booked as an expense: where the evidence is decisive it shows harm, and elsewhere it is inconclusive at our sample sizes."

We also adopted a single analysis rule throughout, as Reviewer DAtN asked (W3), and we describe the bootstrap and the Holm families in full in our response to Reviewer mM8x (Q2).

With the added results, the contribution and the claims of the paper are unchanged, and the evidence for them is stronger than in the submission. The column with the largest reported gain is now measured under both reasoning settings, and no artifact is recognised under either. The harm on GPT-5.5 is reproduced in independent builds. The value claim no longer rests on the gate's convention. We explain each of these points in our response to Reviewer DAtN.

All runs added during the discussion period are labelled as such in the paper, and each is reported whichever way it came out. Changes in the revised PDF are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.