- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Self-evolving agents write what they learn into a persistent artifact, such as a playbook or memory store. This paper asks what the deployer actually owns once evolution stops. It proposes a bookkeeping protocol: a ledger records the cost of every model call, the artifact is frozen, and it is then judged by its value on held-out tasks, its build cost, its carry cost (the extra per-task cost of keeping it in the prompt) and its lifetime until the backbone model is retired. A recognition gate refuses to count the artifact as an asset unless its value is clearly positive. The protocol runs inside the AgentStream suite, mostly with ACE playbooks, on open-weight models and on GPT-5.4 and GPT-5.5 in Tau2-telecom, with a shuffled playbook as a placebo. No artifact passes the gate, and GPT-5.5's own playbook is harmful. The paper concludes that every measured artifact is an expense.
Critical strengths
- The question is well posed and under-studied. Freezing the artifact and testing it on disjoint held-out tasks is the right counterfactual, and placing the online gain next to the frozen value on the same model shows why it matters.
- The instrumentation is careful and auditable. Untagged calls leave a poison record that invalidates the run, and the ledger reconciles closely with the provider's bill.
- Cache share by call position gives a clear account of churn (a cache miss caused by rewriting the artifact), and the snapshot-lifetime dataset is tied to a usable pay-back rule.
- The paper shows that a token-count projection overstated carry several times over because it could not see caching. This is a useful lesson for anyone pricing agents.
Critical weaknesses
- The abstract and the first sentence of the conclusion say every measured artifact is an expense. That holds only under the paper's convention that a refused artifact is booked as an expense. The body is more careful: it says the value account is "bounded rather than measured" and that refusal is not evidence of absence. That qualification does not reach the abstract. Most rounds sit on columns where AgentStream reports no prior gain. At these sample sizes the gate only recognizes large effects, so refusal there was close to certain.
- The paper's no-asset GPT-5.4 baseline solves far more Tau2-telecom build tasks than AgentStream reports. The paper discloses this but cannot attribute it. Section 6 then reads the missing gain as the expected regression of a winner's-curse estimate. Regression to the mean predicts a smaller gain, not a large shift in the no-asset arm. The claim that the gap shifts both arms equally is also untested, since headroom can interact with the treatment.
- Analysis rules are inconsistent or unstated. GPT-5.4 crash sessions are scored as failures, while a stalled GPT-5.5 session is excluded, with no rationale given. One rule should be used, with the other as a sensitivity check. The paper does not say whether the corrected or uncorrected bound governs the gate, and the Holm family omits the transplant, placebo and online-gain contrasts.
Questions for authors
- What differs between your Tau2-telecom no-asset setup and AgentStream's? Does their exact configuration recover their baseline and reported gain?
- Do independent GPT-5.5 builds reproduce the harm and its MMS concentration? Would any GPT-5.4 build pass the gate?
- Is there a configuration, such as a hand-written playbook, where the gate passes and depreciation can be reported with an interval?
4 comments
Response to Reviewer DAtN (Part 2/4)
W2 and Q1. The baseline gap, the regression argument, headroom, and whether AgentStream's configuration recovers their baseline and gain
Thank you for raising this point. Your question led us to check our configuration against the code released with AgentStream, item by item, and that check identified the cause.
The cause is reasoning effort. Reasoning effort is the API setting that controls how many hidden reasoning tokens the model spends before it writes its answer. At none it answers directly, and at medium it reasons first. AgentStream's paper states that the model is "GPT-5.4-medium" (their Section 4), and we ran GPT-5.4 at reasoning effort medium to match their paper.
After your review we reran their baseline with the released code, as released. The only change was to disable the simulator-side cache, without which the released configuration does not run in our environment. The baseline solves 0.44 of the 50 build tasks (95% CI [0.30, 0.58]; 0.45 under their rule of dropping sessions without a result). This is close to the 0.50 in their paper. At medium the same agent solves 0.82 [0.70, 0.92], and the paired difference between the two settings is −0.38 [−0.52, −0.24].
In reproducing their baseline we found that the released code does not run GPT-5.4 at medium. It runs it at none: the released scripts leave the reasoning-effort argument unset, and our ledger records 0 reasoning tokens per task in that run, against about 2,400 per task at medium. The gap between our baseline and theirs therefore comes from an inconsistency between AgentStream's paper and its code. It does not come from an error in our runs: we followed the setting stated in their paper.
The full comparison, with concrete values:
- Reasoning effort, agent calls. AgentStream's paper: medium. AgentStream's released code: none (the scripts leave the argument unset). Our submission: medium.
- Reasoning effort, reflector and curator calls. Released code: none. Ours: none (same code path).
- Temperature, agent calls. Released code: 1.0. Ours: not sent (provider default).
- Temperature, reflector and curator. Released code: 0.0. Ours: 0.0.
- Agent model id. Released code: alias gpt-5.4. Ours: gpt-5.4-2026-03-05. The alias resolved to this snapshot on every response.
- User simulator. Released code: GPT-5.4, temperature 0.0. Ours: the same, dated snapshot.
- Benchmark, tasks, trials. Released code: tau2-bench v0.1.3, 50 tasks, 1 trial. Ours: the same version, the same 50 tasks, 1 trial.
- Step limits. Released code: 200 steps, 10 errors. Ours: the same.
- Session without a result. Released code: dropped from the score. Ours: scored 0.
- Client response cache. Released code: on. Ours: off.
- API endpoint. Released code: chat completions. Ours: the same.
The inconsistency is confined to this one baseline. The GPT-5.4 Tau2-telecom column is the only AgentStream result that our paper replicates; GPT-5.5 and the open-weight models do not appear in their tables. We also recomputed the numbers of the submission's result tables (Tables 1, 3 and 8) from the archived ledgers. 32 of the 34 quantities reproduce exactly, and the other two trace to one rounded intermediate value in the upgrade-price calculation, which the revision corrects.
Response to Reviewer DAtN (Part 3/4)
W2 and Q1 (continued)
Does their configuration recover their baseline? Yes: 0.44 on their 50 tasks, against the 0.50 in their paper. The level is also stable across runs. The same setting gives 0.39 and 0.45 in our two no-artifact runs on the held-out tasks.
Does it recover their reported gain? We ran our full protocol at reasoning effort none and temperature 1.0 (label C1n), and checked both settings on each of the 4,067 requests.
- Online. ACE evolving over the 50 build tasks solves 0.34, against 0.44 for the vanilla agent: −0.10 [−0.30, +0.10]. The online gain cannot be determined at this number of tasks, and this is one build. ACE solves 6 of the first 10 tasks, 5 of the next 15 and 6 of the last 25, while the playbook grows from 0 to 112 entries.
- Frozen. On the 64 held-out tasks, with two replicates per arm, success is 0.25 with the frozen playbook and 0.42 without: −0.17 [−0.29, −0.05] (harmful; Holm-adjusted p = 0.14). The two replicates with the playbook agree on 60 of 64 tasks.
This is not a failure of the run: no session reached a step limit, no response was truncated, and every session with a result ended in the same way with and without the playbook.
The regression argument is removed. As you point out, regression of a selected estimate predicts a smaller gain and cannot produce a higher no-artifact arm. The difference in the no-artifact arm is a difference in setting.
Headroom. We also removed the sentence that the gap "shifts both arms of every paired contrast equally", which we had not tested. The frozen value is now measured at two levels of headroom on GPT-5.4. With the no-artifact arm at 0.42 (reasoning effort none) the playbook is harmful. With it at 0.82 to 0.87 (medium) two builds are inconclusive. The two settings differ in more than headroom, so this is not a clean test of the interaction. Within the medium setting we split the held-out tasks by whether the no-artifact run solved them in one replicate and estimated the effect on the other replicate: for C1 it is +0.02 [−0.07, +0.11] on solved tasks and +0.05 [−0.18, +0.29] on failed tasks. Whether the effect depends on headroom cannot be determined at this number of tasks.
What this means for the claims of the paper. They are unchanged, and the evidence for them is stronger, for three reasons. First, the submission measured the column with the largest reported gain under one reasoning setting. It is now measured under both, including the one that reproduces AgentStream's baseline, and no artifact is recognised under either. Second, under the setting of their code the result is a measured negative value and not a failed gate, so it does not depend on the convention you questioned in W1. Third, the ledger confirmed the inconsistency directly, because it records reasoning tokens for every call: 0 under their code and about 2,400 per task at medium. This is the kind of audit the protocol is meant to make possible.
Changes in the paper. The abstract reports both settings in place of "the online gain we reproduce". Section 5 gives the exact settings in place of "as in AgentStream". Finding 1 is rewritten around the two settings. The regression paragraph in Section 6 and the "shifts both arms equally" sentence are removed. The comparison above, with file and line references to the released code, is in Appendix H.2.
W3. Analysis rules are inconsistent or unstated
One session rule is now used everywhere, as you asked: a session without a valid result scores 0, for both models. Dropping such sessions is reported as a sensitivity check in Appendix H.3. The rule changes these numbers:
- GPT-5.5 own playbook (C2). Submission: −0.24 [−0.39, −0.12], n = 49. Single rule: −0.22 [−0.36, −0.08], n = 50.
- GPT-5.5 success without an artifact. Submission: 0.918. Single rule: 0.90.
- GPT-5.4 playbook transplanted to GPT-5.5. Submission: 0.00 [−0.08, +0.08]. Single rule: +0.02 [−0.06, +0.12].
No gate outcome changes under either rule.
The gate uses the uncorrected two-sided 95% bound of the round. Holm-adjusted p-values are reported next to it and do not govern the gate. Section 2 now says this explicitly.
The Holm family in the submission omitted the transplant, the placebo contrast and the online-gain contrast. We now report two families and list every member in Appendix H.4.
- Family A (11 members) is the eight rounds of the submission plus those three contrasts. In it only C2 is significant, with adjusted p = 0.029.
- Family B (46 members) adds every contrast run during the discussion period. In it C2 is at 0.081. We do not call any discussion-period contrast pre-specified.
The revision uses 20,000 bootstrap resamples throughout, where the submission mixed 2,000 and 20,000. The construction is described in our response to Reviewer mM8x (Q2).
Response to Reviewer DAtN (Part 4/4)
Q2. Do independent GPT-5.5 builds reproduce the harm and its MMS concentration? Would any GPT-5.4 build pass the gate?
The harm reproduces on GPT-5.5. We rebuilt the ACE playbook from scratch twice more (C2b, C2c; C2 is the build in the submission). Each build was frozen and scored on the same 50 held-out tasks, 25 of which are MMS tasks, against a no-artifact run from the same month.
- C2: 53 entries, with 0.68, without 0.90, difference −0.22 [−0.36, −0.08], tasks lost / gained 13 / 2, lost tasks that are MMS 13 of 13.
- C2b: 53 entries, with 0.62, without 0.92, difference −0.30 [−0.44, −0.16], tasks lost / gained 17 / 2, lost tasks that are MMS 13 of 17.
- C2c: 49 entries, with 0.74, without 0.92, difference −0.18 [−0.32, −0.04], tasks lost / gained 12 / 3, lost tasks that are MMS 12 of 12.
The MMS concentration reproduces: 38 of the 42 lost tasks are MMS tasks. One detail of the submission does not reproduce in every build: the lost conversations are longer with the playbook in C2 and C2c (median 113 against 87.5 messages, and 104 against 91) and shorter in C2b (48 against 73).
Neither GPT-5.4 build as shipped passes the gate. We rebuilt the playbook once more on GPT-5.4 at medium (C1b; C1 is the build in the submission). On 64 held-out tasks with two replicates per arm, C1 gives +0.02 [−0.05, +0.10] and C1b gives +0.04 [−0.05, +0.12].
One point of disclosure. Against the no-artifact runs archived in September, C1b would give +0.09 [+0.01, +0.16] and would pass. We reran the no-artifact arm in November, in the same month as C1b, and against that run C1b is inconclusive. We use the same-month run as the reference because a drift of the no-artifact level between the two months (+0.05 [−0.05, +0.14]) cannot be excluded. Both numbers are in Appendix H.1.
The harm goes with what GPT-5.5 writes for itself. We gave GPT-5.5 the two playbooks built by GPT-5.4. On the same 50 tasks it scores 0.92 and 0.96 with them, against 0.68, 0.62 and 0.74 with its own. The difference between the two groups is +0.26 [+0.16, +0.36] (Holm-adjusted p < 0.005). The GPT-5.4 playbooks are about twice as long, so length does not explain the harm. Our claim is about this difference. We do not claim that the GPT-5.4 playbooks are harmless on GPT-5.5, and we do not offer a mechanism.
Q3. Is there a configuration where the gate passes and depreciation can be reported with an interval?
There is one configuration that passes the gate, and it is exploratory. Section 6 of the submission noted that 76 of the 106 entries of the C1 playbook were never marked helpful by ACE's own curator. Keeping only the 30 entries that were marked helpful gives +0.09 [+0.03, +0.16] on GPT-5.4 at medium (two replicates), which the gate recognises. The rule uses counters from the build phase only.
The paper states its limits next to the result. We tried the rule after the main results were known. It passes on one playbook of three: the same rule gives −0.01 [−0.11, +0.09] on C1b and −0.28 [−0.42, −0.14] on C2. It is not separable from a word-shuffled placebo of the same pruned playbook (+0.04 [−0.02, +0.12]), and it does not survive Holm correction (adjusted p = 0.11). One control is consistent with the reading that the removed entries matter: on their own they give −0.16 [−0.27, −0.06], and the pruned playbook scores 0.26 higher than they do (95% CI [0.16, 0.37]; adjusted p < 0.005).
Depreciation cannot be estimated for it. Transplanted to GPT-5.5, the pruned playbook gives +0.06 [−0.02, +0.14]. GPT-5.5 already solves 0.92 of these tasks without any artifact, so the largest transplant value that could be observed is +0.08. Depreciation is a ratio of two noisy differences under a ceiling, and at 50 tasks its interval covers every bucket from retained to toxic. We report the two differences and the ceiling, and no point estimate.
Presentation
The table of gate outcomes for all rounds moves from the appendix into Section 3. Section 2 gains a table of symbols for the four accounts. Observation 2 is split into shorter paragraphs. The exploratory results are reported briefly in the text, labelled as exploratory, with the full tables in Appendix H.5. Neither the abstract nor the conclusion relies on them.
On soundness
Your soundness concerns were the convention behind the headline claim (W1), the unexplained baseline and the regression argument (W2), and the analysis rules (W3). The headline claim now states its convention and no longer rests on it. The baseline difference is traced to an inconsistency between AgentStream's paper and its code, and the protocol is run under both settings. One rule is used throughout. If anything remains open, we would be glad to address it.
Response to Reviewer DAtN (Part 1/4)
Thank you for the close reading. Each of your points led to a new experiment or a correction. We give a summary first and then the details, in the order of your review. Run labels are explained where they first appear.
Summary
W1. The abstract and conclusion say every measured artifact is an expense
The abstract and the conclusion now state the convention together with the claim. The closing sentence of the abstract reads: "By these books none of the artifacts we measured is recognised as an asset, and under the gate's convention each is therefore booked as an expense: where the evidence is decisive it shows harm, and elsewhere it is inconclusive at our sample sizes." The qualification you cite from the body is now part of it.
Each artifact is reported with one of three outcomes: recognised (95% interval above zero), harmful (interval below zero), or inconclusive (interval contains zero).
As you note, refusal was close to certain in the open-weight rounds, which have one replicate on 50 tasks and sit on columns where no prior gain was reported. The revision reports them for the cost accounts and the protocol only. They are not used as evidence about value.
The value claim now rests on the two closed models, where six artifacts as shipped were measured.
For the four harmful artifacts the outcome does not depend on the convention, because the interval itself lies below zero. We regard the replication across three independent builds as the main evidence of harm. For completeness, after Holm correction over all 46 contrasts the second GPT-5.5 build remains significant (adjusted p = 0.007) and the other three do not (0.081, 0.48 and 0.14).