- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how effectively language-model agents improve through experience while their model weights remain fixed. It introduces agent plasticity, which measures performance improvement relative to cumulative learning expenditure, together with a saturation-based metric for summarizing learning efficiency. The evaluation protocol alternates between acting and reflection: agents use training experience to update a persistent inventory of tools, code, instructions, and memory, while subsequent acting episodes begin with fresh contexts. In the board-game experiments, held-out evaluation trajectories and scores are withheld from reflection. Experiments across chess, Go, Hex, and NetHack reveal substantial variation in improvement across models and tasks, including pronounced gains, stagnation, and regressions following inventory updates. Beyond reporting learning curves, the paper analyzes persistent artifacts and execution traces to distinguish failures without a covering artifact, failures where an available artifact is unused, and failures despite artifact use. These analyses help characterize how agents translate accumulated experience into subsequent behavior. The main contribution is an evaluation framework and empirical study of persistent agent adaptation, emphasizing learning efficiency and adaptation dynamics alongside initial task performance.
Critical strengths
- A clear and well-controlled protocol for evaluating persistent agent improvement. The paper addresses the important question of how effectively agents improve across interactions while their model weights remain fixed. Fresh acting contexts and explicit inventory inheritance establish a clear boundary between transient interaction history and persistent adaptation. Withholding held-out trajectories and scores from reflection strengthens the board-game evaluation, while including rejected reflection attempts in learning costs makes the expenditure accounting more complete. Together, these choices provide a coherent framework for examining improvement beyond initial capability (§§2.1–2.2 and §3.1).
- Informative longitudinal experiments supported by inspectable behavioral evidence. The experiments across chess, Go, Hex, and NetHack reveal substantial model and task dependence, including stagnation and harmful updates. Reporting full learning curves exposes these dynamics, such as the late Go regression following a tool rewrite (§3.3). The artifact analysis carefully distinguishes relevance, failure coverage, and observed use, grounding its judgments in execution traces and engine evidence. The guard-removal experiment in Figure 15 additionally provides concrete evidence that a particular learned artifact changes a subsequent decision.
Critical weaknesses
- The specific contribution of training feedback remains unquantified. The experiments measure improvement under the complete interaction-and-reflection process. As acknowledged in §§3.2 and 4, they do not separate the incremental benefit of training feedback from additional artifact-development computation. Figure 15 provides useful local evidence of a feedback-derived correction, but its contribution to the aggregate improvement remains unclear. This limits conclusions about the specific value of experience. The distinction between improvement under the complete protocol and the contribution of feedback should remain explicit throughout the framing; a targeted comparison with the same development budget would strengthen the latter interpretation.
- The statistical reporting supporting the headline plasticity comparisons is incomplete. Section 2.2 specifies the curve family, fitting criterion, and decision thresholds, but leaves important implementation details unclear, including the construction of the binomial likelihood, calibration of the flat-curve test, and sources of variability captured by the parametric bootstrap. Under , and are unidentified, making the test calibration relevant to interpreting the significance gate. Figure 1 also presents plasticity estimates without uncertainty intervals. These omissions limit assessment of the precision of the reported comparisons, although the broad differences visible in the learning curves remain informative.
Questions for authors
- Beyond the local intervention in Figure 15, is there evidence quantifying the incremental contribution of training feedback relative to additional artifact-development computation? Has a comparison with the same development budget but without trajectory or outcome feedback been examined in any representative setting?
- How is the stated binomial likelihood constructed for chess scores that include half-point draws and for the equally weighted combined-game curve? How is the likelihood-ratio test against a flat curve calibrated, given that and are unidentified when ?
- How many independent adaptation lineages were evaluated for each board-game condition, and what sources of variation does the parametric bootstrap capture? Could the corresponding uncertainty intervals be reported for the plasticity estimates in Figure 1?
6 comments
Author Response (Part 2/3)
W2/Q2: The fitting objective, draws, and combined curve
Draws. At checkpoint , we define the total score across games. With , the fitted objective is
together with weak penalties on the baseline and asymptote logits. The generalized binomial coefficient, defined using gamma functions for fractional , is constant with respect to the fitted parameters.
We agree that this is a fractional-score objective, not the true likelihood of win/draw/loss outcomes. For game score with mean ,
so a Bernoulli likelihood does not correctly represent score variance when draws occur. The objective remains well-defined for fitting the mean performance curve, but we do not use its likelihood-based standard errors for uncertainty estimation.
Combined curve. Chess first averages independent lineages within each difficulty and weights Easy and Hard equally. Chess, Go, and Hex then receive equal weight.
The combined equals this weighted mean score multiplied by the total number of games. This preserves our intended equal-task performance measure, but treating the weighted aggregate as a single binomial does not generally yield correctly calibrated uncertainty, since task weights need not match game-count proportions and outcomes include draws.
We therefore separate the fitting objective from uncertainty estimation and use outcome-level resampling for the latter. We clarify these choices in §2.2.
W2/Q2: Simulation-calibrated flatness test
We agree that under the null hypothesis of no rise (), the parameters and are unidentified.
We therefore recalibrated the test by simulation. For each of the 26 board-game curves, we estimate each held-out game's win/draw/loss probabilities from its 21 checkpoint outcomes and hold these probabilities fixed across checkpoints. We generate 1k no-learning histories preserving the observed evaluation structure, game counts, learning costs, and task weights.
Each simulated history is refitted using the full procedure, including 15 initialization starts, parameter bounds, regularization, and dispersion adjustment. The simulated p-value is
All 26 final plasticity classifications remain unchanged when combining the recalibrated test with our separate 10-point minimum-rise criterion, although individual p-values change:
- Combined Luna: : 0.235 → 0.077, but the fitted 90% rise is only 2.6 points, below the required 10 points; hence .
- Chess GPT-5.5: : 0.35 → 0.043, but the fitted rise is only 1.8 points, also below the minimum-rise criterion.
Thus, the classification depends on both statistical evidence and a minimum improvement magnitude, rather than significance alone.
Scope. The reference distribution is built from each game's own observed outcomes and reproduces the actual evaluation design (game counts, draws, learning costs and task weights), rather than relying on an asymptotic approximation; its independence assumption is stated in Appendix G.2. Under this calibration all 26 classifications are unchanged, showing that they are robust to how the no-rise test is calibrated. Where independent lineages exist, their variation is captured by the bootstrap intervals (W2/Q3).
Author Response (Part 3/3)
W2/Q3: Independent lineages and bootstrap uncertainty
Independent lineages. In chess, Sol, Opus 5, Opus 4.8, GPT-5.5, and Gemini 3.1 Pro have three independent lineages per difficulty; Fable, Opus 4.6, and Luna have one. Go, Hex, and NetHack each have one lineage per model. In the new feedback controls, each Argon condition has two lineages, while Flash and Carbon have one.
Limitations of the submitted bootstrap. Our original procedure sampled each checkpoint score from Binomial$(n_t,\hat p_t)$, held learning costs fixed, and refitted the curve 100 times. This captured only checkpoint-level game-outcome noise. It did not preserve draws, the same held-out game's outcomes across checkpoints, or independent-lineage variation; 100 resamples also provided coarse estimates of interval tails.
Revised bootstrap (Figure 1). We now report 90% intervals from 2,000 resamples. Each resample:
- Resamples independent lineages where available (three per chess difficulty for five models).
- Resamples held-out games by identity, preserving each sampled game's win/draw/loss outcome sequence across checkpoints.
- Refits the full estimation pipeline, including the saturation rule.
This preserves the longitudinal evaluation structure and accounts for the variability represented by the available lineages and held-out games. For models with one lineage, the intervals reflect variation across held-out games.
The main conclusions remain: GPT-5.6 Sol and Opus 5 have the highest reported saturation-plasticity point estimates, with overlapping bootstrap intervals. Both intervals lie clearly above those of Fable 5 and Opus 4.8. We therefore describe Sol and Opus 5 as leading models without asserting a statistically established ordering between them. GPT-5.5 is reported separately using plasticity to date, which is not directly the same estimand as saturation plasticity.
As an additional robustness check, varying the saturation threshold to 85% or 95% leaves the combined ordering unchanged.
We have revised the ranking language in §3.2–3.4 and Finding 2 accordingly.
Paper revisions (in blue)
All changes in the revised manuscript are marked in blue:
- Appendix M, §3.2, Discussion: Feedback-versus-reflection-only experiments, Argon failure/reuse analysis, and qualitative reflection comparisons (W1/Q1).
- §2.2: Exact fractional-score fitting objective, regularization, combined-task weights, and clarification that likelihood-based uncertainty is not used (W2/Q2).
- Appendix G.2, §2.2: Simulation-calibrated flatness test, its assumptions, and the separate 10-point minimum-rise criterion (W2/Q2).
- Figure 1, §2.2: New 2,000-resample bootstrap intervals preserving held-out-game identity and independent lineages where available, with explicit lineage counts (W2/Q3).
- §3.2–3.4, Finding 2: Qualified model rankings where bootstrap intervals overlap (W2/Q3).
We are grateful for the reviewer's careful and constructive feedback. We hope these additions answer the questions raised, and we would appreciate it if the reviewer would take them into account in the final assessment. We are happy to provide further analyses during the discussion period.
Author Response (Part 1/3)
Thank you for your careful review and for recognizing how our protocol isolates persistent adaptation, accounts for rejected reflection attempts in learning cost, and demonstrates the causal effect of a learned artifact through the guard-removal experiment (Fig. 15). We address your concerns with the following new results:
(W1/Q1) Contribution of feedback relative to artifact-development compute: Across three Gemini models on chess and Hex, training feedback yields higher held-out performance than reflection alone (+3.8 to +21.7 points). Qualitative analysis shows that feedback directs reflection toward specific game-losing mistakes, whereas reflection alone predominantly optimizes code efficiency.
(W2/Q2) Statistical validity: We clarify the fractional-score fitting objective and replace the asymptotic χ² reference with simulation-based calibration. All 26 plasticity classifications remain unchanged.
(W2/Q3) Uncertainty intervals: For Figure 1, we replace the original bootstrap with 2,000 resamples that preserve draws, held-out-game identity across checkpoints, and independent lineages where available. Figure 1 now reports 90% bootstrap intervals, and we qualify model rankings accordingly.
We ran the suggested reflection-only control. Both conditions start with empty artifact inventories and run 20 update rounds.
We evaluate three recent Gemini models (Flash 3.8, Argon, Carbon; released recently, accessed via Google AI Studio) on chess and Hex, with two independent Argon runs per game.
Held-out ID score over checkpoints 0–20 and Psat (points per 1k USD), averaged across chess and Hex (and both Argon runs):
All three models reach higher performance with feedback (3.8–21.7 points), although the benefit varies across models and tasks. Feedback also gives higher plasticity (Psat): 1,357 vs 396 points per 1k USD for Argon, 3,045 vs 1,383 for Carbon, and 863 vs 0 for Flash, whose reflection-only runs show no reliable rise.
What does feedback change?
1. Both conditions construct and reuse strong tools. Our failure/reuse analysis (§3.5, Appendix H), pooled over checkpoints 1–20 of the original Argon chess runs, shows:
Argon resembles the strong improvers in our original study (Fable 5, Opus 5, Opus 4.8, GPT-5.6 Sol): 94–98% reuse, with 83–99% of remaining failures occurring despite artifact use. Unlike Sol and Opus 4.8, which begin below 20% reuse at checkpoint 1, surprisingly, Argon already reaches approximately 99%.
In both conditions, nearly all reuse involves executable tools; strategies and memory notes are used in approximately 1% of relevant moves. Thus, tool construction and reuse are not exclusive to learning from training games.
2. Feedback directs reflection toward task failures. We manually inspected reflection summaries from the two original Argon chess runs:
Both conditions develop capable engines. However, supplied feedback directs reflection toward mistakes that actually lose games, consistent with the higher chess score with feedback (95.8% versus 81.3%). Figure 15 provides complementary intervention evidence that a feedback-derived fix can affect subsequent behavior.
Conclusion: Supplied training feedback is associated with additional improvement and promotes reflection toward observed failures, a core requirement for agents that can learn from experience.