acceptodds
Under review as a conference paper at ICLR 2027

Learning Not to Forget How to Improve: Meta-Forgetting in Self-Evolving Agents

Abstract

Self-evolving language-model agents promise sustained adaptation, yet rising task performance can conceal the loss of previously acquired abilities to generate effective agent updates. We investigate this failure, termed meta-forgetting, and seek to preserve improvement ability as agents continue to evolve. We introduce Improvement Function Replay (IFR), which revisits a small set of frozen historical agent states and evaluates how effectively candidate improvement strategies modify them. IFR selects strategy updates that maximize gains on current problems while limiting declines from fixed historical reference gains, preserving the effectiveness of past improvement strategies without requiring identical edits. Across five benchmarks and three frozen language-model backbones, IFR attains higher reported mean task performance. With Qwen3-32B, its paired mean improvements in unseen ALFWorld success, OfficeQA exact accuracy, and Seal-Hard accuracy are 2.3, 2.5, and 2.6 percentage points, respectively, over Current-only + search, the strongest tested controlled alternative. IFR makes retention of improvement ability an explicit objective, complementing task-level evaluation and supporting continued adaptation with a limited replay budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-evolving agents, large language models, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_D12EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies meta forgetting in evolving agents, where the improver can lose its ability to modify earlier agent states even as task performance improves. The authors propose IFR, which constrains improver updates by replaying historical editing problems. Experiments across five benchmarks and three backbone models show improvements in final task performance and gains from individual edits on common agent states.

Critical strengths
  1. The paper shows that improving task performance can coexist with declining ability to improve historical agent states. This offers a useful perspective for evaluating evolving agents beyond task scores.
  2. IFR incorporates capability retention directly into improver updates by replaying historical editing problems and limiting declines relative to fixed reference gains. The method is simple and practically useful.
Critical weaknesses
  1. Section 5.2 evaluates gains from individual edits on common agent states but does not compare subsequent adaptation over multiple steps when different improvers start from the same current agent. Direct evidence that retaining historical editing capabilities supports continued improvement is still missing.
  2. In the offline analysis in Section 5.3, IFR often retains the incumbent improver. The experiments lack controls that match the update frequency or freeze an earlier improver after it has improved. They therefore do not distinguish the benefits of historical capability retention from those of more conservative updates.
Questions for authors
  1. Starting from the same current agent, does the IFR improver still outperform other improvers during subsequent adaptation over multiple steps?
  2. What is the actual frequency of improver updates? How do the results change when this frequency is matched or an earlier improved improver is frozen?
3 comments
AuthorsOpenReview ↗

Response to Reviewer D12E

Thank you for your time and for the concrete suggestions on common-start evaluation and conservative-update controls. We present the results below, including comparisons with matched update counts and an earlier learned improver held fixed.

W1. Multi-step adaptation from the same current agent (also Q1)

Response.{\color{green}\mathbf{Response.}} In the comparison you proposed, IFR supports larger gains over 12 edit steps from the same current agent. Five paired runs each use two starting states from an independent construction trajectory. For each starting state, all methods share the complete agent snapshot and audit tasks. The learned improvers stay fixed while the task agents evolve through three four-attempt blocks, A→B→A′A\to B\to A'. Type A returns with new instances. Held-out audit items are reserved for evaluation.

BenchmarkFixed improverJ0J_0 (%)ΔJ4\Delta J_4 (pp)ΔJ8\Delta J_8 (pp)ΔJ12\Delta J_{12} (pp)AUC (pp)
ALFWorldCurrent-only + search60.03.04.65.43.43
ALFWorldEarly learned IFR checkpoint60.03.45.26.43.93
ALFWorldFinal IFR60.04.06.27.84.70
OfficeQACurrent-only + search50.02.43.64.42.73
OfficeQAEarly learned IFR checkpoint50.02.84.25.23.20
OfficeQAFinal IFR50.03.45.46.64.03
BenchmarkIFR minus comparator12-step advantage (pp, 95% CI)AUC advantage (pp, 95% CI)
ALFWorldCurrent-only + search+2.40 [1.60, 3.40]+1.27 [0.87, 1.70]
ALFWorldEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.77 [0.40, 1.03]
OfficeQACurrent-only + search+2.20 [1.60, 2.80]+1.30 [1.03, 1.60]
OfficeQAEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.83 [0.53, 1.13]

AUC is the normalized trapezoidal area above J0J_0. CIs use 2,000 complete-run paired bootstraps, with the two starts averaged within each run. IFR's advantage persists across the continuation checkpoints and is positive in both final gain and AUC. This extends Table 4's common-state evaluation to repeated adaptation from the same current agent.


W2. Actual update frequency and freezing an earlier learned improver (also Q2)

Response.{\color{green}\mathbf{Response.}} We measured actual installations in five paired reruns and tested both controls you requested. The table reports the deployment counts and task scores:

MethodALFWorld installations / 60OfficeQA installations / 60ALFWorld task score (%)OfficeQA task score (%)
Current-only + search424265.856.2
Current-only, rate-matched161766.457.1
Early learned IFR checkpoint frozen8867.257.8
IFR161768.158.7

The current-only control uses only current gain, with thresholds of 9 pp on ALFWorld and 10 pp on OfficeQA under the same scoring protocol. Aggregate installation counts match IFR. Each method chooses its own update times.

The frozen control shares IFR's first six stages. It then selects a checkpoint using independent development probes and holds the complete improver fixed as the task agent continues to evolve. All selected checkpoints come from stages 2–5 and improve on P0P_0 by 2.7–3.8 pp on the development probes. The eight installations count newly generated versions in the shared prefix. The one-time checkpoint selection is separate from this installation count.

IFR and Current-only task scores are the reference values in Table 1. The other scores and installation counts come from the supplementary experiments. The paired continuation results provide a direct comparison of the learned improvers:

BenchmarkIFR minus comparator12-step advantage (pp, 95% CI)AUC advantage (pp, 95% CI)
ALFWorldRate-matched+2.00 [1.20, 3.00]+1.07 [0.73, 1.43]
ALFWorldEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.77 [0.40, 1.03]
OfficeQARate-matched+1.80 [1.20, 2.40]+1.03 [0.83, 1.23]
OfficeQAEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.83 [0.53, 1.13]

The early checkpoint retains substantial editing capability. IFR's final improver achieves larger paired continuation gains and higher AUC than both controls, supporting the value of preserving useful editing responses throughout evolution.

I appreciate the authors' efforts in addressing my concerns through the additional results. After considering the rebuttal, my final rating is 3.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for taking the time to consider our rebuttal and additional results. We appreciate your constructive feedback and thoughtful evaluation of our work.

Reviewer_Bf9nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies meta-forgetting in self-evolving LLM agents, where an improver gradually loses its ability to improve historical agent states even as task performance increases. The authors propose Improvement Function Replay, which uses frozen historical agent states and fixed reference gains to preserve previously acquired improvement abilities. Experiments across five benchmarks and three LLM backbones show that IFR reduces historical improvement loss and achieves better task performance than the tested baselines.

Critical strengths
  1. The paper investigates an interesting and relatively underexplored problem in self-evolving agents. The distinction between task performance and improvement ability offers a useful perspective on evaluating agent evolution.

  2. The experimental evaluation is fairly comprehensive, covering multiple benchmarks, LLM backbones, and baseline methods. The authors also provide ablation studies and independent evaluations of improvement ability, which help support their findings.

Critical weaknesses
  1. I am not fully convinced by the motivation for preserving an improver's ability to modify historical task agents. In a typical self-evolution process, the task agent and improver evolve together, and the improver mainly works on the current task agent. If earlier problems have already been addressed, why does the current improver still need to be effective at modifying those older agent states? The paper shows that this ability can decline over time, but I would like the authors to explain more clearly how such a decline harms future self-evolution and why it should be considered a problem worth addressing.

  2. I also question whether One-Edit Gain on frozen historical agent states is a reliable proxy for future self-improvement potential. The shared-state evaluation provides some evidence that IFR improves the quality of edits on independent states, but it does not directly establish how well an improver can support continued evolution over time. I would like to see experiments involving longer evolution trajectories, recurring task types, or cross-domain transfer to better establish the connection between retaining historical improvement ability and improving future adaptation.

  3. In the offline analysis in Figure 4, the IFR selection rule retains the incumbent improver in most cases. This raises a concern about what actually drives the performance gains. Is IFR effective because it preserves historical improvement ability, or simply because it reduces the frequency of meta-updates and avoids potentially harmful revisions? I think a useful baseline would be a Current-only variant that accepts meta-updates at a similar rate to IFR, without applying the historical retention constraint. This comparison would help isolate the contribution of the proposed mechanism.

  4. I think the paper is missing a relevant baseline that archives historical improver checkpoints and retrieves or selects them when appropriate. In the paper's setting, improvers are updated through textual instructions and persistent state rather than model parameter updates, so storing previous versions should incur relatively little storage overhead. It seems reasonable to preserve earlier improvement strategies by keeping an archive of improvers and selecting a suitable one when similar problems arise. I would like to see how IFR compares with this approach, including the additional cost of selecting or retrieving archived improvers.

Questions for authors

See Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bf9n (1/2): W1–W2

We appreciate the time you spent reviewing our paper and your thoughtful questions about the practical value of retained editing capabilities. We address them below with results on continued adaptation, update frequency, and checkpoint reuse.

W1. Why preserve the ability to edit historical agents?

Response.{\color{green}\mathbf{Response.}} An editing strategy can remain useful on new instances after it has repaired an earlier agent. IFR preserves this reusable capability. Frozen historical agents provide fixed targets for measuring it, while deployment continues from the current agent. HGM also shows that current task performance and the ability to produce improved descendants can diverge [1].

Table 4 evaluates editing gains on independent common states. To test the practical value you ask about, we evaluated continued adaptation from identical current agents. Each learned improver stays fixed for 12 task-edit attempts in an A→B→A′A\to B\to A' sequence. Each block has four attempts. A′A' revisits the task type with fresh instances that are disjoint from replay examples.

BenchmarkFixed improver12-step gain (pp)Gain on revisiting type A (pp)
ALFWorldCurrent-only + search5.41.2
ALFWorldEarly learned IFR checkpoint6.41.2
ALFWorldFinal IFR7.82.8
OfficeQACurrent-only + search4.40.8
OfficeQAEarly learned IFR checkpoint5.21.2
OfficeQAFinal IFR6.62.4

IFR's paired 12-step advantage over Current-only + search is +2.4 pp [1.6, 3.4] on ALFWorld and +2.2 pp [1.6, 2.8] on OfficeQA. These gains show that retained editing strategies help with new instances and further adaptation from the current agent.

[1] Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine.


W2. Does one-edit gain translate into multi-step adaptation?

Response.{\color{green}\mathbf{Response.}} Your requested continuation test directly measures this connection. One-edit gain evaluates an improver on a fixed editing problem. Appendix A.6.2 distinguishes this raw gain from the gain after deployment acceptance. The continuation experiment follows the deployment acceptance rule over 12 steps. As in the fixed-improver evaluation in HyperAgents [2], the improver stays fixed while the task agent evolves.

Each of five paired runs uses two starting states from an independent construction trajectory. For each state, all methods receive the same complete agent snapshot and share 50 held-out audit items. These items are disjoint from optimization and replay. All branches complete 12 attempts under the same update rule and resource ceilings. Initial scores are 60.0% for ALFWorld and 50.0% for OfficeQA.

BenchmarkFixed improverΔJ4\Delta J_4ΔJ8\Delta J_8ΔJ12\Delta J_{12}Normalized AUC
ALFWorldInitial P0P_01.82.83.42.10
ALFWorldCurrent-only + search3.04.65.43.43
ALFWorldEarly learned IFR checkpoint3.45.26.43.93
ALFWorldFinal IFR4.06.27.84.70
OfficeQAInitial P0P_01.42.22.61.63
OfficeQACurrent-only + search2.43.64.42.73
OfficeQAEarly learned IFR checkpoint2.84.25.23.20
OfficeQAFinal IFR3.45.46.64.03

All entries are pp relative to the common starting score. AUC is the trapezoidal area above that score, divided by 12. The paired IFR advantages are:

BenchmarkComparatorΔJ12\Delta J_{12} difference [95% CI]AUC difference [95% CI]
ALFWorldCurrent-only + search+2.40 [1.60, 3.40]+1.27 [0.87, 1.70]
ALFWorldEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.77 [0.40, 1.03]
OfficeQACurrent-only + search+2.20 [1.60, 2.80]+1.30 [1.03, 1.60]
OfficeQAEarly learned IFR checkpoint+1.40 [0.60, 2.40]+0.83 [0.53, 1.13]

CIs use 2,000 paired bootstraps over five complete runs, averaging the two starts within each run. IFR leads at each measured checkpoint, including after type A returns. Its higher final gains and AUC show sustained improvement across the tested 12-step continuation.

[2] HyperAgents.

AuthorsOpenReview ↗

Response to Reviewer Bf9n (2/2): W3–W4

W3. Could fewer meta-updates explain the gains?

Response.{\color{green}\mathbf{Response.}} Your proposed control uses only current gain. It installs the best new candidate when its gain exceeds the incumbent's by more than λ\lambda. We kept K=4,R=3K=4,R=3 and used pilot-selected thresholds of λ=0.09\lambda=0.09 (9 pp) on ALFWorld and 0.100.10 (10 pp) on OfficeQA. Both thresholds reject the smallest positive gain difference, 1/121/12, making the gate stricter than ordinary Current-only selection.

BenchmarkMethodNew installations / 60 stagesTask score (%)GsharedG_{\mathrm{shared}} (pp)
ALFWorldCurrent-only + search4265.810.2
ALFWorldCurrent-only, rate-matched1666.410.4
ALFWorldIFR1668.110.8
OfficeQACurrent-only + search4256.29.0
OfficeQACurrent-only, rate-matched1757.19.2
OfficeQAIFR1758.79.5

The installation counts come from the supplementary deployment runs. IFR and Current-only endpoint scores are the reference values in Tables 1 and 4. Rate-matched scores come from the supplementary experiment. For a direct paired comparison, we evaluated the learned improvers in the common-start continuation:

BenchmarkIFR / rate-matched 12-step gains (pp)Paired gain difference [95% CI]Paired AUC difference [95% CI]
ALFWorld7.8 / 5.8+2.00 [1.20, 3.00]+1.07 [0.73, 1.43]
OfficeQA6.6 / 4.8+1.80 [1.20, 2.40]+1.03 [0.83, 1.23]

At equal aggregate update counts, IFR achieves larger 12-step gains and higher AUC on both benchmarks. Matching applies to the five-run totals, with each method choosing its own update times.


W4. Historical improver archive and retrieval costs

Response.{\color{green}\mathbf{Response.}} Your proposed archive is a useful alternative for retaining editing strategies. Reusing historical agents is also central to DGM [3]. We tested an archive that stores complete improver manifests and persistent state. At each stage, it scores all archived versions and three new proposals on the current development probe. Selection uses only current gain. It can restore an earlier improver while keeping the current task agent.

BenchmarkMethodTask score (%)GsharedG_{\mathrm{shared}} (pp)Optimization tokens (M)Episodes
ALFWorldArchive + current-gain selection67.610.627.66,210
ALFWorldIFR68.110.822.84,980
OfficeQAArchive + current-gain selection58.09.429.16,420
OfficeQAIFR58.79.525.05,030

IFR scores and costs are the reference values in Tables 1, 4, and 12. Archive results come from the supplementary runs. All methods complete 12 stages within the same 30M-token/10,000-episode ceilings. Storage is small: approximately 0.14 MiB per run on ALFWorld and 0.15 MiB on OfficeQA. The archive restores an earlier version 12 and 10 times, respectively, across 60 stages.

On ALFWorld, version selection costs 8.9M tokens and 2,210 episodes per run. On OfficeQA, it costs 9.6M tokens and 2,430 episodes. These costs are included in the totals above.

The archive also remains competitive in the common-start continuation test. Its library is fixed, with development-based selection at each task-type block:

BenchmarkContinuation system12-step gain (pp)AUC (pp)Tokens (M)Episodes
ALFWorldArchive-select7.24.333.04638
ALFWorldFixed final IFR improver7.84.700.76160
OfficeQAArchive-select5.83.573.24665
OfficeQAFixed final IFR improver6.64.030.80164

Costs cover optimization at both starts per run. The archive retains useful strategies and performs competitively. IFR achieves higher mean continuation gains with lower optimization cost. These results support retaining reusable editing capabilities within a single active improver, with the archive's additional cost coming from repeated version selection.

[3] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your thoughtful feedback, and we would welcome your thoughts on our response.

Our new experiments on ALFWorld and OfficeQA show stronger 12-step adaptation from identical current agents with recurring task types, gains over a control with matched update counts, and higher mean gains than archive-based selection at lower cost.

Please let us know if any concerns remain, as we would be happy to discuss them and provide further clarification or evidence during the remaining discussion period. If these results address your concerns, we would be grateful if you could reconsider your rating.

Thanks!

Reviewer_CgD8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether an evolving agent can retain task competence while losing the ability to generate useful agent modifications. It calls this failure meta-forgetting and measures it by evaluating successive improvers on identical frozen agent states and diagnostic traces. Improvement Function Replay (IFR) selects improver revisions for current improvement subject to a constraint on their mean positive shortfall from fixed historical improvement gains. Historical probes are stored in a bounded reservoir. The analysis establishes an invariant for cached retention loss and extends it to expected loss under explicit sampling assumptions. Experiments evaluate task performance, improvement on independently constructed common states, historical retention, and resource consumption. The controlled comparison with Current-only + search reallocates replay resources to current-state optimization. Reported task and common-state gains favor IFR across five matched runs on three benchmarks.

Critical strengths
  1. Frozen targets and shared diagnostic traces separate changes in the improver from changes in the agent being edited. The independent common-state bank addresses the difficulty of comparing improvement gains on different methods’ trajectories. Separate screening and confirmation trials for positive-reference probes also avoid reusing the same favorable estimate to select and evaluate previously acquired abilities. These controls make the measurement of improvement ability informative beyond final task scores.

  2. The study includes a frozen improver, current-only evolution, additional current-state search, task-score preservation, edit imitation, and full-history replay. The resource accounting includes rejected proposals and replay calls, and Table 12 reports realized consumption as well as common ceilings. IFR uses only about 1–2% more tokens than Current-only + search on the primary functional benchmarks. Table 3 reports positive paired differences in all five runs for both task performance and shared-state improvement. The paper also acknowledges that full-history replay retains historical responses slightly better and that historical MBWT compares method-specific targets. The consistent paired improvements under the resource control are the main empirical reason for my weak-accept recommendation.

Critical weaknesses
  1. The ALFWorld tolerance ablation appears inconsistent with the stated scoring resolution. Appendix B.1 uses binary task success; B.1 specifies four scoring tasks per probe, and B.3 specifies three development repetitions and a buffer of at most eight probes. With the unweighted averaging in Equations (3) and (5), a probe’s cached gain contrast is a multiple of 1/12. Therefore, with a nonempty buffer of size n ≤ 8, the smallest positive mean shortfall is 1/(12n), which is at least 1/96 (approximately 0.0104167), and therefore exceeds 0.01. Under this protocol, ε = 0 and ε = 0.01 admit exactly the same candidates for any fixed cache. Yet Table 10 reports different task success and MBWT for these settings using the same five paired run seeds. This could reflect an omitted scoring detail, numerical handling, or stochastic execution differences. It needs reconciliation before those rows can support a tolerance trade-off.

  2. The mechanism evidence stops short of showing what happened in the deployed selector. Figure 4 uses separately estimated diagnostic scores, and Appendix B.4.4 explicitly warns that its offline winners need not equal the installed improvers. The reported 46–48 incumbent selections out of 60 therefore do not establish the actual revision rate. Actual cache feasibility, installed-candidate counts, and their paired independent audit outcomes would help determine how much of the benefit comes from preserving particular improvement responses versus making fewer meta-updates.

Questions for authors
  1. What exact ALFWorld per-execution score and averaging rule were used in Table 10? Please reconcile the ε=0\varepsilon=0 and ε=0.01\varepsilon=0.01 rows with the 1/(12n)1/(12n) resolution, and clarify whether execution nondeterminism or another implementation detail can produce different trajectories despite using the same paired run seeds. This matters for interpreting the tolerance ablation and whether it reflects the stated selector.

  2. From the original optimization caches, how many stages actually installed a new improver, rejected the current-gain winner, or retained the incumbent? Can these decisions be linked to the existing independent retention audits? Please separate these deployed counts from Figure 4's offline selections. These records would help assess whether the proposed explanation describes the deployed method.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CgD8

Thank you for taking the time to review our work and for your detailed comments on the scoring protocol and selector analysis. Below, we clarify the tolerance settings and present the deployment audit.

W1. Scoring resolution and the tolerance ablation (also Q1)

Response.{\color{green}\mathbf{Response.}} Your calculation of the scoring resolution is correct. Each ALFWorld execution is scored as success or failure. Scores are averaged uniformly over K=4K=4 scoring tasks and R=3R=3 repetitions. The shared parent score cancels when comparing the candidate and anchor gains. Thus, for a nonempty buffer,

\quad\Longrightarrow\quad \widehat L_B(P)\geq\frac{1}{12|B|}\geq\frac{1}{96}>0.01.$$ The two tolerances therefore implement the same constraint. Integer-count cache replay confirms identical feasible sets and selected hashes: | Same-cache comparison: $\epsilon=0$ versus $0.01$ | Disagreements | |---|---:| | Candidate feasibility | 0 stages | | Selected improver hash | 0 stages | The effective tolerance comparison is between $0$, $0.02$, and $0.05$: | $\epsilon$ | ALFWorld success (%) | $G_{\mathrm{shared}}$ (pp) | MBWT (pp) | New improver installations / 60 stages | |---|---:|---:|---:|---:| | 0 | 66.7 | 10.0 | −0.5 | 8 | | 0.01, replay of the $\epsilon=0$ cache | 66.7 | 10.0 | −0.5 | 8 | | 0.02 | 68.1 | 10.8 | −1.1 | 16 | | 0.05 | 68.7 | 11.0 | −2.6 | 27 | The values for $0$, $0.02$, and $0.05$ are the reported optimization endpoints. The $0.01$ row comes from replaying the $\epsilon=0$ cache. Table 10's original $0.01$ result of 67.5% came from a separate execution with the same run seeds. Nondeterministic inference or environment calls produced different candidate or scoring records before selection. The difference from 66.7% therefore reflects variation between the independently executed trajectories. Both tolerances implement the same effective constraint. Across the distinct thresholds $0$, $0.02$, and $0.05$, larger tolerances allow more installations and yield higher task scores, accompanied by more negative MBWT. This captures the retention–plasticity trade-off. The fixed-cache invariant applies to each tolerance. --- > #### **W2. Actual selector decisions and independent retention audits (also Q2)** ${\color{green}\mathbf{Response.}}$ To examine the deployed selector, we audited five paired 12-stage reruns. Figure 4 provides a complementary offline analysis using separately scored diagnostic candidates. The deployment counts below cover valid, distinct online candidates and exclude shadow proposals: | Benchmark | New improver installed / stages | Incumbent retained | Current-gain winner vetoed | Feasible / valid new candidates | |---|---:|---:|---:|---:| | ALFWorld | 16/60 | 44 | 47 | 61/169 | | OfficeQA | 17/60 | 43 | 49 | 54/166 | | Seal-Hard | 16/60 | 44 | 46 | 58/170 | Every stage either installs a new improver or retains the incumbent. A veto can lead to either outcome. Figure 4's incumbent counts of 47/48/46 describe the offline diagnostic selections. For every vetoed stage, eight fresh paired edit trials compare the selected improver with the unconstrained winner on identical frozen probes. Differences below are selected minus unconstrained, with run-level 95% CIs: | Benchmark | Audited / vetoed stages | Historical-probe gain difference (pp) | Current-probe gain difference (pp) | |---|---:|---:|---:| | ALFWorld | 47/47 | +3.0 [2.2, 3.8] | −2.2 [−2.9, −1.4] | | OfficeQA | 49/49 | +2.7 [1.8, 3.6] | −2.4 [−3.2, −1.6] | | Seal-Hard | 46/46 | +2.6 [1.8, 3.4] | −2.0 [−2.7, −1.3] | The selected improver achieves higher gains on historical probes while accepting a lower current-probe gain. Using identical targets makes these differences directly comparable and shows which editing responses the constraint preserves. To test the conservative-update explanation you raise, we used a gate based only on current gain: 9 pp on ALFWorld and 10 pp on OfficeQA, with unchanged $K=4,R=3$. The aggregate installation counts match IFR. A separate 12-step continuation then compares their learned improvers from common starting agents: | Benchmark | IFR / control installations | IFR minus control: 12-step gain (pp, 95% CI) | |---|---:|---:| | ALFWorld | 16/16 | +2.0 [1.2, 3.0] | | OfficeQA | 17/17 | +1.8 [1.2, 2.4] | Each improver stays fixed during continuation, and the comparison uses five paired runs. The aggregate update totals are equal, with each method choosing its own update times. IFR achieves larger continuation gains in both benchmarks, linking the retained editing capability to further adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.