- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies meta forgetting in evolving agents, where the improver can lose its ability to modify earlier agent states even as task performance improves. The authors propose IFR, which constrains improver updates by replaying historical editing problems. Experiments across five benchmarks and three backbone models show improvements in final task performance and gains from individual edits on common agent states.
Critical strengths
- The paper shows that improving task performance can coexist with declining ability to improve historical agent states. This offers a useful perspective for evaluating evolving agents beyond task scores.
- IFR incorporates capability retention directly into improver updates by replaying historical editing problems and limiting declines relative to fixed reference gains. The method is simple and practically useful.
Critical weaknesses
- Section 5.2 evaluates gains from individual edits on common agent states but does not compare subsequent adaptation over multiple steps when different improvers start from the same current agent. Direct evidence that retaining historical editing capabilities supports continued improvement is still missing.
- In the offline analysis in Section 5.3, IFR often retains the incumbent improver. The experiments lack controls that match the update frequency or freeze an earlier improver after it has improved. They therefore do not distinguish the benefits of historical capability retention from those of more conservative updates.
Questions for authors
- Starting from the same current agent, does the IFR improver still outperform other improvers during subsequent adaptation over multiple steps?
- What is the actual frequency of improver updates? How do the results change when this frequency is matched or an earlier improved improver is frozen?
3 comments
I appreciate the authors' efforts in addressing my concerns through the additional results. After considering the rebuttal, my final rating is 3.
Thank you for your feedback
Thank you for taking the time to consider our rebuttal and additional results. We appreciate your constructive feedback and thoughtful evaluation of our work.
Response to Reviewer D12E
Thank you for your time and for the concrete suggestions on common-start evaluation and conservative-update controls. We present the results below, including comparisons with matched update counts and an earlier learned improver held fixed.
Response. In the comparison you proposed, IFR supports larger gains over 12 edit steps from the same current agent. Five paired runs each use two starting states from an independent construction trajectory. For each starting state, all methods share the complete agent snapshot and audit tasks. The learned improvers stay fixed while the task agents evolve through three four-attempt blocks, A→B→A′. Type A returns with new instances. Held-out audit items are reserved for evaluation.
AUC is the normalized trapezoidal area above J0. CIs use 2,000 complete-run paired bootstraps, with the two starts averaged within each run. IFR's advantage persists across the continuation checkpoints and is positive in both final gain and AUC. This extends Table 4's common-state evaluation to repeated adaptation from the same current agent.
Response. We measured actual installations in five paired reruns and tested both controls you requested. The table reports the deployment counts and task scores:
The current-only control uses only current gain, with thresholds of 9 pp on ALFWorld and 10 pp on OfficeQA under the same scoring protocol. Aggregate installation counts match IFR. Each method chooses its own update times.
The frozen control shares IFR's first six stages. It then selects a checkpoint using independent development probes and holds the complete improver fixed as the task agent continues to evolve. All selected checkpoints come from stages 2–5 and improve on P0 by 2.7–3.8 pp on the development probes. The eight installations count newly generated versions in the shared prefix. The one-time checkpoint selection is separate from this installation count.
IFR and Current-only task scores are the reference values in Table 1. The other scores and installation counts come from the supplementary experiments. The paired continuation results provide a direct comparison of the learned improvers:
The early checkpoint retains substantial editing capability. IFR's final improver achieves larger paired continuation gains and higher AUC than both controls, supporting the value of preserving useful editing responses throughout evolution.