- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper develops and RL-based policy for controlling the memory available to an agent. This addresses many of the basic problems of memory management in language model agents: adding useful memories can actually degrade performance, repeating memories can cause stickiness and degrade exploration/diversity, etc. The method developed in the paper uses RL to train a memory controller while keeping the agent fixed, and while simultaneously limiting feedback from a "test set" of tasks to prevent overfitting. The approach, at least conceptually, makes a lot of sense, but there are a number of details that probably matter in getting good practical performance. These are not discussed in particular detail or carefully justified but the experimental evaluation is fairly thorough, suggesting that the choices are at least reasonable. There are some issues with exposition/clarity and I have a couple of technical questions I will raise below.
Critical strengths
The paper addresses a real problem in modern language model agents, namely that agents behave rather strangely in long context scenarios, where "useful" memories can degrade performance, result in lack of exploration, etc. When agent are learning from experience in-context, we would like each experience to improve the agent, but this does not reliably happen. Some type of memory management policy is a natural choice, and the approach proposed here is conceptually one of the most straightforward learning-based approaches. (This is a good thing!)
Critical weaknesses
Two main weaknesses in my mind:
- Exposition/clarity is kind of poor. Especially from section 3 onward. The experiment in the preliminary study is not clearly explained (semantic score is undefined, for example). The setting/preliminaries are omitted, and the derivation of the algorithm is kind of rushed. I can grok that it is basically policy gradient for the memory controller on a natural "empirical-based" objective. But many questions about the details are poorly explained: what exactly is the action space of the controller? Does it write in natural language or propose a structured action space? (It seems to be some combination of the two...). The lack of clarity makes it quite hard to appreciate the technical novelty of the approach. (There are more such clarity issues in the experiment section. It would be worthwhile to give the paper a very careful pass for readability)
- I have some issues about the "overfitting" motivation and the technical approach to resolve this. The first is that I'm not sure this is a "real problem" in the sense that it actually makes a difference in practice. Were there any experiments to assess whether overfitting is a real problem that needs to be addressed in this particular setting? The second issues is with the stochastic rounding. It seems like there is a much simpler approach to address this, which is basically the "sgd" style approach. Here every time we want to query for feedback, we use just a single fresh example/task from the feedback set (so we can only do n queries). Then we have no risk of overfitting, since each query uses a fresh sample, everything is conditionally independent with the correct expectation. The only issue is the variance, but that is naturally bounded by the range, i.e., at most 2 or something. Anyway it just seems much easier and much more standard than the stochastic rounding approach, and possibly has a better guarantee, with no dependence on r?
Questions for authors
See above, in particular, 1. Is there experimental evidence that overfitting is a problem in this setting and 2. why not use the "sgd style" argument to avoid overfitting?
4 comments
Response to Reviewer wt9s (2/2)
W2(b) / Q2. Why not use one fresh feedback task per update?
Your fresh-sample argument is correct when each feedback task is independent of the current history. With a fixed pool of twelve tasks, fresh-only feedback permits twelve updates. ICME studies how this small pool can support continued learning when .
The fixed-pool comparison lets every method complete 768 visible edits. Fresh-once freezes editor parameters after its twelve feedback opportunities:
| Method | Distinct feedback tasks | Feedback-update opportunities | Editor | Final-bank |
|---|---|---|---|---|
| Fixed editor | 0 | 0 | .556 | .558 |
| Fresh-once, then frozen | 12 | 12 | .572 | .570 |
| Reused full-precision mean | 12 | 768 | .650 | .642 |
| ICME-SR-4 | 12 | 768 | .636 | .632 |
This comparison fixes the available task pool and shows the benefit of continued feedback reuse. A second experiment tests your proposal at equal update and feedback-evaluation counts. Both conditions use full-precision single-task gains, 192 updates, 192 visible edits, and 384 paired-evaluation episodes, with caching disabled:
| Feedback sampling | Available feedback groups | Editor | Final-bank |
|---|---|---|---|
| One fresh task per step | 192 | .615 | .606 |
| Repeated sampling from the original pool | ≤12 | .589 | .575 |
These ten-run means favor fresh feedback by .026 in and .031 in audit utility. Your alternative is effective when independent tasks are plentiful. With only twelve tasks available, continued reuse raises from .572 for fresh-once to .636 for SR-4. The two experiments therefore identify the data regimes in which each approach is useful.
For the variance argument, the policy-gradient update multiplies the bounded gain by . Bounding its variance also requires control of this score-function factor. Fresh sampling gives a conditionally unbiased local signal. Generalization guarantees for the final learned output require further assumptions, such as those used in SGD stability analysis [1]. ICME's analysis controls final-bank optimism through the information released by its feedback interface.
[1] Train faster, generalize better: Stability of stochastic gradient descent.
Thanks!
Thanks for the responses. It all makes sense to me and I do feel the paper is above the bar. I'll keep my score.
Thank You for Your Encouraging Feedback
We sincerely thank the reviewer for taking the time to carefully consider our responses and for the encouraging assessment of our work. We greatly appreciate your thoughtful feedback and continued support!
Response to Reviewer wt9s (1/2)
Thank you for taking the time to review our paper and for your thoughtful suggestions on clarity and feedback reuse. Below, we clarify the learning setup and examine your fresh-sample alternative through analysis and experiments.
Response. Your reading of the action space is correct. Each action is a JSON
ADD,REPLACE,DELETE, orKEEPpatch. Thewhenandstrategyfields contain natural language. The editor receives the current bank and visible episode experience. Invalid or over-capacity patches leave the bank unchanged. Valid patches are applied even when harmful (Appendix A.1). The actor and retriever stay frozen throughout editor training.In the preliminary study, six design-data streams produce 192 append events, each evaluated on twelve paired probes. The semantic score is a frozen judge's 1–5 assessment of whether a lesson is supported by its source experience and appears reusable (Appendix B.8.1). This score is used only for the preliminary analysis. Table 5 shows that 29/122 high-scoring writes are harmful, motivating an outcome-based editing objective.
Editor training uses the actual change in whole-bank utility. At fixed state s with incumbent bank M−,
Δ(p)=JF(Apply(M−,p))−JF(M−),
[\widehat\Delta(p)\nabla_\phi\log q_\phi(p\mid s)] =\nabla_\phi\mathbb E_p[\Delta(p)].$$ The pre-edit utility is independent of the sampled action, and stochastic rounding preserves the expected gain. Together, these give the local empirical-gradient identity in Eq. (7). Evaluating the whole bank captures the edit's effects on both memory content and retrieval. --- > #### **W2(a) / Q1. Is feedback overfitting a real issue, and is the feedback set a test set?** ${\color{green}\mathbf{Response.}}$ $D_F$ is a private **training-side feedback set**, disjoint from the audit set $D_A$ and official evaluation tasks. It is queried only during training (Appendix B.2). Table 11 measures the final bank's feedback–audit gap. The additional matched-noise control gives the following comparison: | Feedback | $J_F$ | $J_A$ | $J_F-J_A$ | | --- | ---: | ---: | ---: | | Fixed editor | .561 | .558 | .003 ± .006 | | Full precision | .675 | .642 | .033 ± .010 | | SR-4 | .653 | .632 | .021 ± .008 | | Full precision + matched noise | .665 | .638 | .027 | The intervals are the original 95% CIs. The additional row is a ten-run mean. Training trajectories also show widening gaps against a fixed, disjoint diagnostic split $D_D$: | Step | Full-precision $J_F-J_D$ | SR-4 $J_F-J_D$ | Matched-noise $J_F-J_D$ | | --- | ---: | ---: | ---: | | 0 | .003 | .003 | .003 | | 192 | .024 | .014 | .023 | | 384 | .026 | .016 | .024 | | 576 | .027 | .021 | .024 | | 768 | .037 | .024 | .031 | $D_D$ is a separate diagnostic set from $D_A$. These read-only evaluations use prespecified checkpoints and remain outside the training loop. The results show that feedback optimism grows during reuse and exceeds the fixed-editor reference. SR-4 reduces the final audit gap from .033 to .021. Full precision achieves higher absolute audit utility (.642 versus .632), making the utility–optimism trade-off explicit.