acceptodds
Under review as a conference paper at ICLR 2027

Information-Constrained Memory Evolution for Self-Improving Language Agents

Abstract

External memory helps language agents improve through experience without retraining their acting models, but seemingly useful lessons can harm decisions and repeated feedback can encourage overfitting. We aim to learn memory edits that improve task performance while controlling how much information the editor receives from reused feedback tasks. We propose Information-Constrained Memory Evolution (ICME), which evaluates each edit through its effect on the entire memory bank, keeping the acting model and retriever fixed. A limited-bit feedback channel trains the editor from evaluations of the same tasks before and after each edit, preserving the expected learning signal; after training, the editor updates fresh memory banks without evaluator feedback. In matched experiments with Qwen3-14B, ICME improves success rates over Auto-Dreamer by 3.9 percentage points on unseen ALFWorld tasks and 5.9 points on ScienceWorld. By linking feedback capacity to a bound on the final memory bank's expected overestimation of performance, this work provides a principled basis for learning from memory edits under repeated feedback reuse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, agent memory, self-improvement, memory editing, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_wt9sOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops and RL-based policy for controlling the memory available to an agent. This addresses many of the basic problems of memory management in language model agents: adding useful memories can actually degrade performance, repeating memories can cause stickiness and degrade exploration/diversity, etc. The method developed in the paper uses RL to train a memory controller while keeping the agent fixed, and while simultaneously limiting feedback from a "test set" of tasks to prevent overfitting. The approach, at least conceptually, makes a lot of sense, but there are a number of details that probably matter in getting good practical performance. These are not discussed in particular detail or carefully justified but the experimental evaluation is fairly thorough, suggesting that the choices are at least reasonable. There are some issues with exposition/clarity and I have a couple of technical questions I will raise below.

Critical strengths

The paper addresses a real problem in modern language model agents, namely that agents behave rather strangely in long context scenarios, where "useful" memories can degrade performance, result in lack of exploration, etc. When agent are learning from experience in-context, we would like each experience to improve the agent, but this does not reliably happen. Some type of memory management policy is a natural choice, and the approach proposed here is conceptually one of the most straightforward learning-based approaches. (This is a good thing!)

Critical weaknesses

Two main weaknesses in my mind:

  1. Exposition/clarity is kind of poor. Especially from section 3 onward. The experiment in the preliminary study is not clearly explained (semantic score is undefined, for example). The setting/preliminaries are omitted, and the derivation of the algorithm is kind of rushed. I can grok that it is basically policy gradient for the memory controller on a natural "empirical-based" objective. But many questions about the details are poorly explained: what exactly is the action space of the controller? Does it write in natural language or propose a structured action space? (It seems to be some combination of the two...). The lack of clarity makes it quite hard to appreciate the technical novelty of the approach. (There are more such clarity issues in the experiment section. It would be worthwhile to give the paper a very careful pass for readability)
  2. I have some issues about the "overfitting" motivation and the technical approach to resolve this. The first is that I'm not sure this is a "real problem" in the sense that it actually makes a difference in practice. Were there any experiments to assess whether overfitting is a real problem that needs to be addressed in this particular setting? The second issues is with the stochastic rounding. It seems like there is a much simpler approach to address this, which is basically the "sgd" style approach. Here every time we want to query for feedback, we use just a single fresh example/task from the feedback set (so we can only do n queries). Then we have no risk of overfitting, since each query uses a fresh sample, everything is conditionally independent with the correct expectation. The only issue is the variance, but that is naturally bounded by the range, i.e., at most 2 or something. Anyway it just seems much easier and much more standard than the stochastic rounding approach, and possibly has a better guarantee, with no dependence on r?
Questions for authors

See above, in particular, 1. Is there experimental evidence that overfitting is a problem in this setting and 2. why not use the "sgd style" argument to avoid overfitting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wt9s (1/2)

Thank you for taking the time to review our paper and for your thoughtful suggestions on clarity and feedback reuse. Below, we clarify the learning setup and examine your fresh-sample alternative through analysis and experiments.


W1. Clarify the setting, semantic score, action space, and algorithm derivation.

Response.{\color{green}\mathbf{Response.}} Your reading of the action space is correct. Each action is a JSON ADD, REPLACE, DELETE, or KEEP patch. The when and strategy fields contain natural language. The editor receives the current bank and visible episode experience. Invalid or over-capacity patches leave the bank unchanged. Valid patches are applied even when harmful (Appendix A.1). The actor and retriever stay frozen throughout editor training.

In the preliminary study, six design-data streams produce 192 append events, each evaluated on twelve paired probes. The semantic score is a frozen judge's 1–5 assessment of whether a lesson is supported by its source experience and appears reusable (Appendix B.8.1). This score is used only for the preliminary analysis. Table 5 shows that 29/122 high-scoring writes are harmful, motivating an outcome-based editing objective.

Editor training uses the actual change in whole-bank utility. At fixed state ss with incumbent bank M−\mathcal M^-,

Δ(p)=J^F(Apply⁡(M−,p))−J^F(M−),\Delta(p)=\widehat J_F(\operatorname{Apply}(\mathcal M^-,p))-\widehat J_F(\mathcal M^-),

[\widehat\Delta(p)\nabla_\phi\log q_\phi(p\mid s)] =\nabla_\phi\mathbb E_p[\Delta(p)].$$ The pre-edit utility is independent of the sampled action, and stochastic rounding preserves the expected gain. Together, these give the local empirical-gradient identity in Eq. (7). Evaluating the whole bank captures the edit's effects on both memory content and retrieval. --- > #### **W2(a) / Q1. Is feedback overfitting a real issue, and is the feedback set a test set?** ${\color{green}\mathbf{Response.}}$ $D_F$ is a private **training-side feedback set**, disjoint from the audit set $D_A$ and official evaluation tasks. It is queried only during training (Appendix B.2). Table 11 measures the final bank's feedback–audit gap. The additional matched-noise control gives the following comparison: | Feedback | $J_F$ | $J_A$ | $J_F-J_A$ | | --- | ---: | ---: | ---: | | Fixed editor | .561 | .558 | .003 ± .006 | | Full precision | .675 | .642 | .033 ± .010 | | SR-4 | .653 | .632 | .021 ± .008 | | Full precision + matched noise | .665 | .638 | .027 | The intervals are the original 95% CIs. The additional row is a ten-run mean. Training trajectories also show widening gaps against a fixed, disjoint diagnostic split $D_D$: | Step | Full-precision $J_F-J_D$ | SR-4 $J_F-J_D$ | Matched-noise $J_F-J_D$ | | --- | ---: | ---: | ---: | | 0 | .003 | .003 | .003 | | 192 | .024 | .014 | .023 | | 384 | .026 | .016 | .024 | | 576 | .027 | .021 | .024 | | 768 | .037 | .024 | .031 | $D_D$ is a separate diagnostic set from $D_A$. These read-only evaluations use prespecified checkpoints and remain outside the training loop. The results show that feedback optimism grows during reuse and exceeds the fixed-editor reference. SR-4 reduces the final audit gap from .033 to .021. Full precision achieves higher absolute audit utility (.642 versus .632), making the utility–optimism trade-off explicit.
AuthorsOpenReview ↗

Response to Reviewer wt9s (2/2)

W2(b) / Q2. Why not use one fresh feedback task per update?

Response.{\color{green}\mathbf{Response.}} Your fresh-sample argument is correct when each feedback task is independent of the current history. With a fixed pool of twelve tasks, fresh-only feedback permits twelve updates. ICME studies how this small pool can support continued learning when T≫nT\gg n.

The fixed-pool comparison lets every method complete 768 visible edits. Fresh-once freezes editor parameters after its twelve feedback opportunities:

MethodDistinct feedback tasksFeedback-update opportunitiesEditor J32J_{32}Final-bank JAJ_A
Fixed editor00.556.558
Fresh-once, then frozen1212.572.570
Reused full-precision mean12768.650.642
ICME-SR-412768.636.632

This comparison fixes the available task pool and shows the benefit of continued feedback reuse. A second experiment tests your proposal at equal update and feedback-evaluation counts. Both conditions use full-precision single-task gains, 192 updates, 192 visible edits, and 384 paired-evaluation episodes, with caching disabled:

Feedback samplingAvailable feedback groupsEditor J32J_{32}Final-bank JAJ_A
One fresh task per step192.615.606
Repeated sampling from the original pool≤12.589.575

These ten-run means favor fresh feedback by .026 in J32J_{32} and .031 in audit utility. Your alternative is effective when independent tasks are plentiful. With only twelve tasks available, continued reuse raises J32J_{32} from .572 for fresh-once to .636 for SR-4. The two experiments therefore identify the data regimes in which each approach is useful.

For the variance argument, the policy-gradient update multiplies the bounded gain by ∇log⁡qϕ(p∣s)\nabla\log q_\phi(p\mid s). Bounding its variance also requires control of this score-function factor. Fresh sampling gives a conditionally unbiased local signal. Generalization guarantees for the final learned output require further assumptions, such as those used in SGD stability analysis [1]. ICME's analysis controls final-bank optimism through the information released by its feedback interface.

[1] Train faster, generalize better: Stability of stochastic gradient descent.

Reviewer_wt9sOpenReview ↗

Thanks!

Thanks for the responses. It all makes sense to me and I do feel the paper is above the bar. I'll keep my score.

AuthorsOpenReview ↗

Thank You for Your Encouraging Feedback

We sincerely thank the reviewer for taking the time to carefully consider our responses and for the encouraging assessment of our work. We greatly appreciate your thoughtful feedback and continued support!

Reviewer_cAT4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ICME, a method for optimizing memory management in LLM agents by learning a memory editor. The authors use the change in the utility of the entire memory bank before and after an edit as the reward, and encode the reward into a finite-bit feedback signal through stochastic quantization. The paper further analyzes the relationship between limited feedback and the generalization of the final memory bank from an information-theoretic perspective, and evaluates the method's performance and transfer capabilities across multiple interactive environments.

Critical strengths
  1. The problem is practically meaningful. The paper focuses on the impact of memory edits on downstream task utility and provides feedback for memory editing based on changes in the utility of the entire memory bank. (Sections 4.1–4.3)

  2. The experimental coverage is broad. The authors evaluate ICME across multiple interactive environments and model configurations, examining editor transfer, memory transfer, and performance under different feedback budgets. (Sections 5.1–5.3)

  3. The paper provides a targeted analysis of feedback compression. The authors compare full-precision feedback, deterministic quantization, and stochastic quantization at different precision levels, and evaluate the utility gap between the feedback set and the audit set for the final memory bank. (Appendix C.4, Tables 9–11)

Critical weaknesses
  1. The methodological novelty needs to be clarified further. Although the paper discusses related memory management methods, it does not sufficiently explain what substantive methodological advances these design choices offer over existing LLM memory RL methods. At present, the main innovation I can identify is the use of stochastic quantization. (Sections 3–4)

  2. The practical explanatory power of the theoretical guarantee is limited. Theorem 1 analyzes the relationship between feedback information capacity and the expected optimism of the final memory bank. However, the authors acknowledge that, under the setting T=192,r=4,n=12T=192, r=4, n=12, the bound degenerates into a vacuous bound. Consequently, the result does not provide a useful quantitative guarantee under this setting. (Section 4.4, Lines 261–277)

  3. The mechanism underlying the effect of stochastic quantization has not been sufficiently isolated. The experiments show different trade-offs in utility and in the gap between the feedback-set and audit-set performance under stochastic quantization. However, stochastic quantization also changes the reward noise. The current results are insufficient to clearly distinguish the respective contributions of information-capacity constraints and stochastic optimization noise. (Section 5.3; Appendix C.4, Tables 9–11)

  4. The figures are not clearly presented. I could not understand Figure 2, particularly Figure 2b and its explanation in Appendix B.8.3.

Questions for authors
  1. What are the specific core methodological innovations of ICME compared with existing memory management methods? What benefits does stochastic quantization provide beyond conventional reward quantization?

  2. Could additional experiments be conducted to distinguish the effects of feedback information-capacity constraints from those of the optimization noise introduced by stochastic quantization on final generalization performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cAT4 (1/2)

Thank you for your time and thoughtful feedback on the contribution, theory, and experiments. Below, we clarify the role of the feedback constraint and present the additional control experiment you suggested.


W1 / Q1. What is the methodological novelty, and what does stochastic quantization contribute?

Response.{\color{green}\mathbf{Response.}} ICME's central idea is to learn useful memory changes under an explicit budget on feedback exposure. A plausible lesson can displace useful memory, so an edit's value depends on its effect on the whole bank. Repeatedly optimizing this value on a small feedback set can also make its measured performance optimistic. The editing objective and feedback constraint address these two connected effects.

The experiments motivate both parts: 23.8% of high-semantic-score writes are harmful (Table 5), and full-precision feedback produces a .033 feedback–audit gap versus .003 for the fixed editor (Table 11). SkillOS [1] learns from grouped downstream outcomes. Auto-Dreamer [2] learns region consolidation with downstream and masking-based rewards. ICME evaluates each actual bank transition through a finite, isolated feedback interface. This makes cumulative feedback exposure explicit and supports an analysis of the final bank's dependence on the reused tasks.

Stochastic rounding implements this constraint while preserving

\qquad \widehat\Delta=0\ \text{when}\ \Delta=0.$$ Consequently, Eq. (7) preserves the expected one-step empirical policy gradient. Nearest-grid rounding generally introduces bias. The matched comparison in Table 9 illustrates the practical difference: | Feedback | Bits/update | Editor $J_{32}$ | AUC | MDD ↓ | | --- | ---: | ---: | ---: | ---: | | Nearest-grid | 4 | .619 | .575 | .057 | | ICME stochastic | 4 | .636 | .592 | .044 | Unbiased stochastic quantization is an established technique [3]. In ICME, it preserves the expected local learning signal within the feedback budget. The same interface also gives the information bound in Theorem 1. The noise-matched experiment below examines its effect on optimization. [1] SkillOS: Learning Skill Curation for Self-Evolving Agents. [2] Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents. [3] QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. --- > #### **W2. Theorem 1 is numerically vacuous at the experimental sample size.** ${\color{green}\mathbf{Response.}}$ You are right about the numerical scale. The theorem gives an architecture-independent bound on the information acquired through repeated feedback. Under its isolation and independent-sampling assumptions, $$I(D_F;\mathcal M_T\mid V) \le (T-1)\ln(2^r-1)-\sum_{t<T}\mathbb E h_2(\rho_t).$$ The $T-1$ prefix follows because the final edit precedes its reward. The expression shows how feedback precision, update count, and private rounding entropy control the information bound. Dependence stored in editor parameters persists across memory resets. Information-theoretic generalization analysis [1] then bounds the final bank's expected signed optimism. At $T=192,r=4,n=12$ and at the default $T=768$, the worst-case bound reaches the trivial range limit (§4.4 and Appendix D.4.11). Tables 9–11 directly quantify the practical precision–utility trade-off: | Feedback | Editor $J_{32}$ ↑ | Final-bank $J_A$ ↑ | Feedback–audit gap | | --- | ---: | ---: | ---: | | Full precision | .650 | .642 | .033 | | SR-2 | .607 | .607 | .013 | | SR-4 | .636 | .632 | .021 | | SR-8 | .645 | .640 | .026 | The theory identifies the dependence controlled by the feedback interface. The measurements show the resulting operating choices: coarser feedback has a smaller gap and lower absolute utility in these experiments. Reporting both makes the cost and benefit of each precision level explicit. [1] Controlling Bias in Adaptive Data Analysis Using Information Theory.
AuthorsOpenReview ↗

Response to Reviewer cAT4 (2/2)

W3 / Q2. Can you separate limited feedback from stochastic optimization noise?

Response.{\color{green}\mathbf{Response.}} To test your suggestion, we evaluated a high-precision control with the same conditional reward mean and variance as SR-4. For h=1/7h=1/7 and the same within-bin position ρ\rho, it returns

\epsilon\mid\Delta\sim\mathrm{Uniform}[-\sqrt{3}\sigma,\sqrt{3}\sigma], \quad \sigma^2=h^2\rho(1-\rho).$$ The control transmits float32 without clipping and preserves exact zero. All conditions share $T=768$, feedback tasks, paired seeds, and the editing schedule. The table reports ten-run means. | Feedback | Editor $J_{32}$ ↑ | Final-bank $J_F$ | Final-bank $J_A$ ↑ | Gap | | --- | ---: | ---: | ---: | ---: | | Full precision | .650 | .675 | .642 | .033 | | Full precision + matched noise | .642 | .665 | .638 | .027 | | SR-4 | .636 | .653 | .632 | .021 | Matching the reward variance reduces the observed gap from .033 to .027. SR-4 reaches .021, giving a further .006 reduction in the ten-run means. The noise-matched control therefore captures part of the observed reduction. The remaining comparison involves both the finite alphabet and differences in higher moments, support, sign behavior, and zero-code probabilities. Theorem 1 establishes the finite alphabet's information bound. --- > #### **W4. Figure 2, especially panel b, is difficult to interpret.** ${\color{green}\mathbf{Response.}}$ You can read Figure 2 as **one write, twelve paired task outcomes**. The study contains 192 append events on design data. All scores are used for post-hoc analysis, independently of writer training. In panel a, each row is a write and each column is a probe. Color shows the change in success ($-1,0,+1$), and rows are ordered by semantic score. In panel b, each point is a write. Its horizontal position shows bank occupancy **before** writing, and its vertical position shows the fraction of retrieval results that change. Color represents mean utility change. Point size indicates how many probes change from success to failure or vice versa. Thus the plot shows both changes in retrieved memory and their task-level consequences. The contours smooth the fraction of harmful writes. Table 5 gives the underlying counts: | Group | Definition | Harmful writes | | --- | --- | ---: | | High semantic score | Score ≥4/5 | 29/122 = 23.8% | | Low competition | Occupancy <.30 and displacement <.30 | 11/60 = 18.3% | | Populated/perturbed | Occupancy >.65 and displacement >.50 | 13/22 = 59.1% | Even among high-scoring lessons, 23.8% of writes reduce utility. The harmful-write rate is also higher in populated banks with substantial retrieval displacement. The contours visualize this association, and the counts show its magnitude directly. This motivates evaluating an edit through its effect on the whole bank.
Reviewer_5G1NOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ICME, a method for training a memory editor model for memory-based continual learning agents. At each step, the memory banks before and after an edit are evaluated on a separate feedback set DfD_f to quantify the quality of the edit. This signal is then used to train the memory editor model using RL. ICME uses an rr-bit limited feedback channel to provide coarse-grained rewards, with the aim of limiting feedback exposure and controlling overfitting to the reused feedback set.

ICME is empirically evaluated on ALFWorld, ScienceWorld, WebShop, WebArena, and AppWorld, and outperforms baselines such as ReasoningBank and Auto-Dreamer in matched comparisons. In addition to these evaluations, the paper investigates design decisions such as the feedback precision rr, disentangles content and retrieval effects, and examines behavioral differences between trained and untrained editors.

Critical strengths
  • The work is well-organized and quite readable. (Disclaimer: I did not closely check the theoretical analysis in section 4.2-4.4, as it is outside my expertise.)
  • Some findings are useful: as shown in Figure 2, even semantically plausible lessons do not always improve performance on future tasks.
  • The visualizations present the findings in an accessible way.
Critical weaknesses
  • The paper could be strengthened by providing qualitative examples of memory edits produced by the trained and untrained editors. This would make the argument more convincing.
  • Certain experimental details are missing, or the reasoning behind the experimental design is not explained clearly. See questions below.
Questions for authors
  • What is the rationale for setting a memory limit of 1024 tokens while also using a retriever? One possible alternative is to include all 1024 tokens in the context. Would it be possible to report the performance of this alternative for reference?
  • How is the online evaluation in Table 1 conducted? Do you create 32-task streams and reset the memory after each stream? For example, ALFWorld has 140 seen and 134 unseen evaluation tasks. Are these divided into 32-task streams, or is memory maintained across the entire evaluation sequence?
  • I am quite surprised that RL training using REINFORCE, with a single sampled edit per update, leads to stable training. Did I miss any details that help stabilize training? Could the authors include training curves in a future version of the paper and describe the observed training dynamics in the rebuttal?
  • How would you characterize the similarities and differences between your work and SkillOS: Learning Skill Curation for Self-Evolving Agents?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5G1N (1/2)

Thank you for the time you spent reviewing our paper and for your helpful questions about the editor's behavior and evaluation. We address each point below with paired editing examples, additional results, and protocol details.


W1. Qualitative examples of trained and untrained memory editors.

Response.{\color{green}\mathbf{Response.}} Here are two examples of the comparison you suggest. In each case, both editors receive the same pre-bank and visible experience. The output fields are quoted verbatim.

Case 1 — ALFWorld: recognizing sufficient memory. The actor stores an already-clean mug directly in a cabinet. Pre-bank entry 4 already says: “Check whether the object already satisfies the requested state. If so, move it directly to the target receptacle.” Entry 9 specifies washing for dirty objects.

Output fieldFixed editorTrained editor
OperationADDKEEP
whenWhen putting an object away—
strategyWash the object in the sink before placing it in the requested receptacle.—

The fixed edit introduces an unconditional washing rule. The trained editor's KEEP preserves the existing rule that already covers this experience. The bank and its retrieval results stay unchanged, avoiding an unnecessary competing rule.

Case 2 — ScienceWorld: refining a rule's scope. The visible experience heats water to a target temperature and then measures it. Pre-bank entry 6 says “Heat the sample before measuring its temperature” for all temperature-related tasks.

Output fieldFixed editorTrained editor
OperationADDREPLACE, ID 6
whenWhen handling a sample for temperature-related tasksWhen a task requires temperature control or measurement
strategyHeat the sample before using a thermometer to obtain a reliable temperature reading.First identify whether the task requires changing the sample temperature or measuring its current state. If a target temperature is specified, heat or cool the sample before measuring. If only the current temperature is requested, measure it directly without changing its state.

The trained edit separates temperature control from measurement and states when each procedure applies. These cases illustrate when to retain existing memory and when to refine its applicability. Table 12 provides the broader behavioral evidence: retrieval-dominant harmful edits decrease from 24.1% to 11.6%.


W2. Experimental details and design choices need clarification.

Response.{\color{green}\mathbf{Response.}} We address your four questions individually below, covering the memory budget, online evaluation, training dynamics, and relationship to SkillOS.


Q1. Why use a retriever with a 1,024-token bank? How does full injection perform?

Response.{\color{green}\mathbf{Response.}} We evaluated the full-injection alternative you suggest. The bank stores up to C=1024C=1024 tokens, and the default retriever supplies up to L=512L=512 tokens per episode. With both editor checkpoints frozen, full injection gives similar mean success rates at a higher input cost. The additional results are ten-run means:

EditorMemory injectionALF seen SR (%)ALF unseen SR (%)ScienceWorld SR (%)
FixedTop-4, L=512L=51267.960.744.0
FixedAll entries, L=1024L=102467.460.243.6
FixedMost recent, L=512L=51266.259.142.7
TrainedTop-4, L=512L=51276.469.754.8
TrainedAll entries, L=1024L=102475.969.354.4
TrainedMost recent, L=512L=51273.566.751.7

Top-4 success rates are from Table 1. The alternative injection results are additional evaluations. All conditions retain C=1024C=1024.

CostTop-4All entriesMost recent
Mean injected memory tokens/episode≈410≈870≈430
Relative episode time1.001.121.02

Top-4 uses roughly half as many memory tokens and has a relative episode time of 1.00 versus 1.12 for full injection. Its success rates are within 0.5 percentage points of full injection across all six comparisons. This performance–cost balance supports the default retrieval budget.

AuthorsOpenReview ↗

Response to Reviewer 5G1N (2/2)

Q2. Does Table 1 reset memory every 32 tasks or retain it throughout each evaluation split?

Response.{\color{green}\mathbf{Response.}} Memory is retained throughout each evaluation split. ALFWorld seen (140 tasks) and unseen (134 tasks) each start with an empty bank and share no memory. Each split runs in full, in a prespecified order. After each task's first attempt, its visible experience updates the bank for subsequent tasks. Actor and editor parameters remain frozen, and deployment never queries DFD_F.

The 32-task reset is a training setting. In Table 1, all tasks, including the final tasks in each split, follow the same continuous-memory protocol. Table 8 uses a separate, grouped editor-transfer evaluation.


Q3. How does single-edit REINFORCE train stably? Can you show training curves?

Response.{\color{green}\mathbf{Response.}} Each sampled edit is evaluated on the entire feedback set using identical before/after task snapshots. The pre-edit utility serves as an action-independent baseline. Gains lie in [−1,1][-1,1], and exact zero gains produce no update. Training uses rank-8 query/value LoRA and plain sequence-level SGD with η=10−6\eta=10^{-6} (Table 4 and Appendix B.4), without clipping, a KL penalty, resampling, or a reward-based acceptance gate.

The learning curves you requested are summarized below as ten-run means. Each frozen checkpoint is evaluated on the same fresh streams, starting from empty memory:

Training stepSR-4 J32J_{32}Full precision J32J_{32}Matched-noise J32J_{32}
00.5560.5560.556
1920.5800.5880.585
3840.6040.6150.612
5760.6220.6360.629
7680.6360.6500.642

The SR-4 and full-precision endpoints match Table 9. These are read-only evaluations of prespecified checkpoints, with query results kept separate from training and checkpoint selection. The SR-4 update diagnostics are:

MethodStepZero-update rate*All-step medianNonzero-update median (normalized)Nonzero-update P95 (normalized)
SR-419268%01.001.00
SR-438466%00.940.96
SR-457664%00.880.91
SR-476862%00.840.88

* Rates correspond to zero-feedback-code steps, which give exactly zero SGD updates. The last two columns condition on nonzero updates and are normalized separately to their step-192 values. The all-step median is reported directly as zero.

Mean utility rises across checkpoints while the median and P95 norms of nonzero updates decrease. Together, these measurements show improving checkpoint performance with smaller typical nonzero updates. Eq. (7) explains the expected one-step learning signal through its conditional gradient identity.


Q4. How does ICME relate to SkillOS?

Response.{\color{green}\mathbf{Response.}} Both methods train a curator with a frozen executor. SkillOS [1] (§3) learns skill curation across related task groups using GRPO. Its reward combines subsequent-task outcomes, operation validity, judged content quality, and compression.

ICME focuses on learning useful memory edits when a small feedback set must be reused. An edit can change both stored content and retrieval, so its reward measures the actual before/after change in whole-bank utility. Reusing this reward also exposes information about the feedback tasks. ICME limits that exposure through a finite feedback interface while preserving the expected local gradient. This connects the editing objective directly to the final-bank analysis in Theorem 1.

[1] SkillOS: Learning Skill Curation for Self-Evolving Agents.

Thank you for your response! The two qualitative examples were helpful to understand the benefits brought by ICME. The training curve summary also addressed my concerns on training stabilities.

Overall I think this paper presents meaningful contributions, so I've adjusted my score to 4. For future work, the authors may consider scaling the method to more challenging settings (longer-horizon agent tasks/learning over a larger number of tasks) or training a universal memory editor over multiple datasets.

AuthorsOpenReview ↗

Thanks for Encouraging Feedback and Suggestions

We sincerely thank the reviewer for the encouraging feedback and positive reassessment of our work! We are glad that the additional qualitative examples and training curve analysis helped address your concerns. We also appreciate your constructive suggestions which are valuable directions for future research!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.