acceptodds
Under review as a conference paper at ICLR 2027

Learning to Accumulate Knowledge with Mutual Information

Abstract

Large language model (LLM) agents can improve their performance by reusing knowledge distilled from past interactions. However, curating new experiences into a knowledge bank that becomes more useful as it grows remains challenging. Effective knowledge accumulation should limit redundant overlap among entries and ensure that new knowledge contributes beyond what the bank already provides. Yet training a curator with Group Relative Policy Optimization (GRPO) on standalone task success can reinforce general guidance even when it duplicates existing knowledge. Therefore, we propose Knowledge Weaver, a reinforcement learning framework that trains a language model to curate reusable knowledge from agent trajectories. We couple feedback inspired by token-wise mutual information (MI) with marginal success rewards to guide knowledge accumulation. Together, these signals encourage the curator to preserve distinct information from experience and produce entries that improve task success when added to existing knowledge. Standalone success rewards also favor entries that are useful on their own. On ALFWorld and WebShop, Knowledge Weaver achieves mean success rates of 54.0% and 42.0% with retrieved entries, exceeding GRPO by 16.9 and 18.7 percentage points, respectively. Its knowledge banks also outperform the evaluated prompt-based and established banks, including human-written banks, in overall ALFWorld success rate and WebShop score with the executor frozen.

Primary area:
reinforcement learning
Keywords:
Large Language Model Agents, Knowledge Accumulation, Mutual Information, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m44zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to extract reusable skills from agent execution trajectories while reducing redundancy in external knowledge banks. The authors train a curator with GRPO using multiple reward signals, while keeping the executor frozen. Experiments show that the resulting knowledge banks improve task success rates and transfer to other executors.

Critical strengths

The paper addresses an important distinction between the usefulness of individual skills and the utility of an accumulated knowledge bank. Its reward design considers standalone utility, marginal gains over existing knowledge, and preservation of experience-specific information, providing a well-motivated approach to learning useful and complementary skills.

Critical weaknesses

The ablation lacks an SR + MSR baseline without MI feedback. Since MSR already rewards contributions beyond existing knowledge, this comparison is needed to establish the additional value of MI in the full framework. The current results show that MI improves over SR-only training, but do not isolate its contribution when marginal success feedback is already available.

Questions for authors

Could the authors report an SR + MSR baseline without MI feedback under the same training and evaluation protocol? This comparison would clarify the additional value of MI beyond marginal success feedback. Evidence of a meaningful improvement over this baseline could affect my assessment.

Reviewer_R4ZYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Knowledge Weaver, a reinforcement learning framework designed to train a curator agent that extracts reusable, natural-language guidance from past trajectories while keeping the downstream executor model frozen. The curator is trained using a composite reward signal that combines three components: standalone task execution success, the marginal improvement in success rate when retrieved guidance is added, and a token-level mutual information bonus based on generation likelihood differences across source contexts. During training, a memory bank expands by admitting guidance entries that yield positive marginal gains. Once curator training is complete, it constructs a final, separate knowledge bank used for test-time evaluation.

Critical strengths

The method addresses a meaningful distinction between individual usefulness and usefulness within an existing bank.

Critical weaknesses
  1. Unclear Conceptual Novelty and Distinction from Prior Skill Curators

The paper frames Knowledge Weaver as learning reusable knowledge rather than raw episodic memory, but operationally, it acts as a learned writer of external procedural guidance. This approach closely mirrors established precedents: SkillOS [1] trains an RL-based skill curator while keeping the executor frozen to enable cross-executor transfer, and Mem-α\alpha [2] optimizes memory-writing operations via downstream execution feedback. SKL [3] provides an example of knowledge definition and trains a model to formalize it. The paper needs to better position its contributions relative to these frameworks to clarify what its specific approach contributes to the broader external memory learning literature.

  1. Theoretical and Operational Gaps in the MI-Inspired Signal

The proposed source-contrast reward is motivated as capturing complementary knowledge, but source-dependent token generation does not necessarily equate to useful abstraction. Two guidance entries might use different surface-level phrasing while conveying identical operational rules, or conversely, source-specific wording might simply capture incidental environment details. Furthermore, Equation (6) rewards absolute log-likelihood contrasts—including tokens that are less likely under their true source context—without a formal derivation showing that this clipped local bonus optimizes mutual information or guarantees bank-level complementarity.

  1. Incomplete Validation of Knowledge Accumulation

While the paper claims to improve knowledge accumulation, the experiments primarily evaluate static, pre-generated memory banks. Figures 3 and 4 vary the number of retrieved items at test time rather than evaluating progressive bank expansion over sequential environment steps. Additionally, metrics like co-retrieval Jaccard similarity measure retrieval overlap rather than semantic non-redundancy, leaving open whether successive bank updates actually yield more complementary, reusable knowledge over time.

[1] Ouyang, S., Yan, J., Chen, Y., Han, R., Wang, Z., Mishra, B. D., ... & Lee, C. Y. (2026). Skillos: Learning skill curation for self-evolving agents. arXiv preprint arXiv:2605.06614.

[2] Wang, Y., Takanobu, R., Liang, Z., Mao, Y., Hu, Y., McAuley, J., & Wu, X. (2025). Mem-{\alpha}: Learning memory construction via reinforcement learning. arXiv preprint arXiv:2509.25911.

[3] Song, Y., Feng, X., Liu, B., Cui, X., Fu, H., Liu, Z., ... & Wang, J. (2026). Learning stateful predictive knowledge from experience. arXiv preprint arXiv:2607.28638.

Questions for authors
  1. What specific capability or property does Knowledge Weaver learn beyond generating useful procedural guidance for a frozen executor? Please clarify its relationship to existing memory-writing and frozen-executor skill-curation methods, particularly Mem-α\alpha and SkillOS.

  2. What is the performance of the SR+MSRSR+MSR baseline (without the MI-inspired signal) under the same bank-admission schedule and compute budget?

  3. How does the curator perform when trajectory evidence is removed or shuffled while retaining task descriptions?

  4. Were all baselines and ablations strictly matched for total executor rollouts, cross-context scoring steps, and overall compute budget?

Reviewer_sxxkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This draft proposes Knowledge Weaver, which trains an LLM curator to extract reusable knowledge from agent trajectories while keeping the downstream executor frozen. Its objective combines standalone task success, marginal success improvement when an entry is appended to retrieved knowledge, and token-level feedback based on how generation probabilities change across source contexts. The latter uses a capped absolute log-likelihood contrast and is described as MI-inspired feedback. During training, entries with positive measured marginal gains enter an evolving bank. After training, the curator generates separate evaluation banks. The paper also introduces an idealized Gaussian utility-coverage model to illustrate how overlapping entries can provide less collective coverage despite equal individual utility.

Critical strengths

The paper identifies knowledge that improves an existing collection rather than individually useful knowledge. This distinction is important because optimizing reusable knowledge requires evaluating its contribution in context, not merely its standalone quality. In the paper the marginal-success reward operationalizes this distinction by comparing bank-plus-entry execution with bank-only execution on matched tasks. Such formulation permits negative gains from interference rather than assuming every additional entry is beneficial.

On ALFWorld and WebShop, the method reports success rates of 54.0% and 42.0%, respectively, improving over standalone-success GRPO by 16.9 and 18.7 percentage points. The gains over standalone-success GRPO are large on both benchmarks. The SR+MI ablation also improves substantially over SR-only training, particularly on WebShop.

Critical weaknesses

Table 2 compares SR, SR+MI, and SR+MI+MSR, but omits SR+MSR. Consequently, it establishes that MI helps standalone-success training, but not that MI adds value once bank-relative execution feedback is present. That missing comparison is central to judging whether the proposed information signal is necessary for the full method.

Equation 6 rewards the absolute value of a source-versus-alternative likelihood contrast. A token receives positive credit even when it is substantially less likely under its source context than under alternative contexts. The draft appropriately calls this “MI-inspired,” but does not establish why this particular unsigned reward preferentially preserves useful experience-specific information rather than incidental details.

Questions for authors

What decisions, if any, were influenced by monitoring ALFWorld valid_unseen? Were the step-40 checkpoint, hyperparameters, and bank-admission schedule fixed before examining these results? Was any separate selection split used?

Do the three reported seeds correspond to independently trained curators, separately sampled evaluation banks from one checkpoint, or execution repetitions? What variability is observed for the reward ablations, particularly the additional WebShop gain from MSR?

1 comment

Thank the authors for your responses and the additional experimental results, which have addressed my main concerns. I have raised my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.