- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a method to search for a python program to compress the text history of LLM in a task specific way. In the setup, a LLM work as an agent that receive text observation and return text action, which form the memory. The python program is used to process the memory into a more compact form but still aim to finish the task at hand. The program modification is done via another LLM.
The experiment results show that the memory length is reduced significantly while keeping a high task success rate.
Critical strengths
-
The method produces a practical, low-cost method. The learned memory is a plain Python function that runs without any LLM call at deployment. Summarization approaches such as ACON and the LLM Summarizer instead require online LLM inference whenever they compress.
-
The experiments show that the resulting memory is still informative and rich for specific tasks,
-
Also the memory in text as well as the python program is interpretable and bring insights on tasks at hand.
Critical weaknesses
-
The methodology has unclear novelty. Using LLM to search for program for memory manipulation is not novel. For example: Learning to Continually Learn via Meta-learning Agentic Memory Designs (ALMA)
-
With the main goal being the memory saving, the tasks used all induce pretty managable full history length. In this case, it is not clear to me the point of trading memory reduction at the cost of even 1% lower success rate.
-
It seems that modifying text history is not KV cache friendly and may induce re-computation.
-
What do the existing search records show about improvement beyond the first LLM-generated structured updater, and which improvements depend on population-based search rather than ordinary execution-guided code repair? If the task specification is given in great detail, it is possible that an LLM can directly give a pretty good program in one go?
Questions for authors
-
How does the method differ from ALMA beyond the choice of a deterministic recursive update?
-
Can authors discuss the impact on the usage of KV cache and inference efficiency?
-
Can authors provide some abliation on the effectivenss of different search components?
3 comments
Response by Authors [2/2]
W1. Novelty relative to ALMA.
Our contribution is learning a policy-relative, within-episode state sufficient for control. A candidate state is admissible only if it preserves the fixed reference agent's success within a tolerance. Among admissible states, we optimize compactness. This follows a policy-relative notion of sufficiency that builds on classical state abstraction (see Sections 2.2–3). Proposition 1 (Section 3.1) shows that this criterion is a relaxation of both behavior and performance matching.
ALMA addresses a different problem. It meta-learns memory designs for continual learning across tasks and selects programs based on how well they help an agent improve over a stream of tasks.
Our method asks what information from the current episode must be retained so that the same fixed LLM agent preserves its competence when the raw history is removed. We added this distinction to Sec. 2.4.
W2. Manageable history lengths and trading success for memory.
- Success is preserved (see Appendix D.1). Across 254 held-out ALFWorld and ScienceWorld instances, is +1.7 pp relative to full history when averaged over three runs. The 95% CI is [−1.8, +5.3].
- Long horizons (see Appendix E). On BabyAI GoTo, full history reaches 27.5k tokens per call while remains at about 250. Mean policy input tokens fall by 95%, with no detectable loss in success (exact McNemar test, ).
- Savings scale with the horizon (see Appendix E.1). Full history grows by 24–36 tokens per step in every original family. In 7 of 10 families, the state produced by grows by only 2–8 tokens per step.
Memory saving is not the paper's primary goal. It is a consequence of identifying a compact state sufficient for control.
W3. KV-cache friendliness.
Full history is cache-friendly because its prefix never changes. However, its KV cache and attention context grow at every step.
With , the static instructions and task remain cacheable. Only the bounded recurrent state is re-encoded. On BabyAI, this state is about 250 tokens compared with 27.5k tokens for full history. The trade-off is prefix reuse versus bounded context and KV memory.
W4. Improvement beyond the first program and the role of search.
- One-shot generation is generally insufficient. The first LLM-generated program satisfies the feasibility constraint in only 2 of 10 task families.
- Every later improvement comes from evolutionary search (see Appendix F.1). Of the 26 best-so-far improvements, 10 come directly from mutation or crossover proposals. The other 16 improve only after reflector-guided repair of such a proposal. In several families, including Pick & Place, Heat & Place, and Cool & Place, the final program appears only after multiple generations.
Q1. Difference from ALMA.
ALMA meta-learns memory designs for continual learning across tasks. We learn a policy-relative, within-episode state that must preserve a fixed agent's success, optimizing compactness among such states (see W1 and Section 2.4).
Q2. KV cache and inference efficiency.
Full history allows prefix-cache reuse, but its KV cache and attention context grow at every step. With , the static prefix stays cacheable and the recurrent state stays bounded. On BabyAI, the state is about 250 tokens compared with 27.5k tokens for full history (see W3).
Q3. Effectiveness of search components.
The first generated program is feasible in only 2 of 10 families. Among the 26 later improvements, 10 arise directly from evolutionary proposals and 16 require reflector-guided repair. This shows that the gains are not explained by one-shot generation and that both evolutionary proposal generation and repair contribute (see W4 and Appendix F.1).
Reminder
As the discussion period closes today, we would be grateful if the reviewer could take a look at our response and the revised paper. We are happy to answer any remaining questions.
Response by Authors [1/2]
Opening statement. We thank the reviewer for the review, which credits the method as practical, low-cost, and inspectable. The revision addresses each concern. We first summarize the main points of our response, and then address each weakness and question in detail.
We hope these additions address the reviewer's concerns and would be grateful if the reviewer would reconsider their score.