- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
A well motivated work studying and proposing how to combine textual and code memory more effectively. This paper is very well written and easily to follow with insightful study upfront on the first 1-2 pages immediately attract readers. Then the authors propose an intuitive method (Metis) inspired by their study to consolidate both memory and show strong empirical evidence to demonstrate the effectiveness of Metis.
Overall, I think is a very solid work with minor improvement suggestions.
Critical strengths
- I really enjoy the writing of this paper where authors clearly demonstrate the novel problem in the introduction.
- The proposed method is intuitively and well motivated by the actual problem, and the writing does not over-complicate the approach with just enought math to define and clarify the method.
- Experiments are comprehensive and clearly demonstrate the benefit of Metis with improved performance and reduced token usage.
Critical weaknesses
- Some design choices can be further clarified or motivated. For example, why textual memory is defined with the 3 components between L203-204.
- There are some hyper-parameters defined in the approach such as N in L222-223 and budget in L258. Provide supporting study on these hyper-parameters can make the paper stronger.
- Some notations can be further improved. For example, L252-253, p is defined as a collection of functions with p as inputs. This is not very rigorous. Suggestion modify the notations.
- The analysis should include the textual / coding memory ratio for all baselines compared with Metis - this will show better where the gains of Metic come from.
Questions for authors
- L252-253, potential typo of "3"?
- Do you think the design of the current memory can overfit the existing tasks? A justification of why it is generally applicable would be better besides showing empirical results on the 3 benchmarks (which can also be cherry picked).
5 comments
W2.a Related-Task History Budget ().
When codification is triggered, the codifier receives two sets of historical queries: candidate queries , newly collected since the previous codification round; and related queries , previously consumed by the codifier. Their sizes are governed by the codification threshold and history budget , respectively. Similar to , the related-task history provides additional context to help the codifier generate more generalizable tools. We reconstruct Metis with different values and evaluate each configuration twice, fixing and . The averaged results are shown below.
| 0 | 6 | 9 | 12 | |
|---|---|---|---|---|
| Acc | 79% | 84% | 87% | 82% |
| Code Memory Size | 19 | 11 | 15 | 20 |
| Unused Code Memory Ratio | 5.5(28.95%) | 2(18.18%) | 5(33.33%) | 9.5(47.50%) |
Analysis. The results suggest that both insufficient and excessive historical context can hurt codification. With , the codifier lacks previously observed task variations, making it more prone to generating narrowly scoped tools. Although its unused-tool ratio is lower than those at and , accuracy drops to 79%. Conversely, excessive history introduces greater task variation, making it harder to identify tools that are stable for shared routines. The increasing unused-tool ratio at larger budgets further reflects this issue. Overall, a moderate history budget provides a better balance between generalization and codification quality.
W2.c Codification Threshold ().
The codification threshold controls the size of candidate query buffer and how frequently tool generation is triggered, with larger values making codification more selective. We reconstruct Metis with different thresholds and evaluate each configuration twice, fixing . Here, represents Eager codification, while effectively corresponds to the Text-only variant. The averaged results are shown below.
| 1 | 3 | 5 | 7 | 100 | |
|---|---|---|---|---|---|
| Acc | 83% | 84% | 85% | 91% | 82% |
| Number of Tools | 52 | 14 | 8 | 7 | 0 |
| Unused Code Entry Ratio | 27.5(52.88%) | 4(28.57%) | 0 | 0.5(7.14%) | 0 |
| Code Reflection Cost | 27,639,996 | 6,821,503 | 2,879,854 | 2,483,042 | 0 |
| Codification Count | 98 | 31 | 10 | 14 | 0 |
| ReAct Turns | 7.3 | 7.66 | 7.88 | 7.91 | 8.45 |
Analysis. Increasing from 1 to 7 reduces the number of generated tools from 52 to 7 and code construction cost by 91%, while improving accuracy from 83% to 91%. The unused-tool ratio also drops substantially (52.88% to 7.14%), suggesting that recurrence helps identify tools with greater reuse potential. Although more conservative codification settings slightly increase ReAct turns, they achieve a substantially better balance between accuracy and construction cost.
**
W2.d Retrieval Top-$k$**
We evaluate different retrieval top- values on all three STATE-Bench domains using the same previously constructed memory. Top- limits the candidate pool passed to the Memory Manager, avoiding the cost of examining the entire memory library as it grows. The results are shown below.
| Metrics | 2 | 5 | 10 | 15 | 20 | 30 | 50 |
|---|---|---|---|---|---|---|---|
| Acc | 76.0% | 80.0% | 81.3% | 80.0% | 79.3% | 80.0% | 79.3% |
| Manager tokens | 3,999 | 7,987 | 12,261 | 14,719 | 16,698 | 19,497 | 21,147 |
| Steps | 7.03 | 7.07 | 7.24 | 7.22 | 7.37 | 7.25 | 7.19 |
Analysis. The results show that exposing more memories to the Manager does not necessarily improve performance. A small may exclude useful entries, reducing accuracy to 76.0% at . Conversely, increasing beyond 10 brings no further accuracy gains while steadily increasing Manager token consumption. Overall, achieves the highest accuracy (81.3%) with moderate retrieval overhead, demonstrating the importance of balancing memory coverage and selection cost.
[W3: Improve the notation definitions.]
We agree that the original notation is unclear. Our intention at L252-253 was to emphasize that these fields are specific to each plan. We will instead define each plan directly as a tuple (e.g., ,,) to denote its associated fields. We will revise the paper to improve our notation.
[W4: Textual / Coding Memory Ratio.]
Thanks for the suggestion! We actually found an interesting feature of Metis through this experiments: it automatically adjust its memory components ratios to suits different workloads. We provide a breakdown of the memory composition of Metis, SkillX, and ASI across all three benchmarks. ACE maintains a compound textual playbook and is therefore excluded from the entry-level ratio comparison. For SkillX, we separately report planning and atomic skills as textual entries and functional skills as code-bearing entries.
STATE-Bench
| Text Memories | Code Memories | Code : Text Ratio | |
|---|---|---|---|
| Metis | 266 | 29 | 0.109 |
| SkillX | 211 | 400 | 1.896 |
| ASI | - | 422 | - |
OfficeBench
| Text Memories | Code Memories | Code : Text Ratio | |
|---|---|---|---|
| Metis | 45 | 36 | 0.8 |
| SkillX | 77 | 38 | 0.494 |
| ASI | - | 56 | - |
AppWorld
| Text Memories | Code Memories | Code : Text Ratio | |
|---|---|---|---|
| Metis | 52 | 40 | 0.769 |
| SkillX | 80 | 252 | 3.15 |
| ASI | - | 162 | - |
The results reveal two observations:
- Selective codification substantially reduces tool construction. Compared with ASI, Metis generates 93.1%, 35.7%, and 75.3% fewer callable tools on STATE-Bench, OfficeBench, and AppWorld, respectively. Compared with SkillX's code-bearing functional skills, Metis produces 92.8%, 5.3%, and 84.1% fewer code entries on the same benchmarks. This further proves the effectiveness of selective codification.
- Metis adaptively adjusts its memory composition to different workloads. On STATE-Bench, which emphasizes dynamic user interactions and policy-dependent decisions, Metis predominantly retains textual experience, with a code-to-text ratio of only 0.109. In contrast, on OfficeBench and AppWorld, where reusable tool-based procedures are more prevalent, the ratios increase to 0.800 and 0.769, respectively. These results demonstrate Metis' ability to adaptively balance textual guidance and executable memory according to workload characteristics.
[Q1: Typo]
Thanks for pointing this out! We agree that the wording is unclear. "Three" refers to the additional codification-related fields, , , and , while and are shared with other text memories. We will revise the passage to make this distinction explicit.
[Q2: Discussion on the generalization of the proposed memory design.]
Frankly, current memory designs of Metis may not benefit all workloads equally. Its design primarily targets settings where (1) agents need to interact with stateful environments through tools and APIs, and (2) tasks exhibit reusable execution patterns. These conditions are common in personal assistants and potentially embodied agents, but may be less prevalent in chat-only applications or highly heterogeneous tasks. Within the targeting scopes, Metis explicitly mitigates task-level overfitting, as supported by our evaluations across benchmarks that span different task scenarios.
But broadly, the key principle behind Metis is to match memory representations to the characteristics of reusable experience. We do not claim that the current memory schema suits all scenarios. As agents become increasingly integrated into everyday life, we expect them to perform more tasks through tool-based interactions with external environments. We hope Metis's design principles can provide valuable guidance for developing more efficient self-evolving agents with memory components tailored to their specific needs.
Thanks for your kind words and insightful comments! We really appreciate your time and detailed comments. We presents extra clarifications and conducted additional experiments, and sincerely hope they can address your concern.
[W1: Clarification for certain design choices]
Thank you for the constructive suggestion! We agree that some design choices deserve clearer motivation. The three text memory categories are designed to capture distinct types of reusable experience:
uvmay requireuv run pythoninstead of a barepythoncommand.As for other key design choices, such as using recent task queries to guide reflection and retaining previously encountered queries for subsequent codification, we explain their motivations and provide supporting studies in our response to W2, as these choices are directly supported by those additional experiments. We will revise the paper to include these clarifications.
[W2: Hyperparameter Analysis]
We have conducted additional studies on four key hyperparameters: recent task window N; related-task history budget β; codification threshold θ; retrieval budget top-k. Since DeepSeek-V4-Flash is no longer available through its official API, we use DeepSeek-V4.1-Flash for these studies. Given the limited rebuttal timeline and the need to retrain Metis for each configuration, we conduct these studies on the STATE-Bench Travel split (100 training and 50 test tasks). We present the results for each hyperparameter below. We also explain the motivation for their designs, as mentioned in W1.
W2.a Recent Task Window (N).
During text reflection, the reflector receives the recent N task queries alongside the current trajectory. When task patterns exhibit temporal locality, this additional context helps the reflector identify recurring patterns rather than overfitting to the current task. We reconstruct Metis with different N values and evaluate each configuration twice, fixing β=10 and θ=3. We report the average accuracy and the proportion of unused text memory entries.
Analysis. We observe that, for the same training task, reflectors with larger N explicitly reference similar but distinct earlier queries when constructing text memories, producing more generalizable guidance rather than task-specific solutions. This is consistent with the quantitative results: larger windows generally improve accuracy (83% at N=0 vs. 89% at N=14) while reducing the proportion of unused text entries (23.94% vs. 16.67%). Though there are some fluctuations, we can see an increasing pattern here. These findings support the benefit of incorporating recent task context into memory reflection.