- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Intrinsic Memory Agents, where each agent maintains structured memory updated from its own outputs through an additional LLM call. Subsequent contexts prioritize the task description, personal memory, and recent dialogue. Evaluation covers seven benchmarks using gpt-oss-120b with ten seeds, alongside a collaborative pipeline design case study using Llama-3.2-3B. Results vary across tasks, with higher judged quality in the case study and additional token costs.
Critical strengths
- Preserving relevant information and responsibilities during extended multi-agent collaboration is a practically meaningful problem.
- The evaluation covers diverse tasks, reruns multiple baselines with the same backbone, and reports results across ten seeds.
- The paper reports negative results, including degradation on some action tasks and limited benefits from carrying memory across tasks.
Critical weaknesses
-
The novelty argument relies on an inaccurate characterization of prior work. Section 2 suggests that shared memory gives all agents the same retrieved context. However, G-Memory explicitly conditions memory selection on the agent’s role and current query (its Eq. 8), while Generative Agents already maintains individual agents’ experiences. The paper should clarify its specific advance in updating private structured memory during a task and establish its value relative to these existing mechanisms.
-
The engineering contribution is insufficiently isolated and validated. The evaluation does not directly establish whether important role-specific constraints survive context truncation or outdated decisions are corrected. The template and reset/carry comparisons do not isolate private memory from shared summaries or dynamic updates from static guidance. Moreover, the manual template in Figure 4 includes planning knowledge, confounding memory structure with task guidance. Focused controls and an audit of existing trajectories would better support the central claims about role consistency.
-
The most relevant collaborative evaluation provides limited evidence. The pipeline design study uses one task, ten runs, and Llama-3.2-3B, without adequately motivating this backbone or testing whether gains persist with a stronger model. This matters because the case study most directly matches the paper’s motivation. The judge model and configuration are also unclear, and the evaluation does not adequately control for output length or independently verify engineering quality. Higher scores could therefore partly reflect more detailed descriptions.
-
The practical benefit relative to computational cost remains unclear. On HotpotQA, the manual template scores 0.480 versus 0.469 for the strongest existing baseline, while consuming approximately 4.8 times the completion tokens of the no-memory control (Tables 2 and 5). Performance is weaker on several action tasks. These results call for clearer comparisons against the strongest baselines, including statistical uncertainty and comparable computational budgets, to establish when the additional memory maintenance is worthwhile.
Questions for authors
- After correcting the discussion of G-Memory and Generative Agents, what is the principal technical or empirical contribution, and which experiment directly supports it?
- What inputs does each memory updater actually receive: the agent’s own output, observations, or the task trajectory? Could existing trajectories demonstrate constraint retention and correction of outdated decisions?
- Do the benefits survive controls for shared versus private memory and static task guidance versus dynamic updates, under comparable computational budgets?
- How robust are the case study gains to backbone strength, output length, and independent assessment of design quality? Please also identify the judge model and configuration.