- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MemCo studies how independently acting LLM agents can reuse experience collected in different environments. It retains concrete experience locally, abstracts transferable records into shared global memory, and uses a Wilson lower confidence bound to determine promotion eligibility. At decision time, records are retrieved, checked for compatibility, grounded in the current environment, and composed according to state, task phase, and goal. Experiments across four benchmarks evaluate task performance and interaction efficiency, with ablations examining memory scope, promotion, and retrieval.
Critical strengths
Strength1: The framework explicitly handles when shared experience is applicable. Abstraction preserves operational conditions, while decision-time composition binds abstract roles to current objects and available actions (§§3.4–3.5). This connects transferable procedures with concrete execution requirements. The memory-scope and fixed-concatenation comparisons support the value of selectively using shared experience rather than simply exposing the policy to more records (Fig. 4).
Strength2: The empirical benefits extend across different models and task interfaces. Tables 1–2 show improvements beyond a single backbone or environment, while also revealing settings where MemCo is not the strongest method. The interaction-budget comparison further supports its ability to complete tasks with fewer execution opportunities (Fig. 3), making the contribution relevant to agents with constrained interaction budgets.
Critical weaknesses
Weakness1: The aggregate evaluation leaves the source of transfer gains partly unresolved. The merged test set includes both an agent’s acquisition domain and other domains (§4), but the reported results do not separate these cases. G-Memory and the memory-scope ablations provide useful comparisons; however, a simple pooled-record control would more directly test what the proposed local–global organization adds beyond access to broader source experience.
Weakness2: The promotion ablation does not fully separate evidence-aware selection from pruning strength. Replacing Wilson promotion with the empirical support ratio at the same threshold changes both record membership and global-memory size (Figs. 3–4). Similar injected context sizes help exclude prompt length as the sole explanation, but an equally selective admission rule remains a relevant comparison.
Questions for authors
Question1: Could you separate results on each agent’s acquisition domain from those on other domains? A pooled-record control using the same source trajectories and retrieval/composition procedure would additionally help isolate the benefit of local–global organization.
Question2: Could you compare Wilson promotion with selecting the same number of eligible records by empirical support rate, keeping retrieval and composition unchanged? Repeated random tie-breaking where necessary would help distinguish evidence-size-aware selection from generic pruning.
4 comments
Response to the Reviewer (2/2)
Q2. Could Wilson promotion be compared with selecting the same number of eligible records by empirical support rate, keeping retrieval and composition unchanged and repeating random tie-breaking where necessary?
A2. Thank you for suggesting this controlled comparison. Our original experiments show higher accuracy under Wilson promotion than empirical-support promotion, while retaining fewer records. At threshold 0.35, Wilson retains 426 rather than 736 records and achieves Familiar/Held-out accuracies of 97.40%/97.86%, compared with 95.13%/96.43% for empirical support (Table 10).
In the repeating equal-count experiment, both methods select 426 records from the same pool of 749 eligible records, obtaining results consistent with the above finding. We constructed this pool from the four agents’ frozen local memories by applying abstraction, merging, and checks for structural validity and informativeness. Wilson promotion retained 426 of these records at the original threshold of 0.35, so we also selected the top 426 by empirical support rate. Because 735 records tied for the highest empirical support rate (1.0), we randomly selected 426 from this tied group in each of three repeats. We used no secondary ranking by evidence count. We held the local memories, record evidence, retrieval, grounding, and composition fixed. At this default bank size, Wilson achieves higher accuracy on both layout splits than the mean of the three empirical selections. We use the original Wilson results from Table 10 as the reference. Empirical tie-breaking uses selection seeds 101–103, and all three resulting memory banks are evaluated with inference seed 42 on the same agent–task pairs. We report each empirical selection and the mean ± SD, where SD reflects variation across the three record selections rather than independent memory-acquisition runs.
| Promotion criterion / reference | Records | Familiar Acc. (%) | Familiar Steps | Held-out Acc. (%) | Held-out Steps |
|---|---|---|---|---|---|
| Wilson | 426 | 97.40 | 9.74 | 97.86 | 11.42 |
| Empirical top-K repeat1 | 426 | 96.10 | 10.51 | 96.79 | 12.33 |
| Empirical top-K repeat2 | 426 | 96.75 | 10.39 | 94.64 | 12.51 |
| Empirical top-K repeat3 | 426 | 96.10 | 10.62 | 97.14 | 11.66 |
| Empirical top-K: mean ± SD | 426 | 96.32 ± 0.37 | 10.51 ± 0.12 | 96.19 ± 1.35 | 12.17 ± 0.45 |
Thank you for the detailed rebuttal and the additional controlled experiments. The domain-wise breakdown and pooled-record control clarify the source of the transfer gains, while the matched-record-count comparison with repeated tie-breaking better isolates the effect of Wilson-based promotion. These results adequately address my concerns. I therefore maintain my positive rating of 3.
Thank you very much for your encouraging feedback and for recognizing our additional efforts in addressing your concerns. We sincerely appreciate your positive assessment and continued support for our work.
Response to the Reviewer (1/2)
The suggestion to compare promotion methods at a matched record count directly motivated a valuable controlled experiment. We are grateful for this guidance and the constructive questions that helped strengthen our empirical evaluation.
A1. Thank you for the great suggestions. Following your suggestion to separate the results based on domains, we have added the requested domain breakdown and pooled-record control on ALFWorld with Qwen3-32B to clarify where the transfer gains occur and what local–global organization contributes beyond access to broader experience.
Regarding your 2nd question on the benefit of local-global organization, the submitted ablations support both combining local and global memory and selectively using their contents. Full outperforms Local-only and Global-only in Table 10. With the same underlying local and global memories, Full improves accuracy from 86.69% to 97.40% on Familiar Layouts and from 81.79% to 97.86% on Held-out Layouts compared with Fixed L5+G5, respectively. This comparison supports query-guided retrieval and composition with stored experience held fixed.
The added comparison separates acquisition-domain and other-domain performance and tests Full against directly pooled local records. Own/Other is defined relative to each agent’s acquisition domain within each layout split. Pooled uses the same frozen source memories and preserves record provenance, evidence, and each agent’s local graph. It omits global abstraction, cross-agent evidence aggregation, and promotion, while keeping retrieval, applicability checks, grounding, composition, and budgets unchanged. The largest observed accuracy gains occur on other-domain tasks under Held-out Layouts. Full improves accuracy by 11.90 percentage points over Local-only and 4.29 points over Pooled, while reducing average steps from 15.88 and 12.45 to 11.17. These results support constructing transferable global memory beyond directly pooling local experience. Full does not improve own-domain accuracy over Local-only or Familiar accuracy over Pooled.
The table reports accuracy (%) / average steps, using the original frozen-snapshot Full reference. Each condition evaluates 147 tasks under four agents’ memories. Gains are Full minus Local-only / Pooled, respectively.