- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Memory-as-Skill to improve compositional generalization in embodied multi-agent collaboration. The method abstracts historical interaction trajectories into individual and collaborative skills, and employs an execution-based admission gate to verify whether a skill can be successfully executed in a new episode before adding it to the skill library. At test time, agents retrieve and bind relevant skills according to the current task and state. Experiments on VIKI-R and PARTNR show that the proposed method can generalize to unseen task compositions, task families, and constraint combinations, while overall outperforming several existing memory-based methods.
Critical strengths
- The execution-based admission mechanism is a practically meaningful design choice and is supported by fairly comprehensive experiments.
- The experimental diagnostics and the controls for potential confounding factors are well designed.
Critical weaknesses
- The proposed method and the baselines do not follow the same execution procedure. After retrieving a skill, the proposed method deterministically expands it into primitive actions through role binding, whereas the baselines require the LLM to generate the complete action plan on its own. Under the current setup, it is therefore difficult to determine how much of the observed performance gain can be attributed specifically to Memory-as-Skill itself.
- The compositional generalization experiment on VIKI-R includes only one type of unseen composition, while the more complex three-constraint and four-constraint tasks in PARTNR contain only 11 and 5 examples, respectively. The authors explicitly acknowledge this limitation. However, since compositional generalization is a central claim of the paper, the current experiments are still insufficient to demonstrate that the proposed capability generalizes reliably to a broader range of unseen compositions.
Questions for authors
See weaknesses.
2 comments
We would like to kindly ask the reviewer if there is any further question. Since the deadline is approaching, we would like to make sure we have clear all the confusion for the reviewer.
Rebuttal for Reviewer Leij
W1. Attribution with different execution procedures
Reviewer W1:
Response:
Please see Figure 5 for library construction with grounding fixed, Appendix A.3 for the same-library use control, and Table 2 for PARTNR admission under a common action interface. Figure 4 reports the cooperation and hierarchy ablations, and Appendix A.1 reports the prompt control. These original controls distinguish library construction, skill representation, and executable use.
We additionally evaluated 8B memory pipelines with a common deterministic 14-template expander:
The retrieval-library variant uses training-instance retrieval and achieves higher success on both splits. This additional adaptation supports useful memory retrieval under shared expansion.
W2. Breadth of compositional generalization
Reviewer W2:
Response:
Please see §5.1, Figure 2, and Appendix A.3 for the original VIKI cooperative composition, and Table 2 and Figure 4 for PARTNR constraint transfer. We added evidence on both composition patterns and constraint complexity.
Additional VIKI composition. We evaluated a multi-object state-change pattern with a supplementary memory variant:
The memory variant improves success by 7.3 pp, demonstrating reuse on an additional composition pattern.
Larger constraint-composition pools. The additional 700-task PARTNR-derived evaluation contains 500 R_S tasks and 100 tasks each for H_R_T and H_R_S_T. Under its 8B single-agent protocol, the existing task-level outputs give the following breakdown (revised Appendix A.13, Table 11). Skill-memory Full and G-Memory are averaged over three seeds; ToM and zero-shot use one run each:
Full memory achieves the highest observed mean success in both complex groups. Together with the additional VIKI pattern, these results broaden the evidence for skill reuse across composition patterns and constraint complexity. The original two-agent results and these additional single-agent results retain their respective protocols.