acceptodds
Under review as a conference paper at ICLR 2027

Memory-as-Skill for Compositional Generalization in Embodied Multi-Agent Cooperation

Abstract

Real-world embodied cooperation happens under partial observation and without a central coordinator, so each agent has to learn its own abilities and its teammates' behavior from past interaction. Its memory therefore has to carry both what the agent can do on its own and how it coordinates with teammates. Such a memory is most useful when it generalizes, because new tasks often recombine familiar elements in unseen ways. A memory of raw trajectories supports this poorly, since each trajectory binds actions and coordination to a single episode. We propose Memory-as-Skill, which turns multi-agent interaction traces into a library of individual and cooperation skills, each an abstract behavior with grounded instances. An LLM proposes skills from segmented traces, and a proposal enters the library only after it passes an execution-based admission gate on separate validation episodes. The library is then retrieved at test time to guide execution. On a multi-robot planning benchmark, the admitted library transfers to a held-out subtask combination and to held-out task families with sufficiently capable backbones, and it exceeds existing memory methods that write primitive plans. This transfer requires that skill demos be grounded into the plan by role binding, since a library whose demos the LLM must transcribe by hand fails at the same step as every primitive-plan method. On a decentralized household benchmark, where every memory method shares the same in-context use, the admitted library leads existing memory methods by small margins across open-weight backbones. Ablations show that admission, cooperation skills, and the skill hierarchy each contribute to this transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Large Language Model Agent, Cooperation, Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LeijOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Memory-as-Skill to improve compositional generalization in embodied multi-agent collaboration. The method abstracts historical interaction trajectories into individual and collaborative skills, and employs an execution-based admission gate to verify whether a skill can be successfully executed in a new episode before adding it to the skill library. At test time, agents retrieve and bind relevant skills according to the current task and state. Experiments on VIKI-R and PARTNR show that the proposed method can generalize to unseen task compositions, task families, and constraint combinations, while overall outperforming several existing memory-based methods.

Critical strengths
  1. The execution-based admission mechanism is a practically meaningful design choice and is supported by fairly comprehensive experiments.
  2. The experimental diagnostics and the controls for potential confounding factors are well designed.
Critical weaknesses
  1. The proposed method and the baselines do not follow the same execution procedure. After retrieving a skill, the proposed method deterministically expands it into primitive actions through role binding, whereas the baselines require the LLM to generate the complete action plan on its own. Under the current setup, it is therefore difficult to determine how much of the observed performance gain can be attributed specifically to Memory-as-Skill itself.
  2. The compositional generalization experiment on VIKI-R includes only one type of unseen composition, while the more complex three-constraint and four-constraint tasks in PARTNR contain only 11 and 5 examples, respectively. The authors explicitly acknowledge this limitation. However, since compositional generalization is a central claim of the paper, the current experiments are still insufficient to demonstrate that the proposed capability generalizes reliably to a broader range of unseen compositions.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Leij

W1. Attribution with different execution procedures

Reviewer W1:

The proposed method and the baselines do not follow the same execution procedure. After retrieving a skill, the proposed method deterministically expands it into primitive actions through role binding, whereas the baselines require the LLM to generate the complete action plan on its own. Under the current setup, it is therefore difficult to determine how much of the observed performance gain can be attributed specifically to Memory-as-Skill itself.

Response:

Please see Figure 5 for library construction with grounding fixed, Appendix A.3 for the same-library use control, and Table 2 for PARTNR admission under a common action interface. Figure 4 reports the cooperation and hierarchy ablations, and Appendix A.1 reports the prompt control. These original controls distinguish library construction, skill representation, and executable use.

We additionally evaluated 8B memory pipelines with a common deterministic 14-template expander:

Supplementary pipelineScene-held-out (400 tasks)Task-held-out (800 tasks)
Retrieval-library variant + shared expander67.8%21.0%
MEMENTO + shared expander60.5%16.5%

The retrieval-library variant uses training-instance retrieval and achieves higher success on both splits. This additional adaptation supports useful memory retrieval under shared expansion.

W2. Breadth of compositional generalization

Reviewer W2:

The compositional generalization experiment on VIKI-R includes only one type of unseen composition, while the more complex three-constraint and four-constraint tasks in PARTNR contain only 11 and 5 examples, respectively. The authors explicitly acknowledge this limitation. However, since compositional generalization is a central claim of the paper, the current experiments are still insufficient to demonstrate that the proposed capability generalizes reliably to a broader range of unseen compositions.

Response:

Please see §5.1, Figure 2, and Appendix A.3 for the original VIKI cooperative composition, and Table 2 and Figure 4 for PARTNR constraint transfer. We added evidence on both composition patterns and constraint complexity.

Additional VIKI composition. We evaluated a multi-object state-change pattern with a supplementary memory variant:

Supplementary patternTasksMemory variantSegment
multiobj_statechange43612.6%5.3%

The memory variant improves success by 7.3 pp, demonstrating reuse on an additional composition pattern.

Larger constraint-composition pools. The additional 700-task PARTNR-derived evaluation contains 500 R_S tasks and 100 tasks each for H_R_T and H_R_S_T. Under its 8B single-agent protocol, the existing task-level outputs give the following breakdown (revised Appendix A.13, Table 11). Skill-memory Full and G-Memory are averaged over three seeds; ToM and zero-shot use one run each:

Constraint compositionTasksSkill-memory FullG-Memory FullToMZero-shot
H_R_T10011.0%5.0%6.0%2.0%
H_R_S_T1005.7%4.7%2.0%4.0%

Full memory achieves the highest observed mean success in both complex groups. Together with the additional VIKI pattern, these results broaden the evidence for skill reuse across composition patterns and constraint complexity. The original two-agent results and these additional single-agent results retain their respective protocols.

We would like to kindly ask the reviewer if there is any further question. Since the deadline is approaching, we would like to make sure we have clear all the confusion for the reviewer.

Reviewer_qne9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Memory-as-skill which is a memory mechanism for LLM agents in embodied settings. The paper proposes storing experience as a library of executable skills instead of raw trajectories of experiences. The skills use variables in place of specific objects, rooms and partners, and keeps grounding instances from past episodes so it can be applied to new ones. Skills can be individual or cooperative.

The library of skills is created offline. An LLM marks the steps in the traces where subgoals are completed and proposes a candidate skill for each segment. A candidate is admitted only if it passes three checks. First, it must work for separate validation episodes, covers the source segment and recurs in other traces. During test time, skills are retrieved without LLMs, filtered through preconditions, and placed in the prompt.

Critical strengths
  1. The admission criteria is simple. A skill gets stored only if it executes after working on disjoint episodes, covers the source segment and the recurs in other traces.

  2. There is a thorough set of of control to help understand the contribution

  3. The coordination metrics seem interesting and go beyond the standard performance metrics to my knowledge. It would be nice to see this in the main paper rather than the appendix.

  4. Reporting of numbers appears to be honest. Authors decline significance claims on small margins along with GPT 4o being hurt by the memory usage.

Critical weaknesses
  1. The large VIKI-R margins come from a comparison that the authors note in §5.2, "changes both the library and how it is used". the proposed method grounds skill demos by role binding, so the LLM names effects and the grounding step writes out the primitive actions, whereas every baseline writes primitive plans directly. The primitive-output control (App. A.3) shows that the same eight skills, when the LLM must write primitive actions itself, reach 0.0% / 3.0%, on par with G-Memory. No baseline is evaluated with an equivalent grounding step, so most of the VIKI-R gap is attributable to grounded use rather than to the memory content.

  2. the paper does not isolate or fully define its central contribution. The evidence for admission compares the admitted library with keeping every proposal. These libraries differ in size by an order of magnitude: 8 against 123 skills on VIKI-R, and 29–31 against 450–470 on PARTNR. There is no size-matched random-subset control and no ablation of the individual conditions in Eq. 1. The recurrence condition, supp(c) > 1, is a frequency filter that could account for much of the effect on its own. Finally, the claim that “the gate, not the proposer” decides the library rests on one VIKI-R comparison in which the ungated replay library is never evaluated, and it is not tested on PARTNR. Finally, the claim that "the gate, not the proposer" decides the library rests on one VIKI-R comparison in which the ungated replay library is never evaluated, and it is not tested on PARTNR.

  3. the claim that memory can replace partner modelling and communication is not tested in a setting that needs either. Partner modelling baseline is weak. It would be nice to see approaches like Mindforge (https://proceedings.neurips.cc/paper_files/paper/2025/hash/f82cb74b5d40004cb96d1426fc12fccf-Abstract-Conference.html) that do partner modelling and allow for communication.

Questions for authors
  1. Can the authors please report two admission controls: a size-matched random subset of proposals (8 on VIKI, about 30 on PARTNR), and ablations of each condition of Eq. 1 individually.

  2. How are cover(c) and supp(c) computed: by rules or by LLM-based matching? What counts as a matching segment?

  3. How do the 17 replay candidates perform without the gate? Can replay mining, or a different proposer, be run on PARTNR to support the claim that the gate rather than the proposer decides the library?

  4. Which proposing backbone built the library used for each row of Table 1?

  5. How faithful is the ToM baseline to Li et al. (2023)? Can you provide the VIKI-L2 ToM prompt and compare against a partner-modelling baseline with a persistent belief or intention state?

  6. At test time on PARTNR, is the partner the same backbone running the same method? Is there any experiment with a different or held-out partner policy?

  7. Given the GPT-4o result and the narrowing margins at larger scale, do the PARTNR gains hold on current frontier backbones?

  8. How does the method compare with execution-verified skill induction such as ASI, or with Voyager-style self-verification, under the same traces?

9 .Why does CG without images consistently outperform CG with images? Does the image input interfere with entity naming?

8 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer qne9

W1. Grounded use and memory-content attribution

Reviewer W1:

The large VIKI-R margins come from a comparison that the authors note in §5.2, "changes both the library and how it is used". the proposed method grounds skill demos by role binding, so the LLM names effects and the grounding step writes out the primitive actions, whereas every baseline writes primitive plans directly. The primitive-output control (App. A.3) shows that the same eight skills, when the LLM must write primitive actions itself, reach 0.0% / 3.0%, on par with G-Memory. No baseline is evaluated with an equivalent grounding step, so most of the VIKI-R gap is attributable to grounded use rather than to the memory content.

Response:

Please see Appendix A.3 for the original same-library grounded-use versus primitive-output control, which demonstrates the contribution of executable role binding. Figure 5 holds grounding fixed while varying library construction, Table 3 compares candidate sources under common use, and Appendix A.1 reports the PARTNR prompt control. Together, these controls assess library construction and executable use. The additional shared-expander experiment in our response to vjn3 Q4 compares memory pipelines under a common backend.

Q1 / W2. Random-subset and individual gate controls

Reviewer W2:

the paper does not isolate or fully define its central contribution. The evidence for admission compares the admitted library with keeping every proposal. These libraries differ in size by an order of magnitude: 8 against 123 skills on VIKI-R, and 29–31 against 450–470 on PARTNR. There is no size-matched random-subset control and no ablation of the individual conditions in Eq. 1. The recurrence condition, supp(c) > 1, is a frequency filter that could account for much of the effect on its own. Finally, the claim that “the gate, not the proposer” decides the library rests on one VIKI-R comparison in which the ungated replay library is never evaluated, and it is not tested on PARTNR. Finally, the claim that "the gate, not the proposer" decides the library rests on one VIKI-R comparison in which the ungated replay library is never evaluated, and it is not tested on PARTNR.

Reviewer Q1:

Can the authors please report two admission controls: a size-matched random subset of proposals (8 on VIKI, about 30 on PARTNR), and ablations of each condition of Eq. 1 individually.

Response:

Fixed-size selection. We ran an additional experiment on 1,218 VIKI-L2 tasks with the 72B vision-language model. All libraries have the same size, use the same candidate pool, and include every skill in the prompt under a common primitive-plan interface:

LibrarySelection ruleSuccess
Execution-screenedRetain candidates whose stored behaviors pass execution checks.14.0%
Frequency-selectedSelect the most frequent candidates in training traces at the same library size.10.9%
Random subsetsSample same-size subsets from the candidate pool; average over five frozen libraries.10.3%

Execution screening improves success by 3.1 pp over frequency selection and approximately 3.7 pp over random selection. This supports skill-content selection at fixed size and provides evidence beyond frequency-based selection alone.

Execution-admission control. Please see Table 2 for the original PARTNR comparison with and without execution admission, which favors execution-admitted memory in both the R-only and R+S settings. Together with the new fixed-size results, this supports content selection and checking behavior for reuse. Equation 1 additionally requires necessary-action coverage and recurrence; these controls do not separately quantify each condition's contribution. Appendices A.1 and A.3 provide the original recurrence and completeness evidence.

Q2. Coverage, support, and segment matching

Reviewer Q2:

How are cover(c) and supp(c) computed: by rules or by LLM-based matching? What counts as a matching segment?

Response:

Section 4.1 defines a segment as a contiguous action sequence ending at a subgoal boundary, with its effect determined by the observed state change. cover(c) requires all source actions necessary for that effect; supp(c) counts recorded traces containing a matching segment. Admission requires support above one and successful rebound execution in at least two source-disjoint episodes without invalidating the episode goal. Please see Appendix A.1 for recomputed support and independent recovery across families, and Appendix A.3 for the incomplete-candidate rejection example. We will clarify the construction-time matching procedure; the test-time self-condition filter is rule-based (§4.2).

Q3. Ungated replay and alternative proposers

Reviewer Q3:

How do the 17 replay candidates perform without the gate? Can replay mining, or a different proposer, be run on PARTNR to support the claim that the gate rather than the proposer decides the library?

Response:

Please see Table 3 and Appendix A.3 for the original gated replay comparison and candidate rejection analysis. The original ungated replay pool has no reported downstream score. We additionally evaluated deterministic mining on 500 single-agent PARTNR-derived tasks to examine admission with another candidate source:

Additional deterministic-miner constructionLibrary entriesSeed 17Seed 29Seed 43Mean R+S success
Full907.2%8.4%8.6%≈8.1%
Without gate1788.0%8.4%7.8%≈8.1%

In this separate single-agent deterministic-miner setting, admission reduces the library from 178 to 90 entries at the same reported mean success, supporting compactness with an alternative candidate source. The original grounded VIKI and alternating two-agent PARTNR admission comparisons report success benefits in their respective settings (Figure 5; Table 2). The miner result complements those controls with evidence of a smaller library at retained mean performance; our proposer-flexibility conclusion remains tied to the original replay comparison (Table 3).

Q4. Construction and evaluation backbones

Reviewer Q4:

Which proposing backbone built the library used for each row of Table 1?

Response:

Our Table 1 skill library was constructed offline with Llama-70B and reused across evaluation backbones (§5.1); Appendix A.1 describes the PARTNR baseline memory sources. Please also see Appendix A.8, Figure 11 for the original comparison of libraries induced by different proposing backbones, which supports constructing our representation with different models. Revised Appendix A.14 distinguishes this construction analysis from the Table 1 evaluation library.

Q5. ToM fidelity, prompts, and persistent state

Reviewer Q5:

How faithful is the ToM baseline to Li et al. (2023)? Can you provide the VIKI-L2 ToM prompt and compare against a partner-modelling baseline with a persistent belief or intention state?

Response:

The original ToM-inspired template, covering belief formation, hypothesis generation, and prediction/planning, is in Appendix A.10, Listing 2. Please see Table 5 and Appendix A.3 for the original plan-funnel diagnosis of object validity and final execution. In the revised manuscript, we provide the full additional VIKI coordination adapter in Appendix A.17, Listing 4, including capability reasoning, partner-task inference, and complementary action assignments. This additional prompt version is appended to the system message. The new persistent-state comparisons are given in Q6 below and in Appendix A.15, Figure 15 of the revision.

Q6. Different and held-out partners

Reviewer Q6:

At test time on PARTNR, is the partner the same backbone running the same method? Is there any experiment with a different or held-out partner policy?

Response:

We encoded cooperation skills through observable partner situations to make interaction knowledge reusable (§4.1). We evaluated the original method with alternating two-agent execution and will make both actors' configurations explicit. To test sensitivity to partner conditions, we compared memory and persistent-state focal variants using a 72B focal agent, 700 tasks, and goals assigned before execution. This diagnostic uses parity execution: both agents act before a joint environment step.

PartnerSkill-memory focal variantPersistent-state focal
Persistent-state, 72B66.7%64.3%
Zero-shot, 72B66.7%64.9%
Zero-shot, 8B36.9%36.1%
Constructed held-out scripted policy78.0%73.7%

The memory-focal variant achieved higher joint-success point estimates across all tested partner conditions, supporting its utility under this diagnostic protocol.

Q7. Stronger backbones and GPT-4o

Reviewer Q7:

Given the GPT-4o result and the narrowing margins at larger scale, do the PARTNR gains hold on current frontier backbones?

Response:

Our focus is reusable cooperation knowledge for embodied agents, including smaller open-weight planners relevant to deployments with limited compute. Skill memory lets these agents reuse validated individual and cooperative behaviors without increasing the planning backbone's size (§4). The gains on the evaluated 7B/8B backbones support this practical value (Table 1). Narrower margins on some stronger backbones are consistent with less headroom for memory assistance. Please see Table 8 and Appendix A.9 for the closed-model results: gains on both success and completion with Claude and the highest completion with GPT-4o, with strict-success benefits varying by backbone. Together, these results support the value of reusable embodied experience across the evaluated model capacities.

Q8. ASI and Voyager

Reviewer Q8:

How does the method compare with execution-verified skill induction such as ASI, or with Voyager-style self-verification, under the same traces?

Response:

We clarify the relationship to ASI and Voyager through their skill representations and verification mechanisms. Our experiments test cooperation-aware skills, execution after cross-episode role rebinding, and frozen-memory reuse. ASI's WebArena programs and Voyager's Minecraft feedback/repair loops provide relevant precedents but use different learning and execution protocols. We focus numerical evaluation on the paper's embodied-memory baselines and controls; full-system comparisons would require aligned program interfaces, trace access, and feedback budgets.

W3. MindForge, partner modeling, and communication

Reviewer W3:

the claim that memory can replace partner modelling and communication is not tested in a setting that needs either. Partner modelling baseline is weak. It would be nice to see approaches like Mindforge (https://proceedings.neurips.cc/paper_files/paper/2025/hash/f82cb74b5d40004cb96d1426fc12fccf-Abstract-Conference.html) that do partner modelling and allow for communication.

Response:

Please see Figure 4 for the original cooperation-skill ablation, which supports the contribution of stored interaction knowledge to coordination. We study this mechanism within the paper's protocol without adding explicit communication. MindForge combines partner modeling, perspective-taking, communication, and memory; these mechanisms can complement our skills. We will explain the comparison scope through communication permissions and interaction budgets.

Q9. Images and entity naming

Reviewer Q9:

Why does CG without images consistently outperform CG with images? Does the image input interfere with entity naming?

Response:

Please see §5.4 and Figure 6 for the original asset-name ablation and its explanation of entity naming in grounded execution. We additionally tested image utility under canonical naming with a 72B no-memory Segment diagnostic on separate task pools:

Additional diagnosticReal image + canonical namingNo image + canonical namingGray-image control
Scene-held-out, 400 tasks6.5%3.0%3.3%
Task-held-out, 800 tasks5.4%2.1%2.4%

Real images improved success relative to both controls within each split under canonical naming, supporting the value of visual input when the naming representation is controlled.

Coordination-metric presentation

Reviewer Critical Strength 3:

The coordination metrics seem interesting and go beyond the standard performance metrics to my knowledge. It would be nice to see this in the main paper rather than the appendix.

Response:

We thank the reviewer's advice and will try to figure out a way to add that in the main paper. We do have that in the main paper in several previous version but decided to move to the appendix since there are more expeirments results to be added. We will try to move that in the final version.

Reviewer_vjn3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a memory for LLM-driven agents that cooperate in embodied tasks without explicit communication or a central coordinator. An LLM identifies individual and cooperation skills from multi-agent interaction traces. Skills are written with placeholders (roles) instead of task specifics, and each has a name, description, type, effect and preconditions, plus an instance (a demo and the entities it was observed with). Each agent retrieves from the library on its own, so coordination comes from retrieved cooperation skills, with no messages or partner-belief module.

A proposed skill is admitted only if it passes a gate on different validation episodes. It must execute with its stated effect when rebound in multiple other episodes, keep all actions of its source segment that the effect needs, and have a matching segment in more than one trace. At test time the library is frozen. Skills are retrieved by similarity of name and description to the query, instances are ranked by context similarity, and a rule-based filter checks the agent's own conditions while the partner condition is free text for the LLM.

The authors claim transfer to a held-out subtask combination, held-out task families and constraint compositions. They report large gains on VIKI-R and small gains on PARTNR over ToM, zero-shot and three memory baselines. Ablations show an instruction-only library fails and that traces, admission and role binding matter. On PARTNR, removing cooperation skills or the hierarchy hurts.

Critical strengths
  1. A sound memory design. Ablations show traces, admission and role binding each matter for multi-agent collaboration.
  2. Extensive ablations. Varying memory sources, backbones, library pieces and constraints shows when the approach works.
  3. Gains over several baselines. The method wins on both benchmarks chosen by the authors: by a large margin on VIKI-R and a small one on PARTNR.
  4. Candid reporting. The authors say the PARTNR margins are not claimed significant, that GPT-4o does worse with memory than without, and are honest about shortcomings and results on different ablations.
Critical weaknesses
  1. No size-matched control. The no-admission library is much larger, so the drop in performance may come from harder retrieval rather than lower skill quality. Subsampling it to the same size, or sweeping library size, would be a good control experiment.
  2. The standard admission rule isn't tested directly. The paper argues existing libraries in other approaches admit a skill if it succeeded in its source episode, but only compares against an ablation that skips the execution check entirely. A baseline with task-success based gating would provide valuable insights.
  3. PARTNR evidence is thin. The gain over the best baseline is only a few points on most backbones and over a limited number of episodes, and uncertainty is reported only across repeated runs.
  4. The necessity of the LLM is unclear. On VIKI-R an LLM-free library mined from reference plans and certified by the same gate matches the LLM-proposed one. It's unclear whether the LLM adds value without reference plans.
Questions for authors
  1. How does performance change when the "no-admission-gate" library is subsampled to match the size of the gated library?
  2. How does it change when task-success based admission gate is created?
  3. Can you report variance, confidence intervals, or significance tests for the PARTNR results, is it possible to increase the number of episodes used for experiments here?
  4. How do the baselines perform when given the same non-LLM expander used on VIKI-R? Is it even possible to conduct this experiment?
3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer vjn3

Q1 / W1. Size-matched admission control

Reviewer W1:

No size-matched control. The no-admission library is much larger, so the drop in performance may come from harder retrieval rather than lower skill quality. Subsampling it to the same size, or sweeping library size, would be a good control experiment.

Reviewer Q1:

How does performance change when the "no-admission-gate" library is subsampled to match the size of the gated library?

Response:

We ran an additional frozen-library experiment to test skill selection at fixed size. The libraries use the same candidate pool and contain the same number of skills. We evaluated them with the 72B vision-language model on 1,218 VIKI-L2 tasks, including every skill in the prompt and using the same primitive-plan interface:

LibraryHow candidates are selectedSuccess
Execution-screenedRetain candidates whose stored behaviors pass execution checks.14.0%
Frequency-selectedSelect the candidates occurring most frequently in training traces, matching the screened library's size.10.9%
Random subsetsRandomly sample same-size subsets from the candidate pool; report the mean across five frozen libraries.10.3%

The screened library achieved the highest observed success, exceeding frequency selection by 3.1 pp and the random mean by approximately 3.7 pp (random-library SD: 1.8 pp). These results support selecting reusable skill content at fixed size. For the original admission comparison under common grounded use, please see Figure 5.

Q2 / W2. Task-success-based admission

Reviewer W2:

The standard admission rule isn't tested directly. The paper argues existing libraries in other approaches admit a skill if it succeeded in its source episode, but only compares against an ablation that skips the execution check entirely. A baseline with task-success based gating would provide valuable insights.

Reviewer Q2:

How does it change when task-success based admission gate is created?

Response:

When all source episodes are successful, a source-task-success gate retains every candidate and reduces to keeping all proposals. Under that premise, the relevant existing control is the keep-all-proposals ablation labeled “No execution admission” in Figure 5, where execution admission improves success under common grounded use. Our gate checks candidate behavior after rebinding, directly targeting reuse. The new fixed-size controls in Q1 additionally support skill-content selection beyond library size. For the original candidate-completeness evidence and targeted recovery control, please see Appendices A.3 and A.4, respectively.

Q3 / W3. PARTNR variance, confidence intervals, and sample size

Reviewer W3:

PARTNR evidence is thin. The gain over the best baseline is only a few points on most backbones and over a limited number of episodes, and uncertainty is reported only across repeated runs.

Reviewer Q3:

Can you report variance, confidence intervals, or significance tests for the PARTNR results, is it possible to increase the number of episodes used for experiments here?

Response:

Please see Table 1 for the original backbone-wise means and run-to-run standard deviations; our method achieved the highest mean success on each open-weight backbone. We additionally estimated task- and scene-level uncertainty on 700 mixed R_S/H_R_T/H_R_S_T tasks across 37 scenes in a single-agent PARTNR-derived setting:

ComparisonDifference (pp)Task-bootstrap 95% CI (pp)Scene-cluster 95% CI (pp)
Full − zero-shot+4.9[2.5, 7.3][2.1, 7.5]

Both intervals support a positive Full−zero-shot difference in that setting. We report the original Table 1 gains as mean improvements without significance claims on the small margins.

Q4. Baselines with a shared expander

Reviewer Q4:

How do the baselines perform when given the same non-LLM expander used on VIKI-R? Is it even possible to conduct this experiment?

Response:

We adapted 8B memory pipelines to a common deterministic expander, which converts selected templates into primitive action sequences using a shared set of 14 templates. The retrieval-library variant retrieves training instances. We evaluated the adapted pipelines on held-out scenes and tasks:

Supplementary pipelineScene-held-out (400 tasks)Task-held-out (800 tasks)
Retrieval-library variant + shared expander67.8%21.0%
MEMENTO + shared expander60.5%16.5%

The retrieval-library variant achieved higher success on both splits, showing useful memory retrieval under this shared backend. For the original abstract-skill demo-binding controls, please see Appendix A.3 and Figure 5. Appendix A.1 reports the original PARTNR prompt control and its preserved method ordering.

W4. Value of the LLM proposer

Reviewer W4:

The necessity of the LLM is unclear. On VIKI-R an LLM-free library mined from reference plans and certified by the same gate matches the LLM-proposed one. It's unclear whether the LLM adds value without reference plans.

Response:

The LLM proposer provides an automatic induction route and is optional when reference plans are available. Please see Table 3 and Appendix A.3 for the original replay-mining comparison under the same gate and grounding. Its matching CG performance and additional ID coverage support proposer flexibility. Section 4.1 describes the LLM's automatic trace segmentation and abstraction into roles, effects, and preconditions; Figure 3 and Table 1 report PARTNR construction and subsequent evaluation. Our contribution centers on the validated cooperation-skill representation and its reuse.

Reviewer_vjn3OpenReview ↗

Thanks for the rebuttal

Thanks for the response. The size-matched admission control addresses my main concern, and the shared-expander comparison helps, though the gains are modest. The task-success gating argument holds only if all source episodes succeed, which should be stated. The value of the LLM proposer without reference plans is still unclear, and PARTNR significance against the best baseline in the original setting is untested. Keeping my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal For Memory as Skills

We would like to thank the reviewers for recognizing the memory design, ablations, and transparent reporting. Our contribution combines cooperation-aware skill representation, cross-episode execution-based admission, and grounded reuse. Together, these components turn interaction traces into reusable cooperation knowledge. We address all of the reviewers' weakness/questions one by one using the original paper's results and new supplementary experiment results. We are also eager to hear from the reviewers for their further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.