acceptodds
Under review as a conference paper at ICLR 2027

SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

Abstract

Latent multi-agent systems reduce inference cost by keeping intermediate reasoning and inter-agent communication in continuous representations. Reusable skills, however, are commonly supplied as text or stored as layer-wise Key-Value states or skill-specific model parameters, introducing repeated processing or substantial per-skill storage. We introduce SkillMem, which represents each reusable skill as a small external latent memory and makes it directly available during latent multi-agent reasoning. A role-conditioned cross-attention reader allows the same memory to guide different agents according to their roles and current hidden states. We train the skill memories and shared reader through behavioral distillation, matching the change in token predictions induced by a textual skill while keeping the backbone frozen. For our experiments, we obtain textual skills from paired successful and failed multi-agent trajectories, although existing skills can also be compiled directly. Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem provides strong gains for smaller backbones while preserving efficient latent multi-agent inference across model scales. We use the same memory configuration across all evaluated model scales, requiring 9.6 KB of skill-specific storage per skill, including its retrieval embedding. For a library of 100 skills, SkillMem reduces total storage by more than compared with alternative skill representations and achieves up to a end-to-end inference speedup. These results show that compact procedural memories can be reused without sacrificing the efficiency of latent multi-agent reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent multi-agent systems, procedural skills, latent memory, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tCysOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SkillMem compiles each textual skill into eight learned memory vectors. A shared role-conditioned reader uses the retrieved memory to update agents' hidden states while the backbone remains frozen. Behavioral skill distillation trains the memories and reader to reproduce the effect of the textual skill on token predictions, while preserving base behavior where skill support is limited. Agents share the retrieved memory and continue to communicate through latent states. Across six tasks and three model scales, the experiments compare SkillMem with text and latent communication baselines, KV skills, soft prompts, and separate LoRA adapters for each skill.

Critical strengths

The compact external memory is attractive: each skill needs only one stored memory, and the reader is shared across skills and agent roles. Table 2 includes the 34.5 MB reader in the total rather than reporting only the 9.6 KB marginal cost per skill. Using the same skill library and routing across the skill representations makes the comparison easier to interpret.

The 4B results are strong. SkillMem raises GSM8K accuracy from 72.40% to 89.90% and MedQA from 21.84% to 61.50% relative to LatentMAS, and exceeds the KV, soft prompt, and LoRA alternatives on all six tasks at that scale. The role, capacity, and loss ablations provide evidence about several design choices. Appendix F's unsuccessful attempts to bypass textual skill discovery and distillation are informative, and Section 6.2 appropriately acknowledges that the gains become less consistent at larger scales.

Critical weaknesses
  1. It remains unclear how much of the gain requires the identity of the retrieved skill. The memory bank and shared reader are jointly trained, with separate libraries for each benchmark and backbone (Section 4 and Appendix A). A shared reader trained on those examples could improve the latent pipeline even if the differences among skill memories contributed little. Training a model with a single shared memory on the same examples, or permuting the trained memories at inference while keeping the reader fixed, would help determine whether the system uses distinct retrieved procedures. This distinction is central to the claim of an external skill library.

  2. The accuracy of the Text Skill condition is missing from the comparison. Section 6.1 and the caption of Table 1 identify “Text” as TextMAS without skills, whereas Table 2 reports Text Skill latency of 12.98 seconds without the corresponding task accuracy. SkillMem takes 7.95 seconds, but these results do not quantify how task performance changes when the original textual skills are replaced by learned memories. The comparisons with KV skills, soft prompts, and LoRA are useful, but they do not directly provide this reference. I recognize that the reported 4.23× speedup uses TextMAS as its baseline and that Section 6.3 already describes the latency as comparable to LatentMAS. My concern is therefore the missing accuracy and latency comparison with Text Skill under the same backbone, skill selection, and reasoning protocol. Reporting both measurements would clarify what is gained or lost through compilation.

Questions for authors
  1. Have you tested whether correct skill selection matters when using the trained reader, for example by permuting trained memory identities at inference or by training one shared memory on the same examples? These controls would distinguish the benefit of a retrieved library from adaptation of the latent pipeline.

  2. What accuracy does the Text Skill condition in Table 2 achieve under the same routing and evaluation settings, and does its reported latency include the same retrieval operations as SkillMem? This is the most direct missing comparison for the claim that compilation preserves the behavior of the textual skill.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer tCys (1/3)

Thank you for your helpful suggestions. We conducted both requested experiments and added the missing Text Skill comparison.

W1 / Q1. Does the gain depend on retrieving the right skill? What happens if memories are permuted, or if one shared memory is used?

We tested both settings using Qwen3-4B with the same protocol and seeds as Table 1. We also included additional controls to understand whether the improvement comes from skill content or simply training the reader.

VariantGSM8KGPQA-DMedQAMBPP+Accuracy gain (pp)Gain retained
LatentMAS (no memory)73.718.021.833.000%
Full SkillMem90.934.760.665.4+25.1100%
Swapping memories between skills89.934.159.466.1+24.698%
One shared memory for all skills88.729.362.431.3+14.357%
Training without skills72.521.623.234.2+0.62%
Training with irrelevant text85.523.448.437.0+11.345%

Accuracy gains are measured relative to LatentMAS without memory, pooled over four benchmarks. All comparisons use paired evaluations.

The results show three things:

  • Exact skill selection matters less on these benchmarks. Swapping memories between skills retains 98% of the gain. The difference from Full SkillMem is only 0.5 ± 1.2 points. This suggests that skills within the same benchmark often provide similar benefits.

  • Separate skill memories still matter. Training one memory shared by all skills retains only 57% of the gain. On MBPP+, accuracy drops from 65.4% to 31.3%. Different skills can induce different behaviors during training, which are harder to preserve in one shared memory.

  • The improvement depends on skill content, not just the reader. Training without skills retains only 2% of the gain. Replacing useful skills with irrelevant text reduces it to 45%.

In summary, correct skill retrieval has limited impact in our original benchmarks, but the content of the skills and their separate memories are important during training. The skills within each benchmark are closely related, which may explain why their trained memories have similar effects at inference. In the next experiment, we examine skills with more distinct behaviors and test whether SkillMem can preserve those differences.

AuthorsOpenReview ↗

Reply to Reviewer tCys (2/3)

W1 / Q1 (continued). Can SkillMem preserve the behavior of distinct skills?

To test this, we found code skills that clearly improve TextMAS on MBPP+ problems. Unlike our original skill libraries, these skills provide clear improvements in TextMAS on held-out problems.

We compiled each skill into SkillMem using our original distillation objective. We also trained memories from irrelevant text as a control.

Model / SkillText skill gainSkillMem gainSkillMem − TextGain retainedSkillMem − irrelevant memory
8B Skill 1+4.4 ± 1.7+4.1 ± 1.7−0.2 ± 0.995%+3.8 ± 1.7
8B Skill 2+5.2 ± 2.1+5.1 ± 2.0−0.2 ± 1.697%+2.4 ± 1.8
14B Skill 1+6.0 ± 2.0+5.3 ± 2.0−0.7 ± 1.489%+3.6 ± 1.7

These results show that SkillMem preserves 89–97% of the textual skill's gain at 8B and 14B, with no statistically significant decrease relative to skill text in these experiments. Memories trained from useful skills also outperform those trained from irrelevant text by 1.8–3.8 points.

Comparison with other skill representations. We also evaluated KV Skill, Soft Prompt, and S2L using the same skills and training data:

MBPP+8B Skill 18B Skill 214B Skill 1Added positionsStored values per skill
Skill text+4.4+5.2+6.0285–381—
KV Skill+2.2+5.0+4.8308–404 (cached)23–33M
Soft Prompt+2.1+4.4+3.7833–41K
S2L+4.0+5.7+4.607.7–10.5M
SkillMem+4.1+5.1+5.302K

SkillMem is the only compiled representation without a statistically significant drop relative to skill text across all three settings. It also adds no skill tokens to the prompt and stores 3,700–5,100× fewer values per skill than S2L, using one shared reader across the library.

We will include these experiments in the revised paper.

AuthorsOpenReview ↗

Reply to Reviewer tCys (3/3)

W2 / Q2. Text Skill's accuracy is not reported. Does the latency in Table 2b include retrieval?

Thank you for pointing this out. We clarify the different Text Skill settings, provide the missing accuracy results, and explain how latency is measured.

(1) Clarifying Text Skill vs. TextMAS + Skill Text.

First, Text Skill in Table 2b runs inside LatentMAS, not TextMAS. It directly inserts the retrieved skill text into the LatentMAS agent's prompt. Therefore, its latency (12.98 s) measures LatentMAS + Text Skill, rather than TextMAS + Skill Text.

We use these two settings for different purposes:

  • TextMAS + Skill Text vs. TextMAS (no skill): Measures how much the original textual skill helps the text-based pipeline.
  • LatentMAS + SkillMem vs. LatentMAS (no skill): Measures how much of the skill's benefit can be brought into latent communication.

Our goal is to preserve the benefits of textual skills in LatentMAS, not necessarily to outperform TextMAS in absolute accuracy.

The following table shows the accuracy gains (percentage points) over each pipeline's own no-skill baseline, using the original Table 1 skill libraries:

Accuracy gain (pp)GSM8KARC-EARC-CMedQAGPQA-D
4B TextMAS + Skill Text+2.9−0.5+1.0−1.6−3.6
4B LatentMAS + SkillMem+17.5+4.9+5.9+39.7+17.7
8B TextMAS + Skill Text+0.1−0.2−0.1+0.5−0.8
8B LatentMAS + SkillMem+0.7−1.1−0.6+1.6+1.2

At 4B, the large SkillMem gains mainly come from reducing failures in latent communication. At 8B, the original textual skills provide little improvement even in TextMAS, and SkillMem similarly shows small changes. This helps explain the limited gains in the original Table 1.

Importantly, our new MBPP+ and ALFworld experiments (General Response, Point 2,3) show that when textual skills provide clear improvements, SkillMem preserves most of those gains.

ALFWorld settingSuccess rate (%)
TextMAS32.5
TextMAS + Skill Text43.8 (+11.3)
LatentMAS5.8
LatentMAS + Skill Text26.6 (+20.8)
LatentMAS + SkillMem24.8 (+19.0)

(2) Text Skill accuracy inside LatentMAS.

For a fair comparison of skill representations, all methods in Table 1 use the same LatentMAS pipeline, backbone, retrieved skills, and decoding settings. They differ only in how the skill is injected: Text Skill adds the skill text to the prompt, and SkillMem uses a trained latent memory and shared reader.

We now provide the missing Text Skill accuracy for all three model sizes.

Qwen3-4B (accuracy, %):

MethodGSM8KARC-EARC-CMBPP+MedQAGPQA-D
LatentMAS72.492.783.732.721.818.0
Text Skill84.296.488.240.230.025.6
SkillMem89.997.589.665.461.535.6

Qwen3-8B (accuracy, %):

MethodGSM8KARC-EARC-CMBPP+MedQAGPQA-D
LatentMAS92.398.992.470.267.836.5
Text Skill92.698.092.869.465.740.5
SkillMem93.197.891.871.669.537.7

Qwen3-14B (accuracy, %):

MethodGSM8KARC-EARC-CMBPP+MedQAGPQA-D
TextMAS94.098.493.466.970.437.7
LatentMAS94.398.794.272.171.631.7
Text Skill94.899.294.070.970.930.5
SkillMem92.698.993.967.369.735.3

The results are consistent with our earlier observations:

  • 4B: SkillMem outperforms Text Skill on all six benchmarks, mainly because it reduces failures in latent communication.
  • 8B: Results are mixed, with SkillMem performing better on GSM8K, MedQA, and MBPP+, but worse on the other three benchmarks. The original skills provide little benefit at this scale, even as text.
  • 14B: Results remain task-dependent. SkillMem performs better on GPQA-D but worse on GSM8K and MBPP+. However, our new MBPP+ experiments show that SkillMem can preserve most of the gains when more effective textual skills are available.

(3) Does Table 2b include retrieval latency?

No. Table 2b measures the complete Planner → Critic → Refiner → Judger execution, excluding retrieval for all methods.

  • SkillMem (7.95 s): Includes all reader operations during inference.
  • Text Skill (12.98 s): Runs inside LatentMAS and includes the prefill cost of skill text.
  • Retrieval (0.244 s): Uses the same query encoding and similarity search for both methods.

Including retrieval, SkillMem takes 8.19 s, compared with 13.22 s for Text Skill (Qwen3-8B, 100 MedQA questions).

We will add Text Skill accuracy to Table 1 and clarify the comparison settings and retrieval cost in Table 2b.

Reviewer_YtdoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose SkillMem, which stores each reusable skill as a small latent memory (8 vectors, 9.6 KB) for latent multi-agent systems, instead of putting the skill text into the prompt or storing its KV cache or LoRA. A shared role-conditioned cross-attention reader lets different agents read the same memory based on their roles. The memory and reader are trained with behavioral distillation, which matches the change in token log-probabilities caused by the text skill, with the backbone frozen. On six benchmarks with Qwen3-4B/8B/14B, SkillMem gives large gains on the 4B model and much smaller storage than KV or LoRA skills.

Critical strengths
  1. Reusing skills in latent MAS without re-processing skill text is a practical problem, and the shared reader keeps per-skill storage very small.
  2. The behavioral distillation objective is reasonable, and the ablations show that role conditioning and each loss term help.
Critical weaknesses
  1. The large gains on Qwen3-4B mainly come from a weak baseline, as LatentMAS gets only 21.84% on MedQA and 32.74% on MBPP+ there, while at 8B and 14B the gains are small or negative.
  2. SkillMem is trained to copy the effect of the text skill, but it is much better than KV Skill, which is equivalent to the text skill (e.g., 61.50 vs 33.57 on MedQA-4B). This suggests the gain may come from the reader fixing the latent pipeline rather than from the skill itself.
Questions for authors

How does a reader trained without any skill (same calibration data) perform?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ytdo

Thank you for your thoughtful review, and the positive rating! We conducted additional experiments to better understand where SkillMem's improvements come from.

W1. The 4B gains come from a weak LatentMAS baseline.

We agree that the large improvements at 4B are partly due to the weak LatentMAS baseline. However, this weakness is also an important problem that SkillMem addresses. With Qwen3-4B, latent communication often breaks down, causing the final agent to repeat itself or produce no answer. We analyzed all no-skill LatentMAS-4B outputs across three seeds:

BenchmarkLatentMAS collapse rateSkillMem gain (pp)
MedQA59.8%+39.7
GPQA-D56.6%+17.7
MBPP+28.4%+32.6
GSM8K15.3%+17.5
ARC-C7.4%+5.9
ARC-E3.4%+4.9

SkillMem substantially reduces these failures. For example, on MBPP+, the collapse rate drops from 28.4% to 1.3%. With SkillMem, LatentMAS-4B reaches accuracy comparable to TextMAS on GSM8K and ARC, and is only 2.4 points lower on MBPP+, while achieving 3.5–6.7× lower latency.

At 8B and 14B, LatentMAS is already much stronger, leaving less room for improvement. We acknowledge this limitation in Section 6.2 (lines 408–412). Importantly, our new experiments show that SkillMem can still preserve the benefits of effective textual skills at these larger model sizes (General Response, Points 2–4).

W2 / Q. SkillMem is far above KV Skill, so is the gain from the skill, or from the reader repairing the pipeline? How does a reader trained without any skill, on the same calibration data, perform?

The improvement does not come from training the reader alone. To directly answer your question, we trained SkillMem without skills, using the same calibration data, architecture, and training objective. We also tested irrelevant text, shared memories, and incorrect skill selection.

All experiments use Qwen3-4B with the same protocol as Table 1.

VariantGSM8KGPQA-DMedQAMBPP+Accuracy gain (pp)Gain retained
LatentMAS (no memory)73.718.021.833.000%
Full SkillMem90.934.760.665.4+25.1100%
Training without skills72.521.623.234.2+0.62%
Training with irrelevant text85.523.448.437.0+11.345%
One shared memory for all skills88.729.362.431.3+14.357%
Swapping memories between skills89.934.159.466.1+24.698%

These results help separate the contributions of skill content, memory representation, and retrieval:

  • The reader alone is not sufficient. Without skills, SkillMem retains only 2% of its original improvement. This shows that simply training the reader on the same data does not explain the gains.

  • Skill content matters. Replacing useful skills with irrelevant text reduces the retained gain to 45%. Thus, the training signal provided by the skills is important.

  • Separate skill memories matter. Using one shared memory for all skills reduces the retained gain to 57%, with a particularly large drop on MBPP+. Different skills can induce different behaviors, which are difficult to preserve in a single shared memory.

  • Exact skill selection matters less in these benchmarks. Swapping memories within the same skill library retains 98% of the gain. This suggests that skills discovered for the same benchmark often provide similar benefits. However, replacing them with irrelevant text substantially reduces performance.

Why does SkillMem outperform KV Skill? Text Skill and KV Skill inject skill information into LatentMAS, but they do not directly address its latent communication failures. SkillMem instead trains the reader and memories to reproduce the behavior induced by textual skills. Both components matter: removing the skill signal eliminates nearly all improvement, while removing role conditioning reduces accuracy by 7.8 points (Table 3a).

Overall, SkillMem's improvement comes from learning skill-conditioned behavior through the reader and memories, not from training a reader alone. We will include these ablations and clarify this distinction in the revised paper.

Reviewer_6L4kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SkillMem, a framework for incorporating reusable skills into latent multi-agent reasoning through compact external memories. A shared, role-conditioned cross-attention reader allows agents to access the same skill memory according to their roles and current hidden states. The memories and reader are trained through behavioral distillation to reproduce the effects of textual skills on token predictions while keeping the backbone frozen. Experiments across six benchmarks and multiple Qwen3 model sizes demonstrate performance improvements, particularly for smaller models, while maintaining efficient inference and reducing storage requirements relative to KV-based and LoRA-based skill representations. Ablations examine the contributions of role conditioning, memory capacity, and the training objective.

Critical strengths
  • Compact skill representation. The external memory provides a storage-efficient alternative to per-skill KV caches or LoRA adapters, which could be useful when maintaining a large skill library.

  • Useful component analysis. The ablations examine role conditioning, memory capacity, and the distillation objective, providing some evidence for the design choices despite the limited overall performance gains.

Critical weaknesses
  1. Model-specific training limits portability. SkillMem appears to require a separately trained reader and skill memories for each backbone. This limits the reuse of an existing skill library across models and introduces additional training costs when adopting a new backbone. The paper should clarify whether the reader or skill memories can transfer across models.

  2. Very limited performance gains and misleading highlighting. The performance improvements over competitive baselines are very limited overall and inconsistent across model scales. Table 1 nevertheless bolds SkillMem results whenever they exceed either TextMAS or LatentMAS, even when they underperform the other baseline. For example, the bolded results for Qwen3-4B on MBPP+ and Qwen3-14B on GPQA-Diamond are below TextMAS. Although disclosed in the caption, this unconventional rule is misleading and visually overstates the gains. Boldface should identify the best-performing method, and the discussion should accurately reflect the limited improvements.

  3. Unclear practical significance of skill-processing overhead. The evaluated tasks have relatively short outputs and low inference latency, making skill-processing overhead potentially more prominent. For large scale tasks such as BrowseComp or SWE-bench, skill descriptions may represent a much smaller fraction of the total token budget. The current evaluation therefore does not establish whether the proposed compression delivers meaningful end-to-end benefits in more demanding agent workloads.

Questions for authors
  1. Is the backbone used to train the reader always the same as the model used for downstream evaluation? Can a trained reader be transferred to a different backbone?

  2. Does the evaluation include a baseline that directly inserts the textual skill into the prompt? Is this represented by “KV Skill,” or is KV Skill a separate baseline that reuses cached key–value states?

  3. Does the reported end-to-end inference latency include skill retrieval and all reader operations? Please clarify which components are included in the timing measurements.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer 6L4k (1/2)

Thank you for your constructive feedback. We have conducted additional experiments to address your concerns, especially the effectiveness of SkillMem on larger models.

W1 / Q1. Is SkillMem portable across backbones? Can a trained reader be transferred to another backbone?

SkillMem currently requires a separate reader and skill memories for each backbone. The reader operates on the backbone's hidden states, so it cannot be directly transferred to another model. This is similar to KV caches and LoRA adapters, which are also tied to specific models.

However, the textual skill library can be reused across models. We only need to compile the skills again for the target backbone, without fine-tuning its weights. This process is relatively inexpensive: training the shared reader and a 10-skill memory bank takes 11.7 minutes, compared with 110 minutes for the corresponding S2L adapters (Table 2b).

We will clarify this limitation and discuss cross-backbone transfer as future work.

W2. The gains are small or task-dependent at 8B and 14B, and the bolding in Table 1 overstates them.

We agree that the improvements in the original Table 1 are less consistent at 8B and 14B. We will revise the bolding and discussion to avoid overstating small numerical differences.

Why are the gains small? As shown in General Response (Point 4), the original textual skills provide little benefit even in TextMAS at 8B (−0.8 to +0.5 points across five benchmarks). SkillMem similarly shows small changes in LatentMAS (−1.1 to +1.6 points). This suggests that the limited gains partly come from the skills themselves, rather than SkillMem failing to preserve useful skill behavior.

What happens when textual skills actually help? To answer this, we conducted two additional experiments on MBPP+ and ALFWorld, where textual skills provide clear improvements.

(1) MBPP+: SkillMem preserves the gains of effective skills at 8B and 14B.

We find new textual skills that improve TextMAS on MBPP+. We then compiled these skills into SkillMem and evaluated them in LatentMAS. The table shows the accuracy improvement over each pipeline's own no-skill baseline (percentage points):

MBPP+8B Skill 18B Skill 214B Skill 1
Skill text in TextMAS+4.4+5.2+6.0
SkillMem in LatentMAS+4.1+5.1+5.3
S2L in LatentMAS+4.0+5.7+4.6
KV Skill in LatentMAS+2.2+5.0+4.8
Soft Prompt in LatentMAS+2.1+4.4+3.7

SkillMem retains approximately 88–98% of the textual skill's improvement across these three settings. This shows that SkillMem remains effective at 8B and 14B when the original textual skills are useful.

(2) ALFWorld: SkillMem also preserves useful skills on interactive tasks.

We additionally evaluated ALFWorld, where procedural skills help agents complete multi-step household tasks. Unlike our original benchmarks, we used an existing library of procedural skills rather than discovering skills ourselves.

We compiled these skills using our original training objective and evaluated Qwen3-8B on 274 held-out samples. Results are success rates (%), with gains relative to each pipeline's no-skill baseline.

ALFWorld settingSuccess rate (%)
TextMAS32.5
TextMAS + Skill Text43.8 (+11.3)
LatentMAS5.8
LatentMAS + Skill Text26.6 (+20.8)
LatentMAS + SkillMem24.8 (+19.0)

SkillMem preserves most of the textual skill's benefit in LatentMAS, with a +19.0-point gain compared with +20.8 points from skill text.

Overall, we do not claim that SkillMem consistently improves every benchmark at every model size. Our results show that when textual skills provide a meaningful benefit, SkillMem can preserve most of that benefit in a compact latent representation. The new MBPP+ results confirm this at 8B and 14B, while ALFWorld provides additional evidence on interactive tasks.

AuthorsOpenReview ↗

Reply to Reviewer 6L4k (2/2)

W3. In long agent workloads, one skill's text is a small share of the tokens, so what does compressing it buy?

We agree that a single skill contributes only a small fraction of the tokens in long agent workloads. However, SkillMem provides greater benefits when multiple skills are used or skills are repeatedly accessed. We conducted additional experiments to verify this.

(1) SWE-bench Lite: Multiple skills. We evaluated Qwen3-8B on 107 instances using the same MBPP+ skill library as Table 1. A single skill accounts for only 0.6% of the processed tokens. However, as more skills are used, SkillMem maintains nearly constant overhead. The following table shows additional latency over LatentMAS (20 instances, fixed 64-token outputs):

Active skills1410
Text Skill+0.02 s+0.08 s+0.21 s
KV Skill+0.16 s+0.21 s+0.31 s
SkillMem+0.05 s+0.05 s+0.05 s

On the full 107 instances, SkillMem takes 17.3 s per instance, compared with 17.7 s for LatentMAS and 51.7 s for TextMAS. Thus, SkillMem maintains the efficiency of latent communication.

(2) ALFWorld: Repeated skill use. In this multi-step environment, the skill text adds approximately 880 tokens per Planner call. Over a complete game, SkillMem processes 25% fewer tokens in LatentMAS compared with skill text.

(3) Storage: Large skill libraries. For 1,000 skills, SkillMem requires only 44 MB, compared with 13.6 GB for KV Skill and 30.7 GB for S2L.

In summary, we agree that compressing a single skill provides limited latency savings. SkillMem's main benefits are compact storage, low overhead with multiple skills, and avoiding repeated processing of skill text.

Q2. Is KV Skill the text-in-prompt baseline?

No. Text Skill is the text-in-prompt baseline. It directly inserts the retrieved skill text into the agent's prompt. KV Skill uses the same text but precomputes its KV cache. The cache is then injected into the agent, preserving the same token positions without re-encoding the skill text. Both baselines (except the TextMAS baseline itself) run inside LatentMAS.

Q3. What does the latency in Table 2b include?

The reported latency covers the complete Planner → Critic → Refiner → Judger execution on Qwen3-8B.

  • SkillMem (7.95 s): Includes all reader operations inside the agents' forward passes.
  • Text Skill (12.98 s): Uses the same LatentMAS pipeline, but includes the additional prefill cost of skill text.
  • Skill retrieval: Not included in Table 2b for any method. All methods use the same retrieval procedure, so we measured this cost separately.

Retrieval consists of a backbone forward pass over the query and a similarity search over the skill embeddings. On 100 Qwen3-8B MedQA queries, it takes 244 ms per query, approximately 3% of SkillMem's latency.

I appreciate the authors' thoughtful responses and the additional experiments.

My main remaining concern is the practical significance of the problem being addressed. The motivation of reusable skills is particularly compelling for complex agentic tasks, such as scientific research and software engineering, where accumulated procedural knowledge can substantially improve task execution. However, the current evaluation mainly focuses on relatively small and bounded tasks, where conventional approaches such as CoT, ReAct, and reflection may already be sufficient to achieve reasonable performance. As the authors also acknowledge, the benefits of skills are relatively limited and inconsistent at larger model scales. In these settings, even if SkillMem reduces token processing, the absolute savings may be too small to translate into meaningful practical benefits.

I am also not fully convinced by the efficiency claims in the more demanding settings. The reported 4.23× speedup over TextMAS is largely inherited from LatentMAS itself. Moreover, including the reported retrieval cost of 244 ms, SkillMem's latency becomes approximately 8.19 s, essentially the same as LatentMAS at 8.18 s. The SWE-bench Lite experiment is a useful addition, but as I know, Qwen-8B is not able to effectively solve the underlying software engineering tasks. Discussing efficiency on tasks where the model demonstrates little to no task-solving capability is essentially meaningless.

I appreciate the authors' attempts to address my concerns and I understand that evaluating the method on substantially more challenging agentic workloads would require considerable additional effort. I would be willing to increase my score by 0.5 points in recognition of the additional experiments, but unfortunately, this option is not available in the reviewing system. So I prefer to leave my score unchanged and defer to the other reviewers and the AC to judge whether the demonstrated contribution meets the acceptance bar of ICLR.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the reviewers. This post summarises the main new results; each reviewer's reply gives the details.

How we compare. Our goal is to show that SkillMem can bring the benefits of textual skills to LatentMAS while keeping the efficiency of latent communication. Therefore, we compare the gain from adding SkillMem to LatentMAS with the gain from adding skill text to TextMAS, each relative to its own no-skill baseline. We do not claim that LatentMAS + SkillMem must outperform TextMAS in absolute accuracy.

Comparison with other skill representations. In Table 1, we use the same LatentMAS pipeline and the same retrieved skills for all methods to ensure a fair comparison. The only difference is how each skill is represented and injected: Text Skill directly adds skill text to the prompt; KV Skill injects precomputed KV representations of the same text, without re-encoding it; Soft Prompt learns a prompt embedding for each skill; and S2L trains a LoRA adapter for each skill.

1. Do the gains come from the skills, and how important is correct skill retrieval? (Ytdo, tCys)

On Qwen3-4B, SkillMem improves LatentMAS by 25.1 percentage points across four benchmarks, using the same setup and seeds as Table 1. To understand where this improvement comes from, we tested four variants:

SkillMem variantWhat it testsAccuracy gain (pp)Gain retained
Full SkillMemOriginal method+25.1100%
Training without skillsIs the reader alone sufficient?+0.62%
Training with irrelevant textDoes skill content matter?+11.345%
Sharing one memory across all skills (instead of a separate memory for each skill in SkillMem)Do we need separate skill memories?+14.357%
Swapping memories between skillsDoes correct skill selection matter?+24.698%

These results show two things:

  • Skill content matters. The reader alone retains only 2% of the gain, while irrelevant text retains 45%. Sharing one memory across all skills also reduces the gain to 57%, showing the benefit of maintaining separate skill memories.
  • Exact skill selection is less important in our current benchmarks. Swapping skill memories retains 98% of the gain. This is because skills discovered within each benchmark are related and often address the same weakness. As a result, even a different skill from the same library can provide similar benefits.

2. Can SkillMem preserve the benefits of effective textual skills? ALFWorld (all reviewers)

Our original benchmarks follow the LatentMAS evaluation setup. However, some benchmarks contain diverse questions, making it difficult to identify useful skills for individual tasks. Others, such as ARC-C and ARC-R, already achieve over 90% accuracy, leaving limited room for improvement. To better evaluate whether SkillMem can preserve the benefits of textual skills, we additionally tested ALFWorld, where procedural skills can directly help agents complete tasks.

We evaluated Qwen3-8B on 274 held-out games (success rate in %):

ALFWorld settingSuccess rate (%)
TextMAS32.5
TextMAS + Skill Text43.8 (+11.3)
LatentMAS5.8
LatentMAS + Skill Text26.6 (+20.8)
LatentMAS + SkillMem24.8 (+19.0)

The results show that SkillMem preserves most of the benefits of textual skills. In LatentMAS, it retains most of the improvement from skill text (+19.0 vs. +20.8 points). These results suggest that SkillMem can preserve useful skill behavior without adding skill text tokens to the prompt, rather than simply improving a weak LatentMAS baseline.

AuthorsOpenReview ↗

General response (2/2)

3. When a skill helps, SkillMem keeps its gain (MBPP+).

A new skill-discovery procedure in TextMAS (following SKILL-KD, Shi et al., arXiv:2607.28048) found code skills that clearly improve TextMAS on MBPP+ problems. We run the experiment again with these new skills:

MBPP+8B skill 18B skill 214B skill 1values stored per skill
skill text (in the TextMAS)+4.4+5.2+6.0—
SkillMem (in the LatentMAS)+4.1+5.1+5.32 K
S2L (in the LatentMAS)+4.0+5.7+4.67.7–10.5 M
KV Skill (in the LatentMAS)+2.2+5.0+4.823–33 M
Soft Prompt (in the LatentMAS)+2.1+4.4+3.733–41 K

SkillMem keeps 89–97% of the skill text's gain, is the only compiled form never significantly below it, and stores 3,700–5,100× fewer values per skill than an S2L adapter (details: reply to tCys).

4. The skill's gain in each pipeline, Table-1 libraries.

To evaluate whether SkillMem preserves the benefits of textual skills, we compare the improvement each method brings over its own baseline: SkillMem vs. LatentMAS (no skill) and TextMAS + skill text vs. TextMAS (no skill). This allows us to measure how much of the original textual skill's benefit is retained by SkillMem.

gainGSM8KARC-EARC-CMedQAGPQA-D
4B TextMAS + skill text+2.9−0.5+1.0−1.6−3.6
4B LatentMAS + SkillMem+17.5+4.9+5.9+39.7+17.7
8B TextMAS + skill text+0.1−0.2−0.1+0.5−0.8
8B LatentMAS + SkillMem+0.7−1.1−0.6+1.6+1.2
  • 4B: LatentMAS collapses, so SkillMem's gain is mostly a repair. LatentMAS-4B often loops or gives no answer (60% of MedQA outputs), and the trained memory removes this failure (point 1). The skill text itself adds little at 4B, even in TextMAS. Here LatentMAS + SkillMem has 3.5–6.7× lower latency than TextMAS.
  • 8B: LatentMAS does not collapse, and SkillMem's gain follows the skill text's. Both gains are small, with the same sign on four of the five benchmarks (up on GSM8K and MedQA, down on ARC-E and ARC-C): these libraries add little even as text, so SkillMem adds little too. Where the skill text helps clearly, SkillMem helps clearly too (points 2 and 3).
  • When a skill helps, SkillMem keeps its gain (8B and 14B MBPP+); see point 3

5. Latency (6L4k, tCys).

Text Skill (use text skill in LatentMAS) takes 12.98 s and SkillMem 7.95 s (Table 2b); neither includes retrieval, which is the same 244 ms call for every method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.