- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SkillMem compiles each textual skill into eight learned memory vectors. A shared role-conditioned reader uses the retrieved memory to update agents' hidden states while the backbone remains frozen. Behavioral skill distillation trains the memories and reader to reproduce the effect of the textual skill on token predictions, while preserving base behavior where skill support is limited. Agents share the retrieved memory and continue to communicate through latent states. Across six tasks and three model scales, the experiments compare SkillMem with text and latent communication baselines, KV skills, soft prompts, and separate LoRA adapters for each skill.
Critical strengths
The compact external memory is attractive: each skill needs only one stored memory, and the reader is shared across skills and agent roles. Table 2 includes the 34.5 MB reader in the total rather than reporting only the 9.6 KB marginal cost per skill. Using the same skill library and routing across the skill representations makes the comparison easier to interpret.
The 4B results are strong. SkillMem raises GSM8K accuracy from 72.40% to 89.90% and MedQA from 21.84% to 61.50% relative to LatentMAS, and exceeds the KV, soft prompt, and LoRA alternatives on all six tasks at that scale. The role, capacity, and loss ablations provide evidence about several design choices. Appendix F's unsuccessful attempts to bypass textual skill discovery and distillation are informative, and Section 6.2 appropriately acknowledges that the gains become less consistent at larger scales.
Critical weaknesses
-
It remains unclear how much of the gain requires the identity of the retrieved skill. The memory bank and shared reader are jointly trained, with separate libraries for each benchmark and backbone (Section 4 and Appendix A). A shared reader trained on those examples could improve the latent pipeline even if the differences among skill memories contributed little. Training a model with a single shared memory on the same examples, or permuting the trained memories at inference while keeping the reader fixed, would help determine whether the system uses distinct retrieved procedures. This distinction is central to the claim of an external skill library.
-
The accuracy of the Text Skill condition is missing from the comparison. Section 6.1 and the caption of Table 1 identify “Text” as TextMAS without skills, whereas Table 2 reports Text Skill latency of 12.98 seconds without the corresponding task accuracy. SkillMem takes 7.95 seconds, but these results do not quantify how task performance changes when the original textual skills are replaced by learned memories. The comparisons with KV skills, soft prompts, and LoRA are useful, but they do not directly provide this reference. I recognize that the reported 4.23× speedup uses TextMAS as its baseline and that Section 6.3 already describes the latency as comparable to LatentMAS. My concern is therefore the missing accuracy and latency comparison with Text Skill under the same backbone, skill selection, and reasoning protocol. Reporting both measurements would clarify what is gained or lost through compilation.
Questions for authors
-
Have you tested whether correct skill selection matters when using the trained reader, for example by permuting trained memory identities at inference or by training one shared memory on the same examples? These controls would distinguish the benefit of a retrieved library from adaptation of the latent pipeline.
-
What accuracy does the Text Skill condition in Table 2 achieve under the same routing and evaluation settings, and does its reported latency include the same retrieval operations as SkillMem? This is the most direct missing comparison for the claim that compilation preserves the behavior of the textual skill.
5 comments
Reply to Reviewer tCys (2/3)
W1 / Q1 (continued). Can SkillMem preserve the behavior of distinct skills?
To test this, we found code skills that clearly improve TextMAS on MBPP+ problems. Unlike our original skill libraries, these skills provide clear improvements in TextMAS on held-out problems.
We compiled each skill into SkillMem using our original distillation objective. We also trained memories from irrelevant text as a control.
| Model / Skill | Text skill gain | SkillMem gain | SkillMem − Text | Gain retained | SkillMem − irrelevant memory |
|---|---|---|---|---|---|
| 8B Skill 1 | +4.4 ± 1.7 | +4.1 ± 1.7 | −0.2 ± 0.9 | 95% | +3.8 ± 1.7 |
| 8B Skill 2 | +5.2 ± 2.1 | +5.1 ± 2.0 | −0.2 ± 1.6 | 97% | +2.4 ± 1.8 |
| 14B Skill 1 | +6.0 ± 2.0 | +5.3 ± 2.0 | −0.7 ± 1.4 | 89% | +3.6 ± 1.7 |
These results show that SkillMem preserves 89–97% of the textual skill's gain at 8B and 14B, with no statistically significant decrease relative to skill text in these experiments. Memories trained from useful skills also outperform those trained from irrelevant text by 1.8–3.8 points.
Comparison with other skill representations. We also evaluated KV Skill, Soft Prompt, and S2L using the same skills and training data:
| MBPP+ | 8B Skill 1 | 8B Skill 2 | 14B Skill 1 | Added positions | Stored values per skill |
|---|---|---|---|---|---|
| Skill text | +4.4 | +5.2 | +6.0 | 285–381 | — |
| KV Skill | +2.2 | +5.0 | +4.8 | 308–404 (cached) | 23–33M |
| Soft Prompt | +2.1 | +4.4 | +3.7 | 8 | 33–41K |
| S2L | +4.0 | +5.7 | +4.6 | 0 | 7.7–10.5M |
| SkillMem | +4.1 | +5.1 | +5.3 | 0 | 2K |
SkillMem is the only compiled representation without a statistically significant drop relative to skill text across all three settings. It also adds no skill tokens to the prompt and stores 3,700–5,100× fewer values per skill than S2L, using one shared reader across the library.
We will include these experiments in the revised paper.
Reply to Reviewer tCys (3/3)
W2 / Q2. Text Skill's accuracy is not reported. Does the latency in Table 2b include retrieval?
Thank you for pointing this out. We clarify the different Text Skill settings, provide the missing accuracy results, and explain how latency is measured.
(1) Clarifying Text Skill vs. TextMAS + Skill Text.
First, Text Skill in Table 2b runs inside LatentMAS, not TextMAS. It directly inserts the retrieved skill text into the LatentMAS agent's prompt. Therefore, its latency (12.98 s) measures LatentMAS + Text Skill, rather than TextMAS + Skill Text.
We use these two settings for different purposes:
- TextMAS + Skill Text vs. TextMAS (no skill): Measures how much the original textual skill helps the text-based pipeline.
- LatentMAS + SkillMem vs. LatentMAS (no skill): Measures how much of the skill's benefit can be brought into latent communication.
Our goal is to preserve the benefits of textual skills in LatentMAS, not necessarily to outperform TextMAS in absolute accuracy.
The following table shows the accuracy gains (percentage points) over each pipeline's own no-skill baseline, using the original Table 1 skill libraries:
| Accuracy gain (pp) | GSM8K | ARC-E | ARC-C | MedQA | GPQA-D |
|---|---|---|---|---|---|
| 4B TextMAS + Skill Text | +2.9 | −0.5 | +1.0 | −1.6 | −3.6 |
| 4B LatentMAS + SkillMem | +17.5 | +4.9 | +5.9 | +39.7 | +17.7 |
| 8B TextMAS + Skill Text | +0.1 | −0.2 | −0.1 | +0.5 | −0.8 |
| 8B LatentMAS + SkillMem | +0.7 | −1.1 | −0.6 | +1.6 | +1.2 |
At 4B, the large SkillMem gains mainly come from reducing failures in latent communication. At 8B, the original textual skills provide little improvement even in TextMAS, and SkillMem similarly shows small changes. This helps explain the limited gains in the original Table 1.
Importantly, our new MBPP+ and ALFworld experiments (General Response, Point 2,3) show that when textual skills provide clear improvements, SkillMem preserves most of those gains.
| ALFWorld setting | Success rate (%) |
|---|---|
| TextMAS | 32.5 |
| TextMAS + Skill Text | 43.8 (+11.3) |
| LatentMAS | 5.8 |
| LatentMAS + Skill Text | 26.6 (+20.8) |
| LatentMAS + SkillMem | 24.8 (+19.0) |
(2) Text Skill accuracy inside LatentMAS.
For a fair comparison of skill representations, all methods in Table 1 use the same LatentMAS pipeline, backbone, retrieved skills, and decoding settings. They differ only in how the skill is injected: Text Skill adds the skill text to the prompt, and SkillMem uses a trained latent memory and shared reader.
We now provide the missing Text Skill accuracy for all three model sizes.
Qwen3-4B (accuracy, %):
| Method | GSM8K | ARC-E | ARC-C | MBPP+ | MedQA | GPQA-D |
|---|---|---|---|---|---|---|
| LatentMAS | 72.4 | 92.7 | 83.7 | 32.7 | 21.8 | 18.0 |
| Text Skill | 84.2 | 96.4 | 88.2 | 40.2 | 30.0 | 25.6 |
| SkillMem | 89.9 | 97.5 | 89.6 | 65.4 | 61.5 | 35.6 |
Qwen3-8B (accuracy, %):
| Method | GSM8K | ARC-E | ARC-C | MBPP+ | MedQA | GPQA-D |
|---|---|---|---|---|---|---|
| LatentMAS | 92.3 | 98.9 | 92.4 | 70.2 | 67.8 | 36.5 |
| Text Skill | 92.6 | 98.0 | 92.8 | 69.4 | 65.7 | 40.5 |
| SkillMem | 93.1 | 97.8 | 91.8 | 71.6 | 69.5 | 37.7 |
Qwen3-14B (accuracy, %):
| Method | GSM8K | ARC-E | ARC-C | MBPP+ | MedQA | GPQA-D |
|---|---|---|---|---|---|---|
| TextMAS | 94.0 | 98.4 | 93.4 | 66.9 | 70.4 | 37.7 |
| LatentMAS | 94.3 | 98.7 | 94.2 | 72.1 | 71.6 | 31.7 |
| Text Skill | 94.8 | 99.2 | 94.0 | 70.9 | 70.9 | 30.5 |
| SkillMem | 92.6 | 98.9 | 93.9 | 67.3 | 69.7 | 35.3 |
The results are consistent with our earlier observations:
- 4B: SkillMem outperforms Text Skill on all six benchmarks, mainly because it reduces failures in latent communication.
- 8B: Results are mixed, with SkillMem performing better on GSM8K, MedQA, and MBPP+, but worse on the other three benchmarks. The original skills provide little benefit at this scale, even as text.
- 14B: Results remain task-dependent. SkillMem performs better on GPQA-D but worse on GSM8K and MBPP+. However, our new MBPP+ experiments show that SkillMem can preserve most of the gains when more effective textual skills are available.
(3) Does Table 2b include retrieval latency?
No. Table 2b measures the complete Planner → Critic → Refiner → Judger execution, excluding retrieval for all methods.
- SkillMem (7.95 s): Includes all reader operations during inference.
- Text Skill (12.98 s): Runs inside LatentMAS and includes the prefill cost of skill text.
- Retrieval (0.244 s): Uses the same query encoding and similarity search for both methods.
Including retrieval, SkillMem takes 8.19 s, compared with 13.22 s for Text Skill (Qwen3-8B, 100 MedQA questions).
We will add Text Skill accuracy to Table 1 and clarify the comparison settings and retrieval cost in Table 2b.
Reply to Reviewer tCys (1/3)
Thank you for your helpful suggestions. We conducted both requested experiments and added the missing Text Skill comparison.
We tested both settings using Qwen3-4B with the same protocol and seeds as Table 1. We also included additional controls to understand whether the improvement comes from skill content or simply training the reader.
Accuracy gains are measured relative to LatentMAS without memory, pooled over four benchmarks. All comparisons use paired evaluations.
The results show three things:
Exact skill selection matters less on these benchmarks. Swapping memories between skills retains 98% of the gain. The difference from Full SkillMem is only 0.5 ± 1.2 points. This suggests that skills within the same benchmark often provide similar benefits.
Separate skill memories still matter. Training one memory shared by all skills retains only 57% of the gain. On MBPP+, accuracy drops from 65.4% to 31.3%. Different skills can induce different behaviors during training, which are harder to preserve in one shared memory.
The improvement depends on skill content, not just the reader. Training without skills retains only 2% of the gain. Replacing useful skills with irrelevant text reduces it to 45%.
In summary, correct skill retrieval has limited impact in our original benchmarks, but the content of the skills and their separate memories are important during training. The skills within each benchmark are closely related, which may explain why their trained memories have similar effects at inference. In the next experiment, we examine skills with more distinct behaviors and test whether SkillMem can preserve those differences.