- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MSCPM (Multi-level Skill Composition for Procedural Memory) is a training-free procedural-memory framework for LLM agents, motivated by two analogies to human procedural memory: hierarchical chunking and dynamic co-activation. It builds a three-layer hierarchical skill graph bottom-up — operation, subgoal, and task-domain skills — via LLM annotation, embedding-based clustering, and PrefixSpan frequent-pattern chunking, storing only downward cross-layer edges. At inference, a three-stage cascade composes skills per task: coarse-to-fine subgoal selection (locate task domains → expand to candidate subgoals → LLM-select), typed plan orchestration (each plan step tagged Inherit/Adapt/Create), and dynamic operation assembly (co-activation edges among sibling operations induced transiently and ranked by personalized PageRank). A skill-iteration mechanism uses seeded constrained clustering to absorb or create nodes across batches, and Appendix C proves the memory bank's size is upper-bounded by the number of underlying patterns KH (not by trajectory count) under an ε-separability assumption. On AppWorld and BFCL-V3 with Qwen3-8B/14B/32B, MSCPM beats the strongest baseline (among AWM, ReMe, GoS, SkillX) by 1.94–2.58 Avg@4 / 1.99–3.97 Pass@4, with larger relative gains on smaller backbones, and smaller MSCPM-equipped agents reportedly surpass larger memory-free ones. Ablations attribute most of the gain to the hierarchy and to dynamic assembly.
Critical strengths
-
Strong, honest comparison against the right baselines — this is the main thing the other procedural-memory submissions in this area lack. Unlike papers that compare only against their own memoryless ablation, MSCPM evaluates against four representative, current competitors spanning the exact taxonomy it critiques: flat (AWM), insight-rewriting (ReMe), graph (GoS), and hierarchical (SkillX). It reports standard deviations over three runs, uses two benchmarks with genuinely different structure (AppWorld's 457 APIs and train–test shift vs. BFCL-V3's narrow tool set), and the gains are consistent across all three backbone sizes. The ablation story is also internally coherent and falsifiable: the authors predict that the task-domain layer and subgoal selector should help more on AppWorld than BFCL-V3 (because BFCL exposes only 10–30 tools) and the data confirm it (−1.58/−0.84 on AppWorld vs −0.38/−0.05 on BFCL). An ablation whose asymmetry matches a stated mechanism is more convincing than uniform drops.
-
The "transient within-layer co-activation" idea is a genuine design contribution with a matching efficiency argument. The central novelty — storing only cross-layer edges and inducing within-layer relations on demand via shared-parent siblings, then dissolving them after the task — is a clean answer to the real tension the related work identifies (hierarchical graphs under-compose within a layer; full graphs over-connect and are seed-sensitive). It's not just asserted: the growth proof in Appendix C gives the storage claim teeth by bounding node count by pattern diversity KH rather than trajectory count, and the skill-iteration curves (Fig. 2f–j) empirically show new-node creation decaying to absorption. This is a more principled treatment of memory-bank growth than most of this literature offers.
Critical weaknesses
-
The absolute performance is low enough that the headline framing ("small + MSCPM beats large memory-free") rests on a handful of points near the noise floor, and the pipeline's cost is entirely unreported. On AppWorld, even the best 32B result is 34.52% Avg@4 — these are hard tasks where all methods are failing most of the time, so the 1.94–2.58-point margin over SkillX, while consistent, is small in absolute terms and the std bands overlap in several cells (e.g., 8B AppWorld: MSCPM 19.40±0.17 vs SkillX 16.52±0.68 is clear, but 14B BFCL MSCPM 56.94±0.54 vs SkillX 54.67±0.73 is ~2 points with ~0.7 std — tight but not overwhelming). The "smaller-beats-larger" claim is the kind of statement that needs a significance test, not just non-overlapping means read by eye. More seriously, MSCPM is by far the heaviest pipeline here — six LLM modules offline (annotators, inducers) plus three LLM-driven composition stages online (selector, orchestrator, PageRank assembly) — yet the paper reports no token cost, latency, or construction cost anywhere, on either axis. A framework whose selling point over flat methods is "graph flexibility at near-hierarchical cost" cannot omit the cost numbers that would substantiate "near-hierarchical." Without them, a reader cannot tell whether the 2-point gain is bought with 2× the inference calls of AWM.
-
The convergence proof is real but largely decorative — it proves a property no one disputes, under an assumption that the paper's own clustering is designed to violate. Appendix C proves the node count is bounded by KH and new-node creation decays. But this follows almost immediately from ε-separability (Assumption 1): once you assume same-pattern annotations are <ε apart and cross-pattern ≥ε apart, "clusters map injectively to patterns" is nearly definitional, and the decay is a standard missing-mass argument. The assumption is also exactly what fails in practice — LLM-annotated skill names exhibit the "surface-form variation" the paper cites as its reason for clustering, which means same-pattern items are often not within ε and the clean bound degrades to the c-factor and η-fraction extensions the authors relegate to a paragraph. So the proof certifies bounded growth in the regime where clustering is trivial, and says little about the realistic regime. This isn't fatal, but the paper leans on "provably convergent growth" as a headline contribution, and the theorem is weaker than that billing suggests.
Questions for authors
-
Can you report construction cost (tokens/LLM calls for the six offline modules) and per-task inference cost (tokens, latency, LLM calls for the three composition stages) for MSCPM versus AWM, ReMe, GoS, and SkillX? The "near-hierarchical cost" claim is central and currently unsupported.
-
For the small-margin cells and the "smaller+MSCPM > larger memory-free" claim, can you run a paired significance test across the three runs? Specifically: is 8B-MSCPM vs 14B-No-memory significant on each benchmark?
-
How sensitive is ε-separability to real LLM annotation noise? Can you report the empirical η (fraction of cross-pattern mis-clusters) and show how node count and performance behave as the per-layer thresholds ε_op/ε_sg vary around the chosen 0.23/0.28?
-
The whole pipeline depends on Qwen-produced skill annotations. Did you evaluate annotation/clustering quality directly (e.g., human-judged cluster purity on a subset), and how do downstream gains degrade when annotations are noisier?
-
Retrieval is "permitted only once at the beginning of each task" for all baselines — but MSCPM's dynamic assembly induces co-activation graphs per plan step during composition. Is that composition counted as a single up-front retrieval, and does this give MSCPM more effective memory access than the baselines under the stated fairness constraint?
-
BFCL-V3's training split is 50 randomly chosen tasks; AppWorld uses 90. How stable are the results to the BFCL split choice, given you created it yourselves?
7 comments
Response to Reviewer LsfX (2/7) W1 & Q2: Statistical significance of the improvements
Response. We complement the two conclusions, namely (i) the improvement of MSCPM over the strongest baselines and (ii) a smaller model augmented with MSCPM outperforming larger memory-free models, with a significance analysis based on McNemar's exact test.
For each comparison, we pair the binary outcomes (success/failure) of the two systems on each individual (task, run, execution). Let denote the number of pairs where MSCPM succeeds while the baseline fails, and the number of pairs where MSCPM fails while the baseline succeeds. After discarding pairs on which the two systems agree, let be the total number of discordant pairs. The exact two-sided -value is
The following Table A reports the improvement Δ (in percentage points) between MSCPM and the strongest baseline in each cell of Table 1, as well as the corresponding p-value. For Avg@4 cells, binary outcomes are paired per (task, run, execution) (3 runs × 4 executions per task; 2016 pairs on AppWorld and 1800 on BFCL-V3). For Pass@4 cells, pass indicators are paired per (task, run) (3 runs per task; 504 and 450 pairs in total).
Table A: Significance of MSCPM over the strongest baseline in each cell of Table 1.
| Model | AppWorld Avg@4 () | AppWorld Pass@4 () | BFCL-V3 Avg@4 () | BFCL-V3 Pass@4 () |
|---|---|---|---|---|
| Qwen3-8B | +2.58 (0.0237) | +3.97 (0.0339) | +2.34 (0.0273) | +3.78 (0.0009) |
| GLM-4-Z1-9B | +3.32 (0.0118) | +4.17 (0.0086) | +2.50 (0.0312) | +3.33 (0.0107) |
| Qwen3-14B | +2.58 (0.0212) | +3.38 (0.0430) | +2.27 (0.0396) | +3.11 (0.0451) |
| Qwen3-32B | +2.33 (0.0495) | +1.99 (0.0987) | +1.94 (0.0410) | +2.22 (0.0768) |
Out of the 16 comparisons, 14 are statistically significant (). The only two exceptions are the two Pass@4 cells of Qwen3-32B ( and ), for two reasons: (1) the improvement of this model is the smallest among all models (+1.99/+2.22), because the largest model has the strongest intrinsic capability and thus benefits least from memory augmentation, i.e., the memory-augmentation effect diminishes as the underlying model becomes stronger; (2) the pairing granularity of Pass@4 is the coarsest (only 3 pass indicators per task), which limits the statistical power. Moreover, the direction of improvement is perfectly consistent across all 16 comparisons, and having 14 out of 16 simultaneous tests significant is far beyond the chance level.
Table B: Smaller models augmented with MSCPM vs. larger memory-free models (paired in the same way as above). "--" indicates that the smaller MSCPM-augmented model does not exceed the larger memory-free model in that cell.
| Comparison | AppWorld Avg@4 () | AppWorld Pass@4 () | BFCL-V3 Avg@4 () | BFCL-V3 Pass@4 () |
|---|---|---|---|---|
| Qwen3-8B + MSCPM vs. GLM-4-Z1-9B (no mem.) | +3.92 (0.0022) | +6.75 (0.0002) | +5.73 (0.0013) | +10.89 (0.0001) |
| Qwen3-8B + MSCPM vs. Qwen3-14B (no mem.) | -- | -- | +1.06 (0.1102) | +5.11 (0.0029) |
| GLM-4-Z1-9B + MSCPM vs. Qwen3-14B (no mem.) | -- | +0.60 (0.3750) | +3.06 (0.0235) | +6.89 (0.0014) |
| Qwen3-14B + MSCPM vs. Qwen3-32B (no mem.) | +0.89 (0.4421) | -- | +2.94 (0.0449) | +6.66 (0.0003) |
As shown in Table B, compared with slightly larger memory-free models, the smaller MSCPM-augmented models win comprehensively and significantly (Qwen3-8B + MSCPM vs. GLM-4-Z1-9B: all four cells with ). Compared with substantially larger models, almost all cells show significant advantages on BFCL-V3 (e.g., all Pass@4 cells with ), presumably because the training and test distributions of BFCL are closer to each other, so the memory augmentation transfers better (see our response to Q6 below for an analysis of the BFCL-V3 split); on AppWorld, whose task distribution is more complex, the smaller models approach but do not consistently exceed the larger ones. Where cells are not significant, we frame them as the boundary of the claim rather than as counterexamples. The revised text states that smaller agents equipped with MSCPM can surpass larger memory-free ones — an existence claim established by the significant cells above. This boundary also follows from the mechanism: memory augmentation contributes a roughly constant gain (+5–9 Avg@4 and +8–14 Pass@4 over each backbone's no-memory score, Table 1), whereas the scale gap to a memory-free model grows (+8.3 for 8B→14B and +13.1 for 8B→32B on AppWorld). Significance therefore recedes exactly as the scale gap grows — the dose–response pattern expected of a bounded memory gain; conversely, an 8B model with memory beating a 32B one in every cell would be implausible. We do not claim superiority in the cells where the scale gap exceeds the memory gain.
Response to Reviewer LsfX (3/7) W2: Positioning and scope of the growth analysis
W2: Positioning and scope of the growth analysis
Response.
(1). We agree with the assessment and have repositioned the analysis. Under strict -separability, the pattern–node correspondence does follow from the update rule, and the decay calculation is a standard missing-mass argument; the proposition is an idealized coverage analysis, not an unconditional guarantee for the full pipeline. We have restructured Appendix C accordingly and reworded the "provably convergent growth" phrasing in the conclusion to "bounded, coverage-driven growth".
(2). One clarification on surface-form variation. Surface-form variation is a property of annotation text, whereas Assumption 1 concerns the embedding geometry: differently worded annotations of the same skill can still lie within of each other in embedding space — this is precisely why the pipeline clusters embeddings rather than strings. Nevertheless, we agree that strict global separability is not established for LLM-generated annotations and should not be presented as a practical guarantee.
(3). Revisions and empirical support (see Q3 and Q4): growth stability does not require semantic clustering correctness. The critique conflates two questions that Assumption 1 answers simultaneously — whether clusters are semantically correct, and whether the memory grows without bound — and our original presentation did not separate them either. Appendix C now separates them: (i) a new geometric capacity bound (no semantic assumptions): with a frozen encoder and frozen index embeddings, retained node indices form an -packing of the embedding space, so , finite for any fixed — this holds exactly in the regime the reviewer describes, where same-pattern annotations fragment and distinct patterns merge ( is a capacity bound at the threshold's resolution, not a semantic-quality guarantee); (ii) a new decay bound without separability: under i.i.d. annotation batches and no pruning, the uncovered embedding mass is monotonically non-increasing, , and ; (iii) the original result is retained as the idealized semantic special case ( under Assumption 1). We have also corrected several imprecisions in the original statements: the creation bound is now stated in conditional form (the previously stated unconditional form with the random on the right-hand side was ill-posed; averaging gives ); "independent of the number of rounds and trajectories" now refers to the upper bound rather than the node count; per-round monotone decay is claimed only under a fixed annotation batch size; and the extensions are restated with their correct hypotheses (the multiplicity cap must be global across rounds; an -fraction of mis-annotation preserves bounded growth only if the errors are not persistently novel — recurring errors add at most finitely many nodes, while persistently novel ones accumulate but remain capped by the geometric capacity bound). Empirically, practice operates close to the analyzed regime: the negative-silhouette fraction is 1–6% across layers and benchmarks (Q3); node counts decrease monotonically as grows, as the packing capacity predicts; node creation decays monotonically across construction rounds (Figure 3(f)–(h)); and under annotation-level noise injection up to , downstream gains are largely retained (Q4: at the most damaging injection point) — the injected annotations reuse existing (name, desc) pairs and hence recur rather than being persistently novel, the benign regime of the corrected -extension.
Response to Reviewer LsfX (1/7) W1 & Q1: Offline memory-construction / online execution cost
W1 & Q1: Offline memory-construction / online execution cost
Response. We have added a new subsection, §4.4 Cost Analysis (see Figure 2 in the manuscript), specifically to analyze the offline memory-construction cost and the online execution cost. We compare the offline memory-construction and online execution costs of all methods on AppWorld and BFCL-V3 with Qwen3-32B.
Offline memory-construction cost covers LLM calls, token usage, and wall-clock time averaged per iteration round over three iteration rounds. As shown in Figure 2(a)–(c) and (g)–(i), AWM summarizes the successful trajectories of multiple similar tasks into an abstract workflow with LLMabstract, achieving the lowest construction cost. ReMe extracts experiences via LLMsuccess, LLMfailure, and LLMcontrastive, and verifies each with LLMvalidator. SkillX extracts plan, functional, and atomic skills via LLMplan, LLMfunctional, LLMatomic, LLMfeedback, and LLMvalidator. On BFCL-V3, where all APIs are exposed in the system prompt, atomic extraction is unnecessary. For GoS, we feed the option skills extracted by the MSCPM pipeline as the source for building its skill graph, and additionally employ LLMdiscriminator to annotate its edges with four types of relations between relevant skills (dependency, workflow, semantic, and alternative) together with confidence scores. The offline construction cost of MSCPM is higher than that of AWM, yet lower than or comparable to the strong baselines ReMe and SkillX on most metrics, and lower than that of GoS in all cases.
Online execution cost covers the same metrics averaged per task trajectory. As shown in Figure 2(d)–(f) and (j)–(l), all memory-augmented methods reduce the number of online interaction steps compared with the memory-free method, indicating that memory augmentation reduces redundant exploration and execution. AWM's compact workflows consume few tokens and little time, yet it takes the most interaction steps among memory-augmented methods and achieves the lowest success rate, showing that purely abstract workflows cannot effectively guide agent execution. ReMe additionally applies LLMrerank and LLMrewrite to the retrieved memories, incurring extra online cost. SkillX rewrites the retrieved memories into a plan via LLMplan-rewrite and retrieves low-level skills for each plan step, further invoking LLMrerank when many candidates are returned, likewise incurring a high online cost. MSCPM achieves the fewest LLM calls among all methods, together with lower wall-clock time and token usage than the strong baselines SkillX and ReMe.
Summary. MSCPM achieves offline memory-construction and online execution costs lower than those of the strong baselines while improving task success rates, demonstrating the efficiency of our method.