- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Inspired by Complementary Learning Systems (CLS) in cognitive science, this paper introduces SkillBrain, a memory-centric framework for continual skill evolution that reformulates the evolution loop from “Skill -> Trace -> New Skill” to “Skill -> Trace -> Memory -> Skill”. It separates episodic memory, which rapidly records concrete experiences, from semantic memory, which gradually consolidates reusable knowledge, and compiles deployment skills on demand for the current Executor and Harness. Experiments across six standard benchmarks, as well as task shifts, harness shifts, and cross-executor reuse settings, demonstrate performance gains while examining the limitations of SkillBrain’s continual adaptation.
Critical strengths
- The paper clearly distinguishes multi-round skill optimization on a single task from continual skill evolution. It identifies a key limitation of existing methods: treating the executable skill itself as the sole persistent state entangles reusable knowledge with execution details. The motivation is sound and well justified.
- Drawing on Complementary Learning Systems to decouple rapid episodic encoding from gradual semantic consolidation is intuitive and reasonable. The method demonstrates performance improvements across six standard settings spanning search QA, spreadsheet and document processing, mathematical reasoning, and long-horizon interaction.
Critical weaknesses
- Dynamic recompilation underperforms in key stress tests, leaving its adaptation capability insufficiently supported. A central claim is that skills should serve as interfaces dynamically compiled from memory for the current Executor and Harness. However, the two experiments directly testing this capability expose clear limitations: SkillBrain-CL underperforms the conventional SkillOpt-CL baseline in the ten-stage Harness Shift stress test, while recompilation for the target executor reduces performance by 12 percentage points relative to reusing the existing deployment skill and even slightly trails ordinary summary recompilation. Although the ablation in Table 3 supports the role of recompilation on standard benchmarks, its value for adaptation under changing conditions requires further validation.
- The Cost-Aware Meta-Optimization module lacks empirical evaluation and ablation support. Section 3.5 and Appendices D and E devote substantial space to its formal objective, prompt template, and parameter search space. However, the main experiments, ablation study, and appendix analyses do not report actual token or call cost savings, parameter trajectories, or performance and cost comparisons with and without this module. It therefore reads like an empirically unverified addition.
- Incomplete continual-learning and cross-executor evaluations limit support for the core conclusions. Forgetting is evaluated only through Probe Forgetting on five fixed examples per topic, totaling 60 examples, without Full Forgetting over the complete test matrix. EvoSkill-CL and Reflect are also marked as “Not run” in the ten-stage Harness Shift experiment. In addition, the cross-executor experiment uses the target model, Qwen3.6-35B-A3B, to construct persistent states from source-model trajectories, with only 16 shared training tasks and 100 test questions. It therefore does not independently test the transfer of a source-built memory substrate to a heterogeneous target model. Although the paper acknowledges some of these limitations, the corresponding evidence remains incomplete.
- Several formatting and visualization flaws affect readability and presentation quality. Some appendix tables repeat the same numbered caption above and below the table, such as Tables 7 and 10. In Figures 4 and 5, the internal titles and subtitles overlap, and the internal figure numbers disagree with the manuscript captions. These issues are conspicuous in the current version and require systematic checking and correction.
Questions for authors
- Why does Structured Memory + Recompile underperform Structured Memory + Reuse by 12 percentage points and even slightly trail Summary + Recompile? Under frequent harness changes, why does the Deployment Compiler fail to effectively prevent interference from prior interface syntax and perform particularly poorly with interface E’s text-command format? Can the authors further distinguish the effects of information loss during compilation, failure to follow interface constraints, and candidate selection?
- Was the module proposed in Section 3.5 enabled in the main and continual-learning experiments? If so, how much computational cost did it save relative to fixed hyperparameters, including tokens, API calls, or runtime? Is an ablation study available for this module?
- What is the total token consumption of SkillBrain relative to WikiSkill, SkillOpt, and EvoSkill in the main and continual-learning experiments? Does it include trajectory encoding, memory-organizer retrieval and read/write operations, exploration skill generation, and deployment compilation? In the gated continual-learning setting, how much additional cost comes from generating and validating 32 candidates per deployment round? Although the paper describes some budget controls and cross-method budget differences, end-to-end cost reporting is still needed to assess performance gains relative to computational investment.
- Please systematically correct duplicate table captions, overlapping figure text, and inconsistent numbering in the appendices. Can the authors also complete the missing baselines in the ten-stage Harness Shift experiment and the forgetting evaluation over the full test matrix?
5 comments
Response to Reviewer x1cg — Part 2/4
2. Cost-Aware Meta-Optimization (W2, Q2)
We first clarify that all original main-benchmark and continual-learning results used fixed control parameters. Meta-Optimization did not generate those headline gains. It is an optional cost-reduction component for Memory maintenance and deployment decisions. We evaluated it separately on the same SearchQA Task Shift stream to quantify its net accuracy–cost trade-off.
| Setting | Final Macro EM (%) | Total Tokens | Calls |
|---|---|---|---|
| Fixed Parameters | 81.53 | 5.60M | 1,860 |
| Meta-Optimization | 81.18 | 4.76M | 1,630 |
The cost-aware variant reduces total tokens by 15.0% and model calls by 12.4%, at the expense of just 0.35 pp Final Macro EM. Crucially, these totals include the diagnostic, parameter-validation, and rollback costs of Meta-Optimization itself. We therefore report the net end-to-end result, rather than comparing only the downstream costs after ignoring the controller's overhead.
Most savings come from avoiding redundant historical retrieval and adaptively adjusting how often Memory is organized and how much budget is devoted to deployment candidates. In a Memory-Centric system, accumulated experience is valuable but need not be fully reread at every step. This result suggests that the associated costs can be reduced with only a small drop in accuracy; it does not establish that Meta-Optimization improves accuracy or is necessary for continual learning.
We therefore position Meta-Optimization as an optional efficiency mechanism, separate from the fixed-parameter experiments establishing the core approach. The additional ablation makes this distinction explicit.
The adaptive controls adjust historical retrieval, Memory-organization frequency, and the deployment-candidate budget. They reduce redundant processing of accumulated experience, rather than adding supervision. The table reports the net controller effect; it does not isolate each parameter's contribution or provide a complete stage-by-stage parameter trajectory. This keeps the optional module separate from the core results.
3. End-to-End Computational Cost and Fair Comparison (Q3)
We agree that equal training rollouts do not imply equal total compute. The original fixed-rollout study (Section 4.5 / Appendix H.1) instead controls collected experience: with 32 training rollouts, Wide Search reaches 77.1%, Skill-Centric at most 78.2%, and Memory-Centric 83.1% at R=8. This tests whether organizing and reusing experience across rounds helps under a fixed experience budget, not whether the total cost is equal.
The reviewer also asks for per-arm end-to-end costs in this 32-rollout study. The six-benchmark totals below are not token measurements for its three arms, so we do not infer their cost ranking from that table. Instead, the matched-token study directly tests whether SkillBrain's advantage remains under equal inference budgets; the 32-rollout study addresses how experience is organized.
We now additionally count tokens and model calls throughout the complete evolution process: task execution, Skill generation, trajectory encoding, historical retrieval, Memory updates, deployment compilation, and validation. This includes overhead absent from a rollout-only accounting.
(1) Six-benchmark end-to-end cost.
| Method | 4B Tokens | 35B-A3B Tokens | 35B-A3B Avg. |
|---|---|---|---|
| EvoSkill | 9.3M | 14.7M | 63.5 |
| SkillOpt | 9.8M | 15.8M | 63.9 |
| WikiSkill | 11.1M | 17.6M | 63.6 |
| SkillBrain | 12.8M | 20.0M | 66.2 |
For 35B-A3B, SkillBrain spends 26.6% more tokens than SkillOpt (20.0M vs. 15.8M) and improves the average score by 2.3 pp (66.2 vs. 63.9). We report both sides of this trade-off explicitly. Higher performance at a larger budget, by itself, would not establish that the Memory-Centric mechanism is more efficient.
Matched total-token budgets. We therefore also compare the methods with the same overall token allowance, rather than only the same number of collected rollouts.
| Total Token Budget | SkillOpt Avg. | SkillBrain Avg. |
|---|---|---|
| 12.0M | 62.4 | 63.8 |
| 15.8M | 63.9 | 65.1 |
Under matched budgets, SkillBrain still improves on SkillOpt by 1.4 pp at 12.0M tokens and 1.2 pp at 15.8M. These gains cannot be explained solely by giving SkillBrain more tokens. The two experiments answer different questions: fixed rollouts test experience organization; matched tokens test whether its benefits persist under an equalized compute proxy. We report both, together with the method's full overhead.
Response to Reviewer x1cg — Part 3/4
Continued from Q3: validation costs and continual-learning evidence.
(2) Continual-learning and candidate-validation overhead.
On SearchQA Task Shift, the original SkillBrain U and G settings achieve Final Macro EM of 81.53% and 80.41%, respectively. Their deployment and validation costs, however, are markedly different: U produces 16 deployment candidates and consumes 5.60M tokens, whereas G produces 512 candidates, evaluates 544 validation panels, and consumes 18.40M tokens.
G expends 12.80M more tokens than U without achieving better Final Macro EM in this experiment. Accordingly, extensive Validation-Gated selection is not a free or indispensable component: its additional cost must be distinguished from the gains due to accumulated knowledge.
The 12.80M-token difference is an end-to-end U–G gap, not the isolated cost of validation panels. More candidates also incur compilation and evaluation work. Reporting candidate counts, validation panels, and total cost avoids assigning an unsupported share of the difference to any single step.
Crucially, SkillBrain-CL U already achieves +5.15 FWT and +3.72 BWT without a Validation Gate, demonstrating positive transfer in the ungated setting. Revised Appendix J.2 reports end-to-end token usage, model calls, and validation overhead, and clearly distinguishes fixed training rollouts from matched total-token budgets.
4. Continual-Learning Evaluation and Cross-Executor Transfer (W3, Q4)
(1) Full Forgetting and cross-run stability. The original Probe Forgetting metric uses five fixed questions per topic. We agree that this inexpensive diagnostic should not be mistaken for forgetting on the complete test distribution, and therefore added a full-test evaluation and multiple independent runs.
Using the saved deployment Skill at each learning stage, we reconstructed a stage-by-topic performance matrix on all 1,400 test questions and computed Full Forgetting. The evaluation is separate from training: test answers are not used to update Memory, select deployment candidates, or guide later learning.
| Method | U Full Forgetting ↓ | G Full Forgetting ↓ |
|---|---|---|
| SkillOpt-CL | 5.1 | 5.8 |
| EvoSkill-CL | 3.0 | 4.6 |
| Reflect | 2.8 | 4.9 |
| SkillBrain-CL | 2.1 | 2.7 |
Full Forgetting is higher than the original Probe estimates, but SkillBrain remains best in both U and G. This broadens the retention evidence beyond final accuracy or a small probe. Full Forgetting complements the original FWT/BWT metrics, which measure other aspects of knowledge transfer across the learning sequence.
We also ran three independent seeds with different topic orders. In the ungated setting, SkillBrain-CL averages 81.48% Final Macro EM versus 78.89% for EvoSkill-CL, a 2.59 pp lead; the former closely matches the original 81.53%. FWT and BWT are positive in every run. This does not imply zero variance, but supports the robustness of positive transfer beyond one favorable ordering. The reported EM advantage is a three-run mean; without the per-seed EM table, we do not claim SkillBrain exceeds EvoSkill in every individual run. Appendix J.1 documents the experiments.
(2) Completing the ten-stage Harness Shift baselines. Beyond retention under Task Shift, we agree that a changing Executor interface is an important stress test. The original Harness Shift protocol spans five interfaces, ten learning stages, and nine interface transitions. We completed the missing EvoSkill-CL and Reflect comparisons under this same protocol, and evaluated the compilation intervention described in response to W1/Q1.
| Method | U Avg. EM (%) | G Avg. EM (%) |
|---|---|---|
| SkillOpt-CL | 63.63 | 53.70 |
| EvoSkill-CL | 56.8 | 54.6 |
| Reflect | 49.7 | 43.1 |
| SkillBrain-CL | 30.43 | 50.19 |
| SkillBrain-CL + Interface-/Syntax-Constrained Compilation | 62.8 | 68.9 |
For the added SkillBrain configuration, U uses interface constraints and syntax filtering, but no deployment-selection Validation Gate. G additionally checks candidate validity. Holding the underlying Memory and training experience fixed, constrained compilation raises U from 30.43% to 62.8% (+32.37 pp) and G from 50.19% to 68.9% (+18.71 pp). The G result exceeds every original baseline listed here, while U approaches the strongest U baseline, SkillOpt-CL at 63.63%.
We do not discard the poor original Harness Shift result: it identified a genuine failure of deployment compilation. However, the fixed-Memory recovery, interface-error logs, and Flat Memory control together show that interface adaptation is a major bottleneck, separate from structured Memory's capacity to retain experience. We report the original and improved results side by side.
Response to Reviewer x1cg — Part 1/4
We thank the reviewer for recognizing the motivation and Memory-Centric Skill Evolution design of SkillBrain. We appreciate the questions about dynamic recompilation, the costs of Meta-Optimization, continual-learning evaluation, and cross-Executor transfer. In response, we added controlled experiments and clarified three distinct factors: what knowledge Memory retains, how faithfully compilation turns it into executable Skills, and how much computation deployment selection requires.
1. Dynamic Recompilation and Harness Adaptation (W1, Q1)
Motivation and existing evidence. Original Table 3 shows that removing Recompile reduces average performance from 87.1% to 85.4%, supporting the benefit of deployment adaptation. Table 14 shows that Structured Memory + Reuse outperforms Summary + Reuse (73% vs. 60%), supporting structured knowledge retention. Yet recompilation can underperform direct reuse in a cross-Executor setting. We agree this requires attribution: knowledge may remain in Memory but be omitted or incorrectly expressed in the compiled Skill.
The reviewer also notes that Summary + Recompile slightly exceeds Structured Memory + Recompile. This is not contradicted by the reuse-only comparison: under an imperfect compiler, a richer structured representation may be harder to translate faithfully than a shorter summary. Information can be lost through rule selection, missing applicability conditions, or target-inappropriate instructions. We present these as plausible failure modes, not as a measured causal decomposition of the summary/recompilation gap.
(1) Cross-Executor recompilation. To investigate this distinction, we kept the Memory, target Executor, and test set fixed and intervened in compilation fidelity and deployment candidate selection.
Knowledge-Preserving Compilation checks that key rules and their applicability conditions survive translation from Memory into the executable Skill. It raises EM from 61% to 69%, recovering 8 of the 12 pp lost relative to reuse. Selecting among four candidates on an independent validation set further raises EM to 74%, without using test answers.
Because the underlying Memory is unchanged, these results help isolate where performance is lost. The recovery from 61% to 69% reflects improved knowledge preservation; the further rise to 74% also reflects deployment candidate choice. We do not claim recompilation is always better than reuse: a compiler can discard useful information. Rather, the evidence implicates compilation fidelity and candidate selection, instead of showing that structured Memory lacked transferable knowledge. Revised Section 3.4 clarifies this distinction, with controlled results in Appendix J.3.
(2) Harness Shift adaptation. A related but distinct failure occurs when the environment changes its tool interface. Original Appendix G.2 reports that, among 915 erroneous read actions on interfaces B/D for SkillBrain-CL U, 813 (88.9%) used an
openkey belonging to another interface. This is a concrete, observable error signature: the generated action is syntactically incompatible with the current Harness, even when the underlying task may be understood.We therefore fixed the original training trajectories and E/K Memory and modified only deployment compilation: supplying target-interface constraints, filtering syntax inherited from previous interfaces, and checking candidate validity. Training trajectories and the number of deployment candidates were not increased.
The five-interface average improves by 18.71 pp (50.19% to 68.9%), while the most challenging interface E improves by 54.09 pp (0.21% to 54.3%). Importantly, with the same compilation and validation mechanism, structured E/K Memory exceeds Flat Memory by 6.2 pp (68.9% vs. 62.7%). Thus, correcting interface adaptation restores much of the performance, but does not eliminate the benefit of structured knowledge organization.
These results separate two complementary mechanisms: Memory retains knowledge across experiences, while Harness-conditioned compilation expresses it in the current Executor's interface. The original degradation exposes a limitation of the compiler; the fixed-Memory intervention and Flat Memory control show why that limitation should not be conflated with the value of knowledge organization.