acceptodds
Withdrawn from ICLR 2027

When Skills Remain But Access Fails: Factorized Skill Memory for Continual Skill Internalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Skill internalization aims to convert externally specified procedural skills into persistent model capabilities, allowing language-model agents to execute them without retrieving or reinjecting skill descriptions at inference time. Yet when new skills are learned continually, whether previously internalized skills remain usable is still poorly understood. We study forgetting in continual skill internalization and uncover a clear hierarchical structure. Early in subsequent training, a model may lose the ability to invoke an old skill from the raw request, while supplying only the correct canonical skill name substantially restores the learned behavior; with further optimization, this conditional execution also degrades. We call these two stages skill-addressing forgetting and skill-execution forgetting. Guided by this finding, we introduce Factorized Skill Memory (FSM), which separates continual skill memory into a plastic Address Memory and a stable Execution Memory. At each stage, FSM trains a new LoRA executor using only current-stage data, freezes it after acquisition, and appends it to Execution Memory; in parallel, Address Memory incrementally learns to map current requests to executors without replaying historical requests or execution trajectories. At inference time, Address Memory selects one executor from the raw request, and only that frozen executor performs a single language-model generation. On a five-stage benchmark built from 100 TaskBench skills, FSM achieves 65.95% mean exact match and 93.38% addressing accuracy across two task orders, improving over the strongest replay-free parameter-isolation baseline by 14.3 percentage points while remaining within 1.9 points of Replay, which retains historical training examples. These results show that continual skill forgetting unfolds hierarchically, and continual skill retention depends on both stable procedural content and reliable access to it. Code is available at https://anonymous.4open.science/r/fsm-anonymous-release-F7A3/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Agent, LLM, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5gswOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies forgetting when an LLM learns tool-use skills one task family after another. They split 100 TaskBench tools into 5 families and train Qwen3-8B on them sequentially, where the model has to output a tool graph from only the request. Old-family accuracy drops to 0 within a few steps, but adding the gold skill name to the prompt brings it back to ~70%, and with more training even that degrades. They call these skill-addressing and skill-execution forgetting. Based on this they propose FSM, which trains and freezes one LoRA per family and uses a Naive Bayes classifier on the request (updated without old requests) to pick which LoRA to use. FSM gets 65.95% EM over two task orders, 14.3 points over Progressive-LoRA and within 1.9 points of replay.

Critical strengths
  1. The reactivation probe is a useful diagnostic experiment. The same checkpoint that gets 0 on all old T2 requests bounces back to ~70% when given the skill names, while a wrong name from the same family or random text doesn't, and tracking it over training shows the same order of failure for SFT, GRPO, and even SKILL0.

  2. The routing vs. execution breakdown is informative. Comparing the oracle router, FSM's router and the offline router on the same frozen LoRAs isolates which families are limited by routing and which by the LoRA itself, and FSM is effective in getting close to replay without keeping old data while only running one LoRA per request.

Critical weaknesses
  1. Diagnostic: the "correct skill address" is the gold tool names, which is a significant part of the answer graph, so "skills remain but access fails" might just mean the model still knows how to fill in a tool once it's told which tools to use. The paper admits the address reveals the gold node identity and calls it a diagnostic rather than a pure access probe, but there's no control that separates the two things (that confound). Also, the broader pattern that performance drops in LLM continual learning while the knowledge stays recoverable has been shown before in [1] and [2], which aren't discussed.

  2. Novelty and scope: FSM is one frozen LoRA per task with a separate task classifier, which is of a similar design as past works like EPI, AdapterCL, and Expert Gate. The 5 families are very different topics (daily life APIs vs speech vs vision), so routing them from the request is fairly easy for the simple classifier, and with only TaskBench it's unclear the router holds when families overlap more. Additionally, routing to a single LoRA also can't handle a request that needs tools from two families, raising concern over whether the method generalizes to real world tasks that mix tool families.

[1] Spurious Forgetting in Continual Learning of Language Models

[2] Understanding Catastrophic Forgetting in Language Models via Implicit Inference

Questions for authors
  1. Can you replace the tool names with arbitrary aliases, or give the collapsed checkpoint only the family name instead of the tool names? If recovery holds with just the family, it would be strong evidence for the split and also match the level FSM routes at.
  2. How were TaskBench requests whose tools span more than one family handled, and how does routing do when families share vocabulary?
Reviewer_d9UuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forgetting in continual skill internalization. It finds that closed-book performance can collapse while supplying the correct skill name still recovers substantial performance, with further training later reducing this recoverability. Based on this observation, the authors propose Factorized Skill Memory (FSM), which separates a continually updated request-to-executor selector from frozen, stage-specific LoRA executors. Experiments on a five-stage benchmark constructed from TaskBench show improvements over the evaluated replay-free baselines, with performance approaching replay.

Critical strengths
  1. The paper is well written and easy to follow. I find the question meaningful: after learning new skills, a model should still be able to use earlier skills directly from user requests.

  2. The diagnostic experiments are interesting. The incorrect-address and frozen-base controls help interpret the recovery obtained by supplying the correct skill name. The comparisons between selectors using the same execution bank are also helpful for understanding the contribution of routing.

Critical weaknesses

I have two main concerns:

  1. My first concern is whether the benchmark sufficiently captures the skill retention problem motivating the paper.

I find the problem meaningful, and the controlled setting is useful for studying forgetting. However, the current evaluation mainly measures the generation of predefined workflow graphs, without executing the tools or responding to execution feedback. Moreover, each request uses skills from a single family, so the benchmark does not test whether skills acquired at different stages can be combined. It therefore remains unclear how well performance in this setting reflects the retention and reuse of procedural skills in broader agent tasks. I would appreciate a clearer justification of this connection and a more explicit discussion of which aspects of continual skill internalization the benchmark captures.

  1. I also find the coverage of recent continual learning methods insufficient.

I understand that the paper’s main contribution is the problem setting and analysis, rather than primarily a new continual learning algorithm. However, evaluating representative recent methods is important for establishing the benchmark’s value and understanding which challenges remain unresolved. The main continual learning comparisons rely largely on earlier methods such as LwF, EWC, O-LoRA, EPI, and AdapterCL. Although the SKILL0 experiments are helpful, I would also like to see recent methods such as GainLoRA [1] and SLAO [2] evaluated in this setting. Since the paper already includes Replay as a reference, a recent replay-based approach such as FOREVER [3], evaluated with the same historical-example budget, would also be informative. These comparisons would help clarify how challenging the proposed setting is for current methods and how much FSM adds over recent alternatives.

[1] Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models. NeurIPS 2025.

[2] Merge before Forget: A Single LoRA Continual Learning via Continual Merging. ICLR 2026.

[3] FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning. ACL 2026.

I would consider raising my rating if the authors can better justify how the benchmark captures continual skill internalization, clarify the scope of the conclusions, and strengthen the evaluation with representative recent continual learning methods.

Questions for authors
  1. What does the prompt for correct-skill-address reactivation look like? For requests involving multiple skills, are the identifiers supplied as an unordered set or in the order of the target workflow?

  2. How are the adapters in Progressive-LoRA combined or selected at inference time?

  3. I would appreciate more implementation details for Address Memory, especially the hashing dimension, token or n-gram features, and Naive Bayes smoothing settings.

Reviewer_67PFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of iteratively learning skills without forgetting previously learned ones. The authors first observe that the performance on a previously learned skill might degrade when learning a new one, but providing the "skill address" can recover most of the initial performance, showing that the model still has some procedural knowledge associated with the skill. Motivated by this observation, they propose the Factorized Skill Memory framework, which maps each query to a skill identifier, and then trains one LoRA adapter per skill. At inference time, the query is routed to one LoRA adapter based on the predicted skill. Experiments show that the method successfully learns all consecutive skills, while baselines forget previously learned skills.

Critical strengths
  1. Iteratively learning new skills without forgetting is an important problem
  2. The observation that seemingly forgotten capabilities can be recovered by providing the skill address is interesting
Critical weaknesses
  1. The novelty of the method is limited. The proposed framework essentially learns one LoRA adapter per skill, using a simple routing mechanism based on the query. All the baselines evaluated in Section 6.3 train a single model on all skills iteratively, so the main experimental results are not a fair comparison.
  2. The writing throughout the manuscript is quite poor and hard to follow. Despite the overall method being conceptually simple (one LoRA adapter per skill), the paper is full of loosely defined terms (eg "skill address", "request", "shard identity", "executor", "bank", etc) and dense phrasing (for example "The reactivation diagnostic exposes the Address–Execution split at individual-skill granularity" l.86, or "Training may use stage-local workflow supervision and, for skill-guided baselines, textual skill cards." l.140) which makes it unnecessarily difficult to read.
Questions for authors
  1. Can you provide more motivation for why the p_Address mapping is instantiated as a hashing function and a naive bayes classifier? Currently there is not much justification for this design choice which does not seem particularly natural. Wouldn't the hash function give different outputs semantically equivalent queries for example?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.