- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies forgetting when an LLM learns tool-use skills one task family after another. They split 100 TaskBench tools into 5 families and train Qwen3-8B on them sequentially, where the model has to output a tool graph from only the request. Old-family accuracy drops to 0 within a few steps, but adding the gold skill name to the prompt brings it back to ~70%, and with more training even that degrades. They call these skill-addressing and skill-execution forgetting. Based on this they propose FSM, which trains and freezes one LoRA per family and uses a Naive Bayes classifier on the request (updated without old requests) to pick which LoRA to use. FSM gets 65.95% EM over two task orders, 14.3 points over Progressive-LoRA and within 1.9 points of replay.
Critical strengths
-
The reactivation probe is a useful diagnostic experiment. The same checkpoint that gets 0 on all old T2 requests bounces back to ~70% when given the skill names, while a wrong name from the same family or random text doesn't, and tracking it over training shows the same order of failure for SFT, GRPO, and even SKILL0.
-
The routing vs. execution breakdown is informative. Comparing the oracle router, FSM's router and the offline router on the same frozen LoRAs isolates which families are limited by routing and which by the LoRA itself, and FSM is effective in getting close to replay without keeping old data while only running one LoRA per request.
Critical weaknesses
-
Diagnostic: the "correct skill address" is the gold tool names, which is a significant part of the answer graph, so "skills remain but access fails" might just mean the model still knows how to fill in a tool once it's told which tools to use. The paper admits the address reveals the gold node identity and calls it a diagnostic rather than a pure access probe, but there's no control that separates the two things (that confound). Also, the broader pattern that performance drops in LLM continual learning while the knowledge stays recoverable has been shown before in [1] and [2], which aren't discussed.
-
Novelty and scope: FSM is one frozen LoRA per task with a separate task classifier, which is of a similar design as past works like EPI, AdapterCL, and Expert Gate. The 5 families are very different topics (daily life APIs vs speech vs vision), so routing them from the request is fairly easy for the simple classifier, and with only TaskBench it's unclear the router holds when families overlap more. Additionally, routing to a single LoRA also can't handle a request that needs tools from two families, raising concern over whether the method generalizes to real world tasks that mix tool families.
[1] Spurious Forgetting in Continual Learning of Language Models
[2] Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Questions for authors
- Can you replace the tool names with arbitrary aliases, or give the collapsed checkpoint only the family name instead of the tool names? If recovery holds with just the family, it would be strong evidence for the split and also match the level FSM routes at.
- How were TaskBench requests whose tools span more than one family handled, and how does routing do when families share vocabulary?