- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents MSI, a framework for turning skill knowledge into parameters for an agentic model. To learn a skill, the framework generates synthetic tasks that require it; then a teacher model generates demonstrations associated with those skills. Finally, these demonstrations, together with an anchor set that is used to preserve the agent's other behaviors, are used to train the model to learn the skill. The paper evaluates this approach on 5 different benchmarks and 4 different models. Results show that the method performs better compared to Direct Skill Injection and random skill adaptor.
Critical strengths
-
The paper is generally well-written and easy to follow.
-
The problem that the paper studies, which is to turn skill knowledge into parametric representations in the model, is interesting and important.
Critical weaknesses
The two major weaknesses that I identify are as follows:
-
Lack of novelty compared to existing methods. The main method the paper proposes is to train LoRA as a parametric representation of the skill. This approach has essentially been discussed in at least two previous papers [1] [2]. Notably, these papers were public at least three months prior to the ICLR deadline. However, the paper entirely does not mention these two papers. The paper needs to clearly position itself against these two prior works and justify its novelty.
-
Lack of sufficient baseline comparisons. Currently, the baselines that the method chooses are weak and naive baselines, including full skill retrieval, random skill adaptor, oracle skill with demos. However, the paper does not compare itself with other methods, including the training-based skill internalization methods [1] [2], and also test-time skill selection/adaptation methods. Without these, it's unclear whether the improvement that the method gets is solid.
[1] Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents. https://arxiv.org/abs/2606.16769 [2] LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents https://arxiv.org/abs/2606.06087
Questions for authors
-
The explanation of the 'complementarity of skill text and MSI' is not fully justified. The author does not present clear insights into why this happens or what exactly the 'orthogonal contribution' of skill text and MSI is. The explanation of the occurrence of sub-optimality of MSI + skill text is also not fully convincing, as the authors did not explain why this happens.
-
The cost analysis is insufficient. For example, what is the cost of the whole pipeline for internalizing a single skill? What about for each benchmark, considering there are hundreds of skills?
-
(This can be 'critical weakness 3', but since we are suggested to write '1 or 2' critical weaknesses, I write it here) The comparison strategy of Table 3 is weird. Table 3 compares the performance of MSI and baseline models under the same (imperfect) skill selector. However, the (possibly imperfect) selector design should only affect MSI, not baselines.
3 comments
Thank you for your well-structured and concise response. The following concerns remain:
-
While directly comparing the two Skill-to-LORA baselines is not required per ICLR guidelines, the novelty of this paper compared to previous methods is incremental to me.
-
There are very few baseline comparisons with other existing methods that include skill and/or training. The baseline comparison is insufficient and fails to establish that the method achieves better performance compared to previous methods.
-
The cost of training is high (I believe the authors should also compare it to other training-based methods), and it scales linearly with the number of skills.
-
For Q3, the question is not about 'why it needs a selector', but 'why the baseline does not use a golden selector'. A non-perfect selector might be a weakness of your method, but it should not be that of a baseline.
In summary, I think the novelty of this paper compared to existing works is incremental and not well justified, and the experiments are insufficient to establish the methods' advantage compared to existing methods. Considering these, I decide to maintain my score.
Thank you for clarifying Q3. The text baseline receives the gold document in Table 1, while MSI retrieves its own skill in Table 3. We place the results side by side below. All entries are micro accuracy (%) on the same 4,037 questions.
| Backbone | Gold text (Table 1) | Gold text + 2 demos (Table 1) | Retrieved text (Table 3) | Retrieved MSI (Table 3) |
|---|---|---|---|---|
| Qwen3.5-2B | 57.2 | 59.3 | 54.5 | 59.3 |
| Llama-3.2-3B | 34.2 | 33.3 | 33.7 | 48.6 |
| Phi-3.5-mini | 36.5 | 40.7 | 34.7 | 43.3 |
| Apertus-8B | 41.8 | 40.6 | 40.3 | 46.4 |
Retrieved MSI exceeds gold-text prompting on all four backbones. Adding two demonstrations to the gold prompt yields a tie on Qwen at the reported precision, while MSI leads on the other three.
Table 3 addresses deployment when the skill library is available but the question does not identify the correct skill. Text injection must then select a document, just as MSI must select an adapter. Using the same retrieved skill lets us compare how well each method applies it.
Baselines. We agree that comparison with additional prior methods would strengthen the evaluation. Our response to Reviewer X8ar (Q2) reports a Shared LoRA comparison, and our earlier reply to you reports results for the released LatentSkill compiler. We will scope our claims to the comparisons evaluated.
Cost. We agree that MSI’s training cost grows approximately linearly with the number of skills at a fixed training budget per skill. This cost is paid once per skill and backbone, while the resulting adapter can be reused across queries and save inference input tokens by omitting the document. In our Qwen experiment, training a Shared LoRA on the same per-skill positive-example budget used a similar amount of training GPU time. As the library grows, MSI can add a skill by training a new adapter, whereas adding it to Shared LoRA requires updating weights used by existing skills.
Thank you for the review and for pointing us to the related work.
W1: Novelty relative to Skill-to-LoRA and LatentSkill.
W2: Baseline comparisons.
We applied the released LatentSkill compiler (Qwen3-8B) to all 408 skill documents, with gold skill assignment. Accuracy (%):
Q1: Why skill text and MSI are complementary.
Q2: Cost.
Per skill, we use 200 training demonstrations, 60 anchors, and 80 validation questions. Single-GPU training hours:
Q3: Selector use in Table 3.