acceptodds
Under review as a conference paper at ICLR 2027

SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge

Abstract

Agentic systems driven by large language models (LLMs) regularly feature two key mechanisms to autonomously solve complex problems: synthesizing text-based knowledge and procedures from past experiences and building parametric (weight-space) skill libraries for recurring sub-goals. To date, research has largely treated these as orthogonal pursuits: either organizing textual knowledge through composition and reflection, or consolidating parametric skills via weight-space merging. Consequently, the seamless integration of text and model weights for targeted performance improvements remains largely unexplored. This work bridges this modality gap by treating model weights as an additional modality that an LLM can natively reason over. We instantiate parametric learning via prefix-tuning and augment an LLM to ingest both prefix weights and rich textual data which capture relationships to a target capability. Our augmented LLM, which we call SkillSmith, synthesizes these inputs to perform instruction-steered parametric synthesis, directly outputting new prefix weights that manifest the target skill. We demonstrate that our approach significantly outperforms both text-only and weight-space-only baselines, unlocking performance gains that are out of reach for uni-modal (text-only or weight-only) adaptations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, prefix tuning, kv caches

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3xd8OpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillSmith, a framework for synthesizing task-specific prefix parameters from existing parametric skills and textual metadata. It uses an augmented language model to jointly process source-task prefix K-V caches, task descriptions, and relational information, and generates a new prefix for a frozen downstream model. SkillSmith is trained end-to-end through the downstream task loss. The paper also introduces Composite-SNI, a synthetic task-composition dataset, and evaluates transfer to SNI and MMLU-ProX using retrieved source tasks. The results suggest that SkillSmith can leverage both textual and parametric information, provide useful initialization for subsequent prefix tuning, and transfer composition capabilities learned from synthetic tasks to natural benchmarks.

Critical strengths
  1. An appealing perspective on combining textual and parametric skills. Jointly processing textual knowledge and existing skill parameters is an interesting direction for reusable agent capabilities. Prefix K-V caches provide a natural interface because both text-derived information and learned task prefixes can be represented in this space. The input ablations provide evidence that source parameters contribute information beyond the textual metadata alone.

  2. Clear presentation and encouraging evidence for initialization and transfer. The paper is clearly written, and the experiments examine both known source-task relationships and the more realistic setting where source tasks must be retrieved. The strongest results show that generated prefixes can provide effective initialization for downstream adaptation. The transfer from Composite-SNI pretraining to SNI and MMLU-ProX is also interesting, suggesting that synthetic task compositions can help compensate for limited meta-training tasks.

Critical weaknesses
  1. The evaluation does not sufficiently establish improvements in task-solving ability or practical utility. The reported results rely on target NLL and Elo ratings derived from NLL comparisons, rather than task-level accuracy or other standard outcome metrics. Lower NLL is meaningful, but does not by itself establish that the model solves more instances correctly; the gains could partly reflect improved probability assignment to reference outputs. Elo further summarizes how frequently a method wins across tasks without directly conveying the magnitude of improvement. Training with cross-entropy does not prevent reporting accuracy at evaluation time, particularly on MMLU-ProX.
    Moreover, before target-task gradient updates, SkillSmith underperforms ICL on Composite-SNI, and the SNI-only variant also underperforms ICL. Although the bootstrapped MMLU-ProX results are encouraging, the overall evidence is stronger for prefix initialization than for consistently superior direct adaptation. Given the cost of constructing the skill library and meta-training SkillSmith, the practical advantage over ICL remains insufficiently established without task-level performance and efficiency comparisons.

  2. The empirical scope and positioning relative to closely related parameter-generation methods are limited. Although the task-module formulation conceptually accommodates different PEFT methods, all experiments use prefix tuning. This choice is well motivated by the K-V representation, but it leaves unclear whether the approach extends to widely used alternatives such as LoRA.
    More importantly, the paper does not discuss or compare with relevant work on generating LoRA parameters from textual inputs, including Doc-to-LoRA, SHINE, and LatentSkill. These methods address related forms of mapping contexts or textual skills into model parameters. SkillSmith’s joint conditioning on existing skill weights and text is a potentially important distinction, but its contribution would be clearer with explicit positioning and a controlled comparison against a learned text-conditioned parameter generator under matched conditions.

Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3xd8

Thank you for the thoughtful and constructive review. We are glad that jointly processing textual knowledge and existing skill parameters came across as an appealing direction for reusable agent capabilities, and that prefix K-V caches read as a natural interface. Below, we address your specific concerns.

Given the cost of constructing the skill library and meta-training SkillSmith, the practical advantage over ICL remains insufficiently established without task-level performance and efficiency comparisons.

Our intent behind SkillSmith is not to construct a skill library for the purpose of transfer. Rather, we assume that we are in a setting where the agent is creating parametric skills as a means of problem solving -- and then ask the question -- how can we compose these skills in an instruction steered way to improve transfer performance. From that standpoint we only constructed the skill library to mimic a setting that we believe may be ubiquitous in the future. In which case we believe, SkillSmith is a viable and competitive alternative to ICL and a productive way to squeeze extra leverage out of a skill library that has already been constructed.

The reported results rely on target NLL and Elo ratings derived from NLL comparisons, rather than task-level accuracy or other standard outcome metrics. … Elo further summarizes how frequently a method wins across tasks without directly conveying the magnitude of improvement.

Please see joint response.

Although the task-module formulation conceptually accommodates different PEFT methods, all experiments use prefix tuning. … it leaves unclear whether the approach extends to widely used alternatives such as LoRA

Please see joint response

Training with cross-entropy does not prevent reporting accuracy at evaluation time, particularly on MMLU-ProX

Please see our joint response on the choice of inner metric

Thank you for the additional analyses and clarifications. However, my main concern remains unresolved: the results establish improvements in NLL, but do not demonstrate corresponding gains in task accuracy. While I appreciate the proposed framework, after reconsidering the empirical evidence, I do not find the current evaluation sufficient to support acceptance. I am therefore lowering my rating from 3 to 2 (weak reject). I encourage the authors to continue exploring this direction with stronger task-level evaluation.

Reviewer_X2wyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SkillSmith, an augmented Gemma-3 4B model that ingests source-task prefix KV-caches together with textual metadata (task descriptions, combination rationales, target-task exemplars) and directly generates new prefix weights for a target task. It is meta-trained end-to-end with a cross-entropy loss through the frozen downstream model. To study this, the authors construct Composite-SNI (~21K synthetic tasks composed from SNI task pairs via Gemini 2.5 Pro, ~17K meta-train tasks), and additionally evaluate on SNI and MMLU-ProX, where ground-truth source mappings are replaced by a retrieval + LLM-selection pipeline. Baselines cover weight-space merging (LERP, Concat, SVD variants, SPoT-style transfer), k-shot ICL, and direct prefix tuning. The main claims are that SkillSmith outperforms uni-modal baselines zero-shot and provides a superior initialization for downstream prefix fine-tuning.

Critical strengths
  1. The framing is genuinely interesting: treating weights as an input modality and letting an LLM perform instruction-steered parametric synthesis is a compelling direction for agentic systems, and end-to-end meta-training through the frozen downstream model is a clean instantiation. The idea deserves a stronger empirical treatment than it receives here — that is what makes the submission worth engaging with at all.

  2. The appendices are unusually complete: full prompts for dataset construction and selection, per-task raw NLL tables (App. L), and the Elo computation code (App. D). This allowed me to verify what was actually measured — and to find results in the authors' own tables that undercut the main-text narrative (W1).

Critical weaknesses

W1. The baselines cannot support the headline claim, and the authors' own numbers undercut it.

  • Every weight-space baseline is prefix-cache arithmetic (LERP / Concat / SVD / SPoT-style transfer). No LoRA or LoRA-composition baseline (LoraHub is cited as motivation but never run), no full or LoRA SFT on target-task data, no RL-based adaptation. And since SkillSmith itself only outputs 32-token prefix caches, the whole comparison stays inside this narrow sandbox; the claim of "definitively outperforms purely uni-modal baselines" (Sec. 3.1) is not established.

  • By the authors' own raw numbers (App. L, Table 2), plain k-shot ICL beats zero-shot SkillSmith on 11 of the 15 CSNI tasks. The advantage only appears after downstream fine-tuning — the actual contribution is a better initialization for prefix tuning, considerably narrower than the framing.

  • On SNI after fine-tuning, the win rate among the top methods converges to 0.5 (Sec. 5.2.1): on one of the three benchmarks there is no measurable advantage at all.

  • The "zero-shot" label is misleading: SkillSmith's input includes target-task few-shot exemplars (combination text, Sec. 3.2), which the weight-merging baselines it is compared against cannot consume.

W2. The evaluation is too small and too noisy to conclude anything, and the figures are below the standard of the venue.

  • Meta-evaluation uses 15 (CSNI), 10 (SNI), and 6 (MMLU-ProX) tasks. Elo ratings are computed from win rates over these handfuls of tasks with no confidence intervals, no seed variance, and no significance testing; an Elo gap over 6 tasks is not meaningful, and the win-rate construction discards effect sizes entirely.

  • The sole metric is NLL. MMLU-ProX is a multiple-choice benchmark, yet accuracy is never reported (conceded in Sec. 7); NLL on answer strings is a weak proxy for task success.

  • The main-result figures are not up to standard: Figs. 3-5 are default matplotlib bar charts with illegible rotated labels and undefined variant names; Fig. 6 is a radar chart with no numbers whose caption ("NLLs scaled to (0,1). ↓ area is better.") explains nothing; Fig. 1's in-box text is too small to read. Fig. 10 additionally truncates the y-axis at 1100, so the 12-point Elo gap between Retrieval+LLM Selection (1530) and Random+LLM Selection (1518) — which the authors themselves describe as "only slightly better" (App. I.1) — fills most of the plotting area.

Minor concerns:

  • Scope: composition is only ever over N=2 source tasks, one 4B backbone, prefix-tuning only; the motivating Twi-legal example of Sec. 1 is never tested.

  • The amortization argument (Sec. 7) is unquantified — no comparison of total compute (hundreds of source KV caches + meta-training a 4B model) against direct prefix tuning per task.

  • Reproducibility: dataset generation, combination text, and pair selection all depend on Gemini 2.5 Pro; no code or data release is mentioned. Several NLL entries of exactly 0.0000 (e.g., tasks 330736 and 625640 in Table 1, and task 625640 in the fine-tuned SkillSmith column of Table 3) look like numerical artifacts.

Questions for authors
  1. Will you add the baselines readers will ask for — LoRA composition (e.g., LoraHub), LoRA/full SFT on target-task data, and at least one RL-based method — under matched adaptation capacity?

  2. Can you report accuracy (not NLL) on MMLU-ProX, and provide uncertainty estimates (multiple seeds, confidence intervals, significance tests) for all Elo comparisons, given only 6-15 evaluation tasks?

  3. In the "zero-shot" setting, SkillSmith sees target-task exemplars in the combination text while weight-merging baselines see no target text at all. How do you justify this as a fair comparison?

  4. Does the approach scale beyond N=2 source tasks, and beyond prefix caches on a single 4B backbone? (Clarification, not a request for new work.)

3 comments
AuthorsOpenReview ↗

Response to Reviewer X2wy

Thank you for the detailed review, and for checking our claims against the raw tables in App. L and the Elo code in App. D. We are glad the framing came across as genuinely interesting (weights as an input modality, with an LLM performing instruction-steered parametric synthesis), and that the complete appendices were useful.

Points shared with other reviewers are in the joint response, so here we add only what is specific to your review.

And since SkillSmith itself only outputs 32-token prefix caches, the whole comparison stays inside this narrow sandbox

The output length is not fixed at 32. Our SkillSmith checkpoint was trained with variable output lengths of up to 128 but we at the time, did not have resources to evaluate the full swath of prefixes from 32 -> 128. We have now evaluated that same checkpoint at four lengths on the 15 CSNI tasks:

Output length0-step NLLFine-tuned NLL
32 (paper)1.1430.451
641.1340.443
1281.1180.456
256 (twice the longest trained length)1.1600.470

At every length, fine-tuned SkillSmith beats each of the eight fine-tuned baselines on at least 13/15 tasks (p ≤ 0.013). The baselines were run at length 32 only, so this shows that our conclusions do not depend on SkillSmith's output length; it does not show how the baselines behave at other lengths.

"Definitively outperforms purely uni-modal baselines"

Please note that this is research question posed in Sec. 3.1, not claim of a result. We make measured claims in our introduction by stating: our SkillSmith on the target task yields representations that either outperform or are competitive with all other methods studied, including the direct training of a prefix-cache initialized from in-context examples.

By the authors' own raw numbers (App. L, Table 2), plain k-shot ICL beats zero-shot SkillSmith on 11 of the 15 CSNI tasks. The advantage only appears after downstream fine-tuning — the actual contribution is a better initialization for prefix tuning, considerably narrower than the framing.

We agree that initialization for fine-tuning is the most robust result, and our introduction and results section do not make any claims otherwise. Please note that we are not only contributing the results -- but also this novel way of thinking about parameter generation and composition.

For your claim of narrower than the framing: note that the ICL lead on CSNI is not statistically significant (p = 0.33) and is specific to that benchmark: on MMLU-ProX, 0-step SkillSmith beats ICL on 6/6 tasks (mean NLL 1.08 vs. 3.12, p = 0.03). Both results are in the joint response ("Elo as outer evaluation metric").

We agree that initialization for fine-tuning is the most robust result, and the abstract and introduction will be modified to be more explicit about this.

The "zero-shot" label is misleading: SkillSmith's input includes target-task few-shot exemplars (combination text, Sec. 3.2), which the weight-merging baselines it is compared against cannot consume.

We will rename the setting to "0-step" and state at first use that the target description and a few exemplars are inputs. The comparison with weight-merging baselines is not an equal-information one, and we will say so.

Every weight-space baseline is prefix-cache arithmetic (LERP / Concat / SVD / SPoT-style transfer). No LoRA or LoRA-composition baseline (LoraHub is cited as motivation but never run), no full or LoRA SFT on target-task data, no RL-based adaptation.

Please see the joint response ("Comparison to other text to PEFT generation methods") for why the paper is scoped to prefix K-V caches at matched capacity, and not LoRA or full fine-tuning.

Meta-evaluation uses 15 (CSNI), 10 (SNI), and 6 (MMLU-ProX) tasks. Elo ratings are computed from win rates over these handfuls of tasks with no confidence intervals, no seed variance, and no significance testing; an Elo gap over 6 tasks is not meaningful …

Please see the joint response ("Size of Evaluation Set")

Scope: composition is only ever over N=2 source tasks, one 4B backbone, prefix-tuning only; the motivating Twi-legal example of Sec. 1 is never tested.

Please see joint response ("Scaling beyond N = 2 source tasks")

The anthor response addressed my concerns and I have reased my score.

I am grateful for the comment from the author and have decided to leave the score unchanged.

Reviewer_DYhwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SkillSmith treats prefix-tuning KV caches as an input modality an LLM can read. An augmented Gemma-3-4B "coprocessor" takes source-task bundles as input. Each bundle is a trained prefix cache, projected by an MLP adapter, together with that task's text metadata. The model also gets "combination text" describing how the sources relate to the target task. It then emits a new 32-token prefix cache for a frozen downstream model, and is meta-trained end to end with cross-entropy through that frozen model.

The authors also build Composite-SNI (~21K Gemini-generated tasks, each composing two SNI tasks) so that the ground-truth source tasks are known. For SNI and MMLU-ProX, where they are not, sources come from an embedding-retrieval plus Gemini selection pipeline. Results are reported as Elo scores computed from held-out NLL.

Critical strengths
  1. New, interesting framing: an LLM acting as a hyper-network over text and weights together, steered by instructions. Using prefix caches is a sensible choice because text and weights then share the same KV space.

  2. Strong initialization: SkillSmith-generated prefixes work well as a starting point for fine-tuning, especially on the harder, data-sparse MMLU-ProX.

Critical weaknesses
  1. Weak evaluation metric: The paper's results use NLL only, aggregated into Elo. Elo hides effect sizes and gives no confidence intervals. No accuracy is reported, even for MMLU-ProX, which is multiple-choice, so it is hard to judge how much the gains matter in practice.

  2. The evaluation set is small: It uses only 15 CSNI, 10 SNI and 6 MMLU-ProX held-out tasks. The paper also fixes N = 2 source tasks, uses one backbone (Gemma-3-4B) and one cache length (32).

  3. Mixed gains: ICL is competitive in the zero-shot setting, and on SNI the top fine-tuned methods are statistically tied (win rate approximately 0.5). The clear win comes mostly from MMLU-ProX with 6 tasks.

  4. Thin data validation: CSNI quality was checked on only 10–30 samples.

Questions for authors
  1. How do results change with N > 2 sources, or with distractor sources?

  2. What happens if the combination text is generated without access to the target task?

1 comment
AuthorsOpenReview ↗

Response to Reviewer DYhw

Thank you for the careful review. We are glad the framing came across as new and interesting (an LLM acting as a hyper-network over text and weights together, steered by instructions), that prefix caches read as a sensible choice because text and weights then share the same KV space.

Below, we address your concerns about our work :

Thin data validation: CSNI quality was checked on only 10–30 samples.

Those 10–30 samples were used only to design and calibrate our automatic filter, not as the validation itself. Every retained task passed four automatic stages (dataset-construction appendix): quality-score and length (≥16 examples) cutoffs, two rounds of pairwise LLM ranking on clarity and diversity, and a per-task correctness check that narrows ~90K candidates to the final ~21K tasks. Downstream transfer also supports data quality: Composite-SNI pretraining lowers 0-step NLL on SNI (1.93 → 1.22) and improves fine-tuned NLL on the unrelated MMLU-ProX in every run (1.038 → 1.010 for N = 2; 1.033 → 0.983 for variable N).

Weak evaluation metric: The paper's results use NLL only, aggregated into Elo. Elo hides effect sizes and gives no confidence intervals. No accuracy is reported, even for MMLU-ProX, which is multiple-choice, so it is hard to judge how much the gains matter in practice.

Please see joint response.

The evaluation set is small: It uses only 15 CSNI, 10 SNI and 6 MMLU-ProX held-out tasks. The paper also fixes N = 2 source tasks, uses one backbone (Gemma-3-4B) and one cache length (32).

Please see joint response.

Mixed gains: ICL is competitive in the zero-shot setting, and on SNI the top fine-tuned methods are statistically tied (win rate approximately 0.5). The clear win comes mostly from MMLU-ProX with 6 tasks.

Please see joint response -- we perform significance testing to show that our results are statistically significant our method wins majority of the time, though not in all cases.

How do results change with N > 2 sources, or with distractor sources?

Please see joint response

Comments on the paper

2 comments
AuthorsOpenReview ↗

Responding to Joint Concerns

We would like to thank all reviewers for their helpful comments. We appreciate that all reviewers found our framing intellectually interesting with promising results. Below, we address some cross cutting concerns

Elo as outer evaluation metric

We agree that Elo on its own hides magnitudes (that's why we comprehensively report raw results in the appendix). But Elo is an effective approach for comparing many methods across many tasks, especially since the magnitudes of the NLLs can vary across tasks. Below, we present tabular results with tests that show that the gains from SkillSmith are significant.

0-step (before target fine-tuning)

Benchmark (tasks)Baseline comparedSkillSmith wins (Wilcoxon p)
Composite-SNI (15)Best 0-step prefix (ICL-init)15/15 (p < 0.001)
Composite-SNI (15)k-shot ICL4/15 (p = 0.33, n.s.)
MMLU-ProX (6)Best 0-step prefix (LERP)5/6 (p = 0.06, n.s.)
MMLU-ProX (6)k-shot ICL6/6 (p = 0.03)
SNI (10)Best 0-step prefix (ICL-init)10/10 (p = 0.002)
SNI (10)k-shot ICL6/10 (p = 0.43, n.s.)

Fine-tuned (after target fine-tuning)

Benchmark (tasks)Baseline comparedSkillSmith wins (Wilcoxon p)
Composite-SNI (15)Direct prefix tuning13/15 (p = 0.004)
Composite-SNI (15)Best baseline (ICL-init + train)14/15 (p < 0.001)
MMLU-ProX (6)Direct prefix tuning6/6 (p = 0.03)
MMLU-ProX (6)Best baseline (Concat + train)6/6 (p = 0.03)
SNI (10)Direct prefix tuning10/10 (p = 0.002)
SNI (10)Best baseline (SVD-Seq + train)7/10 (p = 0.32, n.s.)

For each benchmark, we form the per-task NLL differences di=NLLSkillSmith,i−NLLBaseline,id_i=NLL_{SkillSmith,i} - NLL_{Baseline,i} rank their magnitudes ∣di∣|d_i| (dropping exact zeros and averaging ties), sum the ranks of positive differences (W+W_{+}), and compute the exact two-sided p-value by enumerating all 2^n sign assignments under the symmetric null (H0H_{0}: each difference is equally likely to be ±).

NLL as inner evaluation metric

We reported NLL because it is defined for every task in the three benchmarks, which mix classification with free-form generation. Specifically for MMLU-Pro-X, optimizing for the true metric (accuracy) directly would require reinforcement learning which we did not have the computational resources for. Re reporting accuracy even if we trained with NLL, note that the limited per-task dataset size meant that whilst we could establish correlation between train NLL and validation NLL, we could not establish correlation between train NLL and validation accuracy for any of the methods. Thus our choice in sticking to NLL since optimizing for accuracy during training was infeasible for us.

Size of Evaluation Set

Reviewers X2wy and DYhw raise the question of evaluation set size. We contend that 3 benchmarks and > 5 tasks per-benchmark, comparing ~ 9 different methods and their fine-tuned counterparts is quite a comprehensively sized evaluation benchmark.

Please take into consideration that each evaluation required non-trivial resources, and given the quite exhaustive coverage of our evaluation, we would like to respectfully push back on this claim.

Comparison to other text to PEFT generation methods

Reviewers X2wy and 3xd8 raise concerns regarding comparisons to other parameter efficient fine-tuning methods like LoRA, and other text-to-parameter methods that involve transforming text to LoRA parameters.

We chose prefix K-V caches deliberately: text maps into the same K-V space through a forward pass, so text and skill weights share one representation, which is what makes composing them tractable. LoRA has no such shared space, and extending SkillSmith to it is an open research problem we aim to explore in later work; we will state this scope in the abstract and limitations and restrict our claims to it. Within that scope all comparisons are at matched capacity (a 32-position prefix on the same frozen model), which LoRA or full fine-tuning would not be.

Doc-to-LoRA, SHINE and LatentSkill map text alone to a LoRA adapter in one forward pass. SkillSmith instead takes skills that already exist as trained parameters, plus text, and learns to compose them, so the two are complementary. We will add this positioning to the related work.

AuthorsOpenReview ↗

Continued Joint Response

Scaling beyond N = 2 source tasks (Reviewers DYhw, X2wy)

We trained a single variable-N model on a mixture of N ∈ {2, 4, 6} top-retrieved sources (6 source slots with unused slots masked; nested top-N sets, each with its own rationale) and evaluated fine-tuned target performance at N = 2, 4, 6 s over two training seeds (plus a second seed for fixed N = 2).

Model (N at eval)CSNI-pretrained MMLU-ProX (FT NLL)MMLU-ProX only (FT NLL)
Fixed N = 2 (N = 2)1.010 ± 0.0201.038 ± 0.010
Variable N (N = 2)0.983 ± 0.0051.033 ± 0.021
Variable N (N = 4)0.983 ± 0.0091.034 ± 0.020
Variable N (N = 6)0.982 ± 0.0091.037 ± 0.023

As can be seen from the results, SkillSmith Scales beyond N = 2 and is robust to lower-ranked / weakly related sources .

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.