- Rating
- 2(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SkillSmith, a framework for synthesizing task-specific prefix parameters from existing parametric skills and textual metadata. It uses an augmented language model to jointly process source-task prefix K-V caches, task descriptions, and relational information, and generates a new prefix for a frozen downstream model. SkillSmith is trained end-to-end through the downstream task loss. The paper also introduces Composite-SNI, a synthetic task-composition dataset, and evaluates transfer to SNI and MMLU-ProX using retrieved source tasks. The results suggest that SkillSmith can leverage both textual and parametric information, provide useful initialization for subsequent prefix tuning, and transfer composition capabilities learned from synthetic tasks to natural benchmarks.
Critical strengths
-
An appealing perspective on combining textual and parametric skills. Jointly processing textual knowledge and existing skill parameters is an interesting direction for reusable agent capabilities. Prefix K-V caches provide a natural interface because both text-derived information and learned task prefixes can be represented in this space. The input ablations provide evidence that source parameters contribute information beyond the textual metadata alone.
-
Clear presentation and encouraging evidence for initialization and transfer. The paper is clearly written, and the experiments examine both known source-task relationships and the more realistic setting where source tasks must be retrieved. The strongest results show that generated prefixes can provide effective initialization for downstream adaptation. The transfer from Composite-SNI pretraining to SNI and MMLU-ProX is also interesting, suggesting that synthetic task compositions can help compensate for limited meta-training tasks.
Critical weaknesses
-
The evaluation does not sufficiently establish improvements in task-solving ability or practical utility. The reported results rely on target NLL and Elo ratings derived from NLL comparisons, rather than task-level accuracy or other standard outcome metrics. Lower NLL is meaningful, but does not by itself establish that the model solves more instances correctly; the gains could partly reflect improved probability assignment to reference outputs. Elo further summarizes how frequently a method wins across tasks without directly conveying the magnitude of improvement. Training with cross-entropy does not prevent reporting accuracy at evaluation time, particularly on MMLU-ProX.
Moreover, before target-task gradient updates, SkillSmith underperforms ICL on Composite-SNI, and the SNI-only variant also underperforms ICL. Although the bootstrapped MMLU-ProX results are encouraging, the overall evidence is stronger for prefix initialization than for consistently superior direct adaptation. Given the cost of constructing the skill library and meta-training SkillSmith, the practical advantage over ICL remains insufficiently established without task-level performance and efficiency comparisons. -
The empirical scope and positioning relative to closely related parameter-generation methods are limited. Although the task-module formulation conceptually accommodates different PEFT methods, all experiments use prefix tuning. This choice is well motivated by the K-V representation, but it leaves unclear whether the approach extends to widely used alternatives such as LoRA.
More importantly, the paper does not discuss or compare with relevant work on generating LoRA parameters from textual inputs, including Doc-to-LoRA, SHINE, and LatentSkill. These methods address related forms of mapping contexts or textual skills into model parameters. SkillSmith’s joint conditioning on existing skill weights and text is a potentially important distinction, but its contribution would be clearer with explicit positioning and a controlled comparison against a learned text-conditioned parameter generator under matched conditions.
Questions for authors
See weaknesses above.
2 comments
Thank you for the additional analyses and clarifications. However, my main concern remains unresolved: the results establish improvements in NLL, but do not demonstrate corresponding gains in task accuracy. While I appreciate the proposed framework, after reconsidering the empirical evidence, I do not find the current evaluation sufficient to support acceptance. I am therefore lowering my rating from 3 to 2 (weak reject). I encourage the authors to continue exploring this direction with stronger task-level evaluation.
Response to Reviewer 3xd8
Thank you for the thoughtful and constructive review. We are glad that jointly processing textual knowledge and existing skill parameters came across as an appealing direction for reusable agent capabilities, and that prefix K-V caches read as a natural interface. Below, we address your specific concerns.
Our intent behind SkillSmith is not to construct a skill library for the purpose of transfer. Rather, we assume that we are in a setting where the agent is creating parametric skills as a means of problem solving -- and then ask the question -- how can we compose these skills in an instruction steered way to improve transfer performance. From that standpoint we only constructed the skill library to mimic a setting that we believe may be ubiquitous in the future. In which case we believe, SkillSmith is a viable and competitive alternative to ICL and a productive way to squeeze extra leverage out of a skill library that has already been constructed.
Please see joint response.
Please see joint response
Please see our joint response on the choice of inner metric