- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes SOFTSKILL, which initializes a short block of virtual-token embeddings from a natural-language skill and optimizes that block with next-token prediction while keeping the backbone frozen. Experiments show gains on several QA and agent benchmarks, and controlled ALFWorld interventions suggest that the learned continuous update—not merely a change in nearest vocabulary tokens—drives much of the measured improvement.
Critical strengths
SOFTSKILL is an interesting, simple, and effective approach: it turns an existing textual skill into a compact trainable context while keeping the backbone frozen. The experiments are unusually thorough, spanning QA and agentic tasks, initialization and placement ablations, robustness tests, and mechanistic interventions. Overall, this is a strong and carefully evaluated piece of work.
Critical weaknesses
The choice of a fixed 32-token skill is not sufficiently motivated. Although the paper compares several lengths, performance sometimes declines as the skill grows, but the authors do not explain why. More importantly, initialization uses the first tokens of the skill text; for a very long skill, those first 32 tokens may contain little task-relevant information. Would SOFTSKILL still work with a different 32-token segment—or even a randomly initialized vector block? The paper should test these alternatives and provide a more general, principled method for selecting the skill representation and its length.
Questions for authors
See weakness part.
hank you for the positive assessment and for the precise questions on segment choice, random initialization, and block length. We ran new three-seed experiments for each question on all three QA tasks; the results also yield a principled rule for choosing the representation and its length. All rows below use the skill-slot insertion and report means over three seeds.
Q1. Would a different 32-token segment work?
Yes: on every task, the tested segments perform similarly.
The four segments stay within two points of each other on each task, so the opening segment is not uniquely important.
Q2. Would a random vector block work?
It depends on how much room the task leaves: on LiveMath a norm-matched Gaussian block fails, while on SearchQA and DocVQA every tested start trains to a similar score.
LiveMath is the most informative task here: its no-skill baseline is 22.4, and its seed skill is a real 208-token document. There, a Gaussian block reaches only 26.1, whereas random vocabulary tokens reach 64.0; across 12 initializations, real-token starts land at 60–69 and averaged or Gaussian starts at 26–52. DocVQA leaves little room (the no-skill baseline is already 86.9), so all 12 initializations land within 88.0–90.1. Skill content adds to the effect of geometry where the skill carries task knowledge: on LiveMath, and by a smaller margin on DocVQA, the right skill beats a generic instruction and the wrong task's skill. On SearchQA the seed skill is a 22-token placeholder ("No learned rules yet"), so it carries no content, and all 12 initializations land within 72.9–76.8.
Why does performance sometimes drop as the block grows?
With three seeds there is no systematic decline beyond 32 tokens on any task; the drop in the original single-seed results does not reproduce.
LiveMath rises up to 16–32 tokens and then plateaus; SearchQA and DocVQA are flat across the whole range. This points to a useful distinction: textual skill length determines how much content can be expressed, whereas soft-skill length mainly sets trainable capacity. A block longer than its source text is initialized by repeating the text (SearchQA's 22-token skill becomes about 12 copies at 256 tokens), yet it trains to the same accuracy, so the training signal supplies task-specific behavior beyond the initialization text. Soft-skill length is therefore best viewed as a capacity hyperparameter, like adapter rank, and about 16–32 tokens suffice in the evaluated settings.
A principled selection method
Two rules follow from these results. Representation: initialize from actual token embeddings (any tested skill segment works), not from averaged embeddings or random dense vectors. Length: treat it as a capacity hyperparameter and choose the shortest length within one standard deviation of the best validation score. On our validation results this rule selects 32 tokens for both LiveMath and SearchQA. The revision replaces the single-number justification with the length curve and these two rules.
We hope the new results address the reviewer's concerns and have convinced them to recommend acceptance with confidence.