- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose SKIM, an adaptive framework for compressing procedural skills into reusable soft tokens. It combines progressive training with offline token-budget selection to preserve workflow and tool-use behavior while reducing repeated context costs. Reported fixed-budget configurations retain 15% to 55% of the original token length.
Critical strengths
-
The paper addresses an interesting question of how to reduce the context overhead and inference costs caused by repeatedly loading lengthy procedural skills.
-
The three-stage training design provides a coherent progression from reconstructing skill content to supporting procedural questions and downstream tasks.
-
The framework supports independently compressed skills within a shared context, accommodating tasks that require multiple skills.
Critical weaknesses
-
SKIM builds on an existing compression architecture. However, it is unclear what new technical ideas its training strategy and budget selection introduce beyond adapting existing methods to procedural skills.
-
The selector chooses between only two compressed budgets and full text, while its advantage over simpler budget-allocation rules remains unclear.
-
The same model generates diagnostic questions, produces reference answers, and evaluates compressed answers. This self-evaluation may overlook the same procedural requirements across these steps, making a compressed representation appear faithful even when important instructions are missed.
-
At inference, all selected skills use the largest individual budget. One demanding skill can therefore increase the token allocation for every simpler skill in the same request.
-
The shared LoRA is trained with compressed soft tokens, so returning to ordinary skill text may not preserve the original model’s full-text performance.
-
Character edit distance measures textual change rather than procedural importance. A small edit to a condition, tool argument, and prohibition may require recalibration even when most of the skill remains unchanged.
Questions for authors
Please refer to the Critical Weaknesses.
4 comments
Response to Reviewer S9db (Part 2/2)
W3: Potential Blind Spots in Self-Evaluation
Thank you for raising this concern. First, we used target-model self-evaluation in our main experiments to keep the framework self-contained and ensure a fair comparison without privileged external models or human labels (Appendix H.2). Moreover, self-evaluation is conducted only during the offline exam as part of our method, completely separate from and independent of final downstream benchmark testing, and the downstream performance itself validates the soundness of our approach. Appendix H (Tables 11 and 12) also already evaluated part of your suggestion by replacing the judge with GPT-5.2 and two human annotators, confirming no systematic self-preference bias. To further examine question generation, we replaced both the question generator and the judge with a larger external model, Qwen3-32B, across the 162 skills and 2,083 downstream test instances:
| Question Gen. / Judge | {128..512, Full}: Acc. (%) / Tokens | {256, 512, Full}: Acc. (%) / Tokens |
|---|---|---|
| Qwen3-8B / Qwen3-8B (Self, Table 12) | 75.90 / 666 | 75.47 / 730 |
| Qwen3-8B / GPT-5.2 (Table 12) | 76.00 / 680 | 75.47 / 738 |
| Qwen3-32B / Qwen3-8B | 75.90 / 607 | 75.80 / 659 |
| Qwen3-32B / Qwen3-32B | 75.76 / 600 | 75.66 / 650 |
| Ref: Fixed K=512 / Full Text (Table 12) | 71.58 (536) / 76.86 (999) | Same |
Replacing both question generation and judging with a stronger external model leaves downstream accuracy virtually unchanged while slightly reducing token cost. This confirms that self-evaluation does not miss critical instructions, and as noted in Appendix H.2, substituting a stronger model or human verification for offline routing remains optional for deployment.
W4: Multi-Skill Budget Allocation
We agree with your observation. Since SKIM encodes and prefix-truncates each skill independently (Section 3.2: ), concatenating variable-length soft-token sequences such as has no architectural limitation. Taking the maximum budget in Section 3.5 was simply a shortcut for reusing single-budget runs during offline evaluation. We evaluated per-skill heterogeneous compressed budgets on the two multi-skill datasets (BigCodeBench with 2.76 skills/item and CHAMP with 1.71 skills/item) under the Table 9 dense Adaptive routing (Acc. (%) / Avg. Tokens):
| Model / Dataset | Shared Max Budget (Table 9) | Per-Skill Heterogeneous Budget |
|---|---|---|
| Qwen3-8B / BigCodeBench | 46.05 / 1567 | 46.49 / 1486 (-5% tok) |
| Qwen3-8B / CHAMP | 64.13 / 1303 | 62.33 / 1271 (-2% tok) |
| Phi-4 / BigCodeBench | 50.61 / 2012 | 50.09 / 1827 (-9% tok) |
| Phi-4 / CHAMP | 60.99 / 1161 | 60.99 / 1113 (-4% tok) |
Per-skill heterogeneous allocation further reduces token usage while maintaining comparable accuracy (no paired difference is statistically significant). Since the other three benchmarks are single-skill (Table 5) and the extra token savings here are modest, our overall conclusions remain unchanged. We have added this analysis to Appendix R.3 (Table 26).
W5: LoRA Handling During Full-Text Fallback
As noted in Section 3.5, the base target LLM weights remain frozen, and Stage 3 trains a modular LoRA adapter that engines such as vLLM can toggle per request. LoRA is activated only when SKIM soft tokens are present in the input. When falling back to Full Text (just as when no skill is needed), the adapter is deactivated and inference runs on the unmodified base model, preserving original Full Text performance. We have made this explicit in Section 3.5 in revision.
W6: Character Edit Distance and Recalibration
We respectfully clarify two design details already stated in Appendix I:
- Soft tokens are always re-encoded after every edit (~0.5 s): Appendix I states that after any edit, even changing a single parameter, condition, or prohibition, a compressor forward pass regenerates the soft-token artifact, so modified content is always encoded rather than ignored below 30%.
- The 30% rule of thumb only concerns whether to rerun budget tier selection (~110 s): Moreover, our controlled edit study in Appendix I explicitly instructed GPT-5.2 to make semantically non-equivalent modifications to workflows, libraries, or APIs, and even under such substantive edits, the optimal tier rarely changed below 30% edit distance (Table 13). That said, we agree that when an edit substantially alters branching complexity or tool schemas, rerunning offline calibration is advisable.
Thank you again for your detailed feedback. We hope these experiments and clarifications resolve your concerns, and we are happy to discuss further.
I thank the authors for the rebuttal. My main concerns have been largely addressed, and I have raised my score.
Thank you for taking the time to read our rebuttal and for raising your score. We sincerely appreciate your constructive feedback, which has helped us strengthen the revised manuscript.
Response to Reviewer S9db (Part 1/2)
We thank the reviewer for the thorough review and for recognizing the coherence of our three-stage training pipeline and experimental scope. Below we address each point in detail.
W1: Technical Contributions
As noted in Section 3.2 (citing DRIFT and Liu & Qiu, 2025), the dual-model Compressor + Projector backbone builds on prior soft compression architectures, and we do not claim the backbone itself as our contribution. Our technical contributions lie in three aspects:
W2: Comparison with Simple Budget Allocation Rules
Thanks for the comment. First, our selector is not restricted to two compressed budgets. Via prefix truncation, untrained intermediate tiers
{128, 256, 384, 512, Full}can be included without retraining, reducing average tokens from 1,303 to 978 on Qwen3-8B and from 1,055 to 923 on Phi-4 at comparable accuracy (Appendix E.2 and E.3, Figure 7 and Table 9).Second, to address your question directly, we compared SKIM-Adaptive against three label-free allocation rules that keep the exact per-dataset tier counts chosen by Adaptive and replace only the ranking criterion (higher-scoring skills receive larger budgets): ranking by full-text token length, ranking by structural complexity (total regex counts of numbered steps, code blocks, conditionals, and tool calls), and matched-proportion random allocation. Macro-averaged results (Acc. % / Avg. Tokens) across all five datasets under the Table 9 candidate set
{128, 256, 384, 512, Full}at τ=0.9 are:Long or superficially complex skills often contain verbose explanations or extended examples that compress well, yet static heuristics assign all of them to Full Text and drastically inflate token usage. Conversely, short but logically dense theorems or rules often need higher resolution, yet static heuristics assign them small budgets. Thus, Adaptive consistently outperforms these static rules at equal or lower token cost.
In addition, Appendix H.2 (Table 12) has specifically studied the routing problem under different exam judges (target-model self-judgment, GPT-5.2, and two independent human annotators), showing that all exam-driven adaptive configurations consistently outperform corresponding fixed-budget baselines.