- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ANCHOR, an amortized method for compressing procedural skill documents into KV caches for a frozen language model. It separates document reconstruction from instruction-following training and retains a reconstruction anchor during the latter stage. The paper also introduces DOMAINBENCH to evaluate tool-call fidelity after compression. Experiments show that retaining reconstruction improves the reported fidelity metrics.
Critical strengths
-
Clear distinction between skill compression and instruction following. The paper distinguishes preserving skill content from using that content to follow instructions. The two-stage design and reconstruction anchor allow the authors to examine how instruction-following training affects document fidelity. The ablations provide useful evidence that retaining reconstruction benefits downstream performance.
-
Useful evaluation perspective. The paper examines whether specific information needed for tool use survives skill compression, with separate metrics for tool selection, argument structure, and argument values. This breakdown provides a useful diagnostic perspective on procedural knowledge compression and helps identify which information different methods fail to preserve.
Critical weaknesses
-
Unclear handling of multiple valid solutions in gold supervision. Equation (9) applies token-level supervision to a single gold response, which Appendix B describes as coming from a model-generated simulated trajectory. The paper does not sufficiently explain whether training tasks have unique solutions or how other valid responses are handled. For agent tasks with multiple solutions, reproducing one reference trajectory is not equivalent to completing the task. The authors should clarify whether this objective unnecessarily penalizes valid alternatives and whether its benefits extend to tasks that do not require reproducing a particular reference response.
-
Call-F1 and the no-skill comparison do not sufficiently establish improvements in task completion. Appendix A counts a call as correct only when its tool name, argument keys, and normalized argument values match the reference. The same appendix states that the no-skill model cannot infer the domain-specific API signatures. In this setting, supplying tool definitions and calling information through a skill can predictably improve the matching score. The resulting gap therefore does not isolate the benefit of procedural knowledge from that of access to API information. More fundamentally, agreement with reference calls is not equivalent to task completion, and the paper does not sufficiently validate the relationship between the two. A comparison in which all conditions have access to tool definitions, together with corresponding task-completion results, would make the benchmark substantially more convincing.
Questions for authors
- Are the training tasks restricted to a single valid solution? If so, how is uniqueness established? If multiple solutions are allowed, how are valid alternatives handled by the gold supervision?
- For the gold-loss ablations, do the gains persist when valid alternative solutions are accepted and outputs are assessed by task completion? This would help clarify whether the benefits extend beyond closer agreement with the chosen reference.
- How much of the reported improvement remains against a schema-only baseline that receives the same tool definitions but no procedural instructions? This comparison would help separate the contribution of procedural knowledge from access to API information.
- Do improvements in Call-F1 correspond to higher task-completion rates? An independent audit of existing predictions should report where the two measures disagree and whether the main comparisons between methods still hold under a task-completion evaluation.
3 comments
Dear Reviewer 6rcZ, we are grateful for your review and the helpful suggestions. We have posted a detailed response addressing each of your concerns. We are happy to clarify anything that remains unclear before the discussion period closes.
I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
e thank you for recognising the distinction between preserving a skill and following instructions with it, and the diagnostic value of separate tool, argument and value metrics.
W1 / Q1. Single gold trajectory despite multiple valid solutions.
Our tasks are API tool calls, and each training row has one gold answer. Only the gold term (Eq. 9) pulls the model toward that reference and the on-policy KL term uses no reference: it matches the student's output distribution on its own samples to that of the frozen model reading the full skill, so it rewards any solution the teacher would produce. Q2 tests whether the gold term helps beyond reproducing the reference.
Q2. Do the gold-loss gains hold under task completion?
Yes. We trained three stage-2 variants from the same stage-1 checkpoint (Qwen3-4B, r=4, 2000 steps, 3 seeds each): the full objective, one without the anchor and one without the gold term. Table R5 reports Call-F1 and task completion on DomainBench. Task completion is decided by two LLM judges at temperature 0: judge A is gemma-4-31B-it and judge B is Qwen3-32B. Each judge sees the skill, the request, the gold calls as one valid solution and the model's response, and is told to accept other valid solutions. Because judge A's model also generated the gold and both models are part of the generation pipeline, Q4 adds a third judge from another model family.
Table R5. Gold-term and anchor ablations under task completion.
The gold term and the anchor term both matter a lot. Removing either of them has a negative impact on both Call-F1 and Completion.
W2 / Q3. Schema-only baseline
We added the two settings you asked for. (i) Schema-only: the frozen model receives only the skill's tool schemas (12.6% of the skill's tokens) and no procedural instructions. (ii) ANCHOR + schema: ANCHOR compacts the whole skill document exactly as before, and the same tool schemas are additionally given verbatim in the prompt. ANCHOR alone is the original setting, in which the model receives the skill only through the compacted KV-cache slots.
Table R6. Schema-only baseline (Qwen3-4B).
On its own, ANCHOR beats the schema-only baseline on Call-F1. On task completion the two are close under judge A but ANCHOR is better under judge B.
With the schema in the prompt, ANCHOR's compacted skill raises task completion under both judges over both arms alone.
W2 / Q4. Do Call-F1 gains correspond to task completion?
For single-call tasks, Call-F1 is equivalent to task completion: the task scores 1 only when its call matches the gold tool, argument keys and argument values. For multi-call tasks it gives partial credit (for example, when 2 of 3 gold calls are matched), and Tool-F1 and Schema-F1 can be seen as coarser partial completion. We choose this setting so we do not need to rely on LLM judge as an audit and introduce noise. Additionally, on other two public skill-use benchmarks we evaluated, task-completion is used as metrics on which ANCHOR also performs well. To further address your concern, we also did an experiment using three different LLM judges to score every prediction for task completion and they are also told to accept valid alternatives: A (gemma-4-31B-it), B (Qwen3-32B) and C (gpt-oss-120b, a model family that took no part in building the gold). The results show that they agree well: Cohen's κ is 0.876 (A–C), 0.733 (A–B) and 0.685 (B–C).
Table R7. Task completion under three judges (Qwen3-4B).
Under every judge, ANCHOR completes far more tasks than the other compaction methods. We also see observe that higher Call-F1 goes with higher completion.