acceptodds
Under review as a conference paper at ICLR 2027

ANCHOR: AMORTIZED KV-CACHE COMPACTION FOR PROCEDURAL SKILLS

Abstract

LLM agents increasingly rely on long procedural skills: prompts packaging tool schemas and invocation instructions that consume extensive context on every request. KV-cache compaction promises to reclaim this budget by emitting a compacted KV prefix for unseen skills in a single forward pass, yet existing methods perform poorly on procedural skills, producing missing or inaccurate tool calls. We identify a key training mechanism behind this, the anchoring effect: when knowledge-preservation (reconstruction) and instruction-following (tool-calling) objectives are trained sequentially, the second stage erases most of what the first stage stored, leading to sub-optimal final tool-call accuracy. We therefore present Amortized Neural KV Compaction with Held-On Reconstruction, an emitter network trained with on-policy distillation anchored by a low-weight reconstruction objective that keeps stored knowledge intact while improving tool-call fidelity. We also find that evaluating skill compaction is itself an open problem: on some existing agentic benchmarks, strong base models score nearly as well without the skill in context, leaving no headroom to measure what a compacted prefix preserves. We therefore propose DomainBench: 716 tasks from 100 procedural skills, answerable only with the skill in context and graded on executable tool calls at three nested granularities of tool name, argument keys, and argument values. At 4x compaction ANCHOR recovers 65.1% and 58.9% of full-context tool-use fidelity on two LLM backbones against 5.5–45.0% for four prior methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compaction, LLM, Procedural Skill

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_6rcZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ANCHOR, an amortized method for compressing procedural skill documents into KV caches for a frozen language model. It separates document reconstruction from instruction-following training and retains a reconstruction anchor during the latter stage. The paper also introduces DOMAINBENCH to evaluate tool-call fidelity after compression. Experiments show that retaining reconstruction improves the reported fidelity metrics.

Critical strengths
  • Clear distinction between skill compression and instruction following. The paper distinguishes preserving skill content from using that content to follow instructions. The two-stage design and reconstruction anchor allow the authors to examine how instruction-following training affects document fidelity. The ablations provide useful evidence that retaining reconstruction benefits downstream performance.

  • Useful evaluation perspective. The paper examines whether specific information needed for tool use survives skill compression, with separate metrics for tool selection, argument structure, and argument values. This breakdown provides a useful diagnostic perspective on procedural knowledge compression and helps identify which information different methods fail to preserve.

Critical weaknesses
  • Unclear handling of multiple valid solutions in gold supervision. Equation (9) applies token-level supervision to a single gold response, which Appendix B describes as coming from a model-generated simulated trajectory. The paper does not sufficiently explain whether training tasks have unique solutions or how other valid responses are handled. For agent tasks with multiple solutions, reproducing one reference trajectory is not equivalent to completing the task. The authors should clarify whether this objective unnecessarily penalizes valid alternatives and whether its benefits extend to tasks that do not require reproducing a particular reference response.

  • Call-F1 and the no-skill comparison do not sufficiently establish improvements in task completion. Appendix A counts a call as correct only when its tool name, argument keys, and normalized argument values match the reference. The same appendix states that the no-skill model cannot infer the domain-specific API signatures. In this setting, supplying tool definitions and calling information through a skill can predictably improve the matching score. The resulting gap therefore does not isolate the benefit of procedural knowledge from that of access to API information. More fundamentally, agreement with reference calls is not equivalent to task completion, and the paper does not sufficiently validate the relationship between the two. A comparison in which all conditions have access to tool definitions, together with corresponding task-completion results, would make the benchmark substantially more convincing.

Questions for authors
  1. Are the training tasks restricted to a single valid solution? If so, how is uniqueness established? If multiple solutions are allowed, how are valid alternatives handled by the gold supervision?
  2. For the gold-loss ablations, do the gains persist when valid alternative solutions are accepted and outputs are assessed by task completion? This would help clarify whether the benefits extend beyond closer agreement with the chosen reference.
  3. How much of the reported improvement remains against a schema-only baseline that receives the same tool definitions but no procedural instructions? This comparison would help separate the contribution of procedural knowledge from access to API information.
  4. Do improvements in Call-F1 correspond to higher task-completion rates? An independent audit of existing predictions should report where the two measures disagree and whether the main comparisons between methods still hold under a task-completion evaluation.
3 comments

e thank you for recognising the distinction between preserving a skill and following instructions with it, and the diagnostic value of separate tool, argument and value metrics.

W1 / Q1. Single gold trajectory despite multiple valid solutions.

Our tasks are API tool calls, and each training row has one gold answer. Only the gold term (Eq. 9) pulls the model toward that reference and the on-policy KL term uses no reference: it matches the student's output distribution on its own samples to that of the frozen model reading the full skill, so it rewards any solution the teacher would produce. Q2 tests whether the gold term helps beyond reproducing the reference.

Q2. Do the gold-loss gains hold under task completion?

Yes. We trained three stage-2 variants from the same stage-1 checkpoint (Qwen3-4B, r=4, 2000 steps, 3 seeds each): the full objective, one without the anchor and one without the gold term. Table R5 reports Call-F1 and task completion on DomainBench. Task completion is decided by two LLM judges at temperature 0: judge A is gemma-4-31B-it and judge B is Qwen3-32B. Each judge sees the skill, the request, the gold calls as one valid solution and the model's response, and is told to accept other valid solutions. Because judge A's model also generated the gold and both models are part of the generation pipeline, Q4 adds a third judge from another model family.

Table R5. Gold-term and anchor ablations under task completion.

Stage-2 (mean of 3 seeds)Call-F1Completion, judge A / B
Full objective0.3430.369 / 0.316
Without the anchor0.2530.246 / 0.283
Without the gold term0.2960.284 / 0.279

The gold term and the anchor term both matter a lot. Removing either of them has a negative impact on both Call-F1 and Completion.

W2 / Q3. Schema-only baseline

We added the two settings you asked for. (i) Schema-only: the frozen model receives only the skill's tool schemas (12.6% of the skill's tokens) and no procedural instructions. (ii) ANCHOR + schema: ANCHOR compacts the whole skill document exactly as before, and the same tool schemas are additionally given verbatim in the prompt. ANCHOR alone is the original setting, in which the model receives the skill only through the compacted KV-cache slots.

Table R6. Schema-only baseline (Qwen3-4B).

ConditionCall-F1Completion, judge A / B
Full skill0.7630.728 / 0.611
Schema-only baseline0.3290.321 / 0.347
ANCHOR0.4970.412/ 0.489
ANCHOR + schema0.5650.547 / 0.570

On its own, ANCHOR beats the schema-only baseline on Call-F1. On task completion the two are close under judge A but ANCHOR is better under judge B.

With the schema in the prompt, ANCHOR's compacted skill raises task completion under both judges over both arms alone.

W2 / Q4. Do Call-F1 gains correspond to task completion?

For single-call tasks, Call-F1 is equivalent to task completion: the task scores 1 only when its call matches the gold tool, argument keys and argument values. For multi-call tasks it gives partial credit (for example, when 2 of 3 gold calls are matched), and Tool-F1 and Schema-F1 can be seen as coarser partial completion. We choose this setting so we do not need to rely on LLM judge as an audit and introduce noise. Additionally, on other two public skill-use benchmarks we evaluated, task-completion is used as metrics on which ANCHOR also performs well. To further address your concern, we also did an experiment using three different LLM judges to score every prediction for task completion and they are also told to accept valid alternatives: A (gemma-4-31B-it), B (Qwen3-32B) and C (gpt-oss-120b, a model family that took no part in building the gold). The results show that they agree well: Cohen's κ is 0.876 (A–C), 0.733 (A–B) and 0.685 (B–C).

Table R7. Task completion under three judges (Qwen3-4B).

JudgeFull skillANCHOR r=4SKIMStillAM
A0.7280.4120.1580.0730.013
B0.6110.4890.2390.1030.066
C0.6590.4780.1560.0560.009

Under every judge, ANCHOR completes far more tasks than the other compaction methods. We also see observe that higher Call-F1 goes with higher completion.

Dear Reviewer 6rcZ, we are grateful for your review and the helpful suggestions. We have posted a detailed response addressing each of your concerns. We are happy to clarify anything that remains unclear before the discussion period closes.

I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_pAYqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an amortized emitter that compresses procedural-skill documents into reusable key–value prefixes for a frozen language model: it pools per-layer document hidden states into a length-adaptive number of slots, transforms them with learned residual linear heads, and constructs cache entries through the backbone's frozen normalization and key/value projections. Training first optimizes document reconstruction and then combines on-policy forward-KL distillation, gold-response cross-entropy, and a low-weight reconstruction anchor to mitigate the loss of schema information during instruction-following training. The paper also introduces DomainBench, comprising 716 human-validated tasks over 100 held-out skills, with nested Tool-, Schema-, and Call-F1 metrics. At nominal 4× compaction, ANCHOR achieves Call-F1 of 0.497 on Qwen3-4B and 0.487 on Gemma-4-E4B, versus full-context scores of 0.763 and 0.825, corresponding to 65.1% and 58.9% recovery of the full-context improvement over the no-skill baseline. Across additional benchmarks, ANCHOR leads the compared compactors on SOP-Bench and SWE-Skills-Bench, although it remains below the no-skill baseline on SWE-Skills-Bench and below Cartridges on LongBench v2.

Critical strengths
  • Comparisons initialized from the same reconstruction checkpoint, including instruction-following-only training and an L2 anti-drift alternative, connect preservation of schema identifiers to improved tool-call fidelity rather than relying solely on aggregate benchmark gains.

  • DomainBench combines held-out tool families, three-annotator review with reported agreement, a substantial full-context versus no-skill performance gap, and nested call-matching metrics that help distinguish tool-selection, argument-key, and argument-value errors.

  • The frozen-backbone, query-independent emitter supports reusable skill prefixes without per-document optimization, while experiments across two backbones, four benchmarks, multiple compaction ratios, and architectural ablations examine several dimensions of its utility.

Critical weaknesses

The comparison with the closest amortized baseline, Still, does not isolate the proposed emitter's benefit from training and cache-budget differences, because Still's reported pure-KL/reconstruction-mixed recipe and fixed 768-slot budget differ from ANCHOR's combined losses and length-adaptive budget.

DomainBench's unordered maximum-matching metric can assign perfect Call-F1 to the correct calls presented in an invalid dependency order, so the primary benchmark does not by itself establish the claimed preservation of executable procedural behavior.

  • Could you please re-score cached predictions on the dependency-bearing subset of the 280 multi-call tasks using annotated prerequisite edges and an argument-availability checker, allowing independent calls to be reordered but requiring dependent calls to follow their prerequisites; include deliberately swapped dependent-call sequences as negative controls and report dependency-valid task success, the fraction of perfect-Call-F1 predictions that fail these checks, and whether the method ranking changes.

The practical efficiency benefit remains unquantified because the paper reports compaction ratios but no measured compaction latency, peak memory, or decoding throughput, despite requiring a full document-encoding pass and an emission head with approximately nd2nd^2 weight parameters.

  • Could you profile the existing ANCHOR, Still, and full-context configurations on one Qwen3-4B setup using 50 held-out skills stratified by length and 1, 10, and 100 requests per skill, caching the uncompressed prefix in the full-context baseline; report encoding/compaction time, actual retained KV bytes, peak memory including emitter weights and temporary hidden states, time to first token, decoding throughput, and Call-F1, explicitly stating whether inactive emission slots are allocated or trimmed.
Questions for authors

Please see above.

1 comment

We thank the reviewer for valuing the same-checkpoint comparisons that tie schema preservation to tool-call fidelity, DomainBench's design, and the reusable frozen-backbone emitter.

W1. The Still comparison does not isolate the emitter from recipe and budget.

We share the point that we can not compare cleanly between methods with fixed length budget and the ones with adaptive length governed by compaction ratio so we use the fixed 768 length for Still to have a similar effective compaction ratio with ANCHOR on DomainBench; Still's recipe at 768 latents was selected by the hyperparameter sweep of Appendix C.2. As an additional diagnostic experiment we also tried to control either the budget or the training recipe

  • Budget: Using the ANCHOR checkpoint trained with compaction ratio r=4r=4, we fix the emission slot m=768m=768 so ANCHOR and STILL has the exactly same slot length. On DomainBench ANCHOR reaches 0.383 (0.497 length-adaptive) against 0.125 for Still.

  • Recipe: a Still-like emitter trained with ANCHOR's recipe did not converge in stage 1 (reconstruction cross-entropy 10.47 vs. 0.106), so it cannot separate architecture from recipe and we do not report it.

W2. Unordered matching can reward invalid dependency orders. & W3. Efficiency is unquantified.

Thank you for proposing these two experiments; we ran both as specified.

Dependency-aware scoring. We filtered 76 dependency-aware tasks from the full DomainBench and re-scored with prerequisite edges and an argument-availability check: independent calls may be reordered, a dependent call must follow the call that produces its input, and deliberately swapped gold sequences serve as negative controls. No perfect-Call-F1 prediction of any method fails the check. ANCHOR stays first among the compaction methods under both Call-F1 and dependency-valid success; the two measures swap only two pairs of baselines. One call per round on the 76 order-dependent tasks, ANCHOR solves 0.301 of them against 0.103 for Cartridges and at most 0.040 for SKIM, Still and AM (full context 0.437); see Table R1 in the response to dEWv, W1.

Efficiency. We extended the efficiency analysis with the configuration you specified: Qwen3-4B, 50 held-out skills stratified by length, 1, 10 and 100 requests per skill, and full context with its prefix cached. Table R4 in the response to dEWv, W4 reports build time, retained KV, memory after load and during the build (emitter weights included), TTFT, decoding throughput and per-request latency. The gain of every compaction method is mainly the retained KV, which matches its compaction ratio (102.6 MiB per skill for ANCHOR against 422.1 MiB for full context). ANCHOR keeps no separate copy of the base model as its context encoder: the resident base model encodes the skill, so after loading it holds only the 0.88 GiB emitter beyond the base model (8.48 vs. 7.55 GiB), and it builds a skill in 138 ms. Its build has the largest transient of the profiled methods (3.27 GiB on average), because the emitter computes all 4,096 emission slots before the retained cache keeps only the active ones. After building, ANCHOR's TTFT and decoding speed match full context's (30.8 vs. 30.3 ms, 41.0 vs. 41.9 tok/s). Cartridges trains a cache for each skill: 487 s per DomainBench skill, about 3,500× ANCHOR's build.

Reviewer_92vXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to compress long procedural skill documents while preserving the ability of large language models to make accurate tool calls. The authors find that training a model to first preserve document knowledge and then follow instructions can cause previously learned information to be forgotten. To address this issue, they propose ANCHOR, a KV-cache compression method that maintains a document reconstruction objective during instruction-following training to reduce information loss and improve tool-calling accuracy.

The authors also introduce DomainBench, a benchmark containing 716 tasks across 100 procedural skills, which evaluates tool-calling accuracy at three levels: tool names, argument keys, and argument values. Experiments compare ANCHOR against four existing compression methods on two language models, with additional evaluations on three public benchmarks. At a 4× compression ratio, ANCHOR achieves Call-F1 scores of 0.497 and 0.487 on DomainBench, outperforming all baselines, although its advantages are less consistent on the public benchmarks.

Critical strengths

The paper identifies an important problem: training models to follow instructions from compressed skill documents can cause previously preserved information to be forgotten. It proposes a reconstruction-based training objective to address this issue. Experiments show that this approach effectively preserves key information and significantly improves tool-calling accuracy, providing a useful insight for procedural skill compression.

Critical weaknesses

The paper's main performance gains rely heavily on a synthetic benchmark created by the authors, where skill documents follow relatively uniform structures that may not reflect the complexity and diversity of real-world scenarios. Furthermore, the paper lacks sufficient validation on real-world skill documents. As a result, the current experiments do not convincingly demonstrate that the method would provide similar benefits in practical applications, weakening the paper's central claims.

Questions for authors

The skill documents in DomainBench are generated by the same model using similar templates, while ANCHOR shows inconsistent performance across public benchmarks. How can the authors justify, based on the existing results, that the method's advantages generalize to diverse real-world skill documents rather than being limited to the synthetic benchmark?

2 comments

We thank you for recognising the forgetting problem, and that retaining reconstruction preserves the key information and improves tool-calling accuracy.

Main concern. Do the gains transfer beyond DomainBench?

DomainBench is motivated by the observation that existing real-world benchmarks often provide too little headroom to evaluate KV-cache compaction: the gap between the full-context reference and the no-context baseline is small. We therefore build tasks on tool schemas and interaction patterns from real-world setups rather than on purely random synthetic data: every skill is built around a tool family of the xLAM function-calling dataset [1]. Synthesis lets us control the quality and complexity of the tasks, so DomainBench is reasonably complex and realistic (example in Appendix E) and has a clear gap between the full-context reference and the no-context baseline, which leaves room to measure how much of the full-context performance a compacted KV cache retains. Still and SKIM, the baselines that need training, are trained on the same data as ANCHOR with a similar budget, and Cartridges trains one cache per skill, so the comparison is fair. The gains also transfer beyond DomainBench: the submission reports SOP-Bench, where ANCHOR is the best compactor on both backbones, SWE-Skills-Bench, where it degrades the agent least, and LongBench v2. To address your concern further, we add two public benchmarks with human-written documents, MedCalc-Bench and APIBench (TF Hub) and results show that ANCHOR also performs better than the baselines; please see the response to dEWv, W2 for more details.

[1] Liu, Zuxin, et al. "APIGen: Automated pipeline for generating verifiable and diverse function-calling datasets." arXiv preprint arXiv:2406.18518 (2024)

Reviewer_92vXOpenReview ↗

Offcial Comment by Reviewer 92vX

Thank you for the thorough rebuttal. The clarification of the SOP-Bench results and the additional experiments on benchmarks with human-written documents have largely addressed my concerns about generalization beyond DomainBench. I therefore raise my rating from 2 to 3.

Reviewer_dEWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ANCHOR, an amortized KV-cache compression method for procedural skill documents. Its key idea is to introduce a knowledge reconstruction anchor during instruction-following training to mitigate knowledge forgetting caused by behavioral distillation. The authors also introduce DOMAINBENCH, a benchmark for fine-grained evaluation of procedural knowledge preservation. Experiments across two model backbones and multiple benchmarks demonstrate that ANCHOR consistently outperforms the evaluated compression baselines, particularly in tool-calling fidelity under 4× compression.

Critical strengths

1. Meaningful research problem. The paper identifies an important tension between knowledge preservation and instruction following in procedural skill compression. The proposed reconstruction-anchored training strategy is intuitive, well-motivated, and supported by ablation studies.

2. Valuable evaluation framework. The authors introduce DOMAINBENCH, comprising 100 skills and 716 tasks, with three hierarchical metrics: Tool-F1, Schema-F1, and Call-F1. This provides a more fine-grained evaluation framework for procedural knowledge compression.

3. Promising empirical results. ANCHOR consistently outperforms four compression baselines across two model backbones. The paper also provides systematic ablations of its training objectives and architectural components, supporting the effectiveness of its key design choices.

Critical weaknesses

W1. The DOMAINBENCH evaluation metric does not fully capture sequential tool-calling correctness.

The benchmark adopts set-based matching for multi-call tasks without explicitly accounting for execution order or dependencies between tool calls. Consequently, predictions with incorrect execution orders may still receive high scores if their tools and arguments match. This limits the extent to which Call-F1 reflects actual procedural execution fidelity.

W2. Generalization to real-world procedural skills remains insufficiently demonstrated.

Both training data and DOMAINBENCH rely heavily on synthetically generated skill documents produced by Gemma-4-31B-it, with similar structured formats. Although tool families are held out during training, this does not fully establish generalization to real-world skills with diverse writing styles and distributions. Furthermore, ANCHOR underperforms the no-skill baseline on SWE-SKILLS-BENCH, raising concerns about its practical effectiveness beyond synthetic evaluations.

W3. The baseline comparison is not sufficiently comprehensive.

The evaluation covers four compression methods but excludes other relevant approaches, such as Activation Beacon and KV-Distill. This limits the strength of the claim that ANCHOR outperforms existing KV-cache and context compression methods more broadly.

W4. The practical efficiency claims lack sufficient system-level evidence.

Although the paper emphasizes amortized compression and KV-cache reduction, it does not adequately report actual compression latency, peak memory consumption, emitter parameter overhead, or end-to-end inference throughput. Consequently, the practical efficiency advantages over full-context inference remain unclear.

Questions for authors

Q1. For Gemma-4-E4B, whose sliding-window attention architecture supports KV prefixes in only 7 out of 42 layers, how exactly is the reported 4× compression ratio calculated? Does this ratio reflect the actual reduction in total KV-cache memory consumption?

Q2. Do the authors plan to release the complete DOMAINBENCH dataset, including skill documents, task annotations, tool simulators, and evaluation scripts, to facilitate reproducibility and future research?

4 comments

W1. Set-based Call-F1 ignores execution order.

We share your point. Single-turn Call-F1 is order-free by design. We filtered out 76 order-dependent multi tool-call tasks and added an order- and dependency-aware evaluation, whose gold trajectories we verified again with Claude Opus 5. Every model emits one call per round: a call that matches a remaining gold call receives that call's simulated tool result, a call that does not match uses up its round and receives no result, and the rollout ends when the model stops calling. A task is called successful when every gold call is matched. On Qwen3-4B, ANCHOR is the best compaction method on both measures (Table R1).

Table R1. One call per round (Qwen3-4B), 76 order-dependent tasks.

MethodCall-F1success rate
Full context0.6270.437
ANCHOR r=40.4190.301
Cartridges0.2470.103
SKIM0.0460.008
Still0.0930.040
AM0.0250.000

W2. Generalization beyond synthetic skills; SWE-Skills-Bench.

We acknowledge the similar structure of DomainBench and the training data. DomainBench's tools are not invented from scratch: each skill is built around a tool family of the xLAM function-calling dataset [1]. Our submission therefore also reports SOP-Bench, whose SOP skill files are expert-authored and stay outside the training data distribution. ANCHOR is the best compactor on both backbones by task success: 0.307 vs. 0.253 for Still on Qwen3-4B, and 0.271 vs. 0.221 for Cartridges on Gemma-4-E4B. To strengthen the advantage of ANCHOR, we also added two public benchmarks with human-written documents.

Table R2. Third-party benchmarks with human-written documents (accuracy).

BackboneBenchmarkNo doc.Full doc.ANCHOR r=4StillAMSKIMCartridges
Qwen3-4BMedCalc-Bench (1,100 tasks)0.3540.8220.6350.3270.1850.1150.694
Qwen3-4BAPIBench TF Hub, request's card (685)0.1270.8960.2830.1110.0000.0000.258
Qwen3-4BAPIBench TF Hub, domain's cards (685)0.1200.3010.2620.2530.0010.0000.239
Gemma-4-E4BMedCalc-Bench0.3670.8270.5220.0460.3420.2310.100
Gemma-4-E4BAPIBench TF Hub, request's card0.2860.9650.3690.2410.0790.0000.128
Gemma-4-E4BAPIBench TF Hub, domain's cards0.2950.5880.3610.2530.2190.0000.421

ANCHOR is the best in 4 out of 6 settings, often exceeding Cartridges, which is optimized for individual skill files.

All methods perform under no-skill baseline on SWE-Skills-Bench, which indicates that the task is extremely hard. Also here the full skill itself adds only 0.007 pass rate over no skill (Qwen3-4B), so the benchmark mainly measures how much a compacted skill degrades the agent. ANCHOR degrades it least on both backbones (−0.049 and −0.078, vs. −0.166 to −0.321 for the four baselines). Additionally, ANCHOR is the only compactor that never falls below no-doc in any of the 6 settings.

[1]Liu, Zuxin, et al. "APIGen: Automated pipeline for generating verifiable and diverse function-calling datasets." arXiv preprint arXiv:2406.18518 (2024).

AuthorsOpenReview ↗

Rebuttal-Part2

W3. Activation Beacon and KV-Distill are missing.

KV-Distill has no released code or checkpoints, and Still, which we compare against, reports outperforming it on RULER, HELMET and LongBench. We additionally trained Activation Beacon [1] on Qwen3-4B with its authors' default recipe. Evaluation on DomainBench shows that ANCHOR is ahead of the Activation Beacon with similar compression. We will add full results in the final manuscript.

Table R3. Activation Beacon vs. ANCHOR.

MethodEffective CompressionCall-F1
Beacon, r=4, window 2563.58×0.287
ANCHOR r=40.497
Beacon, r=8, window 2565.76×0.219
ANCHOR r=80.325

Activation Beacon also applies sliding window-based compression with the last shorter window untouched, resulting a slightly different compaction rate. For the two closest settings ANCHOR performs both better.

W4. No system-level efficiency evidence.

We extend the efficiency analysis of the submission with the system-level measurements requested: Qwen3-4B on one H200, bf16, sdpa, batch 1, on 50 DomainBench skills. Each skill is compacted once, and 1, 10 or 100 of its own requests then reuse the compacted state (full context reuses its cached skill prefix); per-request latency is (build + the TTFT of the first N requests) / N. Values are averaged over the 50 skills.

Table R4. System-level profile.

MethodKV / skillBuildMemory after load / build peak (mean over skills)TTFTDecodeLatency per request (N=1 / 10 / 100)
Full context422.1 MiB89 ms7.55 / 8.8 GiB30.3 ms41.9 tok/s121.5 / 39.5 / 31.1 ms
ANCHOR r=4102.6 MiB138 ms8.48 / 11.8 GiB30.8 ms41.0 tok/s170.5 / 44.8 / 32.2 ms
Still108.0 MiB285 ms7.72 / 9.3 GiB34.5 ms35.1 tok/s322.3 / 63.4 / 37.4 ms
SKIM108.4 MiB195 ms7.73 / 8.5 GiB57.5 ms21.3 tok/s255.2 / 77.5 / 59.6 ms
AM (4×)102.3 MiB4,678 ms7.55 / 10.3 GiB34.1 ms35.2 tok/s4,714.8 / 502.3 / 81.0 ms

The gain of every compaction method is mainly the retained KV, which matches its compaction ratio (102.6 MiB per skill for ANCHOR against 422.1 MiB for full context). ANCHOR keeps no separate copy of the base model as its context encoder: the resident base model encodes the skill, so after loading it holds only the 0.88 GiB emitter beyond the base model (8.48 vs. 7.55 GiB), and it builds a skill in 138 ms. Its build has the largest transient of the profiled methods (3.27 GiB on average), because the emitter computes all 4,096 emission slots before the retained cache keeps only the active ones. After building, ANCHOR's TTFT and decoding speed match full context's (30.8 vs. 30.3 ms, 41.0 vs. 41.9 tok/s). Cartridges trains a cache for each skill: 487 s per DomainBench skill, about 3,500× ANCHOR's build.

Q1. What does "4×" mean on Gemma-4-E4B?

It is the slot ratio m = ⌈L/4⌉ in the 4 full-attention layers that own a KV cache. Gemma-4-E4B has 7 full-attention and 35 sliding-window (window size=512) layers; 24 of the 42 own a KV cache, including 4 of the 7 full-attention layers. ANCHOR writes its prefix only into the full-attention layers and emits nothing for the sliding-window layers, the same for AM and Still. SKIM's soft tokens pass through every layer, and Cartridges trains the KV of every layer that owns a cache. Appendix D of the submission wrongly stated that the 7-of-42 restriction applies identically to every method.

Q2. Will DomainBench be released?

Yes. Here is an anonymous repository for the DomainBench data.

https://anonymous-hf.com/a/git2umxhhcc9/

[1] Zhang, Peitian, et al. "Long context compression with Activation Beacon." arXiv preprint arXiv:2401.03462 (2024).

Dear Reviewer dEWv, thanks again for the time and effort you put into the initial review. We have posted a detailed response to address your concerns. If you have follow-up questions or any remaining concerns, we would be glad to address them.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewer dEWv, thank you for your review. As the discussion period comes to an end, we want to briefly recap our response into a concise version as below (please find the detailed response in previously posted comments):

  1. Execution order: on 76 order-dependent tasks with one call per round, ANCHOR solves 0.301 of them against at most 0.103 for the baselines (Appendix G.1).
  2. Real-world skills: on two new benchmarks with human-written documents, MedCalc-Bench and APIBench, ANCHOR is the best compactor in 4 of 6 settings and the only one that never falls below the no-document baseline (Table 4); it is also the best compactor on SOP-Bench on both backbones.
  3. Baselines: ANCHOR outperforms Activation Beacon at comparable compression (Call-F1 0.497 vs. 0.287; Appendix G.3).
  4. Efficiency: ANCHOR builds a skill in 138 ms, keeps about 4× less KV than full context (102.6 vs. 422.1 MiB per skill), and matches its TTFT and decoding speed (Appendix G.5).

Q1 / Q2: we clarified the 4× ratio on Gemma-4-E4B (corrected in Appendix D) and released DomainBench.

We hope these results address your concerns, and please also let us know how could we further improve this paper. Thanks again!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their insightful and constructive reviews. Reviewers dEWv, 92vX and 6rcZ found the problem we identify important: the tension between knowledge preservation and instruction following, where training a compactor to follow instructions can erase skill knowledge it had preserved. They found the reconstruction anchor intuitive and well motivated. dEWv, pAYq and 6rcZ valued DomainBench as a careful evaluation, with held-out tool families, annotator review and fine-grained metrics that separate tool selection, argument structure and argument values.

The most common concern is whether the gains on DomainBench reflect real procedural skill use and carry over beyond it. We built DomainBench for fine-grained diagnosis with enough headroom, since public benchmarks often leave a small gap between the full-context reference and the no-context baseline. Its skills are built around tool families of the xLAM function-calling dataset [3] and follow realistic structures, and the baselines that need training are trained on the same data as ANCHOR, so the methods compare fairly. The gains also carry over to public benchmarks: on SOP-Bench, whose SOPs are expert-written, ANCHOR is the best compactor on both backbones.

We will add the following to the revision:

  1. Two more public benchmarks with human-written documents and real applications outside our training distribution: MedCalc-Bench [1] and Gorilla APIBench (TF Hub) [2]. They add to SOP-Bench, LongBench v2 and SWE-Skills-Bench, which the submission already reports.

  2. Additional analyses: an order- and dependency-aware evaluation with one call per round (dEWv W1, pAYq), a schema-only baseline and task completion under three LLM judges (6rcZ), Activation Beacon as a further baseline (dEWv W3), and an extended system-level efficiency profile (dEWv W4, pAYq).

  3. A new appendix section with every experiment requested in the reviews.

[1] Khandekar, Nikhil, et al. "Medcalc-bench: Evaluating large language models for medical calculations." Advances in Neural Information Processing Systems 37 (2024): 84730-84745.

[2] Patil, Shishir G., et al. "Gorilla: Large language model connected with massive apis." Advances in Neural Information Processing Systems 37 (2024): 126544-126565.

[3] Liu, Zuxin, et al. "APIGen: Automated pipeline for generating verifiable and diverse function-calling datasets." arXiv preprint arXiv:2406.18518 (2024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.