- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to reuse independently computed KV caches without losing too much answer quality. Recomputing the full context can recover quality, but also reduces the benefit of caching. Attuner keeps the cached keys and values unchanged and trains small adapters on the query projections. A full-prefill model provides the training targets, but is not needed at inference time.
Critical strengths
1. The problem is interesting, and the method directly addresses it.
I find the main question useful: does recovering quality require recomputing the cached representations, or can the model learn to read the existing caches better? Attuner tests the second option without changing the caches. This makes the contribution more than another way to speed up a particular task. The oracle experiment also shows that quality can be recovered while keeping the artifact KV fixed. It motivates the method, although it does not establish a unique cause of the quality loss.
2. The main results support a useful quality and latency trade-off.
The evaluation covers two model sizes and different types of cached content. Relative to PIC, the average in-domain scores improve by 5.13 and 7.77 points, while time to first token remains 1.79x and 2.45x faster than full prefill. A shared adapter also improves performance on held-out tasks. The Full-adaptation control is useful: the much smaller gain under Full suggests that the adapter is mainly helping with independent-cache reading rather than simply improving the base model. Attuner is not best on every metric, but the trade-off it offers is meaningful.
Critical weaknesses
The robustness of the trained reader to irrelevant artifacts is still unclear.
The main evaluation directly supplies the annotated relevant artifacts, without additional distractors. The paper has a distractor experiment, but it compares Full and PIC rather than the trained Attuner. The current results therefore show that Attuner reads a relevant set of materials better; they do not yet show whether it can maintain this advantage when irrelevant materials are mixed in.
This matters because identifying which cached content to use is part of a reusable cache-reading method. Adding the trained adapter to the existing distractor experiment would clarify the scope of the result.
Questions for authors
Could the authors evaluate the same trained adapter in the existing distractor setting? In particular, does its gain over PIC remain as more irrelevant artifacts are added, and how does the gap to Full change?
3 comments
Thank you for testing the original adapter with distractors. It still beats PIC, but its gap to full prefill grows from about 1 to nearly 8 pass@1 points as more skills are added. I’m keeping my rating at 3 because identifying which cached material to use is an important part of reading the cache.
Thank you for reviewing the additional results. We would like to clarify that the original adapter was trained only on relevant artifacts, without distractors, so some degradation under this distribution shift is expected, although it continues to outperform PIC at every tested level.
To examine adaptation to this setting, we additionally trained adapters with distractors. As shown in Table 1 PIC + new 4K adapter in Response to Reviewer Uz9u, these improve performance at all four skill counts, achieving results close to or above Full with 4–12 skills and narrowing the remaining gap at 16 skills.
We sincerely thank you for your thoughtful feedback and for helping us strengthen the evaluation and clarify the scope of our work.
Response to Reviewer Uz9u
We sincerely thank the reviewer for recognizing the value of our research question and ATTUNER’s quality–latency trade-off. We appreciate your thoughtful feedback and hope the additional evaluation below addresses your concern about robustness to distractors.
Q1: Could the authors evaluate the same trained adapter in the existing distractor setting? In particular, does its gain over PIC remain as more irrelevant artifacts are added, and how does the gap to Full change?
A1: As discussed in Section 3.1, we distinguish artifact discovery from evidence utilization. We view evidence utilization as a starting point for robust cache reuse. At the same time, artifact discovery is important in practical retrieval settings, where retrieved materials are not accompanied by ground-truth relevance labels. Our focus here is on repairing the reading of cached evidence as a foundation on which broader discovery mechanisms can build.
To address the reviewer's question, we evaluate the agent-relevant skill-use setting built on BigCodeBench, where the model receives both relevant skills and distractors. We use the same Qwen3-4B adapter as in the paper, without additional training. Each condition retains all relevant skills and adds distractors to reach a target total of 4, 8, 12, or 16 skills. The paper adapter was trained with relevant artifacts only. To assess whether training with distractors improves recovery, we additionally train a separate adapter from the base model for each condition, using 4,000 synthetic examples built from training-side skills disjoint from evaluation.
As shown in Table 1, ATTUNER consistently improves over PIC when relevant skills are mixed with distractors, although its gap to Full widens as distraction increases. Training under the corresponding conditions further improves performance at all four skill counts, supporting effective cache reading when retrieved materials include irrelevant content.
Table 1. BigCodeBench pass@1 (%).
We also evaluate inference efficiency. Since PIC primarily accelerates prompt processing, we measure wall-clock TTFT, including cache lookup, CPU-to-GPU transfer, KV assembly, online prefill, and first-token selection. The paper adapter retains essentially the same TTFT as PIC. As more skills are added, a larger portion of the prompt can be reused from cache, increasing the speedup over Full.
Table 2. BigCodeBench warm-cache efficiency. Token counts are per-request means (rounded); TTFT is the median in ms.
From the experiments above, we can see that ATTUNER improves cache reading in this mixed-context setting while retaining near-PIC warm-cache latency. Our contribution is to establish evidence-utilization repair through query-side adaptation as a foundation for effective reuse of independently cached artifacts. Robust artifact discovery may additionally benefit from document-level supervision or selection mechanisms, particularly when evidence is interdependent across artifacts. Please see our response to Reviewer GkER, A2, for the attention diagnostics and discussion of this direction.
We thank the reviewer for highlighting this important point, and we have clarified the paper's scope and the directions for more robust artifact discovery in the revised manuscript (Line 1386, page 26).