acceptodds
Under review as a conference paper at ICLR 2027

SemPIC: Learning Semantic Position-Independent KV Caches

Abstract

Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders. Prefix caching cannot exploit this reuse, while position-independent caching (PIC) remains unreliable because independently compiled KV states lack the future context in which they will be consumed. Our diagnostics show that a learned boundary-conditioned baseline sharply reduces attention deviation near reusable-block boundaries but leaves interior and task-level residuals, motivating adaptation of the document representation itself. We present SemPIC, which trains a LoRA-enabled Writer to compile native per-layer document KVs through behavioral distillation while retaining the pretrained decoder as an unchanged Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface and cache-hit decoding path. We further introduce KV Gradient Checkpointing, which reduces peak training memory without severing gradients through cached KVs. Across three models and four tasks, SemPIC raises mean micro-F1 over KV Packet from 0.53 to 0.60, approaching Full Recompute at 0.62.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language models, KV cache reuse, context reuse, long-context inference, efficient LLM inference, position-independent caching, knowledge distillation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_9sZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies position-independent KV cache reuse, where independently precomputed document caches are reused across different request contexts and document compositions without online document recomputation. The authors first analyze the boundary-based adaptation in KV Packet and identify residual interior attention deviations and task-level performance gaps despite substantial improvements near document boundaries. Motivated by this observation, the author propose SemPIC, which trains a LoRA-enabled document Writer through behavioral distillation to produce native per-layer KVs consumed by an unchanged, frozen Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface during online inference. The paper also introduces KV Gradient Checkpointing to reduce training memory while preserving gradient flow through the cached KVs. Across three evaluated models and four tasks, SemPIC improves average token micro-F1 from 0.530.53 for KV Packet to 0.600.60, approaching Full Recompute at 0.620.62. Additional experiments examine cross-domain transfer, quality–efficiency tradeoffs, and residual attention patterns.

Critical strengths
  1. The paper is clear and easy to follow. It distinguishes positional alignment from contextual mismatch and motivates the method through an analysis of attention deviations near document boundaries and within document interiors. Figures 1 and 3 effectively illustrate the relationship to prior approaches and the training and inference workflows of the Writer–Reader design.

  2. Restricting adaptation to the offline Writer while preserving the Reader and native KV interface is sensible and practically useful. The gains across multiple models and tasks, Joint variant, and transfer analysis provide useful evidence. The checkpointing experiments also demonstrate memory reductions and transitions from OOM to feasible execution under the tested configurations.

Critical weaknesses
  1. The methodological novelty appears incremental. The main change is to extend KV Packet's boundary-parameter adaptation to LoRA adaptation of the document Writer while retaining full-context behavioral distillation. The broader idea of learning composable caches with a frozen backbone is also related to work such as C$^2$KV. Direct comparisons with these closely related methods and controls for adaptation capacity and training budget are missing, making it difficult to isolate the contribution of the adaptation locus itself.

  2. Evidence for generalization and practical benefits remains limited. The transfer averages include in-domain results, and substantial negative transfer occurs: in Table 2, for Qwen3-8B trained on NIAH, Biography performance drops from 0.530.53 with KV Packet to 0.250.25 with SemPIC; in Table 1, Llama MuSiQue also decreases from 0.370.37 to 0.280.28. Input-length distributions and cross-request reuse statistics are also unreported, limiting assessment of the motivating deployment scenarios. Furthermore, although checkpointing demonstrates memory savings and OOM-to-feasible transitions, its time overhead depends on configuration. Without memory and throughput scaling results or maximum trainable context lengths, the practical resource–time tradeoff remains insufficiently characterized.

Questions for authors
  1. Could the authors clarify SemPIC's distinctive contribution relative to KV Packet and C$^2$KV, separating the effects of adaptation locus, capacity, and training budget? As a scope clarification, please also specify the architectural requirements of the native KV interface and whether hybrids such as Qwen3.5, combining full attention with Gated DeltaNet, would require a new state-composition mechanism.

  2. Do the transfer gains persist when in-domain results are excluded? Could the authors provide input-length statistics, and clarify what the current experiments establish about changing document compositions and repeated cross-request reuse, versus what remains an unvalidated application scenario?

  3. For KV Gradient Checkpointing, could the authors report memory usage, training throughput, and maximum trainable context length across input lengths, using fixed hardware, matched forward computations, and clearly specified batch settings? Please also clarify whether the times in Table 3 correspond to complete training runs, so that resource benefits can be assessed alongside their training-time cost.

5 comments

We sincerely thank the reviewer for recognizing the strengths of our work. We address the concerns and questions below.

W1 & Q1.1. Adaptation Mechanism and C²KV Comparison

We acknowledge that SemPIC adopts KV Packet's KL distillation objective, but emphasize the differences in adaptation mechanism and cache representation.

Relative to KV Packet, our boundary–interior analysis reveals residual interior attention discrepancies under boundary-only adaptation. SemPIC instead learns a document-wide KV mapping through a LoRA-adapted Writer, improving native per-token KVs without introducing additional boundary tokens or modifying the frozen Reader.

Relative to C²KV. C²KV, a concurrent work, learns compressed, composable caches through a sidecar Extractor and compression tokens. In contrast, SemPIC directly adapts the document Writer to produce full, native token-level KVs, targeting position-independent reuse fidelity rather than context compression.

To further validate this distinction, we implemented the C²KV architecture under the same Qwen3-8B/MuSiQue training protocol as SemPIC, using identical training data, teacher continuations, KL objective, and training budget, while changing the cache-generation architecture.

MethodCompression ratioF1
C²KV1×0.3067
C²KV2×0.2816
SemPIC (rank 8, 3 seeds)1×0.3679 ± 0.0224

SemPIC achieves higher F1 than C²KV at a 1:1 cache-token ratio, supporting the effectiveness of directly adapting native document KVs under matched training conditions.

Additionally, KV Gradient Checkpointing enables memory-efficient differentiable KV compilation while preserving gradient flow through all cached layers.

W1 & Q1.2. Capacity- and Budget-Matched Comparison

On Qwen3-8B/MuSiQue, rank-1 SemPIC and KV Packet with 117+117 boundary tokens have identical 958,464 trainable parameters. Both use 2,048 training requests, 128 validation requests, identical Full Recompute teacher continuations and KL loss, batch size 16, and five epochs (640 updates). SemPIC uses LR 5×10−45\times10^{-4}; Packet's LR 10−310^{-3} is validation-selected.

MethodF1Document KV (MiB/request)
Packet (117+117 tokens)0.31351002.0
SemPIC (rank 1)0.3723343.9

At matched capacity and training budget, SemPIC achieves higher observed F1 with 65.7% less document KV storage. The paired 95% CI for the F1 difference is [−0.0245, 0.1412].

Q1.3. Native KV Interface and Hybrid Architectures

The current implementation requires causal full-attention layers with conventional per-token, per-layer K/V states and positional relocation. Hybrid architectures such as Qwen3.5 combine full attention with Gated DeltaNet, whose recurrent states cannot be composed through ordinary KV concatenation. Extending SemPIC to such models would require an additional state-composition mechanism, which is an interesting future direction.

W2 & Q2.1. Out-of-Domain Transfer

We recomputed the transfer averages from Table 2, excluding each single-domain Writer's in-domain target and the mixture-trained rows. Each model contributes 12 cross-domain source–target combinations.

ModelKV PacketSemPIC
Llama-3.1-8B0.410.48
Qwen3-4B0.370.37
Qwen3-8B0.360.38
Average0.380.41

SemPIC retains a +0.03 average F1 advantage after excluding in-domain results.

NIAH-to-Biography transfer. Regarding the originally reported NIAH-to-Biography transfer gap, the KV Packet checkpoint achieving 0.53 on Biography had unusually poor in-domain NIAH F1 (0.38). We hypothesize that this checkpoint learned ineffective NIAH-specific boundary adaptations, introducing less task-specific interference on Biography. Across three retraining seeds, KV Packet achieves 0.6446 ± 0.0617 on NIAH but only 0.3022 ± 0.0156 on Biography, compared with SemPIC's 0.7095 ± 0.0120 and 0.2905 ± 0.0280, respectively. The original apparent cross-domain advantage therefore does not persist across retraining. We will update the corresponding results.

W2 & Q2.2. Input-Length Statistics

We additionally report statistics for 100 Qwen3-8B test requests per task, excluding generated answers.

TaskMedian request tokensMax tokensMedian documents/requestMedian tokens/document
Biography435467580
HotpotQA1,4062,66410123
MuSiQue2,443.53,82720104
NIAH7,93211,1328895.5

W2 & Q2.3. Document Composition and Cross-Request Reuse

Document composition and cache reuse. On Qwen3-8B/MuSiQue, we reuse fixed document caches while changing document order or removing distractors. Across three Writer seeds, the original/reversed/random orders achieve mean F1 of 0.3679/0.3613/0.3535. Retaining all supporting evidence while reducing the context to 10/15 paragraphs yields 0.4679/0.4290. All retained document tokens are reused, with zero online Writer calls.

Repeated cross-request reuse. We additionally evaluate the Qwen3-8B mixture-trained Writer from Table 2 on the unseen 2WikiQA benchmark without task-specific adaptation. We organize 100 questions into 50 pairs, each sharing 10 independently compiled document caches across two different queries. SemPIC achieves 0.4761 F1, compared with 0.4005 for KV Packet and 0.2426 for No Recompute.

These experiments demonstrate cache reuse across varying document compositions and multiple requests, while providing further evidence of cross-task generalization.

W2 & Q3. KV Gradient Checkpointing: Memory and Training Cost

Controlled memory and throughput measurements. We evaluated Qwen3-8B on a single A100 80GB GPU, using a BF16 base model, rank-8 LoRA, optimizer batch size 8, and forward batch size 1. Both configurations use identical inputs, losses, and batch settings. Each LL-token request contains L−32L-32 document tokens and 32 query tokens, followed by 32 teacher-forced answer tokens. Measurements are averaged over three synchronized optimizer steps after one warm-up step.

LengthPeak memory, off/on (GiB)Step time, off/on (s)Throughput, off/on (examples/s)
2K30.99 / 17.526.85 / 15.711.17 / 0.51
4K46.63 / 19.6812.90 / 16.440.62 / 0.49
8KOOM / 23.99— / 32.69— / 0.24
16KOOM / 32.61— / 72.20— / 0.11

At 4K, checkpointing reduces peak memory by 57.8%, with a 27.4% increase in step time. It also extends the longest successfully tested training sequence from 4K to 16K. At FBS=8, checkpointing enables 2K/4K training, whereas all tested lengths encounter OOM without checkpointing.

Table 3 training times. The reported times correspond to complete five-epoch Writer training runs. For Biography training at FBS=8, checkpointing reduces peak memory from 75.8 to 33.2 GiB, while increasing total training time only from 23 to 25 minutes. The larger overhead at FBS=1 (34 to 115 minutes) is primarily attributable to Biography's short sequences (slightly over 400 tokens per request), which limit GPU utilization during recomputation.

We will incorporate these additional results and clarifications into the revised paper.

Thank you for the additional experiments and clarifications. They address some of my questions. The clarification on hybrid architecture models is also helpful, though support for these models remains future work.

I still find the methodological contribution of SemPIC incremental, and have reservations about the practical benefits, given the fact that it does not currently support hybrid architectures used across many recent open-source model families.

Overall, this is a borderline rejection case for me, so I will maintain my rating of 2 (weak rejection).

Reviewer_fD6HOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In many settings when you want to do a prefill over multiple documents. Unfortunatelyl prefilling on the individual documents is not equivalent to doing a prefill over all the documents because of the cross-document attention. We'd like to be able to do KV cache reuse to be able to make use of the already computed representations without having to do another prefill. Stitching together can substantially hurt performance. And so this paper proposes a setup where you basically train the language model via LoRA to basically act as if those documents were being processed in one long prefill rather than independently.

Critical strengths

I think it's a fairly clean idea that allows us to do KV cache reuse via a learned method. This is a pretty practical setting because in many settings we've already done a prefill over a relevant document. We don't want to have to do a prefill over this document again just to get its cross attention with the other documents. And I think it's clever to think about ways we can teach a model to act as if it's doing a normal prefill via a learning procedure.

Critical weaknesses

I think the conceptual novelty over existing methods like KV packet is limited. I think also it would be nice to stress test the method on tasks that really require complex reasoning over multiple documents. I'm not convinced that the current tasks really stress that. Could we use like a more realistic agentic workload or more RAG like settings?

Questions for authors

I'm not fully convinced that the baseline that uses boundary tokens has been fairly tuned. What happens as we blow up the number of header/trailer tokens or link tokens in CacheBlend?

3 comments

We sincerely thank the reviewer for recognizing the practical value of our work. We address the concerns below.

Q1. Contribution Beyond KV Packet

We acknowledge that SemPIC adopts KV Packet's KL distillation objective. However, the contribution is not merely replacing boundary tokens with LoRA, but changing the scope and mechanism of cache adaptation.

First, our boundary–interior analysis identifies the limitation of boundary-only adaptation: KV Packet substantially corrects attention near document boundaries but leaves considerable interior discrepancies and task-level errors. This motivates adapting the document representations themselves.

Second, SemPIC learns a reusable document-to-KV mapping without introducing additional boundary-token storage, allowing all document tokens to be adapted through a LoRA-enabled Writer while preserving native per-layer KVs and an unchanged Reader.

Third, KV Gradient Checkpointing enables differentiable Writer–Reader training by preserving per-layer KV gradients and correctly restoring the Writer's LoRA state during backward recomputation. For Qwen3-8B Writer training on Biography at FBS=8, it reduces peak training memory from 75.8 to 33.2 GiB, with training time increasing only from 23 to 25 minutes.

Together, these contributions establish a document-wide, offline learning approach to position-independent cache reuse, distinct from boundary-only adaptation and online cache repair.

Q2. Multi-Document Reasoning and Broader Workloads

Existing multi-document reasoning tasks. Our evaluation already includes MuSiQue (20 independently compiled paragraphs) and HotpotQA (10 paragraphs), both requiring multi-hop reasoning over evidence distributed across documents.

Additional out-of-domain evaluation. To further assess generalization, we directly evaluate the Qwen3-8B mixture-trained checkpoints from Table 2, trained on Biography, HotpotQA, MuSiQue, and NIAH, on three unseen benchmarks: 2WikiQA (multi-hop QA), RepoBench-P (cross-file code completion), and LCC (in-file code completion). SemPIC uses a rank-8 LoRA Writer and KV Packet uses 8+8 boundary tokens, with the same frozen Reader and no task-specific training.

We evaluate 100 examples per task, constructing independent PIC blocks as follows:

  • 2WikiQA: We group 100 questions into 50 shared-document pairs. Each group contains 10 supporting or distractor passages, with each passage's title and text forming one PIC block. Both questions reuse the same compiled caches.
  • RepoBench-P: Each supplied cross-file snippet and its file path form one PIC block. The current file's prefix remains online for next-line prediction.
  • LCC: Each file prefix is divided into approximately 4,096-character blocks at complete-line boundaries, preserving indentation, blank lines, and ordering.

Each block is compiled independently, without access to the query or other blocks.

TaskMetricFull RecomputeNo RecomputeKV PacketSemPIC
2WikiQAMean token F10.45110.24260.40050.4761
RepoBench-PEdit similarity0.60190.51160.40500.5782
LCCEdit similarity0.69170.65170.40470.6623

SemPIC outperforms KV Packet and No Recompute on all three unseen benchmarks, including code completion tasks absent from the training mixture. In particular, RepoBench-P evaluates reuse of independently compiled cross-file contexts.

These results extend our evaluation to unseen multi-document QA and code completion workloads, providing evidence of cross-task transfer without additional training. We acknowledge that end-to-end agentic workflows remain an important direction for further evaluation.

Q3. Baseline Tuning and Fairness

KV Packet. We conducted controlled comparisons on Qwen3-8B/MuSiQue, using identical training data (2,048 requests), validation data (128 requests), Full Recompute teacher continuations, KL objective, batch size (16), and training budget (5 epochs, 640 updates).

For blowing up the number of header/trailer tokens, we increased KV Packet's boundary tokens from 8+8 to 117+117, matching the 0.96M trainable parameters of rank-1 SemPIC. SemPIC uses LR 5×10−45\times10^{-4}; Packet uses validation-selected LR 10−310^{-3}, selected from {10−4,5×10−4,10−3}\{10^{-4},5\times10^{-4},10^{-3}\} for the 117+117 configuration.

MethodTrainable paramsF1Document KV (MiB/request)
Packet (8+8)0.07M0.3327388.9
Packet (117+117)0.96M0.31351002.0
SemPIC (rank 1)0.96M0.3723343.9

At matched capacity, SemPIC achieves higher F1 while using 65.7% less document KV storage. The paired 95% CI for the F1 difference is [−0.0245, 0.1412]. Increasing Packet's boundary-token count does not improve its performance in this experiment and substantially increases cache storage.

CacheBlend. CacheBlend controls quality through selective online token recomputation. Figure 4 already sweeps recomputation ratios of 10/30/50/70/90% on Qwen3-8B/MuSiQue, yielding F1 scores of 0.1617/0.3412/0.3655/0.4086/0.4217, respectively. Online prefill computation increases from 7.18 to 35.75 TFLOPs across this sweep.

Higher recomputation budgets can improve CacheBlend's quality, but incur additional online computation. SemPIC instead performs adaptation offline and requires no document recomputation on cache hits, representing a different quality–efficiency trade-off.

We sincerely thank you for your thoughtful review. We hope our responses and additional experiments have addressed your concerns. If you have any further questions or comments, we would be happy to discuss them.

Reviewer_QkeQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses position-independent caching (PIC), in which the KV cache of each reusable document is prepared once, relocated to its request-time position, and composed with other cached documents. While RoPE re-rotation handles positional alignment, independently constructed KV states remain contextually incomplete.

In particular, the paper aims to reduce this contextual mismatch by learning reusable document KV representations that better approximate full-context predictive behavior.

First, it reports a boundary–interior asymmetry in KV Packet, which uses trainable soft tokens (header and trailer) at the boundaries of reusable documents. It then proposes a LoRA-based solution. The same frozen pretrained model is used as a Writer for offline KV-cache construction and a Reader for online prefill and decoding. LoRA is enabled only for the Writer. The Writer is trained through behavioral distillation from full-context teacher predictions, with gradients passing from the Reader through the cached KVs.

The experiments show improved performance over KV Packet on average, with particularly strong gains on retrieval tasks such as Biography and NIAH.

Critical strengths

A clear distinction between positional alignment and contextual mismatch in reusable KV caches.

A simple but effective solution that achieves better average performance than KV Packet while keeping the Reader unchanged and avoiding document recomputation on cache hits.

Critical weaknesses

A more extensive evaluation under varying document positions and request contexts is needed. Regarding position, the authors could reuse the same compiled document caches while changing their locations and order within the context. Regarding contextual variation, it would be useful to vary the preceding instructions, conversation history, and surrounding documents.

Although the paper reports attention analyses, the changes in KV representations induced by Writer adaptation and their relationship to task quality deserve further investigation. Which changes account for the performance gains, given that each document is still compiled independently of its eventual request context? Building on the existing attention analyses, it would be useful to compare the KV representations of Full Recompute, No Recompute, and SemPIC using vector similarities after positional alignment and analyses at different layers and attention heads, and relate these changes to task performance.

Practical guidance for applying the method to a new domain or task is also limited. The paper reports domain-specific training as well as cross-domain and mixture-training results. However, under what conditions can an existing or mixture-trained Writer be reused without additional training, and when is in-domain adaptation necessary? If adaptation is needed, how should representative queries, document compositions, and teacher outputs be collected?

An important potential application of document-cache reuse is coding with a large codebase. How could the proposed method be applied to this setting? In particular, how should reusable code blocks be defined, and how well would independently compiled caches support tasks that require reasoning across files and their dependencies?

Questions for authors

See Weaknesses.

3 comments

We sincerely thank the reviewer for recognizing the strengths of our work. We address the four concerns below.

Q1. Cache Reuse Under Position and Context Variations

We conducted additional experiments on Qwen3-8B/MuSiQue using 100 questions and existing rank-8 Writers (three seeds). The same independently compiled document caches are reused across request variations without recompilation. The original F1 is 0.3679 ± 0.0224 (mean ± sample SD across seeds).

Position, instruction, and conversation history.

VariationConditionsF1 (mean ± SD)
PositionPrepending 512 / 2,048 / 4,096 tokens0.3528 ± 0.0185 / 0.3661 ± 0.0350 / 0.3526 ± 0.0234
InstructionThree paraphrases0.3576 ± 0.0269 / 0.3544 ± 0.0497 / 0.3527 ± 0.0284
HistoryThree synthetic user–assistant histories0.3336 ± 0.0069 / 0.3575 ± 0.0191 / 0.3569 ± 0.0152

Document order and composition. We reorder the 20 paragraphs or remove distractors while preserving all annotated supporting evidence.

ConditionF1 (mean ± SD)
Original order (20 paragraphs)0.3679 ± 0.0224
Reversed order0.3613 ± 0.0046
Fixed random permutation0.3535 ± 0.0184
Supporting passages first / middle / last0.4051 ± 0.0314 / 0.3859 ± 0.0200 / 0.3916 ± 0.0155
10 paragraphs (evidence preserved)0.4679 ± 0.0246
15 paragraphs (evidence preserved)0.4290 ± 0.0376

Performance remains relatively stable under position shifts, instruction changes, and document reordering, while removing distractors improves accuracy. All configurations achieve 100% reuse of retained document tokens, with zero online Writer calls, demonstrating cache reuse across varying request contexts and document compositions.

Q2. Representation Changes and Task Quality

We further conducted representation analyses and cache interventions on Qwen3-8B/MuSiQue, using 100 matched questions and three Writer seeds.

Representation similarity. After token and RoPE alignment, SemPIC reduces mean K cosine distance to Full Recompute from 0.1001 to 0.0698, while V distance changes slightly from 0.1358 to 0.1395. Thus, improved task performance does not require uniformly closer K/V representations.

K/V interventions. We independently swap K and V sources and evaluate task F1:

K source / V sourceFullNo RecomputeSemPIC
Full0.44830.12250.2865
No Recompute0.00000.05060.1398
SemPIC0.00000.10830.3679

Replacing either SemPIC component, even with its Full Recompute counterpart, reduces performance. This suggests that K/V compatibility matters beyond individual representation similarity.

Layer selection and interventions. We divided Qwen3-8B's 36 layers into six consecutive groups. Using 128 held-out requests, we evaluated each group by inserting its SemPIC KVs into No Recompute and reverting its KVs in SemPIC to No Recompute. Groups were ranked by the average bidirectional effect on the NLL of the same Full Recompute continuation:

\mathrm{score}(G)=\frac{1}{2}\big[& \mathrm{NLL}(N)-\mathrm{NLL}(\mathrm{Add}_G)+\mathrm{NLL}(\mathrm{Remove}_G)-\mathrm{NLL}(S) \big]. \end{aligned}$$ Here, $N$ and $S$ denote No Recompute and SemPIC, respectively. | Layers (0-indexed) | NLL effect | |---|---:| | 0–5 | 0.1250 | | 6–11 | 0.1184 | | **12–17** | **0.2202** | | 18–23 | 0.1685 | | 24–29 | 0.0704 | | 30–35 | 0.0538 | We selected **layers 12–17** based on held-out NLL, then fixed this selection for test evaluation across three seeds. Reverting their KVs reduces mean F1 from **0.3679 to 0.2413** (paired 95% CI for the change: [−0.1891, −0.0757]). **Token- and head-level interventions.** Within the selected layers and heads, reverting supporting-evidence tokens causes a larger F1 drop than reverting equally many random tokens (difference: **0.0658**, 95% CI: [0.0214, 0.1130]). Individual-head effects are less consistent across seeds. These findings support contributions from **coordinated K/V adaptation and evidence-bearing cached states**. Although the Writer remains request-independent, the frozen Reader uses online query states to select and integrate information from cached documents.

Q3. Writer Reuse and Domain Adaptation

We recommend reusing a mixture-trained Writer when it meets quality requirements on representative held-out requests, and performing in-domain adaptation otherwise.

For adaptation, training requests should cover representative queries, instructions, document lengths, compositions, and ordering patterns. The frozen base model generates Full Recompute teacher continuations, which provide KL distillation targets. We then optimize the Writer through the frozen Reader and validate on held-out requests before offline cache compilation.

As discussed in our response on cross-domain behavior, mixture training can mitigate domain-specific degradation. We additionally evaluated the mixture-trained Writer from Table 2 on unseen benchmarks without task-specific adaptation, including 2WikiQA (F1: 0.4761 for SemPIC vs. 0.4005 for KV Packet), as well as the code completion tasks below.

Q4. Code Completion and Cross-File Dependencies

We evaluated the Qwen3-8B mixture-trained checkpoints from Table 2, trained on Biography, HotpotQA, MuSiQue, and NIAH, on RepoBench-P (cross-file completion) and LCC (in-file completion), using 100 examples per task. SemPIC uses a rank-8 LoRA Writer and KV Packet uses 8+8 boundary tokens, without any code-specific training.

Code block construction.

  • RepoBench-P: Each supplied cross-file snippet and its file path form one PIC block. The current file's prefix remains online for next-line prediction.
  • LCC: Each file prefix is split into approximately 4,096-character blocks at complete-line boundaries, preserving indentation, blank lines, and ordering.

Each block is compiled independently without access to other blocks or the completion target.

TaskMetricFull RecomputeNo RecomputeKV PacketSemPIC
RepoBench-PEdit similarity0.60190.51160.40500.5782
LCCEdit similarity0.69170.65170.40470.6623

SemPIC outperforms KV Packet and No Recompute on both benchmarks, approaching Full Recompute on RepoBench-P despite no code-specific adaptation. This provides preliminary evidence that mixture-trained Writers can transfer to reusable codebase caches, including cross-file contexts.

We will incorporate these additional experiments and clarify the deployment guidelines in the revised paper.

Thank the authors for the rebuttal efforts. Thanks to the rebuttal, I think I can maintain the current positive evaluation.

Reviewer_dHyvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies position-independent caching (PIC). In PIC, the KV caches of reusable documents are computed independently offline and linked into later prompts at arbitrary positions. Because independently compiled KVs never attend to the context that precedes them at request time, naive reuse substantially degrades quality.

Starting from a diagnostic of KV Packet, which learns boundary (header/trailer) embeddings, the authors argue that boundary adaptation reduces attention deviation near block boundaries but leaves residual errors in block interiors and at the task level. They propose SemPIC. A LoRA adapter is active only while documents are compiled into KVs (the "Writer"), and the unchanged base model consumes the linked caches at serving time (the "Reader"). The Writer is trained by KL distillation from the base model's full-context output distribution, with gradients propagated through the cached per-layer KV tensors. A per-layer "KV Gradient Checkpointing" scheme keeps this training within GPU memory.

Critical strengths

S1. Deployment-friendly design with no added online cost.

  • Adaptation is confined to offline cache construction. The LoRA adapter is used only by the Writer; the Reader is the unchanged pretrained model operating on standard per-layer KVs.
  • As a result, the cache format, the cache-hit decoding path and the online prefill cost are identical to naive reuse. By construction, requests that do not use cached documents are unaffected.
  • Why it matters: this makes the approach directly compatible with existing serving stacks, unlike methods that modify the decoder or add an online encoder or cross-attention interface.

S2. Meaningful quality recovery at the cost of naive reuse.

  • Across the 12 model–task settings, SemPIC raises overall mean micro-F1 from 0.53 (KV Packet) to 0.60, against 0.62 for Full Recompute (Table 1). On Qwen3-4B, its per-model average matches Full Recompute (0.59). Its interior attention error is lower than KV Packet's in all 12 settings (Fig. 5).
  • Why it matters: this is the main evidence that document-side adaptation can close most of the PIC quality gap. However, the size of the gain varies considerably across models; the Llama-3.1-8B averages are tied (see W1).
Critical weaknesses

W1. The central comparison with KV Packet is not yet conclusive.

The paper states that "the central experimental contrast is the adaptation locus: document-wide for SemPIC and boundary-only for KV Packet" (§6.2, L355–357). Several aspects of the evidence make it difficult to attribute the observed gains to the adaptation locus:

  • Baseline configuration.
    • The training configuration of KV Packet in this paper (training data, epochs, learning rate, number of soft tokens) is not reported. §6.1 and Appendix A describe only SemPIC's settings.
    • KV Packet nearly fails on Qwen3-8B NIAH: 0.38, against 0.37 for No Recompute, while SemPIC reaches 0.70 (Table 1). To my knowledge, the original KV Packet paper does not evaluate this model.
  • Capacity.
    • By my estimate, rank-8 LoRA on the q/k/v/o projections of all 36 layers of Qwen3-8B amounts to roughly 7.7M trainable parameters. KV Packet's 16 soft-token embeddings amount to about 65K, a difference of about two orders of magnitude.
    • Without a capacity-matched boundary baseline, the comparison conflates the locus of adaptation with the amount of trainable capacity.
  • Concentration of gains.
    • Three cells account for about 74% of the summed per-cell improvement over KV Packet (0.62 of 0.84): Qwen3-8B NIAH (+0.32), Qwen3-4B Biography (+0.17) and Qwen3-4B NIAH (+0.13).
    • On Llama-3.1-8B the averages are tied (0.64 vs. 0.64), and SemPIC is lower on MuSiQue (0.28 vs. 0.37).
    • On HotpotQA, the two methods differ by at most 0.01 for every model.
  • Statistical reliability. Each cell uses 100 test examples with a fixed data seed. No repeated training runs, variance estimates or confidence intervals are reported, so many of the smaller differences cannot be distinguished from noise.

Why it matters: the contribution is primarily empirical. If the margin over KV Packet is driven by an under-tuned or lower-capacity baseline, or by sampling variance, the paper's main claim is not established.

W2. Limited novelty relative to KV Packet and prior learned-cache work.

  • Inherited from KV Packet.
    • The training objective is KV Packet's self-supervised KL distillation (L213–216). The paper itself notes that its distinction "is not the distillation loss itself, but its gradient path and adaptation locus" (L245–246).
    • The tasks, baselines and comparison grid also follow KV Packet (L375).
    • Moving the trainable component from boundary embeddings to a LoRA adapter on the document encoder is a natural extension.
  • Precedents in context compression. The Writer–Reader design, in which a LoRA-adapted copy of the LLM produces representations consumed by the frozen LLM, has clear precedents. None of these is discussed:
    • ICAE (Ge et al., ICLR 2024) produces memory-slot representations.
    • 500xCompressor (Li et al., ACL 2025) produces per-layer KV values.
    • Cartridges (Eyuboglu et al., ICLR 2026) trains KV caches offline with a context-distillation objective, for use by an unchanged model.
  • KV Gradient Checkpointing.
    • It is effective. Peak memory falls from 75.8 to 33.2 GiB at forward batch size 8, and training on HotpotQA, MuSiQue and NIAH becomes possible (Table 3).
    • However, it is essentially standard non-reentrant activation checkpointing adapted to return per-layer KVs (Eq. 10).
    • It increases training time from 34 to 115 minutes at forward batch size 1.

Why it matters: taken together, I view the methodological contribution as incremental, which places more weight on the empirical evidence discussed in W1.

Basis of my rating. My rating is driven mainly by W1. The current evidence does not yet establish that the gains over KV Packet reflect the adaptation locus rather than differences in baseline configuration or capacity.

Questions for authors
  1. KV Packet configuration (clarification).
    • How was KV Packet trained in this paper (training data, number of epochs, learning rate, number of header/trailer tokens)?
    • Was it retrained on the same distillation data as SemPIC?
    • Why does it barely improve over No Recompute on Qwen3-8B NIAH (0.38 vs. 0.37)?
  2. Trainable capacity (clarification; limited experiment).
    • What are the trainable-parameter counts of SemPIC, KV Packet and Joint?
    • How would a capacity-matched boundary baseline (e.g., more soft tokens or learned per-layer prefix KVs) perform on a subset of settings?
  3. Variance (clarification; limited experiment). What is the variance across adapter-training seeds, and what are the bootstrap confidence intervals over the 100 test examples for the main comparisons in Table 1?
  4. Out-of-domain behavior (clarification).
    • In Table 2 (Qwen3-8B), the Writer trained on Biography scores 0.32 on NIAH, below No Recompute (0.37).
    • The Writer trained on NIAH scores 0.25 on Biography, close to No Recompute (0.24) and well below KV Packet (0.53).
    • What explains this degradation, and how should a deployment handle document domains not seen during training? For example, is a mixture-trained Writer the recommended default?

Convincing answers to Questions 1–3, showing that the gains over KV Packet persist under a comparable training budget and are statistically reliable, would lead me to reconsider my rating.

6 comments

We sincerely thank the reviewer for recognizing the strengths of our work. We respond the concerns and questions below.

W1.1 & Q1. KV Packet Configuration

We retrained KV Packet using the same distillation data, preprocessing, teacher caches, and sample counts as SemPIC, following its original training recipe (30 epochs, LR 5×10−45\times10^{-4}, 8 header + 8 trailer tokens). Qwen3-8B, not evaluated in the original KV Packet paper, was trained using the same recipe.

For Qwen3-8B on NIAH, the originally reported KV Packet checkpoint achieved only 0.38 F1, barely above No Recompute (0.37). We suspected that training instability prevented its boundary embeddings from learning effective cache adaptation. To investigate this unusually poor result, we conducted three additional training runs, obtaining 0.6446 ± 0.0617 (mean ± sample SD). We will revise the main paper to report the multi-seed results. Such variability is consistent with prior findings on the training instability of parameter-efficient tuning methods[1–3].

W1.2 & Q2. Trainable Capacity

Parameter counts. SemPIC trains rank-8 LoRA (α=16\alpha=16) on the Writer's q/k/v/o projections, keeping the base model and Reader frozen. KV Packet trains 8 header and 8 trailer embeddings shared across cached blocks, while Joint trains both.

ModelSemPIC (rank 8)Packet (8+8 tokens)Joint
Qwen3-4B5.90M0.04M5.94M
Qwen3-8B7.67M0.07M7.73M
Llama-3.1-8B6.82M0.07M6.88M

Capacity-matched comparison. On Qwen3-8B, rank-1 SemPIC and KV Packet with 117 header + 117 trailer tokens each have 0.96M trainable parameters. Both use identical training data, with validation-selected learning rates for KV Packet.

DatasetSemPIC F1Packet F1SemPIC KV (MiB/request)Packet KV (MiB/request)
MuSiQue0.37230.3135343.91002.0
NIAH0.70550.67431173.71414.6

At matched capacity, SemPIC achieves higher F1 while using 65.7% less document KV storage on MuSiQue and 17.0% less on NIAH. Unlike SemPIC, KV Packet introduces additional boundary tokens for every cached block, making its storage overhead particularly substantial for tasks involving multiple short documents.

The paired 95% CIs for the F1 differences are [-0.0245, 0.1412] on MuSiQue and [-0.0873, 0.1470] on NIAH, respectively.

W1.3. Concentration of Gains

We acknowledge that improvements vary across tasks. The originally large gain on Qwen3-8B/NIAH (+0.32) was partly due to an unusually weak KV Packet checkpoint (0.38 F1). With the updated three-seed result (0.6446 ± 0.0617), this gain becomes +0.0557.

Importantly, SemPIC still outperforms KV Packet in 11 of 12 settings, with an average improvement of +0.0486 F1 (paired 95% CI: [0.0227, 0.0767]). Gains are more pronounced on Biography and NIAH, while HotpotQA shows smaller improvements and Llama-3.1-8B/MuSiQue remains a negative case.

Thus, the overall improvement persists beyond the original outlier, although its magnitude varies across tasks. We will update the corresponding results in the paper.

References

[1] Chen, Guanzheng, et al. "Revisiting parameter-efficient tuning: Are we really there yet?." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[2] Chen, Lichang, et al. "Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[3] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "The impact of initialization on lora finetuning dynamics." Advances in Neural Information Processing Systems 37 (2024): 117015-117040.

W1.4 & Q3. Variance and Bootstrap Confidence Intervals

Training-seed variance. To assess the reliability of the reported results, we retrained Qwen3-8B SemPIC (rank 8) and KV Packet (8+8 tokens) on MuSiQue and NIAH across three seeds, using identical data and evaluation protocols.

Test F1 (mean ± sample SD):

DatasetSemPICKV Packet
MuSiQue0.3679 ± 0.02240.3270 ± 0.0330
NIAH0.7095 ± 0.01200.6446 ± 0.0617

On MuSiQue, both originally reported results fall within three standard deviations of their respective means, with SemPIC showing higher mean F1 and lower variability. In contrast, the original KV Packet NIAH result deviates by over four standard deviations. We therefore revise only the anomalous Qwen3-8B/NIAH result.

Bootstrap CIs. For each model–task setting in Table 1, we draw 10,000 paired bootstrap samples from the 100 test examples and recompute corpus-level micro-F1 from aggregated TP/FP/FN. We report percentile-based 95% CIs for the F1 differences (SemPIC − KV Packet). For Qwen3-8B/NIAH, we compare the original SemPIC checkpoint against the three-seed mean of KV Packet, averaging the seed-specific F1 scores within each paired bootstrap sample. These intervals capture test-example variability.

ModelTaskSemPIC F1Packet F1ΔF1Paired 95% CI
Llama-3.1-8BBiography0.98840.9592+0.0292[0.0000, 0.0597]
Llama-3.1-8BHotpotQA0.45220.4404+0.0118[-0.0456, 0.0685]
Llama-3.1-8BMuSiQue0.28270.3697-0.0870[-0.1656, 0.0074]
Llama-3.1-8BNIAH0.82350.7973+0.0262[-0.0041, 0.0567]
Qwen3-4BBiography0.97920.8142+0.1650[0.0779, 0.2696]
Qwen3-4BHotpotQA0.29430.2892+0.0051[-0.0680, 0.0804]
Qwen3-4BMuSiQue0.31340.2416+0.0718[-0.0494, 0.2422]
Qwen3-4BNIAH0.75960.6293+0.1303[0.0655, 0.2110]
Qwen3-8BBiography0.97920.8919+0.0873[0.0358, 0.1496]
Qwen3-8BHotpotQA0.28340.2681+0.0153[-0.0525, 0.0837]
Qwen3-8BMuSiQue0.33560.2625+0.0731[-0.0034, 0.1545]
Qwen3-8BNIAH old0.70030.3811+0.3192[0.1588, 0.4399]
Qwen3-8BNIAH new0.70030.6446+0.0557[-0.0983, 0.2076]
Averageold0.59930.5287+0.0706[0.0434, 0.0924]
Averagenew0.59930.5507+0.0486[0.0227, 0.0767]

Q4. Out-of-Domain Behavior

Task differences and negative transfer. NIAH evaluates sparse evidence retrieval from long contexts, whereas Biography evaluates factual retrieval from short biographical records. Their different context lengths, evidence distributions, and cache-composition patterns may favor different adaptations, potentially causing negative transfer.

The apparent cross-domain advantage of KV Packet. For Qwen3-8B, the originally reported Biography advantage of NIAH-trained KV Packet (0.53 vs. SemPIC's 0.25) came from a checkpoint with unusually poor in-domain NIAH performance (0.38 F1). We hypothesize that this particular checkpoint failed to learn effective NIAH-specific boundary adaptations, consequently introducing less task-specific interference on Biography.

To investigate this, we retrained both methods on Qwen3-8B with three seeds:

MethodNIAH F1Biography F1
SemPIC0.72320.2760
SemPIC0.70380.3229
SemPIC0.70140.2727
SemPIC mean ± SD0.7095 ± 0.01200.2905 ± 0.0280
KV Packet0.68570.2903
KV Packet0.57360.2965
KV Packet0.67440.3199
KV Packet mean ± SD0.6446 ± 0.06170.3022 ± 0.0156

The retrained KV Packet achieves 0.6446 ± 0.0617 on NIAH, but only 0.3022 ± 0.0156 on Biography, compared with SemPIC's 0.7095 ± 0.0120 and 0.2905 ± 0.0280, respectively. Thus, the original apparent cross-domain advantage does not persist across retraining, while both methods remain limited in cross-domain transfer.

Mixed-domain training. Different tasks may require conflicting adaptations, making cross-domain generalization challenging for both methods. The mixed-domain results in Table 2 suggest that training on diverse domains can mitigate this degradation. We therefore agree that a mixture-trained Writer is the recommended default when cross-domain reuse is expected.

To further assess the generalization of mixture-trained Writers beyond the training domains, we directly evaluate the Qwen3-8B mixture-trained checkpoints reported in Table 2 (trained on Biography, HotpotQA, MuSiQue, and NIAH) on three unseen benchmarks: 2WikiQA (multi-hop QA), RepoBench-P (cross-file code completion), and LCC (in-file code completion). SemPIC uses a rank-8 LoRA Writer, while KV Packet uses 8 header + 8 trailer tokens. Both retain the same frozen Reader, with no additional training or task-specific adaptation.

Data construction and PIC configuration. We evaluate 100 examples per task, with documents organized into independently compiled PIC blocks:

  • 2WikiQA: We group 100 questions into 50 pairs sharing the same documents. Each group contains 10 supporting or distractor passages, with each passage's title and text forming one PIC block. The two questions reuse the same compiled caches.
  • RepoBench-P: Each supplied cross-file code snippet, together with its file path, forms one PIC block. The current file's prefix remains online for next-line prediction.
  • LCC: We split each file's prefix into approximately 4,096-character blocks at complete-line boundaries, preserving indentation, blank lines, and ordering. These blocks are independently cached for next-line prediction.

Each block is compiled without access to the query or other blocks. The frozen Reader consumes the positionally aligned KVs and processes the remaining prompt online. For code completion, we preserve the original target text and leave tokens crossing block boundaries online.

TaskMetricFull RecomputeNo RecomputeKV PacketSemPIC
2WikiQAMean token F10.45110.24260.40050.4761
RepoBench-PEdit similarity0.60190.51160.40500.5782
LCCEdit similarity0.69170.65170.40470.6623

SemPIC outperforms KV Packet and No Recompute on all three unseen benchmarks, including code completion tasks absent from the training mixture. These results provide further evidence that the mixture-trained Writer can generalize beyond its training domains without task-specific adaptation, supporting mixed-domain training as a practical strategy for cross-task reuse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.