- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies position-independent KV cache reuse, where independently precomputed document caches are reused across different request contexts and document compositions without online document recomputation. The authors first analyze the boundary-based adaptation in KV Packet and identify residual interior attention deviations and task-level performance gaps despite substantial improvements near document boundaries. Motivated by this observation, the author propose SemPIC, which trains a LoRA-enabled document Writer through behavioral distillation to produce native per-layer KVs consumed by an unchanged, frozen Reader. Adaptation is confined to offline cache construction, preserving the standard KV interface during online inference. The paper also introduces KV Gradient Checkpointing to reduce training memory while preserving gradient flow through the cached KVs. Across three evaluated models and four tasks, SemPIC improves average token micro-F1 from for KV Packet to , approaching Full Recompute at . Additional experiments examine cross-domain transfer, quality–efficiency tradeoffs, and residual attention patterns.
Critical strengths
-
The paper is clear and easy to follow. It distinguishes positional alignment from contextual mismatch and motivates the method through an analysis of attention deviations near document boundaries and within document interiors. Figures 1 and 3 effectively illustrate the relationship to prior approaches and the training and inference workflows of the Writer–Reader design.
-
Restricting adaptation to the offline Writer while preserving the Reader and native KV interface is sensible and practically useful. The gains across multiple models and tasks, Joint variant, and transfer analysis provide useful evidence. The checkpointing experiments also demonstrate memory reductions and transitions from OOM to feasible execution under the tested configurations.
Critical weaknesses
-
The methodological novelty appears incremental. The main change is to extend KV Packet's boundary-parameter adaptation to LoRA adaptation of the document Writer while retaining full-context behavioral distillation. The broader idea of learning composable caches with a frozen backbone is also related to work such as C$^2$KV. Direct comparisons with these closely related methods and controls for adaptation capacity and training budget are missing, making it difficult to isolate the contribution of the adaptation locus itself.
-
Evidence for generalization and practical benefits remains limited. The transfer averages include in-domain results, and substantial negative transfer occurs: in Table 2, for Qwen3-8B trained on NIAH, Biography performance drops from with KV Packet to with SemPIC; in Table 1, Llama MuSiQue also decreases from to . Input-length distributions and cross-request reuse statistics are also unreported, limiting assessment of the motivating deployment scenarios. Furthermore, although checkpointing demonstrates memory savings and OOM-to-feasible transitions, its time overhead depends on configuration. Without memory and throughput scaling results or maximum trainable context lengths, the practical resource–time tradeoff remains insufficiently characterized.
Questions for authors
-
Could the authors clarify SemPIC's distinctive contribution relative to KV Packet and C$^2$KV, separating the effects of adaptation locus, capacity, and training budget? As a scope clarification, please also specify the architectural requirements of the native KV interface and whether hybrids such as Qwen3.5, combining full attention with Gated DeltaNet, would require a new state-composition mechanism.
-
Do the transfer gains persist when in-domain results are excluded? Could the authors provide input-length statistics, and clarify what the current experiments establish about changing document compositions and repeated cross-request reuse, versus what remains an unvalidated application scenario?
-
For KV Gradient Checkpointing, could the authors report memory usage, training throughput, and maximum trainable context length across input lengths, using fixed hardware, matched forward computations, and clearly specified batch settings? Please also clarify whether the times in Table 3 correspond to complete training runs, so that resource benefits can be assessed alongside their training-time cost.
5 comments
W2 & Q2.1. Out-of-Domain Transfer
We recomputed the transfer averages from Table 2, excluding each single-domain Writer's in-domain target and the mixture-trained rows. Each model contributes 12 cross-domain source–target combinations.
| Model | KV Packet | SemPIC |
|---|---|---|
| Llama-3.1-8B | 0.41 | 0.48 |
| Qwen3-4B | 0.37 | 0.37 |
| Qwen3-8B | 0.36 | 0.38 |
| Average | 0.38 | 0.41 |
SemPIC retains a +0.03 average F1 advantage after excluding in-domain results.
NIAH-to-Biography transfer. Regarding the originally reported NIAH-to-Biography transfer gap, the KV Packet checkpoint achieving 0.53 on Biography had unusually poor in-domain NIAH F1 (0.38). We hypothesize that this checkpoint learned ineffective NIAH-specific boundary adaptations, introducing less task-specific interference on Biography. Across three retraining seeds, KV Packet achieves 0.6446 ± 0.0617 on NIAH but only 0.3022 ± 0.0156 on Biography, compared with SemPIC's 0.7095 ± 0.0120 and 0.2905 ± 0.0280, respectively. The original apparent cross-domain advantage therefore does not persist across retraining. We will update the corresponding results.
W2 & Q2.2. Input-Length Statistics
We additionally report statistics for 100 Qwen3-8B test requests per task, excluding generated answers.
| Task | Median request tokens | Max tokens | Median documents/request | Median tokens/document |
|---|---|---|---|---|
| Biography | 435 | 467 | 5 | 80 |
| HotpotQA | 1,406 | 2,664 | 10 | 123 |
| MuSiQue | 2,443.5 | 3,827 | 20 | 104 |
| NIAH | 7,932 | 11,132 | 8 | 895.5 |
W2 & Q2.3. Document Composition and Cross-Request Reuse
Document composition and cache reuse. On Qwen3-8B/MuSiQue, we reuse fixed document caches while changing document order or removing distractors. Across three Writer seeds, the original/reversed/random orders achieve mean F1 of 0.3679/0.3613/0.3535. Retaining all supporting evidence while reducing the context to 10/15 paragraphs yields 0.4679/0.4290. All retained document tokens are reused, with zero online Writer calls.
Repeated cross-request reuse. We additionally evaluate the Qwen3-8B mixture-trained Writer from Table 2 on the unseen 2WikiQA benchmark without task-specific adaptation. We organize 100 questions into 50 pairs, each sharing 10 independently compiled document caches across two different queries. SemPIC achieves 0.4761 F1, compared with 0.4005 for KV Packet and 0.2426 for No Recompute.
These experiments demonstrate cache reuse across varying document compositions and multiple requests, while providing further evidence of cross-task generalization.
W2 & Q3. KV Gradient Checkpointing: Memory and Training Cost
Controlled memory and throughput measurements. We evaluated Qwen3-8B on a single A100 80GB GPU, using a BF16 base model, rank-8 LoRA, optimizer batch size 8, and forward batch size 1. Both configurations use identical inputs, losses, and batch settings. Each -token request contains document tokens and 32 query tokens, followed by 32 teacher-forced answer tokens. Measurements are averaged over three synchronized optimizer steps after one warm-up step.
| Length | Peak memory, off/on (GiB) | Step time, off/on (s) | Throughput, off/on (examples/s) |
|---|---|---|---|
| 2K | 30.99 / 17.52 | 6.85 / 15.71 | 1.17 / 0.51 |
| 4K | 46.63 / 19.68 | 12.90 / 16.44 | 0.62 / 0.49 |
| 8K | OOM / 23.99 | — / 32.69 | — / 0.24 |
| 16K | OOM / 32.61 | — / 72.20 | — / 0.11 |
At 4K, checkpointing reduces peak memory by 57.8%, with a 27.4% increase in step time. It also extends the longest successfully tested training sequence from 4K to 16K. At FBS=8, checkpointing enables 2K/4K training, whereas all tested lengths encounter OOM without checkpointing.
Table 3 training times. The reported times correspond to complete five-epoch Writer training runs. For Biography training at FBS=8, checkpointing reduces peak memory from 75.8 to 33.2 GiB, while increasing total training time only from 23 to 25 minutes. The larger overhead at FBS=1 (34 to 115 minutes) is primarily attributable to Biography's short sequences (slightly over 400 tokens per request), which limit GPU utilization during recomputation.
We will incorporate these additional results and clarifications into the revised paper.
Thank you for the additional experiments and clarifications. They address some of my questions. The clarification on hybrid architecture models is also helpful, though support for these models remains future work.
I still find the methodological contribution of SemPIC incremental, and have reservations about the practical benefits, given the fact that it does not currently support hybrid architectures used across many recent open-source model families.
Overall, this is a borderline rejection case for me, so I will maintain my rating of 2 (weak rejection).
We sincerely thank the reviewer for recognizing the strengths of our work. We address the concerns and questions below.
W1 & Q1.1. Adaptation Mechanism and C²KV Comparison
We acknowledge that SemPIC adopts KV Packet's KL distillation objective, but emphasize the differences in adaptation mechanism and cache representation.
Relative to KV Packet, our boundary–interior analysis reveals residual interior attention discrepancies under boundary-only adaptation. SemPIC instead learns a document-wide KV mapping through a LoRA-adapted Writer, improving native per-token KVs without introducing additional boundary tokens or modifying the frozen Reader.
Relative to C²KV. C²KV, a concurrent work, learns compressed, composable caches through a sidecar Extractor and compression tokens. In contrast, SemPIC directly adapts the document Writer to produce full, native token-level KVs, targeting position-independent reuse fidelity rather than context compression.
To further validate this distinction, we implemented the C²KV architecture under the same Qwen3-8B/MuSiQue training protocol as SemPIC, using identical training data, teacher continuations, KL objective, and training budget, while changing the cache-generation architecture.
SemPIC achieves higher F1 than C²KV at a 1:1 cache-token ratio, supporting the effectiveness of directly adapting native document KVs under matched training conditions.
Additionally, KV Gradient Checkpointing enables memory-efficient differentiable KV compilation while preserving gradient flow through all cached layers.
W1 & Q1.2. Capacity- and Budget-Matched Comparison
On Qwen3-8B/MuSiQue, rank-1 SemPIC and KV Packet with 117+117 boundary tokens have identical 958,464 trainable parameters. Both use 2,048 training requests, 128 validation requests, identical Full Recompute teacher continuations and KL loss, batch size 16, and five epochs (640 updates). SemPIC uses LR 5×10−4; Packet's LR 10−3 is validation-selected.
At matched capacity and training budget, SemPIC achieves higher observed F1 with 65.7% less document KV storage. The paired 95% CI for the F1 difference is [−0.0245, 0.1412].
Q1.3. Native KV Interface and Hybrid Architectures
The current implementation requires causal full-attention layers with conventional per-token, per-layer K/V states and positional relocation. Hybrid architectures such as Qwen3.5 combine full attention with Gated DeltaNet, whose recurrent states cannot be composed through ordinary KV concatenation. Extending SemPIC to such models would require an additional state-composition mechanism, which is an interesting future direction.