- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies a write-once, query-later store for same-backbone LLM agents. Documents are stored as INT8 residual checkpoints at an early layer boundary. A receiver selects a document using its question and locally held evidence, inserts the checkpoint, and reconstructs upper-layer document states under its private prefix. No parameters are trained. Experiments separate BGE-M3 receiver-aware retrieval, matched read paths, and an end-to-end lexical-address system with overlapped transfer/preparation, using Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct.
Critical strengths
-
Reconstruction addresses the mismatch between stored states and the receiver's context. With query-free writes, the stored upper-layer states have never seen the eventual receiver's prefix. Holding the selected document fixed, reconstruction improves over static reuse by 13.54, 6.53, and 5.13 F1 points across the three backbones. Wrong-example checkpoints and random retrieval provide additional evidence that the selected content matters. The description of ragged-cache positions, masks, and execution paths also makes the approximation reasonably clear.
-
The timing comparison is fairly constructed and the limitations are clearly reported. The native-text baseline also overlaps transfer and preparation. Paired timing, process restarts, stage breakdowns, and transferred bytes give useful support for the modest TTFT benefit in this workload. I appreciate that the paper reports the much larger checkpoint traffic alongside the speedup, as well as the small completion-latency changes, shared-GPU limitations, and unresolved Llama quality margins.
Critical weaknesses
-
The practical operating envelope remains narrow and uncharacterized. TTFT improves by 4.23–6.91%, while checkpoint transactions carry approximately 413–661 times as many bytes as native text. The larger transfer is hidden behind preparation under the tested network and short windows; there is no demonstrated break-even range. The 8B runs also share GPUs with uncontrolled processes, and question-bootstrap intervals do not quantify independent load/network variation. Supporting-fact-centered windows, one gold document already held by the receiver, and a guaranteed target in a small directory further restrict workload representativeness. These are legitimate controls, but broader efficiency claims need a limited sensitivity check.
-
Component results do not fully establish the deployed system's contribution. The 15.5–17.2-point retrieval gains use BGE-M3 over document text, whereas the deployed selector uses 96-byte rare-term hash addresses. There is no matched query-only-address ablation to establish receiver conditioning at that budget. Adaptive-cache comparisons also exchange substantially different costs: reconstruction reruns 83–84% of document layer–token computations versus approximately 19–34% for CacheBlend, while transferring fewer bytes. Higher F1 alone does not establish the better systems tradeoff. The existing boundary sweep measures quality without alternative-boundary timing.
Questions for authors
- Based on the recorded transfer and preparation times, at approximately what bandwidth would the method cease to outperform native text? Could a slower-bandwidth or longer-window condition be used to check this estimate?
- For the existing 400-question sample, what happens to target retrieval and answer quality if receiver-document features are removed from the 96-byte-address scorer while keeping the rest of the selector unchanged?
- Could latency, transferred bytes, and recomputation be reported together for one representative adaptive-cache comparison? Alternatively, timing one more boundary from the existing sweep would help show how the reconstruction cost changes with the layer choice.
3 comments
Response to Reviewer vfgt 2/3
Q2 & W2: removing receiver features from the 96-byte scorer
We removed the receiver-document features from the deployed address scorer, turning receiver-aware (RA) selection into query-only (QO) selection, with addresses, directories and everything else unchanged.
| 400 questions | Qwen3-4B | Qwen3-8B | Llama-3.1-8B |
|---|---|---|---|
| Target hits, RA / QO | 310 / 179 | 310 / 179 | 310 / 179 |
| Text F1, RA | |||
| Text F1, QO | |||
| Checkpoint F1, RA | |||
| Checkpoint F1, QO | |||
| Checkpoint EM, RA | |||
| Checkpoint EM, QO |
Table 3: Receiver conditioning under the 96-byte address budget. The selected document changes on 179 of 400 questions: 134 wrong-to-correct, 3 correct-to-wrong and 42 wrong-to-wrong. Every gain is significant. Bold marks the better selector, and subscripts are standard errors.
Table 3 shows that receiver conditioning raises target hits from 44.75% to 77.50% and F1 by 16.7–20.2 points under the deployed budget, not only with BGE-M3 on full text, with nearly equal gains for text and checkpoint delivery. Post 3/3 tests this advantage in harder directories.
Q3 & W2: latency, bytes and recomputation for adaptive caching
We evaluated five reconstruction boundaries, four CacheBlend ratios, native text and static KV on Qwen3-8B, with quality from 800 target-fixed questions and receiver-side timing from 100 questions with five runs each. Table 4 shows representative points of this sweep.
| Read path | Recompute | F1 | After arrival (ms) | Total receiver work (ms) | Payload |
|---|---|---|---|---|---|
| Native text | 100% | 265 B | |||
| Reconstruction, b = 3 | 92% | 262 KB | |||
| Reconstruction, b = 6 | 83% | 262 KB | |||
| Reconstruction, b = 18 | 50% | 262 KB | |||
| CacheBlend 30% | 34% | 9.44 MB | |||
| CacheBlend 84% | 84% | 9.44 MB | |||
| Static KV, INT8 | 0% | 4.79 MB |
Table 4: Representative operating points (Qwen3-8B). "After arrival" assumes the receiver prepared its state in advance, and "total" adds that preparation. Bold marks the best value in each metric column, and subscripts are standard errors.
Table 4 shows three findings. The boundary trades quality for time, with 4.9 more F1 and 26 to 45 ms after arrival from b = 18 to b = 3. All five boundaries take 26–45 ms after arrival, versus 109–113 ms for the four CacheBlend ratios, and transfer 36× fewer bytes. At low budgets CacheBlend scores slightly higher with less recomputation (68.78 F1 at 34% vs. 68.13 at b = 18), though it stays slower and larger. End to end over TCP on 100 questions, reconstruction reaches 77.57 F1 at 82.33 ms TTFT, against 75.32 F1 at 269.08 ms for CacheBlend 84% and 76.39 F1 at 87.31 ms for native text. At ~84% recompute on 800 questions, the two reach similar F1, with reconstruction ahead by 1.44 and 0.87 points on the Qwen models and behind by 0.61 on Llama.
Response to Reviewer vfgt 3/3
W1: workload representativeness
The original workload used favorable controls: annotation-centered 64-token windows, a held gold first-hop document and a small directory that always contains the target. We relaxed each control separately, keeping the others fixed.
Windows and document length. Table 5 replaces the 64-token target window with the whole natural paragraph and with 1K–8K documents that embed the complete evidence paragraph among real paragraphs from other questions.
| F1 | Paragraph | 1K | 2K | 4K | 8K |
|---|---|---|---|---|---|
| Qwen3-4B, text | |||||
| Qwen3-4B, ckpt | |||||
| Qwen3-4B, static KV | |||||
| Qwen3-8B, text | |||||
| Qwen3-8B, ckpt | |||||
| Qwen3-8B, static KV | |||||
| Llama-3.1-8B, text | |||||
| Llama-3.1-8B, ckpt | |||||
| Llama-3.1-8B, static KV |
Table 5: Quality versus document length (same 380 questions and held window). Bold marks the best read path, and subscripts are standard errors.
Table 5 shows that the checkpoint stays within about 2 F1 of text on whole paragraphs, and within 0.9 F1 in a separate 400-question test with both documents as complete paragraphs (64–1,125 tokens). From 1K tokens both text and the checkpoint decline, the checkpoint more, while static KV drops much further, to 16–42 F1.
Held first hop. Table 6 replaces the held gold first-hop document with the top BM25 result for the question.
| Checkpoint pipeline F1 | Qwen3-4B | Qwen3-8B | Llama-3.1-8B |
|---|---|---|---|
| Gold first hop held (paper), RA | |||
| BM25 first hop, RA | |||
| BM25 first hop, QO |
Table 6: Checkpoint pipeline F1 with the first hop held or retrieved (400 questions). Bold marks the better selector under BM25, every RA gain is significant, and subscripts are standard errors.
Table 6 shows that pipeline F1 drops by 34–38 points, because BM25 returns the gold first hop for only 136 of 400 questions, while receiver conditioning still adds 10–12 F1.
Small directory with the target. Table 7 enlarges the directory to the full store of 815 windows, or replaces 30 or all 60 distractors with the BM25 windows that best match the question and the held window among 397,053 2Wiki training-context paragraphs.
| Checkpoint F1, 400 questions | Target hits | Qwen3-4B | Qwen3-8B | Llama-3.1-8B |
|---|---|---|---|---|
| Full store (815 windows), RA | 239 | |||
| Full store (815 windows), QO | 136 | |||
| 30 BM25 hard negatives, RA | 126 | |||
| 30 BM25 hard negatives, QO | 120 | |||
| 60 BM25 hard negatives, RA | 123 | |||
| 60 BM25 hard negatives, QO | 110 |
Table 7: Checkpoint F1 in harder directories (400 questions, original directory in Table 3). Bold marks the better selector, and subscripts are standard errors.
Table 7 shows that RA keeps a 13–14 F1 lead in the full store, where its median target rank is 1 against 46 for QO. Hard negatives retrieved with the same question and held window that RA uses make its signal less discriminative, and the lead narrows to 0.85–2.43 F1.
Response to Reviewer vfgt 1/3
We thank the reviewer for the precise questions, and for recognizing that reconstruction addresses the stored-state mismatch and that our timing comparison is fair. We ran the experiments the review asks for.
Q1 & W1: break-even bandwidth and longer windows
We ran the end-to-end system on Qwen3-8B at six link rates, each with the same 100 questions in three independent batches with process restarts and five timings per request.
Table 1: TTFT versus link rate (Qwen3-8B, 2Wiki), per-question medians of 15 timings averaged over questions. The 50 Mbps point is a monitored rerun. Bold marks the faster path, and subscripts are standard errors.
Table 1 shows that text TTFT does not depend on the link, because text sends only 397 bytes, while checkpoint TTFT falls from 288.80 ms at 10 Mbps to 79.19 ms at 1 Gbps. The sign reversal lies between 50 and 75 Mbps, and all three batches agree in direction at every rate. Both paths overlap transfer with the receiver's prefix preparation, so the checkpoint's extra transfer stays off the TTFT critical path until it outlasts that preparation. Unlike the paper's 8B timings, which shared GPUs with unrelated processes as its appendix states, this sweep and all other rebuttal timings ran on otherwise idle GPUs. Current GPU servers connect at 100–400 Gb/s per network adapter, so even a small fraction of that rate is far above this range.
Longer windows. Table 2 measures receiver-side time after the payload arrives on Qwen3-8B (10 documents, transfer excluded).
Table 2: Receiver-side time after the payload arrives, and payload size, versus document length. Bold marks the better path, and subscripts are standard errors over the 10 documents.
Table 2 shows that the checkpoint saves about 15% of receiver time at every length, while its payload grows by about 4 KB per token. At 8K tokens, serializing 32 MiB takes 268 ms at 1 Gbps, more than the 174 ms saved, but under 3 ms at 100 Gb/s, so by this component-level calculation the 8K break-even is about 1.5 Gb/s. Quality declines with length for both text and the checkpoint, and at 8K checkpoint F1 is 4.05, 5.77 and 12.20 points below text (Table 5, post 3/3).