- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a memory-lifetime mismatch between low-rank KV factors that depend on the full prompt and chunked prefill. Under conventional chunk-major execution, each layer group receives its complete factor source only when the final chunk arrives, causing dense KV histories from multiple groups to overlap in memory. SeamKV changes the traversal to group-major execution: all prompt chunks pass through one factor-producing group before advancing to the next. It accumulates additive statistics during execution, completes global factor computations using bounded source blocks after production finishes, and installs factors and releases dense KV after their final prefill use.
Critical strengths
-
The memory savings are practically meaningful, and the paper carefully examines target fidelity and operating boundaries.
-
The problem and most mathematical construction are well motivated.
-
The central systems insight is useful. A representation may be prompt-global in token scope while its source becomes complete at an intermediate depth.
-
The experiments clearly explain where the memory savings come from.
Critical weaknesses
-
The serving tests show a memory advantage, but do not yet establish the practical throughput gain. CM-tile runs out of memory at a concurrency where SeamKV succeeds (Section 4.5 Fig. 4). However, Appendix A.7 uses only twelve 64K requests, 8-token decode tails, and at most four active requests. A memory-aware CM scheduler could queue excess requests instead of failing. The paper does not report how that choice affects completion time, TTFT, or throughput.
-
The novelty over Layered Prefill is unclear. SeamKV already cites Layered Prefill, which uses layer-group scheduling and supports token chunking. The paper should explain what additional technical challenges it solves beyond applying this schedule to KV factor construction and releasing dense KV as soon as possible. Without this comparison, it is difficult to judge whether SeamKV offers a substantial new systems contribution or mainly an incremental integration of existing techniques.
Questions for authors
-
What does SeamKV add to existing layered prefill? If layered/group prefill uses the same bounded factorization and replaces dense KV at the earliest valid time, is it equivalent to Eq. (3) and Algorithm A.1? If not, please identify the extra operations or constraints. If so, please frame the contribution around factor construction and memory-lifetime analysis.
-
How were the 80 RULER examples in Table 2C selected, and how do the 64K and 128K CM protocols differ? Appendix A.8 reports exact sequence agreement of 68.8%/78.8%, although most task scores match. Please test a larger paired sample against a stated acceptable quality-loss threshold.
-
For Table 3, please compare total completion time, queue length, throughput, and TTFT under the same inputs and memory budget.
4 comments
Response to Reviewer YK3E — Q3 (Part 2/2)
Q3. Fair queueing: completion, queue, throughput, and TTFT
To address the short decode tails and limited workload of the submitted serving experiment, we evaluated a longer-decode mixed workload under fair memory-aware queueing. SeamKV and CM-tile were compared on 2×RTX 4090 under the same 22 GiB/GPU budget, tile-level prefill/decode scheduling, memory-aware FCFS admission, and frozen arrival/input/output traces. Each of three paired streams per load contains 24 requests (12×32K, 12×64K inputs; 64/128/256-token outputs). Offered arrival rates were set to 0.75×/1.10× the CM-tile finite-stream pilot processing rate. All requests completed; TTFT includes queueing delay, and reported values are medians across three paired request streams.
Fair queueing under matched workloads and memory budgets. Each entry reports the relative change versus CM-tile, followed by absolute values (CM-tile → SeamKV).
| Metric | 0.75× pilot | 1.10× pilot |
|---|---|---|
| Peak/GPU (GiB) | −19.1% (17.54→14.18) | −16.0% (19.09→16.04) |
| Mean admission queue (requests) | 0.149→0.040 | 1.612→0.838 |
| Completion time (s) | +0.3% (1347.8→1351.4) | −1.1% (1180.1→1167.0) |
| Output throughput (tok/s) | −0.3% (2.659→2.652) | +1.1% (3.037→3.071) |
| TTFT P99 (s) | +4.0% (116.8→121.5) | −2.4% (308.3→300.9) |
Percentages denote changes relative to CM-tile; lower is better for memory, queue occupancy, completion time, and TTFT, while higher is better for throughput.
With memory-aware queueing, both methods complete all requests under the same memory budget. SeamKV consistently reduces peak GPU memory by 16–19%. At 1.10× the CM-tile pilot rate, mean admission queue occupancy decreases by 48%, while completion time, output throughput, and TTFT P99 remain close to the control (within 2.4%). At 0.75×, memory and queue occupancy also decrease, with comparable completion and throughput and a 4.0% higher TTFT P99. These results show that SeamKV retains its memory-lifetime advantage under fair queueing while maintaining broadly comparable end-to-end completion performance.
Both methods use the same reconstruction-based decode backend. Output throughput denotes total generated tokens divided by the finite request-stream makespan, including arrivals, queueing, prefill, factor construction, and decoding, rather than per-request decode speed.
I am happy with the authors' rebuttal. Most of my concerns have been addressed. I have updated my score for this paper from 2 to 3.
Thank you for your reply. We're really glad that our rebuttal helped address most of your concerns. We sincerely appreciate your feedback.
Response to Reviewer YK3E — Q1–Q2 (Part 1/2)
We thank the reviewer for the precise Layered Prefill question. We first clarify the factor-construction and replacement operations required beyond group-wise traversal, then address RULER sample selection and the 64K/128K reference protocols, and finally report the queueing comparison under matched conditions.
Q1. Novelty beyond Layered Prefill
Layered Prefill can reproduce the group-wise execution order in Eq. (3), but traversal alone does not construct decode-ready full-prompt KV factors or enable their legal dense-state retirement. SeamKV addresses this factor-construction and memory-lifetime problem under chunked prefill. Although a factor depends on all prompt tokens, its source becomes complete once the final chunk exits its producer group, before deeper groups finish prefill. This separates the factor's full-prompt representation scope from its construction and memory lifetime. Under conventional chunk-major execution, producer-completion events cluster near the prompt tail, prolonging the overlap among dense KV histories. Full-prompt dependence does not imply prompt-end construction.
Exploiting this boundary requires a bounded factor-construction and replacement protocol. Source completion does not immediately yield a decode-ready factor: prompt-global refinement and core computations still require the complete source, while retaining all source rows would undermine the memory benefit. SeamKV accumulates additive statistics during production, completes the remaining global computations through bounded source replay, and retires dense K/V only after the factor is installed and its last required prefill use has passed (Sec. 3; Algorithm A.1). The state-accounting model in Eq. (7) explicitly captures the trade-off between avoided dense-state overlap and additional hidden buffering, earlier factor residency, and construction state. The full-prompt xKV factorization target is preserved.
Our matched experiments characterize the resulting memory-lifetime effects. The A→B→C→D controls show that earliest-legal reclamation and production-time accumulation under chunk-major execution do not eliminate dense-state overlap; advancing producer completion reduces 128K Peak/GPU from 20.084 to 15.912 GiB. As a separate physical check, delaying factor installation while holding producer order fixed increases 64K live dense groups from 1 to 8 and Peak/GPU from 11.944 to 14.333 GiB. Together, these measurements connect producer completion and replacement timing to actual dense-state residency.
Under the reviewer's counterfactual, an implementation with the same bounded factor construction and earliest-legal replacement could also reproduce Algorithm A.1 at the abstract level. SeamKV's systems contribution is the formulation and causal characterization of this factor-aware memory-lifetime problem, together with a bounded construction-to-retirement protocol that makes progressive dense-KV replacement possible.
Q2. RULER sample selection, reference protocols, and quality
Selection. Table 2C uses the first eight recorded examples (indices 0–7) from each of ten RULER tasks per context length (80 pairs), with identical input-token digests, answers, and scoring within each pair; this is a deterministic subset.
64K versus 128K reference. The Appendix A.8 quality-reference paths are distinct from the accumulating matched CM used for system scaling. At 64K, the reference directly retains pre-RoPE K; at 128K, it reconstructs pre-RoPE K from post-RoPE K through bounded inverse-RoPE views, avoiding a separate full-prompt pre-RoPE copy. Both target the same grouped full-prompt factorization and settings (rank/group, oversampling, power iteration, QR, replay), but inverse-RoPE changes the BF16 numerical path. Results should therefore be interpreted within length-matched pairs; the original 68.8%/78.8% exact-sequence agreements were not non-inferiority tests.
New BF16 test. We evaluated 160 separately frozen 128K-setting pairs (16 new prompts per fixed RULER-10 task; actual input lengths 113,799–131,035 tokens) against the matched bounded post-hoc reference, with identical inputs, generation limits, and scoring. Reference/SeamKV task-macro scores were 72.76/73.09 (Δ=+0.33 points before rounding), with 153/160 example-level scores tied. The pre-specified one-point non-inferiority margin is met on this fixed task set: fixed-task stratified paired-bootstrap 95% CI [−0.594, 1.375] (lower bound >−1). This new cohort is not pooled with the original 80 pairs; the conclusion concerns task-score non-inferiority, rather than exact-sequence identity.
Q3 (fair queueing: completion, queue, throughput, and TTFT) follows in Part 2/2. (continued in next comment)