acceptodds
Under review as a conference paper at ICLR 2027

SeamKV: Efficient Construction of Full-Prompt KV Factors under Chunked Prefill

Abstract

Long-context LLM inference faces growing KV-cache memory and long-prompt prefill. KV-cache compression reduces resident state, while chunked prefill processes prompts incrementally; they compose poorly when the compact representation itself must be built from the full prompt. In xKV-style cross-layer low-rank sharing, each factor depends on all prompt tokens but is produced by a local layer group. Under chunk-major prefill, factor completion is tied to the final chunk, causing dense K/V from multiple groups to overlap near prompt end. We present SeamKV, which instead sends all prompt chunks through one factor-producing group before advancing deeper. It accumulates factor statistics during forward execution, completes remaining prompt-global products over bounded source blocks, then replaces dense K/V with the finished factor after the group has produced all required rows and its last prefill use has passed, preserving the same full-prompt factorization target. Relative to a matched chunk-major control, SeamKV reduces per-GPU peak memory by 9.5–20.8% across 32K–128K, while cache-ready time is lower at every tested length; at 128K, the peak number of simultaneously live dense groups falls from 8 to 1. Controlled experiments isolate earlier producer completion as the source of the memory reduction; in the matched causal audit, factor-finalization CUDA time differs by at most 0.33%. The schedules compute the same full-prompt operator in exact arithmetic, and under BF16 their paired RULER and LongBench results remain closely matched. The effect transfers to a per-layer full-prompt key-factor geometry, reducing peak memory by 13.6–22.6%; all 120 paired examples produce identical token sequences under SeamKV and the matched control. Prompt-global factorization therefore need not imply prompt-end construction.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language model inference, long-context inference, KV cache compression, chunked prefill, low-rank approximation, LLM serving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YK3EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a memory-lifetime mismatch between low-rank KV factors that depend on the full prompt and chunked prefill. Under conventional chunk-major execution, each layer group receives its complete factor source only when the final chunk arrives, causing dense KV histories from multiple groups to overlap in memory. SeamKV changes the traversal to group-major execution: all prompt chunks pass through one factor-producing group before advancing to the next. It accumulates additive statistics during execution, completes global factor computations using bounded source blocks after production finishes, and installs factors and releases dense KV after their final prefill use.

Critical strengths
  1. The memory savings are practically meaningful, and the paper carefully examines target fidelity and operating boundaries.

  2. The problem and most mathematical construction are well motivated.

  3. The central systems insight is useful. A representation may be prompt-global in token scope while its source becomes complete at an intermediate depth.

  4. The experiments clearly explain where the memory savings come from.

Critical weaknesses
  1. The serving tests show a memory advantage, but do not yet establish the practical throughput gain. CM-tile runs out of memory at a concurrency where SeamKV succeeds (Section 4.5 Fig. 4). However, Appendix A.7 uses only twelve 64K requests, 8-token decode tails, and at most four active requests. A memory-aware CM scheduler could queue excess requests instead of failing. The paper does not report how that choice affects completion time, TTFT, or throughput.

  2. The novelty over Layered Prefill is unclear. SeamKV already cites Layered Prefill, which uses layer-group scheduling and supports token chunking. The paper should explain what additional technical challenges it solves beyond applying this schedule to KV factor construction and releasing dense KV as soon as possible. Without this comparison, it is difficult to judge whether SeamKV offers a substantial new systems contribution or mainly an incremental integration of existing techniques.

Questions for authors
  1. What does SeamKV add to existing layered prefill? If layered/group prefill uses the same bounded factorization and replaces dense KV at the earliest valid time, is it equivalent to Eq. (3) and Algorithm A.1? If not, please identify the extra operations or constraints. If so, please frame the contribution around factor construction and memory-lifetime analysis.

  2. How were the 80 RULER examples in Table 2C selected, and how do the 64K and 128K CM protocols differ? Appendix A.8 reports exact sequence agreement of 68.8%/78.8%, although most task scores match. Please test a larger paired sample against a stated acceptable quality-loss threshold.

  3. For Table 3, please compare total completion time, queue length, throughput, and TTFT under the same inputs and memory budget.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YK3E — Q1–Q2 (Part 1/2)

We thank the reviewer for the precise Layered Prefill question. We first clarify the factor-construction and replacement operations required beyond group-wise traversal, then address RULER sample selection and the 64K/128K reference protocols, and finally report the queueing comparison under matched conditions.

Q1. Novelty beyond Layered Prefill

Layered Prefill can reproduce the group-wise execution order in Eq. (3), but traversal alone does not construct decode-ready full-prompt KV factors or enable their legal dense-state retirement. SeamKV addresses this factor-construction and memory-lifetime problem under chunked prefill. Although a factor depends on all prompt tokens, its source becomes complete once the final chunk exits its producer group, before deeper groups finish prefill. This separates the factor's full-prompt representation scope from its construction and memory lifetime. Under conventional chunk-major execution, producer-completion events cluster near the prompt tail, prolonging the overlap among dense KV histories. Full-prompt dependence does not imply prompt-end construction.

Exploiting this boundary requires a bounded factor-construction and replacement protocol. Source completion does not immediately yield a decode-ready factor: prompt-global refinement and core computations still require the complete source, while retaining all source rows would undermine the memory benefit. SeamKV accumulates additive statistics during production, completes the remaining global computations through bounded source replay, and retires dense K/V only after the factor is installed and its last required prefill use has passed (Sec. 3; Algorithm A.1). The state-accounting model in Eq. (7) explicitly captures the trade-off between avoided dense-state overlap and additional hidden buffering, earlier factor residency, and construction state. The full-prompt xKV factorization target is preserved.

Our matched experiments characterize the resulting memory-lifetime effects. The A→B→C→D controls show that earliest-legal reclamation and production-time accumulation under chunk-major execution do not eliminate dense-state overlap; advancing producer completion reduces 128K Peak/GPU from 20.084 to 15.912 GiB. As a separate physical check, delaying factor installation while holding producer order fixed increases 64K live dense groups from 1 to 8 and Peak/GPU from 11.944 to 14.333 GiB. Together, these measurements connect producer completion and replacement timing to actual dense-state residency.

Under the reviewer's counterfactual, an implementation with the same bounded factor construction and earliest-legal replacement could also reproduce Algorithm A.1 at the abstract level. SeamKV's systems contribution is the formulation and causal characterization of this factor-aware memory-lifetime problem, together with a bounded construction-to-retirement protocol that makes progressive dense-KV replacement possible.

Q2. RULER sample selection, reference protocols, and quality

Selection. Table 2C uses the first eight recorded examples (indices 0–7) from each of ten RULER tasks per context length (80 pairs), with identical input-token digests, answers, and scoring within each pair; this is a deterministic subset.

64K versus 128K reference. The Appendix A.8 quality-reference paths are distinct from the accumulating matched CM used for system scaling. At 64K, the reference directly retains pre-RoPE K; at 128K, it reconstructs pre-RoPE K from post-RoPE K through bounded inverse-RoPE views, avoiding a separate full-prompt pre-RoPE copy. Both target the same grouped full-prompt factorization and settings (rank/group, oversampling, power iteration, QR, replay), but inverse-RoPE changes the BF16 numerical path. Results should therefore be interpreted within length-matched pairs; the original 68.8%/78.8% exact-sequence agreements were not non-inferiority tests.

New BF16 test. We evaluated 160 separately frozen 128K-setting pairs (16 new prompts per fixed RULER-10 task; actual input lengths 113,799–131,035 tokens) against the matched bounded post-hoc reference, with identical inputs, generation limits, and scoring. Reference/SeamKV task-macro scores were 72.76/73.09 (Δ=+0.33 points before rounding), with 153/160 example-level scores tied. The pre-specified one-point non-inferiority margin is met on this fixed task set: fixed-task stratified paired-bootstrap 95% CI [−0.594, 1.375] (lower bound >−1). This new cohort is not pooled with the original 80 pairs; the conclusion concerns task-score non-inferiority, rather than exact-sequence identity.

Q3 (fair queueing: completion, queue, throughput, and TTFT) follows in Part 2/2. (continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer YK3E — Q3 (Part 2/2)

Q3. Fair queueing: completion, queue, throughput, and TTFT

To address the short decode tails and limited workload of the submitted serving experiment, we evaluated a longer-decode mixed workload under fair memory-aware queueing. SeamKV and CM-tile were compared on 2×RTX 4090 under the same 22 GiB/GPU budget, tile-level prefill/decode scheduling, memory-aware FCFS admission, and frozen arrival/input/output traces. Each of three paired streams per load contains 24 requests (12×32K, 12×64K inputs; 64/128/256-token outputs). Offered arrival rates were set to 0.75×/1.10× the CM-tile finite-stream pilot processing rate. All requests completed; TTFT includes queueing delay, and reported values are medians across three paired request streams.

Fair queueing under matched workloads and memory budgets. Each entry reports the relative change versus CM-tile, followed by absolute values (CM-tile → SeamKV).

Metric0.75× pilot1.10× pilot
Peak/GPU (GiB)−19.1% (17.54→14.18)−16.0% (19.09→16.04)
Mean admission queue (requests)0.149→0.0401.612→0.838
Completion time (s)+0.3% (1347.8→1351.4)−1.1% (1180.1→1167.0)
Output throughput (tok/s)−0.3% (2.659→2.652)+1.1% (3.037→3.071)
TTFT P99 (s)+4.0% (116.8→121.5)−2.4% (308.3→300.9)

Percentages denote changes relative to CM-tile; lower is better for memory, queue occupancy, completion time, and TTFT, while higher is better for throughput.

With memory-aware queueing, both methods complete all requests under the same memory budget. SeamKV consistently reduces peak GPU memory by 16–19%. At 1.10× the CM-tile pilot rate, mean admission queue occupancy decreases by 48%, while completion time, output throughput, and TTFT P99 remain close to the control (within 2.4%). At 0.75×, memory and queue occupancy also decrease, with comparable completion and throughput and a 4.0% higher TTFT P99. These results show that SeamKV retains its memory-lifetime advantage under fair queueing while maintaining broadly comparable end-to-end completion performance.

Both methods use the same reconstruction-based decode backend. Output throughput denotes total generated tokens divided by the finite request-stream makespan, including arrivals, queueing, prefill, factor construction, and decoding, rather than per-request decode speed.

I am happy with the authors' rebuttal. Most of my concerns have been addressed. I have updated my score for this paper from 2 to 3.

Thank you for your reply. We're really glad that our rebuttal helped address most of your concerns. We sincerely appreciate your feedback.

Reviewer_JoTTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies req-specific low-rank factors for chunk prefilling. It proposes SeamKV, which runs all chunks by one layer group before moving to the next, which accumlating some svd info during preiflling and release memory. By the results, the author shows that the peak mem is reduced by a lot.

Critical strengths
  • The problem is clearly pointed.
  • Same operator in exact arithmetic, outputs are almost identical.
  • The transfer to a second factor geometry shows it is not specific to xKV, which is good.
Critical weaknesses
  • The novelty is limited. Finishing a layer group before moving on is basically layer-wise or layered prefill, and the insight only applies to the req-specific setting?
  • The gains are not that much, and one-shot xKV still has lower latency at mid lengths.
  • Group-major order works against the main reason for chunked prefill, which is interleaving prefill with decode.
  • Only one setup, an 8B model on 4090s with a 22GiB cap. I am not sure this matters on modern GPUs.
Questions for authors
  • Decode latency under realistic mixed traffic (mayeb like, get a mix dataset)?
  • Results on a larger model (like large MoE?) or H100?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JoTT

We thank the reviewer for recognizing our lifecycle diagnosis, matched controls, and transfer to a second factor geometry. We directly tested the concern that group-major factor construction might compromise decode interleaving, and further clarify the practical trade-offs, novelty beyond Layered Prefill, and hardware scope.

Q2 / W3. Decode latency under mixed traffic

SeamKV yields at group–chunk tile boundaries, so group-major construction need not block decode for an entire producer group (Sec. 3.1; App. A.7). We measured this on 2×RTX 4090, Llama-3.1-8B-Instruct, 22 GiB/GPU using 24-request frozen streams (12×32K, 12×64K inputs; 64/128/256-token outputs), three paired streams per load. Arrival intensities were calibrated at 0.75×/1.10× the CM-tile finite-stream pilot processing rate. Both methods used the same completion-aware prefill/decode scheduler and memory-aware admission; all requests completed, and actual prefill/decode interleaving was verified in every run. A separate queueing study uses a tile-level policy on the same frozen streams; the table below uses the completion-aware policy.

SeamKV vs CM-tile, common completion-aware policy0.75× pilot1.10× pilot
Decode ITL P95−0.47%−0.43%
Decode ITL P99≈0%−0.24%
Peak/GPU−16.59%−7.92%
TTFT P99+1.57%+5.23%
Output throughput−0.47%−1.08%

Values compare medians of per-stream metrics. Decode ITL P95/P99 remains near CM-tile while peak memory falls, at a modest TTFT/throughput cost.

Q1 / W4. Larger models and H100

We prioritized controlled 32K–128K context scaling, BF16 fidelity, and matched causal/serving comparisons on 7B/8B models. H100 and >8B evaluations have not yet been completed; we plan to examine these extensions in follow-up work.

Eq. (7) provides a conditional scaling argument: avoiding overlapping dense-KV residency can benefit other models and accelerators when the saved dense state exceeds added hidden, factor, and construction state. H100 does not change the logical producer-completion/last-use conditions, but its capacity and runtime behavior can change the measured gains. The Mistral-7B 32K replication (9.84% lower matched Peak/GPU) and 8/9 favorable 64K rank/group settings support robustness within the tested range; larger-model and H100 effects remain to be measured.

W1. Novelty beyond Layered Prefill

Layered Prefill's group-wise execution order does not, by itself, construct a decode-ready full-prompt KV factor. Layered Prefill uses group-wise execution to reduce MoE expert-weight reloads and decode stalls; SeamKV must additionally construct a request-specific factor whose source spans every prompt chunk. After the last chunk exits a producer group, the source is complete, but prompt-global refinement/core work, installation into the decode representation, and last-use-safe dense-KV replacement still remain. SeamKV couples these operations to producer completion using production-time accumulation and bounded source replay (Sec. 3; Algorithm A.1), without changing xKV's factor target.

The matched controls isolate why this matters: eager reclamation and accumulation under chunk-major execution do not eliminate the dense-state overlap, whereas advancing producer completion reduces 128K Peak/GPU 20.084→15.912 GiB. As a separate implementation check, with producer order fixed at 64K, delaying factor installation raises live dense groups from 1 to 8 and Peak/GPU from 11.944 to 14.333 GiB. The ShadowKV-aligned per-layer variant you noted also reduces matched Peak/GPU by 13.65%/22.60% at 32K/64K, using a different factor layout and decode path. This supports the applicability of the same producer-completion and last-use mechanism to another factor layout, rather than only xKV's cross-layer layout.

W2. Practical gain versus native xKV

Our matched chunk-major control (CM) isolates the lifecycle effect under the same prompt-global factor target; it is not a substitute for comparing native xKV. Matched peak savings across 32K–128K are 1.08–4.17 GiB/GPU (9.46%–20.78%). At 64K under identical per-GPU allocator budgets, SeamKV/native xKV completed 3/3 vs 0/3 runs at 14 GiB/GPU; both completed 3/3 at 18 GiB/GPU, where native xKV was faster (19.440 vs 25.367 s cache-ready). The lifecycle savings enable a lower-memory feasible operating point, with a construction-latency trade-off when both fit.

Thanks for the authors' reply. Most of my concern are solved. I will maintain my score. :->

Thank you for your reply. We're really glad that our response helped address most of your concerns. We sincerely appreciate your feedback.

Reviewer_wMVuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SeamKV studies how chunked prefill interacts with X KV-style cross-layer low-rank KV factorization. Under conventional chunk-major execution, each full-prompt factor becomes complete only when the final prompt chunk passes through its producing group, so dense K/V from multiple groups overlap near the end of prefill. SeamKV processes all prompt chunks through one factorization group before advancing, accumulates factor statistics during source production, completes prompt-global products over bounded replay blocks, and installs each factor before releasing the corresponding dense K/V. Quality, numerical fidelity, and serving results are primarily evaluated on Llama-3.1-8B; Mistral-7B is a 32K memory and cache-ready replication.

Critical strengths
  • The distinction between full-prompt representation scope and producer-local construction timing is clear, and the proposed schedule follows directly from it.
  • The A→B→C→D matched-control decomposition carefully separates earliest-legal reclamation, production-time accumulation, and producer-oriented execution. Only the last step produces the large peak-memory reduction.
  • The memory results are meaningful: Peak/GPU decreases by 9.46%–20.78% over 32K–128K, and live dense groups fall from 8 to 1 at 128K. The N=1/2/4 and periodic-arrival tests also show a capacity advantage over the matched control.
Critical weaknesses
  • The hidden-state buffer cost needs a per-request breakdown. At 128K, H[1:T] adds about 1.0 GiB while live factor state remains 3.867 GiB. Although the paper includes N=1/2/4 serving tests, it does not report how hidden buffers, factors, sketches, replay state, and decode KV scale under concurrency. This is the key remaining memory question.
  • The memory–latency trade-off needs to be presented as a trade-off. At 64K and 96K, X KV reports cache-ready times of 19.879 s and 34.075 s, while SeamKV reports 25.306 s and 49.027 s. However, X KV also uses 17.079/21.454 GiB versus SeamKV’s 12.193/14.052 GiB and OOMs at 128K. The paper should compare methods at matched memory or quality and report throughput, not only cache-ready tim
  • Quality equivalence is not established. Paired intervals cross or touch zero, and exact-sequence agreement is only 68.8% at 64K, 78.8% at 128K, and 73.3% on LongBench-6. These are useful diagnostics, but they do not establish BF16 non-inferiority; the exact-arithmetic target claim should not be treated as empirical quality matching. The 80-example subsets and missing memory/timing variance further limit interpretation.
  • The contribution and evidence base are narrower than the framing suggests. The core idea is producer-oriented scheduling and progressive replacement; xKV and layer-wise/chunked-prefill prior work are the closest references. The paper should state precisely what remains new beyond the schedule, and it links no code, artifact, or raw logs, so the headline memory and serving results cannot be independently audited.
Questions for authors
  1. Can the authors report N=4/8 state accounting and an iso-memory or iso-quality comparison against X KV?
  2. Can they provide TTFT, throughput, P99, and run-level variance on the existing two-GPU setup?
  3. Can they clarify the precise novelty relative to xKV and layer-wise scheduling, and release code/configuration/logs sufficient to reproduce the memory and serving results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wMVu — Q1–Q2 (Part 1/2)

We thank the reviewer for identifying concurrent-state accounting as a key remaining memory question. We address it with per-request N=4/8 accounting, followed by the native-xKV trade-off and serving measurements. We then clarify what remains novel beyond traversal and provide additional BF16 quality evidence.

Q1. N=4/8 state accounting and equal-budget xKV comparison

We profiled N=4/8 offered 64K requests for SeamKV and the tile-yielding chunk-major control (CM-tile, Fig. 4) under the same memory-aware admission policy. Both completed all requests. CM-tile reached at most 3 active / 2 prefilling requests, versus 4 active / 3 prefilling for SeamKV; N counts submitted requests, not simultaneous residents. The measured GPU0/GPU1 peaks fell from 15.224/19.001 to 14.096/17.308 GiB, despite SeamKV admitting more concurrent work.

Per-request state at each method's GPU1 allocator peak (GiB; N=8). Request IDs p000–p007 denote the eight submitted requests (zero-indexed). Only requests resident at each method's own peak appear below: CM-tile has p002, p003, while SeamKV has p000, p001, p002. The two peaks occur at different times; these are not request-matched snapshots.

State (GiB)CM p002CM p003Seam p000Seam p001Seam p002
Dense KV4.5084.5001.0231.0001.000
Hidden00.0160.5000.5000.500
Factors000.9960.9960.996
Sketch0.1510.1510.0300.0300.030
Replay view00000
Decode KV00000
RoPE0.0310.0310.0310.0310.031
Transient/other1.31301.31300
Tracked total6.0044.6983.8942.5582.558

Across these snapshots, dense KV drops from 9.008 to 3.023 GiB, even after counting 1.500 GiB hidden state and 2.988 GiB factors for SeamKV. Including shared model state (8.291 GiB) and 0.008 GiB unattributed storage reconciles the respective GPU1 peaks with zero allocator-accounting residual. Replay views and persistent decode KV are zero at these snapshots, not necessarily throughout execution; other temporary workspace is included above. The 3/4 maximum-active counts occur at other instants in the runs. N=4 and N=8 have the same peak-state breakdown because additional requests queue under admission.

For native one-shot xKV, we matched the 64K inputs, model, target ranks/group, GPU placement, and per-GPU PyTorch allocator budget, while retaining each method's native factorization backend; this is a budget-matched practical comparison, not an identical-operator ablation. At 14 GiB/GPU, SeamKV finished 3/3 runs and xKV 0/3; at 18 GiB/GPU, both finished 3/3, but xKV had lower median cache-ready latency (19.440 vs 25.367 s, three runs). These are budget-constrained runs, distinct from the submitted default-latency protocol. They establish a memory-feasible region for SeamKV; the matched CM control isolates the lifecycle effect.

Q2. TTFT, throughput, P99, and run-level variance

We repeated a 12-request serving workload (6×32K and 6×64K inputs, 128 output tokens each) three times per method/load, with frozen arrivals, the same two GPUs, 22-GiB/GPU budget, stage-aware prefill/decode scheduling, FCFS request order, and memory-aware admission. Both methods completed all requests. Values are medians of per-run metrics; P99 refers to TTFT and request E2E latency, and relative changes use unrounded medians.

LoadMethodTTFT P99 (s)E2E P99 (s)Output (tok/s)Peak/GPU (GiB)
LightCM-tile25.66855.8802.68614.303
LightSeamKV25.34855.4292.69412.193
HeavyCM-tile356.467386.6213.13819.060
HeavySeamKV339.657375.6163.22415.638

At heavy load, SeamKV reduces peak memory 17.96%, TTFT P99 4.72%, and E2E P99 2.85%, while improving finite-stream throughput 2.73% (makespan 489.428→476.420 s). CM-tile also completes the same workload through queueing. Heavy-load run-level sample SDs (CM-tile/SeamKV) are 0.00223/0.00644 tok/s (throughput), 0.367/0.688 s (TTFT P99), 0.326/0.943 s (E2E P99), and 0.00000/0.00008 GiB (Peak/GPU). TTFT includes queue waiting.

Decode backend and metric. Both SeamKV and CM-tile use the same reconstruction-based decode backend; xKV's optimized selective-reconstruction and fused-decode paths are not integrated into these serving experiments. Output (tok/s) denotes total generated tokens divided by the finite request-stream makespan, including arrivals, queueing, prefill, factor construction, and decoding, rather than per-request decode speed.

Q3 (novelty and reproducibility) and the BF16 quality evaluation follow in Part 2/2. (continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer wMVu — Q3, BF16 (Part 2/2)

Q3. Novelty relative to xKV and layer-wise scheduling; reproducibility

Beyond traversal, SeamKV provides the factor-construction and replacement path that a layered schedule alone does not. xKV defines the prompt-global factor target, and Layered Prefill supplies a possible group-major execution order. Neither by itself makes a factor decode-ready at an intermediate producer boundary: completing its source is not the same as completing the factor. SeamKV accumulates statistics as source rows are produced, finishes the remaining prompt-global refinement/core products through bounded source replay, installs the factor for decode, and releases dense K/V only after its last required prefill use (Sec. 3; Algorithm A.1). The factorization target remains unchanged.

The matched A→B→C→D controls distinguish this construction path from eager reclamation or accumulation alone: advancing producer completion changes 128K Peak/GPU from 20.084 to 15.912 GiB. More directly, with producer order fixed at 64K, delaying factor installation raises live dense groups 1→8 and Peak/GPU 11.944→14.333 GiB; across 14 legal orders, producer-completion position correlates with dense-state live-area (Spearman ρ=0.9802). These results link earlier producer completion and legal factor replacement to the physical memory benefit.

We are organizing the core implementation, experiment configurations, and evaluation scripts into a reproducibility package for public release.

Additional BF16 quality evaluation

Exact-arithmetic factor-target agreement is not BF16 quality evidence. We therefore ran a separately frozen 128K-setting evaluation of 160 new paired prompts (16 per fixed RULER-10 task; actual input lengths 113,799–131,035 tokens) against the matched post-hoc quality reference, with identical inputs, generation limits, and scoring. The original 80-pair diagnostic used the first eight examples per RULER task.

Reference/SeamKV task-macro scores were 72.76/73.09 (Δ=+0.33 points before rounding); 153/160 example-level scores tied. With a pre-specified one-point non-inferiority margin, the fixed-task paired bootstrap gave 95% CI [−0.594, 1.375] (lower bound >−1), establishing task-score non-inferiority for this fixed RULER-10 protocol.

Reviewer_wMVuOpenReview ↗

Response to the authors.

I thank the authors for their comprehensive response which have addressed most of my concerns about the paper. So I would like to raise my score to 3.

Thank you for your reply. We're really glad that our response helped address most of your concerns. We sincerely appreciate your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.