acceptodds
Under review as a conference paper at ICLR 2027

BasisKV: Value Latents for Communication and Key Routing

Abstract

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose **BasisKV**, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality–efficiency tradeoff than existing low-rank baselines, reaching up to end-to-end speedup under 8-GPU tensor-parallel inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, inference optimization, KV cache compression, tensor parallelism, communication optimization, sparse attention, low-rank approximation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_V2Q8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

BasisKV targets long-context tensor-parallel inference, where KV-cache growth and per-GPU attention-output communication are both expensive. The method fits low-rank factors for each tensor-parallel source against the complete attention output, preserves source-specific latents through an AllGather, and reconstructs full-width outputs after communication. For linear Value paths, the same encoder is folded into the Value projection and reused for Value-cache compression. The compressed Value also predicts part of the pre-RoPE Key for page routing, with a low-rank Key residual and exact QK recomputation on selected pages.

Critical strengths
  • Using one low-rank Value latent for communication, Value-cache compression, and routing is a coherent and practically motivated system design.
  • The joint attention-output objective, alternating-least-squares updates, routing-aligned basis rotation, and Page-Fisher residual fitting are described in useful detail. The evaluation spans multiple model families, retention budgets, long-context benchmarks, and tensor-parallel settings.
  • Table 9 provides a useful routing ablation: B16R16 averages 79.48 on RULER, comparable to R32 at 79.52, and improves over R16 with a reported paired interval of [0.80, 5.17]. The Value-derived component therefore has measurable value.
Critical weaknesses
  • The efficiency claim is not quality- or latency-matched. The 1.42× speedup uses V64/50% retention on Llama-3.1-8B, where WikiText-2 perplexity rises from 7.22 to 11.50 and average MCQ falls from 0.709 to 0.673. The quality–efficiency trade-off is also model-dependent: at 75% retention, PaLU G-LRD has lower Llama perplexity (7.98 versus 8.61) but much higher communication traffic. A unified accuracy–memory–latency Pareto comparison, including measured PaLU latency, is still needed.
  • The scope of the memory claim should be explicit. Table 4 and Eq. (9) give useful logical widths and cache-traffic formulas, and Figure 4 reports measured peak decode memory. However, exact Keys and Values remain GPU-resident in the main sparse-decoding experiments, and the paper lacks a state-level byte breakdown for Table 3, page-metadata overhead, and an actual offload measurement.
  • Routing reuse is demonstrated, but its practical advantage needs clearer accounting. Table 9 shows that B16R16 matches R32 while using a 16-dimensional Key residual, so the Value base is useful. However, B16R16 still performs a 32-dimensional full-context routing read, and the aggregate decode results do not isolate whether the saved persistent state translates into routing-specific latency or memory savings.
  • The GQA parameterization, uncertainty, and artifacts need strengthening. The paper first fits an encoder per TP source and later states that query heads sharing a physical KV head share a Value encoder while retaining head-specific decoders; the constrained optimization problem and communication-byte formula are unclear. Main tables lack confidence intervals or calibration-seed variance, several margins are small, and no code artifact is linked. The narrow “first” claim should be retained only as an unverified, explicitly scoped statement.
Questions for authors
  1. What is the exact GQA parameterization, including shared Value encoders, head-specific decoders, and the resulting communication-byte formula?
  2. Can the authors provide a quality-matched Pareto comparison and measured latency for Dense, PaLU, sparse baselines, and BasisKV?
  3. Can they report a state-level byte breakdown for the Table 3 setting, page-metadata overhead, actual offload behavior, and a code artifact sufficient to reproduce the timing and memory results?
4 comments

Response to Reviewer V2Q8 — Part 1/2

We sincerely thank the reviewer for the careful reading, thoughtful questions, and constructive suggestions. Below, Part 1/2 covers the GQA formulation, collective-traffic equations, and multi-seed uncertainty analyses, and Part 2/2 covers unified quality–memory–latency comparisons, 128K state and offload accounting, and our anonymous code repository.

Q1. GQA parameterization and communication

GQA is enforced through parameter tying. Let HqH_q and HkvH_{\mathrm{kv}} be the numbers of query and physical KV heads, and let g(h)g(h) map query head hh to its KV head. With attention weights AhA_h, define Zh=AhVg(h)Z_h=A_hV_{g(h)}. All query heads associated with the same KV head share an encoder Eg∈Rdv×rE_g\in\mathbb R^{d_v\times r}, while retaining separate decoder blocks Dh∈Rr×dmodelD_h\in\mathbb R^{r\times d_{\mathrm{model}}}. For the stacked calibration outputs YY, the constrained objective is

\left\\|Y-\sum_{h=1}^{H_q}Z_hE_{g(h)}D_h\right\\|_F^2.$$ The decoder blocks are fitted jointly against the complete output, and updating a shared encoder accounts for all query heads that use it. In the TP-source notation of Equation 1, $E_p$ is therefore block-diagonal over the query heads owned by rank $p$ with blocks tied to $E_{g(h)}$, rather than a single unconstrained matrix per GPU. We cache $\widetilde V_g=V_gE_g$ once per physical KV head. Each query head computes its own attention-output latent $U_h=A_h\widetilde V_{g(h)}$, which remains distinct because the attention weights differ. The ranks AllGather these latents and reconstruct $\widehat Y=\sum_h U_hD_h$. Consequently, the Value cache scales with $H_{\mathrm{kv}}$, whereas the communicated latent width scales with $H_q$. For a layer with uniform rank $r$, $P$ TP ranks evenly partitioning KV groups without replication, and $s$ bytes per element, the compressed Value cache occupies $sH_{\mathrm{kv}}r/P$ bytes per cached token per rank. Including unchanged dense Keys of width $d_k$, the logical KV storage is $$M_{\mathrm{KV}}=\frac{sH_{\mathrm{kv}}}{P}(d_k+r) \quad\text{bytes per cached token per layer per rank}.$$ For $N$ current query tokens across the batch, the ideal ring collective traffic is $$C_{\mathrm{AllGather}}=\frac{P-1}{P}sNH_qr, \qquad C_{\mathrm{AllReduce}}=2\frac{P-1}{P}sNd_{\mathrm{model}}.$$ These are **bytes sent per rank per layer**, counting one direction; receive traffic is the same. The AllReduce expression refers to the dense full-width attention output, giving the relative traffic ratio $C_{\mathrm{AllGather}}/C_{\mathrm{AllReduce}}=H_qr/(2d_{\mathrm{model}})$ reported in Table 1 at average rank $r$. During one decode step, $N$ is the active batch size. We have made the shared-encoder constraint, head-specific decoders, and accounting units explicit in Section 3.2 and Appendix A.1. ### Additional: Calibration-seed variation and confidence intervals We repeated the Table 1 V64 comparison across C4 calibration seeds 42, 43, and 44, resampling calibration windows and refitting both factors and rank schedules per seed, reporting mean ± SD below: |Model|Method|WikiText-2 PPL ↓|C4 PPL ↓|7-task MCQ (%) ↑| |---|---|---:|---:|---:| |Qwen3-8B-Base|BasisKV|**8.220 ± 0.223**|**10.089 ± 0.023**|**67.34 ± 0.59**| |Qwen3-8B-Base|PaLU M-LRD|14.235 ± 0.892|16.257 ± 0.805|60.06 ± 0.64| |Qwen3-8B-Base|PaLU G-LRD4|10.645 ± 0.153|12.956 ± 0.128|64.94 ± 0.21| |Llama-3.1-8B-Instruct|BasisKV|11.096 ± 0.075|**12.529 ± 0.074**|**67.52 ± 0.58**| |Llama-3.1-8B-Instruct|PaLU M-LRD|15.592 ± 0.546|17.495 ± 0.504|59.59 ± 0.67| |Llama-3.1-8B-Instruct|PaLU G-LRD4|**10.506 ± 0.357**|13.040 ± 0.174|65.41 ± 0.38| Calibration variance is small across seeds: BasisKV leads on MCQ and C4 PPL in every seed on both models, while PaLU G-LRD4 retains lower Llama WikiText-2 PPL. Across 10000 task-stratified paired bootstrap resamples, BasisKV's three-seed MCQ gain over PaLU G-LRD4 is **+2.40 pp** with 95% CI 1.79 to 3.02 on Qwen and **+2.12 pp** with 95% CI 1.60 to 2.64 on Llama. We have added these statistics in Section 5 and Appendix C.2. We have also removed the “first” claim in Section 1 to state our cache-communication co-design, joint-fitting, and representation-reuse contributions directly.

Response to Reviewer V2Q8 — Part 2/2

Q2. Quality–memory–latency comparison

TP8 Pareto comparison. We added a TP8 comparison for Llama-3.1-8B-Instruct on eight L40S GPUs using the Table 1 75% Value-retention checkpoints. For PaLU G-LRD4, we measured both group-latent replication across ranks as PaLU-A and group-latent sharding as PaLU-B, using identical factors and rank allocations. Each request has 4096 input tokens and generates 128 tokens, reporting cohort E2E completion time after one warmup cohort:

MethodWikiT. / C4 PPL ↓MCQ ↑B32 E2E s ↓B64 E2E s ↓B64 logical KV GiB/rank ↓
Dense7.22 / 9.770.70911.4020.394.12
BasisKV8.61 / 10.660.7049.9117.063.61
PaLU-A: replicated7.98 / 10.660.69812.0522.218.25
PaLU-B: sharded7.98 / 10.660.69815.2026.713.61

At B64, BasisKV achieves 1.20× E2E speedup over Dense, 1.30× over PaLU-A, and 1.57× over PaLU-B, while reducing logical KV storage by 12.5% relative to Dense, matching PaLU G-LRD4 on C4 perplexity, and improving 7-task MCQ accuracy. PaLU-A doubles logical KV storage over Dense through latent replication, whereas PaLU-B recovers the cache savings at the cost of extra intra-group communication and higher measured latency. Logical KV sizes describe the live cache. We include this comparison in Section 5 and Appendix D.1.

Sparse baselines under TP1 and TP8. We also compare runtime and GPU memory on L40S GPUs for Llama-3.1-8B-Instruct in Appendix D.3 under the dense-Value sparse setting of Appendix C.3. Across 64K and 128K prefills with 128 generated tokens, the results are shown below with Keys offloaded for BasisKV and LRQK and Values offloaded for ShadowKV:

SettingMethodPrefill-to-ready sE2E sDecode ms/stepGPU resident GiB/rank
TP1, B1, 64KBasisKV11.4715.2229.4419.72
TP1, B1, 64KShadowKV18.2522.0529.8616.46
TP1, B1, 64KLRQK15.7354.23302.9026.04
TP1, B1, 128KBasisKV35.1239.3233.2024.23
TP1, B1, 128KShadowKV46.1150.1331.6617.60
TP1, B1, 128KLRQKOOMOOMOOMOOM
TP8, B16, 64KBasisKV85.4890.8042.5013.08
TP8, B16, 64KShadowKV154.11160.3251.6415.33

Unlike ShadowKV and LRQK, which rely on online, per-request factorization, BasisKV uses purely offline-calibrated linear projections, avoiding any runtime SVD or factor updates. At TP1, ShadowKV lowers GPU memory by offloading Values, matching BasisKV's decode speed at the cost of extra online prefill SVD, whereas LRQK's online fitting and per-step factor updates slow decoding by over 10× and run out of memory during 128K prefill. In our TP8 implementation of ShadowKV, we preserve its joint Key SVD across all KV heads: at each layer, we gather all ranks' Key shards onto a rotating owner GPU, run the online FP32 SVD per request, broadcast the shared temporal basis UU to every rank, and scatter the per-head SV⊤SV^\top slices. Replicating UU across all eight GPUs eliminates ShadowKV's single-GPU memory advantage over BasisKV's sharded Values while nearly doubling prefill-to-ready latency.

Q3. State accounting, offload, and reproducibility

State breakdown and Key offload. We make the memory scope explicit in Section 5 and Appendix B.4: the original Table 3 configuration at 128K on one L40S retains exact Keys and Values on GPU. Its 32.98 GiB PyTorch decode allocation comprises 14.96 GiB model parameters, 7.95 GiB each for exact Keys and Values, 0.99 GiB each for the separate Base and Key-residual caches, and 0.14 GiB for routing metadata, RoPE, and workspace buffers. Rerunning this setup alongside matched Base-reuse and Key-offload variants gives:

ConfigurationDecode GPU GiBPinned host K GiBPrefill peak GPU GiBDecode ms/step
Dense30.86038.3647.35
BasisKV, separate Base cache32.98040.4837.40
BasisKV, Base reuse31.98039.4837.83
BasisKV, reuse + Key offload24.297.9531.7939.48

Routing reuse, offload, and metadata. Reading Base directly from the rotated Values removes the dedicated Base cache while leaving the 32-dimensional routing scan virtually unchanged at 11.90 versus 11.75 ms, confirming that reuse reduces persistent memory rather than scan latency. Offloading historical Keys to pinned host memory and fetching only missed pages on demand substantially lowers GPU decode residency below Dense with only minor on-demand transfer overhead, while page scores and routing metadata occupy only ~0.05 GiB on GPU.

Code and reproducibility. We provide an anonymous BasisKV repository and benchmark guide covering factor fitting, evaluation, TP serving, sparse decoding, and offload accounting.

Reviewer_V2Q8OpenReview ↗

Thanks for your comprehensive response, I wouled raise my score to 3.

Thanks for your comprehensive response, I wouled raise my score to 3.

We sincerely thank the reviewer for reconsidering our work and raising the score. We are glad that our response and additional results were helpful. Your detailed feedback helped us strengthen the experimental evaluation and clarify the presentation. Thank you again for your constructive review.

Reviewer_KCuuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work applies a low-rank decomposition for the output matrix of the value projection in transformers. This operation successfully reduces the value cache size. By synchronizing the value latents in tensor parallelism, the authors manage to reduce the communication cost in distributed deployment. Furthermore, the authors propose a key routing mechanism based on the compressed value latents to perform paged sparse attention. Experiments show that the proposed method can achieve better performance under the same value retention budget while reducing the TP communication cost. It also achieves better performance on long context tasks compared to the sparse attention baseline methods.

Critical strengths
  1. The author proposes a novel approach of KV cache compression under the context of distributed deployment with tensor parallelism. The proposed method is well-motivated and addresses the real needs of LLM deployment.
  2. The proposed method successfully reduces the communication cost in distributed deployment while achieving better performance on language modeling and long context tasks compared to the baselines.
Critical weaknesses
  1. In table 1, the authors compare BasisKV with PaLU. I am wondering if the authors follows the standard implementation of PaLU. If so, under the same value retention budget, PaLU would have less key cache size than BasisKV, which would make the comparison unfair. If not, does it mean the performance gain comes from ALS optimization and this still holds without the context of TP?
  2. I notice that the experiments run on 8 L40S GPUs. L40S does not support NVLink. If we switch to other GPUs with NVLink (like H100 SXM), does BasisKV still bring significant speedup? In Figure 3 it seems that the end-to-end speedup is only about 1.1-1.3x, if the communication cost is already low, the speedup may further reduce. Maybe it brings more insight if the authors could provide more results on common practical TP deployment scenarios with NVLink support.
  3. If we only look at the computation graph without TP, BasisKV is very similar to PaLU. I think in the introduction or method part it worths a reference to this previous work and discuss differences and similarities.

Nonetheless, I would like to give this work a weak accept as it is already the best work I reviewed in ICLR 2027.

Questions for authors

See the weakness section.

3 comments

Response to Reviewer KCuu

We sincerely thank the reviewer for the encouraging assessment and generous feedback. Below, we clarify the Value-only PaLU setup and non-TP fitting gains, report new H100 SXM NVLink TP4 benchmarks as suggested, and clarify our distinction from PaLU.

1. PaLU comparison and quality without TP

Table 1 evaluates Value-only compression: all methods retain the original dense Keys and full attention, so Key-cache size is identical across methods. Our PaLU baselines apply PaLU's activation-aware SVD and Fisher-based rank allocation strictly to the Value projections for both per-head M-LRD and 4-head G-LRD, providing a matched comparison at each Value-retention budget. We now state this Value-only setup explicitly in Section 5.

Yes—the quality gain comes from joint-output ALS and holds directly without TP. Because TP AllGather reconstructs the exact same layer output as single-GPU execution, Table 1 itself reflects non-TP model quality. Whereas PaLU's SVD factorizes each head or 4-head group independently without cross-group output interaction, BasisKV's ALS fits all heads' encoders and decoders jointly against the complete summed attention output, capturing error interactions across the entire layer. Thus, BasisKV improves Value-compression quality on a single GPU, while TP adds the systems benefit of communicating compact latents before output reconstruction.

2. Performance on GPUs with NVLink

Our new H100 TP4 measurements show smaller end-to-end latency gains on a high-bandwidth NVLink node than on PCIe-connected GPUs. We compare Dense with BasisKV V64 on four H100 80GB GPUs for Llama-3.1-8B-Instruct and Qwen3-32B with 4096-token prompts and 1024 generated tokens, reporting total cohort completion time across prefill and decode as the median of three runs after one warmup:

ModelBatch sizeDense total time (s)BasisKV total time (s)Speedup
Llama-3.1-8B-Instruct327.8647.3281.073×
Llama-3.1-8B-Instruct6411.32110.4281.086×
Llama-3.1-8B-Instruct12818.43616.7011.104×
Llama-3.1-8B-Instruct25632.71729.8221.097×
Qwen3-32B3219.59118.9141.036×
Qwen3-32B6427.73426.7211.038×
Qwen3-32B12845.53543.8411.039×
Qwen3-32B256104.585100.3741.042×

End-to-end speedups remain consistent but modest across batch sizes, matching the expectation that communication compression yields smaller latency gains over faster interconnects. BasisKV targets communication-intensive deployments while still lowering completion time and preserving the 50% Value-cache memory reduction on NVLink. We have added these H100 TP4 results in Appendix D.2, signposted in Section 5.

3. Relationship to PaLU

We agree that on a single device, our Value path shares the same low-rank caching and output-projection fusion structure as PaLU, Chang et al., ICLR 2025, as noted in Section 6, with head-grouping TP comparisons added in Section 5 and Appendix D.1.

Our core distinction is reformulating Value low-rank compression as a joint cache-and-communication optimization problem. Using ALS against the complete attention output, BasisKV learns per-KV-head latents that preserve cross-head interactions without merging heads into shared groups. This yields a TP-friendly representation that directly compresses inter-GPU communication and can be further reused for Key page routing.

In contrast, PaLU relies on 4-head Group-LRD for quality, which introduces structural overhead in both Value aggregation and TP deployment. At average per-head rank rr, each query head must aggregate a shared 4r4r-dimensional group latent, incurring 4×4\times the per-head Value-aggregation compute of BasisKV. Moreover, when a group spans multiple GPUs under TP4 or TP8, G-LRD either replicates the group latent across ranks, sacrificing V-cache savings, or shards it at the cost of extra intra-group communication and computation on top of the standard output AllReduce. For concrete cache accounting and deployment measurements comparing these mappings, please see our response to Reviewer V2Q8, Part 2/2, Q2.

Reviewer_KCuuOpenReview ↗

Response to Authors

I thank the authors for their response. Most of my concerns have been addressed, and I decide to maintain my current score. I believe that combining optimization techniques with tensor parallelism (TP) will be a promising direction for future research. Congratulations on receiving positive feedback from all the reviewers!

We sincerely appreciate your kind congratulations and continued support, and are glad that the additional experiments addressed your concerns. Thank you again for your constructive review and for recognizing the promise of this direction.

Reviewer_nPVXOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BasisKV, a method to get more use out of the same compressed Value representation. it treats KV-cache compression and communication between GPUs as a joint problem, where each GPU computes a small attention-output latent, communicates it, and reconstructs the full output afterward. The same latent representation can also be stored directly in the Value cache for standard attention.

The paper then investigates using some of the information in compressed Values to predict which Key pages are worth accessing.It adds a small Key-specific residual. After selecting pages, it retrieves their original Keys and recomputes the QK scores. The reported gains include up to 1.42 speedup with eight GPUs and 1.25 full-model decode speedup at 128K context on one GPU.

Critical strengths
  • The reviewer appreciated that the paper considers where the compressed representation actually goes during inference. Reusing it for both caching and communication gives the compression a clear systems purpose. The final attention output combines contributions from different sources, so fitting those contributions together accounts for error interactions that independent fitting misses.
  • BasisKV generally gives the best average multiple-choice accuracy among the compared compressed methods (although it does not win every individual metric).
  • The paper measures actual model latency and throughput alongside reconstruction and benchmark quality. It also shows where the gains become larger: higher batch sizes and longer contexts.
Critical weaknesses

Several optimization steps rely on approximations: (1) alternating least squares does not guarantee a global optimum, (2)rank allocation approximates interactions between layers, and (3) Page-Fisher uses a local approximation that does not directly guarantee correct page selection. (1)--(3) reasonable engineering choices, but their optimality and sensitivity to approximation parameters remain unclear.

Questions for authors

How sensitive are BasisKV's quality and efficiency to the approximations and parameter choices in alternating least squares, rank allocation, and Page-Fisher fitting, and what evidence suggests these choices yield near-optimal solutions?

3 comments

Response to Reviewer nPVX

We sincerely thank the reviewer for the positive assessment and for highlighting our joint caching-and-communication design alongside the quality–latency evaluation. Below, we examine the sensitivity and near-optimality of all three approximations through ALS convergence sweeps, 1120 pairwise rank-exchange evaluations, and an exact full-dimensional page-scoring reference.

1. ALS initialization and optimization budget

The reconstruction objective is nonconvex jointly in the encoder and decoder factors. We optimize it using ALS, which iteratively updates one block of factors while holding the others fixed. We assess its practical convergence empirically in Figure 5, comparing activation-aware and random orthogonal initialization on Qwen3-8B-Base at ranks 64 and 96 through 16 iterations. The held-out relative reconstruction errors are:

RankInitialization0616
64Activation-aware0.143580.136810.13602
64Random0.212810.142090.13693
96Activation-aware0.058550.055230.05469
96Random0.093820.058470.05523

With the activation-aware initialization used in BasisKV, six iterations capture most of the reconstruction improvement observed through iteration 16 at both ranks. Further iterations reduce reconstruction error only modestly, and these reductions do not consistently translate into better downstream WikiText-2 perplexity. The two initializations also approach similar reconstruction errors, although activation-aware initialization reaches lower error with fewer sweeps. These findings support our preset six-iteration budget as a practical balance between fitting cost and quality. Additional ALS iterations affect offline calibration cost, while the inference-time operators and cache dimensions remain unchanged at fixed ranks.

2. Sensitivity to interactions in rank allocation

With KK candidate ranks for each of LL layers, there are KLK^L possible joint assignments before applying the budget constraint. Scoring each feasible assignment requires model forward passes over the calibration data to measure its output KL, making exhaustive evaluation computationally expensive. Instead, our rank allocation estimates each layer’s sensitivity by decreasing and increasing its rank while keeping the other layers fixed. This requires only 2L2L perturbed configurations, or 72 probes on Qwen3-8B-Base.

To assess this approximation, we further tested 1120 alternative allocations around the selected allocation. Each candidate decreases one layer’s rank by 16 and increases another’s by 16, keeping the total rank budget unchanged. We evaluated each candidate’s output KL relative to the dense model, directly accounting for interactions between the rank changes. This involved approximately 9000 batched forward passes. The best candidate reduced profiling KL by only 0.42%, while its WikiText-2 and C4 perplexity were slightly worse than those of the original allocation.

MetricOriginal allocationBest profiling exchange
Profiling KL0.1069480.106504
WikiText-2 PPL8.31198.3623
C4 PPL10.062010.0658

3. Page-Fisher versus full-dimensional page scoring

We compared Page-Fisher routing with an exact-score sparse reference that uses full-dimensional QK scores to rank pages under the same Value factors and Page4 sparse support budget of 2048 historical plus 64 recent tokens, with both methods computing attention only over the selected support.

Using the paper's ten-task RULER macro average, with 1000 examples per model:

ModelPage-FisherExact-score sparseExact minus Page-Fisher (pp)Paired 95% CI (pp)
Qwen3-8B78.63%79.24%+0.61[−0.447, +1.685]
Llama-3.1-8B-Instruct80.32%79.94%−0.39[−1.142, +0.367]

Page-Fisher differs from exact-score sparse routing by less than 0.7 percentage points in RULER accuracy on both models. The paired 95% CIs are computed using 10000 task-stratified paired bootstrap resamples.

We have added the ALS convergence and exact-score routing analyses in Appendices C.2 and C.3.

Dear Reviewer nPVX, thank you again for your insightful comments. We have added empirical analyses on ALS convergence, pairwise rank-allocation sensitivity, and exact-score routing comparisons on approximation optimality. As the discussion phase is ending soon, we would greatly appreciate your feedback on whether these results resolve your concerns, and we remain available for any further discussion.

We thank the authors for their clarifications. The paper is in a good shape to be recommended for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.