acceptodds
Under review as a conference paper at ICLR 2027

Encrypted Latent KV Caches for FHE-Based LLM Decoding

Abstract

Fully homomorphic encryption (FHE) enables LLM inference over encrypted inputs without exposing plaintext prompts. Applying FHE to autoregressive LLM decoding, however, requires maintaining and repeatedly accessing a growing encrypted KV cache. KV cache compression is a natural way to reduce such state in plaintext serving, but we show that under RNS-CKKS, plaintext compression ratios do not directly predict encrypted capacity or latency. Page geometry determines the cache page count, while execution layout and scheduling also determine runtime and workspace requirements. We present Encrypted Latent Attention (ELA), to our knowledge the first system to realize and evaluate latent KV cache compression for per-step non-interactive LLM decoding under RNS-CKKS. ELA maps shared latent KV states to encrypted pages without reconstructing full keys or values. Page-bucket rank fitting raises latent rank without increasing page count. Head-Diagonal Packing reduces repeated per-head computation on the shared latent. PageFold aggregates aligned page states before applying shared reduction, masking, and refresh. On an RTX A6000, KV compression expands the measured layer-local resident encrypted cache capacity of a 1.7B model by . At a logical context length of 8k tokens, ELA achieves an approximately speedup over page-wise native full-KV in the measured encrypted attention core. We validate the encrypted implementation against a plaintext reference. These results show that effective encrypted KV compression depends on co-designing latent representations, ciphertext layout, and execution scheduling.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully homomorphic encryption, private LLM inference, KV cache compression, RNS-CKKS, multi-head latent attention, encrypted KV cache, long-context decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iPmGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Encrypted Latent Attention (ELA), which applies latent KV cache compression to FHE-based LLM decoding. Building on existing MHA2MLA checkpoints, the authors design an encrypted cache layout and select latent ranks according to ciphertext page capacity. They further propose Head-Diagonal Packing to reduce repeated computation across attention heads and PageFold to share reduction, masking, and refresh operations across cache pages. Experiments on SmolLM models show increased layer-local encrypted cache capacity and reduced attention-core latency.

Critical strengths

The paper identifies a meaningful problem: reducing the plaintext KV cache size does not necessarily reduce encrypted execution cost. The proposed packing and scheduling methods address this mismatch, and the ablation results show that both Head-Diagonal Packing and PageFold contribute to the speedup.

Critical weaknesses
  1. The main weakness is the lack of end-to-end encrypted decoding results. The GPU benchmark measures one layer’s attention core using synthetic weights and two repeatedly reused cache pages, excluding projections, cache updates, FFN, and page transfers. The separate learned-layer checks do not establish the latency or generation quality of the complete system. Therefore, it remains unclear how much of the reported attention-core speedup translates into actual model-level decoding benefits.

  2. The evaluation is limited to the SmolLM family, with the largest model being 1.7B. Since the proposed packing depends on the head count, latent dimensions, and page geometry, its benefits may change for other architectures. Evaluation on a larger model, such as a 7B/8B model, would help establish scalability.

Questions for authors
  1. Can the authors provide end-to-end encrypted decoding results on at least one model, even at a shorter context length, including cache updates and multiple consecutive decoding steps? Both latency and output agreement with the plaintext model would help assess the practical benefits.

  2. Can the authors provide evaluation results on larger models, such as 7B/8B models, to verify whether the proposed methods remain effective at a larger scale?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iPmG

We sincerely thank the reviewer for the careful review and for recognizing that a smaller plaintext KV cache does not by itself reduce encrypted cost and that both HDP and PageFold contribute to the speedup.

W1 & Q1. End-to-end encrypted decoding. We added learned-input accuracy and context-scaling measurements of the encrypted attention, and a multi-step end-to-end decoding measurement at 135M is in progress, whose outcome we will report during the discussion period.

Level 1, attention with learned inputs (new, GPU). Our encrypted attention stays within a 2−82^{-8} per-head relative L2L_2 error (below 0.4%) against exact fp64 attention in 1,1491{,}149 of 1,1521{,}152 cases in each of ELA, full-KV, and DeepSeek-V2-Lite (General Response, item 4). At 135M, its median latencies per layer are 8.2 to 10.0 s for ELA-d8d_8 against 8.2 s for full-KV at 129 keys, and 9.5 to 11.3 s against about 45 s at 8,192 keys, where only cost is measured.

Level 2, a connected attention layer over consecutive steps (new, GPU). We ran one attention layer at 135M over eight consecutive decode steps with the cache kept encrypted, including the RMSNorm, the projections, the cache append, and the residual. Each step takes the layer input that the plaintext model computes for its next generated token. Over the eight steps of three prompts, its largest relative L2L_2 error against the exact fp64 attention layer is 3.37×10−63.37\times10^{-6} for ELA-d8d_8 and 1.32×10−51.32\times10^{-5} for full-KV (Appendix G).

  • Page reuse and transfer (new). The paper times the attention core with two resident cache pages reused across the logical context. With 16 distinct pages instead, the 135M-d8d_8 latency was 0.40% higher in median. We also streamed every cache page from host memory inside the timed region at 1.7B and 8,192 keys. The latency rises from 18.48 to 21.35 s for ELA-d8d_8, which transfers 128 pages, and from 36.25 to 55.52 s for full-KV with PageFold, which transfers 1,024. ELA is then 2.60 times faster instead of 1.96. Each page is a 70 MiB ciphertext, sent from pageable host memory over PCIe 4.0 x8.
  • Scope at 1.7B. With positions up to 2,048, even the full-KV baseline needs a more accurate exponential than the tested double-precision Chebyshev interpolants of degree at most 59 attain, so this limit is not specific to ELA. The learned-input evaluation covers 135M and DeepSeek-V2-Lite (Appendix G).
  • Synthetic weights. They do not change the cost of the timed circuit (Appendix F.3), and General Response item 4 addresses its exponential stage.

Validations already in the paper (Appendix C, CPU). Complete 135M-d8d_8 layers, with the cache append, the FFN, and both residuals, run under CKKS at t=128t=128 with learned weights, and layers 28 and 29 are chained without decryption. A 100-step loop at 135M dimensions with synthetic inputs updates the encrypted cache without re-encryption, with softmax weights formed from decrypted scores, and at d8d_8 the attention top-1 matches an fp64 reference on all 100100 steps. In a plaintext fp64 diagnostic of the same circuit with exact SiLU, the layers deviate from the fp32 checkpoint by at most 1.21×10−51.21\times10^{-5} in relative L2L_2. With the polynomial SiLU of the FFN, which is identical in both representations, the deviation is 0.6260.626 in layer 28 and 0.8000.800 across layers 28 and 29.

Outside the attention core. The absorbed projections cost the same in both representations (response to Reviewer Winy), and RMSNorm, the FFN, their bootstraps, and the LM head are identical (Appendix B).

W2 & Q2. Larger models such as 7B and 8B. At the 7B geometry, ELA is 2.22 times faster than full-KV with PageFold at 8,192 keys in the benchmark circuit, and it holds eight times more tokens per page. The page layout of a model follows from its head count and its latent and RoPE widths, so Section 4 gives it in closed form before any run. The rules are P=nextPow2(dc+dR)P=\mathrm{nextPow2}(d_c+d_R), B=S/PB=S/P, and M=⌈t/B⌉M=\lceil t/B\rceil, and one output group suffices whenever H≤BH\le B.

  • The released MHA2MLA conversion of Llama-2-7B with dkv=16d_{kv}=16 (Ji et al., ACL 2025) has dc=512d_c=512, so the rules give P=1,024P=1{,}024 and B=32B=32. It keeps one output group and the two-head condition P=2dcP=2d_c. Its MHA baseline in the same release has B=4B=4, so ELA holds eight times more tokens per page and needs eight times fewer pages, the same ratio as at 1.7B-d8d_8.
  • We measured both geometries in the benchmark circuit with synthetic weights (Appendix F.4). At 8,192 keys, ELA takes 34.4 s per layer against 76.5 s for the MHA full-KV baseline with PageFold and 668 s under page-wise execution, so ELA is 2.22 and 19.40 times faster. The full-KV times are 2.11 and 2.00 times the 1.7B values as its page count doubles, whereas the ELA time equals that of 1.7B-d16d_{16}, which has the same page geometry.

We are deeply grateful to the reviewer for these constructive questions.

AuthorsOpenReview ↗

Follow-up on W1 & Q1: End-to-End Encrypted Decoding

Thank you again for asking for end-to-end encrypted decoding. In our response to W1 & Q1, we promised to report the outcome of the multi-step decoding at 135M. It is complete, and Section 6.4 and Appendix G of the revised PDF report it (Table 22).

The whole 135M model decodes under encryption, and encryption does not change its output. In plain terms, the encrypted model generates the same tokens as an unencrypted run of the same computation.

  • Setup. All 30 layers, the final normalization, and the LM head run under RNS-CKKS on one GPU, for ELA-d8d_8 and for full-KV, both with PageFold. Each of five held-out prompts is decoded for eight consecutive steps with the cache kept encrypted, and each token produced under encryption is the input of the next step.
  • Encryption fidelity. At all 80 steps, the encrypted token equals the token of the plaintext fp64 evaluation of the same circuit.
  • Agreement with the exact model. At 78 of the 80 steps, the encrypted token also equals the argmax of the exact model evaluated on the same prefix. The two other steps are full-KV steps at which the top two logits of the exact model differ by only 0.036 and 0.042 nat.
  • Stability. The plaintext twin evaluated on the same prefixes showed no approximation-domain or bootstrap-input violations.
  • Cost. Near position 129, a decode step takes 591 s for ELA-d8d_8 and 597 s for full-KV. At 8,192 keys, measured for cost only, ELA-d8d_8 is 3.37 times faster.

The update in the General Response thread summarizes the same results. We would be grateful to hear whether this addresses your concern, and we are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Dear Reviewer iPmG,

As the discussion period closes soon we wanted to follow up on our response. Following your main question, the revised PDF reports E2E encrypted decoding of the whole 135M model. Over eight consecutive steps on five held out prompts, with the cache kept encrypted, the encrypted token equals that of the plaintext evaluation of the same circuit at all 80 steps. For larger models, ELA is 2.22 times faster than full-KV with PageFold at the 7B geometry and 8,192 keys. Changes in the revised PDF are marked in blue.

We would be grateful to know whether these address your concerns, or whether any questions remain. Thank you again for your careful review.

Reviewer_iGkpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper Encrypted Latent Attention (ELA), to improve speed for FHE enabled LLMs. The key ideas used are two fold, the first is using Latent KV states, that reduce the size of KV cache and improves the throughput. The second set of improvements come from practical implementation choices such as Head-Diagonal Packing, Page-rank Fitting and aggregating accross page-folds before applying shared reductions and masking. Overall, these provide a total speed up of up to 18x for the encryption layer.

Critical strengths

The paper motivates the need for improving FHE based encrypted LLM inference and proposes Encrypted Latent Attention, which is unique and provide significant speed-up. The second class of improvements are optimal implementation improvements such as Head Diagonal packing, aggregation before page folds and page bucket ranking. The papers has detailed separation of benefits and ablations for each of the above optimizations and also has clear numerical correctness validation reported.

Critical weaknesses

The results mainly focus on latency of each layer and not E2E latency for inference. The claims suggest 18x improvements, but only 2x improvement is from the key highlight of the paper based on ELA, and the other improvements are achieved by smart implementation tricks. The largest model used in evals are up to 1.7B and most LLMs today are much larger, what are the feasibility at much larger models as well as practical applications that benefit from these smaller models. For attention, the paper uses models with MHA and GQA based techniques and it is not clear how well this will apply to MLA based approaches, will we see the same 2x improvement? How do MoE based models impact Encrypted Latent Attention?

Questions for authors

For completeness, some additional E2E latency numbers should be added to the paper. How well do these approaches work for natively MLA based models (DeepSeek style architectures, which are popular?). Most of the latency results are limited to maximum of 8k context length, how well does latency improvement scale with context length.

6 comments
AuthorsOpenReview ↗

Response to Reviewer iGkp

We are sincerely grateful to the reviewer for the time spent on our paper and for recognizing ELA as unique with a significant speedup, together with the detailed separation of benefits and ablations. We begin with W2 and W4.

W2. Packing and scheduling as contributions. HDP and PageFold are components of ELA, and the reported speedups answer different controlled comparisons in the benchmark attention core. Under FHE, the server cannot skip work depending on the data, so the ciphertext layout, the rotation count, and the order of reductions and refreshes determine the cost of an algorithm. Packing and scheduling are therefore algorithmic contributions in this setting, as in MOAI (Zhang et al., ICLR 2026).

  • The combined design, 18.05×18.05\times. It compares ELA's benchmark attention core with page-wise full-KV.
  • The schedule controlled, 1.96×1.96\times. It applies our PageFold schedule to full-KV as well, which makes the comparison stricter. PageFold alone makes full-KV 9.21×9.21\times faster.
  • The layout alone, 6.45×6.45\times. It compares HDP with per-head execution of the same latent cache under the same schedule. Without HDP, the latent cache is 3.29×3.29\times slower than full-KV with PageFold (119.20 s against 36.25 s), and HDP brings it to 18.48 s.
  • At 1.7B-d8d_8, HDP reduces the rotations from 74,28174{,}281 to 4,4204{,}420 (Table 4), and PageFold reduces the bootstraps of the attention core from 130130 to 33 per layer at 8,192 keys (Section 4.4).
  • Prior HE transformer systems, such as THOR, NEXUS, MOAI, and EncryptedLLM, run a single forward pass without a persistent KV cache, and Cachemir leaves cache compression to future work. ELA designs the encrypted execution of a compressed latent cache (Section 3).

W4. Which attention the evaluation uses. All ELA evaluations use an MLA-style latent-cache representation. The main benchmark tables use geometries derived from MHA2MLA conversions, with synthetic weights for the cost measurements (Section 5). Appendix G additionally evaluates encrypted attention on cache states from DeepSeek-V2-Lite, a natively trained MLA model. MHA and GQA appear only as full-KV baselines. The gain depends on the latent geometry. Against full-KV with PageFold, ELA-d8d_8 is 13.37×13.37\times, 11.92×11.92\times, and 1.96×1.96\times faster at 135M, 360M, and 1.7B, and the wider ELA-d32d_{32} is 1.86×1.86\times slower at 1.7B (Section 6.2).

W1 & Q1. End-to-end latency. Outside attention, both representations use the same RMSNorm and FFN. Under equal non-attention costs, the total per-token saving is the sum of the attention savings across layers (Appendix B). We also ran a connected attention layer at 135M, with the projections and the cache append, over eight decode steps with the cache kept encrypted. Each step uses the next-token layer input from the plaintext model. At 8,192 keys, where only cost is measured, one step takes 11.9 s for ELA-d8d_8 against 48.9 s for full-KV (Appendix G). Full-model encrypted decoding is in progress (General Response, item 2).

W3. Larger models. Encrypted inference is practical today at small scale, and there ELA enlarges the context that fits on one GPU. ELA-d8d_8 keeps 32,76832{,}768 and 16,89616{,}896 tokens of one layer resident at 135M and 360M, against 3,9043{,}904 and 1,9201{,}920 for full-KV, so only ELA holds an 8k-token context of a layer on one RTX A6000 (Section 6.1). At the geometry of the released MHA2MLA Llama-2-7B, ELA is 2.22×2.22\times faster than full-KV with PageFold at 8,192 keys in the benchmark circuit with synthetic weights (Appendix F.4).

W5. MoE. MoE changes the FFN, not attention, so ELA's attention and cache mechanism applies unchanged. DeepSeek-V2-Lite, on which we evaluated our encrypted attention, is an MoE model.

Q2. Natively trained MLA models. ELA applies to them directly. On DeepSeek-V2-Lite, our encrypted attention stays within the bound in 1,1491{,}149 of 1,1521{,}152 cases, the same rate as at 135M (General Response, item 4). The absorption identities of Section 2 depend only on MLA's shared latent and decoupled RoPE state. For DeepSeek-V2 and V3, with 128 heads, the grouping rule of Section 4.3 gives four output groups, beyond the two-group configurations we evaluate.

Q3. Contexts beyond 8k. The advantage grows with context length. In the learned-input circuit at 135M, the full-KV/ELA latency ratio is 4.0 to 4.7 at 8,192 keys and reaches 11.5 at 32,768 keys in layer 28, where only cost is measured (Appendix G). In the benchmark circuit, the ELA-d8d_8 advantage over full-KV with PageFold grows from 1.08 to 2.18 at 1.7B and from 3.56 to 16.52 at 360M between 1,024 and 32,768 keys (Appendix F.4). At 1.7B-d8d_8, PageFold keeps 33 bootstraps per layer for any tt, whereas page-wise execution needs M(t)+2M(t)+2, which is 514514 at 32,768 keys.

We are deeply grateful to the reviewer for these questions, which helped us state the contribution more clearly.

AuthorsOpenReview ↗

Follow-up on W1 & Q1: Full-Model Encrypted Decoding

Thank you again for raising end-to-end latency. Our response to W1 & Q1 noted that full-model encrypted decoding was in progress. It is complete, and Section 6.4 and Appendix G of the revised PDF report it (Table 22).

The whole 135M model decodes under encryption for eight consecutive steps on five held-out prompts, and at all 80 steps its token equals that of the plaintext evaluation of the same circuit. In plain terms, encryption does not change what the model generates.

  • One decode step of the whole model. Near position 129, a step takes 591 s for ELA-d8d_8 and 597 s for full-KV.
  • Longer contexts. Measured for cost only, ELA-d8d_8 is 1.68 times faster at 2,048 keys (657 s against 1,107 s) and 3.37 times faster at 8,192 keys (762 s against 2,567 s).
  • Agreement with the exact model. At 78 of the 80 steps, the encrypted token also equals the argmax of the exact model evaluated on the same prefix. At the two other steps, both full-KV, the top two logits of the exact model differ by only 0.036 and 0.042 nat.

The update in the General Response thread gives the full results. We would be glad to answer any further questions.

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Dear Reviewer iGkp,

As the discussion period closes soon, we wanted to follow up on our response. The revised PDF reports end-to-end encrypted decoding of the whole 135M model, results on DeepSeek-V2-Lite, a natively trained MLA model with MoE layers, and contexts up to 32,768 keys. The advantage over full-KV with PageFold grows with the context, to 2.18 times at 1.7B and 16.52 times at 360M at 32,768 keys. Changes in the revised PDF are marked in blue.

We would be grateful to know whether these address your concerns, including our explanation that HDP and PageFold are components of ELA. We are happy to answer any remaining questions.

Thank you again for your time.

Reviewer_WinyOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an encrypted latent KV-cache design for LLM decoding. It combines a compressed KV representation with ciphertext packing and an attention computation schedule to reduce the memory and computation required for encrypted attention. The evaluation studies attention-core latency, cache capacity, and the model-quality impact of compression.

Critical strengths
  • The paper addresses an important systems problem: the cost of storing and processing KV caches during encrypted LLM decoding.

  • Although the absolute inference latency remains high, I appreciate that the proposed approach achieves a meaningful speedup over encrypted full-KV attention. I see this as a valuable step toward making fully homomorphic encrypted LLM inference more practical.

  • I appreciate the authors' efforts to co-design the latent KV-cache representation and its encrypted execution. In particular, Head-Diagonal Packing (HDP) and PageFold effectively reduce redundant computation across attention heads and cache pages. These optimizations are important because KV-cache compression alone does not necessarily translate into faster encrypted inference.

Critical weaknesses
  • The latency evaluation focuses on the attention core and excludes the query and output projections. Because the proposed latent representation changes these projections, their cost could affect the net performance benefit. A measurement or cost breakdown for the complete attention layer would make the practical benefit easier to assess.

  • The evaluation is restricted to the relatively small SmolLM v1 models converted to MLA using MHA2MLA. It would be valuable to evaluate stronger and more representative small LLMs, such as Qwen3-1.7B or Qwen3-4B. Furthermore, all evaluated models require an MLA conversion step. Evaluating a model with a native MLA architecture, such as DeepSeek-V2-Lite, would strengthen the evidence that the proposed optimizations generalize to existing MLA-based LLMs without requiring model conversion.

Questions for authors
  • The unit for Table 3 is second, right? Please specify it better in the paper.

  • What is the additional cost of the absorbed query and output projections, and how does including these operations affect the attention-layer speedup?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Winy

We sincerely thank the reviewer for the thoughtful and generous review, and in particular for recognizing that KV-cache compression alone does not make encrypted inference faster. We label the weaknesses and questions W1, W2, Q1, and Q2 in the order of the review.

W1 & Q2. Cost of the absorbed query and output projections. Absorption adds no projection work. At 1.7B, the absorbed projections of ELA-d8d_8 use the same numbers of operations as those of full-KV, and their computation times agree within 0.1%. In plain terms, absorption changes which public matrix multiplies the encrypted vector, not the amount of encrypted work.

  • Operations. Each projection is a product of an encrypted vector with a public matrix, evaluated with baby-step giant-step diagonals. The output projection takes 2,0482{,}048 ciphertext-plaintext multiplications and 9898 rotations at one level in both representations. Neither projection needs rotation keys beyond those of the attention core, so they add no key memory.
  • Seconds. Each projection takes 1.561.56 s for the computation alone (1.5641.564 s for full-KV and 1.5651.565 s for ELA-d8d_8 in the output projection), about 1515 s when its plaintext diagonals are also transferred to the GPU, and 2525 to 3737 s when they are also encoded at every step, in both representations.
  • Projected speedup after adding the measured projection costs. The projections add the same cost to both representations. They therefore lower the ratio and leave the saved time unchanged, at 17.817.8 s per layer against full-KV with PageFold and 315315 s against page-wise full-KV. The table adds both separately measured projections to the attention core of Table 2 for ELA-d8d_8 at 1.7B and 8,192 keys, so its ratios are computed rather than measured on a connected 1.7B layer.
Projection cost includedvs. full-KV + PageFoldvs. page-wise full-KV
None (attention core)1.96×1.96\times18.05×18.05\times
Computation only1.82×1.82\times15.6×15.6\times
With transfer of the diagonals1.37×1.37\times7.5×7.5\times
With encoding at every step1.2×1.2\times to 1.3×1.3\times4.4×4.4\times to 5.6×5.6\times
  • Query replication. The query must also be replicated across the token slots of the core's input layout, in both representations. This step is not part of either measurement. We estimate it at about 0.360.36 s per layer for ELA-d8d_8 and 66 ms for full-KV at 1.7B.
  • Measured as one circuit. At 135M, we connected the RMSNorm, the projections, the cache append, the attention core, and the residual with the learned-input circuit. Over eight consecutive decode steps with the cache kept encrypted, its output stays within 3.37×10−63.37\times10^{-6} relative L2L_2 error (about three parts per million) of the exact fp64 attention layer for ELA-d8d_8 and 1.32×10−51.32\times10^{-5} for full-KV. At 8,192 keys, where only cost is measured, one step of this connected layer takes 11.9 s for ELA-d8d_8 and 48.9 s for full-KV. ELA-d8d_8 is 4.11 times faster with the projections included, against 4.65 times for its attention core alone (Appendix G).

W2. Stronger small models and native MLA. ELA applies to natively trained MLA models without conversion, as our evaluation on DeepSeek-V2-Lite shows. On cache states of the released DeepSeek-V2-Lite, our encrypted attention circuit stays within a 2−82^{-8} per-head relative L2L_2 error (below 0.4%) against exact fp64 attention in 1,1491{,}149 of 1,1521{,}152 cases, the same rate as at 135M (General Response, items 3 and 4). This holds because ELA works directly on MLA's shared latent and decoupled RoPE state. For head hh and latent cjc_j, qh⊤WK,hcj=(WK,h⊤qh)⊤cj,∑jahjWV,hcj=WV,h(∑jahjcj),q_h^\top W_{K,h}c_j=(W_{K,h}^\top q_h)^\top c_j,\qquad \sum_j a_{hj}W_{V,h}c_j=W_{V,h}\Big(\sum_j a_{hj}c_j\Big), which is how DeepSeek-V2 itself absorbs the up-projections into the query and output projections (DeepSeek-AI, 2024).

  • For DeepSeek-V2-Lite (16 heads, dc=512d_c=512, dR=64d_R=64), the rules of Section 4 give P=1,024P=1{,}024 and B=32B=32. One output group therefore holds all heads, and the two-head placement condition P=2dcP=2d_c also holds. Moonlight-16B-A3B shares this geometry.
  • Qwen3 models use GQA, and an MLA conversion such as MHA2MLA brings them into this setting, as it does for the SmolLM models we evaluate.

Q1. Units of Table 3. Table 3 reports seconds. The revised caption states the unit.

We are deeply grateful to the reviewer for the encouraging assessment and for questions that improved the precision of the paper.

AuthorsOpenReview ↗

Follow-up on W1 & W2: Model-Level Cost and Native MLA Latency

Thank you again for your questions on the costs beyond the attention core (W1) and on natively trained MLA models (W2). The revised PDF adds two measurements that bear on them.

On DeepSeek-V2-Lite, ELA's advantage in encrypted attention latency grows with the context length. We timed one attention layer at layers 13, 25, and 26 against a full-KV baseline that expands the same latent cache into per-head keys and values. The full-KV/ELA latency ratio is 1.01 to 1.02 at 129 keys and 2.95 to 3.04 at 8,192 keys, where only cost is measured (Section 6.2 and Appendix G, Table 23). In plain terms, one encrypted attention layer of this natively trained MLA model runs about three times faster with ELA at 8,192 keys.

The whole 135M model, with its projections, FFN, and LM head, decodes under encryption. Over eight consecutive steps on five held-out prompts, the encrypted token equals that of the plaintext evaluation of the same circuit at all 80 steps (Section 6.4 and Table 22). At 8,192 keys, measured for cost only, a full decode step is 3.37 times faster with ELA-d8d_8 than with full-KV (Section 6.2 and Appendix G).

A correction to the table in our response. Its row with encoding at every step used the same encoding time for both representations. With each representation's own measured encoding time, ELA-d8d_8 is 1.08 times faster than full-KV with PageFold and 4.77 times faster than page-wise full-KV in that row.

The update in the General Response thread gives the details. We would be grateful to know whether these address your concerns.

AuthorsOpenReview ↗

Follow-up on Q2: Measured 1.7B Attention Layer with Projections

Thank you again for Q2. We have also measured it directly at 1.7B. In the benchmark circuit of Table 2, which measures cost, we ran one connected attention layer with the RMSNorm, the absorbed query and output projections, the cache append, the attention core, and the residual, for ELA-d8d_8 and for full-KV, both with PageFold. Each time is the median of three runs.

With the projections included, ELA-d8d_8 is 1.21 times faster than full-KV at 8,192 keys, and the two are on par at 1,024 keys. In plain terms, the rest of the layer takes about the same time in both representations, so the time saved per layer stays close to that of the attention core.

KeysELA-d8d_8 layerFull-KV layerFull-KV/ELAAttention core, ELA-d8d_8 / full-KV
1,02468.10 s67.72 s0.995.29 s / 5.26 s
8,19281.61 s98.47 s1.2118.68 s / 36.00 s
  • Outside the attention core, the layer takes 62.8 to 62.9 s for ELA-d8d_8 and 62.5 s for full-KV. At 8,192 keys, ELA-d8d_8 saves 16.9 s per layer, close to the 17.3 s saved in its attention core.
  • The plaintext diagonals of the 1.7B projections do not fit in the 48 GB GPU memory next to the attention core. Both representations therefore encode them at every step. For the same reason, the row "Computation only" of our table could not be measured on this GPU.
  • This measurement replaces the computed ratio for encoding at every step in our previous follow-up. Its records are in the anonymized repository.

We are grateful for this question, which led us to measure the complete layer.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. The revised PDF marks all changes in blue.

Summary.

  • A stronger baseline. We made full-KV faster and re-measured it. At 1.7B, ELA-d8d_8 is 18.05 times faster than page-wise full-KV and 1.96 times faster under the same PageFold schedule.
  • Beyond the attention core. The absorbed projections cost the same in both representations. A connected attention layer runs encrypted over eight decode steps, and full-model decoding is in progress.
  • Model generality. ELA applies to natively trained MLA models without conversion, and at the 7B geometry it is 2.22 times faster than full-KV with PageFold.
  • Numerical fidelity. A circuit calibrated to real attention scores stays within 0.4% of exact attention in 1,149 of 1,152 cases, for ELA, full-KV, and DeepSeek-V2-Lite alike.

1. A stronger baseline. We found that full-KV runs faster when its cache pages are first lowered to the level of their first use. To make the comparison stricter, we added this step and re-measured every full-KV cell, so all timed circuits follow the same page-level rule (Appendix F.3). The ELA measurements are unchanged. At 135M and 360M, where the baseline gains more from this step, the advantage under PageFold is 13.37 and 11.92 times. The 1.7B capacity gain is 7.61 times.

2. Beyond the attention core. At 1.7B, the absorbed query and output projections cost the same in both representations, with equal operation counts and computation times within 0.1%. At 135M, we ran a connected attention layer, including the projections and the cache append, over eight consecutive decode steps with the cache kept encrypted. A step takes 10.7 s for ELA-d8d_8 and 12.6 s for full-KV, and the ELA output stays within 3.4×10−63.4\times10^{-6} relative error of the exact attention layer (Appendix G). This is not yet full-model encrypted generation. That measurement is in progress at 135M, and we will report its outcome in the discussion period.

3. Model generality. ELA uses MLA's shared latent and decoupled RoPE state, so it needs no conversion. On DeepSeek-V2-Lite, a natively trained MLA model, our encrypted attention is as accurate as at 135M. At the 7B geometry in the benchmark circuit, ELA takes 34.4 s per layer at 8,192 keys against 76.5 s for full-KV with PageFold (Appendix F.4).

4. Numerical fidelity. We separate the performance benchmark from numerical accuracy. Table 2 measures cost. Its exponential stage is a fixed degree-15 polynomial with synthetic coefficients, so it does not establish exponential accuracy. For accuracy, we added a circuit calibrated to the model's real attention scores and compared its encrypted output with exact fp64 attention on three held-out prompts. It stays within a 2−82^{-8} per-head relative L2L_2 error (below 0.4%) in 1,149 of 1,152 cases in each of 135M ELA-d8d_8, 135M full-KV, and DeepSeek-V2-Lite. Four of the nine cases above the bound have scores outside the calibrated range. The other five match a known overflow in CKKS bootstrapping and did not recur on re-execution. For this deeper circuit, the full-KV/ELA latency ratio is 0.82 to 1.00 at 129 keys and 4.0 to 4.7 at 8,192 keys, where only cost is measured. At 1.7B with positions up to 2,048, even the full-KV baseline needs a more accurate exponential than the tested double-precision Chebyshev interpolants of degree at most 59 attain, so this limit is not specific to ELA (Appendix G).

Contribution. Compression alone does not make encrypted attention faster. Executed head by head under PageFold, the compressed cache of 1.7B-d8d_8 takes 119.20 s, slower than the 36.25 s of full-KV, and HDP brings it to 18.48 s. HDP and PageFold are the co-design that turns KV compression into encrypted speedup. We see the encrypted KV cache as a problem that private generative inference has to solve, and ELA shows how to design its layout, computation, and refresh schedule together.

Thank you again for your time.

AuthorsOpenReview ↗

Update: End-to-End Encrypted Decoding

We thank the reviewers for their patience. As promised in the General Response (item 2), we have completed full-model encrypted decoding and added the results to Section 6.4 and Appendix G of the revised PDF.

Setup. The whole 135M model, with all 30 layers, the final normalization, and the LM head, runs under RNS-CKKS on one GPU for ELA-d8d_8 and for full-KV, both with PageFold. Each of five held-out prompts is decoded for eight consecutive steps with the cache kept encrypted, and each token produced under encryption is the input of the next step. The circuit constants were set without these prompts.

Results.

  • Encryption does not change the output. At all 80 steps, the encrypted token equals the token of the plaintext fp64 evaluation of the same circuit.
  • At 78 of the 80 steps, the encrypted token also equals the argmax of the exact model evaluated on the same prefix. The two exceptions are full-KV steps at which the top two logits of the exact model differ by only 0.036 and 0.042 nat.
  • The plaintext twin evaluated on the same prefixes showed no approximation-domain or bootstrap-input violations.
  • Model-level cost. Near position 129, a decode step takes 591 s for ELA-d8d_8 and 597 s for full-KV. At longer contexts, measured for cost only, ELA-d8d_8 is 1.68 times faster at 2,048 keys (657 s against 1,107 s) and 3.37 times faster at 8,192 keys (762 s against 2,567 s).
  • Latency on a natively trained MLA model. On three layers of DeepSeek-V2-Lite, the full-KV/ELA latency ratio of one attention layer is 1.01 to 1.02 at 129 keys and 2.95 to 3.04 at 8,192 keys, where only cost is measured.

Bootstrapping. The end-to-end runs use the bound K=648K=648 of later OpenFHE revisions (Appendix B). It makes the bootstrapping overflow of General Response item 4 negligible. It adds 2.5% to the time of a bootstrap call.

Thank you again for your time.

AuthorsOpenReview ↗

Dear AC and Reviewers

We are deeply grateful to the area chair and to all reviewers for their time and careful reading. The reviewers' questions led us to run the whole 135M model under encryption, to evaluate a natively trained MLA model, to measure connected attention layers, and to make our baseline stricter. Each of these made the paper stronger. We close with why the problem matters and what ELA contributes beyond its speedups.

Why the encrypted KV cache has to be solved. In non-interactive private decoding, the server keeps the KV cache under encryption and processes all of it at every step. The cache grows with every generated token. It therefore decides both how much context fits on a GPU and how long each step takes. With full-KV, one RTX A6000 holds only 3,904, 1,920, and 496 tokens of one layer at 135M, 360M, and 1.7B. In our whole-model measurement at 8,192 keys, which measures cost only, the attention over the cache takes 89% of a full-KV decode step. Private generation over long contexts is limited by this cache.

Why it remained open. Single-pass FHE transformer inference has become feasible, from THOR (CCS 2025), NEXUS (NDSS 2025), and MOAI (ICLR 2026) to GPU-accelerated EncryptedLLM (ICML 2025). These systems do not address the growing KV cache of autoregressive decoding. Cachemir (IEEE S&P 2027, to appear), concurrent with our work, keeps a full KV cache in non-interactive CKKS decoding and leaves its compression to future work. Interactive protocols handle the cache with communication, through query-aware cache selection in MPCache (NeurIPS 2025) or encrypted cache reuse with secret sharing in CryptoGen (arXiv 2026). To our knowledge, ELA is the first system to realize and evaluate latent KV cache compression for per-step non-interactive LLM decoding under RNS-CKKS.

What ELA contributes beyond its speedups.

  • A finding about representations. A smaller plaintext cache does not by itself make encrypted attention cheaper. At 1.7B, the 8 times smaller latent cache executed head by head under PageFold takes 119.20 s, slower than the 36.25 s of full-KV under the same schedule. Under FHE, page geometry, ciphertext layout, and the refresh schedule set the cost, not the number of stored bytes.
  • A method that turns compression into encrypted savings. Page-bucket rank fitting chooses latent ranks that fill the ciphertext pages, Head-Diagonal Packing evaluates all heads on the shared latent, and PageFold shares reduction, masking, and refresh across pages. Together they bring the same layer to 18.48 s. This is 1.96 times faster than full-KV under the same PageFold schedule and 18.05 times faster than page-wise full-KV. The resident cache capacity grows 7.61 times.
  • A step toward practical private inference. Only ELA holds an 8k-token context of one layer on one RTX A6000 at 135M and 360M. The whole 135M model decodes under encryption with the same tokens as its plaintext twin at all 80 steps of five held-out prompts. The advantage grows with the context. A full decode step is 3.37 times faster at 8,192 keys, measured for cost only, and the 360M attention core is 16.52 times faster than full-KV with PageFold at 32,768 keys. The gain carries to one attention layer of DeepSeek-V2-Lite, a natively trained MLA model, which is 2.95 to 3.04 times faster at 8,192 keys, and to the 7B geometry, which is 2.22 times faster. With the absorbed projections included, a 1.7B layer keeps 16.9 s of the 17.3 s that its attention core saves per step at 8,192 keys.

Status of the discussion. Reviewer Winy wrote that the experiments with the full attention layer and MLA have largely addressed their concern, and kept their rating of weak acceptance. Reviewer iGkp wrote that some of the questions were addressed and stated that they will raise their score from 1 to 2. Reviewer iPmG's main request was end-to-end encrypted decoding with cache updates and output agreement, and Section 6.4 and Table 22 report that setting.

How the discussion improved the paper. The revised Section 6.4 and Appendix G report whole-model encrypted decoding, DeepSeek-V2-Lite, and a connected attention layer. The revised Appendix F reports the stricter page-aligned baseline, contexts up to 32,768 keys, and the 7B geometry. The anonymized artifact shared with the reviewers and the area chair links every revised value to its run records.

We see the encrypted KV cache as a problem that private generative inference has to solve. ELA shows how to design its representation, layout, and refresh schedule together. We would be grateful if the reviewers and the area chair could take these results into account in their final assessment.

Thank you again for your generous engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.