- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Encrypted Latent Attention (ELA), which applies latent KV cache compression to FHE-based LLM decoding. Building on existing MHA2MLA checkpoints, the authors design an encrypted cache layout and select latent ranks according to ciphertext page capacity. They further propose Head-Diagonal Packing to reduce repeated computation across attention heads and PageFold to share reduction, masking, and refresh operations across cache pages. Experiments on SmolLM models show increased layer-local encrypted cache capacity and reduced attention-core latency.
Critical strengths
The paper identifies a meaningful problem: reducing the plaintext KV cache size does not necessarily reduce encrypted execution cost. The proposed packing and scheduling methods address this mismatch, and the ablation results show that both Head-Diagonal Packing and PageFold contribute to the speedup.
Critical weaknesses
-
The main weakness is the lack of end-to-end encrypted decoding results. The GPU benchmark measures one layer’s attention core using synthetic weights and two repeatedly reused cache pages, excluding projections, cache updates, FFN, and page transfers. The separate learned-layer checks do not establish the latency or generation quality of the complete system. Therefore, it remains unclear how much of the reported attention-core speedup translates into actual model-level decoding benefits.
-
The evaluation is limited to the SmolLM family, with the largest model being 1.7B. Since the proposed packing depends on the head count, latent dimensions, and page geometry, its benefits may change for other architectures. Evaluation on a larger model, such as a 7B/8B model, would help establish scalability.
Questions for authors
-
Can the authors provide end-to-end encrypted decoding results on at least one model, even at a shorter context length, including cache updates and multiple consecutive decoding steps? Both latency and output agreement with the plaintext model would help assess the practical benefits.
-
Can the authors provide evaluation results on larger models, such as 7B/8B models, to verify whether the proposed methods remain effective at a larger scale?
3 comments
Follow-up on W1 & Q1: End-to-End Encrypted Decoding
Thank you again for asking for end-to-end encrypted decoding. In our response to W1 & Q1, we promised to report the outcome of the multi-step decoding at 135M. It is complete, and Section 6.4 and Appendix G of the revised PDF report it (Table 22).
The whole 135M model decodes under encryption, and encryption does not change its output. In plain terms, the encrypted model generates the same tokens as an unencrypted run of the same computation.
- Setup. All 30 layers, the final normalization, and the LM head run under RNS-CKKS on one GPU, for ELA- and for full-KV, both with PageFold. Each of five held-out prompts is decoded for eight consecutive steps with the cache kept encrypted, and each token produced under encryption is the input of the next step.
- Encryption fidelity. At all 80 steps, the encrypted token equals the token of the plaintext fp64 evaluation of the same circuit.
- Agreement with the exact model. At 78 of the 80 steps, the encrypted token also equals the argmax of the exact model evaluated on the same prefix. The two other steps are full-KV steps at which the top two logits of the exact model differ by only 0.036 and 0.042 nat.
- Stability. The plaintext twin evaluated on the same prefixes showed no approximation-domain or bootstrap-input violations.
- Cost. Near position 129, a decode step takes 591 s for ELA- and 597 s for full-KV. At 8,192 keys, measured for cost only, ELA- is 3.37 times faster.
The update in the General Response thread summarizes the same results. We would be grateful to hear whether this addresses your concern, and we are happy to answer any further questions.
Follow-up Before the Discussion Closes
Dear Reviewer iPmG,
As the discussion period closes soon we wanted to follow up on our response. Following your main question, the revised PDF reports E2E encrypted decoding of the whole 135M model. Over eight consecutive steps on five held out prompts, with the cache kept encrypted, the encrypted token equals that of the plaintext evaluation of the same circuit at all 80 steps. For larger models, ELA is 2.22 times faster than full-KV with PageFold at the 7B geometry and 8,192 keys. Changes in the revised PDF are marked in blue.
We would be grateful to know whether these address your concerns, or whether any questions remain. Thank you again for your careful review.
Response to Reviewer iPmG
We sincerely thank the reviewer for the careful review and for recognizing that a smaller plaintext KV cache does not by itself reduce encrypted cost and that both HDP and PageFold contribute to the speedup.
W1 & Q1. End-to-end encrypted decoding. We added learned-input accuracy and context-scaling measurements of the encrypted attention, and a multi-step end-to-end decoding measurement at 135M is in progress, whose outcome we will report during the discussion period.
Level 1, attention with learned inputs (new, GPU). Our encrypted attention stays within a 2−8 per-head relative L2 error (below 0.4%) against exact fp64 attention in 1,149 of 1,152 cases in each of ELA, full-KV, and DeepSeek-V2-Lite (General Response, item 4). At 135M, its median latencies per layer are 8.2 to 10.0 s for ELA-d8 against 8.2 s for full-KV at 129 keys, and 9.5 to 11.3 s against about 45 s at 8,192 keys, where only cost is measured.
Level 2, a connected attention layer over consecutive steps (new, GPU). We ran one attention layer at 135M over eight consecutive decode steps with the cache kept encrypted, including the RMSNorm, the projections, the cache append, and the residual. Each step takes the layer input that the plaintext model computes for its next generated token. Over the eight steps of three prompts, its largest relative L2 error against the exact fp64 attention layer is 3.37×10−6 for ELA-d8 and 1.32×10−5 for full-KV (Appendix G).
Validations already in the paper (Appendix C, CPU). Complete 135M-d8 layers, with the cache append, the FFN, and both residuals, run under CKKS at t=128 with learned weights, and layers 28 and 29 are chained without decryption. A 100-step loop at 135M dimensions with synthetic inputs updates the encrypted cache without re-encryption, with softmax weights formed from decrypted scores, and at d8 the attention top-1 matches an fp64 reference on all 100 steps. In a plaintext fp64 diagnostic of the same circuit with exact SiLU, the layers deviate from the fp32 checkpoint by at most 1.21×10−5 in relative L2. With the polynomial SiLU of the FFN, which is identical in both representations, the deviation is 0.626 in layer 28 and 0.800 across layers 28 and 29.
Outside the attention core. The absorbed projections cost the same in both representations (response to Reviewer Winy), and RMSNorm, the FFN, their bootstraps, and the LM head are identical (Appendix B).
W2 & Q2. Larger models such as 7B and 8B. At the 7B geometry, ELA is 2.22 times faster than full-KV with PageFold at 8,192 keys in the benchmark circuit, and it holds eight times more tokens per page. The page layout of a model follows from its head count and its latent and RoPE widths, so Section 4 gives it in closed form before any run. The rules are P=nextPow2(dc+dR), B=S/P, and M=⌈t/B⌉, and one output group suffices whenever H≤B.
We are deeply grateful to the reviewer for these constructive questions.