- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ELLMo, a secure transformer inference framework via fully homomorphic encryption. Specifically, ELLMo proposes ShallowPCMM to reduces rotations and weight memory, and automates parameter selection for nonlinear operation to minimize the number of bootstrapping. For BERT-Base and Llama-3-8B, the paper reported a 1.1x to 12x speedup and 3.2x to 5.2x less weight memory occupation on a A100 GPU.
Critical strengths
-
This paper raise the GPU memory bottleneck of FHE, which is a critical issue for practical FHE-based tranformer inference that single GPU memory is limited. The strategy is interesting that place all the PCMM at lowest level if the bootstrapping is fast enough.
-
The implementation of sparse encoding and corresponding kernel fusion is a useful engineering contribution. The authors also promise to release their source code upon publication.
Critical weaknesses
-
The main concern is algorithmic novelty of ShallowPCMM that it appears closely relate to standard diagonal decomposition of a plaintext linear transformation under row-major packing. And techniques such as BSGS, hoisting, and sparse plaintext encoding have already been explored in prior work. For the amount of KeySwitch operations in Table 2, where I don't see the advantage of ShallowPCMM over optimized diagonal-BSGS PCMM but the author claims it reduces rotations. The permanformance gains comes from low level and other techniques which should be also applicable to other baselines. Apart from that, ShallowPCMM uses 2 multplicative levels whereas a standard plaintext linear transformation only need 1 level usually. Futhermore, a comparison against baselines with the same optimizations for ShallowPCMM is needed.
-
The second concern is that ELLMo places all PCMM at the lowest level, which means it forces bootstrapping. As the bootstrapping is still the slowest operation in FHE, it is unfair to compare matrix multiplication without denoting the level and further influence of following computation. In addition, the bootstrapping placement, the multiplicative level change and non-linear approximation level consumption are unclear to me, which make it hard to assess the efficiency improvement of ELLMo without the code closure. The improvements currently is not well separated and not distinguished from those of existing techniques, such as hoisting, sparse encoding, and GPU kernel fusion, which make improvement seems like a combination of engineering optimization instead of proposed ShallowPCMM.
Questions for authors
-
Could the author clarify the advantage and fundamental difference between ShallowPCMM and diagonal-BSGS PCMM with precomputed plaintext diagonals, and compare it to baselines on the same level and applying the same techniques?
-
Could the authors provide a detailed breakdown of ciphertext levels, bootstrap placements and non-linear approximation consumption for BERT-Base and LLama?
-
Appendix A.3.3 states that decoding uses BSGS matrix-vector products instead of ShallowPCMM. Could the authors clarify the reported decoding performance gains with ELLMo?
4 comments
Thank you for the clarification and additional result.
My concerns about i) fair comparison to separate SPCMM from bootstrapping placement and other optimization settings, ii) no fundamental advantage or novelty of SPCMM over optimized diagonal-BSGS PCMM, iii) what does 8B Prefill (16.95 and 16.98) stand for in full results table? they remains unresolved.
Also, the main papre's Table 4 present ELLMo as 50.2–57.4s lantency, and here is 44.07s. If this improvement comes from further optimizations, the updated results still do not isolate the contribution of SPCMM, making it even harder to assess its actual benefit.
I appreciate the additional clarification, since my primary concerns remains unresovled, I will maintain my original score.
We thank the reviewer for the reply.
i) Isolating SPCMM. Table 8 in the paper (repeated below) incrementally adds one technique per row within our library. Each product is timed independently at Level 2, ensuring the impact of SPCMM is isolated from bootstrapping placement. For the optimization settings, we apply standard techniques such as hoisting and kernel fusion. A comparison with prior art using this same library is provided in Part (ii). The "+ SPCMM" row introduces only SPCMM, prior to the application of sparse encoding, hoisting, or kernel fusion. By itself, this technique cuts latency by 1.6 to 2.6 due to a reduction in ciphertext-side transforms.
| Variant (#KS / ms) | ×W_Q/K/V | ×W_O | ×W_FFN1 | ×W_FFN2 |
|---|---|---|---|---|
| JKLS, complex packing | 2520 / 1505 | 840 / 502 | 3360 / 1986 | 2496 / 1481 |
| + SPCMM | 2358 / 913 | 426 / 241 | 3144 / 1208 | 642 / 567 |
| + Sparse encoding | 2358 / 702 | 426 / 172 | 3144 / 928 | 642 / 359 |
| + Hoisting output tiles | 198 / 280 | 110 / 109 | 222 / 355 | 186 / 262 |
| + Kernel fusion | 198 / 76 | 110 / 36 | 222 / 90 | 186 / 71 |
ii) Comparison with diagonal-BSGS. We reimplemented THOR (diagonal-BSGS), Powerformer, and Tricycle in FIDESlib, verified each against its paper's key-switch counts, and ran each at its lowest level on the same A100. Each implementation used the efficient FIDESlib kernels for dot products and key-switches. Per BERT-Base layer:
| System | Total #KS | Total (s) | ELLMo speedup |
|---|---|---|---|
| THOR | 1593 | 2.385 | 4.7x |
| Powerformer | 1195 | 1.435 | 2.8x |
| Tricycle | 2188 | 2.776 | 5.4x |
| ELLMo | 1759 | 0.510 |
ELLMo does not issue the fewest keyswitches. Compared to THOR, its benefit lies in lower operational levels: THOR requires 4-5 limbs to start every PCMM, whereas ELLMo requires 2. Compared to Powerformer, all additional keyswitches in ELLMo occur in the baby steps of the BSGS, which are hoisted and therefore incur less cost than a regular rotation. For the second most expensive CKKS primitive, Rescale, ELLMo issues 7.4x fewer operations than Powerformer. This is because ELLMo’s plaintext operations are done offline and stored sparsely at no additional cost due to its repetitive diagonal packing. This is also reflected in the PtMult (plaintext x ciphertext element-wise multiplication) kernel calls. Tricycle also runs at the same low levels as ELLMo, 2 limbs per PCMM, but issues more keyswitches. Although we reimplemented Tricycle based on their published paper, we could not find an open-source GPU implementation; therefore, our implementation may suffer from suboptimalities due to the inaccessibility of all their optimizations.
iii) 8B Prefill column. The numbers in the Llama-3-8B column stand for the WikiText-2 perplexities of Llama-3-8B (teacher-forced, 128-token windows). Encrypted is 0.18% above plaintext.
iv) Updated latency. The drop from 50.2 s to 57.4 s to 44 s to 47 s comes from the post-submission optimizations described in our response to Reviewer zXSS. We kindly ask the reviewer to see our response above.
Thank you for your additional experiment. However, (i) hoisting seems contribute more than SPCMM, the ablation is incomplete. (ii) The advantage cannot be simply attributed to limbs number consumption in case SPCMM don't contribute to it. (iii) diagonal-BSGS consumes 1 level while ELLMo consumes 2 levels, lower operational levels advantage is unconvincing.
The experiments result seems unconvincing for me and my concerns remain unresolved. Therefore, I will maintain my original score.
We thank the reviewer for the detailed review.
We present our full results below.