- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work focuses on reducing the number of bootstrappings in FHE-based transformer inference. They proposed 1) cyclic sparse attention in SoftMax, 2) low-rank feedforward, 3) evaluation algorithm for LayerNorm. On a 12-layer BERT-base model, it achieved 2x reduction in bootstrapping and also 2x speedup over MOAI. The motivation is good, but the authors do not provide codes. I hope the authors can consider to open source their implementation in the furture.
Critical strengths
-
The paper focuses on an important topic in privacy-preserving AI, and the motivation that reducing the number of bootstrappings is reasonable since it is the most expensive operation in FHE framework.
-
The optimizations are carefully designed and easy to follow.
-
The paper presents the results clearly.
Critical weaknesses
-
The proposed techniques for SoftMax and FFN look specific to models, and they only provide results on BERT-base model, which cannot guarantee that the ideas work in other models.
-
The baseline is limited. The paper only compared with MOAI, but there are lots of following works like “ARION: Attention-Optimized Transformer Inference on Encrypted Data”, which claims achieving 2.5x speedups over MOAI.
Questions for authors
-
Ciphertext depth in LayerNorm. The paper propose a new evaluation method for LayerNorm to reduce the number of bootstrappings. But in Table 10, they only present that the depth of layernorm changes from 3 to 2 (or 2 to 1). Provide more details about how the ciphertext depth changes in each step of layernorm would be helpful.
-
In the end-to-end framework design, the ciphertext is set to have 20 layers after bootstrapping. However, the higher layers are only used in Softmax evaluation. I think one possible improvement is to redesign the softmax evaluation and reduce the overall ciphertext depth, since the ciphertext depth will affect the time cost of bootstrapping.
2 comments
Response to Reviewer kgLq (2/2)
Q1: Step-by-step LayerNorm levels
Provide more details about how the ciphertext depth changes in each step of layernorm would be helpful.
The 3→2 and 2→1 entries in Table 10 in the paper give the input and output levels of LN1 and LN2. The final multiplication and rescaling produce this one-level drop. Each LayerNorm bootstraps only one packed variance ciphertext to level 20 and evaluates the inverse square root on that branch.
In the other branch, the ciphertexts encoding the scaled centered values remain at level 3 for LN1 and level 2 for LN2 until the branches merge. The following table shows the level at each step for encoder layer 0 in the CPU implementation.
| Step | LN1 | LN2 |
|---|---|---|
| Sum the squares of the scaled centered values | 3→2 | 2→1 |
| Multiply by to obtain the variance | 2→1 | 1→0 |
| Align and bootstrap variance | 1→0→20 | 0→20 |
| Add | 20→20 | 20→20 |
| Approximate the inverse square root | 20→14→6 | 20→16→10 |
| Multiply by , then align to the centered-value branch | 6→5→3 | 10→9→2 |
| Multiply the two branches and rescale | 3→2 | 2→1 |
In the inverse square root step, the first transition comes from evaluating the Chebyshev seed, and the second comes from Newton refinement. LN1 uses a degree-13 seed and four Newton iterations, and LN2 uses a degree-3 seed and three Newton iterations.
Q2: Reducing the Softmax level budget
I think one possible improvement is to redesign the softmax evaluation and reduce the overall ciphertext depth, since the ciphertext depth will affect the time cost of bootstrapping.
Based on the static level count in Table 10, the score bootstrap could restore level 19 instead of level 20. The low-rank FFN projection would then end at level 0. The measurements in Table 1 in the paper use the 20-level configuration so that BLADE and MOAI are compared with the same CKKS parameters and bootstrapping implementation.
We agree that redesigning Softmax could further reduce the level budget. However, the input ranges encountered in real tasks make accurate exponential and reciprocal approximation difficult, often requiring substantial multiplicative depth. Our current Softmax circuit uses mean centering and normalization before and after squaring to maintain accuracy while limiting this depth.
Bootstrapping before the first reciprocal could reduce the level budget required by the score ciphertexts without changing the approximation polynomials. We could bootstrap the denominator ciphertext for each attention head, evaluate the reciprocal with the restored levels, and align its result with the retained exponential outputs.
If we keep the existing bootstrap for Softmax statistics, this option would add 12 ciphertexts to be bootstrapped per layer. We would assess whether restoring fewer levels for the 288 score ciphertexts saves enough time to offset this added cost. We would also verify that inference accuracy is maintained after the added bootstrapping error propagates through reciprocal evaluation.
Code availability
If the paper is accepted, we will release the BLADE implementation as soon as we receive the acceptance notification.
References (Parts 1–2)
[1] Grattafiori, Aaron, et al. “The Llama 3 Herd of Models.” arXiv preprint arXiv:2407.21783 (2024).
[2] Wang, Alex, et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” ICLR (2019).
[3] Yang, Linhan, et al. “ARION: Attention-Optimized Transformer Inference on Encrypted Data.” Cryptology ePrint Archive, Report 2025/2271 (2025).
We hope the comparison with ARION and the details of the ciphertext level schedule address your concerns. We would be happy to follow up on any remaining questions during the discussion.
Response to Reviewer kgLq (1/2)
Thank you for recognizing our motivation to reduce bootstrapping costs and for your positive assessment of the optimizations. We also appreciate your specific suggestions on applicability, the comparison with ARION, and the ciphertext level schedule.
W1: Applicability beyond BERT-Base
We have extended the plaintext evaluation to Llama-3-8B [1], which uses causal attention, SwiGLU, and RMSNorm. We trained dense Llama-3-8B and the BLADE variant under the same training configuration on SST-2, QQP, STS-B, CoLA, and RTE [2].
BLADE closely matches the dense model's plaintext scores on QQP, STS-B, and CoLA.
BLADE retains 112 causal offsets per head and uses a shared rank of 3,072 for the SwiGLU gate and up projections. We refresh their shared low-rank representation before expansion and bootstrap only the packed mean-square statistics within RMSNorm.
For 128-token inputs and 256 interleaved inputs per batch, both schedules bootstrap to level 20. The dense schedule retains MOAI's 19-level RMSNorm circuit, with full-width refreshes before and after each RMSNorm. BLADE reduces the static bootstrap-input count by 53.0%, from 16,416 in the dense schedule to 7,714 ciphertexts per layer per batch.
With Llama's pre-normalization structure and this level schedule, each BLADE block starts and ends at level 6. It therefore avoids the full-width layer-output bootstrap retained in our BERT-Base implementation.
W2: Comparison with ARION
We measured ARION's encoder runtime on BERT-Base using the same HE library as BLADE and MOAI, retaining ARION's level-14 bootstrapping configuration. BLADE's total encoder runtime is 21.5% lower than ARION's. We chose MOAI for its optimized ciphertext packing and bootstrapping placement. It refreshes the 768-dimensional hidden representation before FFN expansion.
ARION's reported 2.5× speedup includes backend and depth differences alongside attention optimizations. ARION uses Lattigo, whereas MOAI uses SEAL, and their multiplicative-depth configurations differ. Appendix G of the same paper [3] also reports faster single-threaded bootstrapping in Lattigo under matched cryptographic parameters.
In plaintext polynomial evaluation on all 872 SST-2 examples [2], ARION's released Base approximation profile [3] produced non-finite values on both the Dense BERT-Base and BLADE checkpoints. Its approximations therefore require recalibration for these models. If maintaining the target accuracy requires higher-degree polynomials or more iterations, the 14-level budget and execution cost need to be reevaluated.
References are listed in Part 2.