acceptodds
Under review as a conference paper at ICLR 2027

Bootstrap Less, Infer Faster: Redesigning Transformers for Fully Homomorphic Inference

Abstract

Fully homomorphic encryption (FHE) enables non-interactive inference on encrypted inputs, but evaluating deep Transformers with CKKS remains costly due to repeated and expensive bootstrapping. Beyond optimizing packing and bootstrap placement within an existing computation, BLADE introduces three new, complementary FHE-specific techniques for attention, FFN, and LayerNorm that jointly reshape Transformer architecture and encrypted dataflow to reduce bootstrap inputs. First, ciphertext-aligned cyclic sparse attention maps each selected relative offset to one complete cyclic score diagonal, so unselected score ciphertexts are never computed or bootstrapped. Second, low-rank feed-forward networks bootstrap compact intermediates before expansion. Third, LayerNorm bootstraps only the packed variance ciphertext while retaining the centered values for subsequent normalization. The three techniques share an execution schedule that preserves sufficient multiplicative depth for retained and refreshed branches. We further develop a numerically robust, mean-centered homomorphic Softmax with normalization before and after squaring. On a complete 12-layer BERT-Base encoder, BLADE reduces the number of ciphertexts submitted to bootstrapping from 37,008 in MOAI to 17,448, a 2.12 reduction, and achieves 2.16 CPU and 2.00 GPU encoder speedups over MOAI. The adapted models score 1.46 metric points below BERT-Base on average across five GLUE tasks with exact nonlinearities. For the SST-2 checkpoint used in the runtime evaluation, CKKS execution matches the 90.02% accuracy of the same adapted checkpoint evaluated in plaintext with exact nonlinearities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully homomorphic encryption, secure transformer inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kgLqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on reducing the number of bootstrappings in FHE-based transformer inference. They proposed 1) cyclic sparse attention in SoftMax, 2) low-rank feedforward, 3) evaluation algorithm for LayerNorm. On a 12-layer BERT-base model, it achieved 2x reduction in bootstrapping and also 2x speedup over MOAI. The motivation is good, but the authors do not provide codes. I hope the authors can consider to open source their implementation in the furture.

Critical strengths
  1. The paper focuses on an important topic in privacy-preserving AI, and the motivation that reducing the number of bootstrappings is reasonable since it is the most expensive operation in FHE framework.

  2. The optimizations are carefully designed and easy to follow.

  3. The paper presents the results clearly.

Critical weaknesses
  1. The proposed techniques for SoftMax and FFN look specific to models, and they only provide results on BERT-base model, which cannot guarantee that the ideas work in other models.

  2. The baseline is limited. The paper only compared with MOAI, but there are lots of following works like “ARION: Attention-Optimized Transformer Inference on Encrypted Data”, which claims achieving 2.5x speedups over MOAI.

Questions for authors
  1. Ciphertext depth in LayerNorm. The paper propose a new evaluation method for LayerNorm to reduce the number of bootstrappings. But in Table 10, they only present that the depth of layernorm changes from 3 to 2 (or 2 to 1). Provide more details about how the ciphertext depth changes in each step of layernorm would be helpful.

  2. In the end-to-end framework design, the ciphertext is set to have 20 layers after bootstrapping. However, the higher layers are only used in Softmax evaluation. I think one possible improvement is to redesign the softmax evaluation and reduce the overall ciphertext depth, since the ciphertext depth will affect the time cost of bootstrapping.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kgLq (1/2)

Thank you for recognizing our motivation to reduce bootstrapping costs and for your positive assessment of the optimizations. We also appreciate your specific suggestions on applicability, the comparison with ARION, and the ciphertext level schedule.

W1: Applicability beyond BERT-Base

The proposed techniques for SoftMax and FFN look specific to models, and they only provide results on BERT-base model, which cannot guarantee that the ideas work in other models.

We have extended the plaintext evaluation to Llama-3-8B [1], which uses causal attention, SwiGLU, and RMSNorm. We trained dense Llama-3-8B and the BLADE variant under the same training configuration on SST-2, QQP, STS-B, CoLA, and RTE [2].

TaskMetricDense Llama-3-8BBLADE
SST-2Accuracy96.9095.99
QQPAccuracy92.2192.18
STS-BPearson/Spearman mean91.8391.96
CoLAMCC66.7666.53
RTEAccuracy90.2586.64

BLADE closely matches the dense model's plaintext scores on QQP, STS-B, and CoLA.

BLADE retains 112 causal offsets per head and uses a shared rank of 3,072 for the SwiGLU gate and up projections. We refresh their shared low-rank representation before expansion and bootstrap only the packed mean-square statistics within RMSNorm.

For 128-token inputs and 256 interleaved inputs per batch, both schedules bootstrap to level 20. The dense schedule retains MOAI's 19-level RMSNorm circuit, with full-width refreshes before and after each RMSNorm. BLADE reduces the static bootstrap-input count by 53.0%, from 16,416 in the dense schedule to 7,714 ciphertexts per layer per batch.

With Llama's pre-normalization structure and this level schedule, each BLADE block starts and ends at level 6. It therefore avoids the full-width layer-output bootstrap retained in our BERT-Base implementation.

W2: Comparison with ARION

The baseline is limited. The paper only compared with MOAI, but there are lots of following works like "ARION: Attention-Optimized Transformer Inference on Encrypted Data", which claims achieving 2.5x speedups over MOAI.

We measured ARION's encoder runtime on BERT-Base using the same HE library as BLADE and MOAI, retaining ARION's level-14 bootstrapping configuration. BLADE's total encoder runtime is 21.5% lower than ARION's. We chose MOAI for its optimized ciphertext packing and bootstrapping placement. It refreshes the 768-dimensional hidden representation before FFN expansion.

MetricBLADEMOAIARION
Ciphertexts submitted to bootstrapping (per batch)17,44837,00837,008
Bootstrapping time (s/input)47.71103.9968.81
Other operations (s/input)43.8979.5247.84
Total encoder time (s/input)91.61183.51116.65

ARION's reported 2.5× speedup includes backend and depth differences alongside attention optimizations. ARION uses Lattigo, whereas MOAI uses SEAL, and their multiplicative-depth configurations differ. Appendix G of the same paper [3] also reports faster single-threaded bootstrapping in Lattigo under matched cryptographic parameters.

In plaintext polynomial evaluation on all 872 SST-2 examples [2], ARION's released Base approximation profile [3] produced non-finite values on both the Dense BERT-Base and BLADE checkpoints. Its approximations therefore require recalibration for these models. If maintaining the target accuracy requires higher-degree polynomials or more iterations, the 14-level budget and execution cost need to be reevaluated.

References are listed in Part 2.

AuthorsOpenReview ↗

Response to Reviewer kgLq (2/2)

Q1: Step-by-step LayerNorm levels

Provide more details about how the ciphertext depth changes in each step of layernorm would be helpful.

The 3→2 and 2→1 entries in Table 10 in the paper give the input and output levels of LN1 and LN2. The final multiplication and rescaling produce this one-level drop. Each LayerNorm bootstraps only one packed variance ciphertext to level 20 and evaluates the inverse square root on that branch.

In the other branch, the ciphertexts encoding the scaled centered values zi=dxi−∑jxjz_i=d x_i-\sum_j x_j remain at level 3 for LN1 and level 2 for LN2 until the branches merge. The following table shows the level at each step for encoder layer 0 in the CPU implementation.

StepLN1LN2
Sum the squares of the scaled centered values3→22→1
Multiply by d−3d^{-3} to obtain the variance2→11→0
Align and bootstrap variance1→0→200→20
Add ϵ\epsilon20→2020→20
Approximate the inverse square root20→14→620→16→10
Multiply by γi/d\gamma_i/d, then align to the centered-value branch6→5→310→9→2
Multiply the two branches and rescale3→22→1

In the inverse square root step, the first transition comes from evaluating the Chebyshev seed, and the second comes from Newton refinement. LN1 uses a degree-13 seed and four Newton iterations, and LN2 uses a degree-3 seed and three Newton iterations.

Q2: Reducing the Softmax level budget

I think one possible improvement is to redesign the softmax evaluation and reduce the overall ciphertext depth, since the ciphertext depth will affect the time cost of bootstrapping.

Based on the static level count in Table 10, the score bootstrap could restore level 19 instead of level 20. The low-rank FFN projection would then end at level 0. The measurements in Table 1 in the paper use the 20-level configuration so that BLADE and MOAI are compared with the same CKKS parameters and bootstrapping implementation.

We agree that redesigning Softmax could further reduce the level budget. However, the input ranges encountered in real tasks make accurate exponential and reciprocal approximation difficult, often requiring substantial multiplicative depth. Our current Softmax circuit uses mean centering and normalization before and after squaring to maintain accuracy while limiting this depth.

Bootstrapping before the first reciprocal could reduce the level budget required by the score ciphertexts without changing the approximation polynomials. We could bootstrap the denominator ciphertext for each attention head, evaluate the reciprocal with the restored levels, and align its result with the retained exponential outputs.

If we keep the existing bootstrap for Softmax statistics, this option would add 12 ciphertexts to be bootstrapped per layer. We would assess whether restoring fewer levels for the 288 score ciphertexts saves enough time to offset this added cost. We would also verify that inference accuracy is maintained after the added bootstrapping error propagates through reciprocal evaluation.

Code availability

If the paper is accepted, we will release the BLADE implementation as soon as we receive the acceptance notification.

References (Parts 1–2)

[1] Grattafiori, Aaron, et al. “The Llama 3 Herd of Models.” arXiv preprint arXiv:2407.21783 (2024).

[2] Wang, Alex, et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” ICLR (2019).

[3] Yang, Linhan, et al. “ARION: Attention-Optimized Transformer Inference on Encrypted Data.” Cryptology ePrint Archive, Report 2025/2271 (2025).

We hope the comparison with ARION and the details of the ciphertext level schedule address your concerns. We would be happy to follow up on any remaining questions during the discussion.

Reviewer_tpQpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes BLADE, it combines cyclic sparse attention, low-rank FFNs, and variance-only LayerNorm bootstrapping to reduce bootstrap inputs. It reports approximately 2x CPU/GPU speedups over MOAI on an adapted 12-layer BERT-Base encoder in a 256-input batched configuration.

Critical strengths
  • The paper presents a concrete integration of architectural changes and selective refreshing, with explicit bootstrap accounting and a detailed execution schedule (Section 3, Figure 1, and Appendix F).
  • The complete-encoder CPU/GPU evaluation and controlled Softmax comparison provide useful implementation and numerical evidence (Table 1 and Appendix E).
Critical weaknesses

My main concern is whether the demonstrated FHE benefit justifies this particular customized, lossy model. The current evaluation leaves three gaps in that argument.

  1. The architectural tradeoff is not established. The runtime attention mask is a cyclic sliding window (Appendix D.2), implemented using established diagonal packing, while the FFN uses low-rank factorization. The value of this combination therefore needs to be demonstrated through its accuracy–encrypted-cost tradeoff. Once model adaptation and quality loss are accepted, why prefer this design over reducing depth or width, or using smaller models like DistilBERT or TinyBERT? There can be many approaches that reduce computation given the task is simple in both plaintext inference and encryepted inference. The starting point of this paper, reducing overhead through lossy methods, is intuitive; however, the fact that no other approaches were considered makes this design choice feel somewhat deliberate or contrived to the proposed method. Meanwhile, F1 does not evaluate the variance-only LayerNorm optimization on an unchanged dense model, leaving unclear how much benefit actually requires the lossy architectural changes.
  2. The baseline choice leaves open whether the advantage depends on MOAI's particular design. Section 1 presents the approach as complementary to existing bootstrap-placement optimizations, but the reported speedup is measured only against MOAI [1], whose full-width refresh schedule provides a clear opportunity for the proposed compact representations (Table 1(c)). The paper does not show whether the redesign remains worthwhile against other competitive dense implementations, such as THOR [2] or Cerium [3]. This is consequential because obtaining the reported gain requires a separately adapted model with reduced attention connectivity and FFN rank. Comparisons should also distinguish single-request latency from amortized throughput.
  3. D1 does not provide a convincing dense-attention reference. It computes and bootstraps all 128 score diagonals, then uses only 24 (Section 4.3). MOAI instead performs no score-diagonal bootstrapping and refreshes 768 attention-output ciphertexts per layer (Table 1(c)). The reported 5.41x operation slowdown therefore measures work on discarded values under BLADE's schedule; it does not establish the advantage of the sparse architecture over efficiently scheduled dense attention.

A further limitation is that the advertised average 1.46-point utility gap does not isolate the cost of the architecture: Appendix D.3 explicitly acknowledges unmatched training procedures. The encrypted prediction-agreement result also uses a separate SST-2 checkpoint with an additional classifier refit that upweights misclassified or low-margin training examples.

Questions for authors
  1. Under comparable adaptation resources and matched task utility, what advantage remains over both an optimized dense model incorporating the applicable LayerNorm/dataflow improvements and a conventionally compressed model?
  2. How do exact accuracy and exact-versus-CKKS prediction agreement change before and after the classifier refit in Appendix D.3?
  3. How do the authors expect the fixed attention-offset budget, FFN ranks, and adaptation process to scale beyond the evaluated BERT/GLUE setting—for example, to larger models or more demanding tasks such as long-context or multi-hop QA? Would preserving accuracy require increasingly model- or task-specific redesign and retraining, and does it worth it? This is a discussion-only question without additional experiments or revision.

References:

[1] Linru Zhang, Xiangning Wang, Jun Jie Sim, Zhicong Huang, Jiahao Zhong, Huaxiong Wang, Pu Duan, and Kwok-Yan Lam. MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference. ICLR, 2026.

[2] Jungho Moon, Dongwoo Yoo, Xiaoqian Jiang, and Miran Kim. THOR: Secure Transformer Inference with Homomorphic Encryption. ACM CCS, 2025.

[3] Siddharth Jayashankar, Joshua Kim, Michael B. Sullivan, Wenting Zheng, and Dimitrios Skarlatos. A Scalable Multi-GPU Framework for Encrypted Large-Model Inference. 2025.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tpQp (1/2)

Thank you for your feedback on task utility, adaptation costs, and comparisons with optimized dense implementations and conventionally compressed models.

W1 / Q1: Comparison with an optimized dense model and conventional compression

Under comparable adaptation resources and matched task utility, what advantage remains over both an optimized dense model incorporating the applicable LayerNorm/dataflow improvements and a conventionally compressed model?

Conventional compression. Compared with a shallower or narrower dense model, BLADE retains the original encoder's depth, hidden width, and nonlinear operations in each layer. Studies of plaintext long-context models show that, on some tasks, larger sparse models can outperform smaller dense models at comparable compute or memory-access budgets [1]. BLADE can also complement conventional compression by creating narrow bootstrapping points within the layers of TinyBERT and DistilBERT. In plaintext inference, experiments in DRONE [2] show that applying low-rank factorization to TinyBERT maintains its SST-2 accuracy and speeds up CPU inference by a further 1.31×.

Dense model with variance-only LayerNorm. For Dense BERT-Base, we statically analyzed a candidate schedule with BLADE's variance-only LayerNorm optimization. The analysis used our circuits and a 20-level budget. Across all 12 encoder layers, it submits 27,816 ciphertexts to bootstrapping per batch, compared with 17,448 in BLADE. BLADE still requires 37.3% fewer bootstrap input ciphertexts than this dense candidate with the same variance-only LayerNorm optimization.

This candidate still requires full-width bootstrapping at both the attention output and the FFN input. Without score bootstrapping, the dense attention output is at level 1, below the two levels needed by LN1's variance branch. Even after refreshing the attention output, LN1 reaches at most level 10, while the dense FFN needs level 11 to reach LN2. Thus, the attention and FFN redesign reduces bootstrap inputs beyond the savings from variance-only LayerNorm in this dense candidate.

W2: Baseline choice, THOR, Cerium, and timing scope

The baseline choice leaves open whether the advantage depends on MOAI's particular design. The paper does not show whether the redesign remains worthwhile against other competitive dense implementations, such as THOR or Cerium. Comparisons should also distinguish single-request latency from amortized throughput.

MOAI already selects relatively narrow bootstrapping points within its dense graph [3]. DaCapo similarly prioritizes candidate points with fewer ciphertexts requiring bootstrapping [4]. MOAI also reports a runtime on a single A100 52.8% below the runtime reported for THOR [3], supporting our choice of baseline.

BLADE and MOAI use the same packing, under which full attention scores and expanded FFN representations occupy 1,536 and 3,072 ciphertexts per layer. MOAI refreshes 768 ciphertexts at each of two points: the attention output and FFN input [3]. BLADE further reduces bootstrap inputs by creating narrow bootstrapping points absent from this dense graph.

BLADE's reductions in bootstrap inputs can complement THOR's matrix multiplication optimizations and Cerium's compiler and runtime optimizations. For the FFN, both public BERT implementations bootstrap after expansion, before GELU [5, 6]. BLADE can instead refresh the low-rank intermediate before expansion. The FFN can retain THOR's packing for the hidden representation and its block matrix multiplication, or Cerium's packing by feature blocks.

For attention, THOR can retain its slot layout and bootstrap the groups containing selected score diagonals [5]. Cerium's DSL can express compactly packed sparse attention and the redesigned FFN for execution with its existing compiler and runtime optimizations [6].

Table 1 in the paper reports amortized encoder runtime per input under the same 256-input interleaved batching configuration for BLADE and MOAI.

W3: Interpretation of D1

D1 does not provide a convincing dense-attention reference.

D1 is not intended to establish that the sparse architecture outperforms efficiently scheduled dense attention. It answers a specific question: If the schedule requires restoring levels before Softmax, how much does narrowing that bootstrapping point save? D1 keeps the bootstrapping location and downstream attention computation fixed to isolate the savings from narrowing the score representation (Table 2 in the paper).

BLADE aims to reduce the bootstrapping cost of the full encoder. Table 1 reports 2.16× CPU and 2.00× GPU speedups over MOAI for the complete encoders under their respective schedules. The total number of ciphertexts submitted to bootstrapping per batch drops from 37,008 in MOAI to 17,448 in BLADE, a reduction of approximately 52.9% (Table 1(c)).

References are listed in Part 2.

AuthorsOpenReview ↗

Response to Reviewer tpQp (2/2)

Q2: Before and after classifier refitting

How do exact accuracy and exact-versus-CKKS prediction agreement change before and after the classifier refit in Appendix D.3?

On SST-2 [7], the refit increases plaintext accuracy with exact nonlinearities from 89.56% to 90.02%. Classifier refitting was not required for exact-versus-CKKS prediction agreement. Before and after the refit, end-to-end CKKS predictions match the corresponding exact-nonlinearity plaintext predictions on every example. We used the refit only to reach our preset 90% plaintext accuracy target for the encrypted experiments.

Q3: Applicability beyond BERT/GLUE and adaptation cost

How do the authors expect the fixed attention-offset budget, FFN ranks, and adaptation process to scale beyond the evaluated BERT/GLUE setting—for example, to larger models or more demanding tasks such as long-context or multi-hop QA? Would preserving accuracy require increasingly model- or task-specific redesign and retraining, and is it worth it?

We have extended the plaintext evaluation to Llama-3-8B. The BLADE variant uses 112 causal offsets per head and a shared rank of 3,072 for the SwiGLU gate and up projections. We trained dense Llama-3-8B and BLADE under the same training configuration on SST-2, QQP, STS-B, CoLA, and RTE [7].

TaskMetricDense Llama-3-8BBLADE
SST-2Accuracy96.9095.99
QQPAccuracy92.2192.18
STS-BPearson/Spearman mean91.8391.96
CoLAMCC66.7666.53
RTEAccuracy90.2586.64

BLADE closely matches the dense model's plaintext scores on QQP, STS-B, and CoLA.

Relative to the dense schedule retaining MOAI's RMSNorm circuit, the static bootstrap-input count decreases from 16,416 to 7,714 ciphertexts per layer per batch, a 53.0% reduction. Both schedules use 128-token inputs, 256 interleaved inputs per batch, and bootstrapping to level 20. For other backbones, we can initialize from dense weights and select FFN ranks according to each layer's sensitivity to low-rank truncation and the target task utility [8].

For long contexts and multi-hop tasks, the arrangement of offsets determines whether evidence from distant locations can be brought together within a limited number of layers. We can replace some local offsets with fixed long-range offsets in selected layers or heads to shorten information paths across segments [9]. At a fixed offset budget, this replacement leaves the number of score ciphertexts bootstrapped per head unchanged. For multi-hop tasks, fine-tuning or distillation on target-task data can help the model combine evidence along these paths. We can then evaluate task utility at the target context length and number of reasoning hops [10]. The results can guide whether to increase the offset budget.

For task-specific adaptation, we reused the same FFN rank schedule and budget of 24 offsets per head in our BERT-Base experiments on SST-2, QQP, STS-B, CoLA, and RTE [7]. QQP, STS-B, and CoLA also used the same offset table. We fine-tune the model separately for each task and can reuse the resulting checkpoint for subsequent encrypted inference. Adaptation is a one-time cost, while the reduction in bootstrap inputs applies to every subsequent inference.

References (Parts 1–2)

[1] Nawrot, Piotr, et al. “The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs.” Findings of ACL (2026).

[2] Chen, Patrick H., et al. “DRONE: Data-aware Low-rank Compression for Large NLP Models.” NeurIPS (2021).

[3] Zhang, Linru, et al. “MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference.” ICLR (2026).

[4] Cheon, Seonyoung, et al. “DaCapo: Automatic Bootstrapping Management for Efficient Fully Homomorphic Encryption.” USENIX Security (2024).

[5] Moon, Jungho, et al. “THOR: Secure Transformer Inference with Homomorphic Encryption.” ACM CCS (2025).

[6] Jayashankar, Siddharth, et al. “Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference.” arXiv preprint arXiv:2512.11269v2 (2026).

[7] Wang, Alex, et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” ICLR (2019).

[8] Yuan, Zhihang, et al. “ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models.” arXiv preprint arXiv:2312.05821 (2023).

[9] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. “Longformer: The Long-Document Transformer.” arXiv preprint arXiv:2004.05150 (2020).

[10] Hsieh, Cheng-Ping, et al. “RULER: What's the Real Context Size of Your Long-Context Language Models?” arXiv preprint arXiv:2404.06654 (2024).

We hope our responses clarify the scope of our comparisons and how BLADE can complement existing optimizations. We would be happy to elaborate on any remaining points during the discussion.

Reviewer_wU9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BLADE combines cyclic sparse attention, low-rank FFNs, and revised nonlinear circuits to reduce bootstrapping in a MOAI-style BERT-base pipeline. Bootstrap input ciphertexts fall from 37,008 to 17,448, yielding 2.16x CPU and 2.00x GPU speedups under matched parameters and hardware.

Critical strengths
  1. The efficiency comparison controls CKKS parameters, bootstrap implementation, hardware, batching, and timing scope. Bootstrap counts and targeted ablations support the reported savings.
  2. All 872 SST-2 encrypted predictions match the exact-nonlinearity plaintext reference on CPU and GPU. The isolated softmax comparison reduces accuracy loss from 5.08 to 0.30 percentage points.
Critical weaknesses
  1. The architectural accuracy cost is not isolated: task-specific training recipes and development-set selection confound the comparison. Encrypted evaluation covers only SST-2, where the runtime checkpoint achieves 90.02%, 2.41 percentage points below the reported BERT-base result.
  2. Most components are established, and the gain is about 2x over one baseline. GELU becomes roughly 10x more expensive, partly offsetting the savings, while the largest layer-output bootstrap cost remains unchanged.
Questions for authors
  1. Clarification: Can you compare BLADE and dense BERT-base with identical training recipes and teachers on at least two tasks, and report MOAI accuracy in your setting?
  2. Clarification: Can you provide end-to-end encrypted accuracy on another task, preferably RTE?
  3. Clarification: How sensitive is the result to the special fine-tuning and reweighted L-BFGS refit? Were approximation intervals fitted using training data only?
2 comments
AuthorsOpenReview ↗

Response to Reviewer wU9f (1/2)

Thank you for your constructive feedback and for recognizing our efficiency comparison and our verification of prediction agreement between encrypted and plaintext inference.

Q1 / W1: Matched-training comparisons and MOAI accuracy

Can you compare BLADE and dense BERT-base with identical training recipes and teachers on at least two tasks, and report MOAI accuracy in your setting?

Following your suggestion, we fine-tuned BLADE and Dense BERT-Base on SST-2, QQP, STS-B, CoLA, and RTE from GLUE [1] under identical training recipes and teachers. For each task, both models used the same training data, loss function, training budget, and checkpoint selection rule. Dense BERT-Base was fine-tuned directly from the full pretrained checkpoint. BLADE inherited the dense weights and was fine-tuned after the architectural changes. The resulting development-set scores are shown below.

TaskMetricDense BERT-BaseBLADE
SST-2Accuracy93.1290.48
QQPAccuracy91.3389.54
STS-BPearson/Spearman mean90.1985.09
CoLAMCC57.0154.72
RTEAccuracy76.1770.76

Table 3 in the paper uses the original Dense BERT-Base scores to show how closely BLADE can match them after architectural changes and fine-tuning. MOAI preserves the architecture and weights of Dense BERT-Base, so its plaintext accuracy with exact nonlinearities is unchanged. Any accuracy differences introduced by MOAI arise from polynomial approximation and CKKS numerical error [2].

For the MOAI accuracy check, we evaluated Dense BERT-Base from Table 3 on all 872 SST-2 development-set examples. Its plaintext accuracy with exact nonlinearities was 92.43%. With MOAI's released implementation and original nonlinear approximation configuration, both plaintext polynomial evaluation and end-to-end CKKS inference produced non-finite values on all 872 examples.

W2: Contribution, higher GELU cost, and layer-output bootstrapping

Most components are established, and the gain is about 2x over one baseline. GELU becomes roughly 10x more expensive, partly offsetting the savings, while the largest layer-output bootstrap cost remains unchanged.

Diagonal packing and low-rank factorization are established, but smaller intermediate representations do not automatically reduce bootstrap inputs. BLADE creates new narrow bootstrapping points where compact representations occupy fewer ciphertexts. We design their packing and CKKS level schedule together so that the refreshed values can support subsequent computation.

  • In attention, we align the sparse attention pattern with cyclic score diagonals to avoid bootstrapping ciphertexts for unselected diagonals.
  • In the FFN, we bootstrap the low-rank intermediate between the two linear factors.
  • In LayerNorm, we bootstrap only the packed variance ciphertext.

We chose MOAI as the dense baseline because it already optimizes ciphertext packing and schedules bootstrapping on relatively compact representations within its dense computation graph. For example, MOAI bootstraps 768 ciphertexts at each of two points, the attention output and the FFN input.

The increase in GELU cost comes primarily from the higher polynomial degrees. MOAI retains the degree-24 power polynomial from its released implementation, whereas BLADE uses layer-specific Chebyshev polynomials of degrees 28–120. The approximation intervals vary substantially across layers; for example, layer 10 uses approximately [−35.12, 95.48] (Table 9 in the paper). We select the degree for each interval to keep the approximation error small across the layer's input range. The higher degrees therefore reflect the accuracy requirements of these wider intervals.

The current schedule requires full-width layer-output bootstrapping because the second LayerNorm outputs ciphertexts at level 1, while the next encoder layer expects inputs at level 8 (Table 10 in the paper). BLADE's goal is to reduce the total number of bootstrap input ciphertexts across the full encoder. With layer-output bootstrapping included, this total falls from 37,008 ciphertexts per batch with MOAI to 17,448 with BLADE over all 12 layers (Table 1(c) in the paper). With both the higher GELU cost and layer-output bootstrapping included, BLADE achieves 2.16× CPU and 2.00× GPU encoder speedups over MOAI.

To reduce this remaining cost, we will explore projecting the layer output to a low-rank representation, bootstrapping it, and reconstructing the hidden representation before the next layer. Unlike factorizing an existing FFN projection, this introduces a bottleneck between encoder layers. We will evaluate whether it can further reduce bootstrap inputs while maintaining accuracy. Thank you for highlighting this remaining cost.

References are listed in Part 2.

AuthorsOpenReview ↗

Response to Reviewer wU9f (2/2)

Q2: Encrypted accuracy on RTE

Can you provide end-to-end encrypted accuracy on another task, preferably RTE?

Following your suggestion, we extended end-to-end CKKS evaluation of BLADE to RTE, achieving 70.40% accuracy on 277 samples. Plaintext evaluation with exact nonlinearities, using the original functions without polynomial approximation, achieved 70.76% accuracy. Plaintext polynomial evaluation also achieved 70.76% accuracy.

Q3: Fine-tuning, classifier refitting, and approximation-interval calibration

How sensitive is the result to the special fine-tuning and reweighted L-BFGS refit? Were approximation intervals fitted using training data only?

The result is not sensitive to the special fine-tuning used for runtime evaluation. We performed end-to-end CKKS evaluation of the BLADE model from Table 3 in the paper on SST-2. This model used a different fine-tuning recipe from the runtime checkpoint. Its CKKS accuracy matched the 90.48% accuracy of the plaintext reference with exact nonlinearities.

Reweighted L-BFGS classifier refitting was not required for CKKS prediction agreement. Before and after refitting, CKKS predictions matched the exact-nonlinearity plaintext reference on every example. For the runtime checkpoint, refitting was used only to reach our preset 90% plaintext accuracy target for the encrypted experiments. It increased plaintext accuracy with exact nonlinearities from 89.56% to 90.02%.

The original approximation intervals were fitted using unlabeled activation statistics from SST-2 development inputs for the CKKS agreement check. We recalibrated the Softmax, LayerNorm, and GELU intervals using only training data and fixed them before development-set evaluation. The approximation methods, polynomial degrees, Newton iteration counts, and evaluation paths were unchanged. In end-to-end plaintext polynomial evaluation, the new intervals preserved all 872 predictions of the exact-nonlinearity plaintext reference, with 785/872 correctly classified.

References (Parts 1–2)

[1] Wang, Alex, et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.” ICLR (2019).

[2] Zhang, Linru, et al. “MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference.” ICLR (2026).

We hope the additional comparisons and encrypted evaluation results address your questions. We would be happy to clarify any remaining points during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.