acceptodds
Under review as a conference paper at ICLR 2027

OBELISK: Co-designing Post-Packing Execution for Efficient Transformer Inference under Fully Homomorphic Encryption

Abstract

Privacy-preserving Transformer inference under CKKS fully homomorphic encryption remains expensive in evaluation time and memory. Prior systems mainly optimize packing layouts and polynomial approximations of nonlinear operators. Packing fixes the logical dataflow, namely which values meet in which operations, but not how that dataflow is executed. Three post-packing decisions remain: the level budget, intermediate-ciphertext lifetime, and slot representation. To our knowledge, OBELISK is the first system to co-design all three for non-interactive secure Transformer inference on a commodity GPU, through budget-operator co-design, Process-in-Pieces (PiP), and Hybrid Real-Complex Representation (HRC). Budget-operator co-design shortens the nonlinear operators and lets a layer-level cost model choose the budget, cutting the layer's full-width bootstraps to two and reinvesting the freed modulus bits in cheaper key switching. PiP processes attention by head and the feed-forward network by chunk, avoiding full-width resident intermediates without a measured increase in evaluation time. HRC screens each operation through three criteria and runs it in the cheaper representation, real or complex, halving the work of the two full-width bootstraps and both ciphertext-ciphertext matmuls without adding a level. On fully encrypted 12-layer GLUE inference, OBELISK runs on a commodity 24 GiB GPU (an RTX 4090) that no evaluated baseline fits, and on the same H200 as the state of the art it reduces end-to-end evaluation time by 6.6x and peak memory by 6.0x (129.3 to 21.6 GiB), within 2.52 pp of plaintext accuracy. We report directly measured peak memory for all compared systems, which no evaluated prior system reports.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully homomorphic encryption, CKKS, privacy-preserving machine learning, secure inference, transformer, bootstrapping, GPU acceleration, memory efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rvopOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OBELISK, a system for efficient non-interactive Transformer inference under CKKS FHE. It jointly optimizes three execution decisions: the modulus-level budget and nonlinear operators, intermediate-ciphertext lifetimes through Process-in-Pieces (PiP), and real vs. complex representations through Hybrid Real-Complex Representation (HRC). The authors' evaluation covers full 12-layer BERT-base inference on four GLUE validation sets, multiple GPUs, and component-level profiling and ablations. The authors report a 6.6× reduction in amortized server evaluation time and a 6.0× reduction in peak GPU memory relative to MOAI on an H200. The deployed system also runs on a 24 GiB RTX 4090, with accuracy losses of up to 2.52 percentage points relative to its plaintext checkpoints.

Critical strengths
  1. The memory reduction makes encrypted inference feasible on more accessible hardware. The system reduces the working set enough to run encrypted BERT inference on a 24 GiB GPU, whereas the evaluated baselines require GPUs with substantially more memory. Memory breakdowns, attention and FFN granularity sweeps, and successful runs across multiple GPUs support this result. This is a meaningful improvement in deployment feasibility.
  2. The paper provides convincing evidence that the execution optimizations complement one another. The level schedules explain how the budget constrains tiling and representation choices. The PiP/HRC factorial ablations show that tiling substantially reduces memory with little runtime overhead, while selective complex pairing reduces evaluation time. Validation of the bootstrap cost model across two libraries further supports the design. Together, these results demonstrate useful opportunities to optimize execution after the packing layout has been chosen
Critical weaknesses
  1. The main performance comparison does not establish the speedup at comparable accuracy. Appendix F acknowledges that Table 1 compares computational cost without matching accuracy. OBELISK loses up to 2.52 percentage points relative to its plaintext model. Appendix H reports larger accuracy losses than THOR on RTE and MRPC, but these comparisons use different fine-tuned models and do not establish their relative accuracy under controlled conditions. A comparison on the same evaluation data, allowing each system its required model adaptation, would clarify how much of the speed advantage holds at comparable accuracy. This matters because the design uses a shorter modulus chain, and the authors attribute most of the remaining accuracy loss to homomorphic noise.

  2. The contribution of budget–operator co-design to the overall speedup is not fully isolated. The base system already includes fused plaintext–ciphertext multiplication, a cumulative rotation-key strategy, and cached bootstrap encoding. Table G.3 shows that these additional optimizations reduce execution time to 32% of the budget-only configuration in an instrumented two-layer experiment. Although the paper measures these improvements and separately evaluates PiP and HRC, it does not provide an end-to-end ablation of budget–operator co-design. Several alternative budgets in Table B.1 also have timing results but do not produce decryptable outputs. This leaves uncertainty about how much the budget and operator choices contribute to the total speedup and how alternative numerically valid configurations compare in accuracy and performance.

Questions for authors
  1. What accuracy does MOAI achieve on the evaluated tasks, and how does it compare with OBELISK? Without this information, how should we interpret the reported speedup in terms of the accuracy–performance tradeoff?

  2. Appendix B.1 reports timings for several alternative budgets that do not produce decryptable outputs. How did you account for numerical correctness when selecting the budget? Did you compare the selected configuration with other budgets that produce correct outputs, including those using fewer special-modulus primes?

  3. How does the 93.7-bit core-SVP estimate in Appendix A support the claim of 128-bit classical security for the deployed sparse secret? Please specify the estimator settings and the standard configuration used for comparison.

  4. How do OBELISK’s GELU, softmax, and LayerNorm approximations differ from MOAI’s in multiplicative depth and accuracy? How much of the reported speedup comes from these differences and their effect on bootstrapping, rather than from the other execution optimizations?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for the careful reading and the insightful comments.

Response to Q1/W1 - Speedup at comparable accuracy

The speedup holds at comparable accuracy. General Response 1 traces most of the earlier drop to implementation errors (miscalibration) rather than to noise on the shorter chain. After the fixes, OBELISK loses at most 0.36 pp, close to the losses MOAI and THOR report. MOAI and THOR use their own fine-tuned models, so we compare the accuracy each system loses under encryption. The table gives encrypted accuracy (%), with that loss in parentheses.

TaskOBELISKTHOR (reported)MOAI (reported)
SST-292.20 (−0.23)90.71 (−0.80)93.8 (−0.1)
QNLI91.62 (+0.07)—90.5 (−0.2)
RTE72.20 (−0.36)71.12 (−1.08)65.5 (−0.2)
MRPC88.24 (+0.49)84.80 (−0.49)—
MRPC (F1)91.67 (+0.32)89.49 (−0.41)—

In a controlled comparison on THOR's released MRPC model and the same 408 validation inputs, OBELISK is within one input of THOR at 17.9× its throughput on an RTX PRO 6000 (19.1× on an A100). We rebuilt only OBELISK's per-task constants from the MRPC training data. MOAI has not released its task models, and THOR has released only its MRPC model. THOR's accuracy and A100 time are as reported, and we measured the other times.

MRPC, THOR's modelPlaintextTHOROBELISK
Correct / 408348346345
Accuracy (%)85.2984.8084.56
F189.9089.4989.41
s / input, RTX PRO 6000—33018.45
s / input, A100—599.631.38

 

Response to Q2/W2 - The co-design ablation and alternative budgets

In General Response 2, the budget–operator co-design makes the baseline with all three base techniques 2.94× faster (row 3 to row 5). Of this, the operator choices give 1.68× and the budget 1.75×. The base techniques themselves give 1.42× over our MOAI rerun. Table G.3's 32% came from adding the base techniques to our cheaper budget in an instrumented two-layer run, where weight encoding takes a larger share.

Every alternative budget we re-ran gives the same predictions as the deployed (2, 16, 5), which remains the cheapest. Accuracy thus does not separate the budgets, so cost alone decides. We ran L = 17 to 21, and L = 16 with fewer special primes, through all 12 layers on 256 MRPC inputs.

The submitted note that budgets with L ≥ 17 do not decrypt was wrong. Each budget gets 224 right, against 223 for float64 plaintext. Budgets with L ≤ 15 need a third bootstrap per layer, so we excluded them. The revised 128-bit bound gives L = 19 and 20 one fewer special prime than the submission. Times are for one layer on an RTX PRO 6000. Every column is relative to the deployed budget. Cosine compares each input's last-layer [CLS] vector under the two budgets.

(B, L, α)Layer timeSame predictionsWorst-case cosine
(2, 21, 1)1.88×256 / 2560.99996
(2, 20, 1)1.81×256 / 2560.99998
(2, 19, 2)1.34×256 / 2560.99996
(2, 18, 3)1.16×256 / 2560.99994
(2, 17, 4)1.07×256 / 2560.99998
(2, 16, 5)1.00×——
(2, 16, 4)1.04×256 / 2560.99997
(2, 16, 3)1.09×256 / 2560.99999
(2, 16, 2)1.21×256 / 2560.99998
(2, 16, 1)1.55×256 / 2560.99988

 

Response to Q3 - 128-bit security

The deployed configuration now reaches 128.2 bits of security through adopting SSE, with less than 1% of runtime overhead. The 93.7 bits in Appendix A came from the estimator's rough core-SVP mode and do not support 128 bits, and the full estimator puts the submitted secret at about 121 bits. We appreciate this question, which led us to the fix. General Response 1 gives the estimator settings and the HE Standard's largest 128-bit point.

 

Response to Q4 - Operator choices against MOAI

Most of the operator gain comes from LayerNorm, which takes 2 levels instead of MOAI's 20. A layer then needs 30 levels instead of MOAI's 63, which halves the full-width bootstraps (row 4 of General Response 2, 1.68×) and allows the shorter chain (rows 4 to 5, 1.75×).

Each of our operators stays within 0.05 pp of the exact model's 90.68%. We use the public TextAttack checkpoints of our main results, as MOAI's are not public. MOAI's LayerNorm and softmax diverge as released on these checkpoints, so we refit them on our training inputs. Its GELU is accurate only on [−17, 10], while inputs here reach 150, and a same-degree refit is worse. Accuracy covers 7,020 validation inputs, one operator replaced at a time in float64.

LayerNorm (×2 per layer)GELUSoftmax
Levels, MOAI20710
Levels, OBELISK21010
Accuracy (%), MOAI89.5358.7689.74
Accuracy (%), OBELISK90.6690.7390.64

Thank you again for the thoughtful review.

Reviewer_8QHwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OBELISK optimizes non-interactive CKKS BERT-base inference through level-budget co-design, Process-in-Pieces scheduling, and complex pairing (HRC). On an H200 at batch 256, it reports 21.54 s/input and 21.63 GB peak memory. Encrypted accuracy remains within 2.52 percentage points of plaintext across four GLUE tasks.

Critical strengths
  1. Transparent baseline measurements and factorial PiP/HRC ablations make the reported memory savings credible. Fitting within roughly 21.6 GB is practically valuable.
  2. The full encrypted 12-layer model is evaluated on the complete validation sets of four GLUE tasks, providing stronger correctness evidence than layer-local tests.
Critical weaknesses
  1. The 6.6x speedup over MOAI is not cleanly attributed. HRC contributes about 1.5–1.6x, budget co-design lacks an end-to-end ablation, and the base includes substantial additional kernel and encoding optimizations.
  2. Methodological novelty is limited: complex pairing is established, PiP is tiling/scheduling, and budget selection enumerates a small configuration set. Layer-specific variants and task calibration also limit evidence of transferability.
Questions for authors
  1. Clarification: Can you separate the gains from budget selection, operator redesign, base optimizations, and HRC? How much remains if MOAI receives the fused kernel and stride key?
  2. Clarification: Were operator variants and approximation intervals selected using training data only? How robust is the full cascade to out-of-distribution inputs?
  3. Clarification: Can you directly justify 128-bit security for the deployed sparse-secret configuration and confirm whether MOAI uses the same secret distribution?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your valuable and constructive feedback.

Response to Q1/W1 - Attribution of the speedup

Our three contributions give 4.70× on top of all three base techniques. If MOAI receives only the fused kernel and the stride key, OBELISK is still 5.24× faster. General Response 2 shows each step end to end.

 

Response to Q2 - Selection data and OOD inputs

Every operator variant, approximation interval and per-task constant is selected using training data only. No validation input or label enters these choices, and the weights are the public checkpoints, unchanged. The corrected accuracies in General Response 1 use exactly these settings.

The cascade also holds on out-of-distribution inputs. Under encryption, we fed each task model 256 natural inputs from two of our other tasks and ran the full encrypted cascade. Only six of the 1,024 predictions change, all near-ties with a plaintext top-two logit gap of at most 0.18.

Calibration datasetOut-of-distribution inputs (128 each)Same prediction as plaintext
SST-2QNLI questions & RTE premises256 / 256
RTEMRPC pairs & QNLI pairs254 / 256
MRPCRTE pairs & QNLI pairs255 / 256
QNLIRTE pairs & MRPC pairs253 / 256
All1,018 / 1,024

On 13,000 more inputs, a plaintext emulation of every approximation stays as close to float64 as on validation inputs. These inputs come from IMDB, Yelp, AG News and CoLA for the SST-2 model and from MNLI, SNLI and QQP for the other three models. Their mean logit error is 0.024 against 0.019, and their 99th percentile is 0.144 against 0.143.

 

Response to Q3 - 128-bit security

The deployed configuration now reaches 128.2 bits of security through deploying SSE, with less than 0.5% of runtime overhead. The submitted sparse secret reached only about 121 bits, and MOAI uses the same secret at almost the same modulus. We are grateful for this question, which uncovered the flaw. General Response 1 gives the full estimator settings and results.

 

Response to W2 - Novelty and transferability

Beyond what each decision does on its own, our contribution lies in choosing the level budget, PiP and HRC together, since each choice changes what the others can do.

  • Shorter operators enable a cheaper budget. Our LayerNorm takes 2 levels instead of MOAI's 20, which halves the full-width bootstraps. The shallower operators then admit a 16-level chain instead of 21, and its freed modulus bits buy cheaper key switching. Rows 4 and 5 of General Response 2 measure these two steps at 1.68× and 1.75×, or 2.94× together.
  • PiP and the budget shape each other. Each PiP stream runs between two full-width bootstraps and uses the budget's levels from the top down. Attention descends from level 15 at the Q, K and V projections to level 1 at W_O. Since a ciphertext's size scales with its level, every piece enters large and leaves small, so only one piece is large at a time and the peak stays low. In turn, the chain must fit this whole 14-level stream, which sets the budget's floor. With the 128-bit cap as the ceiling, only a few budgets remain, so we price every one and take the cheapest.
  • HRC's three criteria shape PiP. They decide where complex pairing is valid and pays off (Section 3.3). Only QKᵀ, SM·V, the full-width bootstraps and the softmax partial bootstraps meet all three. This pairing lowers the attention peak that caps PiP's feed-forward pieces, so without HRC, PiP would pick pieces of 256 columns instead of 128 (Figure 4).

The decisions also transfer beyond our setup. The per-layer normalization variants only save computation, since using the wider one on every layer gives the same accuracy on all four tasks in our plaintext emulation. The layer-2 softmax variant keeps the same depth and is only refit on that layer's training scores. Task calibration is limited to the per-task constants in our response to Q2, all fit on training data. On THOR's released MRPC model, rebuilding only these constants keeps OBELISK within one input of THOR. General Response 3 also applies the same methods to LLaMA-3-8B, with all 32 layers on one 96 GB GPU at plaintext-level accuracy.

We thank you again for the careful review and would be glad to clarify anything further.

Reviewer_NQHCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OBELISK, a co-designed non-interactive secure Transformer inference framework to consider three post-packing factors including the level budget, intermediate-ciphertext lifetime, and slot representation. The proposed budget-operator co-design, Process-in-Pieces (PiP), and Hybrid Real-Complex Representation (HRC) OBELISK run on a commodity GPU. Budget-operator co-design cuts the layer's full-width bootstraps to two and reinvests the freed modulus bits in cheaper key switching. PiP computes attention by head and the feed-forward network by chunk. HRC runs operations in the cheaper representation, real or complex.

Critical strengths
  1. The paper works on an important problem, i.e., secure transformer inference. The paper solves this problem by co-designing three key post-packing decisions.

  2. The paper shows a large memory reduction in a full encrypted model. OBELISK executes a 12-layer BERT forward pass on a 24GB RTX 4090. It achieves 6x lower peak memory than MOAI on a H200 GPU with only 2.52% accuracy decrease.

  3. The paper reports detailed operation-level timing and memory of proposed scheme OBELISK and its baselines.

Critical weaknesses
  1. The paper is NOT well-motivated. Although the memory reduction is significant, the paper does not explain why prior works are not good enough. MOAI does not consider three key post-packing decision co-design, in table 1, it still can run on an A100 GPU. In my opinion, OBELISK just tries to accelerate encrypted inference as fast as possible without analyzing how large the potential is.

  2. Speedups are not fully controlled comparisons. Its baseline results come from the systems' published results. THOR and Powerformer use batch size 1, compared with batch size 256 for OBELISK and MOAI. So the largest reported ratios are throughput comparisons, not latency results. MOAI’s published and rerun results are difficult (i.e., 141.3 vs. 195.1 seconds per input), emphasizing the sensitivity to measurement scope.

  3. The inference latency is still high. A H200 pass uses 5,514 seconds for 256 inputs, which is still too high. This is why I ask about the motivation like how fast is acceptable. The client-side (e.g., key generation, encryption, and decryption) is not reported. Because of the large computing overhead and high latency, this paper is limited to BERT classification. No autoregressive inference. This is also related to the motivation like whether it is good enough.

Questions for authors

Please comment my reviews in the weakness section.

  1. The design motivation: how fast is acceptable and whether no autoregressive inference is acceptable.
  2. The speedup results are not fully controlled comparisons.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors [part 1]

Thank you for the thoughtful review and the clear questions.

Response to W1 - Motivation and remaining potential

Prior systems leave two barriers, and our motivation is to remove them.

First, they need large GPUs beyond any consumer card. MOAI evaluates end to end only on an H200, and its A100 time in Table 1 of the paper is a sum of per-module timings. As released, its code runs out of memory on the 80 GB A100 and on a 96 GB card in our runs, and it peaks at 129.3 GiB on a 141 GB H200. A provider then needs a GPU with H200-class memory for every run. THOR and Powerformer serve one input at a time but still peak at 62.7 and 72.1 GiB. Thus no system in Table 1 of the paper fits a consumer GPU, which holds at most 32 GB. OBELISK removes this barrier. It peaks at 22.2 GiB for 256 inputs and runs on a consumer card such as the 24 GB RTX 4090.

Second, memory decides whether larger models can run at all. With the same contributions, one 96 GB GPU runs LLaMA-3-8B on 256 prompts of 128 tokens at once, with all 32 layers encrypted (General Response 3). Without our budget–operator co-design and PiP, LLaMA-3-8B does not fit even on a 141 GB H200.

As for how large the potential is, OBELISK is already at the cheapest point we found within our design space. Two bootstraps per layer is the minimum with our operators, since one would need a 31-level chain beyond the 128-bit cap. Every alternative budget we re-ran is 1.04× to 1.88× slower per layer, with the same predictions. The remaining potential lies outside this space, in models retrained for FHE such as Powerformer's distilled BERT and in faster FHE libraries and hardware. Our contributions would apply on top of both.

 

Response to W2/Q2 - Controlled comparisons

We agree that the gains over THOR and Powerformer are throughput gains, not latency gains. OBELISK and MOAI batch 256 inputs, while THOR and Powerformer serve one. A batch of 256 inputs suits offline jobs on many records under one key, such as a hospital classifying its notes or a company analyzing the sentiment of hundreds of customer reviews. THOR is faster for a single input, at 330 s against OBELISK's 4,723 s when we run both on an RTX PRO 6000, but OBELISK finishes sooner once there are 15 or more inputs. Table 1 of the paper labels them as throughput gains for this reason. It is also why our headline comparison is with MOAI, which batches 256 inputs on the same engine.

We also agree that Table 1 of the paper quotes published baseline times, but General Response 2 measures MOAI under control. It runs every step from MOAI's configuration to OBELISK end to end on one H200 with the same engine. The gap between MOAI's 141.3 and 195.1 s is one of scope. MOAI's own timers leave out the weight encoding it performs at run time. Our rerun with those timers gives 146.0 s, and 195.1 s adds the encoding. OBELISK is 6.66× faster than the 146.0 s rerun and 8.90× faster than the 195.1 s end-to-end run. Table 1 of the paper uses the lower reported figure, which favors MOAI.

AuthorsOpenReview ↗

Official Comment by Authors [part 2]

Response to W3/Q1 - Latency, client cost and autoregressive inference

We agree that encrypted inference is still far from interactive latency. What is acceptable depends on the task. Offline batch jobs can wait hours, and OBELISK finishes a 256-input batch in 1.6 h on one H200 and in 1.9 h on a consumer RTX 4090. Interactive use needs answers in seconds, which FHE inference has not reached.

The field is moving fast. Bootstrapping takes over half of OBELISK's time, and the table shows how much faster one CKKS bootstrap has become.

YearImplementationPlatformBootstrap time
2020Han & Ki (CT-RSA) [1]CPU, one thread52.8 s
2021Bossuat et al. (EUROCRYPT) [2]CPU, one thread18 s
2021100× (TCHES) [3]V100 GPU328 ms
2023TensorFHE (HPCA) [4]A100 GPU, batched250 ms
2024Phantom (TDSC) [5], used by MOAI and OBELISKRTX PRO 6000 GPU275 ms
2025WarpDrive (HPCA) [6]A100 GPU121 ms
2026Cheddar (ASPLOS) [7]A100 GPU40 ms
2022CraterLake (ISCA) [8]ASIC, simulated6.33 ms
2023SHARP (ISCA) [9]ASIC, simulated3.12 ms
2024Trinity (MICRO) [10]ASIC, simulated1.92 ms
2025FAST (ISCA) [11]ASIC, simulated1.38 ms

CPU rows come from each paper, GPU rows from Cheddar's Table 5 and ASIC rows from WHET's Table 5 [12], and we measured the Phantom row. Parameters differ across rows.

Faster bootstrapping would speed up OBELISK directly. OBELISK reaches its times on Phantom, the same engine as MOAI, whose bootstrap is several times slower than those of the newest GPU libraries. Our three contributions cut how many bootstraps run and how wide they are, so they add to any faster engine.

Dedicated hardware would go further, and OBELISK suits it. The newest accelerator designs bootstrap over 100× faster than our engine, so far in simulation. CraterLake and ARK [13] hold only 256 and 512 MB on chip and are limited by off-chip memory traffic. OBELISK runs the full BERT-base model in 22.2 GiB against MOAI's 129.3 GiB and with fewer bootstraps, which reduces that traffic. We expect such hardware to bring encrypted inference toward acceptable latency.

The client side is light. Key generation takes 4.5 s once per key set on one RTX PRO 6000. Encrypting a 256-input batch takes 11 to 14 s and decrypting its output 2.6 s, under 0.07 s per input together.

Autoregressive generation is outside our scope, as it is for every system in Table 1 of the paper, but we run LLaMA-3-8B prefill end to end with all 32 layers encrypted. Its next-token scores already serve classification and multiple choice, as General Response 3 shows on four tasks. Decoding with a key-value cache needs a packing built for it, such as Cachemir's [14]. Our contributions act after packing, so the same methods would apply on top of such a packing, as they did for LLaMA.

We agree that the paper should state its motivation more clearly, and the revised PDF will add it to the introduction. Thank you again for the constructive questions.

 

[1] Han, K., Ki, D. "Better Bootstrapping for Approximate Homomorphic Encryption." CT-RSA 2020.

[2] Bossuat, J.-P., et al. "Efficient Bootstrapping for Approximate Homomorphic Encryption with Non-Sparse Keys." EUROCRYPT 2021.

[3] Jung, W., et al. "Over 100x Faster Bootstrapping in Fully Homomorphic Encryption through Memory-centric Optimization with GPUs." TCHES 2021.

[4] Fan, S., et al. "TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU." HPCA 2023.

[5] Yang, H., et al. "Phantom: A CUDA-Accelerated Word-Wise Homomorphic Encryption Library." IEEE TDSC 2024.

[6] Fan, G., et al. "WarpDrive: GPU-Based Fully Homomorphic Encryption Acceleration Leveraging Tensor and CUDA Cores." HPCA 2025.

[7] Choi, W., Kim, J., Ahn, J. H. "Cheddar: A Swift Fully Homomorphic Encryption Library Designed for GPU Architectures." ASPLOS 2026.

[8] Samardzic, N., et al. "CraterLake: A Hardware Accelerator for Efficient Unbounded Computation on Encrypted Data." ISCA 2022.

[9] Kim, J., et al. "SHARP: A Short-Word Hierarchical Accelerator for Robust and Practical Fully Homomorphic Encryption." ISCA 2023.

[10] Deng, X., et al. "Trinity: A General Purpose FHE Accelerator." MICRO 2024.

[11] Fan, S., et al. "FAST: An FHE Accelerator for Scalable-parallelism with Tunable-bit." ISCA 2025.

[12] Kim, J., et al. "WHET: Welding Homomorphic Encryption to Accelerator Architectures." arXiv:2606.11541, 2026.

[13] Kim, J., et al. "ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse." MICRO 2022.

[14] Yu, Y., et al. "Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache." IEEE S&P 2027.

I have read the rebuttal. My score is not the bottleneck, so I just keep it.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for reading our response and for confirming your positive assessment. We appreciate your careful review.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/3)

We are grateful to all reviewers for their time and for questions that led to new measurements and a stronger paper. We re-measured our main results with the two corrections below. Every new experiment in our responses also uses them where they apply.

 

Corrected Encrypted Accuracy

Based on the reviewers' feedback on the encrypted accuracy drop of up to 2.52 pp, we traced it to three implementation errors and fixed them. The drop is now at most 0.36 pp across the four GLUE tasks, and no task differs statistically from plaintext on its full validation set (exact McNemar test, p ≥ 0.45). The errors were in the GELU scale, the bootstrap's mean shrink and the layer-2 softmax fit. Most of the drop thus came from these errors rather than from homomorphic noise on the shorter chain, a concern noted by Reviewer rvop. The fixes add no measurable run time and change neither the budget nor any operation.

TaskPlaintext (%)Encrypted, submitted (%)Encrypted, corrected (%)
SST-292.4391.6392.20
QNLI91.5490.0491.62
RTE72.5670.0472.20
MRPC87.7585.5488.24
MRPC (F1)91.3590.1891.67

 

128-Bit Security

Following the reviewers' questions on the 128-bit security of the deployed sparse secret, we now achieve 128.2-bit security through adopting sparse-secret encapsulation (SSE) [1], with less than 1% of runtime overhead and about 0.4 GiB of extra peak memory. The full lattice estimator [2] puts the submitted secret at only about 121 bits. MOAI uses the same secret at almost the same modulus and so reaches about 121 bits, as Reviewer 8QHw also asked us to confirm. The 93.7 bits in Appendix A do not support 128 bits, as kindly pointed out by Reviewer rvop. They came from the estimator's rough core-SVP mode, which skips the hybrid attacks that target sparse secrets.

All computation now uses a main secret of Hamming weight 4096, at 128.2 bits. A sparse secret (Hamming weight 64) remains only inside bootstrapping, on a 341-bit modulus where it reaches 201 bits.

We ran every attack in the lattice estimator, including the sparse-secret hybrids. We report the MATZOV [3] and BDGL16 [4] cost models. For the standard configuration that Reviewer rvop requested, we include the HE Standard's largest 128-bit point [5]. The standard has no point beyond N = 2^15.

KeyModulusMATZOV (bits)BDGL16 (bits)
Submitted: sparse secret (Hamming weight 192)N = 2^16, log QP 1745120.9121.1
MOAI: sparse secret (Hamming weight 192)N = 2^16, log QP 1743121.1121.3
HE Standard's largest 128-bit pointN = 2^15, log Q 881126.1128.1
Revision: main secret, all computation (Hamming weight 4096)N = 2^16, log QP 1745128.2130.1
Revision: sparse secret, bootstrapping only (Hamming weight 64)341-bit base modulus201.4202.0

We thank the reviewers for these points and will update Sections 3.1 and 4.5 and Appendix A accordingly.

 

[1] Bossuat, J.-P., Troncoso-Pastoriza, J., Hubaux, J.-P. "Bootstrapping for Approximate Homomorphic Encryption with Negligible Failure-Probability by Using Sparse-Secret Encapsulation." ACNS 2022.

[2] Albrecht, M. R., Player, R., Scott, S. "On the Concrete Hardness of Learning with Errors." Journal of Mathematical Cryptology 2015. Lattice estimator.

[3] MATZOV. "Report on the Security of LWE: Improved Dual Lattice Attack." 2022.

[4] Becker, A., Ducas, L., Gama, N., Laarhoven, T. "New Directions in Nearest Neighbor Searching with Applications to Lattice Sieving." SODA 2016.

[5] Albrecht, M. et al. "Homomorphic Encryption Security Standard." HomomorphicEncryption.org 2018.

AuthorsOpenReview ↗

General Response (2/3)

End-to-End Ablation of the Speedup over MOAI

To address the reviewers' request to separate the speedup over MOAI, we measured each contribution end to end on an H200. Budget–operator co-design gives 2.94× on top of our base techniques, and all three contributions give 4.70×. Our base techniques alone give 1.42× over our rerun of MOAI. PiP keeps the time and cuts the peak memory from 68.2 to 23.0 GiB.

MOAI uses the same GPU library and bootstrapping as OBELISK. We therefore reproduced MOAI's configuration in our code and added the base techniques and then each contribution one at a time. This also addresses the point on controlled comparisons raised by Reviewer NQHC.

MOAI (reported) is the time that MOAI reports, and MOAI (rerun) is our run of MOAI's code timed with MOAI's own timers. MOAI (end to end) also counts the weight encoding that those timers omit.

RowConfigurations / input× vs MOAI (rerun)× vs MOAI (end to end)× vs row 3Peak (GiB)
MOAI (reported)141.3
MOAI (rerun)146.01.00
MOAI (end to end)195.11.00129.3
Base techniques
1MOAI (end to end) + fused Pt×Ct kernel121.31.201.61108.8
2+ cumulative stride key114.91.271.7098.3
3+ cached bootstrap encoding = baseline103.11.421.891.0098.3
Contributions
4+ operator redesign, (2, 21, 1)61.32.383.181.68103.1
5+ budget, (2, 16, 5)35.14.165.572.9468.2
6+ PiP35.24.155.542.9323.0
7+ HRC = OBELISK21.96.668.904.7022.2

We thank the reviewers for requesting this ablation and will add it to Section 4.4.

AuthorsOpenReview ↗

General Response (3/3)

Transfer to LLaMA-3-8B

In response to the reviewers' concerns that our evaluation is limited to BERT and that our decisions may not transfer, we applied our three decisions to LLaMA-3-8B prefill with all 32 decoder layers encrypted. The encrypted model matches the plaintext prediction on 2,112 of 2,124 inputs across four tasks, and its accuracy is within 0.4 pp of plaintext on every task. All twelve changed predictions are near-ties, with a plaintext top-two logit gap below 0.05.

The same methods as for BERT re-derived the budget, lifetime and representation decisions for this decoder, which uses RMSNorm, SwiGLU and causal attention. Its public weights are unchanged. The approximation intervals were set once on SST-2 training prompts and reused unchanged on the other three tasks. Only a small table of bootstrap correction constants is fit per task, on its training data. This limits the task calibration noted by Reviewer 8QHw.

The full model fits on a single 96 GB RTX PRO 6000, while without our budget and PiP it does not fit even on a 141 GB H200. Each 128-token prompt takes 351 s amortized over a batch of 256.

Task (inputs)Plaintext, float64 (%)Encrypted (%)Same prediction
SST-2, 2-shot (872)94.3894.27871 / 872
RTE (277)69.3168.95274 / 277
MRPC (408)68.1467.89401 / 408
MRPC, F179.4379.04
ARC-Easy, 4-way (567)89.4289.59566 / 567
All (2,124)84.7584.652,112 / 2,124

This evaluation covers prefill and next-token scoring. We thank the reviewers for raising this point and will report these results in Appendix I.

AuthorsOpenReview ↗

Summary of Revisions

Thank you again for the careful reviews and constructive questions. We have uploaded the revised PDF. Changes are highlighted in blue, and the main ones are listed below; the responses cited in parentheses discuss each in detail.

  • Accuracy
    • Corrected encrypted accuracy, at most 0.36 pp below plaintext, in the Abstract, Section 4.5, Table 5, and Appendix H (General Response 1; rvop).
    • Fitting on training data only, and out-of-distribution inputs, in Section 4.5, Table 6, and Appendix H (8QHw).
    • Accuracy-matched comparison on THOR's public MRPC model in Appendix F (rvop).
  • Security
    • 128.2-bit security through sparse-secret encapsulation in Section 3.1 and Appendix A (General Response 1; rvop, 8QHw).
    • OBELISK's times and peaks re-measured under this key in the Abstract, Sections 1 and 3–5, Tables 1–3, and Appendices C, E, and F (General Response 1).
  • Speedup
    • End-to-end ablation from MOAI's code to OBELISK in Section 4.4, Table 4, Figures 5–6, and Appendix G (General Response 2; 8QHw, rvop, NQHC).
    • Our operators against MOAI's, and alternative budgets, in Appendix B (rvop).
  • Transferability
    • LLaMA-3-8B prefill with all 32 layers encrypted in Appendix I (General Response 3; 8QHw, NQHC).
  • Practicality
    • Motivation, client cost, latency, and generation in Sections 1, 4.1, and 5 (NQHC).

Our earlier responses now quote the revised numbers. We have also updated the supplementary material. Its top-level README links the paper's main tables and results to the code, commands, and run logs behind them. We would be glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.