acceptodds
Withdrawn from ICLR 2027

SHIM: Scaling Homomorphic Generative Transformer Inference via Multi-Party Collaboration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Privacy-preserving large language model inference with homomorphic encryption remains impractical due to both latency and memory bottlenecks, particularly during autoregressive decoding. Existing non-interactive FHE systems incur high end-to-end latency from large-precision homomorphic operations and frequent bootstrapping, while hybrid HE-2PC designs reduce bootstrapping but introduce substantial communication overhead, especially for non-linear layers. Moreover, decoding requires a KV cache whose encrypted form leads to severe memory amplification. We propose SHIM, a collaborative framework that integrates threshold HE and secret sharing to enable a threshold-mediated masking paradigm. Our key insight is a hybrid representation of LLM state: linear layers remain encrypted, while KV cache and non-linear function inputs are maintained as cryptographically masked plaintexts. This design enables exact evaluation of Softmax, LayerNorm, and SiLU on masked values without polynomial approximations, simultaneously reducing both activation latency and bootstrapping overhead. Meanwhile, masked KV cache storage reduces memory from 198,GB to 3,GB compared to FHE baselines, enabling long-context decoding that exceeds memory limits in prior systems. SHIM achieves up to lower generation latency compared to recent state-of-the-art baselines, including NEXUS (NDSS'25), THOR (CCS'25), Powerformer (ACL'25), and BOLT (S&P'24), while preserving model utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Homomorphic Encryption, LLM Inference, Non-linear Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_qE6jOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SHIM, a hybrid framework combines FHE and secret sharing schemes for privacy-preserving LLMs inference. Specifically, SHIM uses a mask to maintained KV cache and non-linear function in plaintext. It claims KV cache storage reduces memory from 198 GB to 3 GB, and 15.7x speedup over NEXUS.

Critical strengths
  • This paper identifies the latency and memory bottlenecks brought by non-linear function evaluation and bootstrapping, which are two heavy exponents during inference and valuable to solve for practical secure LLM inference.
Critical weaknesses
  • Security and performance concerns. The paper use masked plaintext as a one-time pad for security, however, this design is problematic under proposed R=230R = 2^{30} and δ=240\delta = 2^{40} parameters, where ∣X~−X∣≤2−10≈0.000977|\widetilde X-X|\le 2^{-10} \approx 0.000977 under additive mask. The secret value is close to masked value which is critical security issue. The multiplicative mask has the same problem. In addition, the integration of secret sharing require server transfer CKKS ciphertexts, and this latency and communication cost are not reflected in Table 1, since reported Activation latency (0.002 s) is less than WAN latency setting (0.04 s) and one CKKS ciphertext under N=65536N=65536 is at least 1 MB, 70 MB communication for every non-linear function in LLaMa-3.1-8B seems unrealistic.

  • Soundness concerns. Comparison seems unfair as some of methods are outdated such as NEXUS where 15.7x speedup comes from. Missing SOTA baselines such as BumbleBee(NDSS '25)[1], MOAI(ICLR '26)[2], Cachemir(S&P '27)[3], ROSETTA(CCS '26)[4] undermine validity of the evaluation.

[1] Lu, Wen-jie, et al. "Bumblebee: Secure two-party inference framework for large transformers." Cryptology ePrint Archive (2023).

[2] Zhang, Linru, et al. "MOAI: Module-optimizing architecture for non-interactive secure transformer inference." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Yu, Ye, et al. "Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache." arXiv preprint arXiv:2602.11470 (2026).

[4] Yu, Jiangrui, et al. "ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation." arXiv preprint arXiv:2609.16915 (2026).

Questions for authors
  1. How do the authors justify the privacy guarantees of mask plaintexts under the actual parameters? Could the authors provide a indistinguishability or leakage analysis, since masked values are extremely close to original values.

  2. Does the reported activation latency is correct and provide the runtime breakdown including the communication latency. Also, there should be a detailed presentation of communication volume. Could the authors provide a breakdown of communication rounds, transferred bytes, and latency, and clarify how the three-server WAN environment was implemented?

  3. Why are more recent published secure LLM inference work(Bumblebee, MOAI, Cachemir, ROSETTA) are not included in comparison? Also, could author provides a fair comparison under the same security and deployment conditions or illustrate the potential unfair conditions.

Reviewer_nfVoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SHIM combines threshold CKKS encryption and secret sharing for privacy-preserving autoregressive Transformer inference. Linear operations remain encrypted, while the computation server receives selected nonlinear inputs and KV-cache entries as masked plaintexts. Encrypted correction terms remove the masks’ contributions from the nonlinear results, and peers help cancel mask terms during attention. The framework uses threshold refresh instead of conventional FHE bootstrapping. The experiments use LLaMA-3.1-8B-Instruct and LongBench, with cache budgets from 4K to 32K. At an 8K budget, Table 1 reports 13.7 seconds of total latency and 3 GB of KV-cache memory, compared with 213.9 seconds and 198 GB for NEXUS. The paper also reports operator errors, downstream F1 scores, and the batch sizes supported at different cache budgets.

Critical strengths
  1. The paper looks at both KV-cache storage and the repeated cost of decoding, rather than just individual encrypted operations. I found the attention protocol interesting: peers compute the mask-dependent terms using bilinearity, and the server cancels them homomorphically. This could offer substantial memory savings if the masking can be made secure.

  2. The experiments use an 8B model and several LongBench tasks, rather than stopping at cryptographic microbenchmarks. Testing different cache budgets and reporting downstream F1 alongside operator errors are sensible choices for evaluating a generative model. The results would be more convincing with the matched controls and clearer cost reporting requested below.

Critical weaknesses
  1. Appendix H.1 defines x~=x+r/Δ\widetilde{x}=x+r/\Delta, with ∣r∣≤R|r|\le R. With the experimental settings R=230R=2^{30} and Δ=240\Delta=2^{40}, this gives ∣x~−x∣≤R/Δ=2−10.|\widetilde{x}-x|\le R/\Delta=2^{-10}. Under this mask-only model, x=0x=0 and x=1x=1 produce disjoint observation intervals. Keeping the mask unknown therefore does not provide the one-time-pad hiding claimed in Lemma L.1. The multiplicative mask in Appendix H.2, M=1+k/ΔM=1+k/\Delta with ∣k∣≤K=2|k|\le K=2, has the same problem: for x≠0x\ne0, the relative changes are at most about 1.82×10−121.82\times10^{-12} for xMxM and 3.64×10−123.64\times10^{-12} for xM2xM^2. This does not support Lemma L.3’s argument. These bounds concern the masks, not the full decoded observations, which also contain CKKS and smudging noise. Any privacy provided by that noise needs its own analysis, including repeated disclosures. Remark L.6 acknowledges the tradeoff but does not establish privacy for the experimental parameters or give another setting with both privacy and numerical stability. Fixing the masking could change the reported performance, so this is not just a gap in presentation.

  2. Section 4.2 gives Xj=c0+djX_j=c_0+d_j for every peer, which counts c0c_0 NS−1N_S-1 times when the shares are summed. It is also unclear how the cited three-party MPC protocol is run when multiplication is assigned to two peers. Algorithms 10–11 combine fresh ciphertexts with retained input ciphertexts, so they do not automatically refresh every branch. Threshold refresh may handle this, but the paper needs to state when it is called and where its cost is included. Table 1’s 0.002-second activation latency cannot be the wall-clock time of a complete, unamortized online interaction over the stated 40-ms WAN link. It may cover local or amortized work. Please show where communication, MPC, re-encryption, corrections, and refresh are counted in the 13.7-second total, with the same timing boundaries for all baselines. The 0.07-GB communication figure also needs a breakdown. Without these details, I cannot tell whether the results account for the full protocol.

Questions for authors
  1. Were the decoded mask definitions in Appendix H, with R=230R=2^{30}, K=2K=2, and Δ=240\Delta=2^{40}, used in Tables 1–3? Please address the mask-only example above and give a privacy bound for the implemented observations, accounting for decoded smudging noise and repeated disclosures. How does independently sampling a scaled distribution at each peer produce the stated aggregate-mask distribution? If different parameters are needed, please show that the full computation remains numerically stable after mask cancellation, re-encryption, homomorphic correction, and rescaling.

  2. How is c0c_0 allocated in the implementation of Equation (3)? What representation is used for the secret-shared multiplications, and how are CKKS ring elements, decoded slots, scaling, truncation, and modular wraparound handled? Which three parties run the cited MPC protocol, what does the computation server observe, and how many corrupted parties does this implementation tolerate?

  3. Could the authors provide the modulus chain, initial level budget, and a trace of ciphertext levels through a complete decoding pass, including retained inputs and residual branches? How many explicit PrefreshP_{\mathrm{refresh}} calls occur per decoding step, and where are their computation and communication costs counted in Table 1? Does the implementation refresh branches that are not shown as refreshed in Algorithms 10–11?

  4. What do the timings in Tables 1 and 3 measure: complete generation latency, decoding-step latency, or amortized time per output token? Please specify the batch size, prompt and output lengths, whether preprocessing is included, and a timing breakdown of the steps on the critical path. How are the three logical servers run on the stated single-A6000 workstation? Please report peak CPU/GPU memory for each role and for the system as a whole. If Table 1’s 3-GB cache figure is per sequence, Table 3’s batch size of 16 would approach the nominal 48-GB GPU capacity before other state is counted. Is that interpretation correct? Where are the model weights, evaluation keys, helper mask shares, and working buffers stored? Please also provide message counts and serialized byte sizes for each protocol to explain the 0.07-GB communication result.

  5. Did the evaluated model use its native RMSNorm or the mean-centered LayerNorm described as the evaluated implementation in Appendix K? Please give the actual normalization computation and explain the “LayerNorm” MAE column. Meta’s reference implementation uses RMSNorm, so it is important to clarify whether the evaluated model was changed. Please also compare plaintext and SHIM with identical cache budgets, cache-management rules, prompts, and decoding settings. Are the operator MAEs measured after the full cryptographic correction pipeline? The paper attributes the F1 gap to the limited cache, but the comparison with an unlimited-cache plaintext baseline does not separate cache effects from protocol errors. Sample counts and uncertainty estimates would help interpret the remaining differences.

Reviewer_fwpZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SHIM combines threshold encryption with secret-shared masks for private language-model inference. Linear operations remain encrypted, nonlinear functions use masked plaintext values and encrypted correction terms, and the KV cache is maintained as masked plaintext. The study reports decoding latency, cache memory and downstream utility on LLaMA-3.1-8B.

Critical strengths
  1. I appreciate the focus on KV-cache expansion in Section 5.2: this limits context length and batch size independently of nonlinear arithmetic. Table 1 reports a reduction from 198 GB to 3 GB. A compact private state would be valuable, although I cannot credit this result as secure without a valid hiding argument.

  2. The correction identities in Sections 4.2 and K offer a coherent way to retain the original nonlinear functional forms. Table 2 also reports relatively small utility differences. This is a useful direction for avoiding polynomial surrogates, but the approximate CKKS arithmetic means the evaluation should not be described as numerically exact.

Critical weaknesses
  1. My main concern is confidentiality. Section H.1 reveals x+r/Δ; the parameters in Section 5.1 give R/Δ=2^-10. Section H.2 uses multiplicative masks within 2^-39 of one. These observations nearly disclose the protected values. Sections L.1–L.2 incorrectly treat unknown bounded masks as one-time pads, and the qualification in Section L.4 supplies no concrete repair.

  2. The speed comparison in Table 1 changes the trust and resource model: SHIM uses collaborating servers and a trusted dealer, whereas the baselines include noninteractive FHE and HE–2PC. Section 5.1 includes CPU–GPU transfers and author-added decoding support, but I still need matched confidentiality and aggregate helper costs to judge the size of the protocol improvement.

  3. Table 2 compares a bounded KV cache with an unrestricted plaintext reference. Section 5.2 attributes the utility gap to cache capacity, but this comparison cannot separate context truncation from encrypted-execution error. I would need a plaintext reference with the same cache budget before accepting that attribution.

Questions for authors
  1. For the concrete parameters in Section 5.1, what bounds the distinguishing advantage between server observations of different admissible activations, including the actual arithmetic noise?

  2. How do the Table 1 gains change under matched confidentiality and trust assumptions, including helper computation and mask/cache memory?

  3. With the same cache and context budget in Table 2, how much downstream utility loss remains relative to plaintext?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.