acceptodds
Under review as a conference paper at ICLR 2027

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

17 views

Abstract

Fully homomorphic encryption (FHE) allows a server to run a language model directly on encrypted user prompts, but current approaches remain prohibitively slow. Ciphertexts natively support only addition, multiplication, and rotation, and multiplications may be composed only to a bounded depth before a costly bootstrapping operation is required to continue. Every nonlinearity must therefore be approximated by an iterative method; each iteration increasing the number of multiplications. A higher iteration count buys precision but exhausts the available depth more frequently and thus triggers more bootstraps, which dominate latency. We introduce **Homomorphic Encryption-Aware Training (HEAT)**, a _fine-tuning_ method that makes the per-nonlinearity iteration counts _learnable_, enabling them and the model weights to co-adapt during training. HEAT optimizes iterations with respect to the task objective, allowing the model to adapt to approximation errors encountered during inference without architectural changes or retraining from scratch. We further relate iteration count to quantization bit width and bound, at fixed weights, the gap between our objective and quantization-aware training. On encrypted GPT-2 decoding, HEAT reduces iterations by , bootstraps by , and end-to-end latency by , while improving decode agreement over the calibrated encrypted baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fully homomorphic encryption, FHE, Fine-tuning, Encryption, CKKS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1dCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce Homomorphic Encryption-Aware Training, a technique optimizes approximation of non-linear function via per-site iteration selection with fine-tuning. The results show that HEAT reduces iteration by 3.1x, bootstraps by 1.6x and end-to-end latentcy by 1.4x on a encrypted GPT-2 (124M).

Critical strengths
  • This paper considers per-site approximation iteration and range as optimization ovjectives and fine-tuning the model collaboratively for adapting the errors introduced by approximation under FHE. This is interesting when prior work either focus on only fine-tuning model or find a optimal approximation on a frozen model.

  • This paper integrates quantization-aware training concept for error tolerance adaption in FHE, which is a inspiring direction for achieving a more effient approximation of non-linear funcitons under FHE.

Critical weaknesses
  • The boundary between finetuning a FHE-friendly variants e.g. AutoFHE[1] and Power-Softmax[2], and a FHE-friendly polynomial variants is ambiguious in paper claims. Training a model to adapt an efficient polynomial approximation of original function and a alternative function with easiser polynomial approximation approach, which is the same objectives for reducing overall task loss and preseving model's performance. Approximation-aware weight adaptation itself does not appear to be the main novelty. In addition, the finetuning process in the paper is full parameter update and produce an FHE-adapted model variant, which is unclear current method have apparent advantages comparing to AutoFHE and Power-Softmax finetuning scheme regarding the training workload (these methods don't require training from scratch). The training costs for larger models are not reported, making it difficult to assess the scalability of the proposed adaptation procedure.

  • The per-site optimization of approximation is not directly resulting in reduction of bootstrappings and multiplicative level drop in a whole computation graph of transformer block. The experiments and detailed analysis are required for illustrating how the reduced iterations are well suited with the strategy of multiplication level organization and bootstrapping placement. It is unclear why choose these micro optimization instead of choosing the critical components to optimize that directly contribute to reduction of bootstrappings and multiplicative level drop.

[1] Ao, Wei, and Vishnu Naresh Boddeti. "{AutoFHE}: Automated adaption of {CNNs} for efficient evaluation over {FHE}." 33rd USENIX Security Symposium (USENIX Security 24). 2024.

[2] Zimerman, Itamar, et al. "Power-softmax: Towards secure llm inference over encrypted data." arXiv preprint arXiv:2410.09457 (2024).

Questions for authors
  1. Could author provide the comparison of these FHE-friendly finetuning scheme to HEAT regarding the the model's performance and training workload? And author may show if HEAT is compatible to parameteric efficient finetuning method, and it would also be helpful to report training time for larger model if possible.

  2. Could author provide the details of how these iteration reduction results in reduction of bootstrapping and multiplicative level drop? It will improve the value of HEAT if author can show the FHE computation design (multiplication level organization and bootstrapping placement) can seamlessly incorporate in current optimization obejectives.

  3. Could author provide the comparison of per-family and per-site fixed count iteration and range finetuning results with current parameters selected by HEAT? It will seperate the contribution from optimal parameters setting and show if co-optimizing training is necessary.

1 comment

Thank you for your additional experiments.

  1. For Power-Softmax, its Appendix A.1 describes adaptation from pretrained checkpoints for both Pythia‑70M and Pythia‑1.4B, and 2 days claim is the maximal time across their experiments, which should refer to the experiment on Pythia‑1.4B. This is conflict with author's claim ( i.e. paper only examines Pythia-70M at 128 tokens ). I think authors need report Power-Softmax time on Pythia‑70M and Pythia‑1.4B, following their receipt in Appendix A.1 and starting from pre-trained checkpoint for a fair comparison.
  2. Another concern on approximation-aware weight adaptation boundary still. Since both jointly finetuning of HEAT and function replacement will change the model's distribution and alter model's behavior, and cannot ensure the model downstream performance. Why not use an alternative function with easiser polynomial approximation approach e.g. Powerformer[1]? Could authors quantify the benefit of HEAT against Powerformer-style function replacement method? And If Powerformer-style function replacement already in place, does HEAT still have incremental gains?
  3. Could author clarify the actual strategy of HEAT bootstrapping placement? And How does HEAT bootstrapping placement compare with a static schedule coordinating with their packing method e.g. Powerformer?

[1] Park, Dongjin, Eunsang Lee, and Joon-Woo Lee. "Powerformer: Efficient and high-accuracy privacy-preserving language model with homomorphic encryption." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Reviewer_cjHNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets non-interactive CKKS inference for transformers, where the iteration counts of the Softmax and LayerNorm solvers set the multiplicative depth and therefore the number of bootstraps. HEAT learns a per-site count during fine-tuning: each site gets a PonderNet-style halting distribution with a negative-binomial prior, an activation-range loss is added, and training runs in three phases. A theorem bounds, at fixed weights, the loss gap to matched-precision QAT. On encrypted GPT-2 small decoding, compared with a per-site calibrated baseline, iterations per forward drop from 712 to 228 and latency from 54.3 to 37.6 s/token. ATLAS, an ablation, and one encrypted ViT experiment are also reported.

Critical strengths
  1. A practical idea, validated end to end under real encryption. The architecture is unchanged, fine-tuning costs 7 GPU-hours, and every latency and fidelity number comes from a deployed CKKS run. Four seeds recover the same 228-iteration circuit.

  2. A useful diagnosis of encrypted drift. Bootstrap-noise injection identifies the Softmax square-and-normalize refinement as the main amplifier of encrypted error. It also explains why ATLAS, "near-lossless in plaintext", collapses on 13/64 sequences under encryption.

Critical weaknesses
  1. The contribution of the core mechanism (learned counts with co-adapted weights) is not quantitatively separated from that of the range squeeze.

    • Training with Lrange\mathcal{L}_{\text{range}} alone and letting the calibrator set the counts (Table 2, row 1, p. 9, l. 438) already yields about 64% of HEAT's e2e latency reduction: (54.3−43.6)/(54.3−37.6), from Table 1, l. 338/341 and Table 2, l. 438/441.
    • E.5 credits HEAT's single refinement pass to "halved score ranges and reduced iterations" (App. E.5, p. 26, l. 1397). However, row 1 reports neither top-1 nor bts/tok (Table 2, l. 437), so the fidelity gain from the squeeze cannot be separated from that of the learned counts.
    • None of the comparisons holds the squeeze fixed. Row 2 freezes the weights and row 3 omits Lrange\mathcal{L}_{\text{range}} (Sec. 4, p. 8, l. 411–412), and ATLAS runs "from the calibrated circuit on frozen weights" (p. 6, l. 318–319). The transplant result is asserted without numbers (p. 8, l. 416–417). Supporting signals come from ViT (Table 5, p. 25, l. 1306–1309) and plaintext scaling (Table 9, p. 28, l. 1464–1466), not from an encrypted GPT-2 control.

    Why consequential: the novelty claim rests on learning the counts, yet the range loss is borrowed (Sec. 3.1, p. 4, below l. 206), and Thm. 1 compares losses "at fixed weights and a fixed count" (App. B.2, p. 18, l. 931). The GPT-2 evidence therefore does not show what the learned counts add.

  2. The encrypted evaluation conditions are narrow relative to the claim of faster FHE inference.

    • All methods share a single-iteration bootstrap with roughly 10–13 bits of precision (App. C.1, p. 19, l. 1013–1016). Under it, the calibrated baseline degrades from 31.7 to 44.6 perplexity (Table 1, l. 338–339). A 2-iteration bootstrap with roughly 2–3 orders of magnitude lower error (Fig. 6, p. 19, l. 972–985) is never used to run a circuit.
    • Encrypted language-model results cover only GPT-2 small on 128-token teacher-forced OpenWebText windows (Table 1 caption, p. 7, l. 325–326). OpenWebText is also the fine-tuning corpus (App. E.3, p. 25, l. 1330–1331), and fidelity is measured "each model against its own plaintext logits" (l. 326).
    • Out-of-domain and scaling results are plaintext only. HEAT has the worst C4 perplexity of the deployed circuits (29.8 vs. 28.0 calibrated; Table 7, p. 26, l. 1355), and Table 9 is not run under FHE (p. 28, l. 1459–1460).

    Why consequential: it is unclear whether the 1.4× gain and the stability advantage survive a more precise bootstrap, other inputs, or larger models.

Questions for authors
  1. On the squeezed GPT-2 checkpoint, with an equal fine-tuning budget, please report it/fwd, bts/tok, e2e, top-1 and collapses for recalibrated counts, ATLAS, and HEAT's final counts trained with phase 3 only. Please also quantify the transplant result (p. 8, l. 416–417). If full HEAT is clearly better, I would raise my score; if these variants come within a few percent of it, I would consider the contribution marginal.
  2. How do the calibrated baseline and HEAT compare under the 2-iteration bootstrap of Fig. 6 (p. 19)? A consistent HEAT advantage would strengthen the claim; if the gap disappears, the fidelity benefit appears tied to the low-precision bootstrap.
  3. Please report encrypted collapse rate and top-1 on an out-of-domain corpus such as C4, on contexts longer than 128 tokens, and against the original GPT-2. Stable results would increase my confidence in deployability.
  4. Which collapse definition is used: "more than half of the final states diverged" (Table 1 caption, p. 7, l. 327) or "more than 50% of the tokens couldn't be decrypted" (Fig. 5 caption, p. 8, l. 397–398)? How many encrypted runs underlie Table 1? (0/64 vs. 5/64, Table 2, l. 438/441, gives Fisher p ≈ 0.06 by my computation.) Please also report the total log(PQ) including the special primes (App. C.1, p. 19, l. 1007–1012). This determines how much weight I give the stability and security claims.
Reviewer_JAeWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Homomorphic Encryption-Aware Training (HEAT), a framework that jointly optimizes model weights and iteration counts of nonlinear approximations for efficient fully homomorphic encryption (FHE) inference. By making iteration counts differentiable through an adaptive computation mechanism, HEAT enables the model to tolerate lower-precision approximations while reducing computational overhead. Experiments on encrypted GPT-2 inference demonstrate reductions of 3.1× in iteration counts, 1.6× in bootstrapping operations, and 1.4× in end-to-end latency.

Critical strengths

Iterative methods such as Goldschmidt and Newton–Raphson are widely used to approximate nonlinear operations, including reciprocal and inverse square root functions, in FHE-based inference. However, their iteration counts are often determined heuristically based on conservative accuracy requirements, leading to unnecessary computational overhead. This paper addresses this practical limitation by optimizing iteration counts with respect to the task objective.

An important contribution is the application of adaptive computation techniques to make discrete iteration counts differentiable and jointly optimizable with model weights. This provides a meaningful approach to reducing the computational cost of FHE inference without modifying the model architecture.

Critical weaknesses

The primary concern is the fairness of the comparisons in Table 1. While HEAT jointly fine-tunes model weights and iteration counts, competing methods, including ATLAS, are evaluated using frozen pretrained weights. Consequently, the reported improvements may partly stem from additional fine-tuning rather than the proposed iteration optimization strategy.

Although Table 2 provides ablation studies of HEAT's components, these experiments do not establish its superiority over existing methods under comparable weight optimization conditions. A fairer comparison with fine-tuned baselines is necessary to isolate the benefits of the proposed joint optimization.

Questions for authors

Could the authors include additional baselines under comparable fine-tuning conditions?

  1. Fine-tuned model + Calibration: Fine-tune the original plaintext model using its exact nonlinear operations, without introducing any approximations, on the same dataset and with a training budget comparable to HEAT. Then, apply conventional approximation calibration to the resulting model weights.
  2. Fine-tuned model + ATLAS: Apply ATLAS to the fine-tuned model obtained in (1), using the same weights to optimize the approximation configurations.

These comparisons, particularly in terms of perplexity and inference latency, would help distinguish improvements arising from weight fine-tuning from those attributable to HEAT's joint optimization of weights and iteration counts, thereby providing stronger evidence of its advantages over existing methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.