- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduce Homomorphic Encryption-Aware Training, a technique optimizes approximation of non-linear function via per-site iteration selection with fine-tuning. The results show that HEAT reduces iteration by 3.1x, bootstraps by 1.6x and end-to-end latentcy by 1.4x on a encrypted GPT-2 (124M).
Critical strengths
-
This paper considers per-site approximation iteration and range as optimization ovjectives and fine-tuning the model collaboratively for adapting the errors introduced by approximation under FHE. This is interesting when prior work either focus on only fine-tuning model or find a optimal approximation on a frozen model.
-
This paper integrates quantization-aware training concept for error tolerance adaption in FHE, which is a inspiring direction for achieving a more effient approximation of non-linear funcitons under FHE.
Critical weaknesses
-
The boundary between finetuning a FHE-friendly variants e.g. AutoFHE[1] and Power-Softmax[2], and a FHE-friendly polynomial variants is ambiguious in paper claims. Training a model to adapt an efficient polynomial approximation of original function and a alternative function with easiser polynomial approximation approach, which is the same objectives for reducing overall task loss and preseving model's performance. Approximation-aware weight adaptation itself does not appear to be the main novelty. In addition, the finetuning process in the paper is full parameter update and produce an FHE-adapted model variant, which is unclear current method have apparent advantages comparing to AutoFHE and Power-Softmax finetuning scheme regarding the training workload (these methods don't require training from scratch). The training costs for larger models are not reported, making it difficult to assess the scalability of the proposed adaptation procedure.
-
The per-site optimization of approximation is not directly resulting in reduction of bootstrappings and multiplicative level drop in a whole computation graph of transformer block. The experiments and detailed analysis are required for illustrating how the reduced iterations are well suited with the strategy of multiplication level organization and bootstrapping placement. It is unclear why choose these micro optimization instead of choosing the critical components to optimize that directly contribute to reduction of bootstrappings and multiplicative level drop.
[1] Ao, Wei, and Vishnu Naresh Boddeti. "{AutoFHE}: Automated adaption of {CNNs} for efficient evaluation over {FHE}." 33rd USENIX Security Symposium (USENIX Security 24). 2024.
[2] Zimerman, Itamar, et al. "Power-softmax: Towards secure llm inference over encrypted data." arXiv preprint arXiv:2410.09457 (2024).
Questions for authors
-
Could author provide the comparison of these FHE-friendly finetuning scheme to HEAT regarding the the model's performance and training workload? And author may show if HEAT is compatible to parameteric efficient finetuning method, and it would also be helpful to report training time for larger model if possible.
-
Could author provide the details of how these iteration reduction results in reduction of bootstrapping and multiplicative level drop? It will improve the value of HEAT if author can show the FHE computation design (multiplication level organization and bootstrapping placement) can seamlessly incorporate in current optimization obejectives.
-
Could author provide the comparison of per-family and per-site fixed count iteration and range finetuning results with current parameters selected by HEAT? It will seperate the contribution from optimal parameters setting and show if co-optimizing training is necessary.
Thank you for your additional experiments.
[1] Park, Dongjin, Eunsang Lee, and Joon-Woo Lee. "Powerformer: Efficient and high-accuracy privacy-preserving language model with homomorphic encryption." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.