- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the high computational cost of Softmax operations in homomorphic encryption (HE) environments. The authors propose CGF-Softmax, an approach that reformulates the Softmax normalization process using the Cumulant Generating Function (CGF). By employing a second-order approximation—retaining only the mean and variance—the method avoids explicit maximum subtraction and homomorphic division, thereby reducing computational overhead. The paper further analyzes the theoretical properties and approximation errors of this method and evaluates the trade-off between accuracy and multiplicative depth across models such as Llama-3.2-1B, ViT, and DeiT. Finally, the authors integrate CGF-Softmax into Cerium, an existing FHE inference framework, and demonstrate improved efficiency for end-to-end encrypted inference using BERT-Base.
Critical strengths
-
The research problem is clearly defined, and the proposed method holds practical significance: Non-linear operations in homomorphic encryption environments are inherently computationally expensive, and Softmax is a widely used operation in Transformer architectures. To address this, the authors reconstruct the Softmax normalization process using CGF, replacing costly computational steps with relatively simple statistical operations. I find this approach reasonable; it targets a genuine bottleneck in current encrypted inference while achieving a favorable balance between computational efficiency and model accuracy.
-
The experimental evaluation is comprehensive, validating not only the algorithm itself but also its performance in a practical system: The authors first analyze the method's efficiency advantages in terms of multiplicative depth and computational complexity. They then conduct downstream task evaluations on language models and Vision Transformers to verify whether the original model accuracy is maintained after replacing the standard Softmax. Furthermore, I appreciate that the paper goes beyond algorithmic analysis; the authors integrate the method into Cerium—a real-world FHE inference framework—and achieve a 36%–40% reduction in end-to-end inference latency on BERT-Base. These experiments provide compelling evidence for the method's practical utility.
Critical weaknesses
-
There remains a gap between the theoretical analysis and the second-order approximation used in practice: The paper provides a theoretical analysis of the properties and asymptotic error of CGF-Softmax; however, the primary error analysis is based on the full-CGF formulation, whereas the practical implementation retains only the mean and variance, omitting higher-order cumulants. Although the authors analyze changes in certain higher-order cumulants before and after fine-tuning in Appendix E, this does not fully capture the error introduced by the second-order truncation, nor does it sufficiently demonstrate the stability of this approximation across different models and input distributions. Given that the second-order approximation is a crucial component of the actual algorithm, I believe the current theoretical analysis provides insufficient support for the practical method.
-
The scope of experimental validation for end-to-end encrypted inference remains limited: The paper evaluates the trade-off between accuracy and multiplicative depth using Llama-3.2-1B and several ViT/DeiT models, representing a reasonable scope of coverage. However, experiments involving the actual deployment of the method within an FHE system to measure end-to-end inference latency are limited to the RTE task on BERT-Base. While the results demonstrate significant efficiency gains, direct system-level experimental validation is currently lacking for the other large-scale models evaluated in the paper. Consequently, I believe the actual speedup effects across a broader range of models and tasks require further confirmation.
Questions for authors
-
Regarding the second-order cumulant approximation used in Section 3.4: Based on existing experimental results, could you further elaborate on the magnitude of the approximation error or normalization bias between the practical CGF-Softmax and the original Softmax? Specifically, do these errors change significantly before and after fine-tuning?
-
Given that end-to-end FHE latency is evaluated only on BERT-Base, could the authors further clarify, based on the existing results, how the reported reductions in multiplicative depth might translate into practical end-to-end speedups for Llama and ViT/DeiT?
2 comments
Response to Authors
I appreciate the author’s clarification regarding my questions and the additional experiments provided. I believe the author has effectively addressed my primary concerns about the paper; therefore, I will raise my score.
We thank the reviewer for the constructive feedback and suggestions.
(W1)
We thank the reviewer for raising this concern. To further examine the behavior of higher-order cumulants across different models and datasets, we extended the analysis in Appendix E to DeiT-Tiny/Base, ViT-Tiny, and Llama-3.2-1B on Banking77, Clinc150, and SST-2. The additional results, presented below, illustrate the behavior of the omitted third- and fourth-order cumulant terms before and after fine-tuning. As shown in the table below, after fine-tuning, a larger proportion of the combined third- and fourth-order terms falls within intervals centered at zero across all evaluated models and datasets.
We acknowledge that the theoretical error bounds apply to the full-CGF formulation and do not directly bound the second-order truncation error. However, our primary objective is to preserve downstream inference accuracy rather than precisely approximate individual softmax outputs. As demonstrated in Figures 1 and 2, the practical second-order CGF-softmax maintains inference accuracy within one percentage point of the exact-softmax baseline after model adaptation, supporting its effectiveness across diverse models and tasks.
(Q1)
We would like to clarify that our primary objective is to preserve downstream inference accuracy rather than precisely approximate individual softmax outputs. CGF-softmax is designed as an efficient, trainable softmax reformulation, where exact normalization is not required. Although the second-order cumulant approximation may introduce normalization bias, model adaptation allows CGF-softmax to maintain high inference accuracy, as demonstrated in Figures 1 and 2. Therefore, our evaluation focuses on the accuracy–depth trade-off rather than minimizing the numerical discrepancy from exact softmax.
(W2, Q2)
We thank the reviewer for this suggestion. Our end-to-end FHE evaluation is based on Cerium, which implements ResNet-20, BERT-Base, and Llama-3-8B. To further validate the scalability of CGF-softmax, we additionally integrated it into Cerium's Llama-3-8B implementation.
We fine-tuned Llama-3-8B on Clinc150, Banking77, and SST-2 using the same knowledge distillation (KD) framework as in our original experiments. The exact-softmax achieved accuracies of 91.51%, 93.87%, and 94.08%, respectively, while CGF-softmax achieved 90.71%, 93.52%, and 93.41%, maintaining accuracy within one percentage point across all three benchmarks.
In Cerium's Llama-3-8B implementation, input sequence length fixed at 128, and the maximum available level after bootstrapping is 21. The original softmax starts at level 16 and outputs at level 9, whereas CGF-softmax starts at level 19 and consumes 10 levels to reach level 9 without internal bootstrapping.
Since Llama-3-8B could not be executed on our single GPU, we conducted the FHE latency evaluation using two NVIDIA RTX PRO 6000 Blackwell GPUs. The FHE inference latencies are reported below in seconds, excluding the model weight encoding time, which is common to both methods. For each component, latency is reported as the maximum execution time across the two GPUs, reflecting the critical path of parallel execution.
Similar to BERT-Base, the primary source of latency reduction in Llama-3-8B is the elimination of bootstrapping within softmax. Although the input sequence length remains 128, as in BERT-Base, the larger hidden dimension and model size of Llama-3-8B reduce the relative contribution of softmax to overall inference latency. Nevertheless, softmax latency decreases from 60.9 s to 6.09 s, achieving a 90.0% reduction, demonstrating that the computational advantages of CGF-softmax extend to larger-scale encrypted language models.