acceptodds
Under review as a conference paper at ICLR 2027

CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption

Abstract

Homomorphic encryption (HE) is a prominent framework for privacy-preserving machine learning, enabling inference directly on encrypted data. However, softmax, widely used in deep neural networks including transformers, remains challenging to evaluate under HE due to its multivariate structure, the large dynamic range induced by exponentiation, and costly division. In this paper, we propose CGF-softmax, which reformulates the softmax denominator through the cumulant generating function (CGF). By eliminating both homomorphic division and explicit maximum subtraction, this reformulation substantially reduces multiplicative depth while preserving key properties of softmax. Extensive experiments on Vision Transformers and large language models show that CGF-softmax provides an efficient and accurate approximation of softmax in encrypted inference. In particular, it achieves inference accuracy close to that of high-depth exact methods, while requiring substantially lower computational cost through reduced multiplicative depth. Integrated into a state-of-the-art multi-GPU FHE inference framework, replacing only the softmax operation reduces end-to-end encrypted BERT-Base inference latency by 36.3% (one GPU) and 39.6% (two GPUs), by eliminating all bootstrapping inside the softmax.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Homomorphic Encryption, Privacy Preserving Machine Learning, Softmax, Moment Generating Function, Cumulant Generating Function, Polynomial Approximation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QvqHOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the high computational cost of Softmax operations in homomorphic encryption (HE) environments. The authors propose CGF-Softmax, an approach that reformulates the Softmax normalization process using the Cumulant Generating Function (CGF). By employing a second-order approximation—retaining only the mean and variance—the method avoids explicit maximum subtraction and homomorphic division, thereby reducing computational overhead. The paper further analyzes the theoretical properties and approximation errors of this method and evaluates the trade-off between accuracy and multiplicative depth across models such as Llama-3.2-1B, ViT, and DeiT. Finally, the authors integrate CGF-Softmax into Cerium, an existing FHE inference framework, and demonstrate improved efficiency for end-to-end encrypted inference using BERT-Base.

Critical strengths
  1. The research problem is clearly defined, and the proposed method holds practical significance: Non-linear operations in homomorphic encryption environments are inherently computationally expensive, and Softmax is a widely used operation in Transformer architectures. To address this, the authors reconstruct the Softmax normalization process using CGF, replacing costly computational steps with relatively simple statistical operations. I find this approach reasonable; it targets a genuine bottleneck in current encrypted inference while achieving a favorable balance between computational efficiency and model accuracy.

  2. The experimental evaluation is comprehensive, validating not only the algorithm itself but also its performance in a practical system: The authors first analyze the method's efficiency advantages in terms of multiplicative depth and computational complexity. They then conduct downstream task evaluations on language models and Vision Transformers to verify whether the original model accuracy is maintained after replacing the standard Softmax. Furthermore, I appreciate that the paper goes beyond algorithmic analysis; the authors integrate the method into Cerium—a real-world FHE inference framework—and achieve a 36%–40% reduction in end-to-end inference latency on BERT-Base. These experiments provide compelling evidence for the method's practical utility.

Critical weaknesses
  1. There remains a gap between the theoretical analysis and the second-order approximation used in practice: The paper provides a theoretical analysis of the properties and asymptotic error of CGF-Softmax; however, the primary error analysis is based on the full-CGF formulation, whereas the practical implementation retains only the mean and variance, omitting higher-order cumulants. Although the authors analyze changes in certain higher-order cumulants before and after fine-tuning in Appendix E, this does not fully capture the error introduced by the second-order truncation, nor does it sufficiently demonstrate the stability of this approximation across different models and input distributions. Given that the second-order approximation is a crucial component of the actual algorithm, I believe the current theoretical analysis provides insufficient support for the practical method.

  2. The scope of experimental validation for end-to-end encrypted inference remains limited: The paper evaluates the trade-off between accuracy and multiplicative depth using Llama-3.2-1B and several ViT/DeiT models, representing a reasonable scope of coverage. However, experiments involving the actual deployment of the method within an FHE system to measure end-to-end inference latency are limited to the RTE task on BERT-Base. While the results demonstrate significant efficiency gains, direct system-level experimental validation is currently lacking for the other large-scale models evaluated in the paper. Consequently, I believe the actual speedup effects across a broader range of models and tasks require further confirmation.

Questions for authors
  1. Regarding the second-order cumulant approximation used in Section 3.4: Based on existing experimental results, could you further elaborate on the magnitude of the approximation error or normalization bias between the practical CGF-Softmax and the original Softmax? Specifically, do these errors change significantly before and after fine-tuning?

  2. Given that end-to-end FHE latency is evaluated only on BERT-Base, could the authors further clarify, based on the existing results, how the reported reductions in multiplicative depth might translate into practical end-to-end speedups for Llama and ViT/DeiT?

2 comments

We thank the reviewer for the constructive feedback and suggestions.


(W1)

… this does not fully capture the error introduced by the second-order truncation, nor does it sufficiently demonstrate the stability of this approximation across different models and input distributions. …

We thank the reviewer for raising this concern. To further examine the behavior of higher-order cumulants across different models and datasets, we extended the analysis in Appendix E to DeiT-Tiny/Base, ViT-Tiny, and Llama-3.2-1B on Banking77, Clinc150, and SST-2. The additional results, presented below, illustrate the behavior of the omitted third- and fourth-order cumulant terms before and after fine-tuning. As shown in the table below, after fine-tuning, a larger proportion of the combined third- and fourth-order terms falls within intervals centered at zero across all evaluated models and datasets.

We acknowledge that the theoretical error bounds apply to the full-CGF formulation and do not directly bound the second-order truncation error. However, our primary objective is to preserve downstream inference accuracy rather than precisely approximate individual softmax outputs. As demonstrated in Figures 1 and 2, the practical second-order CGF-softmax maintains inference accuracy within one percentage point of the exact-softmax baseline after model adaptation, supporting its effectiveness across diverse models and tasks.

ModelDataset[-1,1) before fine-tuning[-1,1) after fine-tuning[-2,2) before fine-tuning[-2,2) after fine-tuning[-3,3) before fine-tuning[-3,3) after fine-tuning
DeiT-TinyImageNet-1k80.47%96.89%87.52%99.60%90.05%99.92%
ViT-TinyImageNet-1k62.95%92.47%78.71%98.06%85.28%99.32%
ViT-BaseImageNet-1k66.58%91.58%80.52%95.47%86.39%96.36%
Llama-3.2-1BClinc15063.66%77.98%78.66%89.19%85.76%93.43%
Llama-3.2-1BBanking7757.48%78.29%73.71%90.10%82.04%94.28%
Llama-3.2-1BSST-251.53%82.23%67.60%92.95%76.75%96.09%

(Q1)

We would like to clarify that our primary objective is to preserve downstream inference accuracy rather than precisely approximate individual softmax outputs. CGF-softmax is designed as an efficient, trainable softmax reformulation, where exact normalization is not required. Although the second-order cumulant approximation may introduce normalization bias, model adaptation allows CGF-softmax to maintain high inference accuracy, as demonstrated in Figures 1 and 2. Therefore, our evaluation focuses on the accuracy–depth trade-off rather than minimizing the numerical discrepancy from exact softmax.

(W2, Q2)

We thank the reviewer for this suggestion. Our end-to-end FHE evaluation is based on Cerium, which implements ResNet-20, BERT-Base, and Llama-3-8B. To further validate the scalability of CGF-softmax, we additionally integrated it into Cerium's Llama-3-8B implementation.

We fine-tuned Llama-3-8B on Clinc150, Banking77, and SST-2 using the same knowledge distillation (KD) framework as in our original experiments. The exact-softmax achieved accuracies of 91.51%, 93.87%, and 94.08%, respectively, while CGF-softmax achieved 90.71%, 93.52%, and 93.41%, maintaining accuracy within one percentage point across all three benchmarks.

In Cerium's Llama-3-8B implementation, input sequence length fixed at 128, and the maximum available level after bootstrapping is 21. The original softmax starts at level 16 and outputs at level 9, whereas CGF-softmax starts at level 19 and consumes 10 levels to reach level 9 without internal bootstrapping.

Since Llama-3-8B could not be executed on our single GPU, we conducted the FHE latency evaluation using two NVIDIA RTX PRO 6000 Blackwell GPUs. The FHE inference latencies are reported below in seconds, excluding the model weight encoding time, which is common to both methods. For each component, latency is reported as the maximum execution time across the two GPUs, reflecting the critical path of parallel execution.

OperationComponentOriginalCGF-softmax
SoftmaxNon-bootstrapping10.416.09
Bootstrapping50.49-
OthersNon-bootstrapping28.8528.84
Bootstrapping289.25287.96
Total369.37314.87

Similar to BERT-Base, the primary source of latency reduction in Llama-3-8B is the elimination of bootstrapping within softmax. Although the input sequence length remains 128, as in BERT-Base, the larger hidden dimension and model size of Llama-3-8B reduce the relative contribution of softmax to overall inference latency. Nevertheless, softmax latency decreases from 60.9 s to 6.09 s, achieving a 90.0% reduction, demonstrating that the computational advantages of CGF-softmax extend to larger-scale encrypted language models.

Reviewer_QvqHOpenReview ↗

Response to Authors

I appreciate the author’s clarification regarding my questions and the additional experiments provided. I believe the author has effectively addressed my primary concerns about the paper; therefore, I will raise my score.

Reviewer_eWYVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies softmax computation for inference under homomorphic encryption (HE). The authors replace the sample mean of exponentials in the softmax denominator with a population expectation. Using the cumulant generating function (CGF), they write the output as (exp⁡(xi−KX(1)−ln⁡n))(\exp(x_i-K_X(1)-\ln n)). The implementation uses the mean and variance of each input row to construct the second-order approximation (KX(1)≈μ+σ2/2)(K_X(1)\approx\mu+\sigma^2/2). The exponential is then evaluated through polynomial approximation, domain scaling, and repeated squaring, avoiding explicit homomorphic division and maximum computation. The paper also introduces a variant that uses a low-degree exponential polynomial during training.

Critical strengths
  1. In addition to operation counts, the paper provides end-to-end measurements and a latency breakdown for encrypted BERT inference. These measurements support the reported speedups with evidence from an implemented system.

  2. The evaluation covers language and vision models, with BPMax tuning, knowledge distillation and an alternative supervised fine-tuning procedure. These experiments document how the softmax replacement affects accuracy and multiplicative depth.

Critical weaknesses
  1. Equation (5) requires conditions ensuring convergence of the cumulant expansion at �0��0�. For example, if �1��1� takes values �2��2� with equal probability, the series has a radius of convergence of �3��3�.
  2. The error analysis in Section 3.3 concerns the full population CGF, while the implementation uses sample statistics and a second-order truncation. Appendix E does not report the resulting normalization error in the final outputs.
  3. Under the Table 7 setting, $d=1d=1 $ and �5��5�, a row of 128 identical logits produces negative outputs. The paper should clarify which properties of the ideal formulation still hold for this variant.
  4. Appendix F does not explain how padding and causal masks are handled when computing the row-wise mean, variance, and effective length. These omissions make the implementation difficult to assess and reproduce.
Questions for authors
  1. After adaptation, how far do output row sums deviate from one, and how often are outputs negative? How accurately does the second-order approximation estimate the exact log-normalizer?

  2. Does the low-degree Llama implementation use any scaling or correction that is not described in the paper? How does it handle the identical-logit example that produces negative outputs under the stated formula?

  3. How are the mean, variance, and effective �0��0� computed when padding and causal masks are present? Is the exponential approximation interval chosen using the inputs after these operations?

3 comments

We thank the reviewers for their careful reading and constructive feedback.


(W1) Equation (5) requires conditions ensuring convergence of the cumulant expansion at t=1t=1. For example, if XX takes values ±2\pm2 with equal probability, the series has a radius of convergence of π/4<1\pi/4 <1.

We thank the reviewers for highlighting the need to state the convergence condition. We qualified Eq. (5) in the revised manuscript with the sufficient condition that KXK_X admits an analytic extension to a complex disk ∣t∣<R|t|<R with R>1R>1, following the standard Taylor expansion theorem.

While the reviewer’s specific two-point example demonstrates that the condition is not universal, it holds for many commonly used distributions, such as Gaussian distribution, as summarized in the table below. In particular, most of the distributions considered in our study satisfy R>1R>1.

DistributionCGF KX(t)K_X(t)Radius RRCondition for R>1R>1
Gaussian N(μ,σ2)N(\mu,\sigma^2)μt+σ2t2/2\mu t+\sigma^2t^2/2∞\inftyAlways
Poisson (λ\lambda)λ(et−1)\lambda(e^t-1)∞\inftyAlways
Uniform [−1,1][-1,1]log⁡(sinh⁡(t)t)\log\left(\frac{\sinh(t)}{t}\right)π\piAlways
Exponential, rate λ>0\lambda>0−log⁡(1−t/λ)-\log(1-t/\lambda)λ\lambdaλ>1\lambda>1
Equal-probability X=±2X=\pm2log⁡cosh⁡(2t)\log\cosh(2t)π/4\pi/4Not satisfied

In addition, Eq. (4) follows directly from the CGF definition and remains valid whenever MX(1)<∞M_X(1)<\infty, independently of convergence of the infinite cumulant series. Moreover, the practical algorithm evaluates a finite second-order approximation rather than the infinite series. With exact exponentiation, this formulation preserves positivity, ordering, exact pairwise ratios, and shift invariance; its output differs from softmax only by a row-dependent positive scale. Fine-tuning allows the model to adapt to the second-order CGF-softmax approximation based on the sample mean and variance, and we evaluate its practical effectiveness through downstream task accuracy and HE cost.

(W2) The error analysis in Section 3.3 concerns the full population CGF, while the implementation uses sample statistics and a second-order truncation. Appendix E does not report the resulting normalization error in the final outputs. (Q1) After adaptation, how far do output row sums deviate from one, and how often are outputs negative? How accurately does the second-order approximation estimate the exact log-normalizer?

Our objective is to preserve downstream inference accuracy at finite nn under a limited HE depth budget. The second-order instantiation is used during fine-tuning, allowing the model to adapt to our CGF reformulation. This is supported by our experiment results. CGF-softmax achieves 87.02%, 92.86%, and 92.43% accuracy on Clinc150, Banking77, and SST-2, respectively, compared with 86.67%, 92.99%, and 93.11% for the exact-softmax baseline (Fig. 1).

Regarding positivity, as established in Section 3.2 (P1) Positivity, CGF-softmax produces strictly positive outputs by construction, whereas BPMax does not guarantee positivity.

(W3) Under the Table 7 setting, d=1d=1 and x0=−3x_0 = -3, a row of 128 identical logits produces negative outputs. The paper should clarify which properties of the ideal formulation still hold for this variant. (Q2) Does the low-degree Llama implementation use any scaling or correction that is not described in the paper? How does it handle the identical-logit example that produces negative outputs under the stated formula?

We would like to clarify that no additional scaling or correction is applied in the low-degree Llama implementation beyond what is described in the paper. Also, unlike the CGF-softmax formulation presented as our main method, the low-degree variant does not satisfy all softmax properties.

Specifically, the degree-1 variant preserves (P2) ordering and (P3) shift invariance because its linear polynomial is strictly increasing and mean-centering makes its input invariant to a common logit shift. However, it does not guarantee (P1) positivity or preserve the exact exponential pairwise ratios in (P2). Thus, negative outputs can occur, including in the reviewer’s example of identical logits. As shown in Figs. 1 and 2, the low-degree variant generally achieves lower accuracy than the standard CGF-softmax.

For comparison, BPMax also preserves (P2) ordering when its polynomial degree pp is a positive odd integer, but it does not generally guarantee (P1) positivity, preserve the exact exponential pairwise ratios in (P2), or satisfy (P3) shift invariance. Nevertheless, our low-degree variant outperforms BPMax on Clinc150, Banking77, and SST-2 at the same multiplicative depth of 2 in the Llama setting (Fig. 1).

(W4) Appendix F does not explain how padding and causal masks are handled when computing the row-wise mean, variance, and effective length. These omissions make the implementation difficult to assess and reproduce. (Q3) How are the mean, variance, and effective nn computed when padding and causal masks are present? Is the exponential approximation interval chosen using the inputs after these operations?

We thank the reviewer for highlighting the need to clarify how the input length and masks are handled.

The length nn denotes the number of valid inputs to CGF-softmax. For ViT and DeiT, n=197n=197, corresponding to the full sequence length. For Llama, the valid length nrn_r can differ across attention rows and is determined by the number of unmasked positions in each row.

We have revised Appendix F of the manuscript to include an algorithm that handles both zero padding and causal masking. The algorithm applies a binary mask before each row-wise summation, ensuring that masked positions do not contribute to the mean or variance. To reduce multiplicative depth, consecutive plaintext multiplications, including masking and exponential domain scaling, are combined wherever possible. The valid length nrn_r is assumed to be public, so the normalization factors and ln⁡nr\ln n_r can be precomputed without additional homomorphic operations.

The exponential approximation interval is selected based on the inputs to the exponential function after computing the mask-aware statistics; masked positions are excluded from interval selection.

The mask-aware algorithm is also available at the following anonymized link:

https://github.com/cgf-anon/cgf-anon-ICLR2027/blob/main/Mask-Aware_Homomorphic_CGF-softmax.png

I appreciate the author’s detailed response and the additional analysis provided. The author has effectively addressed my primary concerns regarding the paper; therefore, I have decided to increase my score.

Reviewer_PPNeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CGF-softmax for CKKS-based encrypted inference. The softmax denominator is reinterpreted as n MX(1)n\,M_X(1), which gives exp⁡(xi−KX(1)−ln⁡n)\exp(x_i-K_X(1)-\ln n) with KXK_X the cumulant generating function (Eq. 4). In practice KX(1)K_X(1) is truncated to the row mean plus half the variance (Sec. 3.4). This removes max subtraction and division, and domain scaling reduces to kk squarings (Prop. 3.3). Props. 3.1–3.6 give properties and full-CGF error bounds. Experiments cover depth/latency vs. Cho et al. (Tables 3–4), post-adaptation accuracy on Llama-3.2-1B and ViT/DeiT vs. Cho et al. and BPMax (Figs. 1–2), and Cerium BERT-Base/RTE, with end-to-end latency −36.3% / −39.6% on 1 / 2 GPUs (Table 5).

Critical strengths
  1. Simple division- and max-free circuit that avoids softmax-internal bootstrapping. Its depth is k+6k+6, compared with ≥8k+9\ge 8k+9 for Cho et al.. Bootstrapping dominates softmax cost: 90.86 of 107.65 s for Cho et al.. This directly targets the main cost of non-interactive CKKS inference.

  2. Credible end-to-end gains at low adaptation cost. In Cerium, BERT-Base latency drops from 49.48 s to 31.51 s on 1 GPU. Table 9 attributes this almost entirely to softmax, whose cost falls from 17.84 s to 0.31 s. A LoRA pipeline stays within 1 pp of exact softmax on Llama-3.2-1B with the same 3 epochs.

Critical weaknesses
  1. The deployed second-order form is a trained surrogate, but it is framed as an accurate softmax approximation with an error bound.

    • The abstract claims an "accurate approximation of softmax in encrypted inference" (Abstract, l. 21–22, p. 1), and the Intro claims "an asymptotic bound on its approximation error" (Sec. 1, l. 78–79, p. 2). However, Props. 3.4–3.6 bound only the full-CGF form (Sec. 3.3, Eqs. 10–12, p. 6), and Sec. 3.4 concedes that the truncation error is not bounded (l. 346–347, p. 7).
    • The omitted terms κ3/3!+κ4/4!\kappa_3/3!+\kappa_4/4! stay heavy-tailed after fine-tuning (std 3.85, down from 6.81; App. E, l. 766–770, Fig. 3, p. 15). "Negligible impact on model performance" (l. 787–789) therefore concerns accuracy after adaptation, not the softmax values.
    • All accuracy results involve training with CGF-softmax in the loop: KD (App. J, l. 1015–1020), LoRA-SFT (Table 8), and RTE fine-tuning (App. K, l. 1041–1045). Training-free accuracy and a mean-only ablation are not reported (Sec. 4.2; App. E, I–J). Most depth comparisons are against approximation methods (Table 2, p. 7), and BPMax is the only trainable replacement compared (Sec. 4.2.1, l. 400–405).

    Why consequential: the headline result, near-exact accuracy at depth 7–10, supports a trainable surrogate that the theory does not cover. It does not yet show the value of the CGF variance term itself.

  2. The HE evaluation does not establish an accuracy–latency advantage over simpler division-free surrogates.

    • Cerium's softmax starts at Level 16 (App. K, l. 1062–1064, p. 20), and CGF-softmax uses depth nine (l. 1061–1062). Much of the gain would therefore apparently also be available to other shallow surrogates such as BPMax at depth 2 (Sec. 4.2.2, l. 427–428). Yet Table 5 compares only with native softmax, on one task and apparently one seed (App. I, l. 982–983).
    • The only HE error metric compares each method with its own plaintext evaluation (App. G, Eq. 36, l. 853–855). The reported 3.05e-10 (l. 386, p. 8) is therefore HE noise, not error relative to true softmax. Table 4's input distribution, kk, and AExp interval are not reported (Sec. 4.1, l. 372–377).
    • Beyond s=215s=2^{15} and L=10L=10 (Sec. 4.1, l. 373–374, p. 7), the paper does not report log⁡QP\log QP, scale, bootstrapping precision, or the security level (not reported in Sec. 2.1, Sec. 4.1, or App. F–K).

    Why consequential: the claimed accuracy–latency advantage over simpler surrogates is not verified under encryption, and the 33× micro-benchmark cannot be checked for security or precision.

Questions for authors
  1. What accuracy does second-order CGF-softmax give without fine-tuning? What is the per-layer distribution of LSE(x)−ln⁡n−μ^−σ^2/2\mathrm{LSE}(x)-\ln n-\hat\mu-\hat\sigma^2/2 before and after adaptation? Near-unit row sums and reasonable training-free accuracy would make me view the method as a genuine approximation and raise my score.

  2. Under the same KD budget (App. I), please compare against a mean-only shift exp⁡(xi−μ^−ln⁡n)\exp(x_i-\hat\mu-\ln n), a learned constant, and third-order truncation. Clear gains of the second-order form would show that the CGF view itself contributes.

  3. Which kk, AExp interval, and input distribution were used in Table 4? At L=10L=10, ⌊(k+6)/L⌋=0\lfloor(k+6)/L\rfloor=0 only for k≤3k\le3 (Table 3), so how does this cover [−128,0][-128,0] (l. 375)? How are causal masking and the zero padding of App. F.1 (l. 797–799) excluded from μ^,σ^2,ln⁡n\hat\mu,\hat\sigma^2,\ln n (Alg. 1, l. 825–834)? Full ≥128-bit parameters and error relative to true softmax would strengthen the efficiency claims.

  4. Could you add Cerium + BPMax under the same RTE adaptation, and at least one more task with several seeds? A clear accuracy advantage over BPMax at similar encrypted latency would move me towards acceptance.

4 comments

We thank the reviewer for the detailed review and constructive suggestions. In response, we clarify the scope of our method and model adaptation, provide additional comparisons and ablation studies, and expand our discussion of the HE implementation and evaluation. Please note that a new Appendix I has been added, so the original Appendices I, J, and K are now Appendices J, K, and L, respectively. We address each of the reviewer’s points individually below.


(W1) The abstract claims an "accurate approximation of softmax in encrypted inference" (Abstract, l. 21–22, p. 1), and the Intro claims "an asymptotic bound on its approximation error" (Sec. 1, l. 78–79, p. 2). However, Props. 3.4–3.6 bound only the full-CGF form (Sec. 3.3, Eqs. 10–12, p. 6), and Sec. 3.4 concedes that the truncation error is not bounded (l. 346–347, p. 7).

We thank the reviewer for pointing out this ambiguity. Our phrase "an efficient and accurate approximation of softmax" was intended to emphasize that CGF-softmax achieves high downstream inference accuracy with low computational cost, rather than accurately approximating individual softmax outputs. We replaced the sentence in the revised manuscript with “an efficient reformulation of softmax, which achieves high inference accuracy.”

The mathematical derivations and asymptotic error bounds in Propositions 3.4–3.6 are based on the full-CGF formulation.

(W1) The omitted terms κ33!+κ44!\frac{\kappa_3}{3!}+\frac{\kappa_4}{4!} stay heavy-tailed after fine-tuning. "Negligible impact on model performance" (l. 787–789) therefore concerns accuracy after adaptation, not the softmax values.

We would like to clarify that our primary objective is to preserve downstream inference accuracy rather than accurately approximating individual softmax outputs. Accordingly, we adopt the second-order cumulant approximation because it maintains high inference accuracy after model adaptation while minimizing the multiplicative depth required under HE. Higher-order cumulants are therefore omitted to avoid additional computational overhead.

(W1) All accuracy results involve training with CGF-softmax in the loop: KD (App. J, l. 1015–1020), LoRA-SFT (Table 8), and RTE fine-tuning (App. K, l. 1041–1045). Training-free accuracy and a mean-only ablation are not reported. (Q2) Under the same KD budget (App. I), please compare against a mean-only shift exp⁡(xi−μ^−ln⁡n)\exp\left(x_i-\hat{\mu}-\ln n\right), a learned constant, and third-order truncation.

Training-free accuracy. CGF-softmax is designed for use with model adaptation rather than as a training-free approximation of softmax. Similarly, softmax replacement methods such as BPMax and Power-softmax generally require additional training to recover model accuracy. Although CGF-softmax does not produce outputs identical to exact softmax, its reformulation preserves key softmax properties, including positivity, ordering, exact pairwise ratios, and shift invariance. After adaptation, CGF-softmax maintains high downstream inference accuracy at low multiplicative depth, including on larger models and more challenging benchmarks where BPMax exhibits substantial accuracy degradation.

Moreover, when task-specific supervised fine-tuning (SFT) is already required, CGF-softmax can be incorporated directly into that process without an additional adaptation stage, as demonstrated in Appendix K of the revised manuscript.

Mean-only ablation. Within this intended setting, we further examine whether the second-order approximation provides a meaningful benefit over a mean-only shift. Specifically, we conducted an additional ablation comparing first-, second-, and third-order cumulant approximations under the same fine-tuning budget on Llama-3.2-1B with Banking77. As shown below, the mean-only (first-order) approximation exhibits accuracy degradation, whereas both the second- and third-order approximations maintain accuracy within one percentage point of the exact-softmax baseline.

Cumulant OrderAccuracy (Exact Softmax: 0.9301)Depth
10.75297
20.92898
30.92219

These results support our choice of the second-order approximation, which achieves high inference accuracy at lower multiplicative depth than the third-order approximation. More broadly, CGF-softmax is a general cumulant-based reformulation encompassing all three variants; the second-order approximation is our preferred practical instantiation rather than the sole contribution.

(W1) Most depth comparisons are against approximation methods, and BPMax is the only trainable replacement compared.

To address the reviewer's concern, we additionally fine-tuned and evaluated Power-softmax [R1], which replaces the exponential function with a power function (xpx^p, with p=4p=4). Although Power-softmax achieves strong accuracy recovery after fine-tuning, it requires a multiplicative depth of 18–20 due to its remaining division operation. The original Power-softmax paper also reports a depth consumption of 22 for Goldschmidt's division algorithm. Thus, while Power-softmax is effective in recovering inference accuracy, it remains computationally inefficient in terms of multiplicative depth. We have included these additional results in Appendix I of the revised manuscript. We also provide an updated accuracy–depth comparison incorporating Power-softmax at the following anonymized link:

https://github.com/cgf-anon/cgf-anon-ICLR2027/blob/main/LLaMA_Pareto_with_PowerSoftmax.png

[R1] Zimerman et al., “Power-softmax: Towards secure LLM inference over encrypted data,” AISTATS 2026.

(W2) … Much of the gain would therefore apparently also be available to other shallow surrogates such as BPMax at depth 2. Yet Table 5 compares only with native softmax, on one task and apparently one seed. (Q4) Could you add Cerium + BPMax under the same RTE adaptation, and at least one more task with several seeds?

Powerformer evaluates BPMax only on BERT-Base, focusing on recovering inference accuracy on GLUE benchmarks, including RTE, MRPC, and SST-2. In contrast, our main evaluation focuses on larger models and more challenging benchmarks, where BPMax exhibits substantial accuracy degradation.

Nevertheless, to address the reviewer's concern, we additionally evaluated CGF-softmax and BPMax on BERT-Base using RTE and MRPC. We fine-tuned BPMax with three random seeds (0, 42, and 123) and learning rates of 1×10−51\times10^{-5}, 3×10−53\times10^{-5}, and 5×10−55\times10^{-5}. The best accuracies of CGF-softmax were 69.43%/86.36% on RTE/MRPC, compared with 69.62%/85.05% for BPMax and 69.31%/86.03% for the exact-softmax.

For FHE evaluation, BPMax uses a degree-5 polynomial with multiplicative depth 3, and we optimized its level configuration from 16→13 to 10→7. On a single GPU, the end-to-end latency decreased from 49.48 s (native Cerium) to 31.51 s (CGF-softmax) and 30.80 s (BPMax). Similarly, on two GPUs, it decreased from 31.96 s (native Cerium) to 19.30 s (CGF-softmax) and 19.20 s (BPMax).

OperationComponentOriginalCGF-softmaxBPMax
SoftmaxNon-bootstrapping3.08±0.000.31±0.000.02±0.00
Bootstrapping14.76±0.01--
OthersNon-bootstrapping19.38±0.0118.93±0.0218.67±0.02
Bootstrapping12.25±0.0112.26±0.0212.01±0.01
Total49.48±0.0331.51±0.0430.80±0.03

As shown in Appendix L (Table 9) of the revised manuscript, the primary source of CGF-softmax's latency reduction is the elimination of bootstrapping within softmax. Since both CGF-softmax and BPMax achieve this, their end-to-end inference latencies differ only marginally.

Additional Cerium Llama-3-8B Evaluation

To further examine the accuracy–latency trade-off on larger models, we additionally evaluated CGF-softmax and BPMax on Cerium’s Llama-3-8B using Clinc150, Banking77, and SST-2. We applied LoRA-based knowledge distillation (KD) rather than full-parameter fine-tuning to make multi-seed evaluation feasible. Both methods were trained for up to three epochs using three random seeds (0, 42, and 123) and learning rates of 5×10−55\times10^{-5} and 9×10−59\times10^{-5}. For BPMax, we adopted (p,c)=(5,5)(p,c)=(5,5), the best-performing configuration reported in Powerformer. In this setting, CGF-softmax achieved accuracy close to that of exact softmax, with multiplicative depths of 9, 8, and 9 on Clinc150, Banking77, and SST-2, respectively. In contrast, BPMax achieved substantially lower accuracy, although it required a multiplicative depth of only 3. The accuracy results are summarized in the table below.

DatasetsExact-softmaxCGF-softmaxBPMax
Clinc15097.33%97.44%87.67%
Banking7792.50%92.11%58.34%
SST-296.79%95.64%88.99%

We also integrated both methods into Cerium's Llama-3-8B implementation. The Cerium’s Llama-3-8B input sequence length was fixed at 128, and the maximum available level after bootstrapping was 21. The original softmax starts at level 16 and outputs at level 9 after multiple internal bootstrapping operations, whereas CGF-softmax starts at level 19 and consumes 10 levels to reach level 9 without internal bootstrapping. BPMax starts at level 12 and outputs at level 9, consuming only 3 levels.

The FHE inference latencies are reported below in seconds, measured using two NVIDIA RTX PRO 6000 Blackwell GPUs and excluding model weight encoding time, which is common to all methods. Component latencies represent the maximum execution time across the two GPUs.

OperationComponentOriginalCGF-softmaxBPMax
SoftmaxNon-bootstrapping10.416.092.81
Bootstrapping50.49--
OthersNon-bootstrapping28.8528.8428.84
Bootstrapping289.25287.96287.99
Total369.37314.87311.62

Similar to BERT-Base, the primary source of latency reduction in Llama-3-8B is the elimination of bootstrapping within softmax. Although the sequence length remains 128, the larger hidden dimension and model size reduce the relative contribution of softmax to overall inference latency. Nevertheless, CGF-softmax reduces softmax latency from 60.90 s to 6.09 s, a 90.0% reduction. While BPMax achieves slightly lower latency, it exhibits substantially greater accuracy degradation on all three benchmarks. These results demonstrate that CGF-softmax provides a more favorable accuracy–latency trade-off for larger-scale encrypted language models.

(W2) The only HE error metric compares each method with its own plaintext evaluation. The reported 3.05e-10 (l. 386, p. 8) is therefore HE noise, not error relative to true softmax.

Our objective is not to numerically reproduce exact softmax outputs, but to preserve downstream inference accuracy while reducing computational cost under HE. The error analysis in Appendix G is intended to demonstrate that CGF-softmax can be evaluated homomorphically with minimal numerical error relative to its plaintext evaluation.

(W2) Beyond s=215s=2^{15} and L=10L=10 (Sec. 4.1, l. 373–374, p. 7), the paper does not report log⁡QP\log QP, scale, bootstrapping precision, or the security level.

The bootstrapping precision has a mean absolute error of 8.51×10−78.51 \times 10^{-7} and a maximum absolute error of 3.05×10−63.05 \times 10^{-6}, with a security level of 128 bits. Although log⁡QP\log QP and scale values are not publicly disclosed by the DESILOFHE library, its documentation (https://fhe.desilo.dev/2.0.1/security_parameters/) explicitly states that the parameter configuration follows the ‘Security Guidelines for Implementing Homomorphic Encryption’ (https://eprint.iacr.org/2024/463.pdf).

(Q1) What accuracy does second-order CGF-softmax give without fine-tuning? What is the per-layer distribution of LSE⁡(x)−ln⁡n−μ^−σ^22\operatorname{LSE}(x)-\ln n-\hat{\mu}-\frac{\hat{\sigma}^{2}}{2} before and after adaptation?

CGF-softmax is designed to be used with model adaptation rather than as a training-free approximation of softmax. Although BPMax also employs fine-tuning, it exhibits substantial accuracy degradation on more challenging models and benchmarks, whereas CGF-softmax maintains high downstream inference accuracy. Moreover, when task-specific supervised fine-tuning (SFT) is already required, CGF-softmax can be incorporated directly into the SFT process without an additional adaptation stage, as demonstrated in Appendix K of the revised manuscript.

(W2) Table 4's input distribution, kk, and AExp interval are not reported. (Q3) Which kk, AExp interval, and input distribution were used in Table 4? At L=10L=10, ⌊(k+6)/L⌋=0\lfloor (k+6)/L \rfloor =0 only for k≤3k \le 3 (Table 3), so how does this cover [−128,0][-128,0] (l. 375)?

In Table 4, CGF-softmax uses k=4k=4 with inputs uniformly sampled from [−128,0][-128,0]. The exponential approximation operates over [−8,0][-8,0], and the original input range is handled using exp⁡(x)=(exp⁡(x/2k))2k\exp(x)=(\exp(x/2^k))^{2^k}, followed by kk squaring steps, as detailed in Appendix H.

Regarding the bootstrapping count, we identified an error in the formula in Table 3. With k=4k=4, CGF-softmax consumes exactly 10 multiplicative levels, matching the available level budget L=10L=10 after bootstrapping. Therefore, the computation completes without requiring bootstrapping. The correct formula is ⌊(k+6−1)/L⌋\lfloor(k+6-1)/L\rfloor. We have corrected the formula in the revised manuscript. We thank the reviewer for pointing out.

(Q3) How are causal masking and the zero padding of App. F.1 (l. 797–799) excluded from μ^, σ^2, ln⁡n\hat{\mu},\ \hat{\sigma}^{2},\ \ln n (Alg. 1, l. 825–834)?

We thank the reviewer for raising this question. To handle zero padding and causal masking, we define a binary mask as the element-wise product of the padding and causal masks. Each mask entry is 11 if the corresponding position is valid and 00 otherwise. The algorithm below uses this mask to exclude invalid positions from the computation of the mean and variance.

Specifically, the mask is applied before each row-wise summation to ensure that masked positions do not contribute to the computed statistics. To minimize multiplicative depth, consecutive plaintext multiplications, including masking and exponential domain scaling, are combined into a single multiplication wherever possible. Furthermore, as commonly assumed in prior work, the number of valid tokens in each row, nrn_r, is public, allowing the normalization factors and ln⁡nr\ln n_r to be precomputed without additional homomorphic operations.

We have revised Appendix F of the manuscript to include the mask-aware algorithm. We also provide the same algorithm at the following anonymized link:

https://github.com/cgf-anon/cgf-anon-ICLR2027/blob/main/Mask-Aware_Homomorphic_CGF-softmax.png

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.