acceptodds
Under review as a conference paper at ICLR 2027

SPA: Structure-Preserving Approximation of Gated Activations under FHE

Abstract

Nonlinear activation evaluation remains a major bottleneck in neural inference under fully homomorphic encryption (FHE). FHE supports efficient addition and multiplication, but lacks native operations for exponential functions, reciprocals, and value-dependent selection. High-degree polynomial approximations on fixed intervals are therefore commonly used to replace nonlinear activations. They can achieve small errors but often incur substantial multiplicative depth and latency. We observe that several widely used nonlinear activations share the algebraic structure with . This class includes GELU, SiLU, fixed-parameter Swish, and the scalar nonlinear branches of GEGLU/SwiGLU. We propose SPA (Structure-Preserving Approximation) to exploit this shared structure. SPA retains the exact odd component and approximates only the even residual as a polynomial in , yielding a comparison-free arithmetic kernel. This construction removes redundant odd degrees of freedom and enables lower-cost polynomial evaluation at a given error target. Plaintext experiments with Gemma-4-E2B and Llama-3.2-3B show that SPA largely preserves downstream task performance without weight updates. Fixed-interval CKKS benchmarks show favorable error-latency trade-offs. On , SPA-balanced achieves a speedup over MOAI (ICLR'26) for erf-GELU with comparable error. On the same interval, SPA-accurate achieves 79.22% lower root mean squared error (RMSE) and a speedup over Euston (S&P'26) for QuickGELU. For SiLU, SPA-accurate achieves 19.02% lower RMSE and a speedup over THOR (CCS'25).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fully Homomorphic Encryption, Structure-Preserving Approximation, Activation Functions, Polynomial Approximation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9SamOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that GELU, SiLU, fixed-parameter Swish, and the scalar branches of GEGLU/SwiGLU share the structure A(x)=x g(x)A(x) = x\,g(x) with g(−x)=1−g(x)g(-x) = 1 - g(x), and proposes SPA, a comparison-free CKKS kernel that keeps the exact odd part x/2x/2 and approximates only the even residual as a Chebyshev polynomial in x2x^2, proving that this attains the same best uniform error as unrestricted polynomials of the corresponding degree and deriving coefficient-specific error bounds. Validation combines frozen-weight evaluations on Gemma-4-E2B (erf-GELU) and Llama-3.2-3B (SiLU), where SPA-balanced changes pooled MCQ accuracy by −0.04-0.04/+0.09+0.09 pp and perplexity by at most 0.21%0.21\%, with fixed-interval CKKS kernel benchmarks against seven prior methods, e.g., a 3.25×3.25\times speedup over MOAI on erf-GELU at about 3×3\times higher RMSE and a 10.40×10.40\times speedup with 79%79\% lower RMSE than Euston on QuickGELU.

Critical strengths

First, SPA is a simple drop-in kernel that uses fewer ciphertext–ciphertext multiplications and levels than every baseline (in shared CKKS contexts, except for Euston), e.g., 77 CT×CT\mathrm{CT}\times\mathrm{CT} products and 66 levels vs. MOAI's 2323 and 88 for erf-GELU on [−5,5][-5,5], while SPA-balanced keeps downstream accuracy close to the reference without retraining; this matters because lower depth can reduce bootstrapping requirements or free depth budget in a full FHE pipeline.

Second, the paper provides coefficient-specific continuous-interval error bounds for the balanced SiLU and QuickGELU configurations on [−5,5][-5,5] and [−20,20][-20,20] under explicitly stated arithmetic assumptions, together with a careful evaluation protocol (paired bootstrap intervals and full failure reporting), which makes its approximation claims easier to audit.

Critical weaknesses

First, the novelty is limited: the algebraic observation and direct residual approximation have clear precedents (BOLT's x2+poly(∣x∣)\tfrac{x}{2} + \mathrm{poly}(|x|) GELU approximation, the HE ReLU construction (x+x sign(x))/2(x + x\,\mathrm{sign}(x))/2, and sigmoid approximations of the form $\tfrac{1}{2} + $ odd polynomial), and the best-approximation equivalence in Theorem 1 follows directly from classical symmetrization, as the paper acknowledges; without a controlled comparison against MOAI-style gate-factor approximation, the paper does not isolate how much of its advantage comes from comparison-free evaluation, the fitting objective, and structural constraints rather than from degree selection, domain extension, and scheduling.

Second, parts of the experimental setup are inconsistent or underspecified: the SiLU [−5,5][-5,5] SPA-fast configuration has polynomial error Epoly=2.05×10−2E_{\mathrm{poly}} = 2.05 \times 10^{-2}, violating the paper's own 10−210^{-2} selection rule even though the k=7k = 7 candidate satisfies it; the baselines' polynomials and range handling on [−20,20][-20,20] and wide channels in the plaintext evaluations, the degrees and errors of the wide-channel polynomials, and test-time out-of-range rates are not reported; and SPA-fast's model-dependent perplexity response (−3.23%-3.23\% on Gemma vs. +3.70%+3.70\% on Llama, WikiText-2) is not analyzed; this weakens the fast-variant and baseline comparisons, and while the kernel gains and SPA-balanced results are supported, robustness outside the calibrated ranges remains unquantified.

Questions for authors
  1. Source of the gains. The MOAI implementation and configuration details referenced in Sections 2 and 5.2 point to a broken "Appendix ??". Please compare SPA with MOAI-style approximation A(x)≈x2(1+f(x))A(x) \approx \tfrac{x}{2}\bigl(1 + f(x)\bigr) with odd ff, which corresponds to the SPA space under the additional constraint q(0)=0q(0) = 0 and minimizes the gate error ∣(2g−1)−f∣|(2g - 1) - f| rather than the residual error, at matched final degree in xx (SPA's kk corresponds to degree 2k2k) and at matched achieved error, reporting latency and level cost. Such controls would clarify how much of the measured advantage comes from the fitting objective and structural constraints versus degree selection and evaluation scheduling.

  2. Selection rule and baselines.

  • For SiLU [−5,5][-5,5] SPA-fast, Epoly=2.05×10−2E_{\mathrm{poly}} = 2.05 \times 10^{-2} exceeds the 10−210^{-2} tolerance, whereas the k=7k = 7 candidate (1.78×10−41.78 \times 10^{-4}) satisfies it. Was a different rule applied, or is this an error, and how do the Llama SPA-fast results change under a compliant configuration?
  • The ciphertext tables report no [−20,20][-20,20] configurations for ENSI/Sylph, and Table 1 marks THOR's SiLU support as unverified. Which polynomials and range-handling rules were used for these baselines on the [−20,20][-20,20] and wide channels in the plaintext evaluations, and which parts of THOR were reused versus re-approximated for SiLU?
  1. Range coverage and anomalies.
  • What are the test-time out-of-range rates in each range group, and what degrees and errors do the wide-channel (R=32R = 32–8080) polynomials have?
  • What explains SPA-fast's opposite perplexity responses on Gemma and Llama, and the single invalid SPA-accurate HellaSwag prediction (1/10,0421/10{,}042), e.g., nonfinite activations or scores?

Future-work suggestion: an end-to-end evaluation of at least one FFN block with mixed-radius packing, reporting bootstrapping counts, would show whether kernel-level savings translate into practical gains.

Reviewer_qD1JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies polynomial approximations of nonlinear activation functions for inference under fully homomorphic encryption (FHE). It considers activations satisfying A(x)=xg(x)A(x)=xg(x) and g(−x)=1−g(x)g(-x)=1-g(x), including GELU, SiLU and Swish with a fixed parameter. SPA separates each activation into the linear odd component x/2x/2 and an even residual. After substituting u=x2u=x^2, the method approximates only the even residual using Chebyshev polynomials obtained through discrete minimax fitting. The resulting polynomial is evaluated homomorphically using a Paterson–Stockmeyer (PS) block evaluation scheme.

The theoretical analysis shows that imposing this structural constraint preserves the best uniform approximation error at the corresponding polynomial degrees. It also gives error bounds for the chosen coefficients over finite intervals, coefficient perturbation bounds, and a decomposition of approximation and homomorphic evaluation errors. Experiments assess task performance after activation replacement in frozen-weight language models, alongside the accuracy and computational cost of encrypted activation kernels.

Critical strengths
  1. The construction is simple and retains the known odd component exactly. The approximation equivalence and coefficient perturbation bounds give it a clear theoretical foundation.

  2. The experiments cover several activation functions and two language models. Results are reported transparently, including paired confidence intervals, failed cases, and homomorphic operation counts.

Critical weaknesses
  1. The comparisons do not isolate the benefit of SPA’s representation from those of polynomial fitting and evaluation. An optimized even-polynomial baseline with the same fitting criterion, degree, and PS schedule would address this gap.
  2. The QuickGELU comparison uses different CKKS scales and modulus chains for SPA and Euston (Table 8). Both methods should be tuned to the same error target and security level to clarify the reported speedup.
  3. Table 7 provides static channel assignments but does not establish input-range coverage during evaluation. The out-of-range rate, handling of such inputs, and cost of wide-range channels remain unreported.
  4. Model performance is evaluated in plaintext, whereas CKKS performance is measured on standalone kernels. An encrypted multilayer experiment is needed to assess error accumulation and the cost of using different approximation ranges.
Questions for authors
  1. How does SPA compare with a direct even-Chebyshev baseline in error and runtime when the fitting criterion, degree, and PS evaluation schedule are held fixed?

  2. With range assignments fixed, how often do test inputs fall outside their assigned intervals, and by how much? What is the encrypted execution cost once wide-range channels are included?

  3. Could the authors evaluate an encrypted multilayer module to show how kernel errors and the number of levels consumed affect model outputs?

Reviewer_FmePOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies scalar gated activations A(x)=x g(x)A(x)=x\,g(x) with g(−x)=1−g(x)g(-x)=1-g(x), a class that includes erf-GELU, SiLU, and QuickGELU. For these functions the odd part is exactly x/2x/2.

SPA keeps x/2x/2 exactly and approximates only the even residual, as a Chebyshev polynomial in x2x^2. The polynomial is fitted by discrete minimax on [−5,5][-5,5] or [−20,20][-20,20] and evaluated with a Paterson–Stockmeyer schedule.

  • Theory: Theorem 1 shows that this structure loses no best-uniform-approximation accuracy. Theorem 2 bounds the error of the stored coefficients.
  • Experiments:
    • Plaintext activation replacement in frozen Gemma-4-E2B (Exact-erf) and Llama-3.2-3B.
    • Isolated CKKS kernels without bootstrapping. Reported speedups are 3.25× over MOAI, 10.40× over Euston, and 2.06× over THOR.
Critical strengths

S1. A provably lossless and fully specified structural reduction.

  • Theorem 1 shows that keeping the exact odd part loses nothing in best uniform error relative to polynomials of degree 2k2k or 2k+12k+1.
  • The closed-form evaluator reproduces every SPA operation and level count, so the method is auditable and needs no retraining.

S2. Strong kernel results.

In plaintext, SPA-balanced is accuracy-neutral: −0.04 pp on Gemma (CI [−0.1422, +0.0711]) and +0.09 pp on Llama (p. 7, ll. 350–351). Certified bounds are clearly separated from sampled-only checks.

Critical weaknesses

W1. The claim that the structure "enables lower-cost polynomial evaluation at a given error target" (Abstract, ll. 022–023) is never isolated from evaluator and baseline-implementation effects.

  • No ablation. SPA is never compared with an unstructured 2k2k/2k+12k+1 Chebyshev fit or a MOAI-style x[1+f(x)]/2x[1+f(x)]/2 fit under the same evaluator and context (not reported in Sec. 5 or App. C–H); fast/balanced/accurate is only a degree sweep (p. 4, ll. 198–200).
  • Theorem 1 makes this decisive. The unstructured fit attains the same best error (Eq. 7, p. 3), so any gain is an evaluation-cost effect, which is exactly what is unmeasured.
  • Baselines are thinly described: one sentence on MOAI (p. 6, ll. 299–302) plus aggregate counts (Table 2). MOAI also preserves the exact odd part (p. 2, ll. 100–101), yet uses 23 vs. 7 CT×CT at R5 (Table 2), partly due to "MOAI's domain extension" (p. 7, ll. 376–377).

Why consequential: The parity idea is elementary, so the contribution rests on efficiency that can be attributed to it, and that evidence is missing.

W2. Two headline comparisons are not error- or context-matched, and the evidence stops at fixed-interval kernels.

  • MOAI at R5. The abstract claims "comparable error" (l. 027), but SPA-balanced has 3.04× MOAI's RMSE (p. 7, l. 376). SPA-accurate shares the same polynomial at R5 (p. 7, l. 377), so no error-matched speedup is reported.
  • Euston context. Euston runs at 35-bit/1660 bits vs. SPA's 46-bit/1132 bits (Table 8, p. 17), partly because of Euston's own 40 levels (Table 2), so 10.40× mixes approximation cost with context choice.
  • Deployment gaps. There is no bootstrapping (p. 7, l. 326), no kernel for the required RwideR_{\rm wide} = 32–80 group (Table 7, p. 16), and no stated handling of out-of-range inputs (not reported in Sec. 5 or App. C–D).

Why consequential: The practical claim is lower encrypted-inference cost at preserved accuracy, but the evidence supports only kernel-level claims.

Questions for authors
  1. Same-evaluator ablation (W1). Please report SPA, an unstructured 2k2k/2k+12k+1 fit, and a MOAI-style fit at matched max error under one evaluator and context (CT×CT, levels, latency).
    • Effect on my score: A clear SPA advantage would raise my rating.
  2. Baselines (W2). Were the baselines original code or re-implementations, and with which degrees? Can you add an SPA point at MOAI's R5 error and run SPA in Euston's 35-bit context?
    • Effect on my score: Substantial matched speedups would resolve most of W2.
  3. Out-of-range and mixed-radius inputs. How are inputs with ∣x∣>R|x|>R handled in the plaintext experiments, and how are R5/R20/wide channels packed (App. D, cited at p. 7, l. 326, does not say)?
    • Effect on my score: No exact-function fallback would make the plaintext results deployment-relevant.
  4. Security and timing. What security level do the Table 8 contexts provide (not stated in the paper)? How many repetitions and threads underlie the "median batch latency" (Table 4 caption)?
    • Effect on my score: Standard 128-bit parameters would remove a soundness concern.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Commitments

We thank the reviewers for their careful evaluation and constructive discussion. We appreciate the reviewers' follow-up comments indicating that our responses and additional measurements have addressed their principal concerns. We also thank Reviewer 9Sam for the specific recommendations for the revision, which we will incorporate.

1. Contribution and evidence clarified through the discussion.

SPA provides a unified, configurable methodology for the polynomial approximation and homomorphic evaluation of GELU, SiLU, and other activation functions with the shared structure A(x)=xg(x)A(x)=xg(x), where g(−x)=1−g(x)g(-x)=1-g(x). One residual-fitting and evaluation procedure adapts coefficients, degree, and schedule to the target activation, public interval, and approximation budget; fast/balanced/accurate are operating points rather than three fixed methods. Building on established symmetry, SPA connects a lossless structural restriction to coefficient-specific error bounds and explicit, comparison-free homomorphic evaluation.

The additional controls distinguish approximation quality from evaluation cost. The tested direct-even and unstructured evaluators take 1.08–1.23× and 1.32–1.44× SPA's time, respectively. The balanced gate-factor controls take approximately 1.3–1.5× SPA's time. Converting the gate-factor controls to residual form recovers SPA's operation counts and levels, demonstrating that the evaluation procedure can also benefit coefficients obtained by other fitting approaches. We will report these controlled gains alongside the larger ratios against specific baseline implementations, without attributing the entire difference to the algebraic representation.

Following Reviewer 9Sam's recommendation, the revised abstract will highlight the new erf-GELU k=15k=15 configuration on [−5,5][-5,5]: a 2.88× speedup over the evaluated MOAI implementation in the same CKKS context, with lower RMSE (2.90e-7 versus 1.53e-6). This replaces the ambiguous “comparable error” description of the original balanced comparison. QuickGELU comparisons will explicitly distinguish identical contexts from comparable total-modulus budgets.

2. Evidence beyond isolated activation kernels.

The added experiments include wide-range kernels, mixed-radius packing controls, and four independently evaluated encrypted Gemma FFN blocks. Each block includes the gate/up projections, activation, gated product, and down projection, without intermediate decryption or bootstrapping. SPA consumes 11–12 levels versus 17 for the tested MOAI configuration. These results connect the scalar construction to encrypted block computation and show why range grouping and chain-length selection matter in deployment. We will distinguish independent-block results from continuous multilayer or end-to-end model evaluation, and explicitly label the reported FFN error averages as using in-range tokens.

3. Incorporating the remaining revision requests.

We will make the out-of-range behavior explicit in the main text: non-padding inputs use their assigned polynomials without range-triggered clipping or exact-function fallback, and higher in-interval accuracy does not ensure more robust extrapolation. We will retain the observed accurate-tier failure and note that the lower-degree variants remained finite in that task. We will clarify that separate range-group evaluation mitigates the observed numerical amplification from shared high-degree intermediates in mixed-radius packing; it does not establish robustness for an out-of-range input evaluated within a single group. The revision will report the corresponding packing and computation costs alongside the stabilization results.

We will correct the stated fast tolerance to 2.1×10−22.1\times10^{-2}, make its application and selection rule explicit across the fast configurations, including wide ranges, and include the strict 10−210^{-2} Llama sensitivity check in the appendix. We will also restore the baseline appendix, repair broken references, and incorporate the new controls, range statistics, wide-channel profiles, and FFN results with their comparison conditions and measurement scopes.

We are grateful that the discussion has strengthened both the experimental evidence and the precision of our claims. We believe the resulting contribution is a useful, reproducible methodology for translating shared activation structure into configurable, low-cost encrypted computation, supported by theory, controlled comparisons, and measured block-level integration. Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.