- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes that GELU, SiLU, fixed-parameter Swish, and the scalar branches of GEGLU/SwiGLU share the structure with , and proposes SPA, a comparison-free CKKS kernel that keeps the exact odd part and approximates only the even residual as a Chebyshev polynomial in , proving that this attains the same best uniform error as unrestricted polynomials of the corresponding degree and deriving coefficient-specific error bounds. Validation combines frozen-weight evaluations on Gemma-4-E2B (erf-GELU) and Llama-3.2-3B (SiLU), where SPA-balanced changes pooled MCQ accuracy by / pp and perplexity by at most , with fixed-interval CKKS kernel benchmarks against seven prior methods, e.g., a speedup over MOAI on erf-GELU at about higher RMSE and a speedup with lower RMSE than Euston on QuickGELU.
Critical strengths
First, SPA is a simple drop-in kernel that uses fewer ciphertext–ciphertext multiplications and levels than every baseline (in shared CKKS contexts, except for Euston), e.g., products and levels vs. MOAI's and for erf-GELU on , while SPA-balanced keeps downstream accuracy close to the reference without retraining; this matters because lower depth can reduce bootstrapping requirements or free depth budget in a full FHE pipeline.
Second, the paper provides coefficient-specific continuous-interval error bounds for the balanced SiLU and QuickGELU configurations on and under explicitly stated arithmetic assumptions, together with a careful evaluation protocol (paired bootstrap intervals and full failure reporting), which makes its approximation claims easier to audit.
Critical weaknesses
First, the novelty is limited: the algebraic observation and direct residual approximation have clear precedents (BOLT's GELU approximation, the HE ReLU construction , and sigmoid approximations of the form $\tfrac{1}{2} + $ odd polynomial), and the best-approximation equivalence in Theorem 1 follows directly from classical symmetrization, as the paper acknowledges; without a controlled comparison against MOAI-style gate-factor approximation, the paper does not isolate how much of its advantage comes from comparison-free evaluation, the fitting objective, and structural constraints rather than from degree selection, domain extension, and scheduling.
Second, parts of the experimental setup are inconsistent or underspecified: the SiLU SPA-fast configuration has polynomial error , violating the paper's own selection rule even though the candidate satisfies it; the baselines' polynomials and range handling on and wide channels in the plaintext evaluations, the degrees and errors of the wide-channel polynomials, and test-time out-of-range rates are not reported; and SPA-fast's model-dependent perplexity response ( on Gemma vs. on Llama, WikiText-2) is not analyzed; this weakens the fast-variant and baseline comparisons, and while the kernel gains and SPA-balanced results are supported, robustness outside the calibrated ranges remains unquantified.
Questions for authors
-
Source of the gains. The MOAI implementation and configuration details referenced in Sections 2 and 5.2 point to a broken "Appendix ??". Please compare SPA with MOAI-style approximation with odd , which corresponds to the SPA space under the additional constraint and minimizes the gate error rather than the residual error, at matched final degree in (SPA's corresponds to degree ) and at matched achieved error, reporting latency and level cost. Such controls would clarify how much of the measured advantage comes from the fitting objective and structural constraints versus degree selection and evaluation scheduling.
-
Selection rule and baselines.
- For SiLU SPA-fast, exceeds the tolerance, whereas the candidate () satisfies it. Was a different rule applied, or is this an error, and how do the Llama SPA-fast results change under a compliant configuration?
- The ciphertext tables report no configurations for ENSI/Sylph, and Table 1 marks THOR's SiLU support as unverified. Which polynomials and range-handling rules were used for these baselines on the and wide channels in the plaintext evaluations, and which parts of THOR were reused versus re-approximated for SiLU?
- Range coverage and anomalies.
- What are the test-time out-of-range rates in each range group, and what degrees and errors do the wide-channel (–) polynomials have?
- What explains SPA-fast's opposite perplexity responses on Gemma and Llama, and the single invalid SPA-accurate HellaSwag prediction (), e.g., nonfinite activations or scores?
Future-work suggestion: an end-to-end evaluation of at least one FFN block with mixed-radius packing, reporting bootstrapping counts, would show whether kernel-level savings translate into practical gains.
General Response and Revision Commitments
We thank the reviewers for their careful evaluation and constructive discussion. We appreciate the reviewers' follow-up comments indicating that our responses and additional measurements have addressed their principal concerns. We also thank Reviewer 9Sam for the specific recommendations for the revision, which we will incorporate.
1. Contribution and evidence clarified through the discussion.
SPA provides a unified, configurable methodology for the polynomial approximation and homomorphic evaluation of GELU, SiLU, and other activation functions with the shared structure A(x)=xg(x), where g(−x)=1−g(x). One residual-fitting and evaluation procedure adapts coefficients, degree, and schedule to the target activation, public interval, and approximation budget; fast/balanced/accurate are operating points rather than three fixed methods. Building on established symmetry, SPA connects a lossless structural restriction to coefficient-specific error bounds and explicit, comparison-free homomorphic evaluation.
The additional controls distinguish approximation quality from evaluation cost. The tested direct-even and unstructured evaluators take 1.08–1.23× and 1.32–1.44× SPA's time, respectively. The balanced gate-factor controls take approximately 1.3–1.5× SPA's time. Converting the gate-factor controls to residual form recovers SPA's operation counts and levels, demonstrating that the evaluation procedure can also benefit coefficients obtained by other fitting approaches. We will report these controlled gains alongside the larger ratios against specific baseline implementations, without attributing the entire difference to the algebraic representation.
Following Reviewer 9Sam's recommendation, the revised abstract will highlight the new erf-GELU k=15 configuration on [−5,5]: a 2.88× speedup over the evaluated MOAI implementation in the same CKKS context, with lower RMSE (2.90e-7 versus 1.53e-6). This replaces the ambiguous “comparable error” description of the original balanced comparison. QuickGELU comparisons will explicitly distinguish identical contexts from comparable total-modulus budgets.
2. Evidence beyond isolated activation kernels.
The added experiments include wide-range kernels, mixed-radius packing controls, and four independently evaluated encrypted Gemma FFN blocks. Each block includes the gate/up projections, activation, gated product, and down projection, without intermediate decryption or bootstrapping. SPA consumes 11–12 levels versus 17 for the tested MOAI configuration. These results connect the scalar construction to encrypted block computation and show why range grouping and chain-length selection matter in deployment. We will distinguish independent-block results from continuous multilayer or end-to-end model evaluation, and explicitly label the reported FFN error averages as using in-range tokens.
3. Incorporating the remaining revision requests.
We will make the out-of-range behavior explicit in the main text: non-padding inputs use their assigned polynomials without range-triggered clipping or exact-function fallback, and higher in-interval accuracy does not ensure more robust extrapolation. We will retain the observed accurate-tier failure and note that the lower-degree variants remained finite in that task. We will clarify that separate range-group evaluation mitigates the observed numerical amplification from shared high-degree intermediates in mixed-radius packing; it does not establish robustness for an out-of-range input evaluated within a single group. The revision will report the corresponding packing and computation costs alongside the stabilization results.
We will correct the stated fast tolerance to 2.1×10−2, make its application and selection rule explicit across the fast configurations, including wide ranges, and include the strict 10−2 Llama sensitivity check in the appendix. We will also restore the baseline appendix, repair broken references, and incorporate the new controls, range statistics, wide-channel profiles, and FFN results with their comparison conditions and measurement scopes.
We are grateful that the discussion has strengthened both the experimental evidence and the precision of our claims. We believe the resulting contribution is a useful, reproducible methodology for translating shared activation structure into configurable, low-cost encrypted computation, supported by theory, controlled comparisons, and measured block-level integration. Thank you for your consideration.