acceptodds
Under review as a conference paper at ICLR 2027

CtrlFormer: Efficient Non-Interactive Secure Transformer Inference without Feature Explosion

Abstract

Fully homomorphic encryption (FHE) enables non-interactive, secure transformer inference. Despite numerous efforts to optimize FHE algorithms and replace non-polynomial functions with FHE-friendly alternatives, existing approaches still suffer from feature explosion and low inference efficiency. To address these issues, we propose CtrlFormer, a novel FHE-based transformer inference framework. Firstly, we propose a bounded, FHE-friendly Softmax alternative that avoids the feature explosion present in many existing Softmax replacements. Secondly, we unify activation evaluation and bootstrapping into a single operation via functional bootstrapping, eliminating the need for separate activation evaluations that typically consume about ten multiplicative levels. Finally, by substantially reducing the multiplicative depth, CtrlFormer supports a smaller ciphertext modulus, enabling faster homomorphic matrix multiplications. CtrlFormer achieves an amortized time of 1.8 minutes per input for the revised BERT-base model on a single GPU, achieving a significant reduction in computational cost while improving performance on the GLUE benchmark compared to prior works.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully homomorphic encryption, secure transformer inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MLmrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CtrlFormer, a secure transformer inference framework based on FHE. To reduce avoid the feature explosion and low inference efficiency of division operations, authors propose an alternative bounded FHE-friendly Softmax for non-polynomial function. Besides, authors also introduce DSF to replace standard LayerNorm to reduce computational cost.

Critical strengths

Authors design non-polynomial activation function, XControl, that operates in element-wise to constrain exponential outputs and improve numerical stability. Such design can reduce likelihood of feature explosion in prior polynomial Softmax alternatives.

Critical weaknesses
  1. Although authors claimed CtrlFormer can avoid feature explosion, there are lack of empirical experiments to report this statement.

  2. In Table 7, authors omitted the baseline such as ARION and NEXUS. It is important to report end-to-end inference time of these two methods to justify how CtrlFormer can improve inference speed with less multiplicative levels.

Questions for authors

In the proposed DSF, how the mean μ\mu is calculated? Is dynamically computed in token level on ciphertexts in inference stage or it is pre-computed during training.

Reviewer_DchyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CtrlFormer adapts a Transformer for non-interactive inference under fully homomorphic encryption (FHE). It replaces Softmax with the bounded XControl function and LayerNorm with the Dynamic Shift Function (DSF). DSF removes input-dependent variance normalization, and the modified model is trained through knowledge distillation. CKKS functional bootstrapping evaluates nonlinear functions while refreshing ciphertexts, and complex-slot packing and lazy operations reduce the remaining arithmetic costs.

In the main experiment, BERT-base processes sequences of 128 tokens on a single NVIDIA A100. The authors report an amortized inference time of 108.62 seconds per input. This is 3.17× faster than Powerformer and 1.40× faster than MOAI-P in the reported comparison. The mean score across RTE, MRPC and SST-2 is about 80.8, with the same scores for plaintext and encrypted inference. Appendix G also covers more tasks, longer sequences, BERT-large and generative models.

Critical strengths
  1. Combining function evaluation with ciphertext refreshing reduces the depth of the encrypted computation. CtrlFormer consumes only five multiplicative levels before entering functional bootstrapping in the reported pipeline. Tables 4, 5, and 7 examine the contributions of bootstrapping, matrix multiplication and the complete system. Together, they make a convincing case for the reported runtime improvements. The 3.17× speedup over Powerformer on the same A100 setup.

  2. XControl directly addresses the numerical-range problem introduced by unbounded Softmax replacements. Table 6 reports the same task scores for plaintext and encrypted inference. Appendix G extends the evaluation to longer sequences, BERT-large and generation tasks. This broader evaluation supports the method's practical value in the tested settings.

Critical weaknesses
  1. The authors have evaluated DSF, including accuracy and runtime ablations in Tables 3, 7 and 11. Figure 4 and Appendix G add numerical-error measurements and observations from generation experiments. These results show that the modified model works in the reported settings. They do not isolate whether DSF and distillation prevent variance degeneration, representation rank collapse or growth in feature norms. Figure 4 measures error between encrypted and plaintext computation, which cannot establish that the underlying representations remain stable.

  2. Replacing LayerNorm with DSF leaves the model's residual connections in place. Distillation supervises logits, embeddings and centered activations, rather than directly matching teacher weights. Such losses can encourage stable behavior, but do not establish stability across inputs and layers. I would like an ablation of DSF and the centered-activation loss, with the remaining training setup held fixed. Layerwise variance, norms, effective rank or singular values, gradients and failure rates across random seeds would provide more direct evidence. This matters for the paper's central claim that the revised architecture avoids feature explosion (Section 3, pp. 4-5; Tables 3 and 7, pp. 8-9; Table 11, p. 27; Figure 4, p. 9; Appendix G).

  3. Appendix B discusses boundedness, limits and smoothness, but does not establish approximation error or specify when the replacement preserves Softmax behavior. The hard-maximum argument in Equation 8 is particularly difficult to justify. For two positive logits, both XControl numerators approach one as the temperature parameter tends to positive infinity. A shared denominator therefore cannot make the weights concentrate on the larger logit. The fixed-denominator function also generally lacks Softmax's unit-sum constraint and translation invariance. XControl may still work well as a learned replacement, but the claimed probabilistic interpretation needs to be narrowed. The authors should also show how the training-derived denominator behaves when inference inputs differ in length or distribution (Appendix B, pp. 15-17, especially Equation 8 on p. 16).

  4. The 1.8-minute figure in Table 7 is an amortized time per input obtained through batching. I could not determine the actual batch size, full-batch wall-clock time and single-request response time together from the main results. Appendix E.2 explains the packing scheme, but those measurements are still needed to interpret the deployment benefit. The longer-sequence and generation experiments in Appendix G are useful additions to the evaluation. Their quality results, however, do not establish complete encrypted execution under the same settings. Please distinguish plaintext quality evaluations from encrypted timing measurements, and report the timing boundaries and peak GPU memory. This would help readers judge how the throughput gain affects request latency.

Questions for authors
  1. How is the mean in DSF computed, including the axes and statistical granularity? Can you ablate DSF and centered-activation distillation while keeping the teacher, residual connections and other training settings fixed? Please report layerwise stability diagnostics and failure rates across several random seeds. Which existing result isolates DSF's contribution to preventing representation degeneration during training or inference?

  2. What assumptions are needed for Equation 8's hard-maximum claim when several positive logits saturate together? How is the fixed denominator estimated and frozen, and how sensitive is it to sequence length or distribution changes?

  3. Are the Table 3 and Table 11 results repeatable across several random seeds? Please provide variability estimates for the roughly 0.2-point improvement over the matched distillation baseline. How do you interpret the lower mean score when DSF is added alone?

  4. What batch size and full-batch wall-clock time does each method use in Table 7? Does the timing include client encryption, decryption and data transfer, and what is the peak GPU memory? Please identify which longer-sequence, generation-quality and Table 18 measurements use plaintext or encrypted execution.

Reviewer_PRdxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Ctrlformer is a FHE based inference framework for transformers. The authors propose a bounded softmax alternative: XControl, and replace layernorm with a variant that drops the divisor. The nonlinearities are evaluated using CKKS functional bootstrapping via a lookup table. The results are interesting with a significant latency gain against Powerformer with similar scores.

Critical strengths
  1. The proposed algo improves over prior FHE approaches by computing the activations during the bootstrap step instead of using expensive polynomial approximations, thus reducing the required multiplication levels.
  2. The ablations are comprehensive and clearly show the effect of each design choice.
  3. Encrypted inference gives the same GLUE scores as plaintext on all three tasks, and the error stays bounded throughout.
Critical weaknesses
  1. The table 2 baseline should be a Plain BERT trained with the same distillation recipe (appendix 80.6 average) instead of BERT w/o distillation for fairness; CtrlFormer gets 80.8 and is lower on SST-2 (Table 11). The results also need error bars.

  2. Related work should also discuss MPC methods.

Questions for authors
  1. The 108.6 s/input number is amortized - how many inputs are in a batch, and how long does one batch actually take end to end?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.