CtrlFormer: Efficient Non-Interactive Secure Transformer Inference without Feature Explosion

Fully homomorphic encryption (FHE) enables non-interactive, secure transformer inference. Despite numerous efforts to optimize FHE algorithms and replace non-polynomial functions with FHE-friendly alternatives, existing approaches still suffer from feature explosion and low inference efficiency. To address these issues, we propose CtrlFormer, a novel FHE-based transformer inference framework. Firstly, we propose a bounded, FHE-friendly Softmax alternative that avoids the feature explosion present in many existing Softmax replacements. Secondly, we unify activation evaluation and bootstrapping into a single operation via functional bootstrapping, eliminating the need for separate activation evaluations that typically consume about ten multiplicative levels. Finally, by substantially reducing the multiplicative depth, CtrlFormer supports a smaller ciphertext modulus, enabling faster homomorphic matrix multiplications. CtrlFormer achieves an amortized time of 1.8 minutes per input for the revised BERT-base model on a single GPU, achieving a significant reduction in computational cost while improving performance on the GLUE benchmark compared to prior works.