Highly Expressive Activation Functions Based on Learnable Soft Quantization

Activation functions (AFs) provide the crucial nonlinearity that underpins the expressiveness of deep neural networks (DNNs). However, widely used AFs such as ReLU and GELU are simple and non-adaptive, limiting the flexibility of DNNs to capture intricate structures in data. In this paper, we introduce Soft Quantization Activation Functions (SQUAFs), a family of trainable, highly expressive AFs constructed as differentiable relaxations of generalized scalar quantizers. We show theoretically that SQUAFs can approximate any continuous 1D function defined over a closed interval with arbitrary precision. Extensive evaluations on standard image classification benchmarks show that SQUAF yields consistent gains over strong fixed and trainable AFs. Moreover, the advantage of SQUAFs is especially pronounced when a task involves modeling complex signals. For partial differential equation (PDE) solving, Physics-Informed Neural Networks (PINNs) equipped with SQUAFs attain the lowest relative error across nine out of ten PDE problems compared to strong baseline AFs. For implicit neural representations (INRs), SQUAF also consistently improves upon existing INR methods across sixteen 2D image representation cases and four 3D shape representation cases, preserving fine-scale details in the fitted signals. Additionally, we show, through knowledge distillation and model depth reduction experiments, that SQUAFs enable smaller models to outperform larger ones trained with conventional AFs, suggesting that increased AF expressiveness can compensate for reduced network size, thus achieving model compression. These results establish SQUAF as a general-purpose, parameter-efficient AF with particular strength in modeling complex signals.