acceptodds
Under review as a conference paper at ICLR 2027

Highly Expressive Activation Functions Based on Learnable Soft Quantization

Abstract

Activation functions (AFs) provide the crucial nonlinearity that underpins the expressiveness of deep neural networks (DNNs). However, widely used AFs such as ReLU and GELU are simple and non-adaptive, limiting the flexibility of DNNs to capture intricate structures in data. In this paper, we introduce Soft Quantization Activation Functions (SQUAFs), a family of trainable, highly expressive AFs constructed as differentiable relaxations of generalized scalar quantizers. We show theoretically that SQUAFs can approximate any continuous 1D function defined over a closed interval with arbitrary precision. Extensive evaluations on standard image classification benchmarks show that SQUAF yields consistent gains over strong fixed and trainable AFs. Moreover, the advantage of SQUAFs is especially pronounced when a task involves modeling complex signals. For partial differential equation (PDE) solving, Physics-Informed Neural Networks (PINNs) equipped with SQUAFs attain the lowest relative error across nine out of ten PDE problems compared to strong baseline AFs. For implicit neural representations (INRs), SQUAF also consistently improves upon existing INR methods across sixteen 2D image representation cases and four 3D shape representation cases, preserving fine-scale details in the fitted signals. Additionally, we show, through knowledge distillation and model depth reduction experiments, that SQUAFs enable smaller models to outperform larger ones trained with conventional AFs, suggesting that increased AF expressiveness can compensate for reduced network size, thus achieving model compression. These results establish SQUAF as a general-purpose, parameter-efficient AF with particular strength in modeling complex signals.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
activation function, quantization, nonlinearity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_VhJbOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose SQUAF, a trainable activation function shaped like a smooth step function. Each layer has one such function, and the step heights, spacing and sharpness are learned during training. Before training, each SQUAF is fitted to the network's usual activation (e.g. ReLU or GELU), so the network starts close to the original. The paper proves that SQUAF can approximate any continuous function of one variable. It reports gains on fitting simple functions, image classification (CIFAR-100, ImageNet), knowledge distillation, physics-informed networks for solving PDEs, and fitting single images and 3D shapes.

Critical strengths
  • The method is simple and has an intuitive reading as a soft, learnable step function. Fitting it to an existing activation before training is a practical way to initialize it.
  • Breadth of evaluation covers CNNs, ViTs, PINNs and INRs, and shows reported gains.
  • SQUAF is smooth with closed-form derivatives, so unlike piecewise activations it can be used where higher-order derivatives are needed, as in PINNs.
Critical weaknesses
  • The novelty of the method is limited. SQUAF is essentially a kernel activation function (KAF) with normalized weights and a learned grid, in an already crowded area (rational/PAU activations, KANs). KAF is tested only on a toy fit, where it does worse than all methods and looks untuned, and is then dropped. Rational and spline activations are never compared. The universality theorem holds for any such family.
  • Most examples are toy examples with PINN examples covering 10 of the 20 datasets (and skip many of the hard data), INR only on image fitting or shape fitting. No NERF or Blender Bot data. All models considered are quite small.
  • There are no error bars anywhere, and margins are often under 1 point. ImageNet has no learnable baselines, and on ViT-T every flexible-shape baseline is below GELU (which feels surprising). The gains are not significant as this method adds more latency.
  • It is not clear where the gains are coming from : on CIFAR, all but the first SQUAF is trained. No classification experiment compares frozen and trained SQUAFs. All models are trained from scratch. Finally the KD claim that students cannot beat their teachers is contradicted by standard KD methods on the same teacher–student pairs [1].
  • Page 7 with all the tables and figures is very hard to read and table 4 is almost incomprehensible.

[1] Decoupled Knowledge Distillation Zhao et al. CVPR 2022

Questions for authors
  • Does learning the activation matter? On CIFAR-100, how do all-frozen and all-trained SQUAFs compare? Alternatively, what happens when a pretrained ViT is fine-tuned with a GELU-fitted SQUAF vs. with GELU?
  • How does SQUAF compare with a tuned KAF and with rational (PAU) activations on ImageNet, with error bars over 3 seeds?
  • Does SQUAF help in more realistic settings, e.g. pretraining a language model such as GPT-2, or distilling from a strong teacher with the DeiT recipe [1]? Also what does the SQUAF student network get when trained normally?
  • Why do the Burgers 2d-C baselines in Table 4 differ from PINNacle's, and why does DiTAC collapse on ViT-T?

[1] Training data-efficient image transformers & distillation through attention Touvron et al. ICML 2021

10 comments
AuthorsOpenReview ↗

Response to Reviewer VhJb (part 1/6)

We thank the reviewer for taking time to review our paper and provide valuable feedbacks. Please find our point-by-point responses below. References to paper contents in our responses correspond to the revised manuscript unless otherwise specified.

Weakness 1: The novelty of the method is limited. SQUAF is closely related to KAF, rational and spline activations, where the universality theorem also holds, and SQUAF is not adequately compared with them.

Response: Our contribution is a general construction based on learnable soft quantization, together with its theoretical properties, closed-form derivatives, and effectiveness as an AF across a variety of model architectures and tasks. The SQUAF formulation is defined through a general CPMF satisfying the stated differentiability and localization conditions, with Gaussian and Laplacian constructions as examples. Theorem 1 establishes sufficient conditions for this construction to approximate continuous functions on a closed interval. Indeed, the universal approximation capability is shared by other function families. However, our universal approximation theorem (Theorem 1) does not automatically hold for any such family. For example, KAF generally fails the localization property P2, and therefore falls outside the assumptions of our theorem. In contrast, the Gaussian and Laplacian-based SQUAF constructions are compliant with our Theorem 1.

Nonetheless, we agree that more comparison with KAF, PAU and splines could better showcase the benefits of our proposed parameterization. To this end, we have added KAF, PAU and B-spline comparisons on all four 3D shape representation tasks in Appendix A.14. All methods use the same PEMLP backbone, one AF parameter set per layer shared across neurons, and initialization to approximate ReLU over [−0.8,0.8][-0.8,0.8]. All Gaussian-kernel variants use the same initial grid A^={0,±q,±2q,…,±kq}\hat{\mathcal{A}}=\{0, \pm q, \pm 2q, \dots, \pm kq\} as SQUAF, with initial q=0.05q=0.05 and k=16k=16. Full SQUAF learns (z,q,α)(\mathbf{z},q,\alpha) with normalized weights as in Eq. (7). Unnormalized SQUAF removes the normalization denominator while retaining these trainable parameters. KAF corresponds to unnormalized SQUAF with fixed (q,α)(q, \alpha); it trains only z\mathbf{z}. We also consider an input-scaled version of KAF ϕscaled(x)=∑i=−1616ziexp⁡[−α(sx−iq)2]\phi_{\mathrm{scaled}}(x)=\sum_{i=-16}^{16}z_i\exp[-\alpha(sx-iq)^2] as suggested by Reviewer DYwK, where qq and α\alpha are fixed while (z,s)(\mathbf{z},s) are trainable.

Tables 17–18 in Appendix A.14 (also shown below) show that full SQUAF achieves the best average Chamfer distance and IoU.

Chamfer distance (lower is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)3.453E-62.082E-62.313E-63.758E-62.901E-6
KAF3.350E-62.024E-62.124E-63.604E-62.776E-6
Input-scaled KAF3.337E-62.003E-62.123E-63.597E-62.765E-6
Unnormalized SQUAF3.336E-61.987E-62.070E-63.520E-62.728E-6
PAU3.391E-62.158E-62.028E-63.154E-62.683E-6
B-spline3.306E-61.948E-62.178E-65.124E-63.139E-6
SQUAF3.246E-61.915E-61.982E-63.026E-62.542E-6
SQUAF (z\mathbf{z} only)3.890E-62.114E-63.220E-63.683E-63.227E-6

IoU (higher is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)9.883E-19.815E-19.767E-19.599E-19.766E-1
KAF9.886E-19.799E-19.727E-19.611E-19.756E-1
Input-scaled KAF9.887E-19.802E-19.746E-19.614E-19.762E-1
Unnormalized SQUAF9.889E-19.814E-19.749E-19.627E-19.770E-1
PAU9.878E-19.774E-19.736E-19.639E-19.757E-1
B-spline9.893E-19.803E-19.733E-19.557E-19.747E-1
SQUAF9.902E-19.823E-19.786E-19.677E-19.797E-1
SQUAF (z\mathbf{z} only)9.872E-19.778E-19.585E-19.580E-19.704E-1

We also incorporate SQUAF's results with only z\mathbf{z} trainable from Table 9 of Appendix A.5 into the tables above, in order to present the following observations. Jointly learning the grid, smoothness and amplitudes is particularly important for the normalized SQUAF: its average Chamfer distance improves from 3.227×10−63.227\times10^{-6} with only z\mathbf{z} trained to 2.542×10−62.542\times10^{-6} with all parameters trained. For the unnormalized family, the corresponding improvement is smaller, from KAF's 2.776×10−62.776\times10^{-6} to 2.728×10−62.728\times10^{-6} of the unnormalized SQUAF. Overall, the combination of normalization and full parameter adaptation, i.e., our original SQUAF, is shown to be the best design.

AuthorsOpenReview ↗

Response to Reviewer VhJb (part 2/6)

Weakness 2: The evaluation mainly covers toy or small-scale cases, omits harder PINNacle problems, and does not include NeRF.

Response: In our original submission, the experimental evaluation includes tasks covering function fitting, image classification on CIFAR-100 and ImageNet-1K (including knowledge distillation and model depth reduction), ten PDE problems from PINNacle, standard 2D/3D INR tasks following the established benchmarks, with numerous evaluated model architectures and compared baseline AFs. It also includes preliminary results for GPT-2 fine-tuning and KAN architectures in appendices. To the best of our knowledge, no other work on AFs has demonstrated comparable experimental breadth to us. There are AF papers dedicated to regression, classification, INRs, KANs, PDE solving, or LLMs, while our evaluation covers all of these. We believe this breadth is important evidence for SQUAF's effectiveness as a general-purpose AF and should not be undervalued. As for the scale of individual experiments, we'd like to clarify that a single ImageNet-1K training run for the larger models considered in this paper (ViT-T and Swin-T) could easily take hundreds of GPU hours, and we believe that they should be considered as large-scale instead of toy experiments. It's also worth noting that since the results of baseline AFs are not generally available in the literature, we have had to reproduce many of them by ourselves, and this also considerably increased the scale of our experimentation. Overall, we believe the breadth and scale of our experiments provide a substantial evaluation of SQUAF’s effectiveness.

Nonetheless, as suggested by the reviewer, we now evaluate SQUAF on the remaining 12 PDE problems from PINNacle, extending beyond the ten cases considered in Sec. 4.3. Table 31 in Appendix A.21 (also shown below) reports the L2RE results for baselines Tanh, LAAF and GAAF (cited from the original PINNacle paper), and compares them with SQUAF results averaged over three runs. Note that given the limited time of the rebuttal period, we were unable to complete experiments for other ten baseline AFs in Table 4, so we only compare with the three baselines that have available results in the literature.

PDETanhLAAFGAAFSQUAF
Poisson 2d-CG6.36E-14.80E-18.71E-13.62E-1
Poisson 2d-MS6.30E-15.93E-19.31E-15.74E-1
Heat 2d-CG3.64E-22.39E-24.61E-11.87E-2
Heat 2d-LT9.99E-19.99E-19.99E-19.99E-1
NS 2d-LT9.96E-19.98E-19.99E-19.93E-1
Wave 1d-C5.88E-14.54E-16.77E-14.24E-1
Wave 2d-CG1.84E08.19E-17.94E-17.99E-1
Wave 2d-MS1.34E01.06E01.06E01.04E0
Chaotic GS3.19E-19.47E-29.46E-29.41E-2
Chaotic KS1.01E01.01E01.00E01.00E0
High dim PNd3.04E-34.14E-37.75E-32.41E-3
High dim HNd3.61E-15.22E-15.21E-12.68E-1

It's shown that SQUAF improves over the best listed baseline on 9 of the 12 cases and matches it on two at the reported precision. This shows that SQUAF is also effective on many of the harder PDE problems.

AuthorsOpenReview ↗

Response to Reviewer VhJb (part 3/6)

Moreover, per your suggestion, we extend the INR evaluation to novel-view synthesis with NeRF on eight scenes from the Blender dataset, following FINER's experimental setup [1]. The SQUAF initialization follows exactly the same as our 2D INR experiments. Table 32 in Appendix A.22 (also shown below) reports the results. All baseline results are cited from FINER's Table 3.

PSNR ↑\uparrow:

MethodsChairDrumsFicusHotdogLegoMaterialsMicShip
PEMLP31.3220.1824.4930.5925.9025.1626.3821.46
Gauss32.6823.1626.1032.1728.2926.1933.5922.28
SIREN33.3124.8927.2632.8529.6027.1333.2822.25
WIRE29.3122.2225.9130.1125.7625.0532.3521.15
FINER33.90‾\underline{33.90}24.90‾\underline{24.90}28.70‾\underline{28.70}33.05‾\underline{33.05}30.04‾\underline{30.04}27.0533.96‾\underline{33.96}22.47‾\underline{22.47}
SQUAF34.0125.2028.7233.1630.4427.06‾\underline{27.06}34.1022.64

SSIM ↑\uparrow:

MethodsChairDrumsFicusHotdogLegoMaterialsMicShip
PEMLP0.9600.8140.9140.9450.9040.9090.9600.754
Gauss0.9670.8830.9330.9560.9320.9160.9800.782
SIREN0.9710.912‾\underline{0.912}0.9470.960‾\underline{0.960}0.9480.9320.9790.788
WIRE0.9380.8580.9310.9380.8860.8980.9780.734
FINER0.973‾\underline{0.973}0.9110.958‾\underline{0.958}0.9590.951‾\underline{0.951}0.928‾\underline{0.928}0.981‾\underline{0.981}0.792‾\underline{0.792}
SQUAF0.9740.9130.9610.9610.9520.928‾\underline{0.928}0.9830.804

LPIPS ↓\downarrow:

MethodsChairDrumsFicusHotdogLegoMaterialsMicShip
PEMLP0.0260.1850.0560.0370.0700.0500.0570.190
Gauss0.0190.0820.0550.032‾\underline{0.032}0.0400.0370.0130.136
SIREN0.0170.051‾\underline{0.051}0.0330.032‾\underline{0.032}0.0300.0280.0140.116
WIRE0.0350.1060.0450.0480.0750.0580.0140.172
FINER0.015‾\underline{0.015}0.0520.024‾\underline{0.024}0.0330.025‾\underline{0.025}0.0320.010‾\underline{0.010}0.108‾\underline{0.108}
SQUAF0.0140.0500.0220.0280.0220.030‾\underline{0.030}0.0090.096

It's shown that SQUAF performs the best on 7 of the 8 scenes, and is the runner-up on the remaining one. These results extend SQUAF's effectiveness on INR beyond fitting images and SDFs.

Reviewer_B1h3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Soft Quantization Activation Functions (SQUAFs) - a novel family of trainable AFs that uses differentiable relaxations of generalized scalar quantizers to increase expressivity of deep neural networks. Theoretical results on universal approximation are provided, in addition to a suite of empirical results. The experiments demonstrate superior performance compared to baseline architectures in tasks including image classification, implicit neural representations of images and shapes, and PDE solution. SQUAFs are also framed as a method for model compression due to their parameter efficiency.

Critical strengths

The idea presented is novel and interesting and based on the results reported, the proposed method would be an asset to the toolkit of ML practitioners. It is clear from the range and diversity of experiments reported that the authors performed extensive empirical validation. Additionally, the theoretical proofs and derivations were rigorous and well-written.

Critical weaknesses

There were a couple instances where average performance was reported with no indication of standard deviations or error bars (eg. Table 5, Figure 2). In Figure 2, SQUAF’s average performance is better than the next best method (FINER), but because the performance margin is quite close, including the variance would make it easier to assess statistical significance.

The discussion on computational timing is omitted from the main body. It is mentioned in the appendix (Tables 13 and 14) but even a small discussion in the main text would improve transparency by showing the tradeoff.

Minor point: the ordering of figures and tables does not match the order of the corresponding text which makes the results section of the paper less easy to follow.

Questions for authors

Under what specific conditions do P1 and P2 fail to hold?

The results in Section 4.1 are strong, but how does it perform on highly non-sinusoidal functions? Given that Theorem 1 applies to continuous functions, how does it perform on functions with discontinuities? Are there any challenges with stability if the networks tries to push α\alpha to ∞\infty to capture sharpness of target functions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer B1h3 (part 1/2)

We thank the reviewer for taking time to review our paper and provide valuable feedbacks. Please find our point-by-point responses below. References to paper contents in our responses correspond to the revised manuscript unless otherwise specified.

Weakness 1: Some average results lack standard deviations or error bars.

Response: Thank you for the suggestion. We now report standard deviations (stds) for all results in the main text which were obtained by averaging over multiple runs. Please refer to Tables 26–28 in Appendix A.19 for stds of the CIFAR-100, PDE and KD results, respectively, and Figure 5 now includes error bars for the depth-reduction results.

For the original Fig. 2 (now Fig. 3) the reviewer referred to, we would like to clarify that the average reported there is the average performance across 16 different images, with one run per image for each compared method. This is different from the repeated-run averages discussed above. However, we agree with the reviewer that the margin between SQUAF and the runner-up FINER is relatively small and including stds could help assess statistical significance.

Therefore, we now repeat SQUAF and FINER experiments five times per image, respectively, and Table 29 (also shown below) reports these per-image PSNR means and standard deviations over 5 runs.

ImageFINERSQUAF
0035.90±0.0935.90\pm0.0937.03 ±\pm 0.21
0142.21±0.1342.21\pm0.1343.11 ±\pm 0.09
0239.30±0.1839.30\pm0.1840.52 ±\pm 0.07
0342.33±0.3742.33\pm0.3742.60 ±\pm 0.15
0440.57±0.0740.57\pm0.0742.22 ±\pm 0.15
0536.90±0.1036.90\pm0.1038.39 ±\pm 0.30
0641.51±0.1141.51\pm0.1142.90 ±\pm 0.07
0739.99±0.0639.99\pm0.0640.40 ±\pm 0.40
0836.74±0.1836.74\pm0.1838.14 ±\pm 0.05
0937.30±0.1537.30\pm0.1538.51 ±\pm 0.07
1045.91±0.1145.91\pm0.1146.25 ±\pm 0.25
1146.88±0.0646.88\pm0.0647.11 ±\pm 0.16
1242.09±0.0842.09\pm0.0842.49 ±\pm 0.22
1339.13±0.2439.13\pm0.2439.22 ±\pm 0.36
1443.07±0.2043.07\pm0.2043.71 ±\pm 0.43
1540.86±0.1540.86\pm0.1542.21 ±\pm 0.14
Avg.40.6741.55

SQUAF has higher mean PSNR on all 16 images; averaging the per-image means gives 41.55 dB for SQUAF and 40.67 dB for FINER. These repeated experiments support the improvement observed in the original single-run comparison.

Weakness 2: A brief discussion of computational timing in the main text would improve transparency.

Response: Thank you for this suggestion. We now add a brief discussion about it in the last paragraph of Section 4.

Weakness 3: The ordering of figures and tables makes the results section difficult to follow.

Response: We have rearranged the figures and tables to better align with the corresponding discussion, and transposed the PDE table to make numbers visually larger and easier to read.

Question 1: Under what specific conditions do P1 and P2 fail to hold?

Response:

  • P1 requires the CPMF Pα(y∣x)P_\alpha(\mathbf{y}\mid x), as a function of y\mathbf{y} and xx, to be differentiable almost everywhere. It fails for pathological CPMFs that are nondifferentiable on a set of positive measure. For example, transform the input with W(x)=∑n=0∞2−ncos⁡(13nπx)W(x)=\sum_{n=0}^{\infty}2^{-n}\cos(13^n\pi x), a continuous nowhere-differentiable Weierstrass function. Then define the CPMF as

    Pα(yi∣x)=exp⁡[−α(W(x)−yi)2]∑j=1Lexp⁡[−α(W(x)−yj)2],α>0.P_\alpha(y_i\mid x)=\frac{\exp[-\alpha(W(x)-y_i)^2]}{\sum_{j=1}^{L}\exp[-\alpha(W(x)-y_j)^2]},\qquad \alpha>0.

    This is a theoretical construction where P1 fails; however, if not deliberately constructing such adversarial examples, P1 holds for most practical cases. Note that P1 only requires differentiability almost everywhere: isolated nondifferentiable points, including those in the Laplacian CPMF, do not violate it.

  • P2 fails when the activation value is not localized to the convex hull of the two nearest amplitudes as α→∞\alpha\rightarrow\infty. For example, for a fixed ε∈(0,1)\varepsilon\in(0,1), define

    Gα,i(x)=e−α(x−yi)2∑j=1Le−α(x−yj)2,Pα(yi∣x)=(1−ε)Gα,i(x)+εL.G_{\alpha,i}(x)=\frac{e^{-\alpha(x-y_i)^2}}{\sum_{j=1}^L e^{-\alpha(x-y_j)^2}},\qquad P_\alpha(y_i\mid x)=(1-\varepsilon)G_{\alpha,i}(x)+\frac{\varepsilon}{L}.

    For this CPMF Pα(y∣x)P_\alpha(\mathbf{y}\mid x), every yiy_i retains probability mass at least ε/L\varepsilon/L, regardless of how large α\alpha is. Take A^={y1,y2,y3}={−1,0,1}\hat{\mathcal{A}}=\{y_1,y_2,y_3\}=\{-1,0,1\} and amplitudes (z1,z2,z3)=(0,0,1)(z_1,z_2,z_3)=(0,0,1). At x=−1x=-1, the amplitudes associated with the two nearest points in A^\hat{\mathcal{A}} are both zero, so their convex hull is {0}\{0\}. However, as α→∞\alpha\rightarrow\infty,

    =(1-\varepsilon)\frac{e^{-4\alpha}}{1+e^{-\alpha}+e^{-4\alpha}} +\frac{\varepsilon}{3} \longrightarrow\frac{\varepsilon}{3}>0.$$ Its distance from that convex hull is at least $\varepsilon/3$, regardless of how large $\alpha$ becomes. Thus, P2 fails because the non-local probability mass never disappears.
AuthorsOpenReview ↗

Response to Reviewer B1h3 (part 2/2)

Question 2: How does SQUAF perform on highly non-sinusoidal or discontinuous functions, and are there stability issues when sharp transitions encourage large α\alpha?

Response: We extend the 1D function fitting experiment in Sec. 4.1 to a discontinuous and highly non-sinusoidal function as defined in Eq. (42) of Appendix A.20. In Table 30 (also shown below), we report the fitting performance of SQUAF-MLP and two SQUAF-only cases as in Table 1, compared to the MLP equipped with the default AF ReLU and the Table 1 runner-up DiTAC.

AFParam.MSE$\downarrow$R2↑R^2\uparrow
ReLU1931.09E-292.25
DiTAC2021.96E-398.68
SQUAF2007.91E-499.43
SQUAF-only1925.00E-599.96
SQUAF-only281.23E-399.16

It can be seen that SQUAF-MLP performs better than ReLU-MLP and DiTAC-MLP. A SQUAF alone with 192 trainable parameters further reduces MSE to 5.00×10−55.00\times10^{-5}. The SQUAF-only case with merely 28 parameters also performs better than both ReLU-MLP and DiTAC-MLP which use far more parameters.

Fig. 11 in the revised manuscript further visualizes the fitted SQUAF, which closely matches the discontinuous target, except for some localized oscillations near the discontinuous points. Also, α\alpha is not pushed towards ∞\infty which can be told from the smooth shape of the fitted SQUAF (an abnormally large α\alpha would instead make SQUAF resemble a rigid step function as explained in Remark (5) of Section 3.1), and thus no instability is observed.

We again thank the reviewer for their time and effort. If our responses above address the concerns, we kindly request that the reviewer considers increasing their score.

I thank the authors for the response and the new experiments in the revised manuscript. My concerns regarding statistical significance and non-sinusoidal/discontinuous functions have been addressed. The layout improvements are appreciated and my questions were answered logically.

I have also considered the concerns raised by other reviewers to inform my holistic assessment of the paper. Appendix A.15 clarifies that training all SQUAF layers degrades accuracy due to overfitting rather than numerical instability, but it seems that the practical requirement to tune dataset and architecture-specific layer freezing schedules as seen in Tables 20–22 makes SQUAF less straightforward as a general-purpose drop-in replacement.

Overall, this is a solid paper with particularly strong results for PDEs and INRs. I have no further questions/requests and stand by my supportive score of 3.

We thank the reviewer for their careful assessment, constructive feedback, and continued support. We appreciate your recognition of our additional experiments and clarifications, as well as the strong results on PDEs and INRs. Your feedback has helped us improve the paper.

Reviewer_DYwKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Soft Quantization Activation Functions (SQUAFs), a family of parameterized activation functions based on learnable soft quantization. The key idea is to replace the hard interval assignments of conventional staircase functions with differentiable probabilistic assignments. SQUAFs achieve flexible nonlinear responses through learnable reference locations, output values, and smoothing parameters. In the main experiments, the authors adopt a Gaussian-based soft quantization formulation with uniformly spaced reference locations and integrate it as an element-wise activation function into MLPs, CNNs, and Transformers. Theoretically, the authors establish that SQUAFs can approximate any continuous univariate function on a closed interval to arbitrary precision with a sufficiently large number of reference points. They also derive analytical gradients with respect to the inputs and learnable parameters. The empirical evaluation covers function approximation, image classification on CIFAR-100 and ImageNet-1K, 10 PDE-solving problems from PINNacle, 16 two-dimensional image INR tasks, and 4 three-dimensional SDF reconstruction tasks. The authors report consistent performance improvements across these benchmarks. SQUAFs achieve the lowest relative error on 9 of the 10 PDE problems and the best reconstruction metrics across the evaluated 2D and 3D INR tasks. Additionally, experiments on knowledge distillation and reducing Transformer depth suggest that SQUAFs may facilitate model compression.

Critical strengths
  1. Simple, generalizable, and parameter-efficient learnable activation function. SQUAFs provide a mathematically concise formulation that enables activation functions to learn more expressive nonlinear responses from data while introducing relatively few additional parameters.

  2. Comprehensive evaluation across diverse architectures and tasks. Beyond standard image classification benchmarks, the paper evaluates SQUAFs on PINNs and INRs, which place greater demands on nonlinear function representation. The experiments cover multiple network architectures and task domains, with generally consistent performance improvements.

Critical weaknesses
  1. SQUAFs have bounded outputs over the entire real domain, which limits their ability to represent unbounded activation functions. This limitation follows directly from the mathematical formulation of SQUAFs. Since the normalized weights satisfy ∑iwi(x)=1,wi(x)≥0,\sum_i w_i(x)=1,\qquad w_i(x)\geq 0, the output is bounded by the minimum and maximum learnable amplitudes: min⁡izi≤ϕ(x)≤max⁡izi.\min_i z_i \leq \phi(x) \leq \max_i z_i. For a uniformly spaced grid with q,α>0q,\alpha>0, the activation also satisfies lim⁡x→+∞ϕ(x)=zk,\lim_{x\to+\infty}\phi(x)=z_k, lim⁡x→−∞ϕ(x)=z−k.\lim_{x\to-\infty}\phi(x)=z_{-k}. Therefore, SQUAFs eventually saturate at both ends of the input domain. This differs fundamentally from ReLU, which grows without bound on the positive half-axis. A finite SQUAF cannot exactly reproduce this unbounded linear behavior over the entire real domain. Consequently, the approximation of existing unbounded activation functions is primarily valid over a predefined bounded input range. Significant shifts in pre-activation distributions during training or out-of-range inputs at inference time may cause saturation and vanishing input gradients. Notably, Appendix A.5 reports that freezing a SQUAF pretrained to approximate ReLU can degrade performance due to changes in the input distribution during training. The authors should discuss this limitation more explicitly and evaluate the robustness of SQUAFs under distribution shifts in pre-activation values.

  2. AC-PKAN (AC-PKAN: Attention-Enhanced and Chebyshev Polynomial-Based Physics-Informed Kolmogorov-Arnold Networks, TMLR 2026) is relevant to the proposed method, particularly regarding PDE solving, learnable nonlinear representations, and extensions to Kolmogorov-Arnold Networks (KANs). The authors should clarify the methodological differences between AC-PKAN and SQUAFs, including their respective nonlinear parameterizations, computational characteristics, and applicability to physics-informed learning. A controlled comparison on shared PDE benchmarks would further strengthen the experimental evaluation.

  3. SQUAFs combine several components, including Gaussian weighting, softmax normalization, learnable output amplitudes ziz_i, learnable reference-point spacing qq, learnable smoothing parameters α\alpha, and initialization through pretraining to approximate existing activation functions. However, the paper lacks systematic ablations that isolate the contributions of these components. Appendix A.5 provides useful evidence that jointly optimizing (q,z,α)(q,z,\alpha) outperforms optimizing only zz on the 3D SDF reconstruction task. Nevertheless, this experiment does not establish whether the specific SQUAF formulation is necessary to achieve the reported improvements. For example, a conventional unnormalized Gaussian RBF activation could serve as a simpler alternative: ϕunnorm(x)=∑izie−α(x−yi)2.\phi_{\mathrm{unnorm}}(x)=\sum_i z_i e^{-\alpha(x-y_i)^2}. Similarly, spline-based activation functions could potentially provide comparable expressive capacity at a lower computational cost: ϕ(x)=SplineInterp⁡(x;z,y).\phi(x)=\operatorname{SplineInterp}(x;\mathbf z,\mathbf y). Another relevant baseline would use fixed Gaussian centers with learnable output amplitudes and a simple learnable input scaling factor. These comparisons would help determine whether the improvements arise from the proposed soft quantization formulation or from the general benefits of introducing additional learnable nonlinear parameters. The current experiments demonstrate the effectiveness of the overall SQUAF design but do not sufficiently establish the necessity or relative importance of its individual components.

  4. Appendix A.4.2 states that only the parameters of the first SQUAF are updated on CIFAR-100, while all remaining SQUAF parameters are frozen to prevent overfitting. This substantially restricts the learnable activation mechanism on one of the main classification benchmarks. Although this training strategy is not inherently problematic, it raises several important questions. First, why does learning SQUAF parameters in deeper layers lead to overfitting? Does this indicate that the increased expressive capacity also introduces optimization instability or a higher risk of overfitting? Second, if updating only the first SQUAF is sufficient to achieve most of the reported improvements, is replacing activation functions throughout the entire network necessary? Third, could simpler alternatives, such as a learnable input transformation, first-layer normalization, or learnable feature scaling, achieve comparable improvements? To address these questions, the authors should compare the following training configurations:

  • Trainable first-layer SQUAF only: Update only the first SQUAF while freezing the remaining SQUAF parameters.
  • Trainable all-layer SQUAFs: Jointly optimize SQUAF parameters across all layers.
  • Trainable last-layer SQUAF only: Update only the final SQUAF while freezing the others.
  • Layer-specific SQUAF replacement: Evaluate different strategies for replacing the original activation functions at selected network depths.

Importantly, the authors should include a baseline that uses SQUAF only in the first layer while retaining the original ReLU or GELU activations in all subsequent layers. This would help establish whether deploying SQUAFs throughout the network provides benefits beyond modifying the first-layer nonlinear transformation. Without these comparisons, the effectiveness of SQUAFs as general-purpose drop-in activation functions remains difficult to assess, particularly given their reliance on architecture-specific training strategies.

Questions for authors

see weaknesses above

11 comments
AuthorsOpenReview ↗

Response to Reviewer DYwK (part 1/5)

We thank the reviewer for taking time to review our paper and provide valuable feedbacks. Please find our point-by-point responses below. References to paper contents in our responses correspond to the revised manuscript unless otherwise specified.

Weakness 1: SQUAF has bounded outputs and saturates at both ends. Please discuss this limitation and evaluate robustness under shifts in pre-activation values.

Response: Yes, SQUAF has bounded outputs and Gaussian-based SQUAFs in equations (6) and (7) saturate on both ends. We now explicitly state these properties in Remarks (6)–(7) of Section 3.1. However, they may not hurt SQUAF's effectiveness in practice. z\mathbf{z} can be initialized to allow SQUAF to reach an arbitrarily large output magnitude, qq can be initialized such that the non-saturated area covers an arbitrarily large input range, and both can be subsequently learned during training to adapt to the data.

Indeed, in Appendix A.5, a frozen ReLU-initialized SQUAF degrades the 3D INR performance. However, that's a controlled setting specific to the ablation study; not the actual setting we used to produce the main 3D INR results, in which SQUAF is not frozen and can adapt to the proper input/output magnitudes. Moreover, even a frozen SQUAF doesn't always hurt model performance. As requested by Reviewer VhJb, we now provide an example on CIFAR-100, where a VGG-8 equipped with ReLU-initialized SQUAFs is trained while all SQUAFs are frozen. Such a trained model reaches 70.75% accuracy on CIFAR-100, higher than the ReLU baseline 70.36%. Thus, even without adapting the SQUAF shapes, they could slightly improve over ReLU in a different task. This is consistent with the findings in [1], which shows training DNNs with activation clipping could improve model generalization. The seemingly contradictory evidence on INR and classification tasks bears a plausible explanation: for CIFAR-100 image classification where overfitting is a problem, clipping/saturating large activations impose a regularization effect such that DNNs cannot exploit spurious features and instead learn more generalizable representations; but for the image and SDF fitting tasks, where fitting the training sample is the ultimate goal while overfitting is not relevant, clipping may hurt the performance as it constrains the range of activations, thus limiting the model from exploiting large activations for memorizing the signal.

The above discussion concerns training DNNs with bounded SQUAFs. As for robustness to test-time pre-activation shifts, we conducted the evaluation below. Given a trained VGG-8 model on CIFAR-100, we perturb its first layer pre-activations during inference. Here, we consider three types of perturbations: additive Gaussian noise, additive constant offset, and multiplicative scaling. Figure 10 in Appendix A.17 of the revised manuscript shows that VGG-8 with SQUAF is more robust to Gaussian noise and constant offset than VGG-8 with ReLU, while the latter is more robust to multiplicative scaling.

Furthermore, our SQUAF construction is also flexible enough to be extended to an unbounded variant. For example, replacing the right endpoint amplitude zkz_k with zk+ReLU⁡(x−kq)z_k+\operatorname{ReLU}(x-kq) gives

ϕ~(x)=ϕ(x)+Pα(kq∣x)ReLU⁡(x−kq).\widetilde\phi(x)=\phi(x)+P_\alpha(kq\mid x)\operatorname{ReLU}(x-kq).

This leaves the original SQUAF unchanged for x≤kqx\leq kq, while its right tail approaches zk+(x−kq)z_k+(x-kq) as x→+∞x\to+\infty. If an unbounded right tail is indeed crucial for a certain application, this variant could be a promising alternative to the original SQUAF in the manuscript. However, we leave its empirical evaluation to future work.

AuthorsOpenReview ↗

Response to Reviewer DYwK (part 2/5)

Weakness 2: Please clarify the relationship to AC-PKAN and provide a controlled PDE comparison.

Response: Thank you for pointing us to AC-PKAN [2]. AC-PKAN proposes a PINN framework combining Chebyshev Type-I-based KANs (Cheby1KANs), auxiliary MLPs, internal attention mechanism, and an adaptive loss–reweighting strategy. On the other hand, our work focuses on proposing a novel family of activation functions, SQUAFs, which can be inserted into all kinds of DNNs and applied to various tasks. Therefore, we believe these two works do not form a direct comparison. However, we find that the model used in AC-PKAN contains a flexible edge function in its KAN layer, which is based on the Chebyshev polynomials of the first kind. To make a fair comparison with SQUAF, we use it as a standard, layer-shared AF in MLP-based PINNs. The exact form of this Chebyshev AF is

ψ(x)=∑m=06cmTm(tanh⁡(x)),\psi(x)=\sum_{m=0}^{6}c_mT_m(\tanh(x)),

where TmT_m is the first-kind Chebyshev polynomial, degree six gives seven trainable parameters per layer, matching SQUAF with k=2k=2, and coefficients cmc_m are initialized following AC-PKAN's official implementation. We then evaluate ψ(x)\psi(x) in the same PINNacle setup as in Sec. 4.3, and compare with SQUAF. Table 25 in Appendix A.18 (also shown below) demonstrates that SQUAF performs better in general, obtaining lower L2RE on nine out of ten PDE problems.

PDEChebyshev AFSQUAF
Burgers 1d-C1.38E-2 ±\pm 1.81E-041.34E-2 ±\pm 7.39E-04
Burgers 2d-C4.47E-1 ±\pm 1.91E-034.42E-1 ±\pm 6.42E-03
Poisson 2d-C6.75E-1 ±\pm 4.62E-024.06E-1 ±\pm 4.95E-03
Poisson 3d-CG4.19E-1 ±\pm 8.21E-023.33E-1 ±\pm 1.92E-02
Heat 2d-VC4.72E-1 ±\pm 3.54E-024.64E-1 ±\pm 4.47E-02
Heat 2d-MS3.76E-2 ±\pm 3.21E-033.57E-2 ±\pm 2.76E-02
NS 2d-C2.11E-2 ±\pm 7.08E-042.51E-2 ±\pm 2.77E-02
NS 2d-CG8.17E-2 ±\pm 1.69E-026.69E-2 ±\pm 8.28E-02
PInv1.38E-1 ±\pm 7.40E-033.41E-2 ±\pm 1.54E-02
HInv1.83E-1 ±\pm 5.10E-031.78E-1 ±\pm 2.06E-01
AuthorsOpenReview ↗

Response to Reviewer DYwK (part 3/5)

Weakness 3: Please isolate the contributions of SQUAF's individual components, and compare against alternatives such as an unnormalized SQUAF, spline, and fixed-center Gaussian activations with learnable input scaling.

Response: We have added these comparisons on all four 3D shape representation tasks in Appendix A.14. All methods use the same PEMLP backbone, one AF parameter set per layer shared across neurons, and initialization to approximate ReLU over [−0.8,0.8][-0.8,0.8]. All Gaussian-kernel variants use the same initial grid A^={0,±q,±2q,…,±kq}\hat{\mathcal{A}}=\{0, \pm q, \pm 2q, \dots, \pm kq\} as SQUAF, with initial q=0.05q=0.05 and k=16k=16. Full SQUAF learns (z,q,α)(\mathbf{z},q,\alpha) with normalized weights as in Eq. (7). Unnormalized SQUAF removes the normalization denominator while retaining these trainable parameters. KAF corresponds to unnormalized SQUAF with fixed (q,α)(q, \alpha); it trains only z\mathbf{z}. Per your suggestion, we also consider an input-scaled version of KAF ϕscaled(x)=∑i=−1616ziexp⁡[−α(sx−iq)2]\phi_{\mathrm{scaled}}(x)=\sum_{i=-16}^{16}z_i\exp[-\alpha(sx-iq)^2], where qq and α\alpha are fixed while (z,s)(\mathbf{z},s) are trainable, and a B-spline AF with degree three, 32 intervals and 35 trainable parameters.

Tables 17–18 in Appendix A.14 (also shown below) show that full SQUAF achieves the best average Chamfer distance and IoU. Among other Gaussian-kernel variants, KAF performs the worst, the input-scaled KAF improves over KAF and unnormalized SQUAF further improves over input-scaled KAF. B-spline also performs decently, but is still outperformed by SQUAF.

Chamfer distance (lower is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)3.453E-62.082E-62.313E-63.758E-62.901E-6
KAF3.350E-62.024E-62.124E-63.604E-62.776E-6
Input-scaled KAF3.337E-62.003E-62.123E-63.597E-62.765E-6
Unnormalized SQUAF3.336E-61.987E-62.070E-63.520E-62.728E-6
PAU3.391E-62.158E-62.028E-63.154E-62.683E-6
B-spline3.306E-61.948E-62.178E-65.124E-63.139E-6
SQUAF3.246E-61.915E-61.982E-63.026E-62.542E-6
SQUAF (z\mathbf{z} only)3.890E-62.114E-63.220E-63.683E-63.227E-6

IoU results (higher is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)9.883E-19.815E-19.767E-19.599E-19.766E-1
KAF9.886E-19.799E-19.727E-19.611E-19.756E-1
Input-scaled KAF9.887E-19.802E-19.746E-19.614E-19.762E-1
Unnormalized SQUAF9.889E-19.814E-19.749E-19.627E-19.770E-1
PAU9.878E-19.774E-19.736E-19.639E-19.757E-1
B-spline9.893E-19.803E-19.733E-19.557E-19.747E-1
SQUAF9.902E-19.823E-19.786E-19.677E-19.797E-1
SQUAF (z\mathbf{z} only)9.872E-19.778E-19.585E-19.580E-19.704E-1

We also incorporate SQUAF's results with only z\mathbf{z} trainable from Table 9 of Appendix A.5 into the tables above, in order to present the following observations. Jointly learning the grid, smoothness and amplitudes is particularly important for the normalized SQUAF: its average Chamfer distance improves from 3.227×10−63.227\times10^{-6} with only z\mathbf{z} trained to 2.542×10−62.542\times10^{-6} with all parameters trained. For the unnormalized family, the corresponding improvement is smaller, from KAF's 2.776×10−62.776\times10^{-6} to 2.728×10−62.728\times10^{-6} of the unnormalized SQUAF. Overall, the combination of normalization and full parameter adaptation, i.e., our original SQUAF, is shown to be the best design. Moreover, we also compare with another relevant AF called PAU [3] following Reviewer VhJb's suggestion, which is also outperformed by SQUAF.

Reviewer_8ESCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SQUAFs, learnable activation functions based on soft quantization. It establishes their universal approximation capability and reports improvements across classification, PDE solving, implicit representations, and model compression. The reported results demonstrate consistent improvements across several architectures and tasks with few additional trainable parameters.

Critical strengths
  1. The evaluation spans multiple architectures and tasks, with consistently positive results and comparisons against specialized INR methods. This breadth supports the method’s practical relevance.

  2. The construction is smooth and flexible with few additional parameters. Random-initialization results for PDEs and controlled 3D ablations provide evidence that learning the activation shape is beneficial.

Critical weaknesses
  1. The advantage over closely related kernel activations remains insufficiently established. KAF already provides flexible Gaussian-kernel expansions, but its layer-shared variant is evaluated only on small function-fitting tasks. The downstream benefits attributable to normalization and learnable grid/smoothness parameters are therefore unclear.
  2. Important performance gains remain difficult to attribute. CIFAR-100 trains only the first SQUAF, while 2D representations use FINER-based initialization. The existing 3D ablations do not directly establish whether improvements in these settings arise from shape adaptation, initialization, or selective freezing.
Questions for authors
  1. Can a representative comparison with KAF under matched sharing and initialization settings clarify the benefits of normalization and grid/smoothness adaptation?
  2. Could you compare freezing strategies on one classification backbone, and frozen versus trainable FINER-initialized SQUAFs on one representative image?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8ESC (part 1/2)

We thank the reviewer for taking time to review our paper and provide valuable feedbacks. Please find our point-by-point responses below. References to paper contents in our responses correspond to the revised manuscript unless otherwise specified.

Weakness 1 / Question 1: Can a representative comparison with KAF under matched sharing and initialization clarify the benefits of normalization and grid/smoothness adaptation?

Response: We have added this comparison on all four 3D shape representation tasks in Appendix A.14. All methods use the same PEMLP backbone, one AF parameter set per layer shared across neurons, and initialization to approximate ReLU over [−0.8,0.8][-0.8,0.8]. All Gaussian-kernel variants use the same initial grid A^={0,±q,±2q,…,±kq}\hat{\mathcal{A}}=\{0, \pm q, \pm 2q, \dots, \pm kq\} as SQUAF, with initial q=0.05q=0.05 and k=16k=16. Full SQUAF learns (z,q,α)(\mathbf{z},q,\alpha) with normalized weights as in Eq. (7). Unnormalized SQUAF removes the normalization denominator while retaining these trainable parameters. KAF corresponds to unnormalized SQUAF with fixed (q,α)(q, \alpha); it trains only z\mathbf{z}. We also consider an input-scaled version of KAF ϕscaled(x)=∑i=−1616ziexp⁡[−α(sx−iq)2]\phi_{\mathrm{scaled}}(x)=\sum_{i=-16}^{16}z_i\exp[-\alpha(sx-iq)^2] as suggested by Reviewer DYwK, where qq and α\alpha are fixed while (z,s)(\mathbf{z},s) are trainable.

Tables 17–18 in Appendix A.14 (also shown below) show that full SQUAF achieves the best average Chamfer distance and IoU.

Chamfer distance (lower is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)3.453E-62.082E-62.313E-63.758E-62.901E-6
KAF3.350E-62.024E-62.124E-63.604E-62.776E-6
Input-scaled KAF3.337E-62.003E-62.123E-63.597E-62.765E-6
Unnormalized SQUAF3.336E-61.987E-62.070E-63.520E-62.728E-6
PAU3.391E-62.158E-62.028E-63.154E-62.683E-6
B-spline3.306E-61.948E-62.178E-65.124E-63.139E-6
SQUAF3.246E-61.915E-61.982E-63.026E-62.542E-6
SQUAF (z\mathbf{z} only)3.890E-62.114E-63.220E-63.683E-63.227E-6

IoU (higher is better):

AFArmadilloAsian DragonLucyThai StatueAvg.
PEMLP (ReLU)9.883E-19.815E-19.767E-19.599E-19.766E-1
KAF9.886E-19.799E-19.727E-19.611E-19.756E-1
Input-scaled KAF9.887E-19.802E-19.746E-19.614E-19.762E-1
Unnormalized SQUAF9.889E-19.814E-19.749E-19.627E-19.770E-1
PAU9.878E-19.774E-19.736E-19.639E-19.757E-1
B-spline9.893E-19.803E-19.733E-19.557E-19.747E-1
SQUAF9.902E-19.823E-19.786E-19.677E-19.797E-1
SQUAF (z\mathbf{z} only)9.872E-19.778E-19.585E-19.580E-19.704E-1

We also incorporate SQUAF's results with only z\mathbf{z} trainable from Table 9 of Appendix A.5 into the tables above, in order to present the following observations. Jointly learning the grid, smoothness and amplitudes is particularly important for the normalized SQUAF: its average Chamfer distance improves from 3.227×10−63.227\times10^{-6} with only z\mathbf{z} trained to 2.542×10−62.542\times10^{-6} with all parameters trained. For the unnormalized family, the corresponding improvement is smaller, from KAF's 2.776×10−62.776\times10^{-6} to 2.728×10−62.728\times10^{-6} of the unnormalized SQUAF. Overall, the combination of normalization and full parameter adaptation, i.e., our original SQUAF, is shown to be the best design. Moreover, we also compare with two other related AFs called PAU [1] and B-spline in the tables above, following Reviewer DYwK and VhJb's suggestions. SQUAF also outperforms these AFs in average Chamfer distance and IoU.

We additionally compare layer-shared KAF and PAU with SQUAF using ResNet-18 on ImageNet-1K. All three are initialized to approximate ReLU on [−16,16][-16,16]. KAF uses the same initial grid as SQUAF with bandwidth 1/81/8 following its own bandwidth rule, and PAU uses degrees (5,4)(5,4). Table 19 (also shown below) reports accuracy (%, mean ±\pm standard deviation over three runs) and shows that SQUAF improves over both baselines.

AFAccuracy
KAF69.87±0.2069.87\pm0.20
PAU70.30±0.1770.30\pm0.17
SQUAF70.86 ±\pm 0.21
AuthorsOpenReview ↗

Response to Reviewer 8ESC (part 2/2)

Weakness 2 / Question 2: Could you compare freezing strategies on one classification backbone, and frozen versus trainable FINER-initialized SQUAFs on one representative image?

Response: We have added both comparisons in Appendix A.15. We examine VGG-8 on CIFAR-100 (whose ReLU baseline accuracy is 70.36%) with SQUAFs at all five activation layers, varying which SQUAFs are trainable while always training the ordinary network weights. Table 20 (also shown below) shows that training only the first SQUAF achieves 72.57%, compared with 70.75% when all SQUAFs are frozen, 68.14% when all are trained, and 68.19% when only the last is trained. Training only the first SQUAF remains the best freezing strategy. The gain over the all-frozen setting shows that activation learning contributes beyond initialization.

Trainable SQUAF layersNone11–21–31–41–5 (all)5 (last)
Accuracy70.7572.5772.1471.2170.3168.1468.19

Note that for this model on CIFAR-100, as we gradually unfreeze deeper SQUAFs, the performance decreases (dropping below the baseline ReLU with the first 4 SQUAFs unfrozen). This is not because SQUAF hurts the classification model when incorporated into deeper layers; it's simply a consequence of overfitting. Incorporating SQUAFs into a model is a way to increase model capacity/expressiveness. It is natural that a model with increased capacity tends to overfit more on the same dataset if we do not change the training recipe. Figure 8 in the revised manuscript shows training curves for the corresponding configurations. All-trained SQUAF reaches the best (lowest) training loss as expected given its highest model capacity, but it has a larger train-test loss gap and lower test accuracy compared to the first-only SQUAF and ReLU. This clearly signals an overfitting phenomenon when all SQUAFs are trained.

This clear overfitting behavior calls for regularization. We could strengthen data augmentation or add regularization terms to the loss, but we choose to retain the original training recipe to make the AF comparison fair and straightforward. Instead, we turn to regularize SQUAF itself. In fact, freezing deeper SQUAFs is an effective regularization method we developed for SQUAF-DNNs to prevent overfitting. During development, we also tried other forms of regularization, such as clipping the amplitudes of z\mathbf{z}, using smaller kk in deeper SQUAFs, etc., some of which work as well, but we find freezing deeper SQUAFs is the most straightforward and effective method, and that's why we apply it in our image classification experiments as needed.

It's worth noting that the preferred trainable-SQUAF depth changes with the task. For ResNet-18 on ImageNet-1K, training only the stem (first) SQUAF gives 69.79%, close to the ReLU baseline of 69.76%, whereas training all SQUAFs except for those in the last stage gives 70.88% (Tab. 22, also shown below). Thus, the first-layer policy used on CIFAR-100 is not a general restriction of SQUAF. Since the dataset gets larger and overfitting becomes less of a problem, we don't need to regularize SQUAFs as much as we do on the smaller CIFAR-100 dataset. Moreover, we don't freeze any SQUAFs in our image and SDF fitting tasks, because fitting the training sample is the ultimate goal there—train-test gap or overfitting is not relevant—and therefore regularization is not needed.

Trainable SQUAFsStemStem + Stage 1Stem + Stage 1–2Stem + Stage 1–3Stem + Stage 1–4
Accuracy69.7970.0370.5370.8869.87

As for the 2D INR comparison the reviewer requested, Table 23 (also shown below) reports PSNR (dB) on image-00, averaged over 5 runs.

AFPSNR
FINER35.90
SQUAF (frozen)35.82
SQUAF (trainable)37.03

It's shown that freezing the FINER-initialized SQUAF yields 35.82 dB, slightly lower than FINER's 35.90 dB, while training the SQUAF raises PSNR to 37.03 dB. We also visualize the training curves of FINER and the frozen FINER-initialized SQUAF in Fig. 9 for your reference. Note that these results are averaged over 5 runs, thus distinct from the single-run results in Fig. 3. Therefore, the improved PSNR comes from adapting SQUAF after FINER-based initialization, rather than simply replacing FINER with a frozen SQUAF-based approximation.

[1] Molina et al., “Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks,” ICLR, 2020.

We again thank the reviewer for their time and effort. If our responses above address the concerns, we kindly request that the reviewer considers increasing their score.

We thank the reviewer again for their time and constructive feedback. As the discussion deadline approaches, we would appreciate hearing whether our responses and additional experiments address your concerns. Please let us know if any points need further clarification. Thank you for your consideration.

Dear Reviewer 8ESC,

With the discussion period closing in a few hours, we wanted to briefly follow up on our rebuttal and revised manuscript. We would be happy to clarify anything or address any remaining concerns. If you have had a chance to review our responses and additional results, we would appreciate hearing whether they address your concerns and whether they warrant any update to your assessment.

Thank you again for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.