acceptodds
Under review as a conference paper at ICLR 2027

COQUAD: CO-OPTIMIZED QUADRATIC ACTIVATIONS FOR EFFICIENT PRIVATE TRANSFORMER INFERENCE

Abstract

Ensuring private inference for Transformer architectures remains challenging, with nonlinear activations serving as a key bottleneck. Quadratic replacement offers inexpensive secure evaluation, but recovering model accuracy requires adaptation. We show that the role of quadratic coefficients depends on the FFN architecture: they can be absorbed into adjacent affine maps in biased standard FFNs, whereas nondegenerate bias-free gated channels retain a coefficient ratio invariant under projection reparameterization. MPCFormer's fixed polynomial prescribes this ratio; CoQuad allows it to adapt by jointly learning layer-wise quadratic coefficients and network weights. DirectFit supplies data-dependent initialization, and Co-Optimize performs teacher-guided joint adaptation while retaining quadratic secure evaluation. On ImageNet-1K, full replacement in ViT-B/16 achieves 79.83% Top-1 accuracy (1.62 percentage points below the 81.45% teacher), exceeding a ViT adaptation of the full MPCFormer method by 1.42 percentage points. Llama experiments extend the evaluation to bias-free SwiGLU: with the same 16 layers replaced and the same 1,500 training steps, CoQuad reaches 72.6% ARC-Easy accuracy, compared with 36.5% for MPCFormer's fixed polynomial. Compared with SHAFT, quadratic GeLU evaluation is estimated to be 47–59× faster and reduces communication by 62×; with the other operators unchanged, measured end-to-end speedups over SHAFT range from 1.227× to 1.530×. These results demonstrate the practical value of joint coefficient and network adaptation across standard and gated Transformer FFNs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy-Preserving Machine Learning, Activation Function Approximation, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k5YoOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoQuad replaces Transformer FFN activations with layer-wise quadratic functions, using data-dependent initialization and joint coefficient–backbone distillation. The paper analyzes coefficient absorption in standard and gated FFNs and evaluates accuracy preservation and private-inference efficiency across ViT, BERT, and Llama.

Critical strengths

The architectural analysis distinguishes coefficient reparameterization in biased FFNs from the channel-wise invariant in bias-free gated FFNs, providing a clear motivation for coefficient learning.

The ViT results are promising: full GeLU replacement achieves 79.83% ImageNet-1K accuracy versus 81.45% for the teacher. Training coefficients improves accuracy by 3.53 percentage points over the frozen-coefficient control.

Evaluation includes multiple model families and two secure-inference backends. The paper distinguishes estimated operator costs from measured end-to-end latency and provides useful implementation details.

Critical weaknesses

Incomplete component attribution. The Llama comparison changes both initialization and coefficient trainability, leaving their individual contributions unclear. The proposed coefficient-ratio explanation would benefit from direct diagnostics or constrained controls. In ViT, optimization and induced regularization effects also remain insufficiently separated.

Efficiency comparisons need strengthening. The main comparison uses SHAFT’s Poly implementation; comparison against its Fourier-based GeLU configuration would better establish the practical advantage. The reported 2.56× speedup additionally changes Softmax and needs a corresponding accuracy measurement.

Limited LLM validation. Evaluation focuses on ARC-Easy with half the SiLU layers replaced. Llama secure-inference results rely on composed activation-cost estimates, leaving full-model latency and broader language-model quality insufficiently established.

Questions for authors

Can the authors provide a matched initialization-by-trainability ablation on Llama and examine the learned coefficient ratios?

How does CoQuad compare with SHAFT’s Fourier configuration, and what accuracy accompanies the combined GeLU–Softmax speedup?

Can additional language-model benchmarks or perplexity measurements support broader quality preservation?

2 comments

Q1

Thank you for suggesting these controls. In a new experiment extending the paper’s 16-layer setting, we successfully replaced SiLU in all 32 SwiGLU layers of Llama-2-7B. After 1,500 steps, CoQuad reached 73.57% on ARC-Easy and 61.54% averaged across seven zero-shot tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA). The baseline model scored 75.6% on ARC-Easy and 62.73% on average across the seven tasks, leaving gaps of just 2 and 1.2 percentage points, respectively, after full replacement.

Two controls in this full-replacement experiment directly address the fixed-coefficient question. We kept the SwiGLU gate intact and trained the backbone under the same distillation recipe. Freezing each layer’s DirectFit coefficients gave 27.69% on ARC-Easy. Using MPCFormer’s fixed polynomial gave 27.74%. From identical DirectFit coefficients, allowing them to train raised ARC-Easy by 45.88 percentage points. Proposition 1 identifies the relevant expressive freedom: in a nondegenerate bias-free gated channel, rescaling the projections preserves the coefficient ratio ac/b2ac/b^2, whereas training the coefficients allows that ratio to change. CoQuad used this freedom in practice: across the 32 layers, the learned ratios ranged from 0.00060.0006 to 0.0360.036, entirely above the DirectFit initialization range of −0.0025-0.0025 to 0.00050.0005. The 32-layer results show the practical importance of this freedom for recovering accuracy under the tested recipe.

For the converse control, we initialized the quadratics in 16 alternating layers with MPCFormer’s coefficients and trained them jointly with the backbone. After 1,500 steps, the final scores were nearly identical to those obtained with DirectFit. The gap was less than 0.5 percentage points for both ARC-Easy accuracy and the seven-task average. DirectFit’s advantage was visible early in training. With DirectFit, the median constant term cc started near zero and ended at 0.013. With MPCFormer initialization, it started at 0.50 and was still 0.224 at step 300. Over steps 260–300, the median distillation KL was 0.155 with DirectFit versus 0.892 with MPCFormer initialization, indicating that DirectFit helped the student match the teacher’s outputs sooner. DirectFit also yielded lower final perplexity on both corpora, with the larger reduction on C4.

Q2

SHAFT’s repository provides two GELU implementations, polynomial and Fourier. Since Fourier evaluation requires multiple trigonometric terms and secure interactions, it is inherently much more expensive than our univariate quadratic approximation; on input shapes (128, 4096) and (256, 4096), SHAFT-Fourier is 2.98× and 2.89× slower than SHAFT-Poly, respectively. Since our Table 1 focuses on runtime, we use SHAFT-Poly as the faster and more conservative runtime baseline. Therefore, comparing against SHAFT-Fourier would only further increase CoQuad’s reported speedup.

CoQuad focuses on FFN activations. For the combined GELU–Softmax experiment, we paired CoQuad’s quadratic GELU with MPCFormer’s 2Quad Softmax, used unchanged, to measure the end-to-end benefit of the two replacements. With 2Quad Softmax held fixed, Proposition 1(i) shows that the trained ViT-B/16 can be reparameterized to use MPCFormer’s fixed quadratic in its FFNs by adjusting the adjacent affine layers, preserving the model’s predictions. The combined configuration delivers a 2.56× end-to-end speedup over the baseline with standard secure GELU and the original Softmax and achieves 79.13% Top-1 accuracy on ImageNet-1K, 0.70 percentage points below CoQuad Full-12 with the original Softmax (79.83%).

Q3

Thank you for the suggestion. We evaluated seven downstream tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA) and report their unweighted mean accuracy. CoQuad achieved 62.34%, compared with 62.73% for the original model. Using MPCFormer’s initial coefficients, mean accuracy was 38.57% with frozen coefficients and 62.08% with trainable coefficients.

We also measured perplexity on WikiText-2 and C4 at context lengths of 1,024 and 2,048 tokens. Across all four settings, CoQuad’s perplexity was 0.20–0.30 above the original model’s and lower than both MPCFormer controls. At 2,048 tokens, for example, CoQuad scored 6.37 on WikiText-2 and 8.36 on C4, compared with 6.10 and 8.16 for the original model. These results extend the evidence for quality preservation beyond ARC-Easy to next-token prediction.

Thank you to the authors for the rebuttal. It addresses some of my concerns and questions to a certain extent. In light of my earlier comments, I have decided to keep my score at 3.

Reviewer_uwvJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the replacement of GeLU or the SiLU component of SwiGLU with layer-wise quadratic activations for private Transformer inference. It contributes: (1) a reparameterization analysis showing that nondegenerate quadratic coefficients are absorbable in a standard FFN with biases, while a single nondegenerate bias-free gated channel retains the ratio ac/b^2; (2) DirectFit, a progressive data-dependent weighted quadratic initialization with output-amplitude alignment; and (3) Co-Optimize, which jointly updates the backbone and quadratic coefficients under teacher guidance. The authors evaluate ViT-B/16 on ImageNet-1K, BERT on RTE, Llama-2/3 on ARC-Easy, and secure-inference costs in SHAFT- and BumbleBee-related settings. The strongest reported accuracy results are 79.83% for full ViT GeLU replacement and 72.6% for replacing SiLU in 16 of 32 Llama-2 layers. The main efficiency claim is a 47-59x estimated GeLU speedup over SHAFT's polynomial implementation, with 1.227-1.530x measured end-to-end speedup when Softmax is unchanged.

Critical strengths
  1. Proposition 1 correctly distinguishes coefficient absorption in biased standard FFNs from a channel-wise invariant in nondegenerate bias-free gated channels. This is the paper's strongest conceptual contribution and gives a principled lens for interpreting explicit activation coefficients.

  2. The reported ViT and Llama accuracies suggest that degree-two activations can be made useful across both standard and gated Transformer FFNs while retaining a low-cost secure-evaluation form. If the Llama result and latency accounting are validated with matched ablations and raw artifacts, the practical contribution would be significant.

Critical weaknesses
  1. CoQuad and the MPCFormer control differ in both initialization and coefficient trainability. The single-channel invariant also does not establish a full multi-channel expressivity gap. Consequently, the 72.6% versus 36.5% result supports the full CoQuad recipe, but not the paper's stronger causal claim that learning ac/b^2 explains the gain. The released single-seed setup and the baseline mismatch further weaken this evidence.

  2. The secure-efficiency evidence is not auditable from the paper. Table 1 is inconsistent with the stated latency model under the natural fixed-protocol interpretation; raw communication, computation, rounds, precision, and security parameters are missing; and no timing variance is reported. The external BumbleBee comparison also depends on an unstated source version. These issues directly affect the paper's headline 47-59x, 62x, and end-to-end claims.

Questions for authors
  1. What exact ARC-Easy protocol produced the 75.6% baseline: model checkpoint, number of shots, prompt format, lm-eval-harness or other evaluator and version, acc versus acc_norm, sequence truncation, and checkpoint-selection rule? Why does llama2_mpcformer_fixed.yaml record a 72.77 baseline while preserving the reported 36.45 control result?

  2. Does cross-channel mixing in a full SwiGLU layer eliminate or weaken the single-channel ratio restriction at practical width? If a full characterization is unavailable, can the claims be narrowed and supported empirically by freezing projections, permuting channels, or measuring whether learned ratios correlate with recovery?

  3. Can a fixed-coefficient, DirectFit-initialized model be defined for the gated case, and if so what is its ARC-Easy result under the exact CoQuad recipe? Conversely, what happens when the MPCFormer polynomial is made trainable without DirectFit?

2 comments

Q1

The 72.77% in llama2_mpcformer_fixed.yaml is the score of the original Llama-2-7b-chat-hf checkpoint. Our method receives 1,500 additional steps of cross-entropy training. Comparing it directly with the 72.77% baseline would therefore conflate the effect of activation replacement with the gain from continued training. To isolate the effect of replacement, we applied the same training recipe to the original checkpoint with SiLU retained in every layer. This raised the baseline from 72.77% to 75.6%, the reference used in Table 3.

We used the final checkpoint at step 1,500 and report zero-shot acc on the full ARC-Easy test set using lm-eval-harness v0.4.12 (HFLM), with the default Question: {question}\nAnswer: prompt, no chat template, and no truncation. We will release the full configurations in our public code repository.

Q2

We thank the reviewer for this question. We now have a layer-level characterization, which we will add to the appendix.

(1) Exact layer-level identity. For any quadratic p(u)=au2+bu+cp(u) = au^2 + bu + c with a,b≠0a, b \neq 0, let χ(p)=ac/b2\chi(p) = ac/b^2. For any other such quadratic qq, there exist scalars ρ,s\rho, s such that p(u)=ρ q(su)+κp(u) = \rho\, q(su) + \kappa, where κ=b2a(χ(p)−χ(q))\kappa = \frac{b^2}{a}\big(\chi(p) - \chi(q)\big). Hence, for a full SwiGLU layer of any width,

Gp(x;Wd,Wg,Wu)=Gq(x;ρWd, sWg, Wu)+κ WdWux.G_p(x; W_d, W_g, W_u) = G_q(x; \rho W_d,\, sW_g,\, W_u) + \kappa\, W_d W_u x .

At the layer level, the ratio is therefore exactly the gain of an ungated linear path. Fixed coefficients tie this gain to the higher-order terms; learnable coefficients release it.

(2) Mixing weakens, but does not eliminate, the restriction. Absorbing the term κ WdWux\kappa\, W_d W_u x with fixed coefficients requires additional channels. At equal width, under mild non-degeneracy conditions on the weights, a fixed-qq layer represents GpG_p if and only if χ(p)=χ(q)\chi(p) = \chi(q). Llama's FFN width lies within this regime. Proposition 1(ii) thus extends from single channels to full layers as a fixed-width statement, and we will phrase our claim accordingly.

Q3

Thank you for suggesting these controls. In a new experiment extending the paper's 16-layer setting, we successfully replaced SiLU in all 32 SwiGLU layers of Llama-2-7B-Chat. After 1,500 steps, CoQuad reached 73.57% on ARC-Easy and 61.54% averaged across seven zero-shot tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA). The baseline model scored 75.6% on ARC-Easy and 62.73% on average across the seven tasks, leaving gaps of just 2.0 and 1.2 percentage points, respectively, after full replacement.

Two controls in this full-replacement experiment directly address the fixed-coefficient question. We kept the SwiGLU gate intact and trained the backbone under the same distillation recipe. Freezing each layer's DirectFit coefficients gave 27.69% on ARC-Easy. Using MPCFormer's fixed polynomial gave 27.74%. Starting from identical DirectFit coefficients, allowing them to train raised ARC-Easy by 45.88 percentage points. Proposition 1 identifies the relevant expressive freedom: in a nondegenerate bias-free gated channel, rescaling the projections preserves the coefficient ratio χ=ac/b2\chi = ac/b^2, whereas training the coefficients allows this ratio to change. The 32-layer results show the practical importance of this freedom for recovering accuracy under the tested recipe.

For the converse control, we initialized the quadratics in 16 alternating layers with MPCFormer's coefficients and trained them jointly with the backbone. After 1,500 steps, the final scores were nearly identical to those obtained with DirectFit: the gap was less than 0.5 percentage points for both ARC-Easy accuracy and the seven-task average. DirectFit's advantage was visible early in training. With DirectFit, the median constant term cc started near zero and ended at 0.013. With MPCFormer initialization, it started at 0.50 and was still 0.224 at step 300. Over steps 260–300, the median distillation KL was 0.155 with DirectFit versus 0.892 with MPCFormer initialization, indicating that DirectFit helped the student match the teacher's outputs sooner. DirectFit also yielded lower final perplexity on both corpora, with the larger reduction on C4.

Thank you for the detailed response; it has addressed most of my questions. Given the contributions of this paper, I will maintain my "weak accept" rating.

Reviewer_x28ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoQuad, which replaces Transformer GeLU/SiLU activations with layer-wise quadratics using data-dependent initialization followed by joint coefficient–backbone distillation. It analyzes how coefficient flexibility depends on the FFN architecture. Experiments demonstrate accuracy recovery with full GeLU replacement in ViT and partial SiLU replacement in Llama, alongside reduced inference costs in MPC and HE-based backends.

Critical strengths
  • Full GeLU replacement retains 79.83% ImageNet accuracy versus the teacher’s 81.45%, with measured 1.23–1.53x end-to-end speedups over SHAFT, demonstrating a useful accuracy–cost trade-off.

  • The matched ViT ablation shows a 3.53 % improvement from coefficient learning, supporting its practical value even when coefficients are algebraically absorbable.

Critical weaknesses
  • Learning polynomial activation coefficients has already been explored by Baruch et al. [R1] and SMART-PAF [R2].

[R1] Moran Baruch et al. “A methodology for training homomorphic encryption friendly neural networks.” International Conference on Applied Cryptography and Network Security. Cham: Springer International Publishing, 2022.

[R2] Jianming Tong et al. “Accurate low-degree polynomial approximation of non-polynomial operators for fast private inference in homomorphic encryption." Proceedings of Machine Learning and Systems 6, 2024.

Questions for authors

What additional benefit does CoQuad provide over existing coefficient-learning approaches, such as Baruch et al. [R1] and SMART-PAF [R2], when polynomial degree and adaptation budget are matched? Could you provide experimental comparisons under these matched conditions?

3 comments

We thank the reviewer for the two references and will add both to the revision. We chose degree 2 from the outset to minimize communication rounds per activation in MPC. Under standard arithmetic secret sharing with binary multiplications, a quadratic requires only one communication round, the minimum for a nonlinear polynomial, whereas higher-degree polynomials exceed this one-round budget and are therefore outside the scope of this study. We regard R1 and R2 as related work to cite, not as missing baselines: R1’s quadratic is a special case of ours whose learnable coefficients are redundant in gated FFNs, and R2 has no degree-2 or Transformer instance. The revision will also state the c=0c=0 result as a corollary next to Proposition 1 and expand the discussion of R2.

R1 and R2 are CNN-only studies: R1 replaces ReLU in AlexNet and SqueezeNet, and R2 replaces ReLU and MaxPooling in ResNet-18 and VGG-19. Neither contains a Transformer, a GeLU or SiLU replacement, a language model, or any analysis of when coefficient learning matters. CoQuad provides all four: full GeLU replacement in ViT-B/16 on ImageNet-1K, SiLU replacement in Llama-2-7B and Llama-3-8B, and Proposition 1. What the three works share is only the general idea of learning or fitting coefficients.

R1’s quadratic, p(u)=au2+bup(u)=au^2+bu, is the c=0c=0 special case of ours, trained for ReLU on two 3-class COVID imaging datasets with 209 and 300 test images, plus CIFAR-10. Without the constant term, its learnable coefficients are redundant in the bias-free gated FFNs of current LLMs: for a,b≠0a,b\ne 0, p(u)=(b2/a) q((a/b)u)p(u)=(b^2/a)\,q((a/b)u) with q(u)=u2+uq(u)=u^2+u, so within a gated channel they are absorbed into WgW_g and WdW_d and what remains is one fixed quadratic. Learning them changes nothing about the functions the channel can represent. This is what CoQuad adds over R1’s quadratic family at the same degree and multiplicative depth: a channel-wise shape parameter, ac/b2ac/b^2, that R1 fixes at zero and CoQuad adapts, at no extra secure-evaluation cost. Table 3 separately shows a 36.1-point gain over MPCFormer’s fixed quadratic under matched layers and steps. R1 contains neither a gated-FFN result nor an analysis of this kind.

R2 does not operate at degree 2 at all. It builds ReLU and Max from composite approximations of sign(x) at multiplication depth 5–8 (Table 2 of R2), five to eight times the depth of a quadratic, and offers no construction for GeLU or SiLU. It trains layer by layer and alternates between coefficients and weights, because joint training converges slowly or diverges in its setting (R2 §1.2); its ResNet-18 run extends beyond 400 epochs (Fig. 9 of R2). R2 also states that the accuracy of quadratic activations on TinyImageNet does not guarantee low degradation on ImageNet (R2 §7). CoQuad achieves full GeLU replacement on ImageNet-1K at degree 2 with 79.83% top-1 accuracy, 1.62 points below the teacher, by training coefficients and weights jointly in one 10-epoch run.

Thank you for the clarifications. My main concern remains whether CoQuad establishes a sufficiently distinct technical contributions over existing approaches to learning polynomial activation functions. Neither replacing GeLU/SiLU instead of ReLU nor extending existing techniques from CNNs to Transformers constitutes a substantial technical contribution in itself.

The relevant prior work extends beyond [R1] and [R2]. As previously noted, [R1] already explored layer-wise trainable quadratics with teacher-guided adaptation. Furthermore, PASNet [R3] employs learnable quadratic activations, including a constant term (Eq. 4), while Alias-Free Convnets [R4] initializes trainable quadratics by fitting GeLU (Eq. 4). Thus, learning quadratic coefficients, and using GeLU-based initialization have already been explored in prior work.

While the experimental evaluation covers several Transformer models, it is limited to replacing their activation functions with trainable quadratics and does not consider other components.

In my opinion, the proposed approach largely builds upon existing techniques for polynomial activation approximation. Hence, I remain unconvinced that the manuscript provides sufficiently technical advances over existing work.

[R1] Moran Baruch et al., “A Methodology for Training Homomorphic Encryption Friendly Neural Networks,” International Conference on Applied Cryptography and Network Security (ACNS), 2022.

[R2] Jianming Tong et al., “Accurate Low-Degree Polynomial Approximation of Non-Polynomial Operators for Fast Private Inference in Homomorphic Encryption,” Proceedings of Machine Learning and Systems (MLSys), vol. 6, 2024.

[R3] Hongwu Peng et al. “PASNet: Polynomial Architecture Search Framework for Two-party Computation-based Secure Neural Network Deployment,” DAC, 2023.

[R4] Hagay Michaeli, Tomer Michaeli, and Daniel Soudry. “Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations,” CVPR, 2023.

We thank the reviewer for pointing us to these related works. We will include references to R1–R4 in the revision and clarify their connections to and differences from CoQuad. We would like to further clarify that a central technical contribution of CoQuad is an architecture-dependent characterization of quadratic-coefficient absorption. Assessing this contribution requires examining the role of the coefficients within the surrounding network architecture, beyond comparing polynomial forms and initialization choices.

For quadratic activations with nonzero leading coefficients, Proposition 1 shows that, in standard FFNs with biases in both adjacent affine maps, coefficient changes can be absorbed into those maps when both are adjustable. Learning the coefficients therefore changes the optimization parameterization without enlarging the function class; strong ViT performance with fixed quadratics is fully consistent with this analysis. Within a single bias-free gated channel, as in SwiGLU, the conclusion differs: when all coefficients of the two quadratics and both projection vectors of the target channel are nonzero, exact equivalence for all inputs through projection reparameterization holds if and only if their ratios ac/b2ac/b^2 coincide. When these ratios differ, the discrepancy cannot be removed by reparameterizing that channel’s projections alone. This within-channel coefficient-absorption criterion was the main technical point highlighted in our previous response. The current comment does not yet discuss this result specifically, and we would appreciate a direct assessment of its correctness, technical significance, and relationship to prior results.

The constant term in R3 and GeLU-fitting initialization in R4 are relevant to our parameterization and initialization, but these precedents do not, by themselves, answer this architecture-dependent question. A methodological comparison also needs to address the concrete procedure. DirectFit proceeds in network order, using actual activation inputs from the student with upstream replacements already installed, with asymmetric weighted fitting and output-amplitude alignment. Co-Optimize subsequently updates the backbone and the coefficients shared within each replaced layer jointly under teacher supervision while keeping the activation polynomial degree at two. In Llama, we replace only the SiLU branch, preserving the other input-dependent branch and the multiplicative gate; the experiments evaluate this adaptation procedure in bias-free SwiGLU FFNs. Assessing CoQuad’s technical advance therefore requires examining this procedure and its connection to the architectural analysis, beyond whether the methods share trainable polynomials or GeLU fitting.

R2’s own discussion also highlights the importance of the training procedure and conditions for scaling. In its setting, R2 §1.2 reports slow convergence at small learning rates and divergence at large learning rates when polynomial coefficients and linear layers are trained jointly, motivating alternating training with separate hyperparameters. Discussing AESPA, R2 §7 also states that quadratic activations’ accuracy preservation on TinyImageNet does not guarantee low accuracy degradation on ImageNet. We cite these observations to emphasize that adaptation performance needs to be assessed in the context of the specific architecture, polynomial degree, and training procedure. R2’s results do not establish performance in CoQuad’s setting; assessing our technical advance therefore still requires examining our specific analysis, adaptation procedure, and experimental results. While keeping the activation polynomial degree at two, CoQuad replaces all twelve GeLU layers in ViT-B/16 and reaches 79.83% ImageNet-1K Top-1 accuracy after ten epochs of joint adaptation. It also studies SiLU replacement and adaptation in Llama-2 and Llama-3 while preserving their multiplicative gates. These experiments address the different roles of coefficients in standard and bias-free gated FFNs, and we hope they can be evaluated against the concrete methods and architectural conditions.

Our focus on activation replacement defines the scope of the paper. We would appreciate a direct assessment of the architectural analysis and adaptation method within that scope. The existence of precedents for individual ingredients and whether those precedents cover the paper’s specific technical contributions are distinct questions. We therefore respectfully ask the reviewer to clarify how existing results encompass the architectural distinction established by Proposition 1 and the corresponding Llama study, or why these specific differences remain insufficient as a technical advance. This would help us understand and address the reviewer’s concern about the significance of the contribution more precisely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.