- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoQuad replaces Transformer FFN activations with layer-wise quadratic functions, using data-dependent initialization and joint coefficient–backbone distillation. The paper analyzes coefficient absorption in standard and gated FFNs and evaluates accuracy preservation and private-inference efficiency across ViT, BERT, and Llama.
Critical strengths
The architectural analysis distinguishes coefficient reparameterization in biased FFNs from the channel-wise invariant in bias-free gated FFNs, providing a clear motivation for coefficient learning.
The ViT results are promising: full GeLU replacement achieves 79.83% ImageNet-1K accuracy versus 81.45% for the teacher. Training coefficients improves accuracy by 3.53 percentage points over the frozen-coefficient control.
Evaluation includes multiple model families and two secure-inference backends. The paper distinguishes estimated operator costs from measured end-to-end latency and provides useful implementation details.
Critical weaknesses
Incomplete component attribution. The Llama comparison changes both initialization and coefficient trainability, leaving their individual contributions unclear. The proposed coefficient-ratio explanation would benefit from direct diagnostics or constrained controls. In ViT, optimization and induced regularization effects also remain insufficiently separated.
Efficiency comparisons need strengthening. The main comparison uses SHAFT’s Poly implementation; comparison against its Fourier-based GeLU configuration would better establish the practical advantage. The reported 2.56× speedup additionally changes Softmax and needs a corresponding accuracy measurement.
Limited LLM validation. Evaluation focuses on ARC-Easy with half the SiLU layers replaced. Llama secure-inference results rely on composed activation-cost estimates, leaving full-model latency and broader language-model quality insufficiently established.
Questions for authors
Can the authors provide a matched initialization-by-trainability ablation on Llama and examine the learned coefficient ratios?
How does CoQuad compare with SHAFT’s Fourier configuration, and what accuracy accompanies the combined GeLU–Softmax speedup?
Can additional language-model benchmarks or perplexity measurements support broader quality preservation?
2 comments
Thank you to the authors for the rebuttal. It addresses some of my concerns and questions to a certain extent. In light of my earlier comments, I have decided to keep my score at 3.
Q1
Thank you for suggesting these controls. In a new experiment extending the paper’s 16-layer setting, we successfully replaced SiLU in all 32 SwiGLU layers of Llama-2-7B. After 1,500 steps, CoQuad reached 73.57% on ARC-Easy and 61.54% averaged across seven zero-shot tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA). The baseline model scored 75.6% on ARC-Easy and 62.73% on average across the seven tasks, leaving gaps of just 2 and 1.2 percentage points, respectively, after full replacement.
Two controls in this full-replacement experiment directly address the fixed-coefficient question. We kept the SwiGLU gate intact and trained the backbone under the same distillation recipe. Freezing each layer’s DirectFit coefficients gave 27.69% on ARC-Easy. Using MPCFormer’s fixed polynomial gave 27.74%. From identical DirectFit coefficients, allowing them to train raised ARC-Easy by 45.88 percentage points. Proposition 1 identifies the relevant expressive freedom: in a nondegenerate bias-free gated channel, rescaling the projections preserves the coefficient ratio ac/b2, whereas training the coefficients allows that ratio to change. CoQuad used this freedom in practice: across the 32 layers, the learned ratios ranged from 0.0006 to 0.036, entirely above the DirectFit initialization range of −0.0025 to 0.0005. The 32-layer results show the practical importance of this freedom for recovering accuracy under the tested recipe.
For the converse control, we initialized the quadratics in 16 alternating layers with MPCFormer’s coefficients and trained them jointly with the backbone. After 1,500 steps, the final scores were nearly identical to those obtained with DirectFit. The gap was less than 0.5 percentage points for both ARC-Easy accuracy and the seven-task average. DirectFit’s advantage was visible early in training. With DirectFit, the median constant term c started near zero and ended at 0.013. With MPCFormer initialization, it started at 0.50 and was still 0.224 at step 300. Over steps 260–300, the median distillation KL was 0.155 with DirectFit versus 0.892 with MPCFormer initialization, indicating that DirectFit helped the student match the teacher’s outputs sooner. DirectFit also yielded lower final perplexity on both corpora, with the larger reduction on C4.
Q2
SHAFT’s repository provides two GELU implementations, polynomial and Fourier. Since Fourier evaluation requires multiple trigonometric terms and secure interactions, it is inherently much more expensive than our univariate quadratic approximation; on input shapes (128, 4096) and (256, 4096), SHAFT-Fourier is 2.98× and 2.89× slower than SHAFT-Poly, respectively. Since our Table 1 focuses on runtime, we use SHAFT-Poly as the faster and more conservative runtime baseline. Therefore, comparing against SHAFT-Fourier would only further increase CoQuad’s reported speedup.
CoQuad focuses on FFN activations. For the combined GELU–Softmax experiment, we paired CoQuad’s quadratic GELU with MPCFormer’s 2Quad Softmax, used unchanged, to measure the end-to-end benefit of the two replacements. With 2Quad Softmax held fixed, Proposition 1(i) shows that the trained ViT-B/16 can be reparameterized to use MPCFormer’s fixed quadratic in its FFNs by adjusting the adjacent affine layers, preserving the model’s predictions. The combined configuration delivers a 2.56× end-to-end speedup over the baseline with standard secure GELU and the original Softmax and achieves 79.13% Top-1 accuracy on ImageNet-1K, 0.70 percentage points below CoQuad Full-12 with the original Softmax (79.83%).
Q3
Thank you for the suggestion. We evaluated seven downstream tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA) and report their unweighted mean accuracy. CoQuad achieved 62.34%, compared with 62.73% for the original model. Using MPCFormer’s initial coefficients, mean accuracy was 38.57% with frozen coefficients and 62.08% with trainable coefficients.
We also measured perplexity on WikiText-2 and C4 at context lengths of 1,024 and 2,048 tokens. Across all four settings, CoQuad’s perplexity was 0.20–0.30 above the original model’s and lower than both MPCFormer controls. At 2,048 tokens, for example, CoQuad scored 6.37 on WikiText-2 and 8.36 on C4, compared with 6.10 and 8.16 for the original model. These results extend the evidence for quality preservation beyond ARC-Easy to next-token prediction.