acceptodds
Under review as a conference paper at ICLR 2027

On Learning Expand-and-Sparsify Representations

Abstract

Expand-and-sparsify (EaS) representations lift an input to a much higher dimension through a linear map and keep only the k largest coordinates. Existing theory builds that map at random and needs the expanded dimension to grow exponentially in the input dimension, which puts the construction out of practical reach. We ask what is gained by learning the map by gradient descent instead, and how that gain depends on how wide the expansion is. We sweep 73 OpenML classification tasks and 48 regression tasks under a protocol that holds the datasets, expanded dimensions, sparsity levels and splits fixed across every method, and we summarise each comparison by a median over datasets and a Wilcoxon signed rank test. We report four findings. First, learning the expansion helps most when the expansion is narrowest, and its advantage over a random map decays to nothing by the widest expansions we run, so a learned EaS layer can be made small. Second, with the map learned, a top-k representation that keeps at most a tenth of its coordinates is indistinguishable from a dense ReLU layer of the same expanded dimension at every expansion factor we test, while GELU and SiLU are not. Third, when the representation is forced to be binary the sparse code is the better of the two, and the reason is that binarization costs it nothing while it costs the dense layer accuracy. Fourth, how sparse the representation can be is governed by how wide it is: the sparsity level that is free rises as the expansion grows, so there is no fixed limit on how sparse it can be. The regression tasks reproduce the first finding and qualify the second and third.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
expand-and-sparsify, sparse representations, top-k activation, learned projections, random projections, winner-take-all, binary codes, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rC7rOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether learning expand-and-sparsify (EaS) representations through gradient descent can improve performance over random projections while remaining competitive with standard dense representations. Through extensive experiments on 121 classification and regression datasets, the authors show that learned EaS representations outperform random projections and achieve performance comparable to dense ReLU representations even with 90% sparsity, while also demonstrating greater robustness to binarization.

Critical strengths
  1. The paper evaluates the method on 121 classification and regression datasets, covering different settings. I think the experiments are quite comprehensive, and the statistical tests make the results more convincing.

  2. The paper shows that learned EaS representations can achieve similar performance to dense ReLU representations even with 90% sparsity. I think this is an interesting result, as it suggests that sparse representations could reduce memory and computational costs without losing much performance.

Critical weaknesses
  1. The paper is quite difficult to read. This is my main concern. Even in the introduction, there is too much technical detail, which makes it hard to follow the main ideas and understand the key contributions. The introduction is also quite long and could be better organized. I think the authors should make the writing more concise and highlight the main ideas and findings more clearly. At the moment, I find it difficult to fully understand the background and motivation of this work, which also makes it hard for me to clearly judge its value and significance.

  2. Although the experiments cover many datasets, they are all based on relatively simple single-hidden-layer networks and tabular data. I think it would be much more convincing to see some experiments on modern architectures, such as Transformers or deeper neural networks, to show whether these findings are useful in practice. Right now, I am not very convinced of the value of this work.

Questions for authors
  1. Could you make the paper easier to read and understand, especially the introduction, so that I can better understand the motivation and contributions and more comprehensively assess the value of this work?

  2. Could you provide additional experiments on more modern architectures, such as Transformers or deeper neural networks? I think this would help demonstrate the practical value of the findings.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer rC7r

Thank you for the review.

W1 / Q1: Readability, especially the introduction. We agree, and we rewrote the introduction; it is about a quarter shorter, and it now states the question in one sentence and the findings with one number each. It now has four short parts:

  • why expanded layers and their sparsity matter;
  • the two questions we ask;
  • five findings, one sentence and one number each;
  • scope.

Protocol details moved to Section 2, and the terms "data-supported", "value-passing", m/dm/d and k/mk/m are defined once, in plain words, before the results. Each result section now states its finding in its title and ends with a one-line takeaway. A reader who reads only the titles and takeaways gets the paper's conclusions.

W2 / Q2: Transformers or deeper networks. The revision adds Section 4, with two deep models trained from scratch: an 8-block MLP-Mixer on CIFAR-10 and a 6-layer Transformer language model on WikiText-103, each at expansion ratios 4, 8 and 16 with five seeds, plus Transformers of width 256 and 512 (51M parameters). Only the activation of the expanded layer changes. With 10% of units active, top-kk matches ReLU within seed noise in both models: pooled over 15 paired runs, −0.04 accuracy points (p=.95p = .95) and −0.1% perplexity (p=.60p = .60). With 3% active it costs 1.8 to 3.5 points or 2% to 5% perplexity; in the Mixer the cost shrinks as the layer widens, as in the tabular study. We kept the tabular study as the controlled core of the paper because there the expanded layer can be isolated and compared across 121 datasets and eight expansion factors with paired tests. The new section shows that the conclusions carry over to the architectures that motivate the work. We also measured what the sparsity is worth in time (new Appendix M): at batch size 1, a plain PyTorch top-kk FFN with 3% to 10% of units active is slower than a dense one for small layers, but up to 1.8× faster on a GPU and 2.5× faster on a CPU for large ones, with no custom kernel.

Thank you again for your review. Since the discussion period closes soon, below each concern from the review is quoted, followed by how the revision addresses it and what remains open.

The paper is quite difficult to read. This is my main concern. Even in the introduction, there is too much technical detail, which makes it hard to follow the main ideas and understand the key contributions. The introduction is also quite long and could be better organized. [...] I find it difficult to fully understand the background and motivation of this work.

The introduction is rewritten and about a quarter shorter. It now starts from the motivation: existing guarantees for expand-and-sparsify layers need a random map and a width exponential in the input dimension, so it has not been clear whether such layers are useful at practical sizes. It then states the question in one sentence and the findings with one number each. Protocol details moved to Section 2, technical terms are defined once in plain words, and every result section states its finding in its title and ends with a one-line takeaway.

Although the experiments cover many datasets, they are all based on relatively simple single-hidden-layer networks and tabular data. I think it would be much more convincing to see some experiments on modern architectures, such as Transformers or deeper neural networks, to show whether these findings are useful in practice.

The new Section 4 tests the same comparison inside an MLP-Mixer on CIFAR-10 and in Transformer language models on WikiText-103, trained from scratch with five seeds, including a Transformer with 51M parameters. With 10% of units active, top-k matches ReLU in both (pooled change -0.04 accuracy points and -0.1% perplexity); with 3% active it costs 1.8 to 3.5 points or 2% to 5% perplexity. Appendix M adds what this sparsity is worth in time: with a plain PyTorch top-k layer and no custom kernel, large layers run up to 1.8 times faster on a GPU and 2.5 times faster on a CPU at batch size 1, while small layers are slower.

What remains open. The deep models are small compared with current large language models; the Limitations section states this. Readability can only be judged by the reader: if some parts are still hard to follow, we would be grateful to know which ones.

We hope the revision addresses these concerns. We would be grateful to know what is still missing: whether one of these points is not resolved to your satisfaction, or whether another concern keeps the paper below acceptance. We will try our best to address it in the time that remains.

I appreciate the effort the authors have put into the revision and the additional experiments. However, the paper has changed quite substantially from the original submission. If this were a journal submission, I would be happy to take another close look in a further round of review. Given the limited time available in the conference review process, I will keep my original score.

Reviewer_tX4KOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission studies supervised learning with expand-and-sparsify representations in single-hidden-layer networks. It compares learned expansion matrices with random and data-supported matricess, evaluates top-k and local winner-take-all activations against dense activations at matched widths, and, separately trains binary variants. The main reported patterns are that learning the expansion helps most at smaller widths, sparse activations often have small aggregate performance differences from ReLU, and the effects of binarization and sparsity depend on width and task type.

Critical strengths
  1. The quesntion is intereting and the paper presentation is clear to me. The simple architecture makes it possible to study the expansion and activation choices with relatively few architectural confounders. Expressing the sweep through m/dm/d and k/mk/m also makes comparisons across input dimensions easier to interpret.

  2. The study has substantial dataset coverage and uses paired comparisons with common splits. I particularly appreciate the fixed-sparsity experiment in Section 3.4.

Critical weaknesses
  1. Appendix H does not establish the stated theoretical guarantee. I saw a few points here. In Lemma H.4, the approximating function uses cell-average weights but omits the required 1/k1/k normalization. For a constant nonzero target f∗(x)=af^*(x)=a, the construction therefore returns kaka, contradicting the claimed zero approximation error for a constant target. In addition, step (d) of Theorem H.2 removes the 2k/m2k/m term using only k≤0.1mk\le0.1m. This bounds the term by a constant; it does not justify the stated vanishing rate. These are substantive problems with the proof.

  2. The paper repeatedly interprets nonsignificant Wilcoxon tests as evidence that sparsity or binarization “costs nothing.” Small median differences suggest competitive performance, but do not establish equivalence. For example, Table 4 reports a median R2R^2 difference of −0.014-0.014 at m=8dm=8d, with 15 wins and 32 losses, yet describes this regime as matching ReLU. The conclusions need practical tolerances and uncertainty estimates, or more cautious wording.

  3. The submission does not provide the OpenML task IDs, complete preprocessing and normalization procedures, or exact straight-through estimators. Appendix F also attributes missing configurations to incomplete sweep coverage without identifying them individually. Since the contribution is primarily empirical, these omissions make the reported comparisons difficult to verify.

Questions for authors

Can you provide a corrected version of Theorem H.2 and Lemma H.4 that resolves the normalization, calibration, and k/mk/m issues, with all assumptions stated explicitly?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer tX4K (1/2)

Thank you for the careful reading, including the proofs. All three weaknesses are correct, and we have fixed them.

W1 / Q: Lemma H.4 and Theorem H.2. You are right on both points, and the appendix (Appendix H in the submission, now Appendix J) is rewritten with every assumption stated.

  • Normalization. The approximating function now uses wj=1kf∗(Cj)w_j = \frac{1}{k} f^{\ast}(C_j) (eq. 8), the normalization required by Dasgupta and Tosh (2020). For a constant target f∗=af^{\ast} = a it returns exactly aa.
  • Assumptions. The rows of MM are drawn uniformly from the unit sphere, and inputs lie on it, as in Dasgupta and Tosh. Assumption J.2: the log-odds f∗=ln⁡η1−ηf^{\ast} = \ln\frac{\eta}{1-\eta} is λ\lambda-Lipschitz (so η\eta is λ/4\lambda/4-Lipschitz and bounded away from 0 and 1, as the appendix now states). Assumption J.3: 1k(∑jf∗(Cj)2)1/2≤R\frac{1}{k}(\sum_j f^{\ast}(C_j)^2)^{1/2} \le R, so that the approximating function lies in FR={w⋅σ(Mx):∥w∥2≤R}\mathcal{F}_R = \lbrace \mathbf{w}\cdot\sigma(M\mathbf{x}) : \Vert \mathbf{w}\Vert _2 \le R \rbrace. fnf_n is the minimizer of the empirical logistic loss over FR\mathcal{F}_R.
  • Calibration. For the logistic cost ϕ(x)=log⁡2(1+ex)\phi(x) = \log_2(1+e^x), H(η)=inf⁡αhη(α)H(\eta) = \inf_\alpha h_\eta(\alpha) is the binary entropy, and the condition of Zhang (2004) holds with exponent 2 and constant 1/21/\sqrt{2}. This gives L(f)−L∗≤2(A(f)−A∗)L(f) - L^{\ast} \le \sqrt{2(A(f) - A^{\ast})}, which the proof now derives explicitly.
  • The 2k/m2k/m term. Theorem J.4 keeps it. With probability at least 1−2δ1 - 2\delta, for any kk and any R≥1/kR \ge 1/\sqrt{k}, L(fn)−L∗≤c(R2k(1+log⁡(2/δ))/n)1/4+(2cdλ(2k/m+2c0(dlog⁡m+log⁡(1/δ))/m)1/(d−1))1/2L(f_n) - L^{\ast} \le c (R^2 k (1+\log(2/\delta))/n)^{1/4} + (2 c_d \lambda (2k/m + 2c_0 (d\log m + \log(1/\delta))/m)^{1/(d-1)})^{1/2}. Both terms vanish only if R2k/n→0R^2 k/n \to 0 and k/m→0k/m \to 0 (Remark J.5). In the regime we study, k=smk = sm with a fixed s≤0.1s \le 0.1, Corollary J.6 and Remark J.7 state that the approximation term is O((s+(dlog⁡m+log⁡(1/δ))/m)1/(2(d−1)))O((s + (d\log m + \log(1/\delta))/m)^{1/(2(d-1))}) and does not vanish as mm grows; with a suitable choice of RR, the estimation term does not grow with mm. We should also have been clearer about the purpose of this appendix. The theory is not one of our contributions. It shows that in the practical regime we study (mm a small multiple of dd and k=smk = sm, rather than mm exponential in dd) the guarantees for a random MM are much weaker, and this is the regime in which learning MM helps most (Section 3.1). The introduction and Appendix D now say this, and the claims of the paper are empirical.
AuthorsOpenReview ↗

Reply to Reviewer tX4K (2/2)

W2: Non-significant tests read as equivalence. We agree. For every comparison and expansion factor, the revision reports the exact 95% CI of the median paired difference, a sign test next to the Wilcoxon test, and TOST equivalence tests. These are summarized by δ∗\delta^\ast, the smallest margin at which equivalence holds at level 0.05 (new Table 7, App. E), so readers can apply their own tolerance.

  • Classification, top-kk vs. ReLU. From m=2dm = 2d, the 95% CI of the median lies within [−0.78, +0.33] accuracy points and δ∗\delta^\ast is between 0.31 and 0.70 (at most 0.83 with Holm correction across the eight widths). At m=dm = d, top-kk is behind (p=0.035p = 0.035, not significant after Holm correction; δ∗=1.53\delta^\ast = 1.53), and the text now says so.
  • Binarizing top-kk costs less than half a point at every m/dm/d (δ∗≤0.36\delta^\ast \le 0.36; 0.43 with Holm correction). The text no longer says "costs nothing".
  • Your example (Table 4, m=8dm = 8d) is one where our wording was wrong. The 95% CI of the median is [−0.028, −0.002] in R2R^2, and the sign test (15 wins, 32 losses) gives p=0.019p = 0.019 (0.11 after Holm correction across widths). On regression, top-kk trails ReLU by a small margin at 4d4d to 16d16d, losing on about two thirds of the datasets, and is within 0.02 R2R^2 from m=16dm = 16d (TOST p≤0.021p \le 0.021; within 0.025 after Holm correction). We removed "matches ReLU once the expansion is at least 4d4d" and "at no measurable cost between 4d4d and 64d64d".
  • Fixed sparsity (Sec. 3.4). At 90%, top-kk is within one point of ReLU at every m/dm/d (TOST p≤0.021p \le 0.021, also after Holm correction). At 97%, it is significantly behind at 4d4d and 8d8d, and at 16d16d before correction (medians −0.59 to −0.33), and within 0.6 points from 64d64d (0.83 after correction). At 99%, it is significantly behind from 4d4d to 128d128d, with the deficit shrinking from 2.42 to 0.35 points at 256d256d, where it is within one point only before correction (TOST p=0.008p = 0.008; 0.064 after). "97% is free from 64d64d" is now "within 0.6 points from 64d64d". While redoing this, we found that the submitted 90% row at m≤8dm \le 8d included a few configurations slightly above our own k≤0.105mk \le 0.105m cap. With the cap applied consistently, the conclusions do not change.

W3: Reproducibility. The new Appendix H "Reproducibility details" gives:

  • the OpenML task ID, sample size, encoded input dimension and task type of all 121 datasets;
  • the preprocessing: one-hot encoding of categorical columns; median imputation (computed on the full dataset before splitting, which we now state); stratified 70/10/20 splits with three fixed seeds; standardization fitted on the training split; projection of each input onto the unit sphere;
  • the model details: rows of MM renormalized to unit norm at every forward pass, no hidden bias, readout initialized to 1/m1/m;
  • the exact straight-through estimator: the forward pass emits the 0/1 mask, and the backward pass uses the gradient of z=y⊙maskz = y \odot \text{mask}, i.e. identity on the selected units and zero elsewhere. For the dense binary baseline the mask is 1[y>0]\mathbb{1}[y > 0], so the backward pass is that of ReLU.

You were right that App. F did not identify the missing configurations. All 31 are now listed (15 classification, 16 regression). Each is a case where the only admissible sparsity level, k/m=0.1k/m = 0.1, rounds to an integer kk just above our k≤0.105mk \le 0.105m cap (e.g. m=16m = 16 gives k=2k = 2). They come from the sparsity grid, not from sweep coverage, and the old sentence was wrong. We reran all 31 with k=⌊0.1m⌋k = \lfloor 0.1m \rfloor, and every number in the revision includes them. Two results changed. At m=dm = d, top-kk is now behind ReLU on classification (p=0.035p = 0.035, though not after Holm correction across widths). On regression at m=4dm = 4d, the sign test now shows top-kk behind ReLU (p=0.040p = 0.040, again not after correction). Code and all result files are in the supplementary material.

Reviewer_EMXjOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper empirically investigates expand and sparsify (EaS) representations, where the input is projected to a higher dimensional space using a linear transformation, and then sparsified by keeping only the top-k of the largest activations. Previous theoretic work utilized random linear transformations that required exponential expansions dimension. The paper investigates whether a learned transformation for expansion using gradient descent can allow for better performance while utilizing fewer expansion dimensions (transformation parameters). The authors motivate this study for applicability to modern deep learning architectures such as the transformer and MLP-mixer.

The experiments were conducted using 121 OpenML tabular datasets comparing gradient descent learned, data-supported (random samples from the dataset), and random (completely random weights) for the expansion transformation. Additionally, the authors compare top-k activation (global, and block local), against the other activation functions such as ReLU, GELI and SiLU as dense baselines. The protocol kept expansion size, sparsity levels and dataset splits consistent, systematically varying the expansion factor, and the authors report the median performance values over the dataset and p-values from the Wilcoxon signed rank test.

The findings reveal that learning the linear transformation for expansion helps over random weights at low expansion factors. As the expansion factor grows the performance difference between learned and random expansion transformation degrades. Second, the learned expansion transformation along with sparsification matches the performance of ReLU activation at the same expansion factor. Binarization of top-k EaS outperforms binarization of dense activation based representations. This is because the performance in dense representations degrades once binarized. Finally, the sparsity achievable without loss in performance grows with the expansion factor.

Critical strengths

The paper is clearly written and well organized: each section maps to one finding, the experimental protocol is described in sufficient detail to reproduce, and the use of paired Wilcoxon tests with explicit p-values throughout adds to the rigor.

The scale (number of runs) of the experiments is another strength of the paper.

Critical weaknesses

The gap between motivation to utilize EaS in modern architectures such as the transformers or MLP-mixers as compared to the evaluation architectures considered undermines the practical claims made by the paper. All the experiments use an MLP with a single hidden layer, and notably the transformer in practice also uses expansion with dense activations.

The findings largely confirm expected behavior at the scale of the architectures. This make the technical contributions of the work limited.

Minor (does not impact the rating): The statistical methodology conflates non-significance of the tests with equivalence. The claims can be re-written appropriately.

Questions for authors

ReLU and Top-k sparsification perform better than other activations. Can the authors provide some additional insights into why this can be the case, and how that can translate into transformers where expansion is used along with GELU or SiLU ?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer EMXj (1/2)

Thank you for the review. We respond point by point.

W1: Gap between the Transformer/MLP-Mixer motivation and the single-hidden-layer experiments. We agree this was the main gap. We studied the expanded layer in isolation on purpose. In a deep model, residual paths, normalization and attention all interact with the activation, which makes it hard to attribute a difference to sparsification. But isolation alone does not show that the result transfers, so the revision adds Section 4. We replace only the activation of the expanded layer in (i) an 8-block MLP-Mixer on CIFAR-10 and (ii) a 6-layer decoder-only Transformer on WikiText-103 (164M training tokens). Both are trained from scratch, with expansion ratios 4, 8 and 16. Arms: ReLU, GELU, and top-kk with 10% or 3% of units active. In the Mixer, top-kk goes in the channel MLP alone or in both mixers. Everything else is fixed, no hyperparameter is tuned for any arm, and we use five seeds with paired comparisons. We also trained two larger Transformers (width 256 at ratios 4 and 8; width 512 with 8 layers and 51M parameters at ratio 4; three and five seeds).

Results (new Table 2):

  • With 10% of units active, top-kk matches ReLU within seed noise. Mean changes are −0.28, +0.16 and +0.01 accuracy points in the Mixer, and −0.4%, −0.1% and 0.0% test perplexity in the Transformer, at ratios 4, 8 and 16. Pooled over the 15 seed-and-ratio pairs: −0.04 points (p=.95p = .95) and −0.1% (p=.60p = .60). At every ratio the seeds disagree in sign, also in the larger Transformers (+0.4% at width 256, −0.3% at width 512).
  • With 3% active, both models pay, and all 15 seed pairs are worse than ReLU in both. In the Mixer the price falls with width (−3.5, −2.2, −1.8 points), mirroring our tabular finding that wider layers can afford more sparsity. In the Transformer it does not (+3.3%, +3.3%, +2.9% perplexity; +4.7% and +3.4% at width 256; +2.2% at width 512).

These results agree with earlier evidence that Transformer FFNs tolerate top-kk (Li et al., ICLR 2023) and are naturally sparse under ReLU. In our Transformer, ReLU keeps on average 13%, 9% and 6% of units active at ratios 4, 8 and 16. A counterpoint: in large-scale pretraining, plain top-kk can cost quality unless combined with other changes (You et al., 2025). Our models are small, and we say so in the limitations.

W2: The findings largely confirm expected behavior. The direction of some findings is expected; for example, a learned map should beat a random one, since only it sees the labels. What was not known in advance is how large the effects are and how they change with width, and the revision now quantifies these with uncertainty:

  • (a) The gain from learning decays with width: a median 3.4 accuracy points at m=dm = d, down to 0.13 at m=256dm = 256d (95% CI [0.00, 0.40], within 0.5 points by TOST). Against a map whose rows are sampled from the data, it does not decay. We trace this to the sampled map running out of distinct rows (App. G).
  • (b) The existing theory needs mm exponential in dd. Yet from m=2dm = 2d on, a learned code with at most 10% of units active is within 0.7 accuracy points of a dense ReLU layer of the same width.
  • (c) The sparsity a layer can afford grows with its width. The highest of our tested levels that stays within one point of ReLU is 90% up to m=16dm = 16d and 97% from 64d64d, with Holm correction across the eight widths (without it, 97% from 16d16d and 99% at 256d256d). The Mixer in Section 4 shows the same trend.

Minor: non-significance vs. equivalence. Agreed and fixed; see the general response, item 2.

AuthorsOpenReview ↗

Reply to Reviewer EMXj (2/2)

Q: Why do ReLU and top-kk beat GELU and SiLU, and how does this translate to Transformers? We see one reason specific to our setup, and Section 4 shows the gap does not transfer as a general rule. (1) Our parameterization bounds the pre-activations. Inputs have unit norm, the rows of MM are renormalized to unit norm in every forward pass, and there is no hidden bias, so every pre-activation lies in [−1,1][-1, 1]. On that interval GELU and SiLU are within 1.2% and 0.4% of a quadratic (maximum error relative to their range), against 9.4% for ReLU. So in our setup they act as a much weaker nonlinearity. We tested this directly (new Appendix L). We multiplied every pre-activation by a fixed c∈{4,16}c \in \lbrace 4, 16 \rbrace, for GELU and SiLU and also for ReLU and top-kk as controls, on 66 to 71 classification tasks at m/d∈{2,4,8,16,64}m/d \in \lbrace 2, 4, 8, 16, 64 \rbrace; the unscaled runs reproduce the paper's cells exactly. Scaling helps every activation at narrow widths, ReLU and top-kk included: at c=16c = 16 and m=4dm = 4d the median gain is +1.4 (GELU), +2.0 (SiLU), +1.1 (ReLU) and +1.0 (top-kk) accuracy points. For ReLU and top-kk, cc only rescales the hidden output, so their gain comes from optimization, not from shape. At equal c∈{4,16}c \in \lbrace 4, 16 \rbrace, GELU and SiLU tie with ReLU at every width (|median| ≤ 0.13 points, equivalence margin ≤ 0.49). With cc chosen on validation for every activation, as kk is, GELU and SiLU match ReLU (margin ≤ 0.34 points), and top-kk is within 1.24, 1.18, 0.87, 0.79 and 0.35 points of ReLU at m/d=2,4,8,16,64m/d = 2, 4, 8, 16, 64, so its small extra cost shrinks as the layer widens. So the tabular GELU/SiLU deficit comes from our normalization, not from the activations, and the paper no longer ranks them. (2) Deep models. With LayerNorm and unconstrained weights the pre-activations are not confined, and the ranking changes with the architecture. GELU is ahead of ReLU in our Mixer (+0.78 points, all five seeds) and behind it in our Transformer (+2.1% perplexity, all five seeds). What transfers in both models is the comparison the paper is about: top-kk with 10% of units active against dense ReLU.

Thank you again for your review. Since the discussion period closes soon, below each concern from the review is quoted, followed by how the revision addresses it and what remains open.

The gap between motivation to utilize EaS in modern architectures such as the transformers or MLP-mixers as compared to the evaluation architectures considered undermines the practical claims made by the paper. All the experiments use an MLP with a single hidden layer, and notably the transformer in practice also uses expansion with dense activations.

The new Section 4 replaces ReLU with top-k in the expanded layer of an MLP-Mixer on CIFAR-10 and of Transformer language models on WikiText-103 (expansion ratios 4, 8 and 16, five seeds, plus larger Transformers of width 256 and 512 with up to 51M parameters). With 10% of units active, top-k matches ReLU within seed noise (pooled over 15 paired runs: -0.04 accuracy points, p = .95, and -0.1% perplexity, p = .60). With 3% active it costs 1.8 to 3.5 points or 2% to 5% perplexity.

The findings largely confirm expected behavior at the scale of the architectures. This make the technical contributions of the work limited.

The revision quantifies how large the effects are and how they change with width, with confidence intervals. Learning the map adds a median 3.4 accuracy points at m = d but only 0.13 at m = 256d; from m = 2d on, a layer with at most 10% of units active is within 0.7 points of ReLU; and the sparsity a layer can afford within one point of ReLU grows from 90% up to m = 16d to 97% from m = 64d.

ReLU and Top-k sparsification perform better than other activations. Can the authors provide some additional insights into why this can be the case, and how that can translate into transformers where expansion is used along with GELU or SiLU?

Appendix L shows that the gap comes from the setup, where every pre-activation lies in [-1, 1] and GELU and SiLU are nearly quadratic there. With the pre-activation scaled by the same factor (4 or 16), GELU and SiLU tie with ReLU at every width tested. In the deep models of Section 4 the pre-activations are not bounded and the ranking depends on the architecture (GELU is ahead of ReLU in the Mixer and behind it in the Transformer), so the paper no longer ranks these activations.

The statistical methodology conflates non-significance of the tests with equivalence. The claims can be re-written appropriately.

Every claim of parity now comes with a 95% confidence interval and a TOST equivalence test (Table 7), with Holm correction across the eight widths (Table 8), and the claims are rewritten to match.

What remains open. The deep models are small (at most 51M parameters), and pretrained large language models are not tested; the Limitations section states this. In the deep models, top-k is compared with ReLU and GELU, not SiLU. Whether the effects above are expected is a judgment that the numbers can inform but not settle.

We hope the revision addresses these concerns. We would be grateful to know what is still missing: whether one of these points is not resolved to your satisfaction, or whether another concern keeps the paper below acceptance. We will try our best to address it in the time that remains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for careful and constructive reviews. They raised four main points. The revised PDF addresses all of them (changes in blue).

  1. Theory (tX4K). Both errors in Appendix H were real: the missing 1/k1/k in the approximant, and removing the 2k/m2k/m term without justification. The appendix (now Appendix J) states a corrected theorem with every assumption explicit. It holds for a random MM and any kk; at a fixed fraction k/mk/m, as in our experiments, its approximation term does not vanish as mm grows, and the paper now says so instead of suggesting that the theory covers our grid. The theory is not one of our contributions: it shows how weak the guarantees are in the practical regime, with mm a small multiple of dd and k=smk = sm rather than mm exponential in dd, which is where learning MM helps most.
  2. Non-significance is not equivalence (tX4K, EMXj). Every claim of parity now comes with a 95% confidence interval for the median paired difference and a TOST equivalence test. We report δ∗\delta^\ast, the smallest margin at which equivalence holds (new Table 7, App. E), and Holm-corrected pp-values and margins across the eight expansion factors (new Table 8). A Bayesian signed-rank test with a region of practical equivalence (Benavoli et al., JMLR 2017) gives a second check. Most claims survive as quantified statements; a result that is significant only without the correction is now marked in the text. Example: from m=2dm = 2d, top-kk is within 0.7 accuracy points of ReLU at every expansion factor (0.83 with Holm correction). Two regression statements did not survive, and we removed them.
  3. Deep architectures (EMXj, rC7r). A new Section 4 replaces ReLU in the expanded layers of an MLP-Mixer trained from scratch on CIFAR-10 and of a Transformer language model trained from scratch on WikiText-103. With 10% of units active, top-kk matches ReLU in both: over 15 paired runs (five seeds, expansion ratios 4, 8 and 16), the mean change is −0.04 accuracy points (p=.95p = .95) and −0.1% perplexity (p=.60p = .60). The same holds for larger Transformers (width 256, and width 512 with 51M parameters). At 3% active it costs 1.8 to 3.5 accuracy points or 2% to 5% perplexity; the Mixer's cost shrinks as the layer widens, as in our tabular results, the Transformer's does not.
  4. Reproducibility and readability (tX4K, rC7r). A new appendix lists every OpenML task ID, the full preprocessing, the exact straight-through estimator, and every previously missing configuration, all of which we have now rerun. The introduction is rewritten and about a quarter shorter, and each result section now states its finding in its title. Code and all result files are in the supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.