- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether learning expand-and-sparsify (EaS) representations through gradient descent can improve performance over random projections while remaining competitive with standard dense representations. Through extensive experiments on 121 classification and regression datasets, the authors show that learned EaS representations outperform random projections and achieve performance comparable to dense ReLU representations even with 90% sparsity, while also demonstrating greater robustness to binarization.
Critical strengths
-
The paper evaluates the method on 121 classification and regression datasets, covering different settings. I think the experiments are quite comprehensive, and the statistical tests make the results more convincing.
-
The paper shows that learned EaS representations can achieve similar performance to dense ReLU representations even with 90% sparsity. I think this is an interesting result, as it suggests that sparse representations could reduce memory and computational costs without losing much performance.
Critical weaknesses
-
The paper is quite difficult to read. This is my main concern. Even in the introduction, there is too much technical detail, which makes it hard to follow the main ideas and understand the key contributions. The introduction is also quite long and could be better organized. I think the authors should make the writing more concise and highlight the main ideas and findings more clearly. At the moment, I find it difficult to fully understand the background and motivation of this work, which also makes it hard for me to clearly judge its value and significance.
-
Although the experiments cover many datasets, they are all based on relatively simple single-hidden-layer networks and tabular data. I think it would be much more convincing to see some experiments on modern architectures, such as Transformers or deeper neural networks, to show whether these findings are useful in practice. Right now, I am not very convinced of the value of this work.
Questions for authors
-
Could you make the paper easier to read and understand, especially the introduction, so that I can better understand the motivation and contributions and more comprehensively assess the value of this work?
-
Could you provide additional experiments on more modern architectures, such as Transformers or deeper neural networks? I think this would help demonstrate the practical value of the findings.
6 comments
Follow-up
Thank you again for your review. Since the discussion period closes soon, below each concern from the review is quoted, followed by how the revision addresses it and what remains open.
The paper is quite difficult to read. This is my main concern. Even in the introduction, there is too much technical detail, which makes it hard to follow the main ideas and understand the key contributions. The introduction is also quite long and could be better organized. [...] I find it difficult to fully understand the background and motivation of this work.
The introduction is rewritten and about a quarter shorter. It now starts from the motivation: existing guarantees for expand-and-sparsify layers need a random map and a width exponential in the input dimension, so it has not been clear whether such layers are useful at practical sizes. It then states the question in one sentence and the findings with one number each. Protocol details moved to Section 2, technical terms are defined once in plain words, and every result section states its finding in its title and ends with a one-line takeaway.
Although the experiments cover many datasets, they are all based on relatively simple single-hidden-layer networks and tabular data. I think it would be much more convincing to see some experiments on modern architectures, such as Transformers or deeper neural networks, to show whether these findings are useful in practice.
The new Section 4 tests the same comparison inside an MLP-Mixer on CIFAR-10 and in Transformer language models on WikiText-103, trained from scratch with five seeds, including a Transformer with 51M parameters. With 10% of units active, top-k matches ReLU in both (pooled change -0.04 accuracy points and -0.1% perplexity); with 3% active it costs 1.8 to 3.5 points or 2% to 5% perplexity. Appendix M adds what this sparsity is worth in time: with a plain PyTorch top-k layer and no custom kernel, large layers run up to 1.8 times faster on a GPU and 2.5 times faster on a CPU at batch size 1, while small layers are slower.
What remains open. The deep models are small compared with current large language models; the Limitations section states this. Readability can only be judged by the reader: if some parts are still hard to follow, we would be grateful to know which ones.
We hope the revision addresses these concerns. We would be grateful to know what is still missing: whether one of these points is not resolved to your satisfaction, or whether another concern keeps the paper below acceptance. We will try our best to address it in the time that remains.
I appreciate the effort the authors have put into the revision and the additional experiments. However, the paper has changed quite substantially from the original submission. If this were a journal submission, I would be happy to take another close look in a further round of review. Given the limited time available in the conference review process, I will keep my original score.
Reply to Reviewer rC7r
Thank you for the review.
W1 / Q1: Readability, especially the introduction. We agree, and we rewrote the introduction; it is about a quarter shorter, and it now states the question in one sentence and the findings with one number each. It now has four short parts:
Protocol details moved to Section 2, and the terms "data-supported", "value-passing", m/d and k/m are defined once, in plain words, before the results. Each result section now states its finding in its title and ends with a one-line takeaway. A reader who reads only the titles and takeaways gets the paper's conclusions.
W2 / Q2: Transformers or deeper networks. The revision adds Section 4, with two deep models trained from scratch: an 8-block MLP-Mixer on CIFAR-10 and a 6-layer Transformer language model on WikiText-103, each at expansion ratios 4, 8 and 16 with five seeds, plus Transformers of width 256 and 512 (51M parameters). Only the activation of the expanded layer changes. With 10% of units active, top-k matches ReLU within seed noise in both models: pooled over 15 paired runs, −0.04 accuracy points (p=.95) and −0.1% perplexity (p=.60). With 3% active it costs 1.8 to 3.5 points or 2% to 5% perplexity; in the Mixer the cost shrinks as the layer widens, as in the tabular study. We kept the tabular study as the controlled core of the paper because there the expanded layer can be isolated and compared across 121 datasets and eight expansion factors with paired tests. The new section shows that the conclusions carry over to the architectures that motivate the work. We also measured what the sparsity is worth in time (new Appendix M): at batch size 1, a plain PyTorch top-k FFN with 3% to 10% of units active is slower than a dense one for small layers, but up to 1.8× faster on a GPU and 2.5× faster on a CPU for large ones, with no custom kernel.