- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes using error correcting output coding (ECOC) to calculate an approximate softmax score. The main application the paper targets is the unembedding (LM head) layer, which can be very expensive for very small models.
Critical strengths
The ECOC-Softmax method is pretty interesting as an approximation for softmax. The writing explains the concept well.
Critical weaknesses
My main concern with this paper is that the targeted problem (making the unembedding faster) is generally not a real problem at large model scales. The only models for which the unembedding is a large cost are very small models with a few billion active parameters. For example, in Kimi K3, a large MoE, there are 104B active parameters and only 7168*160000 = 1.1B parameters in the unembedding. Furthermore, since the unembedding is a dense matmul, the unembedding will generally have higher MFU than other ops, making it relatively cheap. If the concern is the memory of softmax, one can use a tiled online softmax kernel (we only need to store the probability of the correct token for training) or write a fast top-k kernel for sampling. Even if we want full sampling we can also use online gumbel sampling, which has the same marginal probabilities. The test setup in this model (slow GPU, 256 sequence length, not fused/well optimized kernels for baseline softmax) also makes it unclear if the proposed method can be implemented efficiently.
Questions for authors
Are there other uses (e.g. in attention, which is the other key use of softmax) of ECOC-Softmax? I like the idea and its pretty cool, but it just doesn't seem very practical or useful in the proposed setup since the targeted problem isn't a problem at scale.
Connections to extreme classification
Dear authors,
Thank you for the interesting manuscript.
I think the manuscript would benefit from discussing connections to the extreme classification field, which studies classification problems with very large label spaces.
For example, Jasinska and Karampatziakis (2016) and Evron et al. (2018) developed compact, graph-based output representations and efficient decoding methods, with the latter explicitly connecting these representations to ECOC. These approaches represent large label spaces using a logarithmic number of learned scoring functions and explore trade-offs between accuracy, model size, and prediction time.
Additionally, in Evron et al. (2023), we showed that codeword-to-class assignments can substantially affect ECOC performance. In particular, assigning similar codewords to similar classes can make the induced binary subproblems easier and improve generalization, while preserving the codebook’s other important properties. This suggests a potential extension to the binary-index-based assignment with random-bit augmentation described in your Section 4.1: exploring assignments that reflect token similarity. It would be interesting to discuss this connection, perhaps as a direction for future work.
It is important to note that all the aforementioned papers did not utilize ECOC for the softmax layer of neural networks like the manuscript suggests, but were rather limited to basic (mostly linear) models.
Good luck with your submission!