- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces a method named HTSS to speed up the softmax function in the language modeling head module of large language models. Specifically, HTSS first takes scores of target token, K most frequent tokens, and M sampled tokens from the last of vocabulary, then estimates the training loss based on their scores and frequency. Experimental results show that HTSS can achieve better results under the same computing with baseline methods while reducing the peak GPU memory usage.
Critical strengths
- The proposed method achieves better performance while less GPU memory used compared to other baseline methods.
- The effectiveness of HTSS is supported by theoretical analysis, and HTSS is well motivated.
Critical weaknesses
- HTSS mainly focuses on improving the training speed of compact language models with large vocabulary where the forward of language modeling head composes nearly 30% of the overall computation.
- The effectiveness of HTSS will reduce for larger language models and less vocabulary used. Therefore, large language models like LLaMA 70B may not benefit from this method.
Questions for authors
- I find that most of improvement on the zero-shot downstream tasks is less than 1% (Table 4). Are these improvements statistically significant?
3 comments
Thank you for your responses, which address my concerns! I have modified my ratings, correspondingly, Good luck : )
Thank you! We are glad the significance analysis answered your question, and we appreciate your support and the updated score.
Thank you for the review and for raising the question of significance. We have uploaded a revised version, with changes to the text marked in blue. The revision labels which end of Figure 3 holds the most frequent tokens, reports every seed in one table (Table 4, which renumbers the zero-shot table you refer to as Table 5), states both training budgets in the abstract, and adds the significance tests to the caption of Table 5.
Q1 (significance of the zero-shot gains). The gains on LAMBADA are significant, and on the multiple-choice tasks HTSS matches CE while training at a lower cost. In Table 5, HTSS-mc reduces its perplexity from 242.6 to 191.7 (tied) and from 218.9 to 162.1 (untied). Averaged over three seeds, HTSS-mc gains 3.0 accuracy points both tied and untied, and HTSS gains 2.2 and 2.0 at matched steps (Holm p≤1.6×10−15; Table 6). Additionally, in Table 5, no multiple-choice difference from CE is significant. Every difference lies within 1.4 standard errors, below the 1.96 needed at the 5% level, and the largest is 2.1 accuracy points (HTSS on SciQ, tied). Paired per-item tests over three seeds are more sensitive. We compared each HTSS and HTSS-mc run with the CE run of the same seed (exact McNemar tests, Holm-corrected across the seven tasks; App. C.3, Table 6). Across the six multiple-choice tasks, only two differences are significant: both are small gains for HTSS-mc (ARC-c +1.5 points tied; HellaSwag +0.5 untied), and none favors CE. We also ran the evaluations on the 304M vocabulary sweep; in this setting, we get similar results: HTSS-mc is never significantly behind CE on any multiple-choice task under either accuracy score, and its LAMBADA gains are significant at every vocabulary size (App. D.3, Table 8).
W1/2 (scale and vocabulary size). We agree with the point on scale, which §1 already stated as the paper's scope. Since HTSS can save at most as much as the output layer costs, its benefit diminishes as the backbone grows relative to the vocabulary size. On our 304M backbone, HTSS is behind CE at matched compute at 32K and 64K and ahead from 128K (Table 2). HTSS targets compact backbones with large vocabularies. The output layer takes 28.9% of a step in Qwen3.5-0.8B and 17.5% in Llama 3.2 1B, and HTSS-mc beat CE in fine-tuning on both. Because only active parameters count, MoEs such as Qwen3.5-35B-A3B (15.5%) and gpt-oss-20b (14.4%) reach shares close to Llama's, and our revised future work (§5) includes them.