- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Paper suggests a novel way to introduce data statistics information to a language model. These parameters are configured via ngram counts on language data and are not learned during training. Although the resulting table is very large in terms of memory, it can reside in host memory which is considerably cheaper than the gpu memory used for training. The authors show how the use of these ngram counts to weight model's own unembedding matrix leads to gains over equivalent compute with no added learned parameters and adaptability to other domains when these tables are modified post training. The authors also show this table can be shrinked in size with graceful degradation in accuracy.
Critical strengths
the paper introduces a new way to scale models using cheap host memory with little impact on performance. This scaling is novel and shows ability to adapt to new domains.
Critical weaknesses
- seems like the count table was calculated on much larger set of data than was used for training. To reduce the risk for data contamination, I would like to see an experiment where the used data for ngram count is limited to the same data used for training.
Questions for authors
- Did the authors think of a way to introduce the ngram data to the model in way more complaint with distributed embeddings? For example, decompose the table (similar to log-pmi decompositons) and introduce the added information as continues vectors rather than count-weighted sums of embeddings. This will allow tunable parameters that can be interpolated, quantized, used in other ways similar to neural embeddings.
2 comments
I thank authors for their answers and additional results. The experiment where the n-gram count tables are built from the exact training set directly addresses my main concern about contamination. I would encourage the authors to include this experiment. The additional experiment on count-initialized frozen dense embeddings also answers my question. As authors noted, this alternative improves over the baseline but does not match COUNT. Overall, my assessment remains positive.
Response to Reviewer Efff
Thank you for noting the novelty of scaling language models with n-gram counts stored in host memory.
Does improvement rely on counts from unseen data?
Responding to the concern that improvements required n-gram counts from tokens unseen by the pretraining token budget of our models, we have built count tables from the exact 1.9B, 3.8B, and 7.6B tokens used to train the model at D/N = 10, 20, and 40, respectively. When the count table is only a subset of the pretraining data (i.e. D/N = 20 and 40 for 1.9B tokens and D/N = 40 for 3.8B tokens in the experiments below) performance is similar to our headline results. Interestingly when the count table and pretraining tokens overlap completely, generalization is worse. Practically this is easy to avoid by cheaply counting unused data (in the small model infinite data regime of our experiments) or by fitting count tables on only a subset of the training data (with relatively little performance degradation) (Figure 19).
Count-based dense representations
Good idea! Similar to your suggestion, in early experiments we tried fitting dense representations to empirical next-token distributions and using them to initialize learned n-gram embeddings, but this did not improve on random initialization. Now, we try this again but use the initialization as fixed rather than learned embeddings and compare to COUNT. Specifically, before starting pretraining we fit the 190M 1:1 n-gram embeddings to supervision from the empirical next-token distributions with early stopping on the lowest validation KL divergence. We then froze the embeddings but left the projections in the learned embedding architecture trainable. While this does improve loss beyond the standard transformer, it does not do so as well as COUNT (Figure 24).