- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposed Token Markov Graph (TMG), a local co-occurrence statistics counted from the pretraining corpus, which can be used to accelerate early language-model training through two ways: adding a KL loss to the training loss, mixing the hidden state with weighted combination of successor embeddings.
The authors evaluate their methods on transformers (Gemma2 architecture) ranging from 172M to 3B parameters. Their graphs are constructed from 60B and 350B tokens with sequence length 512 and weight decay 0.1. They train the transformers on 100M tokens with a single epoch.
Critical strengths
- Using counted statistics rather than a trained neural teacher is elegant.
- It's good to show experiment results with multiple seeds and check statistical significance.
Critical weaknesses
- This paper is a bit outdated. Every reference is from before 2025 except one citation (the engram paper), even though data-efficient pretraining is a topic with many relevant work from 2025 and 2026.
- Data-constrained pretraining Literatures, including Muennighoff et. al. usually train an overparameterized model on a fixed data budget for multiple epochs. This paper only trains on 100M unique tokens for one epoch, which is an unusual setup. 100M is a very small budget for single-epoch pretraining: In LLM pretraining, it is common that some method outperforms at first but then the gain diminishes as training continues.
Questions for authors
- Why you only train the model on 100M tokens for a single epoch?
- Why you didn't disclose how the 100M tokens were sampled? Is it a subset of DCLM or C4?
- Table 2 shows the gain increases with model size. Do the authors have any understanding on this?
- The model architecture, Gemma2, is an old architecture. The sequence length 512 is too short for pretraining. Why you chose Gemma2 and sequence length 512?
- The calculation of Chinchilla-optimal budget seems incorrect. The authors used 20 * # of non-embedding parameters. Why not use 20 * # of (non-embedding parameters + embedding parameters)?