- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies a prequential pretraining method for language models aimed at predicting compressed bit lengths for chronologically ordered text streams. It constructs a specific large-scale timestamped corpus spanning 2010–2022, with subsets of up to 10B tokens drawn from various web sources. The authors design experiments based on two autoregressive models—an approximately 267M-parameter transformer and a hybrid Gated DeltaNet—and investigate the effects of various training techniques, including temporal ordering, replay, and exact-match retrieval, with results supporting their benefits. Furthermore, numerous quantitative experimental analyses investigating different revision formats and optimizer settings offer detailed insights into the mechanisms of the pretraining process. In short, this work contributes a comprehensive empirical study of temporal memory mechanisms for existing autoregressive language models in the specific scenario of prequential coding.
Critical strengths
Clear problem formulation. The paper frames prequential coding as an autoregressive next-token prediction problem, which is compatible with modern language model architectures and training methods.
Rational experiment design. The 10B-token dataset highlights the challenge of long-context modeling. Ablation studies across different training strategies and hyperparameter settings support the soundness of the proposed method.
Informative analyses of data and limitations. Analyses of staleness and revision formats reveal useful characteristics of the studied text data and expose limitations of existing methods.
Critical weaknesses
Unclear writing: The introduction lacks a concise, accessible definition of prequential coding and its distinction from standard LM pretraining, making it difficult for readers to immediately understand the connection between language modeling and the research problem. The method section would benefit from a schematic for shuffle variants and masking rules, as the implementation details are mostly conveyed through text and are costly to understand.
Ambiguity between theoretical formulation and implementation (need for clarification on how past information enters prediction): The training illustration (Fig. 1) and the cumulative probability formulation (Eq. (2)) suggest that all past tokens are used to predict the bits for the next token, but the implementation appears to depend only on the tokens in the current input window, which may appear to conflict with this logic; see Question 1 below for details.
Questions for authors
-
Specific process at window boundaries: I am confused about the implementation details when processing a current window . The autoregressive formulation in Eq. (2) implies that all tokens in the past windows participate in the forward computation for predicting the first token of . However, the statement “No state is carried across windows” (line 721, Appendix A.1) suggests that previous tokens do not participate in the current forward computation. This also introduces a distinction in how the first token of each window is processed: is it input-only and not scored, or is it predicted? Please clarify this in the paper. Specifically, what is actually used for prediction for , and how is the theoretical chain rule for cumulative probabilities ensured?
-
Order of replayed windows: Based on the performance difference caused by shuffled versus temporal order, how are the sampled past windows ordered during replay? Are the replayed windows fed independently and their gradients averaged, or are they concatenated? Furthermore, considering that the order between windows may affect the causal dependencies that the model learns, does randomly shuffling their order weaken coding performance?
2 comments
Thanks for your responses
Thanks for your clarifications! I have read through the responses and I tend to believe that my previous rating was appropriate.
Thank you for your positive review of our work! We respond to the remaining concerns below.
Thank you for highlighting this, we agree! Please see our shared response for how we plan on improving the writing and motivation.
We will add a clearer visualization that explain our shuffle variants, and we will make the visualization of the masking rule clearer.
Thank you for pointing this out; we should have explained these things more clearly!
Our predictions use earlier tokens of the current window as context, and previous windows through the model’s weights. Eq. (2) allows us, but does not require us, to use all past tokens in context. Fig 2c shows that increasing context length beyond 32k gives little improvements for its cost. For this reason, we split our stream into 32k token windows, with wt[j] being predicted from earlier tokens of wt only. After we score all tokens in wt, we do an optimizer step on it. So when we predict wt, we have already trained on w0,⋯,wt−1.
Some of our runs use previous windows more directly. In the retrieval runs, the retrieval cache is built from tokens in all previous windows, and the model can look up all these. The state in our infinite-context GDN runs depends on all previous tokens.
We score all tokens of our stream except the first. In most runs, our window is 32,769 tokens, and we predict the last 32,768 tokens of each window. We use stride 32,768, meaning the last token of one window is the first token of the next window. This means that we do not need to score the first token of each window, since it was scored in the previous window. The first token of the stream is always BOS, so we don’t need to score it.
The “state” in the appendix refers to the GDN state. We carry this state across windows only for the infinite-context GDN runs. The transformer does not have a state. We will make this clearer in the appendix.
We feed the replayed windows independently, rather than concatenating. The k sampled windows are 32k-token sequences . We process them as a batch and average their gradients. This means that their order does not matter. We only replay windows we have already scored. In Eq. (2), our model fk can depend on all past tokens, which means replay is valid in the prequential setting.
Thank you again for your review. Does the above address your concerns? Please let us know if you have any further questions!