acceptodds
Under review as a conference paper at ICLR 2027

Prequential Pretraining: Learning to Continually Compress the Future

Abstract

A major approach within computational theory of intelligence is prequential coding, which views continual compression of a stream of data as identifying parsimonious models for the underlying data. Prequential coding has been an important motivation for pretraining, leading to language models that can accurately complete single documents. Despite these successes, there has been little work on studying prequential coding on the more general setting of temporal streams of text data. Success in this setting would enable models to make predictions of future texts, across documents and potentially serve as an important primitive for continual learning. We study the empirical feasibility of this problem and curate a high-resolution, verifiably timestamped corpus with a single temporal stream of text up to 10B tokens spanning 2010-2022, including revisions to the same documents over time. We evaluate standard language modeling approaches as coding algorithms and find significant room for improvement in exploiting temporal and long-range structure. Natural interventions, such as long context, reach only 0.69 bits per byte (BPB), compared to retrieving exact-match prefixes from earlier in the stream which reaches 0.49 BPB, revealing substantial headroom for long-context models. Finally, we identify that in principle, it is possible to attain far better compression rates of 0.43 BPB through masking tokens that are in principle knowable from the past.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language modeling, prequential coding, continual learning, long-context modeling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_JvtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a prequential pretraining method for language models aimed at predicting compressed bit lengths for chronologically ordered text streams. It constructs a specific large-scale timestamped corpus spanning 2010–2022, with subsets of up to 10B tokens drawn from various web sources. The authors design experiments based on two autoregressive models—an approximately 267M-parameter transformer and a hybrid Gated DeltaNet—and investigate the effects of various training techniques, including temporal ordering, replay, and exact-match retrieval, with results supporting their benefits. Furthermore, numerous quantitative experimental analyses investigating different revision formats and optimizer settings offer detailed insights into the mechanisms of the pretraining process. In short, this work contributes a comprehensive empirical study of temporal memory mechanisms for existing autoregressive language models in the specific scenario of prequential coding.

Critical strengths

Clear problem formulation. The paper frames prequential coding as an autoregressive next-token prediction problem, which is compatible with modern language model architectures and training methods.

Rational experiment design. The 10B-token dataset highlights the challenge of long-context modeling. Ablation studies across different training strategies and hyperparameter settings support the soundness of the proposed method.

Informative analyses of data and limitations. Analyses of staleness and revision formats reveal useful characteristics of the studied text data and expose limitations of existing methods.

Critical weaknesses

Unclear writing: The introduction lacks a concise, accessible definition of prequential coding and its distinction from standard LM pretraining, making it difficult for readers to immediately understand the connection between language modeling and the research problem. The method section would benefit from a schematic for shuffle variants and masking rules, as the implementation details are mostly conveyed through text and are costly to understand.

Ambiguity between theoretical formulation and implementation (need for clarification on how past information enters prediction): The training illustration (Fig. 1) and the cumulative probability formulation (Eq. (2)) suggest that all past tokens are used to predict the bits for the next token, but the implementation appears to depend only on the tokens in the current input window, which may appear to conflict with this logic; see Question 1 below for details.

Questions for authors
  1. Specific process at window boundaries: I am confused about the implementation details when processing a current window wtw_t. The autoregressive formulation in Eq. (2) implies that all tokens in the past windows w0,…,wt−1w_0,\dots,w_{t-1} participate in the forward computation for predicting the first token of wt[0]w_t[0]. However, the statement “No state is carried across windows” (line 721, Appendix A.1) suggests that previous tokens do not participate in the current forward computation. This also introduces a distinction in how the first token of each window is processed: is it input-only and not scored, or is it predicted? Please clarify this in the paper. Specifically, what is actually used for prediction P(wt[j]∣⋅)P(w_t[j] \mid \cdot) for j≥0j \ge 0, and how is the theoretical chain rule for cumulative probabilities Pft−1(wt[j]∣w0,…,wt−1)P_{f_{t-1}}(w_t[j] \mid w_0,\dots,w_{t-1}) ensured?

  2. Order of replayed windows: Based on the performance difference caused by shuffled versus temporal order, how are the sampled kk past windows ordered during replay? Are the replayed windows fed independently and their gradients averaged, or are they concatenated? Furthermore, considering that the order between windows may affect the causal dependencies that the model learns, does randomly shuffling their order weaken coding performance?

2 comments

Thank you for your positive review of our work! We respond to the remaining concerns below.


Unclear writing: The introduction lacks a concise, accessible definition of prequential coding and its distinction from standard LM pretraining, making it difficult for readers to immediately understand the connection between language modeling and the research problem. The method section would benefit from a schematic for shuffle variants and masking rules, as the implementation details are mostly conveyed through text and are costly to understand.

Thank you for highlighting this, we agree! Please see our shared response for how we plan on improving the writing and motivation.

We will add a clearer visualization that explain our shuffle variants, and we will make the visualization of the masking rule clearer.

Ambiguity between theoretical formulation and implementation (need for clarification on how past information enters prediction): The training illustration (Fig. 1) and the cumulative probability formulation (Eq. (2)) suggest that all past tokens are used to predict the bits for the next token, but the implementation appears to depend only on the tokens in the current input window, which may appear to conflict with this logic; see Question 1 below for details.

Specific process at window boundaries: I am confused about the implementation details when processing a current window wtw_t. The autoregressive formulation in Eq. (2) implies that all tokens in the past windows w0,…,wt−1w_0, \ldots, w_{t-1} participate in the forward computation for predicting the first token of wt[0]w_t[0]. However, the statement “No state is carried across windows” (line 721, Appendix A.1) suggests that previous tokens do not participate in the current forward computation. This also introduces a distinction in how the first token of each window is processed: is it input-only and not scored, or is it predicted? Please clarify this in the paper. Specifically, what is actually used for prediction P(wt[j]∣⋅)P(w_t[j] \mid \cdot) for j≥0j \geq 0, and how is the theoretical chain rule for cumulative probabilities Pft−1(wt[j]∣w0,…,wt−1)P_{f_{t-1}}(w_t[j] \mid w_0, \ldots, w_{t-1}) ensured?

Thank you for pointing this out; we should have explained these things more clearly!

Our predictions use earlier tokens of the current window as context, and previous windows through the model’s weights. Eq. (2) allows us, but does not require us, to use all past tokens in context. Fig 2c shows that increasing context length beyond 32k gives little improvements for its cost. For this reason, we split our stream into 32k token windows, with wt[j]w_t[j] being predicted from earlier tokens of wtw_t only. After we score all tokens in wtw_t, we do an optimizer step on it. So when we predict wtw_t, we have already trained on w0,⋯ ,wt−1w_0,\cdots,w_{t-1}.

Some of our runs use previous windows more directly. In the retrieval runs, the retrieval cache is built from tokens in all previous windows, and the model can look up all these. The state in our infinite-context GDN runs depends on all previous tokens.

We score all tokens of our stream except the first. In most runs, our window is 32,769 tokens, and we predict the last 32,768 tokens of each window. We use stride 32,768, meaning the last token of one window is the first token of the next window. This means that we do not need to score the first token of each window, since it was scored in the previous window. The first token of the stream is always BOS, so we don’t need to score it.

The “state” in the appendix refers to the GDN state. We carry this state across windows only for the infinite-context GDN runs. The transformer does not have a state. We will make this clearer in the appendix.

Order of replayed windows: Based on the performance difference caused by shuffled versus temporal order, how are the sampled kk past windows ordered during replay? Are the replayed windows fed independently and their gradients averaged, or are they concatenated? Furthermore, considering that the order between windows may affect the causal dependencies that the model learns, does randomly shuffling their order weaken coding performance?

We feed the replayed windows independently, rather than concatenating. The kk sampled windows are 32k-token sequences . We process them as a batch and average their gradients. This means that their order does not matter. We only replay windows we have already scored. In Eq. (2), our model fkf_k can depend on all past tokens, which means replay is valid in the prequential setting.


Thank you again for your review. Does the above address your concerns? Please let us know if you have any further questions!

Reviewer_JvtVOpenReview ↗

Thanks for your responses

Thanks for your clarifications! I have read through the responses and I tend to believe that my previous rating was appropriate.

Reviewer_nwT9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames pretraining as prequential coding of a single, temporally ordered text stream. Every 32k window is scored by the weights from before the update on it and then trained on, so the cumulative log loss is a code length. The authors build a corpus with second-level timestamps from Wikipedia full revision history, Stack Exchange post histories, arXiv abstract versions and Global Voices, covering 2010 to 2022, and render revisions as unified diffs against the previous version. Nested hash-sampled streams run from 100M to 10B tokens. A 267M Qwen3-style transformer, warm-started from 5B tokens of pre-2010 text, is trained online with batch size 1. The main findings: temporal order beats several shuffled orderings, staleness from batching erodes the temporal advantage, longer context helps little past 32k, carrying Gated DeltaNet state across windows gains at most 0.3%, replay of uniformly sampled past windows and an exact-match retrieval cache mixed in through a learned gate cut 1B-stream BPB from 0.697 to 0.487, and the representation of revisions matters a lot. A compact edit-script format beats unified diffs by 31% at 1B and 38% at 10B, and masking tokens that a rule-based copier can infer from the past gives a further 12.1% for the best model. Propositions 1 and 2 argue the compact diff is invertible and that the masked loss is itself an achievable prequential code.

Critical strengths
  1. The corpus and the accounting are done right. Timestamps at 1 s or 1 ms resolution, an ordering whose tie-breaking ignores content, ID-hash sampling so that each stream is a subset of the next one, and an explicit list of where retrospective selection leaks the future, on p. 17 lines 897 to 917. A footnote on p. 3 flags that some filters use later information, and Appendix A.2.5 states plainly that absolute losses are probably optimistic because reverted and deleted content is missing. Charged versus shared side information is spelled out on p. 18 lines 966 to 971, including the log2 K cost of selecting hyperparameters on the reported metric. I recomputed the parameter count from the stated architecture and got 267949056, matching line 709, and the stream tables add up. A dataset that anyone can treat as a fixed-length lossless coding task with an exact bit budget is a reusable benchmark for continual learning and long-context work, and that alone carries a lot of the paper.

  2. The representation results are the most interesting science here. The same underlying record sequence costs 1564, 1048 or 722 bits per document at 1B depending only on whether revisions are written as full text, unified diff or compact diff, on p. 26 lines 1377 to 1381. Rule-based masking of copyable tokens is shown by Proposition 2 to be a real code, not an oracle bound. Given its premise that κ and ϕ are computable from the prefix, the proof is correct: q is a valid distribution at every position, masked positions cost 0 bits because κ = 1 forces x_j = ϕ, and the decoder can replay training. This turns the vague claim that the model makes poor use of the past into a measured gap: 1047.6 to 699.6 bits per document for the plain transformer and 732.0 to 643.2 for the best model, both of which I recomputed from Table 6.

Critical weaknesses
  1. The headline value of time mixes order with format. The introduction says temporal information "decreases prequential loss by 60% relative to shuffling documents" on p. 2 line 75, and Figure 1 shows a "2.5× smaller" bar going from 0.62 GB to 0.25 GB. I reconstructed those numbers: 2587 and 1046 bits per article times 1924357 records give 0.62 GB and 0.25 GB, and 1 minus 1046 over 2587 is 59.6%. So the headline is the shuffled plus rediff arm. Section 4.2 is open about this arm, saying it suffers "extreme degradation in performance due to the large sizes of the resulting diffs" on p. 5 lines 267 to 268, and Appendix A.7 notes the 1B stream grows to 6.3B tokens, p. 23 line 1230. The same section also reports the cleaner numbers: 18% for shuffled documents with fixed diffs and 2.9% for shuffled windows, p. 5 line 269 to p. 6 line 272, detailed as 18.3% and 2.9% on p. 23 lines 1235 to 1237. The problem is that the introduction and Figure 1 pick the arm most inflated by representation. Rediffing under shuffled order is a valid code, and part of its cost is a real order effect, but Section 6 shows that format alone moves loss by over 2x, so the 60% cannot be read as the value of time. Even the 18.3% arm carries some representation cost, since its diffs refer to predecessors that may appear later, p. 22 lines 1186 to 1187. Since quantifying "the value of time" in Section 2.1 is the main conceptual selling point, the headline should be the same-format number, or a comparison that uses the best format available for each order, for example compact diffs against the latest seen version, with 2.9% to 18% reported as the range.

  2. The negative long-context and headroom conclusions are not well supported beyond one small configuration. Everything runs at a single scale, the 267M transformer and the 281M hybrid. For 64k and 128k the transformer extends RoPE with YaRN from a 32k warm start, p. 14 line 723, and unlike the hybrid it gets no warm-start updates at the new length. Windows do not overlap, so tokens near a window start get little context. Batch size is one window, so a 128k run also takes 4 times fewer optimizer steps than a 32k run on the same stream, about 2400 against 9600 at 316M, and the paper's own staleness result shows that update frequency matters in this setting. The 316M curve in Figure 2c gets worse at 64k and 128k, which fits fewer updates or a weak extension at least as well as a property of the data. The hybrid model at 32k already reaches 0.6485 BPB and drops 1.1% more at 64k, p. 27 lines 1425 to 1426. That is better than the transformer's 0.697 and not small next to the abstract's "reach only 0.69 bits per byte" on p. 1 line 24. Replay, retrieval and diff-format results use one seed, p. 15 lines 764 to 765, and replay with k equal to 3 processes 4 windows per update, p. 25 line 1312, with no compute-matched baseline such as a larger model or more steps per window. As a result, the claim that standard recipes "underutilize temporal structure" can't be separated from capacity, context-extension and compute effects. The exact-match cache result also has close precedents in continuous and unbounded cache LMs for online language modeling, Grave et al. 2017, and in NNCP-style online transformer compression of enwik9, a Wikipedia dump. Neither is discussed in Related Work on p. 9.

Questions for authors
  1. What is the temporal versus shuffled gap when both orders use a format that only references already-seen text and is not inflated by order? For example, compact diffs against the latest seen version in both cases. If that gap is in the 3% to 18% range, will Figure 1 and line 75 be revised?
  2. Did the 64k and 128k transformer runs get any long-context warm-start training, as the hybrid did with 256 updates? Does the 32k plateau survive with overlapping or sliding-window scoring, or with a sweep that holds the number of optimizer updates fixed across context lengths?
  3. Which stream does Figure 4a use? Its k equal to 0 point sits near 0.705, which matches the 316M baseline rather than the 0.697 of 1B, and its replay all point at k equal to 3 sits near 0.641, while Figure 4b reports 0.635 on 1B.
  4. Do the 32k plateau and the 12% masking gap persist at 1B or more parameters? A compute-matched comparison of replay against a larger model would tell readers whether the headroom is about memory mechanisms or just capacity.
  5. The paper sells the setting as a continual-learning primitive, yet the WSD schedule needs the horizon. At each schedule's best peak, the best horizon-free schedule, constant, trails cosine and WSD by about 0.01 BPB at 100M, 0.777 against 0.767 and 0.768 per p. 21 lines 1123 to 1125. WDS ties cosine only because its single decay is tuned to that horizon. How sensitive are the main conclusions to using constant throughout?
  6. Proposition 2 requires κ to be computable from the token prefix, but Appendix A.3.4 labels characters and masks a token only if every byte it covers is masked. Before decoding a token, the decoder does not know which bytes it covers. How is the token-level decision made from the prefix alone, for example at the end of a copied region, where the tokenization of the last chunk can depend on bytes not yet decoded? The byte-for-byte re-tokenization check on p. 18 establishes that tokenization is invertible, not this property.
4 comments

Thank you for the long review. We have tried our best to answer all questions, but at times it was hard to understand what you were asking for. We’ve tried answering all the questions to the best of our abilities, but let us know if we missed anything!


What is the temporal versus shuffled gap when both orders use a format that only references already-seen text and is not inflated by order? For example, compact diffs against the latest seen version in both cases. If that gap is in the 3% to 18% range, will Figure 1 and line 75 be revised?

For prequential coding, the sum of log losses is the only allowable measurement. We report total bits in Figure 1, which is exactly this up to some constants. This means we can directly compare the losses even if the original streams contain a different number of tokens. See Section 2.1 “Measuring the ability to use temporal information” for further justification (representation invariance/chain-rule paragraph).

That said, for compact diffs, temporal order saves 51% for the 100M corpus (855 vs 1735 bits/doc). This can be partially explained by the shuffled order having larger diffs (5.3x). With full articles (identical tokens) and only order changes, temporal still saves 20% (1406 vs 1762). The 20% reduction undersells the difference, since the temporal order can be represented more compactly with diffs, and Figure 1 is accurate as an upper bound on description length for each approach.

Did the 64k and 128k transformer runs get any long-context warm-start training, as the hybrid did with 256 updates? Does the 32k plateau survive with overlapping or sliding-window scoring, or with a sweep that holds the number of optimizer updates fixed across context lengths?

When adding 256 warm-start updates at the new lengths, the YaRN results barely change (+0.1% at 64k). If we hold the targets and the approximately 10k updates fixed and add unscored preceding context, we see improvements of 5.2% and 7.4% for 64k and 128k respectively at 316M. These roughly match the 5.3% and 8.1% improvements we achieve if we simply do replay on the same tokens with the 32k context model. This indicates that at a fixed amount of compute, longer context does not gain more than replay at shorter context length.

We will update the paper to reflect that while we see gains from longer context length, these are smaller than simply doing more compute at lower context length.

Which stream does Figure 4a use? Its k equal to 0 point sits near 0.705, which matches the 316M baseline rather than the 0.697 of 1B, and its replay all point at k equal to 3 sits near 0.641, while Figure 4b reports 0.635 on 1B.

Correct, Figure 4a uses the 316M stream.

Do the 32k plateau and the 12% masking gap persist at 1B or more parameters? A compute-matched comparison of replay against a larger model would tell readers whether the headroom is about memory mechanisms or just capacity.

If we roughly match FLOPs, replay for a 267M model beats a 940M model, reaching 0.820 BPB, compared to 0.849 BPB for the larger model.

The paper sells the setting as a continual-learning primitive, yet the WSD schedule needs the horizon. At each schedule's best peak, the best horizon-free schedule, constant, trails cosine and WSD by about 0.01 BPB at 100M, 0.777 against 0.767 and 0.768 per p. 21 lines 1123 to 1125. WDS ties cosine only because its single decay is tuned to that horizon. How sensitive are the main conclusions to using constant throughout?

A constant schedule costs around 1% more bits, and all gaps keep their signs and mostly their size. We report each variants code length relative to the temporal baseline with the same schedule. Negative means shorter code length.

VariantWSDConstant LR
Shuffle documents+17.1%+19.2%
Compact-23.7%-23.6%
Replay all-7.6%-7.6%
Gate-20.3%-20.3%

Proposition 2 requires κ to be computable from the token prefix, but Appendix A.3.4 labels characters and masks a token only if every byte it covers is masked. Before decoding a token, the decoder does not know which bytes it covers. How is the token-level decision made from the prefix alone, for example at the end of a copied region, where the tokenization of the last chunk can depend on bytes not yet decoded?

Thank you for catching this! Fixing this small bug brings the cost from 832.2 bits/doc to 834.7 bits/doc (+0.3%). The conclusion remains unchanged, but we will update our code and our paper with the correct numbers

We will discuss the related works you mentioned. A tuned Grave et al. continuous cache reduced code length by 1.8%, against 20.3% for our exact-match gate, if the cost of choosing its settings is charged.


Thank you again for your review. Does the above address your concerns? Please let us know if you have any further questions!

Thanks for the detailed reply and the new runs.

Two things in your reply make me less sure about the original results. First, when you keep the number of training steps the same, longer context does help: 5.2% at 64k and 7.4% at 128k on the 316M stream. So the abstract’s claim that long context helps little was partly caused by the 128k runs getting about 4 times fewer training steps (about 9644 vs 2411 windows at 316M). Second, Proposition 2 says the masked loss is a valid code, but the code did not follow the condition the proposition needs. The fix changes the number only a little (832.2 to 834.7 bits/doc, +0.3%), but the claim and the code did not match.

Some points are still open. The 60% on p. 2, line 75 (1 − 1046/2587) uses the shuffled setup that the paper itself says does badly because its diffs get very large. With the same tokens in both orders, your own numbers give 20% (1406 vs 1762), and the paper reports 18.3% and 2.9% for cleaner setups. So the 60% mixes the effect of order with the effect of the file format, and the paper should say how much each one contributes. For the 0.820 vs 0.849 comparison, you don’t say which stream you used or how you matched compute. Replay, retrieval and format results still use one seed.

First, when you keep the number of training steps the same, longer context does help: 5.2% at 64k and 7.4% at 128k on the 316M stream. So the abstract’s claim that long context helps little was partly caused by the 128k runs getting about 4 times fewer training steps (about 9644 vs 2411 windows at 316M).

A model with 64k context length and the same number of optimizer steps will be trained using more FLOPS. A fair comparison is therefore to compare that 64k context length model with a 32k context length model that does replay.

RunUpdatesContextCompute vs 32kbpb
32k962732k1x0.7044
32k (replay)962732k3.1x0.6506
64k962764k3.4x0.6678
128k9627128k12.3x0.6525

The table shows that a 32k model with replay beats both the 64k and 128k models while using less compute. This means that longer context length helps little compared to the alternative, which is spending the same FLOPS on replay.

Second, Proposition 2 says the masked loss is a valid code, but the code did not follow the condition the proposition needs. The fix changes the number only a little (832.2 to 834.7 bits/doc, +0.3%), but the claim and the code did not match.

All numbers in the paper have been updated, and we have attached a Lean proof for Proposition 2 as supplementary material. Numbers changed only a little, and we found no meaningful differences in the paper’s takeaways.

The 60% on p. 2, line 75 (1 − 1046/2587) uses the shuffled setup that the paper itself says does badly because its diffs get very large. With the same tokens in both orders, your own numbers give 20% (1406 vs 1762), and the paper reports 18.3% and 2.9% for cleaner setups. So the 60% mixes the effect of order with the effect of the file format, and the paper should say how much each one contributes.

We explain in our previous response and in the paper why both are valid code lengths, and why we believe they can be compared fairly, but we have added a plot to the appendix, where we break down the code length for temporal and shuffled document order, both with full article text and with diffs. See Figure 12.

Comparing 267M model vs 940M model

While the 940M model beats the 267M model when trained for the same number of steps, reaching 0.696 BPB compared to 0.729 BPB, this does not hold if we match compute. With replay for the 267M model, we reach 0.673 BPB while using 42% as many FLOPs as the 940M model. We have added this to the appendix under the “Replay against a larger model” section.

Replay, retrieval and format results still use one seed.

We have added more seeds in the updated version of the paper. The results did not meaningfully change. Given the small standard deviations, we did not run some of our more costly experiments with multiple seeds. See the table for how many seeds we used for the different experiments.

Thanks for this follow-up. Have you updated your submission?

Reviewer_cwR5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work focuses on the prequential coding setting, where a model sufficiently trained on a temporally-ordered data stream must predict future entries.

The authors gather a corpus of 10 billion tokens from online sources from 2010-2022 including revisions to said sources into a single temporally ordered string.

Small models trained with batch size 1 are trained on subsamples of this prequential corpus after initial "pre-pre-training" on 5 billion shuffled (not temporally ordered) tokens from pre-2010.

It is claimed that replay (retraining on past data) and retrieval (mixing cached previously seen subsequences into the context) improve bits/byte performance in prequential pre-training on a 1B stream of tokens.

Finally, a compact diff format and a masked loss are studied to improve upon the standard diff format, in an attempt to optimize the data format to get "the best possible learning rates".

Critical strengths

None

Critical weaknesses

This paper is very hard to read and is unclear in its motivations and extent of contributions.

Questions for authors
  • Please share more details on why you constructed this corpus and how it is related to the prequential setting. Was the corpus constructed to train models to predict text revisions of articles?
  • How is the online sampling scheme different from performing random sampling and keeping the data in temporal order?
  • Since retrieval uses the identical cached context, isn't the benefit here due to the future prediction being copied from when it was previously seen? So the model doesn't know the future unless it has specifically seen, cached, then retrieved the future on-demand? Similarly, isn't the longer context also doing something similar?
  • Is there a difference in the token counts between the unified and compact diff, and if so, does this affect the number of updates to the model at the same number of stream tokens? If so, then this calls into question the improvements in Figure 5c.
1 comment

Thank you for your review! We respond to the concerns below.


Please share more details on why you constructed this corpus and how it is related to the prequential setting. Was the corpus constructed to train models to predict text revisions of articles?

Please see our shared reply.

How is the online sampling scheme different from performing random sampling and keeping the data in temporal order?

If we want to train on 10% of our stream, we need to know a priori if we should include each document we receive. If we pick this 10% based on the full set of articles in history after having seen all articles, the sampling leaks future information into the dataset.

To avoid future leakage, we need a fully causal random subsampling scheme. The online sampling scheme is a way of doing this that is both random and fully causal.

Since retrieval uses the identical cached context, isn't the benefit here due to the future prediction being copied from when it was previously seen? So the model doesn't know the future unless it has specifically seen, cached, then retrieved the future on-demand? Similarly, isn't the longer context also doing something similar?

The retrieval can only look up tokens from the past (respecting Eq. 2), so we are not looking up the future. Similarly, we do causal attention, meaning that increasing the context length does not leak the future. While it is true that retrieval and long-context can yield gains when there are shared strings from the past that appear again, this is fair game and many models, like n-gram models operate on this basis.

Even if you think that retrieval is not useful for language modeling, the main goal of the retrieval result is to identify headroom, similar to our masked token loss. The fact that these simple interventions do so well, indicates that there is still substantial room for improvement, and the point of the headroom analysis is to encourage more work on this task.

Is there a difference in the token counts between the unified and compact diff, and if so, does this affect the number of updates to the model at the same number of stream tokens? If so, then this calls into question the improvements in Figure 5c.

We report bits/doc or total bits of the code length, meaning we can directly compare the losses even if the original streams contain different number of tokens. See Section 2.1 “Measuring the ability to use temporal information” for further justification (representation invariance/chain-rule paragraph).

If we do the same number of updates for the compact and unified diffs, the unified diff with masked tokens still does better. We reran the compact diff 100M stream with the same number of optimizer steps as the unified, which reduced loss from 855.5 to 844.0 bits/doc. This is still higher than the unified diff with known tokens masked, which used 834.7 bits/doc.


Does the above address your concerns? Please let us know if you have any further questions!

Reviewer_a1GpOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies prequential pretraining, where language models continually learn from a temporally ordered text stream and are evaluated on their ability to predict future data. The authors construct a timestamped corpus of up to 10B tokens from 2010-2022 Stack Exchange, arXiv and Global Voices, and Wiki that includes revisions. The authors empirically study next-token prediction as a standard language modeling task. Experiments investigate the effects of temporal ordering, context length, batch size, and model architecture, showing that temporal training improves compression but struggles with long-range dependencies. The authors further explore replay, and retrieval to improve compression efficiency, and analyze the remaining headroom through compact document representations and masking of reconstructible tokens.

Critical strengths
  • Clear research question. Sound setup, approach, and experiments
  • The construction of a carefully timestamped temporal corpus
  • Systematic study of how conventional pretraining recipes behave in an online setting, and the use of compression to quantify the limitations of long-range memory
Critical weaknesses
  • The motivation for prequential coding could be strengthened. Why should a broader audience care more about the formulation of prequential coding and its applications? I would imagine such long-term temporal modeling might be helpful for code repos, system logs, predicting news or societal trends, etc.

  • Also about the motivation part, the connection to continual learning is mentioned in the introduction but is not clear in the experiments

Questions for authors
  • The evidence of improvements on prequential coding due to long-term structure modeling is clear, but is there any analysis of negative effects? For instance, I wonder if the BPB improvements are uniform or actually mostly on repetitive events like code diff modification and might regress novel content.

  • I wonder if we using the models trained in the paper to predict the content after 2022, how well they predict the "future"? Is BPB really a good measurement of future predcition in the context of prequential coding?

1 comment

Thank you for your positive review of our work! We respond to the remaining concerns below.


The motivation for prequential coding could be strengthened. Why should a broader audience care more about the formulation of prequential coding and its applications? I would imagine such long-term temporal modeling might be helpful for code repos, system logs, predicting news or societal trends, etc.

Please see our shared response.

Also about the motivation part, the connection to continual learning is mentioned in the introduction but is not clear in the experiments

Continual learning is the eventual goal, which is why we mention it in the intro but we do not have direct experiments. We will make that clearer in the writing. However, we think our experimental settings and many of our experiments will continue to be useful to people studying continual learning, as our setting is the pretraining version of continual learning, and many of our techniques like horizon-free LRs are necessary components.

The evidence of improvements on prequential coding due to long-term structure modeling is clear, but is there any analysis of negative effects? For instance, I wonder if the BPB improvements are uniform or actually mostly on repetitive events like code diff modification and might regress novel content.

While gains are larger for repetitive content, we also see improvements for novel content. The gains hold across all four data sources and mostly increase as the stream size increases. The table shows reduction in bits on novel content from temporal ordering, compared to the document shuffled stream (positive means temporal is better).

SizeStack ExchangearXivWikipediaGlobal Voices (news)All sources
100M9.1%4.8%6.8%3.0%8.3%
316M11.6%7.1%10.2%7.3%10.8%
1B14.7%10.0%4.8%6.4%13.8%

Temporal gains on repetitive content are larger, at 57.8%, 63.4% and 73.8% for 100M, 316M and 1B.

We select novel content by documents whose first revision appears before any of its revisions in the shuffled stream and further restrict to hold keeping documents with position in the temporal and shuffled streams within 5% of each other. In the table above, we compare total bits for this subset.

I wonder if we using the models trained in the paper to predict the content after 2022, how well they predict the "future"? Is BPB really a good measurement of future prediction in the context of prequential coding?

For prequential coding, the sum of log losses is the only allowable measurement. BPB is this measurement up to constants and document format differences. When the document formats differ, we report bits/doc or total bits.

We believe that studying prequential coding is an important part of measuring how well models predict the future, similar to how pretraining is important for general capabilities. For future work, we want to scale up and test emergence of downstream future capability prediction, but we view this as a second step after establishing the dataset, setting and basic methods.


Thank you again for your review. Does the above address your concerns? Please let us know if you have any further questions!

Comments on the paper

1 comment

Thank you to the reviewers for their time. We are glad that they appreciated the prequential setting. We will make several writing changes.

Motivation

In the introduction, between the first and second paragraphs, we will make a more explicit case for prequential coding via this paragraph:

In this work, we build upon the long-running line of research that considers compression of data and prequential coding as a goal in and of itself (Dawid, 1984; Rissanen, 1984; Schmidhuber & Heil, 1996; Bellard, 2021). We take this perspective, as prequential coding is a core, formal justification for LM pretraining. Indeed, works have argued that prequential coding is the reason why compression leads to intelligence, due to its connection to Solomonoff induction (Solomonoff, 1964; Mahoney, 1999; Hutter, 2005; Legg & Hutter, 2007; Grau-Moya et al., 2024) as a computable upper bound to the minimum description length (Rissanen, 1978; Grünwald, 2007; Blier & Ollivier, 2018; Elmoznino et al., 2025). Taking this formal view seriously has led to several fruitful lines of work in AI, including language model pretraining (Radford et al., 2019; Brown et al., 2020; Delétang et al., 2024), tight compression bounds in neural networks (Blier & Ollivier, 2018; Bornschein et al., 2023) and new views of entropy and complexity (Bialek et al., 2001; Voita & Titov, 2020; Finzi et al., 2026). Because of these successes, our work focuses on new forms of pretraining that are natural under prequential coding.

Connection between prequential coding and MDL

In section 2, we will provide an explicit connection between prequential coding and minimum description length, following section 3.4 of Blier and Ollivier 2018, which has a clear exposition of how prequential coding connects to description length and Solomonoff induction.

Related work

Finally, we will move the related work section immediately after the introduction, and highlight the role of prequential coding (and our work) in the context of many existing lines of work, such as prequential coding, minimum description length and online learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.