acceptodds
Under review as a conference paper at ICLR 2027

Specialists Learn Faster: Knowledge Organization in Pre-Training Shapes Knowledge Update Efficiency

Abstract

An ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates. Models trained on chronologically ordered knowledge specialize in the most recently encountered knowledge; we call them specialists. In contrast, models trained on the same knowledge in randomly shuffled order perform well across all represented time periods; we call them generalists. Surprisingly, specialists require far fewer update steps than generalists to acquire new knowledge. To explain this advantage, we derive a law from first principles that relates update cost to the model's optimization dynamics and closely predicts empirical results. Guided by these findings, we design a strategy for organizing pre-training data that preserves stable historical knowledge while enabling efficient updates to current knowledge. We also show that the benefits of temporal knowledge organization extend to real-world pre-trained models and existing knowledge update tools, and generalize to other knowledge orderings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Language model plasticity, Time-varying knowledge, Continual learning, Language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uBq2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the effectiveness of model updates by comparing generalists, models trained i.i.d., with specialists, models trained on temporally-ordered data. It observes that specialists converge faster during model updates compared to generalists. It proposes a theoretical explanation for this phenomenon and applies generalists in different empirical settings.

Critical strengths
  • Generally, the question of how to optimize for forward transfer is still open and fundamental. If well supported theoretically and empirically (which currently it is not), the argument of the paper would be of high interest
Critical weaknesses
  • limited related work section. There is a lot of work on pre-training, continual finetuning, and continual model editing. It should be appropriately discussed and compared against
  • Big claims with limited supporting evidence
    • the paper claims to be about "pre-training", but it studies factual knowledge, which is a limited form of knowledge. It's unclear whether the claims will generalize to standard pretraining setups
    • the claim about convergence speed is measured via the number of Adam updates. However, this quantity depends on the learning rate. Could it be fixed by using a bigger learning rate for the generalist?
    • the theoretical work is not very well supported. It's mostly a theory to explain the empirically observed phenomena.
  • Limited baselines in the experiments: Fig.6 is the only experiment that compars against baselines from the literature, which is not sufficient.
Questions for authors
  • if the advantage is mostly gradient magnitude, can it be recovered by setting a proper learning rate?
  • more than specialist, they seem to be time-aware vs time-agnostic?
  • "...specialists and generalists, trained under CD, perform equally well on the year for which the specialist is optimized. Yet, a specialist clearly has not learned the year information [...]. Why, then, can a model that has not learned the year information perform as well as one that has?"
    • I'm confused by this statement. The specialist is only correct about the current year and it's going to answer with that knowledge regardless of the request. Isn't this obvious? The model completely ignores year information.
Reviewer_322SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the temporal order of pretraining data affects later knowledge update cost. The authors fix the facts and vary only record order, training chronological and shuffled models at three scales. They find that chronological models, called specialists, need roughly 27 to 58 percent fewer update steps than shuffled models, called generalists. They explain this with an empirical step law based on the initial margin gap and initial gradient norm, and then apply the idea to mixed corpora, Gemma continual pretraining, existing editors, and versioned software data.

Critical strengths
  1. The paper introduces the specialist versus generalist distinction and explains it in detail. It then applies this insight to pretraining data ordering and shows a clear improvement in training efficiency.

  2. The step law is interesting and potentially extendable. I think it could serve as a useful starting point for future theoretical work on how data ordering affects model training.

Critical weaknesses
  1. Data ordering effects have been studied before, but the experiments in this paper do not include relevant ordering baselines. In particular, "Understanding Data Temporality Impact on Large Language Models Pre-training" reports findings that are highly similar to this paper. As a result, the core observation about data ordering may not be the main contribution here.

  2. The derivation relies on assumptions that are not very intuitive. For example, the statement around line 1093, "assume that the average speed up to success is a fixed multiple A of the first-step speed," feels like a strong simplification that needs more justification.

  3. The Mix-C ordering method shows clear gains, but it is unclear how long these benefits last into post-training. Pretraining data is not visible during post-training fine-tuning, so the ordering cannot be re-applied later. The paper should discuss how durable the advantage is once the model moves beyond pretraining.

Questions for authors
  1. Can the authors add direct baseline comparisons against prior data ordering work, and clarify what the core contribution is beyond the ordering effect itself?

  2. Can the authors justify the fixed assumptions in the step law and show how long the Mix-C ordering advantage persists into post-training, given that pretraining data is not visible during later fine-tuning?

Reviewer_WBrVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks a simple question: does the order in which a language model sees facts during pre-training change how easy it is to update those facts later? The authors test this on synthetic biographies. Each person has attributes like employer and university that change over the years 2000–2024. They train small models from scratch on the same facts in two ways. In one, records appear in chronological order. In the other, they appear in random order. They call the first kind of model a specialist, because it ends up good at recent facts but poor at older ones. They call the second a generalist, because it handles all years reasonably well. Afterwards, they teach each model a new fact with a small fine-tune (LoRA) and count how many steps it takes until the new value beats the old one. Specialists need about 27–58% fewer steps. The authors then propose a formula that predicts the step count from two things: how far the model starts from the new fact, and how strongly the first gradient pushes toward it. They also propose a mixed training set (shuffled history plus an ordered recent window), test the idea on Gemma-2-9B, on four existing knowledge-editing methods, and on a synthetic software-versions dataset.

Critical strengths
  • The question is new and practical. Most knowledge-editing work takes the pre-trained model as given. This paper asks what the pre-training data layout does to later updates.
  • The main experiment is clean. The facts are identical across conditions and only the order differs.
  • The robustness checks are thorough. The saving holds across optimizers, learning rates, LoRA ranks, full fine-tuning, stopping rules, and (for the smallest model) several random seeds.
  • The authors are open about unflattering results, for example that facts outside the replay data lose about 25% of their past-year answers in the Gemma experiment (Appendix D.2).
Critical weaknesses
  1. The effect may just be recency, not “organization.” A chronologically trained model finishes training on the most recent data. It may be fast to update simply because it saw current facts last. The paper never separates this from a broader claim about how knowledge is organized. Two experiments would help: Take a chronological model partway through training, right after it finishes an early year such as 2010, and update 2010 facts. If the advantage is about recency, it should show up there too. Compare against a shuffled model that gets a short final phase containing only current values. That is a common practice, and it would show whether chronological order adds anything beyond “put recent data last.”
  2. The “roughly half” headline compares models that are doing different jobs. In the undated shuffled model, the same person has conflicting values in the training data and nothing to tell them apart. That model has no real notion of “the latest value.” So “overwrite” means something different for it than for the chronological model. The fairer comparisons are the timestamped pair (27–39% saving for overwrites) and the mixed-corpus pair (about 25–34%). The “half” figure comes mostly from the undated pair, so the abstract and conclusion overstate the effect.
  3. The formula is less principled than it sounds. The abstract says the law is derived “from first principles.” In practice it is an approximation with one fitted correction factor. That factor is not a single constant. The authors’ own fits give 1.19–1.41 for chronological models and 1.42–1.67 for shuffled ones, and the two ranges don’t overlap. So part of the order effect is hiding inside the fudge factor instead of being explained by the formula’s two ingredients. I also checked this against Table 1. If the factor really were shared, the table’s two ratio columns would predict the savings. The predicted savings are larger than the observed ones in 17 of 18 rows. For example, one 110M row predicts roughly 49% against 27% observed. These are ratios of medians, so this is a rough check, but it suggests the decomposition is looser than the text implies. The correlation of 0.91 is also hard to interpret. The factor was fitted on half the updates of the same models, and the correlation spans configurations whose step counts differ by about 10×. What matters more is whether the formula predicts the difference between a specialist and its matching generalist, and that isn’t reported.
  4. Numbers for the same experiment differ across tables. For the same 110M chronological-versus-shuffled overwrite comparison, the saving is 42% in Table 1, 51% in Table 7 (seed 0), and 49% in Table 8. Also, the chronological median in Table 1 (51 steps) is higher than both the slow-fact and fast-fact medians in Table 6 (49 and 39), which shouldn’t happen if Table 1 combines them. This may only reflect different samples of updates, but the authors should explain it. It also suggests the noise in these medians is larger than the significance tests imply. Seed variation is reported only for the smallest model, while the larger models, where the biggest savings appear, are single runs.
Questions for authors
  1. Does the update advantage appear at an earlier checkpoint, such as right after the 2010 phase, when updating 2010 facts? How does it compare with a shuffled model given a short final phase of current values only?
  2. Is the chronological corpus sorted by year across all people, or sorted within each person with people mixed together? Section 3 and Figure 2 read differently.
  3. Can you show the formula’s prediction error within each specialist–generalist pair, with the correction factor fitted on different models than it is tested on?
  4. What causes the larger initial gradient in specialists?
  5. Why do Tables 1, 7 and 8 disagree for the same setup, and how many seeds back the 350M and 682M results?
  6. What happens when updating old facts, or after several rounds of moving the “recent window” forward?
  7. In the Gemma experiment, how much of the accuracy gain is on facts the model was not trained on?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.