acceptodds
Under review as a conference paper at ICLR 2027

LLM Evaluation is a Forecasting Problem

Abstract

Standard procedure in machine learning is to randomly split train and test data, emulating IID draws from one underlying data distribution. In practice, models are trained on past data and deployed in the future, which may come from distinct distributions. In this paper, we observe that text distributions shift continuously over time and thus, because the goal of model evaluation is to estimate how well a model will perform in the future, we argue that evaluation should reflect the temporal structure of the data. Indeed, we find that web text is highly non-stationary, as documents grow more compressible and semantics, domains, and embeddings shift over time. We further observe that these shifts are temporally local, and model performance changes continuously over time across diverse benchmarks; consequently, future model performance is better predicted by measurements made on recent evaluation data, and is best predicted by time-series forecasters applied to historic performance rather than by historic performance itself. Choosing which model to deploy based on forecasted accuracy enables more reliable selection of performant models, thus improving downstream accuracy in the regime practitioners care about: deployment. Taken together, our experiments suggest that time should be treated as a fundamental variable in LLM evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evaluation, LLMs, prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6vytOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work looks at the rols of temporal distribution shifts in LLM evals and argues that future model performance should be modelled as a forecasting problem. The authors strengthen their case by showing that web-based pretraining corpora contains systematic changes across time, and show that language models generalize better across temporally adjacent data. They evaluate 19 models / 12 benchmarks and find that recent eval scores are more predictive of next period performance compared to older eval scores. Building on this, they compare several time-series forecasting methods, including Chronos-2 and classical approaches, and demonstrate improvements in performance prediction and downstream model selection relative to historical averaging. Finally, they investigate the implications of temporal recency for pretraining and continued pretraining data selection.

Critical strengths

1. Interesting and practically relevant perspective on LLM evaluation. The paper highlights an important limitation of conventional static benchmarks by explicitly considering the temporal dimension of model evaluation. The finding that recent evaluations are generally more informative about future performance has practical implications for model assessment/selection. I find this perspective valuable, even independently of the proposed forecasting framework.

2. Comprehensive empirical investigation. The authors provide extensive empirical evidence across several dimensions, including temporal changes in pretraining corpora, controlled experiments on temporal generalization, and evaluations of 19 model configurations across 12 benchmarks. The connection between forecasting accuracy and downstream model selection is particularly interesting, as it demonstrates that improvements in prediction error do not necessarily translate into better deployment decisions.

Critical weaknesses

1. Limited evidence of generalization to real-world deployment. While the forecasting experiments are out-of-sample with respect to historical benchmark scores, they evaluate modern LLMs retrospectively on datasets that largely predate their training cutoffs. Consequently, the experiments do not reproduce genuinely deployment decisions, where models and their historical evaluation scores would need to have been available at the forecast origin. So, although the results demonstrate the value of temporal information, it remains unclear to what extent the proposed framework improves model selection under actual deployment conditions.

2. Limited robustness of the forecasting results. The forecasting experiments consider only one-step-ahead predictions, which may not adequately reflect practical deployment horizons. Furthermore, monthly evaluation scores may contain substantial sampling variability due to differences in the number and composition of examples across periods. Some of the improvements obtained through forecasting methods could therefore reflect smoothing of noisy evaluation scores rather than better predictions of underlying model performance. The incremental benefits of explicit forecasting over simpler recency-aware approaches also remain relatively modest, particularly given the comparable performance of simpler approaches such as persistence and exponential smoothing.

3. The training-data recency experiments in Section 4 are interesting but somewhat disconnected from the central forecasting contribution. Moreover, the comparison between cumulative and five-year training windows is conducted at a single model scale and fixed training budget. It would be useful to investigate whether the benefits of recent data persist at larger scales, where incorporating additional historical data may become advantageous, and whether the conclusions hold when evaluating performance beyond the following year. Though one can argue this is rightfully out of scope for the current work, and less of a criticial weakness therefore, but nonetheless might provide valuable feedback to the authors on how to structure section 4.

Questions for authors
  1. Prospective evaluation (Sections 3.3–3.4): The experiments evaluate modern models retrospectively on historical benchmark data. To what extent would the conclusions hold in a genuinely prospective setting, particularly when evaluating performance on data collected after the models' training cutoffs? Could the authors clarify how the retrospective setup affects the interpretation of the results as improvements in real-world deployment decisions?

  2. Monthly evaluation variability (Sections 3.2–3.3): How much of the observed monthly variation in evaluation scores can be attributed to sampling variability rather than changes in underlying model performance? Would the forecasting improvements persist after accounting for uncertainty in monthly scores or using a coarser aggregation frequency, such as quarterly evaluations?

  3. Forecasting methods and model selection (Section 3.4): Given that simple exponential smoothing achieves forecasting accuracy close to Chronos-2, how does it perform on the model-selection and regret metrics? Are the improvements of Chronos-2 over the strongest classical forecasting methods statistically and practically meaningful?

  4. Forecast horizons (Sections 3.3–3.4): The forecasting experiments focus exclusively on one-step-ahead predictions. How sensitive are the findings to longer forecast horizons, such as 3, 6, and 12 months at monthly frequency and 2–3 years at annual frequency? In particular, do the improvements in forecasting accuracy and model selection persist over longer horizons?

Reviewer_zpSCOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a simple but important question: Is LLM eval shall be a forecasting problem? And to understand this, authors conduct a detailed analysis on the LLM data at different timestamp. And found some interesting pattern strongly correlated with timestamp, and found the generalization difficulity for longer forecasting, so that calling for taking the timestamp gap into consideration when evaluating LLM.

Critical strengths
  1. This paper is very well motivated. Evaluating LLM (especially for pretraining model is extremely important) for fast iteration.
  2. Thinking about testing generalization in terms of time stamp is a novel and interesting perspective but also intuitively makes sense.
  3. The presentation of this work is logically sound, from purposing the question and dive in the details step by step through reasoning.
Critical weaknesses
  1. One challenge of evaluating LLM with future data is, we are already very short for tokens now. And we really want to leverage the close data as much as possible to make the model as strong as possible. Leaving amount of latest tokens for validation might be fine for iteration and ablation but not very ideal for the largest hero run.
Questions for authors

See weakness. In addition, just curious, are there any special pattern after the popular of agent coding model on code data? And since more and more tokens on web are synthetic LLM tokens now, would the forecasting getting easier and the conclusion we get in this paper get broken?

Reviewer_7kF8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that LLM evaluation should be treated as a time-series forecasting problem: since data drifts over time, a model's score on the next time period should be predicted from the trend in past periods rather than from an average over all history. Section 2 shows that pretraining corpora (FineWeb-Edu, DCLM-Baseline, Nemotron-CC) drift over the years, and that 411M models, each pretrained from scratch on a single year, have higher loss on years further away (Fig. 3, 7). Section 3 builds 12 time-stamped datasets (9 new, Table 1) and evaluates 19 existing open-model configurations without training them. There, the most recent period's score reduces annual forecast error from 0.0308 (historical mean) to 0.0200, and Chronos-2 reduces it to 0.0194 (Table 2), and both improve best-of-K model selection (Fig. 1, 11). Section 4 compares pretraining from scratch on all data up to a cutoff against only the last five years (Fig. 13), plus continued pretraining on 1.6M tokens from different years (reported only in the text). The paper describes its contribution as a unified framework connecting these parts and proposes best practices for practitioners.

Critical strengths
  1. A reusable evaluation resource. The nine new benchmarks with carefully recovered dates (App. E) would be useful to the community if released, since time-stamped evaluation data is scarce.

  2. Appendix J is the most informative part of the paper. It shows that lower forecast error does not imply better model selection: monthly persistence improves MAE by 19.8%, but after removing the error shared across all models the gain is only 3.3%, and monthly persistence picks worse models than the plain mean (Fig. 11). This is a concrete warning against selecting models from small recent evaluation slices.

Critical weaknesses
  1. The training setup is unrealistic and does not follow how models are trained over time in practice or in prior continual pretraining work.

The evidence that temporal shift affects models (Section 2.2, Fig. 3 and 7) comes from 411M models that are each pretrained from scratch on a single year and compared with a model trained on the test year itself, L(t, t). Models are not trained this way. In practice, and in time-continual pretraining benchmarks such as TiC-LM (cited by the authors), a model is trained on data accumulated over many years and then updated as new data arrives, often by warm-starting from the previous checkpoint. In those setups there is typically a lag before an updated model catches up on newer data, and the questions that matter are how fast it catches up and how much old data to keep. A single-year model at the Chinchilla budget says little about either. Section 4 does train cumulative models, but each cutoff is again trained from scratch (no continual updates), with one seed (seed 124, App. C), and judged by perplexity only. The continued pretraining experiment uses 1.6M tokens on top of an 8.23B-token model, about 0.02% of the pretraining budget, which is far from the late-stage data mixes (OLMo 3, Qwen3) the section uses as motivation. Its results ("12 of 18 windows", 87% lower regret) appear only in the text, with no table, no list of the windows and no error bars. This matters because the paper's recommendations on training data rely on these experiments, and the pipeline is not representative of the setting the recommendations are meant for.

  1. The main conclusions are expected, and the promised recipe is not delivered.

That recent data matters more for predicting the near future is widely understood. In the current setup the paper's central findings follow almost directly from the existence of a trend. In Fig. 4 the error of a stale score grows roughly as trend slope times its age (about 1.2 points per year on SCIDOCA), so "recent evaluations predict better" restates that per-period scores have trends. Annually, the gain comes from simply using last year's score (0.0308 to 0.0200, Table 2), and Chronos-2 adds 0.06 points. At the same time, the paper does not engage with the main reason practitioners keep old data: repeating past data can help learning on future data. For example, if a model is undertrained on a skill such as coding, older coding data helps it later as well. Section 4 itself says it does not suggest discarding old data and that practitioners "could instead decrease sampling rates with age" (Sec. 4), but this is never tested. The only comparison is between two extremes (all data versus the last five years), with no age-weighted mixtures or replay ratios. The introduction promises "best practices for mitigating this shift", and Section 5 recommends preserving timestamps and using recent validation. These are reasonable but general suggestions, not a full recipe: there is no guidance on how much old data to keep, how to weight it, or how often to update. This matters because the paper's significance rests on giving practitioners something actionable beyond "recent data is more relevant".

Other issues:

  • The paper describes "a unified framework connecting temporal pretraining data shifts, evaluation staleness, forecasting, model selection, and training data selection" (App. A). I see a collection of related studies instead: Section 2 and Section 4 use the authors' 411M models with perplexity, Section 3 uses existing open models on unrelated downstream datasets, and no experiment or formal model links pretraining drift to downstream evaluation staleness. Han et al. (2024), which the paper calls the most closely related work, already combines rolling-window estimation with model selection under drift and is not compared.
  • Several trends may come from how the benchmarks were built rather than drift. WildTime-arXiv is binned by update date, with unusual bin sizes at the 2009/2010 boundary where App. I reports its largest ranking reversal. SCIDOCA's candidate set grows with reference-list length. StreamingQA's older questions carry time phrases that lower F1 from 0.71 to 0.47 or 0.36 in the original paper. Most target periods are also before the models' training cutoffs (App. K).
  • No confidence intervals or significance tests are reported. With about 114 annual targets, the roughly 8-point selection gain at K = 19 corresponds to about 9 extra correct picks, and monthly slices (about 108 examples per month for StreamingQA) have a noise floor comparable to the reported monthly errors.
Questions for authors
  1. [Clarification, affects my score] Can you repeat the key Section 2 and Section 4 analyses in a continual setup, where the model is trained on accumulated data and updated over time from its previous checkpoint, as in time-continual pretraining benchmarks? Do the conclusions about recency still hold there, and how long does the updated model take to catch up on new data?

  2. [Clarification, affects my score] What exactly is the recipe you propose? How much old data should be kept, how should it be weighted by age, and is "decrease sampling rates with age" supported by any experiment in the paper?

  3. [Suggestion] Can you test age-weighted mixtures or replay ratios rather than only "all data" versus "last five years", including a case where older data covers a skill the model is undertrained on?

  4. [Clarification] Please report the continued pretraining results as a table with the list of windows and per-window perplexities, and add confidence intervals for Table 2 and Fig. 1 and 11.

  5. [Clarification] How does the method of Han et al. (2024) compare with persistence and Chronos-2 on your data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.