- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work looks at the rols of temporal distribution shifts in LLM evals and argues that future model performance should be modelled as a forecasting problem. The authors strengthen their case by showing that web-based pretraining corpora contains systematic changes across time, and show that language models generalize better across temporally adjacent data. They evaluate 19 models / 12 benchmarks and find that recent eval scores are more predictive of next period performance compared to older eval scores. Building on this, they compare several time-series forecasting methods, including Chronos-2 and classical approaches, and demonstrate improvements in performance prediction and downstream model selection relative to historical averaging. Finally, they investigate the implications of temporal recency for pretraining and continued pretraining data selection.
Critical strengths
1. Interesting and practically relevant perspective on LLM evaluation. The paper highlights an important limitation of conventional static benchmarks by explicitly considering the temporal dimension of model evaluation. The finding that recent evaluations are generally more informative about future performance has practical implications for model assessment/selection. I find this perspective valuable, even independently of the proposed forecasting framework.
2. Comprehensive empirical investigation. The authors provide extensive empirical evidence across several dimensions, including temporal changes in pretraining corpora, controlled experiments on temporal generalization, and evaluations of 19 model configurations across 12 benchmarks. The connection between forecasting accuracy and downstream model selection is particularly interesting, as it demonstrates that improvements in prediction error do not necessarily translate into better deployment decisions.
Critical weaknesses
1. Limited evidence of generalization to real-world deployment. While the forecasting experiments are out-of-sample with respect to historical benchmark scores, they evaluate modern LLMs retrospectively on datasets that largely predate their training cutoffs. Consequently, the experiments do not reproduce genuinely deployment decisions, where models and their historical evaluation scores would need to have been available at the forecast origin. So, although the results demonstrate the value of temporal information, it remains unclear to what extent the proposed framework improves model selection under actual deployment conditions.
2. Limited robustness of the forecasting results. The forecasting experiments consider only one-step-ahead predictions, which may not adequately reflect practical deployment horizons. Furthermore, monthly evaluation scores may contain substantial sampling variability due to differences in the number and composition of examples across periods. Some of the improvements obtained through forecasting methods could therefore reflect smoothing of noisy evaluation scores rather than better predictions of underlying model performance. The incremental benefits of explicit forecasting over simpler recency-aware approaches also remain relatively modest, particularly given the comparable performance of simpler approaches such as persistence and exponential smoothing.
3. The training-data recency experiments in Section 4 are interesting but somewhat disconnected from the central forecasting contribution. Moreover, the comparison between cumulative and five-year training windows is conducted at a single model scale and fixed training budget. It would be useful to investigate whether the benefits of recent data persist at larger scales, where incorporating additional historical data may become advantageous, and whether the conclusions hold when evaluating performance beyond the following year. Though one can argue this is rightfully out of scope for the current work, and less of a criticial weakness therefore, but nonetheless might provide valuable feedback to the authors on how to structure section 4.
Questions for authors
-
Prospective evaluation (Sections 3.3–3.4): The experiments evaluate modern models retrospectively on historical benchmark data. To what extent would the conclusions hold in a genuinely prospective setting, particularly when evaluating performance on data collected after the models' training cutoffs? Could the authors clarify how the retrospective setup affects the interpretation of the results as improvements in real-world deployment decisions?
-
Monthly evaluation variability (Sections 3.2–3.3): How much of the observed monthly variation in evaluation scores can be attributed to sampling variability rather than changes in underlying model performance? Would the forecasting improvements persist after accounting for uncertainty in monthly scores or using a coarser aggregation frequency, such as quarterly evaluations?
-
Forecasting methods and model selection (Section 3.4): Given that simple exponential smoothing achieves forecasting accuracy close to Chronos-2, how does it perform on the model-selection and regret metrics? Are the improvements of Chronos-2 over the strongest classical forecasting methods statistically and practically meaningful?
-
Forecast horizons (Sections 3.3–3.4): The forecasting experiments focus exclusively on one-step-ahead predictions. How sensitive are the findings to longer forecast horizons, such as 3, 6, and 12 months at monthly frequency and 2–3 years at annual frequency? In particular, do the improvements in forecasting accuracy and model selection persist over longer horizons?