- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether self-improvement training can help T5 models extrapolate to longer sequences from simulated birth–death processes. It varies the number of population states added per self-improvement round (), and evaluates both output structure and numerical prediction across Verhulst, Poisson, and Ricker processes. The results suggest that self-improvement can help on these controlled tasks, with the effect of depending on the process and metric. The question is interesting, but the paper does not yet establish that these findings transfer to general-purpose LLMs or broader long-context forecasting.
Critical strengths
-
Interesting, underexplored application of self-improvement to structured stochastic processes is a useful test case for length generalization. The process setup gives the authors control over the data-generating mechanism and sequence difficulty.
-
The study evaluates multiple process families and complementary measurements to distinguish structural validity from numerical prediction quality. It also includes prompting and fine-tuning comparisons rather than relying on a single training setup.
-
The paper connects the number of states added per round to changes in the variance of the process, using the Chapman–Kolmogorov equation, and tests different values. This offers a concrete starting point for understanding why self-improvement may help in some settings.
Critical weaknesses
-
Limited evidence for broader relevance and unclear positioning against forecasting work. The central experiments use synthetic data and T5-small/T5-base on a narrow family of serialized population sequences. This is not enough to support broad conclusions about contemporary pretrained LLMs, generic long-context OOD generalization, or time-series forecasting. The paper should more clearly position itself relative to work such as Gruver et al. (2023), which studies zero-shot forecasting of numeric series serialized as text. The distinction seems to be that this paper studies self-improvement and length extrapolation on simulated birth–death processes, but it does not establish how much this setup adds beyond prior LLM forecasting results. Nor does it compare against strong time-series foundation models or process-specific statistical baselines, making it difficult to tell whether language-model training is particularly useful here. The application example is also based on simulated extrapolation, rather than a broad evaluation on real population forecasts.
-
The mechanism and training setup need clearer validation. The motivation for is plausible, but increasing changes both sequence length and the distribution/variance of prediction targets. The experiments do not clearly isolate which change causes the observed effects. In particular, Section 3.3 uses linear regression and decision trees to argue that larger creates a more complex mapping; this is not direct evidence about the LLMs or the self-improvement mechanism. The main text also makes it difficult to reconstruct the full training recipe and comparisons. The authors should clarify model initialization, per-round data construction and labels, optimization, data splits, compute/updates, and variation across random seeds.
-
Presentation and claims are not yet sufficiently polished or calibrated. Several plots and tables overflow or are difficult to read, and the prose is repetitive in places. The conclusion that length generalization is “a solved problem for some data” seems too strong for the limited synthetic task families and model sizes studied. The broad claim that length generalization remains difficult is supported mainly by an older citation (Anil et al., 2022), while another statement about prior studies lacks a citation. These claims should be qualified and supported with more recent work. The prompting result is also hard to interpret as a forecasting comparison: the domain-informed prompt performs better than “Find the next number,” but this mainly shows the value of supplying task information, not that the model is a strong general-purpose forecaster.
Questions for authors
-
How does this work extend prior LLM-based numeric forecasting, such as Gruver et al. (2023), beyond applying self-improvement to synthetic birth–death sequences? Can the authors compare with a contemporary pretrained LLM and a strong time-series foundation or process-specific baseline on the same data and protocol, or narrow the claims to the studied T5/synthetic setting? Any claim that TSFMs generally outperform LLMs should be tied to recent, matched benchmarks and qualified by task and evaluation setting.
-
Could the authors provide a compact, reproducible description of the complete training pipeline, including initialization, tokenization, data splits, how model-generated outputs enter each round, optimization budget, and number of seeds? How are comparisons controlled for training tokens, updates, and compute?
-
Section 3.3 uses linear regression and decision trees to motivate the increased-complexity hypothesis. Can the authors clarify that these are diagnostic analyses rather than LLM experiments, and provide an ablation that separates the effect of on sequence length, target variance, and training distribution?
-
The domain-informed prompt provides substantially more task information than “Find the next number.” What conclusion should readers draw from this comparison, and can the prompting baseline be made more comparable to the trained models? Please also update/qualify the broad length-generalization claims and repair the overflowing figures and tables.