acceptodds
Withdrawn from ICLR 2027

Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Empirical scaling laws for autoregressive language models relate prediction loss to model size, data size, and optimization compute, but their theoretical origin is still poorly understood in sequential pretraining settings. We study this question in a tractable teacher–student model where a stable latent linear RNN generates trajectories and a sketched linear recurrent student is trained by safeguarded full-batch WSD gradient descent on next-token prediction. The sketch dimension plays the role of model size, while independent trajectories of length provide the training tokens. We allow the innovation and initialization covariances to have different power-law exponents and . The induced design spectrum produces explicit approximation, optimization, and statistical scaling laws separated by spectral crossovers. When , the original one-scale rates , , and are recovered. When , the heavier initialization tail changes the rates beyond -dependent model and optimization crossovers. The proof uses a covariance event only internally and a globally safeguarded step size on its complement. The variance retains the factor , while sequence length also suppresses the initialization transient, so and cease to be fully interchangeable in the two-scale regime.

Primary area:
learning theory
Keywords:
Neural scaling laws, Autoregressive pretraining, Linear recurrent neural networks, Sequence length, Compute-optimal allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XHGpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper develops a tractable teacher–student model of linear RNN pretraining. A stable latent recurrence generates dependent trajectories, a Gaussian sketch introduces a finite representation width M, and a linear predictor is trained using safeguarded full-batch WSD gradient descent. The analysis derives joint scaling laws in model width M, trajectory count N, trajectory length P, and optimization horizon R, separating approximation error, optimization bias, statistical variance, and a controlled cross term. Distinct initialization and innovation spectra produce two regimes: trajectory count and length are interchangeable at leading order in one regime, while longer trajectories provide additional benefits by suppressing the initialization transient in the other.

Critical strengths
  1. The paper makes a substantive modeling contribution by combining recurrent data generation, finite-width random compression, nonstationary initialization, and finite-time empirical training. These components play distinct roles and enable a joint analysis of model width M, trajectory count N, trajectory length P, and optimization horizon R.
  2. The two-regime analysis identifies a concrete mechanism determining when longer trajectories and more independent trajectories are interchangeable. The explicit spectral crossovers clarify when sequence length provides additional benefits by suppressing the initialization transient and when those benefits saturate.
  3. The theory handles dependent trajectories and empirical GD, provides matching scaling orders for individual error components, and derives compute-allocation consequences within the stated assumptions.
Critical weaknesses
  1. The population evaluation objective changes with P. Increasing trajectory length changes the position-averaged covariance in both training and evaluation, so the reported improvement does not directly establish better learning on a common prediction task. Although the authors acknowledge this limitation, fixed-evaluation comparisons would strengthen the practical interpretation.
  2. The sequence-allocation conclusions require restrictive sample conditions. In particular, Regime II requires N≳ RM, limiting how far trajectory length can increase at a fixed token budget. The compute-optimal allocations also depend on these sufficient conditions, whose necessity remains unclear.
  3. The experiments mainly validate individual error components in the prescribed synthetic model. The bias experiments use population covariance, and the full empirically trained predictor is not directly compared across different N/P allocations at a fixed token budget. The connection to realistic sequence models and learned representations therefore remains limited.
Questions for authors
  1. At fixed D=NP, M, and R, does increasing training trajectory length improve the full predictor when every model is evaluated on the same test distribution, such as a fixed test horizon or stationary trajectories?
  2. Is the condition N≳ RM in Regime II necessary, or is it primarily a proof requirement? Could relaxing it change the compute-optimal allocations?
  3. Which scaling conclusions depend specifically on WSD? Would the same exponents hold for other stable learning-rate schedules with comparable cumulative step sizes?
Reviewer_q7bsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies scaling laws in a simplified linear RNN setting, analyzing how model size, trajectory count, sequence length, and optimization affect prediction error. Its main finding is that longer trajectories can suppress initialization transients, making sequence length and trajectory count inequivalent at the same token budget. The analysis is supported by synthetic experiments and motivated by language-model pretraining.

Critical strengths
  1. The paper investigates how training-data organization can affect learning beyond the total number of tokens, which is a potentially relevant question for language-model pretraining.

  2. The theoretical framework separates approximation, optimization, and statistical errors, providing an interpretable account of their dependence on training variables.

  3. The analysis identifies a concrete mechanism through which sequence length can influence prediction error.

Critical weaknesses
  1. Unclear practical motivation. The paper does not specify whether it addresses context-length selection, document truncation, or data packing, making its relevance to actual LLM training decisions difficult to assess.

  2. Unclear connection to context capability. Longer trajectories mainly dilute initialization effects without explicitly expanding the student’s accessible context. How this mechanism explains improved long-range context use in LLMs remains unestablished.

  3. Insufficient experimental support. Synthetic results alone do not establish applicability to LLMs. Controlled LLM experiments with matched token budgets and a fixed evaluation set are needed to validate the claimed sequence-length benefits.

Questions for authors
  1. What specific LLM training decision does the theory address? Please clarify whether sequence length represents a training context window, document length, or a data-processing choice, and explain how varying it corresponds to a realistic training procedure.

  2. How does the proposed mechanism relate to LLM context use? Please explain why suppressing initialization transients should improve language modeling, and whether the predicted advantage persists when models trained with different sequence lengths are evaluated on the same distribution.

  3. Please add controlled LLM experiments supporting the claimed implications. Vary training sequence length while controlling model architecture, data source, and total training tokens, and evaluate all models on the same held-out data and long-context tasks. These experiments should test whether the predicted advantage appears beyond the synthetic linear setting.

Reviewer_SCowOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces a tractable scaling law for a simple latent linear RNN. The authors introduce a thorough formalism for their model, and describe two regimes for model training: one is initialization-driven, and the other is innovation-driven.

Critical strengths
  • Theoretical analyses of scaling laws provide insights into modeling and training dynamics.
Critical weaknesses
  • Writing and AI usage: this paper is barely understandable in its current form. The abstract and conclusion do not provide any general insight or conclusion. The rest of the paper is extremely hard to parse, with poorly defined phrases ("innovation covariance" or "ambient/sketch dimensions"). Many subsections are very dense with numerical values (Section 4) or notations/definitions (Theorem 3.1). Overall, this paper reads as entirely AI-generated, with minimal efforts to make it readable. Moreover, some closely related citations are missing (1,2).
  • Real-world application and validity: The authors never motivate their assumptions and do not discuss the validity of their model. Given that their model is extremely simple compared to practical sequential models (e.g. LLMs), the fact that their theory does not provide accurate predictions (Figure 3) is concerning for the actual validity of their results.

1: Learning dynamics in linear recurrent neural networks, Proca et al., ICLM 25 (Oral) 2: On the Curse of Memory in RNNs, Li et al., ICLR 21

Questions for authors
  • What are clear takeaways from this paper? What is the intrinsic motivation and realism in the assumptions/design choices that were made?
  • For instance, what does the innovation factor represent?
  • How accurate would you say your theory is, based on the gaps that can be observed in Figure 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.