- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper develops a tractable teacher–student model of linear RNN pretraining. A stable latent recurrence generates dependent trajectories, a Gaussian sketch introduces a finite representation width M, and a linear predictor is trained using safeguarded full-batch WSD gradient descent. The analysis derives joint scaling laws in model width M, trajectory count N, trajectory length P, and optimization horizon R, separating approximation error, optimization bias, statistical variance, and a controlled cross term. Distinct initialization and innovation spectra produce two regimes: trajectory count and length are interchangeable at leading order in one regime, while longer trajectories provide additional benefits by suppressing the initialization transient in the other.
Critical strengths
- The paper makes a substantive modeling contribution by combining recurrent data generation, finite-width random compression, nonstationary initialization, and finite-time empirical training. These components play distinct roles and enable a joint analysis of model width M, trajectory count N, trajectory length P, and optimization horizon R.
- The two-regime analysis identifies a concrete mechanism determining when longer trajectories and more independent trajectories are interchangeable. The explicit spectral crossovers clarify when sequence length provides additional benefits by suppressing the initialization transient and when those benefits saturate.
- The theory handles dependent trajectories and empirical GD, provides matching scaling orders for individual error components, and derives compute-allocation consequences within the stated assumptions.
Critical weaknesses
- The population evaluation objective changes with P. Increasing trajectory length changes the position-averaged covariance in both training and evaluation, so the reported improvement does not directly establish better learning on a common prediction task. Although the authors acknowledge this limitation, fixed-evaluation comparisons would strengthen the practical interpretation.
- The sequence-allocation conclusions require restrictive sample conditions. In particular, Regime II requires N≳ RM, limiting how far trajectory length can increase at a fixed token budget. The compute-optimal allocations also depend on these sufficient conditions, whose necessity remains unclear.
- The experiments mainly validate individual error components in the prescribed synthetic model. The bias experiments use population covariance, and the full empirically trained predictor is not directly compared across different N/P allocations at a fixed token budget. The connection to realistic sequence models and learned representations therefore remains limited.
Questions for authors
- At fixed D=NP, M, and R, does increasing training trajectory length improve the full predictor when every model is evaluated on the same test distribution, such as a fixed test horizon or stationary trajectories?
- Is the condition N≳ RM in Regime II necessary, or is it primarily a proof requirement? Could relaxing it change the compute-optimal allocations?
- Which scaling conclusions depend specifically on WSD? Would the same exponents hold for other stable learning-rate schedules with comparable cumulative step sizes?