When Expressivity Does not Help: A Study of Linear and Nonlinear RNNs for Time Series Forecasting

Time series forecasting requires models to capture temporal dependencies across diverse time scales and dynamics. Recent work has explored more expressive recurrent architectures for forecasting, including mechanisms with input-dependent transitions and explicit state-tracking capabilities. Several studies have reported performance improvements over simpler recurrent models. However, whether these gains systematically arise from greater recurrence expressivity, and whether capabilities emphasized in general sequence modeling are broadly relevant to forecasting, remains unclear. We present a systematic empirical study of recurrent models with different levels of recurrence expressivity for time series forecasting. Overall, by using bootstrap hypothesis testing, we find that greater recurrence expressivity does not significantly improve forecasting accuracy. On controlled synthetic forecasting tasks, models with nonlinearity or explicit state-tracking mechanisms do not show a systematic advantage over less expressive architectures. Furthermore, on standard long-term forecasting and diverse real-world benchmarks, LTSF and GIFT-Eval, competitive linear models such as mLSTM can match or outperform several nonlinear architectures. Moreover, within linear-recurrence models, added expressivity through input-dependent transitions or explicit state-tracking mechanisms does not reliably improve performance. Our results suggest that recurrence expressivity alone is not a reliable predictor of forecasting performance, highlighting the importance of dataset characteristics and architectural inductive biases beyond expressivity.