Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

Multimodal time-series forecasting, which pairs numerical sequences with domain-relevant texts, promises to inject world knowledge into forecasting pipelines. However, we uncover a critical failure mode in existing frameworks that we term text collapse: the text branch of the model converges to a content-independent transformation, contributing negligible discriminative signal regardless of the input content. We argue that text collapse arises from two related problems in multimodal time-series forecasting, namely gradient imbalance and insufficient text supervision. To address these issues, we propose REST-TS (Residual-Exclusive Supervision for Text in Time Series). The numerical backbone first produces an independent forecast, and the text branch is then supervised on the residual between the ground-truth future and this numerical forecast. The resulting auxiliary objective provides explicit supervision and a stronger gradient signal to the text branch, while encouraging it to model predictive information not captured by the numerical backbone. Evaluated across diverse real-world domains and backbone architectures, REST-TS achieves state-of-the-art performance and consistently demonstrates greater text-branch utilization than existing frameworks, providing strong empirical evidence that supervising the text branch on the residual encourages it to extract meaningful information from the input.