acceptodds
Under review as a conference paper at ICLR 2027

Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

Abstract

Multimodal time-series forecasting, which pairs numerical sequences with domain-relevant texts, promises to inject world knowledge into forecasting pipelines. However, we uncover a critical failure mode in existing frameworks that we term text collapse: the text branch of the model converges to a content-independent transformation, contributing negligible discriminative signal regardless of the input content. We argue that text collapse arises from two related problems in multimodal time-series forecasting, namely gradient imbalance and insufficient text supervision. To address these issues, we propose REST-TS (Residual-Exclusive Supervision for Text in Time Series). The numerical backbone first produces an independent forecast, and the text branch is then supervised on the residual between the ground-truth future and this numerical forecast. The resulting auxiliary objective provides explicit supervision and a stronger gradient signal to the text branch, while encouraging it to model predictive information not captured by the numerical backbone. Evaluated across diverse real-world domains and backbone architectures, REST-TS achieves state-of-the-art performance and consistently demonstrates greater text-branch utilization than existing frameworks, providing strong empirical evidence that supervising the text branch on the residual encourages it to extract meaningful information from the input.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-series, Multimodal, Text Collapse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NMZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates text collapse in fusion-based multimodal time-series forecasting: textual representations exhibit limited diversity, weak alignment with forecasting targets, and little influence on predictions. It introduces three complementary diagnostics—effective rank, centered kernel alignment (CKA), and prediction divergence—and proposes REST-TS to mitigate this behavior. REST-TS constructs residual targets using an EMA numerical-input predictor and supervises the text branch on their trend and event components. Stop-gradient prevents the numerical branch from absorbing the auxiliary supervision. Experiments cover Time-MMD, FNF, and FNSPID, eight forecasting backbones, and an adaptation to SpecTF.

Critical strengths
  • Useful diagnostic contribution. The paper evaluates whether models actually use text through complementary measures of representation diversity, target alignment, and prediction sensitivity.
  • Well-motivated method with broad evidence. Residual supervision encourages complementary textual learning, supported by multiple benchmarks, forecasting backbones, and optimization analyses.
Critical weaknesses
  • The diagnosis may depend on limited text representations. Mean-pooled frozen token embeddings omit contextual interactions; experiments with contextualized representations are needed to establish the generality of text collapse.
  • The mechanism and resolution claims remain incompletely supported. The decomposition’s advantage over simpler residual supervision is modest, and improved diagnostic values do not necessarily demonstrate beneficial textual influence or complete resolution of collapse.
Questions for authors
  1. Does text collapse persist with contextualized frozen text representations, and does REST-TS retain its advantage?
  2. Can you provide broader statistical summaries across domains and backbones to qualify the consistency of the forecasting gains?
  3. What additional benefit does trend-and-event supervision provide over supervising the full residual or alternative components?
  4. How is the CKA target projection constructed, and does replacing informative text worsen forecasting error as well as change predictions?
  5. A comparison with TiMi (ICML 2026) would strengthen the positioning of REST-TS. In particular, does text collapse also occur under selective mixture-of-experts integration, and does residual-exclusive supervision provide additional benefits?
Reviewer_MCAvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REST-TS addresses text underuse in multimodal time series forecasting by training the text branch to predict trend and event components of numerical forecast residuals. Experiments across multiple datasets and backbones show improvements in forecasting accuracy and text-utilization metrics.

Critical strengths

Investigating text underuse addresses a meaningful problem in multimodal forecasting. Improvements across multiple datasets and backbones support the practical value of dedicated text supervision.

Critical weaknesses
  • The collapse diagnostics do not convincingly establish whether useful text information is ignored or recovered. Low effective rank can retain predictive information, while sensitivity to a generic replacement sentence does not demonstrate use of relevant text content.
  • The experiments do not isolate the benefit of residual targets from adding text-only auxiliary supervision. Without a matched comparison using the same auxiliary heads to predict the original forecasting targets, the specific contribution of residual-based supervision remains unclear.
Questions for authors
  • Does correctly matched text improve forecasting accuracy over text shuffled across test samples? Is this benefit larger for REST-TS than for the baselines?
  • With the same text-only auxiliary heads and loss weights, does predicting residual components outperform predicting future values directly?
Reviewer_JgThOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This study addresses text collapse in multimodal time-series forecasting, where the text modality is underutilized due to gradient imbalance and inadequate supervision. The authors propose REST-TS, which explicitly supervises the text branch to predict the trend and event components of the residual between ground-truth data and a time-series-only EMA forecast. Comprehensive evaluations confirm that REST-TS substantially improves both overall predictive performance and effective text utilization across eight backbones.

Critical strengths

This study presents a crucial problem formulation supported by complementary diagnostic analyses. It raises the significant question of whether multimodal time-series forecasting models genuinely leverage textual information or merely achieve incidental accuracy improvements.

By employing a combination of effective rank, Centered Kernel Alignment (CKA), prediction divergence, and gradient analysis, the authors offer multiple perspectives on the potential underutilization of text. While these metrics may not be definitive in isolation, their systematic application provides a highly valuable investigation into the issue. Furthermore, the paper delivers an extensive empirical evaluation across diverse forecasting settings, encompassing eight backbones, nine Time-MMD domains, and additional financial datasets.

The substantial performance gains observed, along with robust comparisons against established multimodal methods like SpecTF and OGM-GE, strongly substantiate the claim that residual-based auxiliary supervision effectively enhances forecasting accuracy across various architectural frameworks.

Critical weaknesses
  1. Insufficient evidence validating the resolution of text collapse. The core assertion that REST-TS compels the model to learn meaningful, content-dependent textual representations is not directly substantiated by the proposed diagnostics. A higher effective rank denotes representation diversity but does not guarantee meaningful text utilization. Similarly, CKA assesses alignment with forecasting targets rather than the extraction of complementary residual information. Prediction divergence merely indicates sensitivity to text ablation, not the utility of the resulting prediction shifts; because it is normalized by forecasting error, divergence may artificially inflate simply as accuracy improves. Crucially, the absence of controlled experiments using shuffled or mismatched text leaves a significant gap. Without proving that correctly paired texts outperform unrelated ones, it is ambiguous whether performance gains stem from genuine text utilization or merely the regularizing effect of the auxiliary supervision.

  2. Confounded explanations for text collapse and residual supervision efficacy. The authors primarily attribute text collapse to gradient imbalance and inadequate text supervision, yet the empirical evidence fails to isolate these causal factors. Notably, the increased gradient magnitude in the text branch under REST-TS is a standard consequence of introducing text-specific auxiliary losses, rather than definitive proof of meaningful learning. Additionally, the residual targets are derived from an EMA network coupled to the jointly trained multimodal model rather than an independent numerical-only forecaster. Consequently, it remains uncertain if these residuals genuinely capture information absent from the numerical modality. The provided ablations do not adequately separate the proposed mechanism from general regularization benefits, thereby weakening the causal rationale of the methodology.

Questions for authors

Could the authors evaluate REST-TS employing randomly shuffled or mismatched texts during both training and inference? Specifically, it is vital to determine whether utilizing correctly paired texts yields a statistically significant forecasting improvement over unrelated texts.

How can the authors disentangle the specific benefits of residual-exclusive supervision from those of general auxiliary regularization?

Comments on the paper

2 comments

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

Thank you for the additional results. They address my concern about consistency across runs and settings, but the gain is still small, and I remain unconvinced that residual supervision contributes much to the overall forecasting improvement. I will therefore keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.