acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multimodal Time Series Forecasting: Text Modalities Need Constrained Fusion

Abstract

Recent advances in multimodal learning have motivated the integration of textual information into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities that may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low-rank adapters to filter irrelevant textual information before fusing it into temporal representations. Across over 20K experiments, CFA ranks first among 11 compared methods while adding only 0.61% parameters. Code is available at https://anonymous.4open.science/r/mmcfa-602D.

Primary area:
learning on time series and dynamical systems
Keywords:
Multimodal fusion, Time series forecasting, Language models for time series, Constrained fusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XVi8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to incorporate historical text into numerical time-series forecasting without degrading the temporal model. It compares six naive fusion strategies with four constrained strategies and proposes CFA, a residual adapter consisting of down-projection, LayerNorm, ReLU, and up-projection. The text encoder is frozen, while the time-series model and fusion parameters are trained. The primary study spans nine Time-MMD datasets, 14 forecasting backbones, four text encoders, and four horizons, giving 2,016 configurations. Additional analyses include synthetic matching/contradictory/irrelevant text, rank sensitivity, representation diagnostics, multiple seeds, and ETT experiments using text derived from historical numerical observations. CFA has the best aggregate normalized MSE and average rank among the reported fusion strategies, with small module overhead. However, gains are heterogeneous, and the evidence does not yet isolate low rank as the mechanism responsible for semantic filtering or preservation of temporal information.

Critical strengths
  1. An extensive and useful empirical comparison of fusion choices. The 2,016-configuration study covers several backbones, text representations, datasets, and horizons, with detailed per-configuration results. The supplied recount finds that CFA beats the no-text baseline in 1,204 configurations, ties in 127, and loses in 685. This breadth supports a meaningful aggregate-performance observation while exposing its limits. The paper also includes multi-seed analysis and an additional ETT benchmark, so the concern is not a complete absence of statistical or external evaluation.
  2. A simple, lightweight design with relevant robustness probes. The adapter is easy to integrate and has reported module overheads of 0.61% in parameters and 0.04% in FLOPs, although these exclude the full cost of text encoding. The synthetic matching, contradictory, and irrelevant-text tests address meaningful failure modes. A carefully substantiated empirical finding about such a simple component would be worthwhile even without claiming an entirely new adapter architecture.
Critical weaknesses
  1. The central explanation of text-dependent low-rank filtering is not established, and the stated computation may remove text dependence for some backbones. Appendix P uses a bottleneck width of max(1, floor(C/r)) for several channel-based backbones. With the reported channel counts and default r=8, these bottlenecks have width one. If Equation (1) applies standard LayerNorm over that single feature, its normalized value is constant: it equals the learned bias with affine normalization and zero without it. The resulting residual would then be independent of the input text. This is a conditional mathematical implication of the description, not proof of what the code does; a different normalization axis or special handling could resolve it. Independently, the “without bottleneck” comparison also changes normalization, nonlinearity, and parameterization, so it cannot isolate rank. Low rank constrains the injection subspace but does not itself bound residual magnitude or prevent cancellation of temporal information. These issues directly affect the claimed mechanism behind the contribution. See Equation (1) and Appendices I, J, K, L, and P.
  2. The evidence for robust superiority and semantic selectivity is weaker than the aggregate claims imply. The Friedman/Nemenyi analysis treats 2,016 configurations as blocks despite shared datasets, horizons, backbones, and repeated no-text baselines. This dependence matters for the relatively small CFA–Gating rank difference of 0.361, only slightly above the reported 0.05 critical difference of 0.336. CFA beats Gating in 1,049 configurations but loses in 871, with 96 ties; superiority is not uniform. The semantic-injection diagnostic also reports a matching-versus-contradicting contribution-ratio difference of only about 1.94 × 10^-6 on the original scale. A significant standardized effect does not by itself establish practically meaningful suppression, and text-template differences can explain the effect without checking consistency against the same history. Finally, extreme divergence in some naive-fusion configurations makes matched normalization, initialization, and convergence checks important to interpreting the average advantage. These concerns limit both the statistical and mechanistic conclusions, rather than negating the observed aggregate gains.
Questions for authors
  1. Decisive implementation clarification: For DLinear, FiLM, Koopa, and TSMixer, what are the actual tensor shapes, bottleneck widths, and LayerNorm axes? With numerical history fixed, does replacing the text change the adapter output when the bottleneck width is one? Please reconcile the dimension rules in Appendices I and P with the implementation.
  2. Assessment-changing ablation: When LayerNorm, ReLU, initialization, and training protocol are held fixed, how does low rank compare with full rank? How competitive are constant, zero, or shuffled-text residual controls? These comparisons would determine whether the gain genuinely depends on informative text and the rank constraint.
  3. Assessment-changing statistical check: Does CFA retain an advantage over Gating under dataset-level paired analysis or a suitable clustered/hierarchical analysis? Please provide per-dataset effects and intervals, and reconcile the different CFA–Gating Wilcoxon values in Tables 19 and 20.
  4. Assessment-changing mechanism check: With the same numerical history and matched text templates, does changing direction, timing, or factual content produce meaningful changes in residual norms, predictions, and errors? Please show the contribution ratios on their original scale and explain the practical significance of the observed difference.
  5. Clarification of the comparison protocol: How are hyperparameters selected on validation data, how is normalized MSE computed and aggregated, and do the main conclusions persist after stabilizing naive fusion with matched normalization and initialization or separately reporting divergent runs?
2 comments

Please let me know if you have any further questions or concerns!

If your concerns have been sufficiently addressed, I would greatly appreciate your consideration of a score increase.

Dear Reviewer XVi8,

As we are approaching the end of the discussion period, we appreciate your time and consideration. If you have any remaining questions or comments, we would be happy to address them before the deadline.

Best regards,

The Authors

Reviewer_EZmdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how textual information should be incorporated into time-series forecasting. Its central finding is that naive additive or concatenation-based fusion often underperforms numerical-only forecasting, whereas constrained fusion provides more reliable improvements. The proposed Controlled Fusion Adapter (CFA) injects textual features through a low-rank residual bottleneck. The evaluation covers nine Time-MMD domains, fourteen forecasting backbones, four frozen text encoders, and four forecasting horizons. The paper’s main contribution is a systematic empirical study of fusion strategies, together with a lightweight implementation of constrained fusion.

Critical strengths
  • Broad and systematic empirical evidence. The evaluation spans nine domains, fourteen forecasting backbones, four text encoders, and four horizons, supported by multi-seed results and statistical analyses. This provides substantial evidence that the observed benefits are not restricted to a favorable configuration.

  • A useful finding paired with an efficient solution. The paper demonstrates that naive text fusion can hurt forecasting and offers constrained fusion as a practical remedy. CFA achieves competitive accuracy with only 0.61% additional trainable parameters, making the findings actionable for existing forecasting models.

Critical weaknesses
  • The role of the low-rank constraint is not convincingly isolated. The ablation compares CFA with direct additive fusion, changing normalization, activation, initialization, and projection structure alongside rank. Consequently, the improvements cannot be attributed specifically to low rank. A full-width adapter with otherwise identical components is needed to support the central filtering claim.

  • Generalization beyond the evaluated setting remains uncertain. The real-text experiments are concentrated on Time-MMD, while the external ETT experiment uses text derived from numerical history. Together with frozen text encoders throughout, this limits evidence that CFA’s advantage extends to independent textual sources and jointly adapted multimodal models.

Questions for authors
  1. Does CFA outperform a full-width residual adapter when normalization, activation, initialization, placement, and tuning budget are held fixed?
  2. Are all learning-rate and checkpoint selections performed exclusively on validation data?
  3. Can the generalization claim be supported by evaluation on another benchmark containing real, externally sourced paired text?
  4. Does constrained fusion retain its advantage when the text encoder is fine-tuned?
  5. TiMi (ICML 2026) is a closely related plug-in approach that selectively incorporates textual guidance into time-series Transformers. A comparison on shared backbones and datasets, reporting both accuracy and computational cost, would help establish CFA’s practical advantage.
1 comment

Please let me know if you have any further questions or concerns!

If your concerns have been sufficiently addressed, I would greatly appreciate your consideration of a score increase.

Reviewer_AfbpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of naive multimodal fusion in text enhanced time series forecasting. Through extensive empirical evaluations, the authors observe that directly combining textual and numerical representations through addition or concatenation can degrade forecasting performance compared with unimodal models. To address this issue, they propose Controlled Fusion Adapter, a lightweight plug-in module that constrains textual information through low-rank projections before integrating it into temporal representations via residual connections. The proposed method is evaluated across multiple datasets, forecasting backbones, and text encoders, demonstrating competitive performance with limited additional parameters.

Critical strengths
  1. The paper investigates the potential negative effects of incorporating auxiliary textual information into time series forecasting, which is a practically relevant issue in multimodal learning.
  2. The authors systematically compare naive and constrained fusion strategies across a large number of experimental configurations
  3. Extensive experiments have verified the performance of the model.
Critical weaknesses
  1. The paper fails to clearly identify what important limitations remain unresolved by existing multimodal fusion methods, making the necessity and novelty of the proposed approach less convincing. A substantial body of prior work has developed fusion strategies, including cross-modal attention, gating, alignment, and selective fusion, address the limitations of naive fusion.
  2. Insufficient methodological novelty. CFA mainly employs a low-rank projection followed by residual addition, which closely resembles conventional bottleneck adapters and parameter-efficient fusion techniques.
  3. Some important recent multimodal models have not been compared.
  4. The experimental results show that the performance improvement of CFA is limited compared to multimodal and LLM-based methods, while SpecTF and TaTS achieve better results on multiple datasets.
  5. Marginal improvements over existing constrained fusion strategies. CFA reduces the MSE from 0.261 to 0.256 compared with Gating, corresponding to an improvement of only approximately 1.9%.
Questions for authors
  1. Several existing multimodal time-series methods have already investigated modality interference, irrelevant textual information, and selective or adaptive fusion. What is the fundamental difference between the challenge addressed in this paper and those investigated in prior work?
  2. How does CFA explicitly distinguish task-relevant textual information from irrelevant or contradictory information?
  3. SpecTF and TaTS outperform CFA on several datasets, can authors clarify the specific advantages of CFA over these existing multimodal methods under comparable evaluation settings?
  4. How does CFA perform when the original texts are replaced with shuffled or constant texts, or when an equally sized residual adapter is used without textual inputs?
4 comments

Please let me know if you have any further questions or concerns!

If your concerns have been sufficiently addressed, I would greatly appreciate your consideration of a score increase.

Thank you for the author's response. I believe there are still core issues that have not been resolved. The author clearly stated that the contribution of CFA is to propose a constrained fusion method. However, compared with existing methods such as low-rank fusion and selective fusion, what is the essential difference between this method? The current response has not fully explained the innovativeness of its method. Furthermore, the author believes that by restricting the text information to a low-dimensional subspace, the interference from irrelevant text can be reduced. However, irrelevant information may also exist in the low-dimensional subspace and cannot be guaranteed to be effectively filtered.

Due to the lack of sufficient resolution of the method's innovativeness and the core method issues, I maintain the original score.

We sincerely thank the reviewer for engaging in the discussion and for clarifying the remaining concerns.

We address both points below.

 


[Q1] Essential difference from low-rank fusion and selective fusion

However, compared with existing methods such as low-rank fusion and selective fusion, what is the essential difference between this method?

First of all, we want to emphasize that our main contribution is a FINDING, not a specific module (CFA).

 

As stated in Section 3.3, "our goal is NOT to develop a novel adapter but to design a controlled fusion formulation that constrains the textual signal effectively."

The finding is that "in multimodal TS forecasting, text should enter in a constrained manner":

  • (1) Naive fusion often hurts.
    • The unimodal model outranks all six naive variants (Table 20).
  • (2) Constraining text is what helps.
    • All four constrained strategies, including Gating and FiLM, outrank every naive strategy and attain lower error (Table 20, Figure 3d).

So the question we answer is not which fusion operator to use but how much freedom text should have, and CFA is one simple and effective realization of this principle.

 

That said, CFA also differs from the two families the reviewer mentions.

They constrain DIFFERENT things, as summarized in Table 7 (Appendix D):

  • [1] Low-rank fusion (LMF, Sahay et al., FuseMoE) uses low rank to build a joint representation in which all modalities are symmetric, and it replaces the backbone. The low rank serves the efficiency of the fusion tensor.
  • [2] Selective fusion (Gating, FiLM, orthogonal fusion) decides how much text enters each sample, but the text still acts in the full D-dimensional space, so it can move the temporal representation in any direction.
  • [3] CFA constrains where text can enter. TS keeps all D dimensions as the primary modality, while text is confined to a subspace of dimension at most D/r and added as a residual (Appendix D.3, Appendix L). The backbone is left untouched.

In short, CFA turns the fusion into an asymmetric capacity constraint: Text can adjust the forecast only along a few learned directions and cannot overwrite the temporal signal. This is not the goal of low-rank fusion, and selective fusion does not provide it.

 

The difference is also visible in the results.

  • Gating is the closest selective baseline, yet it falls below the unimodal model on ETT (-1.14%), while CFA improves on it (+1.42%, Table 21).
  • CFA also outperforms Gating on 9 of 9 datasets (p = 0.004, Appendix T.6) with about 1/47 of its parameter overhead (Table 5).

 


[Q2] Can irrelevant information survive in the low-dimensional subspace?

However, irrelevant information may also exist in the low-dimensional subspace and cannot be guaranteed to be effectively filtered.

We agree that the low rank alone does not guarantee that every irrelevant component is removed, and we do not claim perfect filtering.

What the constraint guarantees is that any text can act only within at most D/r directions, so it cannot override the D-dimensional temporal representation (Appendix L). Within this subspace, the directions are learned from the forecasting loss, which favors text that lowers the error.

 

The experiments confirm both parts:

EvidenceIrrelevant or misleading textRelevant text
Text controls (Table R1, Appendix T.1), norm. MSEShuffled 0.513, constant 0.495 (unimodal 0.508)Paired 0.458
Counterfactual edits (Table R7, Appendix T.7), change of residual normOut-of-scope text: CFA -2.0%, naive +68.5%Flipped direction: CFA follows it (+0.23)
Text-contribution ratio (Appendix J.2)Contradicting 31.56Matching 33.50 (p < 0.001)
  • Irrelevant text leaves CFA at the unimodal level instead of hurting it, while paired text clearly helps.
  • On out-of-scope text, the naive residual grows by up to 69%, whereas CFA stays within ±2%.
  • Matching text is injected more strongly than contradicting text, so the learned subspace separates the two.

The bottleneck also helps most precisely when text is irrelevant (+20.04%, Table 13), which is the case the reviewer is concerned about.

 


We believe the response above addresses the reviewer's remaining concerns on both the novelty and the filtering mechanism, and we would sincerely appreciate it if the reviewer could kindly reconsider the evaluation and the score of our paper!

Dear Reviewer Afbp,

As we are approaching the end of the discussion period, we appreciate your time and consideration. If you have any remaining questions or comments, we would be happy to address them before the deadline.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments

General Comments

We have uploaded a revised version of the paper.

We kindly ask the reviewers to download and check it. All revised and newly added parts are shown in purple.

 

We thank all reviewers for their careful and constructive feedback.

We are encouraged that the reviewers found our study

  • to address "a practically relevant issue in multimodal learning" (Afbp),
  • to provide "broad and systematic empirical evidence" and "a useful finding paired with an efficient solution" (EZmd),
  • and to offer "an extensive and useful empirical comparison of fusion choices" with "relevant robustness probes" (XVi8).

 

Main concerns and how we addressed them

1) Novelty and positioning (Afbp)

Existing methods either use naive fusion or rely on architecture-specific designs (Section 2, Table 1). We already show that the unimodal model outranks all six naive variants (Table 20), while CFA is a plug-in that improves on it with +0.61% parameters (Table 5). Our contribution is the finding that how much text enters decides whether it helps, and CFA is its simplest plug-in realization (Section 3.3).

 

2) Does the gain come from the text and the rank constraint? (Afbp, EZmd, XVi8)

The constrained path already beats direct injection for every text type (Table 13), and with irrelevant text from other datasets CFA stays close to the unimodal model, so the gain does not come from parameters alone (Figure 5). Nonetheless, we conducted additional controls with identical components: A full-rank adapter, and zero, constant, and shuffled text. Only paired text with the low-rank constraint clearly lowers the error, while the full-rank adapter (7.4x adapter parameters) and the text-free controls stay at the unimodal level (Appendix T.1, T.3).

 

3) Implementation for channel-space backbones (XVi8)

We added the tensor shapes, bottleneck widths, and LayerNorm axes of all 14 backbones (Appendix I, Table I.1), and set the width to at least 4 so that the residual depends on the text for every backbone. CFA then improves on the unimodal model by 1.8% on average on these backbones (Appendix T.5), and across all 14 backbones CFA keeps the best average rank (Appendix T.6).

 

4) Statistical dependence and the margin over Gating (Afbp, XVi8)

Dataset-level paired tests and a two-way cluster bootstrap confirm that CFA improves on the unimodal model (8/9 datasets, p = 0.027), on the best naive fusion (8/9, p = 0.008), and on Gating (9/9, p = 0.004) (Appendix T.6). CFA achieves this with about 1/47 of Gating's parameter overhead (Table 5).

 

5) Selection and comparison protocol (EZmd, XVi8)

With the LR (and the rank of CFA) chosen on validation data only, CFA remains the best of all 11 methods, and the ordering of the paper is preserved (Appendix T.4). Giving naive fusion CFA's normalization and initialization does not close the gap: CFA wins 69/108 settings (p = 0.005, Appendix T.8). The selection protocol and normalized MSE are now defined in Appendix H.

 

6) Mechanism (Afbp, XVi8)

A counterfactual study with the same history and one template shows that CFA responds to what the text says, not to how it is phrased: It follows a flipped direction, stops following out-of-scope text, and keeps the residual norm within ±2% (Appendix T.7).

 

7) Comparisons and generalization (Afbp, EZmd)

We already compare seven methods, including SpecTF and TaTS (Appendix S), and on a text-paired ETT only CFA improves on the unimodal model in both MSE and MAE (Appendix R, Table 21). Nonetheless, we conducted additional comparisons:

  • (1) TiMi, which CFA matches in accuracy at about 1/100 of its parameter overhead (Appendix T.2)
  • (2) A fine-tuned text encoder, with which CFA remains the best fusion strategy (Appendix T.9)
  • (3) Two external real-news benchmarks (crude oil and FNSPID), where CFA ranks first on both (Appendix T.10)

 

Please let us know if there are any remaining issues you'd like to discuss!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.