acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multimodal Learning for Large Language Models in Time Series Forecasting: The Problem of Pseudo-Multimodality

Abstract

LLM-based time-series forecasting increasingly couples numerical predictors with prompts generated from the same observed series. Although the resulting branches differ in form, neither receives independent sample-specific evidence. They are two representations of a common observational source, a condition we call pseudo-multimodality. Cross-view similarity alone does not establish that the language representation should alter a numerical forecast. We formulate sequence-derived LLM forecasting as a conditional correction problem: given a complete numerical prediction, should a prompt-derived candidate modify it for each sample and variable? Conditional Incremental and Robust Fusion (CIRF) uses the numerical forecast as the default prediction and the language branch as a rejectable correction path. It removes the prompt component predictable from the numerical representation, decodes the remaining conditional residual as a candidate correction, and applies it through a utility-supervised gate. Training also limits the excess perturbation sensitivity introduced by this optional path when both representations inherit the same corrupted observation. We evaluate CIRF under matched input and forecasting-horizon protocols on nine public benchmarks. Across six same-checkpoint routing settings, selective correction yields lower reported MSE and MAE than mandatory correction, while language-route activation ranges from 0.2% to 77.5%.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Large Language Models, Conditional Correction, Pseudo-Multimodality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ssz6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines pseudo-multimodality in time series forecasting: numerical histories and prompts derived from them provide different representations of the same observations. It argues that the language pathway should be assessed by its conditional predictive utility, rather than representational agreement alone. CIRF constructs optional residual corrections to a complete numerical forecast; CIRF-R addresses additional sensitivity under shared perturbations. Benchmark, routing, and representation-replacement experiments investigate correction utility and distinguish it from benefits attributable to language pretraining or matched prompts.

Critical strengths
  1. The pseudo-multimodality perspective is important for multimodal time series research. The paper distinguishes acquiring another observation from deriving another representation, while recognizing that the latter can still help a particular predictor. Assessing this help through conditional utility provides a useful basis for deciding when to incorporate a language-derived correction.
  2. The attribution experiments are thoughtfully designed. LLM replacement/randomization, prompt masking/mismatching, numerical-only models, and Stats-MLP distinguish the roles of pretraining, prompt content, and encoding. Fixed-candidate routing and oracle comparisons separately examine correction potential and selection ability. These controls provide useful guidance for interpreting fusion results, including negative outcomes.
Critical weaknesses
  1. The relationship between agreement and correction utility remains empirically underexplored. The routing experiments establish heterogeneous utility, but do not jointly examine representation agreement, conditional predictability, and actual loss reduction. A focused diagnostic using existing checkpoints would help identify when agreement is informative or misleading and explain which cases benefit from utility-based selection.
  2. The shared-perturbation mechanism is not separately diagnosed. Table 4 supports the combined robustness procedure, but always regenerates prompts from the same corrupted history. It therefore leaves open how coupling the disturbances across views affects sensitivity beyond that of a correction path generally. A matched shared-versus-independent perturbation experiment would distinguish these effects and clarify the conditions in which the proposed common-source sensitivity control matters.
Questions for authors
  1. On existing ETT and Weather tasks, could you relate a justified agreement measure and the conditional predictor's normalized residual to candidate loss reduction, gate decisions, and oracle utility? Keeping candidates fixed, when does agreement identify beneficial corrections, and which cases require utility-based selection?
  2. In one existing corruption setting, could you compare perturbing the numerical input and the history used to generate prompts with shared versus independent disturbances? Holding checkpoints, perturbation distributions, and per-view intensity fixed, how do excess sensitivity and routing behavior change for CIRF and CIRF-R?
2 comments

Thank you for these focused diagnostic suggestions.

Q1. Agreement, residuals and correction utility

Agreement describes shared structure; correction acceptance depends on whether the decoded increment reduces numerical forecast error. Useful candidates also occur at low agreement, supporting CIRF's separation of residual construction and utility-based acceptance.

We associated validation-fitted Procrustes cosine between zX,zPz_X,z_P and normalized conditional residual r=∥zP−Cϕ(zX)∥2/(∥zP∥2+10−8)r=\|z_P-C_\phi(z_X)\|_2/(\|z_P\|_2+10^{-8}) with candidate MSE gain, gate probabilities/decisions and Oracle utility on the same window–variable pairs. Forecasts, candidates and Gate thresholds were fixed; proxy rules were selected on validation and frozen at test.

SettingNumericalAlways-onAgreementResidualGateOracle
Weather-960.1760.1730.1730.1720.1710.158
ETTm2-3360.3030.3130.3030.3030.3030.287

Three-seed mean test MSE from supplementary checkpoints, separate from Table 2 runs. Oracle selects using future targets.

Agreement and benefit. Agreement–utility Spearman correlations are −0.003 to 0.054. On Weather, the Agreement rule accepts 99.07% and operates near Always-on, whereas Gate accepts 75.36% and achieves lower mean MSE than both proxies. These compare policies at their validation-selected acceptance rates. In validation-defined 3×3 agreement–residual strata, Weather's high-residual groups have positive mean utility across all agreement levels and seeds, including low agreement. Thus low agreement alone is not a reason to discard a candidate; the decoded correction's effect is the relevant acceptance criterion.

Utility-based acceptance. ETTm2 illustrates why group averages and individual decisions need to be distinguished. Its group-mean candidate utilities are negative, yet Oracle can still select beneficial individuals. Applying every correction increases error; rejecting every candidate leaves useful corrections unexploited. On Weather, residual–utility correlations are positive (0.083–0.100), but on ETTm2 they are negative (−0.112 to −0.095), while residual–gate-probability correlations remain positive (0.418–0.462 and 0.101–0.166). Residual size therefore needs to be interpreted together with the forecasting context, rather than used as an acceptance score by itself.

For decoded increment d=Y^+−Y^Xd=\hat Y^+-\hat Y_X and numerical error u=Y−Y^Xu=Y-\hat Y_X, U=(2u⊤d−∥d∥22)/HU=(2u^\top d-\|d\|_2^2)/H. Positive utility depends on the increment's direction and magnitude relative to the forecast error. CIRF consequently uses the conditional residual to construct a candidate and learns acceptance from loss reduction (Huber training labels; MSE in this diagnosis).

On ETTm2, Gate accepts 0.23% and avoids most of the error increase from mandatory correction. Its MSE remains 0.341×10−30.341\times10^{-3} above Numerical and above both proxies; Oracle's lower error marks the remaining selection opportunity. These results support optional correction and identify where its acceptance rule can improve.

Q2. Shared versus independent perturbations

Changing whether the two views share noise has little effect on either model's EA or routing in this setting. CIRF-R* retains its lower observed EA under both noise conditions, so this behavior is not restricted to identical perturbations in the two histories.

We fixed each checkpoint and validation threshold on ETTh1-96, using statistics-based prompts and Gaussian scale 0.20 in train-standardized units. Numerical noisy histories were identical across conditions; only prompt-history noise changed from shared to independently drawn at the same strength and distribution. EA uses each checkpoint's own numerical branch as reference.

ModelNoiseMSE/MAEEA ×10³AcceptFlip
CIRFClean0.378/0.393011.4030
CIRFShared0.385/0.4003.11410.2136.272
CIRFIndependent0.385/0.4003.12210.2256.273
CIRF-R*Clean0.389/0.40408.0290
CIRF-R*Shared0.396/0.4120.7838.2814.023
CIRF-R*Independent0.396/0.4120.7828.2744.030

Means over three training seeds using one fixed paired noise set across checkpoints. Accept/Flip are percentages; Flip is relative to clean. CIRF-R* is a supplementary robust-training batch with a different numerical backbone and additional noisy-forecast supervision, reported separately from Table 4.

Shared-minus-independent EA is (−8.01±6.95)×10−6(−8.01\pm6.95)\times10^{-6} for CIRF and (0.65±2.09)×10−6(0.65\pm2.09)\times10^{-6} for CIRF-R* (mean ± sample SD of paired training-seed differences). Switching noise dependence changes 0.0701% and 0.0342% of gate decisions, respectively. These changes are small relative to each model's EA. For these checkpoints and paired draws, the measured sensitivity and routing behavior is therefore largely preserved when the views receive independent perturbations.

Thank you for running both experiments. The Q1 results strengthen the paper’s central insight that representation agreement is a poor proxy for correction utility. The Q2 results clarify that sharing the noise had little effect in the tested setting. The current gate still leaves room for improvement in selecting useful corrections, so I’ll keep my score at 3 and make the case for acceptance to the other reviewers and the AC during discussion.

Reviewer_zvRKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the pseudo-multimodality problem in LLM-aided time series forecasting. The authors argue that when numerical representations and textual prompts are derived from identical historical sequences, distinct representations do not add independent observational information. Hence, cross-view representation consistency cannot validate performance gains brought by the language branch.

To tackle this challenge, the CIRF framework is proposed. It generates base predictions via a numerical backbone, estimates prompt representations from numerical features, converts representation residuals into candidate prediction corrections, and uses a loss-supervised binary gate to selectively enable these corrections. CIRF-R extends this design with paired perturbation training and excess sensitivity regularization to mitigate extra sensitivity under input corruption.

Evaluations are conducted on nine public time-series datasets, and CIRF obtains the best average MSE and MAE across competitors on four datasets. Controlled experiments verify that selective correction surpasses unconditional correction. Further ablations show pretrained LLMs and sample-aligned prompts fail to deliver consistent forecasting improvements.

While the problem is clearly defined and several control experiments are insightful, current evidence cannot fully confirm the uniqueness of conditional representation residuals or justify the full CIRF architecture over simpler alternatives.

Critical strengths
  1. The research question carries clear scientific significance.

This paper properly distinguishes between two concepts: differing representation forms and independent information sources. Instead of treating consistency between numerical and linguistic representations as evidence of complementarity, the authors propose that the utility of auxiliary representations should be evaluated by the actual loss reduction relative to full numerical forecasting.

This perspective is practically relevant to many existing LLM-based time series forecasting methods that build textual prompts from sequence statistics. The work does not arbitrarily dismiss the value of linguistic representations. It acknowledges that pretrained inductive biases and alternative representation learning capacities can still yield conditional predictive benefits, leading to a reasonable problem formulation.

  1. The experimental analysis is candid, and some controlled comparisons are informative.

Table 2 compares pure numerical forecasting, unconditional correction, learned gating and the Oracle using the same model checkpoint. This setup directly validates whether selective correction outperforms forced fusion, eliminating confounding effects from separate training initializations.

Moreover, Table 5 and Table 10 include controls on pretrained weights, prompt matching and lightweight architecture replacements. The authors do not shy away from results unfavorable to the full language branch. They explicitly admit that under the current experimental setup, stable additional benefits from language pretraining and sample-matched prompts cannot be verified.

Reporting such negative findings is a major strength of this paper. The authors also provide relatively complete descriptions of model architecture, training configurations and supplementary experiments in the appendix, with overall clear presentation.

Critical weaknesses

Critical Weakness

1: The independent technical contribution of conditional representation residuals is not sufficiently validated This paper proposes to predict textual representations from numerical ones and leverage the resulting conditional residuals to construct candidate forecast corrections. Though this design is reasonable, existing experiments fail to fully demonstrate that such conditional residuals can deliver stable and unique predictive gains over directly‑using textual representations, numerical representations or simple statistical features.

Table 3 conducts ablation studies on conditional incremental decomposition and reconstruction constraints. However, it only reports results under one random seed and a single forecasting setting, with marginal performance improvements. On the other hand, Table 10 shows that the lightweight Stats‑MLP achieves comparable performance to full CIRF across two Weather forecasting horizons, while Table 11 reveals a substantial computational‑cost gap between them.

These observations do not invalidate the conditional correction mechanism. Nevertheless, it remains unclear whether the full‑fledged language branch and conditional‑residual construction are truly necessary. In particular, attribution experiments with lightweight models under identical protocols are missing for key datasets where CIRF obtains major performance gains, such as Solar‑Energy and Electricity.

Therefore, I recommend the authors further illustrate the advantages of conditional residuals against simpler correction architectures and provide supporting evidence on representative datasets with prominent performance improvements.

Critical Weakness 2:

Room for improvement regarding main‑experiment competitiveness evaluation and partial mechanism attribution This paper reports competitive results across nine public datasets, yet the coverage of baseline comparisons is incomplete. For instance, T3Time (AAAI 2026) is discussed in Related Work but excluded from main‑experiment comparisons. Given its close relevance to this work in terms of language‑aided forecasting and residual fusion, the authors are suggested to elaborate their baseline‑selection rationale and clarify the performance comparison under unified experimental protocols.

Moreover, Table 2 convincingly demonstrates the merit of selective correction over unconditional correction, yet the practical incremental gains vary greatly across datasets. Further analysis on how well the learned gate identifies beneficial corrections would help clarify its success‑ and failure‑mode conditions.

For CIRF‑R, since both perturbation augmentation and sensitivity regularization are enabled simultaneously in existing experiments, the individual contribution of the proposed regularization towards robustness enhancement cannot be isolated. The authors could add a control experiment with perturbation augmentation alone, or supply analyses that disentangle the effects of these two components. These concerns mainly affect the interpretation of methodological contributions and experimental outcomes, rather than indicating definite technical bugs in the proposed algorithm.

Critical Weakness 3:

On the trade-off between inference efficiency and predictive gains:

Although the authors include inference cost analysis in Appendix Table 11, the justification for the extra computational overhead of full CIRF requires further discussion. On the Weather dataset, full CIRF takes roughly 114–130 ms per batch for inference, while Stats-MLP only costs around 2 ms per batch — a 57–66× difference. Nevertheless, Table 10 shows Stats-MLP achieves slightly lower average MSE than full CIRF across two prediction horizons. This indicates the experiments fail to demonstrate predictive benefits that can justify the heavy computational cost introduced by the large language encoder.

Moreover, according to the computation pipeline in Figure 2, the utility gate needs to compute textual representations and conditional residuals before deciding whether to adopt corrections. Even with a low gate activation rate, the model still has to run language encoding and residual calculation. Rejecting corrections therefore does not reduce inference latency automatically.

I suggest the authors further discuss this accuracy-efficiency trade-off. In particular, they should compare predictive performance and inference cost between full CIRF and lightweight alternative architectures on datasets with the main performance improvements, to better validate the practical necessity of the full model design.

Questions for authors

Q1. On the necessity of the conditional residual mechanism

Could the authors conduct comparisons across Solar-Energy, Electricity and Weather with identical training budgets, validation protocols and random seeds, covering full CIRF, pure numerical forecasting, Stats-MLP correction, and direct residual correction without conditional representation prediction? This comparison is critical to identify whether the performance gains in the main experiments stem from conditional residual construction, linguistic representations, or simply a general increase in model capacity. Q2. On recent strong baselines and experimental comparability

T3Time is discussed in the paper but excluded from the main result tables. Could the authors justify this choice and report the performance of CIRF against recent methods such as T3Time under identical data splits, input lengths and evaluation protocols? Besides, please provide multi-seed statistics for datasets with major performance improvements, and further clarify protocol consistency between locally reproduced results and baseline numbers cited from prior literature.

Q3. On the effectiveness of gating decisions

As shown in Table 2, the gate activation rate is extremely low for some ETT setups while much higher on Weather. Could the authors report forecasting errors under different activation rates, together with how well the learned gate identifies beneficial corrections relative to the Oracle? This helps verify whether the gate learns to pick corrections that bring positive predictive benefits, rather than merely preserving the performance of the numerical branch by rejecting corrections.

Q4. On the standalone contribution of robustness regularization

CIRF-R adopts paired perturbation training and excess sensitivity regularization jointly. Could the authors add a control variant using only paired perturbation training without sensitivity regularization, and report gate activation rates under perturbed inputs?

Moreover, Equation (15) in the appendix directly computes the cosine similarity between changes in numerical representations and textual representations. Please explain why these two separately learned representation spaces share sufficient coordinate compatibility, and what practical contribution this cross-representation similarity makes to the final robustness gains.

3 comments

Response to Reviewer zvRK

Thank you for recognizing our formulation and informative controls.

Q1. Conditional residuals

Relative to Direct-prompt, residualization helps when removing predictable overlap leaves a useful forecast increment.

L=96, seed 2021; MSE/MAE. Training budgets and validation rules are matched within each dataset.

Weather:

MethodH=96H=336
CIRF0.170/0.2080.279/0.297
Numeric-only0.173/0.2100.276/0.293
Stats-MLP0.171/0.2110.277/0.300
Direct-prompt0.172/0.2100.276/0.293

Solar-Energy (H=96):

MethodMSE/MAEAcceptance
CIRF0.204/0.24591.64%
Numeric-only0.230/0.2640.00%
Stats-MLP0.200/0.24277.77%
Direct-prompt0.214/0.25417.08%

Full and Direct-prompt share the numerical backbone and GPT-2 encoder, but decode zP−Cϕ(zX)z_P-C_\phi(z_X) with reconstruction/orthogonality losses versus zPz_P. Separating overlap before decoding is intended to ease learning the increment. Solar's lower error and greater correction use fit this rationale; Weather favors Full at H=96 but reverses at H=336.

Stats-MLP uses the same four statistics with conditional prediction, residual decoding and gating. Its competitive Weather and best Solar errors support a lightweight instance of this framework; the Solar gain does not require GPT-2. Both use the same sample observations.

Numeric-only measures the full correction system's effect. Full/Direct-prompt fixes GPT-2 size but changes the predictor and auxiliary losses together, testing the residual package. These are single-seed, not capacity-matched controls. Given time/compute limits, we will complete Electricity-96/336 and Solar-336, then seed repetitions and capacity-matched numerical corrections.

Q2. Recent baselines and mechanism

Thank you for suggesting comparisons with recent methods such as T3Time. Table 1 includes SE-LLM (ICLR 2026), a recent LLM-based baseline. Our core question is which prompts derived from the same history improve a complete numerical forecast and which candidate corrections should be rejected. Unlike fusion before decoding, CIRF separates correction construction from loss-supervised acceptance. With candidates fixed, selective routing reduces error relative to applying every correction in all six settings (Table 2). In the Weather configuration, CIRF uses six GPT-2 blocks, about 83M total/43M trainable parameters.

Q3. Useful selection or protective rejection?

Weather-96 selects useful corrections; ETTm2-336 mainly protects the numerical default. We fix each checkpoint's numerical forecast and candidates. Numerical is its own branch; Q1's Numeric-only is independently trained. MSEs average three seeds; precision/recall pool window-variable decisions. Benefit means lower full-horizon MSE than Numerical.

SettingNumerical MSEGate MSEAlways-on MSEAcceptancePrecisionRecall
Weather-960.1760.1710.17375.36%54.6%80.3%
ETTm2-3360.3030.3030.3130.23%31.5%0.2%

Weather-96 improves on both endpoints. ETTm2-336 avoids most always-on harm, with MSE 0.341×10⁻³ above Numerical. These are 0%, validation-selected and 100% acceptance policies; the full threshold sweep remains pending.

The oracle uses future targets; Gate predicts utility from history. The oracle gap is dominated by harmful acceptance on Weather-336 and missed beneficial corrections on ETTm2-336.

Q4. Regularization and cosine factor

Sensitivity regularization lowers extra output amplification (EA) beyond paired perturbation training. Paired training supervises clean/noisy forecasts against targets; the added penalty constrains correction-induced displacement beyond the numerical branch.

ETTh1, three-seed means; Gaussian train/test scales 0.10/0.20. Entries: paired training / paired training + sensitivity regularization (weight 0.1).

HNoisy acceptance (%)EA (×10⁻³)Noisy MSE
969.796/8.2810.891/0.7830.396/0.396
3365.151/0.2790.581/0.0200.502/0.508

Lower EA and acceptance are consistent with more conservative correction use. At H=96, noisy MSE falls by 0.262×10⁻³. At H=336, clean and noisy MSE rise in all three seeds while the noise-induced increment is nearly unchanged. Thus sensitivity control has an accuracy tradeoff at H=336.

Eq. (15) multiplies ρXρP+ϵ\sqrt{\rho^X\rho^P+\epsilon} by [cos⁡(dX,dP)]+[\cos(d^X,d^P)]_+: at fixed magnitudes, positive agreement raises the weight; nonpositive cosine gives zero. It weights the training penalty. Joint forecasting, orthogonality and gating couple the d-dimensional coordinates; cosine is a learned-coordinate heuristic without an invariance guarantee. Its isolated benefit remains pending. A cosine-only control will set it to 1 on ETTh1-96/336, retaining paired training, three matched seeds, initialization, perturbations, loss weights and validation selection; we will report EA/noisy MSE.

Q1 follow-up: Solar-Energy at H=336

We have now completed the Solar-Energy comparison at H=336, where CIRF achieves the lowest MSE and MAE among the four variants. H=96 results are included for reference (MSE / MAE).

MethodH=96H=336
CIRF0.204 / 0.2450.252 / 0.282
Numeric-only0.230 / 0.2640.289 / 0.296
Stats-MLP0.200 / 0.2420.254 / 0.283
Direct-prompt0.214 / 0.2540.273 / 0.287

At H=336, CIRF reduces MSE by 12.8% versus Numeric-only and 7.5% versus Direct-prompt. CIRF and Direct-prompt use the same numerical backbone and GPT-2 architectures; their comparison supports the conditional-residual construction together with its auxiliary objectives. Removing the component predictable from numerical features is intended to reduce the correction head's burden of separating overlapping and complementary representations, and the observed gain is consistent with this rationale.

Stats-MLP performs best at H=96 and remains close to CIRF at H=336. Both residual variants outperform Numeric-only and Direct-prompt at both horizons, supporting the conditional correction framework with either encoder.

Thank you for getting back to us. If the additional results and clarifications in our rebuttal have changed your assessment, would you consider revisiting your ratings? Please let us know if there’s anything else we can clarify.

Reviewer_qNpoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces pseudo-multimodality in LLM-based time-series forecasting: when textual prompts are generated from numerical histories, the two pathways share the same observations. Consequently, representational agreement does not establish that the language pathway provides additional predictive value. The proposed CIRF constructs a prompt-derived correction to a complete numerical forecast and uses a utility-supervised gate to decide whether to apply it. CIRF-R additionally addresses sensitivity to shared-source perturbations. Experiments include forecasting benchmarks, routing comparisons, and controlled replacements of the language pathway.

Critical strengths
  • Clear conceptual contribution. Pseudo-multimodality distinguishes history-derived prompts from independent textual observations and explains why representational agreement does not establish predictive benefit.
  • Strong controlled analysis. Randomized-model, unmatched-prompt, and Stats-MLP experiments directly test the contribution of language pretraining and prompt content, with transparent reporting of unfavorable findings.
Critical weaknesses
  • Unclear justification for the language pathway. Numerical-only and much cheaper Stats-MLP alternatives achieve comparable or better results, weakening the full model’s practical value.
  • Limited evidence for effective selective correction. Routing often merely matches the numerical default, remains substantially below the oracle, and relies on small improvements without sufficient uncertainty analysis.
Questions for authors
  1. Under what conditions does the language pathway offer a meaningful advantage over numerical-only forecasting or Stats-MLP?
  2. Can you provide uncertainty estimates and paired comparisons for the routing and component-ablation results?
  3. What explains the gap between learned selection and oracle selection?
  4. Does sensitivity regularization improve robustness beyond matched paired perturbation training?
  5. How are protocol differences controlled when combining published and locally reproduced baseline results?
1 comment

Thank you for getting back to us. If the additional results and clarifications in our rebuttal have changed your assessment, would you consider revisiting your ratings? Please let us know if there’s anything else we can clarify.

Comments on the paper

1 comment

Dear Reviewers,

As the author–reviewer discussion period is closing very shortly, I would like to gently remind those who have not yet responded to review the authors' rebuttal and share your feedback.

Thank you very much for your time and effort!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.