acceptodds
Under review as a conference paper at ICLR 2027

QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities

22 views

Abstract

Existing multimodal time series foundation models (TSFMs) typically model heterogeneous modalities through largely shared mechanisms, overlooking the distinct forecasting roles of endogenous and exogenous modalities. In this work, we propose **QiYao-M**, a role-aware multimodal TSFM that models the two types of modalities separately. For endogenous modalities, to capture how they evolve along with the underlying temporal dynamics, we introduce an *Endo-Multimodal Predictor* and *Endo-Multimodal Supervision* to explicitly learn their evolution from history to the future. For exogenous modalities, to generalize across domains and across various modality types and numbers under the scarcity of exo-multimodal pretraining data, we propose an *Exo-Multimodal Retrieval Enhancer* that enables rapid downstream adaptation without updating the TSFM parameters. We further introduce *Endo-Modality Proxy Training* to train this retrieval module without exogenous multimodal pretraining data. Extensive experiments across unimodal and multimodal benchmarks demonstrate strong forecasting performance in scenarios both with and without exogenous modalities.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Time Series Foundation Model, Multimodal Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qAduOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a multimodal time series foundation model that processes endogenous and and exogeneous modalities differently. For endogenous modalities, the work fuses their frozen CLIP representations, while for exogenous modalities, closest and most similar historical cases are incorporated through a retrieval and ranking mechanism. The two types of features are fused through cross attention. Evaluation across many datasets and benchmarks show promising performance.

Critical strengths
  1. Design seems plausible, and allows including external information from exogenous modalities without updating parameters. The proposed training method seems to also enable using exogenous modalities by using endogenous modalities as a proxy, so there is no need to collect a large amount of aligned multimodal data.

  2. There is a lot of evaluation and experiments showing the improvements of the proposed approach against existing work, including with and without exogenous modalities for a variety of datasets and tasks.

Critical weaknesses
  1. Novelty is incremental. Most of the techniques applied are existing in other works. For example, Time-VLM constructs text and images from time series and incorporate frozen vision-language representations like in this work. Retrieval-based time series forecasting also has numerous precedents (e.g., RAF, TimeRAF, RAFT, etc.). The endogenous modeling of modalities uses a predetermined representation and trains by predicting future embeddings as an auxiliary objective. There are other design choices that could have been made, but are not discussed or compared against, such predicting future statistics or other simpler auxiliary objectives under the same training budget. Using the endogenous modality as a proxy for exogenous training seems to be somewhat of a new idea, but I don't see any ablations for this specific part; I believe this is included as part of the retrieval enhancer, but more experiments are needed to show whether this specific component is actually needed under matched training and data capacity, as well as whether or not the diversity of available modalities impacts this part.

  2. Evaluation has several areas of improvement. First, the baselines for incorporating exogenous modalities need to be strengthened. Figure 6 seems to be some kind of ablation study with different components of the retrieval method removed, but comparisons with existing retrieval-based methods are missing. This is important for establishing novelty as well because it seems with simple retrieval-based mechanisms, the performance gain is already quite high. Second, the performance improvements are largely minor (e.g., ~1% on GIFT-Eval and ~0.5% on Time MASE). Finally, there are no end-to-end efficiency measurements that tell us how much does adding these additional mechanisms impact how long it takes (and therefore if it is even worth it to use the methods proposed in this work).

Questions for authors
  1. Please clarify the core novelty of the approach and provide ample evidence to support. Many of the biggest improvements come from similar mechanisms proposed in prior works and improvements from others seem incremental.

  2. Please clarify and quantify the impact of using endogenous modalities as a proxy during training with all parameters the same (e.g., with several different retrieval mechanisms, same training budget, historical bank).

  3. Please evaluate the proposed CLIP-based supervision against other simpler auxiliary objectives (e.g., future statistics or first/second order differences) under the same training capacity.

  4. Please clarify how the historical bank for each baseline is constructed. The point is to see if the baselines are being evaluated fairly (e.g., they should basically all be the same across all baselines).

  5. Please report performance on datasets excluded entirely from the pretraining phase and provide uncertainty estimates to evaluate the generalizability and robustness of the approach. This work proposes a method that does not require fine-tuning to new datasets (e.g., zero-shot), but pretrains on parts of datasets that do not satisfy this "zero-shot" requirement (e.g., GIFT-Eval).

6 comments
AuthorsOpenReview ↗

Response to Reviewer qAdu (Part I)

We sincerely thank Reviewer qAdu for the detailed feedback and for recognizing the plausibility of our design and the breadth of our evaluation. We address the novelty concerns through controlled comparisons, and strengthen the evaluation with retrieval baselines, matched-budget ablations, dataset-exclusion tests, and efficiency measurements below.


Response to Q1 & W1 (Part 1): Clarification on Core Novelty and Empirical Evidence

We thank the reviewer for this constructive feedback. While individual building blocks (e.g., VLM encoders or retrieval) have precedents, equating QiYao-M to prior task-specific or unimodal methods overlooks our paradigm-level formulation and dedicated technical innovations, which we clarify and support with controlled empirical evidence below.

1. Overarching Paradigm

Unlike existing multimodal TSFMs (e.g., ChatTime, Aurora) that process all modalities through role-agnostic shared mechanisms, QiYao-M is the first role-aware multimodal TSFM that explicitly decouples endogenous temporal evolution from exogenous contextual adaptation (Figure 1).

2. Endogenous Modeling (Beyond Time-VLM and Simpler Numerical Objectives)

Prior multimodal forecasters such as Time-VLM are task-specific models that only use constructed text and images as historical inputs. In contrast, our endogenous modeling introduces two key innovations for TSFM pretraining:

  • Multimodal Evolutionary Supervision: Rather than supervising only numerical outputs, QiYao-M is the first TSFM to explicitly predict future representations across multiple endogenous modalities and supervise their evolution from history to the future (Endo-Multimodal Predictor & Supervision, validated in Table 2, Rows 2–3).
  • Pre-aligned VLM Representation Space: Directly predicting raw future statistics or differences struggles to generalize across domains due to drastic variations in numerical scales and noise. Supervising future evolution in CLIP’s pre-aligned latent space abstracts these variations into domain-invariant linguistic regimes and 2D geometric patterns while leveraging rich VLM priors.

To further evaluate the effectiveness and necessity of our Endogenous Modeling, we conducted additional comparative experiments. Please refer to our response to Q3 in Response to Reviewer qAdu (Part II) below.

3. Exogenous Modeling (Beyond Prior Retrieval Methods)

Prior retrieval-based forecasters (e.g., RAF, TimeRAF, and RAFT) focus on unimodal time-series lookup, while existing multimodal TSFMs are restricted to fixed modality inputs (primarily text). In contrast, QiYao-M is the first TSFM to support tuning-free adaptation across varying exogenous modality types and numbers, addressing two fundamental bottlenecks:

  • Forecasting-Oriented Multimodal Architecture: We combine modality-specific predictive-utility analysis (Case Analysis) with Modality-Independent Retrieval, dynamically suppressing uninformative modalities and enabling parameter-free adaptation to varying modality types and numbers (Table 2, Rows 3–5).
  • Pretraining via Endo-Modality Proxy Training: To train this retrieval module without requiring scarce large-scale exogenous pretraining corpora, we dynamically sample combinations of endogenous modalities as retrieval proxies during pretraining.

To quantify the contribution of Endo-Modality Proxy Training and assess its necessity, we conducted additional comparative experiments. Please refer to our response to Q2 in Response to Reviewer qAdu (Part II) below.

Together, these innovations yield state-of-the-art performance across 73 unimodal datasets (Figure 5) and 9 multimodal datasets spanning 2 to 4 modalities (Table 1). The controlled comparisons are provided in Appendices B.2–B.3 (Tables 8–9).

AuthorsOpenReview ↗

Response to Reviewer qAdu (Part II)

Response to Q2 & W1 (Part 2): Isolating and Quantifying Endo-Modality Proxy Training

We thank the reviewer for this constructive suggestion. To isolate the impact of Endo-Modality Proxy Training from the retrieval mechanism itself, we keep the backbone parameters, retrieval architecture, historical bank, and Stage 3 training budget identical across the trained retrieval variants. We replace our proxy training with numerical-only retrieval or randomly constructed candidates. All retrieval variants are evaluated using the same downstream external modalities:

Stage 3 Retrieval Strategy (Same Budget & Bank)Multimodal Downstream RetrievalFinMultiTime (MSE)Time-MMD (MSE)
No retrieval (Table 2, Row 3)✗0.6920.588
Randomly constructed proxies✓0.6890.583
Numerical-only retrieval training✓0.6790.574
Endo-Modality Proxy Training (Ours)✓0.6660.567

Conclusion: This result confirms that our proxy training strategy is essential and contributes significantly beyond retrieval alone.

To isolate the contribution of Endo-Modality Proxy Training beyond retrieval alone, we have added this controlled ablation to Appendix B.3 (Table 9).


Response to Q3 & W1 (Part 3): Evaluating CLIP-Based Endogenous Modeling

We thank the reviewer for raising this critical point.

Directly predicting raw future statistics or differences struggles to generalize across domains due to drastic variations in numerical scales and noise. In contrast, supervising future evolution in CLIP’s pre-aligned latent space abstracts these variations into domain-invariant linguistic regimes and 2D geometric patterns while leveraging rich VLM priors.

To empirically evaluate our CLIP-based endogenous modeling, we compare it with lightweight numerical encoders and prediction heads trained to fuse and predict the same statistics (Appendix A.2) and first- and second-order differences, using the same training budget and matched trainable parameter counts:

Method (Same Information & Training Budget)GIFT-Eval MASEGIFT-Eval CRPSTIME MASETIME CRPS
Simpler Numerical Objectives (Stats & 1st/2nd Diffs)0.6800.4670.6480.544
CLIP-based Endogenous Modeling (Ours)0.6710.4610.6390.537

Conclusion: CLIP-based endogenous modeling achieves lower MASE and CRPS on both benchmarks than the numerical feature baseline, demonstrating the effectiveness of our multimodal formulation.

To demonstrate the benefit of CLIP-based endogenous modeling over numerical features, we have added this controlled comparison to Appendix B.2 (Table 8).

AuthorsOpenReview ↗

Response to Reviewer qAdu (Part III)

Response to Q4 & W2 (Part 1): Historical Bank Construction and Fair Comparison with Baselines

1. Historical Bank Construction & Strict Data Parity (Q4)

  • Training-Only Bank Construction: For each downstream dataset, the historical bank is constructed exclusively from 20% of sliding windows sampled from the official training split (Appendix B.4). Validation and test splits are strictly excluded from bank construction and modality weight calibration, guaranteeing zero data leakage.
  • Strict Data Parity Across Two Baseline Categories:
    1. Zero-Shot TSFMs (0% training data): Standard zero-shot TSFMs (e.g., Chronos-2, Aurora) do not use the downstream training split. For a strictly bank-free comparison, QiYao-M (w/o Exo.) uses no retrieval bank and already outperforms them across most benchmarks.
    2. Full-Shot End-to-End Models (100% training data): Supervised multimodal baselines (e.g., Time-VLM, TATS) update all parameters on 100% of the official training split, whereas QiYao-M (w/ Exo.) only accesses 20% of the training split as a non-parametric bank without updating parameters.

2. Controlled Comparison Across All Four Baseline Categories (W2 & Q4)

To ensure a strictly fair comparison under controlled data access, we evaluate our model against four baseline categories: (i) zero-shot TSFMs (Chronos-2 (ZS), Aurora (ZS)), (ii) full-shot end-to-end multimodal models (Time-VLM, TATS), (iii) few-shot fine-tuned TSFMs updated on the exact same 20% training subset (Chronos-2 (FT), Aurora (FT)), and (iv) unimodal RAG baselines querying the identical 20% historical bank, including full-shot trained RAFT and tuning-free RAF. We report MSE across four representative datasets spanning 2 to 4 modalities:

ModelParadigmParam. UpdateHistorical Data / Bank AccessAgriculture (2-Mod)Health (2-Mod)Tianchi (3-Mod)SP500 (4-Mod)
Chronos-2 (ZS)(i) Zero-Shot TSFM✗None (0%)0.2381.0541.4530.717
Aurora (ZS)(i) Zero-Shot TSFM✗None (0%)0.2721.5531.4030.862
QiYao-M (w/o Exo.)Zero-Shot TSFM (Ours)✗None (0%)0.1990.9151.4230.694
Time-VLM(ii) Full-Shot End-to-End✓100% train split0.2371.5651.3680.756
TATS(ii) Full-Shot End-to-End✓100% train split0.2151.3561.3740.665
Chronos-2 (FT)(iii) Few-Shot Fine-Tuned TSFM✓Identical 20% train subset0.2260.9821.4120.708
Aurora (FT)(iii) Few-Shot Fine-Tuned TSFM✓Identical 20% train subset0.2411.2151.3790.748
RAFT(iv) Full-Shot Unimodal RAG✓100% train + Identical 20% bank0.2311.3841.3580.682
RAF(iv) Tuning-Free Unimodal RAG✗Identical 20% bank0.2240.9761.3960.704
QiYao-M (w/ Exo.)Tuning-Free Multimodal RAG (Ours)✗Identical 20% bank0.1740.9011.3020.660

Conclusion: Under strictly fair and identical bank access, QiYao-M (w/ Exo.) achieves the lowest MSE across all datasets without updating parameters, consistently outperforming all four baseline categories. This confirms that our multimodal retrieval mechanism effectively exploits heterogeneous external information for tuning-free adaptation.

To strengthen the baseline evaluation and make data access explicit, we have added fine-tuning and retrieval baselines alongside their training and bank-access settings in Appendix B.4 (Table 10).

Reviewer_r28jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QiYao-M, a multimodal time series foundation model that explicitly distinguishes endogenous and exogenous modalities. For endogenous modalities, the authors introduce patch-level multimodal fusion and auxiliary representation prediction. For exogenous modalities, they develop a retrieval-based enhancement mechanism trained using endogenous modalities as proxies.

Critical strengths
  1. Well-motivated framework. The distinction between endogenous and exogenous modalities provides a meaningful perspective for designing multimodal forecasting models with different information sources.

  2. Strong empirical performance. The method achieves competitive results across diverse forecasting benchmarks. The component-level ablations also provide useful evidence for the proposed architecture.

Critical weaknesses
  1. Limited justification for endogenous multimodal modeling. The endogenous text and images are deterministic transformations of numerical time series and do not introduce additional information. While the ablations demonstrate improvements from multimodal fusion and supervision, comparisons against simpler numerical feature encoders are missing. Consequently, it remains unclear whether the improvements arise specifically from multimodal representations or from additional engineered features.

  2. Insufficient validation of proxy-based retrieval generalization. A central contribution is training the exogenous retrieval module using endogenous modality proxies. However, the current experiments do not adequately isolate the effectiveness of this strategy from retrieval itself. Furthermore, the method requires historical reference cases with observed future outcomes, while its retrieval and inference overhead is not sufficiently quantified. These limitations weaken the evidence for efficient, generalizable downstream adaptation.

Questions for authors
  1. Can the authors compare CLIP-based endogenous modeling with lightweight numerical feature encoders using the same statistical information?

  2. How does endogenous proxy training compare with alternative retrieval training strategies, such as numerical-only or randomly constructed proxies?

  3. What are the inference latency and memory costs of multimodal encoding and retrieval, relative to the numerical-only backbone?

3 comments

We sincerely thank Reviewer r28j for the detailed feedback and for recognizing the motivation and empirical performance of QiYao-M. We address your concerns through comparisons with lightweight numerical encoders, controlled proxy-training ablations, and end-to-end efficiency measurements below.


Response to Q1 & W1: Clarification on endogenous information and numerical feature controls

We thank the reviewer for this insightful comment. Below, we clarify the theoretical advantages of our endogenous multimodal formulation and provide a controlled empirical comparison against lightweight numerical feature encoders.

1. Why endogenous modalities?

Although endogenous text and images are deterministic transformations that do not add external data, organizing the same statistical information into endogenous text and images—rather than feeding additional engineered features into lightweight numerical feature encoders—holds three key advantages:

  • Cross-Scenario Generalization: Raw statistics and point-wise differences vary drastically in scale, distribution, and noise across domains, hindering the generalization of lightweight numerical feature encoders. In contrast, transforming the same statistical information into linguistic regime descriptions (Endo-Text) and normalized 2D trajectory shapes (Endo-Image) abstracts away raw numerical discrepancies into domain-invariant semantic concepts and geometric patterns, which generalize far more effectively across diverse forecasting scenarios.

  • Naturally Pre-aligned Representation Space: Unlike lightweight numerical feature encoders that must learn to reconcile heterogeneous statistics and differences from scratch, a pretrained multimodal encoder (e.g., CLIP) provides a naturally pre-aligned, unified latent space where linguistic descriptions and visual trajectory shapes are already aligned, offering smooth and synergistic supervision.

  • Integrating VLM Priors & Backbone Scalability: Engineered numerical features are fixed, low-level heuristics. Bridging time series into text and images allows our model to leverage the rich visual-semantic priors of pretrained VLMs, while providing an interface that can seamlessly plug in stronger VLM backbones to further boost performance.

2. Empirical Comparison with Lightweight Numerical Feature Encoders.

To further validate our endogenous multimodal modeling, we compare CLIP-based endogenous modeling with lightweight numerical feature encoders fed and supervised with the exact same statistical information (Appendix A.2) and first-/second-order differences under identical experimental settings and trainable capacity:

MethodGIFT-Eval MASEGIFT-Eval CRPSTIME MASETIME CRPS
Lightweight numerical feature encoders0.6800.4670.6480.544
CLIP-based endogenous modeling (Ours)0.6710.4610.6390.537

Conclusion: Under matched training budgets and trainable parameter counts, CLIP-based endogenous modeling consistently outperforms numerical feature encoders on both benchmarks, demonstrating its advantage over directly fusing and predicting statistics and differences.

To demonstrate the benefit of CLIP-based endogenous modeling over numerical features, we have added this controlled comparison to Appendix B.2 (Table 8).

Response to Q2 & W2: Isolating proxy training from retrieval

We thank the reviewer for raising this important question. To strictly isolate the contribution of Endo-Modality Proxy Training from the retrieval mechanism itself, we keep the Stage 2 backbone, retrieval architecture, and training budget fixed, changing only the retrieval training strategy during Stage 3 to Numerical-only retrieval training and Randomly constructed candidates. All trained variants are optimized with the same forecasting loss and evaluated using the exact same downstream external modalities:

Stage 3 Retrieval Training StrategyMultimodal Downstream RetrievalFinMultiTime (MSE)Time-MMD (MSE)
No retrieval (Table 2, Row 3)✗0.6920.588
Randomly constructed candidates✓0.6890.583
Numerical-only retrieval training✓0.6790.574
Endo-Modality Proxy Training (Ours)✓0.6660.567

Conclusion: Under the exact same retrieval architecture, Randomly constructed proxies (lacking meaningful similarity structure) and Numerical-only retrieval training (lacking multi-modality exposure) yield only partial gains. Endo-Modality Proxy Training achieves the best performance across both benchmarks, confirming that our proxy strategy contributes significantly beyond the retrieval mechanism itself.

To isolate the contribution of Endo-Modality Proxy Training beyond retrieval alone, we have added this controlled ablation to Appendix B.3 (Table 9). We also detail the historical-bank construction and data-access protocol in Appendix B.4 to make the comparison settings explicit.


Response to Q3 & W2: End-to-end latency and memory costs

To evaluate the end-to-end computational overhead, we report the one-time offline retrieval bank construction time (in seconds) and the total inference time across all test windows and prediction horizons (in minutes, at batch size B=8B=8; - indicates exceeding 3 hours):

DatasetTest WindowsBank Build (s)Numerical-Only (Backbone)QiYao-M (w/o Exo.)QiYao-M (w/ Exo.)Chronos-2Toto-2AuroraChatTime
Agriculture3645.160.070.100.430.170.090.0561.25
Health99214.380.250.311.070.400.240.14116.40
Taobao44,50031.318.2411.6122.3218.6210.7816.20-
SP50020,58078.565.186.3823.659.105.0213.06-

We also report the average peak GPU memory across datasets at B=8B=8:

Metric (B=8B=8)Numerical-Only (Backbone)QiYao-M (w/o Exo.)QiYao-M (w/ Exo.)Chronos-2Toto-2AuroraChatTime
Avg. Peak Memory (GiB)1.061.642.810.741.451.8828.27

Conclusion:

  1. Fast Bank Construction: The one-time offline historical bank construction takes only a few seconds (5–79s), introducing minimal preprocessing overhead.
  2. Encoding & Retrieval Overhead: Relative to the numerical-only backbone, endogenous multimodal modeling adds 23%–43% inference time and 0.58 GiB of average peak GPU memory. With exogenous retrieval, the full model uses 2.81 GiB of average peak GPU memory (+1.75 GiB), supporting improved forecasting accuracy and tuning-free adaptation across diverse modality configurations.

To clarify the computational overhead of our method, we have added bank construction time, end-to-end inference time, and GPU memory measurements to Appendix B.5 (Tables 11–12).


We thank the reviewer again for the constructive feedback. We hope these comparisons address your concerns about endogenous representations, proxy-training generalization, and computational cost. If the explanations and evidence resolve your concerns, we would be grateful if you could reconsider your assessment and rating. We welcome further discussion.

AuthorsOpenReview ↗

Waiting for response

Dear Reviewer,

Since the End of the Rebuttal comes soon, we would like to inquire if our response addresses your primary concerns. If it does, we kindly request that you reconsider the score. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper. Thanks again for dedicating your time to enhancing our paper!

Kind Regards,

Authors

Reviewer_QLxUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposed QiYao-M to model the two modalities (endogenous and exogenous) separately.

  1. The Endo-Multimodal Predictor and Endo-Multimodal Supervision to capture endogenous modalities' evolving along with the underlying temporal dynamics
  2. The Exo-Multimodal Retrieval Enhancer to generalize across domains and across various modality types and numbers under the scarcity of exomultimodal pretraining data
  3. The Endo-Modality Proxy Training to train this retrieval module without exogenous multimodal pretraining data.
Critical strengths

S1: The module design makes sense: for "endogenous modalities", e.g., time-series (or related) modalities, it is the right way to do the prediction ("Endo-Multimodal Predictor") and supervised learning ("Endo-Multimodal Supervision");

Critical weaknesses

W1: The definition of "Endogenous modalities" and "Exogenous modalities" are not clearly explained: although the authors defined "Endogenous modalities" as those intrinsically coupled with the time series and evolve with its temporal dynamics and "exogenous modalities" as those not and coming from external sources with "limited multimodal pretraining data and diverse modalities in both type and number across forecasting scenarios". This definition is to my opinion:

  1. very unintuitive: so in the real deployment, we need to always human-label / human-identify which factors are changing with the time series and what is not?
  2. having vague / grey zone: there are modalities that might look independent of the time series in the short term, but not in the long term; and vice versa, the one changing with the modalities may be because there are confounding effects/factors. How human can accurately distinguish "Endo-Modalities" and "Exo-Modalities"?
  3. from Figure 1, there is no obvious difference between the "Endo-Modalities" and "Exo-Modalities", both have image, text, etc.

W2: I got the main idea of "Exogenous modalities" and its adaptation design: basically, it is trying to retrieve from the historical text/image/etc modalites to find the top time-series targets to help the current time series prediction. But what is the motivation to do so? Do we assume if historically two text info / or two image info are similar, they will have similar time series values?

W3: In the Exo-Multimodal Retrieval Enhancer: it uses all the modalities including "Text" "Image" and "TS History" as query to do retrieval: the cost can be high! And I suspect that using "TS History" to do "TS Target" retrieval contribute the most performance gain, since the cross-modality retrieval e.g, Text --> TS and Image --> TS has not considered the modality gap, so how this cross-modality retrieval can be convincing?

W4: To summarize, in the " Endo-Multimodal" part, it is standard prediction / supervised manner with patch design; the "Exo-Multimodal" part is to retrieval top-k similar cases in the history using the text/image/TS history. The contribution is minor.

Questions for authors

Q1: W1 Q2: Why do "Exo-Modalities" have to be "limited" multimodal pretraining data? if they are not "limited", the "retreival" / "adapation" then doesn't make sense? Q3: In ablation study, how can you do Row 4 setting, that has "retrieval enhancer" but no "case analysis"? In Fig 3b, the Query "Text""Image""TS History" rely on the W_Text etc from Fig 3a, so in the ablation study Row 3: those W_Text etc are randomized?

5 comments

We sincerely thank Reviewer QLxU for the detailed feedback and for recognizing the soundness of our endogenous prediction and supervision design. We appreciate the opportunity to clarify several misunderstandings regarding our core mechanisms and definitions below, supported by controlled experiments and efficiency measurements.


Response to W1 & Q1: Clarification on endogenous and exogenous modalities

Our definitions of endogenous and exogenous modalities appear in Lines 41–43 of the original manuscript, whereas Lines 49–51 (referenced in your comment) describe their forecasting characteristics rather than their definitions. The original definitions are:

  1. “Endogenous modalities, which are derived from the time series itself and provide alternative views of the same underlying temporal dynamics;”
  2. “Exogenous modalities, which originate from external sources and provide additional evidence that may influence future evolution.”

The defining distinction is the source of information (internal code generation vs. external input). Endogenous modalities are automatically generated by code directly from the time series itself, such as line plots (Figure 8) or statistical text descriptions (Figure 7), offering alternative views of that series. Exogenous modalities are externally provided inputs, such as satellite cloud imagery for weather forecasting, that supplement the time series with additional information.

Based on these definitions, we address your questions directly:

  1. No manual annotation is required in deployment: endogenous modalities are automatically constructed by code from the input series, while any externally supplied data are treated as exogenous modalities.
  2. There is no grey zone: since categorization depends solely on information source (internal code generation vs. external input) rather than statistical correlation, short-/long-term dependencies or confounding effects are irrelevant.
  3. Figure 1 shows their origins through arrows, highlighted in blue in the revised version: the time-series signal for endogenous modalities, and other sensors or external sources for exogenous modalities.

To make this distinction explicit in the manuscript, we have revised the Introduction to emphasize information source as the key criterion and updated Figure 1 to label the sources of endogenous and exogenous modalities.


Response to W2: Clarification on retrieval motivation and assumptions

We hope the following explanation helps resolve the misunderstanding about our motivation.

Our motivation is to exploit task-specific associations between exogenous modalities and future responses recorded in historical cases. Similar text or images may identify informative cases, but do not necessarily imply similar future time-series values.

We do not blindly assume that similar exogenous inputs necessarily imply similar future time-series values. Instead, we specifically design Case Analysis to evaluate the predictive utility of each modality: it retrieves other historical cases using that modality and measures how accurately their aggregated futures predict the reference case’s future (Eqs. 4–5). These errors determine the modality’s Forecasting Contribution Weight for retrieval.

The retrieved futures serve as candidate evidence, rather than the final prediction. The Candidate-Aware Enhancer combines them with the current time-series representation through candidate-aware cross-attention (Eqs. 7–8). Thus, our approach evaluates the forecasting relevance of modality similarity on historical data and uses the resulting evidence in the context of the current series.

Response to W3: Retrieval cost, gains, and modality matching

1. Retrieval cost

To evaluate the end-to-end computational overhead, we report the one-time offline retrieval bank construction time (in seconds) and the total inference time across all test windows and prediction horizons (in minutes, at batch size B=8B=8; - indicates exceeding 3 hours):

DatasetTest WindowsBank Build (s)QiYao-M (w/o Exo.)QiYao-M (w/ Exo.)Chronos-2Toto-2AuroraChatTime
Agriculture3645.160.100.430.170.090.0561.25
Health99214.380.311.070.400.240.14116.40
Taobao44,50031.3111.6122.3218.6210.7816.20-
SP50020,58078.566.3823.659.105.0213.06-

We also report the average peak GPU memory across datasets at B=8B=8:

Metric (B=8B=8)QiYao-M (w/o Exo.)QiYao-M (w/ Exo.)Chronos-2Toto-2AuroraChatTime
Avg. Peak Memory (GiB)1.642.810.741.451.8828.27

Conclusion:

  1. Fast Bank Construction: The historical bank requires only 5–79 seconds to construct, keeping one-time preprocessing lightweight.
  2. Practical Inference Overhead: Without exogenous modalities, QiYao-M (w/o Exo.) achieves latency and memory usage comparable to unimodal TSFMs; with exogenous modalities, QiYao-M (w/ Exo.) maintains practical inference efficiency with just 2.81 GiB of average peak GPU memory, while improving forecasting accuracy and enabling tuning-free adaptation across diverse modality configurations.

To clarify the computational overhead of our method, we have added bank construction time, end-to-end inference time, and GPU memory measurements to Appendix B.5 (Tables 11–12).

2. Gains beyond TS-only retrieval

We agree that TS-history retrieval contributes substantially, while external modalities also provide consistent additional gains. Extending the SP500 analysis in Section 4.3 (Figure 6) to four datasets (2 to 4 modalities), adding external modalities consistently improves over TS-only retrieval within our own model:

QiYao-M Retrieval SettingAgriculture (2-Mod)Health (2-Mod)Tianchi (3-Mod)SP500 (4-Mod)
TS-Only Retrieval0.1910.9101.3650.672
All Modalities (QiYao-M w/ Exo.)0.1740.9011.3020.660

3. Clarification on cross-modality retrieval.

We do not perform cross-modality retrieval (i.e., never matching Text or Image directly against TS). Instead, each historical case is a key–value pair (Figure 3), and we use within-modality key matching (Eq. 6) followed by value lookup: Text queries match historical Text keys and Image queries match historical Image keys, after which we simply look up their paired future time-series values. Similarity is computed only within each modality, so a shared cross-modal embedding space is unnecessary. The forecasting utility of these matches is assessed by Case Analysis, as explained in our response to W2.

Response to W4: Clarification on novelty

We would like to clarify that reducing our framework to standard patch prediction and generic top-kk retrieval overlooks both our paradigm-level formulation and dedicated technical designs:

Overarching Paradigm: Unlike existing multimodal TSFMs (e.g., ChatTime, Aurora) that treat all modalities uniformly through role-agnostic shared mechanisms, QiYao-M is the first role-aware multimodal TSFM that explicitly decouples endogenous temporal evolution from exogenous contextual adaptation (Figure 1).

1. Endogenous modeling (Beyond Input Fusion)

While constructing text/images with frozen VLMs (e.g., in task-specific Time-VLM) and supervised prediction are established tools, our endogenous contribution concerns what is predicted and supervised during TSFM pretraining.

Beyond using endogenous modalities merely as historical inputs (as in Time-VLM), QiYao-M is the first TSFM to explicitly predict the future from multiple modality-specific perspectives and impose supervision on each modality. This helps the model learn temporal evolution from complementary multimodal perspectives beyond numerical forecasting alone (Table 2, Rows 2–3, and outperforming Time-VLM across all 9 datasets in Table 1).

Comparison with Simpler Auxiliary Objectives: We also compare our CLIP-based endogenous modeling against simpler numerical objectives that directly fuse and predict the exact same statistics (Appendix A.2) and 1st-/2nd-order differences under the same training budget and capacity:

Method (Same Information & Training Budget)GIFT-Eval MASEGIFT-Eval CRPSTIME MASETIME CRPS
Simpler Numerical Objectives (Stats & 1st/2nd Diffs)0.6800.4670.6480.544
CLIP-based Endogenous Modeling (Ours)0.6710.4610.6390.537

These results demonstrate the advantage of CLIP-based endogenous modeling over directly fusing and predicting numerical statistics and differences. To demonstrate the benefit of CLIP-based endogenous modeling over numerical features, we have added this controlled comparison to Appendix B.2 (Table 8).

2. Exogenous Modeling (Beyond Naive Top-kk Retrieval):

Rather than standard similarity lookup, our exogenous design addresses two fundamental bottlenecks in TSFM adaptation:

  • Architecture: We propose a forecasting-oriented retrieval mechanism that combines empirical predictive-utility weighting (Case Analysis) with Modality-Independent Retrieval, enabling parameter-free adaptation to arbitrary downstream modality types and numbers (Table 2, Rows 3–5).

  • Pretraining (Endo-Modality Proxy Training): To train this retrieval module without requiring scarce large-scale exogenous pretraining corpora, we dynamically sample combinations of endogenous modalities as retrieval proxies during pretraining.

Isolating Proxy Training from Retrieval: Keeping the backbone, retrieval architecture, historical bank, and Stage 3 training budget identical, our proxy training consistently outperforms alternative retrieval training strategies (Appendix B.3, Table 9):

Stage 3 Retrieval Strategy (Same Budget & Bank)FinMultiTime (MSE)Time-MMD (MSE)
No retrieval (Table 2, Row 3)0.6920.588
Randomly constructed candidates0.6890.583
Numerical-only retrieval0.6790.574
Endo-Modality Proxy Training (Ours)0.6660.567

Together, these innovations yield state-of-the-art performance across 73 unimodal datasets (Figure 5) and 9 multimodal datasets spanning 2–4 modalities (Table 1, Table 2 Rows 3–5).

Comments on the paper

1 comment

Dear Reviewer r28j,

Thank you for your prompt feedback and for recognizing that our experiments address Q1 and Q2 well. We gladly clarify our experimental protocol and the reasons behind the observed trends:

1. Regarding Multiple Experimental Trials and Random Seeds

We clarify that our experiments were conducted using a single fixed random seed rather than multiple training trials, for three reasons:

  1. Large-Scale Pretraining Stability: During pretraining, the model is exposed to millions of samples and undergoes extensive stochastic sampling across batches and proxy combinations. Prior studies on large-scale pretraining [1] and recent time series foundation models [2, 3] have shown that at this data scale, the optimization trajectory is highly stable, and a single seed has negligible impact on the final converged performance.

  2. Standard Protocol & Deterministic Downstream Evaluation: Following standard practice for TSFMs (e.g., Chronos-2, Toto-2), pretraining and evaluations are performed with a single seed. Crucially, our downstream evaluation involves no stochastic processes: both our template-based endogenous modality construction (Appendix A.2) and historical case retrieval are strictly deterministic at inference time.

  3. Time and Computational Constraints: Although we did not need to re-pretrain all stages from scratch, re-pretraining even a single stage on millions of multimodal samples remains computationally expensive and time-consuming. Within the limited rebuttal timeframe, running multiple trials per variant was not feasible.

2. Why the Results Align with Intuition

Because all variants within each ablation start from the exact same pretrained checkpoint, the step-by-step improvements directly reflect how each design overcomes the limitations of the simpler variants:

  • In Q1 (Endogenous Modeling): Lightweight numerical encoders must learn to reconcile heterogeneous statistics and differences from scratch, making them sensitive to cross-domain scale variations. In contrast, our CLIP-based modeling maps these statistics into a pre-aligned semantic and visual latent space, naturally yielding better generalization.

  • In Q2 (Retrieval Training): Starting from the same frozen Stage 2 backbone (No retrieval), Randomly constructed candidates lack meaningful similarity structure and only marginally activate the retrieval head; Numerical-only retrieval captures temporal similarity but lacks exposure to heterogeneous modalities; finally, Endo-Modality Proxy Training (Ours) explicitly trains the retriever to weight and aggregate diverse modality combinations, endowing the model with much stronger generalization across diverse downstream modalities and thus achieving the best performance.


We hope these details clarify our experimental setup and fully address your concerns. Thank you again for your time and constructive engagement!

[1] Kaplan et al., "Scaling Laws for Neural Language Models," arXiv 2020.

[2] Das et al., "A decoder-only foundation model for time-series forecasting," ICML 2024.

[3] Ansari et al., "Chronos-2: From univariate to universal forecasting," arXiv 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.