- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents a multimodal time series foundation model that processes endogenous and and exogeneous modalities differently. For endogenous modalities, the work fuses their frozen CLIP representations, while for exogenous modalities, closest and most similar historical cases are incorporated through a retrieval and ranking mechanism. The two types of features are fused through cross attention. Evaluation across many datasets and benchmarks show promising performance.
Critical strengths
-
Design seems plausible, and allows including external information from exogenous modalities without updating parameters. The proposed training method seems to also enable using exogenous modalities by using endogenous modalities as a proxy, so there is no need to collect a large amount of aligned multimodal data.
-
There is a lot of evaluation and experiments showing the improvements of the proposed approach against existing work, including with and without exogenous modalities for a variety of datasets and tasks.
Critical weaknesses
-
Novelty is incremental. Most of the techniques applied are existing in other works. For example, Time-VLM constructs text and images from time series and incorporate frozen vision-language representations like in this work. Retrieval-based time series forecasting also has numerous precedents (e.g., RAF, TimeRAF, RAFT, etc.). The endogenous modeling of modalities uses a predetermined representation and trains by predicting future embeddings as an auxiliary objective. There are other design choices that could have been made, but are not discussed or compared against, such predicting future statistics or other simpler auxiliary objectives under the same training budget. Using the endogenous modality as a proxy for exogenous training seems to be somewhat of a new idea, but I don't see any ablations for this specific part; I believe this is included as part of the retrieval enhancer, but more experiments are needed to show whether this specific component is actually needed under matched training and data capacity, as well as whether or not the diversity of available modalities impacts this part.
-
Evaluation has several areas of improvement. First, the baselines for incorporating exogenous modalities need to be strengthened. Figure 6 seems to be some kind of ablation study with different components of the retrieval method removed, but comparisons with existing retrieval-based methods are missing. This is important for establishing novelty as well because it seems with simple retrieval-based mechanisms, the performance gain is already quite high. Second, the performance improvements are largely minor (e.g., ~1% on GIFT-Eval and ~0.5% on Time MASE). Finally, there are no end-to-end efficiency measurements that tell us how much does adding these additional mechanisms impact how long it takes (and therefore if it is even worth it to use the methods proposed in this work).
Questions for authors
-
Please clarify the core novelty of the approach and provide ample evidence to support. Many of the biggest improvements come from similar mechanisms proposed in prior works and improvements from others seem incremental.
-
Please clarify and quantify the impact of using endogenous modalities as a proxy during training with all parameters the same (e.g., with several different retrieval mechanisms, same training budget, historical bank).
-
Please evaluate the proposed CLIP-based supervision against other simpler auxiliary objectives (e.g., future statistics or first/second order differences) under the same training capacity.
-
Please clarify how the historical bank for each baseline is constructed. The point is to see if the baselines are being evaluated fairly (e.g., they should basically all be the same across all baselines).
-
Please report performance on datasets excluded entirely from the pretraining phase and provide uncertainty estimates to evaluate the generalizability and robustness of the approach. This work proposes a method that does not require fine-tuning to new datasets (e.g., zero-shot), but pretrains on parts of datasets that do not satisfy this "zero-shot" requirement (e.g., GIFT-Eval).
6 comments
Response to Reviewer qAdu (Part II)
Response to Q2 & W1 (Part 2): Isolating and Quantifying Endo-Modality Proxy Training
We thank the reviewer for this constructive suggestion. To isolate the impact of Endo-Modality Proxy Training from the retrieval mechanism itself, we keep the backbone parameters, retrieval architecture, historical bank, and Stage 3 training budget identical across the trained retrieval variants. We replace our proxy training with numerical-only retrieval or randomly constructed candidates. All retrieval variants are evaluated using the same downstream external modalities:
| Stage 3 Retrieval Strategy (Same Budget & Bank) | Multimodal Downstream Retrieval | FinMultiTime (MSE) | Time-MMD (MSE) |
|---|---|---|---|
| No retrieval (Table 2, Row 3) | ✗ | 0.692 | 0.588 |
| Randomly constructed proxies | ✓ | 0.689 | 0.583 |
| Numerical-only retrieval training | ✓ | 0.679 | 0.574 |
| Endo-Modality Proxy Training (Ours) | ✓ | 0.666 | 0.567 |
Conclusion: This result confirms that our proxy training strategy is essential and contributes significantly beyond retrieval alone.
To isolate the contribution of Endo-Modality Proxy Training beyond retrieval alone, we have added this controlled ablation to Appendix B.3 (Table 9).
Response to Q3 & W1 (Part 3): Evaluating CLIP-Based Endogenous Modeling
We thank the reviewer for raising this critical point.
Directly predicting raw future statistics or differences struggles to generalize across domains due to drastic variations in numerical scales and noise. In contrast, supervising future evolution in CLIP’s pre-aligned latent space abstracts these variations into domain-invariant linguistic regimes and 2D geometric patterns while leveraging rich VLM priors.
To empirically evaluate our CLIP-based endogenous modeling, we compare it with lightweight numerical encoders and prediction heads trained to fuse and predict the same statistics (Appendix A.2) and first- and second-order differences, using the same training budget and matched trainable parameter counts:
| Method (Same Information & Training Budget) | GIFT-Eval MASE | GIFT-Eval CRPS | TIME MASE | TIME CRPS |
|---|---|---|---|---|
| Simpler Numerical Objectives (Stats & 1st/2nd Diffs) | 0.680 | 0.467 | 0.648 | 0.544 |
| CLIP-based Endogenous Modeling (Ours) | 0.671 | 0.461 | 0.639 | 0.537 |
Conclusion: CLIP-based endogenous modeling achieves lower MASE and CRPS on both benchmarks than the numerical feature baseline, demonstrating the effectiveness of our multimodal formulation.
To demonstrate the benefit of CLIP-based endogenous modeling over numerical features, we have added this controlled comparison to Appendix B.2 (Table 8).
Response to Reviewer qAdu (Part III)
Response to Q4 & W2 (Part 1): Historical Bank Construction and Fair Comparison with Baselines
1. Historical Bank Construction & Strict Data Parity (Q4)
- Training-Only Bank Construction: For each downstream dataset, the historical bank is constructed exclusively from 20% of sliding windows sampled from the official training split (Appendix B.4). Validation and test splits are strictly excluded from bank construction and modality weight calibration, guaranteeing zero data leakage.
- Strict Data Parity Across Two Baseline Categories:
- Zero-Shot TSFMs (0% training data): Standard zero-shot TSFMs (e.g., Chronos-2, Aurora) do not use the downstream training split. For a strictly bank-free comparison,
QiYao-M (w/o Exo.)uses no retrieval bank and already outperforms them across most benchmarks. - Full-Shot End-to-End Models (100% training data): Supervised multimodal baselines (e.g., Time-VLM, TATS) update all parameters on 100% of the official training split, whereas
QiYao-M (w/ Exo.)only accesses 20% of the training split as a non-parametric bank without updating parameters.
- Zero-Shot TSFMs (0% training data): Standard zero-shot TSFMs (e.g., Chronos-2, Aurora) do not use the downstream training split. For a strictly bank-free comparison,
2. Controlled Comparison Across All Four Baseline Categories (W2 & Q4)
To ensure a strictly fair comparison under controlled data access, we evaluate our model against four baseline categories: (i) zero-shot TSFMs (Chronos-2 (ZS), Aurora (ZS)), (ii) full-shot end-to-end multimodal models (Time-VLM, TATS), (iii) few-shot fine-tuned TSFMs updated on the exact same 20% training subset (Chronos-2 (FT), Aurora (FT)), and (iv) unimodal RAG baselines querying the identical 20% historical bank, including full-shot trained RAFT and tuning-free RAF. We report MSE across four representative datasets spanning 2 to 4 modalities:
| Model | Paradigm | Param. Update | Historical Data / Bank Access | Agriculture (2-Mod) | Health (2-Mod) | Tianchi (3-Mod) | SP500 (4-Mod) |
|---|---|---|---|---|---|---|---|
| Chronos-2 (ZS) | (i) Zero-Shot TSFM | ✗ | None (0%) | 0.238 | 1.054 | 1.453 | 0.717 |
| Aurora (ZS) | (i) Zero-Shot TSFM | ✗ | None (0%) | 0.272 | 1.553 | 1.403 | 0.862 |
| QiYao-M (w/o Exo.) | Zero-Shot TSFM (Ours) | ✗ | None (0%) | 0.199 | 0.915 | 1.423 | 0.694 |
| Time-VLM | (ii) Full-Shot End-to-End | ✓ | 100% train split | 0.237 | 1.565 | 1.368 | 0.756 |
| TATS | (ii) Full-Shot End-to-End | ✓ | 100% train split | 0.215 | 1.356 | 1.374 | 0.665 |
| Chronos-2 (FT) | (iii) Few-Shot Fine-Tuned TSFM | ✓ | Identical 20% train subset | 0.226 | 0.982 | 1.412 | 0.708 |
| Aurora (FT) | (iii) Few-Shot Fine-Tuned TSFM | ✓ | Identical 20% train subset | 0.241 | 1.215 | 1.379 | 0.748 |
| RAFT | (iv) Full-Shot Unimodal RAG | ✓ | 100% train + Identical 20% bank | 0.231 | 1.384 | 1.358 | 0.682 |
| RAF | (iv) Tuning-Free Unimodal RAG | ✗ | Identical 20% bank | 0.224 | 0.976 | 1.396 | 0.704 |
| QiYao-M (w/ Exo.) | Tuning-Free Multimodal RAG (Ours) | ✗ | Identical 20% bank | 0.174 | 0.901 | 1.302 | 0.660 |
Conclusion: Under strictly fair and identical bank access, QiYao-M (w/ Exo.) achieves the lowest MSE across all datasets without updating parameters, consistently outperforming all four baseline categories. This confirms that our multimodal retrieval mechanism effectively exploits heterogeneous external information for tuning-free adaptation.
To strengthen the baseline evaluation and make data access explicit, we have added fine-tuning and retrieval baselines alongside their training and bank-access settings in Appendix B.4 (Table 10).
Response to Reviewer qAdu (Part I)
We sincerely thank Reviewer qAdu for the detailed feedback and for recognizing the plausibility of our design and the breadth of our evaluation. We address the novelty concerns through controlled comparisons, and strengthen the evaluation with retrieval baselines, matched-budget ablations, dataset-exclusion tests, and efficiency measurements below.
Response to Q1 & W1 (Part 1): Clarification on Core Novelty and Empirical Evidence
We thank the reviewer for this constructive feedback. While individual building blocks (e.g., VLM encoders or retrieval) have precedents, equating QiYao-M to prior task-specific or unimodal methods overlooks our paradigm-level formulation and dedicated technical innovations, which we clarify and support with controlled empirical evidence below.
1. Overarching Paradigm
Unlike existing multimodal TSFMs (e.g., ChatTime, Aurora) that process all modalities through role-agnostic shared mechanisms, QiYao-M is the first role-aware multimodal TSFM that explicitly decouples endogenous temporal evolution from exogenous contextual adaptation (Figure 1).
2. Endogenous Modeling (Beyond Time-VLM and Simpler Numerical Objectives)
Prior multimodal forecasters such as Time-VLM are task-specific models that only use constructed text and images as historical inputs. In contrast, our endogenous modeling introduces two key innovations for TSFM pretraining:
To further evaluate the effectiveness and necessity of our Endogenous Modeling, we conducted additional comparative experiments. Please refer to our response to Q3 in Response to Reviewer qAdu (Part II) below.
3. Exogenous Modeling (Beyond Prior Retrieval Methods)
Prior retrieval-based forecasters (e.g., RAF, TimeRAF, and RAFT) focus on unimodal time-series lookup, while existing multimodal TSFMs are restricted to fixed modality inputs (primarily text). In contrast, QiYao-M is the first TSFM to support tuning-free adaptation across varying exogenous modality types and numbers, addressing two fundamental bottlenecks:
To quantify the contribution of Endo-Modality Proxy Training and assess its necessity, we conducted additional comparative experiments. Please refer to our response to Q2 in Response to Reviewer qAdu (Part II) below.
Together, these innovations yield state-of-the-art performance across 73 unimodal datasets (Figure 5) and 9 multimodal datasets spanning 2 to 4 modalities (Table 1). The controlled comparisons are provided in Appendices B.2–B.3 (Tables 8–9).