acceptodds
Under review as a conference paper at ICLR 2027

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

Abstract

Real-world time series come with text: metadata, descriptions, news, reports. Yet time series foundation models process numerical sequences in isolation, and the multimodal text-and-time-series models that attempt to bridge the two all adapt a pretrained language model post hoc, inheriting representations shaped without ever seeing temporal data. These models are also evaluated almost exclusively against other multimodal baselines, not against the strongest unimodal foundation models in either domain, leaving open whether joint training is needed at all. We present Chronicle, a compact 324M-parameter decoder-only transformer trained from scratch on natural language and time series within a single unified architecture. Both modalities share the same transformer blocks, attention mechanism, and residual stream; the bulk of pretraining uses unimodal batches so cross-modal capability emerges purely from shared parameters, with a short alignment stage that interleaves the two. To our knowledge, Chronicle is the first model jointly pretrained on text and time series from scratch, and the first multimodal model evaluated against dedicated foundation models in both domains. It matches Gemma-3-270M-PT on 19 NLU tasks, sets a new bar for frozen-embedding time series classification on 24 UCR/UEA datasets, and produces multimodal forecasts on Time-MMD that beat every supervised fusion baseline, all from a single backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Time-Series, Natural Language Processing, Multi-Modality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_6arsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Chrnicle, the first model to learn the relationship between text and time series and train the model from scratch. The authors conduct extensive experiments by comparing it with dedicated foundation models over 19 NLU tasks on 24 datasets.

Critical strengths
  1. In this paper, the authors proposed a unified framework called Chrnicle, whcile is the first model to learn the relationship between text and time series and train the model from scratch.

  2. The authors conduct extensive experiments and achieves competitive performance across over various tasks.

  3. The authors compare the proposed method with strong unimodal foundation models.

Critical weaknesses
  1. Although section 5.3 shows the effectiveness of the cross-modal alignment stage by comparing two stages checkpoint, this experiment can only partially demonstrate whether the proposed method effectively learns relationship between text and time series. The paper lacks the ablation study to demonstrate the effectiveness of the proposed method over the single modality variant, such as text-only and time series only model using the same pretrained Chronicle checkpoint. This aims to enhance the motivation of this paper that "these models are also evaluated almost exclusively against other multimodal baselines, not against the strongest unimodal foundation models in either domain, leaving open whether joint training is needed at all".

  2. In section 5.2.1, the authors mention that supervised DL baselines are trained per-dataset. My concern is that Chronicle may benefits from additional pretraining data. Directly comparing Chronicle with supervised DL baselines trained per-dataset is not fair. So, the experimental results cannot demonstrate the overall superiority of Chronicle for time-series classification. Rather, it serves the evidence as the effectiveness of the proposed method with frozen representations. The authors should frame it carefully.

  3. In Fgiure 3 and table 7, the results show that Stage 2 does not consistently improve multimodal performance across different setting. The paper lacks a detailed analysis for the inconsistency.

  4. In Figure 2, the number of the parameters for many baseline models are not reported, making it hard to compare the effectiveness of the proposed method and other baseline methods fairly.

Questions for authors

Q1: Why does TimeMMD LoRA adaption achieves worse performance in Figure 3 and Table 7? Can you provide some analysis for the inconcistency of the improvement across different settings?

6 comments

Thank you for the review and for the concrete suggestions, each of which we have carried out. The ablation you propose in W1 is done and supports the motivation mentioned. We would like to clarify a few things about the framing concerns in W2 and the comparisons with other pretrained models. The inconsistency in W3 resolves into a consistent pattern once the evaluations are grouped by what they measure.

W1. No ablation against text-only and series-only variants of the same pretrained checkpoint.

We ran this ablation. The Stage 2 checkpoint is frozen and the same head is trained on TimeCAP with joint, text-only and series-only input, plus a control in which captions are shuffled across samples, in training and test alike. Shuffling keeps the text and breaks only its link to the series. Results are means over 5 seeds and 3 domains.

TimeCAP, frozen Stage 2Macro-F1AUC
Joint (text and series)0.6110.763
Text only0.5960.739
Series only0.4570.616
Shuffled captions (same text, wrong series; r = 1)0.3780.500

The joint model beats each single-modality variant in all 5 seeds. With shuffled captions it falls to chance at repetition factor r = 1 and stays below series only at r = 64 (AUC 0.588), so it relies on the text matching the series and not on the presence of extra text.

We also tested whether the frozen model relates text and series before any downstream training. We take 771 held-out (series, description) pairs from the alignment data. Each description is placed after its true series and after 9 other series drawn at random, and we rank the 10 series by the likelihood the frozen model assigns to the description. No head or parameter is trained.

Frozen model, no trainingTrue series 1st of 10Top 5 of 10
Chance10%50%
Stage 111.0%50.5%
Stage 219.5%71.7%

Stage 1 is at chance. Stage 2 ranks the true series first about twice as often as chance, and the 95% bootstrap CIs exclude chance.

Finally, we checked that this ability comes from joint training on paired data and not from more training. Three runs continue from the same Stage 1 checkpoint with the same architecture, schedule and compute (2B tokens). They differ only in a slice of 15% of the batches: real text–series pairs, the same series and descriptions re-paired at random, or no pairs (more time series instead).

Equal compute from Stage 1True series 1st of 10Top 5 of 10
Real pairs19.1%65.6%
Random pairs12.1%51.4%
No pairs9.5%49.0%

Only real pairs produce the ability (real against random pairs: +7.0 points, 95% CI [3.9, 10.2]). The same amount of extra training without pairs leaves the model at chance. This is direct evidence for the motivation you quote: the relationship between the modalities is learned by joint training and is measurable in the backbone itself.

W2. Concerns regarding Chronicle benefitting from additional pretraining data.

Section 5.2.1 also compares Chronicle with pretrained time series foundation models that were trained on far more time series tokens than Chronicle. This like-for-like comparison is in Table 3: three pretrained TSFMs (Chronos-2, Moirai-2, TimesFM) that also benefit from large-scale pretraining, evaluated with the identical frozen probe. They spend all of their pretraining on time series, whereas Chronicle spends about 8% of its compute on it. With 5 seeds on the 24 UCR/UEA datasets:

Frozen probe, UCR/UEAChronicle wins / losses (datasets)Wilcoxon
vs. Chronos-224 / 0p < 0.001
vs. TimesFM22 / 2p < 0.001
vs. Moirai-216 / 8p = 0.08 (n.s.)

Chronicle has the best mean accuracy (0.753) and the best average rank (3.67 among the 10 baselines and both Chronicle checkpoints). We will make this clearer within the paper, thank you for your feedback.

W4. Figure 2 does not report parameter counts for many baselines. We will update Figure 2 to list every model's parameter count from its public model card (e.g. TimesFM-2.5 200M, Toto-Base 151M, Moirai-Large 311M, TiRex 35M, YingLong 300M) for clarity. Chronicle has 323.8M parameters, of which 189.6M process time series. The 134M text embedding table is never used for series inputs.

W3. Stage 2 does not improve performance consistently across settings, and the paper lacks an analysis.

The results are consistent once they are grouped by what each protocol measures. Some evaluations read the frozen representation, through a small head or with no training. Others ask the model to generate text or forecasts zero-shot. Stage 2 replaces a small share of unimodal generation training with alignment data, so the expected outcome is a gain in the representation and a small cost in zero-shot generation. That is what we observe.

What is measuredEvaluationStage 1 → Stage 2
Frozen representationRetrieval, no training (1st / top 5 of 10; see W1)11.0% / 50.5% → 19.5% / 71.7%
Frozen representationUCR/UEA accuracy (5 seeds)0.736 → 0.753 (15 datasets better, 7 worse; p < 0.01)
Frozen representationTime-MMD frozen head, NMAE ↓0.524 → 0.514
Frozen representationTimeCAP frozen head, macro-F1 (r from validation)0.607 → 0.609 (tie)
Zero-shot generationNLU accuracy0.411 → 0.406 (n.s.)
Zero-shot generationGIFT-Eval MASE / CRPS ↓0.978 / 0.690 → 1.053 / 0.754

Stage 2 improves or ties every evaluation of the frozen representation, which is how a foundation model is used downstream. The cost is confined to zero-shot generation.

The UCR/UEA numbers are 5-seed results for every model on all 24 datasets under the probe configuration stated in the paper (Appendix F.4). They supersede the submitted Table 7 entry (0.729 ± 0.057), and Tables 3 and 7 will be updated accordingly. Section 5.3 will be rewritten around this analysis.

Q1. Why is Time-MMD LoRA worse with Stage 2?

We broke the result down by domain. Stage 2 is better than Stage 1 on 4 of the 9 domains and essentially tied on two more (Economy +0.003 and Public Health +0.005 MAE). The average is affected by two small single-channel domains: Security (+0.225 MAE), which has 207 training rows, and Social Good (+0.128 MAE), where the three seeds range from 0.50 to 0.79 MAE (standard deviation 0.146).

LoRA is the less reliable protocol at this data scale for both checkpoints: it is worse than the frozen head on 5 of 9 domains for Stage 1 and on 7 of 9 for Stage 2. The LoRA average therefore reflects the instability of fine-tuning on very small datasets and not an effect of the alignment stage. Under the frozen head, which has no such instability, Stage 2 is the better checkpoint (0.514 vs. 0.524).

Summary. Joint input beats both single-modality variants of the same checkpoint (W1). Against other pretrained models under one frozen probe, Chronicle gives the strongest time series representations (W2). Stage 2 improves or ties every evaluation of the frozen representation, and the one apparent exception traces to two very small domains (W3, Q1). Parameter counts are added (W4). We hope this resolves your concerns, and we would be grateful if you would reconsider your rating.

Reviewer_6arsOpenReview ↗

Response to Rebuttal

Thank you for the detailed explanation and additional results. Most of my concerns are addressed. For W2, my concern is about the comparison between Chronicle, which benefits from large-scale external pretraining, and supervised DL baselines trained independently on each downstream dataset without comparable pretraining. However, the results in Section 5.2.1 compares Chronicle with pretrained time series foundation models that were trained on far more time series tokens than Chronicle. I agree that comparisons with other pretrained time-series foundation models provide useful evidence of representation quality. However, this does not resolve the fairness concern regarding supervised DL baselines.

One thing I want to remind you is that you can update the new version of the pdf file during the rebuttal phase.

Reviewer_gGggOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores joint pretraining of natural language and time series within a unified foundation model, motivated by the limitations of existing approaches that typically adapt pretrained language models to temporal tasks. The authors introduce Chronicle, a compact 324M-parameter decoder-only transformer trained from scratch on both modalities using a shared backbone with modality-specific input and output interfaces. Its training process consists of predominantly unimodal pretraining followed by a short cross-modal alignment stage using interleaved text and time series data. The model is evaluated on language understanding, time series forecasting and classification, and multimodal downstream tasks, demonstrating competitive frozen-representation transfer performance while retaining language understanding capabilities.

Critical strengths
  • Chronicle uses a shared Transformer backbone with modality-specific interfaces, enabling different modalities to be processed without complex task-specific fusion components.
  • The authors evaluate the model on language understanding, zero-shot forecasting, time-series classification, and multimodal tasks, providing a relatively comprehensive assessment of its capabilities.
Critical weaknesses
  • Experiments lack direct comparisons with alternative pretraining paradigms under matched architectures, data, and computational budgets.
  • Parameter sharing alone does not guarantee semantic alignment or meaningful cross-modal information exchange.
  • Stage 2 introduces explicit alignment data, but its contribution is not sufficiently isolated from additional training.
  • The strong Time-MMD performance is primarily achieved after training a downstream forecasting head, making it unclear whether cross-modal understanding is learned during pretraining or introduced mainly through supervised adaptation.
Questions for authors
  • How can the authors establish that it learns genuine cross-modal alignment rather than separate unimodal representations within a shared backbone?
  • Can the authors compare Stage 1 with independent unimodal pretraining and examine the effects of paired versus randomly paired alignment data?
  • Why are recent multimodal time-series models such as TaTS not included in the main Time-MMD comparisons?
  • How much of the reported multimodal improvement comes from genuine semantic interactions between text and time series?
6 comments

Thank you for the detailed review. We understand your concerns regarding clarity on if Chronicle learns cross-modal alignment or only hosts two unimodal models in one backbone, and whether the effect of Stage 2 can be separated from extra training. We have added several experiments to alleviate these concerns.

W2 / Q1. Shared parameters do not guarantee alignment. How can genuine alignment be established?

We further explore this by comparing the two stages of training. With shared parameters alone the model is not yet aligned, and alignment appears primarily when paired text and series are added to training.

We measure alignment by comparing cross-modal alignment between the two stages. We take 771 held-out (series, description) pairs from the alignment data. Each description is placed after its true series and after 9 other series drawn at random, and the frozen model scores the likelihood of the description in each case. We rank the 10 series by that likelihood. No head or parameter is trained.

Frozen model, no trainingTrue series 1st of 41st of 10Top 5 of 10
Chance25%10%50%
Stage 1 (shared parameters only)24.0%11.0%50.5%
Stage 2 (after alignment)40.6%19.5%71.7%

Stage 1 is at chance. Stage 2 ranks the true series first about twice as often as chance, and the 95% bootstrap CIs exclude chance in every column. In the aligned model, how likely a description is depends on the content of the series before it, which is what alignment means in operational terms.

In three runs continued from Stage 1 at equal compute (described under W3 / Q2 below), retrieval rises only when the added data are real pairs:

Continued from Stage 1, equal compute (2B tokens)Retrieval, 1st of 10 (chance 10%)Retrieval, top 5 of 10 (chance 50%)
Real pairs (each series with its own description)19.1%65.6%
Random pairs (same data, re-paired at random)12.1%51.4%
No pairs (extra unimodal training)9.5%49.0%

Real pairs rank the true series first 7.0 points more often than the same data re-paired at random (95% CI [3.9, 10.2]), and extra unimodal training stays at chance. Alignment is therefore produced by paired multimodal data on top of the shared backbone, and not by parameter sharing or further training alone.

W3 / Q2. Stage 2's contribution is not isolated from additional training. Compare with unimodal pretraining, and paired with randomly paired data.

We made both comparisons at equal compute. We continued training from the same Stage 1 checkpoint three times. The runs share the architecture, the text and time series data and their order, the schedule, and the compute (2B tokens). Each spends 80% of its batches on text, 5% on time series and 15% on an alignment slice, and they differ only in the content of that slice:

  • Real pairs: each series appears with its own description, as in Stage 2.
  • Random pairs: the same series and descriptions, re-paired at random. The model sees the same mixed sequences, but the text no longer describes the series.
  • No pairs (unimodal only): the slice is replaced by more time series, i.e. extra training only.

The slice is larger than Stage 2's 5% because the runs are short: 15% of 2B tokens is about 0.3B paired tokens, still fewer than the roughly 0.75B that Stage 2 sees.

All three are evaluated identically: the retrieval test described under W2 / Q1, and Time-MMD forecasting with text, using the same frozen-backbone head and training settings.

Equal compute from Stage 1Retrieval, 1st of 10 (chance 10%)Retrieval, top 5 of 10 (chance 50%)Time-MMD with text, NMAE ↓
Real pairs19.1%65.6%0.517
Random pairs12.1%51.4%0.516
No pairs (unimodal only)9.5%49.0%0.529
  • Unimodal continuation gives no alignment (retrieval at chance) and the worst forecast from mixed input.
  • Random pairs match real pairs on forecasting, because the backbone still learns to process mixed sequences, but give almost no alignment (12.1% vs. 19.1% first of 10; difference 7.0 points, paired bootstrap 95% CI [3.9, 10.2]).
  • Real pairs are the only condition that produces alignment, and they match the Stage 2 checkpoint on retrieval (19.1% vs. 19.5%).

Mixed sequences teach the backbone to take both modalities in one input. Correct pairing teaches it how they relate. The gain of Stage 2 therefore comes from its alignment data and not from the additional training.

The forecasting column is a single seed; the seed-to-seed standard deviation of this head is below 0.002 on the Stage 1 and Stage 2 checkpoints. All evaluation inputs fit within Stage 1's context window (Section 5.3), so Stage 2's longer context is not a factor.

W1. No comparison with alternative pretraining paradigms under matched architecture, data and compute.

The matched runs under W3 / Q2 are this comparison for joint alignment against continued unimodal pretraining: same architecture, data and compute, and only the real-pairs run aligns (19.1% retrieval against 9.5%, chance 10%).

The paper also covers the main alternative paradigm at full scale. The FM Fusion baselines in Tables 4 and 5 pretrain a text model (BERT, GPT-2) and a time series model (Chronos-2, Moirai-2, TimesFM) independently and join them with a trained fusion head. Chronicle outperforms every such pair on Time-MMD (NMAE 0.514 vs. 0.588 for the best pair) and has the highest mean macro-F1 and AUC on TimeCAP. The third paradigm, adapting a pretrained LLM post hoc, was examined by Tan et al. (2024), who found that replacing the LLM with simple layers does not hurt forecasting.

@misc{tan2024languagemodelsactuallyuseful, title={Are Language Models Actually Useful for Time Series Forecasting?}, author={Mingtian Tan and Mike A. Merrill and Vinayak Gupta and Tim Althoff and Thomas Hartvigsen}, year={2024}, eprint={2406.16964}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2406.16964}, }

W4. Time-MMD performance is reached after training a head, so the cross-modal understanding may come from supervised adaptation.

We address these concerns with several new experiments.

  1. The retrieval test (W2 / Q1) uses no head and no training, and Stage 2 identifies the true series twice as often as chance. On Time-MMD specifically, each baseline we compare to receives equal or more training on this task.
  2. In the matched runs (W3 / Q2) the head and its training are identical across runs and only the backbone's pretraining differs, yet the forecast changes from 0.529 to 0.517.
  3. In the TimeCAP controls (Q4 below) the backbone is frozen and the head receives one pooled vector, yet on the three-domain average that vector beats either modality alone, and mismatched captions take it below series only.

The head is also the smallest trained component in Table 5. Every method in Table 5 is trained on Time-MMD including the foundation model baselines. Chronicle is the only model in the table that can also forecast zero-shot from text and series, and that row improves from 1.040 to 0.835 NMAE with Stage 2.

Q3. Why is TaTS not in the main Time-MMD comparison?

Thank you for the suggestion. TaTS will be added to Table 5. We re-ran its public model code with its default training settings under our splits, windows and metric, with all 4 backbones and both text encoders. The two encoders give nearly identical scores, and we show the better one.

TaTS's pipeline averages every forecast 50/50 with a historical-average column supplied in the dataset files. This step is outside the model, and no other method in Table 5 uses that column. We therefore report two settings: one in which no method uses the averaging, and one in which every method does, with Chronicle's head trained under the same averaging.

Time-MMD, NMAE ↓Common protocolBoth with TaTS's averaging
TaTS + PatchTST0.5240.497
TaTS + iTransformer0.5300.500
TaTS + TimesNet0.5400.498
TaTS + DLinear0.6610.561
Chronicle (frozen backbone, 0.69M head)0.5150.484

Chronicle is ahead of every TaTS configuration in both settings, while TaTS trains a separate forecasting model for every domain and horizon. We thank you for this feedback, we believe these new results have improved the robustness and soundness of our results.

Q4. How much of the multimodal improvement comes from genuine semantic interaction?

Joint input could help because the model receives more input, or because it relates what the text says to the series. To separate the two, we froze the Stage 2 backbone and trained the same classification head on TimeCAP with four inputs: joint text and series, text only, series only, and joint input with the captions shuffled across samples, in training and test alike. Shuffling keeps the amount and style of text and breaks only its link to the series. Results are means over 5 seeds and 3 domains, with the repetition factor r selected per domain and seed on the validation split.

TimeCAP, frozen Stage 2Macro-F1AUC
Joint (matched text and series)0.6110.763
Text only0.5960.739
Series only0.4570.616
Shuffled captions (same text, wrong series; r = 1)0.3780.500

On the three-domain average, joint input with matched captions beats both single modalities, and it is ahead of text only on AUC in all 5 seeds. With shuffled captions, performance falls to chance at r = 1, where the caption dominates the pooled vector (Section 5.2), and stays below series only when the series is up-weighted (AUC 0.588 at r = 64, against 0.751 for matched captions at the same r). Extra text therefore helps only when it matches the series. Together with the retrieval result, where the frozen model links a description to its own series with no training, this shows the gain comes from semantic interaction inside the pretrained backbone.

Summary. Each address each question/weakness with several new experiments: alignment with no training (W2, Q1), real pairs against random pairs and against extra training at equal compute (W1, W3, Q2), pretraining against the head (W4), TaTS under a common protocol (Q3), and matched against shuffled text (Q4). We hope these results address your concerns about soundness and informativeness, and we would be grateful if you would reconsider your assessment in light of these additional experiments.

Comments on the paper

4 comments

Thank you for the thorough review, and for recognizing the evaluation design, the alignment ablation and the reproducibility of the paper. We have added several experiments to address your concerns and further improve the quality of our paper: uncertainty and equivalence tests for NLU, a decontamination audit, a fixed-compute modality-ratio experiment, a horizon breakdown, unimodal heads, validation-selected tuning, and paired significance tests. In every case the result supports the claim as made in the paper, and for forecasting and classification it supports it more strongly than before.

W1a. The NLU claim rests on one aggregate score with no uncertainty, and the tie could flip under seed or prompt variation.

We quantified the tie with Gemma-3-270M in three ways from the per-task results.

Chronicle − Gemma-3-270M, mean accuracyResult
Difference, 95% CI over examples+0.0009, [−0.010, +0.012]
TOST equivalence, margin ±1 pointequivalent, 90% CI [−0.008, +0.010]
Resampling the 19 tasks, 95% CI[−0.015, +0.017]
Wilcoxon over tasksp = 0.81

The per-task significant differences split 6 wins for Chronicle (reasoning, reading comprehension) and 4 losses (recall-heavy knowledge), which is the expected pattern for a model that saw 43× less text. Prompts and few-shot examples are fixed and identical for every model.

W1b. The baselines differ in architecture and saw far more text, so the comparison does not isolate the cost of joint training.

The data difference works against Chronicle: the baselines saw 43–87× more text, and Chronicle still matches Gemma-3-270M. Within our own backbone, the two checkpoints measure the effect of changing the data mix directly. Stage 2 gives 5% of its training batches to paired text and series, and NLU changes by −0.005, which is not significant. We therefore see no measurable cost of joint training to language understanding. We add some additional experiments below to further explore the effects of stage 1 vs stage 2 training.

W1c. The no-overlap claim needs substantiation, and the decontamination procedure should be described.

We checked all 89.5k evaluation examples for 13-gram overlap (after normalizing punctuation) that also occur in the text pretraining sources: the FLAN and Wikipedia data used in pretraining and a random sample of 20 FineWeb-Edu shards, 2.2B words in total.

What could inflate a score is overlap that covers the gold answer, and this occurs in at most 2.1% of examples for any task (SQuAD 2.1%, Winograd 1.8%, all others below 1%). Passage-level overlap is concentrated in the Wikipedia-based reading-comprehension tasks (SQuAD, BoolQ, CoQA), as for any model trained on web text. Appendix D will describe the procedure and give the per-task table. It is important to note that the models we compare against generally do not publish full training information with regards to data and therefore we cannot determine how much of their training datasets may overlap with the evaluation datasets.

W2. The attribution of the forecasting gap to compute allocation and the autoregressive objective is asserted, rather than demonstrated.

We ran both analyses you suggest.

Modality ratio at fixed compute. Two runs continue from the same Stage 1 checkpoint with the same backbone, schedule and compute (2B tokens), and both spend 80% of their batches on text. One contains 5% of its batches to time series (the remaining 15% are paired text–series sequences), and the other recieves the full 20% to time series.

Equal compute from Stage 1GIFT-Eval MASE (relative) ↓GIFT-Eval CRPS (relative) ↓UCR/UEA accuracy ↑
5% time series1.0001.0000.735
20% time series0.945 (−5.5%)0.859 (−14%)0.748

The 20% run is better than the 5% run on 62 of 97 GIFT-Eval tasks for MASE and on 64 of 97 for CRPS (Wilcoxon p < 0.002), after only 2B tokens. At equal compute the architecture supports that a larger time series share directly into forecasting accuracy, which is the controlled evidence for the paper's compute-allocation explanation. The modality mix is a design choice that can be adjusted towards more time series when forecasting is the priority.

Horizon breakdown. An architectural gap should be roughly constant across horizons. A gap caused by autoregressive next-patch decoding, where each predicted patch is fed back as input and errors compound, should grow with horizon.

Chronicle (Stage 1) / TimesFM-2.5ShortMediumLong
MASE ratio1.351.431.45
CRPS ratio1.351.461.51

The gap grows steadily with horizon. The breakdown uses the Stage 1 checkpoint, the stronger zero-shot forecaster of the two (Table 7).

Representation quality. The same backbone gives the best frozen time series representations on the 24 UCR/UEA datasets (5 seeds, identical frozen probe for every pretrained model): the best mean accuracy (0.753) and average rank of all models compared, ahead of TimesFM on 22 of 24 datasets and of Chronos-2 on all 24 (Wilcoxon p < 0.001 for both). The architecture captures time series well, and the gap appears only when generating long forecasts.

Forecast error falls when the time series share is raised at fixed compute, and the gap to TimesFM-2.5 widens with horizon, while the frozen representation already matches or beats dedicated models. This supports a budget- and objective-driven gap over an architectural one.

W3a. The multimodal evidence lacks unimodal controls and tuning parity.

On the frozen Stage 2 checkpoint we trained the same TimeCAP classification head with joint input, text only, series only, and joint input with the captions shuffled across samples, in training and test alike. Shuffling keeps the amount and style of text and breaks only its link to the series. Results are means over 5 seeds and 3 domains.

TimeCAP, frozen Stage 2Macro-F1AUC
Joint0.6110.763
Text only0.5960.739
Series only0.4570.616
Shuffled captions (r = 1)0.3780.500

On the three-domain average, joint input beats both single modalities, and it is ahead of text only on AUC in all 5 seeds. The text-only head measures how much label signal the captions carry. The better single modality differs by domain: series on Weather, text on Finance and Healthcare. One joint head matches the better one in each domain (within 0.02) without being provided information on the specific domain, and so it beats either fixed single modality overall. The series therefore contributes information that its generated caption does not. With shuffled captions the head falls to chance at r = 1 and stays below series only at r = 64 (AUC 0.588), so the text helps only when it matches the series.

A separate test, which uses no labels and no trained head, confirms that the backbone itself relates text to series, so the joint result does not come from the captions carrying the label. We take 771 held-out (series, description) pairs from the alignment data. Each description is placed after its true series and after 9 other series drawn at random, and we rank the 10 series by the likelihood the frozen model assigns to the description. Nothing is trained.

Frozen model, no trainingTrue series 1st of 10Top 5 of 10
Chance10%50%
Stage 1 (before alignment)11.0%50.5%
Stage 2 (after alignment)19.5%71.7%

Stage 1 is at chance. Stage 2 ranks the true series first about twice as often as chance, and the 95% bootstrap CIs exclude chance. In three runs continued from Stage 1 at equal compute, this ability appears only when the added data are real text–series pairs (19.1% first of 10, against 12.1% for the same data re-paired at random and 9.5% for extra unimodal training). The frozen representation that the TimeCAP head reads therefore already encodes which text belongs to which series.

W3b. The repetition factor r is tuned on the evaluation domains, and the baselines have no analogous control.

We select r on the validation split only: per domain and seed, r is chosen by validation AUC from a fixed grid (1 to 128), and both metrics are reported from that one head.

Chronicle, frozen Stage 2 (5 seeds)Macro-F1AUC
r = 1 (no repetition)0.5940.739
r = 64, fixed0.5980.751
r selected on validation0.6110.763

Selection from validation without test data gives the best scores of the three. It does not settle on one value: it chooses large r on Weather (48 to 128), where the series is the more informative input, and small r on Healthcare (1 to 32), where the caption is. The baselines have no counterpart to r because they encode text and series separately into fixed-size embeddings, so there is no imbalance between modalities to correct.

W3c. Margins are small relative to seed noise. Significance testing is needed.

We tested Chronicle against each of the 12 baselines with a paired Wilcoxon signed-rank test (two-sided) over the 15 seed × domain cells. Chronicle scores 0.611 macro-F1 and 0.763 AUC with its backbone frozen. The p-values compare Chronicle with the baseline in that row, and bold marks p < 0.05.

BaselineMacro-F1p (F1)AUCp (AUC)
TimesNet + BERT0.6000.3890.7560.454
BERT + Moirai-20.5950.2080.7580.421
DLinear + BERT0.5920.1690.7450.095
BERT + Chronos-20.5890.1070.7330.022
PatchTST + BERT0.5820.0830.7250.012
GPT-2 + Moirai-20.5700.1210.7530.151
DLinear + GPT-20.5680.0180.7200.001
TimesNet + GPT-20.5600.0950.7320.252
PatchTST + GPT-20.5410.0100.7070.012
BERT + TimesFM0.4900.0010.6580.003
GPT-2 + TimesFM0.4680.0020.648<0.001
GPT-2 + Chronos-20.4660.0030.6980.008

No baseline is ahead of Chronicle on either metric, and Chronicle wins the majority of the 15 cells against every baseline. It is significantly better than 7 of the 12 on AUC and than 5 on macro-F1. Chronicle achieves this performance with one frozen backbone and no task-specific training of the encoder, which is the setting a foundation model is meant for.

Summary. We again want to thank you for recognizing our contribution, completeness and focus on transparency and reproducibility of these results. We believe the requested experiments further help to strengthen our paper. The NLU tie is a statistical equivalence within one point, with negligible answer-level contamination (W1). Forecasting improves when the time series share is raised at fixed compute, and the gap to TimesFM-2.5 widens with horizon, as a budget-driven gap predicts (W2). Joint input beats the better single modality on TimeCAP with r chosen on validation data, and Chronicle's lead holds under paired tests (W3). We hope these experiments have resolved any remaining concerns you may have with regards to the soundness, transparency and robustness of the results.

Dear reviewer zwC6, thank you again for your feedback and recognizing the completeness and importance of our work. We wanted to note that the revised PDF is now uploaded, with changes in blue. The experiments you asked for are in the paper:

  • Modality ratio at fixed compute and horizon breakdown (W2): new Table 7 in Section 5.3 and Appendix L.3.
  • NLU uncertainty and overlap audit (W1): Appendix L.6 and Appendix D.
  • Unimodal heads, validation-selected r, and paired significance tests (W3): Appendix L.5.

Thank you again for the careful review. If anything remains unclear we would be glad to answer it in the time left, and if these additions resolve your concerns we would be grateful if you would consider reflecting that in your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.