acceptodds
Under review as a conference paper at ICLR 2027

Forecast or Recall? Attributing Training Data Memorization in Time Series Foundation Models with TimeCanary

Abstract

Time series foundation models are evaluated on public benchmarks, yet their pretraining data are seldom documented at the series level. Low forecast error can thus reflect generalization or recall, leaving data owners unable to tell whether a model learned from their series. We find that a post-training audit depends on the baseline forecaster and lacks enough valid comparison pairs for a reliable conclusion. We introduce **TimeCanary**, which marks an owner's original series before release with a sparse Fourier signature scaled to each series. The mark extends across training windows. For each marked span, a recorded coin flip chooses one of two candidate keys for the mark; the other key provides a comparison. We score both keyed windows by model loss and use a matched model trained with reversed key roles to cancel pre-existing key differences. The test also requires reference models and diagnostic checks. To our knowledge, this is the first effective method shown to attribute use of marked original time series in a controlled audit. The paired score met its support rule across diverse models, including when only one quarter of eligible host spans were marked.

Primary area:
learning on time series and dynamical systems
Keywords:
time series foundation models, training data attribution, data provenance, memorization, Fourier canaries

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BK8NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes TimeCanary, a way to essentially watermark time series datasets so that we can identify if foundation models have been trained on proprietary datasets. This also allows for estimating if the model has truly generalized or if the model just memorized its training corpus. TimeCanary allows hosts to add a mixture of sinusoids to datasets before being used to train time series foundation models (TSFMs). Therefore, when the models are being tested on different perturbations being added to the same datasets, the model predictions can reveal if the model has memorized a particular pattern induced by the hosts.

Critical strengths
  1. This is an important problem, and the proposed solution is sound.

  2. The idea of having embedded and sham keys to watermark and identify datasets being used in the pretraining corpus is interesting and new to the time series community.

Critical weaknesses
  1. As the foundation models keep getting much better at generalization (the likes of Chronos2, Toto2, and TimesFM3 are much better models than the ones used in the paper), how would this approach be able to clearly differentiate between the embedded and the sham keys?

  2. Synthetic datasets like KernelSynth, which follows similar modeling paradigm to the keys in this paper, and more broadly, synthetic time series datasets from structural causal models have been used in the most recent models like Chronos2, Toto2, TimesFM3, etc. In such situations, how would simple keys like the Fourier sums be able to differentiate between model memorization and generalization capabilities?

Questions for authors

Check critical weaknesses. Additionally,

  1. Why is the key based only on periodic waveforms with a combination of different frequencies?

  2. Could it be the case for a few datasets that x(t) by itself is hard to learn? In which case, the loss for embedded and sham keys could just be dominated by the difficulty of the real dataset, and we might not be able to confidently identify the usage of a particular dataset in the training corpus?

  3. Does altering the pretraining datasets with these perturbations affect the model performance in general, when compared to the base model's (trained with no perturbations) performance?

I am willing to update my score towards acceptance if the questions are sufficiently answered.

2 comments

We thank the reviewer for the thoughtful review.

W1. Can newer models distinguish the training key from the sham key? We score both keyed versions of the same span (Figure 2). The single-model test asks whether the target has lower loss on its training key. An easier key can also give lower loss. In the paired test, a matched reference trains on the sham key and scores both versions. Combining the loss differences cancels any advantage shared by both models and tests whether lower loss follows the key each model saw.

We tested Chronos-2, TimesFM 2.5, and Toto 2.0 (313M) at 0.12σ and at 5e-5. The second Toto run kept the same spans, keys, and seeds. The paired test requires training, so we used TimesFM 2.5 because the public TimesFM 3 code only supports inference. The paired test supported attribution in all four settings. The single-model test only supported attribution on TimesFM 2.5 and Toto at 1e-3 (Table A).

W2. What if synthetic pretraining already teaches the model these patterns? A model that learns periodic patterns may forecast both marks well or find one easier. Random assignment gives both keys the same chance of being embedded, and the role swap cancels shared differences in key difficulty (Appendix E.3). Chronos-2’s pretraining included synthetic data from temporal causal models and KernelSynth. The paired test still detected exposure to the planted keys (Table A).

Q1. Why use multiple sinusoids? Recurring patterns in time series motivate using sinusoids. Combining a few frequencies and phases gives each key a reproducible pattern across training patches. Scaling by the span's standard deviation controls the mark's size relative to the signal (Section 5.1 and Appendix E.1). Whether more complex marks improve detection enough to justify any forecasting cost remains open.

Q2. Can a hard-to-forecast dataset hide the attribution signal? It can. Both keyed versions may have high forecast loss, but TimeCanary compares them on the same span, so the baseline loss common to both cancels (Appendix E.3). If the series affects the two keyed forecasts differently, the remaining loss difference can still be noisy. Importantly, the paired test supported attribution in separate audits on ETTm2 and MetroPT-3 (Table 9). These experiments show that the signal can be detected for an individual dataset.

Q3. Does marking hurt forecasting? The 0.06σ mark still supported attribution on Chronos-T5-Small (Table 9), with a forecasting trade-off. To isolate the cost of marking, we continued training with and without marks at equal budget. We measured scaled absolute error of the median forecast on held-out unmarked windows. The original checkpoint gives the net change from continued training. On ETTm1 and weather, marked continuation still improved forecasts over the starting checkpoint, though the gains were smaller than with unmarked continuation. On ETTm2, which was excluded from continued training, marking raised error relative to both unmarked continuation and the original checkpoint (Table B). These results show that the attribution signal can coexist with forecasting gains from continued training, with a cost that depends on where the model is evaluated.

Thank you for offering to revisit your assessment. If these results address your concerns, we would appreciate it if you reconsidered your rating.

Table A. Attribution results. Win rate is the share of spans favoring the training key. “Yes” means the test supports attribution.

ModelLearning ratePairedSingle-modelToto paired gap
Chronos-25e-50.748, yes0.648, no claim¹—
TimesFM 2.5 (200M)5e-50.956, yes0.856, yes—
Toto 2.0 (313M)5e-50.640 (160/250), yes0.524 (131/250), inconclusive²+0.0041
Toto 2.0 (313M)1e-30.872 (218/250), yes0.760 (190/250), yes+0.0444

¹ Chronos-2’s single-model check found a preference between two keys absent from training (p = 0.059).

² Toto’s single-model result was inconclusive (p = 0.22).

Table B. Forecast error after marking. Positive means higher error. “Clean” means unmarked continuation. “Base” means the starting checkpoint.

Dataset0.12σ vs. clean0.12σ vs. base0.06σ vs. clean0.06σ vs. base
ETTm1+11.6%−9.7%+4.6%−19.1%
Weather+8.9%−4.9%+7.7%−11.2%
ETTm2+6.2%+6.2%+4.2%+1.5%

Thank you for the response. However, my concerns are not fully addressed

Q1. Similar to periodicity, auto correlation is also typical in time series. Why not construct marks with AR processes?

Q3. The explanation says that in certain datasets the marking helps and in others, the marking hurts. This is not systematic.

I think this paper needs more rigorous experimentation and rewriting for easier understanding. Therefore, I will retain my score.

Reviewer_zYSXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses problems related memorization of individual time-series in training of time-series foundation models. (I would provide a more concise summary if I were able to extract it from the paper - but have not been able to do so - see comments below)

Critical strengths

The problem of data memorization in time-series foundation models is an important one (but its not at all clear that this paper makes any significant contribution to this topic)

Critical weaknesses

The single critical weakness of this paper is that the writing is incoherent, making it impossible (for this reviewer at least) to understand what the contributions, methodology, experimental methods, and conclusions are. As an example, the writing in Sections 3, 4, and 5 (pages 3 to 7, describing the methodology) is very unclear. If we look at the first few paragraphs of Section 3 for example (lines 146 to 164) the terminology is not defined and the sentences don't make sense in terms of how they flow from one to the next. What is a "break window" or an "anchor" or an "arm" or "candidate member set" or "control set"? These terms need precise definitions in the standard fashion of any technical paper if the reader is to understand the text around Equation 1. What do sentences like the following mean?: "The anchor is regression fit only before the break" and "A candidate signal needs a paired bootstrap 96% CI excluding 0" and "A separate NLL score tests membership that we create and therefore know". These sentences make no sense in the context of the sentences around them.

The section of the paper above, at the start of Section 3, is just one example. The rest of the paper continues in this fashion as far as I can tell.

My guess is that the paper is largely AI-generated (but its hard to know for sure). If it is AI-generated, I plead to the authors to not submit papers like this again going forward and subject reviewers to try to understand - this is wasting everyone's time. Please cease doing this.

Questions for authors

Did you read your own paper before submitting it? if not, please do so in the future and make sure it is readable and understandable. You owe this to the reviewers who will try to read your paper and assess it.

2 comments

Thank you for identifying where the terminology in Sections 3–5 was difficult to follow.

Sections 3–4 ask what we can infer after a model has already been trained. The anchor is a baseline forecaster fit on observations before the forecast window. We compare how much the model improves over that baseline on windows from a candidate dataset and on matched comparison windows. The result depends on the choice of baseline, and the audit yielded no statistically supported discoveries across the five datasets in Table 7.

Section 5 addresses that uncertainty by setting up the comparison before training. A recorded coin flip assigns one of two keys to each marked span. We score the same span with both keys and use a matched model trained with the opposite key assignment to account for differences in how easy the keys are to forecast. The resulting test asks whether the model favors the key it saw during training.

Thanks for the response....but my basic complaint, that the paper appears to be AI-generated and incoherent to read has not been addressed. So I will maintain the original score.

Reviewer_7wCfOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TimeCanary for training-data attribution in TSFMs. It embeds sparse Fourier signatures before data release and uses randomized keys and a role-swapped matched model to detect training-specific key preferences. Controlled continued-pretraining experiments across multiple model families demonstrate its feasibility.

In my view, training-data attribution is a highly valuable research direction for current TSFMs. As the field moves toward training larger models on increasingly large datasets, methods that provide evidence of dataset use can help data owners audit how their data are used and protect their rights. Such attribution methods could also help identify potential overlap between pretraining and evaluation data when assessing zero-shot generalization. This is particularly important for benchmark datasets intended to support fair and reliable comparisons between models.

Critical strengths
  1. The paper addresses a clearly defined research problem that has value.
  2. The paper employs rigorous controls and places considerable emphasis on identifying and ruling out spurious attribution.
  3. The paper conducts validation experiments across multiple models and datasets. Overall, the experimental evaluation is comprehensive.
Critical weaknesses
  1. The authors selected α=0.12\alpha=0.12 through the exploratory experiments reported in Table 11. This choice is not supported by a theoretical derivation or proof. Since the exploratory comparison appears to be based on single training runs, I am concerned that this value may not be suitable for all time series and settings. Its stability may depend on the random seed, model, and data.
  2. The authors should investigate the choice of three sinusoidal components ncompn_{\mathrm{comp}} in greater depth. I consider this hyperparameter to be as important as α\alpha. I therefore suggest conducting a joint grid search over the number of components and α\alpha, and examining parameter sensitivity under different settings.
  3. The results in Figure 6 appear to indicate that adding sinusoidal marks can degrade forecasting performance to some extent. Is it possible to perform attribution while keeping forecasting performance approximately unchanged?
  4. TimeCanary’s main attribution test relies on a reference model, MSM_S, that is strictly matched to the target model. In practical audits where the target model’s training pipeline cannot be fully reproduced, have the authors evaluated how mismatches in background data, training steps, or random processes affect the attribution support rate and the rate of erroneous attribution decisions?
Questions for authors

See the Critical Weaknesses section for specific Questions .

2 comments

We thank the reviewer for the thoughtful questions.

W1. Why 0.12σ, and how stable is it? At 3,000 steps, increasing the mark from 12% to 25% of the span's standard deviation barely changed the win rate (paper Table 11). Training longer at 0.12σ helped more, so we kept the smaller mark. Larger marks gave slightly higher win rates at 8,000 steps (Table A), but increased forecast error in all four comparisons in Figure 6b. Five-seed paired tests support 0.12σ as a practical default across models and datasets, with mixed results at low exposure (Table B).

W2. Why three sinusoids, and does the count matter? We chose three sinusoids to give each key a reproducible pattern across patches using a few frequencies (Section 5.1). We tested three counts at two amplitudes using the paper's Chronos-T5-Small checkpoint and datasets. The Table C shared keys, spans, and seeds, with equal mark power across counts at each amplitude. With five components, both tests supported attribution at both amplitudes after correcting for six comparisons. With one or three, both tests found preferences between keys absent from training, so neither test supported attribution in those new runs. However, three components supported attribution in the paper's runs with different keys, spans, and seeds (Table C). We believe that more draws would help assess how reliably each count works.

W3. Can we retain attribution with little forecasting cost? At 0.06σ, both tests supported attribution (paper Table 9), with a measurable cost on ETTm1 and weather. Figure 6b measures the cost for simple forecasters fitted to marked series. To isolate the cost for a foundation model, we continued training Chronos-T5-Small with and without marks at equal budgets, then scored held-out unmarked windows (Table D). At 0.03σ, error still rose on ETTm1 and the weather result was inconclusive. Neither test supported attribution because both detected preferences between keys absent from training. The experiments therefore show that a smaller mark can still work, with a forecasting trade-off.

W4. What if the reference is mismatched? The matched reference trains on the opposite key to cancel differences in key difficulty shared by both models (Section 5.2). We recombined runs with different seeds or step counts (Table E). In most studies, every pair with different seeds met the thresholds and showed no detectable preference between keys absent from training. A reference trained for fewer or more steps than the target gave a lower paired win rate, but both comparisons passed the same checks. However, a marked reference could push the paired score past its statistical thresholds even when the target never saw the mark.

Table A. Moirai-small single-model win rates (trainable encoder)

Amplitude3k steps8k steps
0.12σ0.5800.672
0.25σ0.5880.692¹
0.50σ0.5600.716¹

¹ Same keys, spans, and seed as at 0.12σ.

Table B. Paired attribution at 0.12σ

SettingSupported
Paper Table 9, first two blocks12/14
Chronos-T5-Small, 5%Yes
Moirai-1.1-R-small, 5%No (0.576 < 0.60)

Low exposure uses owner data in 5% of batches.

Table C. Win rates by component count

Components0.06σ0.12σSupport at both
10.922 / 0.7060.950 / 0.738Neither test
3 (new grid)0.922 / 0.6100.966 / 0.702Neither test
50.922 / 0.6100.954 / 0.638Both tests
3 (paper Table 9)0.924 / 0.6080.948 / 0.680Both tests

Win rates are paired (target and reference) / single-model (target alone).

Table D. Forecast error change vs. unmarked baselines

AmplitudeRidge, weatherChronos, ETTm1Chronos, weather
0.12σ+9.2%+11.6%+8.9%
0.06σ+3.5%+4.6%+7.7%
0.03σ+1.0%+3.0%+2.2%³

³ Inconclusive.

Table E. Paired test under reference mismatch

MismatchMeasureResult
Different seedsThresholds23/25
Different seeds+ key check16/25
8k / 32k stepsWin rate0.740 → 0.724
32k / 8k stepsWin rate0.904 → 0.724
Unmarked / markedThresholds only22/25

Pairs list the target first and then the reference. Arrows go from matched to mismatched at 5% exposure. The last row counts recombinations.

Thanks to the author for the reply. My concerns were resolved to some extent. I also took note of other reviewers' comments and decided to maintain my current rating but lower the confidence level of my review. Good luck.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.