- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes TimeCanary, a way to essentially watermark time series datasets so that we can identify if foundation models have been trained on proprietary datasets. This also allows for estimating if the model has truly generalized or if the model just memorized its training corpus. TimeCanary allows hosts to add a mixture of sinusoids to datasets before being used to train time series foundation models (TSFMs). Therefore, when the models are being tested on different perturbations being added to the same datasets, the model predictions can reveal if the model has memorized a particular pattern induced by the hosts.
Critical strengths
-
This is an important problem, and the proposed solution is sound.
-
The idea of having embedded and sham keys to watermark and identify datasets being used in the pretraining corpus is interesting and new to the time series community.
Critical weaknesses
-
As the foundation models keep getting much better at generalization (the likes of Chronos2, Toto2, and TimesFM3 are much better models than the ones used in the paper), how would this approach be able to clearly differentiate between the embedded and the sham keys?
-
Synthetic datasets like KernelSynth, which follows similar modeling paradigm to the keys in this paper, and more broadly, synthetic time series datasets from structural causal models have been used in the most recent models like Chronos2, Toto2, TimesFM3, etc. In such situations, how would simple keys like the Fourier sums be able to differentiate between model memorization and generalization capabilities?
Questions for authors
Check critical weaknesses. Additionally,
-
Why is the key based only on periodic waveforms with a combination of different frequencies?
-
Could it be the case for a few datasets that x(t) by itself is hard to learn? In which case, the loss for embedded and sham keys could just be dominated by the difficulty of the real dataset, and we might not be able to confidently identify the usage of a particular dataset in the training corpus?
-
Does altering the pretraining datasets with these perturbations affect the model performance in general, when compared to the base model's (trained with no perturbations) performance?
I am willing to update my score towards acceptance if the questions are sufficiently answered.
2 comments
Thank you for the response. However, my concerns are not fully addressed
Q1. Similar to periodicity, auto correlation is also typical in time series. Why not construct marks with AR processes?
Q3. The explanation says that in certain datasets the marking helps and in others, the marking hurts. This is not systematic.
I think this paper needs more rigorous experimentation and rewriting for easier understanding. Therefore, I will retain my score.
We thank the reviewer for the thoughtful review.
W1. Can newer models distinguish the training key from the sham key? We score both keyed versions of the same span (Figure 2). The single-model test asks whether the target has lower loss on its training key. An easier key can also give lower loss. In the paired test, a matched reference trains on the sham key and scores both versions. Combining the loss differences cancels any advantage shared by both models and tests whether lower loss follows the key each model saw.
We tested Chronos-2, TimesFM 2.5, and Toto 2.0 (313M) at 0.12σ and at 5e-5. The second Toto run kept the same spans, keys, and seeds. The paired test requires training, so we used TimesFM 2.5 because the public TimesFM 3 code only supports inference. The paired test supported attribution in all four settings. The single-model test only supported attribution on TimesFM 2.5 and Toto at 1e-3 (Table A).
W2. What if synthetic pretraining already teaches the model these patterns? A model that learns periodic patterns may forecast both marks well or find one easier. Random assignment gives both keys the same chance of being embedded, and the role swap cancels shared differences in key difficulty (Appendix E.3). Chronos-2’s pretraining included synthetic data from temporal causal models and KernelSynth. The paired test still detected exposure to the planted keys (Table A).
Q1. Why use multiple sinusoids? Recurring patterns in time series motivate using sinusoids. Combining a few frequencies and phases gives each key a reproducible pattern across training patches. Scaling by the span's standard deviation controls the mark's size relative to the signal (Section 5.1 and Appendix E.1). Whether more complex marks improve detection enough to justify any forecasting cost remains open.
Q2. Can a hard-to-forecast dataset hide the attribution signal? It can. Both keyed versions may have high forecast loss, but TimeCanary compares them on the same span, so the baseline loss common to both cancels (Appendix E.3). If the series affects the two keyed forecasts differently, the remaining loss difference can still be noisy. Importantly, the paired test supported attribution in separate audits on ETTm2 and MetroPT-3 (Table 9). These experiments show that the signal can be detected for an individual dataset.
Q3. Does marking hurt forecasting? The 0.06σ mark still supported attribution on Chronos-T5-Small (Table 9), with a forecasting trade-off. To isolate the cost of marking, we continued training with and without marks at equal budget. We measured scaled absolute error of the median forecast on held-out unmarked windows. The original checkpoint gives the net change from continued training. On ETTm1 and weather, marked continuation still improved forecasts over the starting checkpoint, though the gains were smaller than with unmarked continuation. On ETTm2, which was excluded from continued training, marking raised error relative to both unmarked continuation and the original checkpoint (Table B). These results show that the attribution signal can coexist with forecasting gains from continued training, with a cost that depends on where the model is evaluated.
Thank you for offering to revisit your assessment. If these results address your concerns, we would appreciate it if you reconsidered your rating.
Table A. Attribution results. Win rate is the share of spans favoring the training key. “Yes” means the test supports attribution.
¹ Chronos-2’s single-model check found a preference between two keys absent from training (p = 0.059).
² Toto’s single-model result was inconclusive (p = 0.22).
Table B. Forecast error after marking. Positive means higher error. “Clean” means unmarked continuation. “Base” means the starting checkpoint.