acceptodds
Under review as a conference paper at ICLR 2027

TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models

Abstract

Time series foundation models (TSFMs) are increasingly pretrained on large corpora, raising concerns that evaluation datasets may have been exposed during pretraining and thus yield overly optimistic performance estimates. Auditing such contamination is challenging in time series because signals are continuous and heterogeneous, and often lack corpus documentation. We formalize pretraining source-exposure auditing for TSFMs and propose TSFMAudit, a method based on probe adaptation dynamics. Our key hypothesis is that prior exposure can support efficient adaptation: a short fine-tuning probe may achieve faster loss reduction with smaller backbone movement. Candidate-to-reference differences and ratios compare this behavior across heterogeneous forecasters. We evaluate TSFMAudit on six TSFMs and 137 datasets using documented training-source evidence, with a separate evaluation on 50 TIME datasets. Against documentation-derived labels, combining reference-relative and candidate features yields candidate-averaged MCC 0.317, compared with 0.050 for the strongest tested scalar baseline and 0.169 for candidate-only dynamics. We further validate TSFMAUDIT under ground-truth membership by pretraining Chronos-T5-mini and Moirai-1.0-R-Small from random initialization on controlled dataset manifests, obtaining MCCs of 0.356 and 0.618, respectively.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Foundation Models, Time Series Forecasting, Foundation Models, Pretraining Contamination, Probe Adaptation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HSktOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents TSFMAudit, a framework for auditing dataset-level pretraining source exposure in time-series foundation models for forecasting. The method uses short fine-tuning probes to characterize the adaptation behavior of a candidate model and heterogeneous reference forecasters. It then combines candidate and reference-relative features using a HistGB auditor. The study evaluates six publicly released TSFMs on 137 datasets and includes a separate deployment on 50 TIME datasets, alongside complementary controlled-membership and transfer experiments. The strongest public-model configuration achieves a model-averaged MCC of 0.317, compared with 0.169 for candidate-only dynamics and 0.050 for the strongest tested scalar baseline. The controlled pretraining experiments provide additional evidence under known dataset membership.

Critical strengths

S1. Important problem and well-motivated approach. The paper investigates contamination auditing for TSFMs, an important issue for trustworthy evaluation. Distinguishing pretraining exposure from intrinsically easy-to-forecast time series is challenging, which motivates the use of adaptation responses and heterogeneous references. I also appreciate the paper’s careful distinction between documented source exposure and exact evaluation-window leakage.

S2. Broad and complementary empirical evidence. The authors evaluate multiple public TSFMs and datasets, with feature and reference ablations, calibration sensitivity analysis, candidate-held-out testing, and a fixed-auditor check on TIME. The manifest-controlled experiments are particularly valuable, providing evidence under known pretraining membership rather than relying solely on documentation-derived labels. The overall results are promising and support the paper’s core contribution.

Critical weaknesses

W1. The interpretation of feature contributions could be slightly clearer. The paper already includes useful feature-family and restricted-feature ablations. The relative benefits vary across references and between model-averaged and pooled metrics. A brief synthesis of Tables 4 and 9 would help readers distinguish the effectiveness of the overall framework from the contributions of individual signals.

W2. A more consolidated explanation of the operating setting would improve accessibility. The manuscript specifies family-aware splitting, development-set FP-0 calibration, and separate test evaluation. The balance of documented positive and negative labels varies considerably across candidates. A brief explanation of how this variation affects the interpretation of the operating point would be useful. Likewise, clarifying how the full public-model feature configuration differs from the core-feature configurations used in some additional experiments would help readers connect the results.

W3. Minor rendering issue in Figure 2. In the Figure 2, some dashed outlines appear incomplete or missing. Please check the figure’s rendering and restore the missing segments so that the intended visual grouping remains clear.

Questions for authors

Q1. Could the authors briefly explain how Tables 4 and 9 should be interpreted together, particularly when model-averaged and pooled metrics favor different feature choices? A short explanation distinguishing what the results establish about the complete method from what they establish about individual features would be sufficient.

Q2. Could the authors clarify the intended practical interpretation of FP-0, given that its threshold is calibrated on development negatives? If already available from the reported runs, per-candidate test FPR/TPR or confusion counts for the main configuration would be helpful. A brief explanation of the relationship among the public, controlled, and TIME feature configurations would also improve clarity.

4 comments

Thank you for the positive evaluation and for pointing out where the feature ablations, the FP-0 rule, and the controlled experiments need to be connected more clearly. We will bring the explanations below into Sections 4.3, 5.4, and 5.6 and the appendix.

W1 and Q1. Reading Tables 4 and 9 together

Table 4 isolates one feature family at a time within All Ref+Det. Full dynamics gives MCC 0.317, compared with 0.169 for loss including AULC, 0.155 for loss drop, 0.096 for displacement, and 0.066 for efficiency. Loss has the highest standalone MCC. Loss drop has the highest standalone Macro-F1 by a small margin (0.486 vs. 0.485 for loss). Table 9 instead retains the loss trajectory, including loss drop, while removing all gradient-, displacement-, and efficiency-dependent inputs. It applies this restriction in six configurations, from No Ref through All Ref.

For All Ref in Table 9, full features give MCC 0.392 vs. 0.365 for the loss-based subset when test decisions are pooled (Pair-MICRO). With equal candidate weights (Model-MACRO), the order reverses, 0.233 vs. 0.273. In fact, full features have higher pooled MCC in all six rows, while the loss-based subset has higher Model-MACRO MCC in four. Pair-MICRO computes one MCC from pooled confusion counts. Model-MACRO first averages split-level MCCs within each candidate, then gives each candidate equal weight. To illustrate the imbalance in test appearances, Kairos and the two Moirai models contribute 5,093 of 6,143 appearances in the All Ref+Det counts below. Those counts illustrate the weighting issue. They do not identify which candidates drive the Table 9 differences.

Table 9 ends at All Ref. We also restricted the public All Ref+Det auditor in an additional refit. Its Model-MACRO MCC was 0.255 when explicit efficiency terms were removed, and 0.186 when only loss-trajectory inputs were retained, compared with 0.317 for the full configuration. These within-configuration comparisons quantify the value of groups of features together with the other inputs.

The two tables thus support a specific synthesis for Section 5.4. The complete method has the highest candidate-averaged MCC among the Table 2 configurations. Loss and loss drop carry the strongest standalone signals in Table 4. The benefit of adding non-loss features depends on the reference configuration and on whether decisions are pooled or candidates are weighted equally.

W2 and Q2 (Part 1). Interpreting FP-0 in practice

FP-0 sets a threshold on development negatives. On each family-aware split the development partition fits the scorer, selects the probe epoch, and supplies the largest documented-negative score at that epoch, \tau_0. A test score is flagged only if it strictly exceeds \tau_0. Zero calibration exceedances follow by construction. The table gives the observed test FPR and TPR for All Ref+Det. The denominators are test appearances over repeated splits, so they can include the same candidate–dataset pair several times.

CandidateDocumented neg / pos (Table 1)Calibration negatives per split (≈80%)Test FPR (FP / documented-negative appearances)Test TPR (TP / documented-positive appearances)
Chronos118 / 19~9413 / 360 (3.6%)10 / 15 (66.7%)
TiRex91 / 46~7314 / 270 (5.2%)11 / 15 (73.3%)
TimesFM2.0125 / 12~10016 / 375 (4.3%)11 / 15 (73.3%)
Kairos20 / 117~1616 / 60 (26.7%)1,016 / 1,515 (67.1%)
Moirai220 / 117~16–1723 / 59 (39.0%)1,081 / 1,514 (71.4%)
Moirai15 / 132~3–41 / 20 (5.0%)808 / 1,925 (42.0%)

Appendix B.1 targets 80% of documented negatives for development and a 1:1 class ratio where feasible. With roughly 73–100 calibration negatives, Chronos, TiRex, and TimesFM2.0 have pooled FPRs of 3.6–5.2%. Each has only 15 positive test appearances, giving a 6.7-point change in pooled TPR per detection. With roughly 16–17 calibration negatives, Kairos and Moirai2 reach FPRs of 26.7% and 39.0%. For Moirai1, 5.0% represents one false positive in 20 negative appearances. Thus the development rule supplies an operating point, while its test behavior must be measured for the candidate in question. We will explain this in Section 4.3 alongside the FP-1 sensitivity results (Figures 4–5) and the threshold-free AUROC (Figure 3).

W2 and Q2 (Part 2). Public, TIME, and controlled feature configurations

The shared principle is to compare short adaptation traces. The fitted TSFMAudit auditors use HistGB, with different feature inputs and different units held out. Appendix D.2 directly compares paired traces without fitting an auditor.

ExperimentFeature inputs and evaluation protocol
Public benchmark, Table 2All Ref+Det (55 features): 48 reference-relative features (13 each for ScratchCNN, ScratchTransformer, and VisionTS; 9 for Stat after masking the four drop- and efficiency-dependent coordinates that are undefined for it; Appendix B.3) plus 7 direct candidate features [l_t, d_t, A_t, \gamma_t, w_t, a_t, \gamma_t/(l_t+\epsilon)]. Each development partition fits the scorer, selects the probe epoch, and sets the FP-0 threshold.
TIME deployment, Section 5.5 and Appendix C.3All Ref (48 features): the 48 reference-relative features above, without the 7 direct candidate features. The scorer, selected epoch, and FP-0 threshold are fixed on the 137-dataset source benchmark before TIME is scored zero-shot.
Manifest-controlled membership, Table 10 and Appendix D.1Core trace features: the core variables of Section 4.1 (loss drop d_t, backbone displacement w_t, and adaptation efficiency a_t) and their elementwise differences and ratios against the four references (Section 4.2). Membership labels come directly from the pretraining manifest (69 IN / 68 OUT).
Complementary checkpoints, Appendix D.2Core trace quantities (d_t, w_t, a_t): two checkpoints pretrained on complementary manifests are probed on identical datasets. The analysis counts the sign of the paired trace differences and fits no classifier.
Candidate-held-out transfer, Appendix D.3The core representation and four references from D.1 are retained. Each public candidate is held out entirely from fitting, epoch selection, and threshold calibration. Labels are documentation-derived, and a dataset can occur with another candidate during fitting.
Low-exposure continuation, Appendix ECore trace features and reference contrasts with a class-balanced HistGB scorer, an outer leave-one-dataset-out holdout, and inner grouped selection on the two endpoint doses (k=0 vs. k=16). Context/horizon is 512/64 (public default 128/24).

The public benchmark tests the full 55-feature configuration against documentation. TIME asks whether a pre-fitted 48-feature auditor can be deployed on an external benchmark. Table 10 tests the core features against known membership on separately pretrained checkpoints. Its test FPRs on manifest-OUT pairs are 16.67% for Chronos-T5-mini and 18.18% for Moirai-1.0-R-Small, further illustrating the distinction between FP-0 calibration and test FPR. We will summarize the configurations in the appendix and clarify the Table 10 caption and Section 5.6.

W3. Figure 2

Thank you for pointing this out. We will re-export Figure 2 with clearer dashed group boundaries and check the affected outlines in the compiled PDF at normal reading zoom.

Thank you to the authors for the thorough and candid responses. All of the points raised in my initial review have been satisfactorily resolved:

  1. Feature Contributions and Aggregation Weighting (W1 & Q1): The rebuttal clearly reconciles Tables 4 and 9 by showing that Kairos and the two Moirai models dominate the pooled Pair-MICRO counts, explaining the divergence between pooled and equal-candidate weighting. Furthermore, the additional refit and permutation analyses on the 55-feature All Ref+Det setup (where removing explicit efficiency terms drops Model-MACRO MCC from 0.317 to 0.255, and restricting to loss trajectory alone drops it to 0.186) confirm that the full auditor genuinely benefits from combining loss, gradient, and adaptation-efficiency signals.
  2. Operating Point Behavior and Experimental Configurations (W2 & Q2): The newly provided per-candidate confusion counts and TPR/FPR breakdown make the practical behavior of FP-0 concrete: when enough documented negatives are available for calibration, test FPR remains well-controlled at 3.6%–5.2% with 66.7%–73.3% TPR, whereas sparse negative documentation naturally increases threshold variance. The side-by-side summary table of feature configurations across the public benchmark, TIME deployment , and controlled/transfer evaluations will also make the paper much easier to navigate.
  3. Additional Evidence Across the Rebuttal: Beyond my own questions, I appreciate the new experiments added in response to the other reviewers—particularly the comparisons against adapted time-series MIA baselines (the dataset-level Koren et al. adaptation and 8/16-shadow-model LiRA on controlled Chronos-T5-mini), the all-subsets reference ablation, and the synthetic-only Chronos-2 negative control. Together with the manifest-controlled results in Table 10, these experiments further strengthen the empirical case for reference-relative adaptation auditing.

I trust the authors will integrate these tables, explanations, and the updated Figure 2 into the final revision. I am raising my Soundness rating to 3 and increasing my overall score to 4.

Thank you for your encouraging feedback and recognition of our contributions. We sincerely appreciate your thoughtful evaluation and will carefully incorporate the suggested improvements into the final revision.

Reviewer_iC58OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Benchmarking time series foundation models has become difficult due to data contamination. The authors propose TSFMAudit which produces a contamination score using probe adaptation dynamics. They find that when finetuning a TSFM on pretrained data the models learn faster (larger loss drops) and the model parameters change less. TSFMAudit trains a histGB model on both the fine-tuning statistics of the TSFM in question and reference models and finds that their approach outperforms baseline approaches with an MCC of 0.317.

Critical strengths
  • The paper targets an important problem in the field where benchmark results are hard to trust due to data contamination. They are the first to study contamination auditing specifically in TSFMs.
  • The paper is well written and easy to understand. Experimental setups are well defined.

The research community would benefit from this line of work. The methodology and motivation are well setup but the paper is held back by a weaker experiment section.

Critical weaknesses
  • Experimental results are weak and point to method instability
    • Results vary significantly amongst candidate TSFMs. While a per candidate table version of Table 2 is not included, the large inter-model differences are worrying: TSFMAudit All Ref+Det MCC was 0.317+/- 0.211 and Figure 6 show that TSFMAudit performance is highly heterogeneous across models.
    • Included baselines are poor with LiRA and frequency baselines having MCC ~0. The authors fail to cite prior time series MIA and LiRA adaptations [1,2]. Additionally, Macro-F1 scores are only marginally better than a random coin flip which would have a macro-F1 of ~0.42 compared to the best setup ref+det 0.576 while the baselines are even below a fair coin.
    • The paper claims that a contaminated model would “exhibit faster loss reduction with relatively little backbone movement” (L191-192). However, their feature ablation experiment indicates that adaptation efficiency and parameter displacement on their own are non-discriminative and that using loss (w/ reference models) is equivalent to the candidate-only dynamics from Table 2. It’s unclear from this ablation that the gains are from an adaptation efficiency mechanism rather than a flexible model learning something else.
  • The paper doesn’t explain where the errors occur. They include the aggregate numerical scores but don’t give a full per-dataset table or examples of false positives or false negatives. The paper would benefit from expressing how errors correlate with dataset features like frequency/seasonality or domain. This and HistGB feature importances can give an idea as to what the TSFMAudit system model is learning.

Weak and inconsistent results alongside easy to beat baselines makes me hesitant to accept this paper. Provided stronger evidence and/or explanation of errors, I'll revisit potentially raising the score to a weak accept.

[1] Johansson, Nicolas, et al. "Privacy risks in time series forecasting: User-and record-level membership inference." 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2026.

[2] Koren, Noam, et al. "Membership inference attacks against time-series models." arXiv preprint arXiv:2407.02870 (2024).

Questions for authors

An interesting experiment would be to look at how TSFMAudit would perform on a fully synthetic model like Chronos-2-synth or PFN based models given that we know there won’t be any data contamination.

5 comments

Thank you for identifying the evidence needed to judge the public-model results. Below we give the per-candidate scores, two dataset-level adaptations of the MIA methods you cited, group-wise feature attributions, examples of errors, and the synthetic negative control. We identify the evaluation setting for each comparison.

1. Heterogeneity across candidates

The table breaks down the All Ref+Det row of Table 2 by candidate and adds the dataset-level Koren baseline described in item 2. MCC and Macro-F1 average the split-level metrics within each candidate. The two counts on the right sum test appearances over the evaluated splits, so a dataset can contribute more than once.

CandidateDocumented negatives / positives (Table 1)MCC, ours / Koren [2] adaptationMacro-F1, ours / Koren [2] adaptationTest TP / positive appearancesTest FP / negative appearances
Chronos118 / 190.484 / 0.2650.723 / 0.61610 / 1513 / 360
TiRex91 / 460.509 / 0.2110.731 / 0.59211 / 1514 / 270
TimesFM2.0125 / 120.528 / 0.0660.738 / 0.52611 / 1516 / 375
Kairos20 / 1170.165 / 0.0780.472 / 0.3961,016 / 1,51516 / 60
Moirai220 / 1170.138 / 0.1320.482 / 0.4441,081 / 1,51423 / 59
Moirai15 / 1320.077 / 0.0250.310 / 0.359808 / 1,9251 / 20
Equal-candidate mean—0.317 / 0.1290.576 / 0.489——

The SD of 0.211 in Table 2 is the spread across these six candidate means, and the means fall into two groups. Chronos, TiRex, and TimesFM2.0 reach MCCs of 0.484–0.528 (Macro-F1 0.723–0.738), whereas Kairos, Moirai2, and Moirai1 reach 0.077–0.165. The division tracks the documentation labels in Table 1: the first group has 91–125 documented negatives, the second only 20, 20, and 5.

The development sampler targets 80% of the documented negatives and a 1:1 positive-to-negative ratio where labels permit (Appendix B.1). Chronos, TiRex, and TimesFM2.0 have roughly 73–100 calibration negatives per split. Their pooled FPRs are 3.6–5.2%, but each contributes only 15 positive test appearances in these counts. One detection changes its pooled TPR by 6.7 percentage points. Kairos and Moirai2 have roughly 16–17 calibration negatives and pooled FPRs of 26.7% and 39.0%. Moirai1 has roughly 3–4 calibration negatives, one false positive in 20 negative appearances, and 808 detections in 1,925 positive appearances.

Figure 6 also shows substantial split-to-split variation for Chronos and TiRex in the configurations displayed. In addition to the calibration imbalance above, the public evaluation relies on incomplete training records. A source without documented use is labeled negative, so an undocumented exposure can count as a false positive. With only five documented negatives for the released Moirai1, fitting the scorer and estimating its operating point are especially sensitive to which examples enter development. The public comparison alone cannot separate these effects from differences in architecture and pretraining corpus. We will include the per-candidate results for all Table 2 methods in the appendix.

The controlled pretraining experiment tests both architectures against known membership. With 69 sources included and 68 excluded, the core traces and four references yield MCC 0.356 and Macro-F1 0.665 for Chronos-T5-mini, and MCC 0.618 and Macro-F1 0.809 for Moirai-1.0-R-Small (Table 10). Both independently trained checkpoints show a membership signal, with the larger effect on Moirai.

The distinction between thresholded decisions and ranking also matters for use in practice. On the six public candidates, mean MCC is 0.317 and Macro-F1 is 0.576 at FP-0, while mean AUROC reaches 0.806 (Figure 3). A maintainer can already use those scores to decide which training records to inspect first. In TabArena, a benchmark for tabular foundation models, an anomalous score prompted a provenance check that identified likely training exposure on two datasets. TSFMAudit supplies a ranked list of sources for precisely this kind of follow-up. TabArena issue #648

2. Stronger baselines and chance level

We will discuss Johansson et al. [1] and Koren et al. [2] in Section 2 and explain what changes when the unit of inference is an entire dataset source. We evaluated the following two adaptations.

Koren et al. [2] on the six public candidates. We averaged the 132-dimensional window vectors (seasonality, trend, forecast, and error statistics) within each dataset. The comparison uses the same forecast windows, proxy labels, HistGB settings, FP-0 rule, and matched test splits for the two methods. The equal-candidate MCC gain is +0.188, with a 95% interval of [0.134, 0.223] from 2,000 paired Bayesian-bootstrap draws over 62 source families. Leaving out one candidate at a time gives gains of +0.133 to +0.224. All six MCC differences favor TSFMAudit, although the difference for Moirai2 is small (0.138 vs. 0.132). Macro-F1 favors TSFMAudit on five candidates and favors the Koren adaptation on Moirai1 (0.359 vs. 0.310).

Shadow-model LiRA [1] under known membership. We evaluated a source-level adaptation with 8 and 16 shadow models on controlled Chronos-T5-mini. Table 10 supplies the TSFMAudit row. The two LiRA rows are additional experiments. The audit uses development FP-0, while LiRA uses its native decision threshold.

Controlled Chronos-T5-mini resultMCCMacro-F1Operating point
TSFMAudit (Table 10, Appendix D.1)0.35630.6645Development FP-0
LiRA, 8 shadow models0.10150.4096Native LiRA threshold
LiRA, 16 shadow models0.16850.4753Native LiRA threshold

Increasing the shadow pool raises LiRA's MCC from 0.1015 to 0.1685 and Macro-F1 from 0.4096 to 0.4753. Both reported operating points fall below the Table 10 result. Shadow pretraining for this architecture took about 53 and 106 A800 GPU-hours, respectively. TSFMAudit requires no shadow pretraining.

Chance level. In 10,000 simulations on the evaluated splits, Macro-F1 averaged 0.359 for a fair coin and 0.419 for random predictions drawn at the development positive rate. The full auditor scores 0.576. The strongest scalar baseline, Loss Drop Rate (0.394), lies between these two chance references. All other scalar baselines in Table 2 score below 0.359.

[1] Johansson et al. (2026). Privacy risks in time series forecasting: User- and record-level membership inference. SaTML.

[2] Koren et al. (2024). Membership inference attacks against time-series models. arXiv:2407.02870.

3. What the scorer uses, and the evidence for the adaptation pattern

The two MCCs of 0.169 come from different feature sets. Candidate-only dynamics in Table 2 uses all seven direct candidate features. The Loss row of Table 4 uses the loss level and area under the loss curve (AULC), including their reference contrasts. The equal scores show that neither feature set alone reaches the full auditor's 0.317.

To test which inputs the flexible scorer uses, we first refit HistGB on restricted feature groups in the public All Ref+Det analysis. With all 55 features the MCC is 0.317. It falls to 0.255 without explicit efficiency terms, 0.151 without gradient features, and 0.186 with loss trajectory alone (level, area, drop, and loss–drop interaction). We then kept HistGB fixed and permuted feature groups. Permuting loss level and area, gradient features, or loss drop lowers MCC by 0.175, 0.153, and 0.144, respectively.

These are complementary measures of what HistGB uses. Refitting measures performance with a group removed. Permutation measures the fitted scorer's reliance on the group. The drops are conditional on the remaining, partly related inputs and cannot be added to apportion the gain among mechanisms. Loss and gradient features both contribute predictive information, and the explicit efficiency terms improve the full configuration on these splits.

The most direct test of the proposed adaptation pattern is the classifier-free comparison in Appendix D.2. On the same 137 datasets, two Chronos checkpoints of identical architecture were pretrained on complementary manifests. The checkpoint exposed to a dataset had larger loss reduction on 108 datasets, smaller backbone displacement on 99, and higher efficiency on 108. These are separate directional counts for the three quantities. The audited data and preprocessing are fixed within each pair, while the other pretraining sources also differ. In a separate experiment, Table 10 combines the core traces and their reference contrasts under known membership and obtains MCCs of 0.356 and 0.618 on Chronos and Moirai, respectively.

We will connect Observation 1 (lines 185–195) to this paired evidence and make explicit in Section 5.4 that the full public scorer also uses loss- and gradient-derived inputs. This preserves the empirical claim about the direction of adaptation and identifies what improves prediction in the public audit.

4. Where errors occur

A recurring false positive. Table 6 labels traffic_hourly as a documented negative for Chronos. In all three of its test appearances it ranked first among 24 held-out documented negatives and exceeded the threshold. It accounts for 3 of Chronos's 13 false-positive appearances. Six other documented-negative transport datasets appeared eight times across those splits and none was flagged. This identifies a specific source for provenance follow-up. We score all three flags as errors against the documentation.

A recurring false negative. Table 6 records tourism_monthly as positive for five candidates. Misses / test appearances were 13/14 for Kairos, 11/15 for Moirai2, 5/15 for Moirai1, 1/3 for Chronos, and 0/5 for TiRex. For Kairos, the source scored below the median of the other 100 held-out documented positives in all 14 appearances, including its one detection. The score ranking helps explain why this source was repeatedly missed.

Frequency, domain, and seasonality. Among documented positives, the candidate-averaged miss rate was 0.633 for hourly data and 0.517 for daily data. It was 0.643 for energy/buildings and 0.269 for the M4 group. For Kairos, a larger share of context-window spectral energy at the dominant peak was associated with fewer misses (Spearman ρ=−0.503\rho=-0.503, 117 pairs from 49 families). TimesFM2.0 gave the opposite sign (ρ=+0.394\rho=+0.394), based on only 8 pairs from 5 families. The associations describe these candidates and groups. We will include per-dataset FP and FN counts with test denominators in the appendix.

5. A Chronos-2 negative control with synthetic-only pretraining

Following your suggestion, we audited a Chronos-2 checkpoint pretrained exclusively on TempoPFN-generated series. On the 97 GIFT-Eval configurations used for this check it obtained normalized MASE 0.736 and normalized CRPS 0.509 (seasonal naive = 1, lower is better). This gives us a checkpoint with forecasting skill on real series and synthetic-only pretraining.

We used the 48-feature All Ref representation, the same feature choice as the TIME deployment in Section 5.5. Here the evaluation held out each of 62 source families in turn. The scorer, probe epoch, and FP-0 threshold were determined from the remaining source families of the public Chronos candidate. We then scored the synthetic checkpoint on the held-out family. Observations from the synthetic checkpoint were used only for evaluation.

The auditor flagged 13 of 137 real datasets (9.5%) and left 124 unflagged (90.5%). All 13 flags are false positives under this synthetic-only training setup. This measures false alarms when a forecaster generalizes to real data without training on the audited sources. Sensitivity to actual exposure is evaluated separately in the manifest-IN/OUT experiment (Table 10).

Reviewer_59fUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Benchmarking Time Series Foundation Models (TSFMs) is becoming increasingly challenging because these models are pretrained on large-scale, multi-source datasets whose composition is often not fully documented. As a result, benchmarks may overlap with pretraining data, leading to contamination and overstated performance estimates. To address this issue, the paper introduces TSFMAudit, a contamination-auditing framework that estimates contamination risk by analyzing the adaptation dynamics of fine-tuning probes relative to a set of reference forecasting models. As the first framework specifically designed for contamination auditing in TSFMs, TSFMAudit is evaluated against several baseline methods and demonstrates increased performance across 137 datasets and six time-series foundation models.

Critical strengths

Strength 1:

  • The paper is well-positioned within the existing literature and proposes a novel and well-motivated framework.

Strength 2:

  • The experimental evaluation is thorough, covering multiple objectives and evaluating across multiple models, datasets, and baselines, which highlights the enhancements provided by TSFMAudit.
Critical weaknesses

Weakness 1:

  • Although the proposed method seems to outperform the considered baselines, the overall performance remains relatively modest, with the best TSFMAudit variant achieving a mean Macro-F1 score of only 0.567. This raises questions about the effectiveness of the framework and suggests that there remains room for improvement.

Weakness 2:

  • The results obtained with the Reference Suite appear highly dependent on the particular models included as shown in the ablation. However, the rationale behind selecting these four reference models is insufficiently explained, making the choice seem somewhat arbitrary. It remains unclear whether the selected reference models are optimal, other model choices may have a more beneficial impact on the framework's performance and robustness.
Questions for authors

Clarification requests:

  1. Provide more detailed captions and explanations for tables and figures so that they are largely self-contained and can be understood without extensive reference to the main text.
  2. Include a brief explanation of the proxy CC in the main text, particularly within Section 3 (Objective), rather than relegating its description solely to the appendix. This would improve the readability of the methodology.
  3. Report statistical significance analyses, such as a Critical Difference (CD) diagram, to support the comparisons presented in Table 2 and strengthen the empirical claims.
  4. Investigate additional models for inclusion in the Reference Suite. Exploring a broader range of reference models could further improve the performance and robustness of the proposed framework.
  5. Expand the conclusion section by more clearly highlighting the main findings and contributions of the work. Additionally, consider including a dedicated Future Work section to discuss next steps for enhancing the proposed framework.
5 comments

Thank you for asking how far the public-model scores support the audit and how sensitive they are to the choice of references. We report a reference-subset analysis, a fifth-reference experiment on Chronos, and significance tests for Table 2 below.

W1. What the reported scores support

On the six public TSFMs, All Ref+Det reaches mean MCC 0.317 and Macro-F1 0.576 at the FP-0 operating point (Table 2). Candidate-only dynamics reaches 0.169 and 0.474, and the strongest scalar baseline reaches 0.050 and 0.394, respectively. The full auditor improves on both, although its thresholded predictions still leave room for improvement.

Two features of the public evaluation affect how we interpret those scores. A negative label means that official sources do not document pretraining use, so an undocumented exposure can count as a false positive (Appendix A.2). Label balance also varies across candidates (Table 1). Chronos, TiRex, and TimesFM2.0 have roughly 73–100 documented negatives per split for FP-0 calibration, with observed test FPRs of 3.6–5.2% across repeated test appearances. Kairos and Moirai2 have only about 16–17 calibration negatives, and their test FPRs reach 26.7% and 39.0%.

Table 10 evaluates two independently pretrained checkpoints with known, nearly balanced membership (69 sources included, 68 excluded). Using the core adaptation traces and the same four references, TSFMAudit reaches MCC 0.618 and Macro-F1 0.809 on Moirai-1.0-R-Small, and MCC 0.356 and Macro-F1 0.665 on Chronos-T5-mini. The auditor thus distinguishes included from excluded sources in both architectures, most clearly on Moirai.

For a benchmark maintainer, a practical question is which sources to investigate first. Alongside balanced accuracy of 0.761 at FP-0, the public auditor achieves mean AUROC 0.806 across thresholds (Figure 3). This ranking is already useful for directing limited provenance checks to the highest-scoring sources. A case from tabular foundation models illustrates the value of that workflow. In TabArena, an anomalous benchmark result prompted an investigation that identified two datasets likely to have been used in training. TSFMAudit gives maintainers a systematic way to prioritize such checks. Training and preprocessing records can then establish what was actually used. TabArena issue #648

Q3. Statistical significance for Table 2

We assessed the aggregate comparison in Table 2 in two ways. The first uses the six candidate models as blocks. The second keeps observations from the same source family together, including repeated test appearances.

Across all 16 methods, the Friedman test uses the six candidates as blocks. With 100,000 within-candidate permutations, the test statistics were 56.797 for MCC, 50.385 for Macro-F1, and 56.715 for balanced accuracy (permutation p ≈ 10⁻⁵ for each). This is an omnibus difference among methods on these candidates.

For the pairwise analysis, we compared All Ref+Det with each of the other 15 methods using a two-sided paired permutation test with the source family as the permutation unit (62 families, 20,000 permutations). Holm correction covers the 15 comparisons within each metric. Each cell gives the raw and adjusted p-values, in that order. Bold marks adjusted values below 0.05.

All Ref+Det compared withMCCMacro-F1Balanced accuracy
Raw Loss0.05115/0.097350.00330/0.019800.01090/0.02450
Loss Drop Rate0.04065/0.097350.00705/0.030250.01805/0.02450
LiRA Ratio (ScratchCNN)0.00940/0.078650.00015/0.001800.00090/0.01080
LiRA Ratio (ScratchTransformer)0.00840/0.078650.00005/0.000750.00100/0.01100
LiRA Ratio (Stat)0.01080/0.078650.00005/0.000750.00210/0.01680
LiRA Ratio (VisionTS)0.00805/0.078650.00005/0.000750.00135/0.01215
TS MinK FFT (K = 10%)0.00770/0.078650.00055/0.005000.00500/0.02450
TS MinK FFT (K = 20%)0.01035/0.078650.00070/0.005600.00490/0.02450
TS MinK FFT (K = 30%)0.00805/0.078650.00050/0.005000.00285/0.01820
Candidate-only dynamics (No Ref)0.00640/0.076800.00605/0.030250.00045/0.00585
+ ScratchCNN0.03245/0.097350.02640/0.030250.00695/0.02450
+ ScratchTransformer0.00525/0.068250.01100/0.030250.00115/0.01150
+ Stat0.00715/0.078650.00850/0.030250.00005/0.00075
+ VisionTS0.00045/0.006300.00115/0.008050.00005/0.00075
All Ref0.00020/0.003000.00015/0.001800.00260/0.01820

Macro-F1 and balanced accuracy favor All Ref+Det over all 15 comparators after Holm correction (adjusted p ≤ 0.031 and ≤ 0.025, respectively). For MCC, two comparisons meet the corrected threshold, + VisionTS (0.00630) and All Ref (0.00300). In particular, adding the seven candidate features to All Ref improves all three metrics. Against candidate-only dynamics, the MCC gain is +0.148 (raw p = 0.00640, adjusted p = 0.07680). The Macro-F1 and balanced-accuracy gains over candidate-only dynamics remain significant after correction.

These tests condition on the evaluated splits, fitted scores, and thresholds. We will add the pairwise results to the appendix and show the mean ranks across the six candidates in the requested Critical Difference diagram. The diagram summarizes candidate-level ranks. The p-values above come from the source-family analysis.

W2. Why these four references

Section 4 motivates references as different views of how difficult the same dataset is to forecast. We chose four forecasters that supply these views without pretraining on real time-series corpora. ScratchCNN and ScratchTransformer learn from random initialization with convolution and self-attention, respectively. Stat provides a statistical anchor using forecasters such as ETS. VisionTS transfers representations learned from natural images. This gives each reference a distinct role in the comparison (Section 5.1).

We kept the seven direct candidate features (+Det) fixed and varied which reference blocks were appended. We evaluated all 15 non-empty subsets and the candidate-only baseline. Each configuration is summarized across the six candidates. The table gives the mean across configurations of each size and the lowest and highest MCC among those configurations.

ReferencesConfigurationsMean MCCMCC range across configurationsMean Macro-F1Mean AUROC
010.1691—0.47360.7044
140.20410.1870–0.21850.50750.7503
260.25400.2273–0.29680.53760.7766
340.29690.2845–0.31150.56160.8011
410.3170—0.57580.8058

All four single-reference configurations with +Det exceed the candidate-only MCC of 0.1691 (range 0.1870–0.2185). Table 2 asks a different question. Its single-reference rows use only the reference-relative block, giving MCCs of 0.173 for ScratchCNN, 0.123 for ScratchTransformer, 0.218 for Stat, and 0.096 for VisionTS. Thus, retaining the candidate features matters when interpreting the single-reference result. Both comparisons average across candidates.

As suite size increases from zero to four, mean MCC, Macro-F1, and AUROC all increase. The full suite has higher MCC (0.3170) than even the best three-reference subset (0.3115). Composition still matters. The best pair (0.2968) scores above the weakest triple (0.2845), and differences within each size show why the choice of members deserves attention. The 0.0055 MCC margin over the best triple is descriptive. This supports using the four complementary references as a default in the tested suite. The candidate averages leave open whether a given addition helps each individual model, and other forecasters may improve the suite further.

Q4. An additional reference

We tested a fifth reference pretrained on synthetic time series, adding a time-series-pretrained transformer to the four views above. The checkpoint is a Chronos-2 architecture trained on TempoPFN-generated series. Used separately as an audit target, it was flagged on 13 of the 137 real datasets by the 48-feature All Ref auditor. The test of its value as a reference is the downstream comparison below.

On the public Chronos candidate, adding this reference raises MCC from 0.484 to 0.505 and Macro-F1 from 0.723 to 0.730. AUROC falls from 0.906 to 0.892. The fifth reference improves the two thresholded metrics on this candidate, with a trade-off in ranking. We will report all three metrics and identify evaluation on the other five candidates as future work.

Q1 and Q2. Captions and the documentation proxy

We will expand the existing proxy sentence in the Objective paragraph of Section 3. For a public candidate–dataset pair, C~=1\tilde C=1 means the candidate's paper, model card, dataset card, or official repository explicitly documents pretraining on the source. A zero means we found no such record. This label identifies documented source use. The idealized CC is the fraction of audited samples derived from pretraining source series, and identifying reuse of a particular evaluation window requires finer provenance (Appendices A.2–A.3).

For Tables 1–5 and Figures 1–3 we will put the comparison, metric direction, and unit of aggregation in the captions where appropriate. For example, the Table 2 caption will explain that higher scores indicate better agreement with the candidate–dataset documentation labels, that FP-0 allows zero development-negative exceedances, and that entries report the mean and sample SD of the six candidate means. The Figure 1 caption will identify the TiRex traces and documentation-label groups. Figure 3 will distinguish reference-relative blocks from inputs that include the direct candidate block. Figure 6 will identify error bars as within-candidate split SD.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback on baseline strength (iC58), experimental reliability (59fU, iC58, HSkt), and practical relevance (59fU, HSkt). Below, we summarize the main revisions and clarifications made in response.

Stronger baselines. We added dataset-level adaptations of two time series MIA methods (Koren et al. and Johansson et al.) and outperformed both in their respective evaluation settings(MCC 0.317 vs. 0.129 and 0.356 vs. 0.169). This likely occurs because TSFMs already forecast many unseen time series well without prior exposure, which makes prediction-based membership signals less discriminative.

Controlled validation. Beyond documentation-derived labels, our original submission already included controlled pretraining of Chronos and Moirai from scratch, with 69 datasets included and 68 excluded. Under known membership, our method achieved MCCs of 0.356 and 0.618. Complementary checkpoint experiments and a synthetic negative control further support the auditing approach.

Practical relevance. Modern TSFMs often rely on large collections of public datasets without disclosing complete pretraining manifests, which makes contamination difficult to verify. To our knowledge, TSFMAudit is the first method specifically designed to detect pretraining data contamination in TSFMs. By identifying and prioritizing potentially exposed sources (mean AUROC 0.806), it helps improve fairness and transparency in increasingly competitive TSFM benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.