- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents TSFMAudit, a framework for auditing dataset-level pretraining source exposure in time-series foundation models for forecasting. The method uses short fine-tuning probes to characterize the adaptation behavior of a candidate model and heterogeneous reference forecasters. It then combines candidate and reference-relative features using a HistGB auditor. The study evaluates six publicly released TSFMs on 137 datasets and includes a separate deployment on 50 TIME datasets, alongside complementary controlled-membership and transfer experiments. The strongest public-model configuration achieves a model-averaged MCC of 0.317, compared with 0.169 for candidate-only dynamics and 0.050 for the strongest tested scalar baseline. The controlled pretraining experiments provide additional evidence under known dataset membership.
Critical strengths
S1. Important problem and well-motivated approach. The paper investigates contamination auditing for TSFMs, an important issue for trustworthy evaluation. Distinguishing pretraining exposure from intrinsically easy-to-forecast time series is challenging, which motivates the use of adaptation responses and heterogeneous references. I also appreciate the paper’s careful distinction between documented source exposure and exact evaluation-window leakage.
S2. Broad and complementary empirical evidence. The authors evaluate multiple public TSFMs and datasets, with feature and reference ablations, calibration sensitivity analysis, candidate-held-out testing, and a fixed-auditor check on TIME. The manifest-controlled experiments are particularly valuable, providing evidence under known pretraining membership rather than relying solely on documentation-derived labels. The overall results are promising and support the paper’s core contribution.
Critical weaknesses
W1. The interpretation of feature contributions could be slightly clearer. The paper already includes useful feature-family and restricted-feature ablations. The relative benefits vary across references and between model-averaged and pooled metrics. A brief synthesis of Tables 4 and 9 would help readers distinguish the effectiveness of the overall framework from the contributions of individual signals.
W2. A more consolidated explanation of the operating setting would improve accessibility. The manuscript specifies family-aware splitting, development-set FP-0 calibration, and separate test evaluation. The balance of documented positive and negative labels varies considerably across candidates. A brief explanation of how this variation affects the interpretation of the operating point would be useful. Likewise, clarifying how the full public-model feature configuration differs from the core-feature configurations used in some additional experiments would help readers connect the results.
W3. Minor rendering issue in Figure 2. In the Figure 2, some dashed outlines appear incomplete or missing. Please check the figure’s rendering and restore the missing segments so that the intended visual grouping remains clear.
Questions for authors
Q1. Could the authors briefly explain how Tables 4 and 9 should be interpreted together, particularly when model-averaged and pooled metrics favor different feature choices? A short explanation distinguishing what the results establish about the complete method from what they establish about individual features would be sufficient.
Q2. Could the authors clarify the intended practical interpretation of FP-0, given that its threshold is calibrated on development negatives? If already available from the reported runs, per-candidate test FPR/TPR or confusion counts for the main configuration would be helpful. A brief explanation of the relationship among the public, controlled, and TIME feature configurations would also improve clarity.
4 comments
W2 and Q2 (Part 2). Public, TIME, and controlled feature configurations
The shared principle is to compare short adaptation traces. The fitted TSFMAudit auditors use HistGB, with different feature inputs and different units held out. Appendix D.2 directly compares paired traces without fitting an auditor.
| Experiment | Feature inputs and evaluation protocol |
|---|---|
| Public benchmark, Table 2 | All Ref+Det (55 features): 48 reference-relative features (13 each for ScratchCNN, ScratchTransformer, and VisionTS; 9 for Stat after masking the four drop- and efficiency-dependent coordinates that are undefined for it; Appendix B.3) plus 7 direct candidate features [l_t, d_t, A_t, \gamma_t, w_t, a_t, \gamma_t/(l_t+\epsilon)]. Each development partition fits the scorer, selects the probe epoch, and sets the FP-0 threshold. |
| TIME deployment, Section 5.5 and Appendix C.3 | All Ref (48 features): the 48 reference-relative features above, without the 7 direct candidate features. The scorer, selected epoch, and FP-0 threshold are fixed on the 137-dataset source benchmark before TIME is scored zero-shot. |
| Manifest-controlled membership, Table 10 and Appendix D.1 | Core trace features: the core variables of Section 4.1 (loss drop d_t, backbone displacement w_t, and adaptation efficiency a_t) and their elementwise differences and ratios against the four references (Section 4.2). Membership labels come directly from the pretraining manifest (69 IN / 68 OUT). |
| Complementary checkpoints, Appendix D.2 | Core trace quantities (d_t, w_t, a_t): two checkpoints pretrained on complementary manifests are probed on identical datasets. The analysis counts the sign of the paired trace differences and fits no classifier. |
| Candidate-held-out transfer, Appendix D.3 | The core representation and four references from D.1 are retained. Each public candidate is held out entirely from fitting, epoch selection, and threshold calibration. Labels are documentation-derived, and a dataset can occur with another candidate during fitting. |
| Low-exposure continuation, Appendix E | Core trace features and reference contrasts with a class-balanced HistGB scorer, an outer leave-one-dataset-out holdout, and inner grouped selection on the two endpoint doses (k=0 vs. k=16). Context/horizon is 512/64 (public default 128/24). |
The public benchmark tests the full 55-feature configuration against documentation. TIME asks whether a pre-fitted 48-feature auditor can be deployed on an external benchmark. Table 10 tests the core features against known membership on separately pretrained checkpoints. Its test FPRs on manifest-OUT pairs are 16.67% for Chronos-T5-mini and 18.18% for Moirai-1.0-R-Small, further illustrating the distinction between FP-0 calibration and test FPR. We will summarize the configurations in the appendix and clarify the Table 10 caption and Section 5.6.
W3. Figure 2
Thank you for pointing this out. We will re-export Figure 2 with clearer dashed group boundaries and check the affected outlines in the compiled PDF at normal reading zoom.
Thank you to the authors for the thorough and candid responses. All of the points raised in my initial review have been satisfactorily resolved:
- Feature Contributions and Aggregation Weighting (W1 & Q1): The rebuttal clearly reconciles Tables 4 and 9 by showing that Kairos and the two Moirai models dominate the pooled Pair-MICRO counts, explaining the divergence between pooled and equal-candidate weighting. Furthermore, the additional refit and permutation analyses on the 55-feature All Ref+Det setup (where removing explicit efficiency terms drops Model-MACRO MCC from 0.317 to 0.255, and restricting to loss trajectory alone drops it to 0.186) confirm that the full auditor genuinely benefits from combining loss, gradient, and adaptation-efficiency signals.
- Operating Point Behavior and Experimental Configurations (W2 & Q2): The newly provided per-candidate confusion counts and TPR/FPR breakdown make the practical behavior of FP-0 concrete: when enough documented negatives are available for calibration, test FPR remains well-controlled at 3.6%–5.2% with 66.7%–73.3% TPR, whereas sparse negative documentation naturally increases threshold variance. The side-by-side summary table of feature configurations across the public benchmark, TIME deployment , and controlled/transfer evaluations will also make the paper much easier to navigate.
- Additional Evidence Across the Rebuttal: Beyond my own questions, I appreciate the new experiments added in response to the other reviewers—particularly the comparisons against adapted time-series MIA baselines (the dataset-level Koren et al. adaptation and 8/16-shadow-model LiRA on controlled Chronos-T5-mini), the all-subsets reference ablation, and the synthetic-only Chronos-2 negative control. Together with the manifest-controlled results in Table 10, these experiments further strengthen the empirical case for reference-relative adaptation auditing.
I trust the authors will integrate these tables, explanations, and the updated Figure 2 into the final revision. I am raising my Soundness rating to 3 and increasing my overall score to 4.
Thank you for your encouraging feedback and recognition of our contributions. We sincerely appreciate your thoughtful evaluation and will carefully incorporate the suggested improvements into the final revision.
Thank you for the positive evaluation and for pointing out where the feature ablations, the FP-0 rule, and the controlled experiments need to be connected more clearly. We will bring the explanations below into Sections 4.3, 5.4, and 5.6 and the appendix.
W1 and Q1. Reading Tables 4 and 9 together
Table 4 isolates one feature family at a time within All Ref+Det. Full dynamics gives MCC 0.317, compared with 0.169 for loss including AULC, 0.155 for loss drop, 0.096 for displacement, and 0.066 for efficiency. Loss has the highest standalone MCC. Loss drop has the highest standalone Macro-F1 by a small margin (0.486 vs. 0.485 for loss). Table 9 instead retains the loss trajectory, including loss drop, while removing all gradient-, displacement-, and efficiency-dependent inputs. It applies this restriction in six configurations, from No Ref through All Ref.
For All Ref in Table 9, full features give MCC 0.392 vs. 0.365 for the loss-based subset when test decisions are pooled (Pair-MICRO). With equal candidate weights (Model-MACRO), the order reverses, 0.233 vs. 0.273. In fact, full features have higher pooled MCC in all six rows, while the loss-based subset has higher Model-MACRO MCC in four. Pair-MICRO computes one MCC from pooled confusion counts. Model-MACRO first averages split-level MCCs within each candidate, then gives each candidate equal weight. To illustrate the imbalance in test appearances, Kairos and the two Moirai models contribute 5,093 of 6,143 appearances in the All Ref+Det counts below. Those counts illustrate the weighting issue. They do not identify which candidates drive the Table 9 differences.
Table 9 ends at All Ref. We also restricted the public All Ref+Det auditor in an additional refit. Its Model-MACRO MCC was 0.255 when explicit efficiency terms were removed, and 0.186 when only loss-trajectory inputs were retained, compared with 0.317 for the full configuration. These within-configuration comparisons quantify the value of groups of features together with the other inputs.
The two tables thus support a specific synthesis for Section 5.4. The complete method has the highest candidate-averaged MCC among the Table 2 configurations. Loss and loss drop carry the strongest standalone signals in Table 4. The benefit of adding non-loss features depends on the reference configuration and on whether decisions are pooled or candidates are weighted equally.
W2 and Q2 (Part 1). Interpreting FP-0 in practice
FP-0 sets a threshold on development negatives. On each family-aware split the development partition fits the scorer, selects the probe epoch, and supplies the largest documented-negative score at that epoch, \tau_0. A test score is flagged only if it strictly exceeds \tau_0. Zero calibration exceedances follow by construction. The table gives the observed test FPR and TPR for All Ref+Det. The denominators are test appearances over repeated splits, so they can include the same candidate–dataset pair several times.
Appendix B.1 targets 80% of documented negatives for development and a 1:1 class ratio where feasible. With roughly 73–100 calibration negatives, Chronos, TiRex, and TimesFM2.0 have pooled FPRs of 3.6–5.2%. Each has only 15 positive test appearances, giving a 6.7-point change in pooled TPR per detection. With roughly 16–17 calibration negatives, Kairos and Moirai2 reach FPRs of 26.7% and 39.0%. For Moirai1, 5.0% represents one false positive in 20 negative appearances. Thus the development rule supplies an operating point, while its test behavior must be measured for the candidate in question. We will explain this in Section 4.3 alongside the FP-1 sensitivity results (Figures 4–5) and the threshold-free AUROC (Figure 3).