acceptodds
Under review as a conference paper at ICLR 2027

DaTS-Mix: Selecting Pretraining Mixtures for Time-Series Foundation Models Using Domain and Temporal Statistics

Abstract

Domain-aware pretraining-mixture selection has been widely studied for large language models, but its use in time-series foundation model (TSFM) pretraining is less well established. We study this problem and emphasize temporal statistics as a complement to domain information when selecting source proportions. We propose DATS-MIX, which represents each mixture using source proportions, domain shares, and the first and second raw moments of 14 temporal statistics computed under the training sampler. We condition a TabPFN response predictor on outcomes from 128 runs of a 7.07M-parameter forecasting proxy and use it to select a mixture for larger-model pretraining. On later evaluation windows from the same 28 sources in seven domains, the selected mixture reduces geometric-mean forecasting loss by 10.85% relative to size-proportional sampling and by 5.57% relative to a matched selector without temporal statistics, under the same 20k-update budget. The mixture selected with both moment blocks achieves the lowest mean native loss and weighted quantile loss among the evaluated methods; a Mean-only ablation separates the incremental role of the second-moment block. These results support proxy-guided TSFM data-mixture design and highlight the value of temporal statistics beyond source and domain proportions.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Foundation Models, Pretraining Data Mixture Optimization, Proxy-Based Performance Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nG2SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DATS-Mix addresses the challenge of selecting optimal pretraining data proportions for time-series foundation models. It combines dataset proportions, domain information, and temporal statistics, using small forecasting models and TabPFN to predict and select the best data mixture. Experiments show that it reduces forecasting loss by 10.85% compared with size-proportional sampling and 5.57% compared with selection without temporal statistics.

Critical strengths
  1. Beyond the existing dataset mixture strategy such as RegMIx that generally learns from domain proportions, this paper further proposed temporal-aware mixture representation that leverage statistics of time-series dynamic. This is the major contribution of this paper.
  2. In addition, proxy-guided mixture optimization is introduced for time-series foundaion models. The experiment demonstrates that outcomes from small-scale experiments can guide mixture slection.
  3. The experiments showed that DATS-mix achieve 10.85% low forecasting loss than naive size-proportional sampling. Extensive ablation studies have conducted to assess the effectiveness of each component.
Critical weaknesses
  1. The major drawback of the proposed method is that it does not establish general out-of-domain transfer. The experiment only proved the method effectiveness on datasets related to its pretraining sources but failed to demonstrate that the mixture strategy can generalize to entirely unseen domains.
  2. Another minor weakness is that the method requires 128 proxy-model training experiments which can still be relatively expensive.
Questions for authors

Authors can address my concerns in the critical weakness section above.

1 comment

Dear Reviewer nG2S,

We apologize for the delay in our reply. We thank you for the time and effort devoted to reviewing our work and for recognizing (S1) the temporal-aware mixture representation, (S2) proxy-guided mixture optimization for TSFMs, and (S3) the 10.85% gain over size-proportional sampling together with the ablations.

The replies below cover (W1) transfer to entirely unseen domains and (W2) the cost of 128 proxy runs. Revisions are marked in blue in the updated PDF, and each reply concludes with their location.

W1. Transfer to unseen domains

Reviewer comment: "...does not establish general out-of-domain transfer... failed to demonstrate that the mixture strategy can generalize to entirely unseen domains."

The external evaluation now covers the full GIFT-Eval benchmark instead of the short-horizon subset, and DaTS-Mix has the lowest mean WQL there.

Thank you for raising this point. GIFT-Eval covers short to long horizons (371,330 windows per model), and the original subset is retained in Appendix C.5. For TSFMs pretrained on each mixture, the table reports the validation loss (a domain-balanced mean log loss on held-out future windows of the training sources) and the domain-balanced GIFT-Eval weighted quantile loss (WQL), averaged over three training seeds.

Pretraining mixtureValidation lossGIFT-Eval WQL
DaTS-Mix0.71270.2776
DaTS-Mix w/o second moments0.72380.2795
DaTS-Mix w/o temporal statistics0.77000.2827
RegMix0.73020.2852
Size-proportional0.82750.2852

DaTS-Mix remains the lowest after excluding the 11 configurations known to overlap our training sources (2.9% lower than size-proportional sampling, p=0.04p=0.04, paired tt-test).

Revision: Sections 4 and 5.2, Appendix C.5 (Tables 1 and 2).

W2. Cost of 128 proxy runs

Reviewer comment: "...requires 128 proxy-model training experiments which can still be relatively expensive."

The 128 small proxies are a one-time cost comparable to prior work, and with temporal statistics 64 proxies already give about the prediction accuracy that 128 give without them.

Thank you for this observation. The proxy budget is compared with prior work below:

MethodProxy runsEach proxy
DaTS-Mix1287.07M, 2k updates
RegMix5121M, 1B tokens
CLIMB112350M (also 62M)
Data Mixing Laws40 mixtures70M–410M, 30B tokens

The 128 proxies cost about 0.76 full-size runs (parameters × training windows) once, regardless of the target model size. Toto 2.0 likewise reused one search for models from 4M to 2.5B parameters. In return, DaTS-Mix reaches the final loss of size-proportional sampling with 62% fewer updates (about 7.6k of 20k), so the saving grows with the target budget, whereas the proxy cost remains fixed.

As RegMix showed, the number of proxies matters. When our regressor is refitted on random subsets of the proxy runs, its prediction error on 48 held-out mixtures decreases with diminishing gains up to 128:

Proxy runs326496128
DaTS-Mix prediction RMSE0.03000.02570.02320.0218

With 64 runs, DaTS-Mix already predicts about as accurately as the selector without temporal statistics does with all 128 (0.0257 versus 0.0255), so temporal statistics also reduce the proxy budget needed for a given accuracy.

Revision: Section 5.6, Appendix D.6 (Table 25).

To improve the clarity of the paper, the revision also explains why second moments are added (Sections 1 and 3.1), contrasts DaTS-Mix with recent TSFM data-selection work (Sections 1 and 2.2), and summarizes the robustness analyses in Section 5.6.

We thank you again for your encouraging assessment and hope that these additions further strengthen your confidence in the work.

Reviewer_VjGWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DATS-Mix, a method for selecting pretraining mixtures for time-series foundation models. Building on proxy-guided mixture optimization, it augments source proportions and domain shares with the first and second moments of 14 temporal statistics. A TabPFN surrogate, trained on outcomes from 128 small-model experiments, selects a mixture for larger-model pretraining. On 28 sources across seven domains, DATS-Mix improves geometric-mean forecasting loss by 10.85% over size-proportional sampling and 5.57% over a matched predictor without temporal features. Improvements over the Mean-only ablation are smaller and statistically inconclusive.

Critical strengths
  • Relevant problem: Optimizing TSFM pretraining mixtures is an important problem and systematic isolated analyses are rare.

  • Promising results: The method delivers meaningful improvements over size-proportional sampling, with favorable results on a complementary GIFT-Eval subset.

  • Careful experimental reporting: Matched ablations, paired-seed evaluations, detailed methodological specifications, and transparent discussion of limitations are appreciated.

Critical weaknesses

1. Limited methodological novelty and insufficient isolation of the proposed inductive bias

The approach closely follows proxy-based mixture optimization (Liu et al., 2025), while dataset representations and temporal attributes have previously been explored for data selection (Zhang et al., 2025; Saurav et al., 2026).

Moreover, several existing TSFMs explicitly design and evaluate their pretraining mixtures (Ansari et al., 2024; Woo et al., 2024; Das et al., 2024). Most notably, Toto 2.0 (Khwaja et al., 2026) already optimizes pretraining-source proportions using a small forecasting proxy and transfers the selected mixture to substantially larger TSFMs. This closely anticipates the broader proxy-guided selection contribution, yet is not discussed.

The main remaining novelty is therefore the temporal-statistics representation. However, these features are deterministic linear functions of the source proportions already supplied to TabPFN. They introduce no additional information about candidate mixtures for a fixed source pool, but instead modify the predictor's inductive bias. While potentially useful, the paper does not establish that the gains arise from meaningful temporal information rather than generic feature augmentation. Dimension-matched random-feature or permuted-statistic controls are needed to substantiate this central contribution.

2. The incremental value of second moments is not convincingly demonstrated

Relative to Mean-only, DATS-Mix achieves just 1.10% lower native forecasting loss (p=0.277, favorable in 2/3 seeds) and a similarly inconclusive WQL improvement (p=0.220). Surrogate prediction accuracy is nearly identical, and all representations achieve the same finite-set selection regret. Furthermore, the three full-model seeds evaluate fixed mixtures rather than independently repeated selection procedures. Thus, the results suggest that temporal feature augmentation may be useful, but do not establish a reliable incremental benefit from second moments, which constitute a central part of the proposed method.

3. Limited generalization evidence and missing experimental controls

Given the modest methodological novelty, stronger experimental validation is needed to establish the contribution's significance.

The baseline comparisons are incomplete: simple uniform-source and uniform-domain sampling, stronger proxy-based mixture optimizers (e.g., Toto-style TPE), and compute-matched comparisons are missing. The RegMix comparison also changes multiple components beyond the feature representation.

Furthermore, the main evaluation uses later windows from the training sources, the GIFT-Eval evaluation covers only a development-inspected short-horizon subset, and experiments are restricted to one source pool and architecture family. These limitations leave the generality of the findings uncertain.

References

Ansari, A. F., et al. (2024). Chronos: Learning the Language of Time Series. Transactions on Machine Learning Research (TMLR).

Das, A., Kong, W., Sen, R., and Zhou, Y. (2024). A Decoder-Only Foundation Model for Time-Series Forecasting. ICML.

Khwaja, E., Lettieri, C., Woo, G., et al. (2026). Toto 2.0: Time Series Forecasting Enters the Scaling Era. arXiv:2605.20119.

Liu, Q., Zheng, X., Muennighoff, N., et al. (2025). RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR.

Saurav, K., Baghel, V., Jain, A., and Guruprasad, R. B. (2026). Time Series Attributes Guided Pretraining Data Selection for Time Series Foundation Models. ICASSP.

Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. (2024). Unified Training of Universal Time Series Forecasting Transformers. ICML.

Zhang, M., Tissue, H., Wang, L., and Qiu, X. (2025). Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture Without Training. ICML.

Questions for authors
  1. Can the authors compare temporal features against dimension-matched random projections, permuted source statistics, or other non-temporal transformations of source proportions to isolate the benefit of temporal information?
  2. What evidence establishes the incremental contribution of second moments, given the inconclusive Mean-only comparison and nearly identical surrogate prediction results?
  3. How sensitive are the selected mixtures and their performance to independently sampled proxy observations, training seeds, and predictor configurations?
  4. How does DATS-Mix compare against stronger mixture-selection baselines, particularly proxy-based TPE optimization and simple uniform or capped sampling strategies?
  5. Can the authors demonstrate improvements on genuinely unseen datasets or across different TSFM architectures?

I am willing to reassess if authors provide convincing controlled evidence that temporal features outperform generic feature augmentation and established mixture-selection baselines, with improvements that persist across independent selection runs; the claimed contribution of second moments would also need stronger support or substantially narrower framing.

4 comments

Dear Reviewer VjGW,

We thank you for the time and effort devoted to reviewing our work and for recognizing (S1) the relevance of TSFM mixture optimization, (S2) the gains over size-proportional sampling, and (S3) the careful reporting. We apologize for the delay in our reply.

Following your criteria, the replies below cover (W1/Q1) novelty and controls, (W2/Q2) second moments, (Q3) sensitivity of the selection, and (W3/Q4–Q5) baselines and evaluation scope. Validation loss denotes the confirmation loss J7J_7 of the paper, measured on a panel disjoint from the one used for selection, and "w/o temporal statistics" and "w/o second moments" are the Ratio+domain and Mean-only ablations. Revisions are marked in blue in the updated PDF. A second comment continues the response.

W1 / Q1. Novelty and generic-augmentation controls

Reviewer comment: "...Toto 2.0... is not discussed... these features are deterministic linear functions of the source proportions already supplied to TabPFN... Dimension-matched random-feature or permuted-statistic controls are needed..."

Temporal statistics are what make regression-based search effective for TSFM mixtures. Without them the same search is no better than uniform sampling (0.7700 versus 0.7651), with them it yields the best of nine mixtures, and matched controls do not reproduce the gain.

Thank you for the references and for proposing these controls. The works you cited, including the mixture rules of Chronos, Moirai and TimesFM, are now discussed in the revision. The closest are compared below:

CharacteristicRegMixSaurav et al.Toto 2.0DaTS-Mix (ours)
Target modelLLMTSFMTSFMTSFM
ObjectiveMixture optimizationDataset selectionMixture optimizationMixture optimization
SearchRegression-basedkNN-basedSequential (TPE)Regression-based
Uses domain information○×× (5 author-defined groups)○
Temporal statistics×○×○
Target-agnostic selection*○×○○

Domain information refers to the application-domain labels assigned by dataset designers. *No target-specific feature matching, so a mixture can be reused when evaluation targets change.

Toto 2.0 itself describes TSFM data curation as ad hoc and its own mixture as found empirically rather than by principled selection, the gap we address.

We agree that our temporal features are linear functions of the source proportions, as are the domain shares of RegMix. Their benefit lies in which directions are exposed to the predictor: the random and permuted controls below are equally linear in the proportions, yet do not reproduce the gain. As you proposed, we replaced the 28 temporal features with dimension-matched controls (100 draws each) and measured the RMSE of predicted proxy loss on 48 held-out mixtures and in cross-validation (CV). Win rate is the share of draws in which ours has the lower RMSE.

FeaturesHeld-out RMSEWin rateCV RMSEWin rate
Temporal statistics0.02183—0.03123—
Noise0.02475100%0.03521100%
Random source statistics0.02899100%0.0324587%
Permuted source statistics0.02863100%0.0338698%

Revision: Sections 1, 2 and 5.6, Appendices D.1 and D.3 (Table 21).

W2 / Q2. Incremental benefit of second moments

Reviewer comment: "...the three full-model seeds evaluate fixed mixtures rather than independently repeated selection procedures... stronger support or substantially narrower framing."

Second moments lower both losses in all four independent selections. We frame them as a refinement of the temporal descriptor, with temporal statistics as the main contribution.

Thank you for stating this concern so precisely. The original comparison rested on a single selection, so we repeated the entire selection independently (details in Q3) and trained the re-selected mixtures at full size:

DaTS-Mix vs. w/o second momentsValidation lossGIFT-Eval WQL
Original selection (mean of 3 seeds)−1.10%−0.69%
Re-selection A−2.59%−2.61%
Re-selection B−0.46%−0.82%
Re-selection C−1.19%−2.72%

Each re-selection uses new proxy runs and candidates, so the four selections are independent, and all four agree in direction on both metrics. Because each one requires pretraining a TSFM from scratch, the number of selections is small, so we report the direction and size of the effect (0.5–2.6%) rather than a significance test. Temporal statistics are the main contribution (3.1–5.6% lower loss in all four selections, Q3), and second moments refine it at no extra proxy cost. We will align the paper's wording with this framing in the final version. The full descriptor remains the only selector significantly better than both the selector w/o temporal statistics and RegMix (p=0.028p=0.028 and 0.0210.021).

Revision: Abstract, Sections 1, 5.1–5.3, 5.6 and 6 (Tables 2, 11, 23 and 24).

Q3. Sensitivity of the selection

Reviewer comment: "How sensitive are the selected mixtures... to independently sampled proxy observations, training seeds, and predictor configurations?"

The three selectors keep the same full-size order in the original selection and in all three independent re-selections, and DaTS-Mix predicts most accurately under every TabPFN setting.

Thank you for this question. As you requested, we repeated the entire selection procedure ten times, each with new proxy runs and one million new candidates shared by the three selectors. Temporal statistics changed the selected mixture in eight of the ten runs, and DaTS-Mix had the lower proxy loss in six of these eight. Following a pre-specified rule that does not depend on performance, the three runs in which the three selectors choose different mixtures were trained at full size with one seed each.

DaTS-Mix compared withw/o temporal statisticsw/o second moments
Proxy validation loss, mean of 10 runs−1.65%−0.51%
Full-size validation loss, the three runs−4.8 / −3.1 / −4.7%−2.6 / −0.5 / −1.2%

By Tables 1 and 24, the full-size loss is always lowest for DaTS-Mix and highest w/o temporal statistics. If the selectors were equivalent, three independent re-selections would all reproduce this order with probability (1/6)3≈0.005(1/6)^3\approx0.005.

Under other TabPFN settings:

Held-out RMSE by TabPFN ensemble sizew/o temporal statisticsw/o second momentsDaTS-Mix
40.024770.021740.02149
8 (default)0.025490.022250.02183
160.025370.022710.02199

The order also holds without TabPFN's feature preprocessing, and the selection of DaTS-Mix is unchanged across ensemble sizes.

Revision: Section 5.6, Appendices D.4 and D.5 (Tables 23 and 24).

W3 / Q4. Baselines and compute

Reviewer comment: "...simple uniform-source and uniform-domain sampling, stronger proxy-based mixture optimizers (e.g., Toto-style TPE), and compute-matched comparisons are missing. The RegMix comparison also changes multiple components beyond the feature representation..."

DaTS-Mix has the lowest full-size validation loss of nine mixtures and is lower than every requested baseline in all three seeds.

Thank you for specifying these baselines. As you suggested, we added four mixtures, each trained at full size with three seeds. TPE, the search used by Toto 2.0, runs over the 28 source proportions.

MixtureValidation lossDaTS-Mix vs.
DaTS-Mix0.7127—
DaTS-Mix w/o second moments0.7238−1.10%
RegMix0.7302−1.73%
Uniform domain (added)0.7387−2.56%
TPE (added)0.7406−2.75%
Size-proportional, capped (added)0.7423−2.91%
Uniform source (added)0.7651−5.11%
DaTS-Mix w/o temporal statistics0.7700−5.57%
Size-proportional0.8275−10.85%

RegMix is run with its published components (LightGBM regressor, top-100 averaging) and size-based allocation within domains, whereas our two ablations change only the features. TPE and RegMix also use 128 proxy runs, so these comparisons are compute-matched, and the proxies are a one-time cost of about 0.76 full-size runs that does not grow with the target model.

Revision: Sections 4 and 5.6, Appendices D.1 and D.6 (Table 11).

W3 / Q5. Unseen datasets and evaluation scope

Reviewer comment: "...the GIFT-Eval evaluation covers only a development-inspected short-horizon subset, and experiments are restricted to one source pool and architecture family..."

The evaluation now covers the full GIFT-Eval benchmark instead of the short-horizon subset, and DaTS-Mix has the lowest mean WQL there.

Thank you for raising this point. The revision uses GIFT-Eval itself, with short to long horizons, as the main external evaluation and retains the original short-horizon subset in Appendix C.5. The mean WQL of DaTS-Mix remains the lowest after excluding the 11 configurations known to overlap our training sources (2.9% lower than size-proportional sampling, p=0.04p=0.04), so the result holds on datasets outside our pool.

The procedure needs only training-data statistics and a proxy of the target architecture, so it carries over to other architectures and source pools. Like Toto 2.0, which tunes its mixture on a 10M-parameter proxy of its own architecture, we keep proxy and target in one architecture family so that the transfer remains controlled.

Revision: Sections 2.2, 4 and 5.2, Appendix C.5 (Table 2).

In summary, on each of your criteria: temporal statistics outperform generic feature augmentation, DaTS-Mix is lower than every requested baseline in all three seeds, the three selectors keep the same full-size order in all four independent selections, and second moments, as a refinement, improve every full-size comparison.

We thank you again for the precise criteria. We hope that these additions address your concerns, and we would be glad to clarify anything further.

I thank the authors for their substantial additional experiments and detailed rebuttal. The new feature controls, independent re-selections, stronger baselines, and expanded GIFT-Eval evaluation meaningfully address several of my original concerns.

In particular, the random/permuted feature controls provide evidence that temporal statistics offer a useful inductive bias beyond generic feature augmentation. The additional baseline comparisons strengthen the empirical contribution, and I appreciate the narrower framing of second moments.

Some concerns remain. The feature controls assess surrogate prediction rather than downstream mixture-selection performance. The additional full-size re-selections are conditionally selected from ten runs and evaluated with only one seed each; their consistent ordering is encouraging, but the proposed (1/6)3(1/6)^3 significance argument is not sufficiently justified. Finally, the methodology remains incremental, with generalization beyond one source pool and architecture family untested.

Overall, the rebuttal strengthens the empirical contribution. I therefore increase my rating to 2 (weak reject). My remaining reservations primarily concern methodological novelty and the breadth of evidence needed to establish ICLR-level significance.

Dear Reviewer VjGW,

Thank you for the careful follow-up and for reconsidering our work. We reply briefly to the remaining points.

As you noted, the (1/6)3(1/6)^3 argument in our earlier reply was too crude a measure. The original selection (mean of three seeds) and the re-selections (one seed each) differ in precision, so combining them is not ideal, but as a more standard analysis we apply the paired tt-test of the paper, taking each independent selection as one replicate.

Across the four independent selections, temporal statistics lower full-size loss by 4.6% on average (p=0.003p=0.003) and second moments by 1.3% (p=0.06p=0.06, not significant).

The table gives the change in full-size loss for each selection:

SelectionDaTS-Mix vs. w/o temporal statisticsDaTS-Mix vs. w/o second moments
Original selection (mean of 3 seeds)−5.57%−1.10%
Re-selection A (added, 1 seed)−4.83%−2.59%
Re-selection B (added, 1 seed)−3.15%−0.46%
Re-selection C (added, 1 seed)−4.67%−1.19%
Mean of the four selections−4.6%−1.3%
Paired tt-test over the four selectionsp=0.003p=0.003p=0.06p=0.06

The values are those of Tables 11 and 24. We count the original selection once because its three seeds retrain the same pair of mixtures and are not independent selections (if n=6n=6, p=0.04p=0.04 for second moments). As you point out, the full-size runs were chosen conditionally: we trained the three runs in which the three selectors recommend different mixtures, not the seven in which some of them choose the same mixture.

Mixture optimization for TSFMs has hardly been studied. We want to share that domain information can be used for it, and that temporal features tailored to time series make it more effective.

To our knowledge, no prior TSFM work optimizes mixtures over application domains. Combining domain shares with temporal statistics, DaTS-Mix gives the lowest loss of nine mixtures (10.85% lower than size-proportional sampling), whereas the same search without temporal statistics is no better than uniform sampling (0.7700 versus 0.7651).

Our setting of one corpus and one architecture family follows the one used to validate RegMix, CLIMB, Data Mixing Laws and Toto 2.0.

We are grateful for your positive consideration of our additions, and we thank you again for the constructive discussion.

Reviewer_PHSQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DaTS-Mix, a proxy-guided data mixture selection method for time-series foundation model (TSFM) pretraining. The authors incorporate 14 temporal statistics, including their first and second moments, into the mixture representation, and use TabPFN to predict forecasting performance based on 128 small-scale proxy experiments. The predicted best mixture is then used to pretrain a larger TSFM from scratch. Experiments show the potential of temporal statistics for optimizing TSFM pretraining data mixtures.

Critical strengths
  1. This paper explores an important but relatively underexplored problem in TSFM pretraining, providing empirical evidence that optimizing data mixtures can substantially improve forecasting performance.
  2. The incorporation of temporal statistics is intuitive and well-motivated, and the authors conduct relatively comprehensive experiments, including matched ablations, response prediction diagnostics, and domain-level analyses, to validate the effectiveness of the proposed method.
Critical weaknesses

The design of the 14 temporal statistics lacks sufficient justification and fine-grained ablation studies. Since temporal statistics are the key contribution distinguishing DaTS-Mix from RegMix, it would be valuable to investigate whether all these statistics are necessary and identify which ones contribute most to the performance improvements.

Questions for authors
  1. Cross-scale transferability. Optimal data mixtures may vary across model sizes and training budgets, especially as smaller sources experience different levels of repeated exposure. How well does the mixture selected by the small proxy transfer to the full-size model? Additional scaling experiments would strengthen this claim.
  2. Coverage of the candidate search space. Sampling (10^6) mixtures from Dirichlet(1) may provide limited coverage of the high-dimensional mixture space. Could more structured or adaptive search strategies lead to better mixture selection?
  3. RegMix versus Ratio+domain. In Table 1, RegMix using only (d) outperforms Ratio+domain using ([w,d]). Could the authors further explain this result, particularly considering their different predictors and selection procedures?
  4. Limited contribution of second raw moments. DaTS-Mix shows relatively small improvements over Mean-only in several experiments. Could the authors further analyze the limited additional benefits of second raw moments and identify when they are most useful?
6 comments

Dear Reviewer PHSQ,

We thank you for the time and effort devoted to reviewing our work and for recognizing (S1) the importance of TSFM mixture selection, (S2) the well-motivated temporal statistics, and (S3) the comprehensive experiments. We apologize for the delay in our reply.

The replies below cover (W1) the design of the 14 temporal statistics, (Q1) proxy-to-full-size transfer, (Q2) candidate coverage and adaptive search, (Q3) RegMix versus Ratio+domain, and (Q4) when second moments help. Revisions are marked in blue in the updated PDF, and each reply concludes with their location. A second comment continues the response.

W1. Design of the 14 temporal statistics

Reviewer comment: "...investigate whether all these statistics are necessary and identify which ones contribute most..."

The 14 statistics act jointly and improve the selected mixture significantly. The spectral group contributes most, and no single group accounts for the gain.

Thank you for this constructive comment. Following your suggestion, we removed each statistic group in turn and measured the RMSE of predicted proxy loss on 48 held-out mixtures (0.0218 with all groups):

Removed groupRMSE change
Spectrum (entropy, dominant period, peak and high-frequency ratios)+4.3%
Increments (difference-variance ratio, median absolute increment)+1.6%
Half-window (mean shift, log-variance ratio)+1.1%
Autocorrelation (lags 1–32)−1.6%
All four groups (Ratio+domain)+16.8%

Removing one group changes the error by at most 4.3%, against 16.8% for all four, so the gain comes from the groups jointly rather than from any single one. The groups follow standard descriptors of dynamics: catch22 (Lubba et al., 2019) includes spectral, autocorrelation and successive-difference features, spectral entropy measures forecastability (Goerg, 2013), and half-window comparisons reflect non-stationarity (Liu et al., 2022). The 14 statistics also outperform dimension-matched random and permuted statistics in every held-out draw, and the mixture selected with them has 5.57% lower validation loss at full size (118.74M parameters, 20k updates) than the one selected without them (p=0.028p=0.028). Validation loss denotes the confirmation loss J7J_7 of the paper, a domain-balanced mean log loss on held-out windows of the training sources, measured on a panel disjoint from the one used for selection.

Revision: Sections 2.2, 5.4 and 5.6, Appendix D.3 (Tables 21 and 22).

Q1. Cross-scale transfer

Reviewer comment: "How well does the mixture selected by the small proxy transfer to the full-size model?"

The mixture selected at proxy scale has the lowest full-size loss of the nine mixtures trained with three seeds, proxy and full-size rankings agree (Kendall τ\tau of 0.37 to 0.61), and full-size rankings are already stable after 2k updates (τ=0.83\tau=0.83).

Thank you for raising the question of cross-scale transfer. DaTS-Mix selects its mixture with 7.07M-parameter proxies trained for 2k updates, and at full size this mixture has the lowest loss of the nine mixtures trained with three seeds: the five of the main comparison and four added baselines. In the original selection and in each of three independent re-selections, its choice also has the lowest full-size loss among the three selectors. Beyond the selected mixture, we compared mixture rankings across model scales and across training lengths:

Rankings comparedMixturesKendall τ\taupp
Proxy vs. full: all170.370.042
Proxy vs. full: independent re-selections90.610.025
Full size, 2k vs. 20k updates: three-seed mixtures90.83<0.001

Independent re-selections repeat the entire selection procedure ten times with new proxy runs and candidates, and three of them were trained at full size for DaTS-Mix and its two ablations (nine mixtures). The last row uses these nine three-seed mixtures. Although full-size training draws ten times as many windows and therefore revisits small sources more often, its 2k and 20k rankings agree, so this additional exposure leaves the ranking largely unchanged.

Revision: Section 5.6, Appendix D.2 (Table 20).

Q2. Search coverage

Reviewer comment: "Sampling 10610^6 mixtures from Dirichlet(1) may provide limited coverage of the high-dimensional mixture space. Could more structured or adaptive search strategies lead to better mixture selection?"

With the same number of proxy runs, DaTS-Mix attains lower full-size loss than adaptive TPE search in all three seeds, and it keeps a lower average proxy loss than its ablations across ten newly drawn candidate sets.

Thank you for this question on the search space. As you suggested, we ran TPE, the adaptive search used by Toto 2.0, over the 28 source proportions instead of scoring a fixed candidate set, with the same budget of 128 proxy runs, and trained its best mixture at full size:

Search over 128 proxy runsFull-size validation lossDaTS-Mix vs.Seeds lower
TPE (adaptive)0.7406−2.75%3/3
DaTS-Mix (predictor-guided)0.7127——

TPE is not restricted to the candidate bank, so it can reach mixtures that the bank misses, yet its best mixture has the higher full-size loss. Across the ten re-selections of Q1, each with a newly drawn set of one million candidates, DaTS-Mix also lowered the average proxy loss relative to Ratio+domain and the selector without second moments (−1.65% and −0.51%).

Revision: Section 5.6, Appendices D.1 and D.5 (Tables 11 and 24).

Q3. RegMix versus Ratio+domain

Reviewer comment: "...RegMix using only (d) outperforms Ratio+domain using ([w,d])..."

RegMix uses two mechanisms that our Ratio+domain ablation (without temporal statistics) does not, and an added baseline shows how strong one of them is. DaTS-Mix outperforms RegMix without either.

Thank you for the close reading of Table 1. RegMix is a prior-work baseline run with its published components on the same 128 proxy runs, whereas Ratio+domain changes only the features of DaTS-Mix:

ComponentRegMixRatio+domainDaTS-Mix
(a) Regression modelLightGBM on domain sharesTabPFN on source and domain sharesTabPFN, plus temporal statistics
(b) Averaging the 100 best-predicted candidatesYesNoNo
(c) Within-domain ratiosFixed by data sizeSearchedSearched
Full-size validation loss0.73020.77000.7127

(a) is unlikely to be the cause, because LightGBM predicts less accurately than TabPFN on the same inputs (RMSE 0.0359 versus 0.0255). (b) lowered the proxy loss of the RegMix selection by 1.9%. (c) is a strong prior by itself: equal domain shares with size-based allocation within domains, a fixed rule that we added as a baseline, already reach 0.7387. Searching all 28 source proportions from 128 runs is the harder problem. Without temporal statistics this search does not pay off (0.7700), and with them it gives the lowest loss (0.7127). DaTS-Mix outperforms RegMix significantly, both at 20k updates (p=0.021p=0.021) and over the entire learning curve (area −3.38%, p=0.007p=0.007).

Revision: Sections 4, 5.1 and 5.6, Appendices B.5 and D.1.

Q4. Second moments

Reviewer comment: "...analyze the limited additional benefits of second raw moments and identify when they are most useful?"

Second moments lowered both full-size losses in all four independent selections, and they matter most when candidates differ in spread rather than in mean.

Thank you for this question. In the main comparison, DaTS-Mix, with both moments, is the only selector that is significantly better than both Ratio+domain (W1) and RegMix (Q3), whereas with means alone p=0.12p=0.12 and 0.380.38. We then repeated the entire selection with new proxy runs and candidates, and at full size second moments lowered both losses every time:

DaTS-Mix vs. w/o second momentsOriginal selectionRe-selection ARe-selection BRe-selection C
Full-size validation loss−1.1%−2.6%−0.5%−1.2%
GIFT-Eval weighted quantile loss−0.7%−2.6%−0.8%−2.7%

The original selection is averaged over three seeds.

With the mean, the second moment determines the variance ∑iwiVari+∑iwi(μi−μw)2\sum_i w_i\mathrm{Var}_i+\sum_i w_i(\mu_i-\mu_w)^2: within-source variation plus between-source dispersion. Equal-mean mixtures (e.g., moderately seasonal versus strongly seasonal plus aperiodic sources) can differ in this spread, which the first moments do not capture. Second moments are therefore most useful when the leading candidates have similar mean statistics. This follows established practice in describing distributions beyond their means: domain adaptation aligns second-order statistics (CORAL; Sun et al., 2016) and higher-order marginal moments (CMD; Zellinger et al., 2017), and GIFT-Eval characterizes time series by the variances of window-level means and variances (stability and lumpiness).

Revision: Sections 1, 3.1, 5.1 and 5.6, Appendix D.5 (Table 24).

We thank you again for your questions. We hope that these additions address your concerns, and we would be glad to clarify anything further.

Thank you for the detailed responses and additional experiments, which have addressed most of my concerns. I have two remaining questions:

  1. Autocorrelation features (W1). Removing the autocorrelation features (lags 1–32) actually reduces RMSE by 1.6%. Does this suggest that these features are redundant or potentially detrimental? Would removing them also improve the final mixture selection?

  2. Independent re-selections (Q4). The authors conducted ten independent re-selections, but only three were evaluated at full scale. How were these three selected? Could this introduce selection bias into the reported improvements from second moments?

Overall, the additional evidence has substantially addressed my concerns. I would appreciate clarification on these remaining points, and am inclined to raise my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.