acceptodds
Under review as a conference paper at ICLR 2027

Tstar: Temporal Structure-Aware Training Data Construction for Time-Series Models

Abstract

Time-series foundation models increasingly rely on large, heterogeneous pretraining datasets, yet how to construct these datasets has received far less systematic study than in natural language processing and computer vision. Dataset scale and source diversity alone do not determine how training exposure is allocated across temporal structures, so even a diverse corpus can leave some structures rarely seen. To address this, we introduce Tstar, a temporal structure-aware method that organizes training windows from heterogeneous datasets into an interpretable atlas of temporal structures. The atlas guides real-window selection and sampling, with optional targeted synthesis, to increase exposure to underrepresented patterns. We pretrain TinyTimeMixer and MOMENT-341M from random initialization under fixed budgets. For TinyTimeMixer, coverage-guided selection reduces the six-seed means of four TIME and FEV error aggregates by 23.41%–41.81% relative to matched uniform selection, with every paired seed improving. For MOMENT-341M, real-window rebalancing alone reduces MASE by 3.43% on TIME and 7.67% on FEV relative to uniform sampling, with both 95% bootstrap intervals excluding zero. A supporting study on six conventional architectures shows lower average error.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series foundation models, data-centric machine learning, pretraining data, dataset construction, time-series forecasting, synthetic data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gFQsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Tstar, a method for constructing training data for time-series models. It assigns windows to 18 predefined temporal archetypes using statistical fingerprints and rule-based labels, then uses this structural atlas to guide window selection and sampling, with optional targeted synthesis. Under matched training budgets, Tstar achieves lower forecasting errors than uniform-selection or uniform-sampling baselines for TinyTimeMixer and MOMENT-341M. A supporting study evaluates forecasting and imputation across six conventional architectures.

Critical strengths

Tstar improves forecasting performance over the uniform-selection and uniform-sampling baselines under matched training budgets. This suggests that allocating training exposure according to temporal structure can improve performance without increasing the number of training updates or the batch size.

Critical weaknesses
  • The presentation makes the method harder to follow than necessary. The abstract and introduction use boldface so frequently that it becomes visually distracting and dilutes the emphasis on the main points. The contribution statements could also be more concise and easier to scan.
  • The narrative references to Mix, Don't Pick read awkwardly. Referring to the authors, for example, "Nagpal et al. (2026)," could improve the flow of these sentences.
  • Readers must also reconcile several overlapping concepts across the main text and appendices. These issues make it difficult to reconstruct the intervention and assess which components explain the reported gains.
Questions for authors
  1. The abstract states, "The atlas guides real-window selection and sampling." Could "atlas" be defined more concretely at first use?
  2. In Section 3.2, what does "a capped random sample of windows" mean, and what cap is applied?
  3. How are the 18 named temporal patterns defined?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gFQs (Part 1 of 2: [W1, W2, W3, Q1])

Response to Reviewer gFQs (Part 1 of 2: [W1, W2, W3, Q1])

Thank you for showing us how to make the paper much clearer and easier to read. We will carefully fix every writing, formatting, and citation point you raised. During rebuttal, we also added a new 3-seed scratch comparison against BLAST (a strong feature-based balancing baseline), where our primary policy (Tstar-Broad) beats BLAST on all four TIME and FEV metrics (+5.29% to +8.97% paired mean task gain, winning 3/3 paired MASE seeds). Below we answer each point in plain English (Part 2 of 2 answers [Q2] and [Q3]).

[W1] Heavy Boldface and Concise Contributions

We will remove all inline boldface throughout the abstract and introduction (keeping headings only) and shorten our contributions into two scannable points:

  1. Temporal structure-aware data construction: We introduce Tstar, using an 18-category atlas to guide real-data selection, sampling, and optional synthesis.
  2. Empirical evidence on real-data composition: We show real-window selection and sampling can improve forecasting at fixed pools and budgets, including at matched size, quotas, and 18-category coverage in TinyTimeMixer (TTM).

[W2] Narrative Citations to "Mix, Don't Pick"

We will replace every awkward narrative title reference with standard author-year citations, "Nagpal et al. (2026)."

[W3] What Each Variant Means and Which Component Drives the Gains

The core idea is simple: raw time-series corpora are flooded with repetitive flat or noisy windows, so rebalancing real training windows with the 18-category atlas drives our main gains, while finer subclustering and synthesis are optional and exploratory.

To make the variants easy to separate on a first read, we will add a concise plain-English summary table showing what each variant does and what it achieves under fixed compute budgets:

Method VariantWhat It Does (Plain English)Key Empirical Result
Tstar-Broad (primary)Reweights real training windows across 18 temporal patterns and 8 observation states (no synthesis)MOMENT: +3.43% TIME and +7.67% FEV MASE skill vs Original; +5.29% TIME and +8.35% FEV task gain vs BLAST (3/3 MASE seeds)
TTM real selectionSelects a balanced real-window subset from 18 public sources under matched quotas (0% synthetic)TTM: 23.41% to 41.81% lower 6-seed mean forecasting error across four metrics (6/6 seeds)
Tstar-Refined (exploratory)Subdivides the 18 patterns into 35 finer clusters using unsupervised clusteringTrails the simpler Tstar-Broad policy across all four MOMENT metrics
Refined+Synthetic (exploratory)Adds targeted synthetic windows on top of Tstar-Refined to boost rare clustersBeats BLAST (3/3 MASE seeds), while 95% CIs vs Tstar-Refined cross zero on 3 of 4 metrics

In short, the simple real-window policy (Tstar-Broad on MOMENT and real-window selection on TTM) does the heavy lifting at zero extra pretraining updates, model parameters, or inference cost.

[Q1] Concrete Definition of "Atlas" at First Use

At first use, we will define "atlas" as a finite 18-category frequency table over fixed-length training windows that counts how often each temporal pattern occurs, so underrepresented patterns receive higher training exposure.

Standard sampling draws by raw volume and overexposes common stationary noise; the atlas shows which local dynamics are overrepresented and which rare dynamics need upweighting.

AuthorsOpenReview ↗

Response to Reviewer gFQs (Part 2 of 2: [Q2, Q3])

[Q2] Meaning of "A Capped Random Sample of Windows" and the Exact Cap

The cap applies only to our fast offline profiling step before training, limiting profiling to at most 16 series per file and 64 non-overlapping windows per series so huge files do not dominate the atlas counts; training itself uses the full data pool.

To prevent any confusion between offline profiling and actual pretraining, we will clearly separate them:

  • Offline Pile profile: 96,020 profiled windows sampled under the 16 series/file and 64 windows/series cap from internal Pile files only (external TIME and FEV benchmarks never enter profiling).
  • MOMENT-341M pretraining: samples from the full pool of 80,437,730 training crops across 2,087 Pile datasets.
  • TinyTimeMixer pretraining: trains on all 197,787 matched real windows from 18 public sources.

[Q3] Plain-English Definitions of All 18 Named Temporal Patterns (TAT-18)

Each window is tested top-to-bottom from Priority 1 to 18 and assigned to the first matching pattern, so rare structural patterns are captured before common stationary noise.

Priority & PatternPlain-English Definition
1. Sparse binaryMostly binary (0 or 1) or two-level discrete values
2. Bursty countNonnegative integer counts with sharp bursts or spikes
3. Count eventNonnegative integer counts without sharp bursts
4. Positive multiplicativeStrictly positive, non-unit-bounded series with strong multiplicative volatility and periodicity
5. Bounded oscillatorOscillatory signal confined within the unit interval [0, 1]
6. Heavy-tail noiseExtreme tail spikes or high outlier rate
7. Intermittent pulseMostly near-zero baseline with isolated pulses
8. HeteroskedasticStrong volatility change across halves without a mean level jump
9. Smooth trendStrong, steady upward or downward drift
10. Piecewise trendUpward or downward drift combined with a clear mean level shift
11. Regime switchClear mean level jump with high autocorrelation
12. Drifting periodPeriodic signal whose dominant frequency drifts over time
13. Strong single seasonOne sharp, dominant periodic cycle
14. Multiple seasonsMultiple distinct periodic cycles
15. Long-memory-likeSlowly decaying autocorrelation with persistent long-range dependence
16. Random walkHigh lag-1 persistence without a strong global linear trend
17. Chaotic-likeTwo fixed heuristic rules for irregular, low-periodicity dynamics (uncertified for physical chaos)
18. Stationary stochasticFallback category assigned when rules 1 to 17 do not trigger

For scaling, spectral, correlation, volatility, and shape descriptors are normalized or scale-invariant, while value-support checks (binary, integer, nonnegative, unit-interval [0, 1]) and the trend-slope guard inspect raw values so bounds, quantization, and exact zeros are preserved.

Thank you again for helping us make the paper much clearer and easier to follow. We specify the full method, TAT-18 rules, and exact configurations transparently and will release all code and manifests openly upon publication; if these clarifications and our new BLAST baseline results address your concerns, we would be sincerely grateful if you would raise score.

Reviewer_F3QHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Tstar, a framework for constructing time series pretraining data based on temporal structures. It extracts statistical features from temporal windows and uses fixed rules to organize them into an atlas of 18 temporal structure categories, with optional finer subdivisions. The atlas guides real window selection, sampling weight adjustment, and optional targeted data synthesis. Under a fixed training budget, the authors pretrain MOMENT-341M and TinyTimeMixer from scratch and evaluate the approach using six additional standard architectures. The reported improvements across multiple forecasting metrics demonstrate the potential value of carefully designing the composition of pretraining data.

Critical strengths
  • The research problem is practical. The paper addresses imbalanced structural coverage and limited compute allocation in heterogeneous time-series pretraining. Improving performance by curating training samples rather than altering model architectures has clear practical value.
  • The main experiments show encouraging empirical results. Both MOMENT and TTM are evaluated under fixed budgets with paired comparisons. Broad real-data resampling improves four forecasting metrics on MOMENT, and coverage-guided selection consistently improves four aggregate metrics across six random seeds on TTM, supported by initial ablations and sensitivity checks.
Critical weaknesses
  • The paper does not explain how its categories, thresholds, and rule priorities were chosen. A window may satisfy multiple criteria but is assigned to the first matching category, while some thresholds depend on raw data scales. Without sensitivity analysis, the stability and validity of the taxonomy remain unclear.
  • The standalone benefit of the temporal structure taxonomy is not isolated. MOMENT's sampling combines the 18 structure classes with 8 observation states, while TTM's selection mixes representativeness, density, and diversity. Without separate ablations for these factors, the results only show that the whole pipeline works, not how much the structure taxonomy itself helps.
  • More complex extensions show no clear advantage over Broad. Refined increases the number of categories from 18 to 35 but performs worse than Broad on all four main metrics. Since it also changes the sampling weights, the cause of this decline remains unclear. Targeted synthesis improves some results over Refined but does not establish a consistent advantage over Broad.
  • The data processing pipeline has potential leakage, and result reporting is incomplete. For MOMENT, the structural profile and feature scalers are fitted before the train/validation/test split. Also, the paper mainly reports aggregate metrics across tasks (especially for TTM), making it unclear whether the gains hold broadly across individual datasets.
Questions for authors
  • How are the 18 classes, thresholds, and first-match priorities in TAT-18 chosen? Why assign a single label to windows that match multiple structures, and how sensitive are the taxonomy and downstream results to rule order and data scaling?
  • Can the authors isolate the contribution of the temporal structure taxonomy from other sampling factors?
  • How does the 35-class split perform if it keeps the exact same weighting scheme as Broad? Also, do the final synthesized windows actually pass the target structure checks after generation?
  • Could the current experimental setup lead to data leakage? Finally, can the authors provide per-task results for MOMENT and TTM to show how gains are distributed across datasets?
5 comments
AuthorsOpenReview ↗

Part 1 of 2 covers [W4, Q4, W2, Q2];

We thank Reviewer F3QH for noting our "clear practical value" and "encouraging empirical results" on "imbalanced structural coverage and limited compute allocation."

[W4, Q4] Split Isolation, Hardware Symmetry, and Per-Task Coverage

External TIME and FEV benchmarks never enter our profiling, scaling, or internal split construction, and our gains hold after overlap exclusions and on matched GPUs.

You were right to check our split wording. Our 0.6, 0.1, and 0.3 split partitions internal Time-Series Pile files only (2,087 datasets; 96,020 profiled windows; 80,437,730 training crops), and both profiling and scalers fit strictly on internal Pile data. Our post-hoc FEV SHA-256 audit found 0 whole-file hits and 27,113 series-hash hits across 4 FEV tasks; excluding those 4 tasks lifts Tstar-Broad vs Original 3-seed geometric FEV MASE skill from +7.67% to +7.92% (96 tasks, [5.36, 11.00]), and excluding the 4 ETT tasks gives +7.96% (96 tasks, [5.37, 10.99]). TIME/TTM hash and near-duplicate audits remain open.

Original and Tstar match GPUs on seeds 20260617 (1 GPU) and 20260619 (DDP2), while Original ran DDP2 vs 1 GPU on 20260618 (20,000 group bootstrap 95% CIs; Table R1).

Table R1: MOMENT-341M vs Original (3-seed and GPU-matched 2-seed geometric skill %).

PolicyMetric3-Seed [95% CI]Tasks W/T/LGroups+Seeds 17 & 19 [95% CI]
Tstar-BroadTIME MASE+3.43 [2.34, 4.85]71/0/2732/39+4.28 [2.97, 5.96]
Tstar-BroadTIME CRPS+3.85 [2.47, 5.95]70/0/2832/39+5.22 [3.49, 7.88]
Tstar-BroadFEV MASE+7.67 [5.12, 10.66]81/0/1935/39+9.26 [5.98, 13.10]
Tstar-BroadFEV SQL+7.68 [4.98, 10.85]81/0/1934/39+9.25 [5.84, 13.28]
Refined+Synthetic (Ref+Syn)TIME MASE+4.38 [2.92, 6.10]73/0/2533/39+4.71 [3.10, 6.54]
Ref+SynFEV MASE+7.72 [4.96, 10.69]80/0/2034/39+8.21 [5.11, 11.49]

Matched seeds widen every gain. In a new 3-seed scratch comparison, Tstar-Broad also beats BLAST on MASE across all 3 matched-GPU seeds and short/medium horizons (Table R2). We use 2,000 paired seed/cluster bootstrap 95% CIs (shared step-0 files on 20260618, 20260619; on 20260617 Original/Tstar reconstructed init deterministically while BLAST lacks archived shared step-0 proof).

Table R2: Tstar-Broad vs BLAST (paired mean task MASE gain %, GPU symmetry, horizon tiers).

SliceGPUTIME MASE (W/T/L)FEV MASE (W/T/L)
3-seed [95% CI]Same+5.29 [2.84, 7.70] (71/0/27, 31/39)+8.35 [4.23, 12.66] (81/0/19, 31/39)
Seeds 17, 18, 191, 1, DDP2+6.31, +3.38, +5.51+7.39, +5.49, +11.10
Short, Medium, LongSame+8.86, +3.55, -0.41+14.18, +7.09, -3.41

BLAST leads long FEV horizons (> 96) and PeakTimingError (-4.53% Broad, -3.53% Ref+Syn), while Broad wins 3/3 MASE seeds and short/medium horizons vs BLAST. For per-task breakdowns, our 3 follow-up posts below list all 39 TIME and 39 FEV groups plus all 98 TIME and 100 FEV tasks for MOMENT; TTM archives retain benchmark-level 6-seed totals only.

[W2, Q2] Controlled Evidence Across Three Model Tiers

Real-window curation improves forecasting across all three model tiers at fixed budgets (Table R3), though we did not run sub-term ablations separating the 18 classes from the 8 observation states or TTM scoring terms.

Table R3: Controlled curation across three model tiers.

TierSettingMetricBaseline -> TstarConsistency
MOMENT (3 seeds)18x8 real (0% syn)TIME; FEV MASE1.4334->1.3778; 2.8483->2.324071/0/27; 81/0/19
TTM (6 seeds)Real select (197,787 win)TIME MASE; FEV test error2.5788->1.6770; 7.4687->5.72046/6 (-34.97%; -23.41%)
6 Models (5 seeds)Strict Tstar (0.21% syn)Forecast; Impute NRMSE1.3570->1.3466; 0.8119->0.809863.9% (-0.0029); 55.5% (-0.0016) over 490 settings

Even without sub-term ablations, TTM keeps total size (197,787), 18 source quotas, 108,391 missing values, and 18-class coverage identical with zero synthesis, isolating coverage-guided real-window selection on 6/6 seeds.

AuthorsOpenReview ↗

Part 2 of 2 covers [W1, Q1, W3, Q3]

TAT-18 checks rare patterns first and assigns one label per window so common stationary noise never hides rare dynamics, and both granularities beat uniform sampling (Table R1).

Single-label assignment keeps quotas mutually exclusive across three ordered stages: (1) discrete, bursty, bounded, heavy-tail, pulse, and heteroskedastic windows; (2) trend, regime-switch, and seasonal windows; (3) persistent, chaotic-like (two fixed heuristics, uncertified chaos), and stationary fallback windows. For scaling, spectral, correlation, volatility, and shape features are normalized or scale-invariant, while support ratios read raw bounds and trend rules check raw slope magnitude > 0.0005. Rule order and thresholds stayed fixed without sweeps.

Table R1: Granularity vs Original (3-seed and matched Seeds 17 & 19 geometric skill %).

MetricBroad (18x8, inv-sqrt): 3-Seed; 17&19Refined (35, exp): 3-Seed; 17&19 (3-Seed W/T/L, Grp+)
TIME MASE+3.43 [2.34, 4.85]; +4.28 [2.97, 5.96]+2.48 [1.38, 3.88]; +4.12 [2.97, 5.63] (67/0/31, 30/39)
TIME CRPS+3.85 [2.47, 5.95]; +5.22 [3.49, 7.88]+1.90 [0.96, 3.24]; +4.50 [3.09, 6.69] (61/0/37, 26/39)
FEV MASE+7.67 [5.12, 10.66]; +9.26 [5.98, 13.10]+6.36 [4.31, 8.62]; +8.87 [5.97, 12.18] (78/0/22, 32/39)
FEV SQL+7.68 [4.98, 10.85]; +9.25 [5.84, 13.28]+6.03 [3.86, 8.50]; +8.38 [5.40, 11.95] (77/0/23, 34/39)

[W3] Why Tstar-Broad Leads Tstar-Refined

We elevate the simpler Tstar-Broad as our primary policy, and our manifest audit shows Refined reweighted conservatively relative to its [0.25, 4.0] clip bounds (Table R2).

Your audit drove both updates, which we will credit in the revision. In the frozen manifest, Refined's exponential rule compressed dataset weights into [0.631, 2.219], leaving effective sample size high (95.5% dataset ESS; 93.0% window ESS) and raising cell entropy only modestly (2.0329 -> 2.3566 unweighted -> 2.4848 nats weighted). Consistent with this conservative reweighting, Broad leads Refined on all four point estimates (3-seed CIs > 0 on 3 of 4 metrics; Table R2).

Table R2: Direct geometric skill (%) across Broad, Refined, and Refined+Synthetic (Ref+Syn).

MetricBroad vs Refined: 3-Seed; 17&193-Seed W/T/LRef+Syn vs Refined: 3-Seed; 17&193-Seed W/T/L
TIME MASE+0.98 [0.13, 1.92]; +0.17 [-0.37, 0.78]56/0/42+1.95 [0.91, 3.07]; +0.62 [-0.58, 1.72]69/0/29
TIME CRPS+1.99 [0.82, 3.74]; +0.75 [0.10, 1.63]59/0/39+1.12 [-0.13, 2.48]; -0.80 [-2.28, 0.43]67/0/31
FEV MASE+1.40 [-0.21, 3.12]; +0.43 [-1.11, 1.98]65/0/35+1.45 [-1.20, 3.59]; -0.72 [-3.43, 1.37]63/0/37
FEV SQL+1.76 [0.30, 3.39]; +0.95 [-0.28, 2.31]65/0/35+2.02 [-0.29, 3.91]; +0.11 [-1.99, 1.79]62/0/38

We did not run a 35-cluster inverse-sqrt control, so granularity, 8-state conditioning, and weighting remain coupled. This audit diagnoses Refined's conservative weights, while Ref+Syn fixes that 35-cell base to isolate 80% synthesis.

[Q3] Targeted Synthesis Verification

Targeted synthesis passes all 14 pre-postprocessing integrity checks across 35 reachable cells and remains exploratory.

  • Candidate selection: picking the best of 4 candidates in 29-D scale-stable space lowers mean centroid distance (1.7686 -> 1.6813), improving 1,222 of 1,638 synthetic slots (74.6%) across all 5 generator families.
  • Batch and schedule hygiene: a 2,048-window batch passes with 0 fallbacks, 0 exact copies, 0 support violations, and 0 non-finite outputs (14/14 checks pass), and the 1,000,000-ID schedule matches targets closely (total variation 0.0000284).
  • Honest scope boundary: post-selection moment matching, jitter, and clipping skip a second TAT-18 rule pass, and Ref+Syn vs Refined CIs cross zero on 3 of 4 metrics, so we keep synthesis strictly exploratory.

Thank you again for your soundness review, which drove our split-isolation audit, manifest ESS check, and primary Tstar-Broad policy. We specify the full method, TAT-18 rules, and exact protocols transparently and will release all code and manifests openly upon publication; if these audits and controls address your concerns, we would be sincerely grateful if you would raise your soundness and overall score.

AuthorsOpenReview ↗

Response to Reviewer F3QH (Results 1 of 3): All 39 TIME and 39 FEV Dataset Groups (MOMENT-341M)

Tstar-Broad lowers forecasting error on 32 of 39 TIME groups and 35 of 39 FEV groups (MASE) across all three paired seeds. In response to [Q4], the tables below report MOMENT-341M Tstar-Broad versus Original uniform across all 39 TIME and 39 FEV dataset groups. We average paired log forecasting error ratios over the three training seeds and all tasks in each group, then convert to percentage geometric skill. Positive values indicate lower forecasting error with Broad, and negative values indicate higher forecasting error.

TIME

Dataset groupTasksMASE (%)CRPS (%)
Australia_Solar3+12.41+10.42
Auto_Production_SF1+14.37+14.88
azure2019_D1-0.59-0.34
azure2019_I1-2.47-1.59
azure2019_U1+5.25+5.15
Coastal_T_S12+1.60+1.95
Commodity_Import1+12.59+17.55
Commodity_Production1+6.72+3.88
CPHL9+1.24+1.77
Crypto1+12.56+10.20
current_velocity15+3.85+1.24
ECDC_COVID2+10.55+19.83
epf_electricity_price3+4.81+5.01
EWELD_Load3+3.20-0.22
Finland_Traffic3-1.02+8.69
Global_Influenza1+4.37+16.87
Global_Price1-0.30-4.72
Housing_Inventory1+11.64+12.82
Job_Claims1+14.19+4.92
JOLTS1+3.79+3.54
MetroPT-33+0.57+0.14
NE_China_Wind3+2.20+2.61
Oil_Price1+3.23+6.45
Online_Retail_2_UCI1+1.94+1.56
OpenElectricity_NEM3+0.38+1.66
Port_Activity2+0.04+1.36
SG_Carpark3+1.62+3.57
SG_PM253+0.46+0.41
SG_Weather3+2.55+3.74
Smart_Manufacturing3-0.07-0.02
Supply_Chain_Customer1+0.62+0.49
Supply_Chain_Location1+0.24+0.24
Uncertainty_1M1-6.40-7.82
US_Labor1+11.80+1.47
US_Term_Structure1+12.39+2.99
Vehicle_Sales1+13.52+23.27
Vehicle_Supply1+7.56+16.36
Water_Quality_Darwin3+4.01+10.12
WUI_Global1-2.26-4.83

FEV

Dataset groupTasksMASE (%)SQL (%)
australian1-0.85-1.09
bizitobs_l2c2+1.33+1.25
boomlet15+4.68+3.69
ecdc1+6.37+6.37
entsoe3+1.62+2.10
epf5+9.43+10.17
ercot4+8.47+9.28
ETT4+0.55+0.26
favorita_stores3+2.51+2.40
favorita_transactions3+6.36+6.70
fred4+24.08+24.50
gvar1+15.66+16.69
hermes1+7.80+7.16
hierarchical_sales2+1.33+1.27
hospital1+1.63+1.68
hospital_admissions2+0.07+0.10
jena1+5.87+5.27
jena_weather2+5.01+4.46
kdd1+2.84+3.60
kdd_cup_20222-0.23-0.40
LOOP_SEATTLE3+1.02+1.27
m53-0.50-0.58
M_DENSE2+7.22+6.94
proenfo3+11.98+13.24
redset3+5.51+3.74
restaurant1+0.04-0.07
rohlik_orders2+8.91+9.53
rohlik_sales2+2.56+2.53
rossmann2+1.21+2.08
solar2-0.61-0.77
solar_with_weather2+20.90+20.63
SZ_TAXI2+2.42+2.17
uci_air_quality2+2.86+3.14
uk_covid_nation4+20.81+21.13
uk_covid_utla2+19.95+20.11
us1+13.27+13.49
us_consumption2+26.17+26.61
walmart1+3.31+3.15
world3+20.69+21.22
Reviewer_DBtnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Tstar, a method for constructing time series training datasets based on the temporal patterns present across individual windows. They extract statistical features from time series windows and use certain predefined rules to classify them into 18 categories. These categories are then used for training window selection and sampling, and there is also an optional synthetic data generation component for underrepresented patterns. The authors evaluate Tstar by pretraining MOMENT-341M and TinyTimeMixer from scratch under fixed training budgets. They compare several sampling strategies on TIME and FEV benchmarks, and also conduct additional experiments on six other non-TSFM architectures for forecasting and imputation. Their results show that changing how training windows are selected or sampled can improve downstream performance at fixed training budget.

Critical strengths
  • I find the overall breadth of the experimental evaluation to be quite good. The authors pretrain MOMENT-341M and TinyTimeMixer from random initialization and keep the architectures and training budgets fixed across different comparisons, which helps isolate the impact of changing the training data. The improvements are quite substantial in some settings and across six paired training seeds. I think these results provide reasonable evidence that training data selection and sampling can have a meaningful impact on downstream forecasting performance.

  • I also like that the authors try to understand which parts of the data construction process actually help. For TinyTimeMixer, both the original and Tstar datasets already cover all 18 temporal categories, but Tstar still improves performance by increasing the representation of less common patterns. This is an interesting observation since it shows that simply covering every temporal pattern may not be sufficient. The ablations on MOMENT are also interesting, particularly how the Broad strategy performs better than Refined on several metrics, and how adding synthetic data provides relatively limited additional improvements. I find these findings useful because the paper does not simply assume that adding more categories or synthetic data will lead to better performance.

Critical weaknesses
  • My biggest concern is that the paper does not compare against stronger existing data sampling approaches, particularly BLAST. BLAST already investigates balancing time series pretraining data using statistical features, although it operates at the series level rather than the window level. The authors explain the difference in Section 2, but the experiments only compare Tstar against uniform sampling or selection. I believe this is an important missing comparison because the results establish that structure-aware sampling is better than uniform sampling, but do not show whether it is better than existing feature-based balancing approaches. Balancing training data using statistical characteristics is not a completely new idea, and I feel a stronger comparison is required to gauge the contribution of the proposed method.

  • I also have major concerns about the choice of models used for the main pretraining experiments. Both MOMENT and TinyTimeMixer were introduced in 2024, while several newer forecasting foundation models, such as Chronos-2, TimesFM 2.5, and Moirai 2.0, are now available. MOMENT is primarily a masked-reconstruction model, and was used in a linearly probed setting even in the original paper for forecasting, while the TinyTimeMixer variant used here has only around 805K parameters and is trained for 10K steps. Although I understand that the authors want to compare sampling strategies under fixed budgets, I feel the evaluation on these two older architectures is insufficient to establish how well Tstar would work for more recent and stronger forecasting models. This is particularly important because the largest improvements are reported on TinyTimeMixer, which just might have the largest headroom, and I am unsure whether such large improvements would hold for stronger models with newer architectures or with longer pretraining. Longer pretraining in particular can diminish the impact of data sampling. The supporting experiments on conventional architectures also show mixed results, since forecasting performance got worse for DLinear, TCN, and GRU.

Questions for authors
  • The authors explain how Tstar differs from BLAST, especially in using window-level temporal categories rather than whole-series statistical features. However, their experiments only compare against uniform sampling. Could the authors provide evidence that supports the advantage of their proposed method over existing feature-based balancing approaches? Is there any existing analysis that helps establish whether the gains come from the specific temporal categories rather than simply increasing the representation of less common statistical patterns?

  • The main pretraining experiments use MOMENT and TinyTimeMixer, both introduced in 2024. Could the authors clarify why these architectures were selected instead of more recent forecasting foundation models? Also, can the authors provide evidence either from existing experiments or by running additional experiments, that the improvements would remain similar with longer pretraining or stronger architectures?

  • TinyTimeMixer experiments report large improvements, but Appendix A.5 mentions that both datasets were constructed using a single selection seed. Could the authors clarify how sensitive the selection procedure is to the random seed, and provide results for more selection seeds or stability analyses? I feel this is important given the magnitude of the reported improvements.

  • Appendix A.1 mentions that the MOMENT profile was fitted before the loader's train/validation/test split and that complete benchmark-overlap and near-duplicate audits were not performed. Could the authors clarify what checks were performed to prevent overlap between the training data and the TIME and FEV evaluation sequences? A similar clarification for the TinyTimeMixer experiments would be useful.

  • The authors use a model they refer to as MOMENT-341M, whereas the original MOMENT paper reports approximately 385M parameters for MOMENT-Large. Appendix A.3 provides the parameter count and some training details, but does not explicitly explain this difference. Could the authors clarify whether they used the original MOMENT-Large architecture or a modified configuration? Also, since the authors mention that their pretraining recipe differs from the original MOMENT recipe, can they clarify the main differences and their motivation behind those?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DBtn (Part 1 of 2)

Part 1 of 2 ([W1, Q1, Q4])

Thank you for noting our "overall breadth of the experimental evaluation" across "six paired training seeds" and candid ablations. Your BLAST suggestion (Shao et al., 2025) became our most informative addition, revealing horizon complementarity we will credit in the revision. BLAST's whole-series grids average out local bursts and regime switches; Tstar classifies 512-point windows across 18 categories and 8 states at zero extra updates, parameters, or inference cost (Part 2 covers [W2, Q2, Q3, Q5]).

[W1] Controlled 3-Seed Comparison Against BLAST

Both Tstar policies beat BLAST across all four metrics in 3-seed evaluation, winning 3/3 MASE seeds (Tables R1, R2). We pretrained five MOMENT-341M policies for 50,000 updates on 98 TIME and 100 FEV tasks.

Table R1: Raw arithmetic task and seed means across MOMENT-341M policies.

PolicyTIME MASETIME CRPSFEV MASEFEV SQL
Original1.43340.32452.84832.7285
BLAST1.45360.33622.69972.5966
Tstar-Broad1.37780.31532.32402.2127
Tstar-Refined1.39310.31892.45052.3512
Refined+Synthetic1.36750.32322.32902.2056

Heavy-tailed FEV scales lower BLAST's raw mean while paired ratios favor Original (2,000 paired seed and dataset-cluster bootstrap 95% CIs; no Refined vs BLAST CI).

Table R2: Paired mean task gain (%) [95% CI] (W/T/L; groups, MASE seeds).

ComparisonTIME MASETIME CRPSFEV MASEFEV SQL
Broad vs BLAST+5.29 [2.84, 7.70] (71/0/27; 31/39, 3/3)+5.61 [1.66, 9.64] (67/0/31)+8.35 [4.23, 12.66] (81/0/19; 31/39, 3/3)+8.97 [4.86, 13.34] (82/0/18)
Refined+Syn vs BLAST+6.24 [4.24, 8.44] (78/0/20; 33/39, 3/3)+5.19 [1.95, 8.29] (76/0/22)+8.21 [4.42, 11.87] (80/0/20; 35/39, 3/3)+9.05 [5.38, 12.89] (79/0/21)
BLAST vs Original-3.08 [-6.57, -0.29] (39/0/59; 15/39)-3.53 [-7.85, -0.21] (40/0/58)-2.38 [-5.96, 0.47] (43/0/57; 15/39)-3.11 [-6.83, -0.27] (42/0/58)

Both Tstar policies beat BLAST on every metric with positive 95% CIs. Window balancing drives short/medium horizons; BLAST leads long FEV horizons and peak timing (Table R3).

Table R3: MASE paired mean task gain (%) over BLAST by horizon tier, plus FEV PeakTimingError.

Slice (FEV, TIME)FEV BroadFEV Ref+SynTIME BroadTIME Ref+Syn
Short (FEV <=24: 46; TIME: 50)+14.18% (44/0/2)+13.86% (43/0/3)+8.86% (42/0/8)+9.23% (43/0/7)
Medium (FEV 25-96: 35; TIME: 24)+7.09% (32/0/3)+8.63% (33/0/2)+3.55% (17/0/7)+6.09% (20/0/4)
Long (FEV >96: 19; TIME: 24)-3.41% (5/0/14)-6.21% (4/0/15)-0.41% (12/0/12)+0.18% (15/0/9)
FEV PeakTimingError-4.53%-3.53%n/an/a

Tstar and BLAST match GPUs on all 3 seeds (step-0 logs absent only on BLAST 20260617), where Tstar wins 3/3 MASE seeds. Original ran DDP2 vs 1 GPU on 20260618; on GPU-matched seeds (20260617, 20260619), Broad gains +4.28% TIME MASE and +9.26% FEV MASE skill over Original.

[Q1] Category Specificity vs. Generic Statistical Balancing

Rebalancing rare patterns lifts every TTM seed and frozen probe metric vs uniform (Table R4). In TTM, Raw and Tstar cover all 18 classes with real windows; rebalancing lifts effective archetypes (6.55 to 7.44) and cuts forecasting error across all six seeds. Frozen probes test 40 name-screened sets.

Table R4: Existing uniform-control results on TTM (6-seed) and MOMENT probes (40 sets).

MetricUniform (Raw/Orig)Tstar (Broad; Ref+Syn)Delta & Consistency
TTM TIME CRPS2.52751.6353-35.30% (6/6)
TTM TIME MASE2.57881.6770-34.97% (6/6)
TTM FEV mean test error7.46875.7204-23.41% (6/6)
TTM FEV WAPE1.51720.8828-41.81% (6/6)
Probe Acc (%)70.9671.75; 72.17+0.79 pp; +1.21 pp
Probe BalAcc (%)64.5765.52; 66.15+0.95 pp [0.14, 1.97] (26/2/12); +1.59 pp
Probe Macro-F1 (%)63.6364.44; 65.09+0.81 pp; +1.46 pp

We did not run a 512-window generic-statistics MOMENT control, so granularity and TAT-18 stay coupled on MOMENT; rebalancing lifts 6/6 TTM seeds, all 3 probe metrics, and 3/3 BLAST MASE seeds.

[Q4] Pre-Split Profiling and Benchmark Hygiene

Our filters, FEV SHA-256 audit, and overlap masks keep external benchmarks outside profiling and confirm FEV gains hold after overlap exclusions (Table R5).

Table R5: Hygiene, FEV SHA-256 audit, and ETT sensitivity.

StageVerified Counts
Pile guard2,434 -> 2,118 -> 2,101 files; 2,087 sets, 80,437,730 crops (0.6 of 0.6, 0.1, 0.3)
Pile profile96,020 internal Pile windows only (TIME/FEV untouched)
FEV hash/ETT0 whole-file SHA-256; 4 series hits cut -> +7.92% MASE (96 tasks); 4 ETT cut -> +7.96%
Probes/TTM26 of 66 probes cut (23 exact, 26 name/path -> 40 clean); 40,320 solar pts cut in TTM

TIME/TTM hash and near-duplicate audits stay open; FEV gains hold after overlap and ETT exclusions.

AuthorsOpenReview ↗

Response to Reviewer DBtn (Part 2 of 2)

[W2, Q2] Open-Science Testbed Rationale, Scale Ladder, and Supervised Architectures

We chose MOMENT-341M and TinyTimeMixer (TTM) because controlled multi-seed pretraining needs open code, reconstructible open data, and feasible compute (Table R1). We agree that Chronos-2, TimesFM 2.5, and Moirai 2.0 have stronger architectures and higher leaderboard accuracy. However, newer models pair public data with proprietary pipelines (TimesFM 2.5), non-public operational data beyond open LOTSA (Moirai 2.0), or massive synthesis (Chronos-2). Retraining them on public subsets alone leaves a wide gap to official checkpoints, which unfairly underrepresents those models and mixes missing data with sampling. Open corpora at the Time-300B scale also exceed academic GPU budgets over 15 MOMENT-scale scratch runs.

[Q3] TTM Selection Seed and Multi-Seed Stability

Fixed source quotas and a fixed coverage-guided scoring rule guide TTM selection; Tstar wins 6/6 TTM seeds vs Raw and 3/3 MOMENT MASE seeds vs BLAST. We built the 18-source TTM corpus using fixed source quotas and fixed scoring weights under one data-selection seed (20260726). We did not run extra selection seeds; on this 18-source corpus Tstar wins all six TTM training seeds, and on the Time-Series Pile Tstar wins all three MOMENT MASE seeds vs BLAST.

[Q5] MOMENT-341M Configuration

We use the official MOMENT research implementation with a randomly initialized, encoder-only Flan-T5-large backbone. Our instantiated configuration contains 341,248,520 trainable parameters, which we report explicitly as MOMENT-341M rather than the approximate 385M stated in the original paper.

Thank you again for prompting our 3-seed BLAST benchmark and horizon-complementarity analysis, which clearly strengthened the paper. We specify the full method, TAT-18 rules, and exact configurations transparently and will release all code and manifests openly upon publication. Because Tstar adds zero extra pretraining updates, model parameters, or inference cost, if these new results and clarifications address your concerns, we would be sincerely grateful if you would raise your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Reviewers and Area Chair

We sincerely thank Reviewers DBtn, gFQs, and F3QH for their constructive feedback, which strengthened our evaluation, audits, and presentation. Tstar rebalances window-level temporal structure at zero extra pretraining updates, parameters, or inference cost, lifting TinyTimeMixer across 6/6 seeds and MOMENT-341M across 98 TIME and 100 FEV tasks. In our new controlled 3-seed comparison, Tstar-Broad beats BLAST on 3/3 MASE seeds (TIME +5.29%, 71/98 tasks; FEV +8.35%, 81/100 tasks), while external TIME and FEV benchmarks never enter internal splitting, profiling, or scaling (post-hoc FEV hash/ETT exclusions raise Broad FEV MASE skill from +7.67% to +7.92% / +7.96%). We also elevate 18-class Tstar-Broad as primary, diagnose Refined's conservative weights (95.5% ESS), and clarify the atlas definition, profiling caps, and 18-class rules. All full tables—including the 3-seed BLAST breakdown, matched-GPU seeds, 14/14 synthesis checks, and all 39 TIME / 39 FEV groups and 198 tasks—are provided in our responses above for easy verification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.