acceptodds
Under review as a conference paper at ICLR 2027

Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach

Abstract

Heterogeneity in time series data is far more pronounced than in vision or language, manifesting at two coupled levels: inter-domain discrepancies in sensing modalities, sampling rates, and physical processes, and intra-domain variability such as latent sub-domains and temporal concept drift. We empirically show that this bi-level heterogeneity is the root cause of two failures of mixed-batch centralized pretraining of time series foundation models (TSFMs), namely representation collapse and gradient conflict, which existing federated learning (FL) methods only partially address since they assume each client is internally homogeneous. We propose FedTRL, a heterogeneity-aware FL pretraining method whose two components target the two levels respectively: a local objective that combines diffusion-based reconstruction with sub-domain adversarial regularization and prototype alignment to reduce intra-domain conflict without disturbing temporal modeling, and a domain-aware aggregation (DaG) on the server that re-weights clients by prototype discriminability and global semantic alignment to control inter-domain interference. Extensive experiments across point/probabilistic forecasting show FedTRL matches or surpasses centralized and federated TSFM baselines and degrades least under domain imbalance, drift, and unseen-domain transfer, turning bi-level heterogeneity from an obstacle into a controllable training signal.

Primary area:
learning on time series and dynamical systems
Keywords:
Federated learning, time series forecasting, time series pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_egXgOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

FedTRL aims to train a shared time series model across different data sources. It reduces differences within each source and uses feature-based weights when combining the models, so both intra and inter domain heterogeneity. The experimental setups cover point and probabilistic forecasting on several benchmarks.

Critical strengths

The experiments cover several benchmarks and forecasting setups and show improvements over the federated baselines, also tested on top of some existing backbones

Critical weaknesses

The method uses domain classification loss as a heuristic measure of source reliability. But a source may be easy to identify simply because of source specific patterns or shortcuts. Although the weighting improves the results, the link between this score and reliability can still benefit from better analysis and evidence.

Questions for authors

Clarification: Under what conditions can combining different data sources hurt prediction, and how well does FedTRL handle these cases?

Clarification: Are the main comparisons matched in terms of total data seen and training budget?

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.


W1 [Whether ℓk\ell_k reflects source reliability or source-specific shortcuts]

DaG does not rely on the server-side classification loss ℓk\ell_k of a client's prototype alone. Instead, it combines ℓk\ell_k with the prototype deviation δk\delta_k through sk=−αℓk−βδks_k=-\alpha\ell_k-\beta\delta_k. Imagine a client whose series share a periodic pattern absent from all other clients. This client is easy to identify (low ℓk\ell_k), but its prototype lies far from the global prototype (high δk\delta_k), allowing δk\delta_k to offset its low ℓk\ell_k. We have added two analyses showing that DaG weights align more closely with measured client utility than ℓk\ell_k alone or sample size, while suppressing the weight of such a shortcut-driven client. Specifically, we address the link to reliability from two aspects below:

(1) Weights versus Measured Utility. We measure each client's utility as the increase in TSLib zero-shot MSE when that client is excluded from 500 additional rounds of uniform aggregation, which start at round 9,500 of the large-scale model and keep the total number of updates fixed. Uniform aggregation keeps this measurement independent of the weighting rule, and Tables R1 and R1b below report the results. The three clients with the largest DaG weights are also the three with the highest utility, and the only client whose removal lowers the error (C8) receives the second-smallest weight.

Table R1. Spearman correlation between aggregation weight and client utility.

WeightingSpearman ρ\rho
DaG (ℓk\ell_k and δk\delta_k)0.78
ℓk\ell_k only (β=0\beta=0)0.65
δk\delta_k only (α=0\alpha=0)0.47
Sample size (FedAvg)0.18

Table R1b. DaG weight and utility of each client.

ClientC1C2C3C4C5C6C7C8C9
DaG weight0.1840.1530.1370.1060.1190.0830.0910.0660.061
Utility (×10⁻³ MSE)4.65.26.33.42.12.70.9−0.31.4

(2) Shortcut Client. We add the client of the example above as a tenth client that holds 10% of all samples. It is the easiest to identify ($\ell_k=0.05vs.amedianof0.29)buthasalargeprototypedeviation(0.05 vs. a median of 0.29) but has a large prototype deviation (\delta_k=$2.28 vs. 0.24). Table R2 below compares aggregation rules differing only in their client weights, with scores and weights averaged over the last 1,000 rounds. The ℓk\ell_k-only rule assigns this client more weight than its sample share and incurs the largest RW-Bench MSE increase. In contrast, DaG assigns it less than half its sample share and reduces the increase to +0.004.

Table R2. Results with the added shortcut client. RW-Bench change is relative to the same rule without this client.

AggregationWeight of added clientTSLibRW-BenchRW-Bench change
Sample size (FedAvg rule)0.1000.3470.675+0.007
ℓk\ell_k only0.1290.3390.645+0.014
DaG (FedTRL)0.0460.3250.603+0.004

We have clarified the role of ℓk\ell_k in Sec. 3.2 and App. B.2 of the revised manuscript.


Q1 [Conditions under which combining sources hurts, and how well FedTRL handles them]

Combining sources can hurt prediction when their temporal dynamics conflict, leading to negative transfer under joint training. For example, exchange rates evolve almost as a random walk, whereas electricity load follows daily cycles, so fitting both with a single model can degrade prediction accuracy. We have added a comparison against training on each dataset alone (Table R3 below): FedAvg exhibits negative transfer on five of eight datasets, whereas FedTRL improves on all eight. Each local-only model is trained in the same way as FedAvg.

Table R3. In-domain MSE of local-only and joint training.

MethodETTh1ETTh2ETTm1ETTm2Elec.TrafficWeatherExch.
Local only0.4710.4140.4040.2960.1930.4390.2590.383
All Mixed0.4580.3990.3720.2860.1890.4280.2540.390
FedAvg0.4760.4100.3990.2990.1980.4340.2640.403
FFTS0.4630.3950.3800.2900.1870.4200.2520.390
FedTRL0.4480.3820.3750.2780.1810.4260.2410.375

Consistent with the example above, both All Mixed and FFTS underperform local-only training on Exchange. Such conflicting dynamics manifest as opposing gradients across domains. Tab. 24 shows that gradient conflict increases with the number of mixed domains under centralized training, but grows much more slowly under FedTRL. FedTRL also achieves the lowest error at every tested level of domain imbalance (Tab. 25) and when a related domain is withheld from pretraining (Tab. 22).


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


Q2 [Matched total data seen and training budget in the main comparisons]

Yes. The comparisons behind our claim that federated pretraining matches or outperforms centralized pretraining are matched in data, backbone, and number of updates. In the in-domain setting, all methods use the same eight datasets and Transformer backbone, the federated methods follow one protocol (App. C.1), and All Mixed pools these data in a single centralized run. In the large-scale setting, all methods use the same 302M backbone, the Time-MoE-300B corpus, and the same total number of sequences and updates across clients (App. C.6). We have added the GPU-hours of every method, and FedTRL uses only 2.7% more than All Mixed (1,102 vs. 1,073), so the budgets also match in computation. Centralized TSFMs are compared through their public results, so their pretraining data and budgets are fixed by their original training. Among them, Time-MoE is pretrained on the same Time-MoE-300B corpus (Tab. 11).


We have added Tables R1 and R2 to App. E and stated the matched training budgets in App. C.6 of the revised manuscript.

We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Dear Reviewer egXg,

As the discussion period draws to a close, we would appreciate knowing whether our responses have addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly. We would also be happy to address any remaining questions during the remaining discussion period.

Best regards,
The Authors

Reviewer_vfq1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FedTRL, a federated‑learning‑based pretraining framework for time‑series foundation models that addresses bi‑level heterogeneity. Locally, it combines diffusion‑based reconstruction with sub‑domain adversarial regularization and prototype alignment to mitigate within‑client conflicts. The authors conduct extensive evaluations on in‑domain, full‑shot and zero‑shot point and probabilistic forecasting tasks across multiple benchmarks, including a newly‑collected real‑world weather RW‑Bench. Results show FedTRL outperforms existing federated baselines and is competitive with centralized pretrained time‑series foundation models, with supporting ablations, representation analysis, and scalability tests verifying each core component.

Critical strengths

The paper identifies a practical gap in existing federated time‑series learning because prior work mainly addresses cross‑client (inter‑domain) heterogeneity while ignoring intra‑domain conflicts inside each client. The authors empirically verify that mixed‑batch centralized pretraining produces representation collapse and gradient conflict originating from this two‑level heterogeneity, rather than only relying on conceptual argument.

The work conducts extensive evaluation covering in‑domain, full‑shot, zero‑shot point and probabilistic forecasting across multiple public benchmarks plus the newly‑introduced RW‑Bench real‑world weather dataset. It includes rigorous ablations for each core component, backbone scalability tests, robustness checks under domain imbalance, leave‑one‑domain‑out generalization studies, and communication‑computation overhead analysis. FedTRL demonstrates that federated pretraining can match or outperform centralized time‑series foundation models without collecting raw client data.

Critical weaknesses

FedTRL relies on several client‑side prerequisites, i.e., sub‑domain labels (metadata or k‑means pseudo‑labels) for adversarial regularization, plus multiple hyperparameters for DaG aggregation. While ablations test label noise, the method does not evaluate real‑world scenarios where sub‑domain signals are ambiguous or cannot be reliably derived. When sub‑domain partitioning fails, the core intra‑domain mitigation mechanism degrades.

Empirical evaluation is almost entirely restricted to forecasting (point and probabilistic). Only a quick linear‑probe sanity‑check is provided for time‑series classification, while imputation, anomaly detection and other common time‑series foundation‑model use‑cases remain untested. Furthermore, the provided theoretical convergence proof only covers the server‑side DaG aggregation, and explicitly excludes the saddle‑point GRL adversarial local optimization dynamics.

Questions for authors
  1. Sub‑domain adversarial regularization depends on sub‑domain labels, either from metadata or k‑means pseudo‑labels. In your current experiments you test only limited label‑flip noise. Can you report performance under scenarios where latent sub‑domains are heavily overlapping and cannot be cleanly partitioned (hard clustering setting)?

  2. The theoretical convergence analysis only proves guarantees for DaG server‑side aggregation and explicitly excludes the GRL saddle‑point local optimization loop. Can you provide empirical convergence diagnostics (e.g. training objective curves, representation probe trends) tracking stability of the full end‑to‑end FedTRL training procedure?

  3. You show a small linear‑probe result for time‑series classification as a sanity check, but all major benchmarking is limited to forecasting. Can you present full comparable quantitative results on at‑least two standard held‑out time‑series classification benchmarks using the exact same pretrained FedTRL encoder without further tuning?

  4. DaG aggregation uses two hyper‑parameters α and β weighting discriminability and prototype deviation. While you provide a sensitivity sweep, what practical signal can a practitioner use to set α and β for a completely new federated dataset with no held‑out validation set?

4 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.


W1 & Q1 [Sub-domain labels when latent sub-domains overlap heavily and cannot be cleanly partitioned]

We have added experiments with heavily overlapping sub-domains. FedTRL outperforms FedAvg at all overlap levels, even when k-means can no longer reliably recover the underlying sub-domains. This robustness is consistent with FedTRL's design: sub-domain labels are used only by the local classifier on the prototype path, whereas prototype alignment requires no sub-domain labels, and DaG trains its server classifier using client indices.

(1) Settings in the Paper. Beyond label-flip noise, Tab. 7(a) evaluates k-means pseudo-labels derived without metadata and shows that GRL reduces prediction errors when clean metadata labels are available. When no reliable sub-domain structure can be derived, a single label per client serves as the fallback, disabling GRL while retaining the other components.

(2) Heavily Overlapping Sub-Domains. To simulate sub-domain overlap, each standardized pretraining series x\mathbf{x} is replaced by (1−ρ)x+ρx~(1-\rho)\mathbf{x}+\rho\tilde{\mathbf{x}}, where x~\tilde{\mathbf{x}} is randomly sampled from another sub-domain of the same client. Hard labels are obtained from k-means, with MkM_k selected by the silhouette score, while soft labels use Gaussian-mixture posteriors as classifier targets. Table R1 below reports the MSE together with the silhouette score for label-free cluster separation and the adjusted Rand index (ARI) for recovery of true sub-domains. As overlap increases, ARI drops from 0.76 to 0.06, and the benefit of GRL diminishes. Nevertheless, FedTRL with hard labels outperforms its single-label variant at every overlap level. Soft labels further improve performance, particularly under heavy overlap, and even the single-label variant outperforms FedAvg.

Table R1. MSE (Weather in-domain / RW-Bench zero-shot) under increasing sub-domain overlap.

Overlap ρ\rhoSilhouette / ARIFedTRL, hard k-meansFedTRL, soft labelsFedTRL, single label (no GRL)FedAvg
00.39 / 0.760.244 / 0.6030.242 / 0.6010.260 / 0.6170.264 / 0.729
0.20.16 / 0.340.248 / 0.6100.246 / 0.6070.263 / 0.6210.268 / 0.734
0.40.04 / 0.060.258 / 0.6200.254 / 0.6160.262 / 0.6240.265 / 0.738

Q2 [Empirical stability of the full end-to-end training procedure]

*We have added empirical diagnostics of the full FedTRL training procedure, including GRL-based local optimization (Table R2 below). The validation reconstruction loss steadily decreases, with fluctuations below 0.005 within each window. The normalized adversarial loss Ldom/ln⁡Mk\mathcal{L}_{\text{dom}}/\ln M_k approaches 1 (uniform prediction over MkM_k balanced sub-domains), while the within-client probe accuracy drops from 69% to 42%, indicating progressively reduced sub-domain information. For DaG, the minimum weight remains at least 0.053, consistent with Cor. D.5, and the between-round weight change decreases from 0.064 to 0.004. Together with the improvement in TSLib zero-shot MSE, these results indicate stable end-to-end FedTRL training.

Table R2. Training diagnostics of the 302M FedTRL model, averaged over 100-round windows.

RoundValidation reconstruction lossLdom/ln⁡Mk\mathcal{L}_{\text{dom}}/\ln M_kSub-domain probe accuracymin⁡kwk\min_k w_k∥wt−wt−1∥1\lVert \mathbf{w}^{t}-\mathbf{w}^{t-1}\rVert_1TSLib zero-shot MSE
5000.4170.5769%0.0530.0640.373
1,0000.3410.7458%0.0570.0410.350
2,5000.2890.8851%0.0560.0170.337
5,0000.2650.9445%0.0600.0110.330
7,5000.2520.9344%0.0620.0060.328
10,0000.2480.9742%0.0610.0040.324

Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


W2 & Q3 [Evaluation beyond forecasting with the same frozen encoder]

Beyond the point and probabilistic forecasting tasks targeted by FedTRL, we have added evaluations on UCR and UEA classification, imputation, and anomaly detection. FedTRL achieves the best aggregate results across all three tasks using the same pretrained 302M encoder without further tuning. All compared encoders are also frozen and evaluated under the same downstream protocols, with federated baselines sharing our backbone and pretraining corpus. Specifically, we address this from three aspects below:

(1) Classification on UCR and UEA. Mean-pooled encoder representations are classified using an RBF-kernel SVM, with hyperparameters selected by cross-validation on each training split. Neither archive overlaps with the pretraining corpus, and Table R3 below reports the results. FedTRL also outperforms FFTS on 91 of 128 UCR and 22 of 30 UEA datasets.

Table R3. Average classification accuracy.

Frozen encoderUCRUEA
FedTRL0.7830.706
FFTS0.7590.681
FedAvg0.7460.674
All Mixed0.7520.669
Time-MoE (base)0.7310.657

(2) Imputation. A linear head on all token representations is trained on each target training split, and Table R4 below reports the results.

Table R4. Imputation MSE averaged over point-masking ratios of {12.5, 25, 37.5, 50}%.

Frozen encoder + linear headETTh1ETTm1Weather
FedTRL0.0850.0330.032
FFTS0.0910.0390.036
FedAvg0.0980.0380.039
All Mixed0.0930.0410.035

(3) Anomaly Detection. For each of the 250 series, a linear reconstruction head is trained on the frozen token representations of its anomaly-free training segment, and its reconstruction error serves as the anomaly score. Table R5 below reports the results.

Table R5. Anomaly detection on the UCR Anomaly Archive.

Frozen encoderAdjusted best F1VUS-ROC
FedTRL0.6470.739
FFTS0.6130.724
All Mixed0.6190.718
FedAvg0.5980.711

Q4 [Setting α and β for a new federation without a validation set]

We use two training-time signals to set α\alpha and β\beta: the across-client variation in ℓk\ell_k and δk\delta_k, and the effective client ratio Keff/KK_{\text{eff}}/K. We have added a fixed rule based on these signals, which requires no validation set and performs comparably to the default setting. Specifically, we address this from two aspects below:

(1) Default Values. FedTRL uses α=1.0\alpha=1.0 and β=0.5\beta=0.5 unchanged across 8 TSLib clients with a 4-layer encoder and 9 Time-MoE domains with a 302M encoder (Tab. 10). In Fig. 7, TSLib zero-shot MSE varies by at most 0.005 for β∈[0.1,0.7]\beta\in[0.1,0.7] and 0.001 for α∈[0.7,1.0]\alpha\in[0.7,1.0], showing limited sensitivity to their exact values.

(2) Rule without a Validation Set. To account for different scales of ℓk\ell_k and δk\delta_k, we discard any term whose across-client standard deviation is below 1% of its mean, standardize the remaining terms, and set α=β=1\alpha=\beta=1 in each round. We then compute Keff=1/∑kwk2K_{\text{eff}}=1/\sum_k w_k^2, where Keff/KK_{\text{eff}}/K near 1 indicates nearly uniform aggregation weights. If this ratio falls below 0.5, both coefficients are halved until it reaches 0.5 and then reset to 1 in the next round. We evaluate this fixed rule on both the default and 18-client partitions (Tab. 7(b)), and Table R6 below reports the results. Across both partitions, the fixed rule achieves MSE within 0.002 of the default setting, enabling validation-free selection of α\alpha and β\beta.

Table R6. Default and rule-based settings of α\alpha and β\beta for the 302M model.

SettingKeff/KK_{\text{eff}}/KTSLib zero-shotRW-Bench zero-shot
Default α=1.0\alpha=1.0, β=0.5\beta=0.50.890.3240.599
Standardized ℓk,δk\ell_k,\delta_k, α=β=1\alpha=\beta=10.850.3260.600
Default, 18-client partition0.870.3260.602
Standardized rule, 18-client partition0.840.3270.602

We have added Tables R1 to R5 to App. E of the revised manuscript.

We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Dear Reviewer vfq1,

Thank you for your positive assessment of our work and your constructive feedback.

As the discussion period draws to a close, we would appreciate knowing whether our responses have adequately addressed your concerns. We would also be happy to provide any further clarification during the remaining discussion period.

Best regards,
The Authors

AuthorsOpenReview ↗

Clarification on the updated rating

Dear Reviewer vfq1,

We noticed that your rating was lowered from Weak acceptance (3) to Weak rejection (2) without any follow-up comments. Could you kindly clarify which concerns remain unresolved after our detailed rebuttal and additional experiments? We would appreciate the opportunity to address them.

Best regards,
The Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for your feedback and valuable comments. Below is our point-by-point response.


W1 [Novelty, suitability for bi-level heterogeneity, and alternative strategies]

Our contribution lies in coupling GRL and DaG through the uploaded client prototype to address bi-level heterogeneity, rather than combining established techniques. Imagine a weather client whose stations lie both on the coast and in the mountains. Client partitioning separates it from other domains but does not resolve the client's conflicting dynamics: its prototype blends both patterns into a summary that fits neither and can mislead server-side aggregation. FedTRL addresses this using the same prototype: GRL removes within-client subdomain variation, while DaG uses the resulting prototypes to discriminate among and weight clients across domains. Specifically, we address this from three aspects below:

(1) Fit to Bi-Level Heterogeneity. Tab. 23 shows that FedTRL separates domains while mixing sub-domains within each client, consistent with the intended bi-level structure. Tab. 26(b) shows that the advantage of diffusion over masked reconstruction increases with the number of sub-domains per client, while Tab. 7(b) shows that fixing DaG weights after their initial computation degrades zero-shot performance.

(2) Alternative Strategies. Tab. 12 shows that the gain does not come from integration itself: FedPer, FedRep, and pFedMe, integrated into the same pretraining, do not improve over FedAvg on any dataset. We have added experiments that replace one component at a time in the 38M model (Table R1 below). FedTRL is best in both columns, and token-level GRL, the standard DANN form, is worse than no regularizer. Alternative regularizers are tuned over the λdom\lambda_{\text{dom}} grid, and the FedDisco-style rule uses prototype distance, as pretraining has no labels.

Table R1. Zero-shot MSE when replacing one component of FedTRL, 38M model.

Local regularizer (DaG fixed)TSLib / RW-BenchAggregation (local objective fixed)TSLib / RW-Bench
GRL on prototype (FedTRL)0.339 / 0.624DaG (FedTRL)0.339 / 0.624
None0.348 / 0.641Sample size (FedAvg rule)0.357 / 0.681
GRL on token features (DANN)0.350 / 0.644Uniform averaging0.351 / 0.674
MMD across sub-domains0.343 / 0.636Loss-based (q-FFL style)0.354 / 0.669
CORAL across sub-domains0.346 / 0.632Cosine to global prototype0.346 / 0.653
Discrepancy-aware (FedDisco style)0.345 / 0.646

(3) Component Coupling. Tab. 27 shows that GRL makes client prototypes more discriminable, strengthening the signal used by DaG. We have added an interaction analysis: GRL reduces RW-Bench MSE by 0.017 under DaG but only 0.010 under sample-size weighting (difference 0.007 ± 0.003 over three seeds). Randomly permuting DaG weights across clients in each round preserves their distribution but yields 0.353 / 0.677, no better than uniform averaging, so the gain depends on assigning the weights to the right clients.

Together, these results demonstrate that the coupling, rather than any individual component, drives FedTRL's advantage under bi-level heterogeneity.


W2 [Comparison with centralized models under identical architecture/budget]

Tab. 1 compares FedTRL and All Mixed across 7 of 8 datasets using the same backbone, data, and training protocol. We further examine the budget at scale, model sizes, and adaptation protocols below.

(1) Matched Comparison at Scale. All methods share the 302M backbone and Time-MoE-300B, with All Mixed and FedAvg using FedTRL's diffusion objective, and 1.0× denotes the same total number of updates across clients. We have added All Mixed with 1.5× updates and the GPU-hours of each method (Table R2 below). Even with 46% more GPU-hours, All Mixed at 1.5× underperforms FedTRL on all benchmarks.

Table R2. Large-scale results with a matched backbone & corpus.

MethodUpdatesGPU-hoursTSLibRW-BenchGIFT-Eval MASEFEV MASE
All Mixed1.0×1,0730.3450.6140.8230.894
All Mixed1.5×1,6070.3410.6120.8170.885
FedAvg1.0×1,0710.3480.7290.8720.890
FFTS1.0×7830.3370.6140.7660.876
FedTRL1.0×1,1020.3240.5990.6750.836

(2) Model Size and Adaptation Protocols. Tab. 3 shows that FedTRL (302M) outperforms the 113M, 453M, and 2.4B Time-MoE models pretrained on the same corpus. In zero-shot evaluations, FedTRL, All Mixed, and Time-MoE use no target adaptation. TimesFM and MOMENT are fine-tuned following the Time-MoE protocol because their output heads are not designed for the evaluated horizons. In the full-shot setting (Tab. 2), FedTRL is adapted for only one epoch.

These results establish that FedTRL's advantage persists under matched architectures, data, and update budgets.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


Q1 [Incorrect or unstable labels, and an unknown or changing number of sub-domains]

FedTRL remains robust to unreliable sub-domain labels: across all tested settings, its MSE differs from that with metadata labels by at most 0.009 on Weather and 0.011 on RW-Bench, and it outperforms its single-label variant. Tab. 7(a) already evaluates 20% and 50% label flips. We have added experiments on misspecified MkM_k, periodic re-clustering, and the emergence of new sub-domains during training (Table R3 below). Re-clustered labels are aligned across rounds by Hungarian matching, while MkM_k is re-selected by the silhouette score when new sub-domains appear.

Table R3. MSE (Weather in-domain / RW-Bench zero-shot) under unreliable labels.

Sub-domain labelsWeatherRW-Bench
Metadata0.2410.599
k-means, MkM_k by silhouette0.2440.603
k-means, MkM_k halved0.2480.608
k-means, MkM_k doubled0.2460.604
k-means re-clustered every 10% of training0.2470.610
Half of the sub-domains appear at mid-training0.2500.606
Single label, no GRL0.2600.617

Q2 [How GRL preserves forecasting-relevant temporal patterns, with direct evidence]

GRL acts only on the pooled prototype, suppressing series-level sub-domain information while the reconstruction objective preserves predictive temporal patterns. With pˉ=Mean(H)\bar{\mathbf{p}}=\mathrm{Mean}(\mathbf{H}), each token receives the same direct adversarial gradient 1N∂Ldom/∂pˉ\frac{1}{N}\partial\mathcal{L}_{\text{dom}}/\partial\bar{\mathbf{p}}, so the adversarial signal acts on the series-level mean rather than differences between time steps.

We have added linear probes as direct evidence (Table R4 below). The probes use token representations from the frozen causal 302M encoder, with a temporal 80/20 split. Period labels are the dominant FFT period class (daily, weekly, or none). Compared with no GRL, prototype-level GRL reduces sub-domain probe accuracy from 74% to 42% while preserving period accuracy and improving both next-patch and zero-shot MSE. In contrast, token-level GRL further reduces sub-domain accuracy to 36% but lowers period accuracy to 70.8% and increases both prediction errors.

Table R4. Linear probe results. Zero-shot MSE is averaged over ETTh1, ETTm1, and Weather.

VariantSub-domain probe ↓Period probe (balanced acc.) ↑Next-patch MSE ↓Zero-shot MSE ↓
No GRL74%83.4%0.4160.335
GRL on prototype (FedTRL)42%82.9%0.4040.329
GRL on token features36%70.8%0.4410.347

Q3 [Many clients, intermittent participation, independent drift, and cost relative to FedAvg]

We have added experiments with up to 72 clients, 25% participation, and independent distribution drift (Table R5 below). FedTRL outperforms FedAvg in all configurations.

(1) Scale, Participation, and Drift. Each Time-MoE-300B domain is divided among multiple clients, each retaining several sub-domains. Active clients are sampled uniformly each round, with the total number of rounds unchanged, and only these clients enter DaG. Under independent drift, each client follows its own random schedule of changes in amplitude, period, and noise level.

Table R5. RW-Bench zero-shot MSE and DaG time per round, 38M model.

ClientsParticipationIndependent driftFedAvgFedTRLDaG time (s)
9100%no0.7430.6241.8
36100%no0.7560.6315.9
72100%no0.7750.63312.1
7225%no0.7890.6433.4
7225%yes0.8090.6583.6

(2) Cost Relative to FedAvg. In the 302M comparison (Table R2), FedTRL requires only 2.9% more GPU-hours than FedAvg, which uses the same diffusion objective. Each participating client uploads one additional prototype per round (at most 4 KB in float32), adding negligible communication overhead (App. E.8). Table R5 shows that DaG time scales with the number of active clients, decreasing from 12.1 s at full participation to 3.4 s at 25% participation.


We have added Tables R1 and R3 to R5 to App. E of the revised manuscript.

We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

Thank you for your time and effort in reviewing and handling our paper. We have posted point-by-point responses to each reviewer and summarize the main points below.


Paper Overview and Strengths

[Paper Overview]: FedTRL is a federated pretraining method for time series foundation models that targets bi-level heterogeneity across and within clients. A local objective combines diffusion-based reconstruction with sub-domain adversarial regularization and prototype alignment, and a domain-aware aggregation (DaG) re-weights clients. It matches or surpasses centralized and federated baselines and degrades least under imbalance, drift, and unseen-domain transfer.

[Strengths]: We are encouraged by the recognition of the problem formulation, which identifies a practical gap (vfq1) and addresses inter-domain and intra-domain heterogeneity jointly (zYfG), of the empirical evidence of representation collapse and gradient conflict (vfq1), of the well-motivated combination of local regularization and domain-aware aggregation (zYfG), of the extensive evaluation (vfq1), of the improvements over federated baselines on several benchmarks and backbones (egXg), and of the clear presentation (vfq1, zYfG).

Additional Experiments for Concerns

To address the main concerns, we conducted the following experiments.

  • [Unreliable Sub-Domain Labels] (vfq1, zYfG): Heavily overlapping sub-domains with hard and soft labels (Table R1 for vfq1), and a misspecified number of sub-domains, re-clustering, and emerging sub-domains (Table R3 for zYfG), where FedTRL outperforms its single-label variant.
  • [Training Stability] (vfq1): Diagnostics of the full procedure with GRL over 10,000 rounds (Table R2 for vfq1), indicating stable training.
  • [Tasks Beyond Forecasting] (vfq1): UCR and UEA classification, imputation, and anomaly detection with the same frozen encoder, where FedTRL is best in every task (Tables R3 to R5 for vfq1).
  • [α\alpha and β\beta Without a Validation Set] (vfq1): A fixed rule based on the spread of ℓk\ell_k and δk\delta_k and the effective number of clients, performing comparably to the default values on two partitions (Table R6 for vfq1).
  • [Reliability of DaG Weights] (egXg): DaG weights correlate with measured client utility more closely than ℓk\ell_k alone or sample size (Tables R1 and R1b for egXg), and a client with a source-specific shortcut is down-weighted (Table R2 for egXg).
  • [Negative Transfer] (egXg): Relative to training on each dataset alone, FedAvg shows negative transfer on several datasets, whereas FedTRL outperforms it on all eight (Table R3 for egXg).
  • [Alternative Strategies and Coupling] (zYfG): Replacing one component at a time with alternative regularizers or aggregation rules leaves FedTRL best in both groups (Table R1 for zYfG), and GRL yields a larger gain under DaG than under sample-size weighting.
  • [Mechanism of GRL] (zYfG): Linear probes show that prototype-level GRL suppresses sub-domain but not period information, whereas token-level GRL suppresses both (Table R4 for zYfG).
  • [Scale, Participation, and Drift] (zYfG): Up to 72 clients, 25% participation, and independent drift, where FedTRL outperforms FedAvg in every configuration with server-side cost linear in the number of active clients (Table R5 for zYfG).

These results validate our motivation and design.


Summary of Responses to Reviewers' Questions

  • [Novelty] (zYfG): The contribution is the coupling of GRL and DaG through the uploaded prototype. Integrating established mechanisms into the same pretraining is not sufficient (Tab. 12), and the gain disappears with standard token-level GRL or permuted DaG weights.
  • [Role of ℓk\ell_k] (egXg): DaG never uses ℓk\ell_k alone but combines it with the prototype deviation δk\delta_k, so a client identifiable only through a source-specific pattern receives a low weight.
  • [Controlled and Matched Comparisons] (egXg, zYfG): Tab. 1 compares FedTRL with All Mixed under the same backbone, data, and protocol, and the large-scale comparison matches backbone, corpus, sequences, updates, and GPU-hours, where All Mixed with 1.5× updates underperforms FedTRL (Table R2 for zYfG).

Revisions During the Rebuttal

All changes are highlighted in blue.

  • [App. E]: Tables R1 to R5 for vfq1, R1, R1b, and R2 for egXg, R1 to R5 for zYfG, and the interaction analysis.
  • [Sec. 3.2 and App. B.2]: the role of ℓk\ell_k as one of two terms of the DaG score.
  • [App. C.6]: the matched training budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.