acceptodds
Desk rejected at ICLR 2027

Ada-MoF: Adaptive Mixture of Frequency-domain Experts for Time Series Forecasting

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multivariate time series forecasts are widely used, such as meteorology, transportation and financial forecasts. However, the dominant frequencies in time series may shift with the evolving spectral distribution of the data. Traditional Mixture of Experts (MoE) models, which employ a fixed number of experts, struggle to adapt to these changes, resulting in frequency coverage imbalance issue. Specifically, too few experts can lead to the overlooking of critical information, while too many can introduce noise. To this end, we propose Ada-MoF, an adaptive Mixture of Frequency-domain Experts model. Ada-MoF integrates univariate spectral intensity and cross-variable frequency response features to adaptively determine the number of experts, ensuring alignment with the input data's frequency distribution. This approach prevents both information loss due to an insufficient number of experts and noise contamination from an excess of experts. The experimental results show that our model achieves state-of-the-art performance on six public benchmarks with only 0.2 million parameters.

Primary area:
learning on time series and dynamical systems
Keywords:
Mixture of Expert, Adaption, Gauss

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VddrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Ada-MoF, an adaptive mixture-of-experts framework in the frequency domain for time series forecasting. The authors propose an adaptive expert allocation mechanism to address the frequency coverage imbalance caused by fixed expert selection, along with Gaussian band-pass filters to mitigate the edge oscillations associated with the Gibbs phenomenon. Experiments on public benchmark datasets demonstrate competitive forecasting performance compared with seven baseline methods.

Critical strengths
  1. The overall framework is straightforward and easy to follow, with a clear motivation for adaptive expert selection and frequency-domain feature decoupling.

  2. The proposed method achieves promising forecasting improvements with a relatively small number of additional parameters. The resulting model remains lightweight, with approximately 0.2 million parameters in the reported configuration.

Critical weaknesses
  1. Ada-MoF appears to function more as a plug-in module than as an independent forecasting model, and its methodological contribution should be more clearly positioned. Although integrating Ada-MoF into existing baselines generally improves their forecasting performance, the comparison between TimeMixer + Ada-MoF and standalone baselines in Table 2 may not be entirely fair, as the former benefits from an additional module and increased model capacity. Furthermore, the implementation details of how Ada-MoF is integrated into different forecasting backbones should be more thoroughly described to ensure reproducibility.

  2. It remains unclear to what extent adaptive expert selection improves forecasting performance compared with conventional fixed Top-K selection. A controlled comparison under the same backbone and expert configurations would help isolate the contribution of the adaptive expert budget. In addition, more in-depth analyses, such as the distribution of activated experts across different samples, are needed to validate the effectiveness of the proposed mechanism. It is also unclear whether the learnable frequency partitioning actually produces frequency-specialized experts. Visualizations of the learned Gaussian filters and expert activation patterns across frequencies would provide stronger evidence for this claim.

Questions for authors

Does Equation (13) mathematically guarantee non-overlapping frequency subbands? If so, could the authors clarify how this property is achieved?

1 comment

@W1

We agree that Ada-MoF is a plug-in, not a standalone forecasting model. Table 2 is a system-level comparison and cannot isolate causality.

For TimeMixer, Ada-MoF is inserted independently at every temporal scale, right after temporal embedding and before Past Decomposable Mixing (PDM). Here BB is mini-batch size, LL the original look-back length, VV the number of input variables (called NN in the implementation), ss a scale index, LsL_s the sequence length at scale ss with L0=LL_0=L, DD the embedding dimension (d_model), Fs=⌊Ls/2⌋+1F_s=\lfloor L_s/2\rfloor+1 the number of one-sided real-FFT bins, MM the number of candidate experts, Kb,sK_{b,s} the sample-specific number of active experts at scale ss, and HH the forecasting horizon. The product B⋅VB\cdot V means each variable of each mini-batch item is treated as one channel-independent sequence.

At each scale ss, input X∈RB×L×VX\in\mathbb{R}^{B\times L\times V} is normalized and downsampled to X(s)∈RB×Ls×VX^{(s)}\in\mathbb{R}^{B\times L_s\times V}. It is permuted to [B,V,Ls][B,V,L_s] and reshaped to [B⋅V,Ls,1][B\cdot V,L_s,1], folding the variable axis into the batch axis without merging variables, then mapped to DD latent channels to give E(s)∈RB⋅V×Ls×DE^{(s)}\in\mathbb{R}^{B\cdot V\times L_s\times D}. Transposing the last two axes yields E~(s)∈RB⋅V×D×Ls\widetilde E^{(s)}\in\mathbb{R}^{B\cdot V\times D\times L_s}, so Ada-MoF operates along the temporal axis while DD holds latent embedding channels rather than the original VV variables.

A real FFT along LsL_s gives Z(s)∈CB⋅V×D×FsZ^{(s)}\in\mathbb{C}^{B\cdot V\times D\times F_s}. The router summarizes ∣Z(s)∣|Z^{(s)}| to predict MM expert scores and the budget Kb,sK_{b,s}, then activates the corresponding experts. Each selected Gaussian spectral filter multiplies the complex FFT coefficients, changing spectral amplitude while retaining the input phase. The gated expert spectra are combined, and an inverse real FFT of length LsL_s returns E^(s)∈RB⋅V×D×Ls\widehat E^{(s)}\in\mathbb{R}^{B\cdot V\times D\times L_s}. Transposing back gives [B⋅V,Ls,D][B\cdot V,L_s,D]. All scales are passed to PDM, which mixes decomposed seasonal/trend information across scales, and the prediction head maps them to the future horizon and restores the channel-independent layout, yielding [B,H,V][B,H,V].

As shown in Table 6 of the original appendix, TimeMixer+Ada-MoF adds only 2M parameters, which is under 0.5% of the total, yet brings about a 2% MSE improvement over TimeMixer, and Table 7 further validates its generalization by showing consistent gains when Ada-MoF is added to multiple models.

@W2:

For ETTh1-96, the four scales collectively exhibit selections from K=1 through K=5. This view exposes both sample-level and run-to-run routing variation while preserving the original counts. Adaptive K is directionally better than validation-tuned fixed K=3 and compute-matched fixed K=4 in mean MSE.

ScaleK=1K=2K=3K=4K=5Mean K
Scale 028.78%8.05%2.65%0.50%60.01%3.55
Scale 10.00%2.27%17.73%0.19%79.81%4.58
Scale 20.00%1.71%33.29%25.23%39.77%4.03
Scale 30.00%23.18%16.82%45.05%14.95%3.52
Budget ruleValidation MSETest MSETest MAE
Fixed K=10.79700.41320.4242
Fixed K=20.74180.40810.4202
Fixed K=30.70710.39050.4085
Fixed K=40.71170.39740.4152
Fixed K=50.72020.40360.4176
Adaptive (default Kmax=5)0.70490.38550.4031

@Q1:

Equation (13) does not guarantee non-overlapping sub-bands and is not the implemented operation. The corrected positive Gaussian windows have infinite tails and are therefore softly overlapping.

For the representative ETTh1-96, the learned finest-scale Gaussian windows have a mean pairwise cosine overlap of 0.051, the measured overlaps across the four scales are approximately 0.05, 0.05, 0.06, and 0.13, with the coarsest scale selecting K=2K=2 for 3,333 routed samples and K=3K=3 for 16,162; since real positive windows multiply complex FFT coefficients, the expert-specific spectra retain the original phase for the inverse FFT.

ScaleMean filter overlapBand-width rangeExperts with nonzero activationTop-3 activation countsExpert spectral peaks (normalized frequency)
Scale 00.0520.078–0.1369E3:19495, E4:19495, E5:194810.08, 0.17, 0.25, 0.33, 0.50, 0.50, 0.58, 0.71, 0.81, 1.00
Scale 10.0530.069–0.1439E2:19495, E4:19495, E1:194780.04, 0.17, 0.29, 0.33, 0.50, 0.50, 0.67, 0.75, 0.83, 1.00
Scale 20.0620.077–0.1308E9:19445, E6:19366, E1:186880.08, 0.08, 0.25, 0.33, 0.42, 0.58, 0.67, 0.67, 0.83, 0.92
Scale 30.1270.067–0.1188E7:19495, E8:19338, E6:90550.17, 0.17, 0.17, 0.33, 0.50, 0.67, 0.67, 0.67, 0.83, 1.00
Reviewer_YP74OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Seems very close to FreqMoE except the number of expers (hence how fine the spectrum is sliced per expert) is made adaptive. Such approach are prone to overfitting, this aspect is not explored well though.

Critical strengths

a fairly detailed comparison in Table 2, though without confidence intervals.

Critical weaknesses

Please see my introductory comments.

Questions for authors

Please elaborate on the learning head architecture. Also explain why freq only representation is adequate, as it goes against 80+ years of work on time series in different fields.

Reviewer_FGKgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ada-MoF, a frequency-domain mixture-of-experts module for multivariate time-series forecasting. It combines learnable Gaussian-based spectral filters with a router that uses frequency-wise and variable-wise magnitude summaries to predict routing scores and a sample-specific number of active experts. A straight-through soft-hard router is used for training, while inference executes the selected experts. The paper evaluates the module with several forecasting backbones and reports component ablations, computational costs, and five-seed results.

Critical strengths

S1. The proposed module is evaluated as an add-on to several forecasting backbones, rather than only in a single architecture. Table 1 reports improvements in multiple settings; for example, PatchTST on ETTh2 improves from 0.385/0.410 to 0.371/0.399 in MSE/MAE. This provides some evidence that the module may be useful beyond one backbone.

S2. The added parameter cost is relatively modest in the configurations reported in Table 6, and the paper includes five-seed results in Table 10. These are useful steps toward assessing practicality and run-to-run stability.

Critical weaknesses

W1. Equation (13) subtracts two Gaussians with the same width and centers at f1 and f2. At the midpoint (f1+f2)/2, the two terms are equal, so H(f)=0, even though the text describes this interval as the passband whose energy should be retained. The paper also describes the resulting bands as non-overlapping and the expert inputs as statistically independent, but these properties do not follow from the displayed expression. It is unclear whether this is a typographical error or the filter actually used in the experiments. Since frequency decomposition is central to Ada-MoF, this ambiguity makes the method difficult to verify.

W2. The argument around Equations (2)–(3) treats the effective frequency span as if it were covered by roughly K regions of characteristic bandwidth w, but the paper does not define how f_min, f_max, or w are estimated in the implemented model, nor does it establish that the learned Gaussian filters behave like such a partition. Consequently, the coverage-ratio discussion is an intuition rather than a validated explanation of why a sample-specific expert budget should improve forecasting.

W3. Table 3 removes the adaptive learner as a whole, which changes more than the budget decision. Tables 4 and 9 compare input features for the learner, while Table 8 varies the maximum number of experts. These experiments do not provide a controlled comparison between adaptive K and a tuned fixed-K router while holding the filter bank, score network, and training protocol constant. The evidence therefore supports the usefulness of the combined module more clearly than it supports adaptive expert count as a distinct contribution.

W4. The paper argues that the Gaussian filters produce compact, non-overlapping sub-bands, that experts receive clean frequency slices, and that the router suppresses noise-dominated bands. However, the reported results focus primarily on downstream forecasting errors. The paper does not show filter responses, the learned band boundaries, representative per-expert spectra, or the distribution of selected K values that would allow readers to verify these mechanism-level claims. This leaves the proposed explanation for the gains under-supported.

W5. In Section 4.2, the paper says that adding the adaptive learner on top of Gaussian experts reduces Solar error by 14.2% from the baseline. Table 3 reports Solar MSE changing from 0.216 to 0.208, which is approximately a 3.7% reduction. Table 9, meanwhile, lists a Solar baseline MSE of 0.242 while retaining 0.208 for Dual Feature. These differences may reflect different settings, but the paper does not explain them. This makes it harder to interpret the claimed size of the ablation effect and compare results across sections.

Questions for authors

1.Does Equation (13) match the filter implementation used in the experiments? If so, how is the zero response at the midpoint of the stated passband reconciled with the claimed band-pass behavior? If it is a typographical error, please provide the actual expression.

2.How are f_min, f_max, and the characteristic bandwidth w operationally defined or estimated? Is there empirical evidence that the learned filters form the non-overlapping sub-bands described in the paper?

3.Can the authors compare adaptive K with a tuned fixed K while keeping the filter bank, routing-score network, and training protocol unchanged, preferably at a similar average number of active experts?

4.Can the authors provide visualizations or quantitative summaries of the learned filter responses, expert-specific spectra, and the distribution of selected K values to substantiate the specialization and noise-suppression claims?

5.Do Tables 3 and 9 use different Solar settings? Please reconcile the baseline MSE values and the 14.2% improvement stated in Section 4.2 with the values in Table 3.

1 comment

Q1:

We are sorry for the confusion this caused to your understanding. We made some writing errors, and the correct setting in the code should be one-sided rather than two-sided.

For the correction, we use ordered continuous boundaries

c_i=\frac{b_i+b_{i+1}}{2},$$ and define each learned interval as the full width at half maximum of a positive Gaussian, $$\sigma_i=\frac{b_{i+1}-b_i}{2\sqrt{2\ln 2}},\qquad H_i(f)=\exp\left[-\frac{(f-c_i)^2}{2\sigma_i^2}\right].$$ This expression has $0 < H_i(f) \le 1$ and no phase inversion. Gaussian tails are nonzero, so we replace non-overlapping and statistically independent with localized, softly overlapping windows and measure overlap directly. On ETTh1-96, the finest-scale mean off-diagonal cosine overlap is 0.051. ## Q2: For the representative ETTh1-96, the learned finest-scale Gaussian windows have a mean pairwise cosine overlap of 0.051, the measured overlaps across the four scales are approximately 0.05, 0.05, 0.06, and 0.13, with the coarsest scale selecting $K=2$ for 3,333 routed samples and $K=3$ for 16,162. Since real positive windows multiply complex FFT coefficients, the expert-specific spectra retain the original phase for the inverse FFT. | Scale | Mean filter overlap | Band-width range | Experts with nonzero activation | Top-3 activation counts | Expert spectral peaks (normalized frequency) | |---|---:|---:|---:|---|---| | Scale 0 | 0.052 | 0.078–0.136 | 9 | E3:19495, E4:19495, E5:19481 | 0.08, 0.17, 0.25, 0.33, 0.50, 0.50, 0.58, 0.71, 0.81, 1.00 | | Scale 1 | 0.053 | 0.069–0.143 | 9 | E2:19495, E4:19495, E1:19478 | 0.04, 0.17, 0.29, 0.33, 0.50, 0.50, 0.67, 0.75, 0.83, 1.00 | | Scale 2 | 0.062 | 0.077–0.130 | 8 | E9:19445, E6:19366, E1:18688 | 0.08, 0.08, 0.25, 0.33, 0.42, 0.58, 0.67, 0.67, 0.83, 0.92 | | Scale 3 | 0.127 | 0.067–0.118 | 8 | E7:19495, E8:19338, E6:9055 | 0.17, 0.17, 0.17, 0.33, 0.50, 0.67, 0.67, 0.67, 0.83, 1.00 | ## Q3: As shown in the following table, adaptive K is directionally better than validation-tuned fixed K=3 and compute-matched fixed K=4 in mean MSE. | Budget rule | Validation MSE | Test MSE | Test MAE | |---|---:|---:|---:| | Fixed K=1 | 0.7970 | 0.4132 | 0.4242 | | Fixed K=2 | 0.7418 | 0.4081 | 0.4202 | | Fixed K=3 | 0.7071 | 0.3905 | 0.4085 | | Fixed K=4 | 0.7117 | 0.3974 | 0.4152 | | Fixed K=5 | 0.7202 | 0.4036 | 0.4176 | | Adaptive (default Kmax=5) | 0.7049 | 0.3855 | 0.4031 | ## Q4: Same as Q2. For the representative ETTh1-96, the learned finest-scale Gaussian windows have a mean pairwise cosine overlap of 0.051, the measured overlaps across the four scales are approximately 0.05, 0.05, 0.06, and 0.13, with the coarsest scale selecting $K=2$ for 3,333 routed samples and $K=3$ for 16,162; since real positive windows multiply complex FFT coefficients, the expert-specific spectra retain the original phase for the inverse FFT. | Scale | Mean filter overlap | Band-width range | Experts with nonzero activation | Top-3 activation counts | Expert spectral peaks (normalized frequency) | |---|---:|---:|---:|---|---| | Scale 0 | 0.052 | 0.078–0.136 | 9 | E3:19495, E4:19495, E5:19481 | 0.08, 0.17, 0.25, 0.33, 0.50, 0.50, 0.58, 0.71, 0.81, 1.00 | | Scale 1 | 0.053 | 0.069–0.143 | 9 | E2:19495, E4:19495, E1:19478 | 0.04, 0.17, 0.29, 0.33, 0.50, 0.50, 0.67, 0.75, 0.83, 1.00 | | Scale 2 | 0.062 | 0.077–0.130 | 8 | E9:19445, E6:19366, E1:18688 | 0.08, 0.08, 0.25, 0.33, 0.42, 0.58, 0.67, 0.67, 0.83, 0.92 | | Scale 3 | 0.127 | 0.067–0.118 | 8 | E7:19495, E8:19338, E6:9055 | 0.17, 0.17, 0.17, 0.33, 0.50, 0.67, 0.67, 0.67, 0.83, 1.00 | For ETTh1-96, the four scales collectively exhibit selections from K=1 through K=5. This view exposes both sample-level and run-to-run routing variation while preserving the original counts. | Scale | K=1 | K=2 | K=3 | K=4 | K=5 | Mean K | |---|---:|---:|---:|---:|---:|---:| | Scale 0 | 28.78% | 8.05% | 2.65% | 0.50% | 60.01% | 3.55 | | Scale 1 | 0.00% | 2.27% | 17.73% | 0.19% | 79.81% | 4.58 | | Scale 2 | 0.00% | 1.71% | 33.29% | 25.23% | 39.77% | 4.03 | | Scale 3 | 0.00% | 23.18% | 16.82% | 45.05% | 14.95% | 3.52 ## Q5: The manuscript mixed an earlier unmatched baseline 0.242 with a corrected matched setting 0.216. The reconciled component values are reported in the following table. | Setting | MSE | MAE | |---|---:|---:| | baseline | 0.216 | 0.280 | | Gaussian experts only | 0.213 | 0.276 | | Gaussian + adaptive learner | 0.208 | 0.272 |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.