acceptodds
Under review as a conference paper at ICLR 2027

Not All Bands Need All Modalities: Frequency-Gated Multimodal Fusion for Time Series Forecasting

Abstract

Multimodal time series forecasters augment numerical inputs with text and image views, but existing designs absorb both auxiliary views indiscriminately: regardless of what these views actually describe, one fusion decision is applied to the whole series and therefore to every frequency component alike. We argue that fusion should instead be selective in the spectral domain. For example, the trend and seasonality summarised by text describe slow, low-frequency structure. High-frequency components should therefore absorb less from the text view than low-frequency ones, since fusing it across the whole spectrum can inject noise into the forecast. To address this issue, we propose **FreqGate**, a frequency-gated multimodal forecaster that assigns fusion weights to each frequency band of the input series. Soft Frequency Partitioning first learns a soft assignment of the spectrum of each input series to low-, mid- and high-frequency bands, rather than imposing a fixed partition shared by all series. Band-Gated Fusion then maps band energies to gates for the text and image encoders. Within each band, numerical tokens attend to the encoders through cross-attention weighted by the corresponding gates. As a result, each auxiliary view contributes only to the bands it can express, learned per series from its spectrum alone. Extensive experiments on eight benchmarks under full-shot, few-shot and zero-shot regimes demonstrate that FreqGate outperforms fifteen existing forecasters.

Primary area:
learning on time series and dynamical systems
Keywords:
multimodal time series forecasting, spectral routing, cross-modal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vWTtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes FreqGate, a multimodal forecaster that adds text and image views built from the input window (a statistics caption and a GAF, recurrence plot and line chart image, both encoded by frozen ViLT). The main idea, called spectral locality, is that these views mainly describe low and mid frequency content, so their fusion should be controlled per frequency band. Soft Frequency Partitioning learns a per-window soft split of FFT bins into three bands, and the band energies drive a sigmoid gate for each modality and band. Each band view of the numerical tokens then cross-attends to the text and image tokens, scaled by its gate. Experiments cover eight LTSF benchmarks at L=96 under full-shot, few-shot and zero-shot settings against fifteen baselines, with ablations and an analysis of the learned gate values.

Critical strengths

[1] The question is reasonable and the mechanism is simple.

Most tri-modal forecasters mix the auxiliary representation into the whole series at one strength, so asking whether a rendered image or a template caption is useful only for some frequencies is a fair hypothesis to test. The method is cheap (an FFT, a few small MLPs and K cross-attentions), and the released code follows the equations in Section 3 closely, which made it possible for me to check what the ablations actually do.

[2] The evaluation covers many settings, and Table 11 is a useful control.

The paper uses eight datasets, three regimes and fifteen baselines under one protocol, with per-horizon tables in Appendix A. The controlled Time-VLM branch removal in Table 11 is also helpful, since it shows that auxiliary views stop helping on ECL, Traffic and Exchange.

Critical weaknesses

[1] The experiments do not show that the gains come from frequency-selective multimodal fusion.

The most important missing result is a numerical-only FreqGate (same encoder and head, no text or image). On Traffic and Exchange the learned gates are almost closed in Table 5, yet FreqGate is still the best model on Traffic. Therefore, on these datasets the improvement must come from the numerical backbone or the band decomposition, not from fusion.

The ablations in Table 4 are also hard to reconcile with the spectral locality story. From the code, "w/o SFP" keeps the learned gates and only fixes the partition, yet Traffic MSE rises from 0.377 to 0.445, even though fusion is nearly off there. Meanwhile "w/o BGF", which fuses both modalities everywhere with gate 1, is clearly better than "w/o SFP". The "Fixed gates" variant, which is the hand-coded version of the paper's own principle, is the worst variant overall. I could not find an explanation for these results in the paper.

There is also a confound in how the views are built. In the code, both views are computed from the average over all channels and shared by every channel. For Traffic and ECL, with hundreds of channels, such views say little about any single channel, which by itself could explain why the gates close on exactly these datasets. The spectral labels in Table 5 are never measured, so the gate pattern may follow channel count rather than frequency content.

[2] Several headline claims are overstated or rest on non-comparable numbers.

The claim that 5% data leaves the average MSE "virtually unchanged" compares averages over different horizon sets, since the few-shot tables drop the longest horizons on several datasets (Table 8). For example, Exchange goes from 0.385 to 0.217, which is a horizon effect and not robustness. Thus, the few-shot conclusion is not supported as stated.

Some baseline numbers (for example Aurora on ETTh1 and DMMV on Exchange) look like failed runs, and the paper uses them to argue that multimodal augmentation without gating is insufficient. No standard deviations are reported despite three seeds, while the margin over VisionTS is small. The inference-time sparsity promoted in the abstract is also never measured.

[Smaller issues]

Close multimodal competitors (TimeCMA, T3Time) and frequency-domain models (FITS, FreTS, FilterNet) are not compared. The ablation variants are not defined in the paper text, and only L=96 is tested.

Questions for authors

Q1) What is the MSE of FreqGate with the text and image encoders removed, on all eight datasets, and what are the seed standard deviations for Tables 1 and 4? If the numerical-only model is within noise of the full model, the multimodal contribution is not established.

Q2) Why is "w/o SFP" so much worse on Traffic and ECL than "w/o BGF", given that the gates on these datasets are near zero? Please also define each ablation variant in the paper.

Q3) How do the gates and the Table 5 pattern change if the views are built per channel instead of from the channel average? A measured spectral descriptor per dataset, such as the energy fraction per band, would also tell me whether the gates follow frequency content or channel count.

Q4) Could you report full-shot and few-shot results on the same horizon subset, and add a direct test of spectral locality, for example forcing the high band gates open at test time? This would show whether the few-shot robustness and the frequency selectivity hold under a like for like comparison.

Reviewer_jgoXOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies an overlooked problem in multimodal time series forecasting: existing methods fuse auxiliary views (text/images) with a uniform strength over the entire sequence/entire spectrum regardless of what the auxiliary view actually describes, causing noise to be taken into account. The authors propose the principle of "spectral locality," fully exploiting the multi-scale advantages of multimodality (text → low frequency, images → mid frequency, numerical values → high frequency). Based on this, proposing FREQGATE. On 8 benchmarks, 15 baselines, and under full-shot / few-shot / zero-shot settings, it achieves the best average MSE/MAE, and with only 5% of the data the average MSE barely degrades (0.311 → 0.312).

Critical strengths
  1. The method is concise and elegant: SFP+BGF involve few components, are end-to-end trainable, and naturally support input-dependent sparse inference with skipping.
  2. The experiments are comprehensive and well-regulated: three training settings + 8 benchmarks + 15 baselines .
  3. Strong few-shot robustness: performance barely degrades with 5% of the data, which is a practically meaningful finding.
Critical weaknesses
  1. Limited relative gains: the average MSE improvement over the strongest baseline, VisionTS, is only 2.5%, and it does not win on 2 out of 8 datasets.
  2. All views are self-derived: the method still cannot address the "external modality" scenario.
Questions for authors
  1. The text/image views currently both come from the input itself. If they are replaced with external modalities (e.g., external descriptions, sensor images), how will the method perform?
  2. On long sequences (L > 96) or data whose spectral characteristics are unlike ETT, is the offset network in SFP stable?
Reviewer_B7apOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

FreqGate proposes "spectral locality": auxiliary text and image views, which are deterministically derived from the input series, carry predictive content mainly at low and mid frequencies. Soft Frequency Partitioning learns per-series soft low/mid/high band masks, applied to the numerical tokens. Band-Gated Fusion maps band energies to per-band gates for frozen ViLT text and image encoders, with band-wise cross-attention. The paper reports SOTA over 15 baselines in full-shot, few-shot, and zero-shot settings at L = 96.

Critical strengths

The question is reasonable and clearly framed: should auxiliary-view fusion strength depend on frequency? The paper gives an intuitive story for why templated text summaries mainly describe low-frequency structure.

Critical weaknesses
  1. "From full-shot to 5%, average MSE changes by only 0.001 (0.311 → 0.312)" compares quantities that are not comparable. The 5% averages drop the long horizons (720 on ETTh1, ETTh2, and Traffic; 336 and 720 on Exchange), which are the hardest. Exchange alone falls from 0.385 (full-shot, 4 horizons) to 0.217 (5%, 2 horizons). The conclusion that "gains stem from spectral gating rather than pretrained knowledge" is not supported.

  2. DMMV scores 3.679 MSE on Exchange (full-shot) and 5.809 (5%). Aurora scores 2.132 on ETTh1 and 1.695 on ETTm1. These are an order of magnitude worse than the published results and than trivial forecasters, which points to failed reproductions. Since the 15-baseline sweep is central to the evidence, every comparison involving these methods is unreliable.

  3. In Eq. (7), the band-k query attends to the unsplit H_T and H_V, and the attention output is added back without band-k filtering. The injected content is therefore not band-limited, and the text can still contribute high-frequency components to the "high" band view. The gates also depend only on the input spectrum p, not on any property of the auxiliary views. The very clean, monotone gate pattern in Table 5 is consistent with the gates simply learning dataset-level spectral statistics.

Questions for authors
  1. Please report the few-shot results restricted to the horizons shared with full-shot (e.g., H ∈ {96, 192, 336} only) for both regimes. Does the "no degradation" claim still hold?

  2. What exact commits and configurations were used to reproduce DMMV and Aurora, and can you reconcile your numbers with their published L = 96 results?

  3. If you band-pass filter the cross-attention output with A_k before adding it in Eq. (7), does performance change? This directly tests whether spectral locality, and not extra capacity, is responsible.

  4. Since both views are deterministic functions of X, how does FreqGate compare against an otherwise identical numerical-only model given the same parameter and FLOP budget (e.g., replacing the frozen ViLT branches with extra numerical layers)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.