- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes FreqGate, a multimodal forecaster that adds text and image views built from the input window (a statistics caption and a GAF, recurrence plot and line chart image, both encoded by frozen ViLT). The main idea, called spectral locality, is that these views mainly describe low and mid frequency content, so their fusion should be controlled per frequency band. Soft Frequency Partitioning learns a per-window soft split of FFT bins into three bands, and the band energies drive a sigmoid gate for each modality and band. Each band view of the numerical tokens then cross-attends to the text and image tokens, scaled by its gate. Experiments cover eight LTSF benchmarks at L=96 under full-shot, few-shot and zero-shot settings against fifteen baselines, with ablations and an analysis of the learned gate values.
Critical strengths
[1] The question is reasonable and the mechanism is simple.
Most tri-modal forecasters mix the auxiliary representation into the whole series at one strength, so asking whether a rendered image or a template caption is useful only for some frequencies is a fair hypothesis to test. The method is cheap (an FFT, a few small MLPs and K cross-attentions), and the released code follows the equations in Section 3 closely, which made it possible for me to check what the ablations actually do.
[2] The evaluation covers many settings, and Table 11 is a useful control.
The paper uses eight datasets, three regimes and fifteen baselines under one protocol, with per-horizon tables in Appendix A. The controlled Time-VLM branch removal in Table 11 is also helpful, since it shows that auxiliary views stop helping on ECL, Traffic and Exchange.
Critical weaknesses
[1] The experiments do not show that the gains come from frequency-selective multimodal fusion.
The most important missing result is a numerical-only FreqGate (same encoder and head, no text or image). On Traffic and Exchange the learned gates are almost closed in Table 5, yet FreqGate is still the best model on Traffic. Therefore, on these datasets the improvement must come from the numerical backbone or the band decomposition, not from fusion.
The ablations in Table 4 are also hard to reconcile with the spectral locality story. From the code, "w/o SFP" keeps the learned gates and only fixes the partition, yet Traffic MSE rises from 0.377 to 0.445, even though fusion is nearly off there. Meanwhile "w/o BGF", which fuses both modalities everywhere with gate 1, is clearly better than "w/o SFP". The "Fixed gates" variant, which is the hand-coded version of the paper's own principle, is the worst variant overall. I could not find an explanation for these results in the paper.
There is also a confound in how the views are built. In the code, both views are computed from the average over all channels and shared by every channel. For Traffic and ECL, with hundreds of channels, such views say little about any single channel, which by itself could explain why the gates close on exactly these datasets. The spectral labels in Table 5 are never measured, so the gate pattern may follow channel count rather than frequency content.
[2] Several headline claims are overstated or rest on non-comparable numbers.
The claim that 5% data leaves the average MSE "virtually unchanged" compares averages over different horizon sets, since the few-shot tables drop the longest horizons on several datasets (Table 8). For example, Exchange goes from 0.385 to 0.217, which is a horizon effect and not robustness. Thus, the few-shot conclusion is not supported as stated.
Some baseline numbers (for example Aurora on ETTh1 and DMMV on Exchange) look like failed runs, and the paper uses them to argue that multimodal augmentation without gating is insufficient. No standard deviations are reported despite three seeds, while the margin over VisionTS is small. The inference-time sparsity promoted in the abstract is also never measured.
[Smaller issues]
Close multimodal competitors (TimeCMA, T3Time) and frequency-domain models (FITS, FreTS, FilterNet) are not compared. The ablation variants are not defined in the paper text, and only L=96 is tested.
Questions for authors
Q1) What is the MSE of FreqGate with the text and image encoders removed, on all eight datasets, and what are the seed standard deviations for Tables 1 and 4? If the numerical-only model is within noise of the full model, the multimodal contribution is not established.
Q2) Why is "w/o SFP" so much worse on Traffic and ECL than "w/o BGF", given that the gates on these datasets are near zero? Please also define each ablation variant in the paper.
Q3) How do the gates and the Table 5 pattern change if the views are built per channel instead of from the channel average? A measured spectral descriptor per dataset, such as the energy fraction per band, would also tell me whether the gates follow frequency content or channel count.
Q4) Could you report full-shot and few-shot results on the same horizon subset, and add a direct test of spectral locality, for example forcing the high band gates open at test time? This would show whether the few-shot robustness and the frequency selectivity hold under a like for like comparison.