- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PatchVQFormer. Each instance-normalized patch is split by fixed FFT masks into M orthogonal frequency bands; band boundaries come from a heuristic energy/autocorrelation score on training patches. For each band a k-means codebook of time-domain waveform prototypes is fitted offline and frozen. Historical patches enter the model only through prototype-indexed embeddings (summed across bands); a Transformer attends across variables and then across patches, with future query tokens built from positional and calendar information. Independent heads output a categorical distribution per band, and the forecast is the softmax-weighted expectation of the frozen prototypes, summed over bands and overlap-added. Training combines cross-entropy on hard target codes with a sequence reconstruction loss. Results are reported on eight benchmarks and four horizons against seven continuous baselines (values taken from the literature at differing input lengths), with ETT ablations of the Fourier split, the codebook and patching, a band-count study on ETTm2 and a capacity study on one ETTh2 setting. Short analytical statements (additive distortion across orthogonal bands, a nearest-prototype margin condition, monotone distortion in capacity) accompany the method.
Critical strengths
-
An interesting representational idea. Factorizing a patch into orthogonal bands so that a small set of per-band prototypes yields a combinatorially large vocabulary, with exactly additive distortion across bands, is a neat way to give discrete time-series tokens compositional structure. If validated properly, it would be a useful design point between whole-patch codes (TOTEM-style) and coefficient tokens (WaveToken-style).
-
Unusual frankness in the appendix. Appendix A.2 and B list the limits of the theory, the absence of multi-seed, cost and robustness measurements, and even code paths that conflict with the stated protocol. This transparency is what makes the paper assessable, and it would make a corrected resubmission easy to trust.
Critical weaknesses
-
The main text and the appendix describe materially different methods and protocols, and the appendix discloses test-informed selection. I cannot tell which system produced Table 1.
- Capacity selection. Abstract, Sections 3 and 4.1: band capacities are "selected on validation data". Appendix B.3: the main runs "do not sweep … on validation data"; capacities are script defaults or an "auto" rule, and they differ per dataset–horizon cell (Table 4).
- History encoding. Section 4.1: one hard nearest-prototype index per band. Appendix B.5: a distance-weighted Top-K mixture of embeddings (K = 4, or 8 / 2 for "selected ETTh1 cells"). The stability result of Section 4.4 concerns the hard assignment that is not what is implemented, as the appendix itself notes.
- Attention mask. Section 4.2: future queries attend to history and to all future queries. Appendix B.5: a future query attends to history and to itself only.
- Objective. Section 4.3: L = L_CE + λ·L_seq with λ = 1. Appendix Eq. 9: cross-entropy + 2·(1 − cosine) + 8·MSE + 2·first-difference MSE, with per-cell overrides of the weights.
- Temperature. Section 4.3: fixed, shared, "used unchanged in training and inference". Appendix: temperature 1 in training and a per-cell "test-time" temperature between 0.9 and 1.3 (Table 4), with no statement of how it was chosen.
- Embeddings. Main text: trainable. Appendix B.4: frozen by default, trainable in three ETTm2 cells.
- Model selection. Appendix B.2 states that the repository "contains a baseline-selection script that chooses the best input length by test MSE, and some configuration comments record test-informed tuning". The model's own input length varies over {96, 336, 512, 720} per cell (the L = 336 ablation gives ETTh1 average MSE 0.427, whereas the Table 1 ETTh1 rows average 0.404), patch length, band count, capacities, temperature, loss weights, batch size and width are all set per cell, there is one seed, and the test loader drops the last incomplete batch. The non-monotone errors in Table 1 (ETTh1 and ETTh2 are better at H = 336 than at H = 192) are consistent with cell-specific tuning. Under these conditions the reported numbers cannot be treated as held-out estimates.
-
Even taken at face value, the results do not support the headline claims, and the central "fully discrete, no continuous decoder" claim is not tested.
- Averaging the 32 rows of Table 1, PatchTST (2023) obtains 0.289 / 0.340 MSE/MAE, better than PatchVQFormer's 0.291 / 0.342 on both metrics; WPMixer has a better MAE (0.336). The abstract quotes PatchVQFormer's average without any baseline average. The method is the worst listed on Exchange at H = 96 and H = 720 and trails clearly on Weather and Electricity at long horizons; the "first-place count" is carried by ETT and Solar.
- Baseline input lengths differ ("a broad literature comparison", in the authors' words), so the ETT long-horizon improvements of 1.0–1.8% are not controlled.
- The paper is positioned against discrete forecasters (TOTEM, WaveToken, ReCast, Sparse-VQ, VQ-AR, Chronos) but compares with none of them. The question it poses, whether a codebook-only output path can match discrete alternatives, is therefore unanswered.
- The output is a softmax-weighted linear combination of fixed vectors trained with a heavily weighted MSE term, and the input is a soft Top-K mixture. Functionally this is continuous regression onto a fixed dictionary, not discrete prediction. The appendix says the evaluation code records hard-argmax decoding and oracle-code reconstruction, but neither is reported; these are the numbers that would show whether the discrete vocabulary carries the forecast.
- Removing the codebook costs 3.4% MSE on ETT and improves ETTh1 MAE; removing patching improves ETTm1; with one seed these effects are not distinguishable from noise. Several cells use P = H (e.g. 720), where the "patch" is the whole horizon and the compositional-reuse argument does not apply.
- The analysis in Section 4.4 is the Pythagorean identity, a triangle-inequality margin and nestedness of feasible sets; it does not bear on forecasting accuracy.
Questions for authors
Clarifications that could change my assessment:
- Which description is correct for the Table 1 runs: Section 4 or Appendix B (hard vs. Top-K history, future-to-future attention, loss, temperature, frozen vs. trainable embeddings, capacity selection)? Please make the paper describe one system.
- How was the test-time temperature in Table 4 chosen for each cell? How was the input length chosen for each cell? Was any quantity in Table 1 selected using test metrics?
- Please report a fully held-out protocol: one input length for all methods, hyperparameters selected on validation only, ≥3 seeds, no dropped test batches, with baselines re-run under it.
- Please report hard-argmax decoding and oracle-code reconstruction error next to the soft-expectation result for every dataset.
- Please compare with TOTEM, WaveToken and ReCast under the same protocol.
- In "w/o Codebook", what replaces the codebook input and output paths?
Suggestion for future work (does not affect my rating): modeling the joint distribution over band codes and sampling from it would make the discrete formulation pay off for probabilistic forecasting.