- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes PatchBasis, a tokenizer for patch-based long-term forecasting that compresses M input patches into K tokens. Each channel's history is split into M patches, which are linearly embedded, given sinusoidal positional encodings and passed through a residual convolution. A shared two-layer MLP (BasisLearner) scores every patch for each of the K output slots. A softmax over patches turns these scores into an input-dependent aggregation map, so each token is a convex combination of patch embeddings plus a learned slot embedding. A "Time-Decay Prior" then mixes slot k with the k-th most recent patch using the logistic weight. The K tokens are fed to a channel-independent Transformer trained with an MAE loss and RevIN. The authors claim three contributions, i.e. a view of tokenization as instance-adaptive selective aggregation of non-contiguous history, an architecture that combines this aggregation with preservation of recent patches, and an empirical study covering accuracy, ablations, efficiency and sensitivity.
Critical strengths
The paper reports its results transparently. It includes results that go against its own method. The plug-in hurts two of the four backbones, the input-dependent variant gives no gain, and inference is slower. It also discloses protocol exceptions that many papers would leave out. This matters because it makes the work easy to audit, and the authors' own diagnostic test in App. F is the right tool for answering my main concern.
The empirical study is broad and yields a useful finding. The method is compared against several tokenizers across three backbone settings and nine datasets. The analysis shows what a learned pooling tokenizer actually converges to, namely recent patches plus daily patterns. This matters because it shows the module is a workable tokenizer and gives the community a clear picture of what such tokenizers learn, even though it does not yet support the specific mechanism the paper claims.
Critical weaknesses
The paper's main claim is not supported, and the mechanism is not new. The paper attributes its gains to aggregation weights that adapt to each input. However, it never tests this against fixed weights in the same pipeline. What it shows about the learned weights is mostly a preference for recent patches plus daily repetition. The aggregation step is also an existing, uncited technique. This matters because it is the paper's main contribution, and without it the novelty is too limited for ICLR
The evidence does not reliably support the reported gains. There are three problem, i.e. (1) the released code implements a different version of the method from the one the paper describes. (2) strong baselines in Table 2 perform far below their published results. (3)main results come from a single run, with margins smaller than the variation shown in the paper's own repeated runs.
Questions for authors
The paper describes a fixed recency budget, but the released code predicts it from each input. Which version produced the main results? What values of K, rho and tau_a were used for each dataset?
In Table 2, how were the shared training settings chosen, and were they tuned for PatchBasis? Several strong baselines score well below their published results (for example, iTransformer on Electricity). Did the baselines use the same loss and checkpoint rule as PatchBasis?
Do the three-seed runs in Table 10 use the same configurations as the main results in Table 11? They are worse by up to 0.015 MSE, which is enough to reverse the comparisons on ETTm1 and ETTh2.
Can you compare against the same pipeline with fixed (input-independent) aggregation weights? Can you also apply your own/mean/swap test to the aggregation weights? If input-specific weights clearly help, the central claim would be much better supported.
How does PatchBasis compare with simply using the most recent K patches, with no aggregation?
3 comments
Response to Reviewer jrtX (2/3)
[Q1] […] Which version produced the main results? What values of K, rho and tau_a were used for each dataset?
Response: The main results use a fixed recency budget. Dynamic budgeting was our initial design. It did not consistently outperform a fixed budget, so we simplified the model and reran the experiments. The released repository contained the fixed-budget implementation, but it was not the default. This was an upload oversight, and we apologize for the confusion.
The repository is now updated. The fixed-budget implementation is at line 687 of layers/PatchBasisLayers.py in the current version. Dynamic budgeting remains a viable alternative: it removes the need to select a fixed-budget hyperparameter. The two variants have similar overall performance, with dataset-dependent trade-offs.
The temperature τₐ is 1 in every experiment and was not tuned. We have removed this parameter from the manuscript. We previously released the shared settings for manuscript Table 12 but not all model-specific values. The table below provides K and ρ for the listed tasks; we will also release them with the configurations.
Task-specific token counts and recency parameters. K is unchanged across the four horizons within each listed dataset; τₐ = 1 throughout.
| Dataset | K | ρ at H = 96 | ρ at H = 192 | ρ at H = 336 | ρ at H = 720 |
|---|---|---|---|---|---|
| ETTh1 | 16 | 10.000 | 10.000 | 1.000 | 2.500 |
| ETTh2 | 8 | 3.500 | 3.500 | 1.000 | 1.000 |
| ETTm1 | 8 | 5.000 | 4.000 | 1.000 | 1.000 |
| ETTm2 | 4 | 3.000 | 3.000 | 3.000 | 3.000 |
| Weather | 8 | 3.500 | 2.500 | 1.500 | 1.500 |
| Electricity | 2 | 0.750 | 0.750 | 0.250 | 0.500 |
| Traffic | 2 | 0.500 | 0.500 | 0.250 | 0.250 |
| Solar-Energy | 2 | 0.500 | 0.500 | 0.150 | 0.150 |
| Exchange | 16 | 3.000 | 6.000 | 3.000 | 3.000 |
The baselines retain their published best model-specific settings. We select ρ as a model-specific hyperparameter using validation MSE. We do not view replacing a learned budget with a selected fixed budget as an attempt to favor PatchBasis. The dynamic alternative avoids this hyperparameter and gives comparable overall performance. We used the fixed variant in the main text for simplicity.
[Q2] In Table 2, how were the shared training settings chosen, and were they tuned for PatchBasis? […] Did the baselines use the same loss and checkpoint rule as PatchBasis?
Response: The shared configurations were selected using PatchBasis’s average validation MSE, which can favor PatchBasis. The baselines nevertheless use the same training and evaluation procedure, including MAE loss and validation-MAE checkpoint selection. We distinguish this shared-configuration selection bias from unequal treatment during training.
We checked and reran baselines with large discrepancies from their published results and released their checkpoints in the anonymous repository. The Electricity issues for TimePerceiver, SEER, and iTransformer were associated with the learning rate of 0.010. We changed the learning rate to 0.001 for all models on Electricity. This correction is documented in the revised experimental settings.
[Q3] Do the three-seed runs in Table 10 use the same configurations as the main results in Table 11? […]
Response: The two tables use different configurations. Manuscript Table 10 is a recent-candidate ablation. Manuscript Table 11 reports results after an extensive hyperparameter search. Table 10 was not a repeated-seed evaluation of Table 11’s best configurations. The revised manuscript states: “Recent-candidate ablation. Figure 4 and Tables 10 and 3 use L = 512 and H = 192 on five datasets.” Appendix A.3 lists the input-length search range: “including input lengths L ∈ {96, 192, 320, 512}.”
For example, the relevant ETTm1 main-table configuration uses L = 320, whereas the ablation uses L = 512. The ablation also isolates allocator initialization from the random-number stream of shared modules. This keeps shared-module initialization comparable across Recent, Uniform, and no Recent, but differs from the main experiments. The discrepancy cannot be attributed entirely to seed variation. Table 10 alone does not establish the stability of the main-table configurations. We have now separately rerun the main comparisons with seeds 2024, 2025, and 2026.
Response to Reviewer jrtX (3/3)
[Q4] Can you compare against the same pipeline with fixed (input-independent) aggregation weights? […] apply your own/mean/swap test to the aggregation weights? […]
Response: We provide both a shared-weight comparator and direct interventions on the learned aggregation matrix. Section 3.1 already describes TimeBase as input-independent: “The aggregation weights remain shared across inputs:”
Manuscript Table 4 includes this comparator. Its coverage of APE, TOTEM, and other native tokenizers requires method-specific representations. We therefore also report the earlier shared lightweight pipeline below.
The lightweight comparison uses TimeBase-MLP-SameRank-PM and BasisLearner (Instance-TL + Position). It uses six datasets, L = 720, H = 336, and three seeds. K, depth, GELU, dropout, and the forecasting backbone are matched. The maximum relative difference in total parameter count is 1.153%. Each method selects its learning rate by validation MSE at patch size 24, then reuses it for the other patch sizes.
Shared lightweight pipeline; L = 720, H = 336, patch size 24. Three-seed mean test MAE.
| Dataset | TimeBase MLP | BasisLearner |
|---|---|---|
| ETTh1 | 0.428 | 0.421 |
| ETTh2 | 0.418 | 0.410 |
| ETTm1 | 0.411 | 0.386 |
| ETTm2 | 0.363 | 0.332 |
| Weather | 0.340 | 0.299 |
| Electricity | 0.280 | 0.259 |
BasisLearner achieves lower MAE on all six datasets at patch size 24 and on all 16 evaluated dataset–patch combinations. Its scoring structure and weight constraints still differ from TimeBase MLP. We therefore do not describe this as changing only input dependence in the complete PatchBasis pipeline.
For the matrix intervention, we use the trained BasisLearner without Recent and do not retrain it. Own uses the current window’s weights. Mean averages weights over training windows while preserving channels and output slots. Swap takes weights from another test window while leaving the current input unchanged.
Aggregation-matrix interventions without retraining. Test MSE / MAE.
| Weight source | ETTh1 | ETTm1 | Weather |
|---|---|---|---|
| Own | 0.370 / 0.384 | 0.293 / 0.332 | 0.158 / 0.194 |
| Mean | 0.380 / 0.395 | 0.328 / 0.355 | 0.165 / 0.207 |
| Swap | 0.392 / 0.403 | 0.348 / 0.367 | 0.167 / 0.210 |
Mean and Swap worsen both metrics on all three datasets. The trained model therefore uses the match between the input and its aggregation weights. This tests A(x), not the recency-budget diagnostics in Appendix F. It supports input matching, but does not rule out a separately retrained static model achieving similar performance.
[Q5] How does PatchBasis compare with simply using the most recent K patches, with no aggregation?
Response: The full model outperforms using only the most recent K patches in all three tested settings. Recent-only retains the latest K patches after the same preprocessing and LocalContext, then uses the same Transformer and prediction head without aggregation. K, the remaining architecture, and the tuning budget are matched within each dataset.
Recent-only and branch ablations; L = 192, H = 96. Three-seed mean test MSE / MAE.
| Method | ETTh1 | ETTm1 | Weather |
|---|---|---|---|
| Recent-only | 0.378 / 0.390 | 0.303 / 0.336 | 0.162 / 0.198 |
| BasisLearner without Recent | 0.370 / 0.384 | 0.293 / 0.332 | 0.158 / 0.194 |
| Full model with Recent | 0.366 / 0.382 | 0.291 / 0.331 | 0.155 / 0.190 |
At L = 192 and H = 96, the full model with Recent model reduces mean MSE by 3.110%, 3.790%, and 4.510% relative to recent-only on ETTh1, ETTm1, and Weather. BasisLearner without Recent also improves both metrics on all three datasets. Historical aggregation therefore contributes beyond simply keeping recent patches.
We hope these responses address your concerns, and we would be happy to provide further clarification during the discussion period.
Response to Reviewer jrtX (1/3)
Thank you for your careful review and constructive comments.
Response: The manuscript already compares against input-independent aggregation through TimeBase. Section 3.1 explicitly identifies its shared weights, and manuscript Table 4 compares TimeBase and BasisLearner under the same downstream backbone within each setting. It repeats this across multiple backbone settings. We therefore disagree that no fixed-weight comparator was tested. The tokenizer representations still differ, as discussed in Q4.
Our contribution concerns how to construct a useful forecasting token basis, why it should adapt to the input, and how recent information can restore details lost through aggregation. The existence of pooling operators does not by itself resolve these questions. Related Work already distinguishes our design from other aggregation schemes.
TokenLearner is the closest operator-level counterpart in our comparison. It targets visual representations and reduces tokens within a ViT stack. PatchBasis builds its compact historical representation before the forecasting backbone and explicitly supplements it with recent patches. Similar operators therefore need to be assessed in the context of their architecture, objective, and evidence.
The introduction defines the forecasting objective: “We explore tokenization from the perspective of instance-adaptive selective aggregation. It reorganizes forecast-relevant information across non-contiguous historical patches into a compact token basis.” Section 3.2 explains how the token basis changes with the input: “Recomputing A from each input lets the same token slot draw on different historical regions across instances.” The Recent branch follows from the observed aggregation weights. Section 3.3 states: “BasisLearner assigns high weights to recent patches in some tokens without an explicit prior (Figure 2).” “Recent patches provide a starting point for forecasting. Preserving them also supplements features lost during aggregation.” The manuscript explicitly combines these components within one budget: “BasisLearner and the Time-Decay Prior jointly retain historical patterns and recent details within a fixed token budget.”
The shared-pipeline comparison, own/mean/swap intervention, and recent-only control in Q4 and Q5 go beyond visual inspection of recency and daily patterns. They test the roles of input matching and historical aggregation directly.
We have added a direct TokenLearner discussion and citation to Section 2, clarifying these forecasting-specific differences.
Response: The main experiments use the fixed-budget model, and we have corrected the repository default and rerun the main comparisons with three seeds. Q1 identifies the implementation and hyperparameters. Q2 explains the baseline settings and Electricity learning-rate correction. Q3 explains the earlier configuration mismatch. The updated main results are no longer based on a single run.