- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies forecasting when only monthly aggregates are available, and asks whether changing how each observation is represented can improve accuracy without adding information. It proposes flat monthly tokenization: each monthly average is repeated across k tokens (k = 90 in the main experiments), and each token carries known calendar features (day of week, month of year), a mask and scaling statistics. A two-layer GRU with about 174k parameters reads this sequence and outputs seven monotone quantiles. A short proposition shows that the dense representation contains no information beyond the monthly history and the calendar, so any gain must come from how the model computes, not from new data.
The authors build monthly tasks from two public daily datasets: Online Retail (2071 products) and UCI electricity (336 clients). This lets them compare against the same network trained on the daily series. Compared with a one-token-per-month GRU, the 90-token model lowers scaled CRPS by 20-49% on retail and 28-36% on electricity at horizons 1-3. It matches the daily-input reference on electricity and beats it on retail. The paper compares against six models trained from scratch (including N-HiTS, PatchTST and an LSTM) and six zero-shot foundation models.
Ablations are used to attribute the gain to three sources: more recurrent updates per observation, calendar features that change across tokens, and the flat input's invariance to how values are distributed within a month. A gate analysis argues that the dense GRU makes many small state corrections rather than integrating slowly. Splitting the score into median and non-median parts shows where the probabilistic improvement occurs.
Critical strengths
- Clear question with a principled framing. The paper separates observation frequency (how many measurements exist) from token density (how many sequence positions represent each measurement). Proposition 1 says clearly that the dense representation adds no information. Any improvement therefore has to come from how the model learns and computes. This framing is useful and could apply beyond this paper.
- Simple, cheap method. Flat tokenization needs no extra parameters, no external data and no model that guesses daily values. The 90-token GRU trains in 2-4 minutes on a laptop GPU. That makes it easy to adopt in practice, especially when only aggregated data exist.
- Informative experimental design. The authors build monthly tasks from daily datasets, so the same network can be trained on the daily series. This lets them measure directly how much accuracy aggregation costs and how much the representation recovers. The comparison is rarely available and is a good idea.
- Careful statistics and transparent reporting. Every comparison is paired across models, and intervals come from bootstraps that resample series, with a two-way version that also resamples seeds. The selection rule was fixed before the density experiments. Screening uses 3 seeds and confirmation uses 8. The paper also reports results that work against it: validation and test disagree, the foundation models win on median error, the best GRU is close to the naive forecast, and some ablations were run after the test evaluation. The core contrast between the 90-token and monthly models stays significant on both datasets even when seeds are resampled too.
- Thorough analysis of where the gain comes from. The ablations separate input map, density, varying versus fixed calendar, weekday and month of year channels, dropout, and how many recurrent steps run per month. Together with the gate measurements and the median/non-median split of the score, they go well beyond reporting one leaderboard number. The finding that the dense GRU makes many small state corrections, rather than integrating slowly, is a nice mechanistic result.
- Reproducibility. Code, the scored results behind every table, and regeneration scripts are provided. Datasets are public, training is deterministic given the seed, and compute is fully reported.
Critical weaknesses
- Claims go beyond the evidence. Token density is presented as a general modeling choice, but the gain appears only for recurrent models. PatchTST, iTransformer and DLinear at 90 tokens do worse than the GRU (Table 4). The claims should be limited to recurrent forecasters.
- Weak baseline. The 20-49% gain is measured against a monthly GRU that does poorly on retail. Its CRPS at horizons 1-3 is 0.884 / 1.369 / 1.888, versus 0.804 / 0.974 / 1.178 for the monthly LSTM and 0.716 / 0.795 / 0.924 for naive. Going to 90 tokens improves the LSTM by only 2-5%. The 90-token GRU is about level with naive at horizon 1 and worse at horizons 2-3.
- Test-set selection. GRU (best, ours) in Tables 4 and 5 is the best configuration on the test set. On electricity it is the one cycle model, not the validation selected one. On retail it is the 4-layer model, trained after the test evaluation. Comparisons with baselines should use only models selected on validation.
- Small evaluation. There are two constructed datasets with 2-3 test origins each, and validation and test rankings reverse (Fig. 7, App. B). The motivating cases (GDP, inflation) are never tested. Truly monthly benchmarks (M4, Tourism, FRED-MD) with rolling origins are needed.
- Limited novelty, missing related work and baselines. On retail, identical tokens cost at most 2% (Table 12), so the retail gain is largely Repeat-RNN. The paper omits mixed-frequency state-space and nowcasting models, MIDAS, and ODE-RNNs. These run a recurrence on a finer grid than the observations. Missing baselines: those models, Repeat-RNN reported as a baseline, and alternatives with matched compute (the 90-token model uses about 80x the forward FLOPs).
- Post-test analysis and caveats left out of the main text. Most ablations supporting the three-source explanation (6.4) were run after the test evaluation. Neither the abstract nor the main text mentions the validation/test reversal, the near-naive retail accuracy, or the foundation models' lower median error.
Questions for authors
- Does the density effect appear in any model that isn't recurrent, such as a Transformer or a state-space model with dense tokens rather than patches? If not, would you limit the claims to recurrent forecasters?
- Why does the monthly GRU do so much worse than the monthly LSTM and the naive forecast on retail? How large is the 90-token gain over the strongest monthly baseline you have, including naive?
- Can you report Tables 4 and 5 using only the configurations selected on validation? Which conclusions in §5.4 and §7 still hold?
- Can you add results on truly monthly or quarterly benchmarks (e.g., M4 monthly, Tourism, FRED-MD) with rolling-origin evaluation? Does the method work at other aggregation ratios, such as quarterly to monthly?
- On validation, accuracy does not improve steadily with density, and the monthly model is best at retail horizon 1. What explains the reversal on test, and how sensitive are the results to the choice of test origins?
- How does flat tokenization differ from mixed-frequency state-space models that run a recurrence on a finer grid than the observations? Can you compare against such a model, and against Repeat-RNN as a standalone baseline?
- Does the 90-token model beat a monthly model given the same compute, through extra depth, internal micro-steps or an ODE-RNN?
- Do the ablations run after the test evaluation reproduce on the validation set, so that the three-source explanation does not rest on test-time analysis alone?
2 comments
Dear Authors, thank you for the detailed response! LImiting the claims to recurrent models, adding Repeat-RNN and a wider GRU, adding validation results for each ablation, registering predictions before running the native benchmarks - all these make the paper clearer and more honest. However, the new results confirm my main concerns. On retail dataset, the 90-token model is worse than monthly GRU and one-token model. Several comparisons reverse with two-way significance (including the gain over the daily input model). On FRED-MD, M3 and Tourism, it is worse than monthly GRU on validation in every cell and on test for two of the three sets. Repeat-RNN without its flag matches the 90-token model. So the gain seems to come mainly from plain repetition and shorter memory, not from the dense calendar tokens. It helps when the earlier months mislead the model (which is hard to know in advance), and it hurts on real monthly data where history is informative. Retail accuracy is also still on par or behind the naive forecast and zero-shot foundation models. I think the paper is now a careful analysis of when repetition helps a recurrent forecaster but its contribution is narrower than the paper claims, so I will keep my score.
Thank you for the careful review. Your questions led to further monthly baselines, a validation result for every ablation, and three native monthly benchmarks. On test, the 90-token GRU outperforms the monthly GRU on both datasets and all six trained baselines on retail, and the native sets mark where it stops helping.