- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a useful trade-off, that is restricting a forecasting model can remove predictive information but make learning easier. It derives restriction costs in several simplified settings, measures data requirements using linear reference predictors, and combines these measurements with architecture descriptions to recommend forecasters before training them.
Critical strengths
-
Clear conceptual framework. Separating the best performance a model class can attain from its estimation and optimization costs helps explain why more expressive models do not always perform better.
-
Mathematically sound, carefully scoped results. The main propositions appear correct under their stated assumptions. The paper distinguishes exact identities, measured reference contrasts, and heuristic score components.
-
In experiments average top-three regret is 5.9%, versus 6.8% for ZiCo, and the authors report substantial ablations and failure cases.
Critical weaknesses
-
The full score offers little improvement over simpler rules on the main metric. For example, top-three regret is 5.9% for the full score, 6.0% without its learning-cost term, and 5.9% for a simple external-sign rule. This is not a significant improvement. Although the full score improves ranking correlation and token-layout predictions, its added complexity is not clearly justified by the primary recommendation metric.
-
The central finite-sample tradeoff is not directly tested. The paper changes N when re-computing scores, but does not report learning curves obtained by retraining the same architecture pairs with different amounts of data. Such an experiment should test whether restricted models win with little data and more expressive models become preferable as data increases.
-
The main recommendation benchmark covers a limited forecasting setting. The ten-model recommendation study is concentrated at L=H=96. Longer histories and horizons are examined in controlled experiments, but the full architecture-ranking benchmark is not repeated across these settings. Recommendation quality at horizons such as 192, 336, and 720 therefore remains unclear.
Questions for authors
See weakness. I might adjust my score given the reply of the authors.
5 comments
Response to Reviewer Pxgv (3/4): learning curves
[W2] Learning curves
Such an experiment should test whether restricted models win with little data and more expressive models become preferable as data increases.
Thank you for this suggestion. We ran the experiment, and the switch you describe appears where the lattice predicts a moderate gain. We retrained the eight controlled pairs on 5–100% of the training rows of 21 datasets, with a new search at each share, and re-measured the lattice on the rows used (Section 6.2, Figure 3d, Appendix D.1). The table gives the median test gain of the relaxed child over the restricted parent, with pair–dataset cells split into thirds by predicted gain (defined in Appendix C.2):
| Cells by predicted gain | 5% of rows | 100% of rows |
|---|---|---|
| Top third | +3.9% | +10.3% |
| Middle third | −1.5% | +2.5% |
| Bottom third | −3.5% | −0.1% |
In the middle third, the restricted parent wins with little data and the relaxed child with more. The child wins 34% of these cells with 5% of the rows and 71% with all of them. In the top third, relaxing already helps with 5% of the rows and helps more with all of them. In the bottom third, keeping the restriction is as good or better at every size.
Re-measured on the rows used, the lattice tracks these gains at every size: its Spearman correlation with the test gain rises from 0.43 at 5% of the rows to 0.65 at 100% (Table 13).
Response to Reviewer Pxgv (4/4): other input lengths and horizons
[W3] Other input lengths and horizons
Recommendation quality at horizons such as 192, 336, and 720 therefore remains unclear.
We repeated the full benchmark at these three horizons and at a longer input, on 20 datasets (18 at H = 720). In each setting, we re-tuned every forecaster, re-measured the lattice, and trained the four record-based selectors on that setting's own results. The probes are reused from L = H = 96 (Appendix D.2, Table 18):
| Setting | Top-three regret (%), score | Top-three regret (%), selectors | Spearman , score | Spearman , selectors |
|---|---|---|---|---|
| H = 192 (L = 96) | 5.3 | 6.5–7.7 | 0.45 | 0.34–0.48 |
| H = 336 (L = 96) | 7.6 | 9.7–10.5 | 0.46 | 0.27–0.38 |
| H = 720 (L = 96) | 10.4 | 15.1–17.4 | 0.46 | 0.16–0.28 |
| L = 336 (H = 96) | 4.7 | 4.6–5.1 | 0.26 | 0.39–0.44 |
Bold: the score is best on that measure; selectors: range over the four.
At every longer horizon, the score keeps the lowest top-three regret, significantly lower than that of two of the four selectors, and at H = 720 its Spearman correlation is significantly higher than every selector's (Holm-adjusted). At L = 336, its regret is similar, but every selector's Spearman correlation is significantly higher.
Thank you again for your review, and especially for the learning-curve suggestion, which now tests the paper's central trade-off directly. We would be glad to answer any further questions.
Summary of the revision (1/4)
Thank you for the constructive review, and for finding the framework clear and the propositions sound under their assumptions. Each of your points led to a new experiment or analysis in the revised PDF, now uploaded. Below we summarize the revision, and the next three comments answer your points on the full score and simpler rules (2/4), learning curves (3/4), and other input lengths and horizons (4/4).
The revision takes up your three points: whether the full score earns its complexity over simpler rules, learning curves that test the trade-off, and recommendation quality at longer horizons.
Following the other reviews, we also grew the benchmark from 10 forecasters on 14 datasets to 17 published forecasters on 25 datasets, each searched over twelve configurations and trained with three seeds (Section 6.1). Correlations and comparisons now have 95% source-family bootstrap intervals and Holm-adjusted p-values (Section 6.1, Appendix C.5), and the code and numerical results are in an anonymous repository.
In the revised experiments, lattice gains rank the test gains of single structural changes (Spearman 0.52 to 0.88 over eight controlled pairs; Section 6.2). With a few minutes of probes and no results from other datasets, the score ranks forecasters at least as well as selectors that learn from them (Section 6.3). A short run on its top three selects a forecaster whose regret after its full search is 3.3%, the lowest among rankings of similar selection cost (Section 6.4).
We removed the distribution-prediction extension and the per-core synthetic experiments, and renumbered the propositions (Appendices A.5, A.7). In the PDF, changes for your points are purple, with your reviewer ID as the margin tag, and changes shared across the reviews are green with the tag GR.