acceptodds
Under review as a conference paper at ICLR 2027

Pricing Structural Restrictions in Time-Series Forecasting

Abstract

The same architectural choice can improve some time-series predictions and worsen others. We examine this variation by comparing the predictive cost of structural restrictions with the learning cost they can save. Here, learning cost is the prediction loss above a structure's best attainable performance due to imperfect estimation and optimization with limited data and resources. In linear prediction settings, we express structural restriction costs through properties of the data distribution, termed data requirements. An orthogonal-design ridge comparison then identifies when estimation savings outweigh the cost of restricting inputs. These results motivate held-out comparisons of restricted linear reference predictors to estimate selected data requirements. In controlled experiments, larger measured requirements tend to accompany larger gains from relaxing the corresponding restriction. Combining these measurements with candidate configurations and a finite-sample approximation to the estimation component of learning cost yields recommendations without training candidates or using performance results from other datasets. Across ten forecasters and 14 datasets, the three recommended candidates have a mean excess test error of 5.9% over the best candidate overall, comparable to the strongest training-free proxy (6.8%) and selectors trained on other datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, inductive bias, structural restrictions, data requirements, training-free model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PxgvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a useful trade-off, that is restricting a forecasting model can remove predictive information but make learning easier. It derives restriction costs in several simplified settings, measures data requirements using linear reference predictors, and combines these measurements with architecture descriptions to recommend forecasters before training them.

Critical strengths
  1. Clear conceptual framework. Separating the best performance a model class can attain from its estimation and optimization costs helps explain why more expressive models do not always perform better.

  2. Mathematically sound, carefully scoped results. The main propositions appear correct under their stated assumptions. The paper distinguishes exact identities, measured reference contrasts, and heuristic score components.

  3. In experiments average top-three regret is 5.9%, versus 6.8% for ZiCo, and the authors report substantial ablations and failure cases.

Critical weaknesses
  1. The full score offers little improvement over simpler rules on the main metric. For example, top-three regret is 5.9% for the full score, 6.0% without its learning-cost term, and 5.9% for a simple external-sign rule. This is not a significant improvement. Although the full score improves ranking correlation and token-layout predictions, its added complexity is not clearly justified by the primary recommendation metric.

  2. The central finite-sample tradeoff is not directly tested. The paper changes N when re-computing scores, but does not report learning curves obtained by retraining the same architecture pairs with different amounts of data. Such an experiment should test whether restricted models win with little data and more expressive models become preferable as data increases.

  3. The main recommendation benchmark covers a limited forecasting setting. The ten-model recommendation study is concentrated at L=H=96. Longer histories and horizons are examined in controlled experiments, but the full architecture-ranking benchmark is not repeated across these settings. Recommendation quality at horizons such as 192, 336, and 720 therefore remains unclear.

Questions for authors

See weakness. I might adjust my score given the reply of the authors.

5 comments
AuthorsOpenReview ↗

Summary of the revision (1/4)

Thank you for the constructive review, and for finding the framework clear and the propositions sound under their assumptions. Each of your points led to a new experiment or analysis in the revised PDF, now uploaded. Below we summarize the revision, and the next three comments answer your points on the full score and simpler rules (2/4), learning curves (3/4), and other input lengths and horizons (4/4).

The revision takes up your three points: whether the full score earns its complexity over simpler rules, learning curves that test the trade-off, and recommendation quality at longer horizons.

  • A simpler score. The score now uses two measurements, both taken before any forecaster is trained (Sections 4–5). On the data side, we compare a ridge reference that relaxes each set of restrictions with the fully restricted reference on held-out rows of the training split. The difference estimates the net gain of relaxing that set, and the gains of all 32 sets form the gain lattice. On the forecaster side, probes at initialization check whether a forecaster's design can do what each restriction forbids, for example whether it can use other channels. A forecaster's score is set by the largest lattice gain among the sets it relaxes (Eq. 5), with no fitted combination weights and no separate learning-cost term. We derived these changes from the theory, checked the score on the tuning errors of the ten submitted forecasters, and fixed it before running the seven added forecasters and eleven added datasets.
  • Simpler rules. We compare simpler versions of the score, including revised counterparts of the two rules you cite, before and after training (Appendices D.2–D.3).
  • Learning curves. We retrained the eight controlled pairs on 5–100% of the training rows and re-measured the lattice on the rows used (Section 6.2, Appendix D.1).
  • Longer horizons. We repeated the benchmark at H = 192, 336, and 720 and at a longer input (Appendix D.2).

Following the other reviews, we also grew the benchmark from 10 forecasters on 14 datasets to 17 published forecasters on 25 datasets, each searched over twelve configurations and trained with three seeds (Section 6.1). Correlations and comparisons now have 95% source-family bootstrap intervals and Holm-adjusted p-values (Section 6.1, Appendix C.5), and the code and numerical results are in an anonymous repository.

In the revised experiments, lattice gains rank the test gains of single structural changes (Spearman 0.52 to 0.88 over eight controlled pairs; Section 6.2). With a few minutes of probes and no results from other datasets, the score ranks forecasters at least as well as selectors that learn from them (Section 6.3). A short run on its top three selects a forecaster whose regret after its full search is 3.3%, the lowest among rankings of similar selection cost (Section 6.4).

We removed the distribution-prediction extension and the per-core synthetic experiments, and renumbered the propositions (Appendices A.5, A.7). In the PDF, changes for your points are purple, with your reviewer ID as the margin tag, and changes shared across the reviews are green with the tag GR.

AuthorsOpenReview ↗

Response to Reviewer Pxgv (3/4): learning curves

[W2] Learning curves

Such an experiment should test whether restricted models win with little data and more expressive models become preferable as data increases.

Thank you for this suggestion. We ran the experiment, and the switch you describe appears where the lattice predicts a moderate gain. We retrained the eight controlled pairs on 5–100% of the training rows of 21 datasets, with a new search at each share, and re-measured the lattice on the rows used (Section 6.2, Figure 3d, Appendix D.1). The table gives the median test gain of the relaxed child over the restricted parent, with pair–dataset cells split into thirds by predicted gain (defined in Appendix C.2):

Cells by predicted gain5% of rows100% of rows
Top third+3.9%+10.3%
Middle third−1.5%+2.5%
Bottom third−3.5%−0.1%

In the middle third, the restricted parent wins with little data and the relaxed child with more. The child wins 34% of these cells with 5% of the rows and 71% with all of them. In the top third, relaxing already helps with 5% of the rows and helps more with all of them. In the bottom third, keeping the restriction is as good or better at every size.

Re-measured on the rows used, the lattice tracks these gains at every size: its Spearman correlation with the test gain rises from 0.43 at 5% of the rows to 0.65 at 100% (Table 13).

AuthorsOpenReview ↗

Response to Reviewer Pxgv (4/4): other input lengths and horizons

[W3] Other input lengths and horizons

Recommendation quality at horizons such as 192, 336, and 720 therefore remains unclear.

We repeated the full benchmark at these three horizons and at a longer input, on 20 datasets (18 at H = 720). In each setting, we re-tuned every forecaster, re-measured the lattice, and trained the four record-based selectors on that setting's own results. The probes are reused from L = H = 96 (Appendix D.2, Table 18):

SettingTop-three regret (%), scoreTop-three regret (%), selectorsSpearman ρ\rho, scoreSpearman ρ\rho, selectors
H = 192 (L = 96)5.36.5–7.70.450.34–0.48
H = 336 (L = 96)7.69.7–10.50.460.27–0.38
H = 720 (L = 96)10.415.1–17.40.460.16–0.28
L = 336 (H = 96)4.74.6–5.10.260.39–0.44

Bold: the score is best on that measure; selectors: range over the four.

At every longer horizon, the score keeps the lowest top-three regret, significantly lower than that of two of the four selectors, and at H = 720 its Spearman correlation is significantly higher than every selector's (Holm-adjusted). At L = 336, its regret is similar, but every selector's Spearman correlation is significantly higher.

Thank you again for your review, and especially for the learning-curve suggestion, which now tests the paper's central trade-off directly. We would be glad to answer any further questions.

Reviewer_HrSxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper investigates the problem of matching model architectures to data characteristics in time-series forecasting. To this end, the authors propose a quantitative formulation of prediction risk that characterizes both the attainable upper bound on model performance and the practical upper bound imposed by the modeling process. Furthermore, the authors propose using simple proxy models to identify suitable architectural structures for different datasets. Finally, experimental results are presented to demonstrate the effectiveness of the proposed model selection strategy.

Critical strengths

The problem of matching model architectures to data characteristics is indeed an important research topic in time-series forecasting. The authors provide a theoretical analysis of several architectural constraints, including cross-variable information access, parameter sharing, historical lookback window length, and low-dimensional representations. Under specific linear forecasting settings, they derive corresponding risk expressions, offering a relatively unified and interpretable perspective on the relationship between model architecture and forecasting performance. Building on these theoretical insights, the authors propose an efficient model selection method for time-series forecasting and provide empirical evaluations of its effectiveness.

Critical weaknesses
  1. Limited novelty relative to existing work. The research problem itself is not particularly novel. Prior work, such as TimeRecipe and related studies, has also investigated the relationship between architectural choices and data characteristics in time-series forecasting. The authors should more clearly articulate how their work differs from these existing approaches and what fundamentally new insights or contributions it provides.

  2. Insufficient justification of the risk formulation. I find it difficult to understand why the first equation in Section 3 provides an appropriate characterization of prediction risk. In particular, the definition of the learning cost as R(f^m)−Rm∗R(\hat{f}_m) - R_m^* requires further justification. The authors should provide relevant theoretical references or, at a minimum, explain the motivation, assumptions, and derivation underlying this formulation.

  3. Presentation issues in Figure 1(b). Figure 1(b) lacks a legend, and some text labels overlap, making the figure difficult to interpret.

  4. Insufficient statistical analysis. Although the authors use Spearman's rank correlation coefficient to support their analysis, they do not report the corresponding p-values or adjusted p-values when multiple comparisons are involved. This makes it difficult to assess the statistical significance and robustness of the reported correlations.

  5. Unclear advantage of the complete selection method. Based on Table 13, the complete model selection method does not appear to demonstrate a clear performance advantage over the alternative methods. This raises questions about the practical benefits of the proposed selection strategy.

  6. Uncertainty in the reported correlation. In Figure 3(b), the reported correlation is ϕ=0.4\phi = 0.4, with a confidence interval of [−0.24,0.78][-0.24, 0.78]. Since this interval includes both zero and negative values, the results remain compatible with no correlation or even a negative correlation. Therefore, the evidence is insufficient to establish a reliably positive association.

  7. Serious reproducibility concerns. The authors do not provide the original source code, and I could not locate the actual numerical forecasting error results from the experiments anywhere in the manuscript. Without access to the implementation and underlying experimental results, it is difficult to independently verify the reported findings. These omissions substantially undermine my confidence in the reliability of the empirical conclusions.

Questions for authors

Also see Weaknesses. My primary concern is the reproducibility and verifiability of the experimental results. I strongly request that the authors provide runnable source code and the underlying experimental data and numerical results. Without this evidence, I do not see sufficient grounds for increasing my current rating. I would also appreciate detailed responses to the other concerns raised in the Critical Weaknesses section. If the authors can adequately address these issues, I would be willing to reconsider my evaluation and potentially increase my score. Overall, I believe the research direction has considerable potential, and I look forward to the authors' responses.

6 comments
AuthorsOpenReview ↗

Summary of the revision (1/4)

Thank you for the detailed review and for stating clearly what would change your assessment. We have uploaded a revised PDF and released the code and numerical results. Below we summarize the revision; the next three comments answer your points on the code and numerical results (2/4); related work, the risk formulation, and Figure 1 (3/4); and the statistics and the full method (4/4).

The revision addresses what you asked for above all: the code and numerical results, a clear contrast with TimeRecipe, adjusted p-values, and evidence that the full method improves on simpler alternatives.

  • Code and numerical results. Appendix E lists the test MSE and MAE of every forecaster on every dataset. The code and stored results are in an anonymous repository, linked from the Reproducibility Statement.
  • Contrast with TimeRecipe. Section 2 now contrasts our approach with TimeRecipe and TSCOMP, and the introduction states the contribution as measuring restriction costs from the data and using them to recommend forecasters.
  • Statistics. Correlations and comparisons now have 95% source-family bootstrap intervals, and the tables report or mark Holm-adjusted p-values (Section 6.1, Appendix C.5).
  • The full method against simpler rules. We compare simpler versions of the score before and after training (Appendices D.2–D.3). The full score leads in rank correlation and, most clearly, after training.

We also simplified the score (Sections 4–5). It now uses two measurements, both taken before any forecaster is trained. On the data side, we compare a ridge reference that relaxes each set of restrictions with the fully restricted reference on held-out rows of the training split. The difference estimates the net gain of relaxing that set, and the gains of all 32 sets form the gain lattice. On the forecaster side, probes at initialization check whether a forecaster's design can do what each restriction forbids, for example whether it can use other channels. A forecaster's score is set by the largest lattice gain among the sets it relaxes (Eq. 5), with no fitted combination weights and no separate learning-cost term. We derived these changes from the theory, checked the score on the tuning errors of the ten submitted forecasters, and fixed it before running the seven added forecasters and eleven added datasets.

Following the other reviews, we also grew the benchmark from 10 forecasters on 14 datasets to 17 published forecasters on 25 datasets, each searched over twelve configurations and trained with three seeds (Section 6.1). We added learning curves (Section 6.2, Appendix D.1) and three longer horizons (Appendix D.2).

In the revised experiments, lattice gains rank the test gains of single structural changes (Spearman 0.52 to 0.88 over eight controlled pairs; Section 6.2). With a few minutes of probes and no results from other datasets, the score ranks forecasters at least as well as selectors that learn from them (Section 6.3). A short run on its top three selects a forecaster whose regret after its full search is 3.3%, the lowest among rankings of similar selection cost (Section 6.4).

We removed the distribution-prediction extension and the per-core synthetic experiments, and renumbered the propositions (Appendices A.5, A.7). In the PDF, changes for your points are orange, with your reviewer ID as the margin tag, and changes shared across the reviews are green with the tag GR.

AuthorsOpenReview ↗

Response to Reviewer HrSx (2/4): code and numerical results

We start with reproducibility, your main concern, and answer Points 1–6 in the next two comments.

[Point 7] Code and numerical results

The authors do not provide the original source code, and I could not locate the actual numerical forecasting error results from the experiments anywhere in the manuscript.

The code and all numerical results are now in an anonymous repository, linked from the Reproducibility Statement. It contains:

  • the code for the lattice, the probes, and the score;
  • the 17 forecasters with their search spaces and selected configurations;
  • data preparation and the splits;
  • the comparison methods;
  • the stored per-seed test errors.

For each table and figure, the README names the script that regenerates it from the stored results. For example, python paper/rankings_before_training/all_methods.py reproduces the before-training rankings of Section 6.3 (Tables 14–15) in about a minute on a CPU. Appendix C.6 describes the protocol and the stored files, and the paper itself lists the test MSE and MAE of every forecaster on every dataset in Appendix E (Tables 22–23).

AuthorsOpenReview ↗

Response to Reviewer HrSx (3/4): related work, the risk formulation, and Figure 1

[Point 1] Difference from TimeRecipe and related work

The authors should more clearly articulate how their work differs from these existing approaches and what fundamentally new insights or contributions it provides.

The main difference is what the recommendation is learned from. TimeRecipe fits LightGBM to dataset features such as trend and seasonality, and TSCOMP fits a predictor to meta-features. Like FFORMA and FFORMS, both learn from training results on other datasets. Our score uses no performance records from other datasets. It estimates the gains from relaxing structural restrictions on the target dataset with a linear reference, and uses probes to check which restrictions each forecaster can relax (Sections 4–5).

In practice, adding a forecaster takes a few minutes of probes instead of training it on other datasets (Table 2), and each recommendation identifies the restrictions contributing to its score (Table 5). Section 2 now explains this distinction.

We did not run TimeRecipe or TSCOMP on our benchmark. Our empirical comparison uses FFORMA and FFORMS, which also learn from records of other datasets. Without using such records, the score achieves comparable or lower top-three regret and a higher dataset-specific correlation (0.41 vs. at most 0.22; Section 6.3, Table 14).

[Point 2] Justification of the risk formulation

I find it difficult to understand why the first equation in Section 3 provides an appropriate characterization of prediction risk.

The equation separates the risk of a trained forecaster into the best risk available within its model class and the additional error from learning within that class. It is an identity, obtained by adding and subtracting the class risk:

R(f^m)=Rm⋆+[R(f^m)−Rm⋆].R(\hat f_m)=R_m^{\star}+\bigl[R(\hat f_m)-R_m^{\star}\bigr].

The first term is the class risk, and the bracketed term is the learning cost. The class risk is defined as

Rm⋆=inf⁡f∈FmR(f).R_m^{\star}=\inf_{f\in\mathcal F_m}R(f).

The identity requires only finite risks. In the terminology of Bottou and Bousquet (2007), class risk consists of Bayes risk and approximation error, while learning cost accounts for estimation and optimization error.

We use this decomposition because a restriction cannot lower class risk but can lower learning cost. The balance between these effects determines whether relaxing the restriction helps (Eq. 1). Proposition 2 derives both terms for the linear reference. Applying the resulting gains to trained forecasters requires the additional proxy assumption in Assumption 1. Section 6 evaluates this proxy empirically. Section 3 now explains the decomposition and cites Bottou and Bousquet.

[Point 3] Figure 1(b)

We redrew Figure 1(b). The colours are now explained on the axis instead of in a legend, and the bars are labelled directly without overlap.

Reviewer_PCEjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper recasts the bias-variance tradeoff in the context of times series forecasting to propose a cheap ridge regression-based model selection method. It argues that structural restrictions (inductive biases) may improve learning cost at the cost of predictive cost, or how much the lowest loss can attain. For example, models that do not allow channel mixing restrict structurally but learn more efficiently. The authors experimentally compare linear estimators with and without channel mixing, with more or less history, and measure the top-3 regret metric of selected models, which were manually categorized based on these features.

Critical strengths

The authors show that their selection method has lower regret than other methods.

Critical weaknesses

W1 - Novelty

I don't see novel theoretical contributions in this work. The Propositions in the papers are restatements of well-known elementary results. For instance:

  • The risk decomposition stated in the paper is the well known bias-variance tradeoff with different names.
  • Proposition 1 follows from basic statistics. "Adding an input helps only if it changes the conditional mean given ZZ". Trivially true, a readout with more information gives a better predictor.
  • Proposition 3 says that shared encoders agree with per-channel encoders in cost iff the same subspace is preferred by all encoders. Again this is elementary and well-known: shared representations for features that align to a subspace are costless.
  • Proposition 5 is the classical mean-squared-error criterion for linear regression.

W2 - Weak experiments

The experiments consist of testing the selector method by running ridge regression on cross-channel input, per-channel weights and longer windows and measuring regret, then selecting transformer/SSM forecasters based on whether they have these properties.

Beyond being limited in novelty, the results are weak: gains relative to other selection methods are all either marginal or within noise, including one ranking method that requires no fitting.

Questions for authors
  1. How does Eq. 4 differ from the classical mean-squared-error criterion for linear restrictions, apart from the oracle-ridge penalty?
  2. Can the authors explain the limited gains and the large rror bars in Figure 4?
  3. What is the main novelty of this work, in the author's words?
  4. The results are all with 96 step prediction horizons. How does the method perform against longer horizon forecasts?
5 comments
AuthorsOpenReview ↗

Summary of the revision (1/4)

Thank you for the careful review, and for noting that our selection method has lower regret than the alternatives. We have uploaded a revised PDF. Below we summarize the revision, and the next three comments answer your points on novelty (2/4), the propositions and Eq. 4 (3/4), and the experiments and longer horizons (4/4).

The revision addresses your three main concerns: what is new beyond classical results, whether the gains exceed the noise, and how the method does at longer horizons.

  • Novelty. The introduction now states the contribution as measuring restriction costs from the data and using them to recommend forecasters, and each classical result is cited where we use it (Sections 3, 4.3; Appendix A.7).
  • Size of the gains. We grew the benchmark from 10 forecasters on 14 datasets to 17 published forecasters on 25 datasets, each searched over twelve configurations and trained with three seeds (Section 6.1). We also added an experiment that compares rankings by training each ranking's top three (Section 6.4). Correlations and comparisons now have 95% source-family bootstrap intervals, and the tables report or mark Holm-adjusted p-values (Section 6.1, Appendix C.5).
  • Longer horizons. We repeated the benchmark at three longer horizons and at a longer input (Appendix D.2).

We also simplified the score (Sections 4–5). It now uses two measurements, both taken before any forecaster is trained. On the data side, we compare a ridge reference that relaxes each set of restrictions with the fully restricted reference on held-out rows of the training split. The difference estimates the net gain of relaxing that set, and the gains of all 32 sets form the gain lattice. On the forecaster side, probes at initialization check whether a forecaster's design can do what each restriction forbids, for example whether it can use other channels. A forecaster's score is set by the largest lattice gain among the sets it relaxes (Eq. 5), with no fitted combination weights and no separate learning-cost term. We derived these changes from the theory, checked the score on the tuning errors of the ten submitted forecasters, and fixed it before running the seven added forecasters and eleven added datasets.

Following the other reviews, we also added learning curves (Section 6.2), the test errors of every forecaster on every dataset (Appendix E), and the code in an anonymous repository.

In the revised experiments, lattice gains rank the test gains of single structural changes (Spearman 0.52 to 0.88 over eight controlled pairs; Section 6.2). With a few minutes of probes and no results from other datasets, the score ranks forecasters at least as well as selectors that learn from them (Section 6.3). A short run on its top three selects a forecaster whose regret after its full search is 3.3%, the lowest among rankings of similar selection cost (Section 6.4).

We removed the distribution-prediction extension and the per-core synthetic experiments. Proposition 3 is now Proposition 8 (Appendix A.5), and Proposition 5 and Eq. 4 are now Proposition 10, Corollary 11, and Eq. 11 (Appendix A.7). In the PDF, changes for your points are red, with your reviewer ID as the margin tag, and changes shared across the reviews are green with the tag GR.

AuthorsOpenReview ↗

Response to Reviewer PCEj (2/4): novelty

[W1, Q3] Novelty

The Propositions in the papers are restatements of well-known elementary results. … What is the main novelty of this work, in the author's words?

In our words, our contribution is to price structural restrictions on the target dataset and use those prices to rank forecasters before any of them is trained. The revision now makes the classical foundations of the propositions explicit and cites the relevant sources.

The pricing starts from Eq. 1, which compares a restricted forecaster f^0\hat f_0 with a relaxed one f^1\hat f_1:

E[R(f^0)−R(f^1)]=Δ−(δ1−δ0).\mathbb{E}[R(\hat f_0)-R(\hat f_1)]=\Delta-(\delta_1-\delta_0).

The predictive cost Δ≥0\Delta\ge0 depends only on the two classes and the data. The learning-cost difference δ1−δ0\delta_1-\delta_0 also depends on the learner and the sample size. As you note, relaxing a restriction never raises the class risk. For recommendation, what matters is that the expected risk after training is higher whenever δ1−δ0>Δ\delta_1-\delta_0>\Delta. This class-dependent view of predictive value is related to work on usable information (Xu et al., 2020; Ethayarajh et al., 2022).

Table 1 shows how we turn these prices into a recommendation. For each set of restrictions TT, the lattice gain Gd(T)G_d(T) estimates the right side of Eq. 1 for a linear reference at the dataset's own sample size. Probes at initialization show which sets a forecaster relaxes (Lemma 3), and the score is set by the largest of these gains (Eq. 5). It uses no fitted combination weights and no results from other datasets. Section 6 then tests each step on trained forecasters.

PEMS08 shows why the target dataset matters. There, cross-channel access improves the MLP mixer by 38%, and the best forecaster is SOFTS, which mixes channels but has a mean rank of 9.8 of 17 on the other datasets. Reading PEMS08's own lattice, the score ranks SOFTS third. Single best, which follows the usual ranking, ranks it twelfth (Section 6.3, Table 5).

The introduction now states the contribution in these terms, and Section 3 introduces the decomposition as the standard identity of Bottou and Bousquet (2007).

AuthorsOpenReview ↗

Response to Reviewer PCEj (3/4): the propositions and Eq. 4

[W1] The listed propositions

The revision cites each result and says what it is used for:

Your point (submission numbering)Role and place in the revision
Risk decompositionSection 3: the identity of Bottou and Bousquet, applied to a pair of classes in Eq. 1
Proposition 1 (adding an input)Still Proposition 1, true for class risk as you note; it gives Δ\Delta from the data distribution for three of the five restrictions (Section 4.1)
Proposition 3 (shared encoders)Now Proposition 8 in Appendix A.5; the score does not use it
Proposition 5 and Eq. 4 (classical criterion)Now Proposition 10 and Corollary 11 (Eq. 11, Appendix A.7); see Q1 below

[Q1] Eq. 4 and the classical criterion

How does Eq. 4 differ from the classical mean-squared-error criterion for linear restrictions, apart from the oracle-ridge penalty?

With one penalty for all blocks, Eq. 4 (now Eq. 11) recovers the classical mean-squared-error criterion in ridge form (Toro-Vizcarrondo and Wallace, 1968; Wallace, 1972). The revision now states this in Section 4.3 and Appendix A.7. Beyond the penalty, the lattice differs in three ways.

What is compared. Eq. 4 compares one restricted fit with one unrestricted fit. The lattice gives each restriction its own block and penalty, as in generalized ridge (Hoerl and Kennard, 1970), and compares all 32 combinations. This accounts for the overlap between blocks in real data. On the median dataset, relaxing all five restrictions gains only 0.61 of the sum of their single gains (Section 5.1).

How the gain depends on the sample size. With optimal penalties, adding block kk, with pkp_k coefficients and predictive cost γk\gamma_k, lowers the risk by γk2/(γk+νk)\gamma_k^2/(\gamma_k+\nu_k), where νk=σ2pk/n\nu_k=\sigma^2p_k/n and σ2\sigma^2 is the noise variance (Proposition 2). The gain is small when γk≪νk\gamma_k\ll\nu_k and approaches γk\gamma_k as nn grows. The same restriction is thus worth little to relax at small samples and nearly its full predictive cost at large ones.

How it is measured. Penalties are chosen on validation rows and fits are scored on separate rows, so each lattice gain estimates the net gain at the dataset's own sample size. When data are scarce, it can be negative (Section 4.3).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Area Chair and Reviewers,

We have uploaded a revised PDF that addresses the reviews. In brief:

  • The score now rests on two measurements taken before any forecaster is trained: a gain lattice measured on the training split, and probes at initialization.
  • The benchmark grows to 17 published forecasters on 25 datasets, with new experiments on learning curves, on training each ranking's top three, and on longer horizons.
  • Comparisons come with 95% bootstrap intervals and Holm-adjusted p-values, and the code and numerical results are in an anonymous repository.

In the PDF, changes made for all reviewers are green (margin tag GR), and changes for one reviewer are in that reviewer's colour (red for PCEj, orange for HrSx, purple for Pxgv). A summary of the revision and point-by-point responses are posted under each review.

Thank you again for your time and feedback.

AuthorsOpenReview ↗

Interpreting the nonlinear reference measurements

We further examined what the nonlinear reference captures and how to interpret its measured gains, complementing the forecaster comparisons in Section 6. The reference is linear in its fitted coefficients but nonlinear in the histories. When paths cc and ll are both enabled, it includes 512 fixed random ReLU features of the joint history, whose coefficients are fitted by ridge regression (Appendix B.1). These joint features allow non-additive dependence on the two histories.

Interpreting the reference error. A standard conditional-expectation decomposition relates prediction error to recovery of the signal supplied by the additional history. Let UU and (U,V)(U,V) be the restricted and relaxed inputs, with m0=E[Y∣U]m_0=\mathbb{E}[Y\mid U] and m1=E[Y∣U,V]m_1=\mathbb{E}[Y\mid U,V]. For a fitted f(U,V)f(U,V), assume Y,f∈L2Y,f\in L^2 and define

s=m1−m0,fˉ=E[f∣U],b=f−fˉ,ε=E(f−m1)2.s=m_1-m_0,\quad \bar{f}=\mathbb{E}[f\mid U],\quad b=f-\bar{f},\quad \varepsilon=\mathbb{E}(f-m_1)^2.

Here ss is the signal supplied by the additional history, and bb is the part of the fitted prediction beyond its conditional mean given UU. Holding the fit fixed and taking expectations over an independent test draw gives

ε=E(fˉ−m0)2+E(b−s)2,E(b−s)2≤ε.\varepsilon=\mathbb{E}(\bar{f}-m_0)^2+\mathbb{E}(b-s)^2,\qquad \mathbb{E}(b-s)^2\leq\varepsilon.

To see this, write f−m1=(fˉ−m0)+(b−s)f-m_1=(\bar{f}-m_0)+(b-s). The first term is a function of UU, while the second has conditional mean zero given UU, so the cross term vanishes. This applies to the realized fit, including validation-selected penalties.

Thus, small error relative to the conditional mean also implies small error in recovering the additional-history signal. For a square-integrable restricted fit f0(U)f_0(U) and squared-error risk R(g)=E(Y−g)2R(g)=\mathbb{E}(Y-g)^2, the same decomposition gives

R(f0)−R(f)=Es2+E(f0−m0)2−ε.R(f_0)-R(f)=\mathbb{E}s^2+\mathbb{E}(f_0-m_0)^2-\varepsilon.

Gain also depends on the restricted fit's error, which is why we evaluate target error alongside gain below.

Recovery of known interactions. Independent uniform angles θ,φ∈[0,2π)\theta,\varphi\in[0,2\pi) were encoded in own history (96 steps) and other-channel history (24 steps). The targets were

h1=2cos⁡θcos⁡φ,h2=2cos⁡(2θ)cos⁡(2φ),h_1=2\cos\theta\cos\varphi,\qquad h_2=2\cos(2\theta)\cos(2\varphi),

with independent zero-mean noise of SD 0.5. Both satisfy E[h∣θ]=E[h∣φ]=0\mathbb{E}[h\mid\theta]=\mathbb{E}[h\mid\varphi]=0 and Eh2=1\mathbb{E}h^2=1. Every square-integrable additive predictor q=a(θ)+c(φ)q=a(\theta)+c(\varphi) therefore has

E(h−q)2=1+Eq2≥1,\mathbb{E}(h-q)^2=1+\mathbb{E}q^2\geq1,

since E(hq)=0\mathbb{E}(hq)=0. These targets require non-additive dependence on the two histories.

We used the released feature map, ridge routine, penalty grid, and blockwise selection. For each of three data seeds, we selected penalties using 768 fitting and 256 validation windows, refitted on all 1,024, and tested on 4,096 unseen windows. Gains are MSE reductions over the own-history nonlinear reference. Each joint fit was also evaluated against the noiseless target on 32,768 additional independent input pairs.

TargetGain: separate nonlinear featuresGain: joint referenceJoint reference's noiseless target MSE
h1h_10.0000.9970.0139 ± 0.0088
h2h_2−0.0040.9490.0394 ± 0.0079

Gains are means over three data seeds; target errors are mean ± SD, using the paper's feature seed. Across five additional feature seeds, mean target errors ranged from 0.011–0.018 for h1h_1 and 0.035–0.045 for h2h_2. Here m0=0m_0=0 and m1=hm_1=h, so the target errors estimate ε\varepsilon in the inequality above: accurate prediction also means accurate recovery of the interaction. For the paper's feature seed, the largest within-fit Monte Carlo standard error was 0.0004.

History correspondence on real data. On 25 datasets, we kept each own-history–target pair fixed and reassigned its other-channel history from a randomly selected window in the same time segment, then refitted.

Cross-channel gain (% of base-reference risk)Linear contextNonlinear context
Original histories, median0.811.56
Re-paired histories, median0.000.00
Re-paired histories, largest absolute value0.02790.1323

All re-paired gains were below 0.133% of base-reference risk in absolute value, including the 12 linear-context and 13 nonlinear-context cases originally above 1%. The cross-channel gains therefore depend on the observed history correspondence. This complements the circular-shift control in Figure 6 while leaving own histories and targets unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.