acceptodds
Under review as a conference paper at ICLR 2027

Future-Guided On-Policy Self-Distillation for Time-Series Forecasting

Abstract

Time-series forecasting has a training asymmetry: each training example eventually reveals the future, but a deployed model sees only history. That future contains both history-predictable structure and innovations. Standard pointwise supervision uses the realised values as targets; can their ordered structure also guide learning without entering the deployed forecast? FoldOPSD turns this training-time context into on-policy self-distillation. A teacher view of the current forecaster receives summaries of observed future segments, while a history-only student uses a learned constant correction. Their shared forecasting body lets teacher supervision update the model that will be deployed; detached residual matching and an EMA student add teaching targets. Because both views share the forecasting body, the answer changes the training update rather than the deployed input. Analysis identifies when ideal answer conditioning reduces gradient noise and how answer hints reweight residual directions. At a fixed DLinear/ETTm2 checkpoint, the joint shared-body gradient lowers reference MSE by 28.3% on training and 35.0% on validation, while across-batch variance falls by 33.4% and 33.9%, respectively. With a compatible native output bias, the student correction folds exactly into that bias, adding no inference operations. Across four datasets and five seeds, FoldOPSD lowers mean test MSE in all 24 DLinear and iTransformer settings, by up to 10.85%. True future hints outperform noise and dataset-level summaries in all 16 matched ETT settings with the basic OPSD objective.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Self-Distillation, OPSD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H4qcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

During training, future observations can provide additional guidance for time series forecasting. The authors propose FoldOPSD, which consists of a teacher path and a student path that share the same forecasting model. The teacher path accesses the mean, standard deviation, and trend of the future segment, while the student path uses only past observations and a learned constant correction to predict future values. After training, only the student path is used for inference.

Critical strengths
  1. Using information about the future segment of the training data to guide forecasting model updates through self-distillation is an interesting idea.

  2. The constant correction in the student path can be absorbed into the existing output bias, providing a practical way to improve forecasting accuracy without changing the deployed architecture.

Critical weaknesses
  1. The paper’s presentation needs improvement. The introduction does not clearly state the main contributions, abbreviations such as OPSD and EMA are not defined at first use, and the purpose of Figure 1(a) is insufficiently explained in the main text.

  2. The teacher path introduces additional training costs. It remains unclear whether comparable improvements could be achieved by training the baseline longer under the same computational budget.

  3. The method uses the mean, standard deviation, and slope computed over patches of fixed lengths. However, the contribution of each statistic and the rationale for selecting the patch lengths remain unclear.

  4. The results in Table 6 raise questions about the robustness of FoldOPSD across different models.

Questions for authors

Please the author(s) answer my two questions: point 2 and point 3 in the 'Critical Weaknesses' section.

1 comment

Thanks the authors for the rebuttal. The additional clarifications and experiments have addressed my concerns regarding W2 and W3. However, my concern regarding W4 remains, so I will keep my original score.

Reviewer_o7xVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes FoldOPSD, which uses future values as privileged information during training only. A teacher view gets patch-level summaries of the future through a small correction map, while the student shares the same body and uses a learned constant slot instead. Because the student's correction is constant, it folds into the host's output bias at deployment, and experiments on DLinear and iTransformer show consistent but small MSE reductions.

Critical strengths

The paper nicely uses future values as a training-time correction signal while making information leak at inference impossible by construction. Since the teacher and student share one body and the student's correction folds exactly into the native bias, the deployed model is just the original architecture with the same cost. I find this a clean and practical design.

Critical weaknesses
  1. There is no ablation on the student slot s∅s_\varnothing (e.g., fixed to zero or to the descriptor mean vs. learned), on the descriptor design (statistics, patch length), or on α/τ/ρ. Also, the "detach teacher" intervention removes the label gradient and the hint gradient at once, so it can't tell which one matters. It's hard to know which parts actually do the work.

  2. The paper gives no concrete example or evidence of what s∅s_\varnothing learns. From my reading, Wcs∅+bcW_c s_\varnothing + b_c is just a horizon-wise bias, and s∅s_\varnothing only acts as a baseline to center Q=Wc(sj−s∅)Q = W_c(s_j - s_\varnothing). But the theory needs E[Q∣X]=0\mathbb{E}[Q \mid X] = 0, which a constant can't guarantee, and this is never measured.

  3. Both main figures look AI-styled and pack in every component instead of showing the core idea. I couldn't tell what Figure 1 is meant to convey, since it mixes a 3D quadratic surface, a descriptor illustration, and a single-seed gradient bar chart.

  4. Asking for more baselines is usually not a strong critique, but since FoldOPSD is an additive plug-in, its value depends on whether the gains hold on stronger forecasters. DLinear and iTransformer is a fairly weak setting; more recent models are needed.

  5. Most improvements are 0.4 to 3% MSE, which is marginal given the extra training cost (second view, EMA copy, more hyperparameters).

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer o7xV — future content, the actual update, and exact folding (1/5)

Dear Reviewer o7xV,

FoldOPSD changes how the shared forecaster learns from future structure and preserves the resulting history-only student through exact native-bias folding (Proposition 2). Fixing the student slot to zero while retaining its trainable correction bias preserves all 12 improvements over ERM; its mean paired MSE differs from the learned-slot model by only +0.022%. Content interventions retain teacher label supervision: replacing the realized summary with noise or a dataset mean reduces performance in the basic OPSD controls, and removing only patch means worsens all 12 standalone FoldOPSD pairs (+1.69% mean MSE). These results identify useful future content in the training route and clarify the constant interface’s role at export. Proposition 4 characterizes the implemented gap directly, retaining its conditional mean shift and covariance terms. Proposition 3 identifies the special case in which the gap equals the centered oracle.

W1 — Future-level information contributes to learning through the shared teacher path

The teacher uses the realized summary as context while predicting the full future. Its label loss updates the same forecasting body that the student uses at deployment.

The descriptor ablations identify local future level as its most consistent content contribution. Removing future patch means worsens all 12 paired runs, increasing MSE by 1.69% on average across four settings. These interventions preserve descriptor width and parameter shapes and start from identical student forecasts. Doubling patch length also worsens all 12 pairs. The 72-fit panel covers DLinear/iTransformer × ETTh1/ETTm2, H=96, seeds 0–2 (Supplement S5); positive entries are MSE increases relative to Full.

InterventionDLinear/ETTh1DLinear/ETTm2iTransformer/ETTh1iTransformer/ETTm2
Fixed-zero slot−0.0025%+0.1364%−0.0471%+0.0013%
Remove mean+0.2285%+3.0491%+2.2999%+1.1917%
Remove std+0.0029%−0.2843%−0.0491%−0.0048%
Remove slope−0.0255%−0.0083%+0.0590%+0.0075%
Halve patch length+0.2275%−0.0419%+0.6660%+0.0434%
Double patch length+0.0638%+0.4080%+0.6113%+0.8362%

The resulting design principle is to preserve useful local level information at an effective segmentation. Removing std improves 9/12 pairs by a small mean 0.084%, while slope effects are smaller and mixed. The default patches of 24 points on ETTh1 and 48 on ETTm2 give the best setting-mean MSE among the tested lengths in three of four settings. Length changes jointly vary temporal resolution and correction-map width, so the result guides this implemented descriptor design.

The same future-level content contributes inside a structural-loss combination. In Fold+FreDF on iTransformer/ETTh1, removing only patch means worsens all nine weight–seed comparisons: +1.621%, +0.856%, and +0.582% at FreDF weights 0.2, 0.5, and 0.8. EMA, teacher supervision, KD, and parameter shapes stay fixed. This identifies a contribution from the teacher’s future content after direct frequency supervision is already present. Across the full factorial panel, removing mean at weight 0.8 improves the other three settings, showing that content and training objective act together.

The sensitivity studies characterize how EMA and residual matching balance the training signals (Appendix G.3). Five-seed studies on DLinear/ETTm2 and iTransformer/ETTh1 vary EMA mixing α∈{0.3,0.5,0.7}, decay ρ∈{0.99,0.999}, residual weight λ∈{0.01,0.04,0.14}, and saturation τ∈{3,10,30}. The default (0.5,0.999,0.04,10) gives MSE changes −0.56%/−0.65%; α=0.3 gives −0.56%/−1.01%, ρ=0.99 gives −0.32%/−0.93%, and λ=0.01 gives −0.45%/−0.67%. Both neighboring τ values retain gains. Stronger weights can reverse the effect: α=0.7 gives +0.44% on iTransformer/ETTh1, and λ=0.14 gives +0.37% on DLinear/ETTm2. These five-seed historical recipes are reported alongside the new three-seed panel with their configurations identified.

AuthorsOpenReview ↗

Reply to Reviewer o7xV — linking future content to the training update (2/5)

W1 — Content interventions and gradient measurements expose the learning route

The paired content interventions show that the realized future summary contributes through the teacher route. Teacher-body detachment measures the value of the complete direct path, costing 2.59–5.36 percentage points of relative MSE improvement in four of five settings (Figure 5). Keeping that path active and replacing only its descriptor gives the following comparisons:

True descriptor compared withAdvantage in relative MSE improvementCoverage
Scale-matched Gaussian noise2.20 pp16/16 ETT settings
Dataset-mean descriptor2.17 pp16/16 ETT settings
Within-batch cyclic shift0.79–4.64 pp4/4 settings

The cyclic shift retains descriptor marginals and changes their correspondence to the training example. Its measured effect, together with width-preserving mean removal, connects forecast quality to the future information supplied through the teacher. EMA adds a further mean 0.76 pp, and replacing output-MSE KD with bounded residual matching adds 0.07 pp; the No-KD control evaluates the complete KD term. These interventions identify contributions from information content and from the subsequent organization of its training signal.

The shared-body gradient identity makes this information route explicit. Let gsg_s and gtg_t be the student and teacher label gradients, with κ=β/(1+β)\kappa=\beta/(1+\beta). Their normalized combination is gjoint=(gs+βgt)/(1+β)=gs+κcg_{\mathrm{joint}}=(g_s+\beta g_t)/(1+\beta)=g_s+\kappa c, where c=gt−gsc=g_t-g_s. The teacher context changes c and thereby the update received by the shared forecasting body. The normalization holds the overall supervision scale fixed.

At the saved DLinear/ETTm2 checkpoint, this correction reduces both batch-to-batch gradient variation and the squared distance between the empirical mean and its same-split reference. Recomputing the arrays with the common 1/n moment convention gives the following joint-minus-student changes (Appendix C.2):

Probe / referenceVariance changeSquared mean-distance changeTotal error change
Train / train−0.044996−0.024318−0.069314
Valid / valid−0.013935−0.002838−0.016773

The same-split gradient-reference MSE reductions are 28.258% and 35.013%, with contributions from both variance and mean alignment. Across splits, train/valid improves 34.646%, while valid/train increases error by 7.760%. These four measurements describe how the realized correction acts relative to each reference gradient.

The short-branch study measures the update as a training-state intervention. Its 36 states come from FoldOPSD trajectories at epochs 1/5/15 and retain model, Adam, and EMA states. Branches update all parameters, including correction and slot, match the complete Adam parameter-displacement norm at each step, and share subsequent batches and dropout streams. Independent validation windows measure the 1/5/20-step outcomes.

At early DLinear/ETTm2, Full improves validation MSE by 2.35% relative to the student-only branch endpoint after 20 steps; mean-only gives 2.38%, no-mean 1.50%, shuffled descriptors 1.60%, and direct-summary supervision 4.80%. Across all 36 states, Full improves 13 states and increases mean validation MSE by 0.382%. The panel therefore identifies state-dependent effects of this full-optimizer intervention. Together, the fixed-checkpoint measurements, state interventions, and complete training runs expose the update relationship, its local behavior, and the final forecasting result.

AuthorsOpenReview ↗

Reply to Reviewer o7xV — theory connects the actual update to exact deployment (3/5)

W2 — The theory characterizes the actual training gap and preserves the learned student

The implemented gap is analyzed directly in Proposition 4. It retains mQ(X)=E[Qout∣X]m_Q(X)=\mathbb E[Q_{\mathrm{out}}\mid X] and the approximation error, giving their effects on the update mean and covariance. Proposition 3 describes the oracle gap q∗=E[ϵ∣X,S]q^*=\mathbb E[\epsilon\mid X,S]; its conditional mean is zero. Propositions 1 and 2 establish the residual geometry and exact folding for their respective constructions.

m_Q(X)=a_X\odot W_c(\mathbb E[S\mid X]-s_\varnothing).$$ Writing $Q_{\mathrm{out}}=m_Q(X)+q^*+e$, with the oracle component $q^*$ and approximation error e conditionally centered, and $\kappa=\beta/(1+\beta)$, the normalized label-supervised output gradient satisfies $$\mathbb E[u_Q\mid X]=\delta+\kappa m_Q(X).$$ The identity quantifies how future context shifts the expected update and changes its covariance. At a fixed gap, the conditional-risk target becomes $\mu(X)-\kappa m_Q(X)$ (Supplement S2). The covariance identity retains the approximation-error covariance and cross terms. Proposition 3 gives the reduction for the particular oracle $q^*$; conditional centering alone does not establish that result. The empirical probe in the preceding reply measures fixed-checkpoint batch-gradient moments. It does not directly estimate the conditional mean of the gap at each $X$. | Result | What it establishes | | --- | --- | | Proposition 1 | Exact residual reweighting after profiling a restricted label-supervised objective | | Proposition 2 | Exact export of the trained student through a compatible native bias | | Proposition 3 | Mean-preserving covariance reduction for the conditionally centered oracle gap | | Proposition 4 | Mean shift and covariance identity for the implemented gap | **The learned predictor consists of the trained shared body together with its exportable constant correction.** The deployment object is $v=W_c s_\varnothing+b_c$, and teacher supervision updates the shared body throughout training. The fixed-zero ablation shows that this training benefit is retained with the simpler zero-slot parameterization, identifying the slot as an interface for preserving the student at export. Proposition 2 completes the training-to-deployment construction: the substitution $b\leftarrow b+v$ in a compatible native bias, before scale restoration, preserves the trained student. Float32 verification differs by at most $4.77\times10^{-7}$. The deployed host thus retains the learned forecast with its original operations and parameter shapes. The correction has an exact coordinate invariance (Supplement S1). For invertible T and shift a, changing $S$ to $TS+a$, $s_\varnothing$ to $Ts_\varnothing+a$, $W_c$ to $W_cT^{-1}$, and $b_c$ to $b_c-W_cT^{-1}a$ preserves the teacher/student predictions, v, and their gap. The correction and its effect on learning are therefore the meaningful objects of interpretation. The empirical slot comparison is learned versus fixed zero; a descriptor-mean-fixed slot was not separately trained. The theory gives a connected account of the method: the teacher changes the supervised residual geometry, the actual-gap identities quantify its effect on updates, and exact folding carries the learned student into the original compatible host. The actual-gap, residual-geometry, and folding results cover their respective constructions without requiring the oracle centering condition.
Reviewer_YFweOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In standard time-series forecasting training, future values are used only as regression targets. This paper explores an additional way: during training, using future values as privileged input to a teacher that guides the history-only forecaster. The authors propose FoldOPSD, an on-policy self-distillation framework in which a teacher view of the forecaster receives summaries of the observed future, while a history-only student view uses a learned constant correction. Because both views share the same forecasting body, the teacher's supervision directly updates the deployed model, and an EMA student and bounded residual matching provide additional distillation targets. The paper also analyzes when answer conditioning reduces gradient noise and how it reweights residual directions. Experiments on DLinear and iTransformer across four datasets show that FoldOPSD reduces test MSE relative to a matched reference in all 24 settings, with an extended study covering additional backbones.

Critical strengths
  1. Well-designed control experiments on the future hints. Replacing the true future summaries with scale-matched noise or a dataset-mean descriptor removes the gain almost completely, shuffling the summaries across examples also removes the gain. This shows that the information in the future summaries drives the improvement.
Critical weaknesses
  1. Gains may reflect faster optimization under a short training budget rather than a better solution. All models are trained for only ten epochs, which may lead to insufficient convergence. Some references are clearly weaker than published results under the same lookback. It's possible that the label-derived summaries may mainly act as an auxiliary signal that accelerates convergence.

  2. Missing comparisons with simpler alternatives. The method is compared only with its own reference and ablations. Simpler methods can provide a similar signal without a teacher, such as structural or frequency-domain losses or an auxiliary head that predicts these summaries from history. Without such comparisons, it is unclear whether the proposed self-distillation framework is necessary.

Questions for authors
  1. Does the improvement persist when the reference is trained to convergence (e.g., more epochs with early stopping) or with a compute budget matched to FoldOPSD?
  2. How does FoldOPSD perform with a stronger backbone?
  3. How does FoldOPSD compare with an auxiliary head that predicts the patch summaries from history, or with existing loss-based alternatives such as the patch-wise structural loss (Kudrat et al., ICML 2025) or the frequency-domain loss FreDF (Wang et al., ICLR 2025)?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer YFwe — what future-conditioned supervision adds (1/3)

Dear Reviewer YFwe,

FoldOPSD uses the same future labels through a different training route: the observed summary conditions a shared teacher whose target remains the full future, and exact folding preserves the learned history-only student on compatible hosts. The tested removable head predicts the same summary; FoldOPSD improves both MSE and MAE in all 12 paired comparisons. On iTransformer/ETTh1, adding FoldOPSD improves all nine weight–seed comparisons for each of FreDF and PS-loss at fixed weights. Within Fold+FreDF, removing only future patch means worsens all nine comparisons in that setting, identifying a contribution from the content itself. With validation-based learning-rate selection for both methods, FoldOPSD improves 12/12 pairs at 15 epochs and 9/12 under a common measured training-and-validation time cap, with four-setting mean MSE reductions of 2.50% and 1.50%, respectively. These results show how the route of future supervision changes the learned predictor.

W1 / Q1 — Forecasting value under matched training budgets

Both methods search the same native-LR multipliers {0.3,1,3} on DLinear/iTransformer × ETTh1/ETTm2, L=H=96, seeds 0–2 (Supplement S7). The reported averages weight the four settings equally. Negative table entries indicate lower FoldOPSD test MSE; wins count paired seeds.

Setting15-epoch MSE changeWinsMeasured-time-prefix MSE changeWins
DLinear / ETTh1−1.444%3/3+2.025%0/3
DLinear / ETTm2−1.547%3/3−1.547%3/3
iTransformer / ETTh1−3.240%3/3−2.726%3/3
iTransformer / ETTm2−3.749%3/3−3.749%3/3

The same training benefit is also present when ordinary supervision receives a longer optimization trajectory. Under the archived recipes, giving ERM the original FoldOPSD run’s measured training-plus-validation time yields 12/12 improvements for FoldOPSD; extending both methods to 30 epochs also yields 12/12 (Supplement S3). Each arm selects its own validation checkpoint.

SettingFixed-recipe equal-time changeFixed-recipe 30-epoch change
DLinear / ETTh1−0.269%−0.286%
DLinear / ETTm2−3.524%−3.524%
iTransformer / ETTh1−3.245%−3.240%
iTransformer / ETTm2−1.752%−1.752%

Aggregation (Table 15). The fixed-recipe equal-time and 30-epoch columns average seed-paired percentage changes. The learning-rate search columns use the ratio of the three-seed mean MSEs within each setting. The headline 2.50% and 1.50% reductions weight the four settings equally.

Protocol. The follow-ups use archived formal recipes, reported separately from the submitted main-table recipes. All 24 extended trajectories reproduce the archived run prefixes. The 30-epoch runs retain type1 decay: the initial learning rate is held for two epochs and then halved each epoch.

The time-prefix comparison retrospectively selects only completed epochs from measured 15-epoch trajectories. Each seed’s common cap is determined from the median time of the three ERM candidates through epoch 10 and the common feasible endpoint. Time includes training and per-epoch validation; candidate/checkpoint selection is locked before test evaluation.

Reference strength. In the main comparisons, each pair shares the host, lookback, input features, and validation rules (Appendix A.1–A.2). Our ETT iTransformer uses the disclosed ReLU implementation without calendar covariates. Published configurations can use different inputs, widths or schedules at the same lookback. The paired results and the independent LR search measure the training increment within the disclosed forecasting setup.

AuthorsOpenReview ↗

Reply to Reviewer YFwe — why the route of future supervision matters (2/3)

W2 / Q1/3 — Using future summaries as context creates learning value

Future summaries are more useful through the proposed teacher route than through the tested summary-prediction head. Both use the same future descriptors, and both remove the auxiliary machinery at export. FoldOPSD improves MSE and MAE in all 12 paired comparisons after validation selects the auxiliary loss weight. This establishes an empirical distinction between supplying future structure as training context and making it an additional prediction target.

SettingFoldOPSD vs. auxiliary-head MSEMSE winsMAE wins
DLinear / ETTh1−0.377%3/33/3
DLinear / ETTm2−13.439%3/33/3
iTransformer / ETTh1−3.364%3/33/3
iTransformer / ETTm2−2.826%3/33/3

Auxiliary-head protocol. An H-to-d affine map predicts S from the history-normalized student forecast; the objective is student MSE plus a summary-prediction loss, with weight chosen from {0.01,0.1,1}. The constant-slot student interface is retained, and the head is deleted at export (Supplement S4). All four settings select 0.01. DLinear/ETTm2’s large gap includes the auxiliary arm performing worse than ERM; the central result is the 12/12 consistency on both metrics.

The submitted Table 10 complements this comparison by studying the transfer objective. It retains student and teacher label supervision and replaces output KD with summary supervision in a history-generated-slot interface; that auxiliary transfer objective is better in three of four settings. The new teacher-free, removable-head experiment evaluates the training route under the main constant-slot deployment interface.

The result has a direct optimization interpretation. An auxiliary head learns to predict S from history features. Our teacher receives S and is supervised on the complete future Y, so S changes the full-horizon residual that updates the shared forecasting body. The head comparison demonstrates the value of this route through the prediction task itself.

Standalone comparisons at 15 epochs. The following values are three-seed mean test MSE at the native learning rate. Each direct loss uses its validation-selected weight. FreDF and PS-loss each outperform standalone FoldOPSD in three of the four settings, giving strong reference points for the combination experiment.

SettingERMFoldOPSDFreDFPS-loss
DLinear / ETTh10.3962600.3951270.3913590.386709
DLinear / ETTm20.1928540.1860530.1832780.182819
iTransformer / ETTh10.3918100.3791140.3833360.383262
iTransformer / ETTm20.1838030.1805800.1787190.174869

Future-conditioned supervision also improves learning when a structural objective already uses the future labels. On iTransformer/ETTh1 at horizon 96, the validation-selected FoldOPSD+FreDF and FoldOPSD+PS combinations outperform both their structural-loss counterpart and standalone FoldOPSD. At 15 epochs:

MethodTest MSE
ERM, native LR0.391810
FoldOPSD0.379114
FreDF0.383336
FoldOPSD + FreDF0.374995
PS-loss0.383262
FoldOPSD + PS-loss0.376047

On iTransformer/ETTh1, the validation-selected combinations lower MSE by 2.176% relative to FreDF and 1.883% relative to PS-loss. The FoldOPSD+FreDF comparison retains a 2.040% improvement over FreDF under the measured time prefix. The complete combination results are:

SettingFold+FreDF vs. FreDFFold+PS vs. PS
DLinear / ETTh1+0.219%+0.030%
DLinear / ETTm2−0.116%−0.030%
iTransformer / ETTh1−2.176%−1.883%
iTransformer / ETTm2+0.489%−0.084%

The combination results extend the contribution from standalone accuracy to how supervision can be composed. Future structure can act through both a direct structural penalty and a contextual teacher residual. The next comparison holds the structural-loss weight fixed and intervenes on the teacher’s future content, connecting the added performance to the proposed information route.

Structural-loss implementation. FreDF uses temporal-rFFT complex-error magnitude. PS-loss retains adaptive patches, correlation/KL/mean terms, and gradient-norm balancing, with explicit 1e−12 stabilization at the common constant-output initialization. The retained check agrees with the official PS objective and gradients on nondegenerate inputs.

AuthorsOpenReview ↗

Reply to Reviewer YFwe — future content adds value alongside structural supervision (3/3)

W2 / Q3 — Future-level context contributes to the structural-loss combination

The teacher route adds predictive value at each of three fixed structural-loss weights. On iTransformer/ETTh1, adding FoldOPSD improves all nine paired comparisons for FreDF and all nine for PS-loss. Each row below fixes the loss weight, native LR, and 15-epoch allowance and compares three paired seeds (Supplements S7 and S9).

Structural lossFixed weightMSE change after adding FoldOPSDWins
FreDF0.2−2.176%3/3
FreDF0.5−1.144%3/3
FreDF0.8−0.923%3/3
PS-loss1−2.048%3/3
PS-loss3−0.995%3/3
PS-loss10−0.418%3/3

Future patch means contribute to that added value. On iTransformer/ETTh1, zeroing only the teacher descriptor’s mean entries within Fold+FreDF worsens all nine weight–seed comparisons. Descriptor width, parameter shapes, EMA, teacher supervision, KD, learning rate, paired seeds, and training allowance stay fixed.

FreDF weightFull Fold+FreDF MSEWithout future meansMSE increaseWorse pairs
0.20.3749950.381073+1.621%3/3
0.50.3744390.377642+0.856%3/3
0.80.3746410.376820+0.582%3/3

The combination establishes a content-dependent increment. The teacher route adds predictive value on iTransformer/ETTh1 at each fixed weight, and removing future-level information reduces that value. At FreDF weight 0.8, mean removal instead improves the combination in the other three settings (Appendix G.3). The content contribution therefore depends on the accompanying objective and setting. The standalone FoldOPSD mean ablation is a separate comparison (Table 17).

The training benefit transfers to FACT with exact native-bias export

The submitted cross-host evaluation already includes FACT, with improvements in 20 of 24 settings and a 2.17% macro-average MSE reduction. The full extension evaluates FACT and MixLinear alongside DLinear and iTransformer over six datasets, four horizons {96,192,336,720}, and five seeds per cell (96 cells; Table 6 and Appendix E).

HostCells with lower mean MSEMacro MSE changeNative-bias folding
DLinear20/24−5.91%Compatible
FACT20/24−2.17%Compatible
iTransformer20/24−2.05%Compatible
MixLinear8/24+0.78%No compatible horizon-wise bias

FACT improves all 16 ETT and all four Weather cells, and its native projection bias absorbs the trained constant. This establishes that the learning route works beyond the two main hosts and that its resulting student can be exported through a third architecture’s existing parameters. The full host table reports the measured accuracy outcomes; both MixLinear arms retain the same external H-vector.

AuthorsOpenReview ↗

Reply to Reviewer YFwe — Closing Remarks

Your review pinpointed the exact empirical questions needed to test whether FoldOPSD learns a better history-only predictor rather than acting as a short-budget shortcut. Building on the future-hint controls you highlighted as a strength, executing your suggested experiments directly shaped our revised manuscript and addressed each diagnostic concern you raised:

  • Convergence, compute-matched budgets, and baseline protocol (W1 / Q1): Across DLinear and iTransformer on ETTh1/ETTm2 (L=H=96L=H=96), FoldOPSD retains consistent gains over ERM under fixed-recipe equal measured fit time (8–18 completed ERM epochs, 12/12 wins) and extended 30-epoch type1 trajectories (12/12 wins), as well as independently LR-tuned 15-epoch budgets (12/12 wins, −2.50%-2.50\% mean MSE) and retrospective common measured-time prefixes (9/12 wins, −1.50%-1.50\% mean MSE, transparently retaining the DLinear/ETTh1 short-cap outcome). All compared arms strictly match host configuration, lookback, covariate access, and validation selection so that differences isolate the training objective.
  • Transfer to a stronger modern backbone (Q2): Beyond DLinear and iTransformer, our 96-cell cross-host study evaluates FACT (ICLR 2026) across six datasets and four horizons (H∈{96,192,336,720}H \in \{96, 192, 336, 720\}, five seeds), lowering MSE in 20/24 settings (−2.17%-2.17\% macro MSE and −1.37%-1.37\% macro MAE, improving all 16 ETT and all 4 Weather cells) while folding the student correction exactly into FACT's native projection bias at zero inference cost.
  • Comparison with simpler teacher-free alternatives (W2 / Q3): Under the same constant-slot deployment interface, FoldOPSD outperforms a validation-selected removable summary-prediction head (γ∈{0.01,0.1,1}\gamma \in \{0.01, 0.1, 1\}) in all 12/12 MSE and MAE pairs (−0.377%-0.377\% to −13.439%-13.439\% MSE), establishing the value of using future summaries as teacher context rather than as an auxiliary prediction target from history. Against FreDF (ICLR 2025) and official-verified PS-loss (ICML 2025), we candidly report where standalone structural losses are stronger (3/4 native-LR settings) and demonstrate consistent complementary gains when adding FoldOPSD on iTransformer/ETTh1 (−2.176%-2.176\% over selected FreDF, −2.040%-2.040\% under the measured-time prefix, and −1.883%-1.883\% over selected PS-loss, improving all 9/9 fixed-weight pairs for each).

Your suggestions substantially sharpened both the empirical rigor and the scientific scope of the paper. In light of these complete controls, we hope our responses resolve your concerns and support a positive reassessment of your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.