acceptodds
Under review as a conference paper at ICLR 2027

Learning from Generated Contexts: Adaptive Stagewise Training for Long-horizon Forecasting

Abstract

Long-horizon forecasts of physical and dynamical systems accumulate errors as generated inputs evolve during recursive deployment. We study stagewise training: successive predictors learn true futures from contexts generated by frozen predecessors. We analyze specialization gains and error propagation; population bounds motivate monitoring without prescribing individual error curves. We introduce public-boundary adaptation and sample-specific adaptation, distinguishing model training from inference-time routing. Public adaptation uses aggregate error signals to allocate a compact model chain. Sample-specific training aligns successor windows with individual slope-based handovers; inference policies are then compared on the same frozen chain. On the tokamak benchmark, public slope control uses 30–40% fewer models than a ten-model baseline, with mean-error changes from −3.35% to +2.82%. On engine and chemical benchmarks, model savings can accompany substantial accuracy losses. A diagnostic that switches when the adjacent successor reduces true scaled root mean squared error (sRMSE) by at least five percent improves mean error and q90/q95 at forecast lead 500 over learned routing on this benchmark, but requires future targets and is not uniformly best across all baselines. Learned error predictors fail to recover these gains. Across three datasets and three backbones, generated-context training reduces step-500 mean sRMSE by 17.63%–96.24% in 27 matched comparisons. Reliable individual routing remains unresolved.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Long-horizon time series forecasting, Dynamical systems, Autoregressive prediction, Stagewise training, Adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LBJGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper studies how to train and allocate successive predictors for autonomous long-horizon forecasting.

    • Each predictor learns from contexts generated by frozen predecessors, with true future observations as supervision. The method also differentiates through the current predictor’s own rollout.
    • Public adaptation selects shared handover boundaries from validation errors. Sample-specific adaptation aligns training windows with individual handovers, then evaluates routing policies on a frozen model chain.
  • The validation combines theoretical analysis with experiments on MAST, TEP, and C-MAPSS.

    • The paper reports consistent endpoint improvements from generated-context training across TFT, DAG, and Timer-XL. Adaptive allocation provides useful model-count trade-offs in some settings but substantial failures in others.
    • A future-informed comparator outperforms learned routing on the same frozen chains. The authors correctly present this as a diagnostic rather than a deployable forecasting method.
Critical strengths
  • Generated-context training improves results consistently.

    • All 27 comparisons across three datasets, three backbones, and three stage counts favor generated contexts over observed contexts. Step-500 mean sRMSE falls by 17.63% to 96.24%.
    • The comparisons fix the backbone and stage count. Training time is higher for generated contexts, so the result supports the training strategy rather than an equal-compute advantage.
  • The allocation and routing analyses make useful practical distinctions.

    • Public adaptation does not need future targets at deployment. On MAST, the slope policy uses seven rather than ten models for DAG and Timer-XL and reduces mean error by 3.35% and 1.89%. Longer reuse of selected predictors can therefore be worthwhile.
    • The learned and future-informed routers share the same frozen forecasters. This separates whether a successor can help from whether that advantage can be detected without future observations. The paper also reports controller failures and does not equate fewer models with lower latency.
Critical weaknesses
  • The added value over existing training and allocation strategies is not established.

    • Tables 1 and 2 compare only components and variants within the proposed framework. Section 2 mentions Scheduled Sampling, Flipped Classroom, and Stratify, but none is evaluated. DaD (Venkatraman et al., 2015) learns directly from generated states and deserves explicit discussion.
    • One generated-context baseline (Flipped Classroom, Scheduled Sampling, or DaD-style training) and one allocation baseline (Direct, DirRec, or a blockwise strategy from Stratify) on the same backbone would suffice. Public code exists for Flipped Classroom and Stratify.
    • The observed-context baseline is close to teacher forcing, which is known to suffer from exposure bias. The 27/27 result is therefore largely expected and does not by itself show an advantage over these alternatives.
  • Simpler controls are missing, and the existing evidence is mixed.

    • The paper itself notes that a shared predictor given the stage label can realize the same specialization gain (Appendix A.4). A fixed schedule with the same K and a shared stage- or lead-conditioned model are therefore necessary controls.
    • The three matched-K cases point in different directions. On C-MAPSS/DAG, Linear with five models (0.01132) is about 38% better than Fixed-5 (0.01826) and also beats Fixed-10. On MAST/DAG, Linear with ten models gives exactly the Fixed-10 errors. On TEP/DAG, Linear with ten models is within 0.4% of Fixed-10 in mean but about 8% and 10% worse in q90 and q95.
    • On MAST/TFT, Fixed-5 has lower mean, q90, and q95 errors than the six-model slope policy. The current evidence does not show when adaptive handovers beat simpler capacity allocation.
  • Deployable sample-specific adaptation underperforms simple baselines.

    • PredCmp is the only deployable form of sample-specific adaptation. On every backbone, a fixed or public schedule with a similar or smaller number of models achieves lower mean error.
    • On DAG, PredCmp uses 6.8 models and is about 19% worse than Fixed-5 (0.09230 vs. 0.07758). On TFT, it uses 6.7 models and is still worse than Fixed-5 (0.03774 vs. 0.03739). On Timer-XL, it is worse than public slope with seven models (0.03630 vs. 0.03528).
    • The TrueCmp gap shows potential, but in deployable form this part of the second contribution is currently a negative result.
  • Endpoint-only evaluation does not establish full-horizon reliability.

    • All main tables report only lead 500, and q90/q95 are quantiles across cases at that endpoint. They cannot show whether a method introduces large intermediate errors near handovers. The paper itself states that a single endpoint cannot replace full-horizon evaluation (Appendix D.4). It also notes that Proposition 3.1 concerns full-horizon losses while the tables report lead 500 (Appendix C.5).
    • The temporal curves in Appendix D.4 use different settings from the main experiments. They show local error increases and crossings between adaptive and fixed schedules. Full-horizon errors from the same checkpoints and settings as the main tables are needed.
    • Adaptive gains also depend strongly on configuration. The slope policy raises TEP/TFT mean error by 91.63% and C-MAPSS/DAG error by 10.78 times relative to Fixed-10. With point estimates only, it is unclear whether the smaller favorable differences are stable.
Questions for authors
  • Comparison with existing methods

    • Could the authors compare against one representative generated-context training method and one multi-step allocation strategy? Data, inputs, backbone, and horizon should be aligned, with training and inference costs reported.
    • Does adaptive handover keep an advantage over a uniform schedule with the same model count and a shared stage- or lead-conditioned predictor? These controls would separate the effects of generated-context training, separate parameters, and adaptive boundaries.
    • How was the observed-context baseline tuned? Its errors are unstable across stage counts. MAST/DAG gives 0.48, 0.78, and 0.69 at K = 5, 10, and 20, and TEP/DAG reaches 2.01 at K = 20.
  • Complete rollout

    • For the main-table configurations, what are the full-horizon average errors and the error curves over forecast leads? Please mark handover locations and show whether endpoint gains come at the expense of earlier intervals.
    • Are the favorable adaptive results stable across repeated runs or independent test trajectories? This matters most for the small improvements.
  • Experimental protocol

    • Please report dataset subsets, split procedures, and the numbers of trajectories and forecast windows. How are the 500-step tasks built for TEP and C-MAPSS? A public split or a reproducible procedure is sufficient.
    • Please distinguish the controller and training settings of the main tables from those of Appendix D.4, and specify the learned error predictor used by PredCmp.

  • The generated-context results are consistent, but the main new contribution is adaptive allocation. Its value over established training strategies and simpler allocation controls is not yet shown, and the deployable sample-specific variant underperforms simple baselines.
  • I would consider raising my score if focused baseline comparisons and full-horizon evaluation show a meaningful, reproducible benefit. I do not require superiority in every setting or an exhaustive baseline suite. The priority is evidence that adaptive handovers help beyond generated-context training and added model capacity.
Reviewer_N3jBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework for adaptively switching between models that are trained sequentially in an autoregressive forecasting setting. Every model is trained from contexts generated by previous models.

From this framework, it introduces two methods for deciding when to switch between these models based on error signals: 1. public-boundary adaptation where validation errors define the boundary schedule shared by every test trajectory, which then determines how many separate models it uses and 2. sample-specific adaptation, where at training time, each trajectory’s error curve defines when to switch from its current model to the next shared model (“successor”). At test time, a router decides when to switch, using either a learned error predictor (“Sample-PredCmp”) or true future errors (“Sample-TrueCmp”). The authors clarify that the latter is a diagnostic and not deployable in practice.

For the public adaptation, the switching schedule is constructed offline from the validation error curve, with two rules that define the switching boundaries: 1. Linear rule, which triggers the model switch when validation error exceeds a threshold that grows linearly in time and 2. Slope rule that initiates the model switch when the current error increment exceeds λ\lambda times the median of the preceding increments.

Critical strengths

A well-motivated research question with diverse benchmarks: How long to reuse one model before switching to another is a relevant and interesting research question in long-horizon autoregressive forecasting settings. The chosen datasets (MAST, TEP, C-MAPSS) are well-suited for the problem. A consistent empirical result on generated-context training. Across three datasets and three backbones, training models on contexts generated by preceding models lowers mean sRMSE compared to observed-context baseline. The authors transparently report the additional training cost incurred by generated-context training.

Critical weaknesses

The reported model savings are confounded. The paper does not clearly explain the benefit of using lower number of models, after conceding that it “does not establish runtime savings” nor does it “measure inference latency”. For example, in Table 2(a) and D.5, public adaptation with linear and slope rule are evaluated against Fixed-10 baseline. The relevant comparison should be against fixed-K baselines with matched K and training cost.  Only DAG-Linear row allows for matched-K comparison and its results are mixed (e.g. on TEP, DAG-Linear achieves lower endpoint sRMSE but higher q90 / q95). The paper does not demonstrate the practical benefit of using sample-specific adaptation: The deployable router (PredCmp) has higher mean error than Fixed-10 on all three MAST backbones. It also lacks the relevant comparisons against fixed-K baselines with matched K and training cost. More importantly, the paper does not clarify when or why sample-specific adaptation should be preferred over public adaptation. While the two are never compared directly, Table 2 results show that public adaptation with Linear rule achieves lower mean error than Sample–PredCmp on all three backbones (and two out of three in the case of public adaptation with Slope Rule). Paper clarity could be significantly improved:Terminology is inconsistent across sections, which makes it difficult to  follow and understand the paper. The first method first appears as “public-boundary adaptation” in the abstract, then as “public adaptation” in Figure 1B and Table 2(a), “public policies” in Section 5.3 and “public controllers” in Appendix D.5. Likewise, the second method is inconsistently named “sample-specific adaptation”, “sample-specific training” and “individual training”. Symbols are also overloaded, with GG, ZZ, μ\mu, qq carrying two or more meanings in different parts of the paper. Incomplete experimental details: the paper does not report the switching rule hyperparameters used for the main results in Table 2 (e.g. model budget Kmax⁡K_{\max}, α0\alpha_0 and β0\beta_0 in Linear rule, λ\lambda, smin⁡s_{\min} and ζ\zeta in Slope rule). Basic training details for the backbones like their parameter count, optimizer, learning rate, number of epochs, batch size are also absent from the paper.  Specific architectural or training details for the error predictor used in PredCmp are not specified as well. Appendix C.3 provides a list of possible implementations for it (e.g. non-negative MLP, quantile / distributional heads, a conditional diffusion model) without stating which one of them was evaluated in Table 2(b).

Questions for authors

Could you compare public-boundary adaptation method with Linear / Slope rule against a fixed-K baseline with the same number of models KK (e.g., Fixed-3 for TEP slope with TFT/DAG), and report their training cost (e.g. training hours, FLOPs)? Can you specify which error predictor was evaluated in Table 2(b) in terms of model family / architecture, training objective (e.g. sRMSE, MSE) and how its training examples were constructed?

Reviewer_KL5LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates long-horizon autoregressive time series forecasting by training sequential stagewise predictors on generated contexts and evaluating heuristic public boundary controllers alongside sample-specific routing policies across three physical and engineering datasets.

Critical strengths
  1. The paper provides a transparent empirical comparison between observed-context and generated-context stagewise training, which is consequential because it consistently demonstrates the importance of mitigating exposure bias during multi-stage unrolls.
Critical weaknesses
  1. Both of the two proposed adaptive mechanisms are empirically unreliable, which is inevitable because the core methodological contribution fails to deliver robust practical advantages.
  2. The evaluation omits essential baselines and leaves out critical hyperparameters while mixing incompatible parameter settings between the main tables and appendix figures, which severely undermines both experimental completeness and reproducibility.
Questions for authors
  1. Why do Appendices D.3–D.4 rely on different parameter settings?
  2. How does a single shared backbone conditioned on the step or stage index perform relative to maintaining K∈{5,10,20}K \in \{5,10,20\} separate parameter copies in Table 1 and Table 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.