acceptodds
Under review as a conference paper at ICLR 2027

When Does Forecasting Reveal Temporal Structure? Recoverability and Stability of Forecast-only Explanations

Abstract

Forecast accuracy is often used as a proxy for temporal structure discovery, but predictive performance and structural recoverability are not equivalent. Forecast-only structural selection—choosing among candidate temporal structures based solely on their forecasting performance—provides a practical way to identify predictive temporal dependencies for model interpretation, diagnosis, and downstream decision making. In this work, we study when forecast-only structural selection can be trusted. We show that a vanishing forecast margin does not necessarily imply structural ambiguity, and establish a stability perspective that relates reliable forecast-only structural selection to the margin's robustness against uncertainty in the selection objective. Experiments across controlled and end-to-end settings demonstrate that forecast margin alone is insufficient, while the proposed stability condition provides a sufficient guarantee for correct forecast-only structural selection. We further give a model-based certification procedure that uses independent data to check the selected structure under a declared structural family, abstains when the evidence is insufficient, and does not require knowing the candidates' ideal risks. Our results suggest that predictive accuracy can support structural interpretation when its advantage is sufficiently robust, or when the selected structure is independently certified under a declared structural family. More broadly, our framework provides a principled basis for evaluating when forecast-derived temporal structures can be trusted as model explanations in theoretical analysis and finite-candidate time-series applications.

Primary area:
interpretability and explainable AI
Keywords:
Time-series forecasting, Interpretable time series, Structural recoverability, Forecast stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MKmsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the question of how to determine the reliability of a temporal structure selected solely based on predictive performance. The motivation stems from the fact that predictive accuracy is not equivalent to structural recoverability; furthermore, model fitting errors and fluctuations in finite-sample evaluation can lead to incorrect structure selection. To this end, the authors analyze the relationship between predictive performance and correct structure selection across three dimensions: structural recoverability, prediction intervals, and selection stability. Specifically, the authors derive the relationship between structural recoverability and prediction intervals using Gaussian temporal models, establish sufficient conditions for stability based on prediction intervals and objective function distortion, and propose a post-selection structure validation method utilizing independent data and likelihood ratio inference. Experimental results—based on controlled Gaussian time series, OLS, MLP, and semi-synthetic industrial data—validate the theoretical findings. They demonstrate that the stability conditions guarantee correct selection when satisfied, while the independent validation method can verify certain candidate structures under specified model assumptions and abstain from making a decision when evidence is insufficient.

Critical strengths
  1. The paper focuses on the relationship between prediction accuracy and the recoverability of temporal structure; the research topic holds both theoretical significance and practical value.

  2. The paper distinguishes between model-fitting error and evaluation fluctuations arising from finite samples, and analyzes their respective impacts on the stability of structure selection.

  3. The paper proposes a structure validation procedure based on independent data; this method controls the probability of erroneous validation under explicit model assumptions and allows for abstention when evidence is insufficient.

Critical weaknesses
  1. There is a lack of sufficient logical connection among the three theoretical issues; in particular, the structural certification method in Section 4.4 is not derived from the preceding stability analysis, making the paper resemble a collection of related yet relatively independent studies rather than a cohesive whole.

  2. The theorem is essentially an application of general perturbation inequalities to the problem of selecting from a finite set of candidate structures; it fails to fully leverage the unique properties of time series or explore specific issues inherent to the time series domain.

  3. Section 4.4 primarily employs the existing "Universal Inference" method but fails to explain the rationale behind the two-step process—selecting the structure based on predictive MSE followed by certification via likelihood testing—and lacks a sufficient comparison with direct statistical structure selection methods.

  4. The structural certification controls the joint probability of an incorrect selection being certified, rather than the conditional error probability given certification; furthermore, the certification guarantees rely on a specified family of linear Gaussian structures, limiting the method's applicability to general time series interpretation tasks.

Questions for authors
  1. Is there a more direct theoretical link between the theoretical results in Sections 4.1–4.3 and the structural certification method in Section 4.4? In particular, what role does the previously proposed stability ratio play in the final certification method?

  2. Compared to standard perturbation bounds in general finite-candidate model selection, what new conclusions or theoretical guarantees specific to time series does Theorem 4.3 provide?

  3. What unique advantages does the two-step approach—selecting the structure based on predictive MSE followed by independent certification—offer over directly identifying the true lag using likelihood tests or statistical model selection methods? Could you provide a comparative analysis or experimental results to demonstrate this?

  4. Since Δmin⁡\Delta_{\min} and τtotal\tau_{\mathrm{total}} depend on the ideal risk, how can one determine whether this condition holds in practical applications? What useful guidance does the theory offer for the many cases where this condition is not met, yet the correct structure is nonetheless selected?

  5. The existing theory controls only the joint probability of erroneous certification; could the authors further elaborate on the actual behavior of the conditional error probability post-certification, and discuss whether this guarantee remains practically meaningful when the certification rate is low?

Reviewer_WJwhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when temporal structures selected solely by forecasting performance can be interpreted as reliable explanations. The paper distinguishes three notions that are often conflated: structural recoverability from the full observed trajectory, the forecast margin between candidate temporal structures, and the stability of empirical forecast-based structural selection. In controlled Gaussian delay models, the paper shows that a vanishing per-observation forecast margin does not necessarily imply structural ambiguity, because accumulated trajectory-level information may still permit reliable delay recovery.

The theoretical core is a sufficient stability condition: if the minimum ideal forecast margin exceeds twice the total distortion between ideal risks and the empirical selection objective, then the true candidate structure remains the unique empirical minimizer. Since this oracle condition is not directly computable, the paper additionally proposes a post-selection certification procedure. The selected delay is treated as a forecast-based proposal, and an independent certification trajectory is used with split/profile likelihood inference to either certify the proposal under a declared linear-Gaussian candidate family or abstain.

The paper gives a clearly specified analysis and evaluation protocol within the stated P1/P2 controlled settings. However, I have concerns about the restrictive scope of the computable certification procedure, the limited empirical evidence for the paper's broader claims about forecast-only explanations in realistic time-series applications, and the practical feasibility of the required trajectory-isolation protocol.

Critical strengths

The paper provides a clear theoretical formulation of the distinction between forecast-based candidate selection, structural recoverability, and post-selection certification in the controlled finite-delay setting. The use of an independent certification trajectory is also a sensible attempt to avoid directly equating the lowest forecast-error candidate with a validated temporal structure.

The experiments are generally aligned with the stated controlled setting and provide useful evidence for the proposed selection-and-certification workflow. However, the main concerns below relate to the scope of the certification guarantee, the generality of the empirical evidence, and the practical availability of the required isolated trajectories.

Critical weaknesses
  1. How robust is the computable certification procedure to model misspecification and to the gap between the declared family and the intended explanatory target?

The certificate is valid only when the true conditional law belongs to the declared candidate family HdH_d, the numerator density is frozen before certification, the boundary conditioning is correct, and the exogeneity and data-isolation assumptions hold. In the reported experiments, the candidate structures are delays in a linear-Gaussian P2 family, while the MLP is used only as a frozen numerator. Thus, the theoretical guarantee does not certify arbitrary nonlinear temporal structures, neural temporal masks, attention patterns, or the explanations generated by contemporary forecasting models.

This scope is important because the introduction and conclusion frame the work broadly around ``forecast-only explanations'' and temporal-structure interpretation. In practice, candidate families may be nonlinear, partially misspecified, non-Gaussian, nonstationary, or non-identifiable. Under such conditions, a singleton retained set need not correspond to the intended physical or causal structure; it may only be the most compatible candidate within a misspecified finite family.

Could the authors provide robustness experiments in which the certification model is deliberately misspecified? Useful cases would include nonlinear dynamics, multiple active lags, non-Gaussian innovations, heteroskedasticity, weakly violated exogeneity, omitted lags, and candidate families that do not contain the true temporal mechanism. At minimum, the paper should more prominently delimit its claim: the procedure certifies a forecast-selected candidate only relative to a correctly specified, declared structural family, rather than validating general neural time-series explanations.

  1. Does the empirical evaluation support the paper's broader practical claims beyond controlled delay-recovery settings?

The empirical studies are careful but highly controlled. The main P1/P2 experiments use Gaussian mechanisms with known ground-truth delay structures. The Tennessee Eastman Process experiment uses realistic industrial-process background trajectories, but constructs a new target by injecting a known delay, Y(new,t). As the paper itself notes, this semi-synthetic study is not a benchmark for recovering physical process delays.

This semi-synthetic experiment is useful because it tests whether the workflow can recover a controlled temporal structure on realistic-looking industrial background dynamics. However, recovering an experimenter-injected delay is not equivalent to discovering an originally unknown temporal relationship in a real industrial system. The current experiments therefore validate recovery of known or implanted finite-delay structures, but provide limited evidence that the method can identify naturally occurring, unknown physical, causal, or operational temporal mechanisms.

The paper does include useful predictive comparators, such as the HAC-based frozen-predictor risk comparison and the same-evaluation-budget reselection baseline. However, these comparisons mainly clarify the role of the certificate and do not provide a systematic comparison with established approaches for lag selection, delay estimation, temporal explanation, or uncertainty-aware structural selection. Could the authors add either real-data case studies with credible external validation of temporal relationships, or a more systematic benchmark involving nonlinear, multivariate, nonstationary, and multiple-lag candidate structures? If such experiments are outside the current scope, the claims should be narrowed from a general framework for forecast-only explanations to a principled analysis of finite-candidate delay selection under controlled assumptions.

  1. How practical is the requirement for independent training, selection, and certification trajectories in a real dependent time series?

The certification guarantee requires three isolated trajectories: one for fitting candidate predictors (AA), one for forecast-based structural selection (BB), and one for certification (CC). The paper correctly does not assume that transitions within one continuous trajectory are independent; instead, its likelihood argument is formulated at the conditional trajectory level. In the simulated settings, the required trajectory-level isolation is achieved by generating separate continuous trajectories with independent random streams.

However, many real time-series applications provide only one long dependent trajectory, or a small number of correlated runs, batches, devices, or units. Simply splitting a continuous record into consecutive temporal blocks may not produce the trajectory-level independence required by the guarantee, especially under long memory, repeated operating regimes, nonstationarity, shared latent disturbances, or distribution shift.

The paper should discuss more concretely how its data-isolation requirement could be implemented, approximated, or audited in practice. For example, can the method be applied across independent units, repeated experimental runs, production batches, wafers, lots, devices, or well-separated temporal blocks? If only a long time series is available, how should practitioners choose temporal gaps, assess residual dependence, or diagnose distribution shift between the selection and certification periods? What happens when the certification trajectory is only approximately independent?

Without such guidance, the practical applicability of the certificate may be limited to settings with repeated independent trajectories or strong stationarity and isolation assumptions.

Questions for authors
  1. The paper should distinguish more consistently among ‘structural recovery,’ ‘correct candidate selection,’ ‘forecast-only explanation,’ and ‘physical or causal interpretation.’ Recovering a delay label within a declared finite family is not automatically equivalent to identifying a physical mechanism or causal relation.

  2. The title and introduction frame the work broadly in terms of forecast-only explanations, while the formal analysis and certification procedure focus primarily on prespecified finite delay candidates. Please make this limitation explicit earlier in the introduction and abstract.

  3. Table~1 reports many selections that share fitted predictor pairs and prefix trajectories. The appendix explains this dependence carefully, but the main text could state more prominently that the entries are not independent replications and should not be read as ordinary independent-sample counts.

  4. The certification experiments report zero observed erroneous certifications in several settings, but the MLP experiments use only 40 independent blocks per mechanism in some rows. The appendix correctly gives conservative upper bounds, yet these bounds should be emphasized in the main discussion when interpreting apparent zero-error results.

  5. It would help to provide a concise algorithm box for the full practical workflow: declaration of the candidate family, construction of trajectories AA, BB, and CC, candidate fitting, forecast-based selection, freezing of the numerator and variance estimate, construction of the confidence set, and the certify-versus-abstain decision.

Reviewer_4broOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when a model structure identified solely by forecasting performance is reliable. The paper precisely defines and distinguishes the potential issues involved, and analyzes methods for identifying model structure under different settings. Overall, I think the research question is quite meaningful. Although the authors derive the detector from simple assumptions, it can also be applied to models in a general sense. However, the proposed method has significant limitations in practice.

Critical strengths

The paper has clear definitions and accurately captures the distinction between predictive accuracy and structural accuracy. It also analyzes and distinguishes different sources of structural selection error, and demonstrates the ideal candidate risk, the overall risk of the fitted predictor, and the selection fluctuation under finite selection.

Critical weaknesses

I think there are mainly three weaknesses. First, the proposed method has significant limitations in practical use: on the one hand, it depends on the performance of the frozen predictor; on the other hand, it only applies to scenarios where the true structure is already included in the candidate structures, which is often unrealistic. Second, in nonstationary settings, the applicability of the method may be compromised. Third, the validation scenarios in the experimental section are relatively weak, mainly focusing on linear relationships, with no investigation of nonlinear relationships.

Questions for authors
  1. What happens to the proposed algorithm when the frozen predictor performs poorly? 2. Can it be extended beyond time series, for example to feature selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.