acceptodds
Under review as a conference paper at ICLR 2027

Support-Only Hindcasting for Test-Time Neural Operator Selection

Abstract

Test-time neural operator selection chooses an operator or short composition from a frozen codebook using a short observed trajectory. Existing Neural Operator Splitting (NOS) search ranks candidates by one-step fitting error inside this observed context, whereas deployment exposes the selected candidate to accumulated closed-loop error. We introduce a training-free criterion for reranking a frozen NOS candidate beam. Each candidate is restarted from several earlier observed states, rolled forward autoregressively inside the observed prefix, and ranked by its mean error against the corresponding suffixes. The winning candidate is then deployed into the unseen future. We bound future normalized root-mean-square error (NRMSE) over finite horizons using PDE energy estimates for true-flow stability. Experiments reveal that support-only reranking significantly improves conventional NOS across all three PDEs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Test-time Generalization, Partial Differential Equations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7aQQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HINDCAST-NOS, a support-only reranking method for Neural Operator Splitting (NOS) search. Conventional NOS selects candidates by one-step fitting error, which mismatches the autoregressive rollout used at deployment and can reverse the candidate ranking even for linear autonomous dynamics. HINDCAST-NOS instead restarts each retained candidate from multiple earlier observed states, replays it freely over the observed suffix, and ranks candidates by the resulting mean closed-loop error. Experiments demonstrate the improvements on multiple PDEs, and the authors establish a finite-horizon transfer bound from observed hindcast error to future error theoretically.

Critical strengths
  1. The motivation is both concrete and theoretically grounded. The paper identifies a clear objective mismatch between the one-step fitting criterion used to select operator candidates and the autoregressive rollout used at deployment.
  2. The proposed method is easy to implement. It requires no additional training or gradient computation.
Critical weaknesses
  1. My principal concern concerns the main experimental results. In Table 2, the metrics for the proposed method are very close to those of the baseline NOS. Although the differences are consistent in direction, their magnitude is small enough that the improvement may not be practically meaningful. As presented, the results make the method appear to be a refinement that is effective in a narrow sense but whose benefit in deployment is unclear.
  2. The experiments lack several simple test-time baselines. For example, the authors should compare HINDCAST-NOS against a random selection from the retained beam, the average prediction over the beam (a test-time ensemble), and so on. Without these controls, it is unclear whether the gain comes from the hindcast-based reranking itself or from the variance reduction that any multi-candidate averaging provides. Moreover, the maximum evaluation horizon is only H=34. Since the claimed benefit concerns accumulated autoregressive error, the authors should report results at longer horizons or explain why H=34 is sufficient.
Questions for authors

My main questions are listed in the Critical Weaknesses. Besides, the practical value of Theorem 4.1 is limited. The error radius depends on constants δc\delta_c and qcq_c, and the manuscript does not estimate their magnitudes for the systems studied. As a result, the bound cannot be used to explain the size of the gains observed in deployment, nor to predict when the hindcast-based reranking should outperform one-step selection. The author can relate the bound to the empirical improvement more clearly or provide estimates or admissible ranges for these constants.

Reviewer_xgN9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HINDCAST-NOS, which reranks NOS candidates based on autoregressive rollout errors on observed trajectories. The authors provide theoretical analysis and report small improvements over NOS on three PDE benchmarks.

Critical strengths

The method shows consistent, although relatively small, improvements on three PDE benchmarks without additional training and with little extra computational cost.

Critical weaknesses

The method is mainly a small modification of NOS. It uses the same pretrained model, codebook and beam search, and only changes the final selection criterion from one step error to rollout error. This idea seems quite straightforward. Also, the relative reduction in NRMSE compared to NOS is only about 0.6% to 2.5%. I am not convinced that this is a sufficiently significant contribution for a standalone paper.

Theorems 4.2 and 4.3 rely on a simplified 2D linear model with positive error memory. However, it is unclear how these assumptions relate to the neural operators used in the experiments. Theorem 4.1 is more general, but the authors do not show whether the theoretical bound is meaningful in practice. Therefore, it is unclear how much these theoretical results actually explain the empirical improvements.

Questions for authors
  1. In Table 8, the improvement of multi origin over single origin is very small, especially for Advection Diffusion and Burgers. Are these improvements statistically significant?

  2. Theorems 4.2 and 4.3 rely on a special linear model with positive error memory. Is there any evidence that the learned neural operators in the experiments satisfy these assumptions? If not, how do these theoretical results support the experimental findings?

  3. Are the reported results based on a single pretrained model and codebook for each PDE benchmark? Have the authors tested different random seeds or codebooks? Since the reported improvements are relatively small, it would be useful to know whether they are consistent across different runs.

  4. The reproducibility statement mentions that the code and per-target experimental outputs are included in the supplementary material. However, I cannot find any supplementary material or code in the submission. Could the authors clarify where these materials are available?

Reviewer_ePogOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Forecasting the future of a physical system from a short observed trajectory requires a model that remains accurate over several prediction steps. When several pretrained models are available, we need to choose one using only these observations. The model with the smallest next-step error may not be the best when its predictions are repeatedly fed back as inputs. This paper studies this selection problem for neural operators, which learn maps between spatial fields, such as velocity or temperature fields, and can be used to model PDE dynamics.

The proposed HINDCAST-NOS method reranks candidate operators or short operator compositions produced by Neural Operator Splitting (NOS). Each candidate starts from earlier states in the observed trajectory and predicts subsequent observed states autoregressively. The candidate with the lowest average replay error is then used to forecast the unseen future. The method keeps the pretrained models and candidate search fixed. The authors provide conditional theoretical guarantees and evaluate the selection rule on three PDE forecasting tasks.

Critical strengths
  1. The paper is clearly written. The distinction between one-step fitting and autoregressive prediction is explained well. I also like the small theorems and examples; they clarify what happens, especially why a model with a slightly larger one-step error can still be preferable if its errors decay faster. The theoretical results are useful for understanding the selection rule, with the strongest conclusions established in a simplified model.

  2. The improvement in selecting the best candidate is good. In Table 3, the fraction of targets for which the method selects the candidate with the lowest future error increases from 65.6% to 86.7% on advection-diffusion and from 64.8% to 79.7% on Navier–Stokes. In my opinion, this is the strongest empirical result.

  3. The method is simple and inexpensive to apply. It requires no additional training, and the comparison with NOS keeps the candidate pool and deployment procedure fixed. This makes the effect of the selection rule easy to interpret.

Critical weaknesses

Major:

  1. My main concern is novelty and the absence of a proper related-work section. Selecting a forecaster using its multi-step errors on an observed part of a time series is a familiar idea. Tashman (2000) discusses rolling-origin evaluation and its use for method selection. More specifically, McElroy and Findley (2010) compare ARIMA models through their multi-step forecast errors and discuss criteria combining several forecast horizons. The overlap is the basic selection principle: assess candidates using errors over the prediction horizon of interest.

    The specific setting here is selection among frozen operators and their compositions from a short trajectory. The paper should explain what additional challenge this setting introduces and how the proposed method or analysis addresses it. The short background section mainly covers neural operators and their adaptation, leaving the connection to forecasting and model selection largely unexplored. A proper related-work section would help distinguish the contribution of the selection criterion, the theoretical analysis, and the application to NOS. At present, I see a useful improvement to NOS, but the methodological novelty is not sufficiently clear.

Minor:

  1. The average forecasting improvements are modest. In the main configuration, the relative reductions in mean NRMSE are about 2.48% on advection-diffusion, 1.16% on Navier–Stokes, and 0.61% on Burgers. Some individual trajectories and the wider-beam experiment show larger gains. Still, the substantial improvement in selecting the best candidate translates into a fairly small average reduction in forecasting error. I consider this a minor weakness, given the low overhead. Reporting how much the candidate errors differ would help explain this result.

  2. The abstract could be easier to understand for a broader audience. It starts immediately with “test-time neural operator selection,” followed by a frozen codebook and operator compositions. I would start with the broader problem of forecasting unfamiliar physical dynamics from a short observed trajectory, briefly explain what a neural operator is, and then introduce the selection problem. The motivation becomes clear in the introduction, but the abstract should provide enough context on its own.

  3. Proper references Please update the references to the published versions of NOS (ICML 2026), DISCO and Zebra (ICML 2025), FNO (ICLR 2021), and DeepONet (Nature Machine Intelligence, 2021), unless a specific preprint version is needed. The DeepONet journal version also has a different title and author list.

References:

  1. Leonard J. Tashman. Out-of-sample tests of forecasting accuracy: an analysis and review. International Journal of Forecasting, 16(4):437–450, 2000.

  2. Tucker S. McElroy and David F. Findley. Selection between models through multi-step-ahead forecasting. Journal of Statistical Planning and Inference, 140(12):3655–3675, 2010.

Questions for authors
  1. What additional challenge does selecting frozen operator compositions from a short trajectory introduce relative to existing multi-step forecast model selection? Which part of the proposed method or theory addresses that challenge?

  2. Can the authors report the mean gap to the best future candidate in the retained beam, before and after reranking? This would help explain why the substantial increase in the best-candidate selection rate produces a relatively small decrease in average forecasting error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.