acceptodds
Withdrawn from ICLR 2027

Historical Information in Forecaster Selection: Replay, Retention, and Candidate Exclusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Historical archives contain reports unavailable when forecasts would have been issued. These reports can change both candidate rankings and which checkpoints survive for final selection. We vary the information available at retention and final selection while holding fitted trajectories fixed and testing selected forecasters on common deployment inputs. Our main study predicts vehicle positions 60 seconds ahead across eight refitted episodes at two transit agencies, SEPTA and VTA. Retrospective selection increases position error in seven episodes. At equal additional realistic evaluation counts, retaining runner-up checkpoints and replaying fewer dates improves mean error over complete replay of one winner per group in six episodes, with agency-average gains of 4.48 m at SEPTA and 0.10 m at VTA. Restricting the library to origin-available fits remains competitive. Finitecandidate analysis separates exclusion, replay-score error, and changes in candidate performance between validation and testing. Full-replay controls distinguish the benefit of wider retention from the cost of reduced date coverage. Accurate rescoring cannot recover a discarded alternative, while retaining more candidates need not improve deployment loss. Retention and final selection must therefore be evaluated together.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, model selection, retrospective validation, data availability, delayed observations, data revisions, temporal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MLnaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I do not understand the paper at all.

Critical strengths

NONE.

Only if I could undersatnd the paper can I even judge it. I see no critical strengths as the entire paper was completely incomprehensible.

Critical weaknesses

The paper is incomprehensible as written, so it cannot be evaluated

The examples below are taken from the first few pages; the same pattern continues through Section 3, after which I could no longer follow the paper. I list them so the authors and the ACs can see the problem concretely. Specific examples:

ABSTRACT

At equal additional realistic evaluation counts, retaining runner-up checkpoints and replaying fewer dates improves mean error over complete replay of one winner per group in six episodes, with agency-average gains of 4.48m at SEPTA and 0.10m at VTA.

What is 4.48m and 0.10m? What is "m" here? What is an episode?

Restricting the library to origin-available fits remains competitive

Unclear what this means

The abstract then ends with

Finite-candidate analysis separates exclusion, replay-score error, and changes in candidate performance between validation and testing. Full-replay controls distinguish the benefit of wider retention from the cost of reduced date coverage. Accurate rescoring cannot recover a discarded alternative, while retaining more candidates need not improve deployment loss. Retention and final selection must therefore be evaluated together

I have no idea what any of this means. There is no context given. If the authors would like to convey a point, the abstract should be kept high-level instead of filling it with jargons and terms that have not been defined.

INTRO

Data vintages affect estimates and model rankings

What is a data vintage? The term has not been defined.

does realistic evaluation still have access to the candidates needed for a good deployment decision?

Who has access here? What is exactly meant by "Evaluation has access to candidates"?

Replaying validation with origin-available inputs can correct the ranking of survivors, but cannot recover a discarded checkpoint

Unclear what this means

Restricting the library to origin-available fits is another response, although it may remove useful models trained on more complete historical records. We compare these choices by the selected forecaster’s deployment loss.

What is an origin-available fit (not defined earlier)? What is a forecaster's deployment loss? Is it the loss in future windows not used in the test set at model selection time? All this is unclear.

We compare origin-available (A) and retrospective (F) information at two stages: inner validation retains candidates, and outer validation selects one for deployment. In AA, AF, FA, and FF, the first letter specifies retention; the second specifies final selection. All four procedures use fixed fitted trajectories and common A test inputs (Figure 1). Matched libraries help distinguish which candidates are available from how many survive and how well they score on inner validation

It is completely unclear what this means. What is "origin-available" and what is retrospective? The whole papers hinges on these terms and they are not defined clearly.

  • Figure 1 is confusing. I don't undersatnd a thing.

Our contribution is an empirical account of candidate survival, supported by finite-candidate analysis. The crossed design distinguishes retention from final ranking; matched libraries separate composition from candidate count and inner-validation quality. Full-replay controls then separate the benefit of retaining more candidates from the cost of evaluating fewer dates: at SEPTA, widening gains 6.20 m, while reduced coverage gives back 1.73 m. The analysis explains these outcomes through exclusion, replay-score error, and validation-to-test transfer. These are diagnostics computed after testing; the evaluated policies select using only pre-test information.

The motivation itself is unclear from what's written so far. The last paragraph of the intro (quoted above) discusses the authors' contributions and it is even more unclear. What is SEPTA / VTA? The abstract says they are "transit agencies" but doesn't provide references. What are they? Datasets?

The paragraph doesn't give any information on what the paper does - is it an analysis? A method? What are the takeaways? What are the results?

ON RELATED WORK:

  • There is a lot of incomprehensible text in the related work as well.

Paired replay uses difference estimation (Särndal et al., 1992), related to proxy-plus-correction inference (Angelopoulos et al., 2023). Whereas proxy-assisted LLM ranking studies preference uncertainty (Chatzi et al., 2024), our endpoint is selected forecasting loss

What does this mean?

matched subset-only controls isolate the value of auxiliary scores.

What does this mean? None of these terms are defined.

Section 3

  • The entire section goes on to use jargon and terms that are not defined at all. It has propositions and terms that contain no meaning (with all the context given before). I can pick any sentence at random to show this.

Section 4 on wards

  • The experimental section is also incomprehensible; I have no idea what is being said here.
  • Only meaningful thing I got from this section is that they used the Monash time series somehow with a train-val-test split but it is unclear so far: what the experimental setting is, what the goals are, what models they are evaluating
  • Due to that, I am unable to undersatnd any of the findings of the paper until the conclusion.
Questions for authors

See weaknesses. I have nothing more to ask/clarify as I did not understand anything.

Reviewer_Qkq6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how information that appears in historical records only after a forecast would have been issued (late-arriving or revised reports) affects model selection for time-series forecasters. The authors split selection into two stages: retaining candidate checkpoints from groups of trained models, and then selecting a final model from the surviving candidates. They vary whether each stage uses origin-available or retrospective information (the AA, AF, FA, and FF configurations), while holding the fitted candidates fixed and testing the selected models on common deployment inputs. They decompose the resulting loss into a candidate-exclusion component and a selection component among surviving candidates, and they study, under a fixed evaluation budget, the trade-off between retaining more candidates and replaying them over fewer dates.

The main experiments predict transit vehicle positions 60 seconds ahead at two agencies (SEPTA and VTA) over eight episodes with three fitted seeds. Retrospective selection increases position error in seven of eight episodes (e.g., SEPTA mean error rises from approximately 431 m under AA to 466 m under FF). Wider retention with shorter replay yields agency-average gains of approximately 4.48 m at SEPTA (about 1%) and 0.10 m at VTA (about 0.05%); restricting the library to origin-available fits remains competitive. Additional experiments on the Electricity and Traffic datasets use artificially imposed information delays. The authors conclude that replay cannot recover discarded candidates, that wider retention helps only when it improves the loss of the deployed model, and that retention and final selection must be evaluated together.

Critical strengths
  1. Important distinction between event time and information-availability time. The paper correctly separates when an event occurred from when information about that event actually became available, a distinction that a conventional chronological split does not necessarily capture. This is the main reason I consider the problem legitimate and worth studying.

  2. Careful experimental isolation of model selection. The authors hold the fitted candidate models fixed while changing only the information used during the selection stages, and they evaluate the resulting choices on common test inputs, across multiple episodes, repeated seeds, and additional diagnostic comparisons. The decomposition into candidate exclusion and selection among surviving candidates is a reasonable way to analyse the results. There is clearly a substantial amount of work involved.

My concern is not that the problem is imaginary or that the authors have not done the work; it is that I am not convinced the work adds enough to what is already known (see weaknesses).

Critical weaknesses
  1. Limited novelty and insight. The fundamental premise, that selecting models using information that would not have been available when the forecasts were issued can lead to the wrong model, is something we already know. In my own research on underwater acoustic sensing, I go to considerable lengths to ensure that development and evaluation reflect only the information the deployed sensor will actually have: AIS, weather, or bathymetry may be available during development, but I cannot assume they are available when evaluating how the deployed system will operate. The authors examine a slightly different problem (late-arriving information affecting selection rather than model inputs), but the underlying principle is the same: models must be evaluated under the information conditions that actually hold at the time. The paper references earlier work on historical data availability and realistic forecasting evaluation, yet it does not clearly explain what gap remains in that literature and how this work fills it.

The conclusions reinforce this concern. That replay cannot recover a candidate that has already been discarded, and that retaining more candidates is beneficial only if it improves the loss of the deployed model, are both reasonable but close to self-evident. The retention-versus-evaluation-budget trade-off (more candidates, but less reliable information about each) is likewise intuitive; the authors quantify it, which is useful, but do not establish a general or surprising result. The loss decomposition formalises these relationships, but the paper does not show what the framework allows us to do that we could not already do: it does not provide a better selection algorithm, a rule for how many candidates to retain, or a useful decision rule. After reading the paper, I was left asking what I should now do differently when developing or selecting forecasting models, and what specific mistake this work helps me avoid that established validation practice does not already address.

  1. Limited practical significance and generality. The wider-retention gains are approximately 4.48 m at SEPTA (about 1%) and 0.10 m at VTA (about 0.05%), against position errors of several hundred metres. The authors do not establish that such differences are operationally meaningful, and an improvement of one-tenth of a metre is particularly difficult to justify as an important practical result. Moreover, restricting the model library to origin-available fits remains competitive and, in some comparisons, performs better than wider retention. This raises the question of why one would not simply ensure that the information used for selection reflects what would actually have been available, rather than adopt the more complicated retention-and-replay procedure. The authors also acknowledge that wider retention does not always improve the result and that fewer replay dates do not always make it worse, so the relationship depends on the particular models, data, and validation conditions. With two transit agencies, eight episodes, three seeds, and artificially imposed delays for Electricity and Traffic, it is unclear how far the findings generalise, and whether the paper proposes a broadly useful method or primarily describes what happened in these particular experiments.

Additional concern (clarity). The paper is considerably more difficult to read than it needs to be, particularly for the broad ICLR audience. The underlying problem is simple to state, yet it is introduced through substantial terminology (origin-available, retrospective, retention, fitted identity, rescoring, replay) and notation before the reader has a clear picture of what is happening and why it matters, with little connection to familiar concepts such as trained checkpoints, candidate pruning, validation, and model selection. "Fitted identity" appears to mean a specific trained model instance (configuration plus checkpoint) and does not need such an unusual name. The AA/AF/FA/FF notation would benefit greatly from a small worked example with three or four candidates showing which survive each stage. In Section 3.2, the meaning of L_AA and L_FF is embedded in the mathematical discussion rather than explained plainly beforehand; Section 3.3 introduces the evaluation budget, candidates, validation blocks, and forecast queries in rapid succession; and the error metric in metres should be defined explicitly in terms of predicted and actual vehicle positions. This makes it harder to identify and appreciate whatever contribution the authors are making.

Questions for authors

Questions that could materially affect my assessment:

  1. What precise gap in the existing literature on historical data availability and realistic forecasting evaluation does this paper fill? Which conclusion here could not already be reached from that prior work?

  2. What does the candidate-exclusion / selection-loss decomposition enable in practice? Does it yield a better model-selection algorithm, guidance on when to retain additional candidates, or a decision rule that a practitioner could apply to decide how many candidates to retain in a new forecasting problem?

  3. Given that restricting the library to origin-available fits is competitive and sometimes better, what practical advantage does the retention-and-replay procedure offer over simply ensuring that all selection uses information that would actually have been available at the time?

  4. Are improvements of approximately 4.48 m (about 1%) at SEPTA and 0.10 m at VTA operationally meaningful for transit vehicle position forecasting with errors of several hundred metres? Can the authors provide an application-grounded argument for this?

  5. Are the authors proposing a broadly useful method, or primarily characterising the behaviour observed in these experiments? Would the same trade-offs appear with different forecasting models, different (naturally occurring rather than imposed) information delays, a substantially different application, or a differently constructed candidate library?

Clarification requests:

  1. Please define the position-error metric explicitly in terms of predicted and actual vehicle positions.

  2. Please provide a small worked example (three or four candidates) illustrating which candidates survive each stage under AA, AF, FA, and FF.

  3. Is "fitted identity" simply a specific trained model instance (configuration plus checkpoint)? If so, could standard terminology be used instead?

Reviewer_UStxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the effects of retrospective information and checkpoint retention on final forecaster selection. The studied mechanisms include historical information availability, information-availability aware replay, and fixed-budget retention comparisons.

Critical strengths
  • Mathematical results seem sound

  • The experimental comparisons are careful and controlled

Critical weaknesses
  • Paper's clarity needs substantial improvement.

  • My current understanding is that the main paper conclusion reads as "audit the information available during selection and preserve alternatives for reevaluation". But this is a rather common precaution against look-ahead bias and premature candidate elimination.

  • Related work section does not establish the novelty of the current work and important references are missing

  • The significance of Propositions 1,2 remains unclear. While the propositions formalize intuitions, their additional explanatory value and connection to the evaluated heuristics are unclear.

  • The connection between experiments and the actual claims is insufficiently explained

Questions for authors
  • Key terminology (replay, retention, exclusion) must be defined in the intro. Please explain the initial retrospective evaluation and subsequent realistic replay more clearly. Please also strengthen the motivation for proposed study. If you could present motivating anecdotal examples of when what you study can make a real profound effect on decision making would help substantially.

  • Currently, RQ1-RQ3 broadly describe experimental agenda. Instead, they should be stated explicitly as questions to either establish an empirical effect, investigate its mechanism, or test a specific intervention.

  • RQ1 is trivial as stated. "historical information changes selected forecasters and their test loss across refitted episodes". If the information changes, we know that the answer is yes, right? A better statement of the research question is needed. For example, we might be interested in quantifying the direction, magnitude and mechanism, not yes/no question.

  • Figure 1 is very hard to understand as insufficient context is provided in intro or caption. Instead of helping me appreciate the core paper contribution it confused me. For example, I don't understand what Top-1 is vs Top-2 and why Top-1 is 5x7 whereas Top-2 is 7x5. Why Top-2 produces seven candidates and not ten? Which groups receive extra checkpoints, and why this comparison is central?

  • The paper's actionable insights beyond standard practices are unclear. What novel generalizable understanding do your experiments contribute beyond standard evaluation precautions?

  • Related work section does not do a good job at positioning the current work in the literature. The novelty of the work with respect to existing papers is unclear. Algorithmically, the current paper seems to be related to multi-arm bandit literature on selection under limited evaluation resources. Could you please incorporate references [1-3] in literature review and provide positioning of your work with respect to those?

  • What is the significance of Proposition 1 with respect to the development of the proposed method? What methodological choice does the proposition inform?

  • The paper does not connect Proposition 2 to its proposed checkpoint-retention rule or evaluation budget. What is its role in developing or selecting the proposed method?

  • Several conclusions from experiments are structural observations. For example, "Replay cannot recover an excluded candidate" is true, but does it really require an experiment? Seems like this holds trivially by construction. The authors should distinguish observed outcomes from general methodological conclusions, and explain what each result implies. For example, Top-2 improves over full Top-1 in some settings. But A-only restriction is better on repeated SEPTA, and VTA gains are tiny. What is the concrete criterion for choosing between them? What is the policy change implied by the experiments? I don't see this come through in the current analysis of experiments.

[1] Riccardo Poiani, Rémy Degenne, Emilie Kaufmann, Alberto Maria Metelli, Marcello Restelli, Optimal Multi-Fidelity Best-Arm Identification, https://arxiv.org/pdf/2406.03033

[2] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, Barnabás Póczos, The Multi-fidelity Multi-armed Bandit, https://arxiv.org/pdf/1610.09726

[3] Oded Maron, Andrew W. Moore, Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation, https://proceedings.neurips.cc/paper_files/paper/1993/file/02a32ad2669e6fe298e607fe7cc0e1a0-Paper.pdf

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.