acceptodds
Under review as a conference paper at ICLR 2027

Context-Gain Concentration Is Not Evidence Of Routable Forecast Value

Abstract

A steep context-gain concentration curve is not evidence that context value can be routed. Ranking forecast origins by realized gain and summing those same gains selects favourable noise, so the curve can rise steeply even when conditional value is identical at every origin. We define context value at an origin and horizon and apply a two-leg rule frozen after method development: the constant-value null leg tests whether a prespecified rank-transition signature in realized gains is incompatible with a fitted dependence- and scale-preserving null, and the held-out predictability leg asks whether realized gain can be ranked out of sample. The composed rule is validated against known truth before empirical evaluation. The strict conjunction supports 5 of 9 domain–backbone pairs, covering energy and retail but not finance/news. It was evaluated once on the designated analysis folds. These results support the prespecified transition signature together with held-out rank predictability in the supported pairs, not context-gain concentration or a general routing premise.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, multimodal forecasting, textual context, context routing, selective evaluation, gain concentration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ug3yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that centralized realized context gains do not imply routable value. The paper presents a theoretical counterexample and proposes a policy evaluation method based on matching budgets. Across three domains, the router outperformed random selection in 4 out of 6 zero-shot pairing sets and outperformed uncertainty-based selection in 3 sets. These results are exploratory in nature.

Critical strengths
  1. This paper explains why realized return concentration cannot establish routable value. The counterexample holds, and the proposed evaluation metric is appropriate.
  2. This paper employs a budget-matched baseline, time-segmentation, and paired confidence intervals. It also reports negative results and research limitations.
Critical weaknesses
  1. The data used in this evaluation also served as the basis for the development of the method. There was no completely independent test set, nor was any multiplicity correction performed. The return of the retail strategy was 2/4 that of the random selection strategy and 2/3 that of the uncertainty-based selection strategy. However, since the strategy included only three time blocks, its confidence intervals were unreliable.
  2. The theoretical results are based on established principles of conditional expectation. The router itself employs a conventional design. The main evaluation relies on fully folded, time-specific decisions. Online evaluations and shared-time evaluations are reported separately. The combined approach was not tested against competitive routing baselines.
Questions for authors
  1. Can the authors validate the routing benefits using independent data that includes a sufficient number of time intervals?
  2. Given the same budget, does the online time-window router outperform uncertainty-based methods and the variable-point selection method?
  3. Table 10 reports 4200 recalculated configurations, but the combined total of the "Completed" and "Derived" categories is only 1800. Could the authors please explain this discrepancy?
  4. How were the "one-third of the time" estimates in Table 2 calculated?
Reviewer_nwRgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the question when additional context is beneficial for time series forecasting models and how to predict if additional context will be useful. Their central claim is that observing where additional context helped a forecast in the past does not necessarily tell us where to use context in the future. They provide some theoretical analysis for this, as well as some empirical experiments for context routers.

Critical strengths

S1: The question about whether observed benefits from additional context can actually be predicted in advance seems relevant.

S2: The theoretical analysis provides a simple and clear argument to why concentrated context gains do not necessarily imply that context can be routed effectively.

S3: The experiments cover multiple forecasting domains and models and compare the proposed router against reasonable baselines.

S4: The authors are transparent about the limitations of their experiments and report negative results rather than focusing only on positive findings.

Critical weaknesses

W1: The theoretical contribution seems rather limited. The main point that large gains observed retrospectively are not necessarily predictable in advance seems kind of trivial.

W2: The statistical evidence is quite weak. The retail experiments have only 87 evaluation time points and three independent blocks for the bootstrap, yet account for two of the four positive zero-shot results.

W3: The practical benefit of the learned router remains unclear to me. It outperforms uncertainty-based selection in only three of the six zero-shot settings, and the router itself uses standard methods. The paper also lacks a convincing evaluation of whether the gains justify the additional complexity.

W4 (subjective): The presentation of the paper could be improved significantly. The main text contains a single plot and table. Since there is still lots of space available, I think that the paper could benefit from putting more effort into the writing.

Questions for authors

See my weaknesses.

Reviewer_HgeTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the concentration of realized context gains can justify routing context before outcomes are observed. The authors construct two settings with identical realized-gain distributions but different routing opportunities, demonstrating that concentration curves alone cannot identify routable value. They propose evaluating pre-decision policies through matched-budget net gains and conduct experiments across energy, retail, and finance/news, comparing a learned router with random, uncertainty-based, and change-point-based selection.

The paper makes a valid distinction between retrospective gain concentration and prospective routing value. However, the practical importance of the motivating problem and the substantive novelty of the proposed evaluation framework remain insufficiently established. The absence of independent empirical confirmation and the unclear organization further weaken the case for acceptance. I recommend rejection in its current form.

Critical strengths

The paper distinguishes retrospective gain concentration from predictable policy value. The two-world counterexample intuitively demonstrates why concentration curves alone cannot establish whether routing will be effective.

Critical weaknesses
  1. The motivation lacks concrete support from prior literature. The paper centers on the observation that retrospective gain concentration does not establish prospective routing effectiveness. However, neither the introduction nor the related work clearly identifies studies that actually rely on this inference. The authors should provide specific references and identify the corresponding arguments or experiments. Without such examples, it is difficult to assess the prevalence and practical importance of the problem in existing research.

  2. The empirical results lack independent confirmation. The policy-value analysis was designed after the earlier gate results were known, and the router, data splits, and evaluation rules were developed on the same archives. Therefore, even though the training and evaluation folds are separated for model fitting, the current evaluation does not constitute independent confirmation of the overall research process. The authors disclose this limitation transparently, but confirmation on a new time period or independent dataset under a frozen protocol is still needed to strengthen the empirical conclusions.

  3. The main text is difficult to follow and provides insufficient detail on key methods and experiments. The presentation repeatedly switches between the final matched-budget policy evaluation and the earlier two-leg gate, making it difficult to distinguish primary evidence, auxiliary diagnostics, and development history. Although the authors explicitly state that the earlier gate is not a test of routing value, its results are discussed repeatedly. In contrast, several experiments closely related to the final approach are only briefly summarized, with detailed evidence deferred to the appendices. The main text should contain sufficient information for readers to understand and assess the central claims independently. I recommend condensing the development history, moving essential experimental evidence and interpretation into the main text, and adding a workflow diagram specifying decision-time information, training labels, temporal splits, routing policies, and evaluation metrics. The sole main-text figure primarily illustrates concentration curves and simulation results rather than explaining the complete methodological framework. Together with the frequent shifts in focus, this substantially increases the reading burden.

  4. The substantive contribution beyond existing methods is insufficiently established. The proposed alternative—predicting gains from pre-decision information, evaluating on held-out data, and comparing against matched-budget baselines—largely consists of conventional methods. The authors need to clarify the precise contribution of the non-identification result and matched-budget evaluation beyond existing conditional predictive ability and forecast-selection frameworks. In particular, without concrete examples of prior work relying on the criticized inference, the current presentation does not sufficiently establish that this conceptual clarification and combination of standard evaluation methods constitute a substantial contribution.

Questions for authors
  1. Before the final policy-value analysis plan was frozen, which results from Fold D had the researchers already inspected? Did these results influence the choice of router, data splits, budget, or evaluation metrics?

  2. Which specific studies infer prospective routing effectiveness from retrospective gain concentration? Please identify the relevant passages, figures, or experiments and explain how this paper’s analysis changes their conclusions.

  3. What new theoretical or methodological content does this paper contribute beyond existing conditional predictive ability and forecast-selection frameworks?

  4. Can the main findings be replicated on data entirely untouched by method development? Can the authors provide stronger validation addressing the small number of independent time blocks in the retail evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.