acceptodds
Under review as a conference paper at ICLR 2027

JudgeCast: Time Series Forecasting with Experience-Informed Covariate Judgments

Abstract

Covariate effects vary across contexts and shift over time, requiring forecasters to assess how to use them for each forecasting context. As forecasting proceeds, observations for earlier forecasts become available, providing feedback on past covariate use for subsequent forecasts. However, when multiple covariates act together, the forecast error reveals the numerical discrepancy from the observation but not how the covariates should have been used. We introduce JudgeCast, an experience-based framework for time series forecasting with covariates. Following the judgmental adjustment practice, a frozen TSFM provides the base forecast, while a frozen LLM uses the current context and relevant experience to adjust it. Within the adjustment, assessing covariate effects and determining the numerical adjustment serve distinct roles, so JudgeCast first forms explicit covariate-wise judgments and then determines the adjustment. After observation, JudgeCast uses the observed residual of the base forecast to reconstruct alternative judgments and evaluates the original and alternatives through their resulting adjustments. The best-performing decision is selected and retained as validated experience for subsequent forecasts. Across diverse real-world datasets, JudgeCast outperforms strong baselines. Ablations show that explicit covariate-wise judgment can improve forecast-time adjustment, while residual-guided experience construction yields more reliable forecasting gains than retaining raw decisions as experience.

Primary area:
learning on time series and dynamical systems
Keywords:
JudgeCast makes covariate-wise judgments explicit and revisits them after observation with residual feedback to build reusable experience, yielding consistent gains across datasets.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Bi4jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JudgeCast extends experience-based, LLM-assisted time-series forecasting to settings with covariates. It combines a frozen time-series foundation model (TSFM), which produces a target-only base forecast, with a frozen large language model (LLM), which makes explicit covariate-wise judgments and uses them to determine a numerical adjustment. After the target observation becomes available, the method uses the base-forecast residual to generate and evaluate alternative judgments, retaining a decision as validated experience only when its adjusted forecast improves on the base forecast. Evaluations on the EPF electricity-price datasets and fev-bench tasks show that JudgeCast outperforms the strongest baselines in the paper’s main evaluation. The ablation studies suggest that explicit covariate-wise judgments can improve forecast-time adjustments, while residual-guided reconstruction produces more reliable gains from stored experience than simply retaining the original decisions.

Critical strengths
  1. JudgeCast uses the residual of the target-only base forecast to generate and compare alternative covariate judgments. It retains a decision as reusable experience only when the resulting adjusted forecast outperforms the unadjusted base forecast. This validation step makes experience reuse more selective and reliable than simply applying LLM-generated covariate adjustments without post-hoc evaluation.

  2. JudgeCast builds and continually updates a memory of experience about covariate use in complex forecasting settings. This iterative reuse of validated experience is conceptually similar to recursive self-improvement: past outcomes inform subsequent decisions, helping improve covariate-informed forecasting over time. Importantly, the framework updates its experience memory rather than the parameters of the frozen models.

  3. The ablation studies are thorough and carefully designed. They examine the contribution of explicit covariate-wise judgments, compare alternative experience construction strategies, including raw experience, validation of the original decision, and residual-guided reconstruction, and analyze the effects of experience accumulation and retrieval relevance. Together, these experiments provide substantial evidence for the value of the main components of the framework.

  4. The paper is clearly written, with a well-organized presentation of the motivation, method, and experimental results.

Critical weaknesses
  1. The empirical evaluation is conducted on a relatively narrow set of forecasting tasks. Although the paper includes seven datasets in its main experiments and adds three further fev-bench tasks, the results do not establish how well JudgeCast generalizes across a broader range of benchmarks and domains. The comparison with alternative TSFM backbones is also limited to NP and PJM. A more comprehensive, benchmark-wide comparison, especially against TSFMs using matched covariate conditioning setups would strengthen the empirical evidence.
  2. The framework may have potential applications to multimodal forecasting with covariates, but the paper evaluates only time-series covariates. It does not investigate whether JudgeCast can effectively handle other modalities, such as text or images, or how multimodal information would be incorporated into its judgment and experience retrieval stages.
  3. The paper does not provide a systematic analysis of the costs and benefits of using an LLM. Since the framework requires multiple LLM calls, particularly to generate and evaluate alternative judgments after observations become available, inference latency and API costs may be substantial. It remains unclear whether the forecasting gains justify these additional costs.
  4. Minor: The sentence "This validation does not establish that the selected judgment faithfully reflects the effects of the covariates." (Introduction, paragraph 5) is vague and defensive.
Questions for authors
  1. How should you position JudgeCast relative to MemCast? The framework appears to extend MemCast’s experience-driven approach to forecasting with covariates. What are its substantive methodological contributions beyond this extension, and what aspects could inspire future work?

  2. Are the practical costs of LLM inference manageable? Please report or discuss the number of LLM calls, inference latency, and associated costs, and clarify whether the forecasting gains justify this overhead. Which parts of the pipeline could be optimized to reduce this overhead?

  3. Could the experience-reuse component be developed further by drawing on advances in LLM context management, memory management or skills? What limitations of the current experience-reuse design do the authors see as most important to address?

1 comment

Thank you for the detailed response and the revision. The additional backbone results, the new baselines in Table 1, and the cost analysis are helpful and answer most of my questions.

I also appreciate that the revised paper clearly states in Section 4.2 that direct covariate conditioning with Chronos-2 performs better on the seven main datasets. This is an honest and useful result. JudgeCast performs better on the additional fev-bench tasks and in some misalignment settings (both offsets on NP, 12h on DE), and I agree that the explicit covariate-wise judgments are a valuable direction. I would suggest mentioning in the abstract and introduction that direct covariate conditioning performs better on the main datasets. More evaluation of interpretability would also make the paper stronger.

Overall, I have raised my score to 3. Thank you again for the effort.

Reviewer_qv5GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses an experience-based and covariate-informed framework called JudgeCast. It uses a judgemental adjustment procedure during forecast time, where a frozen time-series foundation model (TSFM) provides base forecasts, and a frozen LLM further uses current context and relevant experience to adjust them. The authors also propose a residual-guided experience construction mechanism with alternative judgements and memory updates for generating reusable experiences.

Critical strengths

Well-motivated question and problem formulation, framing the approach as a training-free (purely forecast time) framework with judgemental adjustment using LLMs reconstruction is pretty good. Using forecast feedback as reusable experience and residual-guided reconstruction, although potentially not entirely novel, is principled. Good ablations with prompts and configurations clearly provided. The models and datasets also seem appropriate.

Critical weaknesses

The paper does not sufficiently discuss related works or compare against papers like RGMR [1] (inference-time), Alphacast [2], FreshCast [3], TFMAdapter [4], and more with similar motivations. Also, the related work section could better motivate and position the problem with respect to papers on experiential learning, retrieval, and hindsights [5, 6]. The paper reports target-only results in Table 1, but does not provide the equivalent covariate conditioning (Direct cov.) results which is natively supported by Chronos-2, except for just two datasets in Tables 4 and 5. In addition, it seems that Direct cov. outperforms JudgeCast with aligned covariates (Section 4.4, Table 5) in many cases, so these results definitely are critical. The paper lacks discussion and experiments on the additional computational costs and latency associated with LLM calls for sequential reasoning and experience construction. The limitations of the paper also have not been addressed.

References:

[1] Wentao Gao et al. Residual-Guided Multi-Resolution Refinement of Foundation Models: A Case Study in Drought Forecasting, ICML 2026. [2] Xiaohan Zhang et al. AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Time Series Forecasting, CIKM 2026. [3] Chao He et al. FreshCast: Timely and Complementary Retrieval for Frozen Time Series Forecasters, arXiv. [4] Afrin Dange et al. TFMAdapter: Lightweight Instance-Level Adaptation of Foundation Models for Forecasting with Covariates, CIKM 2025. [5] Andrew Zhao et al. ExpeL: LLM Agents Are Experiential Learners, AAAI 2024. [6] Kutay Tire et al. Retrieval Augmented Time Series Forecasting, AISTATS 2026.

Questions for authors

Can you include the baseline comparisons like RGMR, AlphaCast, TFMAdapter? Can you specify the details of similarity metrics and normalization methods used? Can you include the additional computational costs associated with your method? Can you add more discussions on how conflicting covariate judgements are resolved?

1 comment
Reviewer_qv5GOpenReview ↗

response to revisions

The authors have addressed several of my concerns and so I am increasing my rating to a 3 (from 2).

Reviewer_h5qpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JudgeCast, an experience-informed framework for time-series forecasting with covariates. A frozen time-series foundation model generates base forecasts, while an LLM produces explicit covariate-wise judgments and corresponding numerical adjustments. After observing the ground truth, JudgeCast reconstructs alternative judgments using forecast residuals, evaluates their adjustments, and stores the best-performing decisions as validated experience for future retrieval. Experiments across seven main datasets demonstrate improvements in MSE and MAE over the reported baselines. Additional ablations and analyses examine experience construction, retrieval strategies, backbone robustness, and sensitivity to temporally misaligned covariates.

Critical strengths
  1. Well-motivated problem formulation. The paper identifies a meaningful limitation of conventional error-based reflection: forecast residuals do not uniquely determine how individual covariates should have influenced a prediction. Separating covariate-wise judgments from numerical adjustments offers a structured approach to this problem.

  2. Promising experience reconstruction mechanism. Rather than directly storing historical forecasting decisions, JudgeCast reconstructs alternative judgments and retains those producing better adjustments. Table 3 provides useful evidence that this procedure is more effective than raw experience storage or validation of original decisions alone.

  3. Strong empirical performance. JudgeCast achieves the lowest MSE and MAE among the main-table baselines across seven datasets. Additional experiments with alternative TSFM and LLM backbones support the applicability of the framework beyond its primary configuration.

  4. Detailed analysis of experience utilization. The paper investigates memory accumulation, judgment and adjustment retrieval, temporal covariate misalignment, and representative failure cases. These analyses help explain when experience-informed adjustments are beneficial and provide some interpretability through explicit covariate-wise decisions.

Critical weaknesses
  1. Incomplete comparison with direct covariate conditioning. The main comparison in Table 1 excludes Chronos-2 with direct covariate conditioning, despite using the same foundation model. Table 5 shows that direct conditioning substantially outperforms JudgeCast with correctly aligned covariates on NP (15.89915.899 versus 19.65719.657 MSE) and DE (73.16473.164 versus 158.678158.678 MSE). Therefore, the reported superiority over the main baselines does not establish an advantage over directly incorporating reliable covariates. A comprehensive comparison across all datasets is necessary.

  2. Insufficient evidence for the correctness of reconstructed judgments. The post-observation reconstruction process selects judgments by minimizing adjustment error against the observed residual. However, this criterion validates predictive fit rather than the correctness of the inferred covariate effects. Different judgment configurations may produce similar numerical adjustments, creating an identifiability problem. The improvements could partially arise from hindsight-based residual fitting rather than learning meaningful covariate relationships.

  3. Limited isolation of the experience reconstruction contribution. Although the paper compares raw and validated experience, it lacks strong non-LLM residual-memory baselines, such as nearest-neighbor residual correction, similarity-weighted adjustment, or lightweight online residual regression. These comparisons are important for determining whether the proposed judgment reconstruction provides benefits beyond retrieving and reusing historical residual patterns.

  4. Limited evaluation of sequential adaptation. Although JudgeCast is motivated by learning from newly observed forecasting outcomes, Section 4.1 states that experience memory is constructed from training windows and remains fixed during testing. Thus, the main evaluation does not directly establish continual adaptation to distribution shifts during deployment. Moreover, uncertainty estimates across independent LLM runs and statistical significance tests are not reported for the main results.

  5. Unclear computational tradeoff. JudgeCast requires multiple LLM invocations for judgment formation, numerical adjustment, alternative generation, and retrospective evaluation. The additional inference latency, token consumption, and experience-construction costs are not systematically quantified. Without these measurements, it is difficult to assess whether the forecasting improvements justify the additional computational overhead.

Questions for authors
  1. Baseline fairness: Could the authors include Chronos-2 with direct covariate conditioning in the main comparison across all seven datasets? How does JudgeCast compare when covariates are correctly aligned and reliable?

  2. Judgment validity: How can the authors distinguish accurate covariate-effect judgments from alternative judgment configurations that merely produce similar adjustments? Could the reconstructed judgments be evaluated using synthetic data with known covariate effects or independently annotated examples?

  3. Residual-memory controls: Can the authors compare JudgeCast against nearest-neighbor residual correction, similarity-weighted residual averaging, and lightweight residual regression using the same historical information? How much improvement specifically comes from LLM-generated judgment reconstruction?

  4. Online adaptation: How does performance change when the experience memory is updated chronologically during testing after each forecast horizon becomes observable? Can the authors evaluate this setting under explicit distribution shifts?

  5. Efficiency and stability: What are the average LLM calls, token consumption, latency, and monetary costs per forecast and per validated experience? Could the authors report variability across independent LLM runs and confidence intervals for the main results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.