- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
JudgeCast extends experience-based, LLM-assisted time-series forecasting to settings with covariates. It combines a frozen time-series foundation model (TSFM), which produces a target-only base forecast, with a frozen large language model (LLM), which makes explicit covariate-wise judgments and uses them to determine a numerical adjustment. After the target observation becomes available, the method uses the base-forecast residual to generate and evaluate alternative judgments, retaining a decision as validated experience only when its adjusted forecast improves on the base forecast. Evaluations on the EPF electricity-price datasets and fev-bench tasks show that JudgeCast outperforms the strongest baselines in the paper’s main evaluation. The ablation studies suggest that explicit covariate-wise judgments can improve forecast-time adjustments, while residual-guided reconstruction produces more reliable gains from stored experience than simply retaining the original decisions.
Critical strengths
-
JudgeCast uses the residual of the target-only base forecast to generate and compare alternative covariate judgments. It retains a decision as reusable experience only when the resulting adjusted forecast outperforms the unadjusted base forecast. This validation step makes experience reuse more selective and reliable than simply applying LLM-generated covariate adjustments without post-hoc evaluation.
-
JudgeCast builds and continually updates a memory of experience about covariate use in complex forecasting settings. This iterative reuse of validated experience is conceptually similar to recursive self-improvement: past outcomes inform subsequent decisions, helping improve covariate-informed forecasting over time. Importantly, the framework updates its experience memory rather than the parameters of the frozen models.
-
The ablation studies are thorough and carefully designed. They examine the contribution of explicit covariate-wise judgments, compare alternative experience construction strategies, including raw experience, validation of the original decision, and residual-guided reconstruction, and analyze the effects of experience accumulation and retrieval relevance. Together, these experiments provide substantial evidence for the value of the main components of the framework.
-
The paper is clearly written, with a well-organized presentation of the motivation, method, and experimental results.
Critical weaknesses
- The empirical evaluation is conducted on a relatively narrow set of forecasting tasks. Although the paper includes seven datasets in its main experiments and adds three further fev-bench tasks, the results do not establish how well JudgeCast generalizes across a broader range of benchmarks and domains. The comparison with alternative TSFM backbones is also limited to NP and PJM. A more comprehensive, benchmark-wide comparison, especially against TSFMs using matched covariate conditioning setups would strengthen the empirical evidence.
- The framework may have potential applications to multimodal forecasting with covariates, but the paper evaluates only time-series covariates. It does not investigate whether JudgeCast can effectively handle other modalities, such as text or images, or how multimodal information would be incorporated into its judgment and experience retrieval stages.
- The paper does not provide a systematic analysis of the costs and benefits of using an LLM. Since the framework requires multiple LLM calls, particularly to generate and evaluate alternative judgments after observations become available, inference latency and API costs may be substantial. It remains unclear whether the forecasting gains justify these additional costs.
- Minor: The sentence "This validation does not establish that the selected judgment faithfully reflects the effects of the covariates." (Introduction, paragraph 5) is vague and defensive.
Questions for authors
-
How should you position JudgeCast relative to MemCast? The framework appears to extend MemCast’s experience-driven approach to forecasting with covariates. What are its substantive methodological contributions beyond this extension, and what aspects could inspire future work?
-
Are the practical costs of LLM inference manageable? Please report or discuss the number of LLM calls, inference latency, and associated costs, and clarify whether the forecasting gains justify this overhead. Which parts of the pipeline could be optimized to reduce this overhead?
-
Could the experience-reuse component be developed further by drawing on advances in LLM context management, memory management or skills? What limitations of the current experience-reuse design do the authors see as most important to address?
Thank you for the detailed response and the revision. The additional backbone results, the new baselines in Table 1, and the cost analysis are helpful and answer most of my questions.
I also appreciate that the revised paper clearly states in Section 4.2 that direct covariate conditioning with Chronos-2 performs better on the seven main datasets. This is an honest and useful result. JudgeCast performs better on the additional fev-bench tasks and in some misalignment settings (both offsets on NP, 12h on DE), and I agree that the explicit covariate-wise judgments are a valuable direction. I would suggest mentioning in the abstract and introduction that direct covariate conditioning performs better on the main datasets. More evaluation of interpretability would also make the paper stronger.
Overall, I have raised my score to 3. Thank you again for the effort.