- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CoRE-TSF, a framework for ranking the reliability of black-box time-series forecasts without requiring historical forecast-error labels or access to model internals. It maps LLM-generated outlooks and numerical forecasts to a shared ordinal trend space and uses their disagreement as a risk score. The primary variant averages risk scores from four LLM agents. Experiments across 22 datasets and five forecasting backbones report positive average within-configuration risk–error association and selective forecasting gains over random retention.
Critical strengths
- The paper offers a fresh angle on the problem of forecast reliability by exploring textual–numerical consistency.
- The experiments are extensive, covering 22 datasets, multiple forecasters and LLM agents, with ablations and sensitivity analyses that help assess robustness.
Critical weaknesses
- The practical motivation for operating without historical forecast-error labels remains unclear. The introduction does not explain why such labels are unavailable or why they cannot be generated through backtesting.
- The method’s central assumption is insufficiently justified, which relies on potentially inaccurate LLM-generated judgments, mapped to discrete trend levels, to assess the reliability of numerical forecasts.
- The LLM is not given the forecast horizon, whereas the numerical forecast covers a specific future interval. The method therefore does not ensure that the two outputs describe changes over the same time span.
Questions for authors
- What should be done when a forecast is identified as unreliable? Discarding it may be impractical in settings such as weather forecasting, where predictions are still needed.
- What is the rationale for choosing seven trend levels, and what evidence supports this choice?
6 comments
Response to Reviewer 1PYZ (2/3)
W1: “The practical motivation for operating without historical forecast-error labels remains unclear. The introduction does not explain why such labels are unavailable or why they cannot be generated through backtesting.”
A: A newly deployed forecaster may have no recorded errors on a target series. As the reviewer suggests, these errors can be generated by rolling-origin backtesting when enough history is available to reconstruct earlier forecast origins and evaluate their subsequent outcomes. This requires additional calls to the target forecaster. CoRE can score the current forecast directly from its numerical output and a qualitative outlook derived from the observed history.
We compared CoRE with BT3, which uses the same forecaster’s errors at three historical origins. All four methods below were evaluated on the same 6,260 window–backbone pairs (70.4% of the primary cohort), spanning 22 dataset families and four backbones. The BT3 comparison excluded TimeMoE and windows with insufficient history for three origins.
| Risk score | 95% CI | |
|---|---|---|
| BT3 | 0.2341 | [0.0811, 0.3881] |
| CoRE-Ensemble | 0.4229 | [0.2755, 0.5613] |
| Rank average (50% CoRE, 50% BT3) | 0.4526 | [0.3373, 0.5644] |
| Rank average (70% CoRE, 30% BT3) | 0.4802 | [0.3627, 0.5934] |
On this common subset, CoRE’s observed utility exceeded BT3’s by 0.1888. Both rank averages improved over BT3, with paired 95% confidence intervals for the differences above zero. Their gains over CoRE alone were smaller (+0.0297 and +0.0573, respectively), and both paired intervals included zero. We will clarify the cold-start motivation in Section 1 and include this comparison to show how CoRE performs when historical errors are available.
W2: “The method’s central assumption is insufficiently justified, which relies on potentially inaccurate LLM-generated judgments, mapped to discrete trend levels, to assess the reliability of numerical forecasts.”
A: We tested whether disagreement between the qualitative outlook and the numerical forecast can still help rank forecast errors when the LLM outlook itself is imperfect. For each of the three segments, we averaged the four agents’ ordinal scores, rounded the average to the nearest integer, and took its absolute difference from the realized segment’s score on the same seven-level scale. We then averaged these three absolute differences to obtain the window-level mismatch, . For example, segment differences of 0, 0, and 1 give and fall in the first group.
| Mean absolute mismatch per segment | Window–backbone pairs | CoRE | 95% CI |
|---|---|---|---|
| 1,970 (22.2%) | 0.3877 | [0.17, 0.59] | |
| 3,435 (38.6%) | 0.3406 | [0.22, 0.46] | |
| 3,220 (36.2%) | 0.2936 | [0.16, 0.42] | |
| 265 (3.0%) | −0.0990 | [−0.39, 0.17] |
CoRE retained positive ranking utility even when the ensemble outlook differed from the realized trajectory by more than 1.5 and at most 2.5 levels on average (, 95% CI [0.16, 0.42]). The point estimates declined as mismatch increased. In the 265 pairs with mean mismatch above 2.5 levels, we found no clear ranking benefit. These results show that perfect LLM judgments are unnecessary for useful ranking in the tested cohort and identify a limitation when the outlook diverges substantially from the realized trajectory. The realized future is used only for this retrospective analysis. At forecast time, CoRE computes risk from the qualitative outlook and the numerical forecast.
Response to Reviewer 1PYZ (3/3)
W3: “The LLM is not given the forecast horizon, whereas the numerical forecast covers a specific future interval. The method therefore does not ensure that the two outputs describe changes over the same time span.”
A: We agree that the original prompt specifies the relative order of the three segments (early, middle, and late) but leaves their physical duration unspecified. The LLM may therefore interpret the time span differently from the numerical forecaster.
We tested the effect of supplying this information explicitly. We compared the original and time-informed prompts using the same Gemini 3.6 Flash agent on all 1,778 windows, evaluating each outlook against five forecasting backbones (8,890 window–backbone pairs). Adding sampling frequency, forecast horizon, and per-segment step counts changed the discrete textual scores on 43.9% of windows, yielding the following paired changes in selective utility:
| Prompt comparison | Information added | Metric | Paired change in | 95% CI for the change |
|---|---|---|---|---|
| Time-informed minus original | Sampling frequency, forecast horizon, and step count for each segment | Raw MAE | −0.049 | [−0.138, +0.034] |
| Time-informed minus original | Sampling frequency, forecast horizon, and step count for each segment | Normalized MAE | +0.013 | [−0.078, +0.108] |
Thus, while textual scores frequently changed, neither metric showed a statistically significant gain in aggregate ranking utility (with both intervals allowing a small decrease or a small improvement). This experiment measures the effect of temporal metadata on aggregate ranking utility, while leaving open how the LLM interprets the duration of an individual segment. We will clarify the temporal information supplied by the original prompt in Section 3.2 and report this ablation alongside it.
We sincerely thank Reviewer 1PYZ for the careful reading and thoughtful questions, which helped us clarify when CoRE can be useful and where its limitations lie. Please feel free to raise any further questions at any point during the discussion, and we would be grateful for the opportunity to address them.
Response to Reviewer 1PYZ (1/3)
We thank Reviewer 1PYZ for the careful reading of our manuscript and for the thoughtful questions about CoRE’s assumptions and practical use.
A: In a setting such as weather forecasting, the forecast can still be issued. CoRE ranks forecasts by estimated risk, so a forecasting team with limited review capacity could prioritize the highest-risk forecasts for a second assessment. The numerical forecast remains available throughout.
Our experiments quantify the benefit of selective retention. Appendix E.3, Table 13 reports retained MAE relative to full-coverage MAE on the primary cohort. At each target coverage, we retain the forecasts with the lowest CoRE risk.
These reductions describe the forecasts retained at each coverage level. In settings where every forecast must be issued, CoRE could help prioritize flagged forecasts for review. The effect of reviewing or replacing those forecasts on overall MAE remains unmeasured. We will clarify this distinction in Section 3.1.
A: We chose seven levels to represent one neutral category and three degrees of change in each direction. This allows the score to distinguish, for example, a weak increase from a strong increase even when the textual outlook and numerical forecast agree on direction.
We evaluated this design by comparing five ordinal grids on the same 8,890 window–backbone pairs under a common three-agent BGE setup.
Seven levels yielded the highest observed utility among the five grids. The closest alternative was nine levels (0.4106), while both coarser grids yielded lower utility (0.3553 and 0.3316). This comparison supports seven levels as a practical default for the tested setting.