acceptodds
Under review as a conference paper at ICLR 2027

CoRE-TSF: When to Trust Time-Series Forecasts via Cross-Modal Consistency

Abstract

Aggregate forecast accuracy does not reveal whether a particular forecast can be trusted. Yet black-box point forecasters may expose only their numerical predictions, providing no direct signal of which individual forecasts are likely to fail. We introduce CoRE-TSF, an output-level framework that cross-checks each numerical forecast against an LLM’s qualitative expectation of future change. Given the same observed history, a textual pathway predicts the direction and magnitude of future change from categorical history descriptors, while a numerical pathway forecasts future values. Neither pathway observes the other’s output. CoRE-TSF maps both outputs to a shared ordinal space and uses their disagreement to compute a forecast-level risk score without requiring realized future values, historical forecast-error labels, or access to forecaster internals. Our primary method, CoRE-Ensemble, averages risk scores from four LLM agents before ranking forecasts. We evaluate five time-series foundation forecasters across 22 dataset families, with two LLM-based forecasters evaluated on a separate matched short-horizon cohort. On the primary cohort, CoRE-Ensemble achieves a hierarchically macro-averaged within-configuration Spearman correlation of and normalized selective utility of , where measures the fraction of improvement over random retention attainable under ideal error-based ordering. These results support cross-modal consistency as an externally observable signal for ranking forecast reliability.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, forecast reliability, uncertainty quantification, selective forecasting, large language models, cross-modal consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1PYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoRE-TSF, a framework for ranking the reliability of black-box time-series forecasts without requiring historical forecast-error labels or access to model internals. It maps LLM-generated outlooks and numerical forecasts to a shared ordinal trend space and uses their disagreement as a risk score. The primary variant averages risk scores from four LLM agents. Experiments across 22 datasets and five forecasting backbones report positive average within-configuration risk–error association and selective forecasting gains over random retention.

Critical strengths
  1. The paper offers a fresh angle on the problem of forecast reliability by exploring textual–numerical consistency.
  2. The experiments are extensive, covering 22 datasets, multiple forecasters and LLM agents, with ablations and sensitivity analyses that help assess robustness.
Critical weaknesses
  1. The practical motivation for operating without historical forecast-error labels remains unclear. The introduction does not explain why such labels are unavailable or why they cannot be generated through backtesting.
  2. The method’s central assumption is insufficiently justified, which relies on potentially inaccurate LLM-generated judgments, mapped to discrete trend levels, to assess the reliability of numerical forecasts.
  3. The LLM is not given the forecast horizon, whereas the numerical forecast covers a specific future interval. The method therefore does not ensure that the two outputs describe changes over the same time span.
Questions for authors
  1. What should be done when a forecast is identified as unreliable? Discarding it may be impractical in settings such as weather forecasting, where predictions are still needed.
  2. What is the rationale for choosing seven trend levels, and what evidence supports this choice?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 1PYZ (1/3)

We thank Reviewer 1PYZ for the careful reading of our manuscript and for the thoughtful questions about CoRE’s assumptions and practical use.

Q1: “What should be done when a forecast is identified as unreliable? Discarding it may be impractical in settings such as weather forecasting, where predictions are still needed.”

A: In a setting such as weather forecasting, the forecast can still be issued. CoRE ranks forecasts by estimated risk, so a forecasting team with limited review capacity could prioritize the highest-risk forecasts for a second assessment. The numerical forecast remains available throughout.

Our experiments quantify the benefit of selective retention. Appendix E.3, Table 13 reports retained MAE relative to full-coverage MAE on the primary cohort. At each target coverage, we retain the forecasts with the lowest CoRE risk.

Target coverageRelative retained MAEReduction from full-coverage MAE
85%0.86213.8%
80%0.82517.5%
70%0.74225.8%

These reductions describe the forecasts retained at each coverage level. In settings where every forecast must be issued, CoRE could help prioritize flagged forecasts for review. The effect of reviewing or replacing those forecasts on overall MAE remains unmeasured. We will clarify this distinction in Section 3.1.

Q2: “What is the rationale for choosing seven trend levels, and what evidence supports this choice?”

A: We chose seven levels to represent one neutral category and three degrees of change in each direction. This allows the score to distinguish, for example, a weak increase from a strong increase even when the textual outlook and numerical forecast agree on direction.

We evaluated this design by comparing five ordinal grids on the same 8,890 window–backbone pairs under a common three-agent BGE setup.

Number of levels357 (default)911
Selective utility (UmacroU_{\mathrm{macro}})0.35530.33160.42530.41060.4055

Seven levels yielded the highest observed utility among the five grids. The closest alternative was nine levels (0.4106), while both coarser grids yielded lower utility (0.3553 and 0.3316). This comparison supports seven levels as a practical default for the tested setting.

AuthorsOpenReview ↗

Response to Reviewer 1PYZ (2/3)

W1: “The practical motivation for operating without historical forecast-error labels remains unclear. The introduction does not explain why such labels are unavailable or why they cannot be generated through backtesting.”

A: A newly deployed forecaster may have no recorded errors on a target series. As the reviewer suggests, these errors can be generated by rolling-origin backtesting when enough history is available to reconstruct earlier forecast origins and evaluate their subsequent outcomes. This requires additional calls to the target forecaster. CoRE can score the current forecast directly from its numerical output and a qualitative outlook derived from the observed history.

We compared CoRE with BT3, which uses the same forecaster’s errors at three historical origins. All four methods below were evaluated on the same 6,260 window–backbone pairs (70.4% of the primary cohort), spanning 22 dataset families and four backbones. The BT3 comparison excluded TimeMoE and windows with insufficient history for three origins.

Risk scoreUmacroU_{\mathrm{macro}}95% CI
BT30.2341[0.0811, 0.3881]
CoRE-Ensemble0.4229[0.2755, 0.5613]
Rank average (50% CoRE, 50% BT3)0.4526[0.3373, 0.5644]
Rank average (70% CoRE, 30% BT3)0.4802[0.3627, 0.5934]

On this common subset, CoRE’s observed utility exceeded BT3’s by 0.1888. Both rank averages improved over BT3, with paired 95% confidence intervals for the differences above zero. Their gains over CoRE alone were smaller (+0.0297 and +0.0573, respectively), and both paired intervals included zero. We will clarify the cold-start motivation in Section 1 and include this comparison to show how CoRE performs when historical errors are available.

W2: “The method’s central assumption is insufficiently justified, which relies on potentially inaccurate LLM-generated judgments, mapped to discrete trend levels, to assess the reliability of numerical forecasts.”

A: We tested whether disagreement between the qualitative outlook and the numerical forecast can still help rank forecast errors when the LLM outlook itself is imperfect. For each of the three segments, we averaged the four agents’ ordinal scores, rounded the average to the nearest integer, and took its absolute difference from the realized segment’s score on the same seven-level scale. We then averaged these three absolute differences to obtain the window-level mismatch, mm. For example, segment differences of 0, 0, and 1 give m=1/3m=1/3 and fall in the first group.

Mean absolute mismatch per segmentWindow–backbone pairsCoRE UmacroU_{\mathrm{macro}}95% CI
m≤0.5m \leq 0.51,970 (22.2%)0.3877[0.17, 0.59]
0.5<m≤1.50.5 < m \leq 1.53,435 (38.6%)0.3406[0.22, 0.46]
1.5<m≤2.51.5 < m \leq 2.53,220 (36.2%)0.2936[0.16, 0.42]
m>2.5m > 2.5265 (3.0%)−0.0990[−0.39, 0.17]

CoRE retained positive ranking utility even when the ensemble outlook differed from the realized trajectory by more than 1.5 and at most 2.5 levels on average (Umacro=0.2936U_{\mathrm{macro}}=0.2936, 95% CI [0.16, 0.42]). The point estimates declined as mismatch increased. In the 265 pairs with mean mismatch above 2.5 levels, we found no clear ranking benefit. These results show that perfect LLM judgments are unnecessary for useful ranking in the tested cohort and identify a limitation when the outlook diverges substantially from the realized trajectory. The realized future is used only for this retrospective analysis. At forecast time, CoRE computes risk from the qualitative outlook and the numerical forecast.

AuthorsOpenReview ↗

Response to Reviewer 1PYZ (3/3)

W3: “The LLM is not given the forecast horizon, whereas the numerical forecast covers a specific future interval. The method therefore does not ensure that the two outputs describe changes over the same time span.”

A: We agree that the original prompt specifies the relative order of the three segments (early, middle, and late) but leaves their physical duration unspecified. The LLM may therefore interpret the time span differently from the numerical forecaster.

We tested the effect of supplying this information explicitly. We compared the original and time-informed prompts using the same Gemini 3.6 Flash agent on all 1,778 windows, evaluating each outlook against five forecasting backbones (8,890 window–backbone pairs). Adding sampling frequency, forecast horizon, and per-segment step counts changed the discrete textual scores on 43.9% of windows, yielding the following paired changes in selective utility:

Prompt comparisonInformation addedMetricPaired change in UmacroU_{\mathrm{macro}}95% CI for the change
Time-informed minus originalSampling frequency, forecast horizon, and step count for each segmentRaw MAE−0.049[−0.138, +0.034]
Time-informed minus originalSampling frequency, forecast horizon, and step count for each segmentNormalized MAE+0.013[−0.078, +0.108]

Thus, while textual scores frequently changed, neither metric showed a statistically significant gain in aggregate ranking utility (with both intervals allowing a small decrease or a small improvement). This experiment measures the effect of temporal metadata on aggregate ranking utility, while leaving open how the LLM interprets the duration of an individual segment. We will clarify the temporal information supplied by the original prompt in Section 3.2 and report this ablation alongside it.

We sincerely thank Reviewer 1PYZ for the careful reading and thoughtful questions, which helped us clarify when CoRE can be useful and where its limitations lie. Please feel free to raise any further questions at any point during the discussion, and we would be grateful for the opportunity to address them.

Reviewer_UAkoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRE-TSF, a framework for ranking the reliability of time-series point forecasts using disagreement between an LLM-generated qualitative outlook and a numerical forecast. Both outputs are mapped to a seven-level direction–magnitude space, and their discrepancy defines a risk score. The primary variant averages scores from four LLMs. Experiments across 22 dataset families and five time-series foundation models show positive associations between the proposed score and realized forecast error, as well as benefits from retaining lower-risk forecasts.

Critical strengths

The experiments cover multiple datasets and forecasting models. The authors also clearly acknowledge that agreement does not guarantee accuracy.

Critical weaknesses

Limited novelty. The method uses disagreement between an LLM’s trend description and a numerical forecast to estimate reliability. Using disagreement between predictors is an established approach, and the paper does not clearly show what is gained by making one of those predictions textual.

The trend definitions are not clearly aligned. Trends depend on the forecast horizon and sampling frequency, but the LLM receives neither. Moreover, textual trend strength is determined through semantic similarity, while numerical trend strength uses fixed relative-slope thresholds. Assigning both the same labels does not ensure that they describe the same behavior. This also matters for evaluation: a forecast can capture the long-term trend while missing short-term fluctuations and therefore still have a large MAE.

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UAko (1/2)

We thank Reviewer UAko for the careful reading and for asking what the textual outlook contributes and how the two branches compare trend strength.

W1: “Limited novelty. The method uses disagreement between an LLM’s trend description and a numerical forecast to estimate reliability. Using disagreement between predictors is an established approach, and the paper does not clearly show what is gained by making one of those predictions textual.”

A: We agree that disagreement-based reliability assessment is an established idea. CoRE’s proposed contribution is to construct a history-derived qualitative reference for auditing black-box point forecasts: an LLM interprets historical descriptors into a future outlook, generated separately from the numerical forecast and compared with it through a shared ordinal representation. This enables risk scoring without access to forecaster internals, predictions from additional numerical forecasters, or a selector trained on historical error labels. Our contribution concerns this specific formulation and its empirical evaluation, rather than disagreement itself. To examine what the textual pathway contributes, we tested whether its pairing with a particular history matters, whether textual features help supervised predictors of forecast error, and how simpler ways of constructing an outlook perform.

ComparisonResult
Original versus scale-matched reassigned outlooksSingle-agent selective utility was 0.3465 with the original history–outlook pairing and 0.2705 on average across 1,000 reassignments within scale-matched blocks (difference +0.0760, one-sided permutation p = 0.001). Each reassignment moved the complete three-segment outlook together and used the same assignment for all five backbones. The 151 blocks covered 1,769 of 1,778 windows.
Numerical features with and without textual featuresIn the original leave-one-family-out feature probes, adding the tested textual features increased macro-averaged selective utility by 0.101 for Ridge and 0.118 for Random Forest. These gains apply to those feature sets. A separate probe that added only the scalar CoRE risk to numerical features and the nine history descriptors showed no clear gain (paired difference −0.0122, 95% CI [−0.035, +0.003]).
Textual outlook versus last-segment carryoverThe rule repeats the final history segment’s direction and magnitude across the forecast horizon. In the tie-aware threshold sweep, its utility ranged from 0.2211 to 0.3232, compared with 0.3465 for single-agent CoRE and 0.4201 for four-agent CoRE under the same evaluation.

The permutation result gives the clearest evidence that the pairing matters: even among windows of similar context scale, using each window’s own outlook yielded higher ranking utility than reassigning outlooks. The carryover sweep has a narrower interpretation because the rule uses only the final segment. In a further full-cohort comparison, online four-agent CoRE achieved a higher point estimate (0.432) than a deterministic rule that carried all three observed segments forward (0.343), though the paired difference was not statistically significant. A cross-family lookup using stored LLM outputs also achieved 0.396 without new LLM calls at evaluation time. We will clarify the contribution accordingly: a history-specific qualitative outlook can support forecast-error ranking, and the benefit of generating it online depends on the alternative used to construct that outlook.

AuthorsOpenReview ↗

Response to Reviewer UAko (2/2)

W2: “The trend definitions are not clearly aligned. ... Assigning both the same labels does not ensure that they describe the same behavior. ... a forecast can capture the long-term trend while missing short-term fluctuations and therefore still have a large MAE.”

A: We agree that the original prompt leaves a trend’s physical duration unspecified, aligning segments only by relative position (early, middle, late). Thus, shared labels like “strong increase” permit ordinal comparison, though their exact rate-of-change correspondence remains uncertain.

Adding sampling frequency, horizon, and per-segment step ranges to Gemini 3.6 Flash across 1,778 windows (5 backbones) changed textual scores in 43.9% of windows. The paired utility change (time-informed minus original) was −0.049 for raw MAE (95% CI [−0.138, +0.034]) and +0.013 for normalized MAE ([−0.078, +0.108]), showing frequent score shifts without a clear aggregate utility gain. Due to space limits, we include the summary table in the immediately following comment.

1. Ordinal granularity and trend strength. In a three-agent BGE evaluation over 8,890 window–backbone pairs, seven levels gave the highest observed utility among the five grids tested.

Ordinal levels357 (default)911
Selective utility0.35530.33160.42530.41060.4055

Holding textual scores fixed, multiplying the numerical slope thresholds by factors from 0.5 to 2.0 yielded macro-averaged selective utility of 0.413–0.438 (default 0.420). Accounting for segment length gave 0.421, and normalizing slope by the context median absolute deviation gave 0.422.

We also examined whether stronger wording corresponded to larger realized movements. Across 5,334 distinct target segments, we grouped outlooks by their textual direction and strength. The table reports the median absolute normalized slope of the realized target, with segment counts in parentheses.

Textual strengthDecrease in outlookIncrease in outlook
Slight0.0026 (74)0.0019 (59)
Moderate0.0030 (845)0.0025 (1,919)
Strong0.0037 (562)0.0036 (949)

For both textual directions, median absolute slope increased from slight to moderate to strong. Neutral outlooks had the lowest median (0.0016, n = 926). Even the two strong medians fell below the numerical branch’s first non-neutral boundary (0.02). The family-macro Spearman correlation between textual level and realized signed slope was 0.024 (95% CI [−0.015, +0.063]). Thus, the labels showed a descriptive ordering in movement size, while their correspondence to specific numerical slope ranges remains unresolved.

2. Forecasts with low trend error but high total MAE. We used STL to decompose targets and forecasts for 8,170 valid window–backbone pairs and examined the failure case raised by the reviewer.

Trend errorTotal MAEPairs (% of valid pairs)Median CoRE riskWithin-group utility [95% CI]
LowLow3,261 (39.9%)0.3610.428 [0.261, 0.592]
LowHigh54 (0.66%)0.361−0.123 [−0.307, 0.127]
HighLow3,448 (42.2%)0.3890.390 [0.216, 0.560]
HighHigh1,407 (17.2%)0.4720.452 [0.299, 0.611]

Of the 1,461 pairs with high total MAE, 1,407 (96.3%) also had high trend error. The remaining 54 pairs accounted for 3.7% of the high-MAE group and 0.66% of all valid pairs. Their median CoRE risk was 0.361, the same as in the low-trend-error, low-MAE group. When trend error was low but pointwise error was high, CoRE showed no clear ability to rank those errors.

3. Combining CoRE with numerical disagreement. We averaged the within-group ranks of CoRE and normalized Pairwise-Point. The reported weights (0.7 CoRE, 0.3 Pairwise-Point) were selected from three weight pairs evaluated on the primary cohort.

MethodMacro-averaged selective utilityPaired gain of hybrid over method [95% CI]
Normalized Pairwise-Point0.303+0.202 [+0.026, +0.394]
CoRE-Ensemble (same evaluation protocol)0.432+0.073 [+0.022, +0.130]
Hybrid0.505—

The hybrid improved overall ranking utility relative to both components on this cohort. Within the 54 low-trend-error, high-MAE pairs, it gained 0.118 over CoRE, leaving utility near zero. The evidence for complementarity is strongest in the overall evaluation, and ranking errors in this subgroup remains a limitation.

We will clarify the temporal assumptions and incorporate these findings in Sections 3–4 and the appendix. We thank Reviewer UAko again for the constructive feedback and would welcome any further questions.

AuthorsOpenReview ↗

Follow-up on W2: Temporal-Metadata Prompt Ablation

Due to character limits, we include the summary table for this temporal-metadata ablation in this comment. As noted in our response to W2 above, below is the summary table for the temporal-metadata prompt ablation using Gemini 3.6 Flash across all 1,778 windows and five forecasting backbones (8,890 window–backbone pairs):

Prompt comparisonInformation addedMetricPaired change in UmacroU_{\mathrm{macro}}95% CI for the change
Time-informed minus originalSampling frequency, forecast horizon, and step count for each segmentRaw MAE−0.049[−0.138, +0.034]
Time-informed minus originalSampling frequency, forecast horizon, and step count for each segmentNormalized MAE+0.013[−0.078, +0.108]

The time-informed prompt gave a lower point estimate under raw MAE and a small positive point estimate under normalized MAE, with both paired intervals including both a decrease and a small increase in utility. This experiment measures the effect of temporal metadata on aggregate ranking utility, while leaving open how the LLM interprets the duration of an individual segment. We will clarify the temporal information supplied by the original prompt in Section 3.2 and report this ablation alongside it.

We sincerely thank Reviewer UAko again for the careful reading and constructive feedback.

Reviewer_JnRvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks how to tell which individual forecasts can be trusted when the forecasting model is a black box that outputs only numbers. The proposed CoRE-TSF asks a language model to describe in words how the series will likely move, using only a coarse summary of its history. This outlook is compared with the numerical forecast: the more they disagree, the riskier the forecast. It needs no training, past error records, or model internals. Combining judgments from four language models, the authors evaluate the score across 22 dataset families and seven forecasters. Higher risk tends to accompany larger errors: keeping the 80% lowest-risk forecasts cuts error by about 17% for the main foundation models, outperforming simple history-based alternatives.

Critical strengths
  1. The paper studies an interesting problem: it5me-series foundation models are increasingly deployed zero-shot, often through APIs that return only numbers. Average benchmark accuracy does not tell a user whether this particular forecast is trustworthy. Therefore, being able to flag a doubtful forecast and send it to a human can prevent costly mistakes.

  2. Rather than asking LLMs to forecast, which earlier work questions, the paper uses them as auditors of a specialized forecaster. This is a new perspective. In addition, averaging several language models improves results.

  3. The method works with any forecaster, including foundation models and LLM-based forecasters.

  4. Authors have provided broad evaluation of the method and achieved clear gain from the results.

Critical weaknesses
  1. For deployment to real applications, I think the authors should consider the cost and latency issue of several commercial LLM calls per forecast.

  2. The error metric is raw MAE, but windows within a dataset can differ enormously in size and activity. The LLM's input labels are computed in raw units, while every baseline is scale-normalized. So the signal may come from the scale of the time series rather than LLM. The authors may consider adding a raw-scale baseline.

  3. The LLM essentially acts as a lookup table, with only 9 course labels as input. The signal also looks coarse. So it is unclear what LLM finally contributes or have utilized its unique capacity.

Questions for authors

It appears that hyperparameters such as thresholds, penalty and the number of segments are fixed with no stated selection procedure.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JnRv (1/4)

We sincerely thank Reviewer JnRv for taking the time to read our manuscript carefully and for the thoughtful comments and questions.

Q1: “It appears that hyperparameters such as thresholds, penalty and the number of segments are fixed with no stated selection procedure.”

A: We selected the direction penalty and segment count after examining results on the Monash evaluation data. We then used these defaults in the primary evaluation. Below, we describe the original choices and reproduce the corresponding sensitivity results from Appendix F.

Direction penalty. We compared α∈{0,0.25,0.50,0.75,1}\alpha \in \{0, 0.25, 0.50, 0.75, 1\} on Monash. The result on tourism_quarterly informed our choice: the Spearman correlation between consistency and error was −0.555 at α=0.75\alpha = 0.75 and −0.516 at α=1\alpha = 1. The selected value applies a 25% discount to consistency when the two segment scores have opposite non-neutral signs.

Appendix F.1, Table 18. Four agents, K=3K = 3, with re-encoded BGE assignments held fixed. I80I_{80} denotes the percentage reduction in retained MAE at 80% coverage relative to full coverage.

Direction penaltyUmacroU_{\mathrm{macro}} [95% CI]ρmacro\rho_{\mathrm{macro}}I80I_{80} (%)
0.000.423 [0.274, 0.567]0.39016.0
0.250.422 [0.273, 0.565]0.39016.0
0.500.420 [0.271, 0.564]0.39316.2
0.75 (default)0.423 [0.273, 0.569]0.39717.5
1.000.423 [0.272, 0.570]0.39717.7

We also selected α\alpha by leave-one-family-out cross-validation, choosing among the same five values on 21 families and evaluating on the held-out family. All 22 folds selected α=1\alpha = 1 (no discount). In this separate evaluation, held-out UmacroU_{\mathrm{macro}} was 0.4556 (95% CI [0.333, 0.574]) for the selected setting and 0.4492 for the fixed default. The paired difference was +0.0064 (p=0.094p = 0.094).

Segment count. We originally compared K∈{3,4,5}K \in \{3, 4, 5\} on Monash and retained K=3K = 3 based on the observed dataset-level results. This choice represents the early, middle, and late portions of the trajectory. Appendix F.2 subsequently tested a different grid, including K=1K = 1 to evaluate the effect of summarizing the entire trajectory in one segment.

Appendix F.2, Table 19. Two agents (Gemini and GPT), α=0.75\alpha = 0.75.

Segment countUmacroU_{\mathrm{macro}} [95% CI]ρmacro\rho_{\mathrm{macro}}I80I_{80} (%)
10.351 [0.210, 0.492]0.30014.8
3 (default)0.352 [0.197, 0.509]0.33314.6
50.372 [0.219, 0.524]0.36417.0

The one- and three-segment settings yielded similar selective utility. Five segments gave the highest point estimate on all three metrics in this two-agent evaluation.

Thresholds. The history descriptors and the numerical forecast scores use different thresholds because they operate on different quantities.

Quantity being discretizedDefault boundaries
History slope for directionNeutral band bounded by −0.02 and +0.02
Absolute history slope for magnitude0.05 and 0.15
History standard deviation for volatility0.5 and 2.0
Absolute normalized forecast slope for the numerical ordinal score0.02, 0.08, and 0.20

We set the numerical forecast boundaries heuristically. The development records do not document a formal selection procedure for the history descriptor boundaries. To assess sensitivity on the forecast side, we held the textual scores fixed and varied the numerical slope thresholds from 0.5×0.5\times to 2×2\times their defaults. Across the tested settings, UmacroU_{\mathrm{macro}} ranged from 0.413 to 0.438, with 0.420 at the default. We will document these selection procedures in Section 3 and link each default to its corresponding sensitivity analysis.

AuthorsOpenReview ↗

Response to Reviewer JnRv (2/4)

W1: “For deployment to real applications, I think the authors should consider the cost and latency issue of several commercial LLM calls per forecast.”

A: We agree that inference cost and latency matter for deployment. The four Agent A models generated 7,112 retained responses across the 1,778 primary-cohort windows. Because these outlooks depend only on the observed history, we reused them across all five forecasting backbones, yielding 8,890 window–backbone evaluations without additional Agent A calls for each backbone.

Following an audit of the response records, we correct our earlier computational-cost summary. The OpenAI agent used in the primary ensemble is GPT-6 Astra. Recorded token usage is available for all four agents and totals 2,914,789 tokens. However, the available records do not establish the applicable billing rates and treatment of reasoning tokens across providers. We therefore withdraw the previously quoted USD 37 estimate and its derived per-window and per-pair costs.

AgentResponses used in evaluationMedian request latency (s)p95 (s)
GPT-6 Astra1,7781.9252.939
Gemini 3.6 Flash1,7785.87228.85
Fable 5.1 Max1,7787.1149.50
DeepSeek V4 Pro1,7785.87810.60

Recorded request durations sum to approximately 12.63 agent-hours across these 7,112 retained responses. These measurements describe individual requests and do not establish total time across unrecorded retries. End-to-end latency for a new window with four concurrent calls remains unmeasured.

To quantify the utility available with fewer calls, Appendix F.5 (Table 23) evaluates all 15 nonempty subsets of the four agents.

Number of agentsSubsets evaluatedMean UmacroU_{\mathrm{macro}}Range across subsets
140.3630.331–0.390
260.3990.352–0.432
340.4150.397–0.433
4 (full ensemble)10.419—

Two- and three-agent ensembles achieved mean utilities of 0.399 and 0.415, respectively, compared with 0.419 for the full ensemble in this sweep. Agent choice also mattered: the best observed two- and three-agent point estimates were 0.432 and 0.433. These results characterize the utility–call-count trade-off; monetary savings depend on the selected agents.

For deployments where new LLM calls are costly, the cross-family lookup described in W3 achieved 0.396 using stored outputs and zero new LLM calls at evaluation time, although constructing the lookup requires previously generated LLM judgments.

We will report the audited token usage and request-latency measurements in Appendix H.2 and connect them to the subset results in Appendix F.5. We thank the reviewer for prompting this closer examination of deployment costs and apologize for the earlier reporting inaccuracies.

AuthorsOpenReview ↗

Response to Reviewer JnRv (3/4)

W2: “...the signal may come from the scale of the time series rather than LLM. The authors may consider adding a raw-scale baseline.”

A: We thank the reviewer for highlighting this potential scale effect. We added raw-scale baselines, evaluated history-normalized errors, and examined CoRE’s association with forecast error after controlling for context standard deviation. The original baselines use normalization to express different forms of relative disagreement or variability. Pairwise-Point is defined in the paper as history-scale-normalized inter-backbone disagreement, and HistWidth divides bootstrap interval width by the history scale SxS_x. PIDEV instead measures forecast deviation relative to the bootstrap interval half-width at each step. History-Trend uses mean-relative slopes to follow the numerical branch’s definition of the shared ordinal representation.

These definitions reduce sensitivity to measurement scale, but can also remove absolute-scale information that is useful for ranking raw MAE. The reviewer’s suggestion therefore motivates an important complementary comparison. We added raw-unit variants and evaluated them against CoRE using the same raw-MAE protocol. For PW and HW, removing normalization recovers the raw score by multiplying by SxS_x. PIDEV uses step-specific denominators, so its raw variant changes the deviation measure rather than simply reversing a single history-scale normalization.

1. Comparison with raw-scale baselines. Most directly relevant to the reviewer’s concern, CoRE achieves higher observed selective utility than the history-scale score SxS_x (0.4230 versus 0.3258, a difference of 0.0972). Its utility also exceeds the point estimates of History-Raw, PIDEV-Raw, and HW-Raw.

MethodUmacroU_{\mathrm{macro}}, raw MAE ↑Spearman, raw MAE ↑MAE reduction at 30% retained coverage (%) ↑
CoRE0.42300.396042.94
PW-Raw0.50300.443540.53
HW-Raw0.41400.374440.64
PIDEV-Raw0.39110.332738.08
History-Raw0.35540.335933.20
History-scale score SxS_x0.32580.383838.67

These point-estimate comparisons support CoRE’s usefulness beyond the tested scale-only score, while showing that raw-scale baselines are competitive. PW-Raw achieves higher aggregate utility and Spearman correlation, whereas CoRE achieves the largest observed MAE reduction at 30% retained coverage among the tested methods (42.94%, compared with 40.53% for PW-Raw).

CoRE also offers a practical advantage in single-forecaster settings. It assesses a target forecast using a history-derived qualitative reference without requiring predictions from additional numerical forecasters, which PW requires.

2. Association after controlling for history scale. Controlling for context standard deviation yields a positive partial Spearman correlation between CoRE risk and raw MAE of 0.266, with positive associations in 18 of 22 dataset families. Together with the direct comparison against SxS_x, this supports an error-ranking association beyond the component captured by context standard deviation. It does not, however, exclude all scale-related effects or isolate the contribution of LLM processing.

3. Evaluation using history-normalized MAE. CoRE achieves the highest observed utility and NMAE reduction at 30% retained coverage among the methods below.

MethodUmacroU_{\mathrm{macro}}, NMAE ↑Spearman, NMAE ↑NMAE reduction at 30% retained coverage (%) ↑
CoRE0.43600.307934.53
PW-Norm0.37530.283731.29
HW-Norm0.40350.297023.68
PIDEV-Norm−0.1203−0.0187−17.49
History-Norm0.15730.172215.43

CoRE exceeds PW-Norm in selective utility by 0.0607 and in NMAE reduction at 30% retained coverage by 3.24 percentage points. These are point-estimate comparisons. The retained-error reduction measures performance at a specific coverage, while utility aggregates performance across coverage levels.

The normalized evaluation requires a qualification. Approximately 14.9% of windows activate the MAD denominator floor and account for approximately 99.9997% of the pooled NMAE sum. This pooled share is distinct from their weight in the hierarchically aggregated family-macro utility. Nevertheless, the denominator-floor effect warrants caution, so we treat these normalized-error results as supplementary evidence.

We will include these raw-scale and supplementary NMAE comparisons in the Appendix (with a brief summary in Section 4) and qualify the comparative claim accordingly. The results support CoRE’s utility for forecast-error ranking, including a positive association after controlling for context standard deviation, while establishing that raw-scale disagreement baselines can be strong alternatives. The reviewer’s feedback has also helped identify scale handling in the history descriptors as a concrete direction for further refinement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Updates

We thank Reviewers UAko, 1PYZ, and JnRv for their careful reading and constructive feedback, and the Area Chair for overseeing the review process. We have uploaded a revised manuscript incorporating the additional analyses and clarifications discussed in our responses.

The main updates are:

  • Contribution and intended use — Sections 1, 3.4, and 5. We clarified CoRE-TSF’s role as a history-derived qualitative reference for assessing individual black-box forecasts, distinguished reliability ranking from forecasting accuracy, and discussed prioritizing review and additional checks in applications where forecasts must still be issued.

  • Scale controls — Section 4.5 and Appendix H. We added raw-scale baseline comparisons, history-normalized error evaluation, and analyses controlling for context standard deviation. We also qualified the comparative claims and documented normalization and denominator-floor effects. The original reliability baseline definitions are provided in Appendix D.

  • Role of the textual outlook — Section 4.5 and Appendices I.1–I.2. We incorporated deterministic carryover and cross-family lookup controls, together with retrospective analyses of utility conditional on outlook–target mismatch.

  • Temporal correspondence — Section 4.5 and Appendices J.1–J.3. We reported the temporal-metadata ablation and additional horizon experiments comparing matched and mismatched outlooks generated from the same history. These analyses examine explicitly specified forecast-step ranges and their relationship to forecast-error ranking.

  • Implementation and computational resources — Section 4.1, Appendices B.1–B.2, F, and K.1–K.2. We clarified hyperparameter selection and sensitivity analyses, verified the provenance of the four Agent A models, and updated the reporting of recorded token usage and request durations. Appendix B.1 provides the history descriptors and prompt, Appendix B.2 documents Agent A inference, Appendix F reports sensitivity analyses, and Appendix K describes reproducibility and computational resources.

Section 4.5 provides a consolidated summary of the additional controls and diagnostics. We also revised the limitations and future-work discussion in Section 5.

We would be grateful if the reviewers could consider the revised manuscript alongside our responses. If any concerns remain, we would welcome the opportunity to clarify them. We sincerely thank UAko, 1PYZ, and JnRv for helping us improve the evaluation and presentation of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.