acceptodds
Under review as a conference paper at ICLR 2027

TSOrchestra: Time Series Agentic Orchestration Framework towards Dynamic and Faithful Forecasting

Abstract

Time series are inherently dynamic and diverse, meaning a single foundation model can seldom maintain universal dominance across varying regimes. Consequently, the frontier of forecasting has shifted from seeking a single optimal architecture to dynamic model orchestration. While Large Language Models (LLMs) offer powerful reasoning capabilities, their direct application to forecasting remains ineffective. We propose TSOrchestra, a novel framework that bridges mathematical execution with agentic reasoning by repositioning the LLM as a metacognitive auditor. While the underlying numerical weights are determined by a continuous optimization solver, the LLM auditor actively steers the optimization trajectory in real-time through iterative hypothesis testing and dynamic metric routing. To instill domain-specific heuristics, we introduce an R1-style reinforcement learning pipeline. The agent's policy is regularized by structural faithfulness rewards, ensuring its audit trails are strictly anchored to empirical, decomposition-based temporal dynamics. Validated on the GIFT-Eval benchmark across 23 datasets and 97 settings, TSOrchestra consistently outperforms all leading time-series foundation models on both CRPS and MASE metrics, establishing new state-of-the-art results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic System, Foundation Model, Time Series, Forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xbxWOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents TSOrchestra, a framework for dynamically combining pretrained time-series forecasting models through simplex-constrained weight optimization. Rather than directly generating forecasts or unrestricted ensemble weights, an LLM-based auditor selects among solver-generated candidates using model information, validation summaries, and optimization traces. The auditor is trained using supervised trajectories followed by R1-style reinforcement learning with GRPO. The framework also introduces the Temporal Incompatibility Index I_T​, intended to quantify divergence between historical patterns and future outcomes and motivate dynamic adaptation across regimes or sliding windows.

The empirical study reports encouraging forecasting performance and includes comparisons against individual forecasting models, random metric routing, supervised versus SFT+GRPO training, and non-LLM adaptive controllers. However, two claims central to the paper are not adequately supported. First, the claimed theorem establishing a strictly positive advantage for the ensemble does not follow from the assumptions stated in the manuscript. Second, the clipped GRPO reward appears to eliminate the faithfulness signal for correct and sufficiently confident responses. These issues substantially weaken the paper’s claims of theoretically justified orchestration and faithful reasoning.

Critical strengths
  1. Sensible separation between numerical optimization and language-based control The LLM auditor does not directly produce numerical forecasts or unconstrained ensemble weights. Instead, it selects among candidates generated by a conventional optimization procedure. This is a more controlled and testable use of an LLM than allowing the model to generate arbitrary numerical outputs.

    The design also makes it possible to distinguish errors in forecasting, optimization, and orchestration. This separation is valuable for both empirical evaluation and operational auditing.

  2. Stronger-than-usual adaptive baselines

    The supplementary evaluation goes meaningfully beyond comparisons with individual forecasting models. It includes random metric routing, repeated inference runs, SFT versus SFT+GRPO, and non-LLM adaptive controllers, including controllers that consume optimization traces.

    These controls are considerably more informative than static or single-model baselines and provide credible evidence that the orchestration interface has empirical value. The alternative and expanded model-pool experiments also suggest that the method is not tied to a single fixed collection of pretrained forecasters.

  3. Potentially useful empirical orchestration framework

    Even if the theoretical guarantee and strong faithfulness claims are removed, the paper may still offer a useful empirical method for adaptively combining heterogeneous pretrained forecasting models. The constrained candidate-selection formulation could be practically attractive in settings where completely unrestricted LLM control would be difficult to validate.

Critical weaknesses
  1. Theorem 3.1 and the corresponding appendix claim a strictly positive advantage .At the stated level of generality, this result appears false. The theorem does not specify:

    • restricted hypothesis class for the monolithic predictor *quantitative smoothness, Lipschitz, capacity, or regularity constraint
    • An information advantage available only to the ensemble or router
    • Approximation or competence assumptions on the constituent predictors
    • Conditions under which the ensemble can identify the relevant local regime
    • A bound on routing or switching error
    • A relationship connecting I_T​ to the stated logarithmic/logistic excess-risk gap

    For example, a monolithic predictor that represents the true forecasting function exactly can have zero loss even when nearby inputs have very different outputs. If that predictor is also included in the model pool, the optimized ensemble can achieve zero loss, but it cannot improve on zero by the strictly positive amount asserted by the theorem. More generally, a sufficiently expressive monolithic model may represent the entire ensemble and routing function.

    The proof sketch does not resolve this issue. It assumes a partition into locally compatible regimes and negligible switching cost, These assumptions also describe an idealized regime-routing construction that is not obviously equivalent to the implemented convex combination of fixed pretrained forecasts.

    The related Kolmogorov-complexity argument appears similarly incomplete. A weighted average of local description lengths does not automatically account for the description length of the complete sequence, regime assignments, routing mechanism, or switching information.

    The paper should either provide a correctly scoped theorem with explicit hypothesis classes, losses, approximation conditions, and routing assumptions, or present I_T​ solely as a heuristic motivation and remove the claimed guarantee.

  2. The reward appears to saturate and remove the faithfulness signal. For a correct and valid response with c≥0.5 and rfaith∈[−1,1]r_{faith} \in [−1,1], the minimum possible unclipped reward is:

    1+0.2(0.5)+0.1(−1)=11+0.2(0.5)+0.1(−1)=1

    Therefore, the final clipped reward is exactly one for every such output. A correct response with a maximally negatively correlated explanation receives the same reward as a correct response with a maximally positively correlated explanation.

    Unless the implementation differs from the manuscript, GRPO therefore cannot use this reward to distinguish faithful from unfaithful explanations among correct, confident responses. This directly undermines the claimed role of reinforcement learning in improving explanation faithfulness.

  3. Attribution agreement is not equivalent to explanation faithfulness. Even with a non-saturating reward, agreement with a Shapley-style attribution does not by itself establish that a generated explanation faithfully represents the auditor’s actual decision process. It may demonstrate consistency with an external attribution rule, but not necessarily causal or mechanistic faithfulness. The paper needs a reproducible description of how free-text explanations are converted into numerical vectors suitable for correlation with the attribution target. It should also clarify the sign convention when converting between loss contributions, forecasting utility, and metric-selection rationales.

  4. The operational adaptation rule is insufficiently specified. The paper appears to describe the system in different places as updating weights for every sliding window, continuously tracking I_T​, or updating periodically or when a regime change is detected. These are operationally different systems. Without extra information, the reported method is difficult to reproduce.

Questions for authors
  1. Theorem 3.1: What explicit assumptions on the monolithic hypothesis class, constituent predictors, information available to the router, and switching cost are required for the claimed strictly positive ensemble advantage? In particular, what excludes a zero-error monolithic predictor or one capable of representing the ensemble itself?

  2. Derivation of the bound: Can the authors provide a complete derivation of the specific I_T-dependent logarithmic/logistic gap in Theorem 3.1? The current proof sketch appears to assume locally compatible regimes and negligible switching cost, then assert the bound without deriving it

  3. Reward saturation: Is the GRPO reward implemented exactly as written? If so, is it correct that every correct, valid response with c≥0.5c receives reward one regardless of its faithfulness correlation?

  4. Effective faithfulness signal: What fraction of training rewards saturate, and what signal allows GRPO to distinguish faithful from unfaithful explanations among correct decisions? Please include an ablation using a non-saturating reward

  5. Faithfulness measurement: How are free-text explanations converted into the numerical attribution vectors used to calculate correlation, and what held-out evidence shows that this agreement reflects the auditor’s actual decision process rather than a post-hoc rationale?

15 comments
AuthorsOpenReview ↗

Response to Reviewer xbxW

We thank the reviewer for recognizing TSOrchestra's separation between numerical optimization and language-based control, the strength of our trace-conditioned adaptive baselines, and the practical value of constrained candidate selection.

TSOrchestra is designed as an empirical orchestration framework where theoretical analysis motivates adaptive coordination and constrained SLSQP together with trace-audited routing drives operational performance. Before addressing each technical point, we highlight the empirical evidence, including new full-benchmark ablations across all 97 GIFT-Eval settings, confirming that our formulation is both sound and effective as designed:

  • Consistent gains over exhaustive search and trace-based controllers: Under the exact same candidate pool and per-window SLSQP budget, TSOrchestra-R1 achieves mean MASE/CRPS ranks of 7.3 / 8.2, outperforming an Exhaustive-CV baseline that solves SLSQP under every candidate objective on every window (8.5 / 8.9), Trace-Transformer (8.2 / 8.7), Trace-GRU (8.4 / 8.8), and Learned Gating (8.7 / 8.9), while reducing geometric-mean MASE/CRPS loss by 7.1% / 9.4% over Moirai-2 and 2.3% / 3.7% over the per-setting hindsight-best individual model.
  • Validation of the Eq. (20) reward design: Removing the attribution term degrades GIFT-Eval MASE/CRPS ranks from 7.3/8.27.3 / 8.2 to 8.3/8.58.3 / 8.5, confirming that Eq. (20) provides an effective learning signal on uncertain and incorrect rollouts. Evaluating the reviewer's suggested unclipped variant (rnosatr_{\text{nosat}}) in Appendix G.1.1 yields closely matched performance, confirming both the robustness of our reward pipeline and the stabilizing effect of upper clipping on downstream confidence ranking.

1. Scope of Theorem 3.1 and Complementary Appendix E Formulations (W1, Q1, Q2)

Theorem 3.1 and the Temporal Incompatibility Index ITI_T are introduced as a conceptual motivation under the structured regime-specialization setting defined in Appendix E (where regimes are locally compatible, candidate models specialize across regimes, and a single fixed predictor faces cross-regime trade-offs), while operational weights are computed strictly via constrained SLSQP in Eq. (3). The counterexample of an unrestricted zero-error monolithic predictor corresponds to the boundary case where cross-regime mismatch is zero.

Rather than altering the main text's narrative flow, we will expand Appendix E to state these underlying regime conditions in explicit algebraic form and add a complementary Proposition E.1 that directly characterizes our fixed-pool SLSQP optimization:

  • Explicit regime conditions and boundary case for Theorem 3.1 (Appendix E): We will make the Appendix E conditions fully explicit: across MM regimes with probabilities πm\pi_m and optimal risks Rm⋆R_m^\star, when a capacity-bounded monolithic predictor incurs cross-regime mismatch Δreg≥0\Delta_{\text{reg}} \ge 0 (such that Rmono≥∑mπmRm⋆+ΔregR_{\text{mono}} \ge \sum_m \pi_m R_m^\star + \Delta_{\text{reg}}) while regime specialists satisfy R(fm,Dm)≤Rm⋆+ϵmR(f_m, \mathcal{D}_m) \le R_m^\star + \epsilon_m under switching cost csw≥0c_{\text{sw}} \ge 0, adaptive orchestration satisfies Rens≤Rmono−(Δreg−∑mπmϵm−csw)R_{\text{ens}} \le R_{\text{mono}} - (\Delta_{\text{reg}} - \sum_m \pi_m \epsilon_m - c_{\text{sw}}). This naturally encompasses the unrestricted zero-error monolithic predictor as the boundary case Δreg=0\Delta_{\text{reg}} = 0, and clarifies that Ω(IT,M)\Omega(I_T, M) serves as an illustrative parameterization of Δreg\Delta_{\text{reg}} under bounded regime complexity.
  • Complementary Proposition E.1 — Fixed-pool convex-combination guarantee (Added to Appendix E): To directly connect this motivation to the window-wise SLSQP solver in Eq. (3) without regime assumptions, let ri∈Rnr_i \in \mathbb{R}^n be the fitting-window residual vector of candidate ii and let j=arg⁡min⁡iLval(ei)j = \arg\min_i L_{\text{val}}(e_i) be the best individual model under Lval(w)=1n∥∑iwiri∥2L_{\text{val}}(w) = \frac{1}{n}\|\sum_i w_i r_i\|^2. The simplex optimum w⋆w^\star satisfies Lval(w⋆)≤Lval(ej)−GvalL_{\text{val}}(w^\star) \le L_{\text{val}}(e_j) - G_{\text{val}}, where Gval=1nmax⁡k(max⁡(∥rj∥2−rj⊤rk,0))2∥rj−rk∥2G_{\text{val}} = \frac{1}{n} \max_k \frac{(\max(\|r_j\|^2 - r_j^\top r_k, 0))^2}{\|r_j - r_k\|^2}. Strict improvement (Gval>0G_{\text{val}} > 0) holds if and only if rj⊤rk<∥rj∥2r_j^\top r_k < \|r_j\|^2 for some candidate kk (and Gval=0G_{\text{val}} = 0 if rj=0r_j = 0). Under validation-test discrepancy dgend_{\text{gen}}, out-of-sample loss satisfies Ltest(w⋆)≤min⁡iLtest(ei)−Gval+2dgenL_{\text{test}}(w^\star) \le \min_i L_{\text{test}}(e_i) - G_{\text{val}} + 2 d_{\text{gen}}.

Together, Theorem 3.1 and Proposition E.1 form a complete two-level justification: Theorem 3.1 provides the regime-level lens explaining why non-stationary temporal dynamics motivate adaptive multi-model coordination, while Proposition E.1 establishes the exact window-level residual-decorrelation condition under which our operational SLSQP convex combination provably improves upon the best single model in a fixed pool.


AuthorsOpenReview ↗

Continue Response

2. Rationale of Clipped Reward Eq. (20) and Non-Saturating Ablations (W2, Q3, Q4)

As specified in Appendix G.1.1 (Line 1521), the confidence term in Eq. (20) is defined piecewise as rconf=min⁡(c,1)⋅I[correct]−max⁡(c−0.5,0)⋅I[incorrect]r_{\text{conf}} = \min(c, 1)\cdot\mathbb{I}[\text{correct}] - \max(c - 0.5, 0)\cdot\mathbb{I}[\text{incorrect}], with α=0.2\alpha = 0.2 and β=0.1\beta = 0.1. While responses that are simultaneously correct (rbase=1r_{\text{base}} = 1) and highly confident (c≥0.5c \ge 0.5) reach the upper clip of 1.0, this clipping is an intentional design choice that works in tandem with SFT initialization and unsaturated exploration groups:

1. Regularization role of upper clipping and active gradient statistics: Prior to GRPO, all 2,000 SFT trajectories are already filtered for high attribution agreement (PCC(CE,EE)>0.8\text{PCC}(CE, EE) > 0.8) over 500 steps, and KL regularization (λ=10−3\lambda = 10^{-3}) in Eq. (21) anchors the policy to this SFT prior. During GRPO, capping the composite reward at 1.0 acts as a safeguard against auxiliary reward over-optimization: once a rollout achieves both a correct routing decision and high confidence (c≥0.5c \ge 0.5), the upper clip prevents the 3-component auxiliary attribution score from distorting the confidence ranking of already well-calibrated routing actions, while concentrating attribution gradients on uncertain (c<0.5c < 0.5) and incorrect (rbase=0r_{\text{base}} = 0) rollouts. Empirically, across our 50-step GRPO logs (800 prompts, n=4n=4 rollouts per prompt), 61.6% of individual rollouts—and 74.2% of the 4-sample rollout groups—contain at least one unsaturated trajectory where rfaithr_{\text{faith}} actively differentiates responses and yields non-zero group advantage variance σr(x)>0\sigma_r(x) > 0 in Eq. (22). Specifically, rfaithr_{\text{faith}} drives gradients on: (i) correct boundary trajectories with moderate confidence c<0.5c < 0.5 (unclipped sums in [0.9,1.1)[0.9, 1.1)), and (ii) incorrect stopping rollouts (rbase=0r_{\text{base}} = 0), where the score is −0.2max⁡(c−0.5,0)+0.1rfaith-0.2\max(c - 0.5, 0) + 0.1 r_{\text{faith}} and positive attribution alignment yields strictly positive clipped rewards in (0,0.1](0, 0.1].

2. Full-benchmark ablations in Appendix G.1.1 (β=0\beta = 0 and unclipped rnosatr_{\text{nosat}}): To verify both that Eq. (20) delivers an effective attribution signal and to test the reviewer's unclipped alternative, we ran two controlled retraining experiments under the exact Table 4 configuration across all 97 GIFT-Eval settings:

  • Without attribution reward (β=0\beta = 0): Retains rbaser_{\text{base}} and rconfr_{\text{conf}} while setting β=0\beta = 0.
  • Unclipped non-saturating variant (rnosatr_{\text{nosat}}): Rescales attribution agreement to F′=(rfaith+1)/2∈[0,1]F' = (r_{\text{faith}} + 1)/2 \in [0, 1] without upper clipping (rnosat=rbase+0.2rconf+0.1F′r_{\text{nosat}} = r_{\text{base}} + 0.2 r_{\text{conf}} + 0.1 F'), followed by group normalization in Eq. (22), applying attribution gradients uniformly across all confidence levels c∈[0,1]c \in [0, 1].
GRPO Reward Formulation (All 97 GIFT-Eval Settings)MASE Rank (↓\downarrow)CRPS Rank (↓\downarrow)Held-Out PCC(CE, EE) (↑\uparrow)
TSOrchestra without attribution reward (β=0\beta = 0)8.38.50.71
TSOrchestra unclipped reward variant (rnosatr_{\text{nosat}})7.48.20.87
TSOrchestra default clipped reward (Eq. 20, β=0.1\beta = 0.1)7.38.20.84

These results confirm both aspects of our design: (i) removing rfaithr_{\text{faith}} (β=0\beta = 0) degrades mean ranks by 1.0 on MASE and 0.3 on CRPS (and drops PCC from 0.84 to 0.71), proving that our default Eq. (20) formulation effectively leverages rfaithr_{\text{faith}} across the 74.2% unsaturated rollout groups; and (ii) while removing the upper clip (rnosatr_{\text{nosat}}) pushes surface attribution correlation slightly higher (0.84→0.870.84 \to 0.87), forcing confident correct decisions to further optimize the 3-component proxy slightly perturbs downstream confidence selection (7.3→7.47.3 \to 7.4 in MASE rank). Thus, Eq. (20) strikes the optimal balance between attribution grounding and routing accuracy. We will add this saturation analysis and ablation table to Appendix G.1.1.


AuthorsOpenReview ↗

Continue Response

3. Deterministic Extraction of EE(x,C)EE(x, \mathcal{C}) and Functional Impact (W3, Q5)

Reproducible rule-based extraction (Appendix F.1): The auditor uses a fixed output schema containing three explicitly tagged component-importance fields in a fixed order: Trend, Seasonality, and Residual. As detailed in Appendix F.1 (Equations 17–18 and Lines 1421–1456), each component is characterized by its ordinal importance role (dominant, secondary, minor, or none, establishing rankings such as trend > seasonal > residual) and its attribution direction (whether the component improves forecast utility or increases error, checked in Equation 18 against Wrong direction, Overstatement, Understatement, and Missed pattern). A deterministic, rule-based parser reads only these three structured fields and maps their categorical levels and direction signs s∈+1,−1s \in \\{+1, -1\\} through a fixed lookup table—mapping {dominant, secondary, minor, none} to s×3,2,1,0∈±3,±2,±1,0s \times \\{3, 2, 1, 0\\} \in \\{\pm 3, \pm 2, \pm 1, 0\\}—to construct the three-dimensional vector EE(x,C)EE(x, \mathcal{C}) along the Trend, Seasonality, and Residual coordinates. The surrounding prose is not parsed, and no secondary LLM, learned extractor, or manual annotation is involved.

Unified sign convention and discrete null significance: Equations (14) and (17) express the same Shapley attribution under opposite sign conventions: Equation (14) uses forecasting error f(S)f(S), whereas Equation (17) defines utility u(S)=−f(S)u(S) = -f(S) (Lines 1402–1406). By Shapley linearity, ϕc(u)=−ϕc(f)\phi_c(u) = -\phi_c(f). In our implementation, both CE(x,C)CE(x, \mathcal{C}) and EE(x,C)EE(x, \mathcal{C}) are converted to the utility convention u=−fu = -f prior to computing Equation (5), with zero-variance vectors defaulting to 0. Because each coordinate of EE(x,C)EE(x, \mathcal{C}) takes 7 signed values ±3,±2,±1,0\\{\pm 3, \pm 2, \pm 1, 0\\} (73−7=3367^3 - 7 = 336 non-constant signed configurations), requiring PCC(CE,EE)>0.8\text{PCC}(CE, EE) > 0.8 enforces simultaneous sign and rank alignment: under a permutation null over the 336 configurations, the chance probability of PCC>0.8\text{PCC} > 0.8 is only 11.3% (expected correlation 0.00), making our held-out correlations of 0.84–0.88 highly significant (p<10−4p < 10^{-4}). We will include the parser pseudocode and unified utility notation in Appendix F.1.

Functional impact on routing decisions (Q5): As we explicitly noted in Appendix F.1 (Lines 1413–1415), this metric is intended as a decomposition-based attribution signal rather than an internal mechanistic probe of LLM parameters. Crucially, three lines of held-out evidence confirm that this attribution analysis actively improves routing decisions rather than serving as an inert post-hoc rationalization:

  1. Causal generation order: The auditor generates the component attribution analysis inside the reasoning block prior to emitting the JSON stopping and metric-routing decision tokens (Section 3.4.1), directly conditioning the routing action on the component diagnosis.
  2. Held-out GIFT-Eval ablations: Explicitly optimizing attribution agreement (β=0→β=0.1\beta = 0 \to \beta = 0.1) improves held-out routing accuracy and lifts GIFT-Eval MASE/CRPS ranks from 8.3 / 8.5 to 7.3 / 8.2.
  3. Controlled 10,800-series Known-DGP synthetic diagnostic (added to Appendix C/F): Across 10,800 synthetic time series with known ground-truth generating processes (varying trend, periodicity, Gaussian vs. heavy-tailed t3t_3 noise, and regime shifts over explicit linear, periodic, and smoothing experts), EE(x,C)EE(x, \mathcal{C}) achieves a mean correlation of 0.84 (p<0.001p < 0.001) and 89.4% dominant-component agreement with exact simulator component-removal effects, enabling TSOrchestra to reduce expected forecasting regret to 1.41% (vs. 1.76% for naive validation selection and 7.66% for the best single expert).

Reviewer_8ZBeOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses an LLM as a controller to choose the optimization objective for a time series forecasting ensemble.

Critical strengths
  1. The problem is well-defined, and the results demonstrate good performance on GIFT-Eval.
  2. The authors also generated a training dataset of optimization dexplanations, which could be beneficial for research on general data science agents.
Critical weaknesses

Incremental benefit of the LLM controller

The controller searches over a relatively small set of optimization objectives within a fixed candidate model pool. Appendix C.2 already provides relevant comparisons with non-LLM controllers, which I view as AutoML style alternatives. These comparisons support the proposed method, but I would appreciate further clarification of the incremental benefit of using an LLM.

In particular, could the authors clarify how the strongest alternatives compare in terms of available information, training supervision, and computational cost? A simple baseline that evaluates each candidate objective and selects the resulting ensemble using temporally ordered validation would also help establish the value of the iterative auditing process. Reporting the forecasting loss improvement alongside ranks would make it easier to assess whether the benefit justifies the additional complexity through data generation, training, and inference.

Reliability of training labels and controlled synthetic experiments

The training labels are derived from empirical forecasting performance. My concern is that the winning candidate on a finite validation sample may not be reliably better in expectation, especially for noisy series or when competing candidates have similar performance. Although the method includes a confidence-related reward, it is unclear how it accounts for uncertainty in the reference labels themselves.

Could the authors complement the real data experiments with controlled synthetic time series whose generating processes are known? I mean synthetic underlying series, rather than synthetic optimization trajectories or explanations. Varying trend, seasonality, noise, and regime changes would help distinguish systematic selection improvements from fitting noise.

As a diagnostic experiment, the candidate pool could include models with explicit, contrasting inductive biases, such as a linear and periodic model. This could test whether the controller selects effective ensembles as the generating structure changes and whether its explanations agree with measured component-ablation effects.

Questions for authors

How is explanation text converted into the vector (EE(x,C)) used for Equation (5)?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8ZBe

We thank the reviewer for the constructive assessment and for highlighting the central empirical question: what incremental value does the LLM auditor provide beyond standard objective search and non-LLM adaptive controllers, and how reliably do finite-sample validation labels reflect underlying data-generating dynamics? Below, we address each point with matched controller comparisons (including supervision and compute budgets), an exhaustive validation-selection baseline, a bootstrap label-stability analysis, controlled temporal-evolution and known-DGP synthetic diagnostics, and the exact extraction specification of EE(x,C)EE(x, \mathcal{C}).

W1: Benefits of the LLM controller, cost, and loss improvements

To clarify the incremental value of the LLM auditor over AutoML-style objective search and non-LLM controllers, we address W1 from three aspects corresponding to Sections 1–3 below: (1) Value of iterative auditing and controlled reasoning isolation, via a new Exhaustive-CV baseline, matched non-LLM controller comparisons under identical SLSQP and trajectory budgets, and within-backbone ablations holding metadata and textual priors constant; (2) End-to-end forecasting loss reductions reported alongside rank improvements across all 97 settings; and (3) Training supervision and runtime latency quantifying the offline and online overhead.

1. Exhaustive validation baseline and matched controller comparison

To test whether TSOrchestra’s gains come simply from evaluating all candidate objectives and picking the validation winner, we ran Exhaustive-CV across all 97 GIFT-Eval settings. For every evaluation window, Exhaustive-CV runs SLSQP under every candidate objective (MSE, MAE, sMAPE) on the earlier fitting folds and evaluates the resulting configurations on the same held-out pre-test selection fold (using held-out MASE for the MASE comparison and held-out CRPS for the CRPS comparison, while leaving the test split untouched). This baseline directly implements the “evaluate every objective, then select by temporally ordered validation” rule.

ControllerMASE rank ↓CRPS rank ↓
Exhaustive-CV8.58.9
Trace-Transformer8.28.7
Trace-GRU8.48.8
TSOrchestra-R17.38.2

Committing to the raw validation argmin is vulnerable to transient window idiosyncrasies (Appendix A, Figures 5–6 and Appendix H, Table 5), whereas iterative auditing evaluates cross-round stability and alignment and retains the highest-confidence candidate (w⋆=arg⁡max⁡w∈WexploredCk(w)w^{\star} = \arg\max_{w \in \mathcal{W}_{\mathrm{explored}}} \mathcal{C}_k(w)).

The controllers in Table 2 (Appendix C.2) also operate under the exact same candidate-generation and numerical-optimization budget. For every evaluation setting, they use the same frozen foundation-model forecasts (Moirai-2, Sundial, Toto) and evaluate the same three SLSQP-generated candidate weight configurations, obtained with the same number of solver calls (at most three calls per window). TSOrchestra audits these three candidates over at most three rounds and selects one configuration; the non-LLM controllers are evaluated over the same candidate set and are not given additional numerical optimizations, with Trace-GRU and Trace-Transformer trained on the same 2,800 synthetic optimization trajectories. Therefore, the reported difference is not caused by TSOrchestra evaluating more candidate forecasts or receiving a larger SLSQP search budget.

The task-relevant numerical information is also matched across controllers, although its representation differs across controller families. Whereas the non-LLM controllers consume numerical features, recent losses, or model ranks directly, TSOrchestra serializes the same numerical state into a language-based audit and additionally expresses dataset metadata and model-related knowledge as textual priors. Under this matched candidate and solver budget, TSOrchestra-R1 obtains average MASE/CRPS ranks of 7.3/8.2, compared with 8.2/8.7 for Trace-Transformer, 8.4/8.8 for Trace-GRU, and 8.7/8.9 for Learned Gating.

Most importantly, Table 3 (Appendix C.4) controls for the additional metadata and textual model priors by holding the Qwen-2.5-3B backbone, input prompt, candidate pool, and SLSQP optimization budget fixed while changing only the reasoning-training stage. Relative to the prompted Qwen-2.5-3B auditor without R1 training, SFT+GRPO increases Web/CloudOps MASE/CRPS selection accuracy from 0.40/0.300.40/0.30 to 0.50/0.600.50/0.60 and Sales accuracy from 0.50/0.500.50/0.50 to 1.00/1.001.00/1.00, while the SFT-only checkpoint gives intermediate results of 0.40/0.400.40/0.40 and 0.70/1.000.70/1.00 (with Table 1 showing consistent gains across Energy, Nature, and Econ/Fin). Because the input metadata and model priors are unchanged across these variants, these within-backbone gains isolate the contribution of the reasoning-alignment procedure directly.

AuthorsOpenReview ↗

Continue Response

2. Forecasting loss improvements alongside ranks

To complement the rank comparisons (where the 25.5% figure in Section 4.2 denotes average MASE rank reduction from 10.0 to 7.3), the geometric mean of per-setting forecasting loss ratios across all 97 settings shows consistent end-to-end error reductions for TSOrchestra-R1:

  • vs. Moirai-2 (best single foundation model): -7.1% in MASE and -9.4% in CRPS.
  • vs. TimeCopilot (automated ensemble baseline): -4.0% in MASE and -3.1% in CRPS.
  • vs. Per-setting hindsight-best pool member: -2.3% in MASE and -3.7% in CRPS.

3. Training supervision and computational cost

The auditor uses a compact Qwen-2.5-3B backbone trained once offline on 2,800 synthetic trajectories (2,000 SFT samples for 500 steps; 800 GRPO samples for 50 steps; Appendix G.3, Table 4). At deployment, the auditor is invoked only during periodic window calibration: a single audit call takes 0.5–2.0 s, and a full calibration takes 3–4 s (≤3\le 3 rounds). Subsequent forecasting within the window is a linear combination of the frozen member outputs with negligible overhead (Appendix D.1).

W2: Reliability of finite-sample training labels

First, regarding the role of rconfr_{\mathrm{conf}} in Equation (20): it calibrates the auditor’s confidence score Ck\mathcal{C}_k on the discrete Accept/Continue stopping decision against the improvement-tolerance rule in Equation (19), rather than estimating the posterior probability that a candidate objective is optimal in expectation.

Second, to directly quantify finite-sample uncertainty in the reference objective labels, we ran an instance-level bootstrap analysis across 812 training-style GIFT-Eval batches (500 bootstrap resamples per batch):

  • In 94.5% of batches, the reference supervision is either a strictly dominant winner (23.9% of batches win in ≥95\ge 95% of resamples) or a near-tie where the top two validation objectives differ by less than 1% in validation loss (70.6% of batches).
  • Only 5.5% of batches exhibit finite-sample label ambiguity across candidates that differ by more than 1%.

Because hard-label flips are heavily concentrated in near-tie regimes (<1% gap), switching between the top two candidates incurs minimal downstream forecasting regret—a property we verify directly in the controlled synthetic diagnostic below. We will add this bootstrap analysis to Appendix G.2.

W3: Controlled temporal-evolution and known-DGP synthetic experiments

We address this from two complementary angles: (1) the controlled temporal-evolution diagnostics already in the appendices (Figures 4–6 and Table 5), which track how TSOrchestra adapts ensemble weights across evolving time windows, horizons, and metrics over models with distinct inductive biases; and (2) a 10,800-series synthetic benchmark with known data-generating processes (Known-DGP) to isolate finite-sample label reliability, controller routing, and ground-truth component ablations.

1. Controlled temporal evolution in the existing appendices (Figures 4–6 & Table 5)

In our base pool, the three members embody contrasting inductive biases (Section 4.1 & Appendix B/I): Moirai-2 (patch-based transformer capturing high-frequency/short-range patterns), Sundial (explicit trend/seasonal decomposition), and Toto (observability-pretrained for local patterns and irregular spikes). The appendices provide three controlled slices of how weight allocation evolves as temporal structure changes:

  • Evolution across rolling temporal windows (Appendix A.3, Figure 6): Tracking the ensemble weights across successive cross-validation folds (T0,T4,…,T24T_0, T_4, \dots, T_{24}) under MSE, MAE, and sMAPE shows substantial temporal drift as the underlying window shifts (e.g., under MSE, Sundial shifts from 0.470.47 at T0T_0 to 0.130.13 at T12T_{12} and 0.400.40 at T24T_{24}, while Moirai shifts from 0.300.30 to 0.600.60).
  • Adaptation across horizons and domains vs. single-metric volatility (Appendix A.1–A.2, Figures 4–5): Across forecast horizons (Figure 4), weight shifts systematically from Moirai on short horizons toward Sundial and Toto on medium and long horizons. Across domains (Figure 5), fixed metrics exhibit high volatility (e.g., in Sales, Sundial drops from 88% under sMAPE to 7% under MAE; in Healthcare, Toto shifts from 51% under sMAPE to 25% under MAE), whereas the LLM Judge row discovers stable, domain-matched allocations (such as prioritizing Toto at 49% in Web/CloudOps and balanced 42%/37%/21% weights in Transport).
  • Validation-to-test transfer and qualitative reasoning checks (Appendix H, Table 5 & Appendix D.2, F.1, I): Table 5 documents concrete series (BizitObs, ETT1, Jena Weather) where cross-validation error and test error diverge across metrics, while the step-by-step traces in Appendix D.2 ("Mirage Trend") and Appendix I show how the auditor detects trend/seasonal dominance (e.g., BizitObs trend strength 0.8690.869) and rejects noise-fitted weights.
AuthorsOpenReview ↗

Continue Response

2. Controlled Known-DGP synthetic benchmark (10,800 underlying series)

To test label reliability, controller selection, and component ablations under known ground-truth expectations, we constructed a synthetic benchmark of 10,800 time series (108 generating conditions×100 series108 \text{ generating conditions} \times 100 \text{ series}) varying trend (none, linear, piecewise), seasonality (none, P24P_{24}, multi-seasonal P24+168P_{24+168}), noise distribution (Gaussian vs. heavy-tailed Student-t3t_3), and regime shifts. The candidate pool uses three transparent experts with explicit, contrasting inductive biases: an OLS linear-trend model, a Fourier-periodic model, and a level/exponential-smoothing model. Simplex ensembles are fitted via SLSQP under MSE, MAE, or sMAPE over W=3W=3 windows, and true expected scaled MAE is evaluated over K=300K=300 independent future noise draws per series.

Controlled Known-DGP RegimeEqual Weights RegretSingle Expert RegretStatic MSE RegretStatic MAE RegretStatic sMAPE RegretNaive Val-Select RegretTSOrchestra RegretRepresentative Component Effect (ΔLoss\Delta\text{Loss}: Trend / Season / Noise)
Stationary Multi-Regime41.83%12.45%2.32%1.86%1.58%1.97%1.42%−0.219-0.219 / −0.215-0.215 / −0.298-0.298
Trend + Seasonality21.71%11.00%1.82%1.70%1.60%1.70%1.38%−0.359-0.359 / −0.316-0.316 / −0.242-0.242
Heavy-Tailed Noise (Student-t3\text{Student-}t_3)20.90%8.14%2.19%1.79%1.54%1.98%1.45%−0.008-0.008 / −1.042-1.042 / −0.057-0.057
Observable Regime Shift11.66%9.30%1.96%1.87%1.53%1.86%1.39%+0.015+0.015 / −0.251-0.251 / −0.089-0.089
All 10,800 Series18.55%7.66%1.88%1.76%1.52%1.76%1.41%—
  • Label stability vs. expected regret: Across all 10,800 series, a single validation window's hard winner differs from the true expected-best objective in 15.7% of series. However, because 67.7% of series are near-ties (<1% expected loss gap between the top two objectives), finite-sample label flips occur primarily between nearly equivalent simplex weight vectors. Constrained SLSQP ensembling reduces expected regret from 18.55% (equal weights) and 7.66% (validation-best single expert) down to 1.41%–1.98%.
  • Why TSOrchestra outperforms static objectives and naive validation selection: Within individual generating conditions, the optimal objective shifts sharply across noise distributions and temporal structures—for instance, under heavy-tailed multi-seasonal dynamics (Piecewise, P24+168, t3), single-expert regret reaches 40.73% and L2L_2-based MSE degrades to 4.02% regret, whereas robust MAE achieves 1.47%. Naive validation argmin (Val-Select) is vulnerable to finite-window outlier spikes (1.76%–1.98% regret across groups, and up to 2.64%–4.38% in heavy-tailed multi-seasonal cells). By incorporating decomposition statistics and residual volatility into its multi-round audit, TSOrchestra dynamically routes optimization toward regime-appropriate objectives and achieves the lowest expected regret across every regime group (1.38%–1.45%, and 1.41% overall)—mirroring the exact gap between Exhaustive-CV (8.5/8.9) and TSOrchestra-R1 (7.3/8.2) on GIFT-Eval.
  • Agreement between controller explanations and ground-truth component ablations: The rightmost column reports the exact expected loss change when removing each generating component in the simulator. Across the synthetic benchmark, the auditor's extracted explanation vector EE(x,C)EE(x, \mathcal{C}) achieves a mean Pearson correlation of rfaith=0.84r_{faith} = 0.84 (with p<0.001p < 0.001 against a permutation null) and 89.4% dominant-component rank agreement with the ground-truth simulator ablation effects ΔLosstrend,ΔLossseason,ΔLossnoise\Delta\text{Loss}_{\text{trend}}, \Delta\text{Loss}_{\text{season}}, \Delta\text{Loss}_{\text{noise}}. Finally, when an unannounced regime shift occurs exactly at the forecast origin, pre-change validation windows cannot observe the post-origin shift; we document this boundary explicitly in Appendix D.2 and Limitations.
Reviewer_t5QFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TSOrchestra, an ensemble orchestration framework in which constrained SLSQP optimization fits weights over candidate time-series foundation models while a language-model 'metacognitive auditor' chooses whether to accept a solution or continue optimizing under another evaluation metric. The auditor is trained with SFT and GRPO; a decomposition-based SHAP attribution alignment term is used to reward explanations. The default ensemble combines Moirai-2, Sundial and Toto. Results on GIFT-Eval (23 datasets, 97 settings) report strong aggregate ranks in MASE and CRPS, and the appendices include non-LLM routing controls, model-pool tests, training ablations and cost discussion.

Critical strengths
  1. The paper addresses the challenge of dynamically coordinating heterogeneous time series foundation models under evolving temporal patterns. Separating numerical weight optimization from LLM-based decision-making provides a reasonable direction for adaptive forecasting.
  2. TSOrchestra combines constrained ensemble optimization with an LLM-based auditor that determines whether to accept the current weights or explore alternative optimization metrics. The proposed SFT and GRPO training pipeline further incorporates decision accuracy, confidence calibration, and decomposition-based explanation faithfulness.
  3. Experiments on the GIFT-Eval benchmark cover 23 datasets and 97 forecasting settings.
Critical weaknesses
  1. Theorem 3.1 claims a strictly positive ensemble advantage determined by the Temporal Incompatibility Index and the number of candidate models. However, the proof in Appendix E relies on assumptions about regime-specific specialization without rigorously deriving the proposed improvement bound. In particular, temporal incompatibility alone does not guarantee that a convex combination of the available forecasting models will outperform the optimal individual model.
  2. The paper states that GRPO improves forecasting performance through reasoning-based optimization. However, the primary reward is defined by the correctness of accept/continue decisions, whose reference labels are constructed using predefined improvement thresholds and iteration limits. It remains unclear whether this training objective directly encourages better out-of-sample forecasts or primarily teaches the agent to reproduce a heuristic stopping policy.
  3. The proposed faithfulness reward measures the Pearson correlation between SHAP-derived temporal component attributions and explanation-implied effects. However, the attribution vector contains only three components: trend, seasonality, and residual. Correlations computed from such low-dimensional vectors may be unstable. Furthermore, Equations (14) and (17) adopt different performance-score conventions, making the sign interpretation of component contributions insufficiently clear.
  4. Appendix C.1.1 refers to "97 datasets," whereas the main text describes 23 datasets with 97 evaluation settings. The terminology should be standardized to avoid confusion.
  5. Several references appear to be duplicated, such as the entries for TEMPO and In-Context Fine-Tuning for Time-Series Foundation Models. The authors should check the bibliography and ensure consistent formatting.
Questions for authors
  1. Could the authors clarify whether the LLM and non-LLM adaptive controllers receive identical input information and operate under comparable optimization budgets? In particular, how much of the improvement can be attributed to LLM reasoning rather than additional dataset metadata and model-specific priors?
  2. How are the probabilistic outputs of Moirai-2, Sundial, and Toto combined to produce the final predictive distribution? Does the SLSQP solver optimize the weighted point forecasts, predictive quantiles, or complete distributions, and how is consistency between the optimization objective and CRPS evaluation ensured?
  3. How are the accept/continue decisions and next-metric selection jointly optimized during GRPO? Since Equation (19) defines decision labels using a threshold-based stopping rule, while Section G.2 describes selecting metrics based on held-out validation errors, could the authors clarify how these two supervision signals are integrated and how they relate to the final forecasting objective?
6 comments
AuthorsOpenReview ↗

Response to Reviewer t5QF

We thank the reviewer for the constructive feedback and for recognizing the separation between numerical weight optimization and language-based control. We address the theoretical scope, training objectives, attribution scoring, and evaluation protocols point by point below.


W1: What ensemble advantage is actually justified?

The reviewer correctly notes that temporal incompatibility (ITI_T) alone does not guarantee that a convex combination of an arbitrary model pool will outperform its best individual member without structural assumptions. Theorem 3.1 was intended to describe the structured regime-specialization setting in Appendix E, where ITI_T serves as a conceptual motivation rather than a statistic estimated during training or inference, while operational weights are obtained through constrained SLSQP optimization in Equation (3).

To address the reviewer's concern without disrupting the main text, we will expand Appendix E to make the regime assumptions of Theorem 3.1 explicit and introduce a complementary Proposition E.1 for fixed-pool convex combinations:

  • Explicit assumptions for Theorem 3.1 (Appendix E): We will clarify in Appendix E that Theorem 3.1 describes a conditional regime-specialization setting (where a monolithic model suffers cross-regime penalty Δreg\Delta_{reg} relative to regime specialists with excess risk ϵm\epsilon_m and switching cost cswc_{sw}), and that Ω(IT,M)\Omega(I_T, M) parameterizes this conditional advantage rather than acting as an unconditional bound.
  • Complementary Proposition E.1 — Fixed-pool convex-combination guarantee (Appendix E): To bridge regime-level motivation with window-wise SLSQP optimization on a fixed model pool, we add Proposition E.1 in Appendix E. For candidate residual vectors ri∈Rnr_i \in \mathbb{R}^n and best single model j=arg⁡min⁡iLval(ei)j = \arg\min_i L_{val}(e_i), the optimal convex combination w⋆∈Δw^\star \in \Delta strictly reduces validation squared error, satisfying Lval(w⋆)≤Lval(ej)−GvalL_{val}(w^\star) \le L_{val}(e_j) - G_{val} with Gval>0G_{val} > 0, if and only if rj⊤rk<∥rj∥2r_j^\top r_k < \|r_j\|^2 for some candidate kk, where Gval=1nmax⁡k(max⁡(∥rj∥2−rj⊤rk,0))2∥rj−rk∥2G_{val} = \frac{1}{n} \max_k \frac{(\max(\|r_j\|^2 - r_j^\top r_k, 0))^2}{\|r_j - r_k\|^2}. With validation-test generalization discrepancy dgend_{gen}, this directly implies Ltest(w⋆)≤min⁡iLtest(ei)−Gval+2dgenL_{test}(w^\star) \le \min_i L_{test}(e_i) - G_{val} + 2 d_{gen}.

We will include the expanded assumptions and complete proofs of both results in Appendix E. Together, Theorem 3.1 provides the conceptual lens for why regime heterogeneity motivates adaptive model coordination, while Proposition E.1 characterizes exactly when a concrete convex combination improves upon its best individual member on a fixed pool. TSOrchestra instantiates this operationally through window-wise SLSQP optimization and auditor-guided metric routing, without estimating ITI_T or evaluating theoretical bounds at runtime.


AuthorsOpenReview ↗

Continue Response

W2, W3, and Q3: What each training component optimizes

We address these questions together as they address three connected elements of our post-training pipeline: the joint action space, the GRPO objective, and the attribution-agreement reward.

1. W2 & Q3: Joint optimization of accept/continue decisions and next-metric routing

At round kk (for k≤3k \le 3), the auditor outputs a structured JSON response containing a confidence score Ck∈[0,1]\mathcal{C}_k \in [0,1], a stopping decision Dk∈{Accept,Continue}\mathcal{D}_k \in \{\text{Accept}, \text{Continue}\}, and a recommended next metric Lk+1∈LL_{k+1} \in \mathcal{L}. A Continue decision triggers another SLSQP solve under Lk+1L_{k+1}; an Accept decision (or reaching the 3-round cap) terminates exploration, and the system selects the explored weight configuration with the highest auditor confidence: w⋆=arg⁡max⁡w∈WexploredCLLM(w)w^\star = \arg\max_{w \in \mathcal{W}_{\text{explored}}} \mathcal{C}_{\text{LLM}}(w). Thus, SLSQP handles continuous weight optimization, while the auditor governs discrete metric exploration and confidence-based selection.

The two supervision signals in Equation (19) and Appendix G.2.1 supervise complementary parts of this action:

  • Stopping target (Equation 19): Defines the reference Accept/Continue label GG using the relative improvement margin Δ≤τ\Delta \le \tau and the 3-round cap.
  • Next-metric target (Appendix G.2.1): Executes SLSQP under each candidate metric on the training split and labels the metric whose resulting weights achieve the lowest error on the held-out validation split as the optimal routing action Lk+1⋆L_{k+1}^\star.

During SFT, trajectories combine both targets so the policy learns joint stopping and metric-routing proposals. During GRPO (Equation 20), the policy generates complete structured responses and receives sequence-level policy-gradient updates from the composite reward r=clip[0,1](rbase+αrconf+βrfaith)r = \text{clip}_{[0,1]}(r_{\text{base}} + \alpha r_{\text{conf}} + \beta r_{\text{faith}}). Because rconfr_{\text{conf}} rewards high confidence only on correct decisions and final deployment selects w⋆=arg⁡max⁡kCkw^\star = \arg\max_k \mathcal{C}_k, GRPO sharpens both when the agent stops and which explored candidate receives the highest confidence. We will revise Section 3.4 and Appendix G to state explicitly that GRPO optimizes a performance-informed surrogate for routing accuracy and confidence calibration, rather than claiming direct end-to-end optimization of out-of-sample forecasting loss.

2. W2: Empirical evidence from staged training

Tables 1 and 3 quantify how each training stage improves downstream metric/weight selection accuracy:

  • Table 3 (staged ablation): Moving from Prompt-only (A: Qwen-2.5-3B) →\to SFT-only (A+SFT) →\to Full R1 (SFT+GRPO) improves MASE/CRPS selection accuracy from 0.40/0.30→0.40/0.40→0.50/0.600.40/0.30 \to 0.40/0.40 \to 0.50/0.60 on Web/CloudOps, and from 0.50/0.50→0.70/1.00→1.00/1.000.50/0.50 \to 0.70/1.00 \to 1.00/1.00 on Sales.
  • Table 1 (across all 5 domains): Relative to Prompt-only Qwen-2.5-3B, A+R1: Qwen-2.5-3B also improves selection accuracy from 0.29/0.25→0.58/0.620.29/0.25 \to 0.58/0.62 on Energy, 0.13/0.27→0.55/0.550.13/0.27 \to 0.55/0.55 on Nature, and 0.33/0.15→1.00/1.000.33/0.15 \to 1.00/1.00 on Econ/Fin.

We will label Table 3 explicitly as measuring held-out selection accuracy and distinguish this metric from final benchmark MASE/CRPS ranks.

3. W3: Role and ablation of the 3-component faithfulness reward (rfaithr_{\text{faith}})

Rather than serving as a standalone population-level correlation test, the 3-component PCC in Equation (5) acts as a deterministic, within-instance auxiliary alignment score over the three fixed STL semantic axes (Trend, Seasonality, Residual), weighted by β=0.1\beta = 0.1 alongside rbaser_{\text{base}} and rconfr_{\text{conf}} (α=0.2\alpha = 0.2). We will refer to this term precisely as decomposition-level attribution agreement.

To test whether this 3-coordinate grounding signal contributes to downstream forecasting performance or introduces instability, we ran a controlled β=0\beta = 0 ablation across all 97 GIFT-Eval settings. Holding the Table 4 training configuration, SFT initialization, rbaser_{\text{base}}, and rconfr_{\text{conf}} identical while setting β=0\beta = 0 (removing rfaithr_{\text{faith}}) yields:

VariantMASE Rank (↓\downarrow)CRPS Rank (↓\downarrow)
TSOrchestra without rfaithr_{\text{faith}} (β=0\beta = 0)8.38.5
TSOrchestra full model (β=0.1\beta = 0.1)7.38.2

Including rfaithr_{\text{faith}} improves mean rank by 1.0 on MASE and 0.3 on CRPS over the full benchmark, confirming that grounding explanations in STL-decomposition attributions regularizes the GRPO policy in a way that benefits downstream ensemble selection.

AuthorsOpenReview ↗

Continue Response

4. W3: Unified sign convention between Equations (14) and (17)

Equations (14) and (17) describe the same Shapley coalition attribution under opposite sign conventions: Equation (14) is written in terms of lower-is-better forecasting error f(S)f(S), whereas Equation (17) defines utility u(S)=−f(S)u(S) = -f(S) (lines 1402–1406). By linearity of Shapley values, ϕc(u)=−ϕc(f)\phi_c(u) = -\phi_c(f).

Thus, a positive utility attribution ϕc(u)>0\phi_c(u) > 0 is equivalent to a negative error attribution ϕc(f)<0\phi_c(f) < 0: both mean that component cc reduces forecasting error. In our implementation, both CE(x,C)CE(x, \mathcal{C}) and EE(x,C)EE(x, \mathcal{C}) are converted to the utility convention u=−fu = -f prior to computing Equation (5). We will unify Equations (14)–(17) under the utility convention throughout Appendix F.1 to eliminate any sign ambiguity.


W4 and W5: Standardizing terminology and bibliography cleanup

We thank the reviewer for catching these editorial issues:

  • Terminology (W4): GIFT-Eval contains 23 datasets evaluated across 97 settings (varying frequencies and horizons). We will standardize all occurrences—including Appendix C.1.1, Figure 8, and Section 5—to read “97 evaluation settings across 23 datasets.”
  • Bibliography (W5): We audited the entire bibliography and merged all duplicate entries, including Cao et al. (2024a/b, TEMPO), Faw et al. (2025a/b, In-Context Fine-Tuning), Gruver et al. (2023a/b, LLMTIME), Luo et al. (2025a/b, Time-R1), Tang et al. (2025a/b), and Wang et al. (2025a/b, ChatTime).

Q1: Matched information, optimization budgets, and isolating LLM reasoning

We evaluate the contribution of the LLM auditor at two complementary levels:

1. Matched candidate pool and SLSQP budget across controller families (Table 2): All controllers in Table 2 operate over the exact same foundation-model forecasts (Moirai-2, Sundial, Toto) and the same per-window numerical optimization budget (at most three SLSQP candidate weight configurations generated per window). None of the non-LLM controllers receive fewer solver calls or a smaller candidate pool. In addition, we evaluated an Exhaustive-CV*baseline that solves SLSQP under all candidate metrics on every window and picks the configuration with the lowest held-out validation loss; Exhaustive-CV achieves mean MASE/CRPS ranks of 8.5 / 8.9.

Regarding input information, Trace-GRU (8.4 / 8.8) and Trace-Transformer (8.2 / 8.7) consume the exact same numerical multi-round optimization trace as TSOrchestra-R1 (7.3 / 8.2). While the underlying numerical evidence and solver budgets are matched, TSOrchestra serializes the state into natural language and incorporates dataset metadata and textual model priors. Therefore, we interpret Table 2 as demonstrating the system-level advantage of the complete language-based auditor over non-LLM sequence and gating controllers under an equal SLSQP budget, rather than claiming a pure causal isolation of language representation from textual priors in Table 2 alone.

2. Strict within-backbone isolation with identical inputs and priors (Tables 1 and 3): To isolate the effect of LLM reasoning alignment while holding dataset metadata, textual model priors, backbone architecture (Qwen-2.5-3B), candidate pool, and SLSQP budget strictly identical, Tables 1 and 3 compare checkpoints across training stages:

  • Prompt-only (A: Qwen-2.5-3B) vs. A+SFT vs. A+R1 (SFT+GRPO): With identical input prompts (including all metadata and priors), selection accuracy increases from 0.40/0.30→0.40/0.40→0.50/0.600.40/0.30 \to 0.40/0.40 \to 0.50/0.60 on Web/CloudOps and 0.50/0.50→0.70/1.00→1.00/1.000.50/0.50 \to 0.70/1.00 \to 1.00/1.00 on Sales (Table 3), as well as 0.29/0.25→0.58/0.620.29/0.25 \to 0.58/0.62 on Energy, 0.13/0.27→0.55/0.550.13/0.27 \to 0.55/0.55 on Nature, and 0.33/0.15→1.00/1.000.33/0.15 \to 1.00/1.00 on Econ/Fin (Table 1).

Because metadata and model priors are held constant across these variants, these gains directly measure the contribution of our SFT+GRPO reasoning-alignment pipeline. We will revise Appendix C.2 to explicitly document the shared candidate sets, equal solver calls, and two-level attribution across Tables 1–3.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.