- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents TSOrchestra, a framework for dynamically combining pretrained time-series forecasting models through simplex-constrained weight optimization. Rather than directly generating forecasts or unrestricted ensemble weights, an LLM-based auditor selects among solver-generated candidates using model information, validation summaries, and optimization traces. The auditor is trained using supervised trajectories followed by R1-style reinforcement learning with GRPO. The framework also introduces the Temporal Incompatibility Index I_T, intended to quantify divergence between historical patterns and future outcomes and motivate dynamic adaptation across regimes or sliding windows.
The empirical study reports encouraging forecasting performance and includes comparisons against individual forecasting models, random metric routing, supervised versus SFT+GRPO training, and non-LLM adaptive controllers. However, two claims central to the paper are not adequately supported. First, the claimed theorem establishing a strictly positive advantage for the ensemble does not follow from the assumptions stated in the manuscript. Second, the clipped GRPO reward appears to eliminate the faithfulness signal for correct and sufficiently confident responses. These issues substantially weaken the paper’s claims of theoretically justified orchestration and faithful reasoning.
Critical strengths
-
Sensible separation between numerical optimization and language-based control The LLM auditor does not directly produce numerical forecasts or unconstrained ensemble weights. Instead, it selects among candidates generated by a conventional optimization procedure. This is a more controlled and testable use of an LLM than allowing the model to generate arbitrary numerical outputs.
The design also makes it possible to distinguish errors in forecasting, optimization, and orchestration. This separation is valuable for both empirical evaluation and operational auditing.
-
Stronger-than-usual adaptive baselines
The supplementary evaluation goes meaningfully beyond comparisons with individual forecasting models. It includes random metric routing, repeated inference runs, SFT versus SFT+GRPO, and non-LLM adaptive controllers, including controllers that consume optimization traces.
These controls are considerably more informative than static or single-model baselines and provide credible evidence that the orchestration interface has empirical value. The alternative and expanded model-pool experiments also suggest that the method is not tied to a single fixed collection of pretrained forecasters.
-
Potentially useful empirical orchestration framework
Even if the theoretical guarantee and strong faithfulness claims are removed, the paper may still offer a useful empirical method for adaptively combining heterogeneous pretrained forecasting models. The constrained candidate-selection formulation could be practically attractive in settings where completely unrestricted LLM control would be difficult to validate.
Critical weaknesses
-
Theorem 3.1 and the corresponding appendix claim a strictly positive advantage .At the stated level of generality, this result appears false. The theorem does not specify:
- restricted hypothesis class for the monolithic predictor *quantitative smoothness, Lipschitz, capacity, or regularity constraint
- An information advantage available only to the ensemble or router
- Approximation or competence assumptions on the constituent predictors
- Conditions under which the ensemble can identify the relevant local regime
- A bound on routing or switching error
- A relationship connecting I_T to the stated logarithmic/logistic excess-risk gap
For example, a monolithic predictor that represents the true forecasting function exactly can have zero loss even when nearby inputs have very different outputs. If that predictor is also included in the model pool, the optimized ensemble can achieve zero loss, but it cannot improve on zero by the strictly positive amount asserted by the theorem. More generally, a sufficiently expressive monolithic model may represent the entire ensemble and routing function.
The proof sketch does not resolve this issue. It assumes a partition into locally compatible regimes and negligible switching cost, These assumptions also describe an idealized regime-routing construction that is not obviously equivalent to the implemented convex combination of fixed pretrained forecasts.
The related Kolmogorov-complexity argument appears similarly incomplete. A weighted average of local description lengths does not automatically account for the description length of the complete sequence, regime assignments, routing mechanism, or switching information.
The paper should either provide a correctly scoped theorem with explicit hypothesis classes, losses, approximation conditions, and routing assumptions, or present I_T solely as a heuristic motivation and remove the claimed guarantee.
-
The reward appears to saturate and remove the faithfulness signal. For a correct and valid response with c≥0.5 and , the minimum possible unclipped reward is:
Therefore, the final clipped reward is exactly one for every such output. A correct response with a maximally negatively correlated explanation receives the same reward as a correct response with a maximally positively correlated explanation.
Unless the implementation differs from the manuscript, GRPO therefore cannot use this reward to distinguish faithful from unfaithful explanations among correct, confident responses. This directly undermines the claimed role of reinforcement learning in improving explanation faithfulness.
-
Attribution agreement is not equivalent to explanation faithfulness. Even with a non-saturating reward, agreement with a Shapley-style attribution does not by itself establish that a generated explanation faithfully represents the auditor’s actual decision process. It may demonstrate consistency with an external attribution rule, but not necessarily causal or mechanistic faithfulness. The paper needs a reproducible description of how free-text explanations are converted into numerical vectors suitable for correlation with the attribution target. It should also clarify the sign convention when converting between loss contributions, forecasting utility, and metric-selection rationales.
-
The operational adaptation rule is insufficiently specified. The paper appears to describe the system in different places as updating weights for every sliding window, continuously tracking I_T, or updating periodically or when a regime change is detected. These are operationally different systems. Without extra information, the reported method is difficult to reproduce.
Questions for authors
-
Theorem 3.1: What explicit assumptions on the monolithic hypothesis class, constituent predictors, information available to the router, and switching cost are required for the claimed strictly positive ensemble advantage? In particular, what excludes a zero-error monolithic predictor or one capable of representing the ensemble itself?
-
Derivation of the bound: Can the authors provide a complete derivation of the specific I_T-dependent logarithmic/logistic gap in Theorem 3.1? The current proof sketch appears to assume locally compatible regimes and negligible switching cost, then assert the bound without deriving it
-
Reward saturation: Is the GRPO reward implemented exactly as written? If so, is it correct that every correct, valid response with c≥0.5c receives reward one regardless of its faithfulness correlation?
-
Effective faithfulness signal: What fraction of training rewards saturate, and what signal allows GRPO to distinguish faithful from unfaithful explanations among correct decisions? Please include an ablation using a non-saturating reward
-
Faithfulness measurement: How are free-text explanations converted into the numerical attribution vectors used to calculate correlation, and what held-out evidence shows that this agreement reflects the auditor’s actual decision process rather than a post-hoc rationale?
15 comments
Continue Response
2. Rationale of Clipped Reward Eq. (20) and Non-Saturating Ablations (W2, Q3, Q4)
As specified in Appendix G.1.1 (Line 1521), the confidence term in Eq. (20) is defined piecewise as , with and . While responses that are simultaneously correct () and highly confident () reach the upper clip of 1.0, this clipping is an intentional design choice that works in tandem with SFT initialization and unsaturated exploration groups:
1. Regularization role of upper clipping and active gradient statistics: Prior to GRPO, all 2,000 SFT trajectories are already filtered for high attribution agreement () over 500 steps, and KL regularization () in Eq. (21) anchors the policy to this SFT prior. During GRPO, capping the composite reward at 1.0 acts as a safeguard against auxiliary reward over-optimization: once a rollout achieves both a correct routing decision and high confidence (), the upper clip prevents the 3-component auxiliary attribution score from distorting the confidence ranking of already well-calibrated routing actions, while concentrating attribution gradients on uncertain () and incorrect () rollouts. Empirically, across our 50-step GRPO logs (800 prompts, rollouts per prompt), 61.6% of individual rollouts—and 74.2% of the 4-sample rollout groups—contain at least one unsaturated trajectory where actively differentiates responses and yields non-zero group advantage variance in Eq. (22). Specifically, drives gradients on: (i) correct boundary trajectories with moderate confidence (unclipped sums in ), and (ii) incorrect stopping rollouts (), where the score is and positive attribution alignment yields strictly positive clipped rewards in .
2. Full-benchmark ablations in Appendix G.1.1 ( and unclipped ): To verify both that Eq. (20) delivers an effective attribution signal and to test the reviewer's unclipped alternative, we ran two controlled retraining experiments under the exact Table 4 configuration across all 97 GIFT-Eval settings:
- Without attribution reward (): Retains and while setting .
- Unclipped non-saturating variant (): Rescales attribution agreement to without upper clipping (), followed by group normalization in Eq. (22), applying attribution gradients uniformly across all confidence levels .
| GRPO Reward Formulation (All 97 GIFT-Eval Settings) | MASE Rank () | CRPS Rank () | Held-Out PCC(CE, EE) () |
|---|---|---|---|
| TSOrchestra without attribution reward () | 8.3 | 8.5 | 0.71 |
| TSOrchestra unclipped reward variant () | 7.4 | 8.2 | 0.87 |
| TSOrchestra default clipped reward (Eq. 20, ) | 7.3 | 8.2 | 0.84 |
These results confirm both aspects of our design: (i) removing () degrades mean ranks by 1.0 on MASE and 0.3 on CRPS (and drops PCC from 0.84 to 0.71), proving that our default Eq. (20) formulation effectively leverages across the 74.2% unsaturated rollout groups; and (ii) while removing the upper clip () pushes surface attribution correlation slightly higher (), forcing confident correct decisions to further optimize the 3-component proxy slightly perturbs downstream confidence selection ( in MASE rank). Thus, Eq. (20) strikes the optimal balance between attribution grounding and routing accuracy. We will add this saturation analysis and ablation table to Appendix G.1.1.
Continue Response
3. Deterministic Extraction of and Functional Impact (W3, Q5)
Reproducible rule-based extraction (Appendix F.1): The auditor uses a fixed output schema containing three explicitly tagged component-importance fields in a fixed order: Trend, Seasonality, and Residual. As detailed in Appendix F.1 (Equations 17–18 and Lines 1421–1456), each component is characterized by its ordinal importance role (dominant, secondary, minor, or none, establishing rankings such as trend > seasonal > residual) and its attribution direction (whether the component improves forecast utility or increases error, checked in Equation 18 against Wrong direction, Overstatement, Understatement, and Missed pattern). A deterministic, rule-based parser reads only these three structured fields and maps their categorical levels and direction signs through a fixed lookup table—mapping {dominant, secondary, minor, none} to —to construct the three-dimensional vector along the Trend, Seasonality, and Residual coordinates. The surrounding prose is not parsed, and no secondary LLM, learned extractor, or manual annotation is involved.
Unified sign convention and discrete null significance: Equations (14) and (17) express the same Shapley attribution under opposite sign conventions: Equation (14) uses forecasting error , whereas Equation (17) defines utility (Lines 1402–1406). By Shapley linearity, . In our implementation, both and are converted to the utility convention prior to computing Equation (5), with zero-variance vectors defaulting to 0. Because each coordinate of takes 7 signed values ( non-constant signed configurations), requiring enforces simultaneous sign and rank alignment: under a permutation null over the 336 configurations, the chance probability of is only 11.3% (expected correlation 0.00), making our held-out correlations of 0.84–0.88 highly significant (). We will include the parser pseudocode and unified utility notation in Appendix F.1.
Functional impact on routing decisions (Q5): As we explicitly noted in Appendix F.1 (Lines 1413–1415), this metric is intended as a decomposition-based attribution signal rather than an internal mechanistic probe of LLM parameters. Crucially, three lines of held-out evidence confirm that this attribution analysis actively improves routing decisions rather than serving as an inert post-hoc rationalization:
- Causal generation order: The auditor generates the component attribution analysis inside the reasoning block prior to emitting the JSON stopping and metric-routing decision tokens (Section 3.4.1), directly conditioning the routing action on the component diagnosis.
- Held-out GIFT-Eval ablations: Explicitly optimizing attribution agreement () improves held-out routing accuracy and lifts GIFT-Eval MASE/CRPS ranks from 8.3 / 8.5 to 7.3 / 8.2.
- Controlled 10,800-series Known-DGP synthetic diagnostic (added to Appendix C/F): Across 10,800 synthetic time series with known ground-truth generating processes (varying trend, periodicity, Gaussian vs. heavy-tailed noise, and regime shifts over explicit linear, periodic, and smoothing experts), achieves a mean correlation of 0.84 () and 89.4% dominant-component agreement with exact simulator component-removal effects, enabling TSOrchestra to reduce expected forecasting regret to 1.41% (vs. 1.76% for naive validation selection and 7.66% for the best single expert).
Response to Reviewer xbxW
We thank the reviewer for recognizing TSOrchestra's separation between numerical optimization and language-based control, the strength of our trace-conditioned adaptive baselines, and the practical value of constrained candidate selection.
TSOrchestra is designed as an empirical orchestration framework where theoretical analysis motivates adaptive coordination and constrained SLSQP together with trace-audited routing drives operational performance. Before addressing each technical point, we highlight the empirical evidence, including new full-benchmark ablations across all 97 GIFT-Eval settings, confirming that our formulation is both sound and effective as designed:
1. Scope of Theorem 3.1 and Complementary Appendix E Formulations (W1, Q1, Q2)
Theorem 3.1 and the Temporal Incompatibility Index IT are introduced as a conceptual motivation under the structured regime-specialization setting defined in Appendix E (where regimes are locally compatible, candidate models specialize across regimes, and a single fixed predictor faces cross-regime trade-offs), while operational weights are computed strictly via constrained SLSQP in Eq. (3). The counterexample of an unrestricted zero-error monolithic predictor corresponds to the boundary case where cross-regime mismatch is zero.
Rather than altering the main text's narrative flow, we will expand Appendix E to state these underlying regime conditions in explicit algebraic form and add a complementary Proposition E.1 that directly characterizes our fixed-pool SLSQP optimization:
Together, Theorem 3.1 and Proposition E.1 form a complete two-level justification: Theorem 3.1 provides the regime-level lens explaining why non-stationary temporal dynamics motivate adaptive multi-model coordination, while Proposition E.1 establishes the exact window-level residual-decorrelation condition under which our operational SLSQP convex combination provably improves upon the best single model in a fixed pool.