- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ECLIPSE, an LLM-based agent framework for time-series forecasting and question answering with textual context. Instead of using an LLM to generate forecasts directly, the method uses it to plan a sequence of operations on the time series. The system first determines the task type and whether additional contextual knowledge is needed. For context-dependent tasks, it uses scenario-specific rules and retrieves reference patterns from a Temporal Archetype Bank. The planner can then modify the observed history before forecasting with Moirai and adjust the resulting forecast according to future events or constraints. For QA tasks, it instead generates a plan to extract the relevant temporal evidence. An audit module checks the generated operations and outputs, with replanning or fallback when needed. The authors evaluate ECLIPSE on CiK and TemporalBench. On CiK, ECLIPSE with DeepSeek-Chat achieves an average RCRPS of 0.174, compared with 0.188 for TimeClaw, although the previously reported Llama-3.1-405B direct-prompting result is better at 0.159. ECLIPSE also achieves the best contextual reasoning accuracy across the four TemporalBench datasets. The paper further includes a shared-toolchain comparison and component ablations to examine the contribution of the different intervention steps.
Critical strengths
1.I find the controlled comparison in Table 3 useful. The authors keep the LLM, forecasting backbone, retrieved references, and tool library fixed, making it easier to assess the contribution of the proposed workflow rather than differences in the underlying models or tools. This is a meaningful addition to the evaluation, especially for an agent-based framework.
2.The paper is fairly transparent about its limitations. The authors acknowledge that ECLIPSE does not outperform the previously reported Llama-3.1-405B result on CiK or the standalone Moirai model on historical-information tasks. They also report the extreme errors in some ablations and the relatively low repair rates of the audit module. I appreciate that these unfavorable results are not omitted.
Critical weaknesses
1.Limited novelty and unclear attribution of the performance gains. My main concern is that the methodological novelty is rather limited. Most components, including LLM planning, retrieval, temporal preprocessing/post-processing, and verification, are already well established. The two-sided intervention design is reasonable, but the contribution seems to be mainly in how these components are combined. More importantly, Table 3 does not control for inference budget: ECLIPSE uses 7.2 LLM calls per task on average, compared with one for the single-agent baseline. It is therefore unclear how much of the gain comes from the proposed orchestration itself. Much of the supporting analysis also relies on the authors' DTR metric rather than the original RCRPS metric.
2.Concerns about robustness and generalization. The ablation results in Table 13 are concerning. Removing certain components leads to extremely large errors on a few instances, suggesting that the planner can generate numerically unstable interventions. The trimmed statistics help characterize typical performance but do not fully address this issue, particularly given the audit module's limited ability to repair unsupported interventions (Table 18). Also, the seven scenario categories are designed around CiK-style tasks, so it is unclear how well the framework generalizes to unseen types of context.
Questions for authors
1.Table 3 controls for models and tools but not inference budget (7.2 vs. 1 LLM calls per task). Could the authors provide a budget-matched comparison, even on a subset? It would also be helpful to see whether the improvement holds under RCRPS rather than DTR.
2.The baselines in Table 2 appear to be taken from the original TemporalBench paper. Were the dataset version and evaluation protocol identical? Also, what explains the large improvement on Causal Chambers-HU (68.7% vs. 28.67% for the strongest baseline), where external context is not required?
3.Table 13 shows catastrophic errors when certain components are removed. Does the executor have general safeguards against numerically unstable interventions? Would simple bounds on intervention magnitudes prevent these failures while preserving the gains in Table 4?
4.Could the authors clarify how the Temporal Archetype Bank is separated from the CiK evaluation data, particularly when they share the same data sources? A brief description of the time ranges and data splits would be helpful.
4 comments
Reply to Q2: TemporalBench evaluation protocol and the improvement on Causal Chambers-HU
Thank you for this question. Yes, we used the same dataset version and evaluation protocol as the original TemporalBench paper. The corresponding baseline results in Table 2 are taken from that paper.
The substantial improvement on Causal Chambers-HU mainly stems from how ECLIPSE handles series scale and relative changes. Our framework focuses on handling the effects of contextual information on time series. Because many contextual effects are expressed as proportions or relative changes, ECLIPSE uses scale-normalized computations in its numerical analysis, which also benefit historical understanding. Specifically, the pressure readings are approximately 96,044 Pa, while fluctuations are only a few pascals; many samples exhibit an end-to-end relative change below 0.05%. Compared with models that directly interpret raw numbers, ECLIPSE better assesses the relative magnitude of these changes, reducing the tendency to mistake small fluctuations for pronounced trends.
HU follows a self-contained numerical analysis path without invoking scenario classification, retrieval, or intervention modules. The gains therefore reflect the benefits of executable numerical analysis without requiring additional context. The T1 solving process does not access reference answers; these are used only for final scoring.
Reply to W2: Generalization across contextual settings
Thank you for this comment. The seven scenario categories were indeed informed by CiK, but they characterize how context participates in forecasting—for example, historical events, future constraints, explicit rules, and indirect reasoning—rather than depending on a particular dataset. New contexts can fall within existing abstractions while requiring different operations, temporal scopes, and parameters. Generalization across settings therefore does not necessarily require introducing a new category. The framework also supports custom scenarios and tools; the seven categories are not a closed set of task templates.
We additionally evaluated ECLIPSE on TemporalBench, whose event-informed forecasting tasks span retail, electricity, experimental control, and clinical domains, providing different information conditions from CiK tasks with explicitly stated effects. For example, PSML provides qualitative event descriptions without specifying effect magnitudes; Causal Chambers provides changes in control variables without explicit response equations; and MIMIC provides medications and doses without directly specifying the target variable’s response. These settings require the system to determine appropriate computations from less complete contextual information. They examine whether the same planning abstractions can accommodate different forms of information and domain mechanisms, rather than whether the system can match a particular CiK task template.
Reply to W2 & Q3: Numerical stability and executor safeguards
Thank you for this suggestion. The executor includes local formula-convergence checks and a configurable magnitude guardrail: when an intervention’s mean relative change exceeds the threshold, the adjustment is reverted to the pre-intervention forecast. The reported runs enabled the local convergence checks but not the general magnitude guardrail.
It is important to distinguish legitimate large changes from amplification caused by incorrect execution. For example, in the “fivefold withdrawals” task, the full program applies a localized fivefold adjustment, whereas the ablated program mistakenly compounds the multiplier daily, producing DTR of approximately 10⁷. Enabling the guardrail reverts the erroneous recurrence while retaining the correct adjustment.
The reference DTR for full ECLIPSE is 0.190. We replayed the saved ablation programs under different thresholds while keeping the programs and forecast inputs fixed:
| Ablation | Unbounded | Threshold 0.8 | Threshold 2 | Threshold 5 |
|---|---|---|---|---|
| −Contextual Priors | 47,853.4 | 0.238 | 0.237 | 0.232 |
| −Audit | 105.2 | 0.191 | 0.190 | 0.187 |
| −Direct Context | 48.5 | 0.206 | 0.195 | 0.191 |
Simple bounds prevent these numerical divergences, and all six paired effects in Table 4 remain statistically significant at the three listed thresholds. At thresholds 2 and 5, ECLIPSE’s improvements over the TSFM are 0.0171 and 0.0184, respectively, compared with the original 0.0170.
However, this protection can also discard grounded interventions that the system can already execute. Exceeding the threshold causes the entire adjustment to be reverted; consequently, legitimate contextual corrections may be discarded when future events or physical relationships require large responses. For example, in PressureFromSpeed, rotational speed increases from 661 to 1590. The 0.8 threshold reverts the physically supported adjustment, increasing DTR from 0.258 to 0.882. SVAR equations and irradiance inference based on zenith angle and cloud conditions can likewise require large changes. Among physically or equation-driven tasks, the 0.8 threshold reverts 21 beneficial interventions, causing a cumulative DTR increase of 4.19; threshold 5 still reverts four, with a cumulative increase of 0.55. Preserving aggregate gains therefore does not mean preserving every attainable benefit from context: some are sacrificed by the threshold rule.
This also motivates the audit module: errors may involve direction, temporal scope, rule interpretation, or contextual grounding, and cannot be judged solely by magnitude. Some incorrect adjustments are small, while some correct operations must produce large changes. The audit checks consistency with context and constraints, complementing deterministic numerical safeguards. Simple bounds provide a numerical backstop but cannot replace checks of an intervention’s grounding and semantics.
Reply to Q4: Separation of the Temporal Archetype Bank from CiK
Thank you for this question. CiK contains 355 evaluation instances, allowing us to inspect each instance’s data source and historical and forecast windows. We have checked this information and separated the Temporal Archetype Bank through non-overlapping time ranges, different stations, and data splits. Sharing an upstream data source does not imply using the same evaluation windows.
For example, the solar irradiance data exclude 2022—the year used by CiK—and come from different stations. The Montreal fire and rescue data use only 2017 and 2020, neither of which overlaps the historical or forecast windows of the corresponding CiK instances. Other sources use their native training splits or training prefixes. Verification combines year, station, and split metadata with additional window-level numerical comparisons for series at matching frequencies. We will add a concise description of the time ranges and data splits to the revised manuscript.
Dear Reviewer 971o,
Thank you for your thoughtful review. We hope our new budget-matched comparisons and RCRPS evaluations, numerical safeguard analysis, and clarifications on the TemporalBench protocol and retrieval-bank separation address your concerns. If so, we would appreciate your consideration of a higher score. We are happy to answer any further questions.
Best regards,
The Authors
Reply to W1: Methodological novelty
Thank you for this comment. The idea that context can improve forecasting is indeed long-standing and intuitive. Our research is motivated precisely by the substantial gap between this idea and the practical ability to realize it: effectively incorporating heterogeneous text into forecasting for series with different dynamics remains challenging. Across settings, what changes is not only the numerical distribution, but also how text conveys information and how that information should influence the forecasting process.
The semantic understanding of LLMs and the execution capabilities of agents offer a new approach to this problem. ECLIPSE makes this concrete by allowing context to determine which historical observations forecasting should rely on, which reference patterns are relevant, and how future trajectories should be adjusted. It translates these judgments into the selection, construction, and execution of programs while retaining the numerical modeling capabilities of TSFMs. We fully acknowledge the established foundations of the individual components. Our intended contribution is a method that coordinates these capabilities to address the alignment between contextual information and time-series computation, together with empirical evidence of its effectiveness.
Reply to W1 & Q1: Computational budget, attribution, and RCRPS
Thank you for this suggestion. We added budget-matched comparisons and RCRPS evaluations covering all 355 instances. To directly examine the contribution of orchestration, we provided an enhanced single agent with the same parsed inputs, category-specific guidance, retrieved information, tools, and executor as ECLIPSE, allowing either one planning call or up to eight planning–execution–checking rounds. We report the budget-matched comparison before audit and additionally include ECLIPSE’s audited result:
With matched inputs and call counts, ECLIPSE retains the lowest numerical errors, although its difference from the enhanced single-call agent is not statistically significant: the paired RCRPS difference is −0.008, with a 95% CI of [−0.035, +0.014]. The multi-round agent consumes more calls and approximately 27.8K tokens without improving average performance. Thus, additional reasoning budget does not automatically translate into gains; meanwhile, a single agent given ECLIPSE’s contextual processing capabilities can achieve comparable performance.
Under RCRPS, the overall ranking with the revised executor remains broadly consistent. ECLIPSE significantly outperforms Moirai, the fixed template, and the original Table 3 single agent. Its difference from RAG + Post-processing is not significant (−0.011, 95% CI [−0.025, +0.004]). We also report results with the original executor separately: ECLIPSE, RAG + Post-processing, and the enhanced single-call agent obtain RCRPS values of 0.184, 0.183, and 0.187, respectively. Finally, the current audit modifies point forecasts but not sampled trajectories, so its point-forecast improvements do not directly improve RCRPS. These additions distinguish the effects of computational budget, executor revisions, and evaluation metrics.