acceptodds
Under review as a conference paper at ICLR 2027

When Events Move the Curve: An Event-Informed Reasoning Framework for Multimodal Time Series Forecasting

10 views

Abstract

Strong time series forecasting models can effectively capture endogenous patterns from numerical histories, but they cannot directly access the external events described in text. Existing multimodal methods either compress text into latent features, underutilizing event semantics, or rely on LLMs to directly generate numerical forecasts, risking imprecision due to the language–number modality gap. Consequently, they struggle to retain the forecasting capability of numerical models while fully exploiting textual event semantics and accurately translating them into future numerical changes. We propose After (Adjustment Framework through Textual Event Reasoning), an event-informed framework that retains a trained numerical backbone and uses textual events to adjust its predictions without additional parameter optimization. After first employs History-Aware Event Reasoning to identify forecast-relevant events from the textual context. It then derives forecast offsets through contextual event reasoning, case-based analogical reasoning over an Event-Response Memory, and direct numerical transfer of retrieved continuations. These complementary offset candidates are robustly integrated and added to the backbone forecast. Experiments across nine real-world multimodal time series datasets and multiple forecasting horizons demonstrate the effectiveness and broad applicability of After against strong numerical and multimodal baselines. Our code is available at https://anonymous.4open.science/r/AFTER-ICLR–0638/

Primary area:
learning on time series and dynamical systems
Keywords:
Multimodal Time Series Forecasting, Textual Event Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dUhfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses multimodal time series forecasting. Numerical forecasters learn endogenous patterns from history but cannot see the external events described in text, while existing multimodal methods either compress text into latent features, underusing event semantics, or let an LLM generate numbers directly, which suffers from the language–number gap. The authors propose AFTER, which keeps a trained numerical backbone and lets an LLM estimate only an offset to the backbone forecast, without training any new parameters. AFTER works in three steps. History-Aware Event Reasoning (HAER) reads the history and the texts, decides whether any event still affects the future (none / weak / active), and writes an event summary. Complementary Offset Reasoning (COR) then estimates offsets along three routes: LLM reasoning from the current event state, LLM reasoning by analogy with similar cases retrieved from a training-set memory, and direct transfer of the retrieved cases’ continuations. Robust Offset Integration (ROI) takes the element-wise median of the three offsets and adds it to the backbone forecast.

Critical strengths

The most valuable aspect is the division of labor: the numerical model forecasts and the LLM only estimates a correction. The backbone does not need retraining, so AFTER can be attached to any trained forecaster; the authors observe gains on all seven backbones, and MSE decreases in 95 of the 108 main settings. When the text carries no useful information, AFTER leaves the forecast unchanged: on Environment, 94% of the samples are labeled none and the predictions barely move, so it does not degrade a good forecast. Table 3(b) shows that letting the LLM generate the whole trajectory greatly increases the error (Economy MSE rises from 0.193 to 0.369), which supports the claim that anchoring on the numerical forecast is more reliable than asking the LLM for numbers directly. This matters because it offers a training-free, traceable, and low-risk way to connect an LLM’s understanding of events to existing forecasting systems.

Critical weaknesses
  1. All test periods end before mid-2024 (e.g., roughly 2009–2024 for Social Good; Health and Traffic both cover the 2020 pandemic), very likely before the knowledge cutoff of DeepSeek-V4-Flash. The HAER and COR prompts contain the absolute input and forecast time ranges and the target variable (e.g., the U.S. unemployment rate), and the texts contain specific figures such as “unemployment increased to 9.5%”. The LLM could therefore recognize the episode and write down what happened next. The de-identification check covers only Economy and Social Good with iTransformer and keeps the specific numbers in the texts, so it does not rule this out. The No Text condition in Table 4 falls back to the backbone by design, so it cannot separate event information in the text from the LLM’s own memory. Since the central claim is that the gains come from reasoning about events, this concern directly determines whether the conclusion holds.

  2. The average gains over the backbones are only 3.34% (MSE) and 2.03% (MAE); MSE drops by only about 0.2%–1.9% on Agriculture, Climate, and Energy, and Environment is slightly worse. The paper does not state the number of seeds, reports no standard deviations or significance tests, and does not describe the randomness of LLM outputs, while the monthly test segments are short (about 60 months for Security) and the test windows overlap heavily. Table 1(a) in the main text shows only three “representative” datasets, yet Tables 17–19 show that MM-TSFLib beats AFTER on Climate, Health, and Environment under all three backbones, and Table 2 omits MM-TSFLib and TaTS. Each sample also needs three LLM calls (about 7.7 s versus 1.8 ms for the backbone), and the cost of building the memory on the training set is not reported. The related work also misses directly relevant studies, such as From News to Forecast (NeurIPS 2024), which uses LLMs to analyze news events for forecasting, retrieval-augmented forecasting (e.g., RAF, TimeRAG), and the classic forecasting literature on judgmental adjustment of statistical forecasts, which may lead readers to think this direction has not been explored before.

Questions for authors

Dear authors, the answers to the following questions could directly affect my assessment.

  1. Could you evaluate on data after the LLM’s knowledge cutoff, for example by collecting 2025–2026 data from the same sources, or use an LLM whose cutoff precedes the test period? A cheaper control is to keep the dates, history, and dataset description but replace the texts with texts from other periods; if the gains remain, they come from the LLM’s memory rather than from the texts.
  2. How many seeds were run? Are LLM outputs sampled, and at what temperature? Could you report mean ± standard deviation and a paired significance test, and show how much of the gain comes from a few extreme periods such as the 2020 pandemic, and what the results are without them?
  3. How was the context_guidance written for each dataset, and was it designed using only the training and validation sets? The problem is formulated for C variables; do the experiments forecast only the OT variable?
3 comments
AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer dUhf,

We sincerely thank you for your time and for the thoughtful and constructive comments on our paper. We have carefully addressed each of your questions and concerns, conducted additional controlled experiments, and revised the manuscript accordingly.

As the discussion period approaches its conclusion, we hope that our responses, new experimental evidence, and manuscript revisions have adequately addressed your concerns. If any questions remain, please do not hesitate to let us know. We would be pleased to provide further clarification promptly and in detail.

Best regards,
The Authors

AuthorsOpenReview ↗

A Kind Follow-up on Our Rebuttal

Dear Reviewer dUhf,

We sincerely appreciate your thoughtful and constructive comments, which have helped us improve our paper.

It has been five days since we submitted our detailed responses, additional experiments, and revised manuscript. As the discussion period is approaching its conclusion, could you kindly review our responses and let us know whether your concerns have been addressed?

We greatly value open and constructive scientific discussion and would welcome the opportunity to address any remaining questions or comments before the discussion period ends.

We fully understand that you may be very busy. Thank you again for your valuable time and effort in reviewing our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Your Feedback on Our Responses

Dear Reviewer dUhf,

We have carefully addressed every concern raised in your Critical Weaknesses and Questions:

  • Historical-knowledge leakage: Following your suggestion, we retained the dates and numerical history while replacing only the event texts with texts from other periods. We also expanded the de-identification experiment.
  • Statistical reliability: We reported five-seed mean ± standard deviation, paired significance tests, and the LLM decoding settings, and separately analyzed pandemic and non-pandemic periods.
  • Framework comparison: We conducted controlled comparisons with MM-TSflib after removing its additional numerical prior and also compared the frameworks when the same type of prior is available.
  • Inference and Memory costs: We reported online token and monetary costs together with offline Memory-construction costs.
  • Context guidance and multivariate forecasting: We clarified that context_guidance contains no sample-specific information. The main experiments forecast only the OT variable because they follow the established Time-MMD protocol adopted by prior methods. We further conducted an additional experiment demonstrating AFTER's applicability to multivariate forecasting.
  • Related work: We added From News to Forecast, RAF, TimeRAG, and judgmental forecast-adjustment studies.

The corresponding revisions have been added to Appendices D, L, M.1, and N, together with the expanded Related Work. As less than one day remains before the discussion period ends, we would greatly appreciate it if you could review our responses and let us know whether they have addressed your concerns. We remain available to promptly answer any further questions.

Best regards,
The Authors

Reviewer_MW6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AFTER, a framework that adjusts forecasts from a trained numerical backbone using textual event information without additional parameter optimization. History-Aware Event Reasoning identifies potentially relevant event states. Three complementary routes then estimate adjustments through contextual reasoning, analogy with retrieved historical cases, and direct transfer of historical continuations. Their outputs are integrated using an element-wise median. Experimental results suggest that AFTER can improve a numerical backbone in many settings while preserving an interpretable record of how textual evidence contributes to a forecast adjustment.

Critical strengths
  • The paper is well-motivated. The paper identifies the difficulty of translating textual event semantics into precise numerical trajectories. Anchoring textual reasoning to an existing numerical forecast is a reasonable alternative.

  • The proposed framework is interpretable. HAER explicitly distinguishes relevant external influences from redundant, stale, or already-observed information. COR uses complementary sources of evidence, while ROI avoids introducing an additional learned fusion module. The intermediate event states, retrieved cases, and route-specific offsets provide a useful audit trail.

  • The evaluation is comprehensive. The experiments cover nine datasets, multiple prediction horizons, and three principal numerical backbones. The paper also explores alternative LLM backends, retrieval configurations, and additional numerical architectures.

Critical weaknesses
  • The paper does not fully establish that the gains result from external event reasoning rather than generic corrections or retrieval effects. Table 4 compares original event texts with a no-text condition and static descriptions derived from historical numerical observations. The results support the usefulness of the original textual context. However, they do not fully isolate the contribution of event semantics from other properties of the text, including entity information, temporal cues, domain-specific vocabulary, and correlations between historical descriptions and future values.

  • The Event-Response Memory retrieves cases using the geometric mean of textual semantic similarity and normalized historical-shape similarity. This is an intuitive retrieval strategy, but similarity in event descriptions and historical trajectories does not necessarily imply similarity in the underlying response mechanism. The paper would provide a more detailed analysis of retrieval quality, including the relationship between retrieval scores and downstream forecast improvement, the frequency of negative transfer, and the performance of the numerical-transfer route on its own.

  • The empirical gains are not uniformly significant. The reported results are encouraging overall, but improvements vary substantially by dataset and metric. For example, Table 1 shows that the adjustments on Environment remain close to the backbone predictions, while some individual comparisons in Table 2 favor alternative multimodal methods on particular metrics.

  • The paper lacks a detailed discussion of inference cost and operational trade-offs. As noted in Appendix F, most of the additional latency comes from the three LLM reasoning stages. Thus, the method is constrained by API latency, token consumption, concurrent throughput, and monetary cost.

  • The information-leakage analysis is useful but does not completely resolve concerns about historical knowledge and temporal confounding. The de-identification experiment covers only two datasets, and removing timestamps or named entities cannot eliminate all potential leakage through recognizable event descriptions or widely known historical mechanisms.

  • The terms “training-free” and “without additional parameter optimization” should be used precisely, given that the numerical backbone itself is trained and the framework still requires substantial inference-time computation.

Questions for authors

Please refer to weakness

3 comments
AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer MW6B,

We sincerely thank you for your time and for the thoughtful and constructive comments on our paper. We have carefully addressed each of your questions and concerns, conducted additional controlled experiments, and revised the manuscript accordingly.

As the discussion period approaches its conclusion, we hope that our responses, new experimental evidence, and manuscript revisions have adequately addressed your concerns. If any questions remain, please do not hesitate to let us know. We would be pleased to provide further clarification promptly and in detail.

Best regards,
The Authors

AuthorsOpenReview ↗

A Kind Follow-up on Our Rebuttal

Dear Reviewer MW6B,

We sincerely appreciate your thoughtful and constructive comments, which have helped us improve our paper.

It has been five days since we submitted our detailed responses, additional experiments, and revised manuscript. As the discussion period is approaching its conclusion, could you kindly review our responses and let us know whether your concerns have been addressed?

We greatly value open and constructive scientific discussion and would welcome the opportunity to address any remaining questions or comments before the discussion period ends.

We fully understand that you may be very busy. Thank you again for your valuable time and effort in reviewing our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Your Feedback on Our Responses

Dear Reviewer MW6B,

We have carefully addressed every concern raised in your Critical Weaknesses:

  • Event semantics and information leakage: We added expanded de-identification and event-mismatch controls, showing that the gains depend on the supplied event mechanisms rather than generic text, identifiers, or retrieval alone.
  • Retrieval quality: We analyzed retrieval scores, numerical-transfer performance, and negative-transfer frequency, and explained how analogical reasoning and ROI limit unreliable transfer.
  • Dataset-dependent gains: We analyzed the effect of text quality, particularly the highly repetitive Environment context, and clarified why AFTER remains conservative when reliable event evidence is unavailable.
  • Operational cost: We reported latency, token consumption, API cost, and Memory-construction cost.
  • Terminology: We revised the relevant terminology throughout the manuscript to ensure more precise and unambiguous descriptions.

The corresponding revisions have been added to Appendices H, L, M.1, and N. As less than one day remains before the discussion period ends, we would greatly appreciate it if you could review our responses and let us know whether they have addressed your concerns. We remain available to promptly answer any further questions.

Best regards,
The Authors

Reviewer_GcZXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AFTER, which uses textual event information to revise forecasts from a numerical model. It first produces a base forecast, then estimates a correction through event reasoning, historical analogies, and numerical transfer. The method combines the correction with the base forecast and is evaluated on nine datasets.

Critical strengths
  1. External events may affect future trends, while models using only historical values cannot directly access this information.

  2. AFTER achieves competitive results across multiple datasets and numerical forecasting models.

Critical weaknesses
  1. Unclear motivation for generating corrections, the paper does not explain why generating corrections should be easier than directly forecasting future values. In Table 3, the full method differs little from direct correction generation, which underperforms the base model on some datasets. The direct forecasting baseline does not receive the base forecast, so this comparison does not isolate the benefit of correction generation.

  2. The paper argues that prior methods underuse textual information but does not define what effective use entails. Table 4 shows that event text helps AFTER. It does not establish that AFTER correctly captures event effects or uses this information more effectively than prior methods.

  3. Heuristic method design. The paper does not adequately motivate the three paths or explain when each is useful. Although it provides ablations, they do not clearly establish the conditions under which each component contributes.

Questions for authors
  1. Why should generating corrections be easier than direct forecasting? Can the authors compare the two when both receive the base forecast?

  2. How do the authors define effective use of textual semantics? What evidence shows that AFTER captures event effects better than prior methods?

  3. What motivates the three paths, and under what conditions does each help?

3 comments
AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer GcZX,

We sincerely thank you for your time and for the thoughtful and constructive comments on our paper. We have carefully addressed each of your questions and concerns, conducted additional controlled experiments, and revised the manuscript accordingly.

As the discussion period approaches its conclusion, we hope that our responses, new experimental evidence, and manuscript revisions have adequately addressed your concerns. If any questions remain, please do not hesitate to let us know. We would be pleased to provide further clarification promptly and in detail.

Best regards,
The Authors

AuthorsOpenReview ↗

A Kind Follow-up on Our Rebuttal

Dear Reviewer GcZX,

We sincerely appreciate your thoughtful and constructive comments, which have helped us improve our paper.

It has been five days since we submitted our detailed responses, additional experiments, and revised manuscript. As the discussion period is approaching its conclusion, could you kindly review our responses and let us know whether your concerns have been addressed?

We greatly value open and constructive scientific discussion and would welcome the opportunity to address any remaining questions or comments before the discussion period ends.

We fully understand that you may be very busy. Thank you again for your valuable time and effort in reviewing our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Your Feedback on Our Responses

Dear Reviewer GcZX,

We have carefully addressed every concern raised in your Critical Weaknesses and Questions:

  • Correction generation: We conducted a controlled comparison in which both output formulations receive the same base forecast. We also clarified that Direct Offset Reasoning removes only HAER and explained how HAER improves correction reasoning and Memory retrieval by converting noisy raw texts into structured, target-aligned event states.
  • Effective use of textual semantics: We provided a precise definition and added de-identification, event-mismatch, and cross-method text-sensitivity experiments.
  • Three reasoning routes: We clarified when each route is useful and added a complete route-combination ablation demonstrating their complementarity.

The corresponding analyses have been added to Appendices E, L, and O. As less than one day remains before the discussion period ends, we would greatly appreciate it if you could review our responses and let us know whether they have addressed your concerns. We remain available to promptly answer any further questions.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers & Summary of Revisions

We sincerely thank all reviewers for their thoughtful comments and constructive suggestions, which have helped us improve our paper.

This paper introduces AFTER, an event-informed framework that retains a trained numerical backbone and uses textual events to adjust its forecasts through history-aware event reasoning, historical analogies, and robust correction integration.

We appreciate that Reviewer GcZX recognized the importance of external-event information and AFTER's competitive results. Reviewer MW6B found the paper well motivated, interpretable, and comprehensively evaluated. Reviewer dUhf highlighted the division of labor between the numerical backbone and LLM correction as the most valuable aspect, and recognized AFTER as a traceable and low-risk way to connect event understanding with trained forecasters.

The reviewers also raised several constructive comments. We have carefully addressed all their concerns through additional controlled experiments, analyses, clarifications, and manuscript revisions. The main updates are summarized below.

  • Base-anchored correction and HAER (Reviewer GcZX). We compare correction generation with corrected-forecast generation under the same history, base forecast, and HAER state. Explicit corrections perform better overall while preserving an auditable backbone anchor. We also clarify that Direct Offset Reasoning removes only HAER; five-seed tests show significant improvements from the full model.
    Newly added: Appendix O.

  • Genuine use of event semantics (Reviewers GcZX, MW6B, and dUhf). We define effective textual utilization as converting event semantics and mechanisms into improved forecasts. De-identification preserves or improves performance, whereas mismatching event texts consistently increases error. MM-TSflib and TaTS remain almost unchanged under the same mismatch, while AFTER degrades consistently, showing stronger sensitivity to event semantics rather than identifiers, generic text, or memorized outcomes.
    Expanded: Appendix L.

  • Why three reasoning routes (Reviewer GcZX). We clarify their conditional roles: contextual reasoning handles current events without reliable analogs; case-based reasoning evaluates the transferability of retrieved mechanisms and phases; and numerical transfer supplies magnitude and temporal-shape evidence when high-quality cases agree. The full three-route model achieves the best average rank under all three backbones.
    Expanded: Appendix E.

  • Retrieval quality and negative transfer (Reviewer MW6B). We analyze retrieval scores, numerical-transfer performance, and negative-transfer frequency on Economy and Traffic. The results show that transferable cases provide useful numerical evidence, while analogical reasoning, retrieval-independent contextual reasoning, and ROI limit adverse effects when retrieved cases are unreliable.
    Expanded: Appendix H.

  • Statistical reliability and framework comparison (Reviewers dUhf). We now report five-seed mean ±\pm standard deviation and paired tt-tests, with deterministic LLM decoding (temperature=0, top_p=1.0). Most consistently improved settings are statistically significant. On selected datasets where MM-TSflib originally outperforms AFTER, removing its extra numerical covariate prior_history_avg degrades its performance and narrows or reverses the gap in several settings. With the same prior, AFTER outperforms MM-TSflib in the corresponding experiments.
    Updated: Appendices D and M.1.

  • Robustness and generality (Reviewers dUhf). We show that AFTER's gains persist after removing all COVID-19-associated windows and clarify that context_guidance is fixed dataset-level information with no sample-specific content. The main experiments follow the established Time-MMD protocol and forecast the OT variable; an additional multivariate experiment shows consistent improvements across all target variables, confirming that AFTER also supports multivariate forecasting. We also expand the related work to cover event-driven, retrieval-augmented, and judgmental forecast-adjustment studies.
    Updated: Related Work.

  • Inference and Memory costs (Reviewers MW6B and dUhf). We report online and offline token, monetary, and storage costs. Online reasoning costs approximately $0.0035 per sample ($3.51 per 1,000 samples), and we discuss deployment under strict latency or cost constraints.
    Newly added: Appendix N.

Together, these revisions strengthen the evidence that AFTER's gains arise from reasoning over supplied external-event mechanisms, clarify when each component is useful, and improve the paper's reliability, generality, and positioning.

We sincerely appreciate the reviewers' valuable suggestions and look forward to your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.