acceptodds
Under review as a conference paper at ICLR 2027

Forecasting Time Series Responses to Counterfactual Exogenous Event Descriptions

Abstract

Real-world time series respond to external events. Prior work shows that exogenous-event text can improve factual forecasts, while counterfactual modeling has been studied for structured interventions and target-derived or target-descriptive text. Yet whether a model produces the appropriate numerical response when the same history is paired with a different free-form exogenous event remains unclear. We propose , a conditional diffusion model with factorized history-event conditioning and matched conditioning-swap objectives that encourage use of both sources. Across traffic, energy, and treatment-response domains, lowers factual forecasting error and improves counterfactual event-specific responses, including direction and magnitude, while generalizing to unseen event configurations.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Multimodal time series forecasting, Exogenous events conditioning, Diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NGUVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to forecast time series under different future events described in natural language. The authors propose MEETS, a diffusion-based model that encodes historical observations and event descriptions separately. They also introduce two auxiliary losses to encourage the model to use both inputs. To evaluate counterfactual predictions, the paper considers not only forecasting error but also the direction and magnitude of the predicted response. Experiments on traffic, energy usage, and simulated tumor growth show improvements over several baselines. The paper also includes ablation studies and experiments on unseen event configurations.

Critical strengths
  1. Interesting and well-motivated problem. The paper addresses an important gap in text-conditioned forecasting. A model can achieve good factual predictions without responding correctly to alternative events. Evaluating the direction and magnitude of these responses is useful, especially when exact counterfactual outcomes are available in the simulator.

  2. Strong experimental results and useful ablations. MEETS performs well across all three datasets and consistently improves most evaluation metrics. The ablations help explain the contributions of separate conditioning, event-level attention, and the auxiliary losses. The experiments on unseen treatment configurations also provide useful evidence of generalization.

Critical weaknesses
  1. Uncertain quality of the counterfactual references. On the two real-world datasets, the counterfactual reference comes from a nearest neighbor selected using DTW. However, similar histories do not necessarily imply similar outcomes under different events. The paper does not provide enough evidence that these matched references are reliable.

  2. Possible temporal confounding in event swapping. The event descriptions include day-of-week and clock-time information. Swapping events may therefore change temporal information along with the event itself. It is unclear whether the measured responses come from the event changes or these additional differences.

  3. Limited evidence for the benefit of natural-language conditioning. Most event descriptions are generated from structured attributes such as event type, timing, and intensity. The paper does not compare MEETS with a model that directly uses these attributes. Such a comparison would help clarify the benefit of representing events in natural language.

  4. Generalization to more diverse events remains unclear. The unseen-event experiments are limited to the tumor simulator. The paraphrasing study is useful, but it only changes the wording of existing events. It remains unclear how well the model would handle new event types or more varied real-world descriptions.

Questions for authors
  1. Can the authors provide evidence that the DTW-matched references are reliable? How sensitive are the results to the matching threshold or donor selection?

  2. Have the authors considered matching or controlling for calendar time when swapping events? Would the results change if day-of-week and clock-time information were controlled?

  3. How would MEETS perform if the events were encoded directly using their structured attributes, while keeping the same backbone and training objectives?

  4. Appendix D.3 mentions that future treatment assignments depend on tumor trajectories. Could the authors clarify whether treatment sequences are fixed across counterfactual branches or generated adaptively?

12 comments

We thank the reviewer for their thoughtful and constructive feedback, and for recognizing that our work targets "an important gap in text-conditioned forecasting". We are glad for the chance to clarify each point below.

W1/Q1: Uncertain quality of the DTW-matched counterfactual references, and unclear sensitivity of the results to the matching threshold or donor selection.

We thank the reviewer for these questions and address each in turn below.

(a) Reliability of the DTW-matched counterfactual references:

The DTW-matched counterfactual references are reliable. We show this by running a validation experiment on Tumor Volume, the only one of our datasets where the true alternate future is known, comparing the DTW-matched reference with the true counterfactual values. For each test window, we select the DTW-nearest other test instance with a different event set, exactly as on the real-world datasets, and compare the donor's future under an event directly with the patient's own future under the same event.

We find that the DTW reference is unbiased and agrees closely with the true alternate future, and that its error is almost entirely noise rather than bias (Table 1.A).

We further validate this by scoring the same forecasts against the donor's realized future and against the patient's true future under the donor's events, and find that the DTW reference raises ResponseMSE for every model and pulls Directional Alignment (DA) and Directional Response Score (DRS) toward zero for every model that responds (Table 1.B).

Together these show that our comparisons of response error remain reliable when the true alternate future is unobserved. On Traffic Volume and Energy Usage, where no true alternate future exists, we test sensitivity to the matching threshold and donor selection in (b) and (c) below. We added both analyses to Appendix H.2.

(b) Sensitivity of the results to the matching threshold:

To test the sensitivity of the results to the matching threshold we keep only the test pairs on Traffic Volume and Energy Usage whose donor is among the closest 50% or 25% of matches, and we find that ResponseMSE is lower for every model at 25% than at 100% while the comparisons between models on response error and direction do not change (Table 1.C below; DA in Appendix H.3).

The comparisons on Traffic Volume and Energy Usage do not change when weaker matches are removed, so the evaluation keeps every nearest-donor pair rather than discarding up to 75% of them, supporting the paper's conclusions about response error and direction. This analysis has been added to Appendix H.

(c) Sensitivity of the results to donor selection:

To test the sensitivity of the results to donor selection we take the reference future on Traffic Volume and Energy Usage from the k-th DTW-nearest donor instead of the nearest (k=2,3,5,7,10), which changes both the reference future and the swapped event set. Every model degrades as the donor moves farther. ResponseMSE is higher for every model than at the nearest donor, and DA falls toward zero or below for distant donors (Table 1.D below; DA in Appendix H.4). The takeaway is that the results are sensitive to donor selection in the expected direction: a farther donor is a poorer match and gives a poorer reference, which is why the evaluation uses the nearest donor. Together with the matching-threshold analysis, this makes the nearest donor with every match kept a stable choice. We added this analysis to Appendix H.

Overall, these analyses establish the quality of our DTW-matched counterfactual references: they are unbiased against the true alternate future, stable to the matching threshold, and best with the nearest donor, which is the one we use. The counterfactual results of Section 5.3 (Table 3) therefore stand.

AuthorsOpenReview ↗

Part-I continued

Table 1.A: Agreement between the DTW-matched reference shift and the true alternate-event shift on Tumor Volume, pooled over 8,370 test windows and 15 future timesteps. The DTW reference is unbiased and moves with the true shift . Restricting donors to observed trajectories does not remove variation: the DTW shift carries more than the true shift.

StatisticValue
Share of reference error from bias0.12%
Share of reference error from scale9.81%
Share of reference error from noise90.07%
Pearson correlation0.92
Spearman correlation0.93
Sign agreement90.7%
Slope of true shift on DTW shift0.87
Mean square of true shift0.93
Mean square of DTW shift1.03

Table 1.B: Response metrics on Tumor Volume for the same DTW-selected pairs and the same forecasts, scored against the true alternate future (Exact) and against the DTW-matched donor's future (DTW). Mean ±\pm std over 10 inference repetitions. The DTW reference raises ResponseMSE for every model and pulls DA and DRS toward zero for every model that responds.

ModelReferenceResponseMSE ↓\downarrowResponseMAE ↓\downarrowDA ↑\uparrowDRS →1\rightarrow 1
MEETSExact0.20±0.000.20 \pm 0.000.06±0.000.06 \pm 0.000.96±0.000.96 \pm 0.000.98±0.010.98 \pm 0.01
MEETSDTW0.33±0.010.33 \pm 0.010.09±0.000.09 \pm 0.000.84±0.010.84 \pm 0.010.85±0.010.85 \pm 0.01
TADiffExact0.22±0.000.22 \pm 0.000.15±0.000.15 \pm 0.000.96±0.010.96 \pm 0.011.44±0.011.44 \pm 0.01
TADiffDTW0.34±0.010.34 \pm 0.010.19±0.000.19 \pm 0.000.88±0.010.88 \pm 0.011.31±0.011.31 \pm 0.01
AuroraExact0.26±0.010.26 \pm 0.010.12±0.000.12 \pm 0.000.87±0.010.87 \pm 0.011.24±0.011.24 \pm 0.01
AuroraDTW0.39±0.010.39 \pm 0.010.17±0.010.17 \pm 0.010.77±0.010.77 \pm 0.011.11±0.011.11 \pm 0.01
EventTSFExact0.93±0.000.93 \pm 0.000.14±0.000.14 \pm 0.00−0.02±0.01-0.02 \pm 0.010.05±0.010.05 \pm 0.01
EventTSFDTW1.02±0.011.02 \pm 0.010.18±0.010.18 \pm 0.01−0.02±0.01-0.02 \pm 0.010.05±0.010.05 \pm 0.01
CLEFExact0.26±0.000.26 \pm 0.000.35±0.000.35 \pm 0.000.74±0.010.74 \pm 0.010.90±0.010.90 \pm 0.01
CLEFDTW0.37±0.010.37 \pm 0.010.38±0.010.38 \pm 0.010.68±0.010.68 \pm 0.010.83±0.010.83 \pm 0.01
IATSFExact0.97±0.030.97 \pm 0.030.34±0.030.34 \pm 0.030.42±0.030.42 \pm 0.03−0.41±0.05-0.41 \pm 0.05
IATSFDTW1.03±0.041.03 \pm 0.040.38±0.030.38 \pm 0.030.39±0.040.39 \pm 0.04−0.36±0.05-0.36 \pm 0.05

Table 1.C: ResponseMSE (↓\downarrow) on Traffic Volume and Energy Usage when only the test pairs whose DTW donor is among the closest 100%, 50%, or 25% of matches are kept (Traffic Volume: 1,003, 502, and 251 pairs; Energy Usage: 44,115, 22,058, and 11,029 pairs). Mean ±\pm std over 10 inference repetitions. From 100% to 25%, closer matches lower ResponseMSE for every model while the gaps between models persist, so the comparisons do not depend on the weaker matches. ResponseMAE, DA, and DRS are reported in Appendix H.3.

ModelTraffic Volume, 100%Traffic Volume, 50%Traffic Volume, 25%Energy Usage, 100%Energy Usage, 50%Energy Usage, 25%
MEETS0.14±0.000.14 \pm 0.000.10±0.000.10 \pm 0.000.09±0.000.09 \pm 0.000.12±0.000.12 \pm 0.000.09±0.000.09 \pm 0.000.07±0.000.07 \pm 0.00
TADiff0.15±0.000.15 \pm 0.000.13±0.000.13 \pm 0.000.10±0.000.10 \pm 0.000.22±0.000.22 \pm 0.000.20±0.000.20 \pm 0.000.18±0.000.18 \pm 0.00
Aurora0.24±0.000.24 \pm 0.000.21±0.000.21 \pm 0.000.18±0.000.18 \pm 0.000.32±0.000.32 \pm 0.000.27±0.000.27 \pm 0.000.27±0.000.27 \pm 0.00
EventTSF0.87±0.010.87 \pm 0.010.85±0.010.85 \pm 0.010.82±0.020.82 \pm 0.020.52±0.000.52 \pm 0.000.50±0.000.50 \pm 0.000.47±0.000.47 \pm 0.00
CLEF0.45±0.010.45 \pm 0.010.42±0.010.42 \pm 0.010.41±0.020.41 \pm 0.020.45±0.010.45 \pm 0.010.42±0.010.42 \pm 0.010.41±0.020.41 \pm 0.02
IATSF1.24±0.031.24 \pm 0.031.21±0.041.21 \pm 0.041.21±0.051.21 \pm 0.051.49±0.031.49 \pm 0.031.46±0.031.46 \pm 0.031.44±0.041.44 \pm 0.04
AuthorsOpenReview ↗

Part-I continued

Table 1.D: ResponseMSE (↓\downarrow) on Traffic Volume and Energy Usage when the reference future is taken from the kk-th DTW-nearest donor (k=1k=1 is the nearest donor, used in the paper). Mean ±\pm std over 10 inference repetitions. ResponseMSE is higher for every model at every farther donor than at the nearest, so a farther donor gives a poorer reference.ResponseMAE, DA, and DRS are reported in Appendix H.4.

DatasetModelk=1k=1k=2k=2k=3k=3k=5k=5k=7k=7k=10k=10
Traffic VolumeMEETS0.14±0.000.14 \pm 0.000.21±0.000.21 \pm 0.000.81±0.000.81 \pm 0.001.34±0.001.34 \pm 0.003.02±0.003.02 \pm 0.005.51±0.005.51 \pm 0.00
Traffic VolumeTADiff0.15±0.000.15 \pm 0.000.96±0.020.96 \pm 0.021.36±0.021.36 \pm 0.025.08±0.015.08 \pm 0.017.38±0.037.38 \pm 0.0378.10±0.0378.10 \pm 0.03
Traffic VolumeAurora0.24±0.000.24 \pm 0.000.76±0.000.76 \pm 0.001.67±0.001.67 \pm 0.006.13±0.016.13 \pm 0.0111.46±0.0011.46 \pm 0.0066.95±0.0066.95 \pm 0.00
Traffic VolumeEventTSF0.87±0.010.87 \pm 0.012.93±0.032.93 \pm 0.034.24±0.054.24 \pm 0.054.56±0.034.56 \pm 0.0371.59±0.0571.59 \pm 0.05248.98±0.04248.98 \pm 0.04
Traffic VolumeCLEF0.45±0.010.45 \pm 0.011.72±0.031.72 \pm 0.033.18±0.063.18 \pm 0.067.05±0.117.05 \pm 0.119.75±0.189.75 \pm 0.1816.80±0.3016.80 \pm 0.30
Traffic VolumeIATSF1.24±0.031.24 \pm 0.038.42±0.068.42 \pm 0.0612.96±0.1212.96 \pm 0.1244.68±0.2244.68 \pm 0.22288.91±0.46288.91 \pm 0.46544.01±1.24544.01 \pm 1.24
Energy UsageMEETS0.12±0.000.12 \pm 0.000.18±0.000.18 \pm 0.000.50±0.000.50 \pm 0.001.28±0.001.28 \pm 0.005.02±0.005.02 \pm 0.0012.71±0.0112.71 \pm 0.01
Energy UsageTADiff0.22±0.000.22 \pm 0.000.87±0.020.87 \pm 0.022.43±0.042.43 \pm 0.048.25±0.068.25 \pm 0.0621.94±0.0421.94 \pm 0.04118.91±0.05118.91 \pm 0.05
Energy UsageAurora0.32±0.000.32 \pm 0.000.62±0.000.62 \pm 0.002.42±0.002.42 \pm 0.009.64±0.009.64 \pm 0.0019.74±0.0019.74 \pm 0.0049.23±0.0049.23 \pm 0.00
Energy UsageEventTSF0.52±0.000.52 \pm 0.002.36±0.022.36 \pm 0.0212.00±0.0412.00 \pm 0.0491.36±0.0891.36 \pm 0.08388.20±0.08388.20 \pm 0.08369.84±0.03369.84 \pm 0.03
Energy UsageCLEF0.45±0.010.45 \pm 0.010.83±0.040.83 \pm 0.041.72±0.071.72 \pm 0.072.25±0.102.25 \pm 0.105.10±0.215.10 \pm 0.219.05±0.349.05 \pm 0.34
Energy UsageIATSF1.49±0.031.49 \pm 0.032.28±0.012.28 \pm 0.0124.56±0.2824.56 \pm 0.2836.88±1.5736.88 \pm 1.5784.13±1.3384.13 \pm 1.33610.13±2.48610.13 \pm 2.48
Reviewer_sspvOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors introduce MEETS, a conditional diffusion model that incorporates future ‘exogenous’ event descriptions into time series forecasts. MEETS separately encodes (1) a historical time series X1:TX_{1:T} into MHM_H, and (2) free-form textual event descriptions (for future events) E from the interval T+1:T+LT+1:T+L into MEM_E, as inputs to a denoising diffusion model (self attention, cross attention to MHM_H, cross attention to MEM_E), in order to ultimately predict future trajectories p(XT+1:TL∣X1:T,E)p(X_{T+1:T_L} | X_{1:T}, E). The approach is evaluated first on factual forecasting: predicting trajectories for events that actually occurred. Then, it is evaluated on counterfactual forecasting: predicting the alternate future under events that did not occur. Three datasets are collated/curated for this time-series evaluation — predictions of traffic volume given weather events, energy usage given extreme-weather events, and tumor volume given combinations of chemotherapy and radiotherapy. MEETS shows stronger performance across all three datasets for both tasks, compared to five recent forecasting methods.

Critical strengths
  • The writing is clear and fairly well situated in related literature.
  • The experiments validate the method across several real-data examples and against relevant baselines/other comparable methods.
Critical weaknesses
  • One weakness in my view is the lack of clarity/preciseness around the framing of ‘exogeneity’ of the events in question, especially when comparing MEETS to other text-conditioned forecasting methods. For example, one description is that “Because the event is an input rather than inferred from the history, we can hold the history fixed, vary the event, and ask whether the forecast changes appropriately.” Can this be restated more precisely as some form of causal assumptions (perhaps an assumed causal structure?) involving the events E, history H, past outcomes Y1:TY_{1:T}, and future outcomes YT+1:T+LY_{T+1:T+L}? This would help to clarify how narrow the assumptions of MEETS are and when they may or may not generalize/apply in other settings.
  • Related to the above, but distinct, what are the structural assumptions being implicitly made by the NN-PEHE-inspired evaluation? When would these assumptions be applicable (or not)? It seems that using EswapE_{swap} as a stand-in alternate future (based on DTW-proximity of the X1:TX_{1:T} values) would introduce some form of bias into the evaluation (for example, choosing pairs that are not adequately similar, or, more importantly, inducing a lack of variability in any latent factors that could be consequential for counterfactual trajectories, given that we restrict ourselves only to existing trajectories in the observed data). As one empirical example of how much bias this method may or may not introduce, can the Tumor Volume data be used to compare the DTW-nearest other test instance substitution to the true counterfactual values (e.g., in the style of an ablation)?
  • A more minor point: it is stated that no position embedding is added for the event encoder because E is a set and “concurrent events have no natural order, and each event’s text already states when it is active.” Perhaps an ablation and/or comparing to an alternative here would show whether or not this choice is helpful?
Questions for authors

See weaknesses above.

  • Can the framing of exogeneity be restated more precisely as some form of causal assumptions (perhaps an assumed causal structure?) involving the events E, history H, past outcomes Y1:TY_{1:T}, and future outcomes YT+1:T+LY_{T+1:T+L}?
  • What are the structural assumptions being implicitly made by the NN-PEHE-inspired evaluation? When would these assumptions be applicable (or not)? It seems that using EswapE_{swap} as a stand-in alternate future (based on DTW-proximity of the X1:TX_{1:T} values) would introduce some form of bias into the evaluation. Can the Tumor Volume data be used to compare the DTW-nearest other test instance substitution to the true counterfactual values (e.g., in the style of an ablation)?
  • Is there empirical evidence in support of (or against) having position embeddings in the event encoder?
7 comments

We thank the reviewer for their thoughtful and constructive feedback. We are encouraged that the reviewer finds our writing clear and recognizes that our experiments validate MEETS's strong performance "across several real-data examples and against relevant baselines/other comparable methods". We address the reviewer's comments below.

W1/Q1: Making the exogeneity framing more precise, to distinguish MEETS from other text-conditioned methods and to clarify its scope and applicability.

We revised the introduction (Section 1) to state what we mean by exogenous events and added formal definition to our problem definition (Section 3). Let TT be the present time, LL the forecast horizon, and XT+1:T+L\mathbf{X}_{T+1:T+L} the target's future trajectory. An event ee, a natural-language description of an external occurrence that is active over an interval I⊆T+1,…,T+LI\subseteq\\{T+1,\dots,T+L\\}, is exogenous if and only if it influences XT+1:T+L\mathbf{X}_{T+1:T+L} but is neither influenced by nor derived from it: XT+1:T+L\mathbf{X}_{T+1:T+L} is not an input to the procedure that constructs ee. Note that we retain our predictive formulation rather than a causal structure. MEETS models the distribution of future trajectories conditional on an observed history and supplied events, and we do not claim that these conditional forecasts identify causal effects.

Distinguishing MEETS's scope and applicability from other text-conditioned methods: Under our problem definition (Section 3), an event description cannot be constructed from the target's future. This is what separates MEETS from prior methods. Prior text-conditioned counterfactual methods may call their conditioning text "external", but the text used in their datasets is either computed from the target's future or directly describes it, so it leaks the outcome into the model's input (Appendix B.1). We instead scope both our data and our model design strictly to this definition.

Our data satisfies this property by construction (Section 5.1; Appendix D), and our model is designed to learn the event's effect from such data (Section 4). MEETS is therefore built to be applicable in settings where the text does not give away the outcome and the model must learn how the history and the event together generate the future. To the best of our knowledge, we are the first to jointly formalize exogeneity for multimodal time-series–text forecasting, construct data that satisfies it across three datasets, and train a forecaster built on this assumption.

W2/Q2: Clarifying the structural assumptions of the NN-PEHE-inspired evaluation and whether the DTW-matched reference introduces bias.

We thank the reviewer for these questions and address each in turn below.

(a) Structural assumptions of the NN-PEHE-inspired evaluation:

Our NN-PEHE-inspired evaluation using DTW as the matching distance makes three structural assumptions, all of which are standard in prior work on nearest-neighbor matching and NN-PEHE [1, 2, 3, 4]: (1) no hidden confounding (no unobserved factor drives both the event a series receives and its response to that event) [2]; (2) smoothness (histories that are close under DTW have close expected futures under the same event) [3, 4]; and (3) overlap (similar histories occur with different events, so a close donor with a different event exists) [2]. We have added discussion of these assumptions to Appendix H.

(b) When would these assumptions be applicable (or not)?

These assumptions hold when the event a series receives is explained by its observed history, similar histories evolve similarly under the same event, and close matches are available. They fail when unobserved factors drive the event and the response, and when similar histories diverge under the same event, or no close donor exists.

Additional experimental support for assumptions: Based on the reviewer's comment, we also tested whether these assumptions hold on all three of our datasets.

Tumor Volume serves as a clean proxy for the real-world data. Since the true alternate future is known there, we apply the same DTW matching and compare the DTW-matched reference (the realized future of the DTW-nearest donor) with the truth. The two match without bias, so close histories evolve similarly under the same event (Table 2.A below and Appendix H.2).

On Traffic Volume and Energy Usage, where the true alternate future is not known, we repeat the evaluation with only the closest 50% and 25% of matches. Response error falls for every model as the matches get closer, so close histories again evolve similarly under the same event (Table 2.C below and Appendix H.3). This supports the assumptions of our NN-PEHE-inspired evaluation holding on our data, which reinforces the validity of our evaluation.

(c) Whether the DTW-matched reference introduces bias:

No, this introduces no bias. We test all sources of bias the reviewer mentions and find that the DTW-matched reference only adds noise to every model rather than a systematic bias.

First, we test for a lack of variability in latent factors, which could arise because references come only from observed trajectories. On Tumor Volume, the simulator gives the true future of each history under a swapped event, so for every test window we have two versions of the change caused by the swap: the true one, and the one given by the DTW-matched reference. Across test windows, the DTW-matched changes are spread at least as widely as the true ones, and their error against the truth is noise rather than bias. Restricting references to observed trajectories therefore does not reduce variability, and a lack of variability in latent factors introduces no bias (Table 2.A below and Appendix H.2).

Second, we test for pairs that are not adequately similar. On Traffic Volume and Energy Usage, we repeat the evaluation with only the closest 50% and 25% of matches. Response error falls for every model, and the differences between models remain. The less similar pairs therefore add noise but do not bias our comparisons (Table 2.C below and Appendix H.3).

(d) How the DTW-matched reference compares with the true counterfactual values on Tumor Volume:

The DTW-matched reference tracks the true counterfactual values closely and its error is noise rather than bias. We tested this as the reviewer suggested and added it to Appendix H.2. For each Tumor Volume test window, we select the DTW-nearest other test instance with a different event set, exactly as on the real-world datasets, and compare its shift with the true shift. None of the reference's errors are bias, and its shift varies at least as much as the true one (Table 2.A below). Scoring the same forecasts against both references raises response error for every model (Table 2.B below), so comparisons of response error remain valid when the true alternate future is unobserved.

Together, these results support the DTW-matched reference on Traffic Volume and Energy Usage with every nearest-donor pair kept. They strengthen the paper's results without changing our conclusions.

AuthorsOpenReview ↗

Part-II continued

Table 2.A: Agreement between the DTW-matched reference shift and the true alternate-event shift on Tumor Volume, pooled over 8,370 test windows and 15 future timesteps. The DTW reference is unbiased and moves with the true shift . Restricting donors to observed trajectories does not remove variation: the DTW shift carries more than the true shift.

StatisticValue
Share of reference error from bias0.12%
Share of reference error from scale9.81%
Share of reference error from noise90.07%
Pearson correlation0.92
Spearman correlation0.93
Sign agreement90.7%
Slope of true shift on DTW shift0.87
Mean square of true shift0.93
Mean square of DTW shift1.03

Table 2.B: Response metrics on Tumor Volume for the same DTW-selected pairs and the same forecasts, scored against the true alternate future (Exact) and against the DTW-matched donor's future (DTW). Mean ±\pm std over 10 inference repetitions. The DTW reference raises ResponseMSE for every model and pulls DA and DRS toward zero for every model that responds.

ModelReferenceResponseMSE ↓\downarrowResponseMAE ↓\downarrowDA ↑\uparrowDRS →1\rightarrow 1
MEETSExact0.20±0.000.20 \pm 0.000.06±0.000.06 \pm 0.000.96±0.000.96 \pm 0.000.98±0.010.98 \pm 0.01
MEETSDTW0.33±0.010.33 \pm 0.010.09±0.000.09 \pm 0.000.84±0.010.84 \pm 0.010.85±0.010.85 \pm 0.01
TADiffExact0.22±0.000.22 \pm 0.000.15±0.000.15 \pm 0.000.96±0.010.96 \pm 0.011.44±0.011.44 \pm 0.01
TADiffDTW0.34±0.010.34 \pm 0.010.19±0.000.19 \pm 0.000.88±0.010.88 \pm 0.011.31±0.011.31 \pm 0.01
AuroraExact0.26±0.010.26 \pm 0.010.12±0.000.12 \pm 0.000.87±0.010.87 \pm 0.011.24±0.011.24 \pm 0.01
AuroraDTW0.39±0.010.39 \pm 0.010.17±0.010.17 \pm 0.010.77±0.010.77 \pm 0.011.11±0.011.11 \pm 0.01
EventTSFExact0.93±0.000.93 \pm 0.000.14±0.000.14 \pm 0.00−0.02±0.01-0.02 \pm 0.010.05±0.010.05 \pm 0.01
EventTSFDTW1.02±0.011.02 \pm 0.010.18±0.010.18 \pm 0.01−0.02±0.01-0.02 \pm 0.010.05±0.010.05 \pm 0.01
CLEFExact0.26±0.000.26 \pm 0.000.35±0.000.35 \pm 0.000.74±0.010.74 \pm 0.010.90±0.010.90 \pm 0.01
CLEFDTW0.37±0.010.37 \pm 0.010.38±0.010.38 \pm 0.010.68±0.010.68 \pm 0.010.83±0.010.83 \pm 0.01
IATSFExact0.97±0.030.97 \pm 0.030.34±0.030.34 \pm 0.030.42±0.030.42 \pm 0.03−0.41±0.05-0.41 \pm 0.05
IATSFDTW1.03±0.041.03 \pm 0.040.38±0.030.38 \pm 0.030.39±0.040.39 \pm 0.04−0.36±0.05-0.36 \pm 0.05

Table 2.C: ResponseMSE (↓\downarrow) on Traffic Volume and Energy Usage when only the test pairs whose DTW donor is among the closest 100%, 50%, or 25% of matches are kept (Traffic Volume: 1,003, 502, and 251 pairs; Energy Usage: 44,115, 22,058, and 11,029 pairs). Mean ±\pm std over 10 inference repetitions. From 100% to 25%, closer matches lower ResponseMSE for every model while the gaps between models persist, so the comparisons do not depend on the weaker matches. ResponseMAE, DA, and DRS are reported in Appendix H.3.

ModelTraffic Volume, 100%Traffic Volume, 50%Traffic Volume, 25%Energy Usage, 100%Energy Usage, 50%Energy Usage, 25%
MEETS0.14±0.000.14 \pm 0.000.10±0.000.10 \pm 0.000.09±0.000.09 \pm 0.000.12±0.000.12 \pm 0.000.09±0.000.09 \pm 0.000.07±0.000.07 \pm 0.00
TADiff0.15±0.000.15 \pm 0.000.13±0.000.13 \pm 0.000.10±0.000.10 \pm 0.000.22±0.000.22 \pm 0.000.20±0.000.20 \pm 0.000.18±0.000.18 \pm 0.00
Aurora0.24±0.000.24 \pm 0.000.21±0.000.21 \pm 0.000.18±0.000.18 \pm 0.000.32±0.000.32 \pm 0.000.27±0.000.27 \pm 0.000.27±0.000.27 \pm 0.00
EventTSF0.87±0.010.87 \pm 0.010.85±0.010.85 \pm 0.010.82±0.020.82 \pm 0.020.52±0.000.52 \pm 0.000.50±0.000.50 \pm 0.000.47±0.000.47 \pm 0.00
CLEF0.45±0.010.45 \pm 0.010.42±0.010.42 \pm 0.010.41±0.020.41 \pm 0.020.45±0.010.45 \pm 0.010.42±0.010.42 \pm 0.010.41±0.020.41 \pm 0.02
IATSF1.24±0.031.24 \pm 0.031.21±0.041.21 \pm 0.041.21±0.051.21 \pm 0.051.49±0.031.49 \pm 0.031.46±0.031.46 \pm 0.031.44±0.041.44 \pm 0.04
Reviewer_JhQpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors present a framework called MEETS, which is designed to probabilistically forecast how a system evolves from its observed time series history given a natural-language descriptions of events that are exogenous to the given/observed time series. The papers shows results for both, traditional forecasting where only factual data is used (i.e., no exogenous events) and results for Tumor Volume forecasting where unseen events are incorporated.

Critical strengths

S1: The studied application where data needs to be forecast for unseen events seems interesting.

S2: The MEETS method seems to perform well for forecasting where only factual data is used. The closest competitor is TADiff, which also uses a diffusion model and hence is probably most similar in its architecture.

Critical weaknesses

W1: The paper is very dense to read, especially in the results section. The authors mention that certain metrics (MSE and MAE) do not show the full picture of how the methods perform. Maybe the details are understandable by reading the complete appendix, but I think the fundamental issues should be understandable from the main paper. Some aspects of the results are presented with very short sentences and very tight explanations.

W2: After reading the abstract and introduction I was under the impression that the main focus and innovation of the paper was on exogenous/unseen events. However, the main paper does not contain the results for this generalization, they only appear in the appendix. Maybe I misread the intention of the paper, but the structure, as it is, seems odd to me. I would expect those results to be in the main paper. It is possible that there is just not enough room for those results, but then maybe this paper might be better published in a journal, where more space would be available.

W3: In Table 3, Tumor Volume, DA, Aurora is underlined as the second best result with a value of 0.87. However, MEETS is calculated at 0.96. Shouldn't this be the second best value?

Questions for authors

Q1: The only general question I have is whether some of the result tables of Section 5.3.1 could be presented in the main paper.

1 comment

We thank the reviewer for their thoughtful and constructive feedback, and appreciate their interest in our exogenous-event forecasting setting and its generalization to unseen events, and their recognition of MEETS's strong performance. We address the reviewer’s concerns below.

W1: Readability of the results section and reliance on appendix to explain metric choices.

We have used the additional page allowed in the revision to bring details from the appendix into the results section (Section 5). Section 5.1 now describes our datasets in detail, and Sections 5.2 and 5.3 each open by describing the metrics used and what each one measures. The reason why MSE and MAE alone are not enough is that they score only a model's mean forecast, so cannot show how well its sampled trajectories are spread or whether the model is sensitive to a change in the event.

We have also divided the findings in Sections 5.2 and 5.3 into separate paragraphs and explained the reasoning behind each of our conclusions. These changes improve only the readability of our paper and do not affect any of our results or conclusions.

W2/Q1: Presenting unseen-event generalization results in the main paper and clarifying how it fits into the paper's main focus.

Presenting unseen-event generalization results in the main paper: We agree that the numerical results of our unseen-event evaluation setting should appear in the main paper next to where they are discussed (Section 5.4, formerly Section 5.3.1) instead of just in the appendix. We have added Figure 3 in Section 5.4 which shows the mean squared error (MSE) and continuous ranked probability score (CRPS) of our method (MEETS) and every baseline on all eleven unseen-event splits of Tumor Volume.

Clarifying how unseen-event generalization fits the focus of our work: The main focus of our paper is time series forecasting under exogenous events. In our setting, an event is considered exogenous when it shapes the future of the series but is neither influenced by nor derived from its future values, so exogeneity is a property of how an event relates to the series, not of whether the model saw it in training.

We have modified the first paragraph of the introduction and the problem definition (Section 3) to state this definition of exogeneity explicitly. Every experiment in our paper conditions on exogenous events: the factual setting (Section 5.2) forecasts under the events that were actually observed, and the counterfactual setting (Section 5.3) under a changed event. Unseen events are simply out-of-distribution (OOD) exogenous event configurations that are withheld during training to test our method's ability to forecast in OOD settings, and they remain the third of our three evaluation settings (Section 5.4).

W3: Table 3 underlining.

We thank the reviewer for catching this mistake. We have corrected the underlining in Table 3, and verified it is correct for all of the other results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Discussion Summary

We thank all three reviewers for their time and thoughtful feedback. Our revised manuscript, now uploaded, incorporates the reviewers’ suggestions and additional experiments that extend our results and reinforce our original conclusions.

We are encouraged by the reviewers’ recognition of our work, which we summarize as follows:

  • Problem setting and evaluation: Reviewers [sspv, NGUV] recognize that our problem is well situated, "interesting and well-motivated", and "addresses an important gap in text-conditioned forecasting." They also find our evaluation, of how a model's forecast shifts in direction and magnitude when the event changes, to be useful.

  • Experiments and ablations: Reviewers [JhQp, sspv, NGUV] find that our experiments "validate" our "method across several real-data examples and against relevant baselines/other comparable methods," and describe our results as strong and our ablations as informative. They also note that our method, MEETS, "shows stronger performance across all three datasets for both tasks, compared to five recent forecasting methods," and that our experiments on unseen event configurations "also provide useful evidence of generalization" as well as being interesting.

Highlights of how we used the reviewers' suggestions to strengthen our work:

1. Formally define exogenous events. We updated our problem definition (Section 3) to formalize our definition of exogenous events. We also updated the first paragraph of our introduction with a constructive example and an accompanying illustration in Figure 1 so that readers immediately know what we mean by exogeneity.

2. Confirm the reliability of DTW-matched counterfactual references. DTW-matched references are what make our counterfactual forecasting evaluation possible on real-world data, where the alternate future is never observed. We performed additional experiments to validate the reliability of these references on both simulated and real-world data (Appendix H).

3. Confirm that model responses to counterfactual events are free of temporal confounding. To isolate the change in a model's forecast under the observed and swapped events due to varying event type rather than timing, we hold event timing fixed. We find that MEETS still responds correctly to the changed event, reinforcing our original counterfactual results (Section 5.3, Appendix I). This kind of controlled experiment is possible only in a simulator, as we note in Appendix A.

4. Empirically show MEETS’s generalization to new event types on real-world data and the benefit of natural language. We added experiments to show that MEETS generalizes to new event types unseen during training (Section 5.4, Appendix J.6), and that its natural-language event inputs allow it to relate these new types to semantically similar ones seen during training (Section 5.5, Appendix K.1.3).

5. Improve presentation of evaluation. We use the additional page for the revision to explain the role of each metric in our experimental results (Section 5) and the findings and reasoning behind each conclusion.

We sincerely appreciate the reviewers' constructive feedback, which has helped further strengthen our paper. Reviewer NGUV noted that the new experiments address most of their concerns and raised their rating from 2 to 3. Reviewer sspv noted that the new experiments address most of their questions and kept their positive rating. We understand Reviewer JhQp may not have had time to consider our responses, but believe our detailed response addresses their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.