- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to forecast time series under different future events described in natural language. The authors propose MEETS, a diffusion-based model that encodes historical observations and event descriptions separately. They also introduce two auxiliary losses to encourage the model to use both inputs. To evaluate counterfactual predictions, the paper considers not only forecasting error but also the direction and magnitude of the predicted response. Experiments on traffic, energy usage, and simulated tumor growth show improvements over several baselines. The paper also includes ablation studies and experiments on unseen event configurations.
Critical strengths
-
Interesting and well-motivated problem. The paper addresses an important gap in text-conditioned forecasting. A model can achieve good factual predictions without responding correctly to alternative events. Evaluating the direction and magnitude of these responses is useful, especially when exact counterfactual outcomes are available in the simulator.
-
Strong experimental results and useful ablations. MEETS performs well across all three datasets and consistently improves most evaluation metrics. The ablations help explain the contributions of separate conditioning, event-level attention, and the auxiliary losses. The experiments on unseen treatment configurations also provide useful evidence of generalization.
Critical weaknesses
-
Uncertain quality of the counterfactual references. On the two real-world datasets, the counterfactual reference comes from a nearest neighbor selected using DTW. However, similar histories do not necessarily imply similar outcomes under different events. The paper does not provide enough evidence that these matched references are reliable.
-
Possible temporal confounding in event swapping. The event descriptions include day-of-week and clock-time information. Swapping events may therefore change temporal information along with the event itself. It is unclear whether the measured responses come from the event changes or these additional differences.
-
Limited evidence for the benefit of natural-language conditioning. Most event descriptions are generated from structured attributes such as event type, timing, and intensity. The paper does not compare MEETS with a model that directly uses these attributes. Such a comparison would help clarify the benefit of representing events in natural language.
-
Generalization to more diverse events remains unclear. The unseen-event experiments are limited to the tumor simulator. The paraphrasing study is useful, but it only changes the wording of existing events. It remains unclear how well the model would handle new event types or more varied real-world descriptions.
Questions for authors
-
Can the authors provide evidence that the DTW-matched references are reliable? How sensitive are the results to the matching threshold or donor selection?
-
Have the authors considered matching or controlling for calendar time when swapping events? Would the results change if day-of-week and clock-time information were controlled?
-
How would MEETS perform if the events were encoded directly using their structured attributes, while keeping the same backbone and training objectives?
-
Appendix D.3 mentions that future treatment assignments depend on tumor trajectories. Could the authors clarify whether treatment sequences are fixed across counterfactual branches or generated adaptively?
12 comments
Part-I continued
Table 1.A: Agreement between the DTW-matched reference shift and the true alternate-event shift on Tumor Volume, pooled over 8,370 test windows and 15 future timesteps. The DTW reference is unbiased and moves with the true shift . Restricting donors to observed trajectories does not remove variation: the DTW shift carries more than the true shift.
| Statistic | Value |
|---|---|
| Share of reference error from bias | 0.12% |
| Share of reference error from scale | 9.81% |
| Share of reference error from noise | 90.07% |
| Pearson correlation | 0.92 |
| Spearman correlation | 0.93 |
| Sign agreement | 90.7% |
| Slope of true shift on DTW shift | 0.87 |
| Mean square of true shift | 0.93 |
| Mean square of DTW shift | 1.03 |
Table 1.B: Response metrics on Tumor Volume for the same DTW-selected pairs and the same forecasts, scored against the true alternate future (Exact) and against the DTW-matched donor's future (DTW). Mean std over 10 inference repetitions. The DTW reference raises ResponseMSE for every model and pulls DA and DRS toward zero for every model that responds.
| Model | Reference | ResponseMSE | ResponseMAE | DA | DRS |
|---|---|---|---|---|---|
| MEETS | Exact | ||||
| MEETS | DTW | ||||
| TADiff | Exact | ||||
| TADiff | DTW | ||||
| Aurora | Exact | ||||
| Aurora | DTW | ||||
| EventTSF | Exact | ||||
| EventTSF | DTW | ||||
| CLEF | Exact | ||||
| CLEF | DTW | ||||
| IATSF | Exact | ||||
| IATSF | DTW |
Table 1.C: ResponseMSE () on Traffic Volume and Energy Usage when only the test pairs whose DTW donor is among the closest 100%, 50%, or 25% of matches are kept (Traffic Volume: 1,003, 502, and 251 pairs; Energy Usage: 44,115, 22,058, and 11,029 pairs). Mean std over 10 inference repetitions. From 100% to 25%, closer matches lower ResponseMSE for every model while the gaps between models persist, so the comparisons do not depend on the weaker matches. ResponseMAE, DA, and DRS are reported in Appendix H.3.
| Model | Traffic Volume, 100% | Traffic Volume, 50% | Traffic Volume, 25% | Energy Usage, 100% | Energy Usage, 50% | Energy Usage, 25% |
|---|---|---|---|---|---|---|
| MEETS | ||||||
| TADiff | ||||||
| Aurora | ||||||
| EventTSF | ||||||
| CLEF | ||||||
| IATSF |
Part-I continued
Table 1.D: ResponseMSE () on Traffic Volume and Energy Usage when the reference future is taken from the -th DTW-nearest donor ( is the nearest donor, used in the paper). Mean std over 10 inference repetitions. ResponseMSE is higher for every model at every farther donor than at the nearest, so a farther donor gives a poorer reference.ResponseMAE, DA, and DRS are reported in Appendix H.4.
| Dataset | Model | ||||||
|---|---|---|---|---|---|---|---|
| Traffic Volume | MEETS | ||||||
| Traffic Volume | TADiff | ||||||
| Traffic Volume | Aurora | ||||||
| Traffic Volume | EventTSF | ||||||
| Traffic Volume | CLEF | ||||||
| Traffic Volume | IATSF | ||||||
| Energy Usage | MEETS | ||||||
| Energy Usage | TADiff | ||||||
| Energy Usage | Aurora | ||||||
| Energy Usage | EventTSF | ||||||
| Energy Usage | CLEF | ||||||
| Energy Usage | IATSF |
Part-I
We thank the reviewer for their thoughtful and constructive feedback, and for recognizing that our work targets "an important gap in text-conditioned forecasting". We are glad for the chance to clarify each point below.
W1/Q1: Uncertain quality of the DTW-matched counterfactual references, and unclear sensitivity of the results to the matching threshold or donor selection.
We thank the reviewer for these questions and address each in turn below.
(a) Reliability of the DTW-matched counterfactual references:
The DTW-matched counterfactual references are reliable. We show this by running a validation experiment on Tumor Volume, the only one of our datasets where the true alternate future is known, comparing the DTW-matched reference with the true counterfactual values. For each test window, we select the DTW-nearest other test instance with a different event set, exactly as on the real-world datasets, and compare the donor's future under an event directly with the patient's own future under the same event.
We find that the DTW reference is unbiased and agrees closely with the true alternate future, and that its error is almost entirely noise rather than bias (Table 1.A).
We further validate this by scoring the same forecasts against the donor's realized future and against the patient's true future under the donor's events, and find that the DTW reference raises ResponseMSE for every model and pulls Directional Alignment (DA) and Directional Response Score (DRS) toward zero for every model that responds (Table 1.B).
Together these show that our comparisons of response error remain reliable when the true alternate future is unobserved. On Traffic Volume and Energy Usage, where no true alternate future exists, we test sensitivity to the matching threshold and donor selection in (b) and (c) below. We added both analyses to Appendix H.2.
(b) Sensitivity of the results to the matching threshold:
To test the sensitivity of the results to the matching threshold we keep only the test pairs on Traffic Volume and Energy Usage whose donor is among the closest 50% or 25% of matches, and we find that ResponseMSE is lower for every model at 25% than at 100% while the comparisons between models on response error and direction do not change (Table 1.C below; DA in Appendix H.3).
The comparisons on Traffic Volume and Energy Usage do not change when weaker matches are removed, so the evaluation keeps every nearest-donor pair rather than discarding up to 75% of them, supporting the paper's conclusions about response error and direction. This analysis has been added to Appendix H.
(c) Sensitivity of the results to donor selection:
To test the sensitivity of the results to donor selection we take the reference future on Traffic Volume and Energy Usage from the k-th DTW-nearest donor instead of the nearest (k=2,3,5,7,10), which changes both the reference future and the swapped event set. Every model degrades as the donor moves farther. ResponseMSE is higher for every model than at the nearest donor, and DA falls toward zero or below for distant donors (Table 1.D below; DA in Appendix H.4). The takeaway is that the results are sensitive to donor selection in the expected direction: a farther donor is a poorer match and gives a poorer reference, which is why the evaluation uses the nearest donor. Together with the matching-threshold analysis, this makes the nearest donor with every match kept a stable choice. We added this analysis to Appendix H.
Overall, these analyses establish the quality of our DTW-matched counterfactual references: they are unbiased against the true alternate future, stable to the matching threshold, and best with the nearest donor, which is the one we use. The counterfactual results of Section 5.3 (Table 3) therefore stand.