- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes REIN, a news-augmented time-series forecasting framework that treats textual information as a bounded modulator of a residual correction on top of a frozen numerical forecaster. Raw news is first processed by an LLM into a structured regime bank containing actionability, topic, regime, confidence, horizon, and embedding information. A PatchTST-based delta module then decomposes the residual into slow-level, intra-horizon shape, and spike components, while news affects the prediction only through bounded modulation variables rather than directly generating the forecast.
The paper further introduces an Active/Blank/Permuted counterfactual evaluation protocol intended to distinguish gains from aligned news content from gains due to generic news availability or temporal correlations. The method is evaluated on electricity price, electricity demand, traffic, and gas-demand datasets against a broad set of numerical, foundation-model, and multimodal baselines, and is additionally studied through residual-channel validation and ablations.
Critical strengths
- The method is well aligned with a meaningful motivation.
The paper goes beyond the usual question of whether text improves forecasting accuracy and asks more operationally relevant questions: how much text should be allowed to change a prediction, which temporal component it affects, and what happens when useful news is absent. The frozen numerical base, structured slow/shape/spike residual, and bounded modulation mechanism are directly motivated by these concerns. This architecture–motivation alignment is one of the strongest aspects of the work.
- The paper attempts to validate mechanism rather than reporting only aggregate forecasting gains.
In addition to the main benchmark table, the authors provide Active/Blank/Permuted counterfactual evaluation and controlled validation of the slow/shape/spike decomposition. The synthetic component study, in particular, gives useful evidence that the three residual branches correspond to the intended temporal structures rather than simply acting as arbitrary parallel heads.
Critical weaknesses
- The central claim that the gains come from aligned news content is not sufficiently isolated from the benefit of the additional time-series residual model.
In the proposed model, the regime relevance mass enters the soft activity gate multiplying the final residual correction. Therefore, under the Blank condition, the delta correction is largely suppressed rather than merely deprived of news semantics. At the same time, the residual itself is produced from time-series features using a PatchTST encoder and calendar covariates. Consequently, the substantial Active-vs-Blank gap does not by itself establish that news semantics are responsible for the improvement; it may partly measure the value of enabling an additional nonlinear residual forecaster on samples declared active.
The most important missing control is therefore a same-capacity TS-only residual baseline: frozen DLinear + the same PatchTST residual module + the same calendar inputs, but without any news/regime-bank information. Without this comparison, I find the main attribution claim insufficiently supported.
- The current counterfactual protocol does not convincingly rule out calendar/availability correlations, and the evidence for fine-grained temporal alignment is relatively weak.
The global date permutation changes not only semantic alignment but potentially seasonal and calendar structure associated with news. A stronger control would preserve the active mask/relevance mass and calendar regime while shuffling only semantic content, or perform within-month/weekday-matched permutations and temporal shift tests.
Moreover, several Electricity Price settings show very small or even negative Active-vs-Permuted differences, while the counterfactual objective is itself used during training and checkpoint selection. Thus, the present results more strongly support the claim that the regime/news activation is useful than the stronger claim that correctly time-aligned news semantics are the main source of the gain.
Questions for authors
- Material experimental question:
Can the authors report a baseline consisting of the frozen DLinear backbone plus the same PatchTST residual module and calendar features, but with no news/regime-bank input? This comparison would materially affect my assessment because it would isolate whether the improvement over DLinear is attributable to news or mainly to the stronger residual forecaster.
- Material attribution question:
Can the authors provide counterfactuals that preserve news activity/relevance mass and calendar structure while perturbing only the semantic content, e.g., within-month or weekday-matched permutations, semantic-vector shuffling with fixed timestamps, and/or ±1/3/7/14-day temporal shifts? Reporting multiple permutation seeds would also help determine whether the relatively small Active-vs-Permuted differences are robust.
- Clarification / empirical validation:
The paper motivates a clean fallback to the numerical base when useful news is absent. Could the authors explicitly report the news-inactive subset with Base, Active, Blank, and Permuted predictions, together with statistics such as the mean or percentile of |y_REIN - y_base|? This would directly validate one of the paper’s main architectural claims. The regime-bank statistics suggest that a large fraction of timestamps are classified as active, so this fallback property is important to quantify empirically.
- Clarification of the theoretical claim:
The deviation guarantee is written as
||y_hat - y_base||_infinity <= lambda_max ||r_delta||_infinity.
Is r_delta itself norm-bounded by the architecture? If not, I think the paper should distinguish between a bound on the modulation coefficient and a fixed absolute bound on the forecast perturbation.
4 comments
Response part (2/4)
Q2 and W2
We agree that full-pack permutation conflates several signals. Nevertheless, aligned packs have lower MAE than the original permutation in 16/20 settings. The new interventions examine what supports this alignment advantage, rather than treating it as proof of precise semantic use.
The submitted permutation moves both content and relevance mass without preserving calendar alignment. Content-only interventions retain the true mass, activity state and gate. Mass-only interventions retain content but change mass and recompute the gate. Calendar-matched interventions replace content within the same month and time slot, or the same weekday and time slot. Each permutation family uses five seeds.
Entries show MAE increases (%) relative to Active on news-active windows using submitted checkpoints, averaged over horizons. Permutation columns average five seeds. Positive values favor aligned news.
| Dataset | Full pack | Mass only | Content only | Content, same weekday and time slot | Content, within month | Content shifted by ±1/3/7/14 days (range) |
|---|---|---|---|---|---|---|
| Electricity Price | +0.20 | +0.09 | +0.10 | +0.20 | −0.16 | −0.45 to +0.06 |
| Electricity Demand | +0.55 | +0.18 | +0.41 | +0.28 | +0.24 | −0.27 to +0.93 |
| Traffic | +2.27 | +1.89 | +0.54 | +0.29 | −0.01 | −0.51 to +0.18 |
| Gas Demand | +0.34 | +0.35 | 0.00 | −0.01 | +0.06 | −0.17 to +0.34 |
- Content-only effects range up to +0.54% for the global permutation, −0.16% to +0.29% for the weekday-matched and within-month permutations, and −0.51% to +0.93% for the shifts.
- Full-pack permutation increases mean MAE on all four datasets. On Traffic and Gas, mass-only changes reproduce much of the effect, with increases of 1.89% and 0.35% versus 2.27% and 0.34% for full-pack permutation.
- Demand shows modest positive content effects, with increases of 0.41% under global permutation, 0.28% under weekday matching and 0.24% under month matching. These support a small local alignment effect, not a general semantic advantage. Short shifts are weaker tests because smoothing and overlapping article validity windows retain some original information.
- The standard deviation over the five permutation seeds is at most 0.23 percentage points. Under a moving-block bootstrap, the variant-level 95% intervals exclude zero in favor of aligned news in 46 of 100 cases for the full pack and 21 of 100 for weekday-matched content, and in favor of the counterfactual in none. For within-month content, 9 of 100 favor aligned news and 7 the counterfactual. For the shifts, the counts are 20 and 14 of 160. These variants overlap and are not corrected for multiple comparisons.
Training with the counterfactual objective limits the protocol’s independence. We will also correct the description of the permutation procedure. The training permutation was fixed (seed 2024), not resampled every batch, and the test used the same one. We ran two checks.
- Four other test permutations change the mean full-pack effect by at most 0.30 percentage points.
- REIN* is trained without and selected by plain validation MAE. Its full-pack effect is −0.40% on Electricity Price, +0.38% on Electricity Demand, +2.15% on Traffic and +0.41% on Gas Demand, against +0.19%, +0.31%, +2.15% and +0.42% for the submitted REIN. Perm is worse than Active in 16 of 20 settings for both.
The positive mean gap survives removal of both the objective term and selection constraint on Demand, Traffic and Gas, but reverses on Price. Thus, these mechanisms are not necessary for the surviving gaps in this run. Their individual effects remain unresolved because we removed them jointly. The submitted-checkpoint interventions identify mass and gate changes as major contributors on Traffic and Gas.
We will retain the observed full-pack alignment advantage in most settings while distinguishing it from precise semantic alignment. Mass includes article selection, confidence and duration, so the remaining signal is still news-derived. However, sensitivity to mass does not establish an independent accuracy gain from it. The additional permutation seeds assess intervention variability, not variability across training seeds.
Response part (3/4)
Q3
We agree that inactive-window behavior must be measured directly. REIN does not exactly revert to the base, but the new evaluation shows substantially smaller corrections on inactive windows and lower MAE than the base in 13/14 settings with such windows.
The following comparison covers news-inactive test windows at using submitted checkpoints. Here σ is the training standard deviation used for z-scoring. Blank zeroes the pack, and Perm uses the submitted date permutation.
| Dataset | Inactive windows | MAE: Base / REIN / Blank / Perm | : mean / p90 / p99 | Same mean on active windows |
|---|---|---|---|---|
| Electricity Price | 217 of 3464 (6.3%) | 50.35 / 45.17 / 45.66 / 41.49 | 0.015 / 0.025 / 0.040 | 0.050 |
| Electricity Demand | 115 of 3464 (3.3%) | 358.22 / 339.56 / 344.17 / 322.89 | 0.032 / 0.081 / 0.112 | 0.190 |
| Traffic | 143 of 1709 (8.4%) | 1.409 / 1.326 / 1.342 / 1.092 | 0.016 / 0.034 / 0.077 | 0.168 |
| Gas Demand | 512 of 1646 (31.1%) | 3.521 / 3.514 / 3.512 / 3.550 | 0.029 / 0.054 / 0.084 | 0.108 |
The supplementary table lists all 14 settings with inactive windows. Six settings have none.
- You are right that most windows are active: 96.7–100% on Electricity Demand, 93.7–100% on Electricity Price, 90.9–100% on Traffic and 64.8–68.9% on Gas Demand. Evidence about fallback therefore comes mainly from Gas Demand.
- Over the 14 settings, the mean deviation on inactive windows is 0.014–0.074σ, which is 9–32% of the active-window mean at the same setting. The p99 reaches 0.347σ (Gas Demand, ).
- The coefficient has a floor of and no hard shut-off was enabled. Mean on inactive windows is 0.053–0.150.
- REIN has lower MAE than the base in 13 of 14 settings, and so does Blank, which is consistent with a residual correction that stays on without news. Perm has lower MAE than the true pack in 9 of 14 settings. A permuted pack can assign these windows a larger mass and a stronger correction, although this does not isolate the cause of the improvement. This agrees with Q1 and is not evidence for aligned content.
The evidence supports attenuated correction on inactive windows, rather than exact fallback. We will retain this measured behavior while withdrawing the “clean fallback” claim. The smaller inactive subsets also limit how broadly these results can be generalized.
Q4
We agree that the architecture does not bound , so an absolute deviation guarantee is unsupported. The coefficient constraint nevertheless provides a precise relative bound on how strongly the learned residual can modify the base. We will state that narrower guarantee explicitly.
\|\hat y_z-\hat y_z^\{\mathrm\{base\}\}\|*\infty \le \lambda*\{\max\},\|r_\{\mathrm\{delta\}\}\|_\infty \.
We will withdraw the absolute deviation guarantee. On the test set, measured in z-space using per-window sup-norms, the deviation from the base has median / maximum 0.17 / 3.41 on Electricity Price, 0.76 / 2.55 on Electricity Demand, 0.82 / 1.56 on Traffic and 0.31 / 2.78 on Gas Demand, while reaches 28.67 on Electricity Price.
Response part (4/4)
Architectural evidence and implementation corrections
The synthetic study applies the target construction of Eqs. (8)–(10) to injected residuals without trained heads. We will clarify that it validates the construction rule, not trained-component fidelity. We therefore also tested the decomposition through a matched-capacity forecasting comparison without news.
The three-head model achieves lower horizon-averaged MAE and MSE than a single residual head on three of four datasets. On Traffic, the single head has 5.75–34.61% higher MAE at and 8.92–81.97% higher MSE at every horizon. On Demand and Gas, its per-horizon MAE increases average 1.05% and 0.94%. Price favors the single head, so we will present decomposition as a useful but dataset-dependent structural choice.
A separate matched-capacity direct-news variant predicts an unbounded horizon-length correction from the regime representation. REIN has lower MAE in 14/20 settings. This supports the practical viability of restricted modulation, although the direct-news variant was not separately tuned and the result does not show that scalar controls capture arbitrary event effects. Full comparisons appear in our response to Reviewer ygEn.
- The implementation computes \lambda=\lambda_\{\min\}+\bigl(\mathrm\{clip\}(\lambda_\{\mathrm\{ts\}\}+\Delta\lambda_\{\mathrm\{news\}\},\lambda_\{\min\},\lambda_\{\max\})-\lambda_\{\min\}\bigr),g_\{\mathrm\{soft\}\}. The gate scales the coefficient above its floor and does not multiply the final residual directly. We will correct the equation to match the implementation. The reported numbers are unchanged.
Revised contributions
We value the way your questions separate forecasting performance from evidence about its source. The added controls expose limitations in the original attribution, while also identifying architectural results worth retaining. We will revise the manuscript around that distinction.
We will narrow the news-attribution claim. Most gains on Demand, Traffic and Gas are recoverable without news. Full-pack alignment is beneficial in most evaluated settings, but the interventions identify substantial mass-related effects and smaller content effects. We will remove the abstract’s semantic-attribution statement and the 42.6% blanking result as supporting evidence, and revise the TL;DR accordingly. These findings do not imply that news contains no useful information.
We will retain the constrained residual interface and the empirical contribution of structured residual learning. The direct-news comparison favors REIN in 14/20 settings, and the three-head comparison favors decomposition in average MAE and MSE on three of four datasets. We will report the exceptions and tuning limitations alongside these results. The guarantee will be relative to residual magnitude, and inactive-window behavior will be described as attenuated correction rather than exact reversion.
We will strengthen the evaluation contribution through no-news retraining and separate mass, content and temporal interventions. This protocol distinguishes effects that Blank and Perm alone conflate. It does not fully isolate every source of accuracy gain or validate individual event-to-target and effect-window alignment.
REIN improves MAE and MSE over the frozen base in all 20 settings. The revisions will preserve this forecasting evidence and the supported architectural comparisons while correcting the attribution and guarantee claims. All new training comparisons use one seed, so we will keep that limitation explicit.
Response part (1/4)
Thank you for pinpointing the controls needed to assess our attribution claim. We completed both, together with the inactive-subset analysis. The results clarify which claims the evidence supports, and we will revise the manuscript accordingly, as explained at the end.
All added runs use one seed (2024, as in the submission), so we report values, directions and counts and claim no statistical significance. Tables show means over the five horizons unless stated, and percent changes are means of per-horizon changes. Per-horizon results and full intervention, deviation and inactive-subset statistics appear in the comment “Supplementary tables”. Traffic MAE is scaled by 10⁴ and Gas Demand MAE by 10⁶.
Q1 and W1
We agree that Blank alone cannot establish news utility. However, the new controls allow us to separate the value of residual learning from sensitivity to news inputs. A zeroed pack closes the soft gate and shrinks the residual coefficient to its floor (λ{min}=00˙5), so Blank keeps only a minimum-strength residual. The Active–Blank gap is 43–91% of the Active–Base gap across the 20 settings and mainly measures the strength of the residual correction. We will remove it as evidence of news value, including the 42.6% figure in the abstract.
NN1 implements the baseline you describe with frozen DLinear, the same PatchTST encoder, the same day-of-week, hour-of-day and holiday embeddings and the same three heads, with no news. All news inputs are zero, the gate is fixed open and the news knobs are fixed at their neutral values. NN uses the same zeroed inputs and open gate but retains the knobs’ ranges, making them learnable constants. Both controls use the submitted training objective and data.
The comparison reports mean MAE, mean percentage gains over the base and the share of REIN’s gain recovered by NN1 at each horizon.
These results support residual learning as the main source of gains on Demand, Traffic and Gas. They also show that REIN remains competitive with the no-news controls, with lower MAE than both in half the settings. We will retain these forecasting results while avoiding semantic attribution. Mass, gating and regularization remain possible contributors to the remaining differences.