- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces REGIME-TIMESFM, a method that combines TimesFM’s numerical forecasts with features—such as market events and policy states—extracted by an LLM from official documents, refining the predictions via a Ridge residual adapter incorporating interaction terms. The approach also utilizes residual quantiles to generate prediction intervals and employs an auxiliary classifier to estimate extreme price risk. Experiments primarily cover 1-, 6-, and 24-hour forecasts for the Guangdong and Jiangsu markets, comparing the original TimesFM against numerical adapters and language-conditioned adapters, while analyzing various market states and text alignment methods. The authors report that the benefits of language features are conditional and note the withdrawal of external test results for Sichuan due to timestamp-related issues.
Critical strengths
The authors clearly distinguish between numerical prediction and text information extraction, retaining extraction failures as missing information rather than defaulting to "no event." The decision to retract scores after discovering timestamp issues in the Sichuan test also demonstrates a prudent approach to handling the boundaries of evidence. While these design choices are sound, they do not yet fully compensate for the shortcomings in the current core validation.
Critical weaknesses
In Table 3, compared to the numerical adapter, the language-conditioned model shows a performance degradation of 0.23 and parity at the three Guangdong forecast horizons; for Jiangsu, it degrades by 0.58 at the 1-hour horizon and improves by only 0.08 at the 6-hour horizon. The most notable improvement occurs at the Jiangsu 24-hour horizon, where the MAE drops from 42.04 to 34.33; however, the original TimesFM achieves 33.18, outperforming it. Thus, these results primarily demonstrate that language features mitigate the issue of a degraded adapter, rather than proving superiority over currently available numerical forecasting methods. More critically, the "strict_observed_future_text_prohibited" control in Figure 5 yields MAEs of 42.04 (Jiangsu 24-hour) and 31.71 (Jiangsu 6-hour), neither of which outperforms the corresponding numerical adapters in Table 3. Conversely, the "correct_aligned_text" setting in Figure 5 aligns with the main table's language-based results, showing MAEs of 34.33 and 31.38, respectively. The paper fails to clearly explain the relationship between these two alignment methods or to establish evidence that the primary gains persist under the strict protocol. Given that preventing temporal leakage is a key contribution, this issue directly impacts the paper's conclusions rather than serving merely as a secondary sensitivity analysis. Furthermore, the text controls do not convincingly demonstrate the superiority of "structured linguistic knowledge" over simple information; the document count control for Jiangsu 6-hour (31.23) outperforms the full language model (31.38), and the vocabulary control for Jiangsu 24-hour (33.72) also outperforms the full language model (34.33). The paper highlights the full model's superiority over certain controls while failing to adequately explain why it underperforms against other, simpler controls. Consequently, the current results make it difficult to disentangle the value of semantic content from the effects of document frequency, lexical characteristics, or adapter design. Time-MMD has already aligned external reports and event texts with numerical sequences, employing LLMs for filtering, fact extraction, and summarization, while providing an evaluation framework that integrates language models with numerical forecasting models. Consequently, there are already direct precedents for "augmenting numerical forecasting with external text" and "text-numerical temporal alignment." The added value of the present paper must therefore stem from a reliable representation of market states or more rigorous evidence of validity—aspects that the current experiments have not yet sufficiently established.
Both Table 5 and Figure 4 report the conditional MAE for the language model minus the numerical adapter, yet the values corresponding to the same market, time horizon, and stratification differ. These discrepancies far exceed the limits of display precision, and the paper offers no explanation attributing them to different data splits, memory windows, or experimental configurations. Conditional stratification underpins the core conclusion regarding "when text is useful"; until these values are reconciled, that conclusion cannot be reliably verified. The "prior-donor" control values reported in Section 6.4 also fail to match the corresponding controls shown in Figure 5, further raising concerns about the inconsistent use of result versions. A line-by-line reconciliation of experimental records is required here, rather than merely revising figure captions. While the title and abstract emphasize cross-market prediction—and Section 5.2 claims to evaluate adaptation to target markets over horizons of 0, 1, 7, and 30 days—the current manuscript presents no results, adaptation curves, or comparative analyses regarding transfer from source to target markets. Local prediction results for the two markets cannot substitute for transfer experiments.
Questions for authors
see weaknesses