- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the model of MAGNet, which combines the text information generated by the large language model with the graph network to predict the time series with exogenous variables. The model learned the relationships between exogenous variables and between exogenous variables and target variables, and conducted experiments on 12 datasets.
Critical strengths
None
Critical weaknesses
-
The overall innovation of the method is relatively limited, mainly combining existing text embedding, graph networks, and attention mechanisms. Although the paper introduces the functions of these modules, it does not fully explain why they need to be designed in this way and what problems they solve compared to existing methods.
-
The paper used graphs to model the relationship between exogenous variables and target variables, but since there is only one target variable, this process is actually very similar to using attention mechanisms to aggregate information from exogenous variables. The author did not fully explain why a graph structure is needed, nor did they provide a direct comparison with regular cross-attention. The support for the effectiveness of each module in the ablation experiment is insufficient. For example, after completely removing the text, there is almost no change in model performance, and the improvement in other modules is also very small. On some datasets, removing these modules can even achieve better results.
Questions for authors
Can you briefly explain what problems this article has solved compared to existing methods?
Can you provide a comparison with aggregation methods such as attention mechanisms?
3 comments
Thank you for conducting additional experiments. These results have addressed my concerns to some extent, but they are still insufficient to convince me of the technological innovation of this method.
I still believe that the core design of MAGNet has significant similarities with existing attention aggregation mechanisms. Especially, it is still debatable whether the cross attention comparison currently provided is sufficient to demonstrate the unique advantages of graph structures. MAGNet itself uses two-stage attention aggregation of Exo-Exo and Exo-Endo, so it may not be entirely fair compared to a simple cross-attention module.
The current supplementary experiments have shown that MAGNet has certain performance advantages, but it has not yet been fully demonstrated that its core design has sufficient clear technological innovation compared to existing deep learning modules. So, I decide to keep my original score at 2.
Thank you for your careful follow-up! We understand your skepticism regarding MAGNet's technical innovation and how it differs from usual aggregation methods. We hope this comment clarifies your concerns.
Concern 1: Technical Innovation
What makes MAGNet stand out
- Explicit exo-exo and exo-endo relationships. MAGNet infers these relationships through correlation and dependency metrics. Usual self-attention or cross-attention networks have to infer them implicitly from the same input. MAGNet's inter-node weights complement such mechanisms and improve accuracy.
- Controlled message passing. The correlation and dependency weights are used directly in graph learning to control how much information one node passes to another. This matters because not all exo variables benefit the prediction of future endo patches equally.
- Text as guidance, not as a mixer. This is one of MAGNet's core contributions, separate from the graph attention network. Current models tend to merge text and numerical data into a shared embedding and forecast from it, so a flawed text can directly corrupt the predicted values. In MAGNet, text only guides the attention.
- Text as global information. Many recent models use patching, which emphasizes local information within each patch. In MAGNet, text describes the overall pattern and statistics of the whole series, so every patch is informed uniformly about the context outside the patch itself.
What we do and do not claim
- MAGNet uses attention. MAGNet revolves around attention mechanisms (a graph attention network). It is a technique we adopt and extend to fit the ExoTSF problem.
- Our main contribution is not the attention network itself. MAGNet aims to solve ExoTSF with a text modality. Our contribution lies in the text-guidance mechanism and the exo-endo separation, not in improving existing attention networks.
PatchTST [1] is a comparable case. It applies attention in a manner similar to Vision Transformers (ViT) [2], yet its core contribution is showing that patching is effective for time series forecasting.
Empirical support for these points
- Removing the exo-exo graph while keeping the SIG prior (w/ SIG-NoExo) or replacing the graph with a homogeneous one (w/ HomGraph) both lower accuracy compared with full MAGNet.
- In the shuffled-exo study, shuffling MAGNet's signal raises MAE by about 20-29% in short-term settings for non-exo backbones (DLinear, PatchTST, TFT), so the signal is not redundant.
Concern 2: Fairness of the Ablation Comparison
We agree that the single cross-attention variant is not a fully matched counterpart to MAGNet's two-stage design, and we do not claim that our graph attention network is superior to every attention-based aggregator. Our claim is narrower: for ExoTSF, explicit exo-exo and exo-endo dependency priors, exo-endo separation and text guidance improve forecasting over models that treat exogenous variables implicitly.
References
[1] Nie et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023.
[2] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.
Thank you for your prompt review.
W1, Q1: Problem and Innovation
This mechanism lacks exo-endo separation (symmetrical) and text-infused
This mechanism highlights exo-endo separation and text-infused
This mechanism highlights exo-endo separation and text is only used to put attention in the correct numbers/variables.
MAGNet's performance compared to other models:
Other baselines categories:
Result: The table shows that MAGNet achieves better performance with its combined mechanism to solve exogenous TSF with text modality.
W2, Q2: MAGNet vs Attention Mechanism
To better support, we extended MAGNet's ablation table to more MAGNet's variants:
Result: Full MAGNet performs best on both overall MAE and MSE, showing that the graph network is important.