- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses how to use noisy textual context, such as news and reports, to improve time series forecasting. Its proposed method, TAG-ICF, uses text to guide attention toward relevant historical examples. The numerical forecasting backbone stays frozen, while a small text-guided attention module is trained. The key insight is that text can help determine which past observations are useful for the current forecast. Experiments show strong overall performance, with text improving the same numerical backbone. However, the benefits depend on text quality, temporal alignment, and the underlying forecasting model.
Critical strengths
TAG-ICF uses text-derived similarity to modulate query–context attention within a frozen numerical ICL backbone. This provides a well-motivated separation of roles: text guides the weighting of historical examples, while the pretrained backbone handles numerical prediction.
Critical weaknesses
- The contribution appears incremental, combining text-based relevance weighting with a frozen ICL backbone. The paper offers limited new insight into when or why this combination is effective.
- The experiments do not compare different ways of using text while keeping the numerical backbone and text inputs the same. This leaves it unclear whether TAG’s attention mechanism offers an advantage over simpler ways of incorporating text.
Questions for authors
- Were different ways of incorporating text evaluated using the same numerical backbone and processed text? If so, how do the results isolate the benefit of TAG’s attention mechanism?
- What does TAG’s attention-level formulation enable beyond existing approaches that use text to select or weight historical examples? Which findings support this distinction?
3 comments
Q2
Attention-level fusion vs. text-based selection/weighting
To our knowledge, TAG-ICF is the first method to fuse text by directly updating an ICL backbone’s query–context attention scores. The closest prior work is context retrieval and selection for ICL models, which acts as a preprocessing step that decides which examples enter the context [1, 2] rather than updating the attention scores. We also believe text-based context selection is not ideal for forecasting, because it breaks the temporal continuity of the context. TAG-ICF instead keeps the full contiguous context and reweights it softly inside the model.
The experiment below supports this argument. We compare against a text-semantic retrieval baseline that selects the top-K context examples, using the same backbone and text embeddings. Simple text-based retrieval does not help forecasting: for both values of K, it is worse than the numerical-only model with full context on all five datasets and both metrics, and TAG-ICF performs much better.
| Dataset (MSE / MAE) | Text retrieval, top 50 | Text retrieval, top 100 | No text (full context) | TAG-ICF |
|---|---|---|---|---|
| Economy | 0.842 / 0.693 | 0.671 / 0.660 | 0.576 / 0.604 | 0.537 / 0.578 |
| Environment | 0.439 / 0.475 | 0.376 / 0.433 | 0.337 / 0.400 | 0.326 / 0.391 |
| Security | 20.038 / 3.366 | 13.545 / 2.989 | 8.491 / 2.517 | 7.681 / 2.343 |
| Social Good | 0.182 / 0.293 | 0.143 / 0.273 | 0.131 / 0.262 | 0.108 / 0.242 |
| Electricity | 1.035 / 0.782 | 0.879 / 0.706 | 0.708 / 0.630 | 0.646 / 0.602 |
In addition, ConTextTab, one of our baselines, feeds text embeddings into an ICL model as input features, so its attention scores combine textual and numerical information implicitly. It shows little or no gain from text on most datasets, whereas TAG-ICF improves on nearly all of them (Tables 2-3). We will include the above discussion and results in the revision.
[1] Thomas et al. Retrieval & Fine-Tuning for In-Context Tabular Models. NeurIPS 2024.
[2] Xu et al. Mixture of In-Context Prompters for Tabular PFNs. ICLR 2025.
Dear Reviewer:
We hope this message finds you well, and thank you again for your initial review. As the discussion deadline approaches, we would appreciate your feedback on our updated responses and additional results.
If our response has addressed your concerns, we kindly ask you to consider raising your score. If any remain, we are happy to discuss which specific criteria still need to be met to support an increase.
Thank you again for your time and consideration.
We thank the reviewer for taking the time to review our paper, and are glad that they found the text guidance in TAG-ICF well motivated.
W1
We thank the reviewer for the comment. The core technical contribution of TAG-ICF is fusion through a text-derived attention guidance mechanism, which uses textual information to reweight historical numerical contexts rather than simply fusing text with numerical inputs. This also differs from approaches that bring text into their frozen or fine-tuned backbones through representation-level fusion or LLM-based prompting.
On why the combination is effective, Figure 1 gives a qualitative example from one of the benchmark datasets. Text guidance changes the backbone’s attention over the historical examples, which corrects the forecast direction. This internal attention change is consistent with the intuition behind the method.
On when it is effective, our experiments cover twelve datasets from diverse domains, where text improves over the same backbone without text on 11 of 12 (Tables 2-3). We also provide ablation studies on the ICL backbone, the text embedding model, text preprocessing, text alignment, etc. (Sections 4.3-4.4, Appendix D), which show that the gain carries over to another backbone and other embedding models, and that it comes from the semantic content of the text. We will add these results to the revision.
Q1/W2
We thank the reviewer for raising this point. Early and late fusion on the same backbone were our starting point. In early-stage experiments, neither could reliably extract a useful signal from the text, and this observation is what motivated similarity-based attention adjustment. We have now run both variants systematically with hyperparameter optimization, using the same TabDPT backbone, inputs, and preprocessing as TAG-ICF in the paper:
Out of the three text-integration mechanisms, our proposed attention guidance is the most effective: TAG-ICF achieves the lowest MSE and MAE on all five datasets. Neither early nor late fusion consistently turns the same GPT-5.2-processed text into forecasting gains. Early fusion is worse than the no-text baseline on every dataset and both metrics. Late fusion improves over the baseline only on Security, which is also the only dataset where it comes close to TAG-ICF. We will add these results in the revision.