- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ForeSeek, a framework for reducing the computational cost of LLM-based time-series forecasting through query-conditioned evidence selection. It selects segments from a mixed sequence of textual prompts and temporal patches using a horizon-aware router. During training, a full-context teacher provides attention-alignment and prediction-distillation supervision. At inference, the teacher is removed, and predictions from the selected-evidence LLM branch are combined with those from a lightweight temporal branch. The framework is instantiated on Time-LLM and S2IP-LLM and evaluated on seven forecasting benchmarks, including long-term, few-shot, and cross-dataset transfer settings. Additional experiments examine selector alternatives, component ablations, random-seed stability, and computational costs. The reported results show competitive forecasting accuracy, improvements across the evaluated ETT transfer tasks, and substantial inference FLOPs reductions, although accuracy and wall-clock benefits vary across datasets and backbones.
Critical strengths
A useful accuracy–efficiency trade-off. On Time-LLM, ForeSeek reduces total inference FLOPs by 52.56% and measured end-to-end latency by approximately 28.5% at the default retention ratio. It also improves average MSE on ETTh2 and ETTm2 by approximately 5% and 8%, respectively. These results provide concrete evidence that selective context processing can reduce deployment costs while maintaining or improving forecasting accuracy on several datasets, supporting the method’s practical value.
Consistent gains in the evaluated transfer tasks. Across the eight ETT cross-dataset transfer tasks, both ForeSeek variants improve average MSE over their corresponding original backbones, and a ForeSeek variant achieves the best average MSE in every task among the evaluated methods. This consistency across Time-LLM and S2IP-LLM strengthens the empirical case for the framework’s usefulness under distribution shift within the ETT benchmark family.
Critical weaknesses
The core method is specified inconsistently across the paper. Equation (4) and Algorithm 1 construct the routing query from both the recent state and the forecast horizon, whereas the backbone-specific implementations in Equations (24) and (29) use only future temporal markers, omitting the recent-state component. Similarly, Equation (20) introduces a supervised teacher loss that is absent from Equation (10) and Algorithm 1. These discrepancies concern central components of the proposed method and make it difficult to determine which formulation produced the reported results. Clarifying the implemented query, teacher training procedure, and complete optimization objective is necessary to assess reproducibility and whether the experiments validate the stated contribution.
The ablations do not sufficiently isolate the contribution of forecast-conditioned routing. Table 16 provides useful matched-budget comparisons, but only ForeSeek’s selector is marked as trainable; the horizon-only and cosine-similarity selectors are not. Consequently, the comparison does not isolate the benefit of combining recent-state and horizon information from the benefit of learning the selector. Moreover, the paper states that gradients do not pass through discrete Top-K selection and that the router is trained through attention alignment. Removing alignment therefore leaves no specified learning signal for the router-specific parameters, complicating the interpretation of Tables 5 and 17. Comparisons using equally trainable recent-only, horizon-only, and full-query routers, together with clarification of how the no-alignment variants are trained, would substantially strengthen the evidence for the paper’s central mechanism.
Questions for authors
Clarification: Which query formulation and training objective produced the reported results, given the inconsistencies between Eqs. (4), (24), and (29), and between Eqs. (10) and (20)?
Clarification: When attention alignment is removed, what learning signal updates the router parameters, given that Top-K selection is non-differentiable?
Targeted validation: Does ForeSeek still outperform equally trainable recent-only and horizon-only routers under the same supervision and token budget?
4 comments
W2 / Q2. What trains router parameters when attention alignment is removed?
For parameters used only to compute routing scores, hard Top-K provides no gradient from the task or consistency loss. With and detached , Eq. (7) gives
=\lambda_{\mathrm{align}}\bigl(p_R(i)-p_A(i)\bigr).$$ Removing alignment therefore removes the specified learning signal for score-only parameters. Prediction heads and shared representations can still receive gradients through their forecasting losses. This ablation measures the effect of removing routing supervision. To compare trainable routers directly, the following controls use real, shuffled, and mean-attention targets. Each has a trainable KL objective, the same full query, and a 57-token budget. Entries are final MSE. | Alignment target | ETTh2 / 96 | ETTh2 / 720 | ETTm2 / 96 | ETTm2 / 720 | | --- | --- | --- | --- | --- | | Sample-specific attention | 0.281 | 0.392 | 0.163 | 0.358 | | Cross-sample shuffled attention | 0.282 | 0.363 | 0.166 | 0.361 | | Training-average attention | 0.288 | 0.367 | 0.164 | 0.365 | | No alignment | 0.288 | 0.387 | 0.179 | 0.371 | Real attention gives the lowest MSE in three settings. Shuffled attention performs best on ETTh2/720. The first three rows compare the information supplied by different training targets. The no-alignment row measures the effect of removing that supervision entirely. --- > #### **W2 / Q3. Does ForeSeek outperform equally trainable recent-only and horizon-only routers?** $\color{green}{\mathbf{Response.}}$ Yes, in all four evaluated settings. The comparison you request uses the same alignment target, downstream architecture, and 57-token budget, varying only the query inputs. Entries are final MSE for ForeSeek-Time. | Trainable query | ETTh2 / 96 | ETTh2 / 720 | ETTm2 / 96 | ETTm2 / 720 | | --- | --- | --- | --- | --- | | Recent only | 0.288 | 0.404 | 0.167 | 0.363 | | Horizon only | 0.299 | 0.410 | 0.170 | 0.367 | | **Recent + horizon** | 0.281 | 0.392 | 0.163 | 0.358 | Relative to the better single-input query, the joint query reduces MSE by 1.4–3.0%. With trainability, supervision, and token budget held fixed, this comparison supports the complementary value of recent state and horizon information in the tested settings.Thank you for the clarifications and additional experiments. The revised formulation and the matched trainable query controls have addressed some of my earlier concerns.
I still have two concerns:
-
It is unclear how the separate full-context prediction head in ForeSeek-Time was initialized. Although it receives no gradient, was it loaded from a pretrained forecasting model? If so, what data was used to train it?
-
The new attention controls do not fully convince me of the benefit of sample-specific attention supervision. The gains over the best shuffled or average targets are small in three settings. More notably, on ETTh2/720, shuffled attention achieves 0.363 MSE, compared to 0.392 with sample-specific attention. These results make it difficult to attribute the improvements specifically to the proposed attention-conservation mechanism.
I appreciate the authors' efforts during the rebuttal, yet I decided to maintain current score.
Thank you for identifying these two remaining issues.Full-context head initialization. In the reported ForeSeek-Time runs, we initialized the separate full-context prediction head using PyTorch’s default linear-layer initialization. We used no forecasting checkpoint or separate training dataset for this head. The two paths share the pretrained GPT-2 backbone. The full-context forecast is detached in the prediction consistency loss, so that loss does not update the separate head. We will make the initialization explicit and clarify the status of this forecast as a consistency target.Sample-specific attention supervision. We agree that the results support a narrower claim. Sample-specific attention yielded modestly lower MSE than the better of the two controls in three of the four settings. On ETTh2 at horizon 720, cross-sample shuffling achieved 0.363 MSE versus 0.392 for sample-specific attention. These results provide limited evidence for an incremental benefit from matching the attention target to each sample. We will revise the manuscript to reflect that limitation when discussing which components account for ForeSeek’s overall forecasting gains.
We are grateful for your time and insightful feedback on the routing mechanism and its training. Below, we clarify the gradient flow and use matched controls to examine the contribution of each query component.
Response. Thank you for pointing out the inconsistency between the main text and the appendix. Both retained implementations construct the routing query as q=Wq(eq+rH), where eq summarizes recent patch tokens and rH encodes known time markers for the forecast interval. We corrected Eqs. (24) and (29) to include eq.
Equation (10) combines fused-forecast MSE, attention alignment, and prediction consistency,
L=Ltask+λalignLalign+λconsLcons.
The auxiliary teacher MSE had weight zero in the reported configuration. Revised Eq. (20) omits that term. The alignment and consistency terms apply stop-gradient to the teacher attention and forecast, respectively. Revised Eq. (22) now uses the same stop-gradient convention as Eq. (9) and Algorithm 1.
We further clarify the two auxiliary objectives in Eq. (10) and their default coefficients. Beyond the task loss, both auxiliary terms are used in all reported experiments with λalign=1 and λcons=0.5.
The Attention Conservation term is
Lalign=KL(sg(pA)∥pR).
This term aligns the routing distribution pR with the stop-gradient teacher attention pA and is the only direct learning signal for the score-only router parameters, which is why removing alignment removes their gradient, as measured by the no-alignment ablation.
The Prediction Consistency term is
Lcons=SmoothL1(Y^ev,sg(Y^T)).
This term distills the teacher forecast Y^T into the evidence path Y^ev to stabilize prediction under compressed context.
The auxiliary teacher MSE in the original appendix was intended to give the teacher branch direct prediction supervision so that its forecast would be better calibrated early in training and thus a more stable target for consistency distillation. The term was assigned weight zero (λaux=0). It therefore contributed zero to the training objective. The full-context branch provided targets through the two stop-gradient objectives above. We have removed the term from Appendix A.1, and the revised Eqs. (20)–(22) now match the three-term objective and stop-gradient convention in the main text and Algorithm 1. Both auxiliary terms act only during training and add no inference cost.
In the saved ForeSeek-Time implementation, the full-context and evidence paths share a pretrained, frozen GPT-2 and have separate prediction heads. The full-context GPT-2 pass runs without gradient tracking. With auxiliary MSE disabled and the teacher forecast detached, the separate full-context head receives no gradient from the ForeSeek objective.
In the saved ForeSeek-S2IP implementation, the two paths share the pretrained GPT-2 and the forecasting head. The full-context GPT-2 pass runs without gradient tracking. The shared head, GPT-2 layer norms, and positional embeddings can receive gradients through the evidence path.