- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces a time series forecasting framework, Navigator. Its idea is that, rather than relying solely on end-to-end optimization to discover predictive patterns, a forecasting model should first extract prior knowledge about which historical information is most predictive of future observations directly from the training data. When modeling information exchange across variables, the model should not rely solely on pairwise similarity between variables, but should also consider whether the information being propagated is geometrically aligned with the underlying relational structure.
Critical strengths
s1: The authors conduct extensive experiments and analyses to demonstrate the effectiveness and robustness of Navigator.
s2: The experimental section is presented in considerable detail, providing a comprehensive evaluation of Navigator.
Critical weaknesses
w1: The paper emphasizes explicitly extracting 'predictive structure'' from the training data and using it as prior knowledge for forecasting. However, the notion of `predictive structure'' does not appear to be clearly and precisely defined. More importantly, the current experiments do not convincingly demonstrate that the performance gains are specifically attributable to exploiting this prior knowledge. In particular, the ablation studies and plug-in experiments demonstrate the effectiveness of Foreseer, but they do not sufficiently isolate the contribution of the proposed predictive structure itself.
w2: The paper appears to address two relatively independent problems: extracting predictive temporal directions from the training data and designing a geometry-aware mechanism for cross-channel interaction. However, the connection between these two problems is not sufficiently clear. It is unclear why they should be addressed jointly within a single framework, or what fundamental relationship motivates their integration. As a result, the overall motivation and narrative of the paper are somewhat difficult to follow.
w3: The performance gains of Geometric Bending Attention (GBA) over existing attention mechanisms appear to be relatively modest in the reported experiments. More importantly, the intuition behind GBA is not entirely clear. In particular, the paper proposes to assess the relationship between the query and key representations while also determining whether the information carried by the value representation is directionally compatible with the corresponding interaction. It is not immediately clear (lines 444) how such directional compatibility should be interpreted or why the geometric relationship between the query--key and key--value transitions provides a meaningful criterion for determining interaction strength.
w4: The paper discusses several limitations of the proposed approach, including the reliance on second-order statistics, the fixed nature of the Predictability Basis after training, and the limited modeling of higher-order joint relationships. However, it is unclear whether these limitations are supported by substantive empirical evidence.
Questions for authors
See the Weaknesses.
5 comments
Response to Reviewer oa89 (Part 2)
W3. Performance Improvements and Geometric Motivation of GBA
A3:
(i) Performance Improvements.
As shown in Table 4, GBA outperforms standard Full Attention across all four evaluated datasets, achieving an average relative MSE reduction of 3.66%. These improvements are obtained under the same forecasting architecture and experimental settings, with only the attention mechanism replaced. Although the magnitude of improvement varies across datasets, the results consistently support the effectiveness of GBA.
(ii) Geometric Interpretation and Motivation.
Standard attention determines interaction strengths based on query–key similarity, while the information ultimately aggregated is carried by the value representation. Consequently, query–key similarity alone does not account for the geometric relationship between the matching process and the information being aggregated.
GBA addresses this by considering the ordered configuration , where the query-to-key transition represents the relationship between the target and source channels, and the key-to-value transition describes the displacement toward the information being aggregated. Directional compatibility measures whether the key-to-value transition continues or deviates from the direction established by the query-to-key transition. By favoring directional continuity, GBA introduces a geometric inductive bias that jointly considers Q–K, Q–V, and K–V relationships when determining interaction strengths.
(iii) Additional Experimental Evidence.
To further examine whether GBA's improvements arise from its specific geometric scoring criterion rather than simply using cosine similarity or incorporating value representations into attention scoring, we conduct additional controlled experiments comparing GBA with Q–K Cosine Attention and Q–V Cosine Attention under the same architecture and experimental settings.
| ETTm1 | Weather | Solar-Energy | |
|---|---|---|---|
| GBA | 0.373/0.376 | 0.237/0.259 | 0.192/0.212 |
| Q–K Cosine Attention | 0.380/0.381 | 0.245/0.271 | 0.204/0.220 |
| Q–V Cosine Attention | 0.377/0.380 | 0.243/0.268 | 0.204/0.219 |
(Results are reported as MSE/MAE, averaged over four prediction horizons.)
As shown, GBA consistently outperforms both alternatives across all three datasets in terms of MSE and MAE. These results indicate that GBA's improvements cannot be attributed solely to cosine-based scoring or the inclusion of value representations in attention scoring, providing direct empirical support for the proposed directional compatibility criterion.
Overall, the geometric interpretation and controlled experiments support GBA as an effective value-aware attention mechanism, with its advantages arising from modeling the ordered query–key–value configuration rather than relying on conventional pairwise similarity alone.
W4. Clarification of the Limitations
A4:
(i) Reliance on Second-Order Statistics.
As described in Sections 3.2 and 4.2.1, the Predictability Basis is constructed from covariance and cross-covariance matrices, which characterize linear statistical relationships between historical and future observations. Therefore, the basis does not explicitly capture nonlinear predictive dependencies that cannot be characterized by second-order statistics. However, this limitation concerns the basis construction rather than the entire Navigator model, which incorporates nonlinear transformations in subsequent layers.
(ii) Fixed Predictability Basis.
As described in Section 4.2.1, the Predictability Basis is constructed by aggregating historical–future statistical relationships across the training data, which may already reflect diverse non-stationary patterns. Although the basis remains fixed, it produces input-dependent representations for different historical windows. Its potential limitation arises when previously unseen predictive relationships emerge, as the basis cannot explicitly update its temporal directions to accommodate such changes.
(iii) Higher-Order Cross-Channel Relationships.
As defined in Eq. (4), GBA computes geometric compatibility scores from individual query–key–value configurations associated with channel pairs. However, the subsequent softmax normalization and weighted aggregation involve all channels, while stacked BendingFormer layers can further capture complex cross-channel dependencies. Therefore, the limitation concerns the absence of an explicit higher-order scoring mechanism involving three or more channels simultaneously, rather than an inability to model higher-order dependencies.
Overall, these limitations reflect the theoretical and architectural boundaries of our current design, rather than experimentally demonstrated shortcomings. We will clarify this distinction in the revised manuscript to avoid overstating their practical implications.
Thanks for the detailed response and the additional experiments. I retain Weak rejection.
Response to Reviewer oa89 (Part 1)
Dear Reviewer oa89:
We sincerely thank you for your constructive feedback. We have carefully addressed your concerns and revised the manuscript accordingly. Our point-by-point responses are provided below.
W1. Clarification of Predictive Structure and Additional Experimental Evidence
A1:
(i) Clarification of Predictive Structure.
In our paper, predictive structure refers to the statistical relationships between historical and future observations, particularly which temporal directions in the historical observation space are more informative about future observations. To characterize this structure, we formally define Temporal Predictability in Definition 1 (Section 3.2) and establish its closed-form characterization in Proposition 1. Based on this theoretical foundation, Section 4.2.1 provides a detailed procedure for constructing the Predictability Basis by identifying and retaining temporal directions with stronger predictive relevance from the training data. Therefore, the Predictability Basis serves as an explicit, data-derived representation of the predictive structure, which Foreseer subsequently incorporates into the forecasting model as prior knowledge.
(ii) Additional Experimental Evidence.
To more directly isolate the contribution of the extracted predictive structure, we have conducted additional controlled experiments, including comparisons with alternative temporal bases and analyses of the learnable biases.
First, we compare the proposed Predictability Basis with four alternative bases, including Random Orthogonal Basis, PCA Basis, DCT Basis, and Identity Basis, under the same forecasting architecture and experimental settings. Unlike the proposed Predictability Basis, these alternatives do not explicitly exploit historical–future statistical relationships: Random Orthogonal Basis uses randomly generated orthogonal directions, PCA Basis captures variance-dominant directions, DCT Basis employs predefined frequency-domain directions, and Identity Basis preserves the original temporal representation. The results show that the proposed Predictability Basis consistently achieves better forecasting performance, providing further evidence of the benefits of explicitly exploiting predictive structure from the training data. For detailed results, please refer to A1(iii) to Reviewer EM36 and A1(v) to Reviewer fvpJ.
Second, we further investigate the role of the learnable biases in Foreseer to distinguish their contribution from that of the Predictability Basis. The corresponding analysis and experimental results are provided in A1(iv) to Reviewer fvpJ.
Taken together, these additional experiments further support that Foreseer's performance improvements arise from the forecasting-relevant information captured by the Predictability Basis, rather than merely from temporal projection or additional learnable parameters, directly addressing your concern.
W2. Motivation for Integrating Foreseer and BendingFormer
A2:
(i) Motivation and Connection.
The key motivation for integrating Foreseer and BendingFormer is to enable geometry-aware cross-channel interaction over forecasting-relevant temporal representations. Specifically, Foreseer constructs a shared Predictability Basis from historical–future statistical relationships, providing a common predictability-oriented representation across channels. BendingFormer then employs GBA to model interactions among these representations based on the joint query–key–value geometry. Thus, Foreseer determines what temporal information is used for cross-channel modeling, while GBA determines how that information is exchanged across channels.
(ii) Additional Experimental Evidence.
To further examine whether Foreseer and GBA provide complementary benefits, we conduct additional controlled experiments comparing Full Attention and GBA under two temporal representations: a linear projection of raw inputs and the representation produced by Foreseer. The results (MSE/MAE), averaged over four prediction horizons, are summarized below.
As shown, GBA achieves larger improvements over Full Attention when combined with Foreseer, as evidenced by ΔGBAForeseer >ΔGBAOriginal . These results provide empirical evidence of the inherent complementarity between Foreseer and BendingFormer, demonstrating that their integration yields greater benefits than their individual contributions alone.