- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CWRE, a transferability estimator for source dataset selection in time-series forecasting. Instead of evaluating individual forecasting windows alone, CWRE considers how changes in learned representations correspond to changes in future sequences across adjacent windows. The method combines cross-window relations, trend–seasonal decomposition, and ridge regression to estimate transferability without target-domain fine-tuning. Experiments across several datasets and forecasting architectures show improved source-ranking agreement over existing methods. I find the cross-window perspective interesting, although its generalization and methodological novelty need further justification.
Critical strengths
-
The paper studies a meaningful problem. Selecting transferable source datasets without expensive target fine-tuning has clear practical value.
-
I find the cross-window relation idea interesting. Considering representation changes rather than only individual windows provides a useful additional perspective on transferability.
-
The empirical evaluation is reasonably comprehensive. The cross-architecture experiments and missing-observation settings are useful, and the ablation results support the contribution of cross-window modeling.
Critical weaknesses
-
The methodological novelty is moderate. Regression-based transferability estimation is well established, and CWRE mainly extends it with temporal differences and decomposition.
-
The experiments use only three target datasets and six candidate sources per target. I am not sure whether the ranking improvements would remain stable with larger source pools.
-
Hyperparameter selection needs more justification. Each source independently selects its best scoring configuration, which may affect the comparability of transferability scores.
-
Adjacent windows overlap heavily. It is still unclear how much of the improvement comes from meaningful temporal evolution rather than this structural dependence.
Questions for authors
-
How does CWRE differ fundamentally from existing transferability estimators such as LinMSE and TEMPLATE, beyond introducing adjacent-window differences?
-
Could the authors clarify why independently optimized hyperparameters produce comparable scores across different source datasets?
-
Would the improvements remain similar with non-overlapping windows or larger temporal gaps?
-
Could the authors provide results on more target datasets or larger candidate-source pools to better demonstrate generalization?
7 comments
Response To Reviewer 9nq7 [Part 2]
Response to W2 and Q4: Generalization Across Target Datasets and Larger Source Pools
Reviewer's Concern (W2): The experiments use only three target datasets and six candidate sources per target. I am not sure whether the ranking improvements would remain stable with larger source pools.
Reviewer's Question (Q4): Could the authors provide results on more target datasets or larger candidate-source pools to better demonstrate generalization? Our original experiments cover three targets, six candidate sources per target, two scoring architectures, and three downstream evaluators, under three missing rates with multiple mask seeds. Following the reviewer's suggestion, we further extend the evaluation to a fourth target and a longer forecasting horizon.
1. Additional Target Dataset: Weather (512→96)
We add Weather as a fourth target, evaluated across MR = 0, 0.3, and 0.6. Results are averaged over PatchTST and TimeEmb, with additionally averaged across three downstream evaluators.
\hline \text{MR} & \text{CWRE }(\tau_w/W) & \text{ETran }(\tau_w/W) \\\\ \hline 0 & \mathbf{0.4093/0.4500} & 0.2497/0.4179 \\\\ 0.3 & 0.3148/\mathbf{0.5072} &\ \mathbf{0.3631}/0.4964 \\\\ 0.6 & \mathbf{0.3209/0.3429} & 0.1946/0.2857 \\\\ \hline \end{array}$$ **Findings:** CWRE achieves higher Kendall's $W$ than ETran across all three missing rates and higher $\tau_w$ in two of three settings, providing additional evidence of effectiveness on a new target domain. **2. Longer Forecasting Horizon: 512→512** We further extend the forecasting horizon from **96 to 512**, evaluating all four targets using two scoring architectures and three downstream evaluators under MR = 0. $$\begin{array}{llcc} \hline \text{Setting} & \text{Method} & \tau_w & W \\\\ \hline 512\rightarrow96 & \text{CWRE} & \mathbf{0.3820} & \mathbf{0.3946} \\\\ & \text{ETran (Best Baseline)} & 0.1978 & 0.3277 \\\\ \hline 512\rightarrow512 & \text{CWRE} & \mathbf{0.0914} & \mathbf{0.2393} \\\\ & \text{MMD (Best Baseline)} & 0.0616 & 0.2268 \\\\ \hline \end{array}$$ **Findings:** CWRE achieves the highest overall ranking agreement on both metrics, supporting its effectiveness beyond the original forecasting horizon. **3. Generalization to Larger Candidate-Source Pools** - **Current Coverage:** Our candidate sources span **five representative time-series forecasting domains**: electricity transformers, electricity consumption, transportation, exchange rates, and weather, providing diverse source distributions for evaluating transferability. - **Limitation:** We agree that evaluating a substantially larger candidate-source pool would provide stronger evidence regarding ranking stability as the number of candidates increases. Since expanding the pool requires source pretraining and downstream reference evaluation for every added dataset and architecture, we do not overclaim this aspect from the current experiments. The additional target and horizon results have been incorporated into **Sections 3.2 and 3.4 (Tables 1, 2, 3, and 4)**, extending our evaluation from **three to four targets and from one to two forecasting horizons**. Since Table 3 reports cross-target averages, we update it to include Weather for all methods, while keeping the original three-target results and evaluation protocols unchanged.Response To Reviewer 9nq7 [Part 3]
Response to W3 and Q2: Shared Hyperparameter Selection and Source-Score Comparability
Reviewer's Concern (W3): Hyperparameter selection needs more justification. Each source independently selects its best scoring configuration, which may affect the comparability of transferability scores.
Reviewer's Question (Q2): Could the authors clarify why independently optimized hyperparameters produce comparable scores across different source datasets?
We apologize that our description may have led to this misunderstanding. No source-specific hyperparameter selection is performed in our experiments, either during source pretraining or CWRE scoring.
1. Actual Hyperparameter Selection Protocol
- Source Pretraining: For a given scoring architecture, all six candidate sources use the same architecture, optimization settings, and training protocol. No source-specific hyperparameter tuning is performed during pretraining.
- Selection Criterion: These parameters are selected using CWRE's own target-domain scoring objective, without downstream fine-tuning performance or ranking evaluation metrics.
- Source Evaluation: All source-pretrained encoders remain frozen. Each source is evaluated using the same configuration:
2. Source-Score Comparability
No source-specific score maximization occurs. All candidates share the same target data, scoring objective, and hyperparameter configuration. Only their source-pretrained representations and the corresponding derived projections differ.
Thus, source scores are compared under a unified evaluation protocol, eliminating the comparability concern arising from independently optimized scoring configurations.
We will clarify this protocol in Section 3.1 and Appendix C.1.2 (Eq. 44), and correct the misleading description that suggested independently optimized configurations for individual sources.
Response To Reviewer 9nq7 [Part 1]
Dear Reviewer,
We sincerely thank you for your careful review and constructive feedback. Your comments on methodological novelty, hyperparameter fairness, overlapping windows, and generalization have helped us clarify our design rationale and strengthen the experimental validation of CWRE.
Response to W1 and Q1: Novelty Beyond Regression-Based and Temporal-Pattern Transferability Estimation
We agree that regression and temporal decomposition are established techniques. However, the novelty of CWRE lies in introducing a new transferability criterion: whether changes in source-pretrained representations correspond to changes in future forecasting targets across successive windows.
1. Beyond Conventional Transferability Criteria: Evaluating Predictive Local Evolution
Existing estimators assess source transferability from different perspectives:
In time-series forecasting, overlapping windows represent successive states of the same temporal process, where historical contexts and future sequences evolve jointly. Thus, CWRE extends transferability estimation from evaluating individual forecasting states to additionally evaluating whether their predictive relations evolve consistently across successive windows.
2. Beyond Simple Temporal Differencing: Constrained and Noise-Aware Cross-Window Evaluation
CWRE evaluates whether representation changes explain the corresponding future changes, providing complementary evidence beyond instance-wise compatibility.
However, overlap can introduce correlations unrelated to prediction. Therefore, CWRE uses instance-wise relations to constrain cross-window relations through a shared linear mapping, limiting the independent fitting of potentially spurious cross-window correlations.
Furthermore, trend–seasonal decomposition mitigates noise amplification from differencing: the trend branch jointly evaluates both relations, while the seasonal branch retains complementary instance-wise information. Ridge regularization and closed-form solutions enable efficient scoring without target-domain fine-tuning.
3. Experimental Evidence
Component ablations. Under MR = 0, averaged over four targets using PatchTST, the results are:
\hline \text{Method} & \tau_w & W \\\\ \hline \mathbf{CWRE} & \mathbf{0.3626} & \mathbf{0.3804} \\\\\\ \text{w/o Cross-Window} & 0.0921 & 0.2643 \\\\ \text{w/o Shared Mapping} & 0.1236 & 0.2839 \\\\ \text{w/o Trend-Seasonal} & 0.2118 & 0.3143 \\\\ \hline \end{array}$$ These results show that CWRE's gains cannot be attributed to regression alone: **cross-window modeling, shared mapping, and decomposition each provide substantial contributions.** **Temporal-step sensitivity.** We evaluate different temporal steps on ETTh1 under MR = 0, averaging results across both scoring architectures. $$\begin{array}{ccc} \hline \text{Temporal Step} & \tau_w & W \\\\ \hline s=0\\ (\text{State-only}) & 0.1347 & 0.4107 \\\\ \mathbf{s=1} & \mathbf{0.5420} & \mathbf{0.5429} \\\\ s=95 & 0.3342 & 0.4536 \\\\ \hline \end{array}$$ **Findings:** CWRE outperforms the state-only variant even at larger temporal gaps, indicating that **cross-window relations provide useful transferability information beyond immediately adjacent windows**, although their effectiveness varies with the temporal step. **Overall, CWRE's novelty is not the individual use of regression, differencing, or decomposition, but introducing cross-window evolution correspondence as an additional transferability criterion and modeling it jointly with instance-wise compatibility under a shared mapping.** We will make these methodological distinctions more explicit in **Sections 2.3–2.4**, supported by the ablation and temporal-step analyses in **Section 3.3 and Appendix C.8**.