- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Learnable Difference-Aware Embedding (LDAE), a lightweight plug-and-play module that constructs channel-wise rate-of-change and curvature features using learnable convolutions initialized with finite-difference stencils. In multivariate time series forecasting, existing embeddings in representative forecasting backbones do not explicitly encode first- and second-order temporal differences at the initial representation stage. LDAE fuses these derivative cues with the backbone representation at the initial embedding inteface and improves forecasting accuracy across several backbones.
Critical strengths
- In MTSF, accurate forecasting requires models to capture not only long-range dependencies and cross-variable interactions, but also non-stationary local dynamics such as channel-specific spikes, asymmetric rate changes, and sharp curvature variations. The authors use first- and second-order differences to capture these features and inject them directly at the embedding stage. This design makes the module easy to apply and transfer across different backbones, without adding more computational cost or design complexity.
- The authors conduct extensive experiments to show the improvements from adding LDAE to different backbones, such as the results in Tables 19 and 20 of the Appendix. They also visualize the trained FD-initialized kernels and analyze training time and memory usage. These experiments make the paper more complete and provide a broader evaluation of the method.
Critical weaknesses
- In Figures 8, 9, and 10, adding LDAE to SimpleTM and iTransformer gives only small and inconsistent improvements. On some datasets, such as ETTh2 and Traffic, LDAE even reduce the backbone's performance. Please explain these results. Are they related to the backbone architecture, the dataset, or limitations in LDAE's generalization?
- The datasets used in the ablation studies and model analysis are not consistent. For example, Tables 12, 13, 14, and 15 use different datasets. This raises a concern that the analysis results may have been selectively presented on specific datasets. Therefore, it is unclear whether the conclusions would still hold if all analysis were conducted on the same datasets. This inconsistency limits the comparability and fairness of the experimental analysis.
- The paper's formatting needs further improvement, especially the tables in the Appendix. Some minor issues also need to be corrected. For example, Section 4.3.2 Ablation Study in the main text is empty.
Questions for authors
- In Table 7, the default values and search ranges of diff_scale, diff2_scale, and diff_dropout vary considerably across datasets. This raises concerns about the method's generalizability. It is unclear whether the reported gains depend on dataset-specific tuning or whether a shared default configuration can achieve similar results. Please explain how these values and search ranges are determined and provide sensitivity analysis for these parameters.
- In Table 8, the 3rd-Diff configuration also achieves competitive results. Could combining 1st-Diff, 2nd-Diff, and 3rd-Diff, or 2nd-Diff and 3rd-Diff, give better results? Please provide experimental results with these configurations.
- Tables 10, 11, and 12 show that removing either the 1st-Diff or 2nd-Diff branch often produces results close to those of the full LDAE and the differences are very small. After excluding the experimental run-to-run variations, does this suggest that the effects of 1st-Diff and 2nd-Diff are actually not that significant?? Does the 2nd-Diff branch contribute more than the 1st-Diff branch? Please clarify the contribution of each branch.
- In the efficiency analysis in Figures 14 and 15, why do memory cost and training time decrease on some datasets when LDAE is added to the backbones? Small cost increases would support LDAE's low overhead, but the large decreases need a detailed explanation, since LDAE adds 1st-Diff and 2nd-Diff operators without changing the backbone's internal structure. Please explain this obeservation and clarify the experimental configuration of efficiency analysis.
3 comments
Thank you for your responses. I also raised several concerns in the Critical Weaknesses section, but it is unclear whether the authors have taken them into account. Considering the overall experimental results and model analysis, I recommend weak rejection.
Q1. How should the small or mixed gains in Figures 8–10 be interpreted?
Figures 8–10 test sensitivity to the lookback window; Tables 22–24 provide the broader comparisons across backbones and datasets. Read together, they show where LDAE’s derivative cues add value and where the effect is smaller. The revised Limitations section discusses the dependence on the host model and the mixed Traffic results. The method’s usefulness is therefore assessed across forecasting settings, rather than inferred from any single curve.
Q2. Are the ablation conclusions comparable across datasets?
Tables 13–15 now evaluate SimpleTM, CASA, and DLinear on the same four datasets, ETTm1, ECL, Solar, and Weather, at all four horizons. Tables 9–10 use those same datasets for the derivative-order comparison. This common set lets readers compare design choices across both datasets and model families, including cases where their effects differ.
Q3. Where is the main-text analysis of these design choices?
The Ablation Study section presents the component results in Table 5 and discusses the effect of injection position using Table 4. Tables 12–15 and 18 provide the detailed branch, component, and horizon-specific results in the appendix. The revised presentation links each main-text conclusion to its underlying comparison.
We hope these directly comparable results and the fuller discussion address the concerns raised in the Critical Weaknesses section. We would appreciate your reconsideration of the revised manuscript.
Q1. How are the defaults and search ranges determined?
The main experiments use one LDAE configuration across all datasets:
diff_scale = 0.10,diff2_scale = 0.03, anddiff_dropout = 0.01. Each baseline/LDAE pair shares the optimizer, loss, data split, normalization, learning-rate schedule, and training budget. The test split is never used for parameter selection.The scales initialize the learnable branch contributions;
diff_dropoutregularizes derivative features. Table 7’s varying values are used only for ablation and sensitivity analysis, not the main comparisons. Tables 10–12 and Appendix F.2 show dataset- and backbone-dependent contributions. We will clarify this in Table 7.Q2. Could combining 1st-, 2nd-, and 3rd-Diff improve performance?
Under the Table 8 protocol, we tested additional combinations with SimpleTM on ETTm1, ECL, Solar, and Weather at H∈{96,192,336,720}. Values average the four horizons.
The standard 1st+2nd configuration has the lowest dataset averages. Neither alternative improves MSE or MAE in any of the 16 dataset–horizon pairs. Higher-order differences may amplify high-frequency noise, supporting first- and second-order cues as the default.
Q3. Are both branches significant, and does the second-order branch contribute more?
Tables 10–12 report matched ablations across SimpleTM, CASA, and DLinear. Below are SimpleTM’s four-horizon means ± pooled standard deviations over three matched iterations.
Removing either branch increases both errors on all three datasets. The first-order branch has a slightly larger average effect on ECL; effects are comparable on ETTm1 and Solar. We therefore describe the branches as complementary, with no claim that the second-order branch is universally more important. The manuscript includes horizon-specific results.
Q4. Why do training time and memory sometimes decrease after adding LDAE?
The decreases in Figures 14 and 15 reflect measurement variation from GPU scheduling, data loading, and memory allocation or reuse. Baseline/LDAE pairs use matched settings; lower measured runtime does not imply fewer optimization steps.
For branch i, grouped convolution and value projection add O(CLKi) and O(CLD) operations per sample, respectively, where D is the token dimension. Across both branches, the added cost is O(CL(K1+K2)+2CLD), excluding normalization and dropout. Shared projection weights still require additional operations and activations.
Summary. The revision clarifies the common defaults, adds derivative-order and branch ablations, and separates computational complexity from measured efficiency.