acceptodds
Under review as a conference paper at ICLR 2027

Learnable Difference-Aware Embedding: A Lightweight Plug-and-Play Module for Multivariate Time Series Forecasting

Abstract

Multivariate time series forecasting supports decision-making in energy, transportation, and other dynamic systems, where models must capture both global dependencies and local temporal changes. However, standard embeddings in representative forecasting backbones do not explicitly encode first- and second-order temporal differences at the initial representation stage. We introduce Learnable Difference-Aware Embedding (LDAE), a lightweight plug-and-play module that constructs channel-wise rate-of-change and curvature features using learnable convolutions initialized with finite-difference stencils. LDAE fuses these derivative cues with the backbone representation at the initial embedding interface, while leaving the backbone's internal temporal modeling unchanged. Across 12 real-world datasets, LDAE improves forecasting accuracy in low-to-moderate-noise, non-stationary settings and transfers across Transformer-, CNN-, and MLP-based backbones. These results demonstrate that explicit derivative cues provide a lightweight and transferable inductive bias for multivariate time series forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series forecasting, time series data, deep learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bq5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Learnable Difference-Aware Embedding (LDAE), a lightweight plug-and-play module that constructs channel-wise rate-of-change and curvature features using learnable convolutions initialized with finite-difference stencils. In multivariate time series forecasting, existing embeddings in representative forecasting backbones do not explicitly encode first- and second-order temporal differences at the initial representation stage. LDAE fuses these derivative cues with the backbone representation at the initial embedding inteface and improves forecasting accuracy across several backbones.

Critical strengths
  1. In MTSF, accurate forecasting requires models to capture not only long-range dependencies and cross-variable interactions, but also non-stationary local dynamics such as channel-specific spikes, asymmetric rate changes, and sharp curvature variations. The authors use first- and second-order differences to capture these features and inject them directly at the embedding stage. This design makes the module easy to apply and transfer across different backbones, without adding more computational cost or design complexity.
  2. The authors conduct extensive experiments to show the improvements from adding LDAE to different backbones, such as the results in Tables 19 and 20 of the Appendix. They also visualize the trained FD-initialized kernels and analyze training time and memory usage. These experiments make the paper more complete and provide a broader evaluation of the method.
Critical weaknesses
  1. In Figures 8, 9, and 10, adding LDAE to SimpleTM and iTransformer gives only small and inconsistent improvements. On some datasets, such as ETTh2 and Traffic, LDAE even reduce the backbone's performance. Please explain these results. Are they related to the backbone architecture, the dataset, or limitations in LDAE's generalization?
  2. The datasets used in the ablation studies and model analysis are not consistent. For example, Tables 12, 13, 14, and 15 use different datasets. This raises a concern that the analysis results may have been selectively presented on specific datasets. Therefore, it is unclear whether the conclusions would still hold if all analysis were conducted on the same datasets. This inconsistency limits the comparability and fairness of the experimental analysis.
  3. The paper's formatting needs further improvement, especially the tables in the Appendix. Some minor issues also need to be corrected. For example, Section 4.3.2 Ablation Study in the main text is empty.
Questions for authors
  1. In Table 7, the default values and search ranges of diff_scale, diff2_scale, and diff_dropout vary considerably across datasets. This raises concerns about the method's generalizability. It is unclear whether the reported gains depend on dataset-specific tuning or whether a shared default configuration can achieve similar results. Please explain how these values and search ranges are determined and provide sensitivity analysis for these parameters.
  2. In Table 8, the 3rd-Diff configuration also achieves competitive results. Could combining 1st-Diff, 2nd-Diff, and 3rd-Diff, or 2nd-Diff and 3rd-Diff, give better results? Please provide experimental results with these configurations.
  3. Tables 10, 11, and 12 show that removing either the 1st-Diff or 2nd-Diff branch often produces results close to those of the full LDAE and the differences are very small. After excluding the experimental run-to-run variations, does this suggest that the effects of 1st-Diff and 2nd-Diff are actually not that significant?? Does the 2nd-Diff branch contribute more than the 1st-Diff branch? Please clarify the contribution of each branch.
  4. In the efficiency analysis in Figures 14 and 15, why do memory cost and training time decrease on some datasets when LDAE is added to the backbones? Small cost increases would support LDAE's low overhead, but the large decreases need a detailed explanation, since LDAE adds 1st-Diff and 2nd-Diff operators without changing the backbone's internal structure. Please explain this obeservation and clarify the experimental configuration of efficiency analysis.
3 comments

Q1. How are the defaults and search ranges determined?

The main experiments use one LDAE configuration across all datasets: diff_scale = 0.10, diff2_scale = 0.03, and diff_dropout = 0.01. Each baseline/LDAE pair shares the optimizer, loss, data split, normalization, learning-rate schedule, and training budget. The test split is never used for parameter selection.

The scales initialize the learnable branch contributions; diff_dropout regularizes derivative features. Table 7’s varying values are used only for ablation and sensitivity analysis, not the main comparisons. Tables 10–12 and Appendix F.2 show dataset- and backbone-dependent contributions. We will clarify this in Table 7.

Q2. Could combining 1st-, 2nd-, and 3rd-Diff improve performance?

Under the Table 8 protocol, we tested additional combinations with SimpleTM on ETTm1, ECL, Solar, and Weather at H∈{96,192,336,720}H\in\{96,192,336,720\}. Values average the four horizons.

Dataset1st+2nd MSE1st+2nd MAE2nd+3rd MSE2nd+3rd MAE1st+2nd+3rd MSE1st+2nd+3rd MAE
ETTm10.3800.3940.3910.3990.3890.399
ECL0.1670.2610.1820.2720.1810.271
Solar0.1860.2460.2080.2740.2110.275
Weather0.2400.2680.2470.2730.2460.271
Overall0.2430.2920.2570.3050.2570.304

The standard 1st+2nd configuration has the lowest dataset averages. Neither alternative improves MSE or MAE in any of the 16 dataset–horizon pairs. Higher-order differences may amplify high-frequency noise, supporting first- and second-order cues as the default.

Q3. Are both branches significant, and does the second-order branch contribute more?

Tables 10–12 report matched ablations across SimpleTM, CASA, and DLinear. Below are SimpleTM’s four-horizon means ± pooled standard deviations over three matched iterations.

DatasetFull LDAE MSE / MAEw/o 1st-Diff MSE / MAEw/o 2nd-Diff MSE / MAE
ECL0.166 ± 0.003 / 0.261 ± 0.0020.184 ± 0.001 / 0.273 ± 0.0010.182 ± 0.001 / 0.272 ± 0.001
ETTm10.382 ± 0.003 / 0.396 ± 0.0030.392 ± 0.003 / 0.400 ± 0.0020.391 ± 0.006 / 0.400 ± 0.004
Solar0.189 ± 0.006 / 0.250 ± 0.0060.209 ± 0.007 / 0.275 ± 0.0080.209 ± 0.005 / 0.273 ± 0.006

Removing either branch increases both errors on all three datasets. The first-order branch has a slightly larger average effect on ECL; effects are comparable on ETTm1 and Solar. We therefore describe the branches as complementary, with no claim that the second-order branch is universally more important. The manuscript includes horizon-specific results.

Q4. Why do training time and memory sometimes decrease after adding LDAE?

The decreases in Figures 14 and 15 reflect measurement variation from GPU scheduling, data loading, and memory allocation or reuse. Baseline/LDAE pairs use matched settings; lower measured runtime does not imply fewer optimization steps.

For branch ii, grouped convolution and value projection add O(CLKi)O(CLK_i) and O(CLD)O(CLD) operations per sample, respectively, where DD is the token dimension. Across both branches, the added cost is O ⁣(CL(K1+K2)+2CLD)O\!\left(CL(K_1+K_2)+2CLD\right), excluding normalization and dropout. Shared projection weights still require additional operations and activations.

Summary. The revision clarifies the common defaults, adds derivative-order and branch ablations, and separates computational complexity from measured efficiency.

Thank you for your responses. I also raised several concerns in the Critical Weaknesses section, but it is unclear whether the authors have taken them into account. Considering the overall experimental results and model analysis, I recommend weak rejection.

Q1. How should the small or mixed gains in Figures 8–10 be interpreted?

Figures 8–10 test sensitivity to the lookback window; Tables 22–24 provide the broader comparisons across backbones and datasets. Read together, they show where LDAE’s derivative cues add value and where the effect is smaller. The revised Limitations section discusses the dependence on the host model and the mixed Traffic results. The method’s usefulness is therefore assessed across forecasting settings, rather than inferred from any single curve.

Q2. Are the ablation conclusions comparable across datasets?

Tables 13–15 now evaluate SimpleTM, CASA, and DLinear on the same four datasets, ETTm1, ECL, Solar, and Weather, at all four horizons. Tables 9–10 use those same datasets for the derivative-order comparison. This common set lets readers compare design choices across both datasets and model families, including cases where their effects differ.

Q3. Where is the main-text analysis of these design choices?

The Ablation Study section presents the component results in Table 5 and discusses the effect of injection position using Table 4. Tables 12–15 and 18 provide the detailed branch, component, and horizon-specific results in the appendix. The revised presentation links each main-text conclusion to its underlying comparison.

We hope these directly comparable results and the fuller discussion address the concerns raised in the Critical Weaknesses section. We would appreciate your reconsideration of the revised manuscript.

Reviewer_Hoh5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Learnable Difference-Aware Embedding (LDAE), a lightweight module for multivariate time series forecasting. It uses channel-wise convolutions initialized with first- and second-order finite-difference stencils, then projects, normalizes, and scales their outputs before adding them to the backbone representation. Experiments cover 12 datasets and several Transformer, CNN, and MLP backbones. The paper also provides component ablations, injection-position comparisons, repeated-run statistics, and theoretical discussions of finite-difference and FIR approximation properties. Improvements depend on the backbone and dataset, with some degradation on Traffic.

Critical strengths
  1. Practical compatibility across forecasting architectures. The module provides a relatively simple way to introduce local temporal priors without changing the backbone’s internal blocks. Its evaluation across several architecture families makes the observed benefits potentially useful beyond a single model.
  2. Useful controlled comparisons and applicability analysis. The fixed-kernel, random-initialization, injection-position, and Times2D replacement experiments help assess the design choices. Reporting negative results on Traffic and limited gains when derivative features are already present also helps define the method’s practical scope.
Critical weaknesses
  1. The motivation and theoretical analysis do not sufficiently establish the proposed mechanism. The absence of explicit differences in an embedding does not by itself establish a limitation: these features are deterministic functions of the input and can be represented by ordinary linear projections or convolutions. Appendix C establishes classical approximation properties, but does not explain when the proposed injection should improve forecasting. Moreover, the learned kernels are unconstrained, so their first- and second-order derivative interpretations need not survive training. Given that Times2D already introduces both derivative orders, the conceptual advance appears limited without stronger evidence connecting the specific design to its claimed benefits.
  2. The evidence for robust improvements in the main configuration remains incomplete. Many average MSE improvements over VPNet in Table 1 are around 0–3%, and the VPNet comparison is absent from the significance tables in Appendix F.3. Several reported long-term improvements are not significant, including SimpleTM and CASA on ECL. There are also substantial differences between the main and repeated-run results: for iTransformer on PEMS03, Table 3 reports 0.152 → 0.132, whereas Table 24 reports 0.3256 → 0.2869 with high variability. Until these reporting differences are explained, it is difficult to assess the robustness of the headline results.
Questions for authors
  1. Mechanism clarification: Are any moment constraints imposed on the kernels during training? If not, can you quantify whether the trained kernels retain the relevant derivative properties, rather than relying on qualitative heatmaps?
  2. Experimental clarification and evidence: What explains the differences between Tables 3 and 24? Please clarify whether the main tables report single runs or means, whether the configurations match, and provide the per-seed, per-horizon VPNet results needed to assess the main improvements.
  3. Efficiency clarification: How should the statement that overhead is typically within ±5% be reconciled with Figure 5, where DLinear training time increases from 102.83 to 157.50 seconds and iTransformer memory increases from 219.83 to 475.93 MB? Does the complexity analysis include the additional value projections as well as the grouped convolutions?
2 comments

Q1. Are any moment constraints imposed on the kernels during training?

No. LDAE uses no moment regularization, projection, clipping, or reparameterization. The kernels are initialized as [−0.5,0,0.5][-0.5,0,0.5] and [1,−2,1][1,-2,1] and remain fully learnable. Thus, “derivative” denotes a structured initialization prior, not an invariant property after training.

We quantify retained structure using centered discrete moments:

μq(w)=∑kwkkq,\mu_q(w) = \sum_k w_k k^q,

where kk denotes the centered kernel offset. The initialization targets are (μ0,μ1)=(0,1)(\mu_0,\mu_1)=(0,1) for the first-order branch and (μ0,μ1,μ2)=(0,0,2)(\mu_0,\mu_1,\mu_2)=(0,0,2) for the second-order branch.

We audited 24 trained checkpoint–branch records from ECL, ETTm1, and Weather at H∈{96,192,336,720}H\in\{96,192,336,720\}.

BranchTarget dominant momentObserved rangeRaw moment errorNormalized shape error
1st-Diffμ1=1\mu_1=10.9650.965–1.0371.0370.1220.122–0.3500.3500.0700.070–0.1910.191
2nd-Diffμ2=2\mu_2=21.8071.807–2.3752.3750.1460.146–0.3730.3730.0900.090–0.2170.217

All records retained a valid dominant-moment scale. The kernels preserve first- or second-order dominant patterns while adapting other moments and detailed shapes. We therefore interpret finite-difference initialization as a persistent derivative-oriented inductive bias, not a hard constraint or an exact derivative guarantee.

Q2. What explains the differences between Tables 3 and 24?

Both tables report averages over three iterations (itr=3) under the same fixed-seed protocol.

Configurations were matched with seq_len=96; otherwise, VPNet uses its original hyperparameters. Table 18 reports horizon-specific results.

The discrepancy arose from an anomalous H=96H=96 result in an earlier multi-GPU iTransformer run on PEMS03:

HHiTransformer MSEiTransformer MAEiTransformer+LDAE MSEiTransformer+LDAE MAE
120.0690.1750.0670.173
240.0980.2090.0950.206
480.1660.2770.1410.255
960.5930.5820.5610.564

After independently rerunning the same configuration on one GPU, the H=96H=96 MSE/MAE changed from 0.593/0.5820.593/0.582 to 0.274/0.3700.274/0.370 for iTransformer and from 0.561/0.5640.561/0.564 to 0.228/0.3320.228/0.332 for iTransformer+LDAE. The corrected four-horizon averages are 0.152→0.1320.152\rightarrow0.132 for MSE and 0.258→0.2410.258\rightarrow0.241 for MAE, consistent with Table 21.

Table 24 retained the earlier version, yielding 0.3256→0.28690.3256\rightarrow0.2869 in MSE. This was a synchronization issue, not a configuration difference. We corrected the manuscript, excluded the anomalous run from the stability analysis, and provide per-iteration records in the supplement.

Q3. How should the overhead statement be reconciled with Figure 5?

We agree that “typically within ±5%\pm5\%” was too broad and unsupported by all Figure 5 measurements. We removed this statement. On ECL, LDAE increases DLinear training time from 102.83102.83 s to 157.50157.50 s (+53.2%+53.2\%) and iTransformer peak memory from 219.83219.83 MB to 475.93475.93 MB (+116.5%+116.5\%).

These are empirical measurements, not complexity guarantees. Lower measured time or memory does not indicate reduced computation because both depend on GPU scheduling, data loading, memory reuse, and allocator behavior. All comparisons use matched data splits, batch sizes, lookback lengths, training budgets, and hardware. Appendix Figures 14 and 15 extend the evidence to seven datasets and four backbones, showing that overhead is configuration-dependent rather than uniformly bounded.

The complexity analysis includes grouped convolutions and value projections. For derivative branch ii, the additional cost is

O(CLKi)+O(CLD),O(CLK_i)+O(CLD),

where CC, LL, KiK_i, and DD denote the number of channels, lookback length, branch kernel length, and token dimension, respectively. With both branches, the added cost is

O ⁣(CL(K1+K2)+2CLD),O\!\left(CL(K_1+K_2)+2CLD\right),

excluding normalization and dropout. Although projection weights are shared with the base embedding, each branch still requires an additional projection and intermediate activations.

We revised the main efficiency discussion, Appendix analysis, and Figures 14–15 captions to distinguish asymptotic complexity from measured runtime and peak memory. LDAE scales linearly in CC and LL, but constant-factor overhead can be substantial for lightweight models or high-dimensional inputs. We therefore no longer claim a universal overhead bound.

Summary. We clarified the unconstrained initialization, synchronized the corrected PEMS03 results, and distinguished asymptotic complexity from measured runtime and memory. See the corresponding Appendix analyses, tables, and figures.

Thank you for the clarifications. The rebuttal addresses my concerns regarding reporting inconsistencies and computational overhead. However, my main concerns about the methodological novelty and the statistical robustness of the improvements remain insufficiently addressed. Therefore, I will maintain my original rating.

Reviewer_NLWCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the issue that standard embeddings in representative forecasting backbones do not explicitly encode first- and second-order temporal differences at the initial representation stage, this paper proposed LDAE, a lightweight plug-and-play module that constructs channel-wise rate-of-change and curvature features using learnable convolutions initialized with first- and second-order finite-difference stencils. After that, an EDNBlock projects, regularizes, and adaptively scales these first- and second-order features. Then, learnable scales αi lets these scaled features fused with the base embedding for further forecasting.

Critical strengths
  1. LDAE operates at the embedding stage, it augments the embedding outputs through EDNBlock-refined additive fusion. It is an early fusion approach which has been proved an effective ways for feature augumentation.

  2. This paper provided enough experiments and discussions covering method validation, generality, efficiency, ablation, and stablility.

  3. The authors listed the details of datasets and models in the appendix partition, which is primary for re-production of the proposed methods.

Critical weaknesses
  1. One concern is the significance of performance improvement in Table 1 and 2. As shown in Table 1 and 2, most increases were in the third position after the dot, which means small benefit. The author should provide depth discussion about such observations to further clarify the efficiency of the proposed method.

  2. Following the first concern, the author should try other augmentation method or evaluation metric to validate the influence of first- and second-order feature.

Questions for authors
  1. Is there other augmentation perspective on embedding features?

  2. Following the first concern in weaknesses, is there other evaluation metric to show the improvement more significantly?

1 comment

Q1. Is there another augmentation perspective on embedding features?

Yes. We view embedding augmentation along three complementary axes:

  1. Temporal prior: LDAE adds explicit rate-of-change and curvature cues to the initial representation.
  2. Channel interaction: The derivative branch may either preserve channel locality or mix information across channels. Tables 10–12 show that replacing grouped channel-wise convolution with channel-mixing convolution generally degrades performance, with DLinear on Solar being an informative exception.
  3. Adaptivity: The derivative prior may be fixed or adaptive. Tables 10–12 compare finite-difference initialization, random initialization, fixed kernels, and learnable scaling.

We also examine the injection position. Table 4 and Appendix D.3 compare embedding-stage and after-mixing injection under matched settings. Appendix B and Table 8 investigate the effect of derivative order. In Appendix D.1 and Table 13, we replace only the handcrafted FSDH branch of Times2D with LDAE while retaining the remainder of the pipeline. This controlled replacement improves MSE in 11 of the 12 dataset–horizon pairs and MAE in 11 of the 12 pairs.

Thus, LDAE represents one specific embedding-augmentation design: channel-local, finite-difference-initialized, learnable temporal augmentation at the embedding interface. We do not claim that derivative cues are the only useful form of embedding augmentation. Instead, our controlled comparisons isolate the respective roles of the temporal prior, channel locality, injection position, and adaptive fusion.

Q2. Is there another evaluation metric that can show the improvement more clearly?

Yes. In addition to MSE and MAE, we report derivative-aware MAE metrics that directly evaluate the local temporal quantities targeted by LDAE:

\frac{1}{(H-r)C} \sum{t=1}^{H-r} \left| \Delta^r\hat{\mathbf{y}}_t - \Delta^r\mathbf{y}_t \right|_1, \qquad r\in{1,2}.$$ Here, dMAE-1 measures the rate-of-change error, whereas dMAE-2 measures the curvature error. Lower values indicate better performance. Both metrics are computed from the same predictions used to calculate MSE and MAE and therefore require no additional training or hyperparameter tuning. The results below complement Figure 3 and Appendix E.1 (Figures 8–10). Each entry reports the mean over three recorded runs for the corresponding prediction horizon. Lower values are better. The lower value within each Base–LDAE comparison is shown in bold, with ties bolded for both variants. | Dataset | $H$ | dMAE-1 (Base → LDAE) | dMAE-2 (Base → LDAE) | |:--------|------:|----------------------:|----------------------:| | ECL | 96 | 0.166 → **0.163** | 0.259 → **0.255** | | ECL | 192 | 0.169 → **0.167** | 0.265 → **0.261** | | ECL | 336 | 0.174 → **0.172** | 0.272 → **0.269** | | ECL | 720 | **0.184** → 0.185 | **0.288** → 0.291 | | ETTm1 | 96 | 0.137 → **0.135** | 0.216 → **0.214** | | ETTm1 | 192 | 0.136 → **0.134** | **0.213** → 0.215 | | ETTm1 | 336 | 0.137 → **0.136** | **0.215 → 0.215** | | ETTm1 | 720 | **0.138** → 0.139 | **0.216** → 0.217 | These derivative-aware metrics complement, rather than replace, MSE and MAE. On ECL, both derivative errors decrease at $H\in\{96,192,336\}$, but increase slightly at $H=720$. On ETTm1, dMAE-1 decreases at $H\in\{96,192,336\}$, whereas dMAE-2 decreases at $H=96$, increases at $H=192$, remains unchanged after rounding at $H=336$, and increases at $H=720$. We therefore present dMAE as mechanism-aligned exploratory evidence rather than as evidence of a universal or statistically significant advantage. **Summary.** We have updated the revised manuscript with a structured discussion of embedding augmentation and derivative-aware evaluation metrics. Please see the corresponding analyses and tables in the Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.