acceptodds
Under review as a conference paper at ICLR 2027

Selective Associative Memory: Rethinking the Representation Learner in Continuous-Time Time Series Forecasting

Abstract

We identify selective associative memory as the unifying principle behind gated RNNs, attention, gated CNNs/MLPs, selective SSMs, and MoE architectures. These models differ not in their fundamental operation, but in four distinct, largely independent design dimensions: operating space, branch count, gating function type, and fusion method. Based on this framework, we derive the minimal gating requirements for time series forecasting: feature and time dimension gating only. We instantiate this minimal design in Alterum, a lightweight continuous-time model with about 10K trainable parameters, combining an exponential-scale Fourier Feature Embedding with a Temporal Decomposition Gated Network and a closed-form ridge regression head. Across multivariate, univariate, and few-shot forecasting, Alterum attains average MSE of 0.276, 0.094, and 0.330, respectively, and remains competitive under 15% random input masking. These results indicate that a framework-driven minimal design can substantially reduce parameter count and improve inference efficiency while maintaining competitive accuracy. Code is available at https://anonymous.4open.science/r/ALTERUM-E23B/.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, continuous-time models, lightweight models, meta-learning, selective associative memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6xLNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed Alterum, which combines random Fourier features, a Temporal Decomposition Gated Network, and a differentiable closed-form ridge regression head. Empirical studies suggest that the model achieve lower MSE over eight standard benchmark.

Critical strengths
  1. Reducing forecasting cost is a worthwhile objective. Alterum’s trainable parameter count is independent of lookback length, and its reported configurations are substantially smaller than many of the included baselines.

  2. The reported forecasting accuracy is competitive across several settings. The experiments are solid and meaningful.

  3. The inner-outer loop optimization makes the behavior of the model clean and clear.

Critical weaknesses
  1. DeepTime explicitly introduces CFF to avoid the hyperparameter sweep associated with a single-scale random Fourier embedding. Its Appendix H lists a fixed collection of eight scales, while Appendix K compares that CFF construction against a separate single-scale sweep. This raise questions about the novelty on this paper.

  2. The main accuracy comparison confounds architecture with access to longer history and different tuning protocols. For example, at H=720, Alterum can search histories from 720 to 6,480 steps (x1, x3, x5, x7, x9), while method like xPatch’s standard lookback is 96.

  3. Can the authors reconcile the equations, configuration tables, and implementation details? For example, Equation 1 and Appendix K.3 assign exactly d/(2N) frequencies to each scale, but Table 25 reports d=8,N=16 for Exchange, which requires one quarter of a frequency per scale. In addtion, in line 1606~1607, the r definition is different from Equation 16.

Questions for authors

See the weakness section.

1 comment

We thank the reviewer. Two documentation problems and one fairness issue. Naming: the revision writes μ for the lookback multiplier (L=μH), reserving τ for the time coordinate.

W1 FFE and CFF

We claim neither random nor multi-scale Fourier features as new, and CFF already spares a per-run single-scale sweep. The difference is what is parameterised. DeepTime lists (0.01,0.1,1,5,10,20,50,100) and uses it unchanged across datasets, having swept single scales; CFF thus answers "one scale does not fit all" with a constant that never says how many scales a new dataset needs. FFE replaces the constant with a rule --- exponential spacing over [ω_min,ω_max], one hyperparameter N under the budget N≤d/2, realised as N_eff=min(d/2,N) scales --- so "which scales" becomes "how many".

N now has a derivation: scales above the L+H grid's Nyquist limit alias and carry no resolvable information, so cutting there gives N=8. Every cell is in Appendix P, Table 30 (168 runs): 17 of 28 differ by at most 1% (median +0.5%); 15 are worse under truncation, 8 better; six degrade by over 2%, largest ETTh1 at H=720 (+18.5%), ECL at 192 (+3.4%), Solar at 720 (+2.8%). Pooled +1.8%, +0.5% without ETTh1. A local cost, not a global equivalence: N=8 suffices when no per-cell optimum is required (within ±1% on 61% of cells at ~1.8%), otherwise search N; N=8 is released in the search space.

The contribution is the scale rule and TDGN's decomposition gating, not multi-scale use. At a matched 10,329-parameter budget, same backbone (Table 19), FFE + TDGN is lowest, alone or tied, in 30 of 32 cells and never beaten; CFF draws level in the rest (median 1.4% worse elsewhere), the sinusoidal encoding worse everywhere (53% median). FFE also concentrates less than CFF (r_90% 8.9 vs 5.8) and still forecasts better, so this is not a concentration effect.

W2 Lookback and tuning protocol

  • μ∈{1,3,5,7,9} is DeepTime's protocol (its table lists that grid; selection by validation loss; its Section 3 gives the loop and the ridge regressor), run on DeepTime itself, now attributed.
  • μ is chosen per dataset and horizon on validation (Appendix A), never on test; in 5 of 32 cells it selects μ=9, where longer windows are untested.
  • Not equalised: the other eight baselines run at released lookbacks; we do not claim they would not improve under a search.
  • One matched case: Exchange keeps μ=1, so all models read L=96 and the margin cannot come from history --- we beat the linear controls at every horizon by 104%/29% (H=96), 4.4%/8.3%, 13%/32%, 48%/79% (720). Those controls got our own μ search and still lose in 24 of 32 cells (means 0.2758 vs 0.3153, 0.3301; cells in Tables B1--B2 of our rfHh reply).
  • Cost at the configurations behind the numbers (Table 5, Appendix J): 8,781 parameters, 35.38 s/epoch, 0.070 GiB; DLinear 0.485M/48.23 s/0.104 GiB, DeepTime 1.315M/80.09 s/2.366 GiB. At μ=9 (L=6480, ETTm2) 27.5 s and 0.0437 GiB at 1,589 parameters, below that dataset's fastest baseline (48.7 s).
  • No fixed-L table: for a discrete model L is input width (DeepTime's Table 11: a fully connected baseline 0.93M→14.1M from L=48 to 5760; the time-index model stays 1.31M), so one fixed window holds nothing equal.

W3 Reconciliation

Both are real documentation errors, and neither changes a reported number.

(a) Eq. 1 presupposes d≥2N (one frequency per scale), never stated in the submission. The implementation halves the column budget for the sine/cosine pair, divides it over N_eff=min(d/2,N) and gives ⌊d/(2N_eff)⌋ frequencies per scale; every other dataset satisfies d=2Nm. Exchange is the exception: with d=8 the realised count is 4, the requested 16 scales collapse to 4 with one column each, and the "quarter frequency per scale" is the trace of a configuration violating d≥2N. The printed N=16 was also wrong; the realised value is 4. Eq. 1 and the appendix now carry N≤d/2, m=⌊d/(2N)⌋, and the Exchange row prints d=8, N=4; the configuration is untouched.

(b) One sentence wrote tr((Σ̂+λI)⁻¹), another tr(Σ̂(Σ̂+λI)⁻¹); they differ by Σ̂ and you are right that they disagree. Tables report the second, the only form bounded by D=2h; reported r never exceeds D, while the first diverges as λ*→0 and would exceed D exactly where r approaches D, so the entries identify the form used. The definition sentence was wrong, not the numbers; the revision rewrites it to the second form, with no table value changing. r_90%/r_95% come from the energy spectrum, independent of λ*.

Closing. The configuration and every table value stay as they are.

Reviewer_rfHhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes selective associative memory (a "memory branch ⊙ selection branch" motif) as a unifying lens for gated RNNs, attention, gated CNNs/MLPs, selective SSMs, and MoE. It characterizes these architectures along four design dimensions: operating space, branch count, gating function type, and fusion method. From this lens, the authors argue that time series forecasting requires only feature- and time-dimension gating, and they instantiate this design as Alterum. Alterum is a time-index model in the style of DeepTime. It embeds normalized time coordinates with a multi-scale Fourier Feature Embedding (FFE), transforms them with a small gated network with trend–seasonal decomposition (TDGN), and predicts with a closed-form ridge regression head fitted on the lookback window. With about 10K parameters, Alterum reports the best average MSE in the multivariate (0.276), univariate (0.094), and few-shot (0.330) settings. The paper also includes a masking-robustness experiment, efficiency comparisons, and a spectral-coverage analysis of FFE.

Critical strengths
  1. Strong efficiency. Alterum uses about 10K trainable parameters, very little GPU memory, and fast training (Tables 6 and 15), while remaining competitive on standard long-term forecasting benchmarks.

  2. FFE is a practical improvement over DeepTime's CFF. Exponentially spaced scales remove DeepTime's per-dataset grid search over scale sets, with similar or slightly better accuracy at a matched parameter budget (Table 5: 0.276 vs. 0.288 for CFF + TDGN).

  3. Reasonably complete reporting. The paper provides per-horizon results with standard deviations over three runs, per-configuration parameter counts (Appendix L), and released code. It also openly acknowledges that the logarithmic error bound rests on an unproven assumption, (A4).

Critical weaknesses
  1. For any fixed configuration, Alterum reduces to an input-independent linear forecaster, so the "selection" never sees the data. The representation learner receives only the relative coordinates τt+i=(i+L)/(L+H−1)\tau_{t+i} = (i+L)/(L+H-1) (Section 3.1), which are identical for every window. Hence Z=hϕ(τ)Z = h_\phi(\tau) is a single fixed matrix per (L,H)(L, H), and the ridge head yields y^H=Z~H(Z~L⊤Z~L+λI)−1Z~L⊤X=AϕX\hat{y}_H = \tilde{Z}_H(\tilde{Z}_L^\top \tilde{Z}_L + \lambda I)^{-1}\tilde{Z}_L^\top X = A_\phi X, where Aϕ∈RH×LA_\phi \in \mathbb{R}^{H\times L} is shared across all windows and channels and has rank at most D+1D+1 (as low as 9 for Weather at H∈{336,720}H \in \{336, 720\}). FFE + TDGN is therefore a 10K-parameter parameterization of a channel-independent, low-rank linear map. Three consequences follow:

    • The gate is not selective. The SiLU gate is a fixed function of time and is identical for every test window. This is not selection in the sense of attention or Mamba (Table 1 itself lists "input-dependent SSM parameters"). The sparsity analysis in Figure 4 describes a static array rather than input-dependent retrieval.
    • Key baselines are missing. The natural baselines are linear forecasters, such as a single linear layer, NLinear/RLinear, or a low-rank H×LH \times L map, trained with the same lookback search. None of these is reported.
    • The gating ablation is hard to interpret. The near-catastrophic "w/o gating" result (ECL MSE 0.587, vs. 0.166 for DLinear) does not show that selection is essential. Removing the gate only changes how the same kind of linear operator is parameterized, so this result looks more like an optimization failure.
  2. The headline comparison is confounded by lookback length, and the margins are small where the comparison is closer to controlled.

    • Lookback mismatch. Alterum searches L=τHL = \tau H with τ∈{1,3,5,7,9}\tau \in \{1,3,5,7,9\} for each dataset and horizon (up to L=6480L = 6480), and the selection criterion is not stated. Baselines use their original lookbacks, with numbers copied from the original papers. The iTransformer and TimeMixer rows appear to match their published L=96L = 96 results, and DLinear matches its L=336L = 336 results. Long lookbacks are well known to benefit linear forecasters, and Alterum's advantage is largest at H=720H = 720 (Figure 3), where the lookback gap is largest.
    • Small margins against linear-type baselines. Against the two lookback-flexible, linear-type baselines, the margin is only 0.001–0.005 MSE on five of eight datasets: ECL 0.163 vs. 0.164 (DeepTime), ETTm1 0.350 vs. 0.354, Solar 0.202 vs. 0.204, Weather 0.230 vs. 0.231, and ETTh1 0.418 vs. 0.423 (DLinear).
    • Gains are concentrated in a few entries. About 84% of the average margin over DeepTime comes from ETTh2 (an author-run DeepTime result) and Exchange, and more than half of the margin over xPatch comes from Exchange alone. On Exchange, the authors' own Raw MLP control is better than Alterum at H=336H = 336 and H=720H = 720 (0.268 vs. 0.269 and 0.460 vs. 0.537; Table 25).
    • Efficiency is measured at a different configuration. Table 6 measures efficiency at L=96L = 96 for all models, not at the configurations that produce Alterum's reported accuracy.
  3. The unifying framework and the "derived" minimal design are not substantiated.

    • The unification is not falsifiable. Eq. (17), f(x)=Memory(x)⊙Selection(x)f(x) = \mathrm{Memory}(x) \odot \mathrm{Selection}(x), does not cover attention or MoE, which use a weighted sum and top-kk selection. This is why "fusion method" is introduced as a free dimension, after which the framework can describe almost any two-branch model. The claim that the four dimensions are "largely independent" (Appendix M.4) is asserted without evidence.
    • Closely related prior work is not discussed. Several associative-memory unifications of attention, linear attention, SSMs, and fast-weight programmers already exist: Hopfield networks (Ramsauer et al., 2021), fast weight programmers (Schlag et al., 2021), structured state space duality (Dao & Gu, 2024), test-time regression (Wang, Shi & Fox, JMLR 2026; three design choices), and Miras (Behrouz et al., ICLR 2026; four design choices). The novelty of Contribution 1 must be positioned against this work.
    • The "derivation" is an informal argument. Sequence-dimension selection is dismissed because forecasting "operates over a short lookback window", yet Alterum itself uses lookbacks of up to 9H=64809H = 6480. No experiment adds sequence-dimension or model-dimension selection at a matched budget to show that it is unnecessary. The TDGN ablation covers only two datasets (ETTh2 and ECL).
    • The paper describes its own model inconsistently. The abstract and introduction claim "feature and time dimension gating", but Appendix M.3 states that the selection branch operates on the feature dimension only. Time enters only through a fixed moving-average split in the memory branch, so TDGN is effectively a GLU-style block with a single feature-wise gate. The gate is also described as "Sigmoid gating (SiLU)", although SiLU is not bounded in [0,1][0,1]; yet Figure 4 shows values only within [0,1][0,1].
  4. The theory and the continuous-time motivation do not connect to the evaluated setting.

    • Aliasing on the discrete grid. Proposition 1 and Table 16 establish log-uniform coverage over [1,e31][1, e^{31}] (44 octaves) with σn=en\sigma_n = e^n. However, the features are only evaluated on L+HL+H grid points in [0,1][0,1]: at most 7,200, and only 192 for H=96H = 96, τ=1\tau = 1. Frequencies with ∣B∣≳L+H|B| \gtrsim L+H alias on such a grid. If σ0=1\sigma_0 = 1, as in Table 16, many configurations have most of their 16 or 32 scales above the grid's Nyquist limit. The statement that "every octave receives a comparable share of the feature budget" therefore says little about the features the model actually uses. The noise-like FFE (N=32N=32) heatmap in Figure 5c is consistent with this.
    • Approximation is not forecasting. The logarithmic bound rests entirely on assumption (A4), which the authors acknowledge is stronger than the standard RFF bound and remains unproven. In addition, within-window approximation error says nothing about extrapolation from the lookback to the horizon, which is what forecasting requires.
    • The continuous-time benefits are never tested. The introduction motivates the work by the sampling-rate dependence of discrete models, but the paper contains no irregular-sampling or resolution-transfer experiment. The masking experiment feeds zeros into the ridge fit as if they were observations, instead of excluding the masked points, which a time-index model could do naturally. It also compares masked Alterum against unmasked baselines, so it cannot establish robustness relative to other models.
    • The weak controls inflate the gaps. The large gaps for Pos. Enc. + TDGN and Raw MLP in Table 5 (+96.0% and +98.6%) are driven by configurations with MSE near or above 1 (e.g., ECL 1.517 and 0.830), which suggests failed training rather than a weak inductive bias. The only meaningful control, CFF + TDGN, differs by 4.3%.
  5. Presentation and figures (secondary).

    • Misleading radar charts. Figure 2 uses irregular, per-axis tick ranges (e.g., the ECL axis spans only 0.163–0.178), which visually magnifies MSE differences of 0.001–0.005.
    • Unreadable figures. Legends, tick labels, and axis text in Figures 2, 3, 5, and 14 are unreadable at print size. Figure 5 never defines "response intensity", and the legend in Figure 3 overlaps the data.
    • Unsupported or mismatched statements. Figure 3's conclusion that "scale amplifies an already sound inductive bias" does not follow from the plot. Its stated next-best value at H=720H = 720 (0.410) does not match Table 10, where the best baseline average is about 0.417 (DLinear).
    • Uninformative visualizations. Figures 6–13 show single windows at L=H=96L = H = 96, which is not the evaluated configuration, and add little.
    • Errors in cross-references and derivations. "Section 4.2.4" is cited for the controlled comparison, which is actually in Section 4.2.1. There are also algebra slips in Eq. (10) and in the expression for m1unifm_1^{\mathrm{unif}}.
Questions for authors
  1. Linear equivalence. Please confirm that hϕh_\phi receives only τ\tau and no observed values. If so, please report the equivalent operator AϕA_\phi, and compare Alterum against (a) a single linear layer and (b) a rank-(D+1)(D+1) linear map, both trained with the same τ\tau search. How do you reconcile an input-independent gate with the "selective" framing?

  2. Lookback-controlled comparison. How was τ\tau selected: on the validation set or on the test set? Please report either Alterum at fixed L∈{96,336}L \in \{96, 336\}, or the main baselines (DLinear, DeepTime, PatchTST, iTransformer) with the same τ\tau search, so that the effect of the architecture can be separated from that of the lookback.

  3. Testing the framework. What prediction does the framework make that could fail? For example, does adding sequence-dimension selection (attention over the lookback) or model-dimension selection (a small MoE) at a matched budget indeed fail to help, as the minimal-design argument implies? How does your framework differ from test-time regression (Wang, Shi & Fox) and Miras (Behrouz et al.)?

  4. FFE on a discrete grid. What value of σ0\sigma_0 is used in each configuration? What fraction of the FFE scales exceeds the Nyquist limit of the L+HL+H grid? Do the results change if the scales are truncated to the resolvable range?

  5. Masking protocol. Please report results with masked points excluded from the ridge fit rather than zero-filled, and apply the same masking to the baselines.

  6. Clarifications.

    • Why does Figure 4 show SiLU values only in [0,1][0,1]?
    • Which baseline corresponds to the "next-best 0.410" at H=720H = 720?
    • What tuning was used for the author-run DeepTime results? For example, its few-shot ETTh1 MSE is 1.160, compared with 0.425 in the multivariate setting, and several headline margins depend on these runs.
6 comments

Part 1/6: the operator

We thank the reviewer for a review that engages the construction; it changed what we claim. Naming: the revision writes μ (L=μH, μ∈{1,3,5,7,9}) for the multiplier, leaving τ for the time coordinate; where your review writes τ, read μ.

What changed. Claims narrowed (feature-axis gate, coordinate-conditioned modulation, "selective retrieval" withdrawn, sparsity figure removed); §4.4 and the limitations state the operator properties (per-variable coefficients over a shared basis, rank ≤ D+1, no variable mixing); we measured the operator's spectrum (Table B1) and ran the two linear comparisons (Table B2); the zero-filled experiment is replaced by position removal with a shared retained set (§5.1.4, Table 4); the lookback rule is stated where applied (Appendix A); W5's presentation errors are corrected. Both tables appear in this response, not the manuscript.

W1.1 The operator

Agreed. With D=2h, for fixed (L,H),

y^H=AϕX,Aϕ=Z~H(Z~L⊤Z~L+λI)−1Z~L⊤,rank⁡(Aϕ)≤min⁡(D+1,H,L),\hat y_H=A_\phi X,\qquad A_\phi=\tilde Z_H(\tilde Z_L^\top\tilde Z_L+\lambda I)^{-1}\tilde Z_L^\top,\qquad \operatorname{rank}(A_\phi)\le\min(D+1,H,L),

which is 9 on Weather at H∈{336,720}. Section 4.4 draws the corollary and Appendix B the derivation; the submission stated the head in prose without drawing it --- our omission.

The property is definitional for closed-form meta-learned time-index models, DeepTime included: φ holds the outer-loop meta-parameters, θ={W^(K)} the base parameters fitted per window, and the inner solve is Bertinetto et al.'s. DeepTime states the consequence itself --- "the adapted time-index model it yields is purely a function of time-index features" --- because the formulation exists to restrict the hypothesis class to ℋ_DeepTime={g_φ*(τ)ᵀ W^(K)}, which is what lets a window be extrapolated from. We agree with the identity; we do not accept the step to "the gate is not selective", since what separates such models is what φ pins the directions to, and baseline (b) measures exactly that. A linear readout is also not a linear function class: given φ, predictions lie in the span of the basis on the horizon --- affine with the raw coordinate, band-limited trigonometric polynomials with FFE, nonlinear mixtures with TDGN (§4.2, Appendix E).

The ridge solve returns W∈ℝ^D× C, b∈ℝ^1× C, so ŷ_H,c=A y_L,c+b_c: one H× L map shared by every variable and window, acting column-wise. Three properties follow, the third now in Section 6 --- one basis and one map shared across variables with coefficients fitted per variable; rank at most D+1, independent of L and H; no variable mixing. Alterum is a per-variable forecaster over a shared basis; cross-variable correlation is not exploited, and we expect that to matter on datasets such as Exchange.

For the framework itself we point to the source rather than restate it: DeepTime's Section 3 sets out the meta-optimisation loop and the closed-form ridge regressor, and its appendix gives the resulting model as a function of time-index features alone. Those are inherited constructions of that paper; ours is the basis generator h_φ and the operator reading of Section 4.4, which is why we measure A_φ rather than re-derive it.

Table B1. From the trained meta-parameters alone (one forward pass of h_φ at the L+H coordinates plus the learned λ, no retraining), H=720. Effective rank is |A_φ|_F²/|A_φ|₂²; r_90% counts the directions carrying 90% of the spectral energy.

DatasetLD+1eff. rankr₉₀%top-1
ECL7206516.6446.0%
ETTh1720172.1747.7%
ETTh2720174.61021.6%
ETTm15040334.01424.9%
ETTm2360091.2285.2%
Exchange96171.3279.2%
Solar2160331.91653.0%
Weather360091.5465.1%

Two readings. The bound is analytic and empirical: energy beyond the first D+1 directions is zero to machine precision (relative tail 2.9×10⁻⁹ to 2.0×10⁻⁸), round-off from a resolvent with condition number up to 6.1×10¹¹, not extra directions. And the operator is not degenerate: ECL needs 44 directions for 90% of its energy, Solar 16, ETTm1 14, ETTh2 10, ETTh1 7, while ETTm2 and Exchange are 1--2-direction dominated (top-1 85.2% and 79.2%); over 32 cells r_90% spans 1--68.

Part 2/6 continues with the two linear comparisons (Table B2) and the gating ablation.

Part 2/6: the linear comparisons, the ablation, the protocol

W1.2 The two linear comparisons you asked for

You are right that a single linear layer and a low-rank map trained under the same lookback search were missing. We ran both: same LL, same HH, same splits, the same μ\mu selection as Alterum, three seeds, 32 cells.

Table B2. Four-horizon average MSE. Degradation is relative to Alterum; positive means the linear arm is worse.

DatasetAlterumsingle lineardegrad.rank-(D+1)(D{+}1)degrad.
ETTh10.4180.4419+5.7%0.4988+19.3%
ETTh20.3400.4954+45.7%0.5221+53.6%
ETTm10.3500.3606+2.9%0.3553+1.5%
ETTm20.2430.2722+11.9%0.2712+11.5%
Exchange0.2610.3557+36.5%0.3976+52.6%
Weather0.2300.2366+2.9%0.2349+2.1%
ECL0.1630.1585−2.6%0.1599−1.7%
Solar0.2020.2013−0.2%0.2012−0.3%
8-dataset mean0.27580.3153+14.3%0.3301+19.7%

Per cell, the better arm has a median degradation of +2.6%, and Alterum is better in 24 of 32. The exceptions, reported rather than averaged: in 8 cells the better linear arm wins --- four ECL (−1.3% to −4.1%), three Solar (−0.3% to −0.6%), ETTm1 at H=96H=96 by 0.2%. On the large-variable datasets the head already carries most of the signal; our margin is concentrated where the variable count is small and D+1D+1 is narrow. The arms average 0.6K trainable parameters (up to 3.63M), so this is not a capacity comparison.

On NLinear and RLinear: the arms above are the function classes those baselines instantiate --- one trained linear map, and a ridge-solved map capped at rank D+1D+1 --- while DLinear is already in the main comparison. Their released implementations differ in normalisation, not in the class of map (NLinear subtracts the window's last value before the map and adds it back).

W1.3 The gating ablation

The ablation now covers all 32 cells rather than the two datasets of the submitted version (revised Table 8; you cite the ECL cell, 0.587 against DLinear 0.166). The gating-free variant is worse than the full model in 32 of 32 cells and worse than the larger single-branch variant in 30 of 32 --- exceptions Exchange at H=336H=336 and H=720H=720. All runs train normally under the standard early stopping, with no NaN and no divergence; the within-cell median coefficient of variation rises from about 1.3% for the single-branch variants to 5.4% without the gate (mean 13.0%).

We accept your interpretation and have re-purposed the experiment: removing the gate changes only how the same kind of linear operator is parameterised, so it cannot show that selection is essential, and we no longer use it that way. What it shows is that the gate conditions the basis and reduces sensitivity to initialisation. The size of the basis contribution is the matched-budget comparison of Appendix N (Table 19): FFE + TDGN is best, alone or tied, in 30 of 32 cells; CFF draws level on the other two but costs a median of 1.4% elsewhere, largest 19.8%.

W2.1 Lookback: the selection rule and the protocol

μ\mu is selected on the validation split, per dataset and per horizon, over {1,3,5,7,9}\{1,3,5,7,9\}, and the test split is evaluated once. Appendix A says so in the revision, and the appendix sentence about μ=7\mu=7 is rewritten as an observation about the sweep rather than a rule: "In this sensitivity sweep, increasing μ\mu reduces MSE, with the lowest average MSE at μ=7\mu=7; this is an observation about the sweep, not the selection rule, which is applied per dataset and horizon on the validation split."

Why the window is searched at all. It is the protocol published with this paradigm, not a licence we took. DeepTime's experimental section states that hyperparameter selection is performed on only one value, the lookback multiplier L=μHL=\mu H, searched over μ=[1,3,5,7,9]\mu=[1,3,5,7,9] and chosen by validation loss, and its hyperparameter table lists that grid. We applied the same grid to Alterum, and to DeepTime itself rather than leaving that baseline at a fixed window. The paradigm's own search already reaches nine horizons of history, against the 96 steps the discrete baselines default to.

Part 3/6 continues: the grid, the Exchange case, the rows you name, and what a long window costs.

Part 3/6: the grid, Exchange, the rows you name, cost

Two notes on the grid. It is defined on the multiplier, not an absolute length, so the same five values serve all horizons and the window is a whole multiple of it (720 to 6,480 at H=720H=720). Spacing is 2 rather than 1 because the ETTm2 sweep already places the turning point inside the grid: MSE improves up to μ=7\mu=7 and then flattens (0.331 at μ=7\mu=7 against 0.334 at μ=9\mu=9 at H=720H=720), so a denser grid would interpolate inside a flat stretch, and per-epoch time and memory grow with LL (Appendix L). What the grid does not settle is its upper edge: in 5 of 32 cells validation selects μ=9\mu=9 (Weather at H=192H=192 and H=336H=336, ECL and Solar at H=96H=96, Solar at H=192H=192), where the search stops at the boundary and a longer window is untested --- a limit of the grid, not an exhaustive search.

Exchange. It is the one configuration where the search finds no gain beyond the shortest setting, so L=96L=96 at all four horizons --- the same lookback xPatch uses --- and the gains there do not rest on a longer window. The gain is also not monotone in the window: averaged over datasets it is best at μ=7\mu=7, and at H=720H=720 it is 0.334 at μ=9\mu=9 against 0.331 at μ=7\mu=7. The parameter count is also independent of LL: a longer window changes how many coordinate--value pairs the inner solve receives, not the size of the network.

On the rows you name. iTransformer and TimeMixer are evaluated at their released lookback of 96 and DLinear at 336, because the tokenisation or the window is part of each model's design. We did not re-run those three at L=6480L=6480, and the reason this does not leave the confound open is Table B2: both linear arms were trained under our own μ\mu search, so every benefit a linear map can extract from a long window is already inside them, and they are still worse in 24 of 32 cells, by a median of +2.6% in the better arm. A long window available to a linear map is not what produces our margin.

Your last point --- that our advantage is largest at H=720H=720, where the lookback gap is largest --- is what the same arms speak to directly. Given the same search, their degradation grows with the horizon instead of shrinking once a long window is available: relative to Alterum's MSE, the single linear arm is +6.4% at H=96H=96, +8.8% at H=336H=336 and +29.5% at H=720H=720, and the rank-capped map +4.3%, +13.9% and +42.9%. The gap is widest where a longer lookback would have helped a linear map most, and widens anyway; what grows with HH is the coordinates the basis must resolve, not the inputs.

Why no fixed L∈{96,336}L\in\{96,336\} grid. Not for the run count: we would rather give the reason than claim the runs were out of reach. For a historical-value model LL is the input width, so moving a discrete forecaster to 6,480 steps changes its parameter count and its work per sample --- DeepTime's Table 11 has a fully connected baseline grow from 0.93M to 14.1M parameters between L=48L=48 and L=5760L=5760 while the time-index model stays at 1.31M --- whereas for a time-index model LL only sets how many coordinate--value pairs the inner solve receives, and the network above it is untouched. One fixed number for both paradigms therefore holds nothing equal: it either places the discrete model outside its designed configuration or the time-index model below the window its published protocol searches. We state that as a structural limitation of the comparison, not a missing run.

What a long window costs. The efficiency table is measured at exactly these μ\mu-selected configurations, so it answers your 6,480-step observation with numbers. At the largest setting the protocol allows (μ=9\mu=9, H=720H=720, L=6480L=6480, ETTm2) an epoch takes 27.5 s and 0.0437 GiB at 1,589 parameters, unchanged from μ=1\mu=1, and is still cheaper than the fastest baseline on that dataset at its own 96-step window (DLinear, 48.7 s). Averaged over the eight datasets at H=720H=720: 35.38 s and 0.070 GiB against DLinear's 48.23 s and 0.104 GiB, with 55× fewer parameters. Had the μ\mu grid been applied to the discrete baselines their cost would have risen with input steps, since each applies its input projection or patch tokenisation to all of them; the trade-off we claim is therefore not an artefact of unequal windows.

The claim we would like weighed. The window decides how much history a forecaster can see, so holding it equal is the fair axis for accuracy, and we report the one case where it is equal. The question that follows is what a model pays for a long window: ours pays nothing in parameters, trains faster per epoch and fits in less memory than a discrete model reading 96 steps, while using up to nine horizons of history --- and that combination, not speed at an equal window, is what the paper is about.

Part 4/6 continues: margins, concentration and the efficiency table.

Reviewer_eCAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces selective associative memory as a unifying perspective on sequence modeling architectures and proposes Alterum, a lightweight continuous-time forecasting model. By combining multi-scale Fourier embeddings, temporal decomposition with gating, and closed-form ridge regression, Alterum achieves competitive forecasting accuracy with approximately 10K trainable parameters.

Critical strengths
  1. Simple and efficient architecture. The combination of Fourier features, temporal decomposition, and closed-form regression provides a computationally efficient alternative to more complex forecasting models.

  2. Useful component analysis. The paper includes extensive ablations, parameter-matched comparisons, and efficiency evaluations that help establish the contributions of individual components.

Critical weaknesses
  1. Insufficient justification for the proposed unifying framework. The selective associative memory perspective is interesting, but the claim that feature- and time-dimension gating are sufficient for forecasting is largely heuristic rather than theoretically established. Moreover, the representation learner and gating mechanism depend only on time coordinates, rather than the observed time-series values. It is unclear how this constitutes adaptive memory selection comparable to attention or input-dependent gating mechanisms.

  2. Potentially unfair experimental comparisons. Alterum tunes the lookback length using L = τH, with τ selected from {1, 3, 5, 7, 9}, whereas most baselines use their original fixed lookback settings. Since longer historical contexts can substantially improve forecasting accuracy, the reported gains may partially reflect differences in available input information rather than architectural improvements.

Questions for authors

Could the authors provide controlled comparisons under matched lookback lengths and report the corresponding computational costs using the actual selected configurations?

1 comment

We thank the reviewer, and we accept both points. Naming: the revision writes μ\mu for the lookback multiplier (L=μHL=\mu H), reserving τ\tau for the time coordinate.

W1 Framework and gate

Three things were conflated in the submitted text.

(1) The feature map depends on coordinates only; the model does not. hϕh_\phi receives only τ\tau, so Z=hϕ(τ)Z=h_\phi(\tau) is one matrix for a fixed (L,H)(L,H); observations enter at exactly one place, the closed-form head. Composition gives y^H,c=AyL,c+bc\hat y_{H,c}=A y_{L,c}+b_c with A=Z~H(Z~L⊤Z~L+λI)−1Z~L⊤A=\tilde Z_H(\tilde Z_L^\top\tilde Z_L+\lambda I)^{-1}\tilde Z_L^\top of rank at most min⁡(H,D+1)\min(H,D+1), applied column by column: coefficients per variable, one shared basis, no variable mixing. Section 4.4 states this and its two consequences. That the map is fixed given (ϕ,λ,L,H)(\phi,\lambda,L,H) is definitional for the closed-form meta-learning family (Bertinetto et al.; DeepTime inherits it --- its Section 3 gives the loop and the ridge regressor), not a defect of it.

(2) Not content-based selection. The gate never reads observed values, so it is not attention-like or input-dependent selection, and the revision no longer says it is; what is adaptive is the inner solve, fitted per window. The lens is renamed gated associative memory, kept as a lens in Sections 2--3 and narrowed: the four dimensions re-describe existing gates rather than vary independently, the "largely independent" claim is withdrawn, and Section 3 positions it as narrower than the associative-memory lenses it builds on.

(3) What the lens now claims. A falsifiable prediction --- selection along the sequence or model axis should fail to help at a matched budget --- stated in Section 6 as untested, with the results that would retract it. What we do test is the representation: at a matched 10,329-parameter budget, FFE + TDGN attains the lowest MSE, alone or tied, in 30 of 32 cells, while the sinusoidal encoding is worse in every cell by a median of 53% (Table 19).

Two measurements we add, tabulated in our rfHh reply (Tables B1--B2). At H=720H=720 the operator needs 44 directions for 90% of its energy on ECL, 16 on Solar, 14 on ETTm1, 10 on ETTh2 and 7 on ETTh1; energy beyond the first D+1D+1 directions is zero to machine precision, round-off at condition numbers up to 6.1×10116.1\times10^{11}. Against a single linear layer and a rank-(D+1)(D+1) map, both given our own μ\mu search, we are better in 24 of 32 cells (four-horizon means 0.2758 vs 0.3153 and 0.3301; exceptions: four ECL cells, three Solar cells, ETTm1 at H=96H=96).

W2 Lookback and fairness

We agree that longer history helps, and do not defend the submitted comparison.

  • μ∈{1,3,5,7,9}\mu\in\{1,3,5,7,9\} is DeepTime's protocol: its hyperparameter table lists exactly this grid, selected by validation loss. We attribute it, and run DeepTime itself under the same grid. μ\mu is chosen per dataset and horizon on the validation split, never on test (Appendix A); in 5 of 32 cells it selects μ=9\mu=9, where a longer window is untested.
  • What we did not equalise, stated rather than inferred: the other eight baselines run at their released lookbacks, and we do not claim they would not improve under a per-dataset search.
  • One matched case: Exchange retains no μ>1\mu>1, so every model reads L=96L=96 and the margin cannot come from history --- against the two linear controls we are better at every horizon, by 104%/29% at H=96H{=}96, 4.4%/8.3% at 192192, 13%/32% at 336336, 48%/79% at 720720.
  • Cost at the configurations behind the accuracy (Table 5; per-dataset cells in Appendix J): 8,781 parameters, 35.38 s per epoch, 0.070 GiB; DLinear 0.485M/48.23 s/0.104 GiB, DeepTime 1.315M/80.09 s/2.366 GiB. At μ=9\mu{=}9 (L=6480L{=}6480, ETTm2) an epoch costs 27.5 s and 0.0437 GiB at the same 1,589 parameters, below that dataset's fastest baseline (DLinear, 48.7 s). Not fastest on two of eight datasets: ECL (67.07 vs 62.56 s), Solar (113.47 vs 112.83 s).
  • No fixed-LL table, for a structural reason: for a discrete model LL is the input width, so one fixed window holds nothing equal (DeepTime's Table 11: a fully connected baseline grows from 0.93M to 14.1M parameters between L=48L{=}48 and 57605760, the time-index model staying at 1.31M). We report what each model pays for the history it uses.

Question

The matched comparison you ask for is Exchange at L=96L=96, plus the two linear controls that got our search and still lose in 24 of 32 cells; we add no fixed-LL table, for the reason above. Cost is reported at the actual configurations, per epoch (per-step time is not comparable when LL differs), including the μ=9\mu=9 endpoint and the two datasets where we are not fastest.

Closing. The submitted text presented a lens as a derivation and a gate as content-based selection. The revision states the lens's scope, drops the input-dependence vocabulary, attributes the protocol, and reports cost at the configurations behind the accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.