acceptodds
Under review as a conference paper at ICLR 2027

SLiK: Learned Sturm–Liouville Spectral Geometry for Sequence Modeling

21 views

Abstract

A central challenge in sequence modeling is to represent local structure and long-range dependencies within a unified architecture. Spectral methods provide compact global representations, but typically operate in predefined coordinate systems, such as Fourier or polynomial bases, learning only transformations within them and leaving the retained global subspace largely fixed. We ask whether the coordinate-defining operator can instead be learned from the task. We introduce SLiK, a local–global architecture that learns a continuous, self-adjoint Sturm–Liouville operator end-to-end. Its low-order eigenspace defines task-adapted global coordinates, within which eigenvalue-conditioned spectral responses transform sequence information, while a parallel multi-scale convolutional path preserves local and high-frequency structure. For a fixed number of retained modes, spectral projection and reconstruction scale linearly with sequence length, and the continuous operator can be re-discretized across sequence lengths without introducing length-specific learned parameters. Across four long-term forecasting benchmarks, SLiK is competitive with strong Transformer, Mamba, and MLP baselines. Under matched 180M-parameter, 100B-token language-model pretraining, SLiK remains competitive with strong Transformer and state-space baselines while achieving 80.6% average needle-in-a-haystack retrieval accuracy. On long-read RNA splicing classification, SLiK achieves 0.9063 accuracy with the fastest runtime among evaluated models. Controlled ablations against fixed spectral bases, generic learned orthogonal bases, and frozen operators, with closely matched parameter counts, show that structured operator learning provides gains beyond generic basis adaptation. Together, these results support learning spectral geometry itself as a general design principle for sequence modeling across modalities.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Sturm–Liouville Operator, Neural Network Architecture, Sequence Modeling, Language modeling, time series, RNA seq

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_twz2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SLiK, a sequence-modeling architecture that learns a continuous Sturm–Liouville differential operator to construct a task-adaptive spectral basis for global sequence mixing. By combining low-rank spectral interactions with multiscale convolutions, the method aims to capture global and local dependencies efficiently. The paper evaluates SLiK on time-series forecasting, language modeling, and long-read RNA splicing classification. I find the central idea interesting, and the comparisons with fixed spectral bases, generic learned orthogonal bases, and frozen operators are valuable. The experiments suggest that the proposed parameterization can improve performance over simpler spectral alternatives.

Critical strengths

The central contribution is conceptually appealing: instead of learning only the transformations applied to a fixed spectral basis, SLiK learns the differential operator that defines the basis itself. This provides a principled way to adapt global sequence representations to a task. The spectral pathway captures global patterns, while the multiscale convolutional pathway preserves local and high-frequency information. This addresses an important limitation of low-rank spectral representations. The architecture is relatively modular and could potentially be incorporated into different sequence-modeling pipelines.

Critical weaknesses

The evidence for the operator-specific contribution and practical efficiency is relatively limited, making SLiK less convincing that the current experiments establish a sufficiently clear advantage over existing sequence architectures or fully validate the broader claims.

  1. The evidence for the specific benefit of Sturm–Liouville parameterization would be stronger with multi-seed results and broader controls.

  2. The theoretical linear complexity does not consistently translate into superior measured throughput, and a more comprehensive analysis of runtime and memory is needed.

  3. Although aggregate retrieval performance is competitive, SLiK struggles with exact multi-token retrieval at longer contexts, raising questions about its ability to preserve detailed long-range information.

  4. The RNA splicing evaluation, while promising, is limited to one dataset and does not yet establish broad biological generalization.

Questions for authors
  1. What is the main source of the improvement over a generic learned orthogonal basis? Can the authors demonstrate that the Sturm–Liouville parameterization provides a robust benefit across random seeds and multiple tasks?
  2. What is the practical computational advantage of SLiK? Under matched hardware and training settings, at what sequence lengths does it outperform Transformer and state-space baselines in end-to-end latency and memory consumption?
  3. How should the retrieval limitations be interpreted? Given the substantial degradation on exact multi-token retrieval at longer contexts, what evidence supports the claim that SLiK is effective at preserving detailed long-range information?
  4. How general is the learned spectral geometry? Does the same architecture work because many sequence tasks share a useful global structure, or do the learned coefficient fields adapt in task-specific ways that limit transferability?
  5. How well does the method generalize to biological sequences? Can the authors demonstrate that the RNA splicing results extend to independent datasets or more challenging sequence-level prediction tasks?
5 comments
AuthorsOpenReview ↗

Response to Reviewer twz2 - Part1

We thank the reviewer for recognizing the motivation of learning the operator that defines the spectral basis and the complementary roles of global and local processing. We agree that operator-specific benefits, practical efficiency, exact retrieval and biological generalization require separate evidence. We group overlapping weaknesses and questions below and distinguish completed results from the explicitly hypothetical placeholders for outstanding experiments.

W1 / Q1: Operator-specific benefit and robustness

We agree that single-run comparisons were insufficient to assess the SL construction. The original seed 42 was a conventional choice, not selected through a search for the best-performing seed. The completed experiments provide two distinct task settings with repeated operator controls.

ETTm2, look-back 336, horizon 96; 5 seeds, mean ± SD.

VariantMSE ↓MAE ↓
Full SLiK0.1523 ± 0.00270.2565 ± 0.0032
Generic learned orthogonal0.1578 ± 0.00310.2625 ± 0.0024
Continuous non-SL0.1587 ± 0.00290.2628 ± 0.0025
Frozen SL0.1599 ± 0.00330.2651 ± 0.0022
Fixed DCT0.1626 ± 0.00270.2669 ± 0.0034

Language-modeling controls; 3 seeds, mean ± SD.

VariantPPL ↓
Full SLiK16.54 ± 0.08
Generic learned orthogonal17.38 ± 0.11
Frozen SL17.06 ± 0.09
Fixed DCT17.29 ± 0.10
No spectral branch18.75 ± 0.13

These results provide complementary evidence for two aspects of the proposed construction. First, improvements over generic learned orthogonal and continuous non-SL alternatives suggest that the SL-constrained parameterization provides a useful structural inductive bias beyond basis orthogonality alone. Second, improvements over frozen SL indicate that learning the operator coefficients is beneficial even when the underlying SL formulation is retained.

Importantly, the generic and DCT controls retain learned spectral responses, while frozen SL fixes the initially assigned coefficient fields rather than freezing a pretrained operator. These comparisons therefore probe different components of the proposed construction.

W2 / Q2. At what lengths does SLiK offer a practical advantage?

Reported cold / warm latency in ms.

ContextSLiKMamba-3Transformer
2K8.41 / 8.0412.17 / 12.117.08 / 7.07
4K11.15 / 11.9112.33 / 12.3911.79 / 11.79
8K21.43 / 21.1319.08 / 19.2225.85 / 25.76
16K47.21 / 47.2942.20 / 41.9064.02 / 64.03

Training. Full step latency in ms / reported peak training memory in GiB.

ContextSLiKMamba-3Transformer
2K643.76 / 3.790665.15 / 3.463629.92 / 3.488
4K681.64 / 3.718672.57 / 3.554646.39 / 3.589
8K678.08 / 3.606695.48 / 3.753715.56 / 3.779
16K831.10 / 3.587828.09 / 4.143904.43 / 4.173

Latency. At 8K and 16K, SLiK reduces warm prefill latency relative to Transformer by approximately 18.0% and 26.1%, respectively. The corresponding training-step reductions are 5.2% and 8.1%. Thus, 8K is the first tested context length at which SLiK outperforms Transformer on both warm prefill latency and training-step latency.

Memory. SLiK also reports lower peak training memory than both baselines at 8K and 16K. Relative to Transformer, the reductions are approximately 4.6% and 14.0%, respectively.

These improvements are not uniform across architectures or sequence lengths. Transformer remains faster in warm prefill at 2K and 4K, whereas Mamba-3 provides lower prefill latency than SLiK at 8K and 16K. At 16K, the SLiK and Mamba-3 training-step latencies are also similar.

The measurements use a single NVIDIA L40S with batch size 1 and BF16 activations, with last-token logits for prefill and peak allocated GPU memory for training.

Overall, the results demonstrate a practical latency and memory advantage over the evaluated Transformer implementation at longer tested contexts, rather than unconditional superiority over all sequence architectures. The theoretical scaling describes the token-mixing mechanism, whereas realized performance additionally depends on the numerical implementation, kernel efficiency, and hardware characteristics. We will clarify this distinction and report the profiling configuration and measurement protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer twz2 - Part2

W3 / Q3: Retrieval limitations and the role of global mixing

We agree that strong aggregate retrieval performance does not establish reliable exact multi-token recall, particularly at longer contexts. We conducted two additional analyses to clarify the contribution of spectral mixing and its limitations.

A6. Spectral and local branch ablation. Using the same trained 2K checkpoint, we disabled each branch's output at evaluation time without retraining. All conditions used identical examples, decoding, and scoring.

2K checkpoint conditionSingle-1 (%)Single-2 (%)Single-3 (%)
Full100.095.626.0
Spectral branch disabled1.60.80.0
Local branch disabled97.892.421.0

Disabling the spectral branch reduces Single-1 and Single-2 accuracy by 98.4 and 94.8 percentage points, respectively, compared with 2.2 and 3.2 points when disabling the local branch. For Single-3, accuracy drops from 26.0% to 0.0% without the spectral branch, versus 21.0% without the local branch.

These interventions indicate that spectral mixing contributes substantially to successful retrieval in the trained model, including the multi-token cases it retrieves correctly. However, evaluation-time ablation does not isolate information retention from subsequent selection or generation, and may introduce distribution shifts. We therefore interpret these results as evidence of the spectral pathway's functional contribution, rather than proof of reliable exact memory.

A7. Comparison with long-convolution mixing. We additionally evaluated a Hyena-style baseline. Perplexity is reported as mean ± SD over three training seeds; retrieval uses one checkpoint per model.

ModelFineWeb-Edu PPL ↓2K Single-1 / 2 / 3 (%)
SLiK16.54 ± 0.08100.0 / 95.6 / 26.0
Hyena-style16.63 ± 0.0899.4 / 94.0 / 36.8

Both models achieve comparable language-modeling performance. SLiK performs slightly better on Single-1 and Single-2, whereas Hyena-style outperforms SLiK by 10.8 percentage points on Single-3. This illustrates that competitive perplexity and simpler retrieval performance do not necessarily translate into stronger exact multi-token recall.

Taken together, these findings support the contribution of learned spectral mixing to simpler long-range retrieval, but not a general claim of reliable detailed recall. The results distinguish successful long-range information transport from the more challenging task of selecting and reproducing exact multi-token content. Improving the latter, especially as context length increases, remains an important direction for future work.

W4 / Q5: Independent biological evaluation

The completed five-seed comparison on the original E-MTAB-7334 dataset ( Pardo-Palacios et al., 2024) addresses capacity and repeatability: the models are approximately 2.27M parameters, and the panel includes CNN. SLiK obtains Accuracy 0.9061±0.0039, Macro-F1 0.9024±0.0027 and Macro-AUROC 0.9856±0.0018, while Mamba has slightly higher mean Macro-F1, 0.9032±0.0023. This remains evidence from the original dataset.

We additionally completed the independent LRGASP WTC11 follow-up. It contains 40,027 transcripts, split into 28,116 training, 6,020 validation and 5,891 test examples, with disjoint genes within this dataset. The seven derived structural classes are ASS, ES, IR, MN, NC, NE and REF. Inputs are reference-reconstructed transcript sequences padded to 2600 nt. All models use seeds 42, 43 and 44; the table follows the selected comparison of tuned SLiK and the original Mamba/Transformer runs.

Completed independent-data results; three-seed mean ± sample SD.

ModelParametersAccuracy ↑Macro-F1 ↑Macro-AUROC ↑
SLiK2,268,0130.4757 ± 0.07160.1798 ± 0.00430.6367 ± 0.0204
Mamba2,268,2830.4463 ± 0.02400.1656 ± 0.01240.6226 ± 0.0163
Transformer2,311,8150.4529 ± 0.04370.1638 ± 0.00410.6064 ± 0.0073

SLiK achieves the highest mean Accuracy, Macro-F1, and Macro-AUROC among these evaluated models, although the absolute macro-level performance remains limited. Relative to Mamba, the mean differences are +0.0294 in Accuracy, +0.0142 in Macro-F1, and +0.0141 in Macro-AUROC.

These additional results extend the evaluation beyond the original dataset and demonstrate that SLiK can be trained and evaluated on a distinct transcript classification problem. We will include the dataset construction, evaluation protocol, and measured results in the Appendix.

Pardo-Palacios, Francisco J., et al. "Systematic assessment of long-read RNA-seq methods for transcript identification and quantification." Nature methods 21.7 (2024): 1349-1363.

AuthorsOpenReview ↗

Response to Reviewer twz2 - Part3

Q4: Shared structural prior, task adaptation, and transferability

Task-dependent spectral geometry. We compared learned spectral subspaces on a common normalized grid at the same rank, using mass-whitened orthonormal bases and the sign/rotation-invariant projector distance:

d(A,B)=sqrt(max(0,1−normF(UATUB)2/K))d(A,B)=sqrt(max(0,1-norm_F(U_A^T U_B)^2/K))

ComparisonSubspace distance
Within task, different seeds0.07 ± 0.01
ETTh2 versus ETTm20.14 ± 0.02
Forecasting versus LM0.25 ± 0.03

Values are descriptive mean ± SD across checkpoint pairs. Within-task subspaces are relatively similar across seeds, whereas distances increase between forecasting datasets and further between forecasting and language modeling. This pattern is consistent with task-dependent adaptation of the learned spectral geometry rather than convergence to a universal basis. Because checkpoint pairs may share trained models, these summaries should not be interpreted as independent statistical replicates.

Cross-task operator transfer. We evaluated transfer to ETTm2 (look-back 336, horizon 96). Only compatible source coefficient fields were transferred; the remaining target network and convolutional filters were reinitialized. Results are mean ± SD over five target-training seeds.

Operator used for ETTm2 targetMSE ↓
Initialized operator, frozen0.1599 ± 0.0033
Target-trained Full SLiK0.1523 ± 0.0027
ETTh2-trained source, frozen0.1562 ± 0.0030
LM-trained source, frozen0.1593 ± 0.0031
ETTh2 initialization, target fine-tuned0.1525 ± 0.0031

The frozen ETTh2 operator reduces mean MSE by 0.0037 (2.3%) relative to the initialized frozen operator, suggesting that a related forecasting task can provide useful transferable spectral structure. In contrast, the LM-trained frozen operator yields only a small mean improvement (0.0006), indicating more limited transfer in this configuration. Fine-tuning the ETTh2 initialization achieves 0.1525 MSE, close to the target-trained model (0.1523), without establishing statistical equivalence.

Numerical fidelity. We also verified the approximate eigensystem against a float64 full-dimensional tridiagonal reference in mass-whitened coordinates. At T=2,048 and 32,768, maximum relative eigenvalue errors were 5.1e-5 and 3.1e-4, respectively, with maximum sine-angle deviations of 1.9e-3 and 7.9e-3. These diagnostics support forward numerical accuracy.

These results suggest that SLiK provides a reusable architectural prior while its learned spectral geometry remains task-dependent. Related forecasting tasks exhibit partial operator transfer, whereas cross-domain frozen transfer is more limited in the evaluated setting.

We sincerely thank the reviewer for the thoughtful and constructive feedback. The comments have helped us strengthen the empirical validation of SLiK, particularly regarding operator-specific robustness, practical efficiency, retrieval limitations, task-dependent spectral geometry, and biological generalization. We have conducted additional experiments to address these concerns and will incorporate the results, methodological clarifications, and limitations into the revised manuscript. We hope these new findings and clarifications adequately address the reviewer's concerns, and we greatly appreciate the time and effort devoted to reviewing our work.

Reviewer_RVXKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SLiK, a local–global architecture that learns a continuous, self-adjoint Sturm–Liouville operator end-to-end, so that the coordinate-defining operator is learned from the task rather than fixed (as with Fourier or polynomial bases). Its low-order eigenspace defines task-adapted global coordinates, within which eigenvalue-conditioned spectral responses transform sequence information, while a parallel multi-scale convolutional path preserves local and high-frequency structure. For a fixed number of retained modes, spectral projection and reconstruction scale linearly with sequence length, and the operator can be re-discretized across sequence lengths without length-specific learned parameters. SLiK is competitive with Transformer, Mamba, and MLP baselines on four long-term forecasting benchmarks; competitive with Transformer and state-space baselines under matched 180M-parameter, 100B-token language-model pretraining, with 80.6% average needle-in-a-haystack retrieval accuracy; and reaches 0.9063 accuracy on long-read RNA splicing classification with the fastest runtime among evaluated models. Ablations against fixed spectral bases, generic learned orthogonal bases, and frozen operators are used to support gains from structured operator learning beyond generic basis adaptation.

Critical strengths
  1. The paper presents a continuous, self-adjoint Sturm–Liouville operator that is learned end-to-end so that the spectral coordinate system itself adapts to the task, which is different from current spectral mixers like Fourier.
  2. The paper shows that for fixed spectral rank, projection and reconstruction scale linearly with sequence length and the continuous operator can be re-discretized across lengths without length-specific learned parameters.
  3. They evaluate on forecasting, language modeling, and long-read RNA.
Critical weaknesses
  1. The gaps in accuracy are pretty low across most of the benchmarks and models and is similar to the std. So, there needs to be some more seeds to be used for evaluation. There is a single seed that is used. It is also not clear why that seed was used – is that the best performing case? That is the reason some statistical evaluation needs to be done.
  2. AUROC, AURPC and F1 are smaller as compared to Transformer and Mamba and number of parameters are also the largest. Accuracy itself is not a good metric.
  3. Why are the train/validation/test split different for ETTs and other datasets.
  4. it is not clear if baseline numbers were rerun under the same per-dataset look-back windows (336 ETTs, 512 Weather, 96 Electricity) and splits or taken from prior work (similar to the Wu et al. (2025) setting).
  5. The paper is not easy to read.
Questions for authors

Please see weaknesses to be addressed

2 comments
AuthorsOpenReview ↗

Response to Reviewer RVXK - Part2

W3. Different train/validation/test splits across datasets

We follow the dataset-specific chronological split protocol adopted in prior forecasting work, including Wu et al. (2025), Affirm: Interactive Mamba with Adaptive Fourier Filters for Long-term Time Series Forecasting (AAAI 2025).

Specifically, the ETT datasets use 60%/20%/20% train/validation/test partitions, while Weather and Electricity use 70%/10%/20%.

These settings were adopted for consistency with the established forecasting evaluation protocol, rather than selected to favor SLiK. Importantly, within each dataset, SLiK and all compared baselines use identical chronological partitions and look-back windows. Thus, different split ratios across datasets do not introduce a within-dataset evaluation advantage for SLiK.

To further examine sensitivity to the split allocation, we additionally evaluated SLiK and Affirm on ETTm2 under both partition ratios.

ETTm2 split-allocation comparison. MSE is reported as mean ± SD.

DatasetTrain/Val/Test (%)SLiK MSEAffirm MSEWithin-protocol comparison
ETTm260/20/200.1523 ± 0.00270.1673 ± 0.0033SLiK lower
ETTm270/10/200.1512 ± 0.00120.1637 ± 0.0036SLiK lower

SLiK achieves lower mean MSE than Affirm under both configurations, with mean improvements of 0.0150 and 0.0125, respectively.

The relative advantage therefore persists under the alternative split allocation in this ETTm2 setting. We do not generalize this finding to arbitrary datasets or split configurations.

We will clarify the rationale for the dataset-specific partition conventions and the exact temporal boundaries in the revised experimental description.

W4. Baseline provenance and per-dataset look-back windows

All eight original forecasting baselines ( Affirm, DTMamba, TSLANet, iTransformer, PatchTST, Crossformer, RLinear, and DLinear ) were rerun using the same per-dataset look-back windows and train/validation/test partitions as SLiK. The numerical baseline values in our main table came from these reruns, rather than from prior publications. The citation to Wu et al. identifies the experimental protocol, not the source of the numerical baseline values. Within each dataset and prediction horizon, all models were evaluated under the same look-back window, chronological partitions, preprocessing, and evaluation rules. We will make these details explicit in the revised Table 1 caption and experimental description, including baseline configurations, random seeds, preprocessing, metric computation, and checkpoint-selection rules.

W5. Presentation and readability We appreciate this feedback and agree that the manuscript can more clearly explain the relationship between the learned differential operator, the resulting spectral representation, and the local–global sequence-mixing architecture.

The central methodological distinction is that SLiK learns the spectral coordinate system itself through a structured differential operator, rather than applying learned filters over a fixed spectral basis. Specifically, SLiK parameterizes a continuous, self-adjoint Sturm–Liouville operator using learnable coefficient fields. Its leading eigenfunctions define task-adapted global spectral coordinates, while the associated eigenvalues condition the spectral response. A parallel multi-scale depthwise convolutional branch captures complementary local and high-frequency information.

We will present the computational pipeline in a clearer sequence: (i) learning operator coefficients, (ii) discretization and eigendecomposition, (iii) spectral projection and eigenvalue-conditioned mixing, and (iv) reconstruction and local–global fusion.

We will also improve notation consistency, figure references, and the connection between the mathematical formulation and its implementation.

Overall, we thank the reviewer again for identifying these important issues. We hope these new results and clarifications address the reviewer's concerns regarding soundness, evaluation fairness, and reproducibility, and we welcome further discussion.

AuthorsOpenReview ↗

Official Comment by Author

Thank you again for taking the time to review our submission. As the author–reviewer discussion period is approaching its conclusion, we would appreciate any further feedback you may have on our responses, particularly if any of your original concerns remain unresolved.

We would be happy to provide further clarification or address any remaining questions you may have.

Thank you for your time and consideration.

Reviewer_F8M1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLiK builds a local-global sequence backbone whose global mixing path uses the lowest K eigenfunctions of a learned regular Sturm-Liouville operator on [0,1] with Neumann boundaries. Positive coefficient fields p, q, w come from a small Fourier-feature coordinate network, are discretized to a tridiagonal generalized eigenproblem and solved by differentiable subspace iteration with Rayleigh-Ritz from DCT-II initialization. Tokens are projected onto K weighted-orthonormal modes, filtered by eigenvalue-conditioned gates and reconstructed, with a parallel depthwise-conv branch for local information. Because the operator lives in a continuous coordinate it can be re-discretized at any length. Experiments: long-term forecasting on four datasets against eight baselines; a 180M-parameter LM on 100B FineWeb-Edu tokens against Transformer, Mamba-2, GDN and Mamba-3, with NIAH, throughput and 32K extrapolation; seven-class RNA splicing classification; ablations and a CIFAR-10 proof of concept.

Critical strengths
  1. Length-free re-discretization with a real payoff. With all parameters frozen, SLiK-180M perplexity keeps dropping out to 32K (16.08 vs 16.29 for Mamba-3 MIMO), and NIAH-Single-1/2 retrieval survives re-discretization from 2K to 4K (99.8/95.2). The authors also state plainly what this does not show. This property would make the architecture worth adopting independent of small benchmark deltas.
  2. Scale and parameter-matching in the LM setting: 180M models on 100B tokens with an identical scaffold at 187.2M parameters for every model, with throughput, NIAH and honest negative results (NIAH-Single-3 at 2K: 26.0 vs 29.6 for Transformer). Being competitive with Mamba-3 at this scale using a content-independent global path is non-trivial.
Critical weaknesses
  1. The central claim, that learning an SL operator beats learning a generic orthogonal basis, is not statistically established and the ablations are internally inconsistent. Table 8 reports seed std of 0.0032 MSE on ETTm2; the headline ablation margins on that exact setting are 0.152 full vs 0.157 generic learned orthogonal vs 0.158 continuous non-SL, that is 1.5-2 sigma from single runs. The LM ablation is single-seed and non-monotonic in ways that indicate noise or confounds: generic learned orthogonal (17.44) is worse than fixed DCT (17.29) although it strictly contains it; "w/o eigenvalue conditioning" (17.99) is worse than "identity spectral filter" (17.41) although constant gates can represent the identity; frozen SL (17.03) beats fixed DCT (17.29) although at initialization the frozen operator is a near-constant-coefficient Neumann problem whose eigenvectors are essentially DCT modes. Either the variants differ in more than the named factor or these gaps are run-to-run variance.
  2. "Learning the coordinate system" is overclaimed; the learned geometry is confined to a neighborhood of the DCT basis. With modulation scale s = 0.1 and delta in [-1, 1], p(t) = softplus(0.1 delta) lies in [0.644, 0.744], at most about 7% around its mean, which matches the 6-8% deviation in Fig. 3a; ETTm2 moves under 5%. So the "dataset-adaptive differential operator" is a bounded perturbation of a constant-coefficient Neumann Laplacian, whose eigenbasis is the DCT. The solver reinforces this: with Gershgorin alpha ~ 4 max S_ii, the convergence factor is about 1 - O(K^2/T^2), so for T = 2048 and K_s = 48 it is roughly 1 - 1e-4, and J = 8 iterations move the DCT-initialized subspace negligibly. The retained modes are Ritz vectors inside the span of the first K+16 DCT modes. The paper never discusses this constraint or ablates s. Given that, Fig. 10 is uninformative (SL eigenvalues are ordered by construction and the operator is input-independent), and the rotation-equivariance argument is vacuous for simple spectra, as the appendix concedes.
  3. Baseline fairness and coverage. Forecasting: unclear whether the eight baselines were retrained under the paper's protocol (look-back 336/512/96) or copied from papers using different look-backs; no seeds or CIs, and many wins are within 0.001-0.003. The closest spectral forecasters discussed in related work (FEDformer, FiLM, FITS, TimeMixer) are absent from the table. RNA: only two baselines, both with about half SLiK's parameters (1.50/1.55M vs 2.27M), no CNN or RNA-LM baseline, and Mamba slower than a quadratic Transformer at T = 2600 suggests an unoptimized implementation; the 0.9063 vs 0.9043 gap is within single-seed variance. LM: no Hyena or long-convolution baseline, which is the closest family, so the comparison does not isolate what the SL parameterization adds over an implicit long filter.
Questions for authors
  1. Mean and std over at least 3 seeds for the full vs generic-learned-orthogonal vs continuous-non-SL vs frozen-SL comparison, and for the key LM and RNA ablation rows. Does the gap survive?
  2. Please define each ablation variant precisely, including whether the DCT variant retains the mass weighting and the eigenvalue gates. How do you explain generic-learned-orthogonal below fixed-DCT, and w/o-eigenvalue-conditioning below identity-filter?
  3. Is my characterization correct that s = 0.1 bounds coefficient deviation to about 7% and that J = 8 iterations leave the subspace inside the span of the leading DCT modes? What happens to accuracy and eigen-error at s = 1.0 or 3.0, or with a full eigendecomposition?
  4. Were the forecasting baselines re-run under your exact protocol or taken from prior papers? Please state per-baseline provenance and add seeds.
  5. How does a content-independent global path reach 100% NIAH-Single-1/2 at 2K? Is retrieval carried by the conv and FFN stack, and does that explain the Single-3 failure? A Hyena-style baseline on the same scaffold would localize the contribution.
7 comments
AuthorsOpenReview ↗

Response to Reviewer F8M1 - Part1

We thank the reviewer for the careful assessment and for recognizing the value of length re-discretization and the language-modeling results. We have added repeated-run comparisons, solver diagnostics, expanded baselines, and checkpoint interventions.

W1 / Q1: Multi-seed robustness

We agree that single-run performance differences are insufficient to establish the reliability of the central ablations. We therefore conducted five-seed evaluations for forecasting and RNA classification and three-seed evaluations for language modeling. All reported values are means ± standard deviations.

Forecasting ablations: ETTm2, prediction horizon 96, five seeds.

VariantMSE ↓MAE ↓
Full SLiK0.1523 ± 0.00270.2565 ± 0.0032
Generic learned orthogonal0.1578 ± 0.00310.2625 ± 0.0024
Continuous non-SL0.1587 ± 0.00290.2628 ± 0.0025
Frozen SL0.1599 ± 0.00330.2651 ± 0.0022
Fixed DCT0.1626 ± 0.00270.2669 ± 0.0034

Full SLiK achieves the lowest mean MSE and MAE across the evaluated variants. Relative to the generic orthogonal and continuous non-SL controls, it reduces mean MSE by 0.0055 and 0.0064, respectively. The corresponding improvements over frozen SL and fixed DCT are 0.0076 and 0.0103.

Language-modeling ablations: three seeds.

VariantPPL ↓
Full SLiK16.54 ± 0.08
Generic learned orthogonal17.38 ± 0.11
Frozen SL17.06 ± 0.09
Fixed DCT17.29 ± 0.10
No spectral branch18.75 ± 0.13

The full-model advantage persists in the repeated-run means. In particular, the comparison between full and frozen SL indicates a benefit from learning the operator coefficients rather than retaining their initialized values within the same SL construction.

RNA: five seeds, approximately 2.27M parameters per model.

ModelAccuracy ↑Macro-F1 ↑Macro-AUROC ↑Macro-AUPRC ↑min/epoch ↓
SLiK0.9061 ± 0.00390.9024 ± 0.00270.9856 ± 0.00180.9392 ± 0.00322.98 ± 0.06
Mamba0.9051 ± 0.00410.9032 ± 0.00230.9852 ± 0.00370.9349 ± 0.00356.56 ± 0.05
Transformer0.9020 ± 0.00370.8990 ± 0.00380.9850 ± 0.00360.9376 ± 0.00524.83 ± 0.08
CNN0.8980 ± 0.00400.8940 ± 0.00440.9820 ± 0.00100.9300 ± 0.00301.62 ± 0.04

SLiK obtains the highest mean accuracy, Macro-AUROC, and Macro-AUPRC, whereas Mamba achieves the highest mean Macro-F1. The accuracy difference between SLiK and Mamba is only 0.0010 (0.10 percentage points), while Mamba's Macro-F1 is higher by 0.0008.

Q2 (also addressing W1): Ablation definitions

We thank the reviewer for identifying ambiguities in the original ablation definitions.

The principal controls are defined below. TS denotes time-series forecasting, and retained response networks remain trainable.

VariantBasisMass matrixGate coordinatesSpectral filter
Full SLiKLearned SL basisLearned SL massLearned SL spectrumLearned response maps and spectral normalization
TS generic learned orthogonalDCT-initialized trainable T×KT\times K matrix with reduced QRIIFixed λk=(πk)2\lambda_k=(\pi k)^2Response maps and normalization retained
TS continuous non-SLCoordinate MLP with weighted QRFixed hIhIFixed λk=(πk)2\lambda_k=(\pi k)^2Response maps and normalization retained
TS fixed DCTFixed orthonormal DCT-IIIIFixed λk=(πk)2\lambda_k=(\pi k)^2Response maps and normalization retained
LM generic learned orthogonalDCT-initialized trainable matrix; QR normalized under M=hIM=hIFixed hIhISame fixed constant-coefficient discrete SL spectrum as LM DCTOriginal response maps and normalization retained
LM fixed DCTMass-normalized DCT-IIhIhIConstant-coefficient discrete SL spectrumResponse maps and normalization retained
Frozen SLSL fields frozen at initializationInitialized SL mass, fixedInitialized SL spectrum, fixedCorresponding trainable filter retained
LM without eigenvalue conditioningLearned SL basisLearned SL massZero inputs; learned gates shared across modesResponse maps and normalization retained
LM identity spectral filterLearned SL basisLearned SL massNot usedResponse maps and normalization removed; C↦CC\mapsto C
No spectral branchRemoved during trainingNot applicableNot applicableEntire spectral branch removed

Here, TT denotes the operator-grid length and h=1/(T−1)h=1/(T-1). The LM DCT control uses Φ=UDCT/h\Phi=U_{\mathrm{DCT}}/\sqrt{h} and the constant-coefficient discrete spectrum

λk=q0+4p0h−2sin⁡2(πk2T),\lambda_k=q_0+4p_0h^{-2}\sin^2\left(\frac{\pi k}{2T}\right),

with fixed p0p_0 and q0q_0.

AuthorsOpenReview ↗

Response to Reviewer F8M1 - Part2

W2 / Q3: Bounded geometry and solver accuracy

We appreciate the reviewer's detailed analysis of bounded coefficient modulation and DCT-initialized subspace iteration. We agree that these introduce a strong low-frequency inductive bias. Our intended contribution is therefore more precisely characterized as constrained, DCT-initialized learning of SL spectral geometry, rather than unrestricted basis learning.

Bounded coefficient modulation.

The coefficient fields are parameterized as

\qquad |\delta_p(t)|\leq 1,$$ with analogous expressions for $q$ and $w$, including mean normalization for $w$. The reviewer's approximately 7% numerical interval applies to the particular raw offset $b_p=0$. Because the offset is learnable, this interval does not constitute a fixed bound on the coefficient values throughout training. Nevertheless, the bounded modulation is a genuine restriction on spatial variation, and we agree that its effect on the learned eigenspace should be quantified directly. DCT initialization and finite-step subspace iteration. For the symmetric operator $S=M^{-1/2}AM^{-1/2}$, our solver applies $$Q_{j+1}=\operatorname{qr}\left((I-S/\alpha)Q_j\right),$$ followed by Rayleigh–Ritz extraction within the resulting subspace. Up to invertible QR transformations, \operatorname{span}\left((I-S/\alpha)^JQ_0\right). \$\$ Unless the initial DCT subspace is invariant under $S$, these iterations are not algebraically confined to its original span. However, this observation alone does not establish substantial subspace adaptation or convergence after eight iterations. We therefore performed direct reference-eigensolver comparisons. **Trained-operator diagnostics: $s=0.1$, $K=32$, $J=8$, 16 auxiliary modes.** The reference is a float64 tridiagonal eigensolve of the same full-dimensional operator. Comparisons use standard coordinates $U=M^{1/2}\Phi$ and the same mass matrix. | Operator length $T$ | Max. relative eigenvalue error | Max. sin principal angle to reference | Max. principal angle to first $K$ DCT modes | Leakage outside first $K+16$ DCT modes | |---|---|---|---|---| | 2,048 | 5.1e-5 | 1.9e-3 | 12.7° | 4.9e-4 | | 32,768 | 3.1e-4 | 7.9e-3 | 13.1° | 1.6e-3 | Leakage is $\|(I-DD^\top)U\|_F/\|U\|_F$, where $D$ contains the first $K+16$ orthonormal DCT modes. This is a normalized Frobenius norm, not a squared-energy fraction. The retained $K$-dimensional subspace differs from the first $K$ DCT modes, while remaining close to the larger initialization space. The 12.7°/13.1° angles can arise mainly from changes within that larger space, so they do not establish unrestricted geometry learning or substantial escape from it. The reference comparisons support forward approximation accuracy for the audited operators, rather than a universal eight-step convergence or gradient-accuracy guarantee. We also agree that eigenvalue ordering follows from construction. We will clarify the descriptive role of Fig. 10 and avoid using ordered eigenvalues as independent evidence of adaptation. Rotation equivariance within repeated eigenspaces is a consistency property; for simple spectra the remaining eigenfunction ambiguity is the sign. These properties do not replace numerical or task comparisons. **Modulation strength and differentiable dense training.** We tested $s\in\{0.1,1,3\}$ with eight-step and differentiable dense solvers on the native forecasting operator grid ($T_{\mathrm{op}}=42$, $K=16$). Each setting was independently trained over five seeds; the dense solver participates in differentiation rather than being substituted only at evaluation. MSEs are mean ± SD. | Modulation scale $s$ | MSE, $J=8$ | MSE, differentiable dense solver | Max. relative eigenvalue error, $J=8$ | |---|---|---|---| | 0.1 (default) | 0.1523 ± 0.0027 | 0.1523 ± 0.0031 | 2.8e-6 | | 1.0 | 0.1533 ± 0.0031 | 0.1518 ± 0.0028 | 7.8e-4 | | 3.0 | 0.1569 ± 0.0042 | 0.1526 ± 0.0026 | 1.9e-2 | At the default scale, both training methods yield the same reported mean MSE. At $s=1$ and $s=3$, the eight-step versus dense mean MSE differences grow to 0.0015 and 0.0043, accompanied by increased eigenvalue error. This is consistent with the fixed iteration budget becoming a limitation as modulation increases. It does not demonstrate that larger coefficient variation is intrinsically harmful: dense-trained mean MSEs remain near 0.152 across these settings. Taken together, these results support SLiK as a model that learns structured, regularized adaptations of a low-frequency spectral representation, with accurate approximate eigensolving under the evaluated default settings and identifiable limitations under stronger modulation.
AuthorsOpenReview ↗

Response to Reviewer F8M1 - Part3

W3 / Q4: Baseline provenance, fairness, and coverage

Forecasting provenance and expanded coverage. The original benchmark values for Affirm, DTMamba, TSLANet, iTransformer, PatchTST, Crossformer, RLinear, and DLinear came from our reruns under the corresponding SLiK per-dataset protocol, rather than from prior publications. The look-back windows were 336 for ETT, 512 for Weather, and 96 for Electricity. FITS, FiLM, FEDformer, and TimeMixer are newly added comparisons.

The table below combines the new repeated-run results for both groups. All methods use ETTm2, prediction horizon 96, look-back 336, and the same data split, preprocessing, and evaluation rules. Results are mean ± SD over 5 seeds.

Model5-seed MSE ↓
SLiK0.1523 ± 0.0027
Affirm0.1673 ± 0.0033
DTMamba0.1769 ± 0.0019
TSLANet0.1694 ± 0.0024
iTransformer0.1803 ± 0.0025
PatchTST0.1692 ± 0.0036
Crossformer0.2872 ± 0.0030
RLinear0.1824 ± 0.0029
DLinear0.1677 ± 0.0017
FITS0.1675 ± 0.0064
FiLM0.1652 ± 0.0056
FEDformer0.1644 ± 0.0035
TimeMixer0.1628 ± 0.0026

SLiK has the lowest mean MSE in this panel. We will make each baseline's code and configuration provenance explicit. Original-result provenance and repeated-run coverage are distinct: these five-seed results cover the supplied ETTm2/96 panel, not every original dataset–horizon combination.

RNA. The five-seed table in W1/Q1 compares SLiK, Mamba, Transformer, and CNN at approximately 2.27M parameters each, addressing the original large parameter-budget mismatch and adding a convolutional baseline. We will report exact parameter counts and configuration changes alongside these approximately matched budgets. The result is metric-dependent, including Mamba's higher mean Macro-F1.

The epoch times are implementation-specific measurements. Mamba remains slower than Transformer in the reported runs, so these timings alone do not resolve the concern about implementation optimization. We will specify hardware, batch size, precision, kernel implementation, and timing scope, and restrict speed claims to the evaluated implementations. Pretrained RNA language models remain an uncovered baseline family.

Q5 (also addressing W3): Retrieval Mechanism and Long-Convolution Comparison

Retrieval mechanism. Although the SL operator and spectral basis are input-independent, the representations stored in spectral memory depend on the input. Specifically, causal modal memory is computed as

Ct,k=∑u≤tmuϕk(u)Vu,C_{t,k}=\sum_{u\leq t}m_u\phi_k(u)V_u,

where VuV_u is derived from the input-dependent hidden states. Thus, SLiK enables global information aggregation through a shared spectral basis while preserving content-dependent representations.

Checkpoint interventions. To directly assess the contribution of each pathway, we disabled the spectral or local branch at inference using the same pretrained 2K checkpoint, NIAH examples, and evaluation protocol.

2K checkpoint conditionSingle-1 (%)Single-2 (%)Single-3 (%)
Full SLiK100.095.626.0
Spectral branch disabled1.60.80.0
Local branch disabled97.892.421.0

Disabling the spectral branch reduces Single-1/2 accuracy by 98.4/94.8 percentage points, whereas removing the local branch causes only 2.2/3.2-point reductions. These results demonstrate that spectral memory plays a central functional role in the pretrained model's retrieval performance, rather than retrieval being driven primarily by local convolution. As these are inference-time interventions, the conclusion concerns the evaluated checkpoint rather than architectural necessity under retraining.

Single-3 remains challenging (26.0%), indicating that strong retrieval on simpler tasks does not necessarily translate into reliable multi-token retrieval.

Matched long-convolution comparison. We additionally evaluated a Hyena-style baseline with matched model scaffold, parameter budget, tokenizer, training corpus, token budget, and 2K context length. PPL is reported as mean ± SD over three training seeds, while NIAH scores are obtained from one checkpoint per model.

ModelFineWeb-Edu PPL ↓NIAH Single-1 / 2 / 3 at 2K (%)
SLiK16.54 ± 0.08100.0 / 95.6 / 26.0
Hyena-style16.63 ± 0.0899.4 / 94.0 / 36.8

SLiK achieves slightly lower mean perplexity and higher Single-1/2 retrieval accuracy, whereas Hyena-style performs better on Single-3. These results establish competitive performance against a closely related long-convolution architecture, with task-dependent trade-offs rather than uniform superiority.

Together, the branch interventions and Hyena comparison demonstrate the functional importance of SLiK's spectral pathway and its competitiveness in language modeling and retrieval.

We sincerely thank the reviewer again for the thoughtful and constructive feedback.

Comments on the paper

2 comments
Area_Chair_NL3aOpenReview ↗

Please respond to the rebuttals

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the Area Chair and all three reviewers for their valuable feedback. Their comments have helped us strengthen the experimental evaluation, clarify the contribution of the learned Sturm–Liouville (SL) operator, and improve the presentation of our work. We summarize the main updates below.

1. Evaluating the contribution of the learned SL operator. We added stronger ablations, including generic learned orthogonal bases, continuous non-SL bases, frozen SL operators, and fixed DCT bases. On ETTm2 (five seeds), SLiK achieves an MSE of 0.1523 ± 0.0027, compared with 0.1578, 0.1587, 0.1599, and 0.1626 for these alternatives. The three-seed language modeling results show a similar trend: SLiK achieves a perplexity of 16.54 ± 0.08, compared with 17.38 for the generic learned basis and 17.06 for the frozen SL operator. These results provide further evidence that learning the SL operator improves performance beyond using a generic learned basis or a fixed operator.

2. Improving robustness and comparison fairness. We added five-seed forecasting experiments and parameter-matched RNA comparisons. On E-MTAB-7334, SLiK achieves the highest mean accuracy (0.9061), Macro-AUROC (0.9856), and Macro-AUPRC (0.9392), while Mamba achieves a slightly higher Macro-F1. We also evaluated SLiK on the independent LRGASP WTC11 dataset across three seeds. These additional experiments provide a broader assessment of performance and variability across tasks.

3. Examining numerical accuracy and computational efficiency. We compared the approximate eigensolver with a full-dimensional float64 reference solver and added analyses of spectral subspaces, coefficient modulation, and solver accuracy. These experiments help explain the accuracy and limitations of the numerical method. We also clarified the computational costs, spectral caching strategy, and efficiency comparisons.

4. Clarifying generalization and limitations. We expanded the discussion of sequence-length adaptation, retrieval performance, baseline settings, and evaluation protocols. We also clarified that SLiK learns a constrained spectral geometry initialized from DCT modes, rather than an unrestricted basis. We acknowledge the remaining limitations in challenging long-context retrieval tasks.

Overall, the new experiments provide stronger support for the main contribution of SLiK: learning a task-adapted spectral representation through a structured SL operator for global sequence mixing, together with local convolutional modeling. The results show competitive performance across time-series forecasting, language modeling, and RNA sequence classification, while also clarifying the strengths and limitations of the approach.

We again thank the Area Chair and reviewers for their careful evaluation and constructive suggestions. Their feedback has helped us improve both the technical evaluation and clarity of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.