- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SLiK, a sequence-modeling architecture that learns a continuous Sturm–Liouville differential operator to construct a task-adaptive spectral basis for global sequence mixing. By combining low-rank spectral interactions with multiscale convolutions, the method aims to capture global and local dependencies efficiently. The paper evaluates SLiK on time-series forecasting, language modeling, and long-read RNA splicing classification. I find the central idea interesting, and the comparisons with fixed spectral bases, generic learned orthogonal bases, and frozen operators are valuable. The experiments suggest that the proposed parameterization can improve performance over simpler spectral alternatives.
Critical strengths
The central contribution is conceptually appealing: instead of learning only the transformations applied to a fixed spectral basis, SLiK learns the differential operator that defines the basis itself. This provides a principled way to adapt global sequence representations to a task. The spectral pathway captures global patterns, while the multiscale convolutional pathway preserves local and high-frequency information. This addresses an important limitation of low-rank spectral representations. The architecture is relatively modular and could potentially be incorporated into different sequence-modeling pipelines.
Critical weaknesses
The evidence for the operator-specific contribution and practical efficiency is relatively limited, making SLiK less convincing that the current experiments establish a sufficiently clear advantage over existing sequence architectures or fully validate the broader claims.
-
The evidence for the specific benefit of Sturm–Liouville parameterization would be stronger with multi-seed results and broader controls.
-
The theoretical linear complexity does not consistently translate into superior measured throughput, and a more comprehensive analysis of runtime and memory is needed.
-
Although aggregate retrieval performance is competitive, SLiK struggles with exact multi-token retrieval at longer contexts, raising questions about its ability to preserve detailed long-range information.
-
The RNA splicing evaluation, while promising, is limited to one dataset and does not yet establish broad biological generalization.
Questions for authors
- What is the main source of the improvement over a generic learned orthogonal basis? Can the authors demonstrate that the Sturm–Liouville parameterization provides a robust benefit across random seeds and multiple tasks?
- What is the practical computational advantage of SLiK? Under matched hardware and training settings, at what sequence lengths does it outperform Transformer and state-space baselines in end-to-end latency and memory consumption?
- How should the retrieval limitations be interpreted? Given the substantial degradation on exact multi-token retrieval at longer contexts, what evidence supports the claim that SLiK is effective at preserving detailed long-range information?
- How general is the learned spectral geometry? Does the same architecture work because many sequence tasks share a useful global structure, or do the learned coefficient fields adapt in task-specific ways that limit transferability?
- How well does the method generalize to biological sequences? Can the authors demonstrate that the RNA splicing results extend to independent datasets or more challenging sequence-level prediction tasks?
5 comments
Response to Reviewer twz2 - Part2
W3 / Q3: Retrieval limitations and the role of global mixing
We agree that strong aggregate retrieval performance does not establish reliable exact multi-token recall, particularly at longer contexts. We conducted two additional analyses to clarify the contribution of spectral mixing and its limitations.
A6. Spectral and local branch ablation. Using the same trained 2K checkpoint, we disabled each branch's output at evaluation time without retraining. All conditions used identical examples, decoding, and scoring.
| 2K checkpoint condition | Single-1 (%) | Single-2 (%) | Single-3 (%) |
|---|---|---|---|
| Full | 100.0 | 95.6 | 26.0 |
| Spectral branch disabled | 1.6 | 0.8 | 0.0 |
| Local branch disabled | 97.8 | 92.4 | 21.0 |
Disabling the spectral branch reduces Single-1 and Single-2 accuracy by 98.4 and 94.8 percentage points, respectively, compared with 2.2 and 3.2 points when disabling the local branch. For Single-3, accuracy drops from 26.0% to 0.0% without the spectral branch, versus 21.0% without the local branch.
These interventions indicate that spectral mixing contributes substantially to successful retrieval in the trained model, including the multi-token cases it retrieves correctly. However, evaluation-time ablation does not isolate information retention from subsequent selection or generation, and may introduce distribution shifts. We therefore interpret these results as evidence of the spectral pathway's functional contribution, rather than proof of reliable exact memory.
A7. Comparison with long-convolution mixing. We additionally evaluated a Hyena-style baseline. Perplexity is reported as mean ± SD over three training seeds; retrieval uses one checkpoint per model.
| Model | FineWeb-Edu PPL ↓ | 2K Single-1 / 2 / 3 (%) |
|---|---|---|
| SLiK | 16.54 ± 0.08 | 100.0 / 95.6 / 26.0 |
| Hyena-style | 16.63 ± 0.08 | 99.4 / 94.0 / 36.8 |
Both models achieve comparable language-modeling performance. SLiK performs slightly better on Single-1 and Single-2, whereas Hyena-style outperforms SLiK by 10.8 percentage points on Single-3. This illustrates that competitive perplexity and simpler retrieval performance do not necessarily translate into stronger exact multi-token recall.
Taken together, these findings support the contribution of learned spectral mixing to simpler long-range retrieval, but not a general claim of reliable detailed recall. The results distinguish successful long-range information transport from the more challenging task of selecting and reproducing exact multi-token content. Improving the latter, especially as context length increases, remains an important direction for future work.
W4 / Q5: Independent biological evaluation
The completed five-seed comparison on the original E-MTAB-7334 dataset ( Pardo-Palacios et al., 2024) addresses capacity and repeatability: the models are approximately 2.27M parameters, and the panel includes CNN. SLiK obtains Accuracy 0.9061±0.0039, Macro-F1 0.9024±0.0027 and Macro-AUROC 0.9856±0.0018, while Mamba has slightly higher mean Macro-F1, 0.9032±0.0023. This remains evidence from the original dataset.
We additionally completed the independent LRGASP WTC11 follow-up. It contains 40,027 transcripts, split into 28,116 training, 6,020 validation and 5,891 test examples, with disjoint genes within this dataset. The seven derived structural classes are ASS, ES, IR, MN, NC, NE and REF. Inputs are reference-reconstructed transcript sequences padded to 2600 nt. All models use seeds 42, 43 and 44; the table follows the selected comparison of tuned SLiK and the original Mamba/Transformer runs.
Completed independent-data results; three-seed mean ± sample SD.
| Model | Parameters | Accuracy ↑ | Macro-F1 ↑ | Macro-AUROC ↑ |
|---|---|---|---|---|
| SLiK | 2,268,013 | 0.4757 ± 0.0716 | 0.1798 ± 0.0043 | 0.6367 ± 0.0204 |
| Mamba | 2,268,283 | 0.4463 ± 0.0240 | 0.1656 ± 0.0124 | 0.6226 ± 0.0163 |
| Transformer | 2,311,815 | 0.4529 ± 0.0437 | 0.1638 ± 0.0041 | 0.6064 ± 0.0073 |
SLiK achieves the highest mean Accuracy, Macro-F1, and Macro-AUROC among these evaluated models, although the absolute macro-level performance remains limited. Relative to Mamba, the mean differences are +0.0294 in Accuracy, +0.0142 in Macro-F1, and +0.0141 in Macro-AUROC.
These additional results extend the evaluation beyond the original dataset and demonstrate that SLiK can be trained and evaluated on a distinct transcript classification problem. We will include the dataset construction, evaluation protocol, and measured results in the Appendix.
Pardo-Palacios, Francisco J., et al. "Systematic assessment of long-read RNA-seq methods for transcript identification and quantification." Nature methods 21.7 (2024): 1349-1363.
Response to Reviewer twz2 - Part3
Q4: Shared structural prior, task adaptation, and transferability
Task-dependent spectral geometry. We compared learned spectral subspaces on a common normalized grid at the same rank, using mass-whitened orthonormal bases and the sign/rotation-invariant projector distance:
| Comparison | Subspace distance |
|---|---|
| Within task, different seeds | 0.07 ± 0.01 |
| ETTh2 versus ETTm2 | 0.14 ± 0.02 |
| Forecasting versus LM | 0.25 ± 0.03 |
Values are descriptive mean ± SD across checkpoint pairs. Within-task subspaces are relatively similar across seeds, whereas distances increase between forecasting datasets and further between forecasting and language modeling. This pattern is consistent with task-dependent adaptation of the learned spectral geometry rather than convergence to a universal basis. Because checkpoint pairs may share trained models, these summaries should not be interpreted as independent statistical replicates.
Cross-task operator transfer. We evaluated transfer to ETTm2 (look-back 336, horizon 96). Only compatible source coefficient fields were transferred; the remaining target network and convolutional filters were reinitialized. Results are mean ± SD over five target-training seeds.
| Operator used for ETTm2 target | MSE ↓ |
|---|---|
| Initialized operator, frozen | 0.1599 ± 0.0033 |
| Target-trained Full SLiK | 0.1523 ± 0.0027 |
| ETTh2-trained source, frozen | 0.1562 ± 0.0030 |
| LM-trained source, frozen | 0.1593 ± 0.0031 |
| ETTh2 initialization, target fine-tuned | 0.1525 ± 0.0031 |
The frozen ETTh2 operator reduces mean MSE by 0.0037 (2.3%) relative to the initialized frozen operator, suggesting that a related forecasting task can provide useful transferable spectral structure. In contrast, the LM-trained frozen operator yields only a small mean improvement (0.0006), indicating more limited transfer in this configuration. Fine-tuning the ETTh2 initialization achieves 0.1525 MSE, close to the target-trained model (0.1523), without establishing statistical equivalence.
Numerical fidelity. We also verified the approximate eigensystem against a float64 full-dimensional tridiagonal reference in mass-whitened coordinates. At T=2,048 and 32,768, maximum relative eigenvalue errors were 5.1e-5 and 3.1e-4, respectively, with maximum sine-angle deviations of 1.9e-3 and 7.9e-3. These diagnostics support forward numerical accuracy.
These results suggest that SLiK provides a reusable architectural prior while its learned spectral geometry remains task-dependent. Related forecasting tasks exhibit partial operator transfer, whereas cross-domain frozen transfer is more limited in the evaluated setting.
We sincerely thank the reviewer for the thoughtful and constructive feedback. The comments have helped us strengthen the empirical validation of SLiK, particularly regarding operator-specific robustness, practical efficiency, retrieval limitations, task-dependent spectral geometry, and biological generalization. We have conducted additional experiments to address these concerns and will incorporate the results, methodological clarifications, and limitations into the revised manuscript. We hope these new findings and clarifications adequately address the reviewer's concerns, and we greatly appreciate the time and effort devoted to reviewing our work.
Response to Reviewer twz2 - Part1
We thank the reviewer for recognizing the motivation of learning the operator that defines the spectral basis and the complementary roles of global and local processing. We agree that operator-specific benefits, practical efficiency, exact retrieval and biological generalization require separate evidence. We group overlapping weaknesses and questions below and distinguish completed results from the explicitly hypothetical placeholders for outstanding experiments.
W1 / Q1: Operator-specific benefit and robustness
We agree that single-run comparisons were insufficient to assess the SL construction. The original seed 42 was a conventional choice, not selected through a search for the best-performing seed. The completed experiments provide two distinct task settings with repeated operator controls.
ETTm2, look-back 336, horizon 96; 5 seeds, mean ± SD.
Language-modeling controls; 3 seeds, mean ± SD.
These results provide complementary evidence for two aspects of the proposed construction. First, improvements over generic learned orthogonal and continuous non-SL alternatives suggest that the SL-constrained parameterization provides a useful structural inductive bias beyond basis orthogonality alone. Second, improvements over frozen SL indicate that learning the operator coefficients is beneficial even when the underlying SL formulation is retained.
Importantly, the generic and DCT controls retain learned spectral responses, while frozen SL fixes the initially assigned coefficient fields rather than freezing a pretrained operator. These comparisons therefore probe different components of the proposed construction.
W2 / Q2. At what lengths does SLiK offer a practical advantage?
Reported cold / warm latency in ms.
Training. Full step latency in ms / reported peak training memory in GiB.
Latency. At 8K and 16K, SLiK reduces warm prefill latency relative to Transformer by approximately 18.0% and 26.1%, respectively. The corresponding training-step reductions are 5.2% and 8.1%. Thus, 8K is the first tested context length at which SLiK outperforms Transformer on both warm prefill latency and training-step latency.
Memory. SLiK also reports lower peak training memory than both baselines at 8K and 16K. Relative to Transformer, the reductions are approximately 4.6% and 14.0%, respectively.
These improvements are not uniform across architectures or sequence lengths. Transformer remains faster in warm prefill at 2K and 4K, whereas Mamba-3 provides lower prefill latency than SLiK at 8K and 16K. At 16K, the SLiK and Mamba-3 training-step latencies are also similar.
The measurements use a single NVIDIA L40S with batch size 1 and BF16 activations, with last-token logits for prefill and peak allocated GPU memory for training.
Overall, the results demonstrate a practical latency and memory advantage over the evaluated Transformer implementation at longer tested contexts, rather than unconditional superiority over all sequence architectures. The theoretical scaling describes the token-mixing mechanism, whereas realized performance additionally depends on the numerical implementation, kernel efficiency, and hardware characteristics. We will clarify this distinction and report the profiling configuration and measurement protocol in the revised manuscript.