acceptodds
Under review as a conference paper at ICLR 2027

Spectral Enhanced Transformer: Multi-Resolution Causal Convolution for Transformer Language Modeling

Abstract

We introduce the Spectral Enhanced Transformer (SET), which augments Transformer blocks with a learned multi-resolution causal convolutional path parallel to self-attention and causal token mixing after the MLP. FFT evaluation with zero padding preserves autoregressive causality and is numerically equivalent to direct evaluation of the same kernels. A complete 30-run WikiText-103 matrix crosses two positional encodings, five parameter-matched arms, and three paired seeds, with 61.44M training tokens per run. Full SET reduces mean validation NLL from 4.01723 to 3.88255 with learned positions and from 3.87901 to 3.84695 with RoPE, with every paired seed favoring full SET over the wider-FFN baseline. MLP-only mixing underperforms the baseline, and a fixed schedule beats depth-varying full SET in every RoPE seed, so the results do not establish a depth-varying-schedule advantage. Earlier synchronized timing measures 1.85 baseline step time, and longer baseline training wins under wall-clock matching. The supported result is a finite-budget equal-token quality–compute tradeoff, not an efficiency or scaling advantage; larger historical runs remain descriptive because of incomplete metadata and undertraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformer language models, causal convolution, spectral mixing, FFT, parameter-matched evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_4ZWVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SET added a multi-resolution causal convolution path parallel to self-attention and an additional causal convolution after the MLP. On WikiText-103, it provides a noticeable improvement under learned positional embeddings but a small improvement with RoPE. SET is also about 1.85× slower per step due to its higher computation complexity.

Critical strengths
  1. The paper provides relatively careful controlled ablations, including parameter matching, and positional-encoding controls.

  2. It is also transparent about negative results and computational cost, which makes the actual scope of the contribution clear.

Critical weaknesses
  1. Limited practical benefit in the tested RoPE setting

The positional argument in Section 4.1 is restricted to learned absolute positional embeddings and does not establish a corresponding limitation of RoPE-based attention. Empirically, the benefit is substantially smaller with RoPE. Separately, the reported implementation requires 1.85× the original baseline’s step time, and longer baseline training achieves lower loss in the tested wall-clock comparisons. These results leave the practical justification for the additional computational cost insufficiently established.

  1. Limited evaluation breadth and evidence of broader applicability

The primary controlled evaluation is concentrated on WikiText-103. The C4 results in Table 10 provide some evidence on a second corpus, but report only language-modeling loss and are subject to acknowledged limitations in training adequacy and experimental metadata. Consequently, the current evidence does not establish whether the observed NLL improvements translate into broader downstream or cross-domain benefits.

Questions for authors

Please see the clarification questions raised in the Weaknesses section.

2 comments

Under RoPE, full SET reduces mean validation NLL from 3.87901 to 3.84695 relative to the parameter-matched wider-FFN Transformer, with all three paired seeds favoring SET. Public WikiText-103 test NLL also improves from 3.8835 to 3.8510.

The same direction appears in the submitted C4 pairs, where SET lowers loss by 3.9%, 5.1%, 4.0%, and 2.9% at the reported 125M, 350M, 760M, and 1B configurations. For incremental generation, caching reduces SET latency from 20.292 to 14.019 ms/token and peak allocated memory from 464041984 to 440651264 bytes at prompt 128 and decode 64. The maximum cached-versus-full logit difference is 0.00003481, and an independent rerun reproduces the 1.447-fold speedup.

SET yields only a small reduction in loss with RoPE, and the paper reports a substantial increase in training time per step. The authors’ response does not explain why this trade-off is worthwhile in practice. The response also provides no new evidence that the lower loss translates into better performance on downstream tasks. Neither of my two concerns has been adequately addressed, so my rating remains unchanged.

Reviewer_L74COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Spectral Enhanced Transformer (SET), which augments Transformer blocks with a multi-resolution causal convolution branch parallel to self-attention and a causal depthwise convolution gate after the MLP, evaluated via zero-padded FFTs. The authors evaluate SET primarily in a controlled 30-run ablation on WikiText-103 (~95M parameters, 61.4M training tokens, sequence length 256), alongside unverified historical runs in the appendix. While SET achieves slightly lower validation NLL than a parameter-matched wider-FFN Transformer at equal training tokens, it is 1.85×1.85\times slower per step and underperforms the baseline when matched for wall-clock training time.

Critical strengths
  • Transparent reporting of negative results: The authors explicitly report that SET loses under wall-clock matching, that depth-varying kernel schedules fail to beat fixed schedules under RoPE, and that FFT convolution is numerically identical to direct convolution.
  • Controlled factorial ablation: The 30-run parameter-matched matrix cleanly isolates the parallel and post-MLP convolutional components across three paired seeds and two positional encodings.
Critical weaknesses
  • Non-standard evaluation on a single dated benchmark: All controlled experiments rely solely on a 0.5-epoch run (61.4M tokens, sequence length 256) of WikiText-103. If SET is intended as a general enhancement to the Transformer, it should be evaluated in standard setups across domains where Transformers are typically benchmarked:
    • Autoregressive LM (nanoGPT setup): Standard small-scale architectural evaluations use the 124M GPT-2 nanoGPT setup [1] trained on OpenWebText [2] or FineWeb [3] at sequence length 1024, rather than 0.5 epochs of WikiText-103 at length 256.
    • Bidirectional NLU: Without the causal constraint, spectral mixing can operate globally; this should be tested in a BERT-style [4] setup on GLUE [5].
    • Vision: Given SET's motivation from GFNet and Feature Pyramid Networks, it should be compared against Vision Transformers (ViT) [6] on ImageNet [7].
  • Inconsistent claims, missing model definitions, and inverse scaling in 25M–1B runs: Section 7.1 claims to evaluate "8 scales from 25M to 1B parameters," yet the main text only evaluates the 85M scale, Table 9 (Model Configurations) only defines 6 scales up to 350M (completely omitting the 760M and 1B architectures), and Table 13 lists 760M†\dagger and 1B†\dagger with an unexplained dagger (†\dagger). Worse, every multi-scale table (Tables 10, 12, 13 on WikiText-103 and C4) exhibits severe inverse scaling—the 1B baseline reaches 589.1 PPL while the 50M model reaches 52.5 PPL—with missing training metadata.
  • Marginal gains with RoPE and poor compute efficiency: Switching from learned absolute positions to RoPE [8] eliminates ∼75%\sim 75\% of SET's benefit (−0.131-0.131 NLL drops to −0.035-0.035 NLL vs. wider-FFN), indicating the convolution largely compensates for missing relative positional bias. Furthermore, SET is 1.85×1.85\times slower per step and loses decisively at equal wall-clock time (3.8843.884 vs. 3.7383.738 NLL).
  • Unvalidated design choices: At n=256n=256, FFT is only 5.1%5.1\% faster than direct convolution with identical loss (Table 5). Moreover, standalone MLP mixing degrades the baseline, and the proposed depth-varying schedule loses to a fixed schedule across all RoPE seeds (Table 2).
Questions for authors
  1. Standard nanoGPT evaluation: Can you evaluate SET in the standard 124M nanoGPT setup [1] on OpenWebText [2] or FineWeb [3] (sequence length 1024) to provide a reproducible comparison on a standard pretraining corpus?
  2. Bidirectional and Vision benchmarks: Can SET be applied to bidirectional encoders like BERT [4] on GLUE [5] or to ViT [6] on ImageNet [7], where global or multi-resolution spectral filters are natural fits?
  3. Missing 760M/1B details and inverse scaling: What are the architectural specifications for the 760M and 1B models omitted from Table 9, what does the †\dagger symbol in Table 13 denote, and why does validation loss degrade catastrophically from 50M to 1B parameters across Tables 10, 12, and 13?
  4. ISO-FLOP comparison: Because convolutional kernels share weights across sequence positions, SET uses more FLOPs per parameter than a wider FFN. Does SET retain its 0.91%0.91\% RoPE gain when matched for training FLOPs?

References

[1] Karpathy, 2022. nanoGPT. https://github.com/karpathy/nanogpt.

[2] Gokaslan et al., 2019. OpenWebText Corpus. http://Skylion007.github.io/OpenWebTextCorpus.

[3] Penedo et al., 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024.

[4] Devlin et al., 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.

[5] Wang et al., 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR 2019.

[6] Dosovitskiy et al., 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.

[7] Deng et al., 2009. ImageNet: A Large-Scale Hierarchical Image Database. CVPR 2009.

[8] Su et al., 2024. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 2024.

2 comments

The primary controlled result is the complete parameter-matched 85M matrix rather than the historical scaling rows. In the controlled RoPE setting, full SET reduces mean validation NLL from 3.87901 to 3.84695, and all three paired seeds favor SET. Public WikiText-103 test NLL preserves the improvement, from 3.8835 to 3.8510.

The direction also holds in the submitted C4 pairs: SET lowers loss by 3.9%, 5.1%, 4.0%, and 2.9% at the reported 125M, 350M, 760M, and 1B configurations. The new kernel-1 control additionally shows that removing post-MLP cross-token mixing retains about 90.3% of full SET's improvement over the Transformer.

Thanks for your reply, I will keep my score. please consider broadening the experiments to other settings where transformers are typically evaluated: Image classification (ImageNet), Masked Language Modeling and downstream evals (GLUE) and Language Modeling (FineWeb).

Reviewer_n61zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a spectral enhanced Transformer to augment Transformer blocks with a learned multi-resolution causal convolutional path parallel to self-attention and causal token mixing after the MLP. However, after reading this paper, I found its motivation and method positioning are unclear and really hard to follow. Besides, the experiments are not solid.

Critical strengths

I must say I can not found any strength of this paper. The motivation is unclear, the proposed modules have not been explained clearly in either Abstract or Introduction, the experiments are far from solid.

Critical weaknesses
  1. The motivation of this paper is not clearly explained in the Abstract and Introduction.

  2. The method is hard to follow and the Figure 1 is not informative.

  3. The experiments only report the loss, without any other metric.

  4. This paper has only 16 references. Given transformer-based language modeling is a big and well-developed domain, I think this is not acceptable. It means authors have not conducted systematical literature review.

  5. There are no representative and SOTA methods included in the experiments.

Questions for authors
  1. Why only report the training loss in the experiments? Could you please also present other metrics in the domain of language modeling?

  2. What's the key advantage of the proposed spectral enhanced transformer?

2 comments

The reported metric is held-out validation NLL rather than training loss. Every entry in the 30-run matrix is measured at step 10000 on the same 247040 validation targets, with no best-checkpoint selection. Under RoPE, full SET obtains 3.84695 mean validation NLL compared with 3.87901 for the parameter-matched wider-FFN Transformer, and every paired seed favors SET.

We also evaluated the final checkpoints on 283136 public WikiText-103 test targets. Full SET obtains 3.8510 mean test NLL compared with 3.8835 for the Transformer, corresponding to test perplexities of 47.04 and 48.59. In the controlled validation matrix, the key advantage is improved held-out language modeling at the same parameter and training-token budget. The kernel-1 control further shows that removing post-MLP cross-token mixing retains about 90.3% of full SET's improvement over the Transformer.

After carefully reading your responses and other reviewers' comments, I think there are multiple flaws exist in the manuscript. For example, lack of SOTA baseline methods, limited literature review and insufficient experimental evaluation. Based on this, I will maintain my score.

Reviewer_q7SCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spectral Enhanced Transformer (SET), which augments Transformer blocks with a multi-resolution causal convolution branch parallel to self-attention and an additional causal convolutional gate after the MLP. Convolutions are evaluated using zero-padded FFTs to preserve autoregressive causality. A 30-run parameter-matched study on WikiText-103 shows lower validation NLL at equal training-token budgets, with reductions of 3.35% under learned positional embeddings and 0.83% under RoPE. However, SET takes approximately 1.85× the baseline step time and is outperformed by longer baseline training under the tested wall-clock comparison. Matched direct convolutions achieve nearly identical losses, while depth-varying kernel lengths show no consistent advantage over fixed lengths. The demonstrated contribution is therefore a finite-budget architectural improvement, rather than an efficiency or scaling advantage.

Critical strengths
  1. Full SET outperforms the parameter-matched baseline in every paired seed under both learned positional embeddings and RoPE. The 30-run component study controls parameter count and training tokens, providing evidence that the gains are not simply due to additional parameters.

  2. The paper compares matched FFT and direct convolutions, fixed and depth-varying kernel schedules, and training under matched wall-clock budgets. These controls clarify which benefits are supported and explicitly reveal the method’s computational limitations.

Critical weaknesses
  1. The post-MLP gate both mixes information across tokens and changes the weighting of the residual stream. The component ablations do not separate these effects, so it is unclear whether the additional gain comes specifically from temporal mixing (Section 5.3, Eq. 6; Table 2). This limits the explanation of why the second convolutional component helps, beyond showing that the full configuration performs better.

  2. The practical value and persistence of the improvement remain unclear. The main controlled experiments use one model configuration, sequence length 256, and a single budget of 61.44M training tokens, without establishing whether SET keeps its lead with longer training (Sections 6–7.1). Additional accuracy and completion results are reported, but they come from historical experiments, and completion accuracy does not improve consistently (Appendices F and H). SET also takes about 1.85× as long per training step, while the baseline achieves lower loss under the tested comparable training-time budget (Sections 5.4 and 7.3). The additional convolutional branches may also slow generation and require extra memory, but cached decoding costs are not systematically measured. Together, these gaps leave it unclear whether the quality gains justify the additional cost in deployment.

Questions for authors
  1. Have you tested a version that keeps the post-MLP gate but removes mixing across tokens, for example by using a kernel of length one? This would help distinguish the benefit of temporal mixing from changes to the residual stream.

  2. Do you have controlled results at larger training-token budgets? Does SET keep its lead when both models are trained for longer, or does the baseline catch up?

  3. How much do the extra convolutional branches affect inference speed and memory use? In particular, do you have measurements for generating tokens one at a time with caching?

  4. Appendices F and H report additional accuracy and completion results, but these come from historical experiments, and the completion results are mixed. Do you have comparable evaluations for the parameter-matched models in Table 2? This would help show whether the lower NLL also leads to better task performance or generation quality.

1 comment

We completed the requested kernel-1 control with three paired RoPE runs of 10000 steps and 61440000 training tokens each. Only the post-MLP temporal kernel is changed to length 1; the parallel spectral path, data order, optimizer, and all other settings are fixed. Kernel-1 validation NLL is 3.849173, 3.842461, and 3.857157 across seeds 604, 605, and 606, compared with 3.844272, 3.844629, and 3.846424 for full SET. The means are 3.849597 and 3.845108, while the Transformer mean is 3.891348. Removing post-MLP cross-token mixing therefore retains about 90.3% of full SET's improvement over the Transformer, identifying the parallel multi-resolution path as the main source of the gain.

For token-by-token generation, we implemented attention KV caches and convolution-history caches and verified them at sequence lengths 8, 32, and 128. At prompt 128 and decode 64, caching reduces SET from 20.292 to 14.019 ms/token and peak allocated memory from 464041984 to 440651264 bytes. The maximum cached-versus-full logit difference is 0.00003481, and an independent rerun reproduces the 1.447-fold speedup.

For the same final checkpoints, public WikiText-103 test NLL improves from 3.8835 to 3.8510. This confirms that the validation improvement persists on the held-out public test split.

Reviewer_UNZaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SET, which adds multi-resolution causal convolution parallel to self-attention and convolutional gating after the MLP, using zero-padded FFTs. The authors evaluate parameter-matched variants on WikiText-103 with learned absolute positions and RoPE, reporting lower validation loss for full SET at a fixed training-token budget.

Critical strengths

The article has made a bold attempt in the combined use of CNN and Transformer, and has conducted excellent ablation experiments and control experiments to demonstrate the participation degree of different modules.

Critical weaknesses
  1. I am concerned that adding a CNN with shared weights to an LLM may encourage memorization of sentence patterns. For example, certain positions within a fixed sentence pattern may systematically receive higher weights than others.

  2. This approach may be more effective for specific tasks. However, as the authors acknowledge, the reported results are insufficient to establish broad language-model capabilities.

  3. The paper also indicates that attention-based models can acquire similar knowledge with longer training and more training data.

Questions for authors
  1. How do you establish that the proposed architecture generalizes sufficiently across LLM applications, comparably to a standard Transformer? Additional zero-shot evaluations could help demonstrate the value of this work. Even without such evaluations, the architecture may still be valuable in specific settings, but this would suggest narrowing the scope of the claims.

  2. If conventional attention can achieve similar results with additional computation time, what advantages does your method offer over attention, particularly given the increased inference latency? Is the benefit simply a reduction in training time? I would appreciate clarification from the authors.

2 comments

SET does not learn separate weights for fixed sentence positions. Its causal kernels are shared across the sequence, so the same learned filter is applied at every position.

Its measured advantage is better language modeling at the same parameter and training-token budget, rather than merely a reduction in training time. Under RoPE, full SET reduces mean validation NLL from 3.87901 to 3.84695 relative to the parameter-matched wider-FFN Transformer, with every paired seed favoring SET. Public WikiText-103 test NLL also improves from 3.8835 to 3.8510. Within SET, incremental caching reduces decoding from 20.292 to 14.019 ms/token and peak allocated memory from 464041984 to 440651264 bytes, with a maximum cached-versus-full logit difference of 0.00003481.

Thank you to the authors for the clarifications regarding the translation-invariant nature of the shared causal kernels, as well as the detailed numbers on WikiText-103 NLL and incremental caching latency.

While I appreciate the rigorous controlled ablations and the encouraging perplexity improvements on WikiText-103, my main reservation remains the gap between the broad language modeling claims and the current empirical scope. Relying solely on WikiText-103 NLL, without broader zero-shot or downstream evaluations, is still insufficient to fully support general LLM capability claims. I believe the work has merit as an architectural exploration and would be much stronger if the claims were more carefully scoped to match the current experimental evidence. Therefore, I will respectfully maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.