acceptodds
Under review as a conference paper at ICLR 2027

A Spectral Perspective on Continuous Diffusion Language Models

Abstract

A spectral Fourier perspective provides a natural lens for understanding visual Diffusion Models. This view inspires a broad range of insights and practical methods. Unlike vision, the text domain does not naturally admit a spectral interpretation due to its discrete nature. However, the rise of Continuous Diffusion Language Models (DLMs), which perform diffusion in the representation space of pretrained text encoders, makes the spectral perspective more natural to investigate in the text domain. In this work, we study continuous DLMs through a Fourier lens. We show that the representation spaces of widely used text models, such as Qwen3.8, Gemma 4, and Nemotron-3, exhibit an approximate power-law spectral structure across sequence length, similar to that observed in images. Notably, this phenomenon persists across different model sizes, architectural families, data domains, training specializations, and instruction-tuning settings. We then analyze the information carried by different frequency bands and uncover a key distinction from vision: high-frequency components in text retain substantially more energy and encode important token-level, positional information, rather than merely fine-grained details. Motivated by these observations, we introduce a frequency-aware diffusion strategy that processes low- and high-frequency components separately across diffusion timesteps, enabling dedicated modeling of high frequencies. Across multiple tasks and representation spaces, this approach yields consistent improvements, with the gains tied to better preservation of positional relationships between tokens.

Primary area:
generative models
Keywords:
text generation, continuous diffusion models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xiemOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that the final-layer representation spaces of pre-trained text encoders, which serve as the diffusion space of continuous DLMs such as ELF, exhibit an approximate power-law spectrum along the sequence dimension that is much flatter than in images and is robust across model scale, family, data domain, training variant, and depth. Through token-distribution analysis, linear probes, reconstruction, and a token-order task, the authors argue that low frequencies encode global information and common tokens, while high frequencies additionally carry rarer tokens and fine-grained positional information. Motivated by this, they propose Haar Delay, a parameter-free scheme that splits representations into low- and high-frequency Haar components and delays denoising of the high-frequency band until a threshold t∗t^*, and report gains for ELF on GSM8K, mainly via fewer NameErrors, and in most settings on four other conditional-generation benchmarks.

Critical strengths
  • The paper is well motivated and clearly presented. It builds from a concrete empirical observation to a characterization of what the low and high frequency bands encode and to the spectral behavior of noisy states across timesteps, and this chain of observations leads naturally into the proposed Haar Delay. The “Key findings” box and the training/inference snippets make the method easy to follow.
  • The spectral observations are documented thoroughly across data and settings: the power-law structure is reported for web text, math, and code, for domain-specialized and instruction-tuned checkpoints, and across layers, and the band analyses are repeated for additional encoders in the appendix.
  • Results are reported across a range of sampling budgets, including the few-step regime, together with CFG and SD3-shift sweeps, matched numbers of denoiser evaluations, and per-task hyperparameters, which makes the comparison with the baseline transparent.
Critical weaknesses

W1. (Circularity of the band analysis.) Low-pass filtering along the sequence and then taking the final token is essentially sequence-wide pooling, while the high band is the final token minus its local average. That low frequencies capture global information and common tokens, and high frequencies capture position, thus follows from the filter definition rather than from text representations. Table 1 shows the confound: Full, which contains strictly more information, underperforms both bands on ListOps. The password-order probe likewise keeps too few low bins to resolve adjacent-token order by construction. The quantitative results are informative, but the interpretive claims in Key finding 2 overreach.

W2. (Dimension-mismatched comparison with vision.) Fig. 1 compares 2D radially averaged spectra of images and latents with 1D spectra of text. Under isotropy a 2D exponent α corresponds to a 1D exponent of α−1\alpha−1, which brings SD-VAE and FLUX latents to roughly −0.8 and −1.0. The gap to text is then small, weakening the claim of substantially more high-frequency energy in text.

W3. (Reliability of the reported gains.) Table 2 and the GSM8K results report no seeds or confidence intervals; most differences are within 1 point and four settings regress. Haar Delay is tuned over an extra hyperparameter and, in several Table 7 rows, a different µtrain, so search budgets are asymmetric. Fig. 6 reports validation examples, leaving it unclear whether selection and reporting share a split, and it is not stated whether the baseline received the extra denoiser evaluation.

W4. (Limited novelty and design justification.) Haar Delay is an adaptation of WaiT to the sequence dimension, so the contribution rests on the analysis; yet the analysis uses narrow Fourier bands while the method uses a one-level Haar split, with no ablation of the transform or decomposition level. Evaluation is confined to ELF and a single baseline, without cheaper alternatives such as a frequency-weighted loss, a shifted timestep distribution, or a two-stage sampler without the decomposition.

Questions for authors

Q1. Beyond the schedule parameters in Table 7 and the brief description in App. D, the training recipe is not specified. Could the authors provide a table, per encoder and task, listing the exact encoder checkpoint (e.g., which T5 variant), ELF architecture details, training data and splits, optimizer and learning rate, batch size, training iterations, and GPU type/count with total training hours? Please also state whether the Haar Delay runs use two denoiser forward passes per iteration (as Snippet 1 suggests), so that “the same training budget” can be interpreted as equal steps or equal compute, and why the T5 L0-Trace and xLAM entries are unavailable.

Q2 (W1). Can the authors disentangle frequency content from pooling, e.g., by comparing the band-limited final token against mean-pooled or smoothed full-spectrum representations in Table 1 and Fig. 2? Why does Full underperform both bands on ListOps?

Q3 (W2). Could the image and latent spectra be reported in a form comparable to the 1D text spectra (1D row/column spectra, or dimension-corrected exponents), and does the “substantially more high-frequency energy” claim hold under that comparison?

Q4 (W3). Please add seeds or confidence intervals to Table 2 and the GSM8K results, state which splits were used for selection and reporting, confirm whether the baseline received the extra denoiser evaluation, and report a baseline tuned with a matched search budget.

Q5 (W4). Can you ablate the transform (Fourier split vs. Haar, number of Haar levels) and compare against the simple alternatives above? Results on a second continuous DLM (e.g., TEncDM or LangFlow) would strengthen the generality claim.

5 comments
AuthorsOpenReview ↗

Reply [1/4]

We thank the reviewer for the thoughtful comments. We have updated the paper, with all changes marked in red, and provide our answers below.


W1. Low-pass filtering and taking the final token is essentially pooling, so the roles of low and high frequencies follow from the filter definition, and Key finding 2 overreaches.

Thanks for raising a meaningful point. We agree that the filter definition partly explains why low frequencies capture global information and common tokens, while high frequencies capture positional information. However, we claim that learned text representations also contribute to these properties. Below, we disentangle the influence of the filter from that of the text representations.

We use the same password-order probe as in the paper, where four marker tokens are placed at adjacent positions inside a random context in all 24 orders, and train linear probes on the low and high Fourier bands along tokens. We compare five representations of the same sequences, namely, the last hidden layer of the trained Qwen3.5-0.8B, its raw input embeddings, its last hidden layer with token positions shuffled by a fixed random permutation, the last hidden layer of a randomly initialized model with the same architecture, and i.i.d. random vectors assigned to each token without any model.

Token-order probe accuracy (%) for low and high frequencies of different representations

RepresentationLow band, acc. %High band, acc. %Diff (High − Low), pp
Qwen3.5, last layer76.3 ± 0.597.6 ± 0.2+21.2 ± 0.3
Raw input embeddings80.1 ± 1.099.7 ± 0.1+19.6 ± 1.0
Shuffled hidden states99.7 ± 0.199.6 ± 0.2−0.1 ± 0.3
Randomly initialized model99.6 ± 0.2100.0 ± 0.0+0.4 ± 0.2
Random vectors, no model98.5 ± 0.1100.0 ± 0.0+1.5 ± 0.1

High frequencies preserve the order for all representations, so this part is indeed a property of the filter. However, low frequencies fail only for representations learned from language, while for shuffled, randomly initialized, and random representations they also recover the order. Therefore, the advantage of high frequencies in positional information is not implied by the filter alone. It appears because language models assign similar vectors to related tokens, so swapping two adjacent similar tokens barely changes the low frequencies, and only high frequencies retain their order. We added this analysis to the revision (end of Section 3.2 and Appendix C, Table 7).

We agree that the claim in Key finding 2 is an overreach. The main goal of this claim is only to state that high frequencies contain more information about the input sequence in text than in images, and that this difference is specific to the representations rather than to the filter definition. We soften it in the revision (end of Section 3).


Q2. Disentangle frequency content from pooling, e.g., with smoothed full-spectrum representations. Why does Full underperform both bands on ListOps?

We add a smoothed full-spectrum representation. The smoothed representation performs on par with the low band on all tasks. This is expected, since a low-pass filter is by definition a local weighted average, so the low band cannot be separated from pooling.

Classification performance using different frequency components and smoothed full-spectrum representations of Qwen

ConfigurationLast layer: ListOpsLast layer: SentimentLast layer: RetrievalLast layer: TopicAvg. across layers: ListOpsAvg. across layers: SentimentAvg. across layers: RetrievalAvg. across layers: Topic
Random Classifier10.050.050.05.010.050.050.05.0
Full27.4 ± 0.489.4 ± 0.355.3 ± 0.356.0 ± 0.319.3 ± 0.183.4 ± 0.051.7 ± 0.335.2 ± 0.1
High Frequencies31.3 ± 2.378.2 ± 0.550.5 ± 0.930.0 ± 0.131.0 ± 0.473.5 ± 0.049.9 ± 0.617.2 ± 0.1
Low Frequencies32.6 ± 0.484.3 ± 0.254.7 ± 1.056.5 ± 0.431.8 ± 0.380.4 ± 0.152.3 ± 0.338.9 ± 0.1
Smoothed full spectrum33.5 ± 0.384.7 ± 0.155.2 ± 1.758.7 ± 0.733.1 ± 0.281.3 ± 0.152.1 ± 0.244.0 ± 0.0

On ListOps, Full underperforms because it uses only the last token, while low frequencies aggregate several neighboring tokens. We added this explanation and the smoothed row to Table 1 in the revision.

AuthorsOpenReview ↗

Reply [2/4]

W2, Q3. Image spectra are two-dimensional while text spectra are one-dimensional. Report comparable one-dimensional image spectra and check whether the high-frequency energy claim holds.

Thanks for raising this important concern. We now compute the spectra of images and image latents with exactly the same 1D estimator as for text, treating every image row and column as a sequence, and fit the power-law slope on a common frequency range.

The claim still holds, since text representations decay more slowly than every image representation. However, the gap is smaller. The slope of autoencoder latents is now about 2 times steeper than that of text, instead of about 3 times in Fig. 1. We slightly soften the claim in Section 3.1 and add the one-dimensional spectra to Appendix B (Figure 9).

1D power-law slope of the spectra of images, image latents and text representations on a common frequency range

Images1D slope βText1D slope β
RGB pixels−2.20GPT-2 Large−0.42
SD-VAE latents−0.79Gemma 4 31B−0.43
FLUX.1 VAE latents−0.98Qwen3.8 27B−0.52
RAE latents−1.34Nemotron-3 30B−0.43
Muse Glimmer 30B−0.39

Q1. Provide the full training recipe per encoder and task, including checkpoints, architecture, data and splits, optimizer, batch size, iterations, and compute.

We thank the reviewer for the suggestion. We added the full table to the revision (Appendix D, Table 8).

Training recipe and schedule parameters per encoder and task

EncoderTaskMax. lengthTraining examplesVal / test examplesEpochsTraining iterationsBaseline μ_trainBaseline shiftHaar Delay μ_trainHaar Delay t*Haar Delay shiftTime (h)
GPT-2 BaseL0-Trace864100,000512 / 2,00010320,085−2.004−1.750.1583.9
GPT-2 BasexLAM94458,218627 / 56017619,888−2.008−1.500.0884.3
GPT-2 BaseSudoku1968,819,6062,000 / 2,000234,450−3.002−2.750.0112.0
GPT-2 BaseWikiSQL14456,0272,000 / 2,00030032,700−1.758−1.250.0281.6
GPT-2 BaseGSM8K51211.78M512 / 1,31920460,120−1.7548−1.750.104832
T5Sudoku2568,819,6062,000 / 2,000234,450−2.252−2.250.02322.1
T5WikiSQL19245,4712,000 / 2,00022719,976−1.5032−1.500.02161.1
Qwen3.5L0-Trace784100,000512 / 2,00010320,085−2.5016−2.500.1245.5
Qwen3.5xLAM91258,230627 / 56017619,888−1.5032−1.250.04326.2
Qwen3.5Sudoku3408,819,6062,000 / 2,000234,450−3.0016−3.250.04165.4
Qwen3.5WikiSQL16055,8892,000 / 2,00030032,700−1.5016−1.500.01163.8
Qwen3.5GSM8K44811.78M512 / 1,31915345,195−1.7564−1.750.1548≈ 71

Do Haar Delay runs use two denoiser forward passes per iteration, as Snippet 1 suggests?

We apologize for the confusion here. Both the baseline and Haar runs have identical training times. The only change in the Haar setup is how inputs and targets are processed for different t, which adds no training overhead. Snippet 1 illustrates how the training step is performed for t < t* and t > t* within a batch, which might be confusing. We have updated Snippet 1 in the revision to resolve this.

Why are the T5 entries for L0-Trace and xLAM unavailable?

The T5 tokenizer does not support some characters in these datasets and either maps them to the unknown token or replaces them with spaces.

  • xLAM: all targets contain the unknown token because every target has { and }. T5-small maps {, }, <, ~, ^, \ and the backtick to the unknown token.
  • L0-Trace: the tokenizer replaces newlines and indentation with single spaces, and all checked prompts lose the indentation that defines the program blocks. For the same reasons, we do not consider T5 for TinyGSM.
AuthorsOpenReview ↗

Reply [3/4]

Q4 (W3). Add seeds or confidence intervals, state the selection and reporting splits, confirm whether the baseline received the extra denoiser evaluation, and report a baseline tuned with a matched budget.

We ran each experiment 3 times with different training seeds and additionally averaged the results over 3 sampling seeds. The updated results now report the mean and std (see the tables below and Table 2 in the revision). We also note that the baseline received the same hyperparameter grid search as the new model.

4 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.0 ± 0.161.3 ± 0.598.7 ± 0.154.7 ± 0.5
GPT-2Haar97.5 ± 0.263.1 ± 0.499.4 ± 0.155.2 ± 0.3
T5Baseline——95.1 ± 0.651.3 ± 0.2
T5Haar——98.6 ± 0.451.1 ± 0.3
Qwen3.5Baseline93.9 ± 1.227.9 ± 0.299.3 ± 0.147.2 ± 0.9
Qwen3.5Haar95.7 ± 0.929.1 ± 0.799.4 ± 0.146.7 ± 1.2

32 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.1 ± 0.164.4 ± 0.599.5 ± 0.155.5 ± 0.5
GPT-2Haar97.4 ± 0.165.1 ± 0.599.7 ± 0.156.5 ± 0.5
T5Baseline——97.6 ± 0.752.1 ± 0.3
T5Haar——99.0 ± 0.352.5 ± 0.4
Qwen3.5Baseline94.6 ± 0.633.1 ± 0.499.4 ± 0.148.3 ± 0.6
Qwen3.5Haar96.1 ± 0.737.8 ± 0.499.5 ± 0.147.8 ± 0.6

For each task, we use a separate validation set, described in the table below and in Appendix D, Table 8.

TaskL0-TracexLAMSudokuWikiSQLGSM8K
Val set512 dev examples627 internal dev examples2,000 dev puzzles2,000 filtered official-dev examples512 screened questions from GSM8K train

The results are already presented for matched training and inference budgets. We note that the Haar setup does not perform an additional step. During inference, we enable higher frequencies at the first timestep t > t* and do not perform an additional step at t*. We added this note to Appendix D and updated Snippet 2 accordingly.


Q5 (W4). Ablate the transform (Fourier split vs. Haar, number of Haar levels) and compare against simple alternatives.

Shifted timestep distribution. We note that we already isolate the effect of the shifted timestep distribution (both in training and sampling) by tuning it for the baseline and Haar models.

Two-stage sampler without the decomposition. The results below show that the two-stage sampler even underperforms the base model. The best two-stage setup splits 32 steps into 25 + 7 for GPT-2 and 64 steps into 54 + 10 for Qwen3.5.

ModelCoarse + refinement stepsBaseTwo-stageHaar
GPT-225 + 730.2 ± 1.629.8 ± 1.033.1 ± 0.8
Qwen3.554 + 1034.8 ± 0.834.2 ± 1.135.8 ± 1.0

Other suggestions. We thank the reviewer for the suggested approaches. We trained the models on TinyGSM with the frequency-weighted loss, with the Haar transform replaced by FFT, and with a two-level Haar transform. The results below show that the two-level Haar transform provides better results. Interestingly, FFT also shows significant gains compared to Haar.

Fourier split vs. Haar, number of Haar levels, frequency-weighted loss.

Model8 steps16 steps32 steps64 steps
Baseline15.5 ± 0.724.8 ± 0.730.2 ± 1.633.1 ± 1.7
Frequency-weighted loss16.8 ± 0.825.2 ± 1.430.2 ± 1.233.1 ± 0.8
Haar → FFT18.6 ± 0.629.8 ± 0.536.5 ± 0.638.7 ± 0.8
Two-level Haar18.0 ± 0.927.9 ± 0.535.3 ± 0.738.5 ± 0.5
Haar (main)17.1 ± 0.827.0 ± 0.833.1 ± 0.835.6 ± 0.4

Results on a second continuous DLM (e.g., TEncDM) would strengthen the generality claim.

In our work, we explore latent DLMs that operate in the latent space of pretrained text encoders. Most recent continuous DLMs either operate on text embeddings or one-hot encodings, or jointly train embeddings / latent representations along with the diffusion model. This is why we consider ELF, which represents a general flow-matching transformer, and explore it with different pretrained latent representations.

To show generalization to other latent diffusion model designs, we train the diffusion model from TEncDM on top of the GPT-2 latents and show the results below.

Model16 steps32 steps48 steps64 steps
TEncDM Baseline7.8 ± 0.310.2 ± 0.810.1 ± 0.710.2 ± 0.3
TEncDM Haar9.4 ± 0.512.3 ± 0.112.8 ± 0.212.9 ± 0.3

We added all ablations to the revision (Section 5 and Appendix D, Tables 10–12).

Reviewer_5d2mOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines continuous diffusion language models from a Fourier lens. The authors observe approximate power-law behavior in the textual representations of various text models and investigate how linguistic information is distributed across frequency bands. Their analyses suggest that high-frequency components are important for distinguishing token positions and reconstructing token order. Building on these observations, they apply Haar Delay, which uses separate denoising schedules for low- and high-frequency components. Evaluations across several benchmarks demonstrate the usefulness of this design.

Critical strengths

The paper presents a thorough and coherent investigation of the spectral properties of text representations. The findings are examined across different model families, sizes, layers, and data domains, while complementary probing and reconstruction experiments help explain the roles of individual frequency bands. In particular, the distinction between preserving token content and preserving its positional arrangement offers valuable insight into continuous language representations.

The work connects its analysis to a simple and effective modeling approach. Haar Delay introduces frequency-dependent generation schedules without adding model parameters, and its benefits are evaluated across multiple tasks, representation spaces, and sampling settings with matched denoiser evaluation budgets. The paper is also clearly written, making both the analytical findings and their practical implications accessible to researchers working on continuous DLMs.

Critical weaknesses

Further evidence would help clarify both the source of the performance gains and their broader applicability. The reduction in NameError cases suggests better consistency across tokens, although it does not by itself demonstrate improved modeling of positional relationships. Measuring denoising errors within individual frequency bands or evaluating carefully controlled token dependencies would strengthen this interpretation. The generation results are also limited to the ELF framework and mostly structured conditional tasks, while estimates of variability across runs would make the smaller gains easier to assess.

Questions for authors

Can the authors provide direct evidence linking improved high-frequency reconstruction to better token ordering or variable consistency?

How reproducible are the reported gains across training seeds, particularly for settings with small improvements?

2 comments

We thank the reviewer for the thoughtful comments. We have updated the paper, with all changes marked in red, and provide our answers below.


W1, Q1. Provide direct evidence linking improved high-frequency reconstruction to better token ordering, e.g., band-wise denoising errors or controlled token dependencies.

Thanks for the important suggestion. We directly verify that our approach improves high-frequency reconstruction. To this end, we consider two tasks.

First, we compare the high-frequency reconstruction error of the Haar Delay approach and the baseline and find that our approach indeed models high frequencies better.

Specifically, we noise the latents to the same high-frequency noise level t for both models, starting from t*, where Haar Delay begins to generate high frequencies. We then measure the relative error of the denoised latent in the highest FFT band, Et=∥H(x^t−x)∥2/∥Hx∥2E_t = \|H(\hat{x}_t - x)\|^2 / \|Hx\|^2, where H is the projection onto this band. The table below shows that Haar Delay reduces this error at high and medium noise levels (t = 0.1–0.4) by 9–68% for GPT-2 and by 9–78% for Qwen3.5.

High-frequency reconstruction error at different noise levels

tGPT-2 Baseline, errorGPT-2 Haar, errorGPT-2 Diff, relative %Qwen3.5 Baseline, errorQwen3.5 Haar, errorQwen3.5 Diff, relative %
0.10.09080.0294−67.6% ± 0.70.09990.0221−77.9% ± 0.5
0.20.01640.0118−27.9% ± 0.80.01690.0107−36.6% ± 0.9
0.30.00770.0065−15.4% ± 1.00.00810.0068−15.8% ± 0.7
0.40.00420.0038−8.6% ± 1.70.00500.0046−8.7% ± 0.9
0.50.00200.0021+3.1% ± 3.60.00310.0029−7.0% ± 1.1

Second, we perform the token-order probe. We insert four numbers in all 24 possible orders into the sequences, noise the latents as above, and denoise them with each model. We then keep only the highest FFT band of the denoised latent and train a linear classifier on the four number positions to predict their order.

Haar Delay achieves higher accuracy than the baseline for both encoders. At low noise levels, both models approach their clean-input accuracy, and the differences are small.

Token-order probe accuracy (%), high FFT band

tGPT-2 Baseline, acc. %GPT-2 Haar, acc. %GPT-2 Diff, ppQwen3.5 Baseline, acc. %Qwen3.5 Haar, acc. %Qwen3.5 Diff, pp
0.37.512.0+4.5 ± 0.75.67.6+1.9 ± 0.5
0.421.530.7+9.2 ± 1.18.811.7+2.9 ± 0.6
0.549.859.7+9.9 ± 1.116.119.2+3.1 ± 0.6
0.681.085.6+4.6 ± 1.328.731.0+2.3 ± 1.2
0.796.197.1+1.1 ± 0.450.149.2−1.0 ± 1.4
0.899.499.30.0 ± 0.172.369.3−3.0 ± 1.3

Together, these results show that Haar Delay models high frequencies better. We added both results to the revision (Section 5, Figure 6, and Appendix D, Table 9).


W2, Q2. The results are limited to ELF and do not report variability across runs. How reproducible are the gains across training seeds?

We report the results averaged over 3 training seeds and 3 sampling seeds in the table below (Table 2 in the revision). Haar Delay improves upon or performs on par with the tuned baseline in all 10 settings at both 4 and 32 steps. Beyond ELF, Haar Delay also improves TEncDM, a different latent diffusion design (Appendix D, Table 12 in the revision).

4 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.0 ± 0.161.3 ± 0.598.7 ± 0.154.7 ± 0.5
GPT-2Haar97.6 ± 0.163.1 ± 0.499.4 ± 0.155.2 ± 0.3
T5Baseline——95.1 ± 0.651.3 ± 0.2
T5Haar——98.6 ± 0.451.1 ± 0.3
Qwen3.5Baseline93.9 ± 1.227.9 ± 0.299.3 ± 0.147.2 ± 0.9
Qwen3.5Haar95.7 ± 0.929.1 ± 0.799.4 ± 0.146.7 ± 1.2

32 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.1 ± 0.164.4 ± 0.599.5 ± 0.155.5 ± 0.5
GPT-2Haar97.5 ± 0.165.1 ± 0.599.7 ± 0.156.5 ± 0.5
T5Baseline——97.6 ± 0.752.1 ± 0.3
T5Haar——99.0 ± 0.352.5 ± 0.4
Qwen3.5Baseline94.6 ± 0.633.1 ± 0.499.4 ± 0.148.3 ± 0.6
Qwen3.5Haar96.1 ± 0.737.8 ± 0.499.5 ± 0.147.8 ± 0.6

We hope that our answers address the reviewer's concerns. If any questions remain, we would be happy to discuss them further.

Reviewer_5d2mOpenReview ↗

RE: Rebuttal

Thank you for the detailed responses and additional experiments. The analyses of high-frequency reconstruction and token ordering, together with the results across multiple seeds, adequately address my questions. I have no further comments and will maintain my positive rating.

Reviewer_yM1QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continuous diffusion language models by analyzing the frequency characteristics of their contextual text representations. Analysis of these frequency bands reveals that low frequencies primarily capture global context, while high frequencies encode specific lexical details and precise token locations. Motivated by the observation that high-frequency signal becomes distinguishable from noise later in the diffusion trajectory, the authors propose Haar Delay, which separates low- and high-frequency components and delays high-frequency denoising until a later stage. The method improves performance in several ELF-based continuous DLM settings.

Critical strengths
  • The analysis spans multiple model families, scales, domains, and both encoder- and decoder-style text models.
  • Through reconstruction tests, the authors prove that high-frequency signals are not merely noise, especially they carry information related to exact token placement and ordering.
  • Haar Delay adds no model parameters and yields nontrivial improvements in some settings, including larger gains on GSM8K with the larger ELF-M model.
Critical weaknesses
  • The paper establishes that Fourier high-frequency components contain useful positional information and separately shows that high-frequency signal becomes distinguishable from noise later in the diffusion process. This motivates treating frequency bands differently over time. The study fails to directly prove that Haar Delay actually enhances the high-frequency information identified in the analysis. It lacks evaluations using frequency-specific reconstruction errors or the previously introduced position probes. There is also a representation mismatch, most of the analysis uses Fourier frequency bands, whereas the intervention uses a one-level Haar decomposition.

  • Because the transform operates along the token-position axis, high-frequency components mathematically retain local variations by default. The observed positional information might simply be a mathematical artifact rather than a learned linguistic feature. Controls using raw embeddings, shuffled hidden-state sequences, randomly initialized models, or synthetic representations with matched local statistics would help determine how much of the effect is specifically learned by the language model.

  • Baseline and Haar Delay are tuned independently, and in multiple settings they use different training-time noise schedules and inference configurations. This makes it impossible to isolate the specific impact of the Haar Delay intervention. A matched ablation keeping the training schedule, inference schedule, CFG, architecture, and optimization settings fixed while toggling only Haar Delay would provide a cleaner test of the method itself.

  • Many reported performance gains are marginal (fractions of a percent), yet the main generation results do not report variance across independent training runs. Without training-run uncertainty, it is difficult to judge which of the smaller improvements are meaningful.

Questions for authors
  1. Can the authors compare baseline and Haar Delay using frequency-specific reconstruction error and the token-order/position probes introduced earlier in the paper? Does Haar Delay demonstrably improve the high-frequency information that motivates the method?
  2. Given that the foundational analysis relies heavily on Fourier space, what is the justification for using a one-level Haar decomposition for the intervention?
  3. Does the high-frequency advantage for exact token ordering remain for raw embeddings, shuffled hidden states, or random/synthetic representations? This would help distinguish learned structure from a generic consequence of high-pass filtering along sequence position.
  4. If the training noise schedule, inference settings, CFG, and all other hyperparameters are held strictly constant, does Haar Delay still outperform the baseline?
  5. Can the authors report results across independent training runs for the main generation experiments, particularly for the settings where gains are small?
5 comments
AuthorsOpenReview ↗

Reply [1/3]

We thank the reviewer for the thoughtful comments. We have updated the paper, with all changes marked in red, and provide our answers below.


Q1, W1. The paper does not directly show that Haar Delay improves the high-frequency information identified in the analysis. Compare the baseline and Haar Delay with frequency-specific reconstruction error and the token-order probe.

Thanks for the good suggestion. First, we compare the high-frequency reconstruction error of the Haar Delay approach and the baseline and find that our approach indeed models high frequencies better.

Specifically, we noise the latents to the same high-frequency noise level t for both models, starting from t*, where Haar Delay begins to generate high frequencies. We then measure the relative error of the denoised latent in the highest FFT band, Et=∥H(x^t−x)∥2/∥Hx∥2E_t = \|H(\hat{x}_t - x)\|^2 / \|Hx\|^2, where H is the projection onto this band. The table below shows that Haar Delay reduces this error at high and medium noise levels (t = 0.1–0.4) by 9–68% for GPT-2 and by 9–78% for Qwen3.5.

High-frequency reconstruction error at different noise levels

tGPT-2 Baseline, errorGPT-2 Haar, errorGPT-2 Diff, relative %Qwen3.5 Baseline, errorQwen3.5 Haar, errorQwen3.5 Diff, relative %
0.10.09080.0294−67.6% ± 0.70.09990.0221−77.9% ± 0.5
0.20.01640.0118−27.9% ± 0.80.01690.0107−36.6% ± 0.9
0.30.00770.0065−15.4% ± 1.00.00810.0068−15.8% ± 0.7
0.40.00420.0038−8.6% ± 1.70.00500.0046−8.7% ± 0.9
0.50.00200.0021+3.1% ± 3.60.00310.0029−7.0% ± 1.1

Second, we perform the token-order probe. We insert four numbers in all 24 possible orders into the sequences, noise the latents as above, and denoise them with each model. We then keep only the highest FFT band of the denoised latent and train a linear classifier on the four number positions to predict their order.

Haar Delay achieves higher accuracy than the baseline for both encoders. At low noise levels, both models approach their clean-input accuracy, and the differences are small.

Token-order probe accuracy (%), high FFT band

tGPT-2 Baseline, acc. %GPT-2 Haar, acc. %GPT-2 Diff, ppQwen3.5 Baseline, acc. %Qwen3.5 Haar, acc. %Qwen3.5 Diff, pp
0.37.512.0+4.5 ± 0.75.67.6+1.9 ± 0.5
0.421.530.7+9.2 ± 1.18.811.7+2.9 ± 0.6
0.549.859.7+9.9 ± 1.116.119.2+3.1 ± 0.6
0.681.085.6+4.6 ± 1.328.731.0+2.3 ± 1.2
0.796.197.1+1.1 ± 0.450.149.2−1.0 ± 1.4
0.899.499.30.0 ± 0.172.369.3−3.0 ± 1.3

Together, these results show that Haar Delay models high frequencies better. We added both results to the revision (Section 5, Figure 6, and Appendix D, Table 9).


W2. The analysis relies on Fourier space, so why use a one-level Haar decomposition for the intervention?

The one-level Haar decomposition is a simple two-band frequency filter. Its low- and high-frequency parts keep a cos⁡2(πf)\cos^2(\pi f) and sin⁡2(πf)\sin^2(\pi f) share of each Fourier frequency f, so they roughly correspond to the lower and upper halves of the Fourier spectrum analyzed in the paper.

We prefer it over an explicit Fourier mask for practical reasons. Haar operates on neighboring token pairs, so it is cheap, exactly invertible, and does not depend on the sequence length. In contrast, the Fourier transform is global, so every token depends on the whole sequence, and its frequency bands change with the sequence length. In the revision, we also compare the one-level Haar transform with a two-level Haar transform and a Fourier split (Section 5 and Appendix D, Table 10).

AuthorsOpenReview ↗

Reply [2/3]

W2, Q3. High frequencies along tokens retain local variations by definition, so the positional information may be a mathematical artifact. Does the advantage remain for raw embeddings, shuffled hidden states, or random representations?

Thanks for raising an important concern. We test whether the advantage of high frequencies in token ordering remains for raw embeddings, shuffled hidden states, or random/synthetic representations, and find an interesting result.

We use the same password-order probe as in the paper, where four marker tokens are placed at adjacent positions inside a random context in all 24 orders, and train linear probes on the low and high Fourier bands along tokens. We compare five representations of the same sequences, namely, the last hidden layer of the trained Qwen3.5-0.8B, its raw input embeddings, its last hidden layer with token positions shuffled by a fixed random permutation, the last hidden layer of a randomly initialized model with the same architecture, and i.i.d. random vectors assigned to each token without any model.

The table below shows that high frequencies preserve the order for all representations. However, the advantage of high over low frequencies appears only for representations learned from language (the trained model and its raw input embeddings). Interestingly, it vanishes for shuffled, randomly initialized, and random representations, where low frequencies also successfully recover the order.

Token-order probe accuracy (%) for low and high frequencies of different representations

RepresentationLow band, acc. %High band, acc. %Diff (High − Low), pp
Qwen3.5, last layer76.3 ± 0.597.6 ± 0.2+21.2 ± 0.3
Raw input embeddings80.1 ± 1.099.7 ± 0.1+19.6 ± 1.0
Shuffled hidden states99.7 ± 0.199.6 ± 0.2−0.1 ± 0.3
Randomly initialized model99.6 ± 0.2100.0 ± 0.0+0.4 ± 0.2
Random vectors, no model98.5 ± 0.1100.0 ± 0.0+1.5 ± 0.1

Language models learn that tokens used in similar contexts have similar representations. For example, the four color markers in our probe have an average cosine similarity of 0.49 in the trained embeddings, compared to 0.02 for random vectors. Low frequencies average neighboring tokens, so when similar tokens stand next to each other, swapping them barely changes the average and their order is lost. That is why learned representations show the worst performance in the low band. In contrast, random vectors and the randomly initialized model assign nearly orthogonal vectors to the markers, so swapping them substantially changes the low-frequency average. High frequencies capture the differences between neighboring tokens and therefore distinguish the order in all cases.

Therefore, the advantage of high frequencies in positional information has two sources. The first is the Fourier transform itself, since high frequencies capture differences between neighboring tokens and thus retain local order for any representation. This is not an artifact but the expected behavior of high frequencies. The second is a learned language feature, namely the similarity between related tokens, which makes low frequencies lose the order of adjacent similar tokens. It is the combination of both that makes high frequencies essential for positional information in language representations and explains the gap between low and high frequencies reported in the paper. We added this analysis to the revision (end of Section 3.2 and Appendix C, Table 7).

AuthorsOpenReview ↗

Reply [3/3]

W3, Q4. Baseline and Haar Delay are tuned independently. Does Haar Delay still outperform the baseline with identical hyperparameters?

We believe that tuning the baseline and the proposed approach separately is important to isolate the gains of the method from the effect of shifting other parameters. Without tuning, we would not know whether Haar’s gains could simply be explained by adapting the time schedules or other standard hyperparameters.

Nevertheless, below we provide the GPT-2 and Qwen3.5 results on TinyGSM under identical standard hyperparameters (Appendix D, Table 11 in the revision). In this setup, Haar provides much higher gains across all steps. We also see that the baseline gains more from hyperparameter tuning, suggesting that the Haar setup might require a smaller hyperparameter search budget in practice.

GPT-2

HyperparametersMethod8 steps16 steps32 steps64 steps
StandardBaseline11.8 ± 0.619.2 ± 0.225.3 ± 0.729.4 ± 0.5
StandardHaar17.3 ± 0.724.9 ± 0.531.6 ± 0.234.6 ± 1.4
TunedBaseline15.5 ± 0.724.8 ± 0.730.2 ± 1.633.1 ± 1.7
TunedHaar17.1 ± 0.827.0 ± 0.833.1 ± 0.835.6 ± 0.4

Qwen3.5

HyperparametersMethod8 steps16 steps32 steps64 steps
StandardBaseline13.6 ± 0.921.3 ± 0.226.8 ± 0.730.3 ± 0.3
StandardHaar18.5 ± 0.426.0 ± 1.231.1 ± 0.432.1 ± 1.0
TunedBaseline19.3 ± 0.530.3 ± 0.334.6 ± 0.434.8 ± 0.8
TunedHaar22.3 ± 0.332.2 ± 0.934.7 ± 0.735.8 ± 1.0

W4, Q5. Many gains are marginal, and the main results do not report variance across training runs.

We thank the reviewer for the suggestion. Below, we report the results for each setup in Table 2, averaged over 3 independent runs and 3 sampling seeds. Haar Delay improves upon or performs on par with the tuned baseline in all 10 settings at both 4 and 32 steps.

4 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.0 ± 0.161.3 ± 0.598.7 ± 0.154.7 ± 0.5
GPT-2Haar97.5 ± 0.263.1 ± 0.499.4 ± 0.155.2 ± 0.3
T5Baseline——95.1 ± 0.651.3 ± 0.2
T5Haar——98.6 ± 0.451.1 ± 0.3
Qwen3.5Baseline93.9 ± 1.227.9 ± 0.299.3 ± 0.147.2 ± 0.9
Qwen3.5Haar95.7 ± 0.929.1 ± 0.799.4 ± 0.146.7 ± 1.2

32 generation steps

ModelMethodL0-TracexLAMSudokuWikiSQL
GPT-2Baseline97.1 ± 0.164.4 ± 0.599.5 ± 0.155.5 ± 0.5
GPT-2Haar97.4 ± 0.165.1 ± 0.599.7 ± 0.156.5 ± 0.5
T5Baseline——97.6 ± 0.752.1 ± 0.3
T5Haar——99.0 ± 0.352.5 ± 0.4
Qwen3.5Baseline94.6 ± 0.633.1 ± 0.499.4 ± 0.148.3 ± 0.6
Qwen3.5Haar96.1 ± 0.737.8 ± 0.499.5 ± 0.147.8 ± 0.6

We hope that our answers address the reviewer's concerns. If any questions remain, we would be happy to discuss them further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.