- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that the final-layer representation spaces of pre-trained text encoders, which serve as the diffusion space of continuous DLMs such as ELF, exhibit an approximate power-law spectrum along the sequence dimension that is much flatter than in images and is robust across model scale, family, data domain, training variant, and depth. Through token-distribution analysis, linear probes, reconstruction, and a token-order task, the authors argue that low frequencies encode global information and common tokens, while high frequencies additionally carry rarer tokens and fine-grained positional information. Motivated by this, they propose Haar Delay, a parameter-free scheme that splits representations into low- and high-frequency Haar components and delays denoising of the high-frequency band until a threshold , and report gains for ELF on GSM8K, mainly via fewer NameErrors, and in most settings on four other conditional-generation benchmarks.
Critical strengths
- The paper is well motivated and clearly presented. It builds from a concrete empirical observation to a characterization of what the low and high frequency bands encode and to the spectral behavior of noisy states across timesteps, and this chain of observations leads naturally into the proposed Haar Delay. The “Key findings” box and the training/inference snippets make the method easy to follow.
- The spectral observations are documented thoroughly across data and settings: the power-law structure is reported for web text, math, and code, for domain-specialized and instruction-tuned checkpoints, and across layers, and the band analyses are repeated for additional encoders in the appendix.
- Results are reported across a range of sampling budgets, including the few-step regime, together with CFG and SD3-shift sweeps, matched numbers of denoiser evaluations, and per-task hyperparameters, which makes the comparison with the baseline transparent.
Critical weaknesses
W1. (Circularity of the band analysis.) Low-pass filtering along the sequence and then taking the final token is essentially sequence-wide pooling, while the high band is the final token minus its local average. That low frequencies capture global information and common tokens, and high frequencies capture position, thus follows from the filter definition rather than from text representations. Table 1 shows the confound: Full, which contains strictly more information, underperforms both bands on ListOps. The password-order probe likewise keeps too few low bins to resolve adjacent-token order by construction. The quantitative results are informative, but the interpretive claims in Key finding 2 overreach.
W2. (Dimension-mismatched comparison with vision.) Fig. 1 compares 2D radially averaged spectra of images and latents with 1D spectra of text. Under isotropy a 2D exponent α corresponds to a 1D exponent of , which brings SD-VAE and FLUX latents to roughly −0.8 and −1.0. The gap to text is then small, weakening the claim of substantially more high-frequency energy in text.
W3. (Reliability of the reported gains.) Table 2 and the GSM8K results report no seeds or confidence intervals; most differences are within 1 point and four settings regress. Haar Delay is tuned over an extra hyperparameter and, in several Table 7 rows, a different µtrain, so search budgets are asymmetric. Fig. 6 reports validation examples, leaving it unclear whether selection and reporting share a split, and it is not stated whether the baseline received the extra denoiser evaluation.
W4. (Limited novelty and design justification.) Haar Delay is an adaptation of WaiT to the sequence dimension, so the contribution rests on the analysis; yet the analysis uses narrow Fourier bands while the method uses a one-level Haar split, with no ablation of the transform or decomposition level. Evaluation is confined to ELF and a single baseline, without cheaper alternatives such as a frequency-weighted loss, a shifted timestep distribution, or a two-stage sampler without the decomposition.
Questions for authors
Q1. Beyond the schedule parameters in Table 7 and the brief description in App. D, the training recipe is not specified. Could the authors provide a table, per encoder and task, listing the exact encoder checkpoint (e.g., which T5 variant), ELF architecture details, training data and splits, optimizer and learning rate, batch size, training iterations, and GPU type/count with total training hours? Please also state whether the Haar Delay runs use two denoiser forward passes per iteration (as Snippet 1 suggests), so that “the same training budget” can be interpreted as equal steps or equal compute, and why the T5 L0-Trace and xLAM entries are unavailable.
Q2 (W1). Can the authors disentangle frequency content from pooling, e.g., by comparing the band-limited final token against mean-pooled or smoothed full-spectrum representations in Table 1 and Fig. 2? Why does Full underperform both bands on ListOps?
Q3 (W2). Could the image and latent spectra be reported in a form comparable to the 1D text spectra (1D row/column spectra, or dimension-corrected exponents), and does the “substantially more high-frequency energy” claim hold under that comparison?
Q4 (W3). Please add seeds or confidence intervals to Table 2 and the GSM8K results, state which splits were used for selection and reporting, confirm whether the baseline received the extra denoiser evaluation, and report a baseline tuned with a matched search budget.
Q5 (W4). Can you ablate the transform (Fourier split vs. Haar, number of Haar levels) and compare against the simple alternatives above? Results on a second continuous DLM (e.g., TEncDM or LangFlow) would strengthen the generality claim.
5 comments
Reply [2/4]
W2, Q3. Image spectra are two-dimensional while text spectra are one-dimensional. Report comparable one-dimensional image spectra and check whether the high-frequency energy claim holds.
Thanks for raising this important concern. We now compute the spectra of images and image latents with exactly the same 1D estimator as for text, treating every image row and column as a sequence, and fit the power-law slope on a common frequency range.
The claim still holds, since text representations decay more slowly than every image representation. However, the gap is smaller. The slope of autoencoder latents is now about 2 times steeper than that of text, instead of about 3 times in Fig. 1. We slightly soften the claim in Section 3.1 and add the one-dimensional spectra to Appendix B (Figure 9).
1D power-law slope of the spectra of images, image latents and text representations on a common frequency range
| Images | 1D slope β | Text | 1D slope β |
|---|---|---|---|
| RGB pixels | −2.20 | GPT-2 Large | −0.42 |
| SD-VAE latents | −0.79 | Gemma 4 31B | −0.43 |
| FLUX.1 VAE latents | −0.98 | Qwen3.8 27B | −0.52 |
| RAE latents | −1.34 | Nemotron-3 30B | −0.43 |
| Muse Glimmer 30B | −0.39 |
Q1. Provide the full training recipe per encoder and task, including checkpoints, architecture, data and splits, optimizer, batch size, iterations, and compute.
We thank the reviewer for the suggestion. We added the full table to the revision (Appendix D, Table 8).
Training recipe and schedule parameters per encoder and task
| Encoder | Task | Max. length | Training examples | Val / test examples | Epochs | Training iterations | Baseline μ_train | Baseline shift | Haar Delay μ_train | Haar Delay t* | Haar Delay shift | Time (h) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-2 Base | L0-Trace | 864 | 100,000 | 512 / 2,000 | 103 | 20,085 | −2.00 | 4 | −1.75 | 0.15 | 8 | 3.9 |
| GPT-2 Base | xLAM | 944 | 58,218 | 627 / 560 | 176 | 19,888 | −2.00 | 8 | −1.50 | 0.08 | 8 | 4.3 |
| GPT-2 Base | Sudoku | 196 | 8,819,606 | 2,000 / 2,000 | 2 | 34,450 | −3.00 | 2 | −2.75 | 0.01 | 1 | 2.0 |
| GPT-2 Base | WikiSQL | 144 | 56,027 | 2,000 / 2,000 | 300 | 32,700 | −1.75 | 8 | −1.25 | 0.02 | 8 | 1.6 |
| GPT-2 Base | GSM8K | 512 | 11.78M | 512 / 1,319 | 20 | 460,120 | −1.75 | 48 | −1.75 | 0.10 | 48 | 32 |
| T5 | Sudoku | 256 | 8,819,606 | 2,000 / 2,000 | 2 | 34,450 | −2.25 | 2 | −2.25 | 0.02 | 32 | 2.1 |
| T5 | WikiSQL | 192 | 45,471 | 2,000 / 2,000 | 227 | 19,976 | −1.50 | 32 | −1.50 | 0.02 | 16 | 1.1 |
| Qwen3.5 | L0-Trace | 784 | 100,000 | 512 / 2,000 | 103 | 20,085 | −2.50 | 16 | −2.50 | 0.12 | 4 | 5.5 |
| Qwen3.5 | xLAM | 912 | 58,230 | 627 / 560 | 176 | 19,888 | −1.50 | 32 | −1.25 | 0.04 | 32 | 6.2 |
| Qwen3.5 | Sudoku | 340 | 8,819,606 | 2,000 / 2,000 | 2 | 34,450 | −3.00 | 16 | −3.25 | 0.04 | 16 | 5.4 |
| Qwen3.5 | WikiSQL | 160 | 55,889 | 2,000 / 2,000 | 300 | 32,700 | −1.50 | 16 | −1.50 | 0.01 | 16 | 3.8 |
| Qwen3.5 | GSM8K | 448 | 11.78M | 512 / 1,319 | 15 | 345,195 | −1.75 | 64 | −1.75 | 0.15 | 48 | ≈ 71 |
Do Haar Delay runs use two denoiser forward passes per iteration, as Snippet 1 suggests?
We apologize for the confusion here. Both the baseline and Haar runs have identical training times. The only change in the Haar setup is how inputs and targets are processed for different t, which adds no training overhead. Snippet 1 illustrates how the training step is performed for t < t* and t > t* within a batch, which might be confusing. We have updated Snippet 1 in the revision to resolve this.
Why are the T5 entries for L0-Trace and xLAM unavailable?
The T5 tokenizer does not support some characters in these datasets and either maps them to the unknown token or replaces them with spaces.
- xLAM: all targets contain the unknown token because every target has
{and}. T5-small maps{,},<,~,^,\and the backtick to the unknown token. - L0-Trace: the tokenizer replaces newlines and indentation with single spaces, and all checked prompts lose the indentation that defines the program blocks. For the same reasons, we do not consider T5 for TinyGSM.
Reply [3/4]
Q4 (W3). Add seeds or confidence intervals, state the selection and reporting splits, confirm whether the baseline received the extra denoiser evaluation, and report a baseline tuned with a matched budget.
We ran each experiment 3 times with different training seeds and additionally averaged the results over 3 sampling seeds. The updated results now report the mean and std (see the tables below and Table 2 in the revision). We also note that the baseline received the same hyperparameter grid search as the new model.
4 generation steps
| Model | Method | L0-Trace | xLAM | Sudoku | WikiSQL |
|---|---|---|---|---|---|
| GPT-2 | Baseline | 97.0 ± 0.1 | 61.3 ± 0.5 | 98.7 ± 0.1 | 54.7 ± 0.5 |
| GPT-2 | Haar | 97.5 ± 0.2 | 63.1 ± 0.4 | 99.4 ± 0.1 | 55.2 ± 0.3 |
| T5 | Baseline | — | — | 95.1 ± 0.6 | 51.3 ± 0.2 |
| T5 | Haar | — | — | 98.6 ± 0.4 | 51.1 ± 0.3 |
| Qwen3.5 | Baseline | 93.9 ± 1.2 | 27.9 ± 0.2 | 99.3 ± 0.1 | 47.2 ± 0.9 |
| Qwen3.5 | Haar | 95.7 ± 0.9 | 29.1 ± 0.7 | 99.4 ± 0.1 | 46.7 ± 1.2 |
32 generation steps
| Model | Method | L0-Trace | xLAM | Sudoku | WikiSQL |
|---|---|---|---|---|---|
| GPT-2 | Baseline | 97.1 ± 0.1 | 64.4 ± 0.5 | 99.5 ± 0.1 | 55.5 ± 0.5 |
| GPT-2 | Haar | 97.4 ± 0.1 | 65.1 ± 0.5 | 99.7 ± 0.1 | 56.5 ± 0.5 |
| T5 | Baseline | — | — | 97.6 ± 0.7 | 52.1 ± 0.3 |
| T5 | Haar | — | — | 99.0 ± 0.3 | 52.5 ± 0.4 |
| Qwen3.5 | Baseline | 94.6 ± 0.6 | 33.1 ± 0.4 | 99.4 ± 0.1 | 48.3 ± 0.6 |
| Qwen3.5 | Haar | 96.1 ± 0.7 | 37.8 ± 0.4 | 99.5 ± 0.1 | 47.8 ± 0.6 |
For each task, we use a separate validation set, described in the table below and in Appendix D, Table 8.
| Task | L0-Trace | xLAM | Sudoku | WikiSQL | GSM8K |
|---|---|---|---|---|---|
| Val set | 512 dev examples | 627 internal dev examples | 2,000 dev puzzles | 2,000 filtered official-dev examples | 512 screened questions from GSM8K train |
The results are already presented for matched training and inference budgets. We note that the Haar setup does not perform an additional step. During inference, we enable higher frequencies at the first timestep t > t* and do not perform an additional step at t*. We added this note to Appendix D and updated Snippet 2 accordingly.
Q5 (W4). Ablate the transform (Fourier split vs. Haar, number of Haar levels) and compare against simple alternatives.
Shifted timestep distribution. We note that we already isolate the effect of the shifted timestep distribution (both in training and sampling) by tuning it for the baseline and Haar models.
Two-stage sampler without the decomposition. The results below show that the two-stage sampler even underperforms the base model. The best two-stage setup splits 32 steps into 25 + 7 for GPT-2 and 64 steps into 54 + 10 for Qwen3.5.
| Model | Coarse + refinement steps | Base | Two-stage | Haar |
|---|---|---|---|---|
| GPT-2 | 25 + 7 | 30.2 ± 1.6 | 29.8 ± 1.0 | 33.1 ± 0.8 |
| Qwen3.5 | 54 + 10 | 34.8 ± 0.8 | 34.2 ± 1.1 | 35.8 ± 1.0 |
Other suggestions. We thank the reviewer for the suggested approaches. We trained the models on TinyGSM with the frequency-weighted loss, with the Haar transform replaced by FFT, and with a two-level Haar transform. The results below show that the two-level Haar transform provides better results. Interestingly, FFT also shows significant gains compared to Haar.
Fourier split vs. Haar, number of Haar levels, frequency-weighted loss.
| Model | 8 steps | 16 steps | 32 steps | 64 steps |
|---|---|---|---|---|
| Baseline | 15.5 ± 0.7 | 24.8 ± 0.7 | 30.2 ± 1.6 | 33.1 ± 1.7 |
| Frequency-weighted loss | 16.8 ± 0.8 | 25.2 ± 1.4 | 30.2 ± 1.2 | 33.1 ± 0.8 |
| Haar → FFT | 18.6 ± 0.6 | 29.8 ± 0.5 | 36.5 ± 0.6 | 38.7 ± 0.8 |
| Two-level Haar | 18.0 ± 0.9 | 27.9 ± 0.5 | 35.3 ± 0.7 | 38.5 ± 0.5 |
| Haar (main) | 17.1 ± 0.8 | 27.0 ± 0.8 | 33.1 ± 0.8 | 35.6 ± 0.4 |
Results on a second continuous DLM (e.g., TEncDM) would strengthen the generality claim.
In our work, we explore latent DLMs that operate in the latent space of pretrained text encoders. Most recent continuous DLMs either operate on text embeddings or one-hot encodings, or jointly train embeddings / latent representations along with the diffusion model. This is why we consider ELF, which represents a general flow-matching transformer, and explore it with different pretrained latent representations.
To show generalization to other latent diffusion model designs, we train the diffusion model from TEncDM on top of the GPT-2 latents and show the results below.
| Model | 16 steps | 32 steps | 48 steps | 64 steps |
|---|---|---|---|---|
| TEncDM Baseline | 7.8 ± 0.3 | 10.2 ± 0.8 | 10.1 ± 0.7 | 10.2 ± 0.3 |
| TEncDM Haar | 9.4 ± 0.5 | 12.3 ± 0.1 | 12.8 ± 0.2 | 12.9 ± 0.3 |
We added all ablations to the revision (Section 5 and Appendix D, Tables 10–12).
Reply [1/4]
We thank the reviewer for the thoughtful comments. We have updated the paper, with all changes marked in red, and provide our answers below.
Thanks for raising a meaningful point. We agree that the filter definition partly explains why low frequencies capture global information and common tokens, while high frequencies capture positional information. However, we claim that learned text representations also contribute to these properties. Below, we disentangle the influence of the filter from that of the text representations.
We use the same password-order probe as in the paper, where four marker tokens are placed at adjacent positions inside a random context in all 24 orders, and train linear probes on the low and high Fourier bands along tokens. We compare five representations of the same sequences, namely, the last hidden layer of the trained Qwen3.5-0.8B, its raw input embeddings, its last hidden layer with token positions shuffled by a fixed random permutation, the last hidden layer of a randomly initialized model with the same architecture, and i.i.d. random vectors assigned to each token without any model.
Token-order probe accuracy (%) for low and high frequencies of different representations
High frequencies preserve the order for all representations, so this part is indeed a property of the filter. However, low frequencies fail only for representations learned from language, while for shuffled, randomly initialized, and random representations they also recover the order. Therefore, the advantage of high frequencies in positional information is not implied by the filter alone. It appears because language models assign similar vectors to related tokens, so swapping two adjacent similar tokens barely changes the low frequencies, and only high frequencies retain their order. We added this analysis to the revision (end of Section 3.2 and Appendix C, Table 7).
We agree that the claim in Key finding 2 is an overreach. The main goal of this claim is only to state that high frequencies contain more information about the input sequence in text than in images, and that this difference is specific to the representations rather than to the filter definition. We soften it in the revision (end of Section 3).
We add a smoothed full-spectrum representation. The smoothed representation performs on par with the low band on all tasks. This is expected, since a low-pass filter is by definition a local weighted average, so the low band cannot be separated from pooling.
Classification performance using different frequency components and smoothed full-spectrum representations of Qwen
On ListOps, Full underperforms because it uses only the last token, while low frequencies aggregate several neighboring tokens. We added this explanation and the smoothed row to Table 1 in the revision.