- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper mainly studies the mechanisms underlying text generation in continuous flow-based language models. The authors focus on LangFlow and ELF, train sparse autoencoders (SAEs) on the clean predictions of these models, and analyze token-level behavior throughout the generation process. They also conduct several experiments, including activation steering, early stopping, and self-distillation.
Critical strengths
The paper introduces SAEs to continuous flow-based language models and uses them to analyze the generation process.
Critical weaknesses
My main concern is that nearly all the experiments, observations, and techniques presented in this paper have already been explored in previous work. Specifically, the claimed contributions have substantial overlap with existing studies:
Sparse Dictionary on Beliefs: See [1], [2], and [3].
Token Commitment: See [4] and [5].
Coarse-to-Fine Token Order: See [6] and [7].
Post-Commit Feature Dynamics: See [3].
Online Stability Stopping: See [4].
Endpoint Self-Distillation: See [8].
Feature Re-timing After Distillation: See [3].
The papers listed above have essentially covered most of the observations and techniques presented in this work, although some behaviors differ between continuous flow-based language models and discrete diffusion language models.
Therefore, my main concern is whether this paper provides any genuinely new insights beyond existing studies. The paper seems to combine SAE-based interpretability analysis, inference acceleration, and self-distillation, but the connections between these components are not sufficiently clear, and it is difficult to identify what new knowledge we gain from the paper.
For example, early stopping in continuous flow-based language models does not necessarily require an SAE. One could simply examine the generated tokens or predictions at different timesteps and determine when they become stable. This raises the question of how the SAE analysis actually contributes to the proposed acceleration methods, rather than merely providing an alternative way to observe behaviors that can already be identified directly from model outputs.
More broadly, while the differences between continuous flow-based and discrete diffusion language models are potentially interesting, the paper does not sufficiently explain why these differences arise or what new understanding they provide.
[1] TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation. arXiv:2503.07050.
[2] SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. ICML 2025.
[3] DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders. arXiv:2602.05859.
[4] Diffusion Language Model Knows the Answer Before It Decodes. ICLR 2026.
[5] Just on Time: Token-Level Early Stopping for Diffusion Language Models. arXiv:2602.11133.
[6] Critical Windows: Non-Asymptotic Theory for Feature Emergence in Diffusion Models. ICML 2024.
[7] Measuring Temporal Linguistic Emergence in Diffusion Language Models. arXiv:2604.23235.
[8] Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.
Questions for authors
a. What are the main new insights provided by this work compared to existing studies? What does the SAE analysis reveal beyond the observations already established for diffusion language models?
b. Why do the token commitment patterns differ from previous findings on discrete diffusion language models [7]?
[7] Measuring Temporal Linguistic Emergence in Diffusion Language Models. arXiv:2604.23235.
7 comments
W2, W3, Qa: New insights, the links between the components, and what the SAE adds beyond the token readout. The belief commitment clock analysis unifies the paper. We extract the clean belief from the velocity field to measure when token decisions stabilize (Section 5.1). We align sparse feature emergence to this clock (Section 5.2-5.4). We operationalize our findings and use readout stabilization for threshold-free early stopping and prefix truncation (Section 5.5). Finally, we shift the clock earlier through endpoint self-distillation and track how the features reorganize (Section 6).
The early-stopping and distillation experiments operationalize our findings. Early commitment and token stability drive the policy design. We do not use the SAE in these policies, because it operates on beliefs that exist only after each denoising step. Conversely, the SAE answers two mechanistic questions that token readouts cannot.
The first question is whether specialized units rise at each position's commitment step, beyond the effect of increasing token concentration over time. The table gives the mean pre-to-post change in specialized units at the true and at the permuted commitment steps. As discussed in Appendix A.7, the permutation moves each commitment step to another position of the same generation and position quartile. The share is the fraction of the rise in activity beyond the permutation (), which is not explained by temporal trends in generation. For the token readout, the units are the top-10 tokens of the readout itself, under the same partition and alignment protocol:
| Basis | Model | True steps | Permuted steps | Share |
|---|---|---|---|---|
| SAE dictionary | LangFlow | 0.952 | 0.570 | 40% |
| Top-10 readout tokens | LangFlow | 0.550 | 0.515 | 6% |
| SAE dictionary | ELF | 4.922 | 2.858 | 42% |
| Top-10 readout tokens | ELF | 0.503 | 0.494 | 2% |
The permutation isolates activity induced by the overall time trend and commitment-related features, which is significant only through the BeliefLens, and overlooked otherwise.
The second question is how self-distillation reorganizes the computation. The token readout shows only that the 16-step student commits earlier. The mechanistic study shows how:
| Self-distilled ELF student, frozen base SAE | Value |
|---|---|
| Timing slope against the base, broad features | 0.79 |
| Timing slope against the base, specialized features | 0.52 |
| Final-token profile cosine, base vs. student | 0.805 |
| Final-token profile cosine, base split-half ceiling | 0.823 |
Broad features keep their timing, and specialized content features advance, as shown in Fig. 6a. Their semantics stay the same: the profile cosine over 5370 specialized features is close to the base split-half ceiling.
The work therefore gives two insights that the cited studies do not. First, CFLMs refine all positions concurrently and have no reveal schedule, yet they surprisingly learn the same commitment order as a masked DLM without inductive bias or progressive objective. This finding driven the inference time acceleration rules that maintained the generation quality with much fewer steps. Second, the SAE shows a feature rise at the commitment step that a token basis cannot show. This insight provides the mechanistic tool for studying the feature re-timing caused by self-distillation.
We have edited the unifying and acceleration discussions and added the top-10 baseline and profile-preservation results to the revision, in Section 4.4 (L235-242), Section 5.5 (L416-425, Table 4), Section 6.3 (L481-485, L511-513, L521-524, Table 5, Table 6, Figure 6a), Appendix A.9 (Table 9), and Appendix A.10 (Table 10).
W4, Qb: Why the commitment order differs from [7], and why CFLMs and DLMs differ. In [7], LLaDA-8B commits numbers ( steps) and nouns () before function words () and punctuation (), the exact opposite of our CFLM order. To understand why, we ran three controlled experiments across MDLM, ELF, and LLaDA-8B-Base.
First, we applied our exact commitment metric and lexical partition to unconditional MDLM rollouts (OpenWebText, 200 samples, 64 steps, ). Second, we controlled for visible context. [7] evaluates masked reconstruction with 60% visible tokens. Thus we kept 30%, 60%, or 90% of the tokens of real OpenWebText visible, and MDLM and ELF filled the hidden positions. The order holds in every setting, and visible context only shifts all classes earlier (mean commitment fraction):
| Model and context | Punctuation | Function | Frequent | Rare |
|---|---|---|---|---|
| MDLM, unconditional | 0.29 | 0.40 | 0.41 | 0.47 |
| MDLM, 30% visible | 0.19 | 0.30 | 0.33 | 0.41 |
| MDLM, 60% visible | 0.09 | 0.18 | 0.24 | 0.32 |
| MDLM, 90% visible | 0.04 | 0.10 | 0.18 | 0.25 |
| ELF, 30% visible | 0.36 | 0.46 | 0.55 | 0.69 |
| ELF, 60% visible | 0.28 | 0.34 | 0.44 | 0.59 |
| ELF, 90% visible | 0.12 | 0.14 | 0.25 | 0.40 |
The discrete unmasking step of MDLM itself has no class order ( to ). Thus coarse-to-fine commitment is an emergent property of the denoiser's beliefs, which discrete and continuous diffusion share.
Third, we reproduced [7] on LLaDA-8B-Base under the protocol that [7] states: WikiText-103 sentences of 10 to 25 tokens, 40% masked, 32 steps, and LLaDA's unmasking rule. These sentences leave only 4 to 10 masked tokens, which mismatches LLaDA's training distribution and cannot model long-form generation. We therefore also ran 256-token windows of the same text. We computed commitment on two readouts. The table gives the mean commitment step out of 32 under confidence unmasking. Bold marks the earliest group:
| Readout | Length | Numbers | Nouns | Verbs | Function | Punctuation |
|---|---|---|---|---|---|---|
| Official LLaDA generate route | 10-25 | 1.88 | 2.09 | 1.85 | 1.05 | 1.70 |
| Official LLaDA generate route | 256 | 5.46 | 6.39 | 7.47 | 3.33 | 2.70 |
| Raw forward argmax | 10-25 | 2.38 | 2.91 | 2.69 | 3.12 | 4.06 |
| Raw forward argmax | 256 | 10.57 | 9.71 | 10.52 | 12.74 | 10.97 |
| Reported in [7] | 10-25 | 3.37 | 4.07 | 4.46 | 4.92 | 4.97 |
On the official LLaDA generate route, which keeps each revealed token fixed, function words commit well before nouns at both lengths, as in our order. At 256 tokens, the full coarse-to-fine order holds (punctuation 0.08, function 0.11, frequent 0.19, rare 0.26 of the trajectory). Random unmasking reveals positions regardless of class and keeps this order. Thus the order comes from the denoiser, as in MDLM. The inverted order in [7] appears only when we compute commitment on the raw forward argmax across all positions. At 256 tokens, even this readout reproduces [7] only in part, with nouns rather than numbers first.
Masked DLMs such as LLaDA apply the training loss only to [MASK] positions. Once the sampler unmasks a token, the token becomes visible input. No loss then trains the model to preserve its identity at the output head of that position. Thus raw logits at already-revealed punctuation and function-word sites can drift late in sampling, with no effect on the generated text.
Thus, when we measure the actual generative decisions of the model, CFLMs, MDLM, and LLaDA all share the same coarse-to-fine commitment hierarchy. The difference is in the decision process. CFLMs reach this order continuously in the belief space , without a discrete unmasking schedule. We have added this cross-family analysis and the citations [1, 2, 7, 8] to the revision, in Section 2 (L111-113, L117-120, L129-131), Section 5.1 (L261-281, Table 1), and Appendix A.12 (Tables 13 and 14).
We thank you for recognizing that the paper introduces SAEs to continuous flow language models and uses them to analyze the generation process.
W1: Overlap with prior work [1--8]. We already position our work against [3, 4, 5, 6] in Section 2. Prior works [1--8] study image diffusion or discrete masked DLMs and do not cover the emerging CFLM. In masked DLMs, the sampler explicitly unmasks tokens progressively. In embedding CFLMs, all positions evolve concurrently in RL×d until t=1, with no unmasking mechanism. This property gives freer feature dynamics in the generation process. Prior works do not cover these dynamics, and they are the core object of our study.
For the sparse dictionary on beliefs, [1, 2] train SAEs on internal DiT and UNet image layers, and [3] trains SAEs on hidden layers of discrete DLMs (Dream, LLaDA). FlowScope instead isolates the predicted clean terminal x^t through the velocity decomposition vt=a(t)xt+b(t)x^t in embedding space. This decomposition removes the time-dependent sampler gain b(t), which otherwise inflates the active dictionary atoms by 2.5×, as shown in Appendix A.2.
For token commitment and online stability stopping, [4, 5] use single-step confidence thresholds to permanently unmask tokens in discrete DLMs. [7] tracks the retrospective argmax in LLaDA. FlowScope shows that commitment emerges in continuous flows without token freezing. It lives in the belief x^t rather than in the noisy state xt, which lags by about 35% of the schedule, shown in Appendix A.3. FlowScope stops when the argmax stops changing, which exploits the early commitment behavior.
For the coarse-to-fine token order, [6] models log-concave image mixtures and has no tokens. [7] reports the opposite order in LLaDA, with numbers and nouns first and punctuation last. FlowScope shows that both CFLMs commit punctuation first, then function words, frequent content, and rare content last, across domains and conditional tasks. The tokens that MDLM and LLaDA-8B actually output follow the same order as we reported in the paper. The symmetry itself is an unexpected result worth a deeper look, as CFLM forms the progressive generation without inductive bias, which undermined its design to refine all tokens until the end of trajectory.
For post-commit dynamics, endpoint self-distillation, and feature re-timing, [3] notes deep-layer feature drift after discrete token unmasking and transfers SAEs to SFT checkpoints, with no distillation. [8] distills step log-probabilities in discrete MDLM, with no mechanistic analysis. FlowScope separates the broad population from the specialized population, which rises at commitment. A within-quartile permutation in Appendix A.7 shuffles commitment steps among positions of the same generation and position quartile. It leaves 40% to 42% of the specialized rise to the feature activity profile. FlowScope then uses the SAE to show that endpoint self-distillation selectively re-times specialized features and preserves their token semantics.
We have added the discussion of [1, 2, 7, 8], the MDLM and LLaDA-8B commitment-order tests, and the distillation preservation results to the revision, in Section 2 (L111-113, L117-120, L129-131), Section 5.1 (L261-269, Table 1), Section 6 (L428-431, L481-485, L511-524, Table 6, Figure 6a), Appendix A.10 (Table 10), Appendix A.11, and Appendix A.12 (Tables 13 and 14).