acceptodds
Under review as a conference paper at ICLR 2027

FlowScope: Interpreting Generation Dynamics in Continuous Flow Language Models

Abstract

Continuous Flow Language Models (CFLMs) generate text by integrating a learned velocity field from noise to the sequences of embeddings that represent texts, providing an alternative to discrete diffusion and autoregressive language models. However, the generation dynamics in the continuous embedding space are not mechanistically studied. This work presents the first mechanistic interpretability analysis of CFLMs, examining the generative process through its central driving force, the velocity field, with sparse dictionary learning. Two embedding CFLMs, LangFlow and ELF, are investigated. We analyze the models based on their distinct designs, use the noise-free per-step estimation of the output state as the proxy for the model’s internal belief, and map the evolution of these beliefs along generation paths. Token identities commit progressively following the learned schedule, in a coarse-to-fine order across token types. The disentangled dictionary separates a small population of frequently activating features from specialized features whose activity increases as trajectories converge, and activation steering then verifies the mechanical causality in these flow models. By examining per-variant trajectory properties, we implement online stability stopping for LangFlow and static schedule truncation for ELF, reducing denoiser evaluations by 12–25% while achieving 0.86–0.94 paired-token agreement, with measurable quality trade-offs. The analysis further examines the dynamics of self-distilled ELF weights, noting observable shifts in their commit clock and feature activation profile.

Primary area:
interpretability and explainable AI
Keywords:
Flow Language Models, Continuous Flow, Interpretability, Explanable AI, Diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2jeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly studies the mechanisms underlying text generation in continuous flow-based language models. The authors focus on LangFlow and ELF, train sparse autoencoders (SAEs) on the clean predictions of these models, and analyze token-level behavior throughout the generation process. They also conduct several experiments, including activation steering, early stopping, and self-distillation.

Critical strengths

The paper introduces SAEs to continuous flow-based language models and uses them to analyze the generation process.

Critical weaknesses

My main concern is that nearly all the experiments, observations, and techniques presented in this paper have already been explored in previous work. Specifically, the claimed contributions have substantial overlap with existing studies:

Sparse Dictionary on Beliefs: See [1], [2], and [3].

Token Commitment: See [4] and [5].

Coarse-to-Fine Token Order: See [6] and [7].

Post-Commit Feature Dynamics: See [3].

Online Stability Stopping: See [4].

Endpoint Self-Distillation: See [8].

Feature Re-timing After Distillation: See [3].

The papers listed above have essentially covered most of the observations and techniques presented in this work, although some behaviors differ between continuous flow-based language models and discrete diffusion language models.

Therefore, my main concern is whether this paper provides any genuinely new insights beyond existing studies. The paper seems to combine SAE-based interpretability analysis, inference acceleration, and self-distillation, but the connections between these components are not sufficiently clear, and it is difficult to identify what new knowledge we gain from the paper.

For example, early stopping in continuous flow-based language models does not necessarily require an SAE. One could simply examine the generated tokens or predictions at different timesteps and determine when they become stable. This raises the question of how the SAE analysis actually contributes to the proposed acceleration methods, rather than merely providing an alternative way to observe behaviors that can already be identified directly from model outputs.

More broadly, while the differences between continuous flow-based and discrete diffusion language models are potentially interesting, the paper does not sufficiently explain why these differences arise or what new understanding they provide.

[1] TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation. arXiv:2503.07050.

[2] SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. ICML 2025.

[3] DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders. arXiv:2602.05859.

[4] Diffusion Language Model Knows the Answer Before It Decodes. ICLR 2026.

[5] Just on Time: Token-Level Early Stopping for Diffusion Language Models. arXiv:2602.11133.

[6] Critical Windows: Non-Asymptotic Theory for Feature Emergence in Diffusion Models. ICML 2024.

[7] Measuring Temporal Linguistic Emergence in Diffusion Language Models. arXiv:2604.23235.

[8] Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.

Questions for authors

a. What are the main new insights provided by this work compared to existing studies? What does the SAE analysis reveal beyond the observations already established for diffusion language models?

b. Why do the token commitment patterns differ from previous findings on discrete diffusion language models [7]?

[7] Measuring Temporal Linguistic Emergence in Diffusion Language Models. arXiv:2604.23235.

7 comments
AuthorsOpenReview ↗

We thank you for recognizing that the paper introduces SAEs to continuous flow language models and uses them to analyze the generation process.

W1: Overlap with prior work [1--8]. We already position our work against [3, 4, 5, 6] in Section 2. Prior works [1--8] study image diffusion or discrete masked DLMs and do not cover the emerging CFLM. In masked DLMs, the sampler explicitly unmasks tokens progressively. In embedding CFLMs, all positions evolve concurrently in RL×d\mathbb{R}^{L \times d} until t=1t=1, with no unmasking mechanism. This property gives freer feature dynamics in the generation process. Prior works do not cover these dynamics, and they are the core object of our study.

For the sparse dictionary on beliefs, [1, 2] train SAEs on internal DiT and UNet image layers, and [3] trains SAEs on hidden layers of discrete DLMs (Dream, LLaDA). FlowScope instead isolates the predicted clean terminal x^t\hat{x}_t through the velocity decomposition vt=a(t)xt+b(t)x^tv_t = a(t)x_t + b(t)\hat{x}_t in embedding space. This decomposition removes the time-dependent sampler gain b(t)b(t), which otherwise inflates the active dictionary atoms by 2.5×2.5\times, as shown in Appendix A.2.

For token commitment and online stability stopping, [4, 5] use single-step confidence thresholds to permanently unmask tokens in discrete DLMs. [7] tracks the retrospective argmax in LLaDA. FlowScope shows that commitment emerges in continuous flows without token freezing. It lives in the belief x^t\hat{x}_t rather than in the noisy state xtx_t, which lags by about 35% of the schedule, shown in Appendix A.3. FlowScope stops when the argmax stops changing, which exploits the early commitment behavior.

For the coarse-to-fine token order, [6] models log-concave image mixtures and has no tokens. [7] reports the opposite order in LLaDA, with numbers and nouns first and punctuation last. FlowScope shows that both CFLMs commit punctuation first, then function words, frequent content, and rare content last, across domains and conditional tasks. The tokens that MDLM and LLaDA-8B actually output follow the same order as we reported in the paper. The symmetry itself is an unexpected result worth a deeper look, as CFLM forms the progressive generation without inductive bias, which undermined its design to refine all tokens until the end of trajectory.

For post-commit dynamics, endpoint self-distillation, and feature re-timing, [3] notes deep-layer feature drift after discrete token unmasking and transfers SAEs to SFT checkpoints, with no distillation. [8] distills step log-probabilities in discrete MDLM, with no mechanistic analysis. FlowScope separates the broad population from the specialized population, which rises at commitment. A within-quartile permutation in Appendix A.7 shuffles commitment steps among positions of the same generation and position quartile. It leaves 40% to 42% of the specialized rise to the feature activity profile. FlowScope then uses the SAE to show that endpoint self-distillation selectively re-times specialized features and preserves their token semantics.

We have added the discussion of [1, 2, 7, 8], the MDLM and LLaDA-8B commitment-order tests, and the distillation preservation results to the revision, in Section 2 (L111-113, L117-120, L129-131), Section 5.1 (L261-269, Table 1), Section 6 (L428-431, L481-485, L511-524, Table 6, Figure 6a), Appendix A.10 (Table 10), Appendix A.11, and Appendix A.12 (Tables 13 and 14).

W2, W3, Qa: New insights, the links between the components, and what the SAE adds beyond the token readout. The belief commitment clock analysis unifies the paper. We extract the clean belief x^t\hat{x}_t from the velocity field to measure when token decisions stabilize (Section 5.1). We align sparse feature emergence to this clock (Section 5.2-5.4). We operationalize our findings and use readout stabilization for threshold-free early stopping and prefix truncation (Section 5.5). Finally, we shift the clock earlier through endpoint self-distillation and track how the features reorganize (Section 6).

The early-stopping and distillation experiments operationalize our findings. Early commitment and token stability drive the policy design. We do not use the SAE in these policies, because it operates on beliefs that exist only after each denoising step. Conversely, the SAE answers two mechanistic questions that token readouts cannot.

The first question is whether specialized units rise at each position's commitment step, beyond the effect of increasing token concentration over time. The table gives the mean pre-to-post change in specialized units at the true and at the permuted commitment steps. As discussed in Appendix A.7, the permutation moves each commitment step to another position of the same generation and position quartile. The share is the fraction of the rise in activity beyond the permutation (1−Permuted/True1 - \text{Permuted}/\text{True}), which is not explained by temporal trends in generation. For the token readout, the units are the top-10 tokens of the readout itself, under the same partition and alignment protocol:

BasisModelTrue stepsPermuted stepsShare
SAE dictionaryLangFlow0.9520.57040%
Top-10 readout tokensLangFlow0.5500.5156%
SAE dictionaryELF4.9222.85842%
Top-10 readout tokensELF0.5030.4942%

The permutation isolates activity induced by the overall time trend and commitment-related features, which is significant only through the BeliefLens, and overlooked otherwise.

The second question is how self-distillation reorganizes the computation. The token readout shows only that the 16-step student commits earlier. The mechanistic study shows how:

Self-distilled ELF student, frozen base SAEValue
Timing slope against the base, broad features0.79
Timing slope against the base, specialized features0.52
Final-token profile cosine, base vs. student0.805
Final-token profile cosine, base split-half ceiling0.823

Broad features keep their timing, and specialized content features advance, as shown in Fig. 6a. Their semantics stay the same: the profile cosine over 5370 specialized features is close to the base split-half ceiling.

The work therefore gives two insights that the cited studies do not. First, CFLMs refine all positions concurrently and have no reveal schedule, yet they surprisingly learn the same commitment order as a masked DLM without inductive bias or progressive objective. This finding driven the inference time acceleration rules that maintained the generation quality with much fewer steps. Second, the SAE shows a feature rise at the commitment step that a token basis cannot show. This insight provides the mechanistic tool for studying the feature re-timing caused by self-distillation.

We have edited the unifying and acceleration discussions and added the top-10 baseline and profile-preservation results to the revision, in Section 4.4 (L235-242), Section 5.5 (L416-425, Table 4), Section 6.3 (L481-485, L511-513, L521-524, Table 5, Table 6, Figure 6a), Appendix A.9 (Table 9), and Appendix A.10 (Table 10).

W4, Qb: Why the commitment order differs from [7], and why CFLMs and DLMs differ. In [7], LLaDA-8B commits numbers (3.373.37 steps) and nouns (4.074.07) before function words (4.924.92) and punctuation (4.974.97), the exact opposite of our CFLM order. To understand why, we ran three controlled experiments across MDLM, ELF, and LLaDA-8B-Base.

First, we applied our exact commitment metric and lexical partition to unconditional MDLM rollouts (OpenWebText, 200 samples, 64 steps, L=256L=256). Second, we controlled for visible context. [7] evaluates masked reconstruction with 60% visible tokens. Thus we kept 30%, 60%, or 90% of the tokens of real OpenWebText visible, and MDLM and ELF filled the hidden positions. The order holds in every setting, and visible context only shifts all classes earlier (mean commitment fraction):

Model and contextPunctuationFunctionFrequentRare
MDLM, unconditional0.290.400.410.47
MDLM, 30% visible0.190.300.330.41
MDLM, 60% visible0.090.180.240.32
MDLM, 90% visible0.040.100.180.25
ELF, 30% visible0.360.460.550.69
ELF, 60% visible0.280.340.440.59
ELF, 90% visible0.120.140.250.40

The discrete unmasking step of MDLM itself has no class order (0.4950.495 to 0.5020.502). Thus coarse-to-fine commitment is an emergent property of the denoiser's beliefs, which discrete and continuous diffusion share.

Third, we reproduced [7] on LLaDA-8B-Base under the protocol that [7] states: WikiText-103 sentences of 10 to 25 tokens, 40% masked, 32 steps, and LLaDA's unmasking rule. These sentences leave only 4 to 10 masked tokens, which mismatches LLaDA's training distribution and cannot model long-form generation. We therefore also ran 256-token windows of the same text. We computed commitment on two readouts. The table gives the mean commitment step out of 32 under confidence unmasking. Bold marks the earliest group:

ReadoutLengthNumbersNounsVerbsFunctionPunctuation
Official LLaDA generate route10-251.882.091.851.051.70
Official LLaDA generate route2565.466.397.473.332.70
Raw forward argmax10-252.382.912.693.124.06
Raw forward argmax25610.579.7110.5212.7410.97
Reported in [7]10-253.374.074.464.924.97

On the official LLaDA generate route, which keeps each revealed token fixed, function words commit well before nouns at both lengths, as in our order. At 256 tokens, the full coarse-to-fine order holds (punctuation 0.08, function 0.11, frequent 0.19, rare 0.26 of the trajectory). Random unmasking reveals positions regardless of class and keeps this order. Thus the order comes from the denoiser, as in MDLM. The inverted order in [7] appears only when we compute commitment on the raw forward argmax across all positions. At 256 tokens, even this readout reproduces [7] only in part, with nouns rather than numbers first.

Masked DLMs such as LLaDA apply the training loss only to [MASK] positions. Once the sampler unmasks a token, the token becomes visible input. No loss then trains the model to preserve its identity at the output head of that position. Thus raw logits at already-revealed punctuation and function-word sites can drift late in sampling, with no effect on the generated text.

Thus, when we measure the actual generative decisions of the model, CFLMs, MDLM, and LLaDA all share the same coarse-to-fine commitment hierarchy. The difference is in the decision process. CFLMs reach this order continuously in the belief space x^t\hat{x}_t, without a discrete unmasking schedule. We have added this cross-family analysis and the citations [1, 2, 7, 8] to the revision, in Section 2 (L111-113, L117-120, L129-131), Section 5.1 (L261-281, Table 1), and Appendix A.12 (Tables 13 and 14).

Reviewer_o1z5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents FlowScope, a mechanistic interpretability framework for Continuous Flow Language Models (CFLMs). By analyzing the predicted terminal state as the model's per-step belief, the authors investigate token commitment and semantic feature evolution using TopK Sparse Autoencoders (SAEs) on LangFlow and ELF. The analysis reveals a progressive commitment pattern, where punctuation and function words stabilize earlier than content words, while specialized features become more active around and after token commitment. The authors further explore activation steering, inference acceleration, and endpoint self-distillation. The experimental results demonstrate that the observed generation dynamics can guide inference acceleration with limited quality degradation, while self-distillation encourages earlier token commitment and maintains competitive generation quality with fewer sampling steps.

Critical strengths

1.Meaningful problem and coherent analysis framework. The paper addresses an underexplored problem in interpreting continuous flow language models. Analyzing the predicted terminal state provides a reasonable and unified perspective for studying token commitment and feature dynamics.

  1. Broad empirical analysis and clear presentation. The paper investigates multiple aspects of generation dynamics across two CFLMs, including sparse feature activation, steering, inference acceleration, and self-distillation. The paper is generally well organized, with clear figures and visualizations that help illustrate the main findings.
Critical weaknesses
  1. Limited methodological novelty in the practical applications. While the paper provides meaningful insights into CFLM generation dynamics, the proposed inference acceleration and self-distillation methods are relatively straightforward extensions of existing techniques. The acceleration results involve a trade-off between computational savings and output consistency, while the self-distillation results demonstrate promising efficiency improvements. These applications are valuable, but their additional methodological contributions appear somewhat limited.

  2. Limited generalizability across different text types and tasks. The token commitment and SAE feature analyses are mainly conducted on unconditional text generation. Although the experiments cover two CFLMs, it remains unclear whether the observed generation dynamics generalize to different text types or conditional generation tasks. A broader analysis would help strengthen the generality of the conclusions.

Questions for authors
  1. Connection between mechanistic analysis and practical applications. Could the authors further clarify how the SAE-based feature analysis specifically informs the proposed inference acceleration and self-distillation methods, beyond the observed token commitment dynamics?

  2. Generalizability across different generation tasks. Do the observed patterns, such as the coarse-to-fine token commitment order and specialized feature dynamics, also hold across different types of text or generation tasks, such as text continuation, question answering, and mathematical reasoning?

4 comments
AuthorsOpenReview ↗

We thank you for recognizing the meaningful problem, the coherent analysis framework, the broad empirical analysis, and the clear presentation.

W1, Q1: The practical applications, and how the mechanistic analysis informs them. Our inference acceleration and self-distillation experiments operationalize the insights from our study of the generation dynamics. We do not propose novel inference acceleration methods, but the practical gains come from exploiting the surprising symmetry between the generation plan of CFLMs and discrete DLMs. We use minimal interventions to test whether the generation properties that FlowScope uncovers, such as early commitment and progressive generation, are actionable at inference.

ApplicationEvaluations savedQuality
Online stability stopping12%0.94 token agreement
Prefix truncation25%0.86 token agreement
Endpoint self-distillation50%maintained perplexity

Stability stopping follows from the analysis of readouts. The cheapest token readout stabilizes reliably without a tuned confidence threshold, so our trigger exploits stability. Its quality metrics stay inside the baseline ranges as shown in Table 4.

Prefix truncation follows from the commitment timing. Token identities settle near the middle of the schedule with mean commitment at 53.9% in ELF, in Fig. 2a, so the trajectory prefix carries the generation decision. A schedule that keeps the 23-step prefix and jumps to the terminal time keeps this decision. Uniform respacing across the same 24 steps loses as shown in Table 4.

For the self-distillation experiments, our focus is to extend the mechanistic framework to study the effect of self-distillation on CFLMs. Our findings on the feature-space clock shift inform how language is compressed in a continuous diffusion. Self-distillation moves the commitment clock: the student reaches 50% coverage at step 6 of 16, against 17 of 32 in the base model as shown in Table 6. In Section 6 and Appendix A.2, the SAE analysis explains how the student reorganizes its computation, which the token readout cannot show. We encoded the beliefs of the student with the frozen base dictionary. Broad features keep their normalized schedule timing, whereas specialized content features selectively advance to earlier steps, shown in Fig. 6. Feature timing advancement is orthogonal to per-feature clarity (r=−0.01r = -0.01 early, −0.10-0.10 late, Appendix A.2).

Moreover, we find that distillation keeps the learned SAE features. To test this directly on disjoint seed halves, we compared the final-token profiles at the sites where each of the 5370 specialized features fires, in the base model and in the student. The base-to-student profile cosine is 0.805. This value nearly matches the split-half ceiling of the base model at 0.823. We have edited the acceleration and self-distillation discussions and added the profile comparison to the revision, in Section 4.4 (L235-242), Section 5.5 (L416-425, Table 4), Section 6.3 (L476-485, L511-524, Table 5, Table 6, Figure 6), and Appendix A.10 (Table 10).

W2, Q2: Generalizability across text types, lengths, conditional tasks, and text continuation. Appendix A.3 already checks 11 sampling schedules, and Appendix A.4 validates the SAE on fresh sequences. We also ran new experiments on longer web text, text continuation, summarization, and machine translation. The table gives the mean commitment time of each class. The coarse-to-fine commitment order holds in every setting:

SettingPunctuationFunctionFrequentRare
LangFlow, LM1B, L=128L=128 (paper)0.1110.2600.3620.611
LangFlow, OpenWebText, L=1024L=10240.1550.3340.4510.659
ELF-B, unconditional (paper)0.3830.5020.5560.666
ELF-B, text continuation0.4160.5350.5850.699
ELF-B, XSum summarization0.1340.1660.2060.346
ELF-B, WMT De-En translation0.0630.0620.0880.159

Strongly conditioned tasks commit earlier overall, because the source text constrains the output space. Yet frequent and rare content words still commit after punctuation and function words. Therefore our commitment order claim holds on the tested tasks.

The specialized feature dynamics also hold. We trained a separate SAE on the OpenWebText beliefs of LangFlow with the same partition rule. It gives the same bimodal split. The cutoff takes place at 0.178 compared to 0.180 on LM1B, producing 139 vs. 143 broad features. Specialized activity places 0.91 of its aligned area at or after commitment compared to 0.84 on LM1B.

The share of the rise beyond the permutation is 1−Permuted/True1 - \text{Permuted}/\text{True}, where both terms are the pre-to-post change in specialized features at permuted and at true commitment steps. The permutation moves each commitment step to another position of the same generation and position quartile. On OpenWebText, this share is 44%, similar to 40% on LM1B. For ELF-B, we repeated the Appendix A.7 within-quartile permutation control in each setting. The table gives the dictionary FVE, the specialized area at or after commitment, the change at true and at permuted commitment steps, and the share (w=2w=2):

ELF-B settingDictionaryFVEAreaTruePermutedShare
Unconditional (paper)paper dictionary0.89 to 0.960.744.9222.85842%
Text continuationfrozen unconditional0.87 to 0.900.704.8192.64945%
XSum summarizationfrozen unconditional0.67 to 0.710.563.0761.60548%
WMT De-En translationtrained on De-En0.95 to 0.960.514.9434.32512%

For De-En, the frozen dictionary explains only 0.53 to 0.60 of the beliefs. Thus we trained a dictionary on De-En beliefs with the recipe and partition rule of the paper (cutoff 0.224). Translation leaves little room before commitment: 49% of De-En positions commit at step 0 or 1 of 64 (median step 2), against 21% for XSum (median step 7).

In the unconditional, continuation, and XSum settings, specialized features also activate substantially later than broad features. For XSum, the unconditional SAE transfers zero-shot to the task-specific checkpoint, and the share of the rise beyond the permutation stays at 48%. Therefore, our findings regarding commitment and specialized feature activity hold true in the tested tasks, except De-En translation. We consider De-En and XSum more conditioned, which is outside the generative language model scope of CFLMs.

No public embedding CFLM checkpoint is currently trained for question answering or mathematical reasoning. Thus our empirical scope covers unconditional generation, text continuation, summarization, and translation. We have added these multi-domain, long-sequence, continuation, and conditional experiments and the scope note to the revision, in Section 5.1 (L261-281, Table 1), Section 5.2 (L295-296), Appendix A.8, and Appendix A.11 (Tables 11 and 12).

We hope this addresses your concerns and are happy to answer any further questions.

Thank you for the detailed rebuttal and additional experiments. The new experiments across different generation tasks substantially address my concerns about generalizability. The additional SAE analyses and controlled comparisons also strengthen the experimental evidence and clarify the contributions of this work.

However, I still have some concerns about the limited methodological novelty of the practical applications. I believe the work could be further strengthened by developing more innovative acceleration strategies that are more closely connected to the mechanistic findings.

Overall, I consider this paper borderline, but the additional evidence is sufficient for me to slightly raise my rating.

Thank you very much for your insightful and constructive feedbacks. We are very encouraged by your reconsideration! Meanwhile, in the latest revision, we incorporated the sae-informed early-stopping results in Appendix A.15, L1423-1511. We hope it can further mitigate your remaining concerns.

Reviewer_kC3mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies generation dynamics in two embedding-space continuous flow language models, LangFlow and ELF. Motivated by the decomposition of sampler velocity into interpolant-state and predicted-terminal components, it analyzes the models’ native clean-terminal predictions using TopK sparse autoencoders. The analysis relates retrospective token commitment to token classes and the activation timing of broad and specialized features. The paper further explores prediction-space steering, model-specific stopping or schedule truncation, and endpoint self-distillation. It reports reductions in denoiser evaluations with measurable output changes, and an ELF student whose generative perplexity at 16 evaluations approaches that of the base model at 32 evaluations.

Critical strengths

A well-motivated representation for studying generation trajectories. Analyzing the native predicted terminal rather than the noisy interpolant provides a useful view of what each model currently predicts. The model-specific treatment of sampling and readout, together with the state-versus-belief comparison and grid-sensitivity checks, makes this choice more convincing than simply applying a dictionary to arbitrary trajectory states. An informative empirical study with meaningful controls. The paper connects commitment timing, sparse representations, interventions, and inference budgets across two models. The appendix includes fresh-sequence reconstruction checks, common-support alignment, a permutation placebo, and explicit accounting of output changes. The endpoint-distillation results also provide an interesting example of changing prediction timing while retaining approximately comparable generative perplexity at a reduced evaluation budget.

Critical weaknesses

The central mechanistic interpretation is not sufficiently distinguished from output-space concentration. In LangFlow, the SAE input is already a posterior-weighted token embedding. As this posterior sharpens, a redistribution from broadly used atoms toward token-specific atoms could arise without establishing a structure-then-content planning mechanism. The same-site controls are useful, but the substantial rise under the permutation placebo indicates that ordinary trajectory-wide time trends also contribute (Appendix A.7). Reconstruction accuracy and token similarity measured in the model’s own readout space do not independently establish the proposed semantic interpretation. I therefore find the descriptive trajectory results more convincing than the stronger claims about semantic planning and inherited feature vocabulary. The steering experiments establish controllability, but do not yet establish the claimed feature-specific semantic mechanism. LangFlow is steered along token-embedding directions, whereas ELF’s selected SAE directions are evaluated using context-derived substring counts rather than independently assessed conceptual applicability (Appendix A.6). Early and late interventions also differ in the number of edits and the remaining propagation horizon. Consequently, their comparison does not isolate commitment timing as the explanation for different steerability. A comparison against simple token/decoder directions and random directions at matched final perturbation budgets would help establish what the learned features contribute beyond lexical control. These concerns currently lead me to weak rejection. I would reconsider this recommendation if focused controls demonstrate that the main feature-timing and intervention findings provide information beyond posterior concentration and straightforward output-space baselines.

Questions for authors

How much of the feature-timing and clarity results remains after controlling for token frequency and time-dependent posterior concentration or belief norm, and using fixed feature cohorts? Could a simple token/decoder-basis baseline or a suitably matched null trajectory distinguish these effects from the proposed semantic planning interpretation? At a matched final perturbation budget, such as token-change rate, do SAE directions outperform nearest-token/decoder directions and random directions on independently assessed conceptual applicability? Such a comparison would materially affect my assessment of the claimed causal validation.

8 comments
AuthorsOpenReview ↗

We thank you for recognizing the well-motivated representation for studying generation trajectories and the informative empirical study with meaningful controls.

W1a, Q2: Effect of output-space concentration and token basis. A top-10 token-readout basis in place of the SAE removes nearly all of the commitment-aligned gain. Thus specialized SAE features capture structure beyond output-space posterior sharpening. Posterior sharpening is a natural part of the generation trajectory. The core question is whether the SAE dictionary shows dynamics beyond the token readout itself. To isolate the effect of token concentration, we replaced the SAE with each model's own top-10 readout tokens, under the same prevalence partition (πj\pi_j) and alignment protocol as in Appendix A.7. We compared both bases against the within-quartile permutation ablation. This ablation keeps the full trajectory and token sharpening, and it permutes the commitment steps within the same generation and position quartile.

For each basis, the table gives the mean pre-to-post change in active specialized units at true and at permuted commitment steps. It also gives the share of the change beyond the permutation (1−Permuted/True1 - \text{Permuted}/\text{True}):

BasisModelTrue stepsPermuted stepsShare
SAE dictionaryLangFlow0.9520.57040%
Top-10 readout tokensLangFlow0.5500.5156%
SAE dictionaryELF4.9222.85842%
Top-10 readout tokensELF0.5030.4942%

The top-1 token defines the commitment step. Yet the specialized tokens within the top-10 readout rise no more at true commitment than at a permuted step. We have added this token-basis baseline to the revision, in Appendix A.9 and Table 9.

W1b, Q1: Effect of time trends, token frequency, and belief norm on feature timing. After controls for trajectory-wide time trends, belief norm, and token frequency, 32–45% of the specialized rise remains tied to commitment.
Global time trends, belief-norm growth, and token frequency are natural parts of the generation trajectory in Section 5.1. The key question is whether specialized features activate at the commitment step of each position, beyond these background trends. The within-quartile permutation (Table 8, Appendix A.7) keeps the full trajectory of every position. It permutes only the commitment index cn,ic_{n,i} within the same generation and position quartile. Thus it keeps all trajectory-wide time trends, posterior sharpening, and belief-norm growth.

To also control for token frequency and lexical class, we repeated the permutation within final-token classes, and within final-token class and position quartile together. The table gives the mean pre-to-post change in active specialized features at true and at permuted commitment steps. It also gives the share of the rise beyond the permutation (1−Permuted/True1 - \text{Permuted}/\text{True}):

PermutationModelTrue stepsPermuted stepsShare
Within quartile (Table 8)LangFlow0.9520.57040%
Within quartile (Table 8)ELF4.9222.85842%
Within final-token classLangFlow0.9520.52345%
Within final-token classELF4.9222.83942%
Within class and quartileLangFlow0.9520.64632%
Within class and quartileELF4.9222.92041%

Across all controls, the pre-to-post jump at the true commitment step significantly exceeds the permuted baseline. Trajectory-wide trends and token frequency explain 55–68% of the raw increase. The remaining 32–45% is tied to the step at which each position commits and feature activities. In LangFlow, content words carry this excess. We have added the stricter permutations and the rise within each final-token class to the revision, in Section 5.2 (L295-301, L347-361, Figure 4) and Appendix A.9 (Table 9).

W1c, Q1: Semantic evidence beyond posterior concentration and readout geometry, with fixed cohorts. Readout-space measurements describe what the model predicts at each step, and this prediction is the generation process that FlowScope studies. We tested whether the semantic reading also holds without posterior concentration, and outside the readout space. Three measurements answer this.

First, we scored clarity on the final text, on fixed cohorts in Table 3, Appendix A.5. Clarity is the mean pairwise cosine of the tokens at the activation sites of a feature. The final text stream uses only the final landed tokens, so posterior concentration does not affect it. In ELF, final-text clarity rises from 0.28 (Early) to 0.68 (Late). On the fixed cohort of the 323 ELF features active in both the Early and Late windows, clarity rises from 0.21 to 0.41, and it rises for 84% of the features. Thus the clarity gain does not come from posterior concentration.

Second, we compared descriptions in language, with no readout geometry in Appendix A.5. An LLM describes each feature from two streams: from the belief readout at its activation step, and from the final text at the same positions. A blind judge sees only the two descriptions and reports whether they name different concepts. The repeat floor is the rate for two seed-variant descriptions of one stream. Each floor cell gives the mean over the two streams. The cohorts are the features active in all three thirds of the schedule:

EarlyMidLate
ELF44.2%9.4%6.7%
ELF repeat floor15.7%8.0%6.1%
LangFlow13.1%3.9%0.8%
LangFlow repeat floor6.5%3.1%4.4%

In the Early third of ELF, the belief readout names a different concept from the final text far above the floor. From the Mid third on, the two streams are at the floor. LangFlow commits earlier, so its rate is low from the Early third on and falls below its Late floor. Thus, measured in language, the meaning of the belief converges onto the meaning of the final text along the trajectory.

Third, we tested the semantics under intervention. We steered ELF along each of 30 specialized feature directions in the late window. A separate AxBench judge model (Qwen3.8-27B) then scored whether the generated text shows the described concept of the feature, at a matched token-change rate of 0.15. The median concept score is 0.36 for the SAE directions, against 0.28 for the token's mean belief and 0.13 for random directions. Thus a push along a feature puts its described concept into new text, outside the readout space.
We have edited the clarity text and added the fixed-cohort clarity and the concept judge to the revision, in Section 5.3 (L305-308, L363-375, Table 3), Appendix A.10, and Appendix A.13 (Table 15).

Comments on the paper

2 comments

Dear Reviewers,

The author-reviewer discussion phase is ending soon. If you have not done so yet, please read the author rebuttals and respond.

Please recall that, as a reviewer, you have agreed to engage in discussion with the authors to clarify and improve the paper. Failure to do so may result in penalties on your own submissions, as per the official ICLR policy. Given the high volume of submissions, your participation is crucial.

Thank you for your time and help.

Best regards,

Your AC

AuthorsOpenReview ↗

Discussion Summary

Dear Reviewers and Area Chairs,

Thank you for your time and efforts in evaluating our paper. We are encouraged to see the reviewers' recognition of our paper's strengths, including the introduction of SAEs to continuous flow language models (2jeY) for an underexplored problem (o1z5), the well-motivated predicted-terminal representation and coherent analysis framework (kC3m, o1z5), the broad empirical study with meaningful controls (kC3m, o1z5), the endpoint self-distillation results (kC3m), and clear presentation (o1z5). We summarize below the new results and revisions, and we answer every point of each review in the individual responses.

Key Contributions

  • FlowScope is the first mechanistic interpretability study of CFLMs. The velocity decomposition exposes the predicted terminal x^t\hat{x}_t as the belief of each step. A sparse dictionary on this belief maps how LangFlow and ELF build text.
  • CFLMs refine all positions concurrently with no unmasking schedule. Yet tokens commit progressively from punctuation and function words to frequent and rare content, the same order that masked DLMs learn. The belief carries this decision, and the noisy state lags by about 35% of the schedule. Specialized features rise at the commitment step of each position.
  • The commitment clock governs control and efficiency. Late steering substitutes targets, while early steering rewrites the text. Readout stability saves 12 to 25% of denoiser evaluations. An endpoint self-distilled student halves the evaluations: it advances its clock and re-times its specialized features.

Key Revisions

The revised PDF marks all changes in blue.

  • Beyond output-space concentration (kC3m, 2jeY). With the top-10 readout tokens of each model as the basis, the share of the rise beyond the within-quartile permutation is 2 to 6%. The SAE gives 40 to 42%. Permutations within final-token class, and within class and position quartile, keep 32 to 45% (Sec. 5.2, Figure 4, App. A.9, Table 9).
  • Feature semantics on fixed feature sets (kC3m). On the 323 ELF features active early and late, final-text clarity rises from 0.21 to 0.41, for 84% of the features. A blind judge compares descriptions from the belief and from the final text. From the middle third of the schedule on, they differ no more than two descriptions of one stream (Sec. 5.3, Table 3, App. A.5, A.10).
  • Steering controls (kC3m). At a matched token-change rate of 0.15, an AxBench judge gives SAE directions a median concept score of 0.36. The mean belief of the token gets 0.28, and its decoder row 0.12. In one window with equal edits and horizon, 69% of edited uncommitted positions change in ELF, against 17% of committed ones. LangFlow gives 42% against 16% (Sec. 5.4, Figure 5b, App. A.13, A.14, Tables 15 and 16).
  • Generality across tasks (o1z5). Four new settings keep the commitment order and the specialized rise: OpenWebText at length 1024, ELF continuation, XSum, and WMT De-En. A source text shifts all classes earlier and keeps the order. No public embedding CFLM targets QA or math, and App. A.8 states this scope (Sec. 5.1, Table 1, App. A.11, Tables 11 and 12).
  • Cross-family comparison and prior work (2jeY). Our metric gives MDLM the same order, while the MDLM unmasking step has no class order (0.495 to 0.502). On LLaDA-8B, the reversed order of Lu et al. (2026) appears only on the raw forward argmax at revealed positions. The official LLaDA generate route follows our order. Sec. 2 now discusses all eight works that 2jeY cited (Sec. 2, Sec. 5.1, App. A.12, Tables 13 and 14).
  • Self-distillation keeps the feature vocabulary (kC3m, o1z5, 2jeY). Over 5,370 specialized features, the base-to-student final-token profile cosine is 0.805, against a base split-half ceiling of 0.823. Broad features keep their timing, and specialized features advance (Sec. 6.3, Table 6, Figure 6, App. A.10, Table 10).
  • SAE-driven acceleration (2jeY, o1z5). SAE-cut stops each sequence from its SAE code alone. It stops once the strongest specialized feature changes at no more than a fraction ε\varepsilon of positions for three consecutive steps. It keeps the full-run task scores (BLEU 28.36 against 27.89, ROUGE-L 20.97 against 20.92) and transfers from De-En to XSum without retuning. It needs no decoder pass and runs at 2.41×2.41\times on XSum, against 1.30×1.30\times for the token-readout rule. We will add SAE-cut to the revision as Appendix A.15.

We hope these results and revisions further strengthen your confidence in our work, and we would be happy to provide any further clarification. Thank you again for your time and consideration.

Sincerely, Authors of Submission 26883

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.