acceptodds
Under review as a conference paper at ICLR 2027

On the design space of continuous diffusion language models

Abstract

Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined , which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians—an issue known as token identifiability. To address it directly, we introduce discrete reveals to , yielding , which achieves discrete diffusion-level performance on TinyGSM. At the B scale, outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.

Primary area:
generative models
Keywords:
language modeling, diffusion, flow matching, flow-based language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T9DtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-conditioning (SC) and embedding geometry in CDLMs. It introduces DoubleCat, a two-level corruption process aimed at reducing training–inference mismatch, and SemiCat, which combines DoubleCat with clean-token reveals. It also investigates how training objectives shape embedding geometry and explores cheaper alternatives to SC in Plaid.

Critical strengths

I appreciate the paper goal to better understand the SC role in CDLMs and find simpler and more natural alternatives to this trick.

I view DoubleCat is a simple and elegant modification motivated by analysing categorical predictions during inference. It largely improves one-hot CDLMs and outperforms the baseline w/ SC without requiring an extra training forward pass. I consider it a valuable contribution to this setting, even though SC provides further improvements.

SemiCat demonstrates significant further empirical gains of combining DoubleCat with the revealing mechanism from CANDI. The analytical posterior used as a cheap conditioning signal is also an interesting exploratory contribution.

Critical weaknesses

W1 | I find several central takeaways in Sections 3 and 4 closely revisit prior work, making it difficult to identify what new understanding the analysis adds:

Section 3.1: Shabalin et al. (2026, §3.3) already connect sampling failures to low-density regions rarely visited during training and interpret SC as helping mitigate this problem.

Section 3.2: CANDI already identifies token-identifiability problems under Gaussian corruption and introduces clean-token reveals to provide reliable conditioning signals.

Section 4.1: CANDI (§5.3 and Appendix C.3) already observes that the transformer operates on projected noisy one-hot inputs and exploits this embedding-space view to accelerate sampling.

Section 4.2: RePlaid and LangFlow already analyze how CE and MSE shape embedding geometry, while CANDI studies token identifiability in embedding spaces.

Section 4.3: CANDI (Appendix B.3.2) explicitly interprets SC as supplying pseudo-clean anchors that address the lack of identifiable conditioning signals.

The paper also acknowledges many connections. Nevertheless, I would appreciate a clearer distinction between established observations, new analytical insights, and the modifications they motivate.

W2 | I have a concern about the practical value of the DoubleCat adaptation to Plaid. The proposed variants improve over Plaid w/o SC but remain largely below standard Plaid w/ SC. Given this accuracy gap, I am not convinced that saving the extra training forward pass offers a worthy trade-off.

Therefore, I would like to see posterior DoubleCat combined with the standard SC and compared against Plaid + SC.

W3 | SC remains largely beneficial in both one-hot CDLMs and Plaid. The results do not establish whether DoubleCat’s gains reflect partial mitigation of the same failures addressed by SC or complementary effects. It would be valuable to see more direct evidence connecting the gains to the proposed mechanism.

W4 | In Figure 5, DoubleCat does not show noticeable improvements on OWT. Thus, the SemiCat gains maybe attributed to the CANDI's revealing mechanism rather than DoubleCat, which is the key methodological contribution, not the revealing mechanism. The scaling experiments likewise do not isolate DoubleCat’s contribution.

Questions for authors
  1. Which analytical insights are new relative to the cited prior work, and how do they specifically motivate the proposed modifications?

  2. Does posterior DoubleCat improve over Plaid + SC when both use the same SC protocol?

  3. What evidence distinguishes a shared mismatch-reduction mechanism from complementary benefits of DoubleCat and SC?

  4. Could the author provide scaling results for DoubleCat alone, ideally also including the base flow with CANDI-style reveals alone?

4 comments

We thank the reviewer for their valuable observations; the resulting changes have been highlighted in blue in the revised PDF.

W1, Q1

Shabalin et al. already connect sampling failures to low-density regions [...]

We concur, and have modified the text to make the connection more explicit (L. 205--206, 253). Our results are complementary, as [1] leverage a multimodal lens, whereas we show that the interpolant construction can be improved, without using SC. We have now highlighted the motivation given by the inference path's shape (L. 197--200), I^t=(1−γt)X0+γtY^t\hat I_t = (1-\gamma_t) X_0 + \gamma_t \hat Y_t.

CANDI already identifies token-identifiability [...]

This is true. We are happy to revise any statement that might suggest otherwise.

CANDI already observes that the transformer operates on projected noisy one-hot inputs [...]

We agree, and have merged 4.1 into a paragraph in the old 4.2. The novelty is that we leverage this to train embedded methods, instead of using that for inference (L. 377).

RePlaid and LangFlow already analyze how CE and MSE shape embedding geometry [...]

RePlaid analyses empirically the embedding geometry of its own model. LangFlow's analysis revolves around eq. (48), the gradients of an MSE loss, and collapse when training only with MSE. We push this further by showing how the DER of an embedded model reduces to that of a one-hot flow, and exploring the gradient of both CE and MSE objectives (App. E.3).

CANDI explicitly interprets SC as supplying pseudo-clean anchors [...]

We concur, and have changed the abstract, introduction and conclusion to better reflect our main contributions, as reviewer yuNT suggested too, with a clearer focus on DoubleCat, its scaling, and the original section 4.2 (embeddings reducing to one-hot DER) (L. 19--22, 54--58, 498--499).

How insights motivate methods. (i) Inference visits states rare at training (prior: [1]); as Y^t\hat Y_t stays off the vertices, DoubleCat trains on such states (ours). (ii) Token identifiability (prior: CANDI) motivates the posterior as SC, and SemiCat's reveals. (iii) One-hot flows are embedded flows (prior: CANDI); we use this to train embedded DoubleCat/SemiCat, and our DER reduction (ours) explains why CE-trained embeddings recover one-hot DERs (§3.1, §3.2, §4.1, §4.2).

Several other explorations are included in App. D and F.

W2, Q2

The proposed variants improve over Plaid w/o SC but remain largely below standard Plaid w/ SC.

We agree that the SC results are better. The goal was to gain understanding on what SC provides, and the token-wise posterior seems to recover part of the performance. While SC remains stronger, it becomes costly when accelerating a model (e.g., a flow map [2] requires 3 forward passes per training step, up to 6 with SC), hence our willingness to bypass it. We add this motivation (L. 167--169).

I would like to see posterior DoubleCat combined with the standard SC and compared against Plaid + SC.

Under the same protocol (pinned endpoints, SC), adding posterior DoubleCat gives 29.64% vs 32.98%; at the default decode it is slightly ahead (26.69% vs 26.08%), but SC alone benefits more from decode tuning. Therefore, the two are not complementary, consistent with addressing the same mismatch. We have included those numbers in App. E.6 (Table 11).

W3, Q3

It would be valuable to see more direct evidence connecting the gains to the proposed mechanism.

We provide new evidence for this. Using OWT trained checkpoints for one-hot flow and DoubleCat, we gathered one-hot and DoubleCat inference trajectories (I^toh\hat I_t^\mathsf{oh}, and I^tdc\hat I_t^\mathsf{dc}, resp.), using 128 NFE Euler integration. We compare the distances between one-hot flow inference and DoubleCat interpolants (ItdcI_t^\mathsf{dc}); DoubleCat inference and DoubleCat interpolants; one-hot flow interpolants (ItohI_t^\mathsf{oh}) and one-hot flow inference. We use the conditional and marginal Wasserstein distances, and MMD. (Details in App. D.7, Figure 8.)

We can see that, for times up to t≈0.75t\approx 0.75 or 0.80.8 (depending on the distance), the train--test gap is lower for DoubleCat, with large advantages until around t=0.5t=0.5. Near t=1t=1, the DoubleCat interpolant's marginals come close to those of one-hot flows by construction, and so the advantage narrows there, and flips slightly in favour of the one-hot flow. Crucially, the one-hot flow inference is closer to DoubleCat's interpolants than it is to its own under all three distances, for all times, although the margin narrows again near t=1t=1.

W4, Q4

the SemiCat gains maybe attributed to the CANDI's revealing mechanism rather than DoubleCat [...]

In our original submission, we provided the 2x2 matrix for this: CANDI is one-hot with reveals (now clarified in the background, L. 148), and SemiCat is DoubleCat+CANDI:

GSM8K accuracy (%, best over samplers)One-hotDoubleCat
Without reveals4.42 (base one-hot flow)16.00 (DoubleCat)
With reveals9.63 (CANDI)32.90 (SemiCat)

This clearly identifies that the gains from CANDI and DoubleCat are complementary, in fact super-additive. DoubleCat is on par with the one-hot flow on OWT, but is clearly superior on TinyGSM: it never hurts, and the gains are substantial for conditional generation. A scaled run is very costly (256 GPUs for about 5 days), and so we find the small scale evidence appropriate; we have nevertheless launched it. We cannot guarantee that it will finish on time, but will report results as soon as possible, and intermediate ones in the meantime. We have not launched a CANDI run.

To conclude, we are thankful for the reviewer's comments that help outline the paper's novelty. We remain available for further clarifications.

References

[1] Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?, Shabalin et al. (2026). [2] Flow Map Matching with Stochastic Interpolants, Boffi et al. (2026).

I sincerely thank the authors for their response, which addresses most of my concerns. I appreciate the clarification of the paper’s novelty, the new analysis of the train–inference mismatch, and the authors’ explicit acknowledgment that adding posterior DoubleCat to Plaid + SC lowers overall accuracy (Tab. 11).

My remaining concern (W4) is whether the gains from DoubleCat alone extend beyond TinyGSM. These gains on TinyGSM are clear, but DoubleCat performs similarly to the baseline on OWT, and the scaled DoubleCat results are on the way.

Overall, I find DoubleCat to be a simple and promising idea supported by a valuable analysis. Therefore, I am raising my score to 3 and look forward to seeing the scaled DoubleCat results.

We thank the reviewer for raising their score, and for a discussion that genuinely improved the paper.

My remaining concern (W4) is whether the gains from DoubleCat alone extend beyond TinyGSM. [...] the scaled DoubleCat results are on the way.

The 1.7B DoubleCat run has not finished yet, so we compare all models at the same fraction of training (57%). DoubleCat alone clearly improves generation over the one-hot flow, at every NFE: at matched entropy, its generative perplexity is 10--26% lower, and SemiCat lowers it by a further factor of ~2 (new App. G, Fig. 14).

On likelihood benchmarks, DoubleCat and the one-hot flow are on par (Table 13). This is expected: these benchmarks score the model on its training interpolant, where no train--inference mismatch arises; the mismatch DoubleCat addresses only appears at sampling time, when the model reads its own predictions.

We have added these results to the paper, and will update them with the final checkpoints.

Reviewer_kuCTOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper carries out an empirical design study for the continuous diffusion language models. The paper proposes DoubleCat, a modified corruption process that interpolates between simplex noise and the target token, then adds Gaussian corruption. Combined with CANDI-style discrete token reveals, the proposed SemiCat provides further performance gains. The paper also explores embedding geometry and self-conditioning. On downstream accuracy-based tasks, the proposed approach achieves performance comparable to or better than the baselines.

Critical strengths
  • The paper provides quite comprehensive empirical studies on the design choices, including the noise schedulers, the intermediate categorical noise, the training objective, and some other hyperparameters. The results may help in developing a CDLM.
  • The paper uses accuracy-based benchmarks like TinyGSM to provide more convincing evaluation results on the model's generative performance, rather than simply relying on Gen PPL and entropy, which are known to be less meaningful.
  • Tab. 3 provides evidence that the proposed SemiCat variant outperforms the compared diffusion baselines, including MDLM/UDLM that operate on discrete tokens.
Critical weaknesses
  • The paper's major results are purely empirical with no theoretical guarantee; yet the paper makes substantive theoretical mistakes and overclaims. These issues significantly lower the credibility of the paper's motivation and soundness.

    • Most critically, the preassumption in Sec 3.1 is theoretically incorrect. The authors claim the mismatch between training and inference is that the model's prediction "never points towards a clean vertex of the simplex". This is incorrect, as the model learns the marginal distribution E[X1∣It=x]\mathbb{E}[X_1\mid I_t=x], so a perfect logit denoiser will naturally be non-one-hot. This never creates a mismatch, as the marginal recovery is already mathematically guaranteed in existing works like MDLM or DFM.

    • Some takeaways are framed in quite overclaimed theoretical terms, despite only empirical observations being available. Others have no concrete, direct experimental evidence to support them.

      Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch.

      This is a fairly strong theoretical claim, although no theoretical analysis is available. The assumed "train–test mismatch" is incorrect. Accuracy on TinyGSM does not automatically translate to likelihood. Similarly, in other takeaways, the accuracy/name error is not directly related to token identifiability. A direct comparison of token recovery accuracy at the same noisy state better supports the claim.

      Cross-entropy training spreads tokens apart, inducing the same DER as a one-hot flow.

      A perfect one-hot flow still learns a marginal logit denoiser; the term "spreading tokens apart" cannot have a meaningful generative implication without specifying a specific time slice. Indeed, by using the same one-hot flow with different time schedulers like t,t2,tt,t^2,\sqrt t, you can arbitrarily advance or postpone the drop in the DER. A more justifiable claim would be, CE promotes earlier logit/probability denoising compared to MSE under the same training and sampling schedulers. Therefore, the current curves in Fig. 3(a) mix the impacts of different time schedulers and the true DER, making the takeaway claim confounded. A direct, fair comparison would involve a variant trained under the ambient setup but with MSE loss instead.

      Large Gaussian noise makes the tokens hard to identify from the current state.

      Existing experimental results in Tab. 2 actually support the opposite. The ambient sphere model consistently gains improvements with a larger embedding size. Plaid also shows non-uniform performance gains with larger embeddings.

      Self-conditioning aids token identifiability as it holds a more identifiable representation of the state, on the convex hull formed by the embeddings.

      The posterior in Eq. 8 can also be viewed as a convex combination of embeddings in WW, so a convex combination does not drive the performance gain.

  • As a primarily empirical study on continuous diffusion language models, the methodological novelty is limited. Most primary techniques are borrowed from existing work, though appropriately cited. For example, the usage of unit norm attributes to CDCD and Plaid; the SemiCat variant from CANDI; and self-conditioning also explored in Fixed-Point Flow. One true difference is the introduction of the double categorical flow. However, due to the theoretical error, I see little concrete motivation for choosing this probability path both theoretically and empirically. Should the introduction of the intermediate uniform state be helpful, then why don't we directly build the probability path directly from Y0Y_0 to X1X_1? Indeed, the resulting UDLM (USDM) shows DoubleCat's consistently inferior performance in Tab. 1, providing weaker evidence for the additional Gaussian prior.

  • The empirical results indicate that the SemiCat is not uniformly the best-performing model on TinyGSM, as shown by combining results in Tab.1 and Tab. 2. Tab.1 alone is misleading, as the available better Plaid results in Tab. 2 for T=1T=1 are not included, so the SemiCat variant appears to achieve the best accuracy overall. Compared to USDM, SemiCat also fails on argmax decoding, runs, and name error.

  • The number of baselines compared remains limited for CDLM in Tab. 3, where there is no embedding-based CDLM. Models like Plaid and CANDI that serve as the direct methodological inspiration should be compared. Additionally, although the paper cites more recent CDLM work that outperforms Plaid (e.g., S-FLM, LangFlow, ELF), it does not compare any of them as additional baselines to more convincingly support the proposed method's better performance.

Questions for authors

Please refer to the weaknesses above. Additionally, I suggest placing captions above the tables.

6 comments

We thank the reviewer for their detailed assessment and astute comments. While we disagree with some of the points that were raised and see a few misunderstandings, we are grateful to the reviewer for raising them, as they have led to direct changes to our submission (highlighted in blue in the revised version). We address the weaknesses that they point out.

W1

[...] the preassumption in Sec 3.1 is theoretically incorrect. The authors claim [...] that the model's prediction "never points towards a clean vertex of the simplex".

We fix this wording (L. 197--203). The optimal denoiser is indeed non-one-hot, and a perfect one recovers the marginals. The mismatch arises with a learnt denoiser, and we now measure it directly (new App. D.7, Figure 8; brief protocol in our answer to T9Dt): on OWT, the states visited by the one-hot flow are nearly disjoint from the training interpolant's for t≥0.75t \geq 0.75 (normalised MMD >0.9> 0.9), and are closer to DoubleCat's training law than to their own at every tt.

Some takeaways are framed in quite overclaimed theoretical terms [...].

Self-conditioning increases the density of highly likely inference regions during training, diminishing the train–test mismatch.

We rely on the well-established results from Shabalin et al. [1], who show that SC mitigates a train--inference mismatch. We correct this lack of context by expounding those findings near the claim (L. 205--206, 253).

A perfect one-hot flow still learns a marginal logit denoiser; the term "spreading tokens apart" cannot have a meaningful generative implication [...]

We clarify the meaning of this sentence in our revision. It is now: "spreading the embeddings of the tokens apart" (L. 428). This was clear in App. E.3, but not in the highlighted sentence. The claim is geometric: since the DER only depends on tt through γt\gamma_t (Prop. 1), curves under a shared schedule differ only through the embedding geometry.

[...] results in Tab. 2 actually support the opposite. The ambient sphere model consistently gains improvements with a larger embedding size. Plaid also shows non-uniform performance gains with larger embeddings.

Our observation holds for Plaid without SC, where its performance drops monotonically. For the ambient sphere, the model is trained with cross-entropy, making token embeddings more identifiable, thus aiding the generation quality on TinyGSM. We had this in the first version at the new L. 424--426. Non-monotonic improvements for Plaid hold for the SC runs, which precisely alleviates identifiability issues.

Self-conditioning aids token identifiability as it holds a more identifiable representation of the state, on the convex hull formed by the embeddings.

While both the token-wise posterior and the SC signal lie in the convex hull of the embeddings, not all signals in the convex hull can improve model performance (e.g., a constant embedding). SC is stronger than the token-wise posterior, but the latter still significantly improves over the no-SC baseline (for T=0.2T=0.2, from 11.98% to 17.97%).

W2

Most primary techniques are borrowed from existing work.

[...] the usage of unit norm attributes to CDCD and Plaid

Unit normalised embeddings are standard, and are not a primary technique in our work.

[...] and self-conditioning also explored in Fixed-Point Flow.

The fixed-point flow paper (which we cite) analyses SC through the lens of fixed-points, and is complementary to our work.

[...] I see little concrete motivation for choosing this probability path both theoretically and empirically.

Empirically, DoubleCat strictly improves on the performance of one-hot-based methods on TinyGSM (e.g., from 4.42% to 15.85% at T=0.5T=0.5). Theoretically, we have added our original motivation: the inference one-hot path takes the form I^t=(1−γt)X0+γtY^t\hat I_t = (1-\gamma_t) X_0 + \gamma_t\hat Y_t, where Y^t\hat Y_t is a path on the simplex (see [3] eq. 82; L. 198--200).

[...] then why don't we directly build the probability path directly from Y0Y_0 to X1X_1? Indeed, the resulting UDLM (USDM) [...]

Building such a path would result in a linear simplex flow, which is known not to scale with vocabulary size (see [2] Figure 2, or [3] Table 28). USDM is a discrete diffusion model.

W3

SemiCat is not uniformly the best-performing model on TinyGSM

Table 1 evaluates one-hot methods and Table 3 (formerly Table 2) embedded ones, hence the separation. Among continuous methods (thus excluding USDM), SemiCat (32.90%, no SC) is on par with the best Plaid + SC variant (32.98% pinned endpoints; 28.43% learnt), and ahead of Plaid + SC in three of four sampler columns (T=0.5T=0.5, T=0.2T=0.2, top-kk=1). Samplers suit models differently, and we tuned both with equal effort.

W4

The number of baselines compared remains limited for CDLM in Tab. 3, where there is no embedding-based CDLM.

Our smaller scale experiments helped us prune the space of CDLMs before launching very expensive 256 GPU runs. SFLM in our reproduction achieves at most 17.8% (now included in Table 3, L. 334--335; matching the paper's number), while SemiCat reaches 32.9%. We have nevertheless launched an SFLM run. We cannot guarantee its finishing in time, but we will report final results as soon as possible, and intermediate ones in the meantime.

Format

We have placed the table titles above, and have reorganised the paper following the other reviewers (Section 5 into a new 3.3, 4.1 merged into 4.2; L. 308, 365).

We thank the reviewer for having raised points that have alleviated some ambiguities in the originally submitted draft. While we do find their rating a bit harsh, we hope that they find our corrections satisfactory.

References

[1] Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It? Shabalin et al. (2026) [2] Dirichlet Flow Matching with Applications to DNA Sequence Design, Stark et al. (2024). [3] Flow Map Language Models, Lee et al. (2026).

I thank the authors for their response. After carefully reading their justification, while some of my concerns about practical baseline comparisons have been partially addressed, I believe the rebuttal only deepens my concerns about the theoretical validity of this work.

Incorrect "Train-Inference Mismatch" Claim

While the authors have acknowledged their incorrect theoretical claim, the current revision does not fully revise it. In Sec. 3.1, the incorrect attribution "instead of following a prediction, follows the path defined by the denoiser on the interior of the simplex" remains. The new sentence added that the mismatch is because the prediction "stays far from the vertices of the simplex", which still uses the same incorrect argument without considering the marginalization.

Furthermore, the comparison in the new Appendix D.7 suffers from the same erroneous claim: YtY_t or ItI_t should be calculated by marginalization over the data distribution. Comparing with a fixed sentence to claim the train-inference gap is not theoretically grounded. This is especially problematic at the stage where most tokens are noisy, yet directly compared to one-hot targets for distributional metrics.

Ungrounded Self-Conditioning Claim

The authors refer to an existing paper. I read the paper and found quantitative results directly related to their density claim in Fig. 2b. In contrast, I see no supporting evidence in this paper. Additionally, since the authors attribute this contribution to existing work, it should not be considered a core contribution, which further deepens my concerns about the paper's limited contributions.

Unaddressed DER Calculation Justification

I acknowledge that DER is only time-dependent via the scheduler γt\gamma_t, which is why I already agreed that the sphere vs. one-hot comparison still holds in my previous review. However, I'm concerned about whether a direct comparison with Plaid provides useful information, since Plaid does not necessarily adopt the same γt\gamma_t as the other two variants. That is why I suggested training a Plaid-like model to isolate each factor and provide a clean comparison. This, however, is never done in the authors' rebuttal.

Remaining Overclaimed Takeaways

Crucially, I emphasize my original intent to scrutinize the paper's takeaways to avoid unsupported or inaccurate claims. For example, the Gaussian variance claim is only applicable to no-self-conditioning and MSE loss, and the self-conditioning improvement is not from a combination from a convex hull. To be clear, I acknowledge the performance improvements in these cases, but the authors' attribution of these gains in their original takeaways is not fully supported. The authors have already acknowledged my points for these two takeaways in their rebuttal, which I appreciate, but the revised manuscript remains misleading, overclaimed, and unqualified as general guidelines for building CDLMs. The current manuscript still makes highly theoretical claims with no direct evidence (e.g., identifiability is not directly measured by recovery rate, but only implicitly by overall accuracy across different tasks).

Motivations For the Proposed Path

I think the authors misunderstood my concern. My concerns can be split into two parts:

  • Theoretically, the authors did not propose justifiable reasons for such probability paths over simple linear paths (the original motivation regarding the train-inference mismatch is theoretically incorrect). This is my major concern.
  • Practically, discrete diffusion models already outperform DoubleCat, so even though DoubleCat outperforms one-hot, my concern is why we should pursue this direction instead of the better-performing UDLM.
Reviewer_yuNTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies several aspects of continuous flows for language modeling, focusing on temporal dynamics, token identifiability, self-conditioning, embedding geometry, and the interactions between these factors.

The authors investigate the role of self-conditioning, propose cheaper alternatives, show that spherical embeddings trained with cross-entropy can exhibit decoding-error behavior similar to one-hot representations, and identify a potential mismatch between the states encountered during training and inference, particularly at times close to t=1t=1.

Motivated by these observations, the paper introduces DoubleCat and SemiCat and evaluates their performance empirically, including experiments at the 1.7B-parameter scale.

Critical strengths
  1. The paper is clear and easy to read.

  2. The problems it tackles are numerous. In addition, it studies how different aspects affect one another.

  3. In particular, I liked the following:

    • Section 4.2: Showing that spherical embeddings suffer from similar token-identifiability issues as one-hot representations, particularly in terms of decoding error rates.
    • Sections 3.2 and 5: Combining DoubleCat with the approach of CANDI, resulting in SemiCat, which achieves strong performance when scaled.
    • Section 3.1: Identifying the potential train/inference mismatch that inspired DoubleCat.
Critical weaknesses
  1. The main weakness, in my opinion, is the emphasis on self-conditioning in the abstract, introduction, and conclusion. It is already well known that self-conditioning helps. Section 4.3 mostly proposes cheaper substitutes for conventional self-conditioning, but does not convincingly establish that improved token identifiability explains their benefits. The analytical posterior is a deterministic function of the noisy input and therefore provides no additional information, although it may provide a more useful representation. Moreover, its use as self-conditioning actually increases name errors relative to the baseline in Table 2. I would shift the focus toward the strongest parts of the paper: Sections 3.1, 4.2, and 5.

  2. While the problem motivating DoubleCat is interesting, the proposed solution is not entirely convincing. Expanding ItI_t in terms of Y0Y_0, X0X_0, and X1X_1 suggests that DoubleCat can be interpreted as a time-dependent modification of the effective source distribution and interpolation schedule. In light of [2], I would investigate whether these changes explain the improvements.

    More importantly, the paper does not establish that DoubleCat assigns considerably more probability mass to inference-like states, particularly for times close to t=1t=1. Even if this could be quantified over some time interval, it would not necessarily show that these states match those actually visited by the learned flow. The fact that ordinary one-hot sampling works best with DoubleCat also deserves further explanation. The intuition is interesting, but in my opinion it has not been explored sufficiently.

  3. I have some reservations about the organization. While I liked that experiments generally followed their corresponding sections, I think Section 5 should precede Section 4. Otherwise, it is unclear whether aspects of Section 4 were incorporated into scaled SemiCat. Section 4.1 complements Section 4.2, but offers limited novelty relative to its length. That space could instead be devoted to Section 3.1, particularly to clarifying the DoubleCat sampling procedure.

Minor Weaknesses

a) The comparison “11.98%, reaching 15.85%” uses different sampling temperatures. At T=0.5T=0.5, the corresponding comparison is 10.84% versus 15.85%.

b) I believe [1] should be cited in the main text, around line 47, given that it was, to my knowledge, among the first papers to propose and test cross-entropy for continuous diffusion with learned embeddings and to identify the importance of time warping.

Questions for authors

Q1. Could the authors describe the sampling of DoubleCat? Q2. Could the authors compare, at least in simplified settings, the true distribution close to time t=1 of their DoubleCat formulation and compare it with the one learned normally at those times by a typical flow [3]?

[1] Dieleman et al. Continuous diffusion for categorical data. 2022 [2] Haxholli. Flow Duality and Source Geometry for Categorical Generation. 2026 [3] Lee et al. Flow Map Language Models: One-step Language Modeling via Continuous Denoising. 2026

1 comment

We thank the reviewer for their appreciation of the paper's contributions, and for their suggestions and remarks. We address them one by one in the following.

W1

The main weakness, in my opinion, is the emphasis on self-conditioning in the abstract, introduction, and conclusion. [...] I would shift the focus toward the strongest parts of the paper.

We agree with the reviewer that the strongest parts of the work are those that they have highlighted, and so we modify the abstract, introduction and conclusion to reflect this better (L. 19--22, 54--58, 498--499). Furthermore, the posterior indeed adds no information, but provides a more useful representation; we have changed the text to state this more clearly (L. 474--475).

W2, Q1, Q2

Could the authors describe the sampling of DoubleCat?

When we studied DoubleCat, we tried a sampler that simulated the two parts of the flow simultaneously (one on the simplex, one in the ambient space). This was a heuristic, and we noticed it performed worse since it dampens the denoiser predictions with an independent draw from the uniform distribution on the simplex (Y0Y_0). We have also realised this was not a valid sampling procedure. Following the reviewer's comment, we found keeping it in the paper to be confusing. We have moved that sampler to App. D.2 (L. 1015) as a side remark, and have deleted that line from Table 1 and L. 241--245. We now only highlight "usual" sampling (Euler integration, L. 214--215), x←x+(γtk+1−γtk)/(1−γtk)(πtkθ(x)−x)x \gets x + (\gamma_{t_{k+1}} - \gamma_{t_k})/(1-\gamma_{t_k})(\pi^\theta_{t_k}(x) - x). DoubleCat should be simply interpreted as a curved stochastic interpolant between Gaussians and one-hots. At inference time, this makes no difference compared to usual flow matching.

In light of [2], I would investigate whether these changes explain the improvements. [...] the paper does not establish that DoubleCat assigns considerably more probability mass to inference-like states

We concur with the reviewer that, following [1], the rewriting It=(1−γt)X0+γt(1−κt)Y0+γtκtX1I_t = (1-\gamma_t) X_0 + \gamma_t(1-\kappa_t)Y_0 + \gamma_t\kappa_t X_1 shows that DoubleCat can be understood as a one-hot flow with a different source distribution and schedule. However, we now show that DoubleCat does close a train--inference gap in the new Figure 8 (App. D.7, pointed to at L. 202--203). We have taken OWT trained checkpoints with simple one-hot flows and DoubleCat, and have gathered, at all times t>0t > 0, basic one-hot inference trajectories (I^toh\hat I_t^\mathsf{oh}) and DoubleCat inference trajectories (I^tdc\hat I_t^\mathsf{dc}), using 128 NFE Euler integration. We compare the distances between one-hot flow inference and DoubleCat interpolants (ItdcI_t^\mathsf{dc}); DoubleCat inference and DoubleCat interpolants; one-hot flow interpolants (ItohI_t^\mathsf{oh}) and one-hot flow inference. We use the conditional and marginal Wasserstein distances, and MMD. (Details are provided directly in App. D.7.)

We can see that, for times up to t≈0.75t\approx 0.75 or 0.80.8 (depending on the distance), the train--test gap is lower for DoubleCat, with large advantages until around t=0.5t=0.5. Near t=1t=1, the DoubleCat interpolant's marginals come close to those of one-hot flows by construction, and so the advantage narrows there, and flips slightly in favour of the one-hot flow. Crucially, the one-hot flow inference is closer to DoubleCat's interpolants than it is to its own under all three distances, for all times, although the margin narrows again near t=1t=1.

W3

We have reorganised the structure, and have moved 5 (Scaling) into a new subsection 3.3, and have merged 4.1 into 4.2, as it is indeed less novel (L. 308, 365, 367).

Minor weaknesses

a) We have fixed the reporting to use the same temperature T=0.5T=0.5 per reviewer's suggestion, where the improvement is even larger (L. 452).

b) We certainly agree that CDCD should appear more prominently in the main part of the paper, given its importance in the field, and have cited it and described it directly on lines 51--53.

To conclude this first round, we thank the reviewer for their time and comments, which have enhanced the paper's structure, clarity, and soundness, especially when coming to show DoubleCat's lower train--test gap. We remain available for further questions.

References

[1] Flow Duality and Source Geometry for Categorical Generation, Haxholli (2026).

Comments on the paper

1 comment

Dear Reviewers,

The author-reviewer discussion phase is ending soon. If you have not done so yet, please read the author rebuttals and respond.

Please recall that, as a reviewer, you have agreed to engage in discussion with the authors to clarify and improve the paper. Failure to do so may result in penalties on your own submissions, as per the official ICLR policy. Given the high volume of submissions, your participation is crucial.

Thank you for your time and help.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.