acceptodds
Under review as a conference paper at ICLR 2027

Tensor-Train Joint Modeling for Few-Step Discrete Diffusion

Abstract

Discrete diffusion promises orders-of-magnitude faster generation than autoregressive (AR) models for sequential discrete data, yet its full potential of few-step generation has remained out of reach due to a fundamental structural limitation. The conditional-independence assumption underlying current discrete diffusion models introduces a systematic parallelization bias that compounds with the number of tokens unmasked per step, becoming severe in the few-step regime that fast generation requires. We address this with the first framework for explicit joint distribution modeling in discrete diffusion via tensor decomposition, which represents the conditional clean distribution as a low-rank tensor with controllable expressivity. The framework supports both Canonical Polyadic (CPD) and Tensor-Train (TTD) decompositions, and we identify a structural bias of TTD toward dependencies between nearby tokens, formalized through Oseledets' theorem relating TT-rank to unfolding-matrix rank, which is well-suited to sequential data such as natural language and line notations for molecular data. To enable efficient generation, we present an iterative marginal inference procedure with specialization for predetermined position schedules. Our framework integrates into pretrained MDMs through lightweight fine-tuning, yielding substantial improvements in few-step generation at a fraction of the cost of training from scratch.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Discrete Diffusion, Tensor Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qhezOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work tackles the problem of errors that arise from parallel decoding from masked diffusion models (MDMs) owing to their independent factorization of tokens. This paper proposes a novel tensor decomposition framework for parameterizing the conditional joint distribution of masked tokens, which leads to an efficient sampling procedure that can be enabled with lightweight fine-tuning of existing MDMs. The authors demonstrate that their method leads to lower text Gen PPL and improved molecular validity, especially for TTD, in few-step settings.

Critical strengths
  • The paper addresses an important limitation of masked diffusion models.
  • The application of tensor decomposition to incorporating cross-token dependencies is novel and represents an interesting approach.
  • The efficient sampling algorithm is interesting.
Critical weaknesses

Improved presentation

  • I think the presentation can be made more readable by better motivating how these decompositions introduce inter-position dependencies. For example, I found it useful to conceptualize CPD as a single shared categorical latent variable that can be marginalized and TTD as a chain of auxiliary states with first-order dependencies (somewhat akin to a bigram model on this latent chain).
  • Moreover, throughout, I believe it would be useful to provide tensor shapes more prominently. This will also clarify what the overall memory overhead is for each approach and how it compares to full dependencies on one extreme vs. vanilla MDLM on the other.

Limited Experimental Results

  • Currently I found the OWT Gen PPL vs entropy results to not be very compelling. While gen ppl improves for both MDLM and VADD this comes at the expense of diversity, and there is not a clear dominance in terms of the entropy vs gen ppl tradeoff (and additionally I have questions about the entropy computation used as the reported values seem significantly different than those previously reported in the literature). I think it is also important to supply example generated samples to better understand the qualitative nature of the model outputs.
  • I was confused why there was not a VADD + CPD/TTD variant reported for the QM9 results. In my opinion novel sample count (which I assume consists of novel and unique samples) is the most important metric, and VADD is markedly superior here for >10 steps.
  • One suggestion would be to explore additional applications. I think results on the popular Sudoku task and tinyGSM dataset would be quite well suited for the proposed method and for highlighting its low-NFE performance.

I am open to adjusting my score with more compelling empirical results.

Questions for authors
  • The Entropy values in Tables 1, for example, seem very high relative to those typically reported for these models which is closer to < 5.5. What might be causing this discrepancy?
Reviewer_65ghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Masked diffusion models (MDMs) predict each masked token independently given the partially masked sequence. This factorization introduces a parallel-decoding bias that grows with the number of tokens unmasked per step.The paper parameterizes the conditional clean distribution pθ(x∣xt)p_\theta(x \mid x_t) explicitly as a low-rank tensor, using either a Canonical Polyadic (CPD) or a Tensor-Train (TT) decomposition. The decomposition is plugged into pretrained MDMs through a lightweight output-head modification, initialized to preserve the pretrained marginals. Experimentally, MDLM and VADD are fine-tuned on OWT and LM1B, and MDLM is used on QM9. Gains are reported mainly in the few-step regime, with TT outperforming CPD.

Critical strengths
  1. The paper is clearly written and well organized. The motivation is easy to follow.
  2. The parameterization generalizes MDMs and keeps likelihood training.
  3. Experimentally, fine-tuning is lightweight, with some improved results in the few-step regime.
Critical weaknesses
  1. Gains are concentrated where absolute sample quality is poor, so their practical significance is unclear.

On OWT, the improvement almost vanishes at 128 steps (122.08 to119.20). At 8 steps, Gen PPL is still above 600, far from fluent text. OWT results for MDLM are usually reported at roughly 1024 sampling steps, but the evaluation here stops at 128.

  1. The quality–latency evaluation is measured only where the quality gain is smallest.

Sampling time is reported only at 128 steps where the gain is not that substantial. At that budget, the Gen PPL gain is about 2.4% and TTD adds about 24% overhead (2.39 to 2.96 s/seq). A rough log-log interpolation of Table 1 suggests that MDLM with about 1.24× steps would match MDLM+TTD at 8–64 steps. If so, the advantage at equal wall-clock time may largely disappear.

  1. Li et al. (2026), "Breaking the Factorization Barrier in Diffusion Language Models," is cited and positioned as close work, but there is no empirical comparison.
Questions for authors
  1. Please add 512/1024-step results for all methods, so readers can see whether the TTD gain persists, vanishes or reverses at standard budgets. Gen PPL–entropy Pareto curves under a temperature sweep would strengthen this further.
  2. Latency at matched quality. Please report wall-clock time and NFE at 8/16 steps for all methods. Please also plot Gen PPL against wall-clock time to show whether TTD improves quality at equal latency.
  3. Qualitative samples. Please provide randomly selected samples in few step regimes with CPD and TTD.
  4. Entropy is computed “over every token in the 1024 generated sequences,” but the units are not stated and no real-data reference is given, so the reported diversity is hard to interpret. Please confirm: whether this is pooled or mean per-sequence unigram entropy; its units; the value for real OWT text under the same definition. I recommend to use per-sequence entropy (the more common convention) or MAUVE, which would make the numbers comparable with prior work. The same definition would also allow comparing DCD (entropy above 8.6) at matched diversity.
  5. Validity of Eq. (21). With a joint decoder plugged in, is Eq. (21) still a valid ELBO or a surrogate?
Reviewer_yQ3jOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CPD and Tensor-Train decomposition (TTD) to model joint token distributions in masked diffusion, aiming to reduce conditional-independence-induced parallelization bias. It develops efficient marginal inference and lightweight fine-tuning, improving few-step text perplexity and molecular generation. The theoretical motivation emphasizes TTD's suitability for local dependencies in sequential data.

Critical strengths

Explicit joint modeling offers a practical approach to mitigating parallelization bias without auxiliary autoregressive models.

The improvements in few-step generation are meaningful, including a 32.7% perplexity reduction for VADD at 8 steps.

Critical weaknesses
  1. Missing theoretical characterization of parallelization error.

Li and Cai (2025) establish information-theoretic sampling-error bounds governed by token mutual information, while Chen et al. (2026) characterize the expected sampling divergence for arbitrary unmasking schedules using information-theoretic dependence measures. These directly relevant results are missing.

Although the paper cites Zhang et al. (2026) on parallelization bias, it does not quantify how TTD changes the corresponding KL/TV errors. In particular, no bound relates TT-rank, approximation error, parallel commit width, and the number of diffusion steps. It remains unclear which existing factorized-sampling limitations are actually overcome and what error remains.

  1. Training–inference consistency and limited theoretical novelty.

The training still uses random masking, without addressing the inference-induced mask distribution mismatch highlighted by PUMA (Kim et al., 2026). Moreover, Eq. (19) approximates tensor contractions without controlling the resulting sampling error.

The locality argument largely invokes Oseledets' classical TT-rank theorem and a constructed example. Its extension to nonnegative, normalized TT cores is not established by ordinary unfolding ranks alone. The paper also reports sampling latency only at 128 steps, leaving the actual few-step quality–latency trade-off insufficiently demonstrated.

Questions for authors
  1. Parallelization-error guarantees and comparison. Could the authors relate their joint sampler to the KL bounds of Li and Cai (2025) and the exact divergence characterization of Chen et al. (2026)? Since TTD removes the factorized-sampling assumption, which error terms disappear, and how do finite rank and approximate marginal inference affect the remaining error? A controlled comparison of KL/TV (including reverse KL), rank, parallel width, and sampling steps would materially strengthen the contribution.

  2. Why left-to-right, and what about training efficiency? Local linguistic dependencies may favor left-to-right generation, but long-range constraints and bidirectional conditioning may favor adaptive ordering. With exact joint sampling, chain-rule order within a fixed token block should not change its distribution. How much of the observed advantage instead comes from different masking trajectories or optimization difficulty?

Relatedly, DPRM (Bu et al., 2026) develops a plug-in Doob (h)-transform-inspired process-reward ordering policy, transitioning from confidence-based to reward-guided ordering and establishing sample-complexity improvements under explicit assumptions. Could the authors compare with DPRM and test whether its adaptive ordering implicitly captures their locality observation? Combining DPRM with TTD may also reveal complementary gains.

  1. Fair efficiency comparison. Could the authors report end-to-end wall-clock latency at 8–16 steps, accounting for sequential conditional draws and approximate contractions? Ben-Hamu et al. (2025) is already evaluated in Appendix E.2, but comparison at matched latency would be more informative. Parallel-in-time methods such as Yao et al. (2026) also deserve discussion as a complementary acceleration approach.

References

Li and Cai. Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models.

Chen, Cong and Li. Optimal Inference Schedules for Masked Diffusion Models.

Zhang et al. Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective.

Kim et al. Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training.

Bu et al. DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models.

Ben-Hamu et al. Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking.

Yao et al. Accelerating Discrete Diffusion Models with Parallel-In-Time Sampling.

Oseledets. Tensor-Train Decomposition.

Cohen and Rothblum. Nonnegative Ranks, Decompositions, and Factorizations of Nonnegative Matrices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.