acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Good Decoding Path? Selecting Parallel-Decodable Trajectories for Diffusion Large Language Models

Abstract

Diffusion large language models (DLLMs) allow flexible decoding trajectories, but their factorized parallel predictions can produce jointly incompatible tokens even when individual predictions are confident. Existing trajectory-selection methods emphasize marginal predictability or immediate dependency risk, leaving less explored how decoding decisions reshape the dependency structure encountered by subsequent updates. In this paper, we derive an information-theoretic identity linking the current and expected residual factorization costs to the information that revealed tokens provide about remaining positions, and propose Factorization-Aware Candidate Transition Selection (FACTS), a training-free sampler that ranks candidate transitions by combining future uncertainty reduction with a joint-compatibility signal obtained by comparing sequential and factorized scores of the same proposed tokens. Experiments on Dream and LLaDA across mathematical reasoning and code generation show that FACTS is particularly effective under wider parallel decoding.

Primary area:
generative models
Keywords:
Diffusion Language Model, Parallel Decoding, Trajectory Selection

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6UaiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates trajectory selection for diffusion large language models (DLLMs) under parallel decoding. The authors propose Factorization-Aware Candidate Transition Selection (FACTS), a training-free sampler that combines a future uncertainty reduction proxy (ΔH\Delta H) with a positive interaction gain (C(a)C(a)) to evaluate candidate transitions. Despite the rigorous theoretical derivation, the approach exhibits significant flaws regarding the cost-benefit ratio, model scale, and novelty.

Critical strengths

Solid theory: The derivation based on total conditional correlation is rigorous and impressive.

Critical weaknesses
  1. Lack of Cost-Benefit Trade-off and Comparison with Native DLLMs. The efficiency evaluation (Table 4) only compares the optimized FACTS with its unoptimized (naive) version, failing to report the actual wall-clock time, FLOPs, and peak memory overhead compared to native DLLM samplers (e.g., standard Confidence or Entropy). Given that FACTS requires candidate proposal, multiple prefix state generation, and sequential probing, its computational complexity scales significantly with generation length. Under such heavy computational costs, the marginal performance gains are negligible, raising serious doubts about its ROI.
  2. Scale of Evaluated Models is Too Small. The experiments are restricted to 7B and 8B models. Given that FACTS is a training-free inference-time optimization method, the authors should have validated its effectiveness on larger or more advanced models (such as the LLaDA 2.x series or larger parameter variants).
  3. Limited Novelty and Over-reliance on Existing Concepts. Discussions on future uncertainty reduction (entropy reduction/information gain) are already widespread in prior literature (e.g., Info-Gain, LookUM). The core novelty of this work seems restricted to the Positive Interaction Gain C(a)C(a); however, relying solely on a heuristic "sequential-versus-factorized score gap" to capture joint compatibility feels somewhat incremental and insufficient to support the novelty of the entire paper.
  4. Contradiction with the Core Acceleration Value Proposition of DLLMs. The primary motivation for adopting DLLMs over Autoregressive models lies in their parallel decoding capability and generation speed. However, FACTS reintroduces massive sequential computational overhead (testing multiple permutation orders and prefix state rollbacks) into the decoding loop, severely harming generation throughput. The paper fails to justify whether sacrificing the core acceleration advantage of DLLMs for marginal accuracy gains is a worthwhile trade-off.
Questions for authors

Same as Critical Weaknesses

Reviewer_SBV9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Although Diffusion Large Language Models (DLLMs) enable flexible bidirectional parallel denoising, their factorized parallel prediction—which assumes positional independence—can easily lead to semantic incompatibility and an irreducible factorization error. Existing trajectory-selection methods are largely confined to evaluating single-step confidence or local dependency risks, overlooking how current decoding decisions fundamentally reshape the dependency structure encountered by subsequent steps.

To address this issue, we adopt an information-theoretic perspective and derive the Transition Factorization Identity, theoretically establishing that minimizing the sum of current and future factorization errors is strictly equivalent to maximizing the mutual information between the revealed tokens and the remaining masked positions. Building on this insight, we propose FACTS, a completely training-free decoding sampler. At each transition, FACTS jointly evaluates two signals: Future Uncertainty Reduction and Positive Interaction Gain.

Furthermore, to mitigate the computational overhead of combinatorial scoring, we develop a subset-lattice prefix-state reuse mechanism, achieving substantial computational speedups while preserving exact mathematical equivalence. Extensive experiments on mathematical reasoning (GSM8K, MATH-500) and code generation (HumanEval, MBPP) using two representative models, Dream-7B and LLaDA-8B, demonstrate that FACTS is particularly effective under wide parallel decoding settings, establishing a new decoding paradigm for high-quality, high-throughput, and few-step DLLM generation.

Critical strengths
  1. The paper derives the Transition Factorization Identity from an information-theoretic perspective, mathematically proving that minimizing the total (current plus residual) factorization error—measured by Total Correlation—across the decoding trajectory is strictly equivalent to maximizing the conditional mutual information between the revealed tokens and the remaining positions. This provides a rigorous mathematical closed-loop foundation for lookahead and information-gain-based trajectory selection in diffusion language models, making a solid theoretical contribution.

  2. The empirical results demonstrate strong and consistent advantages: as the parallel decoding width K increases, traditional baselines suffer catastrophic performance drops, whereas the performance lead of FACTS widens substantially. This remarkable robustness indicates that the proposed method effectively tackles the core bottleneck of large-stride parallel decoding, offering a substantial push toward high-throughput, low-latency DLLM inference.

Critical weaknesses
  1. At each decoding transition, FACTS requires a substantial number of additional forward passes to evaluate N candidate actions and perform local sequential scoring, resulting in computational overhead far exceeding that of standard decoding. Crucially, the paper does not report end-to-end wall-clock latency or throughput.

  2. All empirical evaluations in the paper are conducted under the without KV caching setting. However, KV caching is an indispensable acceleration component in practical LLM deployment; because FACTS frequently modifies the mask states of different positions and evaluates multiple prefix states, the paper leaves unexplored and unverified how this mechanism can be compatibly integrated with modern, highly efficient KV caching schemes.

Questions for authors
  1. Provide a direct comparison of the end-to-end wall-clock time (or generation latency / throughput in tokens per second) between FACTS and the primary baselines on the same hardware? Under an equivalent total computational budget (or latency constraint), does FACTS still maintain a clear and significant advantage on the quality–speed Pareto frontier compared to simpler heuristics running at smaller strides?

  2. Further discuss and clarify the engineering compatibility and feasibility of FACTS with KV caching mechanisms? Specifically, does the dynamic unmasking of different prefix subsets during the prefix-state reuse evaluation disrupt or invalidate standard KV caching?

  3. Extend the evaluation benchmarks to open-ended generation tasks, such as MT-Bench, AlpacaEval, or long-form summarization, to test whether the factorization-aware strategy maintains its efficacy in less rigid, non-math/code scenarios.

Reviewer_ii2YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces Factorization-Aware Candidate Transition Selection (FACTS), a training-free sampling method that considers candidate transitions by scoring based on minimizing the residual factorization error left for the future steps (future uncertainty reduction) and a positive interaction gain term that exploits the joint-compatibility signal from comparing the sequential and factorized scores of the same proposed tokens to have additional predictive support. The authors compare their method against baselines (e.g: KLASS, LookUM, and Info-Gain) and claim improved performance across benchmarks.

Critical strengths
  1. The gap is well-motivated: parallel decoding can commit individually plausible but jointly incompatible tokens, existing methods that use marginal look ahead criteria do not explicitly detect this.

  2. The method is well-formulated: sections 2 and 3 identify the factorization gap, correctly link the future uncertainty proxy to it, clearly define the positive interaction gain and provide a practical implementation for it using exact prefix-state reuse

Critical weaknesses
  1. Lack of statistical significance for benchmarks: no seed variance, means, or confidence intervals are provided for the benchmarks, it is unclear whether the gains (especially in near-tie scenarios) are within seed noise.

  2. Empirical scope is narrow: Primarily comparisons against baselines on efficiency (e.g. wall-clock latency and number of forward passes to completion) under matched hardware, batch size and sequence length are missing.

Questions for authors
  1. Why were the statistical bounds for the results in Tables 1, 2 and 3 omitted? Could you provide the results under multiple seed runs and state the mean and confidence intervals?

  2. Why is positive clipping done in eq. 9? Why is the clipping done before averaging and not after it? For the case when the differences over k candidates have a larger negative magnitude than the positive magnitude, should the overall Positive Interaction Gain be positive or zero?

  3. Could you provide a comparison of FACTS against the baselines on efficiency under matched quality? Primarily providing the wall-clock latency and the number of forward passes to completion figures that produced the benchmark results in Table 1 and 2 under matched hardware, batch size and sequence length?

  4. Does FACTS empirically have a lower residual factorization error and higher positive interaction gain compared to baselines throughout the decode trajectory? Could you provide an experiment that compares the curves for these 2 metrics on some baselines across the decode steps?

Reviewer_szw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies trajectory selection for diffusion large language models under factorized parallel decoding. It argues that a decoding decision affects not only the reliability of currently committed tokens, but also the factorization difficulty faced by subsequent updates. Based on a total-correlation identity linking current and residual factorization costs to the information that revealed tokens provide about remaining positions, the paper proposes FACTS, a training-free sampler that ranks candidate transitions by Future Uncertainty Reduction (ΔH\Delta H) and Positive Interaction Gain (CC). The latter compares sequential and factorized scores of the same proposed tokens under multiple reveal orders, with prefix-state reuse to reduce redundant evaluations.

Experiments on Dream-v0-Instruct-7B and LLaDA-8B-Instruct across GSM8K, MATH-500, HumanEval, and MBPP show that FACTS is particularly effective under wider parallel decoding. The paper also reports component ablations, shared-candidate ranking diagnostics, and prefix-reuse efficiency results.

Critical strengths
  1. Prior work on DLLM trajectory selection largely focuses on current-state confidence/entropy/margin heuristics or on future marginal uncertainty reduction through lookahead and information-gain criteria. This paper unifies future uncertainty reduction with assignment-level joint compatibility under a factorization-aware view, providing a more systematic treatment of parallel-decoding trajectory quality.

  2. To operationalize this view, the paper introduces Future Uncertainty Reduction as a model-based proxy for remaining mutual information, and Positive Interaction Gain as a sequential-versus-factorized compatibility signal. Prefix-state reuse further reduces redundant sequential evaluations without changing the FACTS objective.

  3. Experiments on Dream and LLaDA across math and code benchmarks show that the method is especially effective under wider parallelism. For example, on Dream at K=4K=4, FACTS improves the zero-shot average from 33.4 (LookUM) / 31.0 (Info-Gain) to 42.0, with large gains on MBPP. Component ablations and shared-pool ranking diagnostics further support that combining ΔH\Delta H and CC is beneficial.

Critical weaknesses
  1. The proposed Future Uncertainty Reduction is closely related to existing information-gain / lookahead criteria, while Positive Interaction Gain is a practical probe of joint compatibility. The paper should provide a clearer comparison with recent dependency-aware decoding methods such as PUNT, DEMASK, DAWN, and DOS, discussing methodological differences and empirical effectiveness. In particular, since CC is presented as the distinctive factorization-aware signal beyond information gain, direct experimental comparisons would better justify the claimed benefit.

  2. The experimental evaluation fixes KK and therefore the number of decoding transitions, but FACTS evaluates NN candidates plus multiple sequential prefix states per candidate. Table 4 reports speedups only relative to FACTS without prefix-state reuse, not relative to Info-Gain, LookUM, or local heuristics. It would be useful to report matched total model rows, wall-clock latency, or accuracy-versus-NFE curves to assess whether the gains remain under equal compute.

  3. The LookUM baseline is implemented as a simplified one-step active-block adaptation without the original SMC/NIS path-pool machinery. As a result, the margin over LookUM may partly reflect a weakened baseline rather than a fully matched comparison with the strongest published lookahead system.

  4. The paper does not thoroughly analyze hyperparameter sensitivity for λH/λC\lambda_H/\lambda_C, the number of proposal attempts NN, and the positive-part operator in C(a)C(a). Additional ablations here would strengthen confidence that the reported operating point is robust rather than specially tuned.

Questions for authors
  1. Under a matched total number of model forwards or wall-clock budget, how does FACTS compare with Info-Gain and LookUM on accuracy–NFE or accuracy–latency curves?
  2. Can the paper provide direct comparisons against recent dependency-aware parallel-decoding methods such as PUNT, DEMASK, DAWN, or DOS under the same fixed-KK protocol?
  3. Does the LookUM baseline reproduce the published full LookUM behavior closely enough that the reported margins remain meaningful?
  4. How sensitive are the main results to λH/λC\lambda_H/\lambda_C, the number of proposal attempts NN, and the positive-part operator in C(a)C(a)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.