acceptodds
Under review as a conference paper at ICLR 2027

Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models

Abstract

Diffusion language models are a promising alternative to autoregressive models, yet post-training methods for them largely adapt reward-maximizing objectives. We identify a central failure mode in this setting we call *trajectory locking*: sampled reward-driven updates over-concentrate probability mass onto a narrow set of denoising paths, reducing coverage of alternative correct solutions under repeated sampling. To address this, we propose TraFL (**Tra**jectory **F**low ba**L**ancing), a trajectory-balance objective that trains the policy toward a reward-tilted target distribution anchored to a frozen reference model. We make this practical for diffusion language models with a diffusion-compatible sequence-level surrogate and a learned prompt-dependent normalization. Direct measurements of denoising-path diversity show that TraFL preserves reference-level path diversity (101% on average), compared with 87% for ESPO and 56% for JustGRPO. Across mathematical reasoning and code generation benchmarks, TraFL is the only evaluated post-training method that improves over the base model in every benchmark-length setting, with gains that persist as the sampling budget increases. The improvements transfer to held-out evaluations: TraFL stays above the base model on Minerva Math and is the strongest method on every LiveCodeBench difficulty split.

Primary area:
reinforcement learning
Keywords:
Diffusion Language Models, Reinforcement Learning, Post-Training, Trajectory Balance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3JeWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies trajectory locking in reinforcement learning (RL) post-training of diffusion language models (dLLMs), where probability mass concentrates on a small number of denoising trajectories. To address this, it proposes TraFL, a GFlowNet-inspired method based on trajectory balance that targets a reward-weighted distribution of a frozen reference model. TraFL uses a length-normalized LLaDA ELBO and a learned prompt-dependent normalization term. Theoretically, the paper shows that conventional policy gradients can reduce trajectory diversity even among trajectories producing the same answer. Experiments with LLaDA-8B-Instruct show that TraFL improves Pass@5 across all eight evaluation settings.

Critical strengths

S1. Timely and Important Problem. The concern that RLVR improves single-sample accuracy at the expense of multi-sample coverage is increasingly relevant. Connecting this issue to the unique denoising trajectories of dLLMs is a meaningful contribution.

S2. Consistent Gains with Minimal Overhead. Unlike baselines that underperform the base model in some settings, TraFL improves all eight settings with nearly identical decoding costs.

S3. Direct Mechanistic Evaluation. Beyond accuracy, the paper measures trajectory diversity, conducts LLM-judge evaluations with prespecified statistical tests, and analyzes correct solutions separately. These experiments help connect the proposed mechanism to empirical evidence.

Critical weaknesses

W1. The 1,187-problem “Minerva Math” set appears to be the Algebra subset of the MATH test set, not OCW Courses. It is therefore in-distribution for MATH-trained checkpoints and overlaps with MATH-500, so the held-out/OOD claim holds only for GSM8K-trained models.

W2. All methods score higher on Medium than on Easy, and TraFL scores higher on Hard than on Easy at length 256. This raises concerns about the evaluation pipeline behind the paper’s largest reported gains.

W3. Per-benchmark Pass@1 is not reported. At T=0.6, the gain over the base model shrinks as k increases, which suggests sharpening of already reachable answers rather than expanded coverage.

W4. Proposition 2 shows concentration only within the same rewarded answer, which leaves answer-level diversity unchanged. Theorem 1 therefore cannot establish that trajectory locking limits mode coverage.

W5. Training uses random-order masks independent of the decoding path, while generation and the diversity metric rely on confidence-based unmasking. The preserved trajectory diversity therefore cannot be clearly attributed to the proposed objective.

Questions for authors

Q1. Dataset Composition and Potential Data Overlap

Please clarify whether the Minerva Math benchmark used in the experiments corresponds to the Algebra subset of the MATH test set (lm-eval minerva_math_algebra, 1,187 problems) or the OCW Courses dataset. Additionally, please report the number of overlapping problems with MATH-500 and clarify whether TraFL and each baseline were trained on the original MATH training split (7,500 problems) or the PRM800K split (12,000 problems, including 4,500 test problems).

Q2. Unexpected Difficulty Ordering in LiveCodeBench

In Fig. 3(b), all methods achieve higher performance on Medium than on Easy at both generation lengths (e.g., 61.3 vs. 49.7 for the base model). Notably, at length 256, TraFL achieves higher performance on Hard (62.4) than on Easy (55.4), whereas this reversal is absent in the base model (48.8 vs. 49.7). Could the authors explain these counterintuitive results and why the expected difficulty ordering is reversed?

Q3. Pass@1 Performance and Sources of Improvement

Please provide a table reporting Pass@1 for all methods (base, ESPO, JustGRPO, and TraFL) across benchmarks and generation lengths. Furthermore, at T=0.6, the improvement over the base model decreases from approximately 2.1 percentage points at k=1 to 1.6 percentage points at k=16. Does this suggest that TraFL's gains primarily arise from increasing the probability of already accessible correct answers rather than expanding coverage of distinct solution modes?

Q4. Theoretical Connection Between Trajectory Locking and Mode Coverage

Proposition 2 shows that probability mass can become concentrated on a few trajectories leading to the same rewarded answer. However, such concentration does not necessarily alter the distribution over final answers and therefore does not imply reduced answer-level diversity. In this case, the upper bound in Theorem 1 merely becomes looser. How do Proposition 2 and Theorem 1 establish the claim that trajectory locking limits mode coverage?

Q5. Inconsistency Between Theoretical, Training, and Evaluation Trajectories

It is unclear which sampler defines the trajectory distribution analyzed in the theory. The training objective (Eq. 4) assumes uniformly random unmasking orders, whereas actual generation and trajectory diversity evaluation use confidence-based unmasking. Under random unmasking, policy optimization cannot change the distribution over unmasking orders. Under confidence-based unmasking, however, Eq. 4 does not account for the actual trajectory probabilities induced by the sampler. Could the authors clarify how the trajectories analyzed theoretically, the distribution optimized during training, and the trajectory diversity measured experimentally are connected?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer 3JeW

We thank the reviewer for a very careful review, and for finding the problem "timely and important" and the mechanistic evaluation helpful to "connect the proposed mechanism to empirical evidence."

(W1, Q1, W2, Q2) Out-of-distribution benchmarks

We sincerely thank the reviewer for carefully examining out-of-distribution evaluations. Regarding Minerva Math, the set is indeed lm-eval's minerva_math_algebra from the MATH test set, and 124 of its problems also appear in MATH-500, so it is in-distribution for the MATH-trained checkpoints. All methods are trained on the original MATH training split (7,500 problems, Table 3), not PRM800K, so there is no train–test overlap. In light of these comments, including the unexpected difficulty ordering on LiveCodeBench, we believe that our out-of-distribution evaluation setup, including how out-of-distribution is defined relative to each training set, needs more careful study than the current paper provides, and we do not want to overclaim. We will therefore remove the out-of-distribution section (Sec. 5.4, Fig. 3), the LiveCodeBench judge comparisons, and the corresponding claims from the paper. The remaining results, including the in-distribution benchmarks, the matched-data math comparison, the MATH-500 judge evaluation, and path diversity (Table 1), are unaffected.

(W3, Q3) Pass@1 and the source of the gains

Fig. 1 reports Pass@1 averaged over benchmarks. Per-benchmark Pass@1 / Pass@16 (n=16n=16, T=0.6T=0.6) is below, and we will add it to Table 4.

Task (length)BaseJustGRPOESPOTraFL
GSM8K (256)80.1 / 94.186.5* / 90.783.3* / 94.3*81.9* / 95.7*
GSM8K (512)83.5 / 94.585.6* / 91.583.1 / 94.6*82.8 / 95.5*
MATH-500 (256)34.8 / 60.837.1* / 58.437.6* / 64.8*37.0* / 64.8*
MATH-500 (512)39.1 / 67.040.1* / 62.439.6* / 70.8*39.0 / 68.0*
HumanEval (256)40.1 / 59.837.1 / 51.236.5 / 55.543.8* / 60.4*
HumanEval (512)41.5 / 59.137.5 / 50.041.2 / 61.0*45.5* / 60.4*
MBPP (256)48.5 / 63.453.0* / 68.5*51.8* / 66.5*51.4* / 65.8*
MBPP (512)48.7 / 66.553.2* / 71.6*50.8* / 66.9*51.3* / 67.7*
Avg.52.0 / 70.753.8* / 68.053.0* / 71.8*54.1* / 72.3*

Bold: best per row. *: above the base model. Settings above the base model (Pass@1 / Pass@16, out of 8): JustGRPO 6 / 2, ESPO 5 / 7, TraFL 6 / 8.

On average, TraFL has the highest Pass@1 and Pass@16, although not on every benchmark. At T=0.6T=0.6, part of its gain indeed comes from making accessible answers more likely, but without giving up coverage. TraFL is the only method whose Pass@16 is above the base model in all eight settings, whereas JustGRPO improves Pass@1 as often as TraFL but falls below the base model's Pass@16 in six settings. Moreover, at T=0.9T=0.9 the gap grows with kk (Fig. 1b), and the MATH-500 judge prefers TraFL among correct solutions (Table 5).

(W4, Q4) Proposition 2, Theorem 1, and mode coverage

Theorem 1 is an upper bound, not a monotone relation: complete path collapse precludes multiple answers, while partial concentration need not reduce coverage. We will revise "limits mode coverage" accordingly. Notably, the proof of Prop. 2 (App. D.2) only uses that trajectories share the same reward, so it also applies to trajectories leading to different correct answers. The more probable ones gain mass faster, which concentrates mass on fewer answers and reduces answer-level diversity.

(W5, Q5) Theory, training, and evaluation trajectories

The theory holds for any trajectory distribution induced by the policy, including the confidence-based sampler. Training uses a path-independent score (Eq. 4) and does not model this sampler. However, under confidence-based decoding the unmasking order is determined by the model's token conditionals, which TraFL changes only through its reference-anchored completion-level target. Whether the order distribution stays close to the reference's is then an empirical question, which Table 1 answers with the same sampler for all methods. We will clarify this in Sec. 3.2 and 5.1.

We are grateful for the time and care the reviewer put into this review, in particular the close reading of our out-of-distribution evaluation, which makes the paper more accurate. We hope our responses address the reviewer's concerns, and we would appreciate it if the reviewer would consider reassessing our work in light of them.

Thank you for the detailed responses. I appreciate the authors' efforts, and many of my concerns have been addressed. However, as some of the original claims have been weakened and key theoretical concerns remain, I will maintain my score.

AuthorsOpenReview ↗

reply to reviewer

We thank the reviewer for the response and are glad that many of the concerns have been addressed.

We dropped the OOD claim since the in-distribution / out-of-distribution split is not clean for our checkpoints (Minerva Math algebra is drawn from the MATH test set). The main results do not rely on it. Theorem 1 holds for any trajectory distribution, and our direct evaluation shows that TraFL preserves reference-level path diversity (101% vs. 87% for ESPO and 56% for JustGRPO) while improving Pass@5 and Pass@16 over the base model in all in-distribution settings.

We thank the reviewer again for the valuable feedback.

Reviewer_RdJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The objective of maximizing rewards alone can bring problems, which are referred to as "trajectory locking" in the paper. The author thus proposes distribution matching as another objective.

Critical strengths
  1. The paper does define a dLLM-specific problem. The framing is valid, and the problem indeed exists in many other existing dLLM RL problems.

  2. Good experiment results under fair settings. The mechanism is measured directly. Path diversity, as overlap of unmasking schedules, stays at 101% of the reference model for TraFL vs. 87% for ESPO and 56% for JustGRPO. That’s direct evidence for the claimed mechanism, not just better accuracy.

  3. Presentation is good. Clear and easy to read.

Critical weaknesses
  1. This RL hurts diversity topic is not a new topic. Although it has specific properties related to dLLMs in the paper. The AR side should be included and introduced as well. There is at least work [1][2] on this topic.

  2. The method is an adaptation of FlowRL in dLLMs (which I mean is not a major problem, but it hurts novelty a bit, though).

  3. TraFL’s score is computed on the completion and never looks at the path. So it doesn’t model or balance trajectories. It leaves them alone because the loss ignores them. The locking argument doesn’t distinguish TraFL from ESPO. ESPO also uses a path-independent sequence-level ELBO, so by the paper’s own argument it shouldn’t favor sampled paths either. Yet it loses 13% of path diversity. That suggests the measured loss comes from something else, such as sharper confidence after RL, not from the claimed path-reinforcement mechanism.

[1]Yue et al., 2025, “Does RL really incentivize reasoning beyond the base model? [2]Cui et al., 2025, "The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models"

Questions for authors
  1. The reward-tilted target p_ref·exp(βr)/Z is the known optimum of KL-regularized RL, and FlowRL / GFlowNet fine-tuning already optimizes it with a trajectory-balance loss. Apart from the masked-denoising surrogate, what is new in TraFL?

  2. ESPO also scores whole completions with a path-independent ELBO, so by your argument it shouldn’t reinforce sampled paths either. Why does it still lose about 13% of path diversity?

  3. Does “101% preserved” show more than TraFL staying close to the reference model? How does path diversity relate to KL divergence from the reference, across methods and training steps?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer RdJd

We thank the reviewer for the thoughtful review, and for finding that the paper defines "a dLLM-specific problem" that "indeed exists in many other existing dLLM RL problems," and that the path-diversity measurement is "direct evidence for the claimed mechanism, not just better accuracy."

(W1) Related work on RL and diversity in AR models

Sec. 2 already discusses FlowRL, which addresses diversity under RL for AR reasoning models through distribution matching, and we will add both suggested works there. Yue et al. (2025) show that RL-trained AR models can be overtaken by their base models at large kk, and Cui et al. (2025) attribute entropy collapse to the covariance between action log-probability and advantage. What is specific to dLLMs is that many denoising paths lead to the same completion, so diversity can be lost at the path level even when the completion is fixed.

(W2, Q1) Novelty relative to FlowRL

The reward-tilted target is indeed the optimum of KL-regularized RL, and we will state this explicitly. FlowRL applies trajectory balance to AR models, where a trajectory and a completion coincide and exact token log-probabilities are available. Neither holds for dLLMs, so adapting trajectory balance requires dLLM-specific analysis. Specifically, (i) we identify that terminal reward is indifferent to how mass is split across paths reaching the same completion, and that policy gradient can concentrate this split (Prop. 1, Prop. 2, Theorem 1); (ii) we show that the ideal reward-tilted target reweights completions while preserving the reference path conditional, motivating a practical completion-level approximation that does not explicitly enforce this conditional (Sec. 3.2, App. D.5); (iii) we derive what the masked-diffusion score estimates and characterize its bias (App. C); and (iv) we measure diversity directly at the unmasking-schedule level (Table 1).

(W3, Q2, Q3) Path-independence, ESPO, and the role of the reference

"TraFL's score is computed on the completion and never looks at the path."

This is by design. The ideal target preserves the reference within-completion path distribution (App. D.5). The practical score therefore applies no direct reinforcement based on the realized decoding path, although shared parameter updates can still change path distributions indirectly.

"ESPO also uses a path-independent sequence-level ELBO … Why does it still lose about 13% of path diversity?"

We agree that path-independence alone does not distinguish TraFL from ESPO. The difference lies in their optimization objectives. TraFL minimizes a squared reference-relative balance residual with a learned prompt-dependent normalization, whereas ESPO optimizes group-relative advantages with KL regularization. Although KL-regularized RL can have the same ideal reward-tilted optimum, the practical update dynamics differ. In a group containing both correct and incorrect completions, a correct completion can retain positive advantage under ESPO, subject to its KL and other optimization constraints. Under TraFL, the residual instead decreases as the reference-relative surrogate balance condition is approached.

We hypothesize that this difference reduces concentration over rewarded completions and indirectly helps preserve unmasking-schedule diversity under confidence-based decoding, consistent with the reviewer's explanation involving token confidence. The observed preservation gap (101% for TraFL versus 87% for ESPO) supports the empirical distinction, although it does not establish this explanation causally. We will clarify this in Sec. 5.1.

"Does '101% preserved' show more than TraFL staying close to the reference model?"

Reference-level path diversity is the intended behavior, but the 101% statistic measures unmasking-schedule overlap rather than KL divergence or equality of path distributions. TraFL nevertheless improves Pass@5 over the base model in every benchmark–length setting (Fig. 2) and is preferred for correct-solution diversity on MATH-500 (Table 5). Thus, reference-level measured path diversity coexists with improved task performance. We have not yet measured the relationship between path diversity and reference KL across training, and agree that this would help isolate the mechanism.

We hope these clarifications address the reviewer's concerns and support a more positive assessment of our work.

AuthorsOpenReview ↗

reply to reviewer

Thank you again for your helpful review. Your questions on FlowRL and on ESPO’s path diversity helped us clarify the positioning of our method and the discussion of the path diversity results. Since the discussion period ends soon, we would really appreciate it if you could take a look at our response. We hope it clarifies the points you raised. Thank you for your time and effort.

Reviewer_oPibOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TraFL uses a reference-based, reward-weighted objective and a learned normalization term to post-train a diffusion language model. It improves math and coding results while reporting better preservation of decoding-path and solution diversity.

Critical strengths
  • Preserving multiple correct solutions is a useful goal for diffusion-LM post-training.
  • The evaluation covers sampling budgets, temperatures, and held-out math and coding tasks.
  • The learned-normalization ablation and the discussion of surrogate bias help make the method easier to assess.
Critical weaknesses
  • The practical loss drops the within-completion path term and uses a biased, length-normalized score. The appendix acknowledges this, but it leaves a gap between exact trajectory balance and the mechanism claimed for TraFL.
  • The path-diversity result measures overlap in unmasking positions. This is not trajectory entropy or correct-solution coverage, so the reported preservation percentage does not establish the proposed causal link.
  • The coding baselines use different training data and filtering. A smaller selected dataset is not necessarily a weaker one, so these comparisons do not isolate the training objective.
  • The correct-solution judge compares sets that can contain different numbers of correct answers. Higher accuracy can therefore look like greater diversity without broader coverage at a fixed number of correct samples.
  • Important training details are missing, including the reward scale, prompt batch size, and numeric update budget. This makes the recipe difficult to reproduce from the paper alone.
Questions for authors
  • Can you test the mechanism with a controlled example that separates unmasking-order diversity from terminal solution diversity?
  • Do the coding and judge results hold with matched training data and equal-sized sets of correct solutions?
  • What are the reward scale, prompt batch size, optimizer settings, and number of updates for each checkpoint?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer oPib

We thank the reviewer for the constructive review, and for finding preserving multiple correct solutions "a useful goal" and the ablation and surrogate-bias discussion helpful for assessing the method.

(W1) Gap between exact trajectory balance and the practical loss

Our score is a length-normalized expected log-probability over uniformly random reveal orders (App. C), rather than the exact trajectory likelihood under our sampler. Thus, the practical objective approximates the completion-level balance condition while leaving the within-completion path term unconstrained.

This removes direct path-dependent reinforcement: for a fixed completion, the practical residual is independent of the realized decoding path (App. D.5). Unlike the policy-gradient dynamics in Prop. 2, it does not favor one realized path over another reaching the same completion. Since this alone does not guarantee preserved path diversity, we measure it directly. TraFL retains 101% of the reference path diversity, compared with 87% for ESPO and 56% for JustGRPO (Table 1).

The bias comes from averaging log-probabilities over orders, a lower bound on the uniform reveal-order likelihood (App. C). Prior dLLM RL methods (d1, SPG, ESPO) also rely on approximate likelihoods.

(W2, Q1) Path diversity and the mechanism

Path overlap does not measure solution coverage, so we measure the two separately: unmasking-schedule overlap in Sec. 5.1, and distinct solution approaches via the judge in Sec. 5.5 and App. H.

Our claim is that trajectory diversity imposes an upper bound on solution diversity, not that the two are monotonically related. Theorem 1 and Corollary 1 (Sec. 3.1) establish this necessary condition: complete path collapse precludes multiple terminal modes, although partial path concentration need not reduce coverage. Our results follow this direction. TraFL preserves path diversity and is preferred for correct-solution diversity on MATH-500, while ESPO loses 13% of path diversity and is not more diverse than the base model (Table 5).

(W3) Training data

The math experiments are the controlled comparison, since all methods use exactly the same GSM8K (7,473) and MATH (7,500) training sets (Table 3). Here, TraFL and ESPO reach comparable average Pass@5 (74.2 vs. 74.2), and TraFL is the only method that never drops below the base model (Fig. 2). The difference is in diversity. TraFL keeps 104% and 109% of the reference path diversity on MATH and GSM8K, versus 95% and 79% for ESPO (Table 1), and the judge prefers TraFL over ESPO on MATH-500 (14.3% vs. 5.6% among correct solutions). With data held fixed, TraFL matches ESPO's accuracy while keeping more diversity.

For coding, our filter mainly removes problems a proxy model always solves or always fails (App. F.1), which give zero group-centered reward and would likely cost rollouts without reward signal.

(W4, Q2) Set size in the judge comparison

We compare both models under the same number of samples (16), since what matters in practice is how many distinct correct approaches a user gets from the same number of attempts. Under this view, producing more correct samples is part of the model's coverage rather than a confound, and subsampling correct answers to equal size would measure a different quantity, namely diversity per correct sample. To also control for set size, App. H reports the all-samples setting, where both sides always have 16 responses. There, TraFL is still preferred on MATH-500 over both ESPO (21.8% vs. 14.2%) and the base model (21.8% vs. 15.4%). Notably, ESPO is not preferred over the base model despite its Pass@k gains, which suggests the judge does not simply reward accuracy. The MATH-500 comparisons also use matched training data (see W3).

(W5, Q3) Training details

We train three checkpoints (GSM8K, MATH, code) for one epoch each, with 16 prompts (80 completions) per update, giving 467, 469, and 915 updates. We use β=1.4\beta = 1.4 and AdamW (β1=0.9\beta_1 = 0.9, β2=0.99\beta_2 = 0.99, ϵ=10−8\epsilon = 10^{-8}, weight decay 0.1). We will add these to Sec. 4.1 and release code and checkpoints upon acceptance.

We hope these clarifications address the concerns, and would be grateful if the reviewer would consider a more positive evaluation of our work.

AuthorsOpenReview ↗

Reply to Reviewer

Thank you again for your thoughtful feedback. As the discussion period is closing soon, we wanted to follow up on our rebuttal. We hope it has adequately addressed your concerns. If there are any remaining questions or points that could benefit from further clarification, we would be happy to provide additional details before the deadline. We appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.