- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies trajectory locking in reinforcement learning (RL) post-training of diffusion language models (dLLMs), where probability mass concentrates on a small number of denoising trajectories. To address this, it proposes TraFL, a GFlowNet-inspired method based on trajectory balance that targets a reward-weighted distribution of a frozen reference model. TraFL uses a length-normalized LLaDA ELBO and a learned prompt-dependent normalization term. Theoretically, the paper shows that conventional policy gradients can reduce trajectory diversity even among trajectories producing the same answer. Experiments with LLaDA-8B-Instruct show that TraFL improves Pass@5 across all eight evaluation settings.
Critical strengths
S1. Timely and Important Problem. The concern that RLVR improves single-sample accuracy at the expense of multi-sample coverage is increasingly relevant. Connecting this issue to the unique denoising trajectories of dLLMs is a meaningful contribution.
S2. Consistent Gains with Minimal Overhead. Unlike baselines that underperform the base model in some settings, TraFL improves all eight settings with nearly identical decoding costs.
S3. Direct Mechanistic Evaluation. Beyond accuracy, the paper measures trajectory diversity, conducts LLM-judge evaluations with prespecified statistical tests, and analyzes correct solutions separately. These experiments help connect the proposed mechanism to empirical evidence.
Critical weaknesses
W1. The 1,187-problem “Minerva Math” set appears to be the Algebra subset of the MATH test set, not OCW Courses. It is therefore in-distribution for MATH-trained checkpoints and overlaps with MATH-500, so the held-out/OOD claim holds only for GSM8K-trained models.
W2. All methods score higher on Medium than on Easy, and TraFL scores higher on Hard than on Easy at length 256. This raises concerns about the evaluation pipeline behind the paper’s largest reported gains.
W3. Per-benchmark Pass@1 is not reported. At T=0.6, the gain over the base model shrinks as k increases, which suggests sharpening of already reachable answers rather than expanded coverage.
W4. Proposition 2 shows concentration only within the same rewarded answer, which leaves answer-level diversity unchanged. Theorem 1 therefore cannot establish that trajectory locking limits mode coverage.
W5. Training uses random-order masks independent of the decoding path, while generation and the diversity metric rely on confidence-based unmasking. The preserved trajectory diversity therefore cannot be clearly attributed to the proposed objective.
Questions for authors
Q1. Dataset Composition and Potential Data Overlap
Please clarify whether the Minerva Math benchmark used in the experiments corresponds to the Algebra subset of the MATH test set (lm-eval minerva_math_algebra, 1,187 problems) or the OCW Courses dataset. Additionally, please report the number of overlapping problems with MATH-500 and clarify whether TraFL and each baseline were trained on the original MATH training split (7,500 problems) or the PRM800K split (12,000 problems, including 4,500 test problems).
Q2. Unexpected Difficulty Ordering in LiveCodeBench
In Fig. 3(b), all methods achieve higher performance on Medium than on Easy at both generation lengths (e.g., 61.3 vs. 49.7 for the base model). Notably, at length 256, TraFL achieves higher performance on Hard (62.4) than on Easy (55.4), whereas this reversal is absent in the base model (48.8 vs. 49.7). Could the authors explain these counterintuitive results and why the expected difficulty ordering is reversed?
Q3. Pass@1 Performance and Sources of Improvement
Please provide a table reporting Pass@1 for all methods (base, ESPO, JustGRPO, and TraFL) across benchmarks and generation lengths. Furthermore, at T=0.6, the improvement over the base model decreases from approximately 2.1 percentage points at k=1 to 1.6 percentage points at k=16. Does this suggest that TraFL's gains primarily arise from increasing the probability of already accessible correct answers rather than expanding coverage of distinct solution modes?
Q4. Theoretical Connection Between Trajectory Locking and Mode Coverage
Proposition 2 shows that probability mass can become concentrated on a few trajectories leading to the same rewarded answer. However, such concentration does not necessarily alter the distribution over final answers and therefore does not imply reduced answer-level diversity. In this case, the upper bound in Theorem 1 merely becomes looser. How do Proposition 2 and Theorem 1 establish the claim that trajectory locking limits mode coverage?
Q5. Inconsistency Between Theoretical, Training, and Evaluation Trajectories
It is unclear which sampler defines the trajectory distribution analyzed in the theory. The training objective (Eq. 4) assumes uniformly random unmasking orders, whereas actual generation and trajectory diversity evaluation use confidence-based unmasking. Under random unmasking, policy optimization cannot change the distribution over unmasking orders. Under confidence-based unmasking, however, Eq. 4 does not account for the actual trajectory probabilities induced by the sampler. Could the authors clarify how the trajectories analyzed theoretically, the distribution optimized during training, and the trajectory diversity measured experimentally are connected?
3 comments
Thank you for the detailed responses. I appreciate the authors' efforts, and many of my concerns have been addressed. However, as some of the original claims have been weakened and key theoretical concerns remain, I will maintain my score.
reply to reviewer
We thank the reviewer for the response and are glad that many of the concerns have been addressed.
We dropped the OOD claim since the in-distribution / out-of-distribution split is not clean for our checkpoints (Minerva Math algebra is drawn from the MATH test set). The main results do not rely on it. Theorem 1 holds for any trajectory distribution, and our direct evaluation shows that TraFL preserves reference-level path diversity (101% vs. 87% for ESPO and 56% for JustGRPO) while improving Pass@5 and Pass@16 over the base model in all in-distribution settings.
We thank the reviewer again for the valuable feedback.
Reply to Reviewer 3JeW
We thank the reviewer for a very careful review, and for finding the problem "timely and important" and the mechanistic evaluation helpful to "connect the proposed mechanism to empirical evidence."
(W1, Q1, W2, Q2) Out-of-distribution benchmarks
We sincerely thank the reviewer for carefully examining out-of-distribution evaluations. Regarding Minerva Math, the set is indeed lm-eval's
minerva_math_algebrafrom the MATH test set, and 124 of its problems also appear in MATH-500, so it is in-distribution for the MATH-trained checkpoints. All methods are trained on the original MATH training split (7,500 problems, Table 3), not PRM800K, so there is no train–test overlap. In light of these comments, including the unexpected difficulty ordering on LiveCodeBench, we believe that our out-of-distribution evaluation setup, including how out-of-distribution is defined relative to each training set, needs more careful study than the current paper provides, and we do not want to overclaim. We will therefore remove the out-of-distribution section (Sec. 5.4, Fig. 3), the LiveCodeBench judge comparisons, and the corresponding claims from the paper. The remaining results, including the in-distribution benchmarks, the matched-data math comparison, the MATH-500 judge evaluation, and path diversity (Table 1), are unaffected.(W3, Q3) Pass@1 and the source of the gains
Fig. 1 reports Pass@1 averaged over benchmarks. Per-benchmark Pass@1 / Pass@16 (n=16, T=0.6) is below, and we will add it to Table 4.
Bold: best per row. *: above the base model. Settings above the base model (Pass@1 / Pass@16, out of 8): JustGRPO 6 / 2, ESPO 5 / 7, TraFL 6 / 8.
On average, TraFL has the highest Pass@1 and Pass@16, although not on every benchmark. At T=0.6, part of its gain indeed comes from making accessible answers more likely, but without giving up coverage. TraFL is the only method whose Pass@16 is above the base model in all eight settings, whereas JustGRPO improves Pass@1 as often as TraFL but falls below the base model's Pass@16 in six settings. Moreover, at T=0.9 the gap grows with k (Fig. 1b), and the MATH-500 judge prefers TraFL among correct solutions (Table 5).
(W4, Q4) Proposition 2, Theorem 1, and mode coverage
Theorem 1 is an upper bound, not a monotone relation: complete path collapse precludes multiple answers, while partial concentration need not reduce coverage. We will revise "limits mode coverage" accordingly. Notably, the proof of Prop. 2 (App. D.2) only uses that trajectories share the same reward, so it also applies to trajectories leading to different correct answers. The more probable ones gain mass faster, which concentrates mass on fewer answers and reduces answer-level diversity.
(W5, Q5) Theory, training, and evaluation trajectories
The theory holds for any trajectory distribution induced by the policy, including the confidence-based sampler. Training uses a path-independent score (Eq. 4) and does not model this sampler. However, under confidence-based decoding the unmasking order is determined by the model's token conditionals, which TraFL changes only through its reference-anchored completion-level target. Whether the order distribution stays close to the reference's is then an empirical question, which Table 1 answers with the same sampler for all methods. We will clarify this in Sec. 3.2 and 5.1.
We are grateful for the time and care the reviewer put into this review, in particular the close reading of our out-of-distribution evaluation, which makes the paper more accurate. We hope our responses address the reviewer's concerns, and we would appreciate it if the reviewer would consider reassessing our work in light of them.