acceptodds
Under review as a conference paper at ICLR 2027

Flexible Reveal-Orders for Reinforcement Learning of Masked Diffusion Language Models

Abstract

Masked diffusion language models (MDLMs) have emerged as a promising alternative to autoregressive language models, motivating growing interest in reinforcement learning (RL) to improve their reasoning capabilities. RL for MDLMs, however, requires a tractable surrogate for the intractable order-agnostic likelihood, which marginalizes over a combinatorially large number of possible reveal orders. Existing approaches largely follow two strategies: ELBO-based methods optimize a tractable lower bound, while trajectory-based methods optimize the likelihood along a selected denoising trajectory. In this work, we provide a unified view of these approaches through the distribution over reveal orders underlying the log-likelihood gradient. Specifically, we show that the exact log-likelihood gradient is an average of per-order gradients under the posterior distribution over reveal orders, while ELBO-based and trajectory-based methods correspond to two extremes: a uniform distribution and a Dirac distribution concentrated on a single order, respectively. Motivated by this perspective, we introduce a flexible family of confidence-based distributions over reveal orders that interpolates between these extremes. Our approach can be integrated into existing RL methods without additional model evaluations. Across mathematical reasoning and planning tasks, we show that confidence-based order measures consistently improve both ELBO-based and trajectory-based RL baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion LLM, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CBtzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a unified perspective of the two methods of policy optimization for reinforcement learning in diffusion language models (DLMs): ELBO-based lower bound on sequence likelihood and an exact trajectory likelihood of that specific roll-out. The work shows that the exact likelihood gradient of the sequence is the expectation of per-order gradients where the expectation is over the posterior on the reveal orders. The ELBO uses a proxy distribution of uniform over the reveal orders and trajectory-based methods use a Dirac distribution on a specific trajectory, forming the two points on the spectrum. The paper introduces a flexible single-parameter family of distributions on "reveal orders" based on confidence that interpolates between these two points showing improved performance on held-out checkpoint evaluation, with demonstrable gains in logical reasoning benchmarks (Countdown and some generation lengths for Sudoku).

Critical strengths
  • Providing an interesting mathematical unification of ELBO-based and trajectory-based methods as two approximations of the exact sequence policy gradient.

  • Mathematical theory on gradient error bound and direct motivation for parametrization.

Critical weaknesses
  • [Soundness] Weak results on mathematical reasoning benchmarks.
  • [Novelty] Missing relevant prior work [1], which proposes the "right" partial states/masks and also an interesting trajectory-based method.

[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

  • [Presentation] It is unclear how the intermediate states are constructed. Do we need extra forward passes to get the partial states (unseen in rollouts) in addition to while computing the policy gradients?
Questions for authors
  • Can you include coding benchmarks (MBPP, HumanEval) to strengthen the paper?

  • Can you compare and contrast with the trajectory-based prior work [1]?

[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.

  • Can you please clarify how are unseen partial states sampled and whether they need additional forward passes?
3 comments

We thank the reviewer for their comments, which help us better position the paper within the literature and improve its clarity.

1. Relation to [1] (Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages, https://arxiv.org/abs/2603.12554)

We thank the reviewer for pointing out this relevant recent paper, which we will include and discuss in the revised manuscript. The paper is indeed relevant but has a contribution which is orthogonal to ours. While both works exploit the structure of the denoising process to improve RL, they address distinct questions. Kunde et al. [1] formulate denoising as an MDP and propose selecting high-entropy denoising steps for policy updates, together with stepwise advantage estimation. Their entropy criterion determines which intermediate states along a sampled trajectory should contribute to optimization. The sampled reveal order itself is determined during the rollout phase, as is common in trajectory-based approaches.

In contrast, our work investigates the distribution over reveal orders underlying the sequence-likelihood gradient. We show that the exact gradient averages per-order gradients under the posterior over reveal orders, whereas existing ELBO-based and single-trajectory approaches correspond to different choices of order measure. Our confidence-based distribution provides a temperature-controlled family of such measures, and we study its approximation quality through chi-squared divergence, effective sample size, and gradient error.

Thus, although both methods depart from uniform treatment of denoising decisions, they operate on different dimensions: [1] allocates computation across steps of a trajectory, whereas our method changes the measure over possible trajectories. The two approaches are complementary and could potentially be combined.

2. Partial states and cost

Our method is built on top of the baseline estimator: it only changes which reveal orders are scored, and it uses as confidences the token probabilities that the baseline already computes. It therefore requires no additional forward passes.

τ-ESPO. No new partial states are introduced: masks mm are sampled exactly as in ESPO. ESPO runs one forward pass on ymy_m, which gives pθ(yi∣ym,x)p_\theta(y_i \mid y_m, x) for every masked position ii, and averages their log-probabilities uniformly. We reuse these same probabilities as weights:

∑imiℓlog⁡pθ(yi∣ym,x)⟶∑isg[qc,iτ(m)]milog⁡pθ(yi∣ym,x),qc,iτ(m)∝pθ(yi∣ym,x)1/τ,\sum_{i} \frac{m_i}{\ell} \log p_\theta(y_i \mid y_m, x) \longrightarrow \sum_{i} \mathrm{sg}\big[q^\tau_{c,i}(m)\big] m_i \log p_\theta(y_i \mid y_m, x), \qquad q^\tau_{c,i}(m) \propto p_\theta(y_i \mid y_m, x)^{1/\tau},

where ℓ\ell is the number of masked positions and sg denotes the stop-gradient. The masks and the forward passes are the same as in ESPO.

τ-d2-StepMerge. d2-StepMerge splits an order into NN segments B1,…,BNB_1, \dots, B_N and scores each segment with one forward pass on the state y(n)y^{(n)} at its start (Eq. 16):

log⁡π^θ(y∣x)=∑n=1N∑i∈Bnlog⁡pθ(yi∣y(n),x).\log \hat\pi_\theta(y \mid x) = \sum_{n=1}^{N} \sum_{i \in B_n} \log p_\theta\big(y_i \mid y^{(n)}, x\big).

d2-StepMerge takes the segments from the rollout order. We instead sample them sequentially: the pass on y(n)y^{(n)} gives the probabilities of all masked positions, from which we draw BnB_n with weights ∝pθ(yi∣y(n),x)1/τ\propto p_\theta(y_i \mid y^{(n)}, x)^{1/\tau}; the same pass scores BnB_n, and unmasking it gives y(n+1)y^{(n+1)}. The partial states y(n)y^{(n)} therefore differ from those of the rollout, but there is still one forward pass per segment: NN per order, as in d2-StepMerge.

We acknowledge that the implementation details were not clear enough in the paper. We will fix this by adding an algorithm box for both procedures.

3. Coding benchmarks

We agree with the reviewer that providing results for coding benchmarks would strengthen the paper, but we could not run them during the rebuttal. We will try to include MBPP and HumanEval in the final version and mention their absence as a limitation.

Reviewer_CBtzOpenReview ↗

Reply

Thanks for the response and apologies for the late reply. I want to understand the distinction of your algorithm in ESPO and d2 as baselines clearly:

  1. In τ\tau-ESPO, the rollout procedure is the same as the baseline, but the relative weights of the gradient depends on the τ\tau (and the family of disturbutions).
  2. In τ\tau-d2-StepMerge, the roll procedure changes from baseline to based on the τ\tau family of reveal order distribution, but the there is no additional reweighting of the gradient terms like in the last ESPO case.

Can you confirm that these statements are correct?

Sorry for the late reply. We thank the reviewer for the follow-up. Both statements are correct if the reviewer is not referencing the rollout during the generation of the responses but the rollout during their scoring. In more detail:

  1. In τ\tau -ESPO, the rollouts and the mask sampling are the same as in ESPO and only the weights of each position are changed following our τ\tau dependent approach.

  2. In τ\tau -StepMerge there is indeed no additional reweighting: Eq. (16) is used unchanged, with every token weighted as in d2-StepMerge. What changes is the reveal order used to score the generated response: given a generated response yy, d2-StepMerge uses the order realized during generation phase, whereas we sample a new order of the same response from the confidence measure, segment by segment.

Concerning the generation phase however, we want to stress that responses yy are generated exactly as in the corresponding baseline. Our method only acts on how the log-likelihood gradient of an already generated yy is estimated. The table below summarizes the differences.

ESPOτ-ESPOd2-StepMergeτ-d2-StepMerge
Generation of response yystandardsame as ESPOstandardsame as d2-StepMerge
Partial states scoredmasks mm sampled by ESPOsame masks as ESPOstates along the generation rollout orderstates along an order sampled from qcτq_c^\tau
Weight of each scored token1/ℓ1/\ellqc,iτ(m)q^\tau_{c,i}(m), stop-gradient11
Order measureuniform, via weightingconfidence, via weightingDirac on the generation rollout order, via partial statesconfidence, via partial states
Forward passesone per maskone per maskNN per orderNN per order
Reviewer_UjXLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for improving reinforcement learning in masked diffusion language models by considering the distribution over token reveal orders. The authors first provide a unified view of existing ELBO-based and trajectory-based methods, showing that their likelihood gradients can be expressed as expectations over different reveal-order distributions. This motivates introducing a confidence-based distribution, controlled by a temperature parameter, that interpolates between uniform sampling and greedy confidence-based orders.

The proposed method is then integrated into two existing RL approaches, ESPO and d2-StepMerge, without additional model evaluations according to the authors. On the experimental side, the method is evaluated on mathematical reasoning and planning tasks, and the authors also investigate the choice of temperature through an effective sample size diagnostic.

Critical strengths
  • The paper treats an important problem in the area of RL for diffusion language models, concerning the difficulty of computing sequence likelihoods. It is important because we can not use methods developed for AR LLMs, where the computation of the likelihood is quite easy. I find the approach of unifying existing methods through their distributions over reveal orders interesting, even though the main gradient identity itself follows from Fisher's identity.

  • The proposed method is relatively simple and can be incorporated into existing RL algorithms. The fact that the authors demonstrate improvements on both ELBO-based and trajectory-based methods is interesting, especially if these improvements can indeed be obtained without additional model evaluations.

  • The paper is well written overall, with a clear presentation of the background and existing approaches. I also appreciate the effort to motivate the choice of temperature through the ESS diagnostic and the gradient-distance experiments, rather than relying only on downstream RL performance.

Critical weaknesses
  • My main concern is about the correspondence between the mathematical framework and the algorithms actually used in practice. First, the exact likelihood gradient is derived for a uniform mixture over reveal orders, which does not necessarily correspond to the likelihood induced by the actual confidence-based generation procedure. Second, it is unclear to me how the proposed reveal-order distribution can be sampled exactly in the trajectory-based implementation without additional model evaluations, given the step-merging approximation. These points are important because the main motivation of the paper relies on approximating the exact likelihood gradient.

  • Despite interesting empirical results, I feel that the experimental evidence remains limited. The largest gains are obtained on Countdown and Sudoku, whereas the improvements on mathematical reasoning benchmarks are considerably smaller. Moreover, the experiments are restricted to one base model, and the paper does not clearly report variability across independent training seeds. The 200 random evaluation splits are a useful addition, but they do not measure training variability.

Questions for authors

I have four main questions, mostly concerning the relationship between the theoretical framework and the practical implementation.

  1. My first question concerns the definition of the exact sequence likelihood in Equation (4). The authors define this likelihood by marginalizing over uniformly distributed reveal orders. However, the practical LLaDA sampling procedure uses confidence-based unmasking, with multiple tokens potentially revealed at each denoising step. Correct me if I am wrong, but the resulting generation distribution should therefore differ from the uniform-order mixture considered in Equation (4). In that case, Proposition 1 gives the exact gradient of the uniform-order likelihood, but not necessarily the exact policy gradient corresponding to the actual generation procedure. Could the authors clarify this distinction? In particular, to what extent does approximating the posterior gradient in Equation (7) provide a better approximation of the policy gradient required by the practical RL algorithm?

  2. My second question concerns the computational cost and correctness of sampling from the proposed measure. In Equations (10) and (11), the confidence-based distribution is defined sequentially, with the probability of revealing each position depending on the current partially unmasked sequence. However, in the trajectory-based experiments, the authors use d2-StepMerge, where multiple positions are scored using the same model evaluation (Equation (16)). If I understand correctly, sampling exactly from Equation (11) would require recomputing the probabilities after each reveal, which seems incompatible with the claim that no additional model evaluations are needed. Do the authors instead reuse the probabilities computed at the beginning of each segment? If this is the case, the resulting distribution would not be exactly the one defined in Equation (11). I would appreciate a precise explanation of the implemented sampling procedure and whether an approximation is introduced here.

  3. My third question concerns the practical ESPO implementation and the choice of temperature. In Section 5.2 and Appendix B.5, the authors explain that the mask-state distribution is kept unchanged from ESPO, rather than sampling masks from the distribution induced by the proposed confidence-based measure. Therefore, as correctly acknowledged in the appendix, the resulting gradient is not generally equal to the gradient derived in Equation (13). This raises a question about the theoretical motivation of the practical method. In particular, why should maximizing the ESS of the full reveal-order distribution, as done in Section 4, be a good criterion for selecting the temperature of practical ESPO? Would it be possible to quantify the discrepancy between these two gradients, or evaluate the corresponding event-space approximation introduced in Appendix B.5? I think this would strengthen the connection between the theoretical and experimental parts of the paper.

  4. My last question concerns the experimental validation. The authors report results using 200 random validation/evaluation splits, which is an interesting way to avoid selecting checkpoints directly on test performance. However, if I understand correctly, these splits are performed on the same trained models and therefore do not account for variability across training seeds. Given that some improvements are quite small, especially on GSM8K and MATH, could the authors provide results across independent training seeds, or at least give an estimate of the variability induced by training? Also, do the authors expect similar improvements on other diffusion language models, or are there reasons to believe that the confidence-based measure is particularly suited to LLaDA?

Despite these questions, I think that the paper contains interesting ideas and some encouraging experimental results. However, I currently have reservations about the agreement between the theoretical formulation and its practical implementation, especially regarding the sampling procedure in the trajectory-based setting. I am therefore slightly in favor of rejection at this stage, but I would be open to increasing my score if the authors can clarify these points.

6 comments
AuthorsOpenReview ↗

Reply to Question 1

We thank the reviewer for this argument, which deepens the discussion between ELBO and trajectory-based methods, which is central to our work.

The reviewer is correct. Eq. (4) is the likelihood of the uniform-order, one-token-per-step sampler (Hoogeboom et al., 2022), which is the quantity bounded by the pre-training ELBO. LLaDA's confidence-based, multi-token sampler (Nie et al., 2025) defines a different distribution. Proposition 1 therefore gives the exact gradient of (4), not of the deployed sampler's likelihood. Existing RL methods for MDLMs reflect these two views: ELBO-based methods optimize a bound on (4), averaging over all reveal orders uniformly, while trajectory-based methods usually follow the single order realized by the deployed sampler.

We acknowledge that using the generation distribution, as in trajectory-based approaches, is a valid choice which is closer to an on-policy approach for RL. We believe targeting (4) is also appropriate given it is the likelihood the model pre-training targets through the ELBO. Moreover, many ELBO-based RL methods, such as ESPO, report state-of-the-art results among RL methods for diffusion language models. Finally, unlike a sampler's likelihood, it does not depend on decoding choices (sampling temperature, remasking rule, etc.), which can differ between training and evaluation.

Besides, we would like to emphasize that the main idea of our work is to place the three quantities underlying RL for MDLMs, namely the ELBO, the trajectory likelihood and the order-agnostic likelihood (4), within a single framework. In this framework, each corresponds to a measure over reveal orders. Our measure is designed for exactly this setting. It interpolates between the ELBO-based and trajectory-based extremes, while potentially approximating the gradient of (4) more closely. Experimental results show the advantage of this approach.

AuthorsOpenReview ↗

Reply to Question 2

We thank the reviewer for this question, which allows us to improve the clarity of the proposed method presented in the article.

On sampling from Eq. (11) with d2-StepMerge. We agree with the reviewer that sampling exactly from (11) would require recomputing the confidences after every reveal, i.e. LL model evaluations, which would cancel the cost reduction that d2-StepMerge provides. To maintain the computational benefits of the baseline, we sample the orders from the model evaluations that d2-StepMerge already performs, thereby applying to order sampling the same within-segment approximation that d2-StepMerge applies to the likelihood. Sampling therefore requires no additional model evaluations, at the price of drawing orders from a segment-level approximation q~cτ\tilde q^\tau_c of (11) rather than from (11) exactly.

Implemented procedure. For segment nn, let ymny_{m_n} be the response with all tokens from previous segments revealed. One forward pass on ymny_{m_n} gives pθ(yi∣ymn,x)p_\theta(y_i \mid y_{m_n}, x) for all masked positions. We draw the segment's S=L/NS = L/N positions sequentially without replacement, with probabilities ∝pθ(yi∣ymn,x)1/τ\propto p_\theta(y_i \mid y_{m_n}, x)^{1/\tau} renormalised over the positions still masked, and score them with the same pass in Eq. (16). The only difference from (11) is that the confidences are not updated within a segment. This sampling procedure allows to tradeoff computational cost with exactness of the resulting order distribution.

Gap to (11). The gap is controlled by how much the conditionals differ within a segment, the quantity that also controls the d2-StepMerge likelihood error (cf. εblock\varepsilon_{\text{block}} in Theorem 4.1 of Wang et al., 2026b; OpenReview, arXiv:2509.21474). If revealing tokens within a segment changes the log-probability of any still-masked token by at most ε\varepsilon, then for every order σ\sigma

∣log⁡q~cτ(σ)−log⁡qcτ(σ)∣≤2(L−N) ετ.|\log \tilde q^\tau_c(\sigma) - \log q^\tau_c(\sigma)| \le \frac{2(L-N)\ \varepsilon}{\tau}.

In particular, q~cτ=qcτ\tilde q^\tau_c = q^\tau_c whenever the d2-StepMerge likelihood is itself exact (token by token, for every order), e.g. when N=LN = L. The proof is given in the new Appendix B.6.

Implications and revision. Our estimator is gνg_\nu with ν=q~cτ\nu = \tilde q^\tau_c, a valid measure in our framework to which Proposition 2 applies, but our statements about qcτq^\tau_c hold for it only up to this gap. We will define q~cτ\tilde q^\tau_c explicitly in Section 5.2, show the approximation error wrt. to (11) and add the related proof in the appendix.

AuthorsOpenReview ↗

Reply to Question 3

We thank the reviewer for this question. As noted in Appendix B.5, practical τ-ESPO keeps ESPO's mask distribution, so its expected update g~τ\tilde g_\tau differs from gqcτg_{q_c^\tau}. Since the ESS screening of Figure 2a is computed for qcτq_c^\tau, it is not guaranteed to select a good temperature for the estimator used in training. To check this, we repeat the gradient-error experiment of Figure 2b with the practical τ-ESPO gradient, and find that it agrees with the ESS screening.

Experiment details: We keep the setup of Figure 2b (frozen LLaDA-8B, 32-token blocks, the same reference g⋆g^\star built from 64 importance-weighted orders, the same CountSketch) and change two things:

  • the candidates are now ESPO (g^u\hat g_u) and practical τ-ESPO (g^c\hat g_c), evaluated on the same random masks, so they differ only in the credit given to each masked token: 1/∣m∣1/|m| for ESPO and qc,iτ(m)q^\tau_{c,i}(m) for τ-ESPO;
  • each candidate averages 16 antithetic mask (following the procedure in ESPO) pairs instead of K=4K=4 samples, to reduce Monte Carlo noise, and we use 128 blocks per task.

We report Rτ=∥g^c−g⋆∥2/∥g^u−g⋆∥2R_\tau=\Vert\hat g_c-g^\star\Vert^2/\Vert\hat g_u-g^\star\Vert^2, the geometric mean over blocks:

τ0.5124816
GSM8K1.050.900.840.860.910.95
MATH0.990.830.780.820.880.93
Countdown1.130.910.830.850.900.94
Sudoku1.251.000.890.880.900.94

Results. The new results agree with our previous simulations which identify τ\tau=2 as the optimum for most tasks and the tau range of 1,2,4 is still valid for experimentation.

Revision. We will add this experiment and a corresponding figure to the appendix.

Reviewer_owmHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper develops reinforcement learning for masked diffusion language models (MDLMs), where tokens can be generated in different orders. Unlike autoregressive language models, the likelihood of a complete response involves marginalizing over all possible token-reveal orders. For a response of length L, there are L! possible orders, making exact likelihood computation intractable. The authors identify a unified representation of the likelihood gradient via ELBO and trajectory-based methods. Experiments on GSM8K, MATH, Countdown, and Sudoku show improvements in most evaluated settings, with especially substantial gains on Countdown and some Sudoku configurations.

Critical strengths

The confidence-based sampling distribution is easy to understand, involving only one temperature parameter that can be incorporated into existing RL methods.

The proposed weighting can reuse model predictions already obtained during likelihood evaluation. Thus, the implementation does not require additional model forward evaluations compared with the corresponding baseline.

The empirical gains are convincing for some planning tasks. E.g., Countdown improves by 7.58 points under ESPO and 9.3 points under d2-StepMerge at the training generation length.

Critical weaknesses

The proposed confidence-based sampler is different from posterior over reveal orders. (The exact posterior conditional for the next reveal position depends not only on the current token probability but also on the likelihoods of all possible future reveal sequences.) There is no theoretical result showing that the confidence-based distribution is systematically closer to the posterior than the uniform distribution.

There is a gap between the theory and implementation (though I fully understand the difficulty in engineering.)

Questions for authors

See weakness.

1 comment

We thank the reviewer for raising this point and helping us clarify the gap between the theory and its implementation. We agree that the exact posterior depends not only on the current token probabilities but also on all the future reveals. Our method uses the current token probabilities alone. We can characterize exactly when this approach improves on the uniform measure for a fixed model, prompt, and response.

Formally: Given a prompt xx and a response y=(y1,…,yL)y = (y_1, \dots, y_L).

  • σ=(σ1,…,σL)\sigma = (\sigma_1, \dots, \sigma_L) is a reveal order, and Rt={σ1,…,σt−1}R_t = \{\sigma_1, \dots, \sigma_{t-1}\} is the set of positions already revealed before step tt.
  • ci(R):=pθ(yi∣yR,x)c_i(R) := p_\theta(y_i \mid y_R, x) is the confidence of the model in the realized token at position ii when only the positions in RR are revealed.
  • πσ:=∏t=1Lcσt(Rt)\pi^\sigma := \prod_{t=1}^{L} c_{\sigma_t}(R_t) is the likelihood of the response when it is revealed in the order σ\sigma.
  • qθ(σ):=πσ/∑σ′πσ′q_\theta(\sigma) := \pi^\sigma / \sum_{\sigma'} \pi^{\sigma'} is the posterior over orders, and U(σ):=1/L!U(\sigma) := 1/L! is the uniform measure used by the ELBO.
  • β:=1/τ\beta := 1/\tau, and qcτq_c^\tau is the confidence measure, which reveals a masked position ii with probability qcτ(i∣R)=ci(R)β/∑k∉Rck(R)βq_c^\tau(i \mid R) = c_i(R)^\beta / \sum_{k \notin R} c_k(R)^\beta. The value β=0\beta = 0 (τ=∞\tau = \infty) gives the uniform measure.

The confidence advantage of an order is

Δ(σ):=∑t=1L[log⁡cσt(Rt)−1L−t+1∑j∉Rtlog⁡cj(Rt)].\Delta(\sigma) := \sum_{t=1}^{L} \Big[ \log c_{\sigma_t}(R_t) - \frac{1}{L - t + 1} \sum_{j \notin R_t} \log c_j(R_t) \Big].

Δ(σ)\Delta(\sigma) compares the log-confidence of the token that σ\sigma reveals at each step with the average log-confidence of all other tokens still masked. It is positive when σ\sigma tends to reveal confident tokens first.

Theorem. Let D(β):=KL(qθ∣∣qcτ)D(\beta) := \mathrm{KL}(q_\theta || q_c^\tau).

(i) DD is a convex function of β\beta.

(ii) Its slope at the uniform measure is D′(0)=−Eσ∼qθ[Δ(σ)]D'(0) = -\mathbb E_{\sigma \sim q_\theta}[\Delta(\sigma)].

(iii) Consequently, some finite temperature gives a measure strictly closer to the posterior than the uniform measure if and only if Eqθ[Δ]>0\mathbb E_{q_\theta}[\Delta] > 0. If Eqθ[Δ]≤0\mathbb E_{q_\theta}[\Delta] \le 0, the uniform measure is at least as close as every confidence measure.

Interpretation.

  • Confidence advantage Δ(σ)\Delta(\sigma). As noted above, it compares at each step the log-confidence of the token that σ\sigma reveals with the average log-confidence of all tokens still masked, and it is positive when σ\sigma tends to reveal confident tokens first. For uniformly random orders it averages to zero, Eσ∼U[Δ(σ)]=0\mathbb E_{\sigma \sim U}[\Delta(\sigma)] = 0, because a uniformly chosen masked token has, on average, the average log-confidence.
  • The condition Eqθ[Δ]>0\mathbb E_{q_\theta}[\Delta] > 0. It asks whether the orders that the model finds likely reveal confident tokens first more often than random orders do. If they do, tilting the measure towards confident tokens moves it towards the posterior. If they do not, tilting cannot bring it closer.

Remark. An equivalent result holds for χ2(qθ ∥ qcτ)\chi^2(q_\theta \,\|\, q_c^\tau), the divergence that controls the gradient error in Proposition 2: it is also convex in β\beta, and a finite temperature improves on the uniform measure if and only if the same condition holds, with the average of Δ\Delta taken under weights proportional to qθ2q_\theta^2 instead of qθq_\theta.

We thank the reviewer for this suggestion: we believe the resulting theorem reinforces our empirical results. Figure 2b already showed that the confidence-based gradient estimator is closer to the reference than its uniform counterpart, and the theorem provides its theoretical counterpart. We will include the result in Section 3.1 and its proof in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.