acceptodds
Under review as a conference paper at ICLR 2027

Towards Value-Aware Unmasking: Local Action Value in Diffusion Language Models

Abstract

Diffusion language models (DLMs) must decide which masked position to commit next, yet existing endpoint predictors estimate trajectory-level success, and lookahead methods compare candidate futures. We ask whether the frozen representation already encodes value differences between alternative commitments. We introduce Local Action-Value Readout (LAVR), a framework that formalizes policy-relative action value for unmasking decisions, estimates it through matched forced-and-reference rollout branches with candidate centering, and reads it from frozen hidden states against output-and-trajectory controls. In an 800-prompt GSM8K study on Nemotron-Diffusion-3B, the hidden-state readout improves global state-value prediction over the output-statistics baseline (R² = 0.4812 vs. 0.3380) and supports selective generation. For local decisions, candidate-specific path log-likelihood (Path-LL) value replicates on a disjoint 600-document pool and improves pairwise ranking on an independent 400-document study across all six model–policy/target conditions. By contrast, task-correctness action gains are not reliably positive, and frozen transfer decreases concordance. This contrast shows that readable local continuation value does not automatically imply task-correctness value, making reward alignment central to task-aligned unmasking. Code and evaluation artifacts are available in an anonymous repository.

Primary area:
generative models
Keywords:
diffusion language models, value-aware decoding, unmasking policies, local action value, representation probing, rollout-based evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LhiSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether frozen DLM hidden states contain information about which unmasking decisions lead to better future generation outcomes. It proposes LAVR, which estimates candidate-specific action values through offline rollouts and learns to predict them from current hidden states. The results show statistically significant but small improvements in Path-LL prediction, while task-correctness action-value prediction shows no reliable improvement.

Critical strengths
  • Exploring whether current hidden states encode the future value of unmasking decisions is an interesting and promising direction.
  • The Path-LL prediction gains are statistically significant and reproduced on independent document pools.
Critical weaknesses

Limited practical significance

Path-LL ranking gains are marginal (ΔConcordance ≤ 0.0094). Moreover, the observed weak or inconsistent alignment with task correctness raises questions about whether Path-LL is a suitable surrogate for better unmasking decisions. No closed-loop experiments demonstrate whether these predictive gains improve generation quality.

Limited backbone diversity

The larger-scale experiments use only Nemotron-Diffusion-3B. Evaluation on other representative DLMs, such as LLaDA or Dream, would strengthen the findings, although the computational burden is understandable.

Limited evaluation scope

Local Path-LL is evaluated only on OpenWebText with 169.6M models, while task-correctness prediction is evaluated only on easy mathematical benchmarks (GSM8K and SVAMP) with Nemotron-3B. These differences in models and datasets make it difficult to isolate the effect of the reward choice.

Questions for authors
  • Given the marginal Path-LL gains and weak alignment with task correctness, what practical applications do the authors envision for LAVR?
  • Could the authors discuss the generalizability of their findings to other DLMs and non-mathematical tasks?
  • What are the GPU-hours required for the local Path-LL experiments, including replications A and B? What inference-time overhead would the learned readouts introduce?
2 comments

We thank you for the review and for recognizing that exploring whether hidden states encode the future value of unmasking decisions is an interesting and promising direction, and that the Path-LL gains are statistically significant and reproduced on independent document pools. We address the three weaknesses and three questions below, including new closed-loop decoding, LLaDA-8B, reward-isolation, and cost studies.

W1 & Q1: Practical Significance and Use

Thank you for these questions. The Path-LL gains are measured beyond 100 output, distributional, and trajectory features that include current confidence (App. B.3), and they carry over to decoding. We used the readout to choose actions under MDLM confidence decoding. In the executed-step study, the chosen action is committed on a new 200-document pool and the original policy finishes the generation. In the closed-loop study, the readout chooses the action at every step on 400 new OWT documents under the same NFE budget, and an external language model, independent of the Path-LL target, scores the output:

StudyMetricControl-only readoutLAVRGain [95% CI]
Executed stepReward ↑−3.3830−3.3749+0.0081 [0.0015, 0.0147]
Closed loopExternal NLL ↓2.8462.8300.016 [−0.007, 0.039]

The executed-step gain recovers 7.6% of the gap to an independent oracle (−3.2770). The closed-loop study also compares the standard orderings:

Decoding policyExternal NLL ↓
Original ancestral ordering2.870
Confidence ordering2.857
Control-only readout2.846
LAVR hidden-guided2.830

Path-LL value is therefore a usable decoding signal, and LAVR yields the lowest closed-loop NLL of the four policies under the same NFE budget, as judged by an external model. For Q1, the applications are value-guided unmasking, shown here, and offline selective generation with the global-state readout, which adds 6.12 pp correctness at 50% coverage (Table 2). We have added both decoding studies to the appendix of the revised PDF.

W2, W3 & Q2: Backbones, Data, and Reward Isolation

Thank you for these suggestions. Local Path-LL is demonstrated on MDLM and SEDD on non-math OWT text, and two new studies extend the backbones and isolate the reward. On LLaDA-8B, 400 documents under a fixed policy are split into 240 training, 60 validation, and 100 test documents. On Nemotron-3B, 750 GSM8K C+D prompts keep the states, candidates, and readout architecture fixed and change only the reward target, with correctness labels raised from 8 to 32 completions:

ModelDataTargetΔR² from hidden states [95% CI]
MDLM, SEDD (169.6M)OWTPath-LL6/6 positive intervals (Table 8)
LLaDA-8B (new)400 documentsPath-LL+0.0076 [0.0008, 0.0144]
Nemotron-3B (new)GSM8K C+DPath-LL+0.0115 [0.0040, 0.0190]
Nemotron-3B (new)GSM8K C+DCorrectness+0.0022 [−0.0060, 0.0104]

On LLaDA-8B, pairwise concordance also rises from 0.5234 to 0.5280 (+0.0046 [−0.0007, 0.0099]). On Nemotron-3B, the two targets differ in label reliability:

Nemotron-3B targetLabel reliabilityReliability-corrected ΔR²
Path-LL0.84+0.0137
Correctness0.45+0.0049

Path-LL is therefore readable on both non-math and math text and from 169.6M to 8B parameters, and with model, states, and readout architecture held fixed, the contrast between Path-LL and correctness follows the reward choice itself. We have added both studies to the appendix of the revised PDF.

Q3: Compute Cost and Inference Overhead

Thank you for asking about cost. We measured latency on one A100 80GB GPU with MDLM-169.6M, six candidates, H=16, and matched precision and batching:

QuantityValue
Forced candidate-rollout replicates, original study and replications A and B (Apps. B, F.2)5,356,800
Offline labeling cost in this benchmark30 GPU-hours
Backbone step18.0 ms
Backbone step with LAVR readout18.9 ms (+5.0%)
Direct lookahead step (K=1)252 ms

Rollout replicates differ from model forward-pass counts. A deployed readout uses current activations and a fitted prediction head, so it adds 5.0% inference overhead, launches no candidate rollouts, and is 13.3× faster per step than direct lookahead. We have added this benchmark to the appendix of the revised PDF.

Thanks for the detailed rebuttal and additional experiments. The new experiments address some of my concerns.

However, the improvements are still quite small, and the closed-loop NLL gain is not statistically significant. I'm still not convinced that these gains translate into meaningful improvements in generation quality.

Therefore, I maintain my original score.

Reviewer_pgS2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LAVR, which defines and estimates an "unmasking action value" using matched forced/reference rollouts, and subsequently examines whether the hidden states of a frozen discrete latent model (DLM) can predict this value. Experiments reveal that the overall state value is predictable, while the local "Path-LL" (path log-likelihood) value exhibits a small but reproducible signal. In contrast, the local "task-correctness" value shows no consistent gain and even declines during cross-dataset transfer.

Critical strengths

The motivation is clear. Current token confidence metrics fail to directly capture the impact that unmasking a specific position has on subsequent generation. The paper distinguishes between state-level prediction and within-state action ranking, employing candidate centering and matched rollouts to develop a measurement framework for the latter.

Critical weaknesses
  1. The presentation hinders the understanding of the core pipeline. Section 3 scatters the definitions of value, rollout labeling, and readout across the text; it lacks a clear flowchart distinguishing between offline training, inference, and evaluation. Furthermore, Figure 2 provides an insufficient explanation of the features and their respective branches. The abstract presents a dense list of numerical values ​​before introducing the experimental subjects or metrics, making it difficult for the reader to interpret the significance of these results.
  2. Evidence most directly relevant to unmasking decisions is weak. The pairwise concordance for local Path-LL improves by only 0.0022–0.0094, while gains in task-correctness are statistically insignificant. The paper fails to report the actual reward achieved through action selection or improvements in closed-loop decoding; thus, the practical utility of "readable local values" remains unclear. Moreover, the advantage of the overall state-value largely vanishes when analyzed within the context of individual prompts.
  3. The scope of the models and reward definitions limits the conclusions. Positive results for local Path-LL are validated only on MDLM/SEDD models with approximately 169.6M parameters. Testing on the 3B-parameter Nemotron model yields unstable results, and there is a lack of validation for larger DLMs such as Dream or LLaDA. Additionally, because the positive and negative results span different backbones, datasets, and readout mechanisms, it is difficult to attribute the findings solely to changes in the reward function.
Questions for authors
  1. Why were the positive local results validated only on the ~169.6M-parameter model? Is it possible to conduct experiments on at least a representative subset of larger models, such as Dream or LLaDA?
  2. What is the core takeaway the authors intend for readers to grasp: that a readable "Path-LL" signal exists within the hidden state, or that this signal facilitates unmasking? If the latter, could the authors report held-out top-action reward/regret, or perhaps the results of an actual action-selection experiment?
1 comment

We thank you for the review and for recognizing the clear motivation, as well as the distinction between state-level prediction and within-state action ranking with candidate centering and matched rollouts. We address the three weaknesses and two questions below, including new action-selection, closed-loop, LLaDA-8B, and reward-isolation studies.

W1: Offline Labeling, Inference, and Evaluation

Thank you for pointing out where the pipeline needs to be clearer. Rollouts appear only in offline labeling and evaluation, and inference uses only current-state features (Sec. 3.1–3.3):

ComponentWhat it isWhen it runs
Rollout labelsMatched forced and reference rollouts estimate each candidate's policy-relative advantageOffline, for training and evaluation labels
Control readoutPrediction from 100 output, distributional, positional, and trajectory featuresInference
Hidden readoutThe same prediction with current hidden states addedInference
Evaluation targetsGlobal state value, local Path-LL, and local correctnessOffline evaluation

Candidate centering removes every additive state-only term exactly (Eq. 5, Lemma G.1), so the local target compares alternative commitments from the same state.

W2 & Q2: From Readable Signal to Selected Actions

Thank you for asking which takeaway we intend. The core takeaway is the first one you name, a readable Path-LL signal beyond the evaluated output and trajectory readout, and the new studies show that this signal also selects better actions. Every selector chooses among the same six candidates per state, and independent evaluation rollouts score the chosen action. Offline selection uses the held-out states of study B (100 documents and 600 states per setting), and the executed-step study commits the chosen action on a new 200-document pool under MDLM confidence decoding:

StudySettingΔReward over control readout [95% CI]Regret recoveredTop-1 agreement
OfflineMDLM ancestral+0.0040 [−0.0020, 0.0100]5.0%+0.8 pp
OfflineMDLM confidence+0.0095 [0.0010, 0.0180]8.6%+1.7 pp
OfflineSEDD ancestral+0.0140 [0.0030, 0.0250]10.4%+2.1 pp
Executed stepMDLM confidence+0.0081 [0.0015, 0.0147]7.6%n/a

In the closed-loop study, the readout chooses the action at every step of MDLM confidence decoding on 400 new OWT documents under the same NFE budget, and an external language model scores the output:

Decoding policyExternal NLL ↓
Original ancestral ordering2.870
Confidence ordering2.857
Control-only readout2.846
LAVR hidden-guided2.830

LAVR lowers NLL by 0.016 [−0.007, 0.039] relative to the control-only readout and gives the lowest NLL of the four policies. The within-prompt global ablation (App. D) concerns state value, which is why LAVR separates it from the candidate-centered comparison within each state. The readable local signal therefore raises the reward of the selected action, both offline and after execution. We have added these studies to the appendix of the revised PDF.

W3 & Q1: Larger Backbone and Reward Isolation

Thank you for these questions on scope. We used MDLM and SEDD for local Path-LL because they support controlled measurement of model-defined Path-LL, and Nemotron-3B for global state value and task correctness (Table 1). To answer Q1, we applied the protocol to LLaDA-8B, with 400 documents under a fixed policy split into 240 training, 60 validation, and 100 test documents:

LLaDA-8B metricControl+ HiddenΔ [95% CI]
Downstream R²0.02900.0366+0.0076 [0.0008, 0.0144]
Pairwise concordance0.52340.5280+0.0046 [−0.0007, 0.0099]

To isolate the reward, we used 750 GSM8K C+D prompts on Nemotron-3B, kept the states, candidates, and readout architecture fixed, and changed only the reward target, with correctness labels raised from 8 to 32 completions:

Nemotron-3B targetControl R²+ Hidden R²ΔR² [95% CI]Label reliabilityReliability-corrected ΔR²
Path-LL0.06200.0735+0.0115 [0.0040, 0.0190]0.84+0.0137
Correctness0.00800.0102+0.0022 [−0.0060, 0.0104]0.45+0.0049

The local Path-LL signal therefore holds on an 8B DLM and on Nemotron-3B, and with backbone, states, and readout architecture held fixed, the contrast between Path-LL and correctness follows the reward target itself. We have added both studies to the appendix of the revised PDF.

Reviewer_6d3hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LAVR, a method for studying whether the hidden states of a frozen diffusion language model contain information about which masked position should be revealed next. The authors first use many rollouts to estimate the future value of each candidate position. They then train small readout models to predict this value from the model’s hidden states and regular output features. The results show that hidden states can help predict the overall success of an unfinished generation. They also provide small but consistent improvements in predicting which candidate gives a higher future Path-LL.

Critical strengths
  1. The distinction between global state value and candidate-specific action value is meaningful and relevant to diffusion-language-model decoding.
  2. The matched rollouts and candidate centering provide a reasonable way to estimate and compare candidate values.
  3. The authors use several controls, null tests, held-out data, and new datasets to confirm the Path-LL results.
  4. The paper honestly reports that Path-LL value and task correctness are not always aligned.
Critical weaknesses
  1. The main contribution is unnecessarily difficult to understand. Important terms such as “readout,” “output controls,” global value, local value, and Path-LL are not introduced together with a clear and intuitive explanation. Their definitions are spread across the main paper and appendix. Figure 2 also combines three different readouts, offline rollout-label collection, and possible inference-time use in one diagram. It is difficult to understand what the proposed method actually does and to separate the demonstrated results from possible future applications. This lack of clarity makes the technical contribution difficult to evaluate.
  2. The contribution falls short of the practical goal suggested by the title and motivation. Although the paper studies whether local action value can be read from hidden states, it does not use the learned readout in a closed-loop unmasking policy. Therefore, it remains unclear whether the measured signal can improve real decoding quality or efficiency.
Questions for authors
  1. Is LAVR intended only as a framework for measuring information in frozen hidden states, or is it also intended as a practical unmasking method? Please clarify which parts of Figure 2 are actually used during inference and which parts are only used for offline training and evaluation.
  2. Are the readouts trained separately for each backbone, decoding policy, reward, and dataset? What evidence supports their ability to generalize beyond the setting on which they were trained? In particular, how should readers understand the negative GSM8K C→D transfer result?
  3. What is the computational cost of collecting the Path-LL labels, including the number of model evaluations and total GPU cost? How does this cost compare with performing direct lookahead during decoding?
  4. Given the small offline ranking gains, how likely are they to produce a meaningful improvement in closed-loop decoding?
1 comment

We thank you for the careful review and for recognizing the distinction between global state value and candidate-specific action value, the matched rollouts and candidate centering, and our controls, null tests, held-out data, and new datasets. We address the two weaknesses and four questions below, including new closed-loop decoding, transfer, and cost studies.

W1 & Q1: What LAVR Computes Offline and What Runs at Inference

Thank you for asking us to separate the measurement from its possible uses. The central contribution of LAVR is a candidate-level measurement, and its components split into offline and inference stages (Sec. 3.1–3.3, Figs. 1–2):

ComponentWhat it isWhen it runs
Rollout labelsMatched forced and reference rollouts estimate each candidate's policy-relative advantageOffline, for training and evaluation labels
Control readoutPrediction from 100 output, distributional, positional, and trajectory featuresInference
Hidden readoutThe same prediction with current hidden states addedInference
Evaluation targetsGlobal state value, local Path-LL, and local correctnessOffline evaluation

Local value is the difference in future reward between committing one candidate and following the reference policy from the same state, and candidate centering cancels shared state difficulty (Eq. 5, Lemma G.1). The fitted readout runs no rollouts at inference, so it can score candidates directly as an unmasking policy, which we evaluate under W2 & Q4.

W2 & Q4: Closed-Loop Decoding with the Readout

We appreciate this question about decoding quality. To test it directly, we used the readout to choose actions under MDLM confidence decoding. In the executed-step study, the chosen action is committed on a new 200-document pool and the original policy finishes the generation. In the closed-loop study, the readout chooses the action at every step on 400 new OWT documents under the same NFE budget, and an external language model scores the output:

StudyMetricControl-only readoutLAVRGain [95% CI]
Executed stepReward ↑−3.3830−3.3749+0.0081 [0.0015, 0.0147]
Closed loopExternal NLL ↓2.8462.8300.016 [−0.007, 0.039]

The executed-step gain recovers 7.6% of the gap to an independent oracle (−3.2770). The closed-loop study also compares the standard orderings:

Decoding policyExternal NLL ↓
Original ancestral ordering2.870
Confidence ordering2.857
Control-only readout2.846
LAVR hidden-guided2.830

The local value signal therefore raises the reward of the actions LAVR executes, and LAVR yields the lowest closed-loop NLL of the four policies under the same NFE budget. We have added both decoding studies to the appendix of the revised PDF.

Q2: Training Scope and Transfer

Thank you for this question. Readouts are fitted per evaluated backbone, policy, and reward setting. To test transfer of a fitted readout itself, we froze the Path-LL readouts trained on the original OWT pool and evaluated them on the independent test documents of replication A, against readouts refit on A over the same test set:

SettingFrozen ΔR² [95% CI]Refit ΔR²Retained
MDLM ancestral+0.0046 [−0.0015, 0.0107]+0.008852.3%
MDLM confidence+0.0109 [0.0028, 0.0190]+0.016864.9%
SEDD ancestral+0.0151 [0.0040, 0.0262]+0.020772.9%

The frozen GSM8K C→D result is a within-dataset, cross-batch transfer of the task-correctness readout, with ΔC = −0.0356 (Sec. 5.3). Frozen transfer therefore depends on the reward target, since frozen Path-LL readouts keep more than half of their refit gain on unseen documents while the correctness readout does not transfer. We have added this transfer study to the appendix of the revised PDF.

Q3: Cost of the Path-LL Labels

Thank you for asking about cost. The labels are a one-time offline cost, and the fitted readout replaces per-candidate lookahead at inference. We measured latency on one A100 80GB GPU with MDLM-169.6M, six candidates, H=16, and matched precision and batching:

QuantityValue
Forced candidate-rollout replicates, three Path-LL studies (Apps. B, F.2)5,356,800
Offline labeling cost in this benchmark30 GPU-hours
Backbone step18.0 ms
Backbone step with LAVR readout18.9 ms (+5.0%)
Direct lookahead step (K=1)252 ms
Break-even against direct lookaheadabout 463K decisions

Rollout replicates differ from counts of full-model forward passes. LAVR is 13.3× faster per step than direct lookahead, which recovers the offline cost at the break-even point above. We have added this benchmark to the appendix of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.