acceptodds
Under review as a conference paper at ICLR 2027

ReLPO: Resolution-Lifted Policy Optimization for RL Post-Training of Autoregressive–Diffusion Language Models

Abstract

Autoregressive (AR)–diffusion hybrid language models combine revisable latent refinement with irreversible token realization, creating a distinct challenge for reinforcement learning (RL) post-training in assigning outcome credit across both forms of decision making. We observe that hybrid generation naturally induces a native relaxation hierarchy. Each partially resolved state defines a completion fiber containing all terminal sequences consistent with its committed tokens, while its unresolved latent state determines a continuation distribution over this fiber. Latent refinement reshapes this distribution while preserving possible completions, whereas autoregressive realization progressively contracts the fiber toward a single discrete output. Building on this structure, we introduce ReLPO (Resolution-Lifted Policy Optimization), which propagates terminal outcome information through the hierarchy using a Resolution-Lifted Outcome Value (RLOV). RLOV defines a soft potential over partially resolved states and is learned from completed rollouts and terminal verifier rewards through a consistency relation spanning both refinement and resolution transitions. The resulting shared potential guides continuous latent refinement and autoregressive realization through a coupled policy improvement operator, allowing terminal outcomes to coordinate refinement, deferral, and token commitment throughout generation. Across eight mathematical reasoning and code-generation benchmarks, ReLPO improves the shared Evo backbone from 52.5 to 59.7 average accuracy, compared with 56.4 for the strongest of eight RL baselines. Using separate potentials for the two branches reaches 57.7 under matched capacity, demonstrating the benefit of coupling refinement and realization through a shared potential. Under a unified protocol across six diffusion and AR–diffusion backbones, ReLPO improves every model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, reinforcement learning, Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mMW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RL post-training for autoregressive–diffusion hybrid language models, in which generation interleaves revisable latent refinement with irreversible token commitment. The key observation is that each partially resolved state defines a set of completions consistent with the committed tokens, which is preserved by refinement and contracted by commitment. Based on this observation, a soft value function, termed the Resolution-Lifted Outcome Value (RLOV), is defined on partially resolved states and learned from terminal rewards alone via a soft Bellman recursion. Tilting the reference transition kernel by this shared value gives a KL-regularized policy update for both branches, in which the value gradient guides latent refinement and value differences determine whether and how each position is committed; the update is then distilled into the generator. Experiments on eight mathematical reasoning and code generation benchmarks demonstrate the effectiveness of the method, which improves the average accuracy of Evo-XXL from 52.5 to 59.7 (versus 56.4 for the strongest baseline) and yields consistent gains across six diffusion and hybrid backbones.

Critical strengths
  1. It is well known that, under KL regularization, the optimal policy is the reference policy tilted by a soft value function. The present paper applies this to AR–diffusion generation and notes that, since refinement and commitment are two transition types of the same kernel, a single value suffices to guide both. Of particular note is that commitment timing becomes a decision derived from value differences, learned from terminal rewards alone without process supervision.

  2. The experiments are carefully controlled: all baselines share the same checkpoint, data, verifier, and rollout budget, compute-matched baselines are included, and the method is tested on six backbones. Note also that the paper goes beyond accuracy and checks the mechanism, showing that premature commitment and refinement steps are both reduced, with lower inference latency.

Critical weaknesses

Coverage of the frozen reference. The state values are estimated from the continuation distribution of the reference policy, so if a class of successful solutions has zero probability under this policy, the tilted update cannot reach it either. In practice, only four successors are sampled per state by default, and the paper reports that increasing this number to sixteen still improves the result by 0.2 points, which indicates that the quality of the estimate remains sensitive to the sample size. This may limit the effectiveness of the method on tasks where successful samples are rare or where the policy distribution shifts substantially.

Questions for authors

see weakness

1 comment

Dear Reviewer mMW8,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_tScUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Resolution-Lifted Policy Optimization (ReLPO) for reinforcement learning post-training of autoregressive–diffusion hybrid language models. A shared intermediate-state potential is learned from terminal rewards through soft Bellman consistency and guides both continuous latent refinement and discrete token commitment. Exeriments show that ReLPO outperforms eight RL baselines on all eight benchmark.

Critical strengths
  • The paper addresses a meaningful credit-assignment problem in hybrid generation. Using one potential to guide both revisable latent updates and irreversible token commitment gives the method a coherent design.
Critical weaknesses
  • Training the critic introduces additional computational and memory overhead. ReLPO requires a separate backbone-based critic in addition to the policy, together with an EMA target and intermediate-state storage. Although the authors include comparisons at matched total training FLOPs, these comparisons do not account for the additional memory footprint. Maintaining and training the critic can impose a substantial GPU-memory burden even with LoRA, potentially reducing the feasible batch size or requiring more devices or offloading.
Questions for authors
  • The LLaDA citation leaves the starting checkpoint ambiguous. Section 3.1 and Table 2 refer to “LLaDA 8B,” but the cited Zhu et al. (2026a) paper is LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models, rather than the original LLaDA paper. LLaDA 1.5 has undergone preference optimization and is therefore a different starting point from an SFT-only LLaDA checkpoint. The authors should clarify whether this is a citation error or whether the starting model already includes preference optimization.

  • The provenance of MDLM 9B and BD3-LM 7B is unclear. Table 2 names these large backbones, but to my knowledge, the original BD3-LM paper describes a 110M-parameter experimental architecture, and the official MDLM-OWT model card reports approximately 130M non-embedding parameters. Can the authors provide an explanation?

Reviewer_aYw2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReLPO, an RL post-training method for autoregressive–diffusion language models that treats partially generated states as having different levels of commitment: some content is still revisable, while some tokens are already fixed. It learns a shared value function over these intermediate states from only final rewards, then uses that value to guide both latent refinement and the decision of when and what to commit as discrete tokens. The main claim is that coordinating these two parts of generation through one shared potential gives better credit assignment and stronger post-training than treating them separately.

Critical strengths

I am persuaded that this is a real problem in need of solving: hybrid AR–diffusion generation really does have two qualitatively different decisions - continue refining or commit tokens - and ReLPO gives them a single shared credit signal rather than optimizing them independently.

The experiments are broad and well controlled. ReLPO is compared with eight baselines on a shared backbone, against GRPO and TraDo rerun at matched compute, and across six backbones and three seeds. Ablations cover the learned value, shared versus separate values, the temperature and group size.

The authors also checks how the method actually works: they validate the learned value against Monte Carlo estimates, measure premature commitment and redundant refinement directly, show faster inference, and add a dated evaluation on recent problems to address contamination.

Critical weaknesses

The theory is standard KL-regularized control given new name. The paper admits its key recursion holds for any Markov generator, so the “relaxation hierarchy” adds nothing. The claim that one shared value for refinement and commitment is what matters is tested against a version with two separate values, each learned from only one kind of step. That version misses part of the picture by design, so it is a weak comparison. The fairer one, a standard success-probability critic driving the same update, comes within one point (58.7 against 59.7).

The method is also not iterative policy improvement. Its value function scores states by how the original frozen model would do from that point, not the model being trained. Training pushes the model toward a fixed target - the original model’s own outputs, reweighted to favor the ones that succeed. That is one round of improvement over the starting model, not a repeated improvement. As the trained model moves away from the original, those value estimates fit the states it actually visits less and less, which the paper acknowledges. Refinement is steered by the slope of a learned value network, and the paper checks the value’s accuracy but never whether that slope points somewhere useful. When the term keeping the model close to the original is removed, performance peaks at 56.7 and then falls, consistent with that drift.

The empirical evidence doesn’t fully agree with the proposed idea. The method’s mechanism decides when a position stops being revisable, but the case studies only show the model writing more careful reasoning, such as keeping n = 35k + 17 instead of answering 17. Every step there is ordinary committed text that any RL method could teach. None of the eight baselines was designed for this kind of model; the authors adapted all of them with limited tuning, but there is no test on a purely autoregressive model to show how much of the gain comes from revisable states (left for future work).

Questions for authors

The authors should describe ReLPO as an application of KL-regularized control to hybrid generators, and either drop the relaxation hierarchy as a contribution or prove something that depends on it. One candidate: as committed tokens narrow the remaining possibilities, value targets become less noisy, which would justify sampling more states early in generation. For the shared-value claim, the fair comparison is a factorized version that is not handicapped by design. Appendix C.5 already notes that separate branch values can reproduce the joint update if their totals are calibrated and combined, so that calibrated version should be built and compared. The one-point gap to the success-probability critic should also be reported with per-seed results and a significance test.

Test whether a frozen reference is a limitation and if the refinement gradients help. Compare the current setup against a version that periodically replaces the reference with the current policy and refits the value, which would make the method genuinely iterative. The authors should measure how the value’s accuracy changes over training by checking it against Monte Carlo continuations from the current policy at several checkpoints. For the gradients, a direct test is to nudge latent states a small step along the value’s gradient and check - with fresh continuations - that the success estimate rises more than it does for random directions of the same size. They could also compare gradient-driven refinement against sampling several refinement proposals and reweighting them by the value, which the paper’s own exact update (Eq. 21) allows without any gradient.

The case studies should show the latent mechanism itself: which positions were held back, how the value changed when they were committed, and the matched GRPO and TraDo outputs on the same problems. The authors should run the autoregressive-only version they describe on a standard autoregressive model against GRPO, to separate the gain from revisable states from the gain from the value function. Baselines should also be run on the backbones they were designed for, such as TraDo on BD3-LM or SDAR and d1, WD1 and SPG on LLaDA, with the same tuning budget as ReLPO.

1 comment

Dear Reviewer aYw2,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.