- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies RL post-training for autoregressive–diffusion hybrid language models, in which generation interleaves revisable latent refinement with irreversible token commitment. The key observation is that each partially resolved state defines a set of completions consistent with the committed tokens, which is preserved by refinement and contracted by commitment. Based on this observation, a soft value function, termed the Resolution-Lifted Outcome Value (RLOV), is defined on partially resolved states and learned from terminal rewards alone via a soft Bellman recursion. Tilting the reference transition kernel by this shared value gives a KL-regularized policy update for both branches, in which the value gradient guides latent refinement and value differences determine whether and how each position is committed; the update is then distilled into the generator. Experiments on eight mathematical reasoning and code generation benchmarks demonstrate the effectiveness of the method, which improves the average accuracy of Evo-XXL from 52.5 to 59.7 (versus 56.4 for the strongest baseline) and yields consistent gains across six diffusion and hybrid backbones.
Critical strengths
-
It is well known that, under KL regularization, the optimal policy is the reference policy tilted by a soft value function. The present paper applies this to AR–diffusion generation and notes that, since refinement and commitment are two transition types of the same kernel, a single value suffices to guide both. Of particular note is that commitment timing becomes a decision derived from value differences, learned from terminal rewards alone without process supervision.
-
The experiments are carefully controlled: all baselines share the same checkpoint, data, verifier, and rollout budget, compute-matched baselines are included, and the method is tested on six backbones. Note also that the paper goes beyond accuracy and checks the mechanism, showing that premature commitment and refinement steps are both reduced, with lower inference latency.
Critical weaknesses
Coverage of the frozen reference. The state values are estimated from the continuation distribution of the reference policy, so if a class of successful solutions has zero probability under this policy, the tilted update cannot reach it either. In practice, only four successors are sampled per state by default, and the paper reports that increasing this number to sixteen still improves the result by 0.2 points, which indicates that the quality of the estimate remains sensitive to the sample size. This may limit the effectiveness of the method on tasks where successful samples are rare or where the policy distribution shifts substantially.
Questions for authors
see weakness
Dear Reviewer mMW8,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC