acceptodds
Withdrawn from ICLR 2027

Principled Reinforcement Learning with Verifiable Rewards: Understanding LLM Reasoning via Diffusive Flow

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) increasingly rely on reinforcement learning with verifiable rewards (RLVR) for post-training on complex reasoning tasks. While most RLVR methods are policy-based, value-based approaches have shown promise but remain underexplored in stability, formulation, and theory. In this work, we revisit value-based RLVR through a smoothed consistency framework that stabilizes LLM fine-tuning. With reward augmentation, our framework generalizes the standard KL-regularized objective, connects value-based and policy-based methods, and yields a practical joint policy–value objective using only terminal verifiable rewards. To explain its effectiveness for LLM reasoning, we develop a reasoning-flow characterization over a reasoning-directed acyclic graph (DAG), showing how the learned RLVR objective induces latent stepwise credit assignment and reshapes probability mass over reasoning trajectories. This characterization reveals how the fine-tuned LLM favors high-reward reasoning paths while preserving diversity in structured reasoning. On the theoretical side, we establish a rigorous regret analysis showing that our algorithm achieves sample-complexity guarantees and is robust to the LLM policy drift. Experiments on mathematical reasoning benchmarks show consistent gains over pretrained LLMs and demonstrate that our algorithm is competitive with strong baselines.

Primary area:
reinforcement learning
Keywords:
Generative Modeling, Autoregressive Language Models, Reinforcement Learning with Verifiable Rewards, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Q9KhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper writes KL-regularized RLVR as an entropy-smoothed Bellman equation with a reshaped reward, adds a trust-region term toward the previous iterate, and derives a trajectory-level consistency loss that needs only the terminal verifiable reward. It then recasts the solution as a “reasoning flow” on a reasoning DAG to claim credit-assignment, mode-seeking and diversity properties, and proves a regret bound that depends on local coverage. Its math benchmarks show competitive results against PPO, GRPO and OREO.

Critical strengths
  1. It puts KL-regularized RLVR, soft-Bellman path consistency and the GFlowNet view in one notation, and the trajectory-level consistency result (Thm 3.1) is derived correctly.
  2. It targets the instability of value-based RLVR with a simple KL penalty toward the previous iterate. They evaluate two model families on six benchmarks and release code, so others can check the claims.
Critical weaknesses
  • Novelty: The smoothed Bellman equation, the KL-as-reshaped-reward identity, and Thm 3.1 are standard soft-RL results; Thm 3.1 is essentially Trust-PCL with an added reference anchor. The paper should position itself against PCL/Trust-PCL and RTB instead of presenting these as findings.
  • Motivation for the trust-region term: The justification for adding the ζlog⁡πt−1\zeta\log\pi_{t-1} term is that it can be written as a reward augmentation, which says nothing about whether it should be added.
  • Lemma 4.1 is a definition, not a lemma. It asserts that a function exists but states no property it satisfies.
  • The implemented loss differs from the analyzed one: Policy and value are trained separately, the value loss omits the ζ\zeta term, and the policy loss has an unexplained −V(sH−1)-V(s_{H-1}) term. None of the theory applies to the method in Table 1.
  • A 2k-token cap on long-CoT models leaves baselines far below published numbers, so the gains may come from learning to finish within budget rather than better reasoning.
Questions for authors

Please see weaknesses above.

Reviewer_ArpUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper re-examines value-based reinforcement learning (RLVR) through a smoothing consistency framework, which helps stabilize the fine-tuning process of large language models (LLMs). By introducing a reward-augmentation mechanism, the paper generalizes the standard KL-regularized objective function and establishes a connection between value-based and policy-based methods. However, despite the theoretical arguments presented, the paper lacks sufficient experiments to robustly support its conclusions.

Critical strengths
  1. This paper introduces a smoothed consistency reasoning formulation with reward augmentation.

  2. This paper develops a reasoning-flow characterization of value-based RLVR over a reasoning DAG.

Critical weaknesses
  1. In Lemma 4.1, the terminal reward is expressed as log⁡R\log R, whereas RR is used in Section 3. Why is the target distribution in Section 4 inconsistent with the optimization objective in Section 3?

  2. The appendices in this document contain a wealth of material, yet most of it is not referenced in the main text, and the connection between the content of the appendices and that of the main text remains unclear.

  3. This paper conducts experiments using only the DeepSeek-Distill-1.5B and Qwen3-4B models, lacking experiments that explore different parameter scales within the same model family. Furthermore, the single-sampling performance of the DeepSeek-Distill-1.5B model is inferior to that of GRPO; consequently, the experimental results presented do not fully support the paper's conclusions.

  4. This paper employs a large number of hyperparameters, yet it lacks experimental results for different hyperparameter settings.

  5. This paper lacks experimental validation of its core theory. It does not test target distribution fitting, step-wise credit assignment, or the coverage of reasoning patterns.

Questions for authors
  1. Some equations in this paper are numbered, while others are not.
Reviewer_XMxWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a value-based approach to reinforcement learning with verifiable rewards, combining a smoothed Bellman formulation with regularization toward both a fixed reference policy and the previous policy iterate. It derives a trajectory-level policy–value consistency objective intended to support learning from terminal rewards, and interprets the resulting policy through a reasoning-flow representation over a prefix-based directed acyclic graph. The authors use this representation to argue for mode-seeking behavior, preservation of reasoning diversity, and implicit stepwise credit assignment, and present a finite-class statistical analysis with localized distribution-shift assumptions. Experiments on six mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B show improvements over the starting checkpoints and competitive performance against PPO, GRPO, and OREO, with the strongest results on Qwen3-4B.

Critical strengths
  1. The empirical results provide a meaningful positive signal, especially on Qwen3-4B.
  2. Learning effectively with one rollout per prompt is practically relevant.
Critical weaknesses
  1. Missing comparison with FlowRL and insufficient differentiation from closely related objectives. FlowRL (Zhu et al., 2025) is a particularly relevant missing baseline: it already develops flow-based optimization for LLM reasoning, connects trajectory balance to reverse-KL optimization, and explicitly investigates reasoning diversity. The authors should provide an equation-level comparison clarifying the additional contribution of the previous-iterate regularizer, value-function parameterization, and practical loss modifications. A direct comparison against FlowRL using the same backbone, training data, verifier, and evaluation settings, with clearly reported rollout and compute budgets, is needed to assess the claimed advantages. This comparison should include accuracy, training stability, and correctness-conditioned reasoning diversity.
Questions for authors

See above

1 comment

It seems that the authors forget to submit their rebuttal, I guess?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.