- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper writes KL-regularized RLVR as an entropy-smoothed Bellman equation with a reshaped reward, adds a trust-region term toward the previous iterate, and derives a trajectory-level consistency loss that needs only the terminal verifiable reward. It then recasts the solution as a “reasoning flow” on a reasoning DAG to claim credit-assignment, mode-seeking and diversity properties, and proves a regret bound that depends on local coverage. Its math benchmarks show competitive results against PPO, GRPO and OREO.
Critical strengths
- It puts KL-regularized RLVR, soft-Bellman path consistency and the GFlowNet view in one notation, and the trajectory-level consistency result (Thm 3.1) is derived correctly.
- It targets the instability of value-based RLVR with a simple KL penalty toward the previous iterate. They evaluate two model families on six benchmarks and release code, so others can check the claims.
Critical weaknesses
- Novelty: The smoothed Bellman equation, the KL-as-reshaped-reward identity, and Thm 3.1 are standard soft-RL results; Thm 3.1 is essentially Trust-PCL with an added reference anchor. The paper should position itself against PCL/Trust-PCL and RTB instead of presenting these as findings.
- Motivation for the trust-region term: The justification for adding the term is that it can be written as a reward augmentation, which says nothing about whether it should be added.
- Lemma 4.1 is a definition, not a lemma. It asserts that a function exists but states no property it satisfies.
- The implemented loss differs from the analyzed one: Policy and value are trained separately, the value loss omits the term, and the policy loss has an unexplained term. None of the theory applies to the method in Table 1.
- A 2k-token cap on long-CoT models leaves baselines far below published numbers, so the gains may come from learning to finish within budget rather than better reasoning.
Questions for authors
Please see weaknesses above.
It seems that the authors forget to submit their rebuttal, I guess?