- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VIMPO, a critic-free policy optimization method for LLM reinforcement learning with verifiable rewards (RLVR). Motivated by the coarse trajectory-level credit assignment in GRPO and the additional complexity of learned critics in PPO, VIMPO derives a policy-implied value representation from the optimality conditions of KL-regularized reinforcement learning. Using policy-reference log-ratios and a terminal value boundary condition, the method constructs an outcome-supervised value regression loss that directly updates the policy without a separate critic. It further derives a token-dependent advantage signal for an auxiliary PPO-style actor update. Experiments on mathematical reasoning benchmarks using Qwen3-4B-Base show improvements over GRPO, including under noisy reward conditions. Additional experiments on a terminal-reward Walker2d task and component ablations evaluate the effectiveness of the value regression objective and the auxiliary actor update.
Critical strengths
A promising critic-free optimization approach with encouraging empirical results. The paper explores an interesting direction by deriving policy-implied value and token-level advantage signals without introducing a separate critic. The proposed method achieves consistent improvements over GRPO across four mathematical reasoning benchmarks, including under noisy reward conditions. The additional Walker2d experiments provide further evidence that the value regression objective can effectively train policies, supporting the practical potential of this approach.
Critical weaknesses
1. Limited novelty compared with prior reinforcement learning methods. The core idea of deriving value-related signals from policy probabilities and optimizing trajectory-level consistency has been extensively studied in prior work, particularly Path Consistency Learning (Nachum et al., 2017). Although VIMPO adapts this framework to LLM RLVR, its fundamental distinction from existing consistency-based RL methods remains unclear. The paper lacks direct comparisons or sufficient analysis to establish the novelty and necessity of its proposed formulation.
2. Insufficient theoretical justification and experimental validation. The proposed token-level advantage estimator is derived from an optimality condition but applied to non-optimal policies during training, without sufficient justification that it provides meaningful credit assignment. Moreover, the LLM experiments are limited to a single 4B model and primarily compare against GRPO variants, omitting relevant methods such as FIPO and VAPO. These limitations weaken the evidence for the proposed method's effectiveness and generalizability.
Questions for authors
1. Novelty compared with prior work. How does VIMPO fundamentally differ from Path Consistency Learning (Nachum et al., 2017)? Beyond adapting existing policy–value consistency ideas to LLM RLVR, what are the key methodological innovations?
2. Theoretical justification. The proposed advantage estimator is derived from an optimality condition but applied to non-optimal policies during training. What justifies this approximation, and how can the authors establish that it provides meaningful token-level credit assignment?
3. Experimental comparisons. Why are the LLM experiments primarily compared against GRPO variants rather than more closely related methods such as FIPO and VAPO? Can the authors provide additional comparisons or evidence supporting VIMPO's advantages over these approaches?
4. Generalizability. The LLM experiments use only Qwen3-4B-Base. What evidence suggests that the improvements extend to other model architectures, scales, or reasoning tasks?
Thank you for your thoughtful review and constructive feedback. We respond to your questions below.
1. Relationship to Path Consistency Learning
VIMPO addresses a different starting point from PCL. In LLM post-training, we already have a pretrained model that is useful as a policy, but its token probabilities do not tell us what reward to expect from a partial answer. Standard PCL learns a separate state-value model alongside the policy. Unified PCL uses one model, but it starts from a learned soft Q-function and derives both policy and value from it. VIMPO reverses that direction. It keeps the language model in its original role as the policy and learns expected verifier reward through a value representation derived from the policy itself. That is the distinction beyond sharing parameters.
PCL fits a consistency relation between the values at the two ends of a rollout segment using the rewards and action log-probabilities along it. VIMPO builds its regression target from complete responses. The mean verified reward across completions estimates the value at the prompt, and the value after a completed response is zero. These endpoints let us fit the policy’s prediction directly to the completion’s reward relative to its group, without bootstrapping from an intermediate value estimate. The regression updates the policy directly, and a separate actor loss can reuse the resulting signal for token-level updates. This brings value learning into LLM post-training without changing the policy architecture or requiring a different kind of feedback.
2. Token-level advantages during training
The optimality condition tells us exactly when the policy-implied advantage is correct. It also gives a local approximation away from that point. If the current policy’s log-probabilities at a prefix are within ε of the local Boltzmann policy formed from its own action values, the one-step advantage error is at most 2βε.
The terminal loss gives a second answer that does not require the policy to be near that optimum. In the deterministic, undiscounted autoregressive setting, the true advantage and our KL-centered one-step signal both have zero mean conditional on the prefix. This means their errors at different token positions cannot systematically cancel in the population loss. With independent on-policy completions of the same prompt, the expected squared terminal residual therefore bounds the expected sum of squared one-step errors, up to the group-size factor G/(G−1). We give the derivation in our response to Reviewer Ra4v and will include it in the revised manuscript.
On Qwen3-4B-Base, the value loss alone improves training accuracy, and adding the actor leads to faster gains and higher training accuracy (Figure 5). The ablation shows what the actor adds to optimization, while the analysis above explains when the policy-implied advantage approximates the true one-step advantage.
3. Comparisons with FIPO and VAPO
We chose token-aggregated GRPO for the main experiments because both GRPO and VIMPO adapt PPO to train without a separate critic, but they differ in what they learn from group outcomes. GRPO computes advantages directly from those outcomes, while VIMPO uses them to learn a value signal within the policy. FIPO builds on DAPO, a GRPO variant. It uses changes in the likelihood of the rest of a response as a proxy for a token’s downstream influence and reweights the token’s existing advantage. VIMPO starts from a different principle. Under the local KL-regularized optimality condition, the centered policy-to-reference log ratio gives the one-step advantage. FIPO’s weighting could therefore be applied on top of VIMPO’s actor signal without changing how VIMPO learns it. VAPO instead pretrains a separate critic and adapts advantage estimation to response length so that terminal rewards can inform earlier decisions in long responses.
We compared VIMPO with FIPO on terminal-reward HalfCheetah, where FIPO’s weighting uses the log densities of sampled actions in place of token log-probabilities. Under the same training budget, VIMPO achieved higher held-out return in all three paired runs.
4. Generalizability
On Qwen3-4B-Base, VIMPO outperformed GRPO across four math benchmarks with clean rewards (39.5% vs. 37.4% on average) and retained its lead when 25% of the training rewards were flipped (35.9% vs. 32.2%). In terminal-reward Walker2d, retaining the value loss’s prediction-dependent correction improved mean displacement by 14.79 m across ten paired runs with the actor disabled. This shows that the value objective can support learning in continuous control as well.
We agree that evaluating a single language model limits what we can say about scale. As noted in our response to Reviewer m9Zf, we will include the multi-run comparison on Qwen3-0.6B in the revised manuscript.