acceptodds
Under review as a conference paper at ICLR 2027

VIMPO: Value-Implicit Policy Optimization for LLMs

Abstract

Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment. Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token. Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability. We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from the optimality conditions of KL-regularized reinforcement learning. For autoregressive generation, the resulting value recurrence can be written in terms of policy-reference log-ratios and anchored by the terminal condition that no future reward remains at the end of a trajectory. This gives a simple value loss that incorporates outcome-level verifiable rewards without training a critic. The same derivation also yields a critic-free actor advantage, allowing VIMPO to separate reward incorporation through the value loss from policy improvement through a PPO-style actor update. On mathematical RLVR benchmarks, VIMPO improves over GRPO across MATH-500, AIME 2024, AIME 2025, and OlympiadBench, with especially larger gains on competition-style evaluations. Under noisy rewards, VIMPO retains a consistent advantage over GRPO, suggesting that policy-implied value optimization can provide finer credit assignment while preserving the practical simplicity of critic-free training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qEknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VIMPO, a critic-free policy optimization method for LLM reinforcement learning with verifiable rewards (RLVR). Motivated by the coarse trajectory-level credit assignment in GRPO and the additional complexity of learned critics in PPO, VIMPO derives a policy-implied value representation from the optimality conditions of KL-regularized reinforcement learning. Using policy-reference log-ratios and a terminal value boundary condition, the method constructs an outcome-supervised value regression loss that directly updates the policy without a separate critic. It further derives a token-dependent advantage signal for an auxiliary PPO-style actor update. Experiments on mathematical reasoning benchmarks using Qwen3-4B-Base show improvements over GRPO, including under noisy reward conditions. Additional experiments on a terminal-reward Walker2d task and component ablations evaluate the effectiveness of the value regression objective and the auxiliary actor update.

Critical strengths

A promising critic-free optimization approach with encouraging empirical results. The paper explores an interesting direction by deriving policy-implied value and token-level advantage signals without introducing a separate critic. The proposed method achieves consistent improvements over GRPO across four mathematical reasoning benchmarks, including under noisy reward conditions. The additional Walker2d experiments provide further evidence that the value regression objective can effectively train policies, supporting the practical potential of this approach.

Critical weaknesses

1. Limited novelty compared with prior reinforcement learning methods. The core idea of deriving value-related signals from policy probabilities and optimizing trajectory-level consistency has been extensively studied in prior work, particularly Path Consistency Learning (Nachum et al., 2017). Although VIMPO adapts this framework to LLM RLVR, its fundamental distinction from existing consistency-based RL methods remains unclear. The paper lacks direct comparisons or sufficient analysis to establish the novelty and necessity of its proposed formulation.

2. Insufficient theoretical justification and experimental validation. The proposed token-level advantage estimator is derived from an optimality condition but applied to non-optimal policies during training, without sufficient justification that it provides meaningful credit assignment. Moreover, the LLM experiments are limited to a single 4B model and primarily compare against GRPO variants, omitting relevant methods such as FIPO and VAPO. These limitations weaken the evidence for the proposed method's effectiveness and generalizability.

Questions for authors

1. Novelty compared with prior work. How does VIMPO fundamentally differ from Path Consistency Learning (Nachum et al., 2017)? Beyond adapting existing policy–value consistency ideas to LLM RLVR, what are the key methodological innovations?

2. Theoretical justification. The proposed advantage estimator is derived from an optimality condition but applied to non-optimal policies during training. What justifies this approximation, and how can the authors establish that it provides meaningful token-level credit assignment?

3. Experimental comparisons. Why are the LLM experiments primarily compared against GRPO variants rather than more closely related methods such as FIPO and VAPO? Can the authors provide additional comparisons or evidence supporting VIMPO's advantages over these approaches?

4. Generalizability. The LLM experiments use only Qwen3-4B-Base. What evidence suggests that the improvements extend to other model architectures, scales, or reasoning tasks?

1 comment

Thank you for your thoughtful review and constructive feedback. We respond to your questions below.

1. Relationship to Path Consistency Learning

VIMPO addresses a different starting point from PCL. In LLM post-training, we already have a pretrained model that is useful as a policy, but its token probabilities do not tell us what reward to expect from a partial answer. Standard PCL learns a separate state-value model alongside the policy. Unified PCL uses one model, but it starts from a learned soft Q-function and derives both policy and value from it. VIMPO reverses that direction. It keeps the language model in its original role as the policy and learns expected verifier reward through a value representation derived from the policy itself. That is the distinction beyond sharing parameters.

PCL fits a consistency relation between the values at the two ends of a rollout segment using the rewards and action log-probabilities along it. VIMPO builds its regression target from complete responses. The mean verified reward across completions estimates the value at the prompt, and the value after a completed response is zero. These endpoints let us fit the policy’s prediction directly to the completion’s reward relative to its group, without bootstrapping from an intermediate value estimate. The regression updates the policy directly, and a separate actor loss can reuse the resulting signal for token-level updates. This brings value learning into LLM post-training without changing the policy architecture or requiring a different kind of feedback.

2. Token-level advantages during training

The optimality condition tells us exactly when the policy-implied advantage is correct. It also gives a local approximation away from that point. If the current policy’s log-probabilities at a prefix are within ε\varepsilon of the local Boltzmann policy formed from its own action values, the one-step advantage error is at most 2βε2\beta\varepsilon.

The terminal loss gives a second answer that does not require the policy to be near that optimum. In the deterministic, undiscounted autoregressive setting, the true advantage and our KL-centered one-step signal both have zero mean conditional on the prefix. This means their errors at different token positions cannot systematically cancel in the population loss. With independent on-policy completions of the same prompt, the expected squared terminal residual therefore bounds the expected sum of squared one-step errors, up to the group-size factor G/(G−1)G/(G-1). We give the derivation in our response to Reviewer Ra4v and will include it in the revised manuscript.

On Qwen3-4B-Base, the value loss alone improves training accuracy, and adding the actor leads to faster gains and higher training accuracy (Figure 5). The ablation shows what the actor adds to optimization, while the analysis above explains when the policy-implied advantage approximates the true one-step advantage.

3. Comparisons with FIPO and VAPO

We chose token-aggregated GRPO for the main experiments because both GRPO and VIMPO adapt PPO to train without a separate critic, but they differ in what they learn from group outcomes. GRPO computes advantages directly from those outcomes, while VIMPO uses them to learn a value signal within the policy. FIPO builds on DAPO, a GRPO variant. It uses changes in the likelihood of the rest of a response as a proxy for a token’s downstream influence and reweights the token’s existing advantage. VIMPO starts from a different principle. Under the local KL-regularized optimality condition, the centered policy-to-reference log ratio gives the one-step advantage. FIPO’s weighting could therefore be applied on top of VIMPO’s actor signal without changing how VIMPO learns it. VAPO instead pretrains a separate critic and adapts advantage estimation to response length so that terminal rewards can inform earlier decisions in long responses.

We compared VIMPO with FIPO on terminal-reward HalfCheetah, where FIPO’s weighting uses the log densities of sampled actions in place of token log-probabilities. Under the same training budget, VIMPO achieved higher held-out return in all three paired runs.

MethodMean held-out return
FIPO69.5
VIMPO288.4

4. Generalizability

On Qwen3-4B-Base, VIMPO outperformed GRPO across four math benchmarks with clean rewards (39.5% vs. 37.4% on average) and retained its lead when 25% of the training rewards were flipped (35.9% vs. 32.2%). In terminal-reward Walker2d, retaining the value loss’s prediction-dependent correction improved mean displacement by 14.79 m across ten paired runs with the actor disabled. This shows that the value objective can support learning in continuous control as well.

We agree that evaluating a single language model limits what we can say about scale. As noted in our response to Reviewer m9Zf, we will include the multi-run comparison on Qwen3-0.6B in the revised manuscript.

Reviewer_m9ZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VIMPO, a critic-free RLVR method with two parts. First, a value loss that trains the policy so that the total log-ratio between the policy and the reference model over a response matches how much its reward beats the group average. This is where the reward actually enters. Second, an actor loss that replaces the critic-based advantage in PPO with the per-token log-ratio, which, based on KL-regularized RL theory, should behave like a TD residual. The idea is to get token-level credit assignment like PPO without training a critic. Experiments on math reasoning and a modified Walker2d task show that it outperforms GRPO.

Critical strengths
  1. The idea is simple and well motivated. Instead of adding heuristic token weights on top of GRPO, the paper derives a token-level signal directly from the KL-regularized RL objective and gets it from the policy itself, with no critic to train. It also fits into the usual group-based RLVR setup with very little extra machinery. Since the tension between critic-free simplicity and fine-grained credit assignment is a real problem in current LLM RL, a clean and principled way to address it is a meaningful contribution.

  2. The value loss looks useful on its own. Unlike GRPO-style updates, it regresses toward a target, so it stops pushing once the prediction matches the reward. The Walker2d ablation shows clearly that this part matters, since removing it causes training to fail. To me this is the most solid finding in the paper, and it gives the work value even beyond the actor component.

Critical weaknesses
  1. The main claim about token-level credit assignment is not directly supported. The value loss pushes every token in a response by the same amount, so any per-token difference in the log-ratio has to come from the model’s own generalization rather than from anything that explicitly teaches it which tokens matter. Whether this signal actually reflects real credit is therefore the central empirical question, but the evidence for it is mostly one qualitative example. The credit measurement on Walker2d is inconclusive, and replacing the implied actor with a GRPO-style actor there gives similar results. No such control is done in the LLM setting, which is where the actor is claimed to help. As a result, it is unclear whether the speedup comes from better credit assignment or simply from extra and larger policy updates (which would also be consistent with the faster KL growth and longer responses seen in the ablation).

  2. The LLM evaluation is too limited to judge how robust the gains are. The results come from a single run, the baselines are only GRPO variants, and there is no comparison with critic-based methods like PPO/VAPO. Since the gains over GRPO are modest, it is hard to tell how much of the improvement would hold up.

Questions for authors

I really like this paper and the method, and I’d love to give it a top score. But its main claim about token-level credit assignment isn’t well supported yet. The questions below would help me resolve this.

  1. In the LLM setting, how does VIMPO compare with (a) the value loss plus a GRPO-style actor, and (b) the value loss plus the VIMPO advantage shuffled within each response?

  2. Does the actor still help when compared at a similar KL level, for example against value-only VIMPO with a larger learning rate?

  3. Are there results over multiple seeds for the main LLM comparison, or at least some sense of run-to-run variance?

2 comments

Thank you for the thoughtful review and for suggesting concrete ways to test the role of the actor.

1. Comparison with competitive baselines

We chose token-aggregated GRPO because both GRPO and VIMPO adapt PPO to language-model training without a separate critic, but they make different choices about value learning. GRPO replaces the learned critic with advantages computed by comparing completions for the same prompt. Many methods build on that choice by refining which completions are sampled or how their advantages are used in the update. VIMPO explores a different direction. It removes the separate critic while retaining value learning, so the policy can use verified outcomes to improve its own estimates of return. This makes GRPO a natural baseline. Both approaches learn from groups of verified completions, but they differ in what they learn from that feedback. On Qwen3-4B-Base, VIMPO averaged 39.5% across four math benchmarks versus 37.4% for GRPO. With 25% of the training rewards flipped, the averages were 35.9% and 32.2%, respectively. The gains in both settings show why learning value within the policy is worth pursuing as a direction in its own right.

2. Evidence on the actor and token-level credit

The Walker2d experiment shows what the value objective contributes on its own. With the optimizer fixed and the actor disabled, retaining the prediction-dependent correction improved mean displacement by 14.79 m across ten paired runs. In the language-model ablation, adding the actor improved training accuracy more quickly. The question raised by the review is whether that speedup comes from assigning credit to tokens or from moving the policy farther.

We trained Qwen3-0.6B on the GSM8K training split and evaluated on held-out GSM8K and MATH-500. We first held the value loss fixed and replaced VIMPO’s token-dependent advantages with a GRPO-style outcome advantage.

PolicyGSM8KMATH-500
Base model53.5%51.0%
Value loss + GRPO-style actor66.5%58.2%
VIMPO68.0%58.1%

In a separate comparison, we increased the learning rate for value-only VIMPO to bring its KL to the reference closer to that of VIMPO with the actor.

PolicyMean KL to reference (nats)GSM8KMATH-500
Base model—53.5%49.8%
Value-only VIMPO0.34456.8%51.6%
VIMPO0.40062.7%55.6%

At these KL levels, VIMPO reached higher mean accuracy by 5.9 percentage points on GSM8K and 4.0 points on MATH-500. We will extend this comparison to longer runs in the revised manuscript.

Your suggestion to shuffle the advantages within each response gives us a useful control. It keeps the advantage values fixed while testing whether it matters which tokens receive them. We will run this comparison and report the results in the revised manuscript.

3. Variation across training runs

The main Qwen3-4B-Base comparison uses one run per method. The initial Qwen3-0.6B results are encouraging. Both actor variants improved on the base model on GSM8K and MATH-500 in both runs. We will include the multi-run Qwen3-0.6B comparison in the revised manuscript.

Reviewer_m9ZfOpenReview ↗

thanks the author, maintain my original rating

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

The new KL-controlled comparison provides useful evidence that the actor contributes beyond the value loss alone. However, my main concern regarding token-level credit assignment remains unresolved.

Overall, I continue to find the method interesting, particularly the value-loss formulation, and appreciate the additional evidence. However, I do not think the current results sufficiently establish the central credit-assignment claim. I will therefore maintain my original rating.

Reviewer_Ra4vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VIMPO, a critic-free method that uses policy–reference log-probability ratios to construct implicit value and token-level advantage signals. Its main contribution is an outcome-supervised regression loss, motivated by a KL-regularized optimality identity and a zero-terminal-value condition, combined with an auxiliary PPO-style actor update.

On Qwen3-4B mathematical reasoning, VIMPO outperforms GRPO across four benchmarks, raising average accuracy from 37.4% to 39.5%, and retains an advantage under 25% reward flipping. Ten-seed experiments on terminal-reward Walker2d support the regression loss’s effectiveness, while ablations indicate that the auxiliary actor accelerates LLM training. The LLM evidence remains limited in model scale and training-seed coverage.

Critical strengths

VIMPO turns a policy–reference log-ratio identity and terminal boundary condition into an outcome-supervised loss that directly trains the policy. It also reuses these ratios for token-dependent actor updates. This offers a coherent way to obtain richer learning signals without a separate value network, making the contribution more substantial than heuristic token reweighting. The ten-seed Walker2d experiments show effective learning with the value loss alone, and matched ablations support the prediction-dependent correction beyond a return-only update. Together with improvements across all four mathematical benchmarks, including under reward corruption, this supports the method’s core learning mechanism rather than only the combined algorithm’s performance.

Critical weaknesses

The derivation holds under a stated local optimality condition, but VIMPO substitutes the evolving policy and treats its log-ratios as advantages. Matching a trajectory-level reward does not establish that intermediate implied values or token-level signals are accurate. This leaves the central credit-assignment interpretation less justified than the regression objective’s empirical usefulness. Experiments use one 4B model and one mathematical training dataset, with limited training-seed coverage and comparisons concentrated on GRPO variants. The actor ablation primarily demonstrates improved training accuracy rather than downstream generalization. Although the repeated Walker2d experiments strengthen the regression claim, they do not resolve whether the complete method consistently improves LLM reasoning over competitive alternatives.

Questions for authors

Clarification on credit assignment: What evidence shows that the policy-implied token advantages track differences in expected return during training, away from the optimality condition used in the derivation? Can the terminal regression objective fit outcome rewards while producing misleading intermediate values?

Clarification on empirical reliability: How many independent training runs support the LLM results, and how were the coefficients selected? Do the reported gains exceed variation across training seeds?

1 comment

Thank you for asking how the advantage derivation applies during training. We address each of your concerns below.

1. Advantage approximation during training

The identity in Sec. 3 is exact under the stated local optimality condition. For a current policy π\pi, consider the local Boltzmann update that tilts the reference policy using QπQ^\pi, the expected return after taking an action and then following π\pi. If the current policy’s log probabilities differ from those of this update by at most ε\varepsilon across actions at a given prefix, the KL-centered log-ratio score differs from the true one-step advantage by at most 2βε2\beta\varepsilon. The same bound applies to the error in the score gap between two actions, so their ordering is preserved whenever the true return gap exceeds 2βε2\beta\varepsilon.

The bound depends only on how closely the current policy matches the Boltzmann update at that prefix. As in DPO, the optimal-policy identity motivates the parameterization. The approximation bound gives an explicit condition for using that parameterization during training.

2. What terminal regression constrains

In the deterministic, undiscounted autoregressive setting with deterministic verifier rewards, the sum of the true one-step advantages along a completion equals R−Vπ(s0)R - V^\pi(s_0). The true advantages AtπA_t^\pi and implied scores A^t\widehat{A}_t both have conditional mean zero at each prefix, so errors at different steps are orthogonal in expectation. For G>1G > 1 independent on-policy completions of the same prompt, expanding the group-mean baseline in Eq. 8 gives

2Lpop=(1−1G)E[∑t(A^t−Atπ)2]+1GE[(∑tA^t)2].\displaystyle 2\mathcal{L}_{\mathrm{pop}} = \left(1-\frac{1}{G}\right)\mathbb{E}\left[\sum_t\left(\widehat{A}_t-A_t^\pi\right)^2\right] + \frac{1}{G}\mathbb{E}\left[\left(\sum_t\widehat{A}_t\right)^2\right].

The second term is nonnegative, so the population loss bounds the expected sum of squared one-step advantage errors along on-policy completions. The identity holds for any policy under these assumptions. On a finite rollout batch, errors at different positions can still cancel, allowing an accurate terminal fit alongside misleading intermediate values.

The terminal-reward Walker2d ablation (pp. 8–9 and 18) shows that the regression drives learning with the auxiliary actor disabled. With the optimizer held fixed, retaining the prediction-dependent correction increased mean displacement from −0.14-0.14 m to 14.6414.64 m. The mean paired gain was 14.7914.79 m across ten seeds (simultaneous 95% CI [12.19,17.39][12.19, 17.39]).

In the revised manuscript, we will state the local approximation bound alongside the advantage derivation and add the population-loss identity, its assumptions, and a short proof to the discussion of terminal regression.

3. Training seeds and hyperparameters

The submitted Qwen3-4B-Base comparisons use one training seed. We did not have compute for additional seeds after completing these runs, so we cannot determine whether the gains exceed run-to-run variation. We adopted the shared hyperparameters from the public implementation of Wang et al. (2025) for both methods. For the VIMPO-specific coefficients, we tried values of the form 1×10−x1 \times 10^{-x} or 0.5×10−x0.5 \times 10^{-x}. We selected β=0.0005\beta = 0.0005 and ca=0.005c_a = 0.005 by inspecting training-accuracy curves over the first 200 updates, then held both fixed for the main runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.