acceptodds
Withdrawn from ICLR 2027

V-Trace Policy Optimization: When Multi-Step Correction Helps, and Why

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning for language models increasingly departs from the on-policy ideal: rollouts are reused across optimizer passes, actors lag learners, and inference and training engines disagree on token probabilities. PPO and GRPO correct only the current token, whereas exact trajectory importance sampling corrects the whole sequence at prohibitive variance. We show that prefix, current-action, and suffix ratios reweight the visited history, the sampled action, and its continuation, respectively. We introduce V-Trace Policy Optimization (V-TRACE PO), which attaches a coefficient-truncated suffix trace directly to the per-token score. One reverse scan computes the trace, and on complete terminal-reward responses it requires no learned critic; an exact identity says which continuation policy the truncated trace targets, and whatever the truncation, the correction adds no bias to the per-token score. We evaluate it in two families of individually preregistered experiments: one in which the same rollouts are reused across several optimizer passes, and one in which every sample is fresh but the behavior policy differs from the learner by construction—a lagging generator, a hotter sampler, or sequential minibatches— together with a sweep of the truncation cap, so the correction is also tested at full strength. The change-of-measure argument that motivates the estimator applies to the second family and not the first. The results come out the other way round: V-TRACE PO helps under reuse, and reliably in none of our fresh-sample controls, where widening the correction is if anything harmful. We analyze both halves. On fresh samples the correction does not pay: at every truncation we test, the variance it adds exceeds the bias it removes — even at the mildest mismatch we can construct, the numerical disagreement between the inference and training engines. Under reuse it helps for a reason the construction does not predict: the trace silently becomes an asymmetric learning rule, amplifying whatever an earlier pass over the same batch already did — promoting what succeeded and damping what failed. Under heavy sample reuse the improvement of V-TRACE PO over GRPO is substantial, and it comes with a wider usable trust region — hence less clipping — where that same widening degrades GRPO

Primary area:
reinforcement learning
Keywords:
reinforcement learning for large language models, off-policy correction, importance sampling, grpo, vtrace

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_vfhTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents V-trace PO, which is motivated by v-trace algorithm. The main idea is to use a well-designed truncation to interpolate between suffix ratio and current token ratio, thus reaching a better variance-bias tradeoff compared to the high-variance of suffix ratio and large bias of token ratio. The paper claims this method can help reuse data (if I understand correctly).

Critical strengths

The method looks reasonable and interesting.

Critical weaknesses
  • The paper is basically not human readable: the structure is messy, the flow is discontinuous, the causality is weak.
  • The central claims and insights are not clear.
  • The experiments are limited and weak. There is only one baseline, one model and one task included. Whether the claims can generalize is unknown.
Questions for authors
  • Although the full paper has 58 pages, I didn't find the proof of Proposition 2.2, which is a critical property of the proposed method.
  • What's the point of Section 4 and how it connect to the proposed method?
  • Can you explain what's the key points and insights of your paper?
Reviewer_NMkFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies multi-step off-policy correction for LLM policy optimization. It decomposes the trajectory importance ratio into prefix, current-action, and suffix factors via four exact finite-horizon identities (Prop 2.1). It then proposes V-TRACE PO, which attaches a coefficient-truncated suffix trace (Eq. 7) to each token’s score, computed by a reverse scan, inside a GRPO-style objective. A further identity (Thm 2.3) characterizes the continuation policy induced by any truncation rule, recovering V-trace and Retrace as special cases. Experiments use Qwen3-8B-Base trained on MATH and compare only against matched GRPO. Under fresh-sample mismatch (lagged generator, hotter sampler, sequential minibatches, cap and horizon ladders), the correction yields no reliable gain and can hurt. Under heavy rollout reuse (E=8), V-TRACE PO outperforms GRPO, mainly because GRPO degrades. The authors attribute this benefit to an advantage-sign asymmetry that the trace induces, rather than to the change of measure that motivated the estimator.

Critical strengths
  1. The theory is careful and unifying. The ratio anatomy (Prop 2.1) and the exact characterization of the truncated-trace continuation (Thm 2.3) cleanly relate per-token, suffix, and full-trajectory corrections, and recover V-trace and Retrace as special cases.
  2. The reporting is transparent. The paper reports negative results, per-seed numbers, paired tests, and candid limitations. These are valuable on their own, but they do not compensate for the issues below.
Critical weaknesses
  1. The paper’s central claims are not supported by its own evidence. (a) The change-of-measure argument that motivates the estimator fails empirically. Across all three no-reuse designs, the cap ladder, and the horizon ladder, there is no reliable gain, and the default cˉ=1\bar c=1 is worse than no correction on fresh samples (M.10). (b) The proposed replacement explanation, an advantage-sign asymmetry under reuse, is not established. The key ablation fails its pre-registered manipulation check (residual asymmetry +0.414 vs. a threshold of ±0.10), and its outcome matches neither pre-registered interpretation. V-TRACE PO is not significantly better than the shuffled arm (p=.125) or the fixed-pair arm (p=.156). (c) Most importantly, the fixed-pair arm, a constant weight by sign(A) with no trace at all, already gains +1.39 over GRPO. This suggests that much of the benefit may come from simple asymmetric weighting, as in Asymmetric REINFORCE or TOPR. The necessity of the proposed trace is therefore not demonstrated. As a result, neither the method’s motivation nor its claimed mechanism holds up.
  2. The practical gains are narrow and measured against a degraded baseline. (a) The headline +3.07 at E=8, ϵ=1 comes mainly from GRPO collapsing (8.211 → 5.793) in a configuration practitioners would not use. Comparing each method at its best configuration, the gap is about 0.64 AUC (GRPO 8.229 at E=4, ϵ=.2 vs. V-TRACE PO 8.865 at E=8, ϵ=1). (b) Most primary p-values are .0625–.156, and the authors describe the series as exploratory. (c) At matched updates, E=8 PO-TC trails fresh-data GRPO (Recent −1.89, MATH-500 −7.61), failing the pre-registered non-inferiority criterion. No rigorous compute-matched comparison is given. (d) There is a single model, a single task, and a single training baseline. Methods designed for exactly this regime (e.g., M2PO, GSPO, BAPO, and NFPO, which the authors call “closest in mechanism”) are discussed but never run.
Questions for authors
  1. Can you compare V-TRACE PO against a properly tuned asymmetric baseline (e.g., fixed-pair weights with tuned magnitudes, or Asymmetric REINFORCE / TOPR) under the same reuse setting? If such a baseline matches V-TRACE PO, what remains of the contribution?
  2. Can you provide a compute-matched comparison and run at least one reuse-oriented baseline (M2PO or NFPO), plus a second model or task?
  3. Several cross-references are inconsistent. Please reconcile them: (i) §4 cites a p99 of 6.16 and a log-ratio of .0136 from Table 5, which reports V-TRACE PO-TC, not the default V-TRACE PO; (ii) §6.2 attributes the 0.68 loss to the “untruncated trace”, but in Table 19 this corresponds to cˉ=2\bar c=2; (iii) the E=8, ϵ=1 cell is 5.793/8.865 in Table 22 but 5.51/8.19 in Table 2, and the seeds for the latter are not listed; (iv) M.14 describes the ϵ=.2 half as “a fresh set of 144 training runs”, which contradicts the 144-run total; (v) §6.2 does not identify which of “the sixteen cells” gains at all six seeds; (vi) the dose-matched follow-up in M.12 was run but its results are not reported.
  4. Can you provide verifiable pre-registration records (timestamps or links)? Given that the evaluation pool was replaced after earlier results had been inspected and the experiment order was chosen adaptively, how should readers interpret the confirmatory claims? The manuscript (58 pages) mixes superseded studies, two method arms (PO and PO-TC), and two advantage-normalization conventions. 5. Could the main text and appendix be consolidated around a single protocol so that the evidence for each claim is unambiguous?
Reviewer_SgCyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes V-Trace Policy Optimization, which adds a coefficient-truncated suffix trace to the token-level PPO/GRPO objective to correct the continuation distribution. Experiments on mathematical reasoning show gains over matched GRPO under aggressive rollout reuse but no reliable gains in the tested fresh-sample mismatch settings.

Critical strengths

The paper addresses a practically relevant problem in LLM reinforcement learning, as rollout reuse and behavior–learner mismatch commonly arise in training.

Critical weaknesses
  1. The explanation of the method's empirical gains remains insufficiently established. The method shows no reliable benefit in the tested fresh-sample mismatch settings, while its gains under sample reuse are attributed to a mechanism different from the original change-of-measure rationale. This gap between the motivation and the empirical findings is not sufficiently resolved.

  2. GRPO is the only external algorithmic baseline evaluated, although the paper discusses other methods for off-policy training, suffix weighting, and asymmetric updates (Sections 5 and 7). The comparisons with the method's own variants and ablations do not establish its advantage over these existing approaches. Relevant external baselines are needed to assess the value of the proposed trace in the rollout-reuse setting where it helps.

  3. Minor: The abstract consists of three paragraphs, whereas the official guideline requires a single paragraph.

Questions for authors

None

Reviewer_NYYWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-step correction in LLM reinforcement learning when the behavior policy differs from the learner. It proposes V-Trace PO, which computes a truncated suffix trace in one reverse scan and applies it as a detached multiplier on the per-token PPO/GRPO loss, and derives its relation to V-trace and Retrace. Experiments train Qwen3-8B-Base on MATH in two kinds of settings: reusing each batch for several optimizer passes, and using each sample once while constructing a policy mismatch. The method outperforms GRPO only under heavy reuse and gives no reliable gain with fresh samples. The authors attribute the reuse gain to an asymmetric weighting by advantage sign that the trace acquires on its own.

Critical strengths

The experiments are carefully designed and honestly reported. The design and analysis of each experiment were fixed in advance, seeds are paired, and the paper reports null results and a failed check, and acknowledges that the motivating derivation does not explain the gain.

The negative results with fresh samples are informative. Under several kinds of policy mismatch, suffix correction gives no reliable gain, and when the only mismatch is the numerical difference between the inference and training engines, stronger correction is worse. The method itself is simple, needs no critic, and can directly replace a GRPO loss.

Critical weaknesses

The theoretical motivation does not explain where the method works. The derivation targets off-policy correction under a fixed behavior policy, but in the fresh-sample experiments with a lagging generator, a hotter sampler and sequential minibatches, the method gives no reliable gain, and the default cap is slightly worse than no correction. The method helps only when the same batch is updated repeatedly, where the premise of the derivation no longer holds. The fresh-sample runs also last only tens of updates, which may be too short to show the long-run stability that off-policy correction targets. It therefore remains unclear whether the method is useful in its motivating settings, such as lagging actors and engine mismatch.

The gain under reuse is small relative to a well-tuned GRPO. The main gap is measured against GRPO at E=8 and ε=1, a cell that has itself collapsed: response length blows up and accuracy ends below the untrained model. At the standard ε=0.2 the gap is much smaller, and GRPO with a moderate number of reuse passes comes close to the best V-Trace PO result. The method thus seems mainly to avoid the degradation caused by excessive reuse, which fewer reuse passes would also avoid. The authors should compare the two methods, each at its best setting.

The evidence for the asymmetry mechanism is weak, and simpler alternatives are not compared. The mechanism probes are short single-seed runs, and the asymmetry already appears on the first update, which the explanation does not predict. A pair of constants keyed on the advantage sign, without any trace, recovers about half of the gain, and the follow-up that sets these constants to the trace's measured levels is not reported. If the gain comes from damping negative-advantage updates, methods that treat positive and negative samples asymmetrically, such as Mu-GRPO and BAPO, as well as NFPO, the closest method in mechanism, should be included as baselines.

Questions for authors
  1. Please evaluate the method with longer training in realistic asynchronous or engine-mismatch settings, and give the full setup and cell results of the lagging-generator, hotter-sampler and sequential-minibatch studies.

  2. Please compare GRPO and V-Trace PO, each at its best reuse depth and clipping range, including the standard clipping range.

  3. Please report the follow-up with constants set to the trace's measured levels, and compare with Mu-GRPO, BAPO, NFPO or simple down-weighting of negative advantages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.