acceptodds
Under review as a conference paper at ICLR 2027

Enabling Token-wise Importance Sampling in Asynchronous RLVR with Proximal KL Regularization

Abstract

In asynchronous reinforcement learning with verifiable rewards (RLVR), rollout generation overlaps with policy optimization, so training responses may come from earlier policy versions. Policy-gradient methods commonly reuse these responses with token-wise importance weighting, avoiding products of likelihood ratios across long responses. Under appropriate conditions, we prove that full future KL credit preserves the exact proximal optimizer as the token-wise update's unique stationary policy. This holds despite gradient bias, even when responses come from a mixture of older policies. The update requires neither full-response importance weights nor a fitted critic. We further establish convergence of tabular continuous-time population updates under persistent coverage and time-varying behavior policies. These results motivate **Proximal KL**, a practical asynchronous RLVR recipe that separates the proximal reference from the sampling policy and uses bounded future KL credit in place of ordinary PPO clipping. We evaluate **Proximal KL** on Qwen3-1.7B and Qwen3-4B, finding improvements over our DAPO control: higher off-policy MATH500 and MinervaMath accuracy at both tested learning rates for 1.7B and higher AIME24 and AIME25 mean@16 and pass@16 at 4B. Ablations show better preservation of AIME mean@16 as rollout lag increases and reduced MATH500 sensitivity to baseline perturbations. The horizon sweep favors a finite future window over both tested extremes, while fixed-policy measurements show higher correction variance at longer tested horizons.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NEL8OpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies off-policy learning in asynchronous RLVR. Because rollout generation and policy optimization overlap, responses may be generated by stale behavior policies.

Exact response-level off-policy correction requires multiplying importance ratios across the full response, which can have very high variance for long sequences. In practice, token-wise importance weighting is more stable, but it generally introduces bias.

The main theoretical result is that, under a reverse-KL proximal objective, if each token’s policy-gradient coefficient includes full future KL credit, then token-wise importance weighting preserves the exact proximal optimizer as the unique stationary policy—even when the behavior data come from stale policies or mixtures of actor versions. 26808_Enabling_Token_wise_Impo

The resulting practical method, Proximal KL, separates the proximal reference policy from the behavior policy, uses token-wise importance weighting, introduces bounded future KL correction, and removes ordinary PPO clipping.

Critical strengths
  1. The theoretical question is important and directly relevant to asynchronous RLVR.

    The paper addresses a real practical puzzle: token-wise importance weighting is biased, yet it appears useful in large-scale systems.

  2. The stationary-policy result is conceptually strong.

    The paper does not claim that the estimator is unbiased. Instead, it proves that the biased population update can preserve the desired stationary policy. This is a useful and nontrivial distinction.

  3. The separation between proximal reference and behavior policy is clean.

    This makes the asynchronous setup conceptually clearer and distinguishes the objective-defining policy from the sampling policy.

  4. Future-KL credit is well aligned with the autoregressive structure.

    The score function at the current token receives current and future regularization costs, which is a principled way to represent the response-level KL objective.

  5. The paper is unusually explicit about the gap between theory and implementation.

    Table 3 clearly lists which theoretical assumptions are changed by the practical implementation, including finite horizon, log-ratio clipping, group normalization, and token averaging.

Critical weaknesses
  1. The strongest theorem does not directly apply to the final practical algorithm.

    The stationary-policy guarantee assumes full future KL, unclipped correction, a particular baseline structure, and a population update. The implementation introduces finite �0��0�, log-ratio clipping, group normalization, token averaging, and Adam, all of which can change the expected field and stationary point.

  2. The empirical gains are relatively modest, and the statistical evidence is limited.

    Except for the 1.7B, �1��1� setting, most experiments use a single seed. In the paired bootstrap analysis, all metrics include zero after Bonferroni correction.

  3. The 4B results are mixed rather than uniformly better.

    Proximal KL improves AIME and Minerva metrics but is slightly worse on MATH500 and OlympiadBench in the reported table. 26808_Enabling_Token_wise_Impo

  4. The finite horizon HH becomes a new important hyperparameter.

    The theory favors the full tail, while practice favors an intermediate horizon, leaving open the question of how HH should be selected or adapted.

  5. The main comparison is concentrated on DAPO.

    Additional off-policy or stale-policy baselines would help clarify whether the gain is specific to replacing PPO clipping or reflects a more general advantage.

  6. The empirical scope is largely restricted to mathematical reasoning.

    The theoretical claims are broader than the experimental domain.

Questions for authors
  1. Can any stationary-policy guarantee be retained after introducing finite horizon, log-ratio clipping, and group-normalized advantages?
  2. Since theory favors the full tail but practice favors H=64H=64, can the variance analysis be used to derive an adaptive rule for choosing HH?
  3. Can the authors add multi-seed results for 4B and an additional learning-rate setting?
  4. Given that all Bonferroni-adjusted intervals include zero, how should the practical superiority of Proximal KL be interpreted statistically?
  5. Can the authors include a small-scale exact response-level importance-sampling experiment to directly compare the stationary behavior of exact and token-wise updates?
1 comment

Dear Reviewer NEL8,

We are following up on our November 9 response and revised manuscript, which include updated experimental results and additional theoretical analysis.

As the discussion period is closing, we would appreciate your feedback on whether these updates address your concerns and which issues remain unresolved. We would be happy to provide any further clarification.

Thank you for your time and consideration.

Reviewer_SGHqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether token-wise importance weighting can preserve a KL-regularized policy-optimization target when training uses stale trajectories. Under finite support, behavior coverage, local policy expressiveness, and ideal population updates, it shows that including the full future KL cost makes the exact proximal optimizer the unique stationary policy, despite the gradient remaining biased away from that optimizer. It also proves convergence for tabular continuous-time population dynamics with persistent coverage. These results motivate Proximal KL, which separates the sampling policy from the proximal reference and replaces ordinary PPO clipping with bounded future KL credit. Experiments on Qwen3-1.7B and Qwen3-4B investigate mathematical reasoning, rollout lag, baseline perturbations, training stability, and credit horizon.

Critical strengths
  1. The stationary-policy result answers a precise and useful off-policy question. Theorem 1 establishes more than the familiar Gibbs characterization or the on-policy Future-KL gradient identity: the critic-free update with only a current-token importance ratio still has the desired unique stationary policy under supported stale-policy mixtures. The pathwise cancellation at the optimizer, careful handling of version-specific behavior probabilities, and backward-induction uniqueness argument in Appendix D explain why a biased gradient can retain the correct target. The tabular convergence result further strengthens this contribution.

  2. The implementation and ablations address relevant practical failure modes. Logging the actual sampling probabilities and separating reference refresh from actor refresh give a concrete asynchronous training recipe. The reward-only control, baseline perturbation, and future-horizon experiments examine why simply removing PPO clipping may be insufficient. The authors also provide three-seed results for one main setting and a question-paired uncertainty analysis, and explicitly avoid claiming that the population theorem directly guarantees the Transformer implementation.

Critical weaknesses
  1. The empirical advantage is more qualified than the broad practical motivation suggests. In the three-seed 1.7B comparison at 5e-6, Table 1 improves MATH500 from 67.33 to 68.07, while both AIME25 metrics slightly decrease. At 4B, the AIME metrics improve but MATH500 and OlympiadBench decrease, and this comparison uses one training seed. Appendix A.4 appropriately reports that all seven Bonferroni-adjusted intervals include zero; this does not disprove an improvement, but it limits the strength of the evidence. The rollout-lag comparison also contrasts one seed at lag 0 with three-seed means at lag 4. Consequently, the evidence for reliably better robustness to staleness is suggestive rather than conclusive.

  2. The experiments do not cleanly isolate the practical benefit of the theoretical mechanism from the other recipe changes. The main comparison is against an adapted DAPO control, while the proposed method changes clipping and regularization together. The horizon comparison also changes the KL coefficient: H=64 uses beta=0.075, whereas the full tail uses beta=0.30. Matching beta times H for the H=1 and H=64 comparison controls a maximum correction bound, but does not hold regularization strength or the learning dynamics fixed. Moreover, the scalar correction-variance diagnostic is not a measurement of policy-gradient variance, as Appendix A.3 acknowledges. A matched comparison to a Future-KL/FRPO-style update and a more controlled horizon comparison would make the practical contribution beyond the stationary-policy theorem more convincing.

Questions for authors
  1. Empirical clarification: With matched training seeds and evaluation budgets at lag 0 and lag 4, how consistent is the claimed advantage in robustness to stale rollouts? Please report uncertainty on the change attributable to lag, rather than only the endpoint scores.
  2. Attribution clarification: Under the same asynchronous engine, reference-refresh schedule, and hyperparameter budget, how does Proximal KL compare with a Future-KL/FRPO-style control and with retaining PPO clipping alongside the same future KL correction?
  3. Horizon clarification: Does the finite-window advantage persist when horizon is varied at fixed beta and when each horizon receives the same tuning budget? Which part of Figure 5's difference can be attributed specifically to truncation rather than coefficient choice or the Adam restart documented in Appendix A?

These questions target the existing causal interpretation of the experiments; a full convergence theorem for shared-parameter Transformers is not required to answer them.

1 comment

Dear Reviewer SGHq,

We are following up on our November 9 response and revised manuscript, which include updated experimental results and additional theoretical analysis.

As the discussion period is closing, we would appreciate your feedback on whether these updates address your concerns and which issues remain unresolved. We would be happy to provide any further clarification.

Thank you for your time and consideration.

Reviewer_KpM7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training with data from earlier policy versions in asynchronous RLVR and proposes Proximal KL. The authors prove that, under certain conditions, the token-wise update with full future KL credit has the exact proximal optimizer as its unique stationary policy despite gradient bias, and establish convergence under stronger conditions. Experiments on mathematical reasoning show improvements over the DAPO control on some metrics.

Critical strengths
  1. The paper provides a valuable theoretical analysis of token-wise importance weighting in asynchronous RLVR.
  2. The proposed method is simple and shows some improvements in mathematical reasoning experiments.
Critical weaknesses
  1. Key comparisons are missing, and the method's contribution has not been sufficiently isolated. FRPO[1] already combines future KL with PPO clipping, but the paper lacks a comparison that retains clipping under matched training settings. It is unclear whether removing clipping itself brings improvements.
  2. The horizon ablation does not clearly isolate the effects of horizon length and the KL coefficient. Figure 5 compares H=64,β=0.075H=64, \beta=0.075 with the full tail at β=0.30\beta=0.30, changing both parameters. The current results do not clearly distinguish their respective effects.

[1] Future-KL Regularized GRPO: Process-Level Credit Assignment from ff-Divergence Regularization

Questions for authors
  1. The comparison between H=64,β=0.075H=64, \beta=0.075 and the full tail at β=0.30\beta=0.30 changes both parameters, while matching βH\beta H only matches the clipped correction bound. Does the advantage of the finite horizon remain when β\beta, the training budget, and the optimizer state are held fixed?
  2. For the 4B model, performance improves on AIME but declines on MATH500 and OlympiadBench, and only one training seed is used. In Table 2, lag 0 uses one seed, whereas lag 4 reports an average over three seeds. Why do different settings use different numbers of training seeds?
  3. The symmetric clipping width of 0.2 in Equation (6) differs from the original Clip-Higher settings of 0.2/0.28. Under the same AReaL implementation, data, lag, and training budget, how do DAPO with Clip-Higher and an FRPO-style update combining future KL with PPO clipping perform? How can the effects of future KL, reference refresh, and removing clipping be distinguished?
1 comment

Dear Reviewer KpM7,

We are following up on our November 9 response and revised manuscript, which include updated experimental results and additional theoretical analysis.

As the discussion period is closing, we would appreciate your feedback on whether these updates address your concerns and which issues remain unresolved. We would be happy to provide any further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.