- Rating
- 1(−1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies off-policy learning in asynchronous RLVR. Because rollout generation and policy optimization overlap, responses may be generated by stale behavior policies.
Exact response-level off-policy correction requires multiplying importance ratios across the full response, which can have very high variance for long sequences. In practice, token-wise importance weighting is more stable, but it generally introduces bias.
The main theoretical result is that, under a reverse-KL proximal objective, if each token’s policy-gradient coefficient includes full future KL credit, then token-wise importance weighting preserves the exact proximal optimizer as the unique stationary policy—even when the behavior data come from stale policies or mixtures of actor versions. 26808_Enabling_Token_wise_Impo
The resulting practical method, Proximal KL, separates the proximal reference policy from the behavior policy, uses token-wise importance weighting, introduces bounded future KL correction, and removes ordinary PPO clipping.
Critical strengths
-
The theoretical question is important and directly relevant to asynchronous RLVR.
The paper addresses a real practical puzzle: token-wise importance weighting is biased, yet it appears useful in large-scale systems.
-
The stationary-policy result is conceptually strong.
The paper does not claim that the estimator is unbiased. Instead, it proves that the biased population update can preserve the desired stationary policy. This is a useful and nontrivial distinction.
-
The separation between proximal reference and behavior policy is clean.
This makes the asynchronous setup conceptually clearer and distinguishes the objective-defining policy from the sampling policy.
-
Future-KL credit is well aligned with the autoregressive structure.
The score function at the current token receives current and future regularization costs, which is a principled way to represent the response-level KL objective.
-
The paper is unusually explicit about the gap between theory and implementation.
Table 3 clearly lists which theoretical assumptions are changed by the practical implementation, including finite horizon, log-ratio clipping, group normalization, and token averaging.
Critical weaknesses
-
The strongest theorem does not directly apply to the final practical algorithm.
The stationary-policy guarantee assumes full future KL, unclipped correction, a particular baseline structure, and a population update. The implementation introduces finite , log-ratio clipping, group normalization, token averaging, and Adam, all of which can change the expected field and stationary point.
-
The empirical gains are relatively modest, and the statistical evidence is limited.
Except for the 1.7B, setting, most experiments use a single seed. In the paired bootstrap analysis, all metrics include zero after Bonferroni correction.
-
The 4B results are mixed rather than uniformly better.
Proximal KL improves AIME and Minerva metrics but is slightly worse on MATH500 and OlympiadBench in the reported table. 26808_Enabling_Token_wise_Impo
-
The finite horizon becomes a new important hyperparameter.
The theory favors the full tail, while practice favors an intermediate horizon, leaving open the question of how should be selected or adapted.
-
The main comparison is concentrated on DAPO.
Additional off-policy or stale-policy baselines would help clarify whether the gain is specific to replacing PPO clipping or reflects a more general advantage.
-
The empirical scope is largely restricted to mathematical reasoning.
The theoretical claims are broader than the experimental domain.
Questions for authors
- Can any stationary-policy guarantee be retained after introducing finite horizon, log-ratio clipping, and group-normalized advantages?
- Since theory favors the full tail but practice favors , can the variance analysis be used to derive an adaptive rule for choosing ?
- Can the authors add multi-seed results for 4B and an additional learning-rate setting?
- Given that all Bonferroni-adjusted intervals include zero, how should the practical superiority of Proximal KL be interpreted statistically?
- Can the authors include a small-scale exact response-level importance-sampling experiment to directly compare the stationary behavior of exact and token-wise updates?
Dear Reviewer NEL8,
We are following up on our November 9 response and revised manuscript, which include updated experimental results and additional theoretical analysis.
As the discussion period is closing, we would appreciate your feedback on whether these updates address your concerns and which issues remain unresolved. We would be happy to provide any further clarification.
Thank you for your time and consideration.