acceptodds
Under review as a conference paper at ICLR 2027

Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post Training

Abstract

Fully asynchronous reinforcement learning (RL) improves resource utilization in large language model post-training by overlapping rollout generation with policy optimization, but it also introduces policy lag as trajectories are generated and queued while the trainer continues to update. We study how this lag accumulates over a trajectory's lifetime and how it can be controlled without sacrificing the wall-clock benefits of asynchronous execution. We decompose trajectory staleness into Generation Staleness, accumulated before rollout completion, and Waiting Staleness, accumulated after a completed trajectory enters the pool. Motivated by this decomposition, we introduce PACE (Pool-Aware Control of Effective Staleness). PACE converts excess pool occupancy into an adaptive rejection budget and ranks completed trajectories using an effective-staleness score that combines Waiting Staleness with prefix-aware Generation Staleness. This avoids penalizing long or interrupted rollouts solely because they span multiple policy versions. In single-turn mathematical reasoning, PACE improves the six-benchmark average validation accuracy by 18.7% over unfiltered asynchronous RL at the same wall-clock budget and matches synchronous RL performance with 47.1% less GPU time. PACE also improves validation performance in multi-turn tool-integrated reasoning, outperforming both synchronous and unfiltered asynchronous RL. Further experiments with the mixture-of-experts model and an alternative RL algorithm support its applicability across model architectures and training algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, Asynchronous Reinforcement Learning, Policy Staleness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ft51OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACE is an admission controller for asynchronous GRPO that decides which completed prompt groups enter training. Pool occupancy sets a smoothed rejection budget, while candidates are ranked by waiting age plus generation age discounted by prefix drift. The score separates policy updates crossed during generation from those crossed while a completed group waits in the pool. The controller admits whole groups, preserving the group-relative advantage structure and the form of the policy loss while changing the distribution of training data. Experiments on Qwen3 dense and MoE backbones include a ReTool agentic setting. Matched full/raw comparisons support a benefit from prefix scoring, while the larger ReTool comparison with Async uses different rollout resources.

Critical strengths

S1. The controller treats the two sources of lag differently, and this is the paper's most useful design idea. Occupancy decides how much to reject, while prefix drift discounts only the generation age and leaves the waiting penalty intact. Because admission operates on whole prompt groups, GRPO's group-relative advantage structure survives the filtering.

S2. A controlled comparison shows that the prefix-drift score adds value beyond raw staleness. With matched resources and the same pool controller, full PACE improves the selected six-benchmark validation mean by 2.09 points in both tested settings: 41.29 versus 39.20 for Qwen3-8B and 47.91 versus 45.82 for ReTool.

Critical weaknesses

W1. The 18.67-point ReTool gain over Async comes from runs with different rollout allocations. PACE uses 4r2t and Async uses 2r2t. Extra capacity can change sample supply and pool backlog, so dividing by GPU-hours does not isolate the admission policy's contribution. The matched full/raw comparison does support a 2.09-point gain from prefix scoring, but that control measures an incremental component benefit and cannot explain the size of the full/Async gap.

W2. The MoE result needs a check that the trainer and rollouter agree numerically. Behavior log probabilities are recorded, but the paper does not compare token scores from the two engines at the same checkpoint. Routing replay is disabled, and prefix scoring is performed on the rollout side. A discrepancy between the engines would enter the policy ratio even without a policy-version change.

Questions for authors

Q1. How large is the full PACE/Async gap with the same rollout/trainer allocation and checkpoint-selection rule? An existing matched comparison, or one matched Async run, with available training-run variation would address the attribution issue. Please also report GPU-hours to a common validation target and identify the runs and timing boundaries used for the cost measurements.

Q2. At a fixed MoE checkpoint and on identical sequences, how large are the differences between trainer and rollouter log probabilities? Please report policy-ratio and clipping statistics as well, and investigate routing only if the difference is material.

Reviewer_e87bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE controls trajectory admission in asynchronous LLM RL. It separates generation and waiting staleness, sets rejection pressure from pool occupancy, and uses prefix-policy drift to discount generation age. Experiments on mathematical and tool-integrated reasoning report improved validation performance while retaining asynchronous training efficiency.

Critical strengths
  • Staleness control is a practical problem for asynchronous RL. The pool-based formulation offers a useful systems perspective.
  • The experiments cover both single-turn and tool-integrated reasoning.
Critical weaknesses
  1. The ReTool comparison is confounded by resource allocation: Async uses 2 rollout + 2 training GPUs, while PACE uses 4 + 2 (Appendix C.1). Rollout supply directly affects the pool dynamics under study, so matched-resource comparisons are needed.

  2. The ablations do not isolate the benefit of prefix-aware ranking from simply keeping a fresher queue. Waiting-only filtering or capacity-based queue dropping at matched rejection rates would clarify this.

Questions for authors
  1. Why does PACE outperform Sync despite greater policy staleness?
  2. How does PACE compare with waiting-only filtering or capacity-based queue dropping at matched rejection rates?
  3. Please clarify the dense evaluation counts and averaging. For example, 57.66% on AIME24 cannot arise from one binary evaluation of 30 questions with 32 responses each. Were different counts or repeated evaluations used?
Reviewer_gvjiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies policy lag in asynchronous LLM reinforcement learning. It decomposes token-averaged lag into generation and waiting components, then proposes PACE: pool occupancy sets a rejection budget, while an effective-staleness score combines waiting age with prefix-drift-weighted generation age to rank completed data. The method changes admission rather than the optimization objective. At selected checkpoints, Table 1 reports six-benchmark averages of 41.29% versus 36.34% for unfiltered Async in single-turn math, and 47.91% versus 29.24% in ReTool. Additional dense-model, MoE, and REINFORCE++ experiments provide supporting evidence. The central assessment questions are whether the comparisons isolate PACE's benefit and whether prefix-aware weighting adds value beyond simpler backlog control.

Critical strengths
  1. A practical and coherent intervention point. PACE distinguishes waiting in the completed pool from generation age and places control at admission, without requiring a new policy objective. This is a useful way to exploit surplus rollout supply while retaining asynchronous overlap. The lower waiting-staleness distributions in Figures 6–7 provide direct evidence that the intervention changes the intended operational quantity (pp. 4–6 and 8).
  2. Promising empirical breadth, including a resource-matched corroboration. Table 1 and the full benchmark curves show gains beyond one isolated checkpoint, and the paper includes single-turn math, tool use, smaller dense models, MoE, and another RL algorithm. In particular, the MoE comparison uses 24 GPUs for both synchronous and asynchronous configurations; Figure 9 labels PACE at 27.1% AIME 2024 accuracy versus 23.5% for Async and 29.2% for Sync, with PACE retaining Async-like step time (pp. 9 and 15). This supports the practical direction even though it does not resolve every comparison concern.
Critical weaknesses
  1. The largest reported gain is not isolated from resource allocation and checkpoint selection. For ReTool, Appendix C.1 explicitly assigns Async to 2 rollout plus 2 trainer GPUs, but PACE and M2PO to 4 rollout plus 2 trainer GPUs; Sync uses 4 GPUs (p. 15, lines 771–777). Extra rollout capacity changes supply, backlog, and the candidate data available for filtering, so the 47.91% versus 29.24% comparison cannot by itself attribute the improvement to PACE. Sync is also evaluated every 100 optimizer updates, versus every 25 for the asynchronous methods, while Table 1 selects the best validation average. A common optimizer-step axis does not equalize the opportunities for best-checkpoint selection. Separately, the headline GPU-time claim needs a traceable experiment mapping: the main Qwen3-8B setup is 12r4t, whereas Figure 8 profiles six GPUs. These may be separate valid experiments, but their relationship is not made explicit enough to audit the headline efficiency claim (pp. 2, 7, 9, 15).
  2. The specific value of prefix-aware weighting is not established by the ablations. Raw Staleness fixes the generation coefficient at one, whereas full PACE uses a coefficient between zero and one. Their comparison therefore changes both the amount of generation-age penalty and its trajectory-specific dependence on prefix drift. The generation-only ablation does not separate those effects. A waiting-only or matched constant-discount control under the same pool budget would be more diagnostic. Full PACE exceeds Raw Staleness by 2.09 percentage points in both Table 1 settings, but training-seed uncertainty is not reported. Moreover, Figure 12's separate fixed-policy accuracy–age correlation motivates a possible difficulty bias; it does not show that PACE actually retains harder useful groups or that the prefix score predicts their value. These gaps matter because prefix-aware effective staleness is the main refinement beyond backlog pruning (p. 6, Table 1, pp. 17–18).
Questions for authors
  1. Resource and selection control: Can the ReTool comparison be provided with PACE and Async on the same rollout/trainer allocation, and all methods evaluated at common optimizer checkpoints? Existing logs at common checkpoints would help assess the selection effect; a matched-allocation comparison is needed to assess the supply effect. Please report the corresponding allocated GPU-hours.
  2. Efficiency clarification: Which model, GPU type/count, allocation, checkpoints, and timing components produce Figure 1's 18.7% and 47.1% claims, and how do these runs relate to the six-GPU profile in Figure 8? Please give the actual numerator/denominator values and distinguish cumulative logged training-step time from end-to-end elapsed time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.