acceptodds
Under review as a conference paper at ICLR 2027

ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning

Abstract

Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learninig, Agentic Reinforcement Learning, Efficient Learning, Post-Training Optimization, Learning System

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JsBGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ThunderSyncRL, a framework that overlaps rollout and gradient computation while preserving a synchronous policy update. For GRPO, the method computes trajectory-level score gradients before all group rewards are available and maintains reward-weighted and unweighted gradient sums. These statistics recover the group gradient once its rewards are known. For on-policy distillation, teacher-scored turns release gradient computation before the trajectory finishes. The paper reports up to approximately 1.9× acceleration over synchronous training, together with improved fixed-budget performance relative to its asynchronous baselines.

Critical strengths
  • The paper addresses a practical bottleneck in agent training. Long trajectories and expensive environment interactions can leave learner resources idle, making earlier gradient computation useful.

  • The GRPO decomposition is simple and appealing. It separates trajectory-local gradient computation from group-dependent scalar coefficients without changing the specified batch update. Keeping parameters fixed throughout the logical batch also avoids policy staleness.

  • The evaluation covers both GRPO and OPD, two agent benchmarks, multiple model configurations, and two GPU types. The numerical update-equivalence checks and analysis of rollout–learner balance provide useful support for the approach.

Critical weaknesses
  • The asynchronous comparison does not establish an advantage over appropriately corrected asynchronous training. Appendix D.4 explicitly states that the one-step Async baseline learns from stale trajectories without an additional behavior-policy importance correction. The authors explain that this holds the surrogate fixed across schedules, which is a useful controlled experiment. However, it leaves open whether the reported accuracy gap persists against an asynchronous algorithm designed to handle stale data.

  • Importance sampling does not guarantee identical finite-sample learning behavior: long-horizon weights can have high variance, and practical clipping can introduce bias. Nevertheless, the paper does not investigate whether these effects explain its results. A baseline with an established staleness correction, such as the AReaL-style objective discussed in the manuscript, would make the comparison more convincing. A staleness sweep and diagnostics of policy divergence and importance weights would also help explain when zero staleness matters. Currently, the evidence supports superiority over the specified async baselines, while the broader explanation remains incomplete.

  • Learner scheduling and batching are insufficiently specified. The single-lane scheduling model and Appendix I’s pseudocode suggest sequential processing of ready gradient sources within each learner rank. However, the implementation description does not clearly state whether waiting sources are dynamically batched or whether independent computation graphs can execute concurrently.

  • This distinction matters because samples may arrive during an ongoing backward pass. Processing queued samples individually maximizes scheduling granularity, but batching them could improve GPU throughput. For GRPO, preserving both reward-weighted and unweighted gradient statistics also complicates ordinary batching. Please specify the actual learner batching policy and report an ablation comparing individual-source processing with dynamic microbatching, ideally with learner throughput, memory usage, and end-to-end step time. The fixed-service-time scheduling analysis alone does not resolve this tradeoff.

  • A group-ready synchronous baseline would better isolate the contribution of the GRPO decomposition. Sync waits for the entire logical batch, whereas ThunderSyncRL can start before a group finishes. An intermediate baseline could start ordinary advantage-weighted backward computation as soon as one complete group becomes available, while retaining the same final optimizer barrier. Comparing against this baseline would distinguish the benefit of trajectory-level gradient streaming from the more general benefit of overlapping completed groups with unfinished rollout. The paper discusses group-ready scheduling but does not include it among the reported experimental baselines.

  • Equation (4) introduces the token weights without specifying their values in the main text. Appendix D.1 defines (w_{i,t}=1/(32L_i)), so GRPO does include per-trajectory token normalization. This definition should appear alongside the equation. The authors should also explain why GRPO uses a mean of trajectory-normalized contributions while OPD uses a global token mean. These are different weighting choices, although neither invalidates the streaming construction.

Questions for authors

Please refer to the weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer JsBG (1/5)

Thank you for your detailed questions and constructive suggestions. We address each of your questions and feedback below.

W1. Comparison with corrected asynchronous training

The asynchronous comparison does not establish an advantage over appropriately corrected asynchronous training.

We evaluated ThunderSync against importance-weight-corrected asynchronous training on SWE-bench Verified after training on SWE-smith. AReaL [1], configured with the same GRPO objective, performs best among the corrected baselines we evaluated. Compared with AReaL, ThunderSync reaches the 74.01% pass@3 target 1.21× faster and achieves a 0.80-percentage-point higher pass@3 under a fixed 128 GPU-h budget.

Following AReaL [1], we correct for the difference between the policy that generated each token and the current learner policy. For the same token and context, we divide its probability under the current learner snapshot (πprox\pi_{\mathrm{prox}}) by its probability under the generation policy (πbehav\pi_{\mathrm{behav}}). We cap this ratio at 5 and use it to weight the token’s GRPO contribution. The weight is held constant during backpropagation:

wˉ=min⁡(5,πproxπbehav)\bar w=\min\left(5,\frac{\pi_{\mathrm{prox}}}{\pi_{\mathrm{behav}}}\right)

We set the controller’s staleness limit to η=4, following AReaL’s coding configuration [1]. Both corrected baselines use the same cap of 5 to limit extreme weights without discarding tokens. These settings are fixed before test evaluation. We distinguish the controller setting η from the observed token ages reported in our response to W2.

During generation, we store each scored token’s sampled log-probability and policy-version ID. When weights are updated during an ongoing rollout, subsequent tokens receive the new policy-version ID, while previously recorded probabilities remain unchanged. We rebuild the KV cache after the weight refresh. Before each learner update, we evaluate πprox\pi_{\mathrm{prox}} at the current learner snapshot. Each token’s correction therefore uses the policy that actually generated it, including when the policy changes within a trajectory.

We compare two corrected asynchronous baselines with ThunderSync using Qwen-3.8-27B on eight B200 GPUs. All methods use the same GRPO advantage calculation, trajectory-mean normalization and one full-batch optimizer update per training batch. Async + decoupled GRPO allows a maximum policy lag of one update. AReaL uses the controller staleness limit η=4 and allows weight updates during ongoing rollouts.

We retain the manuscript’s pre-specified 74.01% SWE-bench Verified GRPO pass@3 target (§4.2; Appendix E.3) across the added comparisons. For each seed, time-to-target ends at the first evaluated optimizer-boundary checkpoint meeting the target, without interpolation. GPU-hours count all eight allocated GPUs from the first rollout request, including rollout, tool/verifier waits, learning, publication and checkpoint pauses. One-time setup and offline evaluation are timed separately. The table below reports results as mean ± standard deviation across three training seeds.

MethodPass@3 at 128 GPU-h (%)GPU-h to 74.01%
AReaL73.73 ± 0.12145.1 ± 5.2
Async + decoupled GRPO73.33 ± 0.42153.3 ± 5.0
ThunderSync74.53 ± 0.31120.0 ± 3.3

These comparisons show that ThunderSync retains an advantage in both time to target and fixed-budget performance against the evaluated asynchronous baselines with staleness correction enabled.

[1] Fu et al. (2026). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning.

AuthorsOpenReview ↗

Response to Reviewer JsBG (2/5)

W2. Weight variance, truncation bias and staleness diagnostics

Importance sampling does not guarantee identical finite-sample learning behavior: long-horizon weights can have high variance, and practical clipping can introduce bias.

We agree that importance weighting alone does not guarantee the same learning behavior with finite samples, particularly when weights become uneven or are capped. Our response to W1 compares ThunderSync with corrected asynchronous baselines. Here, we examine how policy staleness relates to correction weights and learning performance.

We profile a 10,000-token pre-update window from SWE-smith training for each configuration and evaluate pass@3 on SWE-bench Verified at 128 GPU-hours. The first four rows vary the allowed policy lag while keeping the generation policy fixed within each trajectory. The final row reports the AReaL configuration used in W1, which allows weight updates during ongoing rollouts.

Execution / lag settingMean token lagCorrected pass@3 at 128 GPU-h (%)Raw KL (nats)Raw weight p95ESS before capESS after capAbove cap 5
Fixed policy / 1173.33 ± 0.420.03291.8393.75%93.75%0.0%
Fixed policy / 21.573.13 ± 0.310.07982.6782.19%82.19%0.0%
Fixed policy / 42.572.80 ± 0.400.19174.0060.68%60.68%0.0%
Fixed policy / 84.572.40 ± 0.400.44706.0036.70%42.79%6.0%
AReaL / η=42.1573.73 ± 0.120.13822.0369.98%74.51%0.35%

Mean token lag measures the observed difference between the learner and generation policy versions. KL measures the difference between their probability distributions. Raw weight p95 is the 95th percentile of the importance weights before capping, and “Above cap 5” reports the fraction exceeding 5.

Effective sample size (ESS) summarizes how evenly the correction weights are distributed. We report it as a percentage of the NN profiled tokens:

ESSN=(∑twt)2N∑twt2.\frac{\mathrm{ESS}}{N}=\frac{\left(\sum_t w_t\right)^2}{N\sum_t w_t^2}.

Equal weights give 100%, while weights concentrated on a smaller number of tokens give a lower value. We compute this measure before and after applying the cap. All tokens remain in the calculation, so a lower ESS does not mean tokens were discarded or represent a count of independent trajectories.

In the fixed-policy sweep, larger lag coincides with greater policy divergence, more uneven weights and lower corrected pass@3. At a lag limit of 8, capping raises ESS from 36.70% to 42.79%. However, capping also changes the estimator and can introduce bias. For the AReaL configuration in W1, ESS rises from 69.98% to 74.51% after capping, with 0.35% of weights exceeding 5.

ThunderSync achieves 74.53 ± 0.31% pass@3 at the same 128 GPU-h budget, compared with AReaL’s 73.73 ± 0.12%, a 0.80-percentage-point advantage. In ThunderSync’s default single-update setting, rollout generation and gradient computation use the same policy snapshot. Its behavior-correction weights are therefore all 1, giving a weight-based ESS of 100%, compared with AReaL’s 74.51% after capping.

These observations are consistent with a finite-sample cost of staleness, but they do not isolate how much of the performance gap comes from weight variance or truncation bias. ThunderSync avoids uneven weighting caused by staleness correction in this setting. Other sources of gradient variance, including sampled trajectories and rewards, remain.

AuthorsOpenReview ↗

Response to Reviewer JsBG (3/5)

W3. Learner scheduling and dynamic microbatching

However, the implementation description does not clearly state whether waiting sources are dynamically batched or whether independent computation graphs can execute concurrently.

Each learner GPU processes one microbatch at a time and completes its forward and backward passes before starting another. Before starting the next microbatch, the learner updates task readiness as rewards arrive and earlier work finishes, then selects the earliest queued task that can run (FIFO). No task type receives special priority.

For GRPO, we allow up to four incomplete groups per node to accumulate gradients, with at most one group per learner rank. Available slots are assigned in the order that groups become ready for early gradient computation. If no slot is available, a trajectory remains queued until a slot opens or all rewards in its group arrive.

The learner processes ready trajectories individually (microbatch size 1) when their group rewards are incomplete. Once group rewards are complete, their advantages are known, so consecutive ready trajectories can be packed together in FIFO order. Each microbatch has a target budget of 16,384 scored tokens per learner rank. A trajectory exceeding this budget runs alone.

For OPD, each turn becomes ready once it has been scored by the teacher and follows the same queue and packing policy.

Only tasks belonging to the current optimizer update can run. All deferred backward computation, normalization and reduction finish before the optimizer updates the parameters.

W4. Individual-source processing versus dynamic microbatching

Processing queued samples individually maximizes scheduling granularity, but batching them could improve GPU throughput.

Under the ready-queue policy in W3, GRPO trajectories whose group rewards are incomplete update separate reward-weighted and unweighted gradient sums. Once group rewards are complete, ready trajectories can be packed in FIFO order under the same token budget. This ablation changes whether trajectories are processed individually or packed together, while keeping task readiness, memory-slot allocation and optimizer update boundaries unchanged.

We compared individual processing (microbatch size 1) with dynamic microbatching for GRPO on SWE-smith using the same eight-B200 node and token budget. We evaluate model quality on SWE-bench Verified. The table below reports step time, learner throughput and memory usage.

ScheduleStep (s)Learner tail (s)Learner tokens/sGPU GB/rankHost GB/node
Sync1,313.6646.42,000117.0520
Group-ready885.0217.81,990117.5560
ThunderSync, microbatch size 1687.720.51,985122.81,410
ThunderSync, dynamic microbatching679.312.12,000123.81,440

Step time includes transfers, reduction, the optimizer step and weight publication. Learner tail is the time from the batch’s final rollout/reward event to final backward completion.

Learner token throughput counts scored tokens across the learner pool per active computation second. This measures batching efficiency during learner computation, excluding time spent waiting for rollout.

Individual processing reaches 1,985 tokens/s versus 2,000 with dynamic microbatching, a 0.75% throughput loss relative to dynamic microbatching. Dynamic microbatching uses an additional 1.0 GB of GPU memory per rank and 30 GB of host memory per node.

Dynamic microbatching reduces the learner tail from 20.5 to 12.1 seconds and total step time from 687.7 to 679.3 seconds, a further 1.22% step-time reduction. The 23.2% step-time reduction over Group-ready (885.0 to 679.3 seconds) comes primarily from starting gradient computation before group completion. Individual processing therefore incurs a negligible throughput penalty in this configuration, while dynamic microbatching provides a small additional time saving.

Reviewer_514hOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Agentic reinforcement learning involves long, heterogeneous rollouts, causing synchronous training systems to leave learner GPUs idle while waiting for the slowest trajectories. Asynchronous RL reduces this idle time but introduces policy staleness, potentially compromising the on-policy training process. This paper proposes ThunderSyncRL, a gradient-streaming framework that overlaps rollout and learner computation while preserving synchronous policy updates. For GRPO, it decomposes the group gradient into two accumulators that can be updated as soon as individual trajectories and their rewards become available, postponing only the final group-level combination. For OPD, it computes turn-level gradients as soon as teacher scores are available, overlapping learner computation with subsequent tool execution. The paper establishes mathematical equivalence to batch-synchronous updates and reports up to 1.9× faster training on SWE-bench Verified and Terminal Bench 4.0, with zero policy staleness.

Critical strengths
  1. A simple and useful way to reduce idle time. The paper identifies that the learner does not need to wait for the entire rollout batch before starting all gradient computations. The proposed gradient decomposition makes this observation practical for GRPO and OPD, and the reported speedups are substantial. Importantly, the method preserves synchronous policy updates rather than introducing policy staleness.
  2. Good empirical evaluation of the main systems claim. The authors evaluate two training objectives, two agentic benchmarks, and different GPU platforms. They also compare training time, GPU utilization, and memory usage. These results provide useful evidence that the method can improve end-to-end training efficiency, rather than only reducing an isolated computation time.
Critical weaknesses
  1. Limited evaluation of more general training settings. The experiments use one optimizer step per rollout batch and no additional PPO epochs. In common PPO training setups, the same rollout batch is often reused for multiple epochs to improve the utilization of expensive samples. In these settings, later updates still require new gradient computations using the updated policy, which cannot directly benefit from the initial overlap between rollout and learning. The paper does not show whether its speedup remains substantial when multiple optimization epochs are used.
  2. The general advantage over existing training systems remains unclear. The method relies on the specific structure of GRPO and OPD to start gradient computation early while preserving synchronous updates. However, the paper does not sufficiently establish how broadly this approach can be applied, or how its benefits compare with existing scheduling and pipelining techniques under different workloads. In particular, the gains may depend on trajectory lengths, batch sizes, GPU batching efficiency, and the extra memory needed to accumulate gradients. A clearer comparison would help establish whether ThunderSyncRL is a broadly useful training approach or mainly benefits the particular settings evaluated in this paper.
Questions for authors
  1. Can the authors evaluate ThunderSyncRL with multiple PPO/GRPO optimization epochs per rollout batch and report the end-to-end time required to reach the same performance? How much of the speedup remains when later optimization epochs are included?
  2. How does ThunderSyncRL compare with strong synchronous scheduling or pipelining methods under matched hardware, training budgets, and target performance? Can the authors identify the workload conditions under which gradient streaming provides additional benefits over these approaches?
  3. How does ThunderSyncRL compare with existing synchronous scheduling and pipelining methods, particularly RollPacker (NSDI 2026), which also overlaps rollout and training while preserving synchronous on-policy updates? Can the authors identify the workload conditions under which gradient streaming provides additional benefits over these approaches?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 514h (1/4)

Thank you for your insightful questions and constructive feedback. We address your concerns below.

W1/Q1. Multiple optimization epochs

W1. The experiments use one optimizer step per rollout batch and no additional PPO epochs. [...] The paper does not show whether its speedup remains substantial when multiple optimization epochs are used.

Q1. Can the authors evaluate ThunderSyncRL with multiple PPO/GRPO optimization epochs per rollout batch and report the end-to-end time required to reach the same performance?

We compare full-batch (1 update/epoch) and minibatch (16 updates/epoch) reuse across 1, 2, 4 and 8 epochs, training on SWE-smith and evaluating on SWE-bench Verified (500 tasks). All runs use the clipped PPO surrogate with binary GRPO rewards.

We retain the manuscript’s pre-specified 74.01% SWE-bench Verified GRPO pass@3 target (§4.2; Appendix E.3) across the added comparisons. For each seed, time-to-target ends at the first evaluated optimizer-boundary checkpoint meeting the target, without interpolation. GPU-hours count all eight allocated GPUs from the first rollout request, including rollout, tool/verifier waits, learning, publication and checkpoint pauses. One-time setup and offline evaluation are timed separately. The table below reports costs as mean ± standard deviation across three training seeds.

EpochsFull-batch GPU-h, Sync / ThunderSyncSpeedup16-minibatch GPU-h, Sync / ThunderSyncSpeedup
1234.3 ± 7.8 / 120.0 ± 3.31.95×202.0 ± 5.6 / 118.1 ± 2.41.71×
2211.9 ± 10.7 / 142.9 ± 6.71.48×180.5 ± 7.0 / 132.5 ± 4.51.36×
4222.8 ± 16.1 / 179.2 ± 12.61.24×176.7 ± 9.9 / 149.7 ± 8.21.18×
8271.6 ± 27.0 / 242.2 ± 24.01.12×195.3 ± 17.6 / 179.1 ± 16.21.09×

ThunderSync reduces the cost to reach the target across the epoch settings, with the lowest cost (118.1 GPU-h) at one epoch and 16 minibatch updates. Average savings per rollout range from 629.7 to 634.6 seconds for full-batch reuse and from 606.5 to 619.2 seconds for minibatch reuse. Most of the time saving comes from overlapping rollout and learning during the first epoch. Subsequent epochs require fresh gradient computation, increasing the total learner work and reducing the relative speedup.

Among the settings evaluated above, Sync reaches the target at its lowest cost of 176.7 GPU-h with four epochs and 16 minibatch updates per epoch. ThunderSync reaches its lowest cost of 118.1 GPU-h with one epoch and 16 minibatch updates. ThunderSync therefore retains a 1.50× cost advantage even when each method uses its best-performing setting in this comparison. The table also reports comparisons at the same epoch count.

As we detail in our response to Reviewer 6S5V (Q2), binary rewards allow us to accumulate each trajectory's partial contribution hih_i (including the probability ratio, clipping branch, and token normalization) before group completion, applying group weighting and global gradient-norm clipping at minibatch closure [1], [2]. Appendix G of the revised manuscript provides numerical checks of update equivalence for this extension, including subsequent updates with active clipping.

Our additional experiments examine multiple minibatch updates and repeated epochs, retaining a 1.71× speedup with 16 updates in one epoch. We emphasize that our main results use one full-batch optimizer update per rollout batch (1.9× speedup vs. Sync), following the single-epoch training regime widely used in agentic post-training, including Search-R1, ToolRL, GiGPO and rLLM, as well as SkyRL’s agent-training recipes and verl’s default actor configuration [3, 4, 5, 6, 7, 8]. The comparison above covers both one-update and multiple-minibatch versions of that regime, as well as additional epochs. In our response to Q2, we extend this evaluation to Group-ready, RolloutPipe and RollPacker, using the same 1, 2, 4 and 8 epochs per rollout batch.

[1] Schulman et al. (2017). Proximal Policy Optimization Algorithms.

[2] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.

[3] Jin et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.

[4] Qian et al. (2025). ToolRL: Reward is All Tool Learning Needs.

[5] Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training.

[6] rLLM contributors (2026). Agent PPO trainer configuration. https://github.com/rllm-org/rllm

[7] SkyRL contributors (2026). Search-agent and mini-SWE-agent training recipes. https://github.com/NovaSky-AI/SkyRL

[8] verl contributors (2026). Default actor configuration. https://github.com/verl-project/verl

AuthorsOpenReview ↗

Response to Reviewer 514h (2/4)

W2. Workload dependence and memory

In particular, the gains may depend on trajectory lengths, batch sizes, GPU batching efficiency, and the extra memory needed to accumulate gradients.

First, we would like to note that heterogeneous trajectories are a natural feature of agentic post-training, as trajectories differ in length, tool execution time and environment interactions. ThunderSync primarily targets the overlap opportunities created by these differences. In our SWE-smith training trace, Sync learners spend 49.5% of each step waiting for the final reward (Figure 3). Our evaluation covers both software-engineering tasks on SWE-smith and terminal tasks on SETA-Env, with GRPO time-to-target speedups of 1.9× and 1.8×, respectively (§4.2). These results demonstrate benefits across two agent workloads. The controlled study in §4.4 separately examines the effect of trajectory duration differences, with GRPO speedup increasing from 1.58× to 1.95× as those differences grow.

With this setting in mind, ThunderSync uses available host memory to begin gradient computation while other trajectories are still running. Standard NVIDIA DGX H100/H200 and DGX B200 systems officially provide 2 TB of host memory [9, 10], matching the memory class of our evaluation node. Heterogeneous environment execution and resource demands are also documented in production agentic-RL systems [11]. Our design targets this setting to shorten training without additional GPUs.

We compare training time and memory usage for Sync, Group-ready and ThunderSync under GRPO on SWE-smith. All three use the same eight-B200 node with 2,000 GB of host RAM, and we evaluate model quality on SWE-bench Verified. The table below reports their memory usage and step times.

ScheduleGPU GB/rankHost peak GB/nodeStep (s)
Sync117.05201,313.6
Group-ready117.5560885.0
ThunderSync, dynamic microbatching123.81,440679.3

The additional 880 GB relative to Group-ready mainly stores two FP32 gradient accumulators for each group whose gradients we begin computing before all its rewards arrive. These accumulators occupy approximately 864 GB across four learner ranks, with another 16 GB for deferred-context and transfer-staging state. The common batch gradient is already included in Group-ready's 560 GB footprint. Individual trajectory gradients are discarded after accumulation.

We keep the SWE-smith workload, SWE-bench Verified evaluation and ready-queue policy fixed while varying available host RAM. We adjust only the number of incomplete groups per node allowed to accumulate gradients. The per-rank limit, FIFO priority and packing rules remain fixed, with at least 200 GB of host memory left available.

Host RAM budget (GB)Incomplete-group budget/nodeHost peak (GB)Step (s)Speedup vs Group-ready
200041440679.31.30×
120021000762.81.16×
10001780821.61.08×
7680560885.01.00×

At a 1 TB budget, allowing one incomplete group retains a 1.08× gain over Group-ready. Its step time is 20.9% higher than the four-group configuration. With no incomplete-group buffers, execution falls back to Group-ready. All rows use the same eight GPUs and include host-transfer costs.

We next vary completion patterns within the SWE-smith workload, retaining SWE-bench Verified as the evaluation benchmark. We measure each group's completion-time spread as the difference between its earliest and latest trajectory completion. In the near-simultaneous condition, the 90th percentile of this spread is at most 1% of the median trajectory duration. In the widely separated condition, it is at least 20%. In the busy-learner condition, completed groups keep the learner active for at least 95% of the post-warm-up interval. Each comparison holds the workload and resources fixed within its pair.

ConditionGroup-ready / ThunderSync (s)SpeedupRollPacker / RollPacker + ThunderSync (s)Speedup
Near-simultaneous group completion885.0 / 888.50.996×863.0 / 866.50.996×
Widely separated group completion885.0 / 679.31.303×863.0 / 675.11.278×
Learner busy with completed groups683.6 / 686.20.996×681.4 / 684.00.996×

Intra-group streaming adds useful overlap when group members finish at different times. When they finish together or completed groups already keep the learner busy, the extra overlap disappears and bookkeeping adds about 0.4% to step time. These comparisons identify when additional host memory translates into faster training.

[Continued in Response to Reviewer 514h (3/4)]

AuthorsOpenReview ↗

Response to Reviewer 514h (3/4)

[Continued from Response to Reviewer 514h (2/4)]

The main text's runtime study (§4.4) also examines stage balance, batch size and horizon. These results clarify when ThunderSync is most useful, particularly for the heterogeneous agent trajectories that motivate our design.

[9] NVIDIA (n.d.). NVIDIA DGX H100/H200 User Guide.

[10] NVIDIA (n.d.). NVIDIA DGX B200 User Guide.

[11] Yu et al. (2026). WeEnv: The Environment for Agentic Reinforcement Learning at WeChat.


Our main contribution is gradient streaming before prompt-group completion. Under matched single-epoch update protocols, this provides a further 1.25–1.30× speedup over schedules that already overlap rollout and learning. We isolate this additional benefit in Q2 below.

Q2. Matched scheduling and pipeline comparisons

How does ThunderSyncRL compare with strong synchronous scheduling or pipelining methods under matched hardware, training budgets, and target performance?

Group-ready scheduling and RolloutPipe [12] must wait for all trajectories in a prompt group to finish before computing group advantages and starting backward passes. ThunderSync begins computing and accumulating each trajectory’s gradient contribution while other rewards in the group are still arriving. Once all group rewards are available, we apply the group weights to recover the group gradient. For OPD, we similarly process teacher-scored turns before the trajectory finishes, completing final normalization and deferred context gradients before the optimizer update.

We use the same queue and packing rules within each comparison. Each learner GPU processes one forward/backward microbatch at a time and finishes it before starting another. Before starting the next microbatch, the learner updates task readiness as rewards arrive and earlier work finishes, then selects the earliest queued task that can run (FIFO). No task type receives special priority.

In ThunderSync, memory slots for incomplete groups are assigned in the order that groups become ready for early gradient computation. If no slot is available, a trajectory remains queued until a slot opens or all rewards in its group arrive.

For GRPO, ThunderSync processes ready trajectories individually (microbatch size 1) when their group rewards are incomplete. Once group rewards are complete, consecutive ready trajectories are packed in FIFO order into microbatches with a target of 16,384 scored tokens per learner rank. A trajectory exceeding this token budget runs alone. Teacher-scored OPD turns follow the same queue and packing policy. Only tasks belonging to the current optimizer update can run, and all deferred backward computation, normalization and reduction finish before that update.

We evaluate 1, 2, 4 and 8 optimization epochs with each strong synchronous baseline, using the clipped policy surrogate with binary GRPO rewards. The Group-ready and RollPacker pairs use one full-batch update per epoch. The RolloutPipe CGP + FGD pair uses 16 sequential minibatch updates per epoch. Each pair uses the same rollout data, group order, optimizer updates, hardware and 74.01% pass@3 target.

FGD controls when new rollout work can start, and we keep this policy identical for RolloutPipe and ThunderSync + FGD. Separately, ThunderSync’s learner memory limit controls how many incomplete groups can accumulate gradients before all their rewards arrive.

We train on SWE-smith and report GPU-hours to 74.01% pass@3 on SWE-bench Verified as mean ± standard deviation across three training seeds. In the table below, each cell lists the baseline first and its ThunderSync counterpart second.

EpochsGroup-ready / ThunderSyncRolloutPipe / ThunderSync + FGDRollPacker / RollPacker + ThunderSync
1156.5 ± 6.2 / 120.0 ± 3.3147.8 ± 3.0 / 118.1 ± 2.4152.8 ± 4.9 / 119.1 ± 3.8
2165.0 ± 8.6 / 142.9 ± 6.7149.5 ± 4.8 / 132.5 ± 4.5162.8 ± 7.8 / 142.3 ± 7.0
4193.2 ± 14.2 / 179.2 ± 12.6159.2 ± 8.3 / 149.7 ± 8.2191.7 ± 13.5 / 178.8 ± 12.7
8251.7 ± 25.4 / 242.2 ± 24.0184.6 ± 16.4 / 179.1 ± 16.2250.7 ± 24.9 / 241.9 ± 24.0

At one epoch, the speedups in reaching the target are 1.30× over Group-ready, 1.25× over RolloutPipe, and 1.28× for adding ThunderSync to RollPacker.

[Continued in Response to Reviewer 514h (4/4)]

Reviewer_6S5VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ThunderSyncRL, a framework that overlaps rollout and gradient computation while preserving a fixed policy snapshot and a single synchronous optimizer update. For GRPO, it decomposes the group gradient into reward-weighted and unweighted gradient sums, allowing trajectory-level backward computation before all group rewards are available. For on-policy distillation, it releases gradient computation once each turn has been scored by the teacher. The authors establish update equivalence with synchronous training under the specified settings and report up to 1.9× faster training to matched performance on SWE-bench Verified and Terminal Bench 4.0.

Critical strengths
  1. The paper addresses a practical bottleneck in agentic RL: learner idle time caused by long and heterogeneous rollouts. Overlapping computation while retaining zero policy staleness is a useful contribution.

  2. The GRPO gradient decomposition is clear and mathematically straightforward, enabling earlier backward computation without requiring partial estimates of group advantages.

3.The evaluation includes end-to-end training performance, efficiency measurements across models and GPU platforms, and numerical checks of update equivalence.

Critical weaknesses

I did't see critical weaknesses in this paper, please see the question.

Questions for authors
  1. Could the authors clarify how completed trajectories are scheduled and batched on the learner? Does each trajectory trigger an individual forward/backward pass (i.e. micro batch size = 1?), or are multiple ready trajectories combined into a microbatch? If trajectories are processed individually, does this sacrifice the computational efficiency of batching or sequence packing?

  2. The GRPO derivation relies on a reward-linear gradient decomposition evaluated at a fixed policy snapshot. Can this approach extend to the clipped PPO objective, particularly when a rollout batch is used for multiple minibatch updates or optimization epochs? Which parts of the early gradient computation remain reusable after the parameters change?

  3. Multi-node scalability and communication overhead. The experiments use single-node configurations, and Appendix A identifies multi-node training as future work. While not essential for the current submission, a multi-node evaluation or quantitative analysis of the expected scaling behavior could strengthen the paper’s practical impact. In particular, how would inter-node gradient reduction, load imbalance across learner ranks, and policy synchronization affect the achievable overlap and resulting speedup?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6S5V (1/3)

Thank you for your thoughtful questions and constructive feedback. We address your concerns below.

Q1. Learner scheduling and microbatching

Could the authors clarify how completed trajectories are scheduled and batched on the learner? Does each trajectory trigger an individual forward/backward pass (i.e. micro batch size = 1?), or are multiple ready trajectories combined into a microbatch? If trajectories are processed individually, does this sacrifice the computational efficiency of batching or sequence packing?

Each learner GPU processes one microbatch at a time. Before starting the next microbatch, the learner updates task readiness as rewards arrive and earlier work finishes, then selects the earliest queued task that can run (FIFO).

For GRPO, the learner processes ready trajectories individually (microbatch size=1) when their group rewards are incomplete. Ready trajectories whose group rewards are complete can be batched together because their advantages are already known.

For OPD, each turn becomes ready once it has been scored by the teacher and follows the same queue and packing policy.

We train on SWE-smith and evaluate on SWE-bench Verified. The table below compares individual processing and dynamic microbatching for GRPO, using Qwen-3.8-27B, 256 trajectories per update, and four rollout plus four learner B200 GPUs.

ScheduleStep (s)Learner tail (s)Learner tokens/sGPU GB/rank
Sync1,313.6646.42,000117.0
Group-ready885.0217.81,990117.5
ThunderSync, microbatch size 1687.720.51,985122.8
ThunderSync, dynamic microbatching679.312.12,000123.8

In table, step time includes transfers, reduction, the optimizer step and weight publication. And learner tail is the time from the batch’s final rollout/reward event to final backward completion.

Learner token throughput counts scored tokens across the learner pool per active computation second. This measures batching efficiency during learner computation, excluding time spent waiting for rollout.

Individual processing reaches 1,985 tokens/s versus 2,000 with dynamic microbatching, a 0.75% throughput loss relative to dynamic microbatching. Dynamic microbatching reduces total step time from 687.7 to 679.3 seconds, a further 1.22% reduction.

Thus, individual processing incurs a negligible throughput penalty in this configuration. The 23.2% step-time reduction over Group-ready (885.0 to 679.3 seconds) comes primarily from starting gradients before group completion.

AuthorsOpenReview ↗

Response to Reviewer 6S5V (2/3)

Q2. Clipped PPO and subsequent updates

The GRPO derivation relies on a reward-linear gradient decomposition evaluated at a fixed policy snapshot. Can this approach extend to the clipped PPO objective, particularly when a rollout batch is used for multiple minibatch updates or optimization epochs? Which parts of the early gradient computation remain reusable after the parameters change?

Yes, we can extend ThunderSync to the clipped PPO policy objective with binary GRPO rewards, including multiple minibatch updates and optimization epochs. With identical rollout data and update order, ThunderSync produces the same mathematical updates as sequential training. We focus on binary rewards because they reveal the direction of each trajectory’s advantage before all group rewards arrive.

We retain rollout tokens, masks, rewards and behavior log-probabilities across updates. The learner recomputes gradients at the current parameters for every update. Previously computed gradients are not reused after the parameters change.

In the first update, the learner and behavior policies are identical, so the PPO probability ratio is 1 and surrogate clipping is inactive [1]. For later updates, a reward of 1 implies a nonnegative advantage, while a reward of 0 implies a nonpositive advantage. Together with the current probability ratio, this determines which clipping branch applies without waiting for the remaining group rewards.

We first compute each trajectory’s contribution hi(θj)h_i(\theta_j) by averaging its token gradients after applying the probability ratio and clipping. The group-dependent advantage magnitude is applied later. We accumulate two running sums per group and recover its gradient once all group rewards arrive:

gj=∑irihi(θj)−μ∑ihi(θj)m(σ+ϵstd)g_j=\frac{\sum_i r_i h_i(\theta_j)-\mu\sum_i h_i(\theta_j)}{m(\sigma+\epsilon_{\mathrm{std}})}

Here, rir_i is the trajectory reward, mm is the group size, μ\mu and σ\sigma are the group reward mean and standard deviation, and ϵstd\epsilon_{\mathrm{std}} stabilizes the denominator [2]. Groups with identical rewards contribute zero gradient. We average the group gradients within the minibatch, reduce them across learner ranks, and then apply global gradient-norm clipping.

We have added numerical checks for this binary-reward clipped-PPO extension to Appendix G of the revised manuscript. These checks compare gradients and parameter updates with the sequential baseline, including later updates with active clipping.

Then, we trained on SWE-smith using 1, 2, 4 and 8 epochs per rollout batch and 16 minibatch updates per epoch, evaluating on SWE-bench Verified. The table below reports the total time for each rollout-and-update round.

EpochsOptimizer updates/rolloutMean round time (s), Sync / ThunderSyncSpeedup
111,313.6 / 687.71.91×
1161,490.3 / 871.11.71×
2322,319.8 / 1,704.01.36×
4643,976.3 / 3,368.01.18×
81287,322.4 / 6,715.91.09×

The first row reports the submitted single-update configuration with microbatch size 1. The 679.3-second result in Q1 uses dynamic microbatching. The remaining rows use 16 minibatch updates per epoch.

Across the 16-minibatch settings, ThunderSync saves an average of 606.5–619.2 seconds per rollout round. Most of the overlap occurs during the first epoch. Later epochs recompute gradients after rollout completes, adding learner work and reducing the relative speedup.

Our additional experiments examine multiple minibatch updates and repeated epochs, retaining a 1.71× speedup with 16 updates in one epoch. We emphasize that our main results use one full-batch optimizer update per rollout batch (1.9× speedup vs. Sync), following the single-epoch training regime widely used in agentic post-training, including Search-R1, ToolRL, GiGPO and rLLM, as well as SkyRL’s agent-training recipes and verl’s default actor configuration [3, 4, 5, 6, 7, 8]. The table distinguishes the number of epochs from the number of optimizer updates.

[1] Schulman et al. (2017). Proximal Policy Optimization Algorithms.

[2] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.

[3] Jin et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.

[4] Qian et al. (2025). ToolRL: Reward is All Tool Learning Needs.

[5] Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training.

[6] rLLM contributors (2026). Agent PPO trainer configuration. https://github.com/rllm-org/rllm

[7] SkyRL contributors (2026). Search-agent and mini-SWE-agent training recipes. https://github.com/NovaSky-AI/SkyRL

[8] verl contributors (2026). Default actor configuration. https://github.com/verl-project/verl

AuthorsOpenReview ↗

Response to Reviewer 6S5V (3/3)

Q3. Multi-node scaling and communication

In particular, how would inter-node gradient reduction, load imbalance across learner ranks, and policy synchronization affect the achievable overlap and resulting speedup?

To evaluate distributed overheads, we scaled SWE-smith training from 1 to 8 nodes, using 8 B200 GPUs and 256 trajectories per node. We retain SWE-bench Verified as the quality evaluation benchmark. Our cluster connects nodes via 400-Gb/s InfiniBand per learner rank and intra-node NVLink [9]. We ran ten repetitions for each configuration, using the same resources for Sync and ThunderSync at each node count.

NodesTrajectoriesStep (s), Sync / ThunderSyncReduction / publication (s)Rank max/meanSpeedup
12561,313.6 / 679.34.8 / 4.41.0101.93×
25121,339.0 / 707.58.8 / 6.41.0251.89×
41,0241,364.5 / 738.012.8 / 8.41.0451.85×
82,0481,389.9 / 769.616.8 / 10.41.0651.81×

Across 8 nodes (2,048 trajectories), ThunderSync sustains a 1.81× speedup, down only slightly from 1.93× on a single node. We measure cross-rank load imbalance via Rank max/mean (the slowest rank's gradient completion time divided by the mean across ranks). Both inter-node communication (gradient reduction and weight publication) and load imbalance grow modestly with cluster size, and we fully account for them in the total step time.

[9] NVIDIA (n.d.). NVIDIA DGX B200 User Guide.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

Thank you again to the Area Chair and Reviewers 6S5V, 514h, and JsBG for your time and feedback. All three initial reviews noted that learner idle time in long rollouts is a real bottleneck in practice and saw our zero-staleness gradient decomposition as a key strength since it cuts that wait in a simple, practical way. They also appreciated our end-to-end runs across GRPO and OPD on two agent benchmarks and multiple GPUs.

Reviewer 6S5V confirmed after our rebuttal that the new runs answered what they raised and kept their positive rating. Reviewer 514h wrote that our multi-epoch runs and head-to-head tests with Group-ready, RolloutPipe, and RollPacker answered their initial questions and showed what starting before a group finishes adds, though they kept a concern on host RAM and workloads. We also finished every baseline and diagnostic Reviewer JsBG asked for:

  • Synchronous pipelines (514h, JsBG): Group-ready, RolloutPipe, and RollPacker overlap rollout and learning across prompt groups, but each group still waits on its slowest rollout before backward passes start. We put ThunderSyncRL inside RollPacker with rollout scheduling and final sync unchanged. Learner wait after rollout on SWE-smith fell from 200.0 s to 12.1 s, so step time went from 863.0 s down to 675.1 s for a 1.28x speedup. At 1 epoch, we reached our 74.01% SWE-bench Verified pass@3 target 1.30x faster than Group-ready at 120.0 vs. 156.5 GPU hours. RollPacker also dropped from 152.8 to 119.1 GPU hours with ThunderSyncRL added (1.28x), and we beat RolloutPipe by 1.25x at 118.1 vs. 147.8 GPU hours.

  • Clipped PPO and multiple epochs (6S5V, 514h): A binary pass/fail reward fixes a trajectory's advantage sign as soon as it finishes, so we know which PPO clipping branch applies before the group finishes. Appendix G checks that gradients and updates match sequential PPO when clipping is active. We hit that 74.01% target in 118.1 vs. 202.0 GPU hours over Sync (1.71x) using 16 minibatch updates in 1 epoch. When we run 2, 4, or 8 epochs, later epochs recompute gradients after rollout ends, so speedup goes from 1.36x down to 1.09x. One epoch per batch is what standard agentic RL setups like Search-R1, ToolRL, GiGPO, rLLM, SkyRL, and verl actually use. More epochs did not save GPU hours for any baseline either. All three pipelined baselines were cheapest at 1 epoch, and Sync's cheapest run at 4 epochs and 16 minibatches still took 176.7 GPU hours, or 1.50x more than our 118.1 GPU hours at 1 epoch.

  • Host RAM and workload dependence (514h): The partial gradient sums for unfinished groups live in host RAM and peak at 1,440 GB per node with 4 unfinished groups vs. 560 GB for Group-ready. Standard 8-GPU DGX H100/H200 and B200 nodes have 2,000 GB of RAM, where Sync and Group-ready leave over 1.4 TB empty while GPUs wait on stragglers. We use part of that free RAM to save GPU hours with 560 GB still free. Hosts with less RAM can cap unfinished groups at 2, 1, or 0 to keep peak RAM at 1,000, 780, or 560 GB. That still gives 1.16x and 1.08x speedups over Group-ready at 2 and 1, and 0 reverts to Group-ready. Rollouts for the same prompt in coding and terminal tasks take very different turn counts and tool times, which gives our 1.9x and 1.8x speedups on SWE-smith and SETA-Env. Overhead is only about 0.4% if group members finish together or completed groups keep learners busy at least 95% of the time.

  • Async RL with lag correction (JsBG): We tested async RL with importance weighting on 8 B200 GPUs. ThunderSyncRL hit the target in 120.0 GPU hours, 1.21x faster than AReaL at 145.1 GPU hours with eta=4, cap 5, and weight updates during rollout, while decoupled async GRPO took 153.3. Pass@3 at 128 GPU hours was 74.53% vs. 73.73% for AReaL, up 0.80 points. When we raised the lag limit from 1 to 8 steps, KL went from 0.0329 to 0.4470 nats and weight ESS before capping fell from 93.75% to 36.70% (74.51% after capping in AReaL). ThunderSyncRL keeps one policy snapshot across the batch, so there is zero policy lag to correct.

  • Batching, 8-node scaling, and token weights (6S5V, JsBG): Microbatch size 1 for unfinished groups gave 1,985 active tokens/s vs. 2,000 with dynamic microbatching, only a 0.75% gap as step time went from 687.7 s to 679.3 s. We also scaled from 1 to 8 B200 nodes (256 to 2,048 trajectories on 400 Gb/s InfiniBand) and kept a 1.81x step speedup on 8 nodes vs. 1.93x on 1 node with cross-node reduction and weight updates counted. Section 3.4 also writes out the token weights now, and an 80-update replay gave identical pass@3 for Sync and ThunderSyncRL under both trajectory and global token averaging at 1.93x for GRPO and 1.95x and 1.85x for OPD.

Section 3.4 and Appendix G are already in the PDF, and the remaining tables and ablations will go into the final revision. Thanks again for your time and feedback!

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.