- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ThunderSyncRL, a framework that overlaps rollout and gradient computation while preserving a synchronous policy update. For GRPO, the method computes trajectory-level score gradients before all group rewards are available and maintains reward-weighted and unweighted gradient sums. These statistics recover the group gradient once its rewards are known. For on-policy distillation, teacher-scored turns release gradient computation before the trajectory finishes. The paper reports up to approximately 1.9× acceleration over synchronous training, together with improved fixed-budget performance relative to its asynchronous baselines.
Critical strengths
-
The paper addresses a practical bottleneck in agent training. Long trajectories and expensive environment interactions can leave learner resources idle, making earlier gradient computation useful.
-
The GRPO decomposition is simple and appealing. It separates trajectory-local gradient computation from group-dependent scalar coefficients without changing the specified batch update. Keeping parameters fixed throughout the logical batch also avoids policy staleness.
-
The evaluation covers both GRPO and OPD, two agent benchmarks, multiple model configurations, and two GPU types. The numerical update-equivalence checks and analysis of rollout–learner balance provide useful support for the approach.
Critical weaknesses
-
The asynchronous comparison does not establish an advantage over appropriately corrected asynchronous training. Appendix D.4 explicitly states that the one-step Async baseline learns from stale trajectories without an additional behavior-policy importance correction. The authors explain that this holds the surrogate fixed across schedules, which is a useful controlled experiment. However, it leaves open whether the reported accuracy gap persists against an asynchronous algorithm designed to handle stale data.
-
Importance sampling does not guarantee identical finite-sample learning behavior: long-horizon weights can have high variance, and practical clipping can introduce bias. Nevertheless, the paper does not investigate whether these effects explain its results. A baseline with an established staleness correction, such as the AReaL-style objective discussed in the manuscript, would make the comparison more convincing. A staleness sweep and diagnostics of policy divergence and importance weights would also help explain when zero staleness matters. Currently, the evidence supports superiority over the specified async baselines, while the broader explanation remains incomplete.
-
Learner scheduling and batching are insufficiently specified. The single-lane scheduling model and Appendix I’s pseudocode suggest sequential processing of ready gradient sources within each learner rank. However, the implementation description does not clearly state whether waiting sources are dynamically batched or whether independent computation graphs can execute concurrently.
-
This distinction matters because samples may arrive during an ongoing backward pass. Processing queued samples individually maximizes scheduling granularity, but batching them could improve GPU throughput. For GRPO, preserving both reward-weighted and unweighted gradient statistics also complicates ordinary batching. Please specify the actual learner batching policy and report an ablation comparing individual-source processing with dynamic microbatching, ideally with learner throughput, memory usage, and end-to-end step time. The fixed-service-time scheduling analysis alone does not resolve this tradeoff.
-
A group-ready synchronous baseline would better isolate the contribution of the GRPO decomposition. Sync waits for the entire logical batch, whereas ThunderSyncRL can start before a group finishes. An intermediate baseline could start ordinary advantage-weighted backward computation as soon as one complete group becomes available, while retaining the same final optimizer barrier. Comparing against this baseline would distinguish the benefit of trajectory-level gradient streaming from the more general benefit of overlapping completed groups with unfinished rollout. The paper discusses group-ready scheduling but does not include it among the reported experimental baselines.
-
Equation (4) introduces the token weights without specifying their values in the main text. Appendix D.1 defines (w_{i,t}=1/(32L_i)), so GRPO does include per-trajectory token normalization. This definition should appear alongside the equation. The authors should also explain why GRPO uses a mean of trajectory-normalized contributions while OPD uses a global token mean. These are different weighting choices, although neither invalidates the streaming construction.
Questions for authors
Please refer to the weaknesses.
7 comments
Response to Reviewer JsBG (2/5)
W2. Weight variance, truncation bias and staleness diagnostics
Importance sampling does not guarantee identical finite-sample learning behavior: long-horizon weights can have high variance, and practical clipping can introduce bias.
We agree that importance weighting alone does not guarantee the same learning behavior with finite samples, particularly when weights become uneven or are capped. Our response to W1 compares ThunderSync with corrected asynchronous baselines. Here, we examine how policy staleness relates to correction weights and learning performance.
We profile a 10,000-token pre-update window from SWE-smith training for each configuration and evaluate pass@3 on SWE-bench Verified at 128 GPU-hours. The first four rows vary the allowed policy lag while keeping the generation policy fixed within each trajectory. The final row reports the AReaL configuration used in W1, which allows weight updates during ongoing rollouts.
| Execution / lag setting | Mean token lag | Corrected pass@3 at 128 GPU-h (%) | Raw KL (nats) | Raw weight p95 | ESS before cap | ESS after cap | Above cap 5 |
|---|---|---|---|---|---|---|---|
| Fixed policy / 1 | 1 | 73.33 ± 0.42 | 0.0329 | 1.83 | 93.75% | 93.75% | 0.0% |
| Fixed policy / 2 | 1.5 | 73.13 ± 0.31 | 0.0798 | 2.67 | 82.19% | 82.19% | 0.0% |
| Fixed policy / 4 | 2.5 | 72.80 ± 0.40 | 0.1917 | 4.00 | 60.68% | 60.68% | 0.0% |
| Fixed policy / 8 | 4.5 | 72.40 ± 0.40 | 0.4470 | 6.00 | 36.70% | 42.79% | 6.0% |
| AReaL / η=4 | 2.15 | 73.73 ± 0.12 | 0.1382 | 2.03 | 69.98% | 74.51% | 0.35% |
Mean token lag measures the observed difference between the learner and generation policy versions. KL measures the difference between their probability distributions. Raw weight p95 is the 95th percentile of the importance weights before capping, and “Above cap 5” reports the fraction exceeding 5.
Effective sample size (ESS) summarizes how evenly the correction weights are distributed. We report it as a percentage of the profiled tokens:
Equal weights give 100%, while weights concentrated on a smaller number of tokens give a lower value. We compute this measure before and after applying the cap. All tokens remain in the calculation, so a lower ESS does not mean tokens were discarded or represent a count of independent trajectories.
In the fixed-policy sweep, larger lag coincides with greater policy divergence, more uneven weights and lower corrected pass@3. At a lag limit of 8, capping raises ESS from 36.70% to 42.79%. However, capping also changes the estimator and can introduce bias. For the AReaL configuration in W1, ESS rises from 69.98% to 74.51% after capping, with 0.35% of weights exceeding 5.
ThunderSync achieves 74.53 ± 0.31% pass@3 at the same 128 GPU-h budget, compared with AReaL’s 73.73 ± 0.12%, a 0.80-percentage-point advantage. In ThunderSync’s default single-update setting, rollout generation and gradient computation use the same policy snapshot. Its behavior-correction weights are therefore all 1, giving a weight-based ESS of 100%, compared with AReaL’s 74.51% after capping.
These observations are consistent with a finite-sample cost of staleness, but they do not isolate how much of the performance gap comes from weight variance or truncation bias. ThunderSync avoids uneven weighting caused by staleness correction in this setting. Other sources of gradient variance, including sampled trajectories and rewards, remain.
Response to Reviewer JsBG (3/5)
W3. Learner scheduling and dynamic microbatching
However, the implementation description does not clearly state whether waiting sources are dynamically batched or whether independent computation graphs can execute concurrently.
Each learner GPU processes one microbatch at a time and completes its forward and backward passes before starting another. Before starting the next microbatch, the learner updates task readiness as rewards arrive and earlier work finishes, then selects the earliest queued task that can run (FIFO). No task type receives special priority.
For GRPO, we allow up to four incomplete groups per node to accumulate gradients, with at most one group per learner rank. Available slots are assigned in the order that groups become ready for early gradient computation. If no slot is available, a trajectory remains queued until a slot opens or all rewards in its group arrive.
The learner processes ready trajectories individually (microbatch size 1) when their group rewards are incomplete. Once group rewards are complete, their advantages are known, so consecutive ready trajectories can be packed together in FIFO order. Each microbatch has a target budget of 16,384 scored tokens per learner rank. A trajectory exceeding this budget runs alone.
For OPD, each turn becomes ready once it has been scored by the teacher and follows the same queue and packing policy.
Only tasks belonging to the current optimizer update can run. All deferred backward computation, normalization and reduction finish before the optimizer updates the parameters.
W4. Individual-source processing versus dynamic microbatching
Processing queued samples individually maximizes scheduling granularity, but batching them could improve GPU throughput.
Under the ready-queue policy in W3, GRPO trajectories whose group rewards are incomplete update separate reward-weighted and unweighted gradient sums. Once group rewards are complete, ready trajectories can be packed in FIFO order under the same token budget. This ablation changes whether trajectories are processed individually or packed together, while keeping task readiness, memory-slot allocation and optimizer update boundaries unchanged.
We compared individual processing (microbatch size 1) with dynamic microbatching for GRPO on SWE-smith using the same eight-B200 node and token budget. We evaluate model quality on SWE-bench Verified. The table below reports step time, learner throughput and memory usage.
| Schedule | Step (s) | Learner tail (s) | Learner tokens/s | GPU GB/rank | Host GB/node |
|---|---|---|---|---|---|
| Sync | 1,313.6 | 646.4 | 2,000 | 117.0 | 520 |
| Group-ready | 885.0 | 217.8 | 1,990 | 117.5 | 560 |
| ThunderSync, microbatch size 1 | 687.7 | 20.5 | 1,985 | 122.8 | 1,410 |
| ThunderSync, dynamic microbatching | 679.3 | 12.1 | 2,000 | 123.8 | 1,440 |
Step time includes transfers, reduction, the optimizer step and weight publication. Learner tail is the time from the batch’s final rollout/reward event to final backward completion.
Learner token throughput counts scored tokens across the learner pool per active computation second. This measures batching efficiency during learner computation, excluding time spent waiting for rollout.
Individual processing reaches 1,985 tokens/s versus 2,000 with dynamic microbatching, a 0.75% throughput loss relative to dynamic microbatching. Dynamic microbatching uses an additional 1.0 GB of GPU memory per rank and 30 GB of host memory per node.
Dynamic microbatching reduces the learner tail from 20.5 to 12.1 seconds and total step time from 687.7 to 679.3 seconds, a further 1.22% step-time reduction. The 23.2% step-time reduction over Group-ready (885.0 to 679.3 seconds) comes primarily from starting gradient computation before group completion. Individual processing therefore incurs a negligible throughput penalty in this configuration, while dynamic microbatching provides a small additional time saving.
Response to Reviewer JsBG (1/5)
Thank you for your detailed questions and constructive suggestions. We address each of your questions and feedback below.
W1. Comparison with corrected asynchronous training
We evaluated ThunderSync against importance-weight-corrected asynchronous training on SWE-bench Verified after training on SWE-smith. AReaL [1], configured with the same GRPO objective, performs best among the corrected baselines we evaluated. Compared with AReaL, ThunderSync reaches the 74.01% pass@3 target 1.21× faster and achieves a 0.80-percentage-point higher pass@3 under a fixed 128 GPU-h budget.
Following AReaL [1], we correct for the difference between the policy that generated each token and the current learner policy. For the same token and context, we divide its probability under the current learner snapshot (πprox) by its probability under the generation policy (πbehav). We cap this ratio at 5 and use it to weight the token’s GRPO contribution. The weight is held constant during backpropagation:
wˉ=min(5,πbehavπprox)
We set the controller’s staleness limit to η=4, following AReaL’s coding configuration [1]. Both corrected baselines use the same cap of 5 to limit extreme weights without discarding tokens. These settings are fixed before test evaluation. We distinguish the controller setting η from the observed token ages reported in our response to W2.
During generation, we store each scored token’s sampled log-probability and policy-version ID. When weights are updated during an ongoing rollout, subsequent tokens receive the new policy-version ID, while previously recorded probabilities remain unchanged. We rebuild the KV cache after the weight refresh. Before each learner update, we evaluate πprox at the current learner snapshot. Each token’s correction therefore uses the policy that actually generated it, including when the policy changes within a trajectory.
We compare two corrected asynchronous baselines with ThunderSync using Qwen-3.8-27B on eight B200 GPUs. All methods use the same GRPO advantage calculation, trajectory-mean normalization and one full-batch optimizer update per training batch. Async + decoupled GRPO allows a maximum policy lag of one update. AReaL uses the controller staleness limit η=4 and allows weight updates during ongoing rollouts.
We retain the manuscript’s pre-specified 74.01% SWE-bench Verified GRPO pass@3 target (§4.2; Appendix E.3) across the added comparisons. For each seed, time-to-target ends at the first evaluated optimizer-boundary checkpoint meeting the target, without interpolation. GPU-hours count all eight allocated GPUs from the first rollout request, including rollout, tool/verifier waits, learning, publication and checkpoint pauses. One-time setup and offline evaluation are timed separately. The table below reports results as mean ± standard deviation across three training seeds.
These comparisons show that ThunderSync retains an advantage in both time to target and fixed-budget performance against the evaluated asynchronous baselines with staleness correction enabled.
[1] Fu et al. (2026). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning.