- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a technique for synchronizing model weights between training and rollout workers, named AREAL-DTE. It accelerates weight synchronization by transferring only changed BF16 weights, based on the observation that few BF16 weight elements change between consecutive policy versions. Specifically, it reconstructs previous weights by inverting AdamW updates, detects BF16 weight changes during layout conversion, and directly transfers sparse updates to inference workers. The evaluation shows that AREAL-DTE achieves meaningful improvements in end-to-end RL training throughput, synchronization latency, and memory efficiency compared to ByteCheckpoint and PULSE.
Critical strengths
-
The paper shows a valuable insight that although FP32 weight updates are dense, the BF16 weight changes observed by inference workers are sparse.
-
The evaluation shows performance improvements over existing methods.
Critical weaknesses
-
The End-to-End Speedup lacks some important experimental settings, such as per-iteration latency breakdown, whether rollout and training are asynchronous, training and rollout batch sizes, and rollout length, raising concerns about the soundness of the experiments. Moreover, all evaluated tasks are non-agentic, making it unclear whether such large speedups can still be achieved on long-horizon agentic tasks.
-
BF16 weight sparsity lacks sensitivity analysis, and the theoretical analysis relies on assumptions that may not hold in practice. Specifically, the derivation of Eq. (25) in Appendix A assumes that the relative positions of weights within their BF16 rounding cells are uniformly distributed.
Questions for authors
-
Could the authors provide more details about the experimental settings, including the per-iteration latency breakdown of rollout, training, and weight synchronization, whether rollout and training are asynchronous, training and rollout batch sizes, and rollout length?
-
Could the authors evaluate whether the reported end-to-end speedups can still be achieved on long-horizon agentic tasks and larger models?
-
Could the authors provide a sensitivity analysis of BF16 weight sparsity under different training settings, such as learning rates and batch sizes?
6 comments
Response to W1/Q2: Multi-turn tool-use evaluation and end-to-end performance
W1 / Q2: Multi-turn tool-use workloads and larger models
Thank you for the suggestion. We add an RL experiment with a Hermes agent on DAPO-Math using Qwen3-30B-A3B, covering multi-turn reasoning, Python tool calls, execution feedback, and subsequent reasoning before submitting a final answer. On this real tool-use workload, AReaL-DTE achieves 227.199 tokens/s, corresponding to and the throughput of PULSE and ByteCheckpoint, respectively.
1. Experimental settings and interaction lengths: training on complete multi-turn trajectories, with weight synchronization after every update.
The experiment uses PPO, a Gbps link, and synchronization at every step (). Each update uses 16 task instances, with one complete interaction trajectory per instance. After all 16 trajectories in the current batch are completed and used for training, weights are synchronized, and the next batch is released only after both inference replicas acknowledge the new version. AReaL-DTE, PULSE, and ByteCheckpoint are compared using the same experimental configuration.
Each model response can generate up to 8,192 new tokens, with a context limit of 65,536 tokens. Each task allows up to 16 Python tool executions, each lasting at most 15 seconds. The 8,192-token limit applies to each response, not to the entire multi-turn trajectory; the limit of 16 tool executions does not cap model interactions at 16. The observed interaction lengths are as follows:
| Statistic | Measured value | Statistic | Measured value |
|---|---|---|---|
| Mean model interactions per trajectory | 3.328 | Mean Python calls per trajectory | 2.245 |
| P90 model interactions | 8 | P90 Python calls | 7 |
| Maximum model interactions in completed trajectories | 22 | Maximum total newly generated tokens in completed trajectories | 46,176 |
These trajectories include actual tool execution and feedback, demonstrating multi-turn interaction beyond single-response tasks.
2. End-to-end performance: throughput gains persist with multi-turn tool interaction.
Performance measurements exclude the first step as warm-up and cover steps 2–40, for a total of 39 measured steps. The following table reports per-step times and end-to-end throughput, covering complete trajectory generation and waiting, training, and weight synchronization.
| Method | Non-sync total (s/step) | Trajectory-batch wait (s/step) | Training (s/step) | Weight synchronization (s/step) | Throughput (tokens/s) |
|---|---|---|---|---|---|
| AReaL-DTE | 160.664 | 148.494 | 11.605 | 475.239 | 227.199 |
| PULSE | 160.612 | 148.681 | 11.350 | 544.804 | 205.157 |
| ByteCheckpoint | 162.303 | 149.984 | 11.743 | 8,736.247 | 16.297 |
Note: Trajectory-batch wait and training are already included in the non-sync total and should not be added again. The full step time is the non-sync total plus weight synchronization time.
AReaL-DTE improves end-to-end throughput over PULSE by approximately 10.7% and achieves ByteCheckpoint's throughput. Non-sync times are similar across the three methods, while AReaL-DTE reduces weight synchronization time from PULSE's 544.804 seconds to 475.239 seconds. This shows that reducing weight synchronization overhead continues to translate into end-to-end performance gains in RL workloads involving multi-turn reasoning and actual tool calls.
Regarding model size, the current evaluation covers Qwen3-8B and Qwen3-30B-A3B. This additional experiment extends the evaluation of Qwen3-30B-A3B to multi-turn tool-use tasks; it does not add measurements on models beyond that size.
Response to W2/Q3 (1/2): Learning-rate and batch-size sensitivity
W2 / Q3: Effects of training settings on BF16 change sparsity and empirical evidence for the uniform-position assumption
Thank you for this suggestion. We add two complementary sources of evidence: six sensitivity experiments varying learning rate and batch size (Appendix D.6, Tables 12–13), and measurements of rounding-position distributions using true weights (Appendix A, Figure 5). We address sensitivity to training settings and the uniform-position assumption separately below.
1. Sensitivity experiments: change rates increase with learning rate, while halving or doubling batch size preserves high sparsity.
The experiments use Qwen3-8B / GSM8K, GRPO, AdamW, and BF16 inference weights, with 40 training steps per setting. The baseline learning rate is , the global training batch size is 16 rollout response sequences (8 prompts with 2 sampled responses each), and weight decay is . Each variant changes only the learning rate or batch size, keeping the remaining settings fixed and synchronizing after every parameter update (). Within each setting, AReaL-DTE and PULSE are compared using the same experimental configuration.
The following table corresponds to Appendix D.6, Table 12. The true BF16 change rate is obtained by comparing the true pre-update and post-update BF16 weights bit for bit, using all model weight elements as the denominator. Percentages are calculated from aggregated counts.
| Setting | True BF16 change rate | Setting | True BF16 change rate |
|---|---|---|---|
| LR | 0.627989% | Baseline: LR , batch | 1.536342% |
| LR | 2.680988% | Batch | 1.411514% |
| LR | 3.414324% | Batch | 1.472879% |
As the learning rate increases from to , the change rate rises from 0.627989% to 3.414324%. At the baseline learning rate, halving or doubling batch size gives change rates of 1.411514% and 1.472879%, respectively, close to the baseline. All six settings record zero missed updates and zero receiver bitwise mismatches at every check.
The original paper's observation of a change rate below 2% comes from the preliminary experiments with learning rates of and . A larger learning rate increases the update magnitude and can cause policy drift to exceed the prescribed trajectory KL budget. Appendix A, Theorem 1, Eq. (12) derives a learning-rate upper bound from this budget, providing a theoretical basis for controlling policy drift through smaller learning rates. Moreover, even when increasing the learning rate raises the change rate above 2%, AReaL-DTE maintains exact synchronization and performance benefits.
The complete corresponding performance results are reported in Table 13: across the six settings, AReaL-DTE achieves –, –, and – the throughput of PULSE at , , and Gbps, respectively. In particular, when the learning rate is increased to four times the baseline and the change rate reaches 3.414324%, AReaL-DTE still achieves the throughput of PULSE at Gbps.
Response to W1/Q1: Experimental settings and per-iteration latency breakdown
Thank you for the suggestion. We provide the training and rollout settings for both models, together with a per-iteration latency breakdown for AReaL-DTE, PULSE, and ByteCheckpoint. All three methods use the same experimental configuration within each setting and follow a synchronous training schedule: each iteration waits for its rollout batch to complete, performs training, and then synchronizes weights for the next rollout.
1. Experimental settings: batch sizes, generation lengths, and synchronization cadence.
Both models use GRPO with a learning rate of 3×10−6, weight decay of 0.01, and BF16 inference weights. The detailed settings are as follows:
Performance measurements exclude the first step as warm-up and use steps 2–40 as the measurement window.
2. Per-iteration latency breakdown: non-sync times are similar, with the main savings coming from weight synchronization.
The following table reports per-iteration times at 0.1 Gbps, in seconds. Each timing cell lists AReaL-DTE / PULSE / ByteCheckpoint, in that order.
Note: Training time includes advantage computation and actor updates. The non-sync total already includes rollout, training, and other runtime overhead, so it should not be added to the first two timing columns again; the full iteration time is the non-sync total plus weight synchronization time. Runtime overhead includes a configured wait after pausing generation to allow in-flight requests to finish: 30 seconds for Qwen3-30B-A3B and 0 seconds for Qwen3-8B, with the same settings across all three methods. As defined in Section 4.1, weight synchronization latency includes update construction, transfer, and receiver-side application.
For the same model and workload, the three methods have similar rollout, training, and non-sync times, while AReaL-DTE has the lowest weight synchronization latency in all eight configurations in the table. For example, on Qwen3-30B-A3B / GSM8K, non-sync totals are 45.17, 46.03, and 44.72 seconds, respectively, whereas weight synchronization takes 477.47, 544.86, and 8,741.18 seconds. This breakdown shows that the end-to-end gains come primarily from reducing weight synchronization overhead.
At 1 and 10 Gbps, each method's rollout, training, and non-sync times for the same model and workload differ from the corresponding 0.1 Gbps measurements by less than 2 seconds in absolute terms; bandwidth primarily affects the weight synchronization stage. Complete synchronization timings at all three bandwidths are reported in Table 3(a) for Qwen3-30B-A3B and Appendix B.2, Table 8(a) for Qwen3-8B.