acceptodds
Under review as a conference paper at ICLR 2027

AReaL-DTE: Sparse Policy-Weight Transfer for Online Agentic Reinforcement Learning

Abstract

Online agentic reinforcement learning implemented with micro-services separates policy model training from rollout generation, improving scalability and modularity while potentially making frequent policy-weight synchronization a critical systems overhead. Shared storage can naturally connect these services across clusters, but vanilla dense policy-weight synchronization could be communication-intensive. Sparse synchronization reduces transferred data, yet checkpoint-oriented approaches can still retain a previous model and materialize complete intermediates to connect heterogeneous training and inference layouts. We present AReaL-DTE, a snapshot-free Delta Transfer Engine that turns inference-visible weight sparsity into end-to-end system efficiency. Across our evaluated workloads, fewer than 2% of BF16 weight elements change between consecutive policy versions. AReaL-DTE reconstructs overwritten weights on demand by inverting AdamW updates, streams reconstructed and current parameters through converter-aligned BF16 change detection, and remaps changed elements directly into receiver-local coordinates. AReaL-DTE supports manifest-committed sparse transfer through shared storage across clusters and a deadlock-safe two-round protocol within a cluster, followed by direct application to inference shards. We evaluate AReaL-DTE on Qwen3-8B and Qwen3-30B-A3B across four online RL workloads. AReaL-DTE achieves speedups of up to over ByteCheckpoint and over PULSE across clusters, and up to and , respectively, within a cluster. In the same-cluster Qwen3-30B-A3B experiments, it reduces peak GPU memory by approximately 41% and peak CPU memory by at least 87%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
online reinforcement learning, distributed RL systems, sparse weight synchronization, machine learning systems

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_7WTEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a technique for synchronizing model weights between training and rollout workers, named AREAL-DTE. It accelerates weight synchronization by transferring only changed BF16 weights, based on the observation that few BF16 weight elements change between consecutive policy versions. Specifically, it reconstructs previous weights by inverting AdamW updates, detects BF16 weight changes during layout conversion, and directly transfers sparse updates to inference workers. The evaluation shows that AREAL-DTE achieves meaningful improvements in end-to-end RL training throughput, synchronization latency, and memory efficiency compared to ByteCheckpoint and PULSE.

Critical strengths
  1. The paper shows a valuable insight that although FP32 weight updates are dense, the BF16 weight changes observed by inference workers are sparse.

  2. The evaluation shows performance improvements over existing methods.

Critical weaknesses
  1. The End-to-End Speedup lacks some important experimental settings, such as per-iteration latency breakdown, whether rollout and training are asynchronous, training and rollout batch sizes, and rollout length, raising concerns about the soundness of the experiments. Moreover, all evaluated tasks are non-agentic, making it unclear whether such large speedups can still be achieved on long-horizon agentic tasks.

  2. BF16 weight sparsity lacks sensitivity analysis, and the theoretical analysis relies on assumptions that may not hold in practice. Specifically, the derivation of Eq. (25) in Appendix A assumes that the relative positions of weights within their BF16 rounding cells are uniformly distributed.

Questions for authors
  1. Could the authors provide more details about the experimental settings, including the per-iteration latency breakdown of rollout, training, and weight synchronization, whether rollout and training are asynchronous, training and rollout batch sizes, and rollout length?

  2. Could the authors evaluate whether the reported end-to-end speedups can still be achieved on long-horizon agentic tasks and larger models?

  3. Could the authors provide a sensitivity analysis of BF16 weight sparsity under different training settings, such as learning rates and batch sizes?

6 comments
AuthorsOpenReview ↗

Response to W1/Q1: Experimental settings and per-iteration latency breakdown

W1 / Q1: Experimental settings and per-iteration latency breakdown

Thank you for the suggestion. We provide the training and rollout settings for both models, together with a per-iteration latency breakdown for AReaL-DTE, PULSE, and ByteCheckpoint. All three methods use the same experimental configuration within each setting and follow a synchronous training schedule: each iteration waits for its rollout batch to complete, performs training, and then synchronizes weights for the next rollout.

1. Experimental settings: batch sizes, generation lengths, and synchronization cadence.

Both models use GRPO with a learning rate of 3×10−63\times10^{-6}, weight decay of 0.010.01, and BF16 inference weights. The detailed settings are as follows:

SettingQwen3-8B and Qwen3-30B-A3B
HardwareNVIDIA H200; Qwen3-8B uses 8 GPUs and Qwen3-30B-A3B uses 16 GPUs
Rollout batch size8 prompts per iteration
Responses per prompt2
Global training batch size16 response sequences per iteration: 8 prompts × 2 responses
Response length limit2,048 newly generated tokens
Total sequence length limit3,072 tokens, including the prompt
Weight synchronization cadenceOnce after each training iteration, K=1K=1

Performance measurements exclude the first step as warm-up and use steps 2–40 as the measurement window.

2. Per-iteration latency breakdown: non-sync times are similar, with the main savings coming from weight synchronization.

The following table reports per-iteration times at 0.10.1 Gbps, in seconds. Each timing cell lists AReaL-DTE / PULSE / ByteCheckpoint, in that order.

ModelBenchmarkRolloutTrainingWeight synchronizationNon-sync total
Qwen3-30B-A3BGSM8K12.60 / 13.63 / 12.542.40 / 2.23 / 2.02477.47 / 544.86 / 8,741.1845.17 / 46.03 / 44.72
Qwen3-30B-A3BHumanEval13.17 / 13.00 / 13.112.75 / 2.09 / 2.65539.64 / 604.60 / 8,741.0146.10 / 45.27 / 45.94
Qwen3-30B-A3BMATH-50013.36 / 14.14 / 13.213.08 / 3.41 / 3.20523.59 / 567.42 / 8,740.8846.62 / 47.72 / 46.60
Qwen3-30B-A3BLogiQA12.96 / 12.99 / 12.762.92 / 2.21 / 2.30511.81 / 559.26 / 8,744.3946.05 / 45.36 / 45.24
Qwen3-8BGSM8K9.78 / 9.83 / 9.681.75 / 1.88 / 1.72117.40 / 142.64 / 2,339.9811.66 / 11.85 / 11.53
Qwen3-8BHumanEval10.01 / 10.14 / 10.102.14 / 2.06 / 2.46118.66 / 127.20 / 2,341.1512.29 / 12.33 / 12.70
Qwen3-8BMATH-50010.23 / 10.19 / 10.112.21 / 2.27 / 2.37127.59 / 138.99 / 2,336.9312.59 / 12.60 / 12.62
Qwen3-8BLogiQA9.96 / 10.09 / 9.942.02 / 2.27 / 2.10143.42 / 158.78 / 2,340.4912.12 / 12.50 / 12.18

Note: Training time includes advantage computation and actor updates. The non-sync total already includes rollout, training, and other runtime overhead, so it should not be added to the first two timing columns again; the full iteration time is the non-sync total plus weight synchronization time. Runtime overhead includes a configured wait after pausing generation to allow in-flight requests to finish: 30 seconds for Qwen3-30B-A3B and 0 seconds for Qwen3-8B, with the same settings across all three methods. As defined in Section 4.1, weight synchronization latency includes update construction, transfer, and receiver-side application.

For the same model and workload, the three methods have similar rollout, training, and non-sync times, while AReaL-DTE has the lowest weight synchronization latency in all eight configurations in the table. For example, on Qwen3-30B-A3B / GSM8K, non-sync totals are 45.17, 46.03, and 44.72 seconds, respectively, whereas weight synchronization takes 477.47, 544.86, and 8,741.18 seconds. This breakdown shows that the end-to-end gains come primarily from reducing weight synchronization overhead.

At 11 and 1010 Gbps, each method's rollout, training, and non-sync times for the same model and workload differ from the corresponding 0.10.1 Gbps measurements by less than 2 seconds in absolute terms; bandwidth primarily affects the weight synchronization stage. Complete synchronization timings at all three bandwidths are reported in Table 3(a) for Qwen3-30B-A3B and Appendix B.2, Table 8(a) for Qwen3-8B.

AuthorsOpenReview ↗

Response to W1/Q2: Multi-turn tool-use evaluation and end-to-end performance

W1 / Q2: Multi-turn tool-use workloads and larger models

Thank you for the suggestion. We add an RL experiment with a Hermes agent on DAPO-Math using Qwen3-30B-A3B, covering multi-turn reasoning, Python tool calls, execution feedback, and subsequent reasoning before submitting a final answer. On this real tool-use workload, AReaL-DTE achieves 227.199 tokens/s, corresponding to 1.107×1.107\times and 13.94×13.94\times the throughput of PULSE and ByteCheckpoint, respectively.

1. Experimental settings and interaction lengths: training on complete multi-turn trajectories, with weight synchronization after every update.

The experiment uses PPO, a 0.10.1 Gbps link, and synchronization at every step (K=1K=1). Each update uses 16 task instances, with one complete interaction trajectory per instance. After all 16 trajectories in the current batch are completed and used for training, weights are synchronized, and the next batch is released only after both inference replicas acknowledge the new version. AReaL-DTE, PULSE, and ByteCheckpoint are compared using the same experimental configuration.

Each model response can generate up to 8,192 new tokens, with a context limit of 65,536 tokens. Each task allows up to 16 Python tool executions, each lasting at most 15 seconds. The 8,192-token limit applies to each response, not to the entire multi-turn trajectory; the limit of 16 tool executions does not cap model interactions at 16. The observed interaction lengths are as follows:

StatisticMeasured valueStatisticMeasured value
Mean model interactions per trajectory3.328Mean Python calls per trajectory2.245
P90 model interactions8P90 Python calls7
Maximum model interactions in completed trajectories22Maximum total newly generated tokens in completed trajectories46,176

These trajectories include actual tool execution and feedback, demonstrating multi-turn interaction beyond single-response tasks.

2. End-to-end performance: throughput gains persist with multi-turn tool interaction.

Performance measurements exclude the first step as warm-up and cover steps 2–40, for a total of 39 measured steps. The following table reports per-step times and end-to-end throughput, covering complete trajectory generation and waiting, training, and weight synchronization.

MethodNon-sync total (s/step)Trajectory-batch wait (s/step)Training (s/step)Weight synchronization (s/step)Throughput (tokens/s)
AReaL-DTE160.664148.49411.605475.239227.199
PULSE160.612148.68111.350544.804205.157
ByteCheckpoint162.303149.98411.7438,736.24716.297

Note: Trajectory-batch wait and training are already included in the non-sync total and should not be added again. The full step time is the non-sync total plus weight synchronization time.

AReaL-DTE improves end-to-end throughput over PULSE by approximately 10.7% and achieves 13.94×13.94\times ByteCheckpoint's throughput. Non-sync times are similar across the three methods, while AReaL-DTE reduces weight synchronization time from PULSE's 544.804 seconds to 475.239 seconds. This shows that reducing weight synchronization overhead continues to translate into end-to-end performance gains in RL workloads involving multi-turn reasoning and actual tool calls.

Regarding model size, the current evaluation covers Qwen3-8B and Qwen3-30B-A3B. This additional experiment extends the evaluation of Qwen3-30B-A3B to multi-turn tool-use tasks; it does not add measurements on models beyond that size.

AuthorsOpenReview ↗

Response to W2/Q3 (1/2): Learning-rate and batch-size sensitivity

W2 / Q3: Effects of training settings on BF16 change sparsity and empirical evidence for the uniform-position assumption

Thank you for this suggestion. We add two complementary sources of evidence: six sensitivity experiments varying learning rate and batch size (Appendix D.6, Tables 12–13), and measurements of rounding-position distributions using true weights (Appendix A, Figure 5). We address sensitivity to training settings and the uniform-position assumption separately below.

1. Sensitivity experiments: change rates increase with learning rate, while halving or doubling batch size preserves high sparsity.

The experiments use Qwen3-8B / GSM8K, GRPO, AdamW, and BF16 inference weights, with 40 training steps per setting. The baseline learning rate is 3×10−63\times10^{-6}, the global training batch size is 16 rollout response sequences (8 prompts with 2 sampled responses each), and weight decay is 0.010.01. Each variant changes only the learning rate or batch size, keeping the remaining settings fixed and synchronizing after every parameter update (K=1K=1). Within each setting, AReaL-DTE and PULSE are compared using the same experimental configuration.

The following table corresponds to Appendix D.6, Table 12. The true BF16 change rate is obtained by comparing the true pre-update and post-update BF16 weights bit for bit, using all model weight elements as the denominator. Percentages are calculated from aggregated counts.

SettingTrue BF16 change rateSettingTrue BF16 change rate
LR =10−6=10^{-6}0.627989%Baseline: LR =3×10−6=3\times10^{-6}, batch =16=161.536342%
LR =6×10−6=6\times10^{-6}2.680988%Batch =8=81.411514%
LR =1.2×10−5=1.2\times10^{-5}3.414324%Batch =32=321.472879%

As the learning rate increases from 10−610^{-6} to 1.2×10−51.2\times10^{-5}, the change rate rises from 0.627989% to 3.414324%. At the baseline learning rate, halving or doubling batch size gives change rates of 1.411514% and 1.472879%, respectively, close to the baseline. All six settings record zero missed updates and zero receiver bitwise mismatches at every check.

The original paper's observation of a change rate below 2% comes from the preliminary experiments with learning rates of 10−610^{-6} and 3×10−63\times10^{-6}. A larger learning rate increases the update magnitude and can cause policy drift to exceed the prescribed trajectory KL budget. Appendix A, Theorem 1, Eq. (12) derives a learning-rate upper bound from this budget, providing a theoretical basis for controlling policy drift through smaller learning rates. Moreover, even when increasing the learning rate raises the change rate above 2%, AReaL-DTE maintains exact synchronization and performance benefits.

The complete corresponding performance results are reported in Table 13: across the six settings, AReaL-DTE achieves 1.061.06–1.31×1.31\times, 1.291.29–1.78×1.78\times, and 1.501.50–1.96×1.96\times the throughput of PULSE at 0.10.1, 11, and 1010 Gbps, respectively. In particular, when the learning rate is increased to four times the baseline and the change rate reaches 3.414324%, AReaL-DTE still achieves 1.50×1.50\times the throughput of PULSE at 1010 Gbps.

Reviewer_kjXNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of efficient policy-weight synchronization in online agentic reinforcement learning. The authors propose AREAL-DTE algorithm to reduce communication and memory overhead. The AREAL-DTE is a snapshot-free framework which combines optimizer-inverted weight reconstruction, sparse delta detection, and efficient weight transfer protocols. The authors provide theoretical analysis to explain BF16 weight sparsity, and conduct experiments on Qwen3 models based on four benchmarks to show the best performance over other baselines.

Critical strengths
  1. In Section 1 and Section 2, the authors clearly address a crucial problem about the communication and bottleneck in online agentic RL, where frequent policy-weight synchronization introduces substantial overhead between training and inference process. This is a very practical motivation for the work of AREAL-DTE.

  2. The AREAL-DTE is well designed to combine optimizer-inverted weight reconstruction, converter-aligned delta detection, and two efficient transfer mechanisms to avoid maintaining historical model snapshots or constructing the full intermediates, which cause the overhead of training.

  3. The corresponding experiments are extensive, covering Qwen3-8B and Qwen3-30B-A3B and four different benchmark. The results demonstrate detailed improvements in synchronization latency, end-to-end training throughput, and memory efficiency compared with other baselines. In addition, the component-wise ablation study shows the contributions of each component to the efficiency improvement of the proposed AREAL-DTE framework.

Critical weaknesses
  1. The core advantage of AREAL-DTE relies heavily on the AdamW inversion, but other optimizers are also frequently to be used, and the paper lacks discussion about the generalizability of the proposed method/mechanism across different optimizers.

  2. The authors acknowledge that the current optimizer rounding strategy may prevent the exact recovery of the previous FP32 weights and AREAL-DTE only uses the result as a numerical reconstruction. However, the potential impact of these numerical errors can not be ignored in larger-scaled models and longer training processes, since the errors may be accumulated to influence the result. Though the Figure 3 shows no observable change in the training dynamics, but the observation is based on only 100 steps, which is insufficient to measure the whole progress of online RL.

Questions for authors
  1. Can the authors discuss about how the proposed mechanism based on AdamW could be generalized to other optimizers? If not considered, could you comment on potential challenges which would be encountered during the process?

  2. Can you demonstrate the longer-term evaluations to verify whether the numerical errors would accumulate over training time? Can the numerical errors be bounded theoretically?

6 comments
AuthorsOpenReview ↗

Response to W1/Q1: Generalization to Adam and other optimizers

W1 / Q1: How can the reconstruction mechanism generalize to other optimizers?

Thank you for the suggestion. AReaL-DTE's reconstruction mechanism also applies to Adam, including Adam with coupled weight decay. The key is to use the optimizer state retained after the update to recompute the update increment that was applied, with a reconstruction rule matching the actual update formula. We add checks during real RL training for three Adam configurations, each run for 40 training steps, with zero missed BF16 updates and zero receiver bitwise mismatches at every step.

1. Reconstruction for Adam: coupled weight decay is already incorporated into the retained moment states.

Following the elementwise notation in Appendix D.1, let xx be the true pre-update weight, yy the current weight, x^\hat{x} the reconstructed pre-update weight, and tt the training-step index of the current update. Adam uses gt+λxg_t+\lambda x to update its first and second moments, where gtg_t is the raw gradient and λ\lambda is the weight decay coefficient. The updated moment states mt,vtm_t,v_t already incorporate the effect of weight decay, so they can be used directly to recompute the adaptive update direction dtd_t and reconstruct the pre-update weight:

dt=mt/(1−β1t)vt/(1−β2t)+ϵ,x^=y+ηtdt.d_t=\frac{m_t/(1-\beta_1^t)}{\sqrt{v_t/(1-\beta_2^t)}+\epsilon},\qquad \hat{x}=y+\eta_t d_t.

Here, ηt\eta_t is the learning rate for this step, β1,β2\beta_1,\beta_2 are the moment decay coefficients, and ϵ\epsilon is the numerical stability term. Because Adam's coupled weight decay is already incorporated into its moment states, reconstruction does not require division by 1−ηtλ1-\eta_t\lambda, as it does for AdamW's decoupled weight decay. The equation above gives the inverse relation in exact arithmetic. After FP32 reconstruction, BF16 change detection still incorporates rounding-boundary protection, and transmitted values always come from the current true weights.

2. Empirical validation: correct synchronization across different weight decay configurations.

The experiments use Qwen3-8B / GSM8K with a learning rate of 3×10−63\times10^{-6}, BF16 inference weights, and synchronization after every parameter update. Each of the three Adam configurations and the AdamW baseline runs for 40 training steps, with the following results:

Optimizer configurationTrue BF16 change rateSelection rate after protectionMissed BF16 updatesReceiver bitwise mismatchesMaximum FP32 absolute reconstruction errorMean FP32 absolute reconstruction error
Adam, weight decay =0=01.606031%1.607711%009.537×10−79.537\times10^{-7}3.090×10−123.090\times10^{-12}
Adam, weight decay =0.003=0.0039.363010%9.363566%004.768×10−74.768\times10^{-7}1.609×10−111.609\times10^{-11}
Adam, weight decay =0.01=0.019.510247%9.510836%009.537×10−79.537\times10^{-7}1.568×10−111.568\times10^{-11}
AdamW, weight decay =0.01=0.011.536342%1.538025%009.537×10−79.537\times10^{-7}1.479×10−91.479\times10^{-9}

Note: The true change rate and selection rate after protection both use the total number of checked model weights as the denominator; percentages are calculated from aggregated counts. Missed updates and receiver bitwise mismatches are zero at every check. Reconstruction errors compare reconstructed pre-update FP32 master weights with the true pre-update FP32 master weights retained for the audit. The maximum is taken over all checked elements and steps; the mean is the sum of all absolute errors divided by the total number of element comparisons.

Adam and AdamW handle weight decay differently, so the same coefficient does not imply the same update magnitude or BF16 change rate. In these experiments, Adam with coupled weight decay has higher change rates, yet reconstruction and change detection still yield zero missed updates and bitwise agreement at the receiver, validating synchronization correctness across these optimizer configurations.

3. Further generalization: match reconstruction to the retained state and the actual update procedure.

Extending the mechanism to another optimizer requires checking whether the state retained after the update is sufficient to recompute the applied update increment, constructing a reconstruction rule that matches the actual computation order and precision, and verifying that numerical protection covers the reconstruction error. Our Adam experiments directly validate this approach; Appendix D provides AdamW reconstruction-error bounds and a proof of synchronization correctness. By developing and validating reconstruction for the specific update rule, the same synchronization workflow—reconstruct pre-update weights, detect changes, and send current true values—can be extended to the corresponding optimizer.

AuthorsOpenReview ↗

Response to W2/Q2 (1/3): Why reconstruction error does not accumulate

W2 / Q2: Can reconstruction error accumulate during long-term training, and can it be bounded theoretically?

Thank you for raising this question about long-term numerical correctness. We add checks over 200 consecutive training steps and synchronizations of Qwen3-8B / GSM8K: the receiver continuously maintains the same live model weights, and all 8,190,735,3608{,}190{,}735{,}360 BF16 weight elements are checked after every synchronization, with zero missed updates and zero receiver bitwise mismatches at every check (Appendix D.5, Table 11). The maximum and mean FP32 absolute reconstruction errors are 9.53674×10−79.53674\times10^{-7} and 1.76757×10−91.76757\times10^{-9}, respectively. Appendix D also provides finite-precision error bounds, a proof that rounding-boundary protection prevents missed updates, and an inductive proof of bitwise agreement after every synchronization, explaining why reconstruction error does not accumulate through synchronization.

1. Long-term synchronization correctness: every synchronization preserves the correct version, so reconstruction error does not accumulate through synchronization.

AReaL-DTE uses reconstructed pre-update weights for change detection and sends the current true BF16 values at all selected coordinates to the receiver. Detection combines BF16 bit comparison with boundary protection: coordinates with different bit patterns are selected for transmission; when the bit patterns match, the detector further checks the distance from the reconstructed FP32 value to the rounding boundaries and also selects coordinates near a boundary. Boundaries are determined by the actual neighboring BF16 values, with default protection coefficient ρ=10−4\rho=10^{-4}. The full rule is given in Section 3.2, Eqs. (3)–(5).

Within the setting analyzed in Appendix D, Proposition 19 proves by induction that, starting from the correct initial BF16 version, the receiver matches the sender bit for bit after every synchronization. Suppose the receiver holds the correct base version before the current synchronization. Then:

  • Selected coordinates: the sender's current true BF16 values overwrite the corresponding receiver coordinates, making them correct after the update. The very few redundant selections also write the correct target values.
  • Skipped coordinates: the no-miss theorem ensures that their true BF16 values have not changed, so retaining the base-version values gives the correct current values.

These two cases cover the entire model, establishing bitwise agreement between the receiver and sender after this synchronization and providing a correct starting point for the next. Consequently, FP32 reconstruction error does not accumulate through synchronization into receiver-weight drift; this induction argument applies to any number of synchronizations.

AuthorsOpenReview ↗

Response to W2/Q2 (2/3): Reconstruction-error bounds and boundary protection

2. Theoretical error bound: the clearance retained by the default boundary protection is strictly greater than reconstruction error.

Following the elementwise notation in Appendix D.1, let xx be the true pre-update FP32 weight, x^\hat{x} its reconstruction from inversion, and yy the current true FP32 weight; quant⁡BF16\operatorname{quant}_{\mathrm{BF16}} is the BF16 quantization operator defined in Section 3.2.

Theorem 17, Eq. (86) in Appendix D.3 bounds single-step FP32 reconstruction error. Lemmas 6 and 12 in Appendix D.2 bound the discrepancy between the forward and inverse update increments and the local reconstruction error near BF16 rounding cells, respectively. Theorem 14 uses these bounds to establish a sufficient condition for boundary protection to prevent missed updates.

Using the fixed AdamW constants in this analysis, Corollary 15 and Table 10 establish that the default ρ=10−4\rho=10^{-4} prevents missed updates whenever

ηhλh≤6×10−6,\eta_h\lambda_h\le6\times10^{-6},

at every step. Here, ηh\eta_h and λh\lambda_h are the learning-rate and weight-decay scalars defined in Appendix D.1. Let Δ32\Delta_{32} be the local FP32 spacing defined in the proof, and let dsafed_{\mathrm{safe}} be the distance from the reconstructed value to the nearest BF16 rounding boundary at an eligible skipped coordinate. Then

∣x^−x∣<3.961Δ32<4Δ32≤dsafe.|\hat{x}-x| < 3.961\Delta_{32} < 4\Delta_{32}\le d_{\mathrm{safe}}.

Thus, the true pre-update weight remains inside the same BF16 rounding cell as the reconstructed value. At coordinates whose bit patterns match and that pass the protection check,

quant⁡BF16(x)=quant⁡BF16(x^)=quant⁡BF16(y).\operatorname{quant}_{\mathrm{BF16}}(x)=\operatorname{quant}_{\mathrm{BF16}}(\hat{x})=\operatorname{quant}_{\mathrm{BF16}}(y).

This establishes that skipped coordinates have unchanged true BF16 values, as used in the induction above. The main experiments use a learning rate of 3×10−63\times10^{-6} and weight decay of 0.010.01, giving a product of 3×10−83\times10^{-8}, within this range. This configuration also admits the tighter reconstruction-error bound of less than 3.77Δ323.77\Delta_{32} (Corollary 16).

Appendix D.3 further covers cases where the relevant multiplications in the forward weight update produce subnormal results under the baseline configuration. After accounting for the corresponding rounding errors, the total reconstruction error remains strictly below 4Δ324\Delta_{32}, while boundary protection retains at least 4Δ324\Delta_{32} of boundary clearance, preserving the no-miss guarantee.

Reviewer_j5BHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the high cost of policy-weight synchronization in online reinforcement learning systems where training and inference are decoupled: every policy update must be delivered to the rollout side. Dense synchronization requires transferring the complete model, while existing sparse synchronization, though transferring only changed values, still retains a model-scale copy of the previous weights and relies on complete checkpoints to bridge the divergent parameter layouts of training and inference engines; the overhead thus grows linearly with model size in all cases. The authors observe that fewer than 2% of the BF16 weight elements visible to the inference side actually change between consecutive policy versions, and they theoretically explain why dense FP32 updates induce only sparse inference-visible changes by deriving a step-size upper bound from the KL-divergence constraint and characterizing the fraction of coordinates that cross BF16 rounding boundaries. Building on this finding, the paper proposes a snapshot-free sparse weight-synchronization system that reconstructs pre-update weights by inverting the optimizer state, detects changes and remaps them to receiver-local shards directly along the training-to-inference format-conversion path, and completes cross-cluster transfer via a two-round communication protocol and shared storage. The authors evaluate the system on Qwen3-8B and Qwen3-30B-A3B across four online RL workloads under multiple bandwidth settings: cross-cluster synchronization latency is reduced by up to 19.9×, end-to-end throughput is improved by more than 3×, and memory consumption is reduced by over 40%, demonstrating that the gains of sparse synchronization stem from a theoretically characterizable regularity rather than workload-specific coincidence.

Critical strengths
  1. The paper elevates an engineering intuition into a verifiable, theoretically grounded conclusion, and the proposed method is novel.

  2. The proposed method achieves significant improvements in cross-cluster end-to-end training throughput, cross-cluster synchronization latency, and intra-cluster CPU/GPU memory consumption.

  3. The authors demonstrate solid theoretical capability, with well-reasoned derivations.

Critical weaknesses
  1. The discussion of the applicability boundaries of the proposed method is insufficient.

  2. The paper lacks an analysis of how the improvement ratios vary across benchmarks, together with the corresponding attribution analysis. Although the experiments validate higher end-to-end throughput on four benchmarks, there is no analysis of how the improvement ratios relate to workload characteristics, such as rollout/synchronization features.

  3. There is a lack of direct validation of the error between the reconstructed weights and the actual pre-update weights, as well as of numerical correctness.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to W1 (1/2): Applicability across optimizers and training settings

W1: Applicability boundaries

Thank you for this suggestion. We clarify AReaL-DTE's applicability along two dimensions: the synchronization mechanism requires sufficient retained optimizer state to reconstruct the update, with reconstruction and detection rules matched to the actual optimizer and inference-weight representation; end-to-end performance benefits depend on the sparsity of inference-visible changes and the fraction of total runtime spent on weight synchronization. We explain these dimensions using additional experiments across optimizers, training settings, inference precisions, and multi-turn tool interaction.

1. Optimizers: AdamW as the foundation, with additional validation of synchronization correctness for Adam.

Sections 3.1–3.2 and Appendix D provide the reconstruction mechanism and numerical analysis for AdamW with FP32 master weights and BF16 inference weights. AReaL-DTE detects changes using reconstructed weights together with rounding-boundary protection; transmitted values always come from the current true weights. The theoretical basis for synchronization correctness is provided in Appendix D and summarized in our response to W3.

We additionally check Adam configurations on Qwen3-8B / GSM8K with a learning rate of 3×10−63\times10^{-6} and weight decay of 00, 0.0030.003, and 0.010.01. Each configuration runs for 40 training steps, with zero missed BF16 updates and zero receiver bitwise mismatches at every step. The latter two configurations use Adam's coupled weight decay: the decay contribution is already incorporated into the retained moment states, and the reconstruction rule is matched to Adam's actual update formula. These experiments provide additional validation of synchronization correctness for the Adam configurations. Extending to other optimizers similarly requires matching reconstruction and protection rules to their retained state, update formulas, and computational precision.

2. Training settings: exact synchronization and performance benefits across learning rates, batch sizes, and synchronization intervals.

Appendix D.6, Tables 12–13 cover eight learning-rate, batch-size, and synchronization-interval settings, with true BF16 change rates ranging from 0.627989% to 3.470902%. Every check records zero missed updates and zero receiver bitwise mismatches, and measured throughput exceeds PULSE at 0.10.1, 11, and 1010 Gbps. Methods are compared using the same experimental configuration within each setting.

The original paper's observation of a change rate below 2% comes from the preliminary experiments with learning rates of 10−610^{-6} and 3×10−63\times10^{-6}. A larger learning rate increases the update magnitude and can cause policy drift to exceed the prescribed trajectory KL budget. Appendix A, Theorem 1, Eq. (12) derives a learning-rate upper bound from this budget, providing a theoretical basis for controlling policy drift through smaller learning rates. Moreover, even when the learning rate is increased to four times the baseline and the true BF16 change rate reaches 3.414324%, AReaL-DTE maintains exact synchronization and achieves 1.50×1.50\times the throughput of PULSE at 1010 Gbps.

For longer synchronization intervals, AReaL-DTE detects changes at every step, accumulates the selected sets, and sends the latest true values at coordinates in their union when synchronization occurs (Appendix D.4, Corollary 20). As synchronization becomes less frequent, the contribution of weight-synchronization optimization to end-to-end performance also changes: at 1010 Gbps, synchronization intervals K=1,2,4K=1,2,4 yield throughput ratios of 1.74×1.74\times, 1.33×1.33\times, and 1.17×1.17\times, respectively. This demonstrates support for synchronization across multiple steps, with benefits varying according to the synchronization frequency.

AuthorsOpenReview ↗

Response to W1 (2/2): FP8 inference and multi-turn tool interaction

3. Inference precision: additional FP8 experiments validate weight and quantization-state synchronization and end-to-end benefits.

We add 40 training steps of experiments with FP8 inference on Qwen3-8B / GSM8K, using a learning rate of 3×10−63\times10^{-6}, batch size 16, weight decay 0.010.01, and synchronization at every step. Training retains BF16 weights and FP32 optimizer states. On the inference side, linear layers in Attention and MLP modules use FP8 E4M3FN weights with FP32 scales, while parameters such as embeddings, the LM head, and normalization layers remain in BF16. Detection is matched to the corresponding quantization rules, selected coordinates carry current true encodings, and scales are synchronized in full at every synchronization.

All 40 training steps record zero missed updates, with bitwise agreement in the actual receiver's FP8 weights, scales, and retained BF16 parameters. Across all FP8 and BF16 model weights, the true encoding change rate is 0.129598636%, the selection rate is 0.129661461%, and redundancy accounts for only 0.048453500% of selected entries. At 0.10.1, 11, and 1010 Gbps, AReaL-DTE achieves 1.46×1.46\times, 1.65×1.65\times, and 1.67×1.67\times the throughput of FP8 PULSE, respectively. This extends empirical validation from BF16 to FP8 inference with synchronization of quantization scales.

4. Workloads: synchronization optimization continues to yield end-to-end benefits with multi-turn tool interaction.

We also evaluate the Hermes agent on DAPO-Math with Qwen3-30B-A3B in a multi-turn Python tool-use workload. The experiment uses PPO, 16 task instances per update (one complete interaction trajectory per instance), synchronization at every step, and 0.10.1 Gbps bandwidth. Trajectories average 3.3283.328 model interactions and 2.2452.245 Python calls, with P90 values of 8 interactions and 7 calls, respectively.

Over the performance measurement window of steps 2–40, AReaL-DTE achieves 227.199 tokens/s, or 1.107×1.107\times the throughput of PULSE, an increase of approximately 10.7%. This result demonstrates that AReaL-DTE continues to deliver end-to-end performance benefits in an online RL workload involving multi-turn reasoning and actual tool calls.

AuthorsOpenReview ↗

Response to W2: Speedup variation and performance attribution

W2: Variation in speedup across workloads and attribution of the gains

Thank you for this suggestion. We add a per-iteration latency breakdown across four benchmarks for Qwen3-30B-A3B at 0.10.1 Gbps, comparing non-synchronization time, weight-synchronization time, and end-to-end speedup. Each run covers 40 training steps, excludes step 1 as warm-up, and reports results over the 39 steps from steps 2–40. Within each benchmark, AReaL-DTE and PULSE use the same experimental configuration with deterministic execution, and we verify that the total numbers of generated tokens match, allowing performance comparison at equal generation work.

BenchmarkNon-sync time (s): DTE / PULSEBF16 change rateSync time (s): DTE / PULSESync speedupEnd-to-end throughput speedup
GSM8K45.17 / 46.031.7389%477.47 / 544.861.141×1.141\times1.131×1.131\times
HumanEval46.10 / 45.271.9766%539.64 / 604.601.120×1.120\times1.109×1.109\times
MATH-50046.62 / 47.721.9114%523.59 / 567.421.084×1.084\times1.079×1.079\times
LogiQA46.05 / 45.361.8658%511.81 / 559.261.093×1.093\times1.084×1.084\times

Note: DTE denotes AReaL-DTE, and times are per iteration. Non-sync time includes rollout, training, and runtime overhead outside the synchronization timing window, including the same configured 30-s drain wait for both methods to allow in-flight rollout requests to clear after generation is paused.

1. Synchronization savings are the main source of end-to-end gains in this setting.

Within the same benchmark, non-sync time differs by at most 1.10 s per iteration between the two methods, whereas AReaL-DTE saves 43.83–67.39 s of weight-synchronization time per iteration. Thus, the reduction in synchronization time plays the dominant role in end-to-end speedup relative to differences in rollout, training, and other non-sync overhead.

For equal generation work, the throughput ratio is the inverse ratio of total execution times. Let TotherT_{\mathrm{other}} and TsyncT_{\mathrm{sync}} denote non-sync and synchronization time, respectively. End-to-end speedup can then be written as

Se2e=TotherPULSE+TsyncPULSETotherDTE+TsyncDTE.S_{\mathrm{e2e}}=\frac{T_{\mathrm{other}}^{\mathrm{PULSE}}+T_{\mathrm{sync}}^{\mathrm{PULSE}}}{T_{\mathrm{other}}^{\mathrm{DTE}}+T_{\mathrm{sync}}^{\mathrm{DTE}}}.

The ratio of PULSE's mean full step time to AReaL-DTE's matches the reported throughput speedup to the displayed precision. Across the four benchmarks, the synchronization and end-to-end speedup rankings agree: GSM8K > HumanEval > LogiQA > MATH-500. For example, GSM8K's synchronization speedup of 1.141×1.141\times corresponds to an end-to-end speedup of 1.131×1.131\times; the respective values for MATH-500 are 1.084×1.084\times and 1.079×1.079\times.

2. BF16 change rates explain differences in synchronization volume and absolute latency, while actual synchronization savings determine relative gains.

The BF16 change-rate ordering is GSM8K < LogiQA < MATH-500 < HumanEval, matching the synchronization-time ordering for both AReaL-DTE and PULSE. Higher change rates mean more weight elements to synchronize and, in this setting, correspond to higher absolute synchronization latency.

Relative speedup depends jointly on the difference in synchronization time between the two methods and its contribution to total runtime. For example, HumanEval has the highest change rate and absolute synchronization time, but not the highest relative speedup. This breakdown connects workload change sparsity, synchronization cost, and the resulting throughput gains, explaining the speedup differences across the four benchmarks.

Reviewer_HCR4OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AReaL-DTE accelerates policy-weight synchronization between training and rollout services in online reinforcement learning. It exploits sparse changes in inference-visible BF16 weights, reconstructs previous weights through AdamW inversion instead of retaining a historical snapshot, and converts and routes updates directly to inference shards. Experiments on Qwen3-8B and Qwen3-30B-A3B across four workloads demonstrate substantial reductions in synchronization latency and memory consumption, together with improved end-to-end training throughput.

Critical strengths
  • Addresses total synchronization overhead beyond network transfer, including snapshot storage, conversion, staging, and receiver-side application.
  • Achieves over 1.5 end-to-end training throughput over the sparse PULSE baseline at 10 Gbps, while reducing latency and peak memory.
  • Comparisons include dense and sparse baselines, multiple bandwidths, dense and mixture-of-experts models, and ablations examining reconstruction, remapping, and sparse transfer.
Critical weaknesses
  • AdamW inversion is not bitwise exact in floating-point arithmetic, so overlapping optimization statistics from one 100-step run do not establish that reconstructed change detection never misses BF16 updates or causes persistent receiver-weight discrepancies.
  • Evaluation is limited to two Qwen3 models using BF16 inference and AdamW, without establishing performance under different settings.
  • Its snapshot-based alternative stages the complete previous model onto the GPU, leaving unclear how much of the reported benefit would remain
Questions for authors
  • Since floating-point arithmetic is not bitwise exact, how do you prove that AdamW inversion errors never miss BF16 updates or cause persistent weight drift over long training runs?
  • How does AREAL-DTE perform under different settings -- such as higher learning rates, varied synchronization intervals, or other inference precisions (like FP8) beyond the evaluated Qwen3 models?
  • How much performance benefit does AdamW inversion retain if compared against a heavily optimized, streamed snapshot baseline instead of full GPU staging?
9 comments
AuthorsOpenReview ↗

Response to W1/Q1 (1/2): Direct numerical validation

W1 / Q1: Can floating-point reconstruction errors cause missed BF16 updates or persistent weight drift?

Thank you for raising this question about synchronization correctness. We added direct bitwise checks of the weights and a finite-precision proof: over 200 training steps of Qwen3-8B / GSM8K, rounding-boundary protection recovered an average of 20,94120{,}941 true BF16 updates per detection that bit comparison alone would miss, with zero missed updates after protection and zero receiver bitwise mismatches at every check. Total redundancy, including boundary protection, accounts for only 0.109700% (approximately 0.1%) of selected entries and 0.001631% of all model weights. Section 3.2 of the revised manuscript explains the detection mechanism, while Appendix D provides reconstruction-error bounds and a proof of bitwise agreement after every synchronization.

1. Direct full-model checks: measuring both FP32 reconstruction error and BF16 synchronization correctness.

Throughout the 200 training steps, the receiver continuously maintained the same live model weights, with 200 parameter updates, synchronizations, and full-model BF16 bitwise checks after synchronization. Each check covered all 8,190,735,3608{,}190{,}735{,}360 weight elements, including both changed and unchanged coordinates. We measured FP32 reconstruction error elementwise, checked for missed updates after forming the protected selected set and before transmission, and independently verified receiver weights after the barrier confirming completion of update application. The sender's full reference weights were placed only in temporary comparison buffers; they did not overwrite or refresh the receiver model. Appendix D.5, Table 11 reports the results; the full statistics and the breakdown of selections from boundary protection are provided below:

MetricValueShare
Missed BF16 updates after protection0 at every check—
Receiver BF16 bitwise mismatches0 at every check—
Weights checked per detection8,190,735,360100% of all weights
True BF16 changes121,618,0621.484825% of all weights
Selected entries after protection121,751,6241.486455% of all weights
Total redundant selections133,5620.109700% of selected; 0.001631% of all weights
True updates recovered by boundary protection20,94115.726430% of added selections
Selections added by boundary protection133,1580.001626% of all weights
Redundant selections before protection21,3450.017531594% of selected
Maximum FP32 absolute reconstruction error9.53674 × 10⁻⁷—
Mean FP32 absolute reconstruction error1.76757 × 10⁻⁹—

Note: Qwen3-8B / GSM8K, 200 training steps, with synchronization at every step and receiver weights checked after synchronization completes. “All weights” refers to all model-weight elements covered by each check; “selected” refers to entries selected after protection; “added selections” refers to entries added by boundary protection. Element counts are per-detection averages, while missed updates and receiver mismatches are zero at every check. Percentages are calculated from aggregated counts. Selections added by boundary protection include recovered true updates and additional redundant entries; total redundancy also includes redundant entries selected before protection. Reconstruction errors compare reconstructed pre-update FP32 master weights with the true pre-update FP32 master weights retained for the audit. The maximum is taken over all checked elements and steps; the mean is the sum of all absolute errors divided by the total number of element comparisons.

AuthorsOpenReview ↗

Response to W1/Q1 (2/2): Detection mechanism and correctness proof

2. Detection and transmission: boundary protection accounts for FP32 reconstruction error, and transmitted values are always current true values.

Following the elementwise notation in Section 3.2 and Appendix D.1, let xx be the true pre-update FP32 weight, x^\hat{x} its reconstruction from inversion, and yy the current true FP32 weight. Denote the current true BF16 value by

q=quant⁡BF16(y).q=\operatorname{quant}_{\mathrm{BF16}}(y).

AdamW inversion can introduce FP32 reconstruction error, placing x^\hat{x} and xx on opposite sides of a BF16 rounding boundary. Comparing only the BF16 bit patterns of reconstructed pre-update and current weights can therefore miss true updates; rounding-boundary protection covers these coordinates near the boundaries.

The detector first compares the bit patterns of quant⁡BF16(x^)\operatorname{quant}_{\mathrm{BF16}}(\hat{x}) and qq. If they differ, the coordinate is selected for transmission. If they match, the detector further checks the distance from x^\hat{x} to the rounding boundaries and also selects coordinates near a boundary. A coordinate is skipped only when its bit patterns match and x^\hat{x} lies in the protected interior of the rounding cell. The default protection coefficient is ρ=10−4\rho=10^{-4}. Boundaries are determined by the actual neighboring BF16 values of qq, which accounts for unequal spacings at exponent transitions (Section 3.2, Eqs. (3)–(5)).

Every selected coordinate is transmitted with its current true BF16 value qq; reconstructed values are used only for detection. The very few redundant selections therefore also write the correct target values, with their fractions reported in the full-model checks above. All AReaL-DTE experiments reported in the paper use boundary protection; its computation and extra transfers are included in every reported performance and memory result.

3. Theoretical guarantee: reconstruction error is smaller than the boundary clearance, and induction establishes bitwise agreement after every synchronization.

Appendix D.2 bounds the discrepancy between the forward and inverse update increments and the local reconstruction error. Building on these bounds, Theorem 14 and Corollary 15 in Appendix D.3 establish a sufficient condition for the default boundary protection to prevent missed updates. For the AdamW setting analyzed there, the default ρ=10−4\rho=10^{-4} satisfies this condition when ηhλh≤6×10−6\eta_h\lambda_h\le6\times10^{-6} at every step. Here, ηh\eta_h and λh\lambda_h are the learning-rate and weight-decay scalars defined in Appendix D.1.

Let Δ32\Delta_{32} be the local FP32 spacing, and let dsafed_{\mathrm{safe}} be the distance from the reconstructed value to the nearest BF16 rounding boundary at an eligible skipped coordinate. Corollary 15 and Table 10 establish

∣x^−x∣<3.961Δ32<4Δ32≤dsafe.|\hat{x}-x| < 3.961\Delta_{32} < 4\Delta_{32}\le d_{\mathrm{safe}}.

Thus, the true pre-update weight xx remains inside the same BF16 rounding cell as the reconstructed value x^\hat{x}. Together with the matching bit patterns at skipped coordinates, this gives

quant⁡BF16(x)=quant⁡BF16(x^)=quant⁡BF16(y)=q.\operatorname{quant}_{\mathrm{BF16}}(x)=\operatorname{quant}_{\mathrm{BF16}}(\hat{x})=\operatorname{quant}_{\mathrm{BF16}}(y)=q.

Skipped coordinates therefore have no true BF16 change, and no update is missed. The main experiments use a learning rate of 3×10−63\times10^{-6} and weight decay of 0.010.01, giving a product of 3×10−83\times10^{-8}, within this range. This configuration also admits the tighter reconstruction-error bound of less than 3.77Δ323.77\Delta_{32} (Corollary 16).

Appendix D.3 further covers cases where the relevant multiplications in the forward weight update produce subnormal results under the baseline configuration. After accounting for the corresponding rounding errors, the total reconstruction error remains strictly below 4Δ324\Delta_{32}, while boundary protection retains at least 4Δ324\Delta_{32} of boundary clearance, preserving the no-miss guarantee.

Building on the results above, Proposition 19 proves by induction that, starting from the correct initial BF16 version, the receiver matches the sender bit for bit after every synchronization. Each synchronization handles two types of coordinates:

  • Selected coordinates receive the current true BF16 values.
  • Skipped coordinates have unchanged true BF16 values, so retaining the previous values gives the correct current values.

These two cases cover the entire model, so each synchronization provides a correct starting point for the next. Consequently, FP32 reconstruction error does not accumulate through synchronization into receiver-weight drift; this induction argument applies to any number of synchronizations.

Results for different learning rates, batch sizes, and synchronization intervals are reported in Appendix D.6, Table 12, and discussed together in our response to W2 / Q2.

AuthorsOpenReview ↗

Response to W2/Q2 (1/3): Learning-rate and batch-size sensitivity

W2 / Q2: Applicability and performance across learning rates, batch sizes, synchronization intervals, and inference precisions

Thank you for this suggestion. We add sensitivity experiments for the learning rate, training batch size, and synchronization interval in Appendix D.6, Tables 12–13 of the revised manuscript, along with additional FP8 inference experiments. All eight BF16 settings record zero missed updates and zero receiver bitwise mismatches, with throughput exceeding PULSE at all three bandwidths. The FP8 experiments also achieve bitwise agreement in weights and scales, together with end-to-end performance gains.

1. Learning rate and batch size: change rates vary with the settings, while synchronization remains exact with very little redundancy.

The experiments use Qwen3-8B / GSM8K, GRPO, AdamW, and BF16 inference weights. The baseline uses a target learning rate of 3×10−63\times10^{-6}, weight decay of 0.010.01, a global training batch size of 16 rollout response sequences (8 prompts with 2 sampled responses each), and synchronization interval K=1K=1, meaning synchronization after every parameter update. Each variant changes only one factor, and all use rounding-boundary protection with the default ρ=10−4\rho=10^{-4}. Within each setting, AReaL-DTE and PULSE use the same experimental configuration. Each of the six settings with K=1K=1 runs for 40 training steps.

The following table corresponds to Appendix D.6, Table 12. After every actual synchronization, receiver bitwise checks cover all 8,190,735,3608{,}190{,}735{,}360 weight elements of Qwen3-8B, including coordinates not selected for transmission.

SettingTrue BF16 change rateSelection rate after protectionRedundancy among selected entriesMissed BF16 updatesReceiver bitwise mismatches
Baseline1.536342%1.538025%0.109426%00
LR =10−6=10^{-6}0.627989%0.628672%0.108642%00
LR =6×10−6=6\times10^{-6}2.680988%2.683717%0.101687%00
LR =1.2×10−5=1.2\times10^{-5}3.414324%3.418514%0.122568%00
Batch =8=81.411514%1.413064%0.109691%00
Batch =32=321.472879%1.474526%0.111697%00
Synchronization interval K=2K=22.408263%2.437073%1.182156%00
Synchronization interval K=4K=43.470902%3.557525%2.434923%00

Note: True changes and redundancy are measured between the current version and the preceding synchronized version. Change rates and selection rates use all checked model weights as the denominator; redundancy fractions use selected entries after protection. Percentages are calculated from aggregated counts. The last two columns are zero at every check. This baseline and the 200-step check in W1/Q1 use different measurement windows, accounting for their different average change rates.

The true BF16 change rate increases with learning rate. At the baseline learning rate, halving or doubling batch size keeps the change rate close to the baseline. Under standard synchronization at every step (K=1K=1), total redundancy across all six settings accounts for only approximately 0.1% of selected entries (0.101687%–0.122568%).

The original paper's observation of a change rate below 2% comes from preliminary experiments with learning rates of 10−610^{-6} and 3×10−63\times10^{-6}. A larger learning rate can cause policy drift to exceed the prescribed trajectory KL budget. Appendix A, Theorem 1, Eq. (12) derives a learning-rate upper bound from this budget, providing a theoretical basis for controlling policy drift through smaller learning rates. Moreover, these experiments show that AReaL-DTE maintains exact synchronization and performance benefits even when increasing the learning rate raises the change rate above 2%.

All settings use weight decay of 0.010.01. The largest learning rate gives ηhλh=1.2×10−7<6×10−6\eta_h\lambda_h=1.2\times10^{-7} < 6\times10^{-6}, within the joint parameter range in Table 10. Here, ηh\eta_h and λh\lambda_h are the learning-rate and weight-decay scalars defined in Appendix D.1; the corresponding correctness analysis is summarized in our response to W1/Q1.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Rebuttal (1/2)

We thank the reviewers for evaluating AReaL-DTE. Our work addresses weight-synchronization overhead in RL training by combining on-demand reconstruction of pre-update weights, streamed layout conversion, and sparse delta transmission, which reduces memory requirements for historical weight snapshots and complete intermediate model representations, as well as end-to-end synchronization overhead. In response to the reviewers' main questions, we added synchronization correctness proofs, full-model bitwise checks, an optimized snapshot baseline, and experiments covering different training settings, FP8 inference, and multi-turn tool interaction. We highlight the key issues we addressed during rebuttal below.

1. Synchronization correctness and error accumulation: theoretical analysis and full-model checks over 200 consecutive steps jointly support exact synchronization.
Address HCR4 W1/Q1, j5BH W3, and kjXN W2/Q2.

Section 3.2 of the revised paper specifies rounding-boundary protection: BF16 bit comparison and rounding-boundary checks jointly determine the selected set. All selected coordinates carry current true BF16 values; reconstructed values are used only for detection. Appendix D establishes reconstruction error bounds and sufficient conditions for preventing missed updates for the AdamW setting analyzed there. It further proves by induction that the receiver and sender are bitwise identical after every synchronization, explaining why FP32 reconstruction error does not accumulate through synchronization into receiver weight drift.

For the AdamW setting analyzed in Appendix D, the default protection coefficient ρ=10−4\rho=10^{-4} satisfies the sufficient condition for preventing missed updates when ηhλh≤6×10−6\eta_h\lambda_h\le6\times10^{-6} at every step. Here, ηh\eta_h and λh\lambda_h are the learning-rate and weight-decay scalars defined in Appendix D.1, and ρ\rho is the relative width of rounding-boundary protection. The local reconstruction error is then below 3.9613.961 FP32 spacings, whereas eligible skipped coordinates retain at least 44 spacings of boundary clearance. The main-paper experiments use a learning rate of 3×10−63\times10^{-6} and weight decay of 0.010.01, within this joint parameter range. Appendix D.3 additionally covers subnormal results from the relevant forward multiplications under the baseline configuration; accounting for the associated rounding errors preserves the no-missed-update guarantee.

In the 200-step continuous training and synchronization audit on Qwen3-8B / GSM8K, the receiver continuously maintains the same actual model weights, and every check covers all 8,190,735,3608{,}190{,}735{,}360 weight elements. Complete reference weights are used only for independent comparison and do not overwrite or refresh the receiver model. The maximum and mean absolute FP32 reconstruction errors are 9.53674×10−79.53674\times10^{-7} and 1.76757×10−91.76757\times10^{-9}, respectively. Boundary protection recovers an average of 20,94120{,}941 true BF16 updates per detection that bit comparison alone would miss, with zero missed updates and zero receiver bitwise mismatches across all 200 synchronizations. Total redundancy accounts for only 0.109700% of selected entries (approximately 0.1%), or 0.001631% of all model weight elements (Appendix D.5, Table 11).

All AReaL-DTE experiments reported in the paper and this rebuttal use rounding-boundary protection. Its computation and additional transfers are included in the corresponding performance and memory results.

AuthorsOpenReview ↗

Summary of the Rebuttal (2/2)

2. Optimized baseline and attribution of gains: throughput and CPU-memory benefits persist over a streamed snapshot baseline.
Address HCR4 W3/Q3, j5BH W2, and 7WTE W1/Q1.

To address whether the snapshot baseline is sufficiently optimized, we add Streamed PULSE, which uses 64 MiB chunks, pinned memory, double buffering, and a bounded queue to overlap GPU-to-CPU transfers with CPU comparison. Timing includes snapshot maintenance. Methods are compared using the same experimental configuration.

Streaming reduces PULSE's peak sender GPU memory from 68.5 GB to 53.2 GB. Compared with this optimized baseline, AReaL-DTE still achieves 1.14×1.14\times, 1.27×1.27\times, and 1.41×1.41\times end-to-end throughput at 0.10.1, 11, and 1010 Gbps, respectively. Peak sender CPU memory decreases from 36.3 GB to 18.7 GB, a reduction of approximately 48.5%, while peak GPU memory is similar (52.4 GB versus 53.2 GB).

We also provide per-step timing breakdowns for Qwen3-8B and Qwen3-30B-A3B across four benchmarks. Under matching configurations, rollout, training, and total non-synchronization times are similar across methods, while AReaL-DTE has the lowest weight-synchronization time in all eight configurations. This shows that end-to-end gains primarily come from reduced overhead in the weight-synchronization stage. These comparisons and breakdowns appear in the corresponding reviewer responses; synchronization times in the paper are reported in Table 3(a) and Appendix B.2, Table 8(a).

3. Applicability and practical workloads: benefits persist with higher change rates, FP8 inference, and multi-turn tool interaction.
Address HCR4 W2/Q2, j5BH W1, kjXN W1/Q1, and 7WTE W1/Q2 and W2/Q3.

Appendix D.6, Tables 12–13 add eight learning-rate, batch-size, and synchronization-interval settings on Qwen3-8B. Every check in all settings records zero missed updates and zero receiver bitwise mismatches, and throughput exceeds PULSE at all three bandwidths. Under standard per-step synchronization (K=1K=1, meaning synchronization after every parameter update), redundancy accounts for only approximately 0.1% of selected entries across all six settings. Even when the learning rate is increased to four times the baseline and the true BF16 change rate reaches 3.414324%, AReaL-DTE achieves 1.50×1.50\times the throughput of PULSE at 1010 Gbps, showing that performance benefits do not require the change rate to remain below 2%.

The additional Qwen3-8B FP8 inference experiments use FP8 E4M3FN linear-layer weights with FP32 scaling factors (scales), while the other relevant parameters use BF16. All 40 steps record zero missed updates and zero receiver bitwise mismatches in FP8 weights, scales, and BF16 parameters. Across all model weights, redundancy accounts for only approximately 0.048% of selected entries. At 0.10.1, 11, and 1010 Gbps, AReaL-DTE achieves 1.46×1.46\times, 1.65×1.65\times, and 1.67×1.67\times the throughput of FP8 PULSE, respectively.

We have also added Llama-3.1-8B experiments in our supplementary responses. All BF16 and FP8 synchronization checks record zero missed updates and zero receiver bitwise mismatches. Across all tested settings and all three bandwidths, AReaL-DTE achieves higher measured end-to-end throughput than PULSE, reaching up to 2.54×2.54\times in BF16 and 1.69×1.69\times in FP8.

In the Hermes agent / DAPO-Math multi-turn tool-interaction experiment with Qwen3-30B-A3B, trajectories include actual Python execution and feedback, with observed maxima of 22 model interactions and 46,176 newly generated tokens per trajectory. AReaL-DTE achieves 227.199 tokens/s, or 1.107×1.107\times and 13.94×13.94\times the throughput of PULSE and ByteCheckpoint, respectively, validating the benefits of synchronization optimization in this practical tool-interaction workload.

We additionally check synchronization correctness for Adam under different weight-decay configurations, all with zero missed updates and zero receiver mismatches. We also add full-model and per-layer measurements of actual rounding-position distributions, with the full-model distribution shown in Appendix A, Figure 5. These measurements provide empirical support for the uniform-position approximation used in the sparsity analysis.

These additions support AReaL-DTE's core contribution through numerical correctness, comparisons against an optimized baseline, and a range of practical training settings: on-demand reconstruction and delta synchronization reduce synchronization time and memory overhead in RL training while preserving inference-weight synchronization correctness. We respectfully ask that you consider the revised paper and the additional evidence in our point-by-point responses when evaluating this work. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.