acceptodds
Desk rejected at ICLR 2027

CoSR: Memory-Efficient FP8 Training for Multitask Reinforcement Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

FP8 reduces compute and memory costs in large-scale deep learning, but bootstrapping, nonstationary targets, and closed-loop feedback between policies and data distributions make deep reinforcement learning sensitive to low-precision perturbations. High-capacity multitask critics further increase training-state memory costs. We find that deterministic writeback in FP8-resident online actor–critic training can suppress updates smaller than the quantization step, degrading long-term performance. Independent stochastic rounding (SR) preserves these updates in expectation but ignores error interactions in forward and backward matrix operations and Adam state updates, allowing locally unbiased noise to accumulate into substantial optimization perturbations. We propose CoSR, an operator-aware joint SR framework. CoSR uses forward/backward channel correlations and the coupled effects of Adam's first and second moments on update directions to coordinate correlations among rounding errors while preserving elementwise rounding probabilities and unbiasedness. It requires no per-parameter compensation buffers, reducing persistent-state memory overhead; major weight matrices and Adam states reside in FP8. Across diverse reinforcement-learning benchmarks, including DeepMind Control Suite and Meta-World, CoSR achieves performance comparable to the high-precision baseline while reducing training-state memory and peak GPU memory by 74% and 63%, respectively. These savings enable wider critics, yielding substantial multitask performance gains under the same hardware budget.

Primary area:
reinforcement learning
Keywords:
Low-Precision Training, Deep Reinforcement Learning, FP8, Memory-Efficient Training, Multitask Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dTcdOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a novel method based on an operator-aware joint stochastic rounding (SR) framework. Unlike independent SR, it jointly considers the impact of rounding on computations involving activations and gradients, which leads to optimal parameter-rounding couplings within a fixed 2×22 \times 2 block partition with independence across blocks. Besides, the Adam optimizer's first and second moments, mm and vv, are jointly rounded to minimize the linearized error in Adam's normalized update. Finally, the LAG method stores the difference between the target and online critic parameters. Its recurrence is equivalent to the standard soft update before quantization, with an explicit characterization of the quantization error, and it improves performance in the experiments.

Critical strengths
  • The paper proposes a rounding method with theoretical guarantees that also works well in experiments. The analysis provides useful insight into how coordinating rounding errors can reduce perturbations in subsequent computations.
  • The experiments are extensive, especially the ablation study covering all 232^3 combinations of weight rounding, Adam-state rounding, and LAG. This helps distinguish the contributions of the individual modules and their combinations.
Critical weaknesses

None.

Questions for authors
  • Do you have any insights into more general rounding designs for weights with theoretical guarantees? The 2×22 \times 2 block-wise construction is interesting and novel, and I wonder whether the guarantees could extend to larger blocks or more general coupling structures. I see this as a possible extension rather than a requirement for the current result.
  • In Figure 1(a), RTN update retention is above zero at an x-coordinate below half the local FP8 gap, which I find confusing. Under a fixed quantization grid, an already-representable weight should remain unchanged for updates smaller than half the gap in the update direction. Could you clarify how retention is defined and whether this point represents a bin that crosses the half-gap threshold, or whether the scale changes during writeback?
  • The LAG analysis establishes equivalence to the standard soft update before quantization and characterizes the quantization error, but does not show that LAG is more accurate than storing absolute target parameters. Are there conditions under which such an advantage can be established theoretically?
  • The additional update-time overhead relative to independent SR is noticeable: approximately 18.4% according to Table 13. Could you clarify which components account for this overhead and whether it could be reduced through better implementation, such as kernel fusion?

Nice work, congrats!

3 comments

Dear Reviewer, thank you for recognizing our work. We address your concerns below, with W denoting the weaknesses and Q the questions you raised.

Q1: More general weight-rounding designs and theoretical guarantees

Thank you for suggesting this extension. The current construction can be generalized in both channel pairing and within-block coupling.

Proposition 1 applies to any predetermined, non-overlapping input- and output-channel pairings. Using the actual forward and backward inner products of these channels, the original XOR construction still simultaneously attains the minimum forward and backward risks under the specified 2×22\times2 partition.

A 2×22\times2 block is the smallest rectangle containing both forward same-column and backward same-row pairs. The two vertical edges share a forward correlation coefficient, and the two horizontal edges share a backward correlation coefficient. Thus, each block needs only one uniform random draw and two inner-product sign checks to achieve optimal coupling on all four edges through row and column flips, making it our first choice.

Larger k×kk\times k blocks can also share one U∼U(0,1)U\sim\mathcal U(0,1), using UU or 1−U1-U according to the XOR of row and column flip labels to preserve each element's SR marginal. Forward and backward risks depend only on row and column flips, respectively. Optimizing each set of flips separately attains both risk minima simultaneously within this construction family. Compared with the two sign checks for 2×22\times2 blocks, larger blocks must account for multiple channel-pair relationships. Enumeration requires comparing 2k−12^{k-1} flip patterns per channel group, creating a computation–error trade-off. We have added this generalization and its proof to Appendix C.2 of the revised manuscript.

We compared block sizes over 5 random seeds and report end-to-end time including environment sampling, evaluation, and updates.

PairingFinal success (%)End-to-end time (h)
2×22\times2 (default)90.7 [89.2, 92.5]15.85
4×44\times490.3 [88.4, 91.7]18.60
8×88\times891.2 [90.3, 92.8]20.56

The results show low sensitivity to block size. The 8×8 setting achieves the highest success rate but requires coordinating more error relationships, increasing runtime by 29.72%. The 2×2 setting has a simple analytical construction, so we adopt 2×2 as the default to balance performance and implementation complexity.

We also compared random pairing and fixed correlation-based pairing on MT50. Random pairs are determined at initialization and then fixed. Correlation-based pairs are greedily matched using absolute normalized correlations from the forward activations and backward gradients of the first training batch, then fixed. Both retain the joint rounding rule within each 2×22\times2 block and use 5 random seeds.

PairingFinal success (%)End-to-end time (h)
Adjacent (default)90.7 [89.2, 92.5]15.85
Random88.3 [86.5, 91.4]16.76
Correlation-based90.1 [88.4, 91.5]16.54

The results show that CoSR supports different pairings. Default adjacent pairing achieves the highest success rate and shortest end-to-end time while avoiding additional correlation matching and channel reordering, balancing performance and cost.

Q2: Clarification of RTN in Figure 1(a)

Your understanding is correct. We define

RS=∑i∈SΔi,E![Δ^i]∑i∈SΔi2.R_{\mathcal S}=\frac{\sum_{i\in\mathcal S}\Delta_i\\,\mathbb E\\!\left[\widehat{\Delta}_i\right]}{\sum_{i\in\mathcal S}\Delta_i^2}.

Here, S\mathcal S is the set of parameter indices in the bin, Δi=zi−wi\Delta_i=z_i-w_i is the intended update, and Δ^i=w^i−wi\widehat{\Delta}_i=\widehat w_i-w_i is the actual writeback update. The expectation is over rounding randomness. RR measures how much of the actual update is retained along the intended direction: 1 indicates full retention of the contribution in that direction, and 0 indicates zero net contribution.

The point you identified corresponds to the [0.1,1)[0.1,1) bin, which includes updates exceeding the half-gap threshold, giving an aggregate R>0R>0. When strictly selecting updates smaller than half the gap, we confirmed that under RTN, the actor and critic have RR values of approximately 0.00012 and −0.00004, respectively, both near zero. These small residuals arise from grid shifts caused by dynamic scaling during writeback.

Thank you for identifying this issue. In the revised manuscript, we use 0.5 as a bin boundary to produce the figure again. In the new [0.5,1)[0.5,1) bin, RTN can round a sub-gap update to a full grid step, so R>1R>1 indicates amplification along the intended update direction.

Q3: Accuracy advantage of LAG over direct storage

Under specific conditions, LAG can be proven to have an accuracy advantage.

Let the target candidate for the current soft update be yy, the online parameters be θ\theta, and the corresponding lag be ℓ=y−θ\ell=y-\theta. Direct storage quantizes yy, whereas LAG quantizes ℓ\ell and reconstructs the target as θ+Q(ℓ)\theta+Q(\ell). Both use the same FP8 format and RTN, with their respective maximum-based scaling. Assuming no overflow and ignoring rounding in FP32 arithmetic,

\le c_{\mathrm{FP8}}\\|x\\|_\infty,$$ where $c_{\mathrm{FP8}}$ depends only on the numerical format. This bound follows directly from the half-gap error bound of RTN and maximum-based scaling. Thus, $$\\|e_{\mathrm{LAG}}\\|_\infty \le c_{\mathrm{FP8}}\\|\ell\\|_\infty, \qquad \\|e_{\mathrm{direct}}\\|_\infty \le c_{\mathrm{FP8}}\\|y\\|_\infty.$$ When the target and online networks are sufficiently close that $$\\|\ell\\|_\infty\le\rho\\|y\\|_\infty, \qquad 0\le\rho < 1,$$ **the above LAG error bound is at most $\rho$ times the corresponding bound for direct storage. Its accuracy advantage comes from matching the quantization scale to the smaller range of differences**. This analysis also connects to long-term target tracking. Along the same online parameter sequence, let $d_t$ denote the error of the reconstructed target relative to the exact soft update. From Appendix C.4, $$d_{t+1}=(1-\tau)d_t+\varepsilon_{t+1}.$$ If $d_0=0$ and the quantization error at each step satisfies $\\|\varepsilon_t\\|_\infty\le\bar\varepsilon$, then $$\\|d_t\\|_\infty \le \bar\varepsilon\\, \frac{1-(1-\tau)^t}{\tau}.$$ Thus, when LAG maintains a smaller uniform per-step error bound throughout training, its long-term target-tracking error bound also decreases accordingly. The experiments also corroborate this result. In Table 1, replacing direct storage of target parameters with LAG while keeping all other modules unchanged **increases MT50 success from 83.7% to 90.7%**. We have added this analysis and its conditions to Appendix C.4 of the revised manuscript. ## Q4: Additional update overhead over independent SR and optimization opportunities In Table 13, update times for CoSR and independent SR are **23.77 ms and 20.08 ms**, respectively, a difference of **3.69 ms, or 18.4%**. Relative to independent SR, CoSR adds forward and backward channel inner-product statistics, coordinated generation of weight-rounding randomness, and joint encoding of Adam's first and second moments. It also uses LAG for target-state reconstruction and writeback. These computations use the current forward and backward operands and update candidates. For a $d_{\mathrm{in}}\times d_{\mathrm{out}}$ weight matrix, weight coordination requires $d_{\mathrm{in}}/2+d_{\mathrm{out}}/2$ sign statistics, which can be shared across channels. We therefore tried combining channel-statistic reductions with the computations producing the corresponding operands, and fusing joint rounding with quantization encoding and state writeback to reduce extra memory accesses and kernel launches. **The optimized parameter update time is 21.48 ms, only 6.97% higher than independent SR**, providing stronger support for CoSR. ### Thank you again for your constructive review and valuable guidance, which helped us further optimize CoSR !
AuthorsOpenReview ↗

A Gentle Follow-Up

Dear Reviewer:

Thank you again for your thoughtful review and for your encouraging assessment of our work! If you have a moment, we would be grateful to hear whether our responses have addressed your questions and whether they have given you greater confidence in your assessment.

Best wishes!

Reviewer_NpVmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies FP8-resident online actor-critic training and proposed coordinated stochastic rounding of weights and Adam moments. It uses weight rounding, which designs forward and backward channel correlations to couple errors within 2-by-2 blocks, preserving scalar SR marginals while minimizing two local perturbation risks. It designs moment rounding to minimize a coordinate-wise linearized Adam-direction error. A relative-lag representation additionally improves target-network storage.

The method is evaluated on MT50, DMC-Hard, and HumanoidBench-Medium. The method achieves FP32 scores with significantly reduced memory.

Critical strengths
  • The paper presents clear motivation and problem definition. The problem is important: how to mitigate the negative effects of FP8 quantization in RL.

  • The proposed method is simple yet effective. The quantized version achieves scores very close to those of FP32.

Critical weaknesses
  • The method can be evaluated on more settings, for example, single-task or other RL settings.

  • While CoSR reduces peak GPU memory and per-update latency, the paper does not report end-to-end RL training time, including environment interaction and evaluation. Consequently, it remains unclear how much the update-level speedup translates into overall training acceleration, particularly when the saved memory is used to train a wider critic.

Questions for authors
  • Can CoSR’s be used for single-task training? Does task diversity create some additional challenges?

  • How were the original-precision baselines in Table 12 validated and tuned? What explains FlashOptim’s low success and unusually high peak memory, and how does conventional mixed-BF16 BRC compare?

5 comments

Dear Reviewer, thank you for your insightful review. We respond below, and the revisions have been included in the revised PDF. W denotes the weaknesses and Q the questions you raised.

W1, Q1: Single-task settings, other reinforcement learning settings, and the effects of task diversity

CoSR can be used effectively across settings, including single-task learning, different RL algorithms (PPO and GRPO), model-based settings, and even agentic RL.

First, theoretically, CoSR's weight rounding depends only on the current linear layer's forward activation and backward gradient statistics. Adam-state rounding depends on the coupled effects of the first and second moments on the update direction. These operations and their theoretical guarantees also apply to other task settings.

Second, task diversity does pose challenges. Differences in activations, gradient magnitudes, and channel correlations across tasks may cause some correlations to cancel in mixed batches, reducing the gains. At each step, CoSR determines the coupling from the current batch's actual statistics to optimize its total perturbation risk. Our experiments cover MT50, DMC-Hard, HumanoidBench-Medium, and joint training on 165 distinct tasks spanning manipulation, locomotion, and dexterous hand control, providing empirical support in heterogeneous multitask settings.

We added five single-task comparisons within BRC, covering MetaWorld manipulation tasks and HumanoidBench locomotion tasks. The table reports means and 95% bootstrap confidence intervals over 5 random seeds.

TaskMetricBRC-FP32CoSR-BRC
PushSuccess rate (%)94.0 [90.0, 98.0]90.0 [90.0, 90.0]
Door OpenSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
Peg Insert SideSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
WalkEpisode return854.41 [851.13, 856.21]825.57 [822.03, 826.91]
StairEpisode return283.88 [159.07, 447.34]358.52 [222.84, 482.28]

The results show performance comparable to full precision, demonstrating CoSR's effectiveness in single-task settings.

For other RL settings, we integrated CoSR into model-based TD-MPC2 and evaluated it on five individual tasks from MetaWorld and DMC. The table reports means and 95% bootstrap confidence intervals over 5 random seeds.

TaskMetricTD-MPC2-FP32CoSR-TD-MPC2
ReachSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
PushSuccess rate (%)90.0 [70.0, 100.0]100.0 [100.0, 100.0]
Cartpole SwingupEpisode return876.07 [866.00, 882.00]860.00 [851.70, 865.78]
Finger SpinEpisode return984.63 [982.10, 989.10]990.00 [986.90, 994.25]
Cheetah RunEpisode return757.60 [593.80, 856.90]760.00 [596.14, 843.45]

CoSR achieves success rates comparable to full precision on all five tasks, demonstrating its effectiveness in model-based RL.

We also completed single-task PPO experiments. Averaged over 5 random seeds, CoSR achieves results comparable to full-precision PPO, demonstrating that its improvement is not limited to one framework.

TaskMetricPPO FP32PPO CoSR
PushSuccess rate (%)100.0 [100.0, 100.0]96.0 [92.0, 100.0]
Door OpenSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
Peg Insert SideSuccess rate (%)92.0 [90.0, 96.0]94.0 [90.0, 98.0]

We also tested CoSR in agentic RL. Compressing the model's MLP layers to FP8 and applying CoSR reduced GPU memory usage during training by 47.13%-58.56%. Using GRPO, we trained Qwen2.5-0.5B-Instruct on Sokoban for 200 iterations and Qwen2.5-3B-Instruct on WebShop for 100 iterations. The tables report mean results over 5 random seeds and GPU memory usage:

Sokoban on Qwen2.5-0.5B-Instruct

MethodSuccess rate (%)Training memory (GiB)
Full precision36.39 [26.95, 48.44]5.521
CoSR44.82 [37.85, 52.11]2.919

WebShop on Qwen2.5-3B-Instruct

MethodSuccess rate (%)Training memory (GiB)
Full precision71.86 [66.95, 78.63]34.488
CoSR74.29 [68.27, 78.44]14.292

The results show that CoSR matches full-precision success rates, demonstrating its generalizability.

We observed an interesting pattern: adding rounding to LLM training improves performance over the original precision, with a larger advantage for the 0.5B model than for the 3B model. Our preliminary investigation suggests implicit regularization from stochastic rounding as one possible explanation. Ozkara et al. (AISTATS 2025) [1] show in their gradient-descent analysis that SR introduces an implicit regularization term related to quantization gaps and gradients, and report better performance than mixed-precision baselines in LLM pretraining. CoSR preserves elementwise SR distributions while coordinating weight and Adam-state rounding errors to reduce forward and backward perturbations and local update distortion. This suggests that CoSR may retain the potential regularization benefits of stochastic perturbations while reducing their interference with learning signals, thereby improving final task performance. We hope to investigate this further in future work.

[1] Kaan Ozkara, Tao Yu, and Youngsuk Park. Stochastic rounding for llm training: Theory and practice. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan, editors, Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 4402–4410. PMLR, 03–05 May 2025.

W2: End-to-end training time

We agree that end-to-end time, including environment sampling, evaluation, and updates, better reflects practical speedups. The table reports these times and the BF16 mixed-precision BRC results on MT50 requested in your next question. All four settings were run independently on a single PRO 6000 Blackwell GPU, with results averaged over 5 random seeds.

MethodSuccess rate (%)Persistent state (GiB)Peak GPU memory (GiB)End-to-end time (h)
BRC FP3290.0 [87.9, 91.8]2.0304.68021.39
BRC BF1689.7 [88.5, 90.3]2.0254.63717.58
SR68.8 [66.5, 70.6]0.5301.75013.37
CoSR90.7 [89.2, 92.5]0.5301.74015.85

These results show that:

  • BF16 completes training faster with comparable success rates but retains high GPU memory usage.
  • SR performs well in GPU memory usage and speed, but it ignores correlated errors and cannot match full-precision success rates.
  • CoSR simultaneously balances success rate, GPU memory usage, and computational performance.

For the wider critic, we also evaluated 3 settings over 5 random seeds. End-to-end times are reported below:

SettingMT50 scoreDMC scoreShadowHand scorePersistent state (GiB)Peak GPU memory (GiB)End-to-end time (h)
BRC FP32(8192)0.89 [0.87, 0.91]0.64 [0.63, 0.65]0.84 [0.82, 0.85]8.0612.5781.7
SR(8192)0.72 [0.71, 0.74]0.49 [0.47, 0.51]0.69 [0.67, 0.70]2.034.6543.8
CoSR(8192)0.88 [0.87, 0.90]0.65 [0.63, 0.67]0.83 [0.81, 0.84]2.034.6552.4

The results demonstrate that with a wider critic, network computation and parameter updates account for a larger share of runtime. CoSR's efficiency gains in these stages translate more fully into end-to-end training speedups, while achieving comparable performance with significantly less GPU memory than the full-precision baseline.

Q2: Baseline validation and tuning, FlashOptim performance, and BF16 mixed-precision BRC

We adapted each method's numerical representation and specific update mechanism to a common BRC framework. For methods with public implementations, we checked our implementations against the official code. We found no official open-source code for ECO or signed-SR, so we implemented them from the algorithms and equations in their papers.

We validated implementations at two levels. First, we used test values to check quantization, dequantization, error compensation, optimizer-state recurrences, and boundary cases. Second, during actual multitask training, we inspected the parameter storage types and ran consecutive updates for further validation.

All methods share the network architecture, number of training steps, and BRC training hyperparameters. Method-specific parameters follow official code defaults or the paper settings when author code is unavailable. The parameters are listed below.

FP16-RL

ParameterValue
init_scale, soft_update_scale10000
increase_every10000
margin1.1
min_eps1e-7
adam_eps1e-4
tanh_threshold10

FlashOptim

ParameterValue
master_weight_bits24
group_size32
quantize, fusedTrue

MS-AMP O2

ParameterValue
DEFAULT_WINDOW_SIZE16
DEFAULT_WGRAD_WINDOW_SIZE1
tensor_scaleTrue

COAT

ParameterValue
qgroup_size128
group_size16
expand_min16
first_order_expansion, second_order_expansionexpand

CQ

ParameterValue
num_levels2
num_clients100
thresholds[0.0, 1.0]
update_thresholdsTrue
rotateFalse

FlashOptim approximates high-precision weights using low-precision master weights and integer correction codes of limited bit width, and compresses Adam's first and second moments to 8 bits. With task distribution differences, group-wise scaling subjects smaller moment components to greater relative quantization perturbations, while the correction codes cannot retain sufficiently small weight updates. These perturbations repeatedly propagate through the critic's bootstrapped targets and policy updates, affecting stability and final performance. This can explain its lower success rate.

Its peak GPU memory mainly comes from the many high-precision intermediate tensors produced by logical-weight reconstruction, optimizer-state decoding, and post-update re-encoding. The INT32 index buffers required to locate adjacent quantization grid points further increase memory usage.

Through kernel fusion, equivalent table lookups, and related optimizations, we reduced peak GPU memory for the FlashOptim baseline at its original precision to 8.67 GiB, still substantially higher than other methods. Considering update latency, peak GPU memory, and task performance, FlashOptim still has room for further adaptation to multitask RL in actor-critic settings.

We added BF16 mixed-precision BRC results on MT50. Dense operations in the actor, critic, and target critic use BF16, with outputs cast back to FP32.

The experiments use the same training budget as the main experiments and run independently on a single PRO 6000 Blackwell GPU. Results over 5 random seeds on MT50 are reported below.

MethodSuccess rate (%)Persistent state (GiB)Peak GPU memory (GiB)Update latency (ms)End-to-end time (h)
BRC BF1689.7 [88.5, 90.3]2.0254.63725.8717.58

The detailed analysis is provided in the preceding response.

Thank you for your very helpful review. We welcome further questions if any concerns remain. If our responses have addressed your concerns, we would be grateful if you could reconsider your assessment. Thank you again for your time and thoughtful feedback.

Reviewer_k45eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes CoSR, a coordinated stochastic rounding method for memory-efficient FP8 training in online multitask RL. The main idea is to exploit correlations between rounding errors rather than treating them independently. Specifically, CoSR uses forward activations and backward gradients to coordinate weight rounding, jointly rounds Adam's first and second moments to reduce update distortion, and introduces a relative lag representation for target network updates. Experiments on MetaWorld, DMC, and HumanoidBench show performance comparable to FP32 training with substantially lower memory usage.

Critical strengths
  • Interesting technical idea. The paper identifies a limitation of independent stochastic rounding: unbiased rounding at the parameter level does not necessarily lead to small errors in forward/backward computations. Coordinating rounding errors based on their effects on downstream operations is a reasonable and interesting idea.
  • Solid theoretical motivation. The authors provide theoretical results showing that their coupling strategy minimizes local forward/backward perturbation risks and a linearized Adam update error under certain assumptions. These results help justify the proposed design.
  • Promising empirical results. CoSR achieves performance close to the FP32 baseline while significantly reducing training-state memory and peak GPU memory. The experiments with wider critics under a fixed memory budget also demonstrate a potential practical benefit.
Critical weaknesses
  • Insufficient justification for adjacent-channel pairing. CoSR only considers correlations between adjacent channels, but the paper does not explain why adjacent channels should be more correlated than other channel pairs. The theoretical guarantees also assume a fixed 2×2 block partition. It is unclear whether this particular pairing strategy is necessary or even preferable to random pairing or correlation-based pairing.
  • Missing ablations on the rounding strategy. While the authors provide ablations for the three main components, I would like to see more analysis of the design choices within CoSR, especially the channel pairing strategy and block size. These choices are important to the proposed method but are not sufficiently examined.
  • Limited generality across RL algorithms. All experiments are built on BRC. Although the paper evaluates several benchmarks, it is unclear whether the method would work equally well with other commonly used RL algorithms or training architectures, especially the model-based category like TD-MPC2[1].
  • Limited scope of the experimental evaluation. The experiments focus on conventional continuous-control and multitask RL benchmarks. Given the growing importance of large-scale agentic RL, it would be interesting to understand whether the proposed method remains useful in those settings. This limits the broader impact of the work in its current form.
  • Unclear end-to-end efficiency gains. The paper reports improvements in memory usage and update latency, but the latency measurements exclude environment interaction and evaluation. Since CoSR introduces additional computations for coordinated rounding, it would be helpful to report end-to-end training time and clarify the actual efficiency gains over independent SR.

[1] Hansen, Nick, Hao Su, and Xiaolong Wang. "Td-mpc2: Scalable, robust world models for continuous control." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors
  • Why adjacent channels? Is there any empirical evidence that adjacent channels have stronger correlations than randomly paired channels? How would CoSR perform with random fixed pairing or correlation-based pairing?

  • How sensitive is CoSR to the block size? The current method uses fixed 2×2 blocks. Have the authors considered larger blocks or alternative coupling structures? Is there a trade-off between computational overhead and error reduction?

  • How well does CoSR generalize beyond BRC? Can the authors provide evidence or discussion of its applicability to other online RL algorithms, such as PPO or other actor-critic methods?

  • What are the end-to-end computational benefits? Can the authors report overall training wall-clock time, including environment interaction, and compare it against FP32 and independent SR? This would help clarify the practical value of the proposed method.

5 comments

Dear Reviewer, thank you for your careful review and the perspectives your comments have offered. We respond in order below, with W denoting the weaknesses and Q the questions you raised.

W1, Q1: Rationale for adjacent-channel pairing and alternatives

We choose adjacent channels primarily to limit the cost of computing statistics and executing the method. Fixed, contiguous channel groups allow direct inner-product computation and batched generation of random numbers for rounding while avoiding overhead from pairing searches and index management.

Theory

The theoretical guarantee in Proposition 1 applies to any predetermined row and column pairings. Specifically, let the input-channel pairs be Ir=ir,jrI_r=\\{i_r,j_r\\} and the output-channel pairs be Jc=uc,vcJ_c=\\{u_c,v_c\\}, and use

\qquad b_c=\langle G_{:,u_c},G_{:,v_c}\rangle.$$ Each block $I_r\times J_c$ of size $2\times2$ still has two vertical edges with coefficient $a_r$ and two horizontal edges with coefficient $b_c$. The original XOR construction simultaneously attains the covariance extrema required by these edges, so the proof in Appendix C.2 applies directly to arbitrary channels. Thus, **random pairing and pairings predetermined by correlation also retain bidirectional optimality guarantees under their respective partitions**. ### Experiments We compared fixed random pairing and fixed correlation-based pairing on MT50. Random pairs are determined at initialization and then fixed. Correlation-based pairs are greedily matched using absolute normalized correlations from the forward activations and backward gradients of the first training batch, then fixed. The table reports means over 5 random seeds. End-to-end time includes environment sampling, evaluation, and updates. | Pairing | Final success (%) | End-to-end time (h) | | --- | --- | --- | | Adjacent (default) | 90.7 [89.2, 92.5] | 15.85 | | Random | 88.3 [86.5, 91.4] | 16.76 | | Correlation-based | 90.1 [88.4, 91.5] | 16.54 | The results show that CoSR supports different pairings. Default adjacent pairing achieves the highest success rate and shortest end-to-end time while avoiding additional correlation matching and channel reordering, balancing performance and cost. ## W2, Q2: Block size and the computation–error trade-off A $2\times2$ block is the smallest rectangle containing both forward same-column and backward same-row pairs. **The vertical and horizontal edges share forward and backward correlation coefficients, respectively, allowing row and column flips to satisfy the optimal coupling requirements of all four edges simultaneously. Each block needs only one uniform random draw and two sign checks to obtain the bidirectional optimality guarantee of Proposition 1**. Larger blocks can account for more channel relationships but must also address their compatibility. To evaluate sensitivity to block size, we extended the construction to $4\times4$ and $8\times8$ blocks. The table reports means over 5 random seeds. | Pairing | Final success (%) | End-to-end time (h) | | --- | --- | --- | | $2\times2$ (default) | 90.7 [89.2, 92.5] | 15.85 | | $4\times4$ | 90.3 [88.4, 91.7] | 18.60 | | $8\times8$ | 91.2 [90.3, 92.8] | 20.56 | The results show low sensitivity to block size. The 8×8 setting achieves the highest success rate but requires coordinating more error relationships, increasing runtime by 29.72%. The 2×2 setting has a simple analytical construction, so we adopt 2×2 as the default to balance success rate and computational performance.

W3, Q3: Applicability beyond BRC, including PPO and TD-MPC2

CoSR does not depend on a specific framework; its modules can be used with different reinforcement learning objectives. Our analysis is based on linear-layer computations and the Adam update map. For PPO, CoSR can be applied to weight and Adam-state writeback in both the policy and value networks. Algorithms with soft-updated target networks can also incorporate LAG.

We completed PPO experiments. Averaged over 5 random seeds, CoSR achieves results comparable to full-precision PPO.

TaskMetricPPO FP32PPO CoSR
PushSuccess rate (%)100.0 [100.0, 100.0]96.0 [92.0, 100.0]
Door OpenSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
Peg Insert SideSuccess rate (%)92.0 [90.0, 96.0]94.0 [90.0, 98.0]

We further integrated CoSR into TD-MPC2 and evaluated it on five individual tasks from MetaWorld and DMC. The table reports means over 5 random seeds.

TaskMetricTD-MPC2-FP32CoSR-TD-MPC2
ReachSuccess rate (%)100.0 [100.0, 100.0]100.0 [100.0, 100.0]
PushSuccess rate (%)90.0 [70.0, 100.0]100.0 [100.0, 100.0]
Cartpole SwingupEpisode return876.07 [866.00, 882.00]860.00 [851.70, 865.78]
Finger SpinEpisode return984.63 [982.10, 989.10]990.00 [986.90, 994.25]
Cheetah RunEpisode return757.60 [593.80, 856.90]760.00 [596.14, 843.45]

CoSR achieves success rates comparable to full precision on all five tasks, demonstrating its effectiveness in model-based RL.

We also applied CoSR to GRPO in agentic RL; the results are presented in the next response.

W4: Potential value for large-scale agentic RL

The agentic RL setting you suggested has considerable practical value. By compressing the MLP layers, which account for the largest share of model parameters, and their Adam states to FP8 and applying CoSR, we reduced GPU memory usage by 47.13%-58.56%. Using GRPO, we trained Qwen2.5-0.5B-Instruct on Sokoban for 200 iterations and Qwen2.5-3B-Instruct on WebShop for 100 iterations. The tables report mean results over 5 random seeds and GPU memory usage:

Sokoban on Qwen2.5-0.5B-Instruct

MethodSuccess rate (%)Training memory (GiB)
Full precision36.39 [26.95, 48.44]5.521
CoSR44.82 [37.85, 52.11]2.919

WebShop on Qwen2.5-3B-Instruct

MethodSuccess rate (%)Training memory (GiB)
Full precision71.86 [66.95, 78.63]34.488
CoSR74.29 [68.27, 78.44]14.292

The results show that CoSR significantly reduces GPU memory usage during training while matching full-precision success rates, demonstrating its generalizability.

We observed an interesting pattern: adding rounding to LLM training improves performance over the original precision, with a larger advantage for the 0.5B model than for the 3B model. Our preliminary investigation suggests implicit regularization from stochastic rounding as one possible explanation. Ozkara et al. (AISTATS 2025) [1] show in their gradient-descent analysis that SR introduces an implicit regularization term related to quantization gaps and gradients, and report better performance than mixed-precision baselines in LLM pretraining. CoSR preserves elementwise SR distributions while coordinating weight and Adam-state rounding errors to reduce forward and backward perturbations and local update distortion. This suggests that CoSR may retain the potential regularization benefits of stochastic perturbations while reducing their interference with learning signals, thereby improving final task performance. We hope to investigate this further in future work.

[1] Kaan Ozkara, Tao Yu, and Youngsuk Park. Stochastic rounding for llm training: Theory and practice. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan, editors, Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 4402–4410. PMLR, 03–05 May 2025.

W5, Q4: End-to-end computational benefits

Overall wall-clock time, including environment sampling, evaluation, and updates, is indeed more compelling. On PRO 6000 Blackwell, we independently measured end-to-end time for the standard network of width 4096 on MT50 and the network with its critic widened to 8192 on 165 tasks. We report means over 5 random seeds for both settings.

MethodSuccess rate (%)Persistent state (GiB)Peak GPU memory (GiB)End-to-end time (h)
BRC FP3290.0 [87.9, 91.8]2.0304.68021.39
SR68.8 [66.5, 70.6]0.5301.75013.37
CoSR90.7 [89.2, 92.5]0.5301.74015.85

These results show that:

  • SR offers advantages in GPU memory and speed, but it ignores correlated errors and cannot match full-precision success rates.
  • CoSR balances success rate, GPU memory usage, and computational performance simultaneously.

For the wider critic, mean end-to-end times are reported below:

SettingMT50 scoreDMC scoreShadowHand scorePersistent state (GiB)Peak GPU memory (GiB)End-to-end time (h)
BRC FP32(8192)0.89 [0.87, 0.91]0.64 [0.63, 0.65]0.84 [0.82, 0.85]8.0612.5781.7
SR(8192)0.72 [0.71, 0.74]0.49 [0.47, 0.51]0.69 [0.67, 0.70]2.034.6543.8
CoSR(8192)0.88 [0.87, 0.90]0.65 [0.63, 0.67]0.83 [0.81, 0.84]2.034.6552.4

The results demonstrate that with a wider critic, network computation and parameter updates account for a larger share of runtime. CoSR's efficiency gains in these stages translate more fully into end-to-end training speedups, while achieving comparable performance with significantly less GPU memory than the full-precision baseline.

Your review has been very helpful. We welcome any further questions about our responses. If they have addressed your concerns, we respectfully ask if you would feel comfortable reconsidering your rating based on the additional experiments and clarifications provided above. Thank you for your time and effort!

Comments on the paper

2 comments
AuthorsOpenReview ↗

With Best Wishes

Thank you again for your time and for the thoughtful discussion. We wish you all the best!

AuthorsOpenReview ↗

A Brief Summary of Our Rebuttal

Dear ICLR Program Chairs, Senior Area Chairs, and Area Chairs,

We sincerely thank you for the time and care you have devoted to guiding the review process for our paper. During the rebuttal, we evaluated CoSR across additional reinforcement learning settings, reported its end-to-end training time, and clarified its theoretical guarantees. We summarize the main additions below. W and Q refer to the weaknesses and questions raised by each reviewer.

  • Applicability across algorithms (k45e W3–W4 and Q3, NpVm W1 and Q1). We reported single-task results for BRC, PPO, and TD-MPC2, as well as GRPO results in agentic RL settings. These results demonstrate CoSR’s effectiveness across a range of reinforcement learning settings.

  • Channel pairing and block size (k45e W1–W2 and Q1–Q2, dTcd Q1). We showed that Proposition 1 applies to any fixed channel pairing and extended the analysis to larger blocks. On MT50, we compared three pairing strategies and three block sizes to assess CoSR’s sensitivity to these choices and the trade-offs they involve.

  • Training efficiency (k45e W5 and Q4, NpVm W2 and Q2, dTcd Q4). We reported end-to-end training times across different task settings. On MT50, our method achieved a 1.35× speedup over the full-precision baseline while maintaining a comparable success rate. When training jointly on 165 tasks with a wider critic, our method achieved a 1.56× speedup.

  • Target quantization (dTcd Q3). We showed when LAG can obtain a tighter error bound by quantizing the difference between the target and online parameters instead of the target parameters themselves. This analysis provides further support for LAG’s design.

We are grateful to the reviewers for confirming that our responses have addressed their main concerns. We appreciate the reviewers’ questions, which helped us improve both the experiments and the analysis. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.