- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a novel method based on an operator-aware joint stochastic rounding (SR) framework. Unlike independent SR, it jointly considers the impact of rounding on computations involving activations and gradients, which leads to optimal parameter-rounding couplings within a fixed block partition with independence across blocks. Besides, the Adam optimizer's first and second moments, and , are jointly rounded to minimize the linearized error in Adam's normalized update. Finally, the LAG method stores the difference between the target and online critic parameters. Its recurrence is equivalent to the standard soft update before quantization, with an explicit characterization of the quantization error, and it improves performance in the experiments.
Critical strengths
- The paper proposes a rounding method with theoretical guarantees that also works well in experiments. The analysis provides useful insight into how coordinating rounding errors can reduce perturbations in subsequent computations.
- The experiments are extensive, especially the ablation study covering all combinations of weight rounding, Adam-state rounding, and LAG. This helps distinguish the contributions of the individual modules and their combinations.
Critical weaknesses
None.
Questions for authors
- Do you have any insights into more general rounding designs for weights with theoretical guarantees? The block-wise construction is interesting and novel, and I wonder whether the guarantees could extend to larger blocks or more general coupling structures. I see this as a possible extension rather than a requirement for the current result.
- In Figure 1(a), RTN update retention is above zero at an x-coordinate below half the local FP8 gap, which I find confusing. Under a fixed quantization grid, an already-representable weight should remain unchanged for updates smaller than half the gap in the update direction. Could you clarify how retention is defined and whether this point represents a bin that crosses the half-gap threshold, or whether the scale changes during writeback?
- The LAG analysis establishes equivalence to the standard soft update before quantization and characterizes the quantization error, but does not show that LAG is more accurate than storing absolute target parameters. Are there conditions under which such an advantage can be established theoretically?
- The additional update-time overhead relative to independent SR is noticeable: approximately 18.4% according to Table 13. Could you clarify which components account for this overhead and whether it could be reduced through better implementation, such as kernel fusion?
Nice work, congrats!
3 comments
Q3: Accuracy advantage of LAG over direct storage
Under specific conditions, LAG can be proven to have an accuracy advantage.
Let the target candidate for the current soft update be , the online parameters be , and the corresponding lag be . Direct storage quantizes , whereas LAG quantizes and reconstructs the target as . Both use the same FP8 format and RTN, with their respective maximum-based scaling. Assuming no overflow and ignoring rounding in FP32 arithmetic,
\le c_{\mathrm{FP8}}\\|x\\|_\infty,$$ where $c_{\mathrm{FP8}}$ depends only on the numerical format. This bound follows directly from the half-gap error bound of RTN and maximum-based scaling. Thus, $$\\|e_{\mathrm{LAG}}\\|_\infty \le c_{\mathrm{FP8}}\\|\ell\\|_\infty, \qquad \\|e_{\mathrm{direct}}\\|_\infty \le c_{\mathrm{FP8}}\\|y\\|_\infty.$$ When the target and online networks are sufficiently close that $$\\|\ell\\|_\infty\le\rho\\|y\\|_\infty, \qquad 0\le\rho < 1,$$ **the above LAG error bound is at most $\rho$ times the corresponding bound for direct storage. Its accuracy advantage comes from matching the quantization scale to the smaller range of differences**. This analysis also connects to long-term target tracking. Along the same online parameter sequence, let $d_t$ denote the error of the reconstructed target relative to the exact soft update. From Appendix C.4, $$d_{t+1}=(1-\tau)d_t+\varepsilon_{t+1}.$$ If $d_0=0$ and the quantization error at each step satisfies $\\|\varepsilon_t\\|_\infty\le\bar\varepsilon$, then $$\\|d_t\\|_\infty \le \bar\varepsilon\\, \frac{1-(1-\tau)^t}{\tau}.$$ Thus, when LAG maintains a smaller uniform per-step error bound throughout training, its long-term target-tracking error bound also decreases accordingly. The experiments also corroborate this result. In Table 1, replacing direct storage of target parameters with LAG while keeping all other modules unchanged **increases MT50 success from 83.7% to 90.7%**. We have added this analysis and its conditions to Appendix C.4 of the revised manuscript. ## Q4: Additional update overhead over independent SR and optimization opportunities In Table 13, update times for CoSR and independent SR are **23.77 ms and 20.08 ms**, respectively, a difference of **3.69 ms, or 18.4%**. Relative to independent SR, CoSR adds forward and backward channel inner-product statistics, coordinated generation of weight-rounding randomness, and joint encoding of Adam's first and second moments. It also uses LAG for target-state reconstruction and writeback. These computations use the current forward and backward operands and update candidates. For a $d_{\mathrm{in}}\times d_{\mathrm{out}}$ weight matrix, weight coordination requires $d_{\mathrm{in}}/2+d_{\mathrm{out}}/2$ sign statistics, which can be shared across channels. We therefore tried combining channel-statistic reductions with the computations producing the corresponding operands, and fusing joint rounding with quantization encoding and state writeback to reduce extra memory accesses and kernel launches. **The optimized parameter update time is 21.48 ms, only 6.97% higher than independent SR**, providing stronger support for CoSR. ### Thank you again for your constructive review and valuable guidance, which helped us further optimize CoSR !A Gentle Follow-Up
Dear Reviewer:
Thank you again for your thoughtful review and for your encouraging assessment of our work! If you have a moment, we would be grateful to hear whether our responses have addressed your questions and whether they have given you greater confidence in your assessment.
Best wishes!
Dear Reviewer, thank you for recognizing our work. We address your concerns below, with W denoting the weaknesses and Q the questions you raised.
Q1: More general weight-rounding designs and theoretical guarantees
Thank you for suggesting this extension. The current construction can be generalized in both channel pairing and within-block coupling.
Proposition 1 applies to any predetermined, non-overlapping input- and output-channel pairings. Using the actual forward and backward inner products of these channels, the original XOR construction still simultaneously attains the minimum forward and backward risks under the specified 2×2 partition.
A 2×2 block is the smallest rectangle containing both forward same-column and backward same-row pairs. The two vertical edges share a forward correlation coefficient, and the two horizontal edges share a backward correlation coefficient. Thus, each block needs only one uniform random draw and two inner-product sign checks to achieve optimal coupling on all four edges through row and column flips, making it our first choice.
Larger k×k blocks can also share one U∼U(0,1), using U or 1−U according to the XOR of row and column flip labels to preserve each element's SR marginal. Forward and backward risks depend only on row and column flips, respectively. Optimizing each set of flips separately attains both risk minima simultaneously within this construction family. Compared with the two sign checks for 2×2 blocks, larger blocks must account for multiple channel-pair relationships. Enumeration requires comparing 2k−1 flip patterns per channel group, creating a computation–error trade-off. We have added this generalization and its proof to Appendix C.2 of the revised manuscript.
We compared block sizes over 5 random seeds and report end-to-end time including environment sampling, evaluation, and updates.
The results show low sensitivity to block size. The 8×8 setting achieves the highest success rate but requires coordinating more error relationships, increasing runtime by 29.72%. The 2×2 setting has a simple analytical construction, so we adopt 2×2 as the default to balance performance and implementation complexity.
We also compared random pairing and fixed correlation-based pairing on MT50. Random pairs are determined at initialization and then fixed. Correlation-based pairs are greedily matched using absolute normalized correlations from the forward activations and backward gradients of the first training batch, then fixed. Both retain the joint rounding rule within each 2×2 block and use 5 random seeds.
The results show that CoSR supports different pairings. Default adjacent pairing achieves the highest success rate and shortest end-to-end time while avoiding additional correlation matching and channel reordering, balancing performance and cost.
Q2: Clarification of RTN in Figure 1(a)
Your understanding is correct. We define
RS=∑i∈SΔi2∑i∈SΔi,E![Δi].
Here, S is the set of parameter indices in the bin, Δi=zi−wi is the intended update, and Δi=wi−wi is the actual writeback update. The expectation is over rounding randomness. R measures how much of the actual update is retained along the intended direction: 1 indicates full retention of the contribution in that direction, and 0 indicates zero net contribution.
The point you identified corresponds to the [0.1,1) bin, which includes updates exceeding the half-gap threshold, giving an aggregate R>0. When strictly selecting updates smaller than half the gap, we confirmed that under RTN, the actor and critic have R values of approximately 0.00012 and −0.00004, respectively, both near zero. These small residuals arise from grid shifts caused by dynamic scaling during writeback.
Thank you for identifying this issue. In the revised manuscript, we use 0.5 as a bin boundary to produce the figure again. In the new [0.5,1) bin, RTN can round a sub-gap update to a full grid step, so R>1 indicates amplification along the intended update direction.