- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CCRT is a single-state optimizer that stores one BF16 gradient-history tensor per parameter, forms a fresh FP32 carry, adds the current gradient, and applies a bounded root-rational transform. The paper contributes the algorithm and exact state semantics, empirical comparisons on a 134.8M decoder-only Transformer, a single-seed 3B reproduction, a 6.98B single-GPU feasibility run, long-horizon CCRT-only telemetry, component/horizon diagnostics, and mathematical analyses of scalar fixed-step dynamics and deterministic smooth-nonconvex stationarity. In two fresh paired 10K runs, CCRT has lower terminal validation CE and validation-CE AUC than AdamW and PowerStep. In the 3B run it reports final perplexity 127.15 versus 246.22 for AdamW and 378.64 for PowerStep, while storing 5.59 GiB versus 22.36 GiB and 11.18 GiB of persistent state. The paper also reports a structured nonzero two-cycle for scalar quadratics and an O(epsilon^-2) first-order stationarity complexity result for the ideal full-precision method.
Critical strengths
- State-memory reduction is concrete. One BF16 full-shape carry reduces persistent optimizer tensor state by approximately 75% versus AdamW and 50% versus PowerStep in the 3B study, and a 6.98B CCRT-only run fits on one A800. This supports a meaningful systems motivation independent of the quality ranking.
- Algorithm and implementation are specified in detail. The carry schedule, fresh-gradient path, root-rational map, BF16 persistence semantics, and evidence limitations are described sufficiently for reproduction.
Critical weaknesses
- Asymmetric and inadequately justified hyperparameter comparison. CCRT is run at
0.01versus0.0006for AdamW and PowerStep in the primary study, a roughly 16.7x raw learning-rate advantage, while only the comparators undergo the documented calibration. The paper does not show that CCRT would win under a comparable search budget or a principled update-scale matching procedure. This directly undermines the central quality claim and is the main reason for rejection. - Insufficient independent evidence. The main result has only two fresh seeds in one 134.8M language-model setup; the 3B comparison is one seed per optimizer and the 7B experiment has no comparator. The full-precision deterministic theory also does not validate the BF16 stochastic training behavior used for the practical claims.
Questions for authors
Clarification requests
- Learning-rate selection: Before the official runs, what CCRT learning-rate and history-horizon candidates were actually evaluated, on which seeds, and under what selection rule? In particular, did any pre-registered or otherwise blinded procedure compare CCRT's
0.01against lower rates using the same calibration budget used for AdamW and PowerStep? Evidence that CCRT received equivalent tuning would materially reduce my main fairness concern. - Large-scale replication: Do tensor-identical, multi-seed 3B runs exist for all three optimizers, or can the authors provide the archived initializations and per-seed results needed to establish this? A replicated 3B result with uncertainty estimates could materially change my confidence in the claimed quality advantage.
Suggestions for future work
- Evaluate BF16 carry-rounding effects against the ideal full-precision recurrence and test the optimizer on additional architectures or non-language-model objectives. These would improve scope and theory-to-implementation coverage, but they do not resolve the current comparison unless accompanied by matched hyperparameter tuning.
Revised Evaluation and Manuscript
We thank the reviewers for their feedback and have uploaded a revised manuscript and supplement that clarify CCRT’s design motivation, expand the related work, and report the new 134.8M and 3B comparisons.