acceptodds
Under review as a conference paper at ICLR 2027

CCRT: A Single-State Optimizer with Retained Gradient History and Bounded Nonlinear Updates

Abstract

Modern optimizers often retain multiple full-shape statistics, making optimizer state a substantial cost at scale. Current Carry Root Transport (CCRT) instead persists one BF16 gradient-history tensor per parameter, reinjects the fresh gradient, and applies a bounded root-rational readout without a second moment. Across three observed 134.8M training trajectories—two fresh paired runs and one earlier independent run—CCRT shows the same terminal validation-CE and validation-AUC ordering against calibrated AdamW and PowerStep. In a complementary single-seed 3B-scale reproduction, CCRT reaches validation perplexity 127.15, versus 246.22 for AdamW and 378.64 for PowerStep, while using 75% and 50% less persistent optimizer state, respectively. CCRT also completes a 1K-step single-GPU run at 6.98B parameters on an 80GB A800; validation CE decreases across all five recorded checkpoints, with a 13.00 GiB BF16 optimizer state. We also characterize the ideal full-precision update mathematically. Exact scalar quadratic dynamics exhibit a structured nonzero fixed-step two-cycle, while a deterministic smooth-nonconvex analysis proves a natural-stationarity residual of order and conventional first-order stationarity complexity. The scalar cycle matches the step-size exponent of the general residual bound. Together, the empirical and theoretical results show that a low-state nonlinear optimizer can combine substantial state savings with effective large-model optimization and an explicit connection between fixed-step dynamics and general nonconvex stationarity complexity.

Primary area:
optimization
Keywords:
optimization, deep learning, memory-efficient optimization, nonconvex optimization, optimizer state, language model training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MSyuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CCRT is a single-state optimizer that stores one BF16 gradient-history tensor per parameter, forms a fresh FP32 carry, adds the current gradient, and applies a bounded root-rational transform. The paper contributes the algorithm and exact state semantics, empirical comparisons on a 134.8M decoder-only Transformer, a single-seed 3B reproduction, a 6.98B single-GPU feasibility run, long-horizon CCRT-only telemetry, component/horizon diagnostics, and mathematical analyses of scalar fixed-step dynamics and deterministic smooth-nonconvex stationarity. In two fresh paired 10K runs, CCRT has lower terminal validation CE and validation-CE AUC than AdamW and PowerStep. In the 3B run it reports final perplexity 127.15 versus 246.22 for AdamW and 378.64 for PowerStep, while storing 5.59 GiB versus 22.36 GiB and 11.18 GiB of persistent state. The paper also reports a structured nonzero two-cycle for scalar quadratics and an O(epsilon^-2) first-order stationarity complexity result for the ideal full-precision method.

Critical strengths
  1. State-memory reduction is concrete. One BF16 full-shape carry reduces persistent optimizer tensor state by approximately 75% versus AdamW and 50% versus PowerStep in the 3B study, and a 6.98B CCRT-only run fits on one A800. This supports a meaningful systems motivation independent of the quality ranking.
  2. Algorithm and implementation are specified in detail. The carry schedule, fresh-gradient path, root-rational map, BF16 persistence semantics, and evidence limitations are described sufficiently for reproduction.
Critical weaknesses
  1. Asymmetric and inadequately justified hyperparameter comparison. CCRT is run at 0.01 versus 0.0006 for AdamW and PowerStep in the primary study, a roughly 16.7x raw learning-rate advantage, while only the comparators undergo the documented calibration. The paper does not show that CCRT would win under a comparable search budget or a principled update-scale matching procedure. This directly undermines the central quality claim and is the main reason for rejection.
  2. Insufficient independent evidence. The main result has only two fresh seeds in one 134.8M language-model setup; the 3B comparison is one seed per optimizer and the 7B experiment has no comparator. The full-precision deterministic theory also does not validate the BF16 stochastic training behavior used for the practical claims.
Questions for authors

Clarification requests

  1. Learning-rate selection: Before the official runs, what CCRT learning-rate and history-horizon candidates were actually evaluated, on which seeds, and under what selection rule? In particular, did any pre-registered or otherwise blinded procedure compare CCRT's 0.01 against lower rates using the same calibration budget used for AdamW and PowerStep? Evidence that CCRT received equivalent tuning would materially reduce my main fairness concern.
  2. Large-scale replication: Do tensor-identical, multi-seed 3B runs exist for all three optimizers, or can the authors provide the archived initializations and per-seed results needed to establish this? A replicated 3B result with uncertainty estimates could materially change my confidence in the claimed quality advantage.

Suggestions for future work

  • Evaluate BF16 carry-rounding effects against the ideal full-precision recurrence and test the optimizer on additional architectures or non-language-model objectives. These would improve scope and theory-to-implementation coverage, but they do not resolve the current comparison unless accompanied by matched hyperparameter tuning.
Reviewer_F5r1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CCRT, an optimizer that stores a single BF16 gradient-history tensor (a running-mean carry that becomes an EMA with β = 31/32 after 32 steps), adds the fresh gradient back to it, and applies a bounded root-rational readout φ(z) = sign(z)·√|z| / (1 + √|z|) with no second moment. Empirically, CCRT achieves lower validation CE than AdamW and PowerStep on a 134.8M model and in a single-seed 3B run. It also completes a 1K-step 6.98B run on one 80GB GPU. The paper shows a nonzero two-cycle for scalar quadratics and proves a deterministic smooth-nonconvex bound with an O(dL³η³) residual and O(ε⁻²) stationarity complexity.

Critical strengths

The author showed a simple and low-state design for concrete measured savings. At 3B model, this persistent optimizer state is 5.59 GiB, while it is 22.36 GiB for AdamW, and peak allocated memory is 31.6% lower. A 6.98B model trains on a single 80GB GPU, which is practically relevant for memory-constrained training. The author also showed a non-trivial and well-matched theory. The deterministic nonconvex bound needs only L-smoothness and a lower bound. Its η³ residual exponent is shown to be tight by the exact scalar two-cycle, a clean link between fixed-step dynamics and the general bound.

Critical weaknesses

I think this experiment runs extremely low training budget (tokens). The 134.8M model runs use about 10.24M tokens (~0.08 tokens/param), and the 3B model run uses 81.9M tokens (~0.03 tokens/param). In this situation, a bounded large-LR update may simply win early, so the results do not support the claim of “effective large-model optimization”? Baseline tuning and coverage. Baseline LRs were chosen from a coarse 3-point grid using 200-step runs, with untuned shared settings (10 warmup steps, β₂ = 0.999) that are atypical for AdamW in LM training, while the origin of CCRT’s LR = 0.01 is not stated. Since the paper itself finds that short-horizon rankings do not predict 10K rankings, this undermines both the 200-step calibration and the 1K-based elimination of Lion, the structurally closest baseline (CCRT is an instance of generalized Lion-κ); other relevant memory-efficient methods (Adafactor, 8-bit Adam, Muon) are also absent.

Questions for authors

Does the ordering against AdamW hold at a reasonable token budget, e.g. ≥1–2B tokens for the 134.8M model? How were CCRT’s learning rates (0.01 at 134.8M, 5e-3 at 3B) selected, and how many CCRT configurations were tried during development? Could AdamW and Lion receive an equal-budget LR sweep (including weight decay) evaluated at the full 10K horizon?

Reviewer_zxPkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CCRT, a single-state optimizer that reduces the memory overhead of optimizer states in large model training. Experiments on language model pre-training show that CCRT outperforms AdamW and PowerStep while using substantially less optimizer memory.

Critical strengths
  1. The paper studies a practically important problem: reducing optimizer memory is increasingly relevant for training large language models.
  2. The proposed method is simple and easy to implement. It keeps only a single state per parameter.
Critical weaknesses
  1. The presentation quality is poor and the paper is difficult to follow. The writing is unpolished, the figures are rudimentary, and the discussion of related work is very thin: only a handful of prior optimizers are cited, given how crowded the memory-efficient optimizer literature is. Much of the main text goes to qualifying the strength of the evidence rather than explaining the method and its motivation.

  2. The experimental comparison is far from sufficient, as only AdamW and PowerStep are included as baselines. Since the paper’s main selling point is reduced optimizer state, it should be compared against established memory-efficient optimizers, such as Adafactor [1], Muon [2], and low-rank or stateless methods like GaLore [3] and SWAN [4]. Some of these keep even less optimizer state than CCRT. Without these comparisons, it is unclear whether CCRT offers any practical advantage over existing alternatives.

  3. The fairness of the comparison is questionable. CCRT is run with a learning rate of 0.01, whereas AdamW and PowerStep use 6e-4. Moreover, the baseline learning rates were selected using very short (200-step) screening runs over only three candidates, while the authors themselves note that short-horizon rankings do not reflect long-horizon behavior. The reported gains may therefore largely reflect insufficiently tuned baselines rather than a genuine advantage of the proposed optimizer.

  4. The models are far from convergence, so the results cannot demonstrate the optimizer’s effectiveness. The final validation perplexity exceeds 100 in all reported settings, including the 3B experiments, which indicates severe under-training. Claims of “effective large-model optimization” are therefore not supported by the current evidence.

[1] N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML, 2018.

[2] K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An Optimizer for Hidden Layers in Neural Networks. Blog post, 2024.

[3] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.

[4] C. Ma, W. Gong, M. Scetbon, and E. Meeds. SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training. ICML, 2025.

Questions for authors

See weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Evaluation and Manuscript

We thank the reviewers for their feedback and have uploaded a revised manuscript and supplement that clarify CCRT’s design motivation, expand the related work, and report the new 134.8M and 3B comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.