acceptodds
Under review as a conference paper at ICLR 2027

Ember: A Kilobyte-State Optimizer for Token Interfaces

Abstract

Language models learn continuous programs over discrete symbols, with the embedding table and LM-head acting as the read/write interface between them. We show that this interface has gradient geometry distinct from dense hidden weights which can be exploited to improve the Pareto frontier across supervised finetuning, RL, and pretraining, while only utilizing kilobytes of optimizer state. We introduce Ember, a lightweight optimizer for embedding and LM-head matrices that utilizes optimizer VRAM, instead of Adam's , and forgoes the need to shard both token table optimizer states. On 50k-step pretraining across two datasets and three paired seeds, Ember lowers val loss by – nats relative to tuned AdamW; we also provide empirical evidence that Ember scales across batch size and parameter count. Finally, we provide a distributed Ember implementation that merges cleanly with existing ZeRO/FSDP setups (code to be released).

Primary area:
optimization
Keywords:
optimization, memory-efficient training, language models, embeddings, optimizer state, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BwMVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Ember, a lightweight optimizer for embedding and LM-head matrices that factorizes the second-moment estimate into row and column statistics, reducing state memory size from O(VD)O(VD) to O(V+D)O(V+D). The paper studies the gradient structure of token-interface parameters. Experiments on different settings show that Ember can match or outperform AdamW while substantially reducing token-interface optimizer state.

Critical strengths
  1. The paper's evaluation result shows a large portion of optimizer-state size reduction.
Critical weaknesses
  1. The presentation is weak and makes the paper difficult to follow. The motivation, technical story, and positioning are not clearly organized, making it hard to identify the paper's central insight and the position in this field.

  2. The novelty is limited. As noted in the paper, Ember's update coincides with Algorithm 2 of Adafactor under matched settings. While the paper validates its effectiveness on LLM, stronger analysis and controlled comparisons are needed to differentiate a research contribution from an engineering adaptation.

Questions for authors

What evidence shows that Ember is particularly suitable for token interfaces, instead of simply reusing Adafactor's update?

1 comment
AuthorsOpenReview ↗

Response to Reviewer BwMV

3.1 Presentation and central question

The presentation is weak and makes the paper difficult to follow. The motivation, technical story, and positioning are not clearly organized, making it hard to identify the paper's central insight and the position in this field.

  • We appreciate the reviewer bringing this up. We will organize the paper around a clear question: can the embedding table and LM head use much less optimizer state while retaining similar validation loss? Our key claim is that Ember can match AdamW, and in some cases outperform it, using V+D optimizer-state values instead of 2VD per matrix.
  • Our primary contribution is revisiting Adafactor's Algorithm 2 for token tables in modern language models, while retaining AdamW or Muon on the transformer body. The update uses V+D optimizer-state values per matrix instead of AdamW's 2VD. In the new comparisons, token-table moment storage fell from 824.18 MB to 0.411 MB, about a 2,000× reduction.
  • We already acknowledge Algorithm 2 in Section 2.1 and Appendix A.6. We will make this attribution explicit in the introduction and center the paper on the empirical results of applying it to these parameter groups.

3.2 Novelty and controlled comparisons

The novelty is limited. As noted in the paper, Ember's update coincides with Algorithm 2 of Adafactor under matched settings. While the paper validates its effectiveness on LLM, stronger analysis and controlled comparisons are needed to differentiate a research contribution from an engineering adaptation.

  • Thank you for recommending stronger controlled comparisons. We added comparisons against AdamW and native Adafactor with cosine annealing. All use Pythia-410M on FineWeb-Edu, with AdamW retained on the transformer body. The token-table optimizer is varied within each comparison.

Mean validation loss [95% CI]; lower is better. Compare methods within each row, since the studies use different held-out data. The 50k row reports means only.

ExperimentAdamWEmberAnnealed native Adafactor
10k updates, batch 32, 3 seeds3.45801 [3.45000, 3.46603]3.46133 [3.45980, 3.46286]3.44976 [3.43712, 3.46241]
5k updates, batch 64, 3 seeds3.54065 [3.53619, 3.54511]3.54604 [3.54442, 3.54766]3.56018 [3.54859, 3.57176]
50k updates, batch 32, 2 recovered seeds3.092023.100963.09067
  • At 10k, Ember minus AdamW was +0.00331 nats [95% CI: −0.00467, +0.01129], and Ember minus native Adafactor was +0.01156 [−0.00228, +0.02540].
  • At batch 64, Ember minus AdamW was +0.00539 nats [95% CI: +0.00225, +0.00853], and Ember minus native Adafactor was −0.01414 [−0.02537, −0.00290]. Ember remains close to annealed native Adafactor and performs better in this batch-64 comparison.
  • For 50k, 3 seeds ran, but an export error left complete results for 2. We report those 2 without a confidence interval.
  • Both Ember and native Adafactor use 0.411 MB of token-table moment storage, about 2,000× less than AdamW's 824.18 MB.

3.3 Evidence for token-table suitability

What evidence shows that Ember is particularly suitable for token interfaces, instead of simply reusing Adafactor's update?

  • Our contribution is evaluating Adafactor’s Algorithm 2 on embedding tables and the LM head in modern language models, rather than proposing a new update rule.
  • The comparisons in 3.2 show comparable validation loss while reducing token-table moment storage from 824.18 MB to 0.411 MB, about 2,000× less. This supports token tables as a useful application, but does not establish that the update is uniquely suited to them.
Reviewer_XoXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ember, an optimizer applied only to untied input embeddings and LM heads. It removes Adam’s first moment and approximates the dense second moment by an outer product of row- and column-wise statistics, reducing token-interface optimizer state from O(2VD) to O(V+D). The Transformer body retains its existing optimizer. Under matched settings, the update is equivalent to Algorithm 2 of Adafactor (Shazeer & Stern, 2018); the claimed contribution is its selective evaluation in modern language-model training and a ZeRO/FSDP-compatible implementation. The principal evidence is a 50k-step Pythia-410M pretraining study on FineWeb-Edu and OpenWebText, with independently tuned learning rates and three paired seeds per corpus. Ember improves frozen validation cross-entropy by 0.080 and 0.088 nats, respectively, on every seed pair, while removing roughly 824 MB of FP32 optimizer state. SFT and GRPO results are approximately matched to AdamW, and a scale sweep reports comparable loss at dramatically lower state memory.

Critical strengths
  1. Token-interface Adam state is a genuine memory and engineering bottleneck, and recent optimizers such as Muon largely leave embeddings untouched.
  2. The main pretraining comparison is carefully controlled, with independent learning-rate selection, late-window averaging, and a consistent gain across paired seeds.
  3. The second-moment diagnostic is informative: the factored estimate tracks Adam’s dense statistic with correlation 0.94 over 38M parameters, and the short ablations test row-only scaling, first-moment restoration, and native Adafactor.
Critical weaknesses
  1. The technical novelty is limited. Ember’s update coincides with Adafactor’s Algorithm 2, and Adafactor already studies factorized second moments and token-frequency effects in embeddings. A focused modern evaluation could still be useful, but the paper presents a known update as a new optimizer.
  2. The 50k-step runs compare only against AdamW; Adafactor, SM3, Coupled Adam, and other memory-efficient methods are missing or confined to 200-step recipe comparisons. Table 3 also does not isolate which design choice causes the observed gap, so it cannot support a long-horizon superiority claim.
  3. The scale sweep shows comparable loss rather than consistent improvement, AdamW performs better at batches 16–256, and SFT and GRPO primarily show parity. The abstract’s Pareto-frontier claim should therefore be narrowed. The evaluation is also limited to 2.8B parameters, a 100-example SFT setting, and a single RL task, with some curves shown only as representative runs.
  4. The systems and theoretical support are thin. The distributed implementation is deferred to supplementary code, without multi-GPU memory, communication, or throughput results; the reported speed difference is below 0.5%. The information-geometry discussion likewise does not establish the row–column approximation for squared minibatch gradients, leaving the empirical correlation to carry that argument.
Questions for authors

Can the authors include native Adafactor, a literal Algorithm 2 implementation, SM3, and Coupled Adam in at least one matched 50k-step pretraining comparison? This is needed to determine whether the gain comes from the proposed selective update or from tuning and bias correction.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XoXF, part 1

2.1 Contribution and attribution

The technical novelty is limited. Ember’s update coincides with Adafactor’s Algorithm 2, and Adafactor already studies factorized second moments and token-frequency effects in embeddings. A focused modern evaluation could still be useful, but the paper presents a known update as a new optimizer.

  • Our primary contribution is to revisit Adafactor’s Algorithm 2 for token tables in modern language models, which has not been done in prior literature to the best of our knowledge. This is practically useful because large token tables require 2VD optimizer-state values per matrix under AdamW. Using V+D values instead reduces token-table moment storage by about 2,000× in one of our experiments (Pythia-410M), lowering VRAM requirements.
  • We appreciate the reviewer highlighting the need for clearer attribution. We acknowledge Adafactor’s Algorithm 2 in Sec 2.1 and Appendix A.6, and will make this explicit in the introduction alongside its token-frequency motivation and use in T5.

2.2 Missing baselines, longer training, and schedule effects

The 50k-step runs compare only against AdamW; Adafactor, SM3, Coupled Adam, and other memory-efficient methods are missing or confined to 200-step recipe comparisons. Table 3 also does not isolate which design choice causes the observed gap, so it cannot support a long-horizon superiority claim.

Can the authors include native Adafactor, a literal Algorithm 2 implementation, SM3, and Coupled Adam in at least one matched 50k-step pretraining comparison? This is needed to determine whether the gain comes from the proposed selective update or from tuning and bias correction.

  • Yes, we added comparisons between AdamW, Ember and native Adafactor with cosine annealing, using the same data within each experiment, as the reviewer recommended. We will include these results in the final paper.

Mean validation loss [95% CI]; lower is better. The 50k row reports means only.

ExperimentAdamWEmberAnnealed native Adafactor
10k updates, batch 32, 3 seeds3.45801 [3.45000, 3.46603]3.46133 [3.45980, 3.46286]3.44976 [3.43712, 3.46241]
5k updates, batch 64, 3 seeds3.54065 [3.53619, 3.54511]3.54604 [3.54442, 3.54766]3.56018 [3.54859, 3.57176]
50k updates, batch 32, 2 recovered seeds3.092023.100963.09067
  • At 10k updates, Ember minus AdamW was +0.00331 nats [95% CI: −0.00467, +0.01129]. Ember minus native was +0.01156 nats [−0.00228, +0.02540]. Both paired intervals include zero.
  • At batch 64, Ember minus native was −0.01414 nats [95% CI: −0.02537, −0.00290].
  • For the 50k comparison, 3 seeds ran, but an export error left complete results for 2. We report those 2 means without a confidence interval.
  • Ember remains close to annealed native Adafactor and performs better in the batch-64 comparison.
  • We did not run SM3 or Coupled Adam, and are happy to include it in the appendix of the final paper if helpful.
AuthorsOpenReview ↗

Response to Reviewer XoXF, part 2

2.3 Batch size, model scale, and downstream claims

The scale sweep shows comparable loss rather than consistent improvement, AdamW performs better at batches 16–256, and SFT and GRPO primarily show parity. The abstract’s Pareto-frontier claim should therefore be narrowed. The evaluation is also limited to 2.8B parameters, a 100-example SFT setting, and a single RL task, with some curves shown only as representative runs.

  • Our key claim is that Ember can match AdamW, and in some cases outperform it, using V+D optimizer-state values instead of 2VD per matrix. Ember is not universally better than AdamW.
  • To address the short pretraining runs, we added the comparisons below. All use Pythia-410M. Validation losses should be compared within each row because the studies use different held-out data.
ExperimentAdamW lossEmber lossEmber − AdamW [95% CI]
10k updates, batch 32, 3 seeds3.458013.46133+0.00331 [−0.00467, +0.01129]
5k updates, batch 64, 3 seeds3.540653.54604+0.00539 [+0.00225, +0.00853]
50k updates, batch 32, 2 recovered seeds3.092023.10096+0.00893, mean only
  • Losses and differences are in nats. The intervals use paired seed results. For 50k, 3 seeds ran, but an export error left complete results for 2; we report those 2 without a confidence interval.
  • At batch 64, token-table optimizer state fell from 824.18 MB to 0.411 MB, about a 2,000× reduction, with the small loss difference shown above.
  • These extend the training horizon, but do not test larger-model pretraining. The SFT and GRPO results remain short exploratory studies. We will narrow the abstract to the measured memory savings and loss trade-offs, and identify curves that show only one run.

2.4 Memory, throughput, and distributed claims

The systems and theoretical support are thin. The distributed implementation is deferred to supplementary code, without multi-GPU memory, communication, or throughput results; the reported speed difference is below 0.5%.

  • Ember's main benefit is reducing optimizer-state memory. We do not claim a reliable speedup from a timing difference below 0.5%.
  • In the direct 10k comparison on a single A100, token-table moment storage fell from 824.18 MB to 0.411 MB, about a 2,000× reduction, and peak allocated memory fell from 19.87 GB to 19.05 GB. Peak reserved memory was 20.12 GB for both methods.
  • We also have a historical 8×H100 comparison with 4 matched seed pairs. Ember was applied to value embeddings with globally pooled statistics; both arms used the same window schedule. Final validation losses at 1,390 updates were:
SeedStock optimizer recipeEmber on value embeddingsEmber − control
13163.27483.2772+0.0024
13173.27603.2771+0.0011
13183.27753.2768-0.0007
13193.27553.2778+0.0023
Mean3.275953.27723+0.00128
  • The paired mean difference was +0.00128 nats [95% CI: -0.00102, +0.00357]. This demonstrates distributed execution of the value-embedding variant. The records for this comparison do not include memory, communication or throughput measurements, so we will keep those claims separate from the verified single-GPU results.

2.5 Theory and squared minibatch gradients

The information-geometry discussion likewise does not establish the row–column approximation for squared minibatch gradients, leaving the empirical correlation to carry that argument.

  • For input embeddings, each token occurrence activates the entire token vector, which motivates a shared row scale tracked in O(V). We observe anisotropy across features, so we add an O(D) column factor, yielding a net O(V+D) state.
  • Appendix A.9 already derives how token frequency enters the squared minibatch-gradient moment; Appendix A.12 discusses the dense LM head separately.
  • The row–column factorization is an approximation, as discussed in Section 3.4. Our component ablations provide empirical support; mostly that the row factor captures majority of the important second moment information. Table 3 already compares both factors against row-only scaling over 3 paired seeds at 200 updates, with mean validation loss ± standard deviation:
VariantBatch 8Batch 256
Ember, row + column7.207 ± 0.0235.842 ± 0.020
Ember, row only7.312 ± 0.0215.890 ± 0.017
  • The separate earlier ablations in Figure 11 also test column-only scaling: losses were 10.899 versus 7.219 at batch 8 and 7.958 versus 5.890 at batch 256, comparing column-only with full Ember.
Reviewer_JSjfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Ember, an optimizer which works on the embedding and LM head of a language model. Ember, is able to reduce the memory cost from O(Vd)O(Vd) to O(V+d)O(V+d) on the two matrices, and is able to do so while lowering final validation loss.

Critical strengths

Ember allows for a cheap and practical drop-in, with savings in memory. The derivations in the paper are correct.

Critical weaknesses

Rebranded Adafactor:

Ember is exactly Algorithm 2 in the Adafactor work (https://arxiv.org/pdf/1804.04235), the authors admit themselves. However, the paper mentions "we introduce ember...". This claim is an overreach. Adafactor already provided the token-frequency reasoning for row and column factorization. It has also been used to train language models (T5).

Confounding baseline:

The "tuned AdamW" is Adam with no weight decay, with only tuning the learning rates. Ablations on weight decay, and betas, along with learning rate are required to confidently state the claims.

Extremely weak evidence:

The paper trains only one model at around 0.2×0.2 \times the compute optimal requirement with a much smaller batch size than the original training recipe. The paper's own batch size sweep shows AdamW winning at batch 16-256.

The claims in the abstract portray more than what is in the paper: Apart from the main headline experiment, every other is extremely short and uninformative.

  1. Pretraining: 200 - 300 steps, with very high perplexities.
  2. Fine-tuning: 250 steps with an extremely small gain in nats (0.008) at one shared learning rate.
  3. RL: 50 steps with no setup described.
  4. Image generation: loss stays within 0.08 nats of random guessing.

Misrepresented motivation:

  1. Frequency argument applied equally to AdamW.
  2. "Momentum is wasted" is incorrect. It is inferred from the negative gradient autocorrelation, which is exactly where momentum helps. Table 3 shows momentum helping at a batch size of 256.

Competitors are missing:

Closest alternatives are cited, but never really compared against: Lau and Su (https://arxiv.org/abs/2605.18106), Adam-mini (https://arxiv.org/abs/2406.16793), SM3 (https://arxiv.org/abs/1901.11150), Scion (not cited), Modula, Coupled Adam.

Minor: The main text leaves important training details in the appendix. Figures read poorly, and many references are not cited.

Questions for authors
  1. Does the saving for pretraining work with more realistic setups? Larger batch sizes, slightly larger models (1 - 4 B params perhaps)?

  2. Can we see the experiments with a fairly tuned baseline for AdamW (with weight decay, tuned learning rates, and jointly tuned betas)?

  3. Contribution. What do you consider to be the novelty beyond the Adafactor work?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JSjf, part 1

1.1 Contribution and attribution

Ember is exactly Algorithm 2 in the Adafactor work (https://arxiv.org/pdf/1804.04235), the authors admit themselves. However, the paper mentions "we introduce ember...". This claim is an overreach. Adafactor already provided the token-frequency reasoning for row and column factorization. It has also been used to train language models (T5).

  • Our primary contribution is to revisit Adafactor’s Algorithm 2 for token tables in modern language models, which has not been done in prior literature to the best of our knowledge. Our key claim is that Ember can match AdamW, and in some cases outperform it, using V+D optimizer-state values instead of 2VD per matrix. Ember is not universally better than AdamW.
  • We appreciate the reviewer highlighting the need for clearer attribution. We acknowledge Adafactor’s Algorithm 2 in Sec 2.1 and Appendix A.6, and will make this explicit in the introduction alongside its token-frequency motivation and use in T5.
  • This is practically useful because large token tables require 2VD optimizer-state values per matrix under AdamW. Using V+D values instead reduces token-table moment storage by about 2,000× in Pythia-410M, lowering VRAM requirements.

1.2 AdamW tuning

Confounding baseline: The "tuned AdamW" is Adam with no weight decay, with only tuning the learning rates. Ablations on weight decay, and betas, along with learning rate are required to confidently state the claims.

  • Thank you for raising this. We reran experiments with 12 candidates each for AdamW and Ember, 2.5k steps each, and found that token decay hurt at all 6 tested settings for each optimizer so we did not include it for the final comparison.
  • Across 3 paired seeds trained for 6,000 steps, Ember minus AdamW was −0.00649 nats (paired 95% CI: [−0.01879, +0.00582]).
  • In a separate 10,000-step comparison with cosine schedules for both optimizers, mean loss was 3.46133 for Ember and 3.45801 for AdamW. The difference was +0.00331 nats [−0.00467, +0.01129].
  • Both confidence intervals include zero.

1.3 Batch size and scale

The paper trains only one model at around 0.2× the compute optimal requirement with a much smaller batch size than the original training recipe. The paper's own batch size sweep shows AdamW winning at batch 16-256.

  • We added a Pythia-410M comparison at batch 64 over 5,000 updates across 3 paired seeds. Mean validation losses were 3.54065 for AdamW and 3.54604 for Ember, which is near-parity, with an Ember–AdamW difference of +0.00539 nats [95% CI: +0.00225, +0.00853], while token-table optimizer state fell from 824.18 MB to 0.411 MB, approximately a 2,000× reduction.
  • This shows similar loss with substantially less optimizer memory at batch 64. Compute-optimal training and larger-model pretraining remain untested.

1.4 Short pretraining comparisons

Pretraining: 200 - 300 steps, with very high perplexities.

  • We already included 50k-step experiments in the paper (Figure 1, Section 3.1 and Appendix A.5), comparing Ember and AdamW with 3 paired seeds per dataset. Table 2 reports the original results:
DatasetAdamW lossEmber lossEmber − AdamW [95% CI]
FineWeb-Edu3.096993.01677−0.08021 [−0.08646, −0.07396]
OpenWebText3.214973.12680−0.08817 [−0.10238, −0.07395]
  • These Pythia-410M runs used 1.6384 billion tokens each. Losses are in nats, averaging 11 validation measurements between steps 45k and 50k for each seed.
  • The 200–300-step results are separate component and batch-size ablations, not the entirety of the pretraining evidence. We also added the 10k-step comparisons across 3 seeds described in 1.2 to strengthen the baseline evaluation.

1.5 Fine-tuning

Fine-tuning: 250 steps with an extremely small gain in nats (0.008) at one shared learning rate.

  • Figure 3C separately includes a representative fine-tuning trajectory extending to approximately 2,000 steps. We will distinguish this longer trajectory from the replicated model-scale comparisons.
  • Our goal with the 250-step run was to measure early fine-tuning loss and memory savings, rather than establishing a downstream performance advantage.

1.6 RL setup

RL: 50 steps with no setup described.

  • We believe this is not accurate, the main text specifically identifies Qwen2.5-3B-Instruct, GRPO, Countdown, rollout batches 64–256, with 3 seeds at batches 64 and 256.
  • We will add any missing RL hyperparams, reward definition, and evaluation protocol. We will make it clear the 50-step result is exploratory and further work can be done to see behavior till convergence.
AuthorsOpenReview ↗

Response to Reviewer JSjf, part 2

1.7 Image generation

Image generation: loss stays within 0.08 nats of random guessing.

  • We ran a 100-update LlamaGen pilot on Imagenette, changing only the input-embedding optimizer and intended to show how this optimizer can generalize outside of text. At the last plotted checkpoint, training losses were 9.6255 for AdamW and 9.6258 for Ember, which is near parity under the experimental setup. The reviewer's concern here is understandable, and we will describe this as an early-training comparison, not evidence of generated-image quality.

1.8 Frequency motivation

Misrepresented motivation: Frequency argument applied equally to AdamW.

  • Our motivation is that AdamW uses O(2VD) floats for each matrix, whereas an O(V+D) approach could suffice and reduce optimizer-state memory. This is not exclusive to Ember: AdamW also captures token-frequency effects. We aim to retain that benefit while storing fewer statistics.
  • In Pythia-410M, token-table moment storage fell from 824.18 MB to 0.411 MB, approximately a 2,000× reduction.

1.9 Momentum

"Momentum is wasted" is incorrect. It is inferred from the negative gradient autocorrelation, which is exactly where momentum helps. Table 3 shows momentum helping at a batch size of 256.

  • In Table 3 (GPT-2-small/FineWeb, 200 updates), adding momentum lowered mean validation loss by 0.02900 nats at batch 256 and raised it by 0.03012 nats at batch 8. Our main text already describes this as having “no consistent benefit” across the two tested batch settings.
  • Omitting the first moment saves VD state values per token table, a 2× reduction in moment storage before factoring the second moment. We will clarify that this is a memory-saving choice evaluated against its effect on loss, rather than a claim that momentum is generally useless.

1.10 Missing competitors

Competitors are missing: Closest alternatives are cited, but never really compared against: Lau and Su (https://arxiv.org/abs/2605.18106), Adam-mini (https://arxiv.org/abs/2406.16793), SM3 (https://arxiv.org/abs/1901.11150), Scion (not cited), Modula, Coupled Adam.

  • Thank you for suggesting these methods. We will expand the related-work discussion to cover them. We have not run matched comparisons against the listed methods.
  • We added matched comparisons against native Adafactor with cosine annealing. Mean validation losses [95% CI] are below; lower is better. The 50k row reports means only. Each row uses its own held-out data, so comparisons should be made within rows.
ExperimentAdamWEmberAnnealed native Adafactor
10k updates, batch 32, 3 seeds3.45801 [3.45000, 3.46603]3.46133 [3.45980, 3.46286]3.44976 [3.43712, 3.46241]
5k updates, batch 64, 3 seeds3.54065 [3.53619, 3.54511]3.54604 [3.54442, 3.54766]3.56018 [3.54859, 3.57176]
50k updates, 2 preserved seeds¹3.092023.100963.09067
  • In the direct 10k comparison, the mean losses were close: Ember minus native was +0.01156 nats [95% CI: −0.00228, +0.02540].
  • At batch 64, Ember minus native was −0.01414 nats [95% CI: −0.02537, −0.00290].
  • Both factored methods use approximately 0.411 MB of token-table moment state in the verified 10k and batch-64 panels, about 2,000× less than AdamW's 824.18 MB.
  • In a separate schedule control, adding cosine to native Adafactor while holding its other settings fixed reduced loss by 0.07429 nats [95% CI: 0.07332, 0.07527].

¹ We ran 3 seeds, but an export error left complete results for 2. We report those 2 means without a confidence interval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.