- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Ember, a lightweight optimizer for embedding and LM-head matrices that factorizes the second-moment estimate into row and column statistics, reducing state memory size from to . The paper studies the gradient structure of token-interface parameters. Experiments on different settings show that Ember can match or outperform AdamW while substantially reducing token-interface optimizer state.
Critical strengths
- The paper's evaluation result shows a large portion of optimizer-state size reduction.
Critical weaknesses
-
The presentation is weak and makes the paper difficult to follow. The motivation, technical story, and positioning are not clearly organized, making it hard to identify the paper's central insight and the position in this field.
-
The novelty is limited. As noted in the paper, Ember's update coincides with Algorithm 2 of Adafactor under matched settings. While the paper validates its effectiveness on LLM, stronger analysis and controlled comparisons are needed to differentiate a research contribution from an engineering adaptation.
Questions for authors
What evidence shows that Ember is particularly suitable for token interfaces, instead of simply reusing Adafactor's update?
Response to Reviewer BwMV
3.1 Presentation and central question
3.2 Novelty and controlled comparisons
Mean validation loss [95% CI]; lower is better. Compare methods within each row, since the studies use different held-out data. The 50k row reports means only.
3.3 Evidence for token-table suitability