- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces COREM, a mechanism for reshaping a matrix-valued momentum state. It normalizes each row of the momentum matrix, uses pairwise cosine similarities to mix the normalized rows, and then restores the original Frobenius norm. The reshaped matrix is used both for the parameter update and as the stored momentum state.
Critical strengths
-
The paper is generally easy to follow. Section 3 clearly presents the COREM method.
-
The spectral diagnostics provide useful insight into how COREM changes the momentum spectrum, showing that the reshaped updates distribute their energy more broadly across singular directions.
Critical weaknesses
-
The motivation is insufficiently convincing. It is unclear why suppressing dominant shared directions and increasing the relative contribution of weaker directions should improve optimization. The paper provides spectral bounds for the transformation, but these do not establish a connection to improved optimization performance. I would appreciate a clearer explanation of when this reshaping is expected to help and why.
-
The empirical evidence is also limited. Evaluation is restricted to a small MLP on CIFAR-10 and a small Transformer on enwik8, leaving the effectiveness of COREM at larger scales unclear. COREM improves accuracy over Muon on CIFAR-10, while its final performance on enwik8 is broadly comparable. These results provide some preliminary support, but do not yet convincingly establish the broader value of the proposed mechanism.
Questions for authors
-
Could the authors clarify why reducing the relative contribution of dominant shared directions in the momentum matrix is expected to improve optimization? Under what conditions would this reshaping help rather than suppress useful gradient information?
-
The spectral bounds characterize the transformation itself, but its connection to optimization performance remains unclear. Could the authors provide theoretical analysis or controlled experiments that help establish this connection?