acceptodds
Withdrawn from ICLR 2027

COREM: Cosine-Relation Momentum Reshaping with Stateful Writeback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Matrix-valued optimizer states may contain relational structure that is not captured by treating their entries independently. We study whether relations within matrix-valued optimizer states can be exploited to improve optimization. To this end, we introduce a unit–relation–transform abstraction and instantiate it as COREM, a Cosine-Relation Momentum Reshaping method with stateful writeback. COREM partitions the momentum state into update units, computes cosine relations among them, and uses these relations to reshape the momentum before writing the transformed state back to the optimizer. This stateful mechanism allows the reshaped momentum to affect not only the current update but also future optimization dynamics. We evaluate COREM on CIFAR-10 with an MLP and on enwik8 with a Transformer. Compared with Muon, COREM shows lower early-stage step efficiency but stronger improvement in the mid-to-late stages of training, achieving better final validation performance on CIFAR-10 and comparable final performance on enwik8. Spectral diagnostics on enwik8 show that COREM consistently increases entropy effective rank and reduces the concentration of singular energy in dominant modes, while preserving an anisotropic spectrum. For square matrix updates, COREM requires approximately 13.3% of the transformation FLOPs of Muon with five Newton–Schulz iterations.

Primary area:
optimization
Keywords:
optimization, relational optimization, stateful writeback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PRNzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces COREM, a mechanism for reshaping a matrix-valued momentum state. It normalizes each row of the momentum matrix, uses pairwise cosine similarities to mix the normalized rows, and then restores the original Frobenius norm. The reshaped matrix is used both for the parameter update and as the stored momentum state.

Critical strengths
  1. The paper is generally easy to follow. Section 3 clearly presents the COREM method.

  2. The spectral diagnostics provide useful insight into how COREM changes the momentum spectrum, showing that the reshaped updates distribute their energy more broadly across singular directions.

Critical weaknesses
  1. The motivation is insufficiently convincing. It is unclear why suppressing dominant shared directions and increasing the relative contribution of weaker directions should improve optimization. The paper provides spectral bounds for the transformation, but these do not establish a connection to improved optimization performance. I would appreciate a clearer explanation of when this reshaping is expected to help and why.

  2. The empirical evidence is also limited. Evaluation is restricted to a small MLP on CIFAR-10 and a small Transformer on enwik8, leaving the effectiveness of COREM at larger scales unclear. COREM improves accuracy over Muon on CIFAR-10, while its final performance on enwik8 is broadly comparable. These results provide some preliminary support, but do not yet convincingly establish the broader value of the proposed mechanism.

Questions for authors
  1. Could the authors clarify why reducing the relative contribution of dominant shared directions in the momentum matrix is expected to improve optimization? Under what conditions would this reshaping help rather than suppress useful gradient information?

  2. The spectral bounds characterize the transformation itself, but its connection to optimization performance remains unclear. Could the authors provide theoretical analysis or controlled experiments that help establish this connection?

Reviewer_JJzgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COREM, an optimization method that adjusts parameter updates by leveraging internal cosine relationships within the momentum matrix and influences subsequent optimization steps through state writeback. The authors compare COREM with Muon on CIFAR-10 and enwik8. The results show that COREM achieves some improvements on CIFAR-10 while performing comparably on enwik8.

Critical strengths
  1. The method demonstrates some novelty by modeling internal relationships within the momentum matrix to adjust the optimization direction, offering a perspective different from that of traditional optimizers.
  2. The experiments include ablation studies and spectral analyses, providing some evidence for the effectiveness of the method and its impact on the optimization process.
Critical weaknesses
  1. The method is evaluated only on CIFAR-10 and enwik8, using relatively small-scale models. Moreover, the comparisons are primarily conducted against Muon, without including other commonly used optimizers. In particular, COREM does not demonstrate a clear advantage on enwik8. Could the authors provide additional experiments to further validate the effectiveness and generalizability of the proposed method?
  2. I noticed that some ablation experiments use different hyperparameter configurations, making it difficult to clearly isolate the individual contributions of each component. Could the authors provide a fairer comparison under consistent hyperparameter settings? Additionally, the claimed computational efficiency advantages are mainly supported by theoretical FLOPs analysis. I would be interested in seeing empirical comparisons of actual training speed and resource consumption to better assess the practical efficiency of the proposed method.
Questions for authors

Please refer to the weakness for questions.

Reviewer_s7KdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes COREM, an optimizer that transforms matrix-valued momentum using relations among its rows. It follows a 5 step straight forward algorithm. The authors present this as an instance of a general units -> relations -> transform abstraction, distinct from Muon’s orthogonalization. The experiments compare against Muon only.

Critical strengths
  1. The method is simple, cheap, and can be used as plug and play.

  2. The authors use a correct and well-used normalization bound.

  3. The authors are honest about the limited ablations for this work.

Critical weaknesses
  1. Ablations that would identify the source of the gains are missing.

  2. The experiments are too small and too narrow to support an optimizer claim. The models are a 3072-256-256-10 MLP at about 60% CIFAR-10 accuracy and a 4-layer, width-192 Transformer reaching 1.58 BPB on enwik8. The work is only compared againts Muon and other baselines like ADAMW is missing.

  3. Both optimizers use a fixed learning rate, that has no warmup and no decay. That’s not how Muon is normally used.

  4. The tuning protocol is unclear, and CIFAR-10 may be tuned on the test set.

  5. No wall clock time measurements are reported. Only FLOPs are reported.

  6. The gains are also very small.

Questions for authors
  1. Can additional ablations help identify the source of the gaints?

  2. How does COREM perform on larger scale experiments (at least GPT 2 scale models)?

  3. How does adding a warmup and decay to the learning rate affect the optimizers? These are essential for an optimizer like Muon.

  4. How was each optimizer exactly tuned? What are the set of hyperparameters tested? What was the validation dataset and what was the final test dataset? Is there any contamination there?

  5. What is the wall clock time speedups of the proposed method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.