- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper asks whether Muon needs fine-grained spectral shaping or whether a two-band reweighting is enough. It proposes BulkBoost, which splits the singular directions of Muon’s momentum into a few leading spikes and a large bulk, using either a fixed fraction or a noise edge, and reweights the two bands with one shared gain. In continued pretraining of Pythia models from 14M to 410M parameters on six corpora, it reports small gains over Muon and over the power-law profile of Freon.
Critical strengths
None
Critical weaknesses
- The central claim is not supported. Neither BulkBoost nor Freon improves on Muon by more than 0.15% of the initial loss on average, and BulkBoost’s gain of about 0.0025 in loss comes with steps that are 20% to 50% slower, so even where it wins it reaches Muon’s final loss later in wall-clock time. The comparison also favors BulkBoost. It uses Polar Express while Muon uses Newton–Schulz, Muon with Polar Express is missing, and the shared learning rate at 70M is above Muon’s own optimum, which is where most of the gain comes from. At 160M, BulkBoost is worse than Muon on four of six corpora. The key quantity, how much of the benefit two bands can capture, is defined but never measured.
- The writing and structure do not meet basic standards. The abstract runs to two paragraphs, although the ICLR template limits it to one. There is no conclusion section, and the conclusion and limitations appear as short paragraphs inside Section 5.1, a subsection on comparing methods. Notation is inconsistent. For example, ∆ is defined as the change in loss during continued pretraining but used for the gap to Muon in Table 2, and q denotes both the number of block iterations and the spike fraction. The main claims are stated as win counts, where a lower three-seed mean at the printed precision of 0.001 counts as a win even when the gap is within the seed standard deviation.
Questions for authors
Does the gain over Muon survive a fair comparison? Please add Muon with Polar Express and tune the learning rate separately for each method, and identify how many of the wins exceed seed noise?