- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Low-Rank Readout Modulation (LRM), a lightweight gate applied to final hidden state right before the LM head. The main idea is to compute a feature-wise scale from the current token embedding through a low-rank bottleneck, and zero-initialize the gate so the model starts identical to the baseline. Empirical evaluations on Transformer and MHAR backbones, at 115M/300M/1B scales, shows that LRM reduces held-out NLL by 0.024-0.043 and reaches baseline loss with 15-30% fewer steps, while adding at most 0.06% parameters.
Critical strengths
-
The method is simple and lightweight. It leaves the transformer blocks and KV cache as is, and adds at most 0.06% parameters with near-equal throughput, as showed in Table 2. The zero-initialization makes it easier to adopt existing architectures.
-
Empirical evaluations report consistent results. Across three seeds, three model sizes and two backbones, LRM improves NLL by 0.024-0.043 with standard deviations reported. (Table 1)
-
LRM seems to be complementary with MHAR, which already modifies the residual stream. Table 1 shows that LRM still gives 0.024-0.038 gain on NLL on top of MHAR.
Critical weaknesses
-
The source of the gain is somewhat unclear. In Table 3, token-conditioned LRM improves over the factorized static gate by only , which is within the error bar. Since the static gate is equivalent to rescaling the final RMSNorm weights (Equation 7), most gain seems to come from reparameterizing the final norm, rather than token conditioning, which is the central claim of this paper. Also, Appendix E shows that the gain drops notably at FP32 model state, from 0.04 to 0.007, but the issue not addressed in the main text.
-
All models are trained on the same token budget (1.31B), which is 11.3/4.3/1.3 tokens per parameter for 115M/300M/1B, far from conventional training regime. The 1B Transformer is even worse than the 300M Transformer in Table 1, although it also uses a different LR schedule. Therefore, it is unclear whether the NLL gain and 15-30% step saving hold with longer training. Also, the LoRA gain on WinoGrande is within error bar (Table 4), so the transfer results are somewhat hard to interpret.
Questions for authors
-
Does a simple change on the final RMSNorm, e.g. larger LR, no weight decay on its weights, give a similar gain? Also, can the authors report the FP32 results with multiple seeds?
-
Does the gain hold with a larger token budget? Can the authors clarify the reason why the 1B model is worse than the 300M model, undertraining or LR schedule?
-
Why is MHAR chosen as the second backbone? Since LRM only acts on the final readout, it seems applicable to other residual-stream variants as well, such as Hyper-Connections (HC) or mHC. Does the gain still hold on top of them?
2 comments
Thank you for the detailed response and the extensive new experiments; I appreciate that unfavorable outcomes are reported transparently. The first-update analysis and the static/direct controls clarify where the improvement comes from, the multi-seed FP32 results show that the readout gain is consistently positive at 115M, and the matched Hyper-Connections study addresses my question about other residual-stream variants (HC; mHC was not tested).
However, the new results also narrow the scope of the main claims. A static gate recovers most of the gain, so the contribution of token conditioning appears small. Under FP32 model/optimizer state, the improvements at 115M are small (about 0.006–0.017 NLL), and at 1B with 5.24B tokens LRM shows no measurable gain (−0.0028 ± 0.0043). The 15–30% savings are therefore demonstrated mainly in the pure-BF16 recipe, where (at least at 115M) the baseline's final-norm weights remain at initialization, and as crossings along a fixed schedule, without demonstrated equivalent savings under a compressed learning-rate schedule. Downstream results are also mixed (e.g., in the 1B BF16 5.24B-token evaluation, LAMBADA accuracy decreases in every seed).
The abstract, Introduction, and Conclusion still state the 15–30% savings without noting that they were obtained in the pure-BF16 model/optimizer-state recipe, and the Conclusion still states that token conditioning further improves the Transformer results, although this added benefit is small and recipe-dependent (e.g., 0.0032 ± 0.0006 over the static gate in the additional BF16 replication, with no consistent advantage over static under FP32 state). I encourage the authors to scope these claims to the tested recipe and to report the 1B FP32 result in the main text.
Overall, I appreciate the thorough response, but since the demonstrated contribution is narrower than claimed, I will maintain my rating.
Thank you for the suggestions and questions.
1. Final-RMSNorm alternatives, FP32 results and conditioning
We tested the larger learning rate and zero-decay settings you suggested at 115M and 1.31B tokens:
These settings do not reproduce LRM's gain. We then tried LR multipliers {1, 4, 8, 16, 32} with decay 0 or 0.1 on a development run and fixed the best setting. LRM still improves over it by 0.0117 in BF16 and 0.0086 in FP32.
Separate FP32 comparisons give 0.0170 in the IOG study and 0.0116 in the JTok study. We report these separately because each has its own baseline.
Our BF16 recipe has no FP32 weights. Small changes to norm coefficients near one can round back to the same stored value. At 115M, all 512 default BF16 final-norm coefficients still equal one after 10,000 updates and earlier checkpoints show nonzero Adam moments despite unchanged coefficients. With FP32 storage, those coefficients move.
LRM's last matrix starts at zero, where BF16 can retain smaller updates. Several entries in that matrix contribute to each feature's scale, so their updates can add up to a larger initial change at the same learning rate. FP32 lets the baseline retain smaller norm updates too, so this advantage should shrink. Section 3 and the Discussion now explain this mechanism and its limits.
We tested the final-norm explanation while leaving the model in BF16:
Both retain matched BF16 rounding of incoming gradient contributions.
Your reading of Equation 7 is correct for positions after the first token in exact arithmetic. In BF16, folding a small gate correction into a norm coefficient can round it away. Learning that correction through matrices also changes the optimizer's updates. Factorized static beats direct static by 0.0243 in BF16 and 0.0114 in FP32.
Finally, most of the gain comes from the factorized scale and the extra contribution from conditioning is smaller:
2. Longer training, the 1B/300M ordering and transfer
We extended the 1B Transformer runs to 5.2B tokens:
Under the original BF16 recipe, LRM reaches the baseline's final held-out loss with 19.58% fewer tokens on average. Its gain grows from 0.0129 at the 1.3B prefix to 0.0210 at 5.24B.
For the 1B/300M ordering, the larger model receives fewer tokens per parameter: about 1.3 versus 4.3. It also uses half the peak LR and a different schedule. On transfer, WikiText word perplexity decreases (LRM − baseline = −1.875 ± 0.391), while PIQA accuracy increases (+1.106 ± 0.904 pp), in every seed of the new BF16-trained 5.24B comparison. MMLU and CommonsenseQA improve slightly.
3. MHAR, Hyper-Connections and mHC
MHAR changes how information is combined across depth before it reaches the readout. We chose it to ask whether LRM could still help when the backbone already changes the residual stream. We have now added a matched four-stream HC comparison at 115M and 1.31B tokens:
LRM improves every run while adding about 1.66% to full-budget runtime. The gain is smaller with FP32 state, as in the Transformer experiments. These results extend the evidence to HC.