acceptodds
Under review as a conference paper at ICLR 2027

Efficient Language Model Pretraining via Low-Rank Readout Modulation

Abstract

Efficient pretraining of language models (LMs) requires reaching a desired model quality with fewer optimization steps and minimal additional cost per step. Most work pursues this goal through data curation, improved optimizers, or more efficient Transformer variants. In this work, we focus on a less explored aspect: the LM's readout head. We introduce Low-Rank Readout Modulation (LRM), a simple, lightweight feature-wise gate applied once between the final normalization and tied LM head. LRM derives its scale from the embedding of the observed token, leaves the Transformer stack and KV cache unchanged, and requires neither another decoder pass nor a custom kernel. Across three seeds at 115M, 300M, and 1B scales, LRM improves held-out negative log-likelihood (NLL) on both standard Transformer and Multi-Head Attention Residuals (MHAR) backbones, with mean gains of 0.024–0.043 NLL. It adds at most 0.06% parameters to the base model and, at similar throughput, reaches loss targets with 15–30% fewer steps. Across six backbone–task pairs, parameter-efficient fine-tuning of models pretrained with LRM improves mean accuracy in five.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model, efficient pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JkMNOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Low-Rank Readout Modulation (LRM), a lightweight gate applied to final hidden state right before the LM head. The main idea is to compute a feature-wise scale from the current token embedding through a low-rank bottleneck, and zero-initialize the gate so the model starts identical to the baseline. Empirical evaluations on Transformer and MHAR backbones, at 115M/300M/1B scales, shows that LRM reduces held-out NLL by 0.024-0.043 and reaches baseline loss with 15-30% fewer steps, while adding at most 0.06% parameters.

Critical strengths
  1. The method is simple and lightweight. It leaves the transformer blocks and KV cache as is, and adds at most 0.06% parameters with near-equal throughput, as showed in Table 2. The zero-initialization makes it easier to adopt existing architectures.

  2. Empirical evaluations report consistent results. Across three seeds, three model sizes and two backbones, LRM improves NLL by 0.024-0.043 with standard deviations reported. (Table 1)

  3. LRM seems to be complementary with MHAR, which already modifies the residual stream. Table 1 shows that LRM still gives 0.024-0.038 gain on NLL on top of MHAR.

Critical weaknesses
  1. The source of the gain is somewhat unclear. In Table 3, token-conditioned LRM improves over the factorized static gate by only 0.006±0.0050.006 \pm 0.005, which is within the error bar. Since the static gate is equivalent to rescaling the final RMSNorm weights (Equation 7), most gain seems to come from reparameterizing the final norm, rather than token conditioning, which is the central claim of this paper. Also, Appendix E shows that the gain drops notably at FP32 model state, from 0.04 to 0.007, but the issue not addressed in the main text.

  2. All models are trained on the same token budget (1.31B), which is 11.3/4.3/1.3 tokens per parameter for 115M/300M/1B, far from conventional training regime. The 1B Transformer is even worse than the 300M Transformer in Table 1, although it also uses a different LR schedule. Therefore, it is unclear whether the NLL gain and 15-30% step saving hold with longer training. Also, the LoRA gain on WinoGrande is within error bar (Table 4), so the transfer results are somewhat hard to interpret.

Questions for authors
  1. Does a simple change on the final RMSNorm, e.g. larger LR, no weight decay on its weights, give a similar gain? Also, can the authors report the FP32 results with multiple seeds?

  2. Does the gain hold with a larger token budget? Can the authors clarify the reason why the 1B model is worse than the 300M model, undertraining or LR schedule?

  3. Why is MHAR chosen as the second backbone? Since LRM only acts on the final readout, it seems applicable to other residual-stream variants as well, such as Hyper-Connections (HC) or mHC. Does the gain still hold on top of them?

2 comments

Thank you for the suggestions and questions.

1. Final-RMSNorm alternatives, FP32 results and conditioning

We tested the larger learning rate and zero-decay settings you suggested at 115M and 1.31B tokens:

NLL improvement over baselineBF16 stateFP32 state
Final-norm LR ×40.0013 ± 0.0113−0.0028 ± 0.0054
Final-norm weight decay 00.0006 ± 0.0026−0.0041 ± 0.0025
LR ×4 and decay 00.0074 ± 0.00780.0016 ± 0.0033
Factorized static0.0437 ± 0.00400.0118 ± 0.0056
LRM0.0441 ± 0.00670.0068 ± 0.0016

These settings do not reproduce LRM's gain. We then tried LR multipliers {1, 4, 8, 16, 32} with decay 0 or 0.1 on a development run and fixed the best setting. LRM still improves over it by 0.0117 in BF16 and 0.0086 in FP32.

Separate FP32 comparisons give 0.0170 in the IOG study and 0.0116 in the JTok study. We report these separately because each has its own baseline.

Our BF16 recipe has no FP32 weights. Small changes to norm coefficients near one can round back to the same stored value. At 115M, all 512 default BF16 final-norm coefficients still equal one after 10,000 updates and earlier checkpoints show nonzero Adam moments despite unchanged coefficients. With FP32 storage, those coefficients move.

LRM's last matrix starts at zero, where BF16 can retain smaller updates. Several entries in that matrix contribute to each feature's scale, so their updates can add up to a larger initial change at the same learning rate. FP32 lets the baseline retain smaller norm updates too, so this advantage should shrink. Section 3 and the Discussion now explain this mechanism and its limits.

We tested the final-norm explanation while leaving the model in BF16:

  • N1: store and update final-norm coefficients and Adam moments in FP32, keeping the coefficient multiplication in BF16.
  • N2: also perform that multiplication and its derivative in FP32.

Both retain matched BF16 rounding of incoming gradient contributions.

ComparisonNLL advantage of LRM
Transformer, versus N10.0115 ± 0.0019
Transformer, versus N20.0074 ± 0.0016
MHAR, versus N1−0.0008 ± 0.0075

Your reading of Equation 7 is correct for positions after the first token in exact arithmetic. In BF16, folding a small gate correction into a norm coefficient can round it away. Learning that correction through matrices also changes the optimizer's updates. Factorized static beats direct static by 0.0243 in BF16 and 0.0114 in FP32.

Finally, most of the gain comes from the factorized scale and the extra contribution from conditioning is smaller:

ComparisonStatic − LRM NLL
Repeat of original 115M recipe, BF16 state+0.0004 ± 0.0065
Repeat of original 115M recipe, FP32 state−0.0050 ± 0.0063
Separate BF16 final-norm control study, Transformer+0.0032 ± 0.0006
Separate BF16 final-norm control study, MHAR+0.0063 ± 0.0050

2. Longer training, the 1B/300M ordering and transfer

We extended the 1B Transformer runs to 5.2B tokens:

1B training recipeBaseline NLLLRM NLLBaseline − LRM
BF16 state; original 2,500-step decay, then fixed LR floor3.3830 ± 0.00543.3620 ± 0.0026+0.0210 ± 0.0033
FP32 state; 40,000-step decay2.7131 ± 0.00552.7159 ± 0.0019−0.0028 ± 0.0043

Under the original BF16 recipe, LRM reaches the baseline's final held-out loss with 19.58% fewer tokens on average. Its gain grows from 0.0129 at the 1.3B prefix to 0.0210 at 5.24B.

For the 1B/300M ordering, the larger model receives fewer tokens per parameter: about 1.3 versus 4.3. It also uses half the peak LR and a different schedule. On transfer, WikiText word perplexity decreases (LRM − baseline = −1.875 ± 0.391), while PIQA accuracy increases (+1.106 ± 0.904 pp), in every seed of the new BF16-trained 5.24B comparison. MMLU and CommonsenseQA improve slightly.

3. MHAR, Hyper-Connections and mHC

MHAR changes how information is combined across depth before it reaches the readout. We chose it to ask whether LRM could still help when the backbone already changes the residual stream. We have now added a matched four-stream HC comparison at 115M and 1.31B tokens:

HC + LRM versus HCNLL improvementFewer target tokensMeasured time to target
BF16 state0.0531 ± 0.007534.17%≥32.22% less
FP32 state0.0128 ± 0.005910.83%≥8.71% less

LRM improves every run while adding about 1.66% to full-budget runtime. The gain is smaller with FP32 state, as in the Transformer experiments. These results extend the evidence to HC.

Thank you for the detailed response and the extensive new experiments; I appreciate that unfavorable outcomes are reported transparently. The first-update analysis and the static/direct controls clarify where the improvement comes from, the multi-seed FP32 results show that the readout gain is consistently positive at 115M, and the matched Hyper-Connections study addresses my question about other residual-stream variants (HC; mHC was not tested).

However, the new results also narrow the scope of the main claims. A static gate recovers most of the gain, so the contribution of token conditioning appears small. Under FP32 model/optimizer state, the improvements at 115M are small (about 0.006–0.017 NLL), and at 1B with 5.24B tokens LRM shows no measurable gain (−0.0028 ± 0.0043). The 15–30% savings are therefore demonstrated mainly in the pure-BF16 recipe, where (at least at 115M) the baseline's final-norm weights remain at initialization, and as crossings along a fixed schedule, without demonstrated equivalent savings under a compressed learning-rate schedule. Downstream results are also mixed (e.g., in the 1B BF16 5.24B-token evaluation, LAMBADA accuracy decreases in every seed).

The abstract, Introduction, and Conclusion still state the 15–30% savings without noting that they were obtained in the pure-BF16 model/optimizer-state recipe, and the Conclusion still states that token conditioning further improves the Transformer results, although this added benefit is small and recipe-dependent (e.g., 0.0032 ± 0.0006 over the static gate in the additional BF16 replication, with no consistent advantage over static under FP32 state). I encourage the authors to scope these claims to the tested recipe and to report the 1B FP32 result in the main text.

Overall, I appreciate the thorough response, but since the demonstrated contribution is narrower than claimed, I will maintain my rating.

Reviewer_dnAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Low-Rank Readout Modulation (LRM), a token-conditioned gate that rescales the final normalized hidden states before the tied LM head. Experiments cover Transformer and MHAR backbones at 115M, 300M, and 1B parameters, each trained on 1.31B tokens. LRM improves held-out NLL and reaches baseline losses with fewer training steps.

Critical strengths

1.The motivation is good, focusing on improving pretraining efficiency through lightweight readout modulation.

2.LRM consistently improves held-out NLL across three model sizes and two backbones, with minimal parameter overhead.

Critical weaknesses

1.Although experiments cover three model sizes, even the 1B model is trained on only 1.31B tokens, limiting the generalizability of the findings to longer pretraining budgets.

2.Section 3 characterizes what LRM computes but does not clearly explain why it accelerates optimization.

3.Factorized static scaling achieves gains close to full LRM, leaving the additional benefit of token conditioning unclear. Ablations at 300M or larger scales with more training tokens would strengthen this claim.

4.Despite discussing IOG and JTok, the paper lacks comparisons with these methods under matched training settings in terms of NLL, parameter overhead, and training time.

Questions for authors

1.Do the NLL and time-to-target improvements persist with longer pretraining budgets, particularly for the 1B model?

2.Could you provide empirical insight into why LRM accelerates optimization?

3.Could you provide a matched comparison with a closely related gating method, such as IOG or JTok, or clarify the challenges that prevent a fair comparison?

3 comments

Thank you for the suggestions.

1. Longer pretraining budgets

We extended the 1B Transformer runs to 5.24B FineWeb-Edu tokens:

1B training recipeBaseline NLLLRM NLLBaseline − LRM
BF16 state; original 2,500-step decay, then fixed LR floor3.3810 ± 0.00543.3620 ± 0.0026+0.0210 ± 0.0033
FP32 state; 40,000-step decay2.7131 ± 0.00552.7159 ± 0.0019−0.0028 ± 0.0043

Under the original BF16 recipe, both gains persist. LRM has lower NLL in every seed and reaches the paired baseline's final held-out loss with 19.58% fewer tokens on average. The NLL gain also grows along these same trajectories, from 0.0129 at 1.3B tokens to 0.0210 at 5.24B.

2. Why LRM can accelerate optimization

The static gate recovers most of LRM's improvement by applying the same feature scales to every token. Most of the gain therefore does not require token conditioning. The gate changes how the scales are learned. After the first token, RMSNorm could apply the same static scales in exact arithmetic. Its coefficients start at one and are updated directly. Our gate learns a correction through two matrices, with the last initialized to zero. Several entries in the last matrix contribute to each feature's scale. Their Adam updates can add up to a larger initial change than a direct coefficient update at the same learning rate. Section 3 derives this first-update effect. We also tested whether simply changing the final RMSNorm optimizer could reproduce the gain:

ComparisonNLL gain (BF16)NLL gain (FP32)
Factorized static gate over direct static scale0.0243 ± 0.00090.0114 ± 0.0060
LRM over selected final-RMSNorm LR/decay setting0.0117 ± 0.00340.0086 ± 0.0062

The static/direct comparison also changes normalization, bounds and regularization, so it does not isolate factorization. We selected the norm optimizer from LR multipliers {1, 4, 8, 16, 32} and decay 0 or 0.1 on a development run, fixing it before confirmation. The selected ×32 is the largest tested; LRM still improves on it.

In our BF16 recipe, parameters and Adam states are stored directly in BF16. Small updates to a coefficient near one can round back to the same stored value. At 115M, all 512 default BF16 final-norm coefficients still equal one after 10,000 updates and earlier checkpoints show nonzero Adam moments despite unchanged coefficients. With FP32 storage, the coefficients move.

LRM's last matrix starts at zero, where BF16 can retain smaller updates. FP32 lets the baseline retain smaller norm updates too, so this advantage should shrink. That fits the reduction in LRM's gain from 0.0441 ± 0.0067 to 0.0068 ± 0.0016, although changing stored precision affects other arithmetic too. We tested the final-norm explanation while keeping the model body in BF16:

  • N1: store and update the final-norm coefficients and Adam moments in FP32, keeping the coefficient multiplication in BF16.
  • N2: also perform that multiplication and its derivative in FP32.

Both controls retain matched BF16 rounding of incoming gradient contributions.

ComparisonNLL advantage of LRM
Transformer, versus N10.0115 ± 0.0019
Transformer, versus N20.0074 ± 0.0016
MHAR, versus N1−0.0008 ± 0.0075

These fixes reproduce most of the BF16 gain and match LRM on MHAR within variation. On Transformer, LRM still improves every seed. It also retains a 0.0171 ± 0.0004 gain when both arms use FP32 norms, QK/RoPE arithmetic and gradient clipping around a BF16.

3. What token conditioning adds

Most of the improvement is already present in the factorized static gate. The table below measures what conditioning adds (positive values mean that LRM has lower NLL than static):

ComparisonStatic − LRM NLL
Repeat of original 115M Transformer recipe, BF16 state+0.0004 ± 0.0065
Repeat of original 115M Transformer recipe, FP32 state−0.0050 ± 0.0063
BF16 final-norm control study, Transformer+0.0032 ± 0.0006
BF16 final-norm control study, MHAR+0.0063 ± 0.0050
Separate BF16 MHAR baseline/static/LRM study+0.0075 ± 0.0052
FP32 norms/QK/RoPE/clipping shared by all arms, BF16 body+0.0035 ± 0.0054
1B, 5.24288B tokens, FP32 state+0.0002 ± 0.0013

Conditioning helps in every seed of the BF16. Its benefit is small or variable in the other comparisons. We therefore describe it as a smaller addition whose benefit depends on the recipe, rather than as necessary for the main gain.

The larger-scale ablation you requested is the 1B, 5.24B-token baseline/static/LRM comparison. Static and LRM finish at nearly the same NLL.

4. Matched IOG and JTok comparisons

We added both comparisons using the same 115M Transformer backbone and 1.3B training tokens.

For IOG, we jointly train its vocabulary-logit gate with the Transformer, using a separate 64-dimensional token embedding. The original method trained the gate on a frozen RNN. We test the sigmoid gate, which starts with scales of 0.5, and the doubled-sigmoid version, which starts at one.

For JTok, we apply the Basic token-indexed gate to every MLP residual branch. We test scaler initializations s=0 and s=1: s=0 starts with the baseline function, while s=1 applies a nonzero gate from the start.

ComparatorAdded parametersDELTA NLL (BF16)DELTA LRM NLL (FP32)
IOG, sigmoid19,599,744+0.0717 ± 0.0050+0.0089 ± 0.0031
IOG, identity-centered (doubled sigmoid)19,599,744+0.0281 ± 0.0067+0.0098 ± 0.0014
JTok, s=1933,500,928+0.01320 ± 0.0050−0.0298 ± 0.0065
JTok, s=0933,500,928+0.0356 ± 0.0037−0.0202 ± 0.0095

Positive values favor LRM, which adds 66,048 parameters. LRM has lower NLL than both IOG versions. It also has 27–29% higher throughput, 21–23% shorter full-budget training time and 36–38% lower peak memory.

Against JTok, the result depends on precision: LRM has lower NLL in BF16, while JTok has lower NLL in FP32. LRM takes about 10–11% less full-budget training time than JTok.

Thank you for the additional experiments. However, the FP32 norm controls recover most of the improvement, and it remains unclear whether the 1B/5.24B-token gains persist against a baseline with corrected norm precision. Static scaling also nearly matches full LRM at 1B, leaving limited evidence for the additional benefit of token conditioning. I therefore maintain my current rating.

Reviewer_ercFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces Low-Rank Readout Modulation (LRM) for efficient pretraining, which is a lightweight feature-wise gate interleaved between the final normalization and tied LM head. Preserving the original transformer stack and KV cache, LRM improves NLL successfully, only requiring small additional parameters, similar throughput, and significantly fewer training steps. The effectiveness of LRM is also supported by the performance improvement of the downstream tasks when evaluating the fine-tuned LRM-pretrained model.

Critical strengths

This paper is well written, and LRM is simple to adapt to any other langauge model architecture. By introducing a small readout module right before LM head, LRM efficienctly reduce the training budget to reach the same level of training loss compared with vanilla transformer stack and MHAR, requring only small number of parameters.

Critical weaknesses

The reason why LRM works for efficient pretraining is not clear. For example, in section 3, authors explain how LRM modifies the LM head but do not clearly explain why it improves pretraining efficiency. Plus, the factorized static gate achieves most of LRM's performance gains, making the contribution of token conditioning unclear.

The downstream accuracy is evaluated on only three tasks after LoRA fine-tuning. A broader evaluation, including zero-shot performance before lora fine-tuning, would help the readers to understnad the effectiveness of LRM.

The paper does not compare LRM with a conventional untied LM head, making it difficult to determine whether LRM mainly addresses limitations introduced by weight tying.

The training budget is too small. All models are trained on only 1.31B tokens, even for 1B models. It remains unclear whether the reported efficiency gains persist under longer pretraining.

Questions for authors

Could the authors explain why LRM improves pretraining efficiency? In addition, why does the factorized static gate achieve most of the performance gains without token conditioning? The gating from the token is really needed?

Could the authors evaluate more downstream tasks, such as ARC-Easy, ARC-Challenge, MMLU, and CommonsenseQA, and so on, including zero-shot accuracy before LoRA fine-tuning?

Could the authors compare LRM with an untied LM head trained from scratch, while accounting for differences in parameter count or any complexity?

Do the improvements in training efficiency persist with larger pretraining token budgets?

+) Please add AI use statement section

3 comments

Thank you for the questions about the source of the gain and the missing comparisons.

1. Why LRM helps, and whether token conditioning is needed

The static gate recovers most of LRM's improvement by applying the same feature scales to every token. Most of the gain therefore does not require token conditioning. The gate changes how the scales are learned. After the first token, RMSNorm could apply the same static scales in exact arithmetic. Its coefficients start at one and are updated directly. Our gate learns a correction through two matrices, with the last initialized to zero. Several entries in the last matrix contribute to each feature's scale. Their Adam updates can add up to a larger initial change than a direct coefficient update at the same learning rate. Section 3 derives this first-update effect. We also tested whether simply changing the final RMSNorm optimizer could reproduce the gain:

ComparisonNLL gain (BF16)NLL gain (FP32)
Factorized static gate over direct static scale0.0243 ± 0.00090.0114 ± 0.0060
LRM over selected final-RMSNorm LR/decay setting0.0117 ± 0.00340.0086 ± 0.0062

The static/direct comparison also changes normalization, bounds and regularization, so it does not isolate factorization. We selected the norm optimizer from LR multipliers {1, 4, 8, 16, 32} and decay 0 or 0.1 on a development run, fixing it before confirmation. The selected ×32 is the largest tested; LRM still improves on it.

In our BF16 recipe, parameters and Adam states are stored directly in BF16. Small updates to a coefficient near one can round back to the same stored value. At 115M, all 512 default BF16 final-norm coefficients still equal one after 10,000 updates and earlier checkpoints show nonzero Adam moments despite unchanged coefficients. With FP32 storage, the coefficients move.

LRM's last matrix starts at zero, where BF16 can retain smaller updates. FP32 lets the baseline retain smaller norm updates too, so this advantage should shrink. That fits the reduction in LRM's gain from 0.0441 ± 0.0067 to 0.0068 ± 0.0016, although changing stored precision affects other arithmetic too. We tested the final-norm explanation while keeping the model body in BF16:

  • N1: store and update the final-norm coefficients and Adam moments in FP32, keeping the coefficient multiplication in BF16.
  • N2: also perform that multiplication and its derivative in FP32.

Both controls retain matched BF16 rounding of incoming gradient contributions.

ComparisonNLL advantage of LRM
Transformer, versus N10.0115 ± 0.0019
Transformer, versus N20.0074 ± 0.0016
MHAR, versus N1−0.0008 ± 0.0075

These fixes reproduce most of the BF16 gain and match LRM on MHAR within variation. On Transformer, LRM still improves every seed. It also retains a 0.0171 ± 0.0004 gain when both arms use FP32 norms, QK/RoPE arithmetic and gradient clipping around a BF16.

2. Evaluation before LoRA

We evaluated the requested tasks before LoRA. For the 1B Transformers trained with BF16 state on 5.24B tokens:

Zero-shot metricLRM − baselineBetter metric value
WikiText word perplexity−1.875 ± 0.391Lower
PIQA accuracy+1.106 ± 0.904Higher
ARC-Easy normalized accuracy+0.477 ± 1.004Higher
ARC-Challenge normalized accuracy−0.398 ± 1.503Higher
MMLU accuracy, average over 57 subjects+0.115 ± 0.049Higher
CommonsenseQA accuracy+0.137 ± 0.095Higher
WinoGrande accuracy−0.184 ± 1.331Higher
LAMBADA accuracy−0.893 ± 0.439Higher

3. Untied output heads

We trained tied and untied models from scratch at 115M, each with and without LRM. The untied output matrix starts as a copy of the randomly initialized input embedding, so the initial logits match. The matrices then train independently.

LRM added to an untied modelNLL improvement
BF16 state0.0351 ± 0.0026
FP32 state0.0056 ± 0.0028

LRM improves every seed and retains about 80–82% of its tied-head gain. Untying alone raises NLL by 0.0084 ± 0.0023 in BF16 and 0.0111 ± 0.0042 in FP32 at this budget.

The parameter costs differ substantially: untying adds 77,791,232 parameters, while LRM adds 66,048. Both still use one vocabulary projection. The revised appendix reports measured memory, throughput and training time. Most of LRM's gain remains after removing weight tying, so tying is unlikely to be its main explanation in this setting.

4. Longer pretraining

The gain persists under the original BF16 recipe. We trained the 1B Transformer for 5.24B FineWeb-Edu tokens:

1B training recipeBaseline NLLLRM NLLBaseline − LRM
BF16 state; original 2,500-step decay, then fixed LR floor3.3830 ± 0.00543.3620 ± 0.0026+0.0210 ± 0.0033
FP32 state; 40,000-step decay2.7131 ± 0.00552.7159 ± 0.0019−0.0028 ± 0.0043

With BF16 state, LRM improves every seed and reaches the baseline's final held-out loss with 19.58% fewer tokens on average. Its gain grows from 0.0129 ± 0.0047 at 1.3B tokens to 0.0210 ± 0.0033 at 5.24B.

5. Please add AI use statement section

Done

Thank you to the authors for the detailed response. I appreciate the additional analyses, and they address several of my original concerns. However, the contribution of token conditioning remains limited, as the static gate reproduces most of the performance gains. Moreover, while LRM maintains its improvements under the original BF16 training recipe, the extended 5.24B token experiments show no consistent advantage under FP32 model/optimizer states and a full learning-rate decay schedule, raising concerns about the generality of its pretraining efficiency gains. Overall, while the rebuttal clarifies several important points, these concerns remain. I will raise my score to 2 and adjust my Soundness rating to 2 as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.