acceptodds
Withdrawn from ICLR 2027

Output-Head Factorization Improves Language Models at No Inference Cost

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing interventions on a language model's output head, including weight tying, learned output maps, mixture models, and low-rank variants, often combine changes to parameter sharing, rank, initialization, or function class. We separate parameter sharing from training-time factorization with matched interventions that change one named axis at a time. Multiplicative factorization of the output head substantially lowers perplexity across the evaluated corpora and architectures, yet folds to a single output projection before inference with no factorization-specific computation. Across dense tied and untied references, perplexity differs by at most 1.8%, whereas the best factorized endpoint is 14.9% lower than its initialization-matched direct reference. The main causal result holds the initial output matrix fixed and changes only its factor split, lowering perplexity by 7.5%. Direct addition provides strong alternatives but does not reach the best factorized endpoint. A local update diagnostic characterizes how factorization changes the output-head update; the endpoint effect is established by intervention. Together, these results show that training-time parameterization of the output head is a consequential deployment-neutral design choice.

Primary area:
foundation or frontier models, including LLMs
Keywords:
output head parameterization, language model pretraining, matrix factorization, overparameterization, initialization, implicit acceleration, optimization geometry, weight tying, structural reparameterization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WKC3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies full-width linear factorizations of language-model output heads, such as W=ARW=AR and W=ARSW=ARS. These factors are optimized during training and folded into one output projection before inference, preserving deployment cost relative to a direct head with the same sharing convention. Experiments examine initialization, factor depth, placement, sharing, and classifier-to-embedding gradient coupling. The central intervention preserves the initial folded matrix W0=E0W_0=E_0 and initial logits while changing the factor split from concentrated to SVD-balanced. Under the reported AdamW protocol, balanced initialization improves perplexity across Qwen2-0.5B, Qwen2-1.5B, and Pythia-410m, with larger gains on less frequent tokens. Local gradient diagnostics and early-loss screening provide additional analyses.

Critical strengths
  1. Concentrated and SVD-balanced factors start with identical folded output weights and logits while holding the architecture and optimizer protocol fixed. Table 6 reports consistent improvements across three model configurations, including larger improvements on less frequent tokens. This establishes an effect beyond changing the initial prediction function.

  2. The input-only factorization control reaches 46.86 PPL versus 44.01 for output-side depth 2, limiting an explanation based on factorizing the input table alone. The stop-gradient control isolates classifier-to-embedding gradient coupling while preserving unrestricted output adaptation through a residual matrix. Folding the factors preserves a single output projection at inference under the same sharing convention.

Critical weaknesses
  1. A tuned direct-head baseline is especially important given the pure-addition result. The direct sum W=W1+W2W=W_1+W_2 reaches 41.36 PPL, versus 47.02 for the direct head and 40.70 for balanced depth 2 (Tables 3, 6, and 10). Numerically, this closes about 90% of that PPL difference. For the same folded initialization, identical component optimizer states and settings, and matching gradient processing, its folded AdamW update is Wt+1=(1−δt)Wt−2ηtutW_{t+1}=(1-\delta_t)W_t-2\eta_tu_t, where δt\delta_t is the per-step decay and utu_t the shared adaptive direction. This corresponds to doubling the direct head's gradient step while preserving its decay multiplier. The actual clipping configuration needs clarification. Since head-specific learning rates were not tuned, the advantage over a simpler optimization adjustment remains unresolved.

  2. The main factorial trains from scratch on approximately 652M tokens, or 1.32 tokens per parameter; Qwen2-1.5B receives 0.423 tokens per parameter (Section 2; Appendix G). Figure 1 shows an early separation followed by a smaller persistent gap, with Qwen2-0.5B approximately stable near −0.072-0.072 nats through the second half. This supports a benefit at the tested horizon, while leaving its persistence under substantially more training unknown. The differing third-factor outcomes in Table 7 further limit general guidance about factor depth.

  3. Table 4 measures hypothetical Euclidean-GD updates in the concentrated gauge using a shared reference matrix. It does not measure the balanced head's actual AdamW updates. Its large rare-token amplification and near doubling with depth characterize that construction; the doubling is an expected consistency check from Eq. (2). The authors explicitly acknowledge these limits, but the diagnostic and Appendix E's weak descriptive correlations do not establish why balanced initialization improves the endpoint, particularly on rare tokens. This limits the explanatory contribution beyond the observed intervention effect.

Questions for authors
  1. How much benefit remains over a tuned direct head? Please specify the component initialization, optimizer states, parameter groups, decay, and clipping for W=W1+W2W=W_1+W_2. Could you compare a small set of head-only learning-rate multipliers, including 22, while keeping W0=E0W_0=E_0 and the backbone schedule fixed and matching per-step decay and gradient processing? Under the standard AdamW convention, doubling the head learning rate requires halving its weight-decay coefficient to preserve that multiplier. Please report endpoint and frequency-bucket PPL against the direct sum and balanced depth-2 head. This would directly test the strongest alternative explanation.

  2. Does the advantage persist with more training? Do existing longer runs, or a targeted extension comparing a tuned direct head with balanced depth 2 under a shared extended schedule and data budget, establish whether the validation gap remains stable? Please distinguish the training-loss trajectories in Figure 1 from held-out validation behavior.

Reviewer_tsc1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the author proposes to factorize the output header during the training phase and merge it into a matrix during deployment. This method can improve the prediction performance of next token by factoring the output header only during training without additional deployment overhead. Through a large number of experiments, it is proved that the factorization initialization method can reduce the PPL of the model, and it is found that the factorization method will affect the gradient direction during training and the update range of different word frequency tokens.

Critical strengths
  1. In this paper, when deployed, the structure of the model is not changed, no additional computational overhead is introduced, the next token prediction performance is improved, and the PPL is reduced by up to 7.5 %.

  2. Through a large number of comparative experiments, the effectiveness of output head factorization in the training process is illustrated.

  3. By keeping the weights before and after folding consistent with the initial logits, this paper conducts independent analysis of factorization to reduce the interference of coupling factors.

Critical weaknesses
  1. In this paper, the existing factorization and initialization strategies are applied to the output header of the language model, and its contribution is more biased towards the systematic experimental analysis of the influence of training parameterization and initialization, rather than proposing a new method. In addition, Table 6-7 still covers a limited model architecture and parameter size, which is not enough to support the applicability of the conclusions under a wider set of settings. It is recommended to supplement experiments on Qwen3.5, Gemma4 and other model series, and further cover models with about 8B parameters to test the generalization ability of conclusions under different model architectures and larger parameter scales.

  2. The additional parameters introduced by the factorization method in training increase the training cost and do not further analyze the training cost. In addition, the current experimental indicators are only PPL, and no downstream tasks are tested, which can only reflect the language modeling ability, and it is difficult to judge the use value in the actual scene. It is recommended to supplement training parameters, memory usage and total training time, and test on MMLU, ARC-E, ARC-C and other evaluations to more comprehensively evaluate the trade-off between performance benefits and training costs.

  3. This paper attempts to explain the improvement of next-token prediction performance brought by factorization through the change of gradient direction and the relative update amplification of the corresponding parameters of very low frequency words. However, the local first-order analysis of Table 4 mainly reveals the differences in update behavior under different parameterization methods, and the direct relationship between these differences and the final performance improvement has not been established. It is recommended to test whether changing the update direction or controlling the update range corresponding to different word frequencies will affect the training results through more targeted intervention experiments, and distinguish the respective roles of these factors, so as to provide more sufficient mechanism evidence for the performance advantages of factorization.

  4. The experiment in this paper mainly focuses on internal control and ablation, and the direct comparison with the existing language model output head improvement method is still insufficient. Although the author has cited Gulordava et al. ( 2018 ) ' s _ How to Represent a Word and Predict It, Too : Improving Tied Architectures for Language Modelling _ and set the relevant hidden-side transformation controls, it is more like ablation. It is recommended to supplement the direct comparison under the condition of matching the training budget and the optimization settings to distinguish the benefits of the design from the benefits of the existing output transformation methods.

  5. At present, multiple sets of control, ablation and mechanism analysis are carried out side by side, and the relationship between the main experiment and the auxiliary experiment is not clear enough. It is suggested to reorganize the experimental part around the main research questions.

Questions for authors
  1. Can we provide the results of downstream tests to prove the effectiveness of the method, such as MMLU type multi-knowledge tasks, or Arc-E, Arc-C and other commonsense tasks. At the same time, the comparison with other existing processing methods for the output head should be increased.

  2. Whether the SVD decomposition has the corresponding truncation, if not, whether it can provide the corresponding additional training parameters and training overhead analysis.

  3. Through more targeted intervention experiments, test whether changing the update direction or controlling the update range corresponding to different word frequencies will affect the training results, and distinguish the respective effects of these factors.

  4. Whether it is effective under the newer model ? Such as qwen3.5 and gemma4.

Reviewer_eB1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-time factorization of language model output heads. By decomposing the output projection into multiple trainable matrices and folding them into a single matrix at inference, the method improves language modeling performance without additional inference overhead. The paper further studies the effects of factorization depth, weight tying, and initialization, highlighting the benefits of SVD-balanced initialization.

Critical strengths
  • The proposed approach is simple and introduces no additional inference overhead, then it shows improvements on perplexity, while controlled comparisons provide useful insights into the effects of different factorization methods.
Critical weaknesses
  • The experiments are mostly limited to a single model configuration and random seed. More comprehensive evaluations across different models, multiple seeds, learning rates, and longer training budgets are needed to establish the robustness and generalizability of the findings. Different factorization methods may also require different hyper-parameter settings for fair comparisons.
  • The paper provides limited guidance on how to choose the best design for different training settings. The analysis in Section 3.4 is mainly descriptive and lacks a stronger theoretical explanation for why these improvements should generalize across models and training conditions.
Questions for authors
  • Although the proposed method introduces no additional inference cost, it increases the number of trainable parameters and optimizer states. Could the authors report the actual training memory usage and computational overhead?
  • As shown in Appendix C, simple addition also achieves substantial improvements. Could these gains be explained by an increased effective learning rate? How does a direct output head with a larger learning rate compare with the additive method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.