- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies full-width linear factorizations of language-model output heads, such as and . These factors are optimized during training and folded into one output projection before inference, preserving deployment cost relative to a direct head with the same sharing convention. Experiments examine initialization, factor depth, placement, sharing, and classifier-to-embedding gradient coupling. The central intervention preserves the initial folded matrix and initial logits while changing the factor split from concentrated to SVD-balanced. Under the reported AdamW protocol, balanced initialization improves perplexity across Qwen2-0.5B, Qwen2-1.5B, and Pythia-410m, with larger gains on less frequent tokens. Local gradient diagnostics and early-loss screening provide additional analyses.
Critical strengths
-
Concentrated and SVD-balanced factors start with identical folded output weights and logits while holding the architecture and optimizer protocol fixed. Table 6 reports consistent improvements across three model configurations, including larger improvements on less frequent tokens. This establishes an effect beyond changing the initial prediction function.
-
The input-only factorization control reaches 46.86 PPL versus 44.01 for output-side depth 2, limiting an explanation based on factorizing the input table alone. The stop-gradient control isolates classifier-to-embedding gradient coupling while preserving unrestricted output adaptation through a residual matrix. Folding the factors preserves a single output projection at inference under the same sharing convention.
Critical weaknesses
-
A tuned direct-head baseline is especially important given the pure-addition result. The direct sum reaches 41.36 PPL, versus 47.02 for the direct head and 40.70 for balanced depth 2 (Tables 3, 6, and 10). Numerically, this closes about 90% of that PPL difference. For the same folded initialization, identical component optimizer states and settings, and matching gradient processing, its folded AdamW update is , where is the per-step decay and the shared adaptive direction. This corresponds to doubling the direct head's gradient step while preserving its decay multiplier. The actual clipping configuration needs clarification. Since head-specific learning rates were not tuned, the advantage over a simpler optimization adjustment remains unresolved.
-
The main factorial trains from scratch on approximately 652M tokens, or 1.32 tokens per parameter; Qwen2-1.5B receives 0.423 tokens per parameter (Section 2; Appendix G). Figure 1 shows an early separation followed by a smaller persistent gap, with Qwen2-0.5B approximately stable near nats through the second half. This supports a benefit at the tested horizon, while leaving its persistence under substantially more training unknown. The differing third-factor outcomes in Table 7 further limit general guidance about factor depth.
-
Table 4 measures hypothetical Euclidean-GD updates in the concentrated gauge using a shared reference matrix. It does not measure the balanced head's actual AdamW updates. Its large rare-token amplification and near doubling with depth characterize that construction; the doubling is an expected consistency check from Eq. (2). The authors explicitly acknowledge these limits, but the diagnostic and Appendix E's weak descriptive correlations do not establish why balanced initialization improves the endpoint, particularly on rare tokens. This limits the explanatory contribution beyond the observed intervention effect.
Questions for authors
-
How much benefit remains over a tuned direct head? Please specify the component initialization, optimizer states, parameter groups, decay, and clipping for . Could you compare a small set of head-only learning-rate multipliers, including , while keeping and the backbone schedule fixed and matching per-step decay and gradient processing? Under the standard AdamW convention, doubling the head learning rate requires halving its weight-decay coefficient to preserve that multiplier. Please report endpoint and frequency-bucket PPL against the direct sum and balanced depth-2 head. This would directly test the strongest alternative explanation.
-
Does the advantage persist with more training? Do existing longer runs, or a targeted extension comparing a tuned direct head with balanced depth 2 under a shared extended schedule and data budget, establish whether the validation gap remains stable? Please distinguish the training-loss trajectories in Figure 1 from held-out validation behavior.