Reachability Floors of Low-Rank Language-Model Heads

Final normalization bounds the hidden states presented to a language model's output head, so a fixed head can realize only a restricted set of token distributions. With this finite state budget, reachability depends on the head's scale as well as its rank. We define the reachability floor as the infimum target loss over normalization-feasible states and compute two-sided bounds at full vocabulary. For one-hot targets the average floor gives an exact decomposition of validation loss into a head-conditioned floor and a realization gap. Across a controlled intervention and two large-model heads, these floors separate into two regimes. In a paired multi-seed experiment, the factorized low-rank head has a much higher one-hot floor than the full-rank-forward control, while their validation losses differ far less. Bias-fixed matching of the learned output-map scale removes most of this contrast. Typical realized losses nevertheless remain well above their floors. Context-conditioned teacher targets reveal the second regime: their floors are smaller, but they make up a substantial share of realized teacher–student divergence, and their absolute floors are comparatively robust to uniform scale matching. A factorization-aware initialization and decay scheme improves both kinds of floor and validation loss relative to same-variance factorization, and an independent twenty-seed replication reproduces the improvement, with mean validation loss close to the full-rank-forward control's. A separate observational comparison of two 2B checkpoints also finds a large separation in teacher-distribution floors. The theory separates the roles of rank and scale, and the experiments distinguish uniform-scale sensitivity from residual learned-head geometry in output reachability. Reachability floors complement validation loss by locating restrictions in the learned output head.