- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the reachability floor, which estimates the lowest loss a fixed output head can achieve. The authors show how rank and scale affect this limit and propose changes to initialization and weight decay that improve validation NLL. However, most experiments use a small Transformer, so the benefits for larger LLMs remain unclear.
Critical strengths
- Useful theoretical insight. The reachability floor helps explain the limits of an output head and distinguish them from other sources of loss.
- Consistent improvements. The proposed changes improve validation NLL, with gains reproduced across 20 random seeds.
Critical weaknesses
- Unclear practical value. The paper does not show whether the method meaningfully improves efficiency or downstream performance in real LLMs.
- Limited experiments. Most results come from a small model, and the main comparison changes both the forward pass and gradient constraints, making it hard to isolate the effect of low-rank factorization.
Questions for authors
- Practical value: How does the reachability floor help in practice? Can it guide output-head design to improve the trade-off between model quality and inference cost?
- Predicting actual loss: The reachability floor differs by 1.62 nats, but validation NLL differs by only 0.101. Why is the gap so large, and when is the floor a useful predictor of actual loss?
- Effect of factorization: Can you isolate the effect of low-rank factorization from the change in gradient constraints?
- Larger models: Can you validate these findings on a pretrained LLM under controlled conditions, rather than relying on small models and observational results?
- Initialization vs. weight decay: How much does each change contribute to the improvement? Does either work well on its own?
- Actual speedup: How much latency and memory does a low-rank head save, and does this translate into a meaningful end-to-end speedup?
4 comments
Isolating factorization, the 2B ladder, and the ablation (2 of 3)
Q3. Isolate factorization from the gradient constraint. Under ordinary backpropagation, factorizing the forward head also restricts the feedback subspace reaching the trunk, so compares two complete training interventions. Isolating the forward restriction needs a custom backward rule that supplies a different feedback map to a factorized forward head, and that arm was not run.
Against the common unrestricted baseline , offered as description only:
| contrast | restriction | ||
|---|---|---|---|
| feedback restricted to 48 dimensions, forward map untouched | +0.163 [−0.010, +0.336] | +0.051 [−0.009, +0.112] | |
| rank-48 factorized forward map, with the feedback restriction it entails under ordinary backpropagation | +1.783 [1.733, 1.833] | +0.152 [0.125, 0.179] |
The floor difference resolves in the twenty-seed replication, at [+0.146, +0.254] nats, reported in Appendix D. The one comparison holding architecture and backpropagation rule fixed is against .
Q4. Validate on a pretrained LLM under controlled conditions. The pretrained 2B evaluation now spans seven head ranks, 32 to 2048, where the submission had two, each against the same teacher on 1024 context positions across the same 728 documents for every cell. Across that ladder the teacher-distribution floor falls more than 38-fold and the realized divergence it bounds less than 3.5-fold. Over the seven evaluated ranks the unremovable share decreases at every step, from about 60.2% at rank 32 to about 5% at rank 2048. All six adjacent comparisons separate under both the numerical enclosures and document-resampling bands, and the rank-32 and rank-512 separation persists at all three measured training stages.
The full ladder is Table 9 of the revision, with the three training stages in Appendix D.2.
These comparisons remain observational. Every cell is at 170k steps against the same teacher on the same contexts, and we did not train the checkpoints, so differences between cells need not come from rank alone. We have not performed a matched large-model training experiment, and that is the main open item. Taken as observational evidence, the ladder is now seven points with every adjacent step separated.
Q5. Initialization versus weight decay. We ran the full two-by-two on the five original seeds, holding architecture, backbone, data order, schedule and factor draws fixed within each seed. Initialization alone improves validation NLL by −0.0865 nats [−0.1151, −0.0579], and the combined treatment by −0.0925 [−0.1166, −0.0684]. Decay alone is +0.0016 [−0.0215, +0.0247], and adding corrected decay on top of corrected initialization gives −0.0060 [−0.0241, +0.0121], which five seeds leave uncertain. Initialization carries the measurable validation benefit.
On the floor, by contrast, every component resolves: −0.5548 [−0.6084, −0.5012] for initialization, −0.0776 [−0.1228, −0.0324] for decay alone, −0.1454 [−0.2220, −0.0687] for incremental decay, and −0.7002 [−0.7613, −0.6391] combined.
We therefore do not assign each component a share of the combined improvement. Our response to Reviewer npSM gives the interaction term and the realization-gap figures.
Measured inference cost and scope (3 of 3)
Q6. Actual speedup. Measured, paired by block over twenty blocks with sign tests in place of confidence intervals, and the answer differs by scale.
At the trained scale the head alone is slower than the dense one. Timed over 128 hidden states the median paired difference is 0.012 ms, with the factorized head slower in 18 of the 20 blocks ( = 0.0004 in bf16, and fp32 agrees in 19 of 20). The full prefill pass over the same positions shows no resolved difference in either direction ( = 0.50 bf16, = 0.82 fp32). The head carries 49.4% fewer parameters, 397,824 against 786,432, and at this size that saving does not become speed.
Factorization roughly halves both the operation count and the stored weight bytes, and the standalone head is slower anyway, so neither reduction delivers a speedup for this implementation. Latency here is nearly independent of the work in any case: the dense head is 1.5 MiB and one call takes about 0.016 ms at batch 1 against 0.020 ms at batch 128. That is 128 times the arithmetic for 1.3 times the time.
At the evaluated 2B dimensions (, ) a synthetic benchmark favours factorization strongly. Weight storage in bf16 is 384.00 MiB dense against 52.00 MiB for explicit rank-512 factors and 3.25 MiB at rank 32. Projection-only latency on one RTX 5090, across batch sizes 1 to 512, is 4.2x to 6.5x faster than dense at rank 512 and 5.7x to 19.5x at rank 32. Here the latency does follow the weights: dense latency is nearly flat from batch 1 to batch 64, 0.305 against 0.354 ms, and the speedup follows the 7.4-fold drop in stored weight bytes, which is what a weight-traffic bottleneck would look like.
The control changes what that means. Densely materializing the rank-32 product restores the full 384.00 MiB and shows no resolved latency difference from dense ( = 0.12 to 1.00). The saving belongs to executing the head as explicit factors, not to the map's numerical rank, and the released checkpoints as stored do not realize it.
The measurements, both precisions, the materialization control and the paired protocol are in Appendix D.5 of the revision.
The full-model timing is prefill-style and the synthetic 2B figures cover the output projection alone, so we make no end-to-end or decoding claim. The actionable finding is the architectural one: a factorized head is cheaper only if it is executed as factors.
W1 and W2. Small-model scope. Accepted and stated in Limitations. Four things were completed in this window, and the two that speak to scope have results worth stating here.
The combined validation improvement survives a change of corpus: it is −0.091 [−0.113, −0.068] on a frozen FineWeb-Edu sample with the per-split token budget matched, against −0.093 [−0.117, −0.068] on WikiText-2. It also survives five times the training: at 6,000 steps against 1,200 it is −0.086 [−0.117, −0.055], and the initialization correction likewise remains an improvement at −0.067 [−0.091, −0.044]. The other two completions are the Q5 ablation and the Q6 inference measurements.
The controlled experiments still use one architecture, and the controlled pretrained-LLM study stays an open item.
What your review produced. A component decomposition, a seven-rank 2B ladder, and inference measurements whose materialization control places the projection saving in executing the head as factors.
Follow-up on our response and revised manuscript
Dear Reviewer TuKY,
Following your questions, we added the component ablation, a second corpus, a training horizon five times longer, and inference measurements. The combined correction improves validation NLL in both breadth studies, and the ablation identifies initialization as carrying the measurable benefit (Appendices D.1 and D.3). The large-model evidence now spans seven ranks, with every adjacent floor comparison separated. These are separately trained checkpoints, so the comparison remains observational (Table 9, Appendix D.2). The inference measurements include a dense-materialization control, which separates the map's numerical rank from executing it as explicit factors (Appendix D.5).
With the discussion closing on 18 November, we would appreciate your assessment of these additions and would welcome hearing which concerns, if any, remain open.
Practical value and the size of the floor-to-loss gap (1 of 3)
Thank you for six questions that go directly at what the paper does and does not establish. Two of them asked for measurements we have now taken, and they produced the clearest new results here, Q6's control most of all.
New for your points: the inference measurements of Q6, including that control; the initialization-versus-decay factorial for Q5; a second corpus and a five-times longer horizon for W1 and W2; and a seven-rank 2B sweep for Q4. The revision marks its changed passages in colour.
Notation. F is the average one-hot reachability floor and L validation NLL, with L=F+G exactly, G being the realization gap. N is the teacher-distribution floor, K the realized divergence, and Q=N/K the share of realized divergence unreachable in principle. Δ denotes a signed paired change, so a negative ΔL is an improvement; N, K and Q are absolute quantities. Arms: A is unrestricted, B restricts the feedback subspace to rank 48 with the forward map untouched, C factorizes the forward map at rank 48 and carries that feedback restriction under ordinary backpropagation, and C′ is C with corrected factor initialization and decay. Paired contrasts below are two-sided 95% Student-t intervals across the five seeds, and floor figures at 2B scale are two-sided numerical enclosures. A contrast is resolved when its interval excludes zero, and a bold value marks a resolved one.
Q1. Practical value. A training benefit at unchanged inference cost. C′ improves validation NLL over C by −0.086 nats across the twenty fresh seeds of the submitted replication, at identical rank and identical head architecture, one-sided 95% upper bound −0.072, about 8.2% lower perplexity. On the same twenty seeds C′ also comes within the 0.05-nat margin of control B (the arm that keeps a full-rank forward map and restricts feedback to 48 dimensions) at a mean LC′−LB of 0.013, one-sided 95% upper bound 0.0262. That 0.05-nat margin is about half the original 0.101-nat difference and corresponds to a 5.1% perplexity increase. It is measured against B: against A, the mean of LC′−LA is 0.065 with two-sided 95% interval [0.046, 0.083]: C′ does not recover unrestricted quality. Because the floor is computed from a fixed head without retraining, it can rule out a target for a head that will be used as given. For a run that trains the head, the floor that binds is the trained head's, and we do not predict that in advance. The measured result here is the training benefit above.
Q2. Why is ΔF = 1.62 so much larger than ΔL = 0.101? Because of what the floor permits. F is the minimum mean target NLL over states chosen independently at each position, so it may use a different target-aware state per token, while a language model must produce states from context through one shared trunk under genuine next-token uncertainty. G absorbs both. Two of the three are measured and the third is fixed by L=F+G: ΔF = +1.620, ΔL = +0.101, and ΔG = −1.519 [−1.623, −1.415].
The floor earns its place in two situations. One is when the question is reachability: an average loss below a fixed head's floor is unattainable for that head whatever the training budget. The other is the regime of context-conditioned teacher targets, where the bound is large relative to the quantity it bounds: Q=N/K is 0.50 to 0.53 for arm C against 0.24 to 0.32 for arm B, and about 60.2% for the rank-32 2B head against 12.4% at rank 512. A substantial N/K means the floor constrains the divergence actually achieved tightly.
For a fixed head and state interface, the floor gives an attainability constraint; it is not a predictor of realized loss. The relation it does give is exact: L=F+G.