acceptodds
Under review as a conference paper at ICLR 2027

Reachability Floors of Low-Rank Language-Model Heads

Abstract

Final normalization bounds the hidden states presented to a language model's output head, so a fixed head can realize only a restricted set of token distributions. With this finite state budget, reachability depends on the head's scale as well as its rank. We define the reachability floor as the infimum target loss over normalization-feasible states and compute two-sided bounds at full vocabulary. For one-hot targets the average floor gives an exact decomposition of validation loss into a head-conditioned floor and a realization gap. Across a controlled intervention and two large-model heads, these floors separate into two regimes. In a paired multi-seed experiment, the factorized low-rank head has a much higher one-hot floor than the full-rank-forward control, while their validation losses differ far less. Bias-fixed matching of the learned output-map scale removes most of this contrast. Typical realized losses nevertheless remain well above their floors. Context-conditioned teacher targets reveal the second regime: their floors are smaller, but they make up a substantial share of realized teacher–student divergence, and their absolute floors are comparatively robust to uniform scale matching. A factorization-aware initialization and decay scheme improves both kinds of floor and validation loss relative to same-variance factorization, and an independent twenty-seed replication reproduces the improvement, with mean validation loss close to the full-rank-forward control's. A separate observational comparison of two 2B checkpoints also finds a large separation in teacher-distribution floors. The theory separates the roles of rank and scale, and the experiments distinguish uniform-scale sensitivity from residual learned-head geometry in output reachability. Reachability floors complement validation loss by locating restrictions in the learned output head.

Primary area:
foundation or frontier models, including LLMs
Keywords:
softmax bottleneck, low-rank factorization, language model output head, reachability floor, convex duality, loss decomposition, layer normalization, knowledge distillation, verified numerical bounds

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TuKYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the reachability floor, which estimates the lowest loss a fixed output head can achieve. The authors show how rank and scale affect this limit and propose changes to initialization and weight decay that improve validation NLL. However, most experiments use a small Transformer, so the benefits for larger LLMs remain unclear.

Critical strengths
  1. Useful theoretical insight. The reachability floor helps explain the limits of an output head and distinguish them from other sources of loss.
  2. Consistent improvements. The proposed changes improve validation NLL, with gains reproduced across 20 random seeds.
Critical weaknesses
  1. Unclear practical value. The paper does not show whether the method meaningfully improves efficiency or downstream performance in real LLMs.
  2. Limited experiments. Most results come from a small model, and the main comparison changes both the forward pass and gradient constraints, making it hard to isolate the effect of low-rank factorization.
Questions for authors
  1. Practical value: How does the reachability floor help in practice? Can it guide output-head design to improve the trade-off between model quality and inference cost?
  2. Predicting actual loss: The reachability floor differs by 1.62 nats, but validation NLL differs by only 0.101. Why is the gap so large, and when is the floor a useful predictor of actual loss?
  3. Effect of factorization: Can you isolate the effect of low-rank factorization from the change in gradient constraints?
  4. Larger models: Can you validate these findings on a pretrained LLM under controlled conditions, rather than relying on small models and observational results?
  5. Initialization vs. weight decay: How much does each change contribute to the improvement? Does either work well on its own?
  6. Actual speedup: How much latency and memory does a low-rank head save, and does this translate into a meaningful end-to-end speedup?
4 comments
AuthorsOpenReview ↗

Practical value and the size of the floor-to-loss gap (1 of 3)

Thank you for six questions that go directly at what the paper does and does not establish. Two of them asked for measurements we have now taken, and they produced the clearest new results here, Q6's control most of all.

New for your points: the inference measurements of Q6, including that control; the initialization-versus-decay factorial for Q5; a second corpus and a five-times longer horizon for W1 and W2; and a seven-rank 2B sweep for Q4. The revision marks its changed passages in colour.

Notation. FF is the average one-hot reachability floor and LL validation NLL, with L=F+GL = F + G exactly, GG being the realization gap. NN is the teacher-distribution floor, KK the realized divergence, and Q=N/KQ = N/K the share of realized divergence unreachable in principle. ΔΔ denotes a signed paired change, so a negative ΔLΔL is an improvement; NN, KK and QQ are absolute quantities. Arms: AA is unrestricted, BB restricts the feedback subspace to rank 48 with the forward map untouched, CC factorizes the forward map at rank 48 and carries that feedback restriction under ordinary backpropagation, and C′C' is CC with corrected factor initialization and decay. Paired contrasts below are two-sided 95% Student-t intervals across the five seeds, and floor figures at 2B scale are two-sided numerical enclosures. A contrast is resolved when its interval excludes zero, and a bold value marks a resolved one.

Q1. Practical value. A training benefit at unchanged inference cost. C′C' improves validation NLL over CC by −0.086 nats across the twenty fresh seeds of the submitted replication, at identical rank and identical head architecture, one-sided 95% upper bound −0.072, about 8.2% lower perplexity. On the same twenty seeds C′C' also comes within the 0.05-nat margin of control BB (the arm that keeps a full-rank forward map and restricts feedback to 48 dimensions) at a mean LC′−LBL_{C'} - L_{B} of 0.013, one-sided 95% upper bound 0.0262. That 0.05-nat margin is about half the original 0.101-nat difference and corresponds to a 5.1% perplexity increase. It is measured against BB: against AA, the mean of LC′−LAL_{C'} - L_{A} is 0.065 with two-sided 95% interval [0.046, 0.083]: C′C' does not recover unrestricted quality. Because the floor is computed from a fixed head without retraining, it can rule out a target for a head that will be used as given. For a run that trains the head, the floor that binds is the trained head's, and we do not predict that in advance. The measured result here is the training benefit above.

Q2. Why is ΔFΔF = 1.62 so much larger than ΔLΔL = 0.101? Because of what the floor permits. FF is the minimum mean target NLL over states chosen independently at each position, so it may use a different target-aware state per token, while a language model must produce states from context through one shared trunk under genuine next-token uncertainty. GG absorbs both. Two of the three are measured and the third is fixed by L=F+GL = F + G: ΔFΔF = +1.620, ΔLΔL = +0.101, and ΔGΔG = −1.519 [−1.623, −1.415].

The floor earns its place in two situations. One is when the question is reachability: an average loss below a fixed head's floor is unattainable for that head whatever the training budget. The other is the regime of context-conditioned teacher targets, where the bound is large relative to the quantity it bounds: Q=N/KQ = N/K is 0.50 to 0.53 for arm CC against 0.24 to 0.32 for arm BB, and about 60.2% for the rank-32 2B head against 12.4% at rank 512. A substantial N/KN/K means the floor constrains the divergence actually achieved tightly.

For a fixed head and state interface, the floor gives an attainability constraint; it is not a predictor of realized loss. The relation it does give is exact: L=F+GL = F + G.

AuthorsOpenReview ↗

Isolating factorization, the 2B ladder, and the ablation (2 of 3)

Q3. Isolate factorization from the gradient constraint. Under ordinary backpropagation, factorizing the forward head also restricts the feedback subspace reaching the trunk, so C−BC - B compares two complete training interventions. Isolating the forward restriction needs a custom backward rule that supplies a different feedback map to a factorized forward head, and that arm was not run.

Against the common unrestricted baseline AA, offered as description only:

contrastrestrictionΔFΔFΔLΔL
B−AB - Afeedback restricted to 48 dimensions, forward map untouched+0.163 [−0.010, +0.336]+0.051 [−0.009, +0.112]
C−AC - Arank-48 factorized forward map, with the feedback restriction it entails under ordinary backpropagation+1.783 [1.733, 1.833]+0.152 [0.125, 0.179]

The B−AB - A floor difference resolves in the twenty-seed replication, at [+0.146, +0.254] nats, reported in Appendix D. The one comparison holding architecture and backpropagation rule fixed is CC against C′C'.

Q4. Validate on a pretrained LLM under controlled conditions. The pretrained 2B evaluation now spans seven head ranks, 32 to 2048, where the submission had two, each against the same d=4096d = 4096 teacher on 1024 context positions across the same 728 documents for every cell. Across that ladder the teacher-distribution floor NN falls more than 38-fold and the realized divergence KK it bounds less than 3.5-fold. Over the seven evaluated ranks the unremovable share Q=N/KQ = N/K decreases at every step, from about 60.2% at rank 32 to about 5% at rank 2048. All six adjacent comparisons separate under both the numerical enclosures and document-resampling bands, and the rank-32 and rank-512 separation persists at all three measured training stages.

The full ladder is Table 9 of the revision, with the three training stages in Appendix D.2.

These comparisons remain observational. Every cell is at 170k steps against the same teacher on the same contexts, and we did not train the checkpoints, so differences between cells need not come from rank alone. We have not performed a matched large-model training experiment, and that is the main open item. Taken as observational evidence, the ladder is now seven points with every adjacent step separated.

Q5. Initialization versus weight decay. We ran the full two-by-two on the five original seeds, holding architecture, backbone, data order, schedule and factor draws fixed within each seed. Initialization alone improves validation NLL by −0.0865 nats [−0.1151, −0.0579], and the combined treatment by −0.0925 [−0.1166, −0.0684]. Decay alone is +0.0016 [−0.0215, +0.0247], and adding corrected decay on top of corrected initialization gives −0.0060 [−0.0241, +0.0121], which five seeds leave uncertain. Initialization carries the measurable validation benefit.

On the floor, by contrast, every component resolves: −0.5548 [−0.6084, −0.5012] for initialization, −0.0776 [−0.1228, −0.0324] for decay alone, −0.1454 [−0.2220, −0.0687] for incremental decay, and −0.7002 [−0.7613, −0.6391] combined.

We therefore do not assign each component a share of the combined improvement. Our response to Reviewer npSM gives the interaction term and the realization-gap figures.

AuthorsOpenReview ↗

Measured inference cost and scope (3 of 3)

Q6. Actual speedup. Measured, paired by block over twenty blocks with sign tests in place of confidence intervals, and the answer differs by scale.

At the trained scale the head alone is slower than the dense one. Timed over 128 hidden states the median paired difference is 0.012 ms, with the factorized head slower in 18 of the 20 blocks (pp = 0.0004 in bf16, and fp32 agrees in 19 of 20). The full prefill pass over the same positions shows no resolved difference in either direction (pp = 0.50 bf16, pp = 0.82 fp32). The head carries 49.4% fewer parameters, 397,824 against 786,432, and at this size that saving does not become speed.

Factorization roughly halves both the operation count and the stored weight bytes, and the standalone head is slower anyway, so neither reduction delivers a speedup for this implementation. Latency here is nearly independent of the work in any case: the dense head is 1.5 MiB and one call takes about 0.016 ms at batch 1 against 0.020 ms at batch 128. That is 128 times the arithmetic for 1.3 times the time.

At the evaluated 2B dimensions (d=4096d = 4096, V=49,152V = 49{,}152) a synthetic benchmark favours factorization strongly. Weight storage in bf16 is 384.00 MiB dense against 52.00 MiB for explicit rank-512 factors and 3.25 MiB at rank 32. Projection-only latency on one RTX 5090, across batch sizes 1 to 512, is 4.2x to 6.5x faster than dense at rank 512 and 5.7x to 19.5x at rank 32. Here the latency does follow the weights: dense latency is nearly flat from batch 1 to batch 64, 0.305 against 0.354 ms, and the speedup follows the 7.4-fold drop in stored weight bytes, which is what a weight-traffic bottleneck would look like.

The control changes what that means. Densely materializing the rank-32 product restores the full 384.00 MiB and shows no resolved latency difference from dense (pp = 0.12 to 1.00). The saving belongs to executing the head as explicit factors, not to the map's numerical rank, and the released checkpoints as stored do not realize it.

The measurements, both precisions, the materialization control and the paired protocol are in Appendix D.5 of the revision.

The full-model timing is prefill-style and the synthetic 2B figures cover the output projection alone, so we make no end-to-end or decoding claim. The actionable finding is the architectural one: a factorized head is cheaper only if it is executed as factors.

W1 and W2. Small-model scope. Accepted and stated in Limitations. Four things were completed in this window, and the two that speak to scope have results worth stating here.

The combined C′−CC' - C validation improvement survives a change of corpus: it is −0.091 [−0.113, −0.068] on a frozen FineWeb-Edu sample with the per-split token budget matched, against −0.093 [−0.117, −0.068] on WikiText-2. It also survives five times the training: at 6,000 steps against 1,200 it is −0.086 [−0.117, −0.055], and the initialization correction likewise remains an improvement at −0.067 [−0.091, −0.044]. The other two completions are the Q5 ablation and the Q6 inference measurements.

The controlled experiments still use one architecture, and the controlled pretrained-LLM study stays an open item.

What your review produced. A component decomposition, a seven-rank 2B ladder, and inference measurements whose materialization control places the projection saving in executing the head as factors.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer TuKY,

Following your questions, we added the component ablation, a second corpus, a training horizon five times longer, and inference measurements. The combined correction improves validation NLL in both breadth studies, and the ablation identifies initialization as carrying the measurable benefit (Appendices D.1 and D.3). The large-model evidence now spans seven ranks, with every adjacent floor comparison separated. These are separately trained checkpoints, so the comparison remains observational (Table 9, Appendix D.2). The inference measurements include a dense-materialization control, which separates the map's numerical rank from executing it as explicit factors (Appendix D.5).

With the discussion closing on 18 November, we would appreciate your assessment of these additions and would welcome hearing which concerns, if any, remain open.

Reviewer_ukSxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new concept called reachability floor. This concept can measure how well a fixed LM head performs on a validation set of tokens. It performs a decomposition L=F+GL=F+G and then conducts experiments to analyze F and G under different cases. The core variables here are the rank of the LM head, the forward pass constraint, and the scale of the weights. It first identifies that low rank can result in a much higher F value while having a slightly higher L value. Then it further identifies that the gap can be attributed to scale rather than the rank difference.

Critical strengths
  1. A novel metric of reachability floor and an interesting ablation study on rank difference results for the reachability floor.
  2. Clean analysis and disentangling of some important factors, such as the impacts of rank and the scale of the weight matrix on the output distribution.
Critical weaknesses
  1. The reachability floor is different from the loss; how can the analysis guide practical LLM training?
  2. More ablation is needed, for example, across different datasets and across different dimension scales and ranks. Current experiments can not fully support the claim.
Questions for authors

See Weaknesses:

  1. How can the analysis guide practical LLM training?
  2. Can you provide more ablation study results to validate the claim?
4 comments
AuthorsOpenReview ↗

What the diagnostic did, measured (1 of 2)

Thank you for identifying that the paper needed broader evidence. Your Q2 turned the 2B comparison from a single pair into a seven-rank study, and it is why the controlled experiment now has a second corpus. Both make the paper stronger.

New for your points: the initialization-by-decay factorial; a second corpus; a longer horizon; and a seven-rank 2B sweep that replaces the two-rank comparison the submission rested on, with two of its ranks re-measured at three training stages. The revision marks its changed passages in colour.

Notation. FF is the average one-hot reachability floor and LL validation NLL, with L=F+GL = F + G exactly; NN is the teacher-distribution floor and KK the realized divergence, so Q=N/KQ = N/K is the share of realized divergence that is unreachable in principle. Arms: AA is unrestricted, BB restricts feedback to 48 dimensions with the forward map untouched, CC factorizes the forward map at rank 48, and C′C' is CC with corrected factor initialization and decay. ΔΔ denotes a signed paired change; NN, KK and QQ are absolute. Intervals on ΔΔ are two-sided 95% Student-t across seeds, and intervals on NN and QQ are numerical enclosures, displayed rounded outward. A contrast is resolved when its interval excludes zero, and a bold value marks a resolved one.

Q1. How can the analysis guide practical LLM training? The floor is a diagnostic measured on a trained head. Here it did two things, both measured: it motivated a training correction we then tested, and it produced a design recommendation.

  1. It motivated a training correction, and the correction holds up. C′C' improves validation NLL at unchanged rank and unchanged head architecture, reproduced on twenty fresh seeds in the replication already in the submission: mean −0.086, one-sided 95% upper bound −0.072, about 8.2% lower perplexity. On the same twenty seeds C′C' comes within the 0.05-nat margin of control BB, the arm that keeps a full-rank forward map and restricts feedback to 48 dimensions, at a mean LC′−LBL_{C'} - L_{B} of 0.013 with one-sided 95% upper bound 0.0262. That margin is about half the original 0.101-nat difference and corresponds to a 5.1% perplexity increase. Against unrestricted AA, C′C' remains 0.065 nats higher in validation NLL, two-sided 95% interval [0.046, 0.083]. The diagnostic motivated the correction; it does not establish that FF is the channel the benefit travels through.
  2. It produced a concrete recommendation. Report effective-map scale alongside rank, since rank alone does not determine the reachable geometry. The theory's own probe of this holds the bias fixed and matches the learned map scale across heads: that removes most of the one-hot contrast and leaves a residual of 0.111 to 0.252 nats across seeds, positive in every one.
  3. It located a restriction that validation NLL does not expose. Arms BB and CC both restrict feedback to 48 dimensions, through different backward operators, while their forward maps differ. Between them the floor differs by +1.620 nats [1.462, 1.777] while validation NLL differs by +0.101 [0.045, 0.156]: resolved, and about 10.6% higher perplexity, but sixteen times smaller in nats. Most realized losses sit well above their floors, which is why a large floor difference can sit under a small loss difference.
  4. The floor is largest relative to the quantity it bounds under context-conditioned teacher targets, where QQ reaches 0.50 to 0.53 for arm CC against 0.24 to 0.32 for arm BB, and about 60.2% at rank 32 against 12.4% at rank 512 at 2B scale.

Those four are what the diagnostic did here, each of them measured.

AuthorsOpenReview ↗

Ablation, second corpus, and the seven-rank sweep (2 of 2)

Q2. More ablation study results, across different datasets. Three additions. Together they take the training benefit to a second corpus (C′−CC' - C at −0.091 [−0.113, −0.068]) and to five times the horizon (−0.086 [−0.117, −0.055]), and widen the 2B study from two heads to seven ranks. Each is given in full below.

The initialization-by-decay two-by-two, on the five original seeds with architecture, backbone, data order, schedule and factor draws held fixed within each seed. Initialization alone improves validation NLL by −0.0865 nats [−0.1151, −0.0579] and the one-hot floor by −0.5548 [−0.6084, −0.5012]. Decay lowers the floor a further −0.1454 [−0.2220, −0.0687] on top of corrected initialization. On validation it adds −0.0060 [−0.0241, +0.0121], which five seeds leave uncertain, so initialization carries the validation benefit we can measure and we do not divide the combined improvement between the two components.

A second corpus, which answers the "across different datasets" part of your question for one further dataset. The controlled experiment was rebuilt on a frozen FineWeb-Edu sample, with the per-split token budget matched exactly to WikiText-2. Both headline contrasts replicate across five seeds: the C−BC - B floor separation is +1.674 [+1.508, +1.840] against +1.620 [+1.462, +1.777] on WikiText-2, and the C′−CC' - C validation improvement is −0.091 [−0.113, −0.068] against −0.093 [−0.117, −0.068].

A seven-rank sweep, replacing the two-rank 2B contrast the submission rested on: seven head ranks of the published suite against one d=4096d = 4096 teacher, on 1024 context positions over the same 728 documents for every cell.

rankfloor NN (nats)KK (nats)QQ (%)
32[0.406590, 0.406605]0.674931[60.24, 60.25]
64[0.206433, 0.206623]0.455776[45.29, 45.34]
128[0.103136, 0.103409]0.327488[31.49, 31.58]
256[0.053104, 0.053270]0.258431[20.54, 20.62]
512[0.028250, 0.028271]0.227932[12.39, 12.41]
1024[0.016099, 0.016161]0.207755[7.74, 7.78]
2048[0.009022, 0.010678]0.198444[4.54, 5.39]

This is Table 9 of the revision, and Appendix D there carries the bracket widths, the three training stages (D.2) and the protocol.

The floor falls more than 38-fold across the ladder and the realized divergence it bounds less than 3.5-fold. Across the seven evaluated ranks the share decreases at every step, from about 60.2% to about 5%. All six adjacent steps separate, under the numerical enclosures and under document-resampling bands, which Table 13 and Appendix D.2 now give with the protocol.

Overall head scale does not explain the separation at the extremes, and the theory supplies the check. Rank 32 carries the largest centered folded spectral scale in the suite: matching every other head up to it uses gains of at least one. By Theorem 1 that is an increase in radius at fixed bias, and the floor is monotone nonincreasing in radius, so the rank-32 floor is unchanged and every other floor can only fall. The separation between rank 32 and rank 2048 can only widen under this matching.

Rank 2048's bracket is 16.8% of the floor against at most 0.38% elsewhere, because the iteration that seeds the bound converges slowest at the highest rank. We report the width as measured. Every claim above holds at every point in the brackets, since each is taken at the endpoints that make it weakest.

The rank-32 and rank-512 separation persists at all three measured checkpoints, by at least 9.12 times at 20k steps, 11.4 at 85k and 14.38 at 170k. Within either rank, the largest floor ratio across those three checkpoints is at most 1.75, taking each enclosure at the endpoint that makes the ratio largest.

The sweep is observational. Every cell is at 170k steps against the same teacher on the same contexts, and we did not train the checkpoints, so differences between cells need not come from rank alone. The training-stage evidence covers two of the seven ranks.

What your review produced. A seven-rank 2B ladder with every adjacent step separated, in place of a single pair, and a second corpus behind the headline floor separation.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer ukSx,

Your request for broader evidence led to two substantial additions: a second-corpus study on FineWeb-Edu and a seven-rank observational study at 2B scale. All six adjacent floor comparisons in the 2B study separate, under both the numerical enclosures and document-resampling bands (Table 9, Appendix D.2). The validation benefit carries over to FineWeb-Edu, and the component ablation identifies initialization as carrying the measurable improvement (Appendix D.1, Table 10).

With the discussion closing on 18 November, we would welcome your assessment of the strengthened evidence and of any concerns that remain.

Thanks for your rebuttal. I will keep my positive score.

Reviewer_npSMOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the realizable set of token distributions of language model heads whose inputs are bounded by the final normalization.The reachability floor is defined as :

TR(q)=min⁡∣h∣≤RDKL(q∣∣softmax⁡(Wh+b))T_R(q)=\min_{|h|\le R}D_{\mathrm{KL}}\left(q || \operatorname{softmax}(Wh+b)\right)

For one-hot targets, averaging gives the head-conditioned average one-hot reachability floor FF, which decomposes the validation loss LL as L=F+GL = F + G where GG is the realization gap (residual).

Using controlled subword setting from prior work, the authors train (using 5 seeds) a 6-layer transformer on WikiText-2 for 1200 steps under a common LR schedule across the 3 different arms; one unrestricted baseline and 2 nominal-rank-48 interventions (backward only and rank-48-factorized-head).

The work finds that two regimes exists and the one-hot and teacher distribution floors behave differently. A relatively high floor characterizes the one-hot targets, and it is very scale sensitive. In contrast, the teacher-distributions have much smaller floors and are more insensitive to uniform rescaling.

Overall this work establishes that comparing language model heads should include effective map scale alongside rank and the reachbility floors alongside realized loss. If a loss if desired below the floor, it requires changing the head or its state interface.

Critical strengths
  1. Experimental results are strong and establish a key separation between reachability and its predictive performance. Section 5 shows that applying gains recovered from spectral matching does not guarantee better performance.
Critical weaknesses
  1. Even though the experiments are well controlled they only span a single architecture, one corpus and a short horizon. Robustness across training regimes is not established.
Questions for authors

Clarifications:

  1. Are there ablation runs with init only and decay only results? This could help clarify whether there is significant material contribution to C′C' from both the theoretical observations.
  2. Does the released code support fresh training or is it meant to only reproduce and verify the artifacts?
4 comments
AuthorsOpenReview ↗

Factorial ablation and verified fresh training (1 of 2)

Thank you for the component ablation request and for the fresh-training question. Both produced results we would not otherwise have, and both are in the revision, which marks its changed passages in colour.

New for your points: the full initialization-by-decay factorial on the original seeds, for Q1; a verified fresh-training run from a clean copy of the published artifact, for Q2; and the controlled experiment rebuilt on a second corpus and at five times the horizon, for W1.

Notation. ΔLΔL is the signed paired change in validation NLL and ΔFΔF the change in the average one-hot reachability floor, so a negative value is an improvement. L=F+GL = F + G exactly, with GG the realization gap. Arms: AA is unrestricted, BB restricts feedback to 48 dimensions with the forward map untouched, CC factorizes the forward map at rank 48, and C′C' is CC with corrected factor initialization and decay, with CinitC_{init} and CdecayC_{decay} applying one correction each. Intervals are two-sided 95% Student-t across seeds unless stated otherwise. A contrast is resolved when its interval excludes zero, and a bold value marks a resolved one.

Q1. Init-only and decay-only ablation results. Yes. We ran the full two-by-two on the five original seeds, holding architecture, backbone, data order, schedule and factor draws fixed within each seed, and the two new arms check that their factor draws are bit-identical to arm CC's before any scaling. All six contrasts we computed:

contrastΔLΔLΔFΔF
initialization only−0.0865 [−0.1151, −0.0579]−0.5548 [−0.6084, −0.5012]
decay only+0.0016 [−0.0215, +0.0247]−0.0776 [−0.1228, −0.0324]
incremental decay, C′−CinitC' - C_{init}−0.0060 [−0.0241, +0.0121]−0.1454 [−0.2220, −0.0687]
incremental initialization, C′−CdecayC' - C_{decay}−0.0941 [−0.1275, −0.0607]−0.6226 [−0.7171, −0.5282]
combined, C′−CC' - C−0.0925 [−0.1166, −0.0684]−0.7002 [−0.7613, −0.6391]
interaction−0.0076 [−0.0466, +0.0314]−0.0678 [−0.1804, +0.0448]

This is Table 10 of the revision, with the full protocol in Appendix D.3. On the floor both components resolve, and initialization carries the validation benefit we can measure. Since L=F+GL = F + G exactly, the combined correction lowering the floor 0.700 nats while validation moves 0.093 means the realization gap rises 0.608 [+0.562, +0.654]. That is arithmetic from the identity, and it makes no claim that the floor change caused the loss change. The decay component's incremental validation effect is unresolved at five seeds, as is the interaction, so we do not divide the combined validation improvement between the two.

Section 5 of the revision now separates the two components, which the submission described as a single treatment, and the twenty-seed replication there holds the bundle against CC, BB and AA.

Q2. Does the released code train, or only verify? Both, and we have now tested the first. reproduce.sh validates stored records and deliberately does not retrain, which its header says but the README does not make obvious. Training goes through src/analysis/murugan_run.py, which the artifact carries with the upstream implementation under external/lmhead/, the tokenized corpus and the run configurations. We took a clean copy of the published artifact and trained arms AA and CC from scratch with it, and the experiment's built-in check that paired arms share data order and initial backbone passed.

Training runs from the artifact. It does not follow that the full evaluation pipeline reruns end to end. As the artifact stands it carries the submitted arms and the records behind them, and the experiments added during the discussion period are documented in Appendix D with their protocols.

AuthorsOpenReview ↗

Second corpus and longer horizon (2 of 2)

W1. One architecture, one corpus, short horizon. Two of the three are now addressed by runs.

Second corpus. The controlled experiment rebuilt on a frozen FineWeb-Edu sample: natural web text, with document-level disjoint splits, a vocabulary fitted only on documents from the training side of the boundary, and a per-split token budget matched exactly to WikiText-2.

contrastFineWeb-EduWikiText-2
C−BC - B, ΔFΔF+1.674 [+1.508, +1.840]+1.620 [+1.462, +1.777]
C−BC - B, ΔLΔL+0.088 [+0.044, +0.132]+0.101 [+0.045, +0.156]
C′−CC' - C, ΔLΔL−0.091 [−0.113, −0.068]−0.093 [−0.117, −0.068]
C′−CC' - C, ΔFΔF−0.763 [−0.811, −0.715]−0.700 [−0.762, −0.639]
C′−BC' - B, ΔLΔL−0.003 [−0.060, +0.054]+0.008 [−0.050, +0.066]

Both headline contrasts replicate across five seeds, on the floor as well as on validation NLL. The remaining contrasts on this corpus, all of them: B−AB - A on validation NLL is +0.034 [−0.031, +0.099], and C−AC - A is +0.122 [+0.077, +0.166].

Longer horizon. Five times the training, 6,000 steps against 1,200, on the same corpus, seeds and schedule shape, with arms CC, CinitC_{init} and C′C'. Training CinitC_{init} lets the horizon question be asked of the initialization component on its own.

contrast6,000 steps1,200 steps
Cinit−CC_{init} - C, ΔLΔL−0.067 [−0.091, −0.044]−0.087 [−0.116, −0.057]
C′−CinitC' - C_{init}, ΔLΔL−0.019 [−0.047, +0.009]−0.006 [−0.025, +0.013]
C′−CC' - C, ΔLΔL−0.086 [−0.117, −0.055]−0.093 [−0.117, −0.068]
Cinit−CC_{init} - C, ΔFΔF−0.205 [−0.254, −0.157]−0.555 [−0.609, −0.501]
C′−CinitC' - C_{init}, ΔFΔF−0.258 [−0.321, −0.194]−0.145 [−0.222, −0.068]
C′−CC' - C, ΔFΔF−0.463 [−0.532, −0.394]−0.700 [−0.762, −0.639]

Both the initialization correction and the combined treatment still improve validation NLL at the longer horizon, with both intervals excluding zero and keeping the sign they had at 1,200 steps.

Testing other architectures stays future work. Construction, schedules and the full contrast tables for both studies are in Appendix D.1 (Tables 11 and 12).

What your review produced. The bundled correction is now separated into its components, and its validation benefit holds on a second corpus and at five times the training.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Dear Reviewer npSM,

Following your suggestions, the revision now includes the full initialization-by-decay ablation (Table 10), and we verified fresh training from a clean copy of the published artifact. The ablation identifies initialization as carrying the measurable validation benefit. We also tested the combined correction on a second corpus, FineWeb-Edu, and at five times the training horizon, and the validation benefit holds in both studies (Appendix D.1).

With the discussion closing on 18 November, we would appreciate your assessment of whether these additions address your concerns, and we would be glad to discuss any remaining questions.

Thanks to the authors for providing the additional experiments and evidences. Most of my concerns are addressed and make the paper stronger, however the robustness to architecture is still unresolved so I've kept my score as is.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision: what is new and where it is

We thank the three reviewers. Between them they asked for a component ablation, a second corpus, a longer training horizon, broader large-model evidence, and real inference numbers. We ran all five. This comment says what the revision contains and where each piece sits; the individual responses follow in each thread.

What is new since submission.

  1. The initialization-by-decay factorial, on the five original seeds with architecture, backbone, data order, schedule and factor draws held fixed within each seed. Section 5 and Table 10, with the protocol in Appendix D.3.
  2. A second corpus. The controlled experiment rebuilt on a frozen FineWeb-Edu sample, with the per-split token budget matched exactly to WikiText-2. Appendix D.1.
  3. Five times the training horizon, 6,000 steps against 1,200, with the initialization component separated from the full correction. Appendix D.1.
  4. A seven-rank 2B sweep, ranks 32 to 2048 against one d=4096d = 4096 teacher, which replaces the two-rank comparison the submission rested on. Table 9, and Appendix D.2 re-measures two of the ranks at three training stages.
  5. Measured inference cost, paired by block over twenty blocks, including a materialization control that changed what we claim. Appendix D.5.
  6. Verified fresh training from a clean copy of the published artifact.

What the paper claims. The floor is a diagnostic, not a predictor. It makes visible a restriction that validation loss alone does not show, and it motivated a training change that improves validation loss at unchanged rank and unchanged head architecture, reproduced on twenty fresh seeds. Over the discussion period we added a component decomposition, a second corpus, a five-times longer horizon, and a 2B ladder of seven ranks in place of the submission's single pair.

The 2B comparison is observational. Every cell is at 170k steps against the same teacher, and we did not train the checkpoints, so differences between cells need not come from rank alone. Limitations states the scope of the controlled experiments, and each reviewer's thread answers the points they raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.