- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ResLorB, a nonlinear low-rank branch added at the input of each Transformer block: , with , and by default. The block reads and writes to the stream , which the branch leaves unmodified: . The branch adds parameters. The paper frames the branch as a source of neuron-level interactions within the residual stream. It motivates the design from Frac-Connection, whose reshape operation fixes which hidden dimensions interact, and derives the branch as a learnable replacement for this fixed structured matrix. The branch combines with multi-state architectures. mHC-ResLorB applies it after the mHC pre-projection, and mHC-ResLorB+ gives it an independent pre-projection. The paper includes a theorem in a nonlinear in-context regression setting with one linear attention layer, in which the branch lowers the in-context risk. Experiments cover LLaMA-style models from 60M to 350M on C4, DeepSeek-V3-style 3B MoE models on 200B tokens under Pre-Norm and Sandwich-Norm, and a 9B mHC model on 450B tokens. The authors report a 0.011 training-loss reduction and a 1.4-point downstream gain at 9B (44.1 45.5), and about a 1-point gain at 3B, with about 1% additional total and 4% additional activated parameters. The appendices compare activation functions, branch placements, ranks, initialization, a stream-injection variant, a gated input variant, an added Transformer layer, eight active experts, and Gated Residual. They also report gradient norms, load balancing, adjacent-layer cosine similarity, and activation magnitudes.
Critical strengths
The ablation suite addresses several simple explanations of the gain. A linear branch (Table 4), a gated input-scaling variant, an eight-expert allocation, and Gated Residual (Figs. 14-15) perform worse than ResLorB, and the gain persists at 9B on top of mHC (Fig. 8, Table 8). This establishes that the pre-block nonlinear branch is a useful component in these settings.
Critical weaknesses
W1 . The central mechanistic claim is unsupported, and the paper defers it to future work.
The title and contributions present ResLorB as acting “inside the residual stream” through “adaptive neuron-level interactions,” motivated by the claim that FC’s fixed partition cannot adapt to the learned representations (p.4). The architecture places the branch outside the stream: in Eq. (6), , so the stream passes unchanged and the branch alters only the block input. Appendix D.5 states this as a design property (preserves the original residual stream and introduces the low-rank transformation only to the block input), and the variant that writes into the stream (Eq. 15) performs worse under Sandwich-Norm. The operation the paper adopts is an additional module on the read path of each block.
The paper derives the method from the HC/FC formulation (Sec. 4.1) and gives no evidence for the mechanism it draws from that lineage. There is no measurement of misalignment between the fixed partition and learned representations, no analysis showing that the branch creates the missing interactions, and no intervention showing that removing those interactions removes the gain. Appendix C.4 attributes the advantage over FC to the interaction mechanism based on an 8-stream vs. 4-stream comparison, which leaves the mechanism untested. Theorem 1 concerns nonlinear feature lifting before attention, and its relation to neuron-level mixing is unclear.
The Conclusion and Appendix E describe the mechanistic understanding as preliminary and leave it to future work. The mechanism is the claim the paper itself puts forward in its title and contributions; deferring it leaves that claim unsupported in this submission.
With the mechanism unsupported, the contribution reduces to the empirical value of an additional pre-block module, and its small gains without variance estimates (W2) cannot be distinguished from selection among design variants.
W2. The empirical gain is not separated from added compute or from run-to-run variance.
With the motivation unsupported, the contribution rests on efficiency, and the evidence for efficiency is narrow:
Adding one Transformer layer comes close to ResLorB, which the paper describes as “slightly outperforms” (p.29, Fig. 14). In activated parameters, which govern compute, ResLorB adds about 0.03B and one extra layer adds about 0.037B. The efficiency claim rests on total parameters (+1% vs. +7%), which in an MoE model reflect expert storage. The reported denominators are also inconsistent: Table 3 lists 0.79B activated parameters for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase. No result reports seeds, standard deviations, or confidence intervals. The reported effects are small: a 0.011 loss reduction at 9B, differences of 0.001-0.014 in Table 4, and a 1.03-point average gain at 3B with lower scores on BBH and PIQA under Pre-Norm (Table 2). In Table 1, FC is better at 60M and NOBLE is better at 350M. The abstract states negligible latency overhead. The paper reports training throughput (57.7 56.0 TFLOP/s/GPU) and memory (44.54 46.09 GB) and contains no latency measurement for prefill or decoding.
W1 removes the stated reason for the design, and W2 leaves the fallback, an efficiency advantage, unestablished at matched compute and with variance accounted for. The combination determines my recommendation.
Questions for authors
Questions that could change my assessment
Q1. What observation led to this design? The paper motivates the branch from the claim that FC’s fixed partition cannot adapt to learned representations. What did the authors observe, in baseline, FC, or pretrained models, that shows this misalignment exists and that the missing interactions matter? Relatedly, which property of the HC/FC formulation remains operative in Eq. (6), which acts on a single state and contains no interaction matrix? The answer determines whether the HC/FC derivation is the reason for the design or a framing applied after the fact.
Q2. Is the gain attributable to neuron-level mixing? Figs. 11–13 show that downstream stream statistics change. Which result separates a change caused by neuron-level mixing from a change caused by added nonlinear capacity on the read path? A direct test: constrain and to be block-diagonal along the FC partition at matched parameter count. If the gain persists, it does not come from cross-partition interactions.
Q3. Is the gain distinguishable from added compute and from variance? Please compare ResLorB with the added-layer baseline at matched activated parameters or FLOPs, and report variance over seeds. The LLaMA scale is sufficient for the variance estimate.
Clarifications (these affect my Soundness score, not my rating)
C1. What does Theorem 1 contribute to the claim about neuron-level mixing? In the construction, the branch acts along a single direction (), and the target requires no interaction among coordinates.
C2. Please state the activated-parameter counts of all variants in Fig. 14 on a common basis (with or without MTP). Table 3 lists 0.79B for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase.
C3. The abstract states “negligible latency overhead.” Please report prefill and decoding latency.
C4. The caption of Table 8 refers to “different training stages,” and the table shows one stage. Fig. 1 omits the outer residual path that Eq. (6) and Algorithm 1 include.
Satisfactory answers to Q1–Q3 would raise my assessment.
7 comments
Response to Reviewer He9J Part II
(Continued from the previous comment.)
Second, in Appendix D.5, we explore whether the branch itself establishes a group structure. We compute the branch's mean-squared Jacobian over all 12 layers and 65,536 held-out tokens at the final iteration, and measure the fraction of off-diagonal Jacobian energy that falls inside the blocks of a partition, normalized by the fraction expected from a random partition with the same block sizes (). Under the fixed FC partition, for attention and for MLP, indicating no block structure. Searching over coordinate permutations with spectral clustering, we find that the best free partition reaches only (attention) and (MLP), against random-weight nulls of and , and its blocks show no clear alignment with the FC groups. The branch therefore mixes coordinates broadly rather than concentrating them into a fixed or recoverable sparse grouping.
The analyses above are motivation and diagnostics. The intervention is the partitioned-branch ablation in our response to Q2, which constrains the architecture and measures the change in loss at matched parameter count: ResLorB is clearly better there (≈1.4e-2 against ≈2e-3 in loss reduction). Taken together, these comparisons are consistent with a material contribution from learned cross-partition interactions, beyond nonlinear capacity or parameter count alone.
1.3 "Theorem 1 concerns nonlinear feature lifting before attention, and its relation to neuron-level mixing is unclear."
We explain the connection again here. The target is . Its nonlinear term depends on all coordinates jointly: it is a function of the sum and cannot be written as a sum of scalar functions of the individual coordinates, so realizing it requires reading neurons jointly. The theorem shows that a pre-block low-rank branch with attains strictly lower in-context risk than the linear-attention baseline ( gap): the baseline cannot form the joint read, and the branch can. For tractability, we consider this relatively simple case, and leave the generalization to more complex cases to future work.
Weakness 2
2.1 "Adding one Transformer layer comes close to ResLorB ... In activated parameters, which govern compute, ResLorB adds about 0.03B and one extra layer adds about 0.037B. The efficiency claim rests on total parameters (+1% vs. +7%), which in an MoE model reflect expert storage. The reported denominators are also inconsistent: Table 3 lists 0.79B activated parameters for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase."
We first explain the inconsistency in the denominators. Table 3 counts activated parameters with the MTP layer: baseline 0.783B (reported as 0.79B, with the components rounded separately), ResLorB 0.814B (+0.031B), and the added layer 0.817B (+0.034B). The sentence on p. 29 (original version) reports the counts without MTP: baseline 0.748B and the added layer 0.783B (+0.0358B, 4.8%), which accounts for the "0.783B" and the "5%" in that sentence. The eight-expert variant uses the same without-MTP basis: 0.823B, +10% over 0.748B; on this basis, ResLorB is 0.780B (+4.3%). The revision reports all statistics on the with-MTP basis (p. 32).
We now turn to the comparison with the added-layer variant. The two methods have closely matched activated-parameter budgets (within half a percentage point), and their loss curves are nearly identical, with ResLorB lower by about 5e-4 under Pre-Norm and 1e-3 under Sandwich-Norm. ResLorB, however, requires fewer total parameters (3.21B, +1%, versus 3.41B, +7%), incurs no additional KV cache, and introduces no additional MoE dispatch and combine operations per token. Measured training throughput under the same configuration (Appendix D.6, More Ablations) also shows the added-layer variant training more slowly than ResLorB, at 54.4 TFLOP/s/GPU against 56.0 (baseline 57.7). Moreover, the rank ablation in Fig. 7 shows that rank 256 is only about 2e-3 worse than rank 512 in training loss, indicating that the performance–efficiency trade-off of ResLorB at can be further improved. Overall, these results indicate that ResLorB is more efficient than adding one Transformer layer.
Response to Reviewer He9J Part III
2.2 "No result reports seeds, standard deviations, or confidence intervals." "The reported effects are small ... In Table 1, FC is better at 60M and NOBLE is better at 350M."
Both the 3B and 9B models are trained far beyond one Chinchilla-optimal epoch — 200B and 450B tokens, ≈66 and 50 tokens per parameter. Our 3B and 9B runs are single-seed. As we show in our response to Reviewer mkpM (1.2), three 3B baseline runs with different seeds converge to a validation-loss spread of ≈1e-3 from 30k steps onward (≈140B tokens), an order of magnitude below the effects reported below. ResLorB reduces the 3B training loss by 0.012 under Pre-Norm and 0.014 under Sandwich-Norm, and the 9B training loss by 0.011 relative to the mHC baseline; it improves the 3B downstream average by 1.03 points and the 9B ten-benchmark average by 1.4 points (44.1 → 45.5), with gains on 8 of 10 benchmarks. For comparison, Frac-Connection's own results at the 1B scale report a training-loss improvement of ≈0.014 (its Fig. 4) and ≈0.7% in downstream average (its Table 4). We therefore consider the improvement significant.
For the LLaMA models, we agree that the improvement over the baseline was relatively minor in the single-seed results. We have added seeds 42, 43, and 44 for every method and scale, with the learning rate searched per method and scale; Table 1 of the revision reports the mean and standard deviation. With seeds included, the advantage is somewhat larger than in the single-seed results: ResLorB is below the baseline at 60M / 130M / 350M by 0.021 / 0.017 / 0.020, against 0.015 / 0.009 / 0.008 for seed 42 alone. At 350M, NOBLE is statistically indistinguishable from ResLorB (2.84821 ± 0.00205 against 2.84729 ± 0.01095), at 8.3% additional parameters versus 1.7% for ResLorB. FC remains marginally ahead of ResLorB at 60M (+0.002), but the difference reverses at 130M (−0.004) and widens at 350M (−0.016), negative values meaning that ResLorB is lower. We therefore read the LLaMA results as a trend that strengthens with model scale rather than as a per-scale significance claim, and have softened the corresponding statements in the Introduction and Sec. 5.1 accordingly.
2.3 "The abstract states negligible latency overhead ... and contains no latency measurement for prefill or decoding."
We have added prefill and decode results for the 3B and 9B models (Appendix D.4, Table 9). We report measured GPU-bound decode latency; for compute-bound prefill, we report per-iteration training time as a proxy. Measurements use a single 8-GPU H200 node with EP=8 and TP=PP=CP=1 and the SwiGLU branch (same parameter budget as the default ReLU branch); at matched parameters the SwiGLU branch performs slightly more matrix multiplications than the ReLU branch, so the overhead is, if anything, an upper bound for the default configuration. For prefill, ResLorB adds 3.8% on 9B and 4.3% on 3B. In the GPU-bound decode regime (batch sizes 8192–32768, where step time is dominated by GPU execution), the overhead is 0.9–4.4% on 9B and 2.5–5.1% on 3B, smallest at the largest batch in both cases. The overhead is thus comparable to or smaller than the ≈4% increase in activated parameters. We have also replaced "negligible" with "small" in the abstract and conclusion.
Questions for Authors
Q1. "What observation led to this design? ... which property of the HC/FC formulation remains operative in Eq. (6)..."
For the first half of the question, please see our response to 1.2 above. For the second half: in Eq. (6), ResLorB replaces FC's interaction matrix with . Both and are learnable, which allows ResLorB to learn neuron-level interactions adaptively rather than fixing them in advance. Beyond learnability, is a low-rank transformation, whereas is a sparse, block-diagonal matrix; the two describe interactions in different forms and can be applied together, as shown for FC in Fig. 5(a).
Response to Reviewer He9J Part I
We thank the reviewer for the careful reading and the detailed review. We answer each point below in the reviewer's order.
Weakness 1
By "inside the residual stream" we mean that the interaction is intra-state: the branch mixes coordinates of the residual state representation that each block reads. The term describes the space in which the mixing occurs, not where the module's output is written. The branch is indeed a module on the read path, and "inside" refers to which coordinates are mixed; nothing in our claim requires the branch to write to the carried residual stream. The definition should have been stated more clearly in the submission, and we now state it explicitly: neuron-level interactions within the residual stream, i.e., among the coordinates of the state that each block reads.
The Eq. (17) variant (Eq. (15) in the submission) is one we designed and tested ourselves. In Fig. 17 it improves over the baseline by 9×10−3 under Pre-Norm and 6×10−3 under Sandwich-Norm, so neuron-level interaction is effective whether the branch only reads the block input or also writes back to the carried residual stream. Under Sandwich-Norm it falls behind the read-only variant; we think this is because writing into the stream changes the effective depth and scaling of each layer, which may require retuning the training configuration, as discussed in Appendix D.6. This is a choice between two implementations, not a constraint implied by our definition of intra-state interaction.
The current draft acknowledges that ResLorB was mainly derived from the algebraic structure of Frac-Connection. After reshaping, FC's pre-projection H^pre⊗Id/n is block-diagonal, so the coordinates that may interact are fixed at initialization. ResLorB replaces this projection with W2ReLU(W1): a learnable, nonlinear map at the same position. The replacement lowers the training loss at 3B and 9B (Figs. 4 and 9) and improves the downstream average at both scales (Tables 2 and 8). We do not claim that the loss reduction measures the misalignment; it shows that the replacement helps.
We did not arrive at the design without any prior observation, however. Dunefsky et al. (2024) find that features in MLP sublayers recovered by sparse autoencoders are dense linear combinations of many neurons: their support is spread across the coordinate basis. A partition that fixes which coordinates may interact therefore cuts across the features themselves, and interactions between such features are not confined to its groups.
To address your concern, we have added two analyses (Section 4.1 and Appendix D.5). The first builds on our hypothesis that the interaction pattern of a nonlinear module should be compatible with the statistical dependency structure of its input, allowing the module to effectively exploit dependencies among correlated neurons. We collect the hidden states at the input of all 24 sublayers of the trained 3B models, at initialization and after training, standardize each neuron across tokens, and average the correlation matrices over sublayers. For FC, the mean absolute correlation within each predefined group is ≈0.03–0.04 at both initialization and after training, and the correlation matrix shows no block structure aligned with the predefined groups. For mHC, by comparison, the same quantity is 1 at initialization and remains 0.54–0.74 after training, concentrated within its predefined groups. These results indicate that FC's fixed grouping may not be appropriate, or that the neurons allowed to interact within each group may be too few. Increasing the number of interacting neurons, however, does not help either: in our comparison of 4-stream and 8-stream FC (App. C.4, Fig. 8), the loss is nearly identical. We therefore believe that the grouping scheme itself should be re-examined, and ideally learned adaptively rather than fixed in advance, which is what ResLorB does.
(Continued in the next comment.)