acceptodds
Under review as a conference paper at ICLR 2027

Inside the Residual Stream: Neuron-Level Mixing with Nonlinear Low-Rank Branches

Abstract

Redesigning residual connections can improve the way information is propagated across deep Transformer layers. Methods such as Hyper-Connection and Attention Residuals enrich connectivity by introducing interactions among different hidden states. However, these approaches primarily operate at the level of whole hidden states, leaving finer-grained interactions within individual states largely unexplored. Frac-Connection explores finer-grained connectivity by partitioning hidden states into multiple streams and applying structured interactions, but its interaction pattern is constrained by a fixed coordinate-wise decomposition. In this work, we propose **ResLorB** (**Res**idual Connections with Nonlinear **Lo**w-**r**ank **B**ranches), which introduces a lightweight nonlinear low-rank transformation at the block input to enable learnable neuron-level interactions. With only minimal additional parameters and computational overhead, ResLorB consistently improves model performance without requiring specialized optimization or infrastructure support. It is naturally compatible with existing multi-state architectures, including mHC, and the low-rank branch can further serve as an additional interactable state to enable richer interaction patterns. Extensive experiments on DeepSeek-V3-style models scaling up to 3B/9B parameters demonstrate consistent improvements over strong baselines, including a 0.011 reduction in training LM loss and 1.4% improvement in downstream average performance at the 9B scale, while introducing only 1% additional parameters and negligible latency overhead. On 3B-scale models, ResLorB improves downstream performance by approximately 1% on average. Combining ResLorB with existing multi-state architectures further yields non-saturating gains and consistent improvements.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, residual connection, low-rank

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_He9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ResLorB, a nonlinear low-rank branch added at the input of each Transformer block: x~l=xl+Wl(2)σ(Wl(1)xl)\tilde{x}_l = x_l + W^{(2)}_l \sigma(W^{(1)}_l x_l), with Wl(1)∈Rr×d,Wl(2)∈Rd×rW^{(1)}_l \in \mathbb{R}^{r\times d},W^{(2)}_l \in \mathbb{R}^{d \times r}, and σ=ReLU\sigma = \mathrm{ReLU} by default. The block reads x~l\tilde{x}_l and writes to the stream xlx_l, which the branch leaves unmodified: xl+1=xl+Fl(x~l)x_{l+1} = x_l + \mathcal{F}_l(\tilde{x}_l). The branch adds 4drL4drL parameters. The paper frames the branch as a source of neuron-level interactions within the residual stream. It motivates the design from Frac-Connection, whose reshape operation fixes which hidden dimensions interact, and derives the branch as a learnable replacement for this fixed structured matrix. The branch combines with multi-state architectures. mHC-ResLorB applies it after the mHC pre-projection, and mHC-ResLorB+ gives it an independent pre-projection. The paper includes a theorem in a nonlinear in-context regression setting with one linear attention layer, in which the branch lowers the in-context risk. Experiments cover LLaMA-style models from 60M to 350M on C4, DeepSeek-V3-style 3B MoE models on 200B tokens under Pre-Norm and Sandwich-Norm, and a 9B mHC model on 450B tokens. The authors report a 0.011 training-loss reduction and a 1.4-point downstream gain at 9B (44.1 →\rightarrow 45.5), and about a 1-point gain at 3B, with about 1% additional total and 4% additional activated parameters. The appendices compare activation functions, branch placements, ranks, initialization, a stream-injection variant, a gated input variant, an added Transformer layer, eight active experts, and Gated Residual. They also report gradient norms, load balancing, adjacent-layer cosine similarity, and activation magnitudes.

Critical strengths

The ablation suite addresses several simple explanations of the gain. A linear branch (Table 4), a gated input-scaling variant, an eight-expert allocation, and Gated Residual (Figs. 14-15) perform worse than ResLorB, and the gain persists at 9B on top of mHC (Fig. 8, Table 8). This establishes that the pre-block nonlinear branch is a useful component in these settings.

Critical weaknesses

W1 . The central mechanistic claim is unsupported, and the paper defers it to future work.

The title and contributions present ResLorB as acting “inside the residual stream” through “adaptive neuron-level interactions,” motivated by the claim that FC’s fixed partition cannot adapt to the learned representations (p.4). The architecture places the branch outside the stream: in Eq. (6), xl+1=xl+Fl(x~l)x_{l+1} = x_l + \mathcal{F}_l(\tilde{x}_l), so the stream xlx_l passes unchanged and the branch alters only the block input. Appendix D.5 states this as a design property (preserves the original residual stream and introduces the low-rank transformation only to the block input), and the variant that writes into the stream (Eq. 15) performs worse under Sandwich-Norm. The operation the paper adopts is an additional module on the read path of each block.

The paper derives the method from the HC/FC formulation (Sec. 4.1) and gives no evidence for the mechanism it draws from that lineage. There is no measurement of misalignment between the fixed partition and learned representations, no analysis showing that the branch creates the missing interactions, and no intervention showing that removing those interactions removes the gain. Appendix C.4 attributes the advantage over FC to the interaction mechanism based on an 8-stream vs. 4-stream comparison, which leaves the mechanism untested. Theorem 1 concerns nonlinear feature lifting before attention, and its relation to neuron-level mixing is unclear.

The Conclusion and Appendix E describe the mechanistic understanding as preliminary and leave it to future work. The mechanism is the claim the paper itself puts forward in its title and contributions; deferring it leaves that claim unsupported in this submission.

With the mechanism unsupported, the contribution reduces to the empirical value of an additional pre-block module, and its small gains without variance estimates (W2) cannot be distinguished from selection among design variants.


W2. The empirical gain is not separated from added compute or from run-to-run variance.

With the motivation unsupported, the contribution rests on efficiency, and the evidence for efficiency is narrow:

Adding one Transformer layer comes close to ResLorB, which the paper describes as “slightly outperforms” (p.29, Fig. 14). In activated parameters, which govern compute, ResLorB adds about 0.03B and one extra layer adds about 0.037B. The efficiency claim rests on total parameters (+1% vs. +7%), which in an MoE model reflect expert storage. The reported denominators are also inconsistent: Table 3 lists 0.79B activated parameters for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase. No result reports seeds, standard deviations, or confidence intervals. The reported effects are small: a 0.011 loss reduction at 9B, differences of 0.001-0.014 in Table 4, and a 1.03-point average gain at 3B with lower scores on BBH and PIQA under Pre-Norm (Table 2). In Table 1, FC is better at 60M and NOBLE is better at 350M. The abstract states negligible latency overhead. The paper reports training throughput (57.7 →\rightarrow 56.0 TFLOP/s/GPU) and memory (44.54 →\rightarrow 46.09 GB) and contains no latency measurement for prefill or decoding.

W1 removes the stated reason for the design, and W2 leaves the fallback, an efficiency advantage, unestablished at matched compute and with variance accounted for. The combination determines my recommendation.

Questions for authors

Questions that could change my assessment

Q1. What observation led to this design? The paper motivates the branch from the claim that FC’s fixed partition cannot adapt to learned representations. What did the authors observe, in baseline, FC, or pretrained models, that shows this misalignment exists and that the missing interactions matter? Relatedly, which property of the HC/FC formulation remains operative in Eq. (6), which acts on a single state and contains no interaction matrix? The answer determines whether the HC/FC derivation is the reason for the design or a framing applied after the fact.

Q2. Is the gain attributable to neuron-level mixing? Figs. 11–13 show that downstream stream statistics change. Which result separates a change caused by neuron-level mixing from a change caused by added nonlinear capacity on the read path? A direct test: constrain W1W^{1} and W2W^{2} to be block-diagonal along the FC partition at matched parameter count. If the gain persists, it does not come from cross-partition interactions.

Q3. Is the gain distinguishable from added compute and from variance? Please compare ResLorB with the added-layer baseline at matched activated parameters or FLOPs, and report variance over seeds. The LLaMA scale is sufficient for the variance estimate.


Clarifications (these affect my Soundness score, not my rating)

C1. What does Theorem 1 contribute to the claim about neuron-level mixing? In the construction, the branch acts along a single direction (1\mathbf{1}), and the target requires no interaction among coordinates.

C2. Please state the activated-parameter counts of all variants in Fig. 14 on a common basis (with or without MTP). Table 3 lists 0.79B for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase.

C3. The abstract states “negligible latency overhead.” Please report prefill and decoding latency.

C4. The caption of Table 8 refers to “different training stages,” and the table shows one stage. Fig. 1 omits the outer residual path xlx_l that Eq. (6) and Algorithm 1 include.

Satisfactory answers to Q1–Q3 would raise my assessment.

7 comments
AuthorsOpenReview ↗

Response to Reviewer He9J Part I

We thank the reviewer for the careful reading and the detailed review. We answer each point below in the reviewer's order.


Weakness 1

1.1 "The central mechanistic claim is unsupported, and the paper defers it to future work. ... The architecture places the branch outside the stream ... the variant that writes into the stream (Eq. 15) performs worse under Sandwich-Norm. ... The operation the paper adopts is an additional module on the read path of each block."

By "inside the residual stream" we mean that the interaction is intra-state: the branch mixes coordinates of the residual state representation that each block reads. The term describes the space in which the mixing occurs, not where the module's output is written. The branch is indeed a module on the read path, and "inside" refers to which coordinates are mixed; nothing in our claim requires the branch to write to the carried residual stream. The definition should have been stated more clearly in the submission, and we now state it explicitly: neuron-level interactions within the residual stream, i.e., among the coordinates of the state that each block reads.

The Eq. (17) variant (Eq. (15) in the submission) is one we designed and tested ourselves. In Fig. 17 it improves over the baseline by 9×10−39\times10^{-3} under Pre-Norm and 6×10−36\times10^{-3} under Sandwich-Norm, so neuron-level interaction is effective whether the branch only reads the block input or also writes back to the carried residual stream. Under Sandwich-Norm it falls behind the read-only variant; we think this is because writing into the stream changes the effective depth and scaling of each layer, which may require retuning the training configuration, as discussed in Appendix D.6. This is a choice between two implementations, not a constraint implied by our definition of intra-state interaction.

1.2 "There is no measurement of misalignment between the fixed partition and learned representations, no analysis showing that the branch creates the missing interactions, and no intervention showing that removing those interactions removes the gain. ... Appendix C.4 attributes the advantage over FC to the interaction mechanism based on an 8-stream vs. 4-stream comparison, which leaves the mechanism untested."

The current draft acknowledges that ResLorB was mainly derived from the algebraic structure of Frac-Connection. After reshaping, FC's pre-projection H^pre⊗Id/n\hat{H}^{\mathrm{pre}} \otimes I_{d/n} is block-diagonal, so the coordinates that may interact are fixed at initialization. ResLorB replaces this projection with W2ReLU(W1)W_2 \mathrm{ReLU}(W_1): a learnable, nonlinear map at the same position. The replacement lowers the training loss at 3B and 9B (Figs. 4 and 9) and improves the downstream average at both scales (Tables 2 and 8). We do not claim that the loss reduction measures the misalignment; it shows that the replacement helps.

We did not arrive at the design without any prior observation, however. Dunefsky et al. (2024) find that features in MLP sublayers recovered by sparse autoencoders are dense linear combinations of many neurons: their support is spread across the coordinate basis. A partition that fixes which coordinates may interact therefore cuts across the features themselves, and interactions between such features are not confined to its groups.

To address your concern, we have added two analyses (Section 4.1 and Appendix D.5). The first builds on our hypothesis that the interaction pattern of a nonlinear module should be compatible with the statistical dependency structure of its input, allowing the module to effectively exploit dependencies among correlated neurons. We collect the hidden states at the input of all 24 sublayers of the trained 3B models, at initialization and after training, standardize each neuron across tokens, and average the correlation matrices over sublayers. For FC, the mean absolute correlation within each predefined group is ≈0.03–0.04 at both initialization and after training, and the correlation matrix shows no block structure aligned with the predefined groups. For mHC, by comparison, the same quantity is 1 at initialization and remains 0.54–0.74 after training, concentrated within its predefined groups. These results indicate that FC's fixed grouping may not be appropriate, or that the neurons allowed to interact within each group may be too few. Increasing the number of interacting neurons, however, does not help either: in our comparison of 4-stream and 8-stream FC (App. C.4, Fig. 8), the loss is nearly identical. We therefore believe that the grouping scheme itself should be re-examined, and ideally learned adaptively rather than fixed in advance, which is what ResLorB does.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer He9J Part II

(Continued from the previous comment.)

Second, in Appendix D.5, we explore whether the branch itself establishes a group structure. We compute the branch's mean-squared Jacobian over all 12 layers and 65,536 held-out tokens at the final iteration, and measure the fraction of off-diagonal Jacobian energy that falls inside the blocks of a partition, normalized by the fraction expected from a random partition with the same block sizes (E=Rwithin/pbaseE = R_{\mathrm{within}} / p_{\mathrm{base}}). Under the fixed FC partition, E=1.0005E = 1.0005 for attention and 1.00081.0008 for MLP, indicating no block structure. Searching over coordinate permutations with spectral clustering, we find that the best free partition reaches only E=1.066E = 1.066 (attention) and 1.0371.037 (MLP), against random-weight nulls of 1.0201.020 and 1.0321.032, and its blocks show no clear alignment with the FC groups. The branch therefore mixes coordinates broadly rather than concentrating them into a fixed or recoverable sparse grouping.

The analyses above are motivation and diagnostics. The intervention is the partitioned-branch ablation in our response to Q2, which constrains the architecture and measures the change in loss at matched parameter count: ResLorB is clearly better there (≈1.4e-2 against ≈2e-3 in loss reduction). Taken together, these comparisons are consistent with a material contribution from learned cross-partition interactions, beyond nonlinear capacity or parameter count alone.

1.3 "Theorem 1 concerns nonlinear feature lifting before attention, and its relation to neuron-level mixing is unclear."

We explain the connection again here. The target is g(x)=x+(1⊤x)21g(x) = x + (\mathbf{1}^{\top}x)^2 \mathbf{1}. Its nonlinear term depends on all coordinates jointly: it is a function of the sum 1⊤x\mathbf{1}^{\top}x and cannot be written as a sum of scalar functions of the individual coordinates, so realizing it requires reading neurons jointly. The theorem shows that a pre-block low-rank branch with r≥d3/4r \ge d^{3/4} attains strictly lower in-context risk than the linear-attention baseline (Ω(d)\Omega(d) gap): the baseline cannot form the joint read, and the branch can. For tractability, we consider this relatively simple case, and leave the generalization to more complex cases to future work.


Weakness 2

2.1 "Adding one Transformer layer comes close to ResLorB ... In activated parameters, which govern compute, ResLorB adds about 0.03B and one extra layer adds about 0.037B. The efficiency claim rests on total parameters (+1% vs. +7%), which in an MoE model reflect expert storage. The reported denominators are also inconsistent: Table 3 lists 0.79B activated parameters for the baseline, and p.29 describes 0.783B for the added-layer model as a 5% increase."

We first explain the inconsistency in the denominators. Table 3 counts activated parameters with the MTP layer: baseline 0.783B (reported as 0.79B, with the components rounded separately), ResLorB 0.814B (+0.031B), and the added layer 0.817B (+0.034B). The sentence on p. 29 (original version) reports the counts without MTP: baseline 0.748B and the added layer 0.783B (+0.0358B, 4.8%), which accounts for the "0.783B" and the "5%" in that sentence. The eight-expert variant uses the same without-MTP basis: 0.823B, +10% over 0.748B; on this basis, ResLorB is 0.780B (+4.3%). The revision reports all statistics on the with-MTP basis (p. 32).

We now turn to the comparison with the added-layer variant. The two methods have closely matched activated-parameter budgets (within half a percentage point), and their loss curves are nearly identical, with ResLorB lower by about 5e-4 under Pre-Norm and 1e-3 under Sandwich-Norm. ResLorB, however, requires fewer total parameters (3.21B, +1%, versus 3.41B, +7%), incurs no additional KV cache, and introduces no additional MoE dispatch and combine operations per token. Measured training throughput under the same configuration (Appendix D.6, More Ablations) also shows the added-layer variant training more slowly than ResLorB, at 54.4 TFLOP/s/GPU against 56.0 (baseline 57.7). Moreover, the rank ablation in Fig. 7 shows that rank 256 is only about 2e-3 worse than rank 512 in training loss, indicating that the performance–efficiency trade-off of ResLorB at r=512r = 512 can be further improved. Overall, these results indicate that ResLorB is more efficient than adding one Transformer layer.

AuthorsOpenReview ↗

Response to Reviewer He9J Part III

2.2 "No result reports seeds, standard deviations, or confidence intervals." "The reported effects are small ... In Table 1, FC is better at 60M and NOBLE is better at 350M."

Both the 3B and 9B models are trained far beyond one Chinchilla-optimal epoch — 200B and 450B tokens, ≈66 and 50 tokens per parameter. Our 3B and 9B runs are single-seed. As we show in our response to Reviewer mkpM (1.2), three 3B baseline runs with different seeds converge to a validation-loss spread of ≈1e-3 from 30k steps onward (≈140B tokens), an order of magnitude below the effects reported below. ResLorB reduces the 3B training loss by 0.012 under Pre-Norm and 0.014 under Sandwich-Norm, and the 9B training loss by 0.011 relative to the mHC baseline; it improves the 3B downstream average by 1.03 points and the 9B ten-benchmark average by 1.4 points (44.1 → 45.5), with gains on 8 of 10 benchmarks. For comparison, Frac-Connection's own results at the 1B scale report a training-loss improvement of ≈0.014 (its Fig. 4) and ≈0.7% in downstream average (its Table 4). We therefore consider the improvement significant.

For the LLaMA models, we agree that the improvement over the baseline was relatively minor in the single-seed results. We have added seeds 42, 43, and 44 for every method and scale, with the learning rate searched per method and scale; Table 1 of the revision reports the mean and standard deviation. With seeds included, the advantage is somewhat larger than in the single-seed results: ResLorB is below the baseline at 60M / 130M / 350M by 0.021 / 0.017 / 0.020, against 0.015 / 0.009 / 0.008 for seed 42 alone. At 350M, NOBLE is statistically indistinguishable from ResLorB (2.84821 ± 0.00205 against 2.84729 ± 0.01095), at 8.3% additional parameters versus 1.7% for ResLorB. FC remains marginally ahead of ResLorB at 60M (+0.002), but the difference reverses at 130M (−0.004) and widens at 350M (−0.016), negative values meaning that ResLorB is lower. We therefore read the LLaMA results as a trend that strengthens with model scale rather than as a per-scale significance claim, and have softened the corresponding statements in the Introduction and Sec. 5.1 accordingly.

2.3 "The abstract states negligible latency overhead ... and contains no latency measurement for prefill or decoding."

We have added prefill and decode results for the 3B and 9B models (Appendix D.4, Table 9). We report measured GPU-bound decode latency; for compute-bound prefill, we report per-iteration training time as a proxy. Measurements use a single 8-GPU H200 node with EP=8 and TP=PP=CP=1 and the SwiGLU branch (same parameter budget as the default ReLU branch); at matched parameters the SwiGLU branch performs slightly more matrix multiplications than the ReLU branch, so the overhead is, if anything, an upper bound for the default configuration. For prefill, ResLorB adds 3.8% on 9B and 4.3% on 3B. In the GPU-bound decode regime (batch sizes 8192–32768, where step time is dominated by GPU execution), the overhead is 0.9–4.4% on 9B and 2.5–5.1% on 3B, smallest at the largest batch in both cases. The overhead is thus comparable to or smaller than the ≈4% increase in activated parameters. We have also replaced "negligible" with "small" in the abstract and conclusion.


Questions for Authors

Q1. "What observation led to this design? ... which property of the HC/FC formulation remains operative in Eq. (6)..."

For the first half of the question, please see our response to 1.2 above. For the second half: in Eq. (6), ResLorB replaces FC's interaction matrix H^pre⊗Id/n\hat{H}^{\mathrm{pre}} \otimes I_{d/n} with W2σ(W1)W_2\sigma(W_1). Both W1W_1 and W2W_2 are learnable, which allows ResLorB to learn neuron-level interactions adaptively rather than fixing them in advance. Beyond learnability, W2σ(W1)W_2\sigma(W_1) is a low-rank transformation, whereas H^pre⊗Id/n\hat{H}^{\mathrm{pre}} \otimes I_{d/n} is a sparse, block-diagonal matrix; the two describe interactions in different forms and can be applied together, as shown for FC in Fig. 5(a).

Reviewer_mkpMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As the title suggests, this paper focuses on residual stream information movement at a finer granularity than the entire hidden state. The authors find that if the Frac-Connection update is rewritten in terms of the hidden-state space, the interacting coordinates are fixed by the initial reshaping. The key point is that its pre-block operator is actually block-diagonal. At the input of each block, ResLorB replaces this operator with a learnable low-rank branch, so the block input becomes x + W2 ReLU(W1 x) before the standard block runs. Roughly, the paper does three things: it proves an in-context risk gap for a synthetic in-context regression setting, describes two ways to wire the branch into mHC/FC, and pretrains LLaMA- and DeepSeek-V3-style models at several scales.

Critical strengths
  1. Overhead is modest: about 1% more parameters, a small throughput drop, and no extra KV cache (Table 3). The more interesting result is that the branch still helps on top of mHC (Table 7). The FC baseline also looks properly tuned, so the comparison isn't against a straw man.
  2. The scale of the evaluation is above average for this kind of architectural change: the 3B model is trained under both Pre-Norm and Sandwich-Norm on 200B tokens, and a 9B mHC model is additionally trained on 450B tokens. The downstream average over the ten benchmarks moves from 44.1 to 45.5 (Table 8), and the training curves show the gap persisting through the annealing phase rather than only at the final checkpoint.
Critical weaknesses
  1. I found the LAUREL comparison hard to pin down. Section 2 says LAUREL already puts a parallel low-rank branch next to the residual stream, which makes the delta here look like pre-block placement, the nonlinearity, and the replacement of FC's block-diagonal map. The last part is the interesting one, but the first two are supported mainly by Table 4, and the gaps there are tiny (2.333 vs 2.339 for placement, 2.333 for both activations). With no seed variance anywhere, I don't know whether those differences are real. If they aren't, the design argument is weaker than the paper suggests — though this is about the ablation, not the 3B/9B results, which look solid. Theorem 1 doesn't help here either: it compares ResLorB with the baseline and constrains r, but says nothing about placement or activation.
  2. The paper also doesn't say how many runs these numbers average over, which makes the small gaps hard to interpret. On FC, Section 5.1 says "competitive or better", but at 60M FC is actually ahead on both loss and parameter overhead, and averaging the three LLaMA scales gives the same loss for both methods. So the "outperforms on average" claim in Section 1 is doing more work than the table supports. I also couldn't tell which scale the authors regard as the main result, or whether the claim is meant to be about cross-scale consistency. The larger 3B/9B results still look like the paper's main evidence to me.
Questions for authors
  1. Could you state whether the Table 4 ablation is on the 3B model, and whether Tables 1 and 4 are single runs? The loss scale suggests 3B, but it isn't spelled out, and it affects how seriously I can read the 2.333 vs 2.339 difference.

  2. Seeds on the three LLaMA scales would help, since the 60M comparison is close. For Table 4, I'd settle for knowing the number of runs; if it is one run and compute allows, repeating just the pre/post pair would be enough.

  3. A bias-only branch and a norm-matched random branch would separate a real interaction from a bias or scaling effect. The linear variant in Table 4 is already worse than ReLU, but it doesn't isolate this. The 350M setting should be enough if 3B is too expensive.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mkpM Part I

We thank the reviewer for the careful reading and the thoughtful review, in particular for acknowledging that our FC baselines are properly tuned and that we evaluate at an appropriate scale. We answer the points in the reviewer's order.


Weakness 1: "I found the LAUREL comparison hard to pin down."

1.1 "Section 2 says LAUREL already puts a parallel low-rank branch next to the residual stream, which makes the delta here look like pre-block placement, the nonlinearity, and the replacement of FC's block-diagonal map. The last part is the interesting one."

We agree with this decomposition, and that the third part is the substantive one. LAUREL updates the stream as xk+1=xk+F(xk)+BAxkx_{k+1} = x_k + F(x_k) + BA x_k (Eq. 3 in Menghani et al., 2025): its low-rank branch is linear and runs in parallel to the residual stream, outside the block. ResLorB instead places the branch at the block input and makes it nonlinear, x~=x+W2σ(W1x)\tilde{x} = x + W_2\sigma(W_1 x), so it enters FF rather than the stream. Replacing FC's fixed block-diagonal operator is the claim we make, and the focus of the main experiments.

1.2 "... the first two are supported mainly by Table 4, and the gaps there are tiny (2.333 vs 2.339 for placement, 2.333 for both activations). With no seed variance anywhere, I don't know whether those differences are real. ... Theorem 1 doesn't help here either."

The Table 4 ablation uses the 3B model under Sandwich-Norm; the revised caption now states the model and the normalization. We believe the placement gap is meaningful at this scale: in Fig. 17, in the same 3B setting, moving from six to eight active experts improves the loss by ≈8e-3 at +10% activated parameters, whereas moving the branch from the post-block to the pre-block position gives a 6e-3 difference. This also bears on the reviewer's concern about variance: while tuning the baseline, we ran three different random seeds (1234, 42, 1) to 34k steps (≈140B tokens each), and the validation loss is as follows.

StepSeed 1234Seed 42Seed 1Spread
10k2.57832.58322.58064.9e-3
20k2.51122.51352.51232.3e-3
30k2.48292.48362.48401.1e-3
34k2.48062.48172.48121.1e-3

While the three runs differ early in training (the spread is 4.9e-3 at 10k steps), they converge to ≈1e-3 from 30k steps onward; the training-loss spread is smaller still, and both lie well below the 6e-3 gap between the pre- and post-block placements.

At the LLaMA scales, we have also added three seeds (42, 43, 44) for every method; Table 1 reports the means and standard deviations (see our response to 2.2 for details).

On Theorem 1, the motivation is to isolate the effect of neuron-level mixing on representational capacity. For tractability and simplicity, we consider an in-context learning setting and a target function g(x)g(x) that cannot be represented without combining neurons. We prove that a single linear attention layer with the low-rank branch attains a strictly lower optimal approximation error than the same layer without it. This gives a theoretical account of that effect in a specific setting. Extending the analysis to more general settings would be interesting, and we leave it to future work.


Weakness 2: "Difficult to understand ResLorB's utility from results"

2.1 "The paper also doesn't say how many runs these numbers average over, which makes the small gaps hard to interpret."

All LLaMA-scale results in Table 1 are averaged over three seeds (42, 43, 44); the 3B and 9B runs are single runs, for the reason given in 1.2. In addition, as described in Q2, we ran the pre-/post-block comparison in the same 3B setting as Table 4, with Kaiming initialization; the gap is nearly the same as that reported in Table 4 (≈6e-3). We therefore consider this gap to be reliable.

2.2 "On FC, Section 5.1 says 'competitive or better', but at 60M FC is actually ahead on both loss and parameter overhead, and averaging the three LLaMA scales gives the same loss for both methods. So the 'outperforms on average' claim in Section 1 is doing more work than the table supports. I also couldn't tell which scale the authors regard as the main result."

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer mkpM Part II

(Continued from the previous comment.)

We regard the 3B and 9B runs as the main results: a DeepSeek-V3-style MoE model trained on 200B tokens at 3B and 450B tokens at 9B. At 3B, ResLorB is ahead of FC by ≈7e-3 under Pre-Norm and ≈1.4e-2 under Sandwich-Norm; at 9B, it reduces training loss over the mHC baseline by 0.011, with a downstream gain of 1.4 points (44.1 → 45.5, ≈3.2% relative; Table 8).

On the LLaMA scales, Table 1 in the revised version now reports three seeds for every method, with the learning rate searched per method and scale. With multiple seeds, the advantage of ResLorB is slightly larger than in the single-seed results: it is below the baseline by 0.021 / 0.017 / 0.020 at 60M / 130M / 350M, and its difference to FC moves from +0.002 at 60M to −0.004 at 130M and −0.016 at 350M — the same direction as at 3B. We agree that FC is marginally ahead at 60M, and we have softened the claim accordingly. The Introduction now states that ResLorB has lower validation loss than FC on average, with the gap widening as the model scale increases; Sec. 5.1 states that FC remains marginally better at 60M, while ResLorB is lowest at 130M and 350M and on the three-scale average (3.099 vs. 3.105).


Questions for Authors

Q1. "Could you state whether the Table 4 ablation is on the 3B model, and whether Tables 1 and 4 are single runs?"

Table 4 evaluates the 3B model under Sandwich-Norm, as specified in the caption in the revised version. Both Table 1 and Table 4 were single runs in the submission; in the revision, Table 1 reports three seeds for the LLaMA scales (see 2.2), while the 3B and 9B runs remain single, as now stated in Sec. 5.1.

Q2. "Seeds on the three LLaMA scales would help... For Table 4, I'd settle for knowing the number of runs; if it is one run and compute allows, repeating just the pre/post pair would be enough."

We have added three seeds (42, 43, 44) for every method at 60M / 130M / 350M; the results are in Table 1 and discussed in 2.2. At 60M, ResLorB remains 0.002 behind FC, but it is ahead by 0.004 at 130M and 0.016 at 350M.

Table 4, by contrast, is a single run. At present we do not have the compute to run the pre-/post-block pair with multiple seeds at 3B. We had, however, previously run the same pair under Kaiming initialization, and we include these runs together with the default-initialization results of Table 4 in Fig. 21 of the appendix. The final losses are close across the two initializations, and the pre-block placement maintains a ≈6e-3 advantage in training loss over the post-block placement. The two initializations provide a robustness check across initialization schemes, although they are not a substitute for independent seed replicates.

Q3. "A bias-only branch and a norm-matched random branch would separate a real interaction from a bias or scaling effect. The linear variant in Table 4 is already worse than ReLU, but it doesn't isolate this. The 350M setting should be enough if 3B is too expensive."

We thank the reviewer for the suggestion. We additionally ran the following controls, implemented as we understand them. The protocols and results are as follows.

  • For the bias-only control, we replace the low-rank branch in x~=x+W2σ(W1x)\tilde{x} = x + W_2\sigma(W_1 x) with a learned bias bb, i.e. x~=x+b\tilde{x} = x + b, with one zero-initialized vector per branch position. This isolates a learned additive offset with no input dependence.
  • For the norm-matched random control, we keep the same rank-64 ReLU bottleneck at all 48 positions, initialize W1W_1 and W2W_2 randomly once and freeze them, and use a per-stage scale αs\alpha_s to align the branch norm with that of the trained ResLorB branch: x~=x+αsW2σ(W1x)\tilde{x} = x + \alpha_s W_2 \sigma(W_1 x). The branch remains input-dependent and nonlinear, but its weights are never trained; the control therefore tests whether an untrained perturbation of matched magnitude can reproduce the gain.

All runs use the LLaMA-style 350M model under Pre-Norm (24 layers, 48 branch positions, seed 43, 60k steps; single run each).

MethodValidation loss
Baseline2.8655
Bias-only2.8681
Norm-matched random2.8793
ResLorB2.8575

The bias-only branch is close to the baseline, and the norm-matched random branch is worse than the baseline; ResLorB is the lowest of the four. Both controls lie above ResLorB, which to some extent supports the view that the gain comes mainly from learned interaction and cannot be accounted for by a learned bias or a scaling effect.

Moreover, two further results at the 3B scale point the same way. Under Sandwich-Norm, the gated ResLorB in Fig. 17 does not improve on the baseline despite a comparable parameter budget, and the FC-partitioned branch in Fig. 18 (our response to Reviewer He9J, Q2) recovers only ≈2e-3 of the ≈1.4e-2 gain when the interaction is confined to FC's groups. Together they indicate that the gain of ResLorB comes mainly from learned interaction.

Reviewer_mkpMOpenReview ↗

Response to Authors

I appreciate the authors' thorough response. The provided explanations have resolved my initial concerns.

Reviewer_HzaHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ResLorB, which is an architectural modification to Transformers that adds a non-linear low-rank transformation to the input to a Transformer layer, and then maintains the standard residual connection after the block. This modification is in place to enable neuron-level interactions and reduce the rank of the input before it is passed to the Transformer MLP or Attn block. Results are shown for small Llama models pretraiend on C4, and a 3B DeepSeek v3 model pretrained on 200B tokens. Results demonstrate loss improvement over the vanilla transformer with slight increase in parameters, and comparable performance to other residual modifications.

Critical strengths
  1. Helpful ablation motivates some of ResLorB's design choices: It is helpful to include Table 4, which demonstrates that the placement and ReLU choices of ResLorB are better than alternatives. ReLU is compared to SwiGLU, and the placement is compared to post-block and a sandwich variety.

  2. Evaluation includes downstream tasks: It is a nice inclusion to have downstream performance of the 3B model in addition to its loss performance; this inclusion gives a fuller picture of the model's capability, and sometimes is not present in architectural papers like these.

Critical weaknesses
  1. Method descriptions contradict each other, more motivation needed: Figure 1 which describes ResLorB and the description of ResLorB on page 4 do not match. The same is true of Figure 2, where equation 8, Figure 2b, and the description of ResLorB again do not align. It is not clear to me how it is possible that the description, which maintains the post-block residual would not match the figure, which applies the residual before the block. It is also likely the case that as Figure 1(c) stands, this architecture would be extremely unstable, which leads me to believe the written description is what was implemented. These discrepancies limit the integrity of the paper as it is not clear what exactly the method is, as well as its reproducibility. Additionally, it is unclear why rank-reducing the input to the Transformer block would be helpful, which is supported by Figure 6 which suggests higher rank is better; I am concerned that the improvements present are due to increased capacity rather than the specific design choices made for ResLorB.

  2. Difficult to understand ResLorB's utility from results: The main results in Table 1 do not clearly demonstrate the utility of ResLorB over similar methods. All runs are done one time which does not account for seed noise, and all margins are fairly small, especially those between FC and ResLorB. Additionally, each method is not isoFLOP adjusted, which makes their comparison difficult; FC uses fewer parameters yet still performs very similarly to ResLorB. In the 3B model case, it is still the case that ResLorB has more parameters than FC, which again makes their direct comparison difficult. Given the minor differences in loss and downstream performance, it is important to be able to attribute these differences to architectural efficiency rather than solely increased parameters.

Questions for authors
  1. How many tokens are each of the models trained for? In the text (lines 292-293), it is stated that each is trained for approximately one epoch following Chinchilla, but the model sizes are quite different, making this statement unclear.
  2. Are there FC results for Table 2 to accompany the comparison present in Figure 3?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HzaH Part I

We thank the reviewer for the detailed comments. We answer each point below in the reviewer's order. Before that, we clarify what our method does.

ResLorB leaves the residual stream unchanged and adds a low-rank update on the read path of each block:

x̃_l = x_l + W_2 σ(W_1 x_l),   x_{l+1} = x_l + F_l(x̃_l)
W_1 ∈ R^{r×d},   W_2 ∈ R^{d×r}

The hidden dimension is unchanged and the branch is purely additive. The block input is x~l\tilde{x}_l. The update W2σ(W1xl)W_2 \sigma(W_1 x_l) has rank at most rr; it is added to xlx_l rather than substituted for it. The description in the summary — "reduce the rank of the input before it is passed to the Transformer MLP or Attn block" — is therefore not accurate.


Weakness 1: "Method descriptions contradict each other, more motivation needed"

1.1 "Figure 1 ... and the description of ResLorB on page 4 do not match ... as Figure 1(c) stands, this architecture would be extremely unstable."

We apologize for the mistake; the written description is what was implemented. Fig. 1(c) omits the outer residual arrow xl→xl+1x_l \to x_{l+1}, so as drawn it shows xl+1=Fl(xl+W2σ(W1xl))x_{l+1} = F_l(x_l + W_2\sigma(W_1 x_l)). Eq. (6) and Algorithm 1 are correct, and the implementation has always followed them; we have redrawn Fig. 1 in the revision with the outer residual path included. With the outer residual restored, the stack retains the standard residual path, so the instability the reviewer anticipates does not arise.

1.2 "... it is unclear why rank-reducing the input to the Transformer block would be helpful, which is supported by Figure 6 which suggests higher rank is better."

The branch does not reduce the rank of the block input. rr is the branch's internal width: the low-rank matrices act inside the branch, and their output is added to xlx_l at full width dd.

Fig. 7 reports the loss difference of ResLorB relative to the baseline, so a more negative value means a larger advantage over the baseline. Larger rr gives the low-rank branch more parameter capacity, which improves the loss; this is the behavior expected of an additive capacity term and is consistent with the view above. The low rank here mainly serves to reduce parameter overhead; we do not claim that a smaller rank is better.

1.3 "The same is true of Figure 2, where equation 8, Figure 2b, and the description of ResLorB again do not align."

Fig. 2 (now Fig. 3) omits the Hres×SH^{\mathrm{res}} \times S term for simplicity, to keep the figure focused on the read-path wiring, and the mHC recurrence otherwise follows Eq. (2) unchanged; the caption now states this explicitly so that it is not read as a removed residual path. Apart from this omission, we find another error in the panel: in Fig. 3(b), Hpre,1H^{\mathrm{pre},1} and Hpre,2H^{\mathrm{pre},2} are swapped relative to Eq. (8). The figure applies the branch to Hpre,1H^{\mathrm{pre},1}, whereas Eq. (8) applies it to Hpre,2H^{\mathrm{pre},2}. Because the two projections share the same parameterization and are instantiated independently, both drawings define the same function family, but the figure should match the equation, and we have corrected the panel in the revision.

1.4 "I am concerned that the improvements present are due to increased capacity rather than the specific design choices made for ResLorB."

We have compared ResLorB with other variants at the 3B scale in Figs. 17 and 19 (Figs. 14 and 15 in the old version). Gated ResLorB (Fig. 17) and Gated Residual (Fig. 19) introduce the same or slightly more parameters than ResLorB, yet both remain clearly behind it in loss. The variant that comes closest to ResLorB is Add_1_Layer (Fig. 17), i.e., adding one additional Transformer layer; it introduces a much larger total-parameter overhead (+7% against +1%) and also adds KV-cache and MoE-communication overhead. In our response to Reviewer He9J (Q2) we further ran an experiment that confines the branch to FC's interaction groups with the parameter count matched to ResLorB (Fig. 18 in the revised version); it recovers only a small part of ResLorB's gain. Taken together, these comparisons are consistent with a material contribution from learned cross-partition interactions, beyond nonlinear capacity or parameter count alone.


AuthorsOpenReview ↗

Response to Reviewer HzaH Part II

Weakness 2: "Difficult to understand ResLorB's utility from results"

2.1 "All runs are done one time which does not account for seed noise, and all margins are fairly small, especially those between FC and ResLorB."

We agree for the LLaMA models. We have added three seeds (42, 43, 44) for every method and scale, with the learning rate searched per method and scale; the results are in Table 1 of the revised version. With seeds included, the advantage over the baseline is in fact larger than in the single-seed experiment: ResLorB is below the baseline at 60M / 130M / 350M by 0.021 / 0.017 / 0.020, against 0.015 / 0.009 / 0.008 for seed 42 alone, and its difference to FC moves from +0.002 at 60M to −0.004 at 130M and −0.016 at 350M (negative values mean ResLorB is lower).

On the margins: the 3B model is trained on 200B tokens, far beyond one Chinchilla-optimal epoch. Our 3B runs are single-seed. As we show in our response to Reviewer mkpM (1.2), three 3B baseline runs whose seeds differ settle to a validation-loss spread of ≈1e-3 from 30k steps onward. Against that, a training-loss improvement above 0.01 from a single run is unlikely to reflect seed noise, and we therefore consider the 0.012 (Pre-Norm) / 0.014 (Sandwich-Norm) advantage over the baseline at 3B to be meaningful. The gap also widens monotonically through the annealing phase under both normalizations and on the MTP objective (Fig. 4). For reference, Frac-Connection's Fig. 4 reports a training-loss improvement of ≈0.014 over its baseline at the 1B scale. Fig. 5(a) further shows that combining ResLorB with FC improves training loss by 0.012 over FC, indicating that the combination is non-degenerate.

2.2 "Each method is not isoFLOP adjusted ... FC uses fewer parameters yet still performs very similarly to ResLorB. In the 3B model case, it is still the case that ResLorB has more parameters than FC, which again makes their direct comparison difficult." "... all margins are fairly small ... Given the minor differences in loss and downstream performance ..."

We do not match FLOPs exactly: the methods differ in architecture, so the same parameter count does not imply the same capacity or compute. We tuned the comparison methods as well as we could. For FC, we report the implementation we use in Table 6, tuned so that the baseline is not under-provisioned; we also compare FC at 4 and 8 streams (Fig. 8) and report the 4-stream version, which has the better performance-to-parameter ratio. On the LLaMA models we have added three seeds for every method, as described in 2.1.

On the comparison between FC and ResLorB: as noted in 2.1, Fig. 5(a) shows that FC and ResLorB can be combined, and that combining them improves training loss by 0.012 over FC, so the gain is non-degenerate. ResLorB alone is ahead of FC at 3B / 200B tokens by ≈7e-3 under Pre-Norm and ≈1.4e-2 under Sandwich-Norm, and the gap widens through the annealing phase. The additional comparisons make a parameter-count-only explanation less likely: FC does not gain from more parameters (Fig. 8), and a branch restricted to FC's interaction groups at matched parameter count recovers only ≈2e-3 of the ≈1.4e-2 gain under Sandwich-Norm (Fig. 18; details in our response to Reviewer He9J, Q2). On downstream performance, the 3B average improves by 1.03 points on a 39.8 baseline, and the 9B ten-benchmark average improves by 1.4 points, with gains on 8 of 10 benchmarks (Table 8); for context, Frac-Connection reports ≈0.7% relative downstream improvement in its Table 4. Overall, we consider the improvement of ResLorB to be meaningful and to come from the method itself.


Questions for Authors

Q1. "How many tokens are each of the models trained for?"

As reported in Table 5, the LLaMA 60M / 130M / 350M models are trained on 1.4B / 2.6B / 7.8B tokens, i.e. ≈23 / 20 / 22 tokens per parameter, following Chinchilla; the 3B model is trained on 200B tokens and the 9B model on 450B tokens.

Q2. "Are there FC results for Table 2 to accompany the comparison present in Figure 3?"

Table 2 reports downstream results for ResLorB and the baseline only. The downstream comparison for the architectures in Fig. 3 (Fig. 4 in the revised version) is reported in Table 7, which evaluates FC, FC-ResLorB, and FC-ResLorB+ under Pre-Norm: the FC model's downstream average is 40.08, compared with 39.83 for the baseline and 40.86 for ResLorB. We did not run a downstream evaluation of FC alone under Sandwich-Norm; FC's training-loss gain is smaller there, and the FC combination is therefore studied under Pre-Norm, where its gain is larger (Fig. 4).

Reviewer_HzaHOpenReview ↗

Response to reviewers

Thank you for your thorough response in clarifying and responding to the review. I appreciate the clarification regarding the stance on what the claim was in terms of "not claiming lower rank is better," which helped clarify my understanding of the experiment varying rank. I also appreciate the updated figure 1 with the key residual connection and the clarification on what was implemented throughout the paper; the justification of why figure 3 appears as it is is additionally helpful with the updated caption.

My concern regarding whether the benefit of ResLorB lies from actual architectual efficiency rather than increased compute still remains, which is also tied to the additional listed concern of the small margins of performance increase present across methods. The existing analysis including seed experiment updates (thank you for addressing this), still do not convincingly show that the benefit from ResLorB is indeed architectural efficiency; the combination of small marginal performance increases and small parameterization overhead does not disentangle thoroughly where the performance is coming from, as two things are varying (total params, and architecture). Therefore, I will increase my score by one point as some of the weak original score came from what were presentation issues.

We thank the reviewer for carefully considering our rebuttal, acknowledging the additional experiments and clarifications, and increasing the score. We understand that some concerns still remain, and we would like to briefly clarify our perspective on these points.

  1. Whether the benefits of ResLorB stem from actual architectual efficiency rather than increased compute. We believe that comparisons with gating-based methods provide the most direct way to examine this question, as they introduce similar additional total and activated parameters. In Fig. 18 and Fig. 20, we have compared ResLorB with both Gated ResLorB and Gated Residual, and ResLorB consistently achieves better performance. We would also like to provide a useful reference from Gated Attention [1](Best Paper), which reports a ppl reduction of 0.095 (≈\approx 0.013 loss reduction) for SDPA elementwise over its baseline (Table 2, 28 layers, 1.7B parameters, most similar to our setting).

  2. On the magnitude of the performance improvements. We believe that the improvement is meaningful: the 3B ResLorB models achieve training loss reductions of 0.014 with Sandwich-Norm and 0.011 with Pre-Norm over the baseline, and ResLorB + FC improves training loss by 0.012 over FC. As a reference, the original FC paper reports an improvement of approximately 0.014 over its baseline (Fig. 4), and our baseline includes more adavanced components such as mHC/MLA/MTP.

  3. On evaluating architectural improvements. More broadly, we believe that architectural comparisons should account for trade-offs across multiple dimensions. For example, a concurrent work [2] replaces mHC's read/write operations with linear up/down projections. While this design uses more parameters and achieves comparable performance, it offers better throughput and simpler implementation. This illustrates that a modest increase in parameter count should not be the sole criterion for evaluating an architectural improvement, as other trade-offs, such as throughput and implementation simplicity, also matter.

We hope these clarifications together with the additional experiments in our rebuttal can provide a more complete picture of ResLorB's architectural contribution. We sincerely appreciate the reviewer's time and consideration.


[1] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [2] Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.