acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Mamba Gate Projection

Abstract

We study whether Mamba's independent gate projection is necessary, or whether its routing signal can instead be derived from a representation the block has already computed. We propose Content-Derived Gating (CDG): a low-rank map applied to Mamba's content projection in place of the full gate. CDG substantially reduces gate parameters while preserving performance below flagship (130M-parameter) scale, where it trains 19% faster and uses 23% less peak memory than the full gate but incurs a persistent 2.6% relative bpb gap. We then show this behavior is not explained by expressivity: CDG and a matched content-blind low-rank gate parameterize exactly the same set of reachable gate functions whenever the content projection has full rank, as it does throughout this study (Proposition prop:reachability). Yet the two behave differently in practice, and the direction reverses with scale: comparable or suggestively better for content-derivation at small scale, but at 130M the content-blind gate improves bpb by over CDG despite fewer parameters, closing more than half its gap to the full gate. Any derived-routing scheme's gains should therefore be checked against a content-blind low-rank baseline of matched size before being credited to the derivation, since that credit can reverse with scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Content-Derived Gating (CDG), Mamba, state space models, parameter efficiency, low-rank gating, model compression, sequential MNIST, scaling behavior, gating mechanisms, architectural ablations, efficient architectures, neural network design, representation routing, flagship-scale models, performance tradeoffs.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QQivOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether Mamba requires an independent full gate projection. It proposes Content-Derived Gating (CDG), which computes the gate through a low-rank transformation of the existing content projection, and compares it with an independent low-rank gate. The authors establish that these parameterizations represent the same gate functions at matched rank when the content projection has full column rank.

Experiments cover synthetic memory tasks, sequential MNIST, and a byte-level language model trained on SlimPajama, with approximately 130M parameters in the full-gate baseline. CDG achieves substantial compression with little observed degradation in the smaller settings. At the largest tested configuration, however, the independent low-rank gate achieves better validation bits per byte (bpb) than CDG. The paper interprets the changed ranking across settings as a scale-dependent effect and recommends careful evaluation when adopting content-derived low-rank gating.

Critical strengths

The independent low-rank baseline provides a useful and consequential control. I agree that the comparing CDG with the full gate alone would not distinguish the benefits of compression from those of sharing the content representation. The independent low-rank baseline directly addresses this distinction. At the largest tested configuration, CDG underperforms over tge naive low-rank, although both remain worse than the full gate. This is a useful negative finding for the use of low-rank projection in the Mamba's gate, and I think the paper’s core strength is its comparison between content-derived and independent low-rank parameterizations with equivalent reachable gate functions.

Critical weaknesses

The experimental design does not isolate model scale as the cause of the reported reversal. The comparison shifts from synthetic tasks and sequential MNIST to language modeling, while also varying width, depth, expansion ratio, training regime, and implementation. There is no model-size sweep within the same language-modeling task. The author calls this result "suggestive" in Section 6.6, but the broader framing and the recommendation to prefer CDG below the largest scale go beyond that evidence. The results establish different rankings across experimental settings; they do not yet establish a scale-dependent crossover. This distinction is consequential because the reversal is central to the paper’s claimed contribution.

The proposed mechanism is not substantiated by the reported results, leaving the broader scientific contribution unclear. The authors hypothesize that coupling the content and gating gradients helps in some regimes and creates conflicting optimization pressures in others. Proposition 1 rules out a difference in reachable gate functions under its assumptions, but I could not find any results to establish this explanation. Sections 4.4–4.5 describe subspace-alignment experiments and an overlap analysis, yet I could not locate their numerical results or empirical conclusions in the supplied manuscript, despite Section 7 stating that they directly test the explanation. Without these results or a controlled scaling study, the strongest supported contribution is the configuration-specific low-rank baseline result. The broader recommendation to validate architectural changes at their intended scale is reasonable and familiar, but does not, by itself, establish a sufficiently specific new insight.

Questions for authors

Quality–efficiency tradeoff The low-rank variant (either the CDG or naive low-rank) reduces its computational cost compared to the full-rank variant. Are there any comparisons in the matched computation or matched quality settings to discuss the positive or negative results by using the low-rank projection in the gating? (What is the "lesson" in this respect? Should we use the low-rank or not for the larger models?)

Mechanistic evidence: Where are the results of the subspace-alignment experiments and overlap analysis described in Sections 4.4–4.5? What do they establish about the proposed coupling of content and gating gradients? In particular, how do the analyses distinguish gradient coupling from other effects of parameterization, initialization, or optimization?

3 comments

We thank the reviewer for a precise review. We agree with the central criticism: the results establish different rankings across experimental settings, not a scale-dependent crossover, and our mechanism section did not report the results it promised. Below we answer both weaknesses and both questions directly, including what the new evidence does and does not show.

W1 — Scale is not isolated; the recommendation to prefer CDG below the largest scale goes beyond the evidence

The reviewer is right on both counts. Moving from the synthetic tasks and sMNIST to the language model changes task, width, depth, expansion, budget and implementation together, and there was no model-size sweep on the language-modeling task.

  • Recommendation corrected. The paper previously recommended content-derived gating below flagship scale. That is not supported: on the synthetic and sMNIST tasks CDG is on par with the plain low-rank gate (apart from one n=3 sMNIST cell), and on the language model it is worse. The paper now recommends the plain low-rank gate and no longer recommends CDG at any scale.
  • "Reversal / scale-dependent" framing removed. The abstract, introduction and Discussion now state only what the data show: no reliable difference on the synthetic and sMNIST tasks, a clear deficit for CDG at 130M, and that model size is not isolated as the cause.
  • Model-size comparison on the same task. We added a ~29M-parameter SlimPajama byte-LM comparison (free, CDG, plain low-rank gate, stop-gradient CDG), 3 seeds, at the same relative gate compression as the 130M setting (gate cost 25% of free's), with the same data, optimizer and block implementation. Width and depth change together with size within this family. It uses a 1000-step budget with our pure-PyTorch scan, so its absolute bpb is not comparable to the 10k-step real-kernel table, as the paper states. Result: the ordering matches 130M. Free 2.3106 ± 0.0032 < plain low-rank gate 2.3478 ± 0.0072 < CDG 2.3774 ± 0.0095 validation bpb. CDG is 1.26% worse than the low-rank gate (z = +4.32) and worse in all three seed-paired comparisons (+0.029, +0.019, +0.041 bpb). So on the language task the CDG deficit does not change sign between 29M and 130M. This is two sizes, not a scaling law, and the much shorter 29M budget means the conclusion is suggestive rather than conclusive.

W2 / Question 2 — Mechanism unsubstantiated; where are the Sec. 4.4–4.5 results?

This is a defect in the submitted manuscript, and we apologize: Section 7 said the explanation was "tested directly" without reporting results. What we now report (Sec. 6.7):

  1. Subspace-Alignment Gate (Sec. 4.4). At 130M (rank 128, 1000 steps, single seed): aligned 2.2948, random 2.2878, orthogonal 2.2755 validation bpb, against free 2.2818 and CDG 2.3269 (reference runs with a different batch order). There is no support for an alignment effect: aligning the gate's frozen reading subspace with the content matrix's dominant directions does not help (if anything the ordering is reversed, but the 0.85% spread is single-seed and short-budget, so we claim only the absence of support).
  2. This experiment cannot test gradient coupling, and the paper wrongly said it did. Its down-projection is frozen, so nothing is shared. We corrected this: the coupling explanation is now labeled an untested conjecture in Sec. 4.1 and the Discussion.
  3. Overlap analysis (old Sec. 4.5) — removed. The required sMNIST retraining did not complete (the T=784 scan did not finish one epoch in 38 minutes on a shared GPU). It was descriptive only; we deleted it rather than report an unfinished analysis. (Sec. 4.5 in the revision is the stop-gradient experiment below.)
  1. A direct test of coupling, which also answers how to separate coupling from parameterization, initialization and optimization. Stop-gradient CDG computes g = UVᵀ·sg(u): same parameters and forward pass as CDG, but the gate loss sends no gradient into the content projection W_u. For a given seed, CDG and stop-gradient CDG have bit-identical initial weights (all 148 tensors) and see the identical sequence of training batches, so they differ only in the backward pass. That removes parameterization, initialization and data order as explanations of any difference between them. What it does not remove: interactions with the optimizer (Adam rescales per-parameter gradients, so removing a gradient component is not a pure ablation of its influence), and it tests one specific form of coupling (gradient into W_u), not every way the tie could matter.
    • 29M, 3 seeds: stop-gradient CDG is 0.92% worse than CDG (2.3995 ± 0.0110 vs. 2.3774 ± 0.0095; z = −2.63; seed-paired difference 0.022 ± 0.017 bpb, same sign in all three seeds, one of them only 0.005), not better.
    • 130M, one matched pair: 2.3472 vs. 2.3622 (0.64% better), the opposite direction. This is one seed, the validation curves cross (worse at steps 500 and 750, better only at 1000), and the seed-matched CDG run differs from our earlier CDG reference run (2.3269, different batch order) by more than twice this difference, so this pair does not separate the two.
    • Reading: the coupling conjecture is not supported in this form. We conclude that this specific mechanism is not supported, not that the cause of CDG's deficit is known.

The paper's strongest supported contributions are therefore, as the reviewer says, the matched low-rank control and the reachability-equivalence proposition, and the paper now describes them that way.

Question 1 — Quality–efficiency tradeoff: matched computation or matched quality; should larger models use low-rank gating?

We have no matched-compute comparison (for example, reinvesting the gate savings into width or depth), and we do not claim one. We can give a matched-quality wall-clock comparison, now also in the paper (real-kernel results subsection), from the 3-seed real-kernel logs at 130M. The runs use constant-learning-rate Adam, so the full gate at an earlier step is a fair stand-in for a full gate trained for less; the curves are mean validation bpb over 3 seeds.

modelval bpb @10kstep timepeak memio paramswall-clock for free to reach this quality
free1.89390.366 s2.70 GB125.8M—
plain low-rank gate (r=128)1.91580.292 s2.02 GB91.8Mfree needs ≈3.4–3.5k s vs. 2.9k s for the low-rank gate: the low-rank gate reaches it ~13–17% sooner
CDG (r=128)1.94310.304 s2.07 GB94.4Mfree needs ≈2.6–3.2k s vs. 3.0k s for CDG: no clear wall-clock advantage (−17% to +5%)

The ranges bracket two estimators (the first step at which the mean free curve reaches the target, and the step after which it stays below it), because the validation curve is noisy; the step time is a single measured value. Peak memory (−25% for the low-rank gate) is not captured by the wall-clock column. So for this model and 10k-step budget, the plain low-rank gate gives a modest efficiency gain at a 1.2% quality cost, and CDG gives none beyond what the low-rank gate gives. Our answer to "should larger models use low-rank gating?" is therefore conditional: if memory and throughput matter more than about 1% quality, the plain low-rank gate is a reasonable choice at the scale we tested and CDG is not. We have not tested whether this holds at larger scale or at convergence, and the paper states that explicitly.

Presentation and informativeness

We accept that the manuscript was hard to follow. Beyond the changes above, we removed references to an unpublished "sibling study", replaced the mix of "gap/SE", "significance bar" and "large standardized difference" with a single term and symbol (z), and removed claims the evidence did not support. We did not restructure the results section in this revision; the extra page was used for the new mechanism and size results (Sec. 6.7), and we tightened the mechanism section and Discussion so the main text ends on page 10.

On informativeness, we agree that "validate at the intended scale" is familiar advice. The specific, new content is (i) that the matched content-blind control changes the conclusion at 130M, (ii) the proof that the two parameterizations reach identical gate functions, and (iii) controlled mechanism tests whose outcomes we now report, including negative ones.

Thank you for the detailed response. The rebuttal and revision have resolved some of my major concerns, i.e., the unsupported claims and recommendations (such as the claim that CDG should be used for the smaller network). I appreciate that the arguments and recommendations in the revised manuscript have been narrowed to those supported by experimental evidence ("no longer recommends CDG at any scale"). However, this would remove one of the paper's main claims (if I understand correctly), and the main claim would be changed. And another important implication in the original manuscript (larger models should use low-rank gating) remains unsupported by the experimental result. I understand the new claim might also have important implications for Mamba's gating design; however, this also needs to be verified through experiments designed for this purpose. Considering these points, I will keep my original rating of 2.

Reviewer_Jks7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Content-Derived Gating (CDG) to replace Mamba’s independent gate projection with a low-rank function.

Critical strengths
  1. Mamba uses a substantial number of parameters for the independent gate projection. Investigating whether those parameters are necessary is relevant to efficient state-space models.

  2. Proposition 1 shows that CDG and a matched-rank independent low-rank gate have equivalent reachable gate functions under the full-rank assumption.

Critical weaknesses

1.The paper provides limited empirical evidence to support its claim. The largest model evaluated is approximately 130M parameters, which is relatively small for drawing conclusions about modern large-scale language models. The paper argues that the relative merits of content-derived and independent low-rank gating change with scale. But the evidence is based on one large-scale operating point.

  1. CDG underperforms the cheaper independent low-rank gate. As shown in Table 4, the independent low-rank gate is better than CDG on all three practical dimensions: model size, training speed, and language-modeling quality, which raises a question about the significance of CDG as a proposed method.
Questions for authors

Could you evaluate Larger Mamba models, ideally at least several hundred million to 1B parameters?

2 comments

We thank the reviewer for the clear statement of the two concerns. Both are fair, and we respond to each, then to the presentation score.

W1 — Evidence rests on ~130M parameters; please evaluate several hundred M to 1B

We agree that our evidence is limited and that nothing here licenses conclusions about models of several hundred million to a billion parameters. We cannot add those runs in this rebuttal window: our compute is a shared pool of GPUs, and a single 130M training run with our pure-PyTorch scan takes roughly 7 hours there, so matched multi-seed comparisons at 300M–1B parameters are out of reach for this response. We would rather say so than report an underpowered single run. What we did instead:

  1. A second operating point on the same task and recipe. We added a ~29M-parameter SlimPajama byte-LM comparison (free, CDG, plain low-rank gate, and a stop-gradient CDG variant), 3 seeds, at the same relative compression as the 130M setting (gate cost 25% of free's), so model size is the main factor that differs. These are 1000-step, custom-scan runs, so their absolute bpb is not comparable to the 10k-step real-kernel table, as the paper states. The ordering matches 130M: free 2.3106 ± 0.0032 < plain low-rank gate 2.3478 ± 0.0072 < CDG 2.3774 ± 0.0095 validation bpb. CDG is 1.26% worse than the low-rank gate (z = +4.32) and worse in all three seed-paired comparisons (+0.029, +0.019, +0.041 bpb). This is two sizes, not a scaling law.
  2. Weaker claims. We removed the claim that the merits of the two gates "reverse with scale". The paper now states what the data support: no reliable difference on the synthetic and sMNIST tasks, and a clear deficit for CDG against the content-blind gate on the language model at 130M, reproduced at 29M, with model size not isolated as the cause.
  3. Explicit limitation. The paper now states that its largest model is ~130M parameters and that conclusions beyond that are untested.

The paper's central message does not depend on extrapolating to larger models: any derived-routing scheme should be compared against a content-blind low-rank control of matched size, which the 130M result shows can change the conclusion.

W2 — The independent low-rank gate beats CDG on size, speed and quality

The reviewer's reading of Table 4 is correct. At 130M on real kernels (3 seeds), the plain low-rank gate vs. CDG: 91.75M vs. 94.37M io parameters (−2.8%), 0.292 vs. 0.304 s/step (−4%), 2.02 vs. 2.07 GB peak memory, and validation bpb 1.9158 vs. 1.9431 (1.4% better, z = 7.62). The size and speed margins are small (about 3–4%); the quality margin is the clear one.

We also added a matched-quality wall-clock comparison to the paper, from the 3-seed real-kernel logs (constant-learning-rate Adam, so the full gate at an earlier step stands in for a shorter run). The full gate needs about 3.4–3.5k s to reach the low-rank gate's final quality, versus 2.9k s for the low-rank gate, which reaches it 13–17% sooner. CDG shows no clear wall-clock advantage over the full gate (−17% to +5%). These rest on single step-time measurements and noisy validation curves, and we did not run a matched-compute comparison.

We agree with the consequence: at 130M, CDG is dominated, and we do not recommend it. CDG was not intended as the paper's contribution, but the original text did not make that clear and even recommended content-derived gating below flagship scale, which our data do not support (on the synthetic and sMNIST tasks CDG is merely on par with the low-rank gate, apart from one n=3 sMNIST cell). The abstract, introduction and Discussion now state that the practical recommendation is the plain low-rank gate, and that the contributions are (i) the matched content-blind control and the finding that it changes the conclusion, (ii) the proof that the two gates reach identical gate functions, so any difference is an optimization or inductive-bias effect, and (iii) controlled tests of two candidate mechanisms. Those tests found no support for either: aligning the gate's frozen subspace did not help (130M, one seed), and stop-gradient CDG did not close the deficit at 29M (0.92% worse than CDG, 3 seeds), while the single 130M pair did not separate the two (Sec. 6.7 of the paper).

Presentation (score 1)

The review does not say which parts were hard to follow, so we addressed the problems we could identify. The text referred to an unpublished "sibling study" and to experiments whose results were not reported (both removed or fixed). The mix of "gap/SE", "significance bar" and "large standardized difference" is replaced by one term and symbol (z). Claims that went beyond the evidence (scale reversal, "tested directly") were removed. We did not restructure the results section in this revision. If the reviewer can point to the sections that were hardest to follow, we will prioritize them in the final version.

Thank you for the response. It is understandable that the author cannot provide evaluation for larger models. Regarding W2, I appreciate the authors’ clarification that CDG is not intended to be the main contribution. Nevertheless, the independent low-rank gate consistently outperforms CDG in the language-modeling experiments while requiring fewer parameters and less computation. Although the matched low-rank comparison and theoretical equivalence result are interesting, I remain unconvinced that the findings constitute a sufficiently substantial contribution. Overall, the rebuttal addresses some of my concerns, particularly regarding the interpretation of the scaling results. However, my main concerns about the limited experimental scope and significance of the contribution remain. Therefore, I will maintain my original rating of 2.

Reviewer_sAn5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether Mamba's independent gate projection can be replaced by Content-Derived Gating (CDG), a low-rank transformation of the content representation. It compares CDG against residual and shared-basis variants and an independent low-rank gate across synthetic memory tasks, sequential MNIST, and a 130M-parameter language model trained on SlimPajama. Proposition 1 establishes that CDG and the independent low-rank gate have equivalent representational capacity under a full-rank content projection. CDG matches the full gate on smaller tasks but incurs a 2.6% relative bpb degradation at 130M parameters, where the independent low-rank gate outperforms it by 1.4%. The authors conclude that low-rank compression, rather than content derivation, explains most of the benefits and emphasize the importance of including an independent low-rank baseline.

Critical strengths
  • Clear and well-motivated research question: The paper investigates whether Mamba's independent gate projection is necessary or whether its functionality can be preserved through a more parameter-efficient design. The question is focused and relevant to efficient state-space models.

  • Useful theoretical analysis: Proposition 1 establishes that CDG and the independent low-rank gate can represent the same set of gate functions under the full-rank assumption on the content projection. This helps rule out differences in reachable gate-function capacity as the explanation for their observed performance differences.

  • Transparent reporting of negative results: The authors openly report that CDG underperforms both the full gate and the independent low-rank alternative at the largest evaluated scale. They also discuss unsuccessful architectural modifications rather than presenting only favorable results.

  • Real-kernel validation and efficiency measurements: The flagship experiments are repeated using actual Mamba CUDA kernels, with numerical parity checks and three random seeds. The inclusion of measured training time and peak memory usage provides practical evidence beyond parameter-count reductions.

  • Meaningful independent low-rank control: The inclusion of a content-blind low-rank gate provides an important comparison that distinguishes the benefits of low-rank compression from those of content-derived gating. This strengthens the paper's methodological contribution and offers a useful recommendation for evaluating similar architectures.

Critical weaknesses
  • Limited advantage of the proposed method: At the largest evaluated scale (130M parameters), CDG underperforms both the full gate (2.6% relative bpb degradation) and the simpler independent low-rank gate, which also uses fewer parameters. While the latter is an informative finding, it limits the practical contribution of CDG itself, and the broader conclusion that gating can be low-rank is closely related to established low-rank parameterization techniques.

  • Sections 4.4 and 4.5 introduce a Subspace-Alignment Gate and an overlap analysis to test whether coupling the gate to the content projection affects optimization. Section 4.1 states that Section 7 “returns to its results,” and Section 7 describes the explanation as “tested directly” by these experiments, yet no results appear in the main text or appendix. The optimization-based explanation, central to the paper’s interpretation, is therefore unverified.

  • Insufficient evidence for a scale-dependent reversal: The reported advantage of content-derived gating at smaller scales is primarily supported by a single sequential MNIST configuration (r = 8, three seeds), which the authors themselves characterize as suggestive. Most other small-scale comparisons show no clear difference. Furthermore, the transition to the 130M experiment changes the task, model architecture, and training conditions, making it difficult to attribute the observed reversal specifically to model scale.

  • Presentation and organization: The manuscript is unnecessarily dense and repetitive in places. References to a separate, unpublished "sibling study" make some methodological motivations difficult to assess independently. The discussion of standardized differences would also benefit from more consistent terminology.

Questions for authors

Refer to Critical weaknesses

2 comments

We thank the reviewer for a careful reading. The reviewer's soundness assessment and listed strengths match our own view of the paper's contribution: a matched content-blind control, a reachability-equivalence result, and real-kernel multi-seed evidence. The weaknesses identified are about completeness of reporting and strength of wording, and the most important of them (W2) is correct. We address each below.

W2 — Sections 4.4/4.5 promise results that do not appear

This is a genuine defect in the submitted manuscript, and we apologize for it. Section 4.1 said the coupling hypothesis was tested by Section 4.4 and that Section 7 "returns to its results", and Section 7 said "tested directly", yet neither the main text nor the appendix reported the outcome. Specifically:

  1. Subspace-Alignment Gate (Sec. 4.4). The experiment had been run but was omitted. At the 130M configuration (custom scan, rank 128, 1000 steps, single seed), validation bpb is:

    Fixed down-projection QVal bpbvs. free (2.2818)vs. CDG (2.3269)
    aligned (top-128 singular vectors of W_u)2.2948+0.57%−1.38%
    random2.2878+0.26%−1.68%
    orthogonal (bottom-128)2.2755−0.28%−2.21%

    There is no evidence that aligning the gate's reading subspace with W_u's dominant directions helps. If anything the ordering is reversed (orthogonal best, aligned worst), but the 0.85% spread comes from one seed at a short budget, so we claim only the absence of support for an alignment effect, not a reversal. All three gates are within 0.6% of free and ahead of CDG at this budget. These results are now in the paper (Sec. 6.7).

  2. Correction to our own framing. The Subspace-Alignment Gate freezes the down-projection, so it tests alignment, not the gradient-coupling mechanism we conjectured in Sec. 4.1 (where W_u receives gradient from both the content and gate paths). We wrongly wrote that it tests coupling. In the revision, coupling is stated as an untested conjecture in Sec. 4.1, the alignment result is reported as speaking only to alignment, and "tested directly" is removed.

  3. Overlap analysis (old Sec. 4.5). The analysis was specified but never completed: the required sMNIST retraining did not finish one epoch in 38 minutes on a shared GPU. Rather than report an unfinished, descriptive analysis, we deleted it and all references to it. (Sec. 4.5 in the revision is the new stop-gradient experiment below.)

  4. Direct test of the coupling conjecture. We added a stop-gradient CDG (g = UVᵀ·sg(u)), which has the same parameters and forward pass as CDG but sends no gate-path gradient into W_u. For a given seed, it and CDG have bit-identical initial weights and see identical training batches, so they differ only in the backward pass. Results (custom scan, 1000 steps):

    • 29M, 3 seeds: stop-gradient CDG is 0.92% worse than CDG (2.3995 ± 0.0110 vs. 2.3774 ± 0.0095; z = −2.63; seed-paired difference 0.022 ± 0.017 bpb, same sign in all three seeds, one of them only 0.005), not better.
    • 130M, one matched pair: 2.3472 vs. 2.3622 (0.64% better), the opposite direction. This is a single seed, the validation curves cross (worse at steps 500 and 750, better only at 1000), and the seed-matched CDG run differs from our earlier CDG reference run (2.3269, different batch order) by more than twice this difference, so this pair does not separate the two.
    • Reading: the coupling conjecture is not supported in this form. Two limits apply: Adam rescales gradients per parameter, so removing a gradient component is not a pure ablation of its influence, and this tests only gradient into W_u, not every way the tie could matter. We conclude that this specific mechanism is not supported, not that the cause of CDG's deficit is known.

W3 — The "scale-dependent reversal" is thin and confounded

We agree on both counts. The small-scale evidence for content-derivation rests on one sMNIST cell (r=8, n=3, standardized difference −2.18, with no correction for the many cells swept), which the paper already called suggestive; the other small-scale cells show no clear difference. And going from sMNIST to the 130M language model changes task, depth, width, budget and kernels at once, so scale is not isolated.

  • Wording. We removed "the direction reverses with scale" from the abstract and introduction. The paper now states what the data support: content-derivation shows no reliable benefit on the synthetic and sMNIST tasks, and a clear deficit relative to the content-blind gate on the language model at 130M, with model size not isolated as the cause.
  • Same-task size comparison. We ran the language-model comparison at a second size (~29M parameters; SlimPajama byte-LM, 1000 steps, 3 seeds, gate at the same 25% relative cost as at 130M, custom scan). The ordering matches 130M: free 2.3106 ± 0.0032 < plain low-rank gate 2.3478 ± 0.0072 < CDG 2.3774 ± 0.0095 validation bpb. CDG is 1.26% worse than the low-rank gate (z = +4.32) and worse in all three seed-paired comparisons (+0.029, +0.019, +0.041 bpb). So on this task the CDG deficit does not change sign between 29M and 130M. This is two sizes, not a scaling law, and the 29M budget is much shorter than the 130M real-kernel runs, so the earlier small-scale parity is best read as specific to the synthetic and sMNIST tasks.
  • Not done. We did not extend the sMNIST r=8 cell beyond n=3 seeds (compute-limited); the paper states this limitation.

What the 130M result establishes regardless of the scale question: at n=3 on real kernels, the content-blind gate beats CDG by a standardized difference of +7.62 (1.40% relative) with a smaller router. This control result does not depend on the scale interpretation.

W1 — Limited advantage of CDG; relation to known low-rank techniques

We agree, and the paper does not claim CDG as its contribution: Sec. 1 states that the strongest conclusion is H2 (the gate signal can be low-rank), not H3 (it must be content-derived), and we do not claim novelty for low-rank parameterization. The contribution is (i) that a published pattern (deriving the routing signal from an existing representation) is confounded with compression unless a matched content-blind control is run, (ii) a proof that the two parameterizations reach the same set of gate functions, so any difference is an optimization or inductive-bias effect, and (iii) a practical result: the content-blind low-rank gate gives ~27% io-parameter savings, ~20% faster steps (0.292 vs. 0.366 s/step) and ~25% lower peak memory at a 1.16% bpb cost. The abstract and Discussion now state the recommendation as a plain low-rank gate rather than CDG, and recommend checking any derived-routing scheme against a matched content-blind baseline. We kept the title.

W4 — Presentation, "sibling study", terminology

  • Sibling study and baselines. References to a "sibling study" are removed. The one fact used from it (a hard gate-to-content tie is invisible at one block but costly when stacked) is now stated with numbers from this paper's own tables. The free and tie_ssm_gate baselines come from an earlier run of the same code and experiment grid; the paper says so plainly, and their per-seed outputs are included in the code and result release.
  • Terminology. One term and symbol throughout: standardized difference, z (replacing "gap/SE" and "significance bar"). The paper still states that z is not a hypothesis test.
  • Structure. We did not restructure Section 6 in this revision; we removed the places that asserted untested mechanisms and replaced them with the results above.
  • Minor. "Four scales" in the introduction referred to four stages (one language-model scale); this is corrected.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

We thank all three reviewers. Their central criticisms overlapped, and we agree with most of them: the mechanism section promised results it did not report, "scale-dependent reversal" was not supported because scale was confounded with task, and the recommendation to prefer content-derived gating (CDG) below flagship scale went beyond the evidence. The revised paper keeps its main-text length within 10 pages. Individual replies to each reviewer are posted separately; this is the shared recap.

What changed in the conclusions

  • Recommendation reversed. The paper now recommends a plain low-rank gate, not CDG. CDG showed no reliable benefit at any scale we tested, apart from one suggestive n=3 sMNIST cell, and is worse than the plain low-rank gate on the language model.
  • "Reversal with scale" removed from the abstract, introduction and Discussion. The paper now says: no reliable difference on the synthetic and sMNIST tasks, a clear deficit for CDG on the language model at 130M, and that model size is not isolated as the cause.
  • Mechanism claims weakened. Gradient coupling is stated as an untested conjecture where it was introduced, and "tested directly" is removed. We also corrected our own error: the Subspace-Alignment Gate freezes the down-projection, so it tests alignment, not coupling.
  • Limitations added. The largest model is ~130M parameters, and nothing is claimed beyond that. The 29M comparison is short-budget.

New experiments

All use the SlimPajama byte-LM, a 1000-step budget and our pure-PyTorch scan, so absolute bpb is not comparable to the 10k-step real-kernel tables. Same-seed runs see identical training batches across modes.

ExperimentResult
29M same-task size comparison, 3 seeds, gate at the same 25% relative cost as at 130Mfree 2.3106 ± 0.0032 < plain low-rank gate 2.3478 ± 0.0072 < CDG 2.3774 ± 0.0095. CDG is 1.26% worse than the low-rank gate (z = +4.32), in all three paired seeds. Same ordering as 130M, so the CDG deficit does not change sign between the two sizes on this task.
Stop-gradient CDG (same parameters and forward pass as CDG, no gate-path gradient into W_u; bit-identical initialization and batches), 29M, 3 seeds2.3995 ± 0.0110, which is 0.92% worse than CDG (z = −2.63). The gradient-coupling conjecture is not supported.
Stop-gradient CDG, 130M, one matched pair2.3472 vs. 2.3622 for CDG (0.64% better), but one seed, crossing curves, and a seed-matched CDG run that differs from our earlier reference run by more than twice this difference. It does not separate the two.
Subspace-Alignment Gate (previously run but unreported), 130M, one seedaligned 2.2948, random 2.2878, orthogonal 2.2755. No support for an alignment effect.
Matched-quality wall-clock (from existing 3-seed real-kernel logs)The full gate needs ~3.4–3.5k s to reach the low-rank gate's final quality vs. 2.9k s for the low-rank gate (13–17% sooner). CDG has no clear wall-clock advantage over the full gate (−17% to +5%).

None of these results supports an explanation for why CDG trails the content-blind gate. We ruled out two specific mechanisms (alignment, and gate-path gradient into W_u) and state that the cause remains open.

Removed or not done

  • Overlap analysis (old Sec. 4.5) removed. The required sMNIST retraining did not complete in the available time, and the analysis was descriptive only.
  • No models above ~130M. Our shared GPU pool could not support matched multi-seed runs at 300M–1B within the rebuttal window, and we did not run a single underpowered run.
  • No matched-compute comparison (reinvesting gate savings into width or depth).
  • sMNIST r=8 not extended beyond n=3 seeds.
  • Results section not restructured. We tightened the mechanism section and Discussion to fit the page limit.

Presentation fixes

  • References to an unpublished "sibling study" removed; the one fact used from it is now stated with numbers from this paper's own tables.
  • One term and symbol for the comparison statistic: standardized difference, z (replacing "gap/SE" and "significance bar"); the paper still states that z is not a hypothesis test.
  • The baselines taken from an earlier run of the same code and grid are described as such, and their per-seed outputs are included in the code and result release.
  • "Four scales" in the introduction corrected to four stages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.