- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether Mamba requires an independent full gate projection. It proposes Content-Derived Gating (CDG), which computes the gate through a low-rank transformation of the existing content projection, and compares it with an independent low-rank gate. The authors establish that these parameterizations represent the same gate functions at matched rank when the content projection has full column rank.
Experiments cover synthetic memory tasks, sequential MNIST, and a byte-level language model trained on SlimPajama, with approximately 130M parameters in the full-gate baseline. CDG achieves substantial compression with little observed degradation in the smaller settings. At the largest tested configuration, however, the independent low-rank gate achieves better validation bits per byte (bpb) than CDG. The paper interprets the changed ranking across settings as a scale-dependent effect and recommends careful evaluation when adopting content-derived low-rank gating.
Critical strengths
The independent low-rank baseline provides a useful and consequential control. I agree that the comparing CDG with the full gate alone would not distinguish the benefits of compression from those of sharing the content representation. The independent low-rank baseline directly addresses this distinction. At the largest tested configuration, CDG underperforms over tge naive low-rank, although both remain worse than the full gate. This is a useful negative finding for the use of low-rank projection in the Mamba's gate, and I think the paper’s core strength is its comparison between content-derived and independent low-rank parameterizations with equivalent reachable gate functions.
Critical weaknesses
The experimental design does not isolate model scale as the cause of the reported reversal. The comparison shifts from synthetic tasks and sequential MNIST to language modeling, while also varying width, depth, expansion ratio, training regime, and implementation. There is no model-size sweep within the same language-modeling task. The author calls this result "suggestive" in Section 6.6, but the broader framing and the recommendation to prefer CDG below the largest scale go beyond that evidence. The results establish different rankings across experimental settings; they do not yet establish a scale-dependent crossover. This distinction is consequential because the reversal is central to the paper’s claimed contribution.
The proposed mechanism is not substantiated by the reported results, leaving the broader scientific contribution unclear. The authors hypothesize that coupling the content and gating gradients helps in some regimes and creates conflicting optimization pressures in others. Proposition 1 rules out a difference in reachable gate functions under its assumptions, but I could not find any results to establish this explanation. Sections 4.4–4.5 describe subspace-alignment experiments and an overlap analysis, yet I could not locate their numerical results or empirical conclusions in the supplied manuscript, despite Section 7 stating that they directly test the explanation. Without these results or a controlled scaling study, the strongest supported contribution is the configuration-specific low-rank baseline result. The broader recommendation to validate architectural changes at their intended scale is reasonable and familiar, but does not, by itself, establish a sufficiently specific new insight.
Questions for authors
Quality–efficiency tradeoff The low-rank variant (either the CDG or naive low-rank) reduces its computational cost compared to the full-rank variant. Are there any comparisons in the matched computation or matched quality settings to discuss the positive or negative results by using the low-rank projection in the gating? (What is the "lesson" in this respect? Should we use the low-rank or not for the larger models?)
Mechanistic evidence: Where are the results of the subspace-alignment experiments and overlap analysis described in Sections 4.4–4.5? What do they establish about the proposed coupling of content and gating gradients? In particular, how do the analyses distinguish gradient coupling from other effects of parameterization, initialization, or optimization?
3 comments
- A direct test of coupling, which also answers how to separate coupling from parameterization, initialization and optimization. Stop-gradient CDG computes g = UVᵀ·sg(u): same parameters and forward pass as CDG, but the gate loss sends no gradient into the content projection W_u. For a given seed, CDG and stop-gradient CDG have bit-identical initial weights (all 148 tensors) and see the identical sequence of training batches, so they differ only in the backward pass. That removes parameterization, initialization and data order as explanations of any difference between them. What it does not remove: interactions with the optimizer (Adam rescales per-parameter gradients, so removing a gradient component is not a pure ablation of its influence), and it tests one specific form of coupling (gradient into W_u), not every way the tie could matter.
- 29M, 3 seeds: stop-gradient CDG is 0.92% worse than CDG (2.3995 ± 0.0110 vs. 2.3774 ± 0.0095; z = −2.63; seed-paired difference 0.022 ± 0.017 bpb, same sign in all three seeds, one of them only 0.005), not better.
- 130M, one matched pair: 2.3472 vs. 2.3622 (0.64% better), the opposite direction. This is one seed, the validation curves cross (worse at steps 500 and 750, better only at 1000), and the seed-matched CDG run differs from our earlier CDG reference run (2.3269, different batch order) by more than twice this difference, so this pair does not separate the two.
- Reading: the coupling conjecture is not supported in this form. We conclude that this specific mechanism is not supported, not that the cause of CDG's deficit is known.
The paper's strongest supported contributions are therefore, as the reviewer says, the matched low-rank control and the reachability-equivalence proposition, and the paper now describes them that way.
Question 1 — Quality–efficiency tradeoff: matched computation or matched quality; should larger models use low-rank gating?
We have no matched-compute comparison (for example, reinvesting the gate savings into width or depth), and we do not claim one. We can give a matched-quality wall-clock comparison, now also in the paper (real-kernel results subsection), from the 3-seed real-kernel logs at 130M. The runs use constant-learning-rate Adam, so the full gate at an earlier step is a fair stand-in for a full gate trained for less; the curves are mean validation bpb over 3 seeds.
| model | val bpb @10k | step time | peak mem | io params | wall-clock for free to reach this quality |
|---|---|---|---|---|---|
| free | 1.8939 | 0.366 s | 2.70 GB | 125.8M | — |
| plain low-rank gate (r=128) | 1.9158 | 0.292 s | 2.02 GB | 91.8M | free needs ≈3.4–3.5k s vs. 2.9k s for the low-rank gate: the low-rank gate reaches it ~13–17% sooner |
| CDG (r=128) | 1.9431 | 0.304 s | 2.07 GB | 94.4M | free needs ≈2.6–3.2k s vs. 3.0k s for CDG: no clear wall-clock advantage (−17% to +5%) |
The ranges bracket two estimators (the first step at which the mean free curve reaches the target, and the step after which it stays below it), because the validation curve is noisy; the step time is a single measured value. Peak memory (−25% for the low-rank gate) is not captured by the wall-clock column. So for this model and 10k-step budget, the plain low-rank gate gives a modest efficiency gain at a 1.2% quality cost, and CDG gives none beyond what the low-rank gate gives. Our answer to "should larger models use low-rank gating?" is therefore conditional: if memory and throughput matter more than about 1% quality, the plain low-rank gate is a reasonable choice at the scale we tested and CDG is not. We have not tested whether this holds at larger scale or at convergence, and the paper states that explicitly.
Presentation and informativeness
We accept that the manuscript was hard to follow. Beyond the changes above, we removed references to an unpublished "sibling study", replaced the mix of "gap/SE", "significance bar" and "large standardized difference" with a single term and symbol (z), and removed claims the evidence did not support. We did not restructure the results section in this revision; the extra page was used for the new mechanism and size results (Sec. 6.7), and we tightened the mechanism section and Discussion so the main text ends on page 10.
On informativeness, we agree that "validate at the intended scale" is familiar advice. The specific, new content is (i) that the matched content-blind control changes the conclusion at 130M, (ii) the proof that the two parameterizations reach identical gate functions, and (iii) controlled mechanism tests whose outcomes we now report, including negative ones.
Thank you for the detailed response. The rebuttal and revision have resolved some of my major concerns, i.e., the unsupported claims and recommendations (such as the claim that CDG should be used for the smaller network). I appreciate that the arguments and recommendations in the revised manuscript have been narrowed to those supported by experimental evidence ("no longer recommends CDG at any scale"). However, this would remove one of the paper's main claims (if I understand correctly), and the main claim would be changed. And another important implication in the original manuscript (larger models should use low-rank gating) remains unsupported by the experimental result. I understand the new claim might also have important implications for Mamba's gating design; however, this also needs to be verified through experiments designed for this purpose. Considering these points, I will keep my original rating of 2.
Response
We thank the reviewer for a precise review. We agree with the central criticism: the results establish different rankings across experimental settings, not a scale-dependent crossover, and our mechanism section did not report the results it promised. Below we answer both weaknesses and both questions directly, including what the new evidence does and does not show.
W1 — Scale is not isolated; the recommendation to prefer CDG below the largest scale goes beyond the evidence
The reviewer is right on both counts. Moving from the synthetic tasks and sMNIST to the language model changes task, width, depth, expansion, budget and implementation together, and there was no model-size sweep on the language-modeling task.
W2 / Question 2 — Mechanism unsubstantiated; where are the Sec. 4.4–4.5 results?
This is a defect in the submitted manuscript, and we apologize: Section 7 said the explanation was "tested directly" without reporting results. What we now report (Sec. 6.7):