acceptodds
Under review as a conference paper at ICLR 2027

DO STATE SPACE MODELS NEED SEPARATE B AND C PROJECTIONS? REVISITING B-C TYING FROM S4D TO MAMBA

Abstract

We study projection redundancy: whether the distinct linear roles a sequence model uses to read and write information can be collapsed onto shared parameters without losing capability. Diagonal state space models (SSMs) such as S4D, S5, and Mamba read and write their hidden state through two matrices, B (input) and C (output); we test tying C = B across four case studies, multi-seeded throughout except where flagged single-seed. For a static SSM (S4D), tying is a clean free lunch — no measurable cost at any depth or task tested, including a byte-level language model — cutting the SSM layer’s parameters by a third. For a selective SSM (Mamba), tying carries a real ∼14% capacity cost, traced to realvalued B,C forcing a tied mode’s contribution to be non-negative (unlike S4D’s complex-valued state); a fixed sign mask fixes this at zero extra parameters, recovering ∼86% of the gap, and a parameter-matched recursive check confirms the fix is a genuine effect rather than fewer parameters in disguise, beating a matchedbudget free model decisively on a real-corpus language model. A plain RNN with a genuine nonlinearity inside the recurrence again costs nothing reproducible, and a systematic study of attention found tying two of three Q,K, V projections costs < 1% relative accuracy (Kayyam et al., 2026). Post hoc, converting an alreadytrained free model to tied is exact for S4D, damaging but fine-tune-recoverable for Mamba, and on a real language model, over 80% recoverable (single seed) using only the frozen model as a synthetic teacher, without the original training data. Together, these four case studies show projection redundancy is architecturedependent in its mechanics, but when tying costs something, the cause is often diagnosable and cheaply fixable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tying read/write projections often cuts parameters with little loss, but the impact depends on architecture, when performance drops, the cause can usually be identified and mostly recovered with simple fixes. Provide your feedback on BizChat

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ncV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The draft recognizes the value of the controlled projection-tying study, the fixed-sign modification, and the post-hoc conversion experiments. Its main reservations concern the relationship to existing diagonal-SSM parameterizations, the scope of the proposed explanation for Mamba, and the strength of the experimental evidence for practical savings and negligible performance loss. The pre-existing weak-rejection rating is retained as a provisional recommendation.

Critical strengths
  1. The paper asks a clear architectural question and investigates it through both algebra and controlled experiments. The distinction between static complex-valued S4D and selective real-valued Mamba is useful, and the fixed-sign modification provides a simple intervention that substantially recovers the synthetic memory-capacity loss. Table 2 reports an increase from 5.872 to 6.688, compared with 6.819 for the untied model, without adding trainable parameters.

  2. The experimental reporting makes a useful effort to distinguish reproducible findings from uncertain ones. The authors extend the Mamba capacity comparison to 30 seeds, report results across several depths, and explicitly discuss single-seed results and reversals under different training budgets. The post-hoc experiments also add practical interest: S4D admits an exact conversion, while Mamba can recover much of the conversion loss through fine-tuning, including a preliminary teacher-only recovery experiment.

Critical weaknesses
  1. The contribution beyond existing diagonal-SSM parameterizations needs clearer positioning. Section 3.3 of the original S4D paper (Gu et al., 2022) already explains that the convolution kernel depends only on the elementwise product BCBC, and discusses directly learning that product or freezing B=1B=1. Consequently, removing one independently learned factor in a static diagonal SSM is already a known option. The complex-square-root construction is valid, but the paper needs to explain what training with C=BC=B contributes beyond these alternatives. A comparison with a directly learned gain or fixed-BB baseline would help establish whether the value lies in optimization behavior, conversion, or the selective-model extension.

  2. The explanation of Mamba's memory-capacity loss is plausible but not yet fully established. The argument in Section 4.2 concerns the same-timestep product CtBˉt≥0C_t\bar B_t\geq 0. However, the memory task reconstructs inputs at positive lags, whose contributions involve Ct(∏r=j+1tAˉr)BˉjC_t(\prod_{r=j+1}^{t}\bar A_r)\bar B_j, with j<tj<t. Under tying, BtB_t and BjB_j can have different signs, so these historical contributions need not be non-negative. The sign-mask intervention is informative, but a fixed per-mode sign also does not restore arbitrary input-dependent sign freedom. The authors should distinguish the proven instantaneous restriction from the proposed explanation of delayed-memory performance, ideally with an intervention that separates real versus complex parameters from static versus selective projections.

  3. The practical conclusions are stronger than some of the experimental comparisons support. The recursive width comparison matches SSM parameter counts while changing the inner width, which also changes the surrounding projections and whole-model computation. Its results therefore need total-model parameter and compute accounting. Appendix H appropriately distinguishes layer-level from total-model savings; Table 10 gives only 0.77% total-parameter savings for the 29.3M Mamba configuration, making measured runtime or memory benefits particularly relevant. In addition, the 300M experiment uses only 1,000 steps and one seed. Small or statistically unresolved gaps in such settings support a limited observation under the tested budgets, rather than establishing performance equivalence after adequate training.

Questions for authors
  1. Could the authors compare C=BC=B against directly learning G=BCG=BC, or fixing B=1B=1, in S4D? Which benefits of tying remain beyond the parameterization options already discussed in the original S4D paper?

  2. Can the authors connect the same-timestep sign restriction to the positive-lag memory-capacity task more directly? For example, can they analyze historical input contributions or provide a controlled ablation that separates selectivity from the real/complex parameter choice?

  3. Could the recursive comparisons report total-model parameters and compute, and could the larger language-model checks report training-token budgets and convergence curves? Confidence intervals relative to a stated acceptable degradation would also make the claims of negligible loss easier to assess.

2 comments

We thank the reviewer for a careful review. We uploaded a revised PDF (section and table numbers below refer to it).

Q1: positioning vs. B=1 / learned gain. You are right: S4D (Sec. 3.3) notes that the kernel depends only on the product BC, and DSS freezes B=1 while training C, which is equivalent to learning the gain G=BC directly. We cited DSS but did not compare. We now do (App. D.1, Table 19). S4D memory capacity, 5 seeds:

SettingfreetiedB=1
Single layer, hard8.061 ± 0.0308.065 ± 0.0398.078 ± 0.041
Stacked, 1 layer25.86 ± 0.3726.26 ± 0.7525.67 ± 0.47
Stacked, 2 / 3 layers32.08 / 32.0832.07 / 32.0732.08 / 32.08

B=1 is indistinguishable from free and tied in all five settings (the single-layer easy row, 31.998, is omitted), as the algebra predicts; L=2,3 sit at the 32 = d_model ceiling. We therefore claim no benefit of C=B over B=1 or learned G for static complex S4D. What remains is the controlled comparison across architectures, the selective case (where B=1 removes the input-dependent write path), and post-hoc conversion. On Mamba, B_t≡1 with C_t free (832 params, same as tied) scores 6.963 ± 0.461 on memory capacity, matching free (6.908 ± 0.511) and ahead of tied (5.872 ± 0.277) and tied_sign (6.649 ± 0.185); it is untested on real tasks. The paper now presents B=C as one of several ways to remove the redundancy, not the best (Sec. 1, 6).

Q2: delayed memory; real/complex vs. static/selective. Agreed. Cor. 1 is the lag-0 result, and our metric covers lags 1–24, where the tied coupling B_t[n]B_s[n] is non-negative only if B_t[n] keeps a stable sign (empirical). On trained models, the |coupling|-weighted share of negative couplings over lags 1–24 is 0.472 (free) vs. 0.003 (tied). A 2×2 ablation (memory capacity, 5 seeds; paired gap/SE of tied vs. free in brackets):

freetiedtied_sign
Static, complex (S4D single layer)32.0032.00—
Static, real, single layer (easy / hard)5.35 / 5.094.88 / 4.11 [−21.0 / −4.0]5.51 / 5.70
Static, real, stacked L1 / L2 / L35.69 / 7.28 / 9.145.36 / 6.52 / 6.34 [−2.4 / −4.3 / −4.3]5.98 / 7.70 / 9.81
Selective, complex (n=15)13.6012.62 [−1.7, n.s.]—
Selective, real (Mamba)6.915.87 [−2.9]6.65

Real-valued state alone, without selectivity, produces a significant tying cost (6–31%) that the sign mask recovers in all five static-real settings, while complex static state shows none. Selective complex state narrows the cost to 7%, not significant at n=15: consistent with, not proof of, a residual selectivity component. Caveats: static real models have far lower capacity (≈5 vs. 32), and for the selective block the deficit vanishes at 3,200 and 8,000 steps (free 3.91 / 6.21 vs. tied 6.77 / 6.92), so it is an 800-step effect (Sec. 4.2, Table 3).

Q3a: total parameters. Agreed. The old comparison matched SSM parameters only. With free models matched on total parameters (within 0.5%) and plain tied (1,000 steps, 3 seeds), the old tied_sign advantage turns out to be a width effect (free 48 → 62 channels gains 0.117 bpb, more than the original 0.064 gap), and we withdraw it. Round 1: tied 2.822 vs. total-matched free 2.856 (p = 0.006); later rounds are not significant. tied_sign is 0.05–0.065 bpb behind total-matched free in every round (Sec. 4.3, Table 7).

Q3b: compute. Tying saves 0.77% of total parameters at 29.3M and 0.46% at 150M. We did not measure wall-clock time or memory, so we claim no speed or memory benefit; the edge-hardware framing is limited to settings where SSM-layer parameters are the binding constraint.

Q3c: token budgets and equivalence. You are right that these are early-training observations: S4D enwik8 983M bytes (3 seeds), Mamba enwik8 492M (3), 150M model 10.2M (3), 300M model 1.0M (1). Paired 90% intervals for tied − free: S4D enwik8 [+0.02, +0.24]% and Mamba enwik8 [+0.07, +0.74]% lie inside ±1%; 150M [−1.91, +1.30]% is too wide; 300M is not assessable (Sec. 4.4, Table 20).

Summary of changes in the revision: B=1 baselines added and the contribution repositioned; Cor. 1 restated as the lag-0 result, with the negative-coupling diagnostic and the 2×2 ablation added; the old recursive advantage withdrawn as a width effect in favor of total-parameter-matched comparisons; and token budgets and equivalence intervals added, with the large-scale claims restricted accordingly.

Reviewer_ncV6OpenReview ↗

Comments

Thank you for the detailed response and additional analyses. The authors' response addresses several specific concerns and clarify the scope of the claims. The evidence for practical benefits is still limited by the small whole-model savings and short training budgets. The reviewer therefore retains the original weak-rejection rating.

Reviewer_zL1DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether state-space models need separate input and output projections. It compares tying these projections with learning them independently in static S4D and selective Mamba.For static diagonal S4D, a complex square-root reparameterization preserves the input-output map. In Mamba, tying lowers memory-task performance, while a fixed sign mask recovers much of the gap. The experiments cover depth and sign-mask ablations, language modeling, nonlinear RNNs and matched SSM parameter budgets. The S4D experiments show little cost from tying.The paper also tests tying the projections of pretrained models. This conversion can initially reduce Mamba performance, although fine-tuning recovers much of the loss.

Critical strengths
  1. The S4D argument clearly explains why tying preserves the input-output map for static diagonal SISO channels.
  2. The fixed sign mask recovers much of the Mamba memory-task gap. The depth ablations show that this improvement is not confined to a single layer.
  3. The extra seeds show how much results vary between runs, and unsuccessful experiments are reported too. The paper distinguishes SSM parameter savings from savings in the full model.
Critical weaknesses
  1. The same-timestep sign restriction does not by itself explain the delayed-memory gap. Products of tied projections at different timesteps can still be negative.

    The sign-mask results support the proposed explanation, but do not clearly distinguish representational limits from optimization effects.

  2. The recursive comparison matches SSM parameters, but the wider tied models have larger non-SSM projections. It therefore does not compare models with the same total parameter budget.

  3. The small enwik8 model's reported means imply about 35% gap recovery, which I could not reconcile with 54–73%.

    The post-hoc claim about avoiding zero-shot damage also needs qualification, given the unsuccessful enwik8 sign-mask conversions.

Questions for authors
  1. What evidence links the same-timestep sign restriction to the delayed-memory gap, beyond the sign-mask ablation?
  2. Can you report total model sizes for each recursive round and include plain tying in that comparison?
  3. How was the 54–73% recovery calculated? Which post-hoc conversions support the claim that sign masking prevents zero-shot damage?
1 comment

We thank the reviewer for a careful read. Two of your points identified errors in our paper, which we correct below. We uploaded a revised PDF (section and table numbers refer to it).

Recovery number (W4, Q3). You are right, and the error is ours. "54–73%" is the fraction of the free–tied gap that remains with tied_sign, not the fraction closed. Final val. bpb on enwik8:

seedfreetiedtied_signgap closed
02.13542.14012.138534%
12.12742.14052.137027%
22.13462.14262.138946%

On the means it is (2.1411 − 2.1381)/(2.1411 − 2.1325) = 35%. The paper now says tied_sign "closes about a third (27–46% per seed) of a small gap" (Sec. 4.4); it remains ~0.0056 bpb behind free. The other recovery figures recompute correctly: 86% = (6.688−5.872)/(6.819−5.872), and 80.7% = (2.7157−2.2491)/(2.7157−2.1378).

Zero-shot claim (W5, Q3). Also correct, and unsupported. Applied post hoc to a trained checkpoint, a fixed mask is worse than plain averaging on both tasks:

memory capacity (5 seeds)enwik8 bpb (1 seed)
free6.822.1378
zero-shot tied (average)3.192.7157
zero-shot tied_sign (random mask)1.366.4942
zero-shot tied_sign_alt2.679.6340

9.63 bpb is worse than an untrained model's 8.0, because the mask is not adapted to the checkpoint. We deleted the claim (Sec. 5). After fine-tuning, masked and plain conversions are indistinguishable on memory capacity (n=5; the ranking reverses between 2,000 and 6,000 steps), and on enwik8 the masked variant is slightly worse (2.1512 vs. 2.1377, single seed). The revised Sec. 5 scopes the mask to models trained tied from scratch.

W1/Q1 and W2: mechanism. Cor. 1 forces only the lag-0 coupling Δ_tB_t² ≥ 0, while our metric covers lags 1–24, where tied couplings B_t[n]B_{t−k}[n] are non-negative only if B_t[n] keeps a stable sign (empirical, not a theorem). On trained single-block models (5 seeds), the |coupling|-weighted share of negative C_t[n]B_{t−k}[n] over lags 1–24 is 0.472 for free vs. 0.003 for tied. Cor. 1 is now stated as the lag-0 result (Sec. 3). The mask ablation alone does not separate representational limit from optimization, so we added a B_t≡1 baseline with tied's parameter count (6.963 ± 0.461 vs. free 6.908 ± 0.511) and a budget test (memory capacity, 5 seeds). Free / tied / tied_sign at 800, 3,200 and 8,000 steps: 6.91 / 5.87 / 6.65, 3.91 / 6.77 / 7.65, and 6.21 / 6.92 / 6.68. The 800-step deficit does not persist, and free is erratic at longer budgets (we have not diagnosed why), so we cannot call it a representational limit. Sec. 4.2 now calls it budget-specific.

Total parameters (W3/Q2). You are right: matching SSM parameters only left tied_sign with 10.6–16.2% more total parameters (179,456 vs. 154,432 in round 1) and 14–16 more channels. We added free models matched on total parameters (d_inner 62, 46, 35, 26; within 0.5%) and plain tied: same data, code, 1,000 steps, batch 16, 3 seeds. Mean val. bpb:

roundtiedfree, total-matchedfree, SSM-matched (old)tied_sign (old)
12.8222.8562.9732.910
22.9142.9263.0482.991
32.9973.0073.1263.062
43.0793.0733.1993.131

(1) The old advantage was a width effect, which we withdraw: widening free from 48 to 62 channels gains 0.117 bpb, more than the original 0.064 gap. (2) Plain tied is as good as or better than total-matched free (free − tied: +0.035, paired p = 0.006, in round 1; +0.012, +0.010, −0.006 after, n.s.). (3) tied_sign is 0.05–0.065 bpb behind total-matched free in every round (p = 0.003–0.046), so we removed any claim that it is never worse than free. Caveats: small 4-layer LM, 1,000 steps, 3 seeds, and tying saves only ~2% of total parameters here (Sec. 4.3, Table 7).

Summary of changes in the revision: the recovery figure corrected to about a third (27–46% per seed); the zero-shot claim deleted and the mask scoped to from-scratch training; Cor. 1 restated as the lag-0 result, with the sign diagnostic and the budget test added; the Mamba cost described as budget-specific; and total-parameter-matched comparisons made primary, with the old recursive advantage withdrawn as a width effect.

Reviewer_A5w2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the parameter redundancy between the input projection B and the output projection C in diagonal state-space models, examining how tying the two as B = C affects expressivity, memory capacity, and task performance. The authors conduct experiments on S4D, Mamba, and conventional RNNs, and theoretically analyze how complex versus real parameterizations affect the tying. Results show that tying substantially reduces SSM-layer parameters while preserving performance on some tasks, but causes noticeable degradation on Mamba's memory-capacity tasks. To address this, the authors propose a Fixed Sign Mask that requires no additional parameters to recover part of the lost expressivity, and further explore the feasibility of post-training tying and fine-tuning. Overall, the paper provides valuable empirical observations, but its methodological novelty, theoretical explanation, and reliability of experimental conclusions all exhibit clear shortcomings, limiting its overall contribution.

Critical strengths
  1. The paper validates the effectiveness of B = C parameter tying across multiple experiments, including synthetic memory-capacity tasks, sMNIST, and enwik8 language modeling. Under the tested settings, this approach reduces SSM-layer parameters by roughly 33% while largely preserving the performance of the original model.

  2. This paper finds experimentally that directly tying B = C causes a noticeable performance loss for Mamba on memory-capacity tasks, and provides a theoretical analysis from the perspective of sign limitations in real parameterizations. To address this, the authors propose a simple Fixed Sign Mask that recovers about 86% of the lost memory capacity without introducing additional trainable parameters.

Critical weaknesses
  1. In the reparameterization subsection, LoRA and LTH have weak direct relevance to this paper, while the more relevant (B=1) parameterization in S4D/DSS is not discussed. The proposed (B=C) in this paper has the same theoretical expressiveness and parameter savings in complex diagonal SISO systems, but it is not directly compared with this existing scheme, nor does the paper explain what additional advantages the squared parameterization offers in terms of optimization or practical performance.

  2. There is misinterpretation of related work in the real and complex recurrence section. The authors claim that Orvieto et al. (2023) demonstrated that real-valued recurrences are generally not necessarily inferior to complex-valued recurrences, whereas the original paper actually enables linear RNNs to achieve the performance and computational efficiency of deep SSMs through designs such as complex diagonalization, stable parameterization, and normalization, rather than proving that purely real-valued recurrences are generally not inferior to complex-valued recurrences.

  3. The theoretical attribution of Mamba's performance loss is insufficient. Corollary 1 only proves that real-valued (B=C) restricts the sign of the single-mode gain, but Mamba's input-dependent recurrence involves (C_tB_s) across time steps, which need not be nonnegative even when tied. Therefore, this corollary is not yet sufficient to fully explain Mamba's overall memory capacity loss. It is worth acknowledging that the proposed Fixed Sign Mask is simple and effective, and can recover most of the performance loss without adding trainable parameters.

  4. The task difficulty and discriminative power of the S4D experiments are limited. Most memory capacity experiment results are already close to the capacity upper bound determined by the task or output dimension. In addition, sMNIST accuracy is near saturation, and the enwik8 experiment only uses a relatively small-scale model. Although these results support that (B=C) tying causes almost no performance loss under the tested settings, they are not yet sufficient to fully validate its training performance on more challenging tasks and larger models.

  5. The cross-task consistency of the Mamba experimental conclusions is insufficient. Although the synthetic memory capacity experiments show that B/C tying causes about a 14% performance loss, this phenomenon is not stable in practical tasks: the two sets of sMNIST experimental results differ, and in larger-scale language models, Tied is even slightly better than Free. In addition, the authors' observation that tying improves training stability comes only from a single synthetic task and lacks further validation.

  6. There appears to be residual Copilot-generated content in the OpenReview Keywords.

Questions for authors

Please refer to the Weakness part.

2 comments

We thank the reviewer for a careful and specific review. Most points are correct. We uploaded a revised PDF (section and table numbers below refer to it) and respond to each, with new experiments where they bear on the point.

W1: B=1 (S4D/DSS). Correct. S4D (Sec. 3.3) describes DSS as freezing B=1 while training C, and we cited DSS but never compared against it. For a complex diagonal channel, B=1 and B=C have the same expressivity (any gain g=BC is reachable by C=g or B=√g, Prop. 1) and the same saving, so we claim no advantage for C=B in that setting. New B=1 baselines (5 seeds, S4D memory capacity; B frozen to 1, C trained):

SettingfreetiedB=1
Single layer, hard8.061 ± 0.0308.065 ± 0.0398.078 ± 0.041
Stacked, 1 layer25.86 ± 0.3726.26 ± 0.7525.67 ± 0.47
Stacked, 2 / 3 layers32.08 / 32.0832.07 / 32.0732.08 / 32.08

The single-layer easy row is 31.997–31.998 for all three (App. D.1, Table 19). L=2,3 sit at the 32 = d_model ceiling; only L=1 has headroom. On a single Mamba block (memory capacity, 5 seeds; B_t≡1 with C_t free, 832 SSM params, same as tied):

free (1,088)tied (832)tied_sign (832)B_t=1 (832)
6.908 ± 0.5115.872 ± 0.2776.649 ± 0.1856.963 ± 0.461

We report this plainly: here B=C is not the best way to remove one projection. B_t=1 removes the input-dependent write path, which language modeling may need, and we have not tested it on sMNIST or LM. We rewrote the contribution as a controlled comparison and diagnosis, not a claim that C=B is best (Sec. 1, 2).

W2: Orvieto et al. Correct, our description was wrong. The LRU is a complex diagonal recurrence, and its ablation compares against a dense real A, not real vs. complex diagonal. We deleted the claim that real recurrences "need not underperform complex ones" (Sec. 2). Since the LRU is complex-state, Prop. 1 predicts lossless tying there (untested), whereas Mamba's real state is where Cor. 1 applies.

W3: Cor. 1 vs. Mamba. Correct. Cor. 1 gives only the lag-0 coupling C_tB̄_t = Δ_tB_t² ≥ 0, while our metric covers lags 1–24. There the coupling from step s to t is C_t[n]B_s[n]G_ts with G_ts > 0, and tied gives B_t[n]B_s[n], non-negative at every lag only if B_t[n] keeps one sign over time: an empirical property, not a theorem. On trained single-block models (5 seeds), the |coupling|-weighted share of negative C_t[n]B_{t−k}[n] over lags 1–24 is 0.472 for free vs. 0.003 for tied, and free models' B_t signs are mostly stable (0.84). To separate real-valued state from selectivity, a 2×2 ablation (Table 3; paired gap/SE of tied vs. free): a static real-state SSM shows a 6–31% tying cost at every setting (single layer 5.35 free vs. 4.88 tied, gap/SE −21; stacked L1–L3 −2.4 to −4.3) that the fixed sign mask recovers (5.51), while complex static state shows none. Selective complex state narrows the Mamba cost to 7% (13.60 vs. 12.62, n=15, gap/SE −1.7, not significant). This is a diagnostic plus interventions, not a proof; the weighting omits G_ts, and tied sign stability is 0.79 (Sec. 4.2).

W4: task difficulty. Partly agreed. Single-layer capacity settings sit near a bound set by D or the number of lags, L=2,3 saturate at 32, and sMNIST is ~99%, so these runs show tied reaching the same ceiling, not more. Only stacked L=1 has headroom (≈26 of 64 lags; free vs. tied 25.9 ± 0.37 vs. 26.3 ± 0.75), and enwik8 at d_model=256 is small. The S4D claim is now "no measurable cost on the tasks tested" (Sec. 4.1 retitled). We ran no harder S4D task such as sCIFAR and list that as a limitation.

W5: cross-task consistency and stability. Agreed that the Mamba cost is not uniform: the sMNIST gap is unsettled across two 3-seed studies, and at LM scale tied is within noise of free. A new budget test (800 / 3,200 / 8,000 steps, 5 seeds) shows the 800-step deficit (tied 5.87 vs. free 6.91) vanishes: at 3,200 steps tied is 6.77 vs. free 3.91 ± 0.77, and at 8,000 steps 6.92 ± 0.57 vs. 6.21 ± 2.31. We withdraw any suggestion of a general Mamba limit. We also withdraw "tied_sign is never worse than free": in a total-parameter-matched small-LM check it is 0.05–0.065 bpb behind free in every round (see our reply to zL1D). On stability, the claim rested on one task (~1-in-12 catastrophic free failures); we removed it from the abstract, results and conclusion.

Keywords. Thank you; we corrected them.

Thanks for the authors' response.

First, I note that the authors acknowledge W1–W3 as well as W5 and W6. While I appreciate their candid disclosure, these weaknesses detract from the value of the work.

Second, the authors frame the omission of sCIFAR as a limitation, yet sMNIST is too simple to offer sufficient discriminative power, and evaluating on sCIFAR (or harder benchmarks like Pathfinder) is standard practice in DSSM research; thus, treating it merely as a limitation downplays a key gap.

Finally, in the AI use statement, the authors claim that "Generative AI tools were not used to generate the paper's text, analysis, ..." yet the paper contains many expressions that appear characteristic of AI generation, such as excessive awkward em dashes ("—"), overused semicolons (";"), and citations appearing in the abstract, while W6 further raises concerns about this issue. Therefore, I am also worried that the authors' use of AI tools actually goes well beyond what is stated in the AI USE STATEMENT section.

Based on the reasons above, I decide to retain my judgment and rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses and changes

We thank the reviewers. Their feedback exposed real errors and overstated claims, and we have uploaded a revised PDF. Individual replies follow under each review; the main changes are below.

Corrections to claims

  • Recovery figure: "54–73%" was the fraction of the gap that remains. The sign mask closes about a third (27–46% per seed) of the small enwik8 gap.
  • Zero-shot claim deleted: a fixed sign mask applied post hoc is worse than plain averaging. The mask is now scoped to models trained tied from scratch.
  • Orvieto et al.: our description was wrong; the LRU is a complex-state model. The claim is deleted.
  • Corollary 1 is now stated as the lag-0 result only. The delayed-memory account rests on a measured sign diagnostic (negative-coupling share 0.472 for free vs. 0.003 for tied), described as a mechanism, not a proof.
  • Stability and "free lunch" language is removed or scoped to the tasks tested.

New experiments

  • B=1 / learned-gain baseline (DSS-style): indistinguishable from free and tied on S4D, so we claim no advantage for C=B there. On Mamba, B_t≡1 matches free on memory capacity and beats tied.
  • 2×2 ablation (real/complex × static/selective): real-valued state alone, without selectivity, produces a significant tying cost that the sign mask recovers; complex static state shows none.
  • Training-budget test: the Mamba tying deficit at 800 steps vanishes at 3,200 and 8,000 steps, so it is budget-specific, not an established representational limit.
  • Total-parameter-matched LM comparison: our earlier recursive advantage for tied_sign was a width effect, and we withdraw it. Plain tied is as good as or better than total-matched free, while tied_sign trails both in this small LM.
  • Token budgets and equivalence intervals: enwik8 intervals lie inside ±1%; the 150M interval is too wide to support equivalence, and 300M is a single run.

Revised positioning. The paper is now a controlled comparison and diagnosis, not a claim that C=B is the best way to remove a projection. Limitations now list the S4D ceiling effects, the untested B_t=1 on real tasks, the early-training LM checks, and the absence of a wall-clock benchmark.

We are grateful for the scrutiny; the paper is more accurate as a result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.