- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The draft recognizes the value of the controlled projection-tying study, the fixed-sign modification, and the post-hoc conversion experiments. Its main reservations concern the relationship to existing diagonal-SSM parameterizations, the scope of the proposed explanation for Mamba, and the strength of the experimental evidence for practical savings and negligible performance loss. The pre-existing weak-rejection rating is retained as a provisional recommendation.
Critical strengths
-
The paper asks a clear architectural question and investigates it through both algebra and controlled experiments. The distinction between static complex-valued S4D and selective real-valued Mamba is useful, and the fixed-sign modification provides a simple intervention that substantially recovers the synthetic memory-capacity loss. Table 2 reports an increase from 5.872 to 6.688, compared with 6.819 for the untied model, without adding trainable parameters.
-
The experimental reporting makes a useful effort to distinguish reproducible findings from uncertain ones. The authors extend the Mamba capacity comparison to 30 seeds, report results across several depths, and explicitly discuss single-seed results and reversals under different training budgets. The post-hoc experiments also add practical interest: S4D admits an exact conversion, while Mamba can recover much of the conversion loss through fine-tuning, including a preliminary teacher-only recovery experiment.
Critical weaknesses
-
The contribution beyond existing diagonal-SSM parameterizations needs clearer positioning. Section 3.3 of the original S4D paper (Gu et al., 2022) already explains that the convolution kernel depends only on the elementwise product , and discusses directly learning that product or freezing . Consequently, removing one independently learned factor in a static diagonal SSM is already a known option. The complex-square-root construction is valid, but the paper needs to explain what training with contributes beyond these alternatives. A comparison with a directly learned gain or fixed- baseline would help establish whether the value lies in optimization behavior, conversion, or the selective-model extension.
-
The explanation of Mamba's memory-capacity loss is plausible but not yet fully established. The argument in Section 4.2 concerns the same-timestep product . However, the memory task reconstructs inputs at positive lags, whose contributions involve , with . Under tying, and can have different signs, so these historical contributions need not be non-negative. The sign-mask intervention is informative, but a fixed per-mode sign also does not restore arbitrary input-dependent sign freedom. The authors should distinguish the proven instantaneous restriction from the proposed explanation of delayed-memory performance, ideally with an intervention that separates real versus complex parameters from static versus selective projections.
-
The practical conclusions are stronger than some of the experimental comparisons support. The recursive width comparison matches SSM parameter counts while changing the inner width, which also changes the surrounding projections and whole-model computation. Its results therefore need total-model parameter and compute accounting. Appendix H appropriately distinguishes layer-level from total-model savings; Table 10 gives only 0.77% total-parameter savings for the 29.3M Mamba configuration, making measured runtime or memory benefits particularly relevant. In addition, the 300M experiment uses only 1,000 steps and one seed. Small or statistically unresolved gaps in such settings support a limited observation under the tested budgets, rather than establishing performance equivalence after adequate training.
Questions for authors
-
Could the authors compare against directly learning , or fixing , in S4D? Which benefits of tying remain beyond the parameterization options already discussed in the original S4D paper?
-
Can the authors connect the same-timestep sign restriction to the positive-lag memory-capacity task more directly? For example, can they analyze historical input contributions or provide a controlled ablation that separates selectivity from the real/complex parameter choice?
-
Could the recursive comparisons report total-model parameters and compute, and could the larger language-model checks report training-token budgets and convergence curves? Confidence intervals relative to a stated acceptable degradation would also make the claims of negligible loss easier to assess.
2 comments
Comments
Thank you for the detailed response and additional analyses. The authors' response addresses several specific concerns and clarify the scope of the claims. The evidence for practical benefits is still limited by the small whole-model savings and short training budgets. The reviewer therefore retains the original weak-rejection rating.
Response
We thank the reviewer for a careful review. We uploaded a revised PDF (section and table numbers below refer to it).
Q1: positioning vs. B=1 / learned gain. You are right: S4D (Sec. 3.3) notes that the kernel depends only on the product BC, and DSS freezes B=1 while training C, which is equivalent to learning the gain G=BC directly. We cited DSS but did not compare. We now do (App. D.1, Table 19). S4D memory capacity, 5 seeds:
B=1 is indistinguishable from free and tied in all five settings (the single-layer easy row, 31.998, is omitted), as the algebra predicts; L=2,3 sit at the 32 = d_model ceiling. We therefore claim no benefit of C=B over B=1 or learned G for static complex S4D. What remains is the controlled comparison across architectures, the selective case (where B=1 removes the input-dependent write path), and post-hoc conversion. On Mamba, B_t≡1 with C_t free (832 params, same as tied) scores 6.963 ± 0.461 on memory capacity, matching free (6.908 ± 0.511) and ahead of tied (5.872 ± 0.277) and tied_sign (6.649 ± 0.185); it is untested on real tasks. The paper now presents B=C as one of several ways to remove the redundancy, not the best (Sec. 1, 6).
Q2: delayed memory; real/complex vs. static/selective. Agreed. Cor. 1 is the lag-0 result, and our metric covers lags 1–24, where the tied coupling B_t[n]B_s[n] is non-negative only if B_t[n] keeps a stable sign (empirical). On trained models, the |coupling|-weighted share of negative couplings over lags 1–24 is 0.472 (free) vs. 0.003 (tied). A 2×2 ablation (memory capacity, 5 seeds; paired gap/SE of tied vs. free in brackets):
Real-valued state alone, without selectivity, produces a significant tying cost (6–31%) that the sign mask recovers in all five static-real settings, while complex static state shows none. Selective complex state narrows the cost to 7%, not significant at n=15: consistent with, not proof of, a residual selectivity component. Caveats: static real models have far lower capacity (≈5 vs. 32), and for the selective block the deficit vanishes at 3,200 and 8,000 steps (free 3.91 / 6.21 vs. tied 6.77 / 6.92), so it is an 800-step effect (Sec. 4.2, Table 3).
Q3a: total parameters. Agreed. The old comparison matched SSM parameters only. With free models matched on total parameters (within 0.5%) and plain tied (1,000 steps, 3 seeds), the old tied_sign advantage turns out to be a width effect (free 48 → 62 channels gains 0.117 bpb, more than the original 0.064 gap), and we withdraw it. Round 1: tied 2.822 vs. total-matched free 2.856 (p = 0.006); later rounds are not significant. tied_sign is 0.05–0.065 bpb behind total-matched free in every round (Sec. 4.3, Table 7).
Q3b: compute. Tying saves 0.77% of total parameters at 29.3M and 0.46% at 150M. We did not measure wall-clock time or memory, so we claim no speed or memory benefit; the edge-hardware framing is limited to settings where SSM-layer parameters are the binding constraint.
Q3c: token budgets and equivalence. You are right that these are early-training observations: S4D enwik8 983M bytes (3 seeds), Mamba enwik8 492M (3), 150M model 10.2M (3), 300M model 1.0M (1). Paired 90% intervals for tied − free: S4D enwik8 [+0.02, +0.24]% and Mamba enwik8 [+0.07, +0.74]% lie inside ±1%; 150M [−1.91, +1.30]% is too wide; 300M is not assessable (Sec. 4.4, Table 20).
Summary of changes in the revision: B=1 baselines added and the contribution repositioned; Cor. 1 restated as the lag-0 result, with the negative-coupling diagnostic and the 2×2 ablation added; the old recursive advantage withdrawn as a width effect in favor of total-parameter-matched comparisons; and token budgets and equivalence intervals added, with the large-scale claims restricted accordingly.