- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulates differential computation for state space models through structured state space duality (SSD). For two branches that share temporal dynamics, the authors propose a single-pass computation method using an augmented state of dimension 2N. They further propose C-only SDC, which adjusts the readout through input-dependent subtraction and normalization. This design is applied to Mamba-2 and Mamba-3. Experiments show that the differential variants improve average accuracy on downstream language modeling tasks and achieve gains on several retrieval tasks.
Critical strengths
-
The C-only variant shares the recurrent state and preserves the state dimension. Compared with duplicating the entire Mamba module, this design reduces the additional structure required to implement differential computation.
-
The language modeling experiments cover multiple model sizes and use 100B-token pretraining. In all comparisons reported in Table 1, the differential variants achieve higher average downstream scores than their corresponding baselines.
Critical weaknesses
-
The justification for the connection between differential readout from a shared state and selective denoising remains insufficient. The paper interprets this design as more precise common-mode noise cancellation, but does not sufficiently explain why the components jointly represented by the two branches mainly correspond to irrelevant information, or why subtraction can suppress these components while preserving the target signal.
-
The gains and regressions in retrieval performance lack explanation. For Mamba-3 MIMO, the score on NIAH-Single-3 at 1024 tokens drops from 95.8 to 61.4. This is a substantial decline that occurs within the training length, especially considering that improving memory utilization is an important motivation of the paper.
Questions for authors
-
What direct evidence shows that the two branches’ contributions from irrelevant information are similar enough to cancel out, while the target information is preserved? Existing experiments provide only limited direct support for this mechanistic explanation. For example, could the authors, as in Differential Transformer, separately quantify how C-only changes the contributions of target and irrelevant information in tasks with clearly defined targets and distractors?
-
Table 4(a) provides evidence that subtraction brings gains in the full construction. Could the authors explain whether this conclusion also holds for the C-only + RMSNorm variant used in the main experiments?
We sincerely thank the reviewers for their time, thoughtful evaluations, and constructive feedback. After carefully considering the reviews, we have decided to withdraw our submission to further investigate the concerns raised and strengthen the work. We greatly appreciate the reviewers' valuable suggestions, which will help guide our future research and revisions.