- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how the effect of pretrained Mamba initialization varies with the timescale geometry, and how to initialize the decay rates using additional scans, and proposes a method called SpanInit to place each new state in the middle of the biggest empty gap between existing timescales. Across nine different checkpoints, ImageNet-1k models have a collapse and no spread in timescales, while ImageNet-21K MambaVision models have a wider span.
Critical strengths
-
The main observation of SpanInit is simple and very cheap to compute. The threefold contributions are validated across multiple checkpoints.
-
The gap between ImageNet-21k and ImageNet-1k checkpoints is well justified and holds across different sizes.
-
The ablations and experiments are well designed, Figure 2 is informative and clearly shows the insertions.
Critical weaknesses
-
The reason for low spread is not checked or validated. In Table 1 and Sec 4.2, the five ImageNet-1k checkpoints has median |A| of 0.99–1.01. Since A = −exp(A_log), this is exactly what you get when A_log is pushed to zero. This also happens when weight decay is applied to A_log during training in most recent libraries by default. If this is the case, the low spread comes from a training setting, not from anything the model learned.
-
One of the major issues in the paper is that spacing the new states evenly (ignoring the pretrained values) already gives +0.30 of the +0.39 gain. The extra +0.09 from SpanInit is within seed noise, and no direct test between the two is shown. This weakness the paper's main contribution.
-
The method is not really "checkpoint-aware" as the title mention; SpanInit is almost one fixed rule. It helps on some checkpoints and hurts on others, and the paper gives no way to know in advance which case you are in.
-
Although the authors evaluated on multiple checkpoints, the evaluation scope itself is limited. It tests only VTAB-1k classification.
Questions for authors
-
Is the weight decay applied to A_log when training each checkpoint? If you remove it and check again, does the low spread go away?
-
Is the gain of SpanInit over even spacing statistically significant? Can you validate that?
-
Why is your Vim-S baseline 0.9 points lower than the published result for Additional-scan, although the linear probe matches closely (52.72 vs. 52.75)?
4 comments
Response to Reviewer y3jx (2/2)
[W4] Evaluation beyond VTAB-1k.
We additionally evaluate on five few-shot fine-grained classification datasets following the NOAH [1] benchmark family. Mean accuracies on MambaVision-B (ImageNet-21k, N′ = 6, 3 seeds) are reported below.
| Dataset | Additional-scan | SpanInit | SpanInit − Additional-scan |
|---|---|---|---|
| Food-101 | 36.52 | 37.80 | +1.28 |
| Oxford Flowers | 93.30 | 93.04 | −0.26 |
| Stanford Cars | 9.88 | 9.69 | −0.19 |
| Oxford Pets | 76.14 | 77.11 | +0.97 |
| FGVC-Aircraft | 12.99 | 11.71 | −1.28 |
Consistent with the VTAB-1k results, the effect varies across datasets: SpanInit improves Food-101 and Oxford Pets, lowers FGVC-Aircraft, and changes little on the other two. Its unweighted five-dataset mean difference is approximately +0.10 points, so this experiment broadens the evaluation setting but should not be presented as evidence of consistent superiority.
[Q3] Vim-S baseline.
Thank you for pointing out this gap. Checking MambaPEFT's released configuration, we found that it zero-initializes the classification head, whereas we used PyTorch's default initialization. Re-running Additional-scan with a zero-initialized head (19 tasks × 3 seeds) closes most of the gap. We re-ran both methods on Vim-S and MambaVision-B (ImageNet-21k) with the zero-initialized head:
| 19 tasks | Additional-scan | SpanInit | SpanInit − Additional-scan [95% CI] |
|---|---|---|---|
| Vim-S | |||
| MambaPEFT (published) | 68.65 | – | – |
| Ours, default head (paper) | 67.74 | 67.49 | −0.25 [−0.59, +0.09] |
| Ours, zero-initialized head | 68.54 | 68.23 | −0.32 [−0.51, −0.13] |
| MambaVision-B (ImageNet-21k) | |||
| Ours, default head (paper) | 69.60 | 69.99 | +0.39 [+0.20, +0.59] |
| Ours, zero-initialized head | 69.71 | 70.27 | +0.56 [+0.38, +0.74] |
The linear-probe objective over frozen features is convex in its classifier parameters, but finite-step optimization can still depend on initialization; therefore, the close linear-probe reproduction alone does not exclude a head-initialization effect. The 19-task re-run supports head initialization as an explanation for most of the gap (from −0.91 to −0.11 points). All arms within our paired comparisons use the same head initialization, and with the zero-initialized head the SpanInit − Additional-scan difference stays within 0.17 points of the paper's value on both checkpoints. We will explicitly state the head initialization in the revision.
Please let us know if our answers do not fully address your concerns.
[1] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural Prompt Search. arXiv:2206.04673, 2022.
Follow up with the authors
Thank you to the authors for the detailed rebuttal and the new comparison with Vim-S and MambaVision-B; I really appreciate your transparency in reporting the numbers. With matched zero-initialized heads, SpanInit improves mean accuracy by 0.56% on MambaVision-B but reduces it by 0.32% on Vim-S. Across the five additional fine-grained datasets, the mean gain is only 0.10. Given the modest, backbone-dependent gains and the limited methodological and analytical contributions, I therefore maintain my rating of Weak Reject.
Further Response to Reviewer y3jx
We thank the reviewer for the continued engagement and for appreciating the transparency of our reporting.
We agree that the aggregate gains are modest; SpanInit changes only the initial A values of the added states and adds no trainable parameters or inference cost. However, we respectfully disagree that the backbone dependence weakens the contribution. It is the central claim of the paper: the effect of initializing added states depends on the checkpoint's timescale geometry, and this geometry can be read from A alone before adaptation, without data or a forward pass. The dependence appears even within a single architecture. On MambaVision-B, SpanInit raises the 19-task mean on the ImageNet-21k checkpoint (+0.39) and lowers it on the ImageNet-1k checkpoint (−0.38). Vim-S is an ImageNet-1k checkpoint with a near-collapsed bank, with every state within a factor of two of |A| = 1. A decrease on Vim-S is therefore consistent with the paper's diagnosis rather than a counterexample to it.
The zero-initialized-head re-runs make this pattern clearer. The gain on MambaVision-B increases from +0.39 to +0.56 [+0.38, +0.74], and the decrease on Vim-S, previously not distinguishable from zero (−0.25 [−0.59, +0.09]), now excludes zero (−0.32 [−0.51, −0.13]). MambaVision-L2 (ImageNet-21k), whose wide span was already reported in the submission, was evaluated on VTAB-1k only afterwards and also improves (+1.14 [+0.63, +1.66]). Across the evaluated checkpoints, all three wide-span checkpoints improve and four of the five near-collapsed checkpoints decrease. The paper therefore motivates choosing between SpanInit and copying by the bank geometry, which can be measured before adaptation, rather than applying a single fixed rule.
Response to Reviewer y3jx (1/2)
We are glad the reviewer found the observation simple and cheap to compute, and the ablations well designed.
[W1/Q1] Weight decay on A_log.
We agree that determining whether A_log was weight-decayed requires the actual optimizer parameter-group mapping, not just the final A_log values or a model attribute. Our audit of available training artifacts indicates nonzero weight-decay settings for A_log (0.075–0.12 in the examined MambaVision ImageNet-1k runs and 0.05–0.1 in the examined Vim runs). We will provide, for each release where the evidence is available, the source artifact and an explicit A_log-to-optimizer-group mapping; where such evidence is absent, we will not claim that its training-time treatment is verified. The ImageNet-21k pretraining history cannot be established from the final fine-tuning checkpoint alone.
To address the second question, we trained MambaVision-T from scratch on a 100-class ImageNet subset for 100 epochs with and without weight decay on A_log, using the values of the released ImageNet-1k MambaVision-B and -L (0.075 and 0.12). Final within-channel span (8× at initialization):
With A_log decayed, the span shrinks from 8× to 1.40× and 1.19×, close to the released ImageNet-1k checkpoints (1.22–1.67×); with A_log excluded, it widens to 18.5–23.1×.
We thank the reviewer for identifying this mechanism. Under the controlled MambaVision-T setup on the ImageNet subset, the comparison supports a causal effect of weight decay on the final within-channel span. It provides a plausible regularization-based explanation for the near-collapsed released checkpoints, but does not establish the exact cause in every original pretraining run. We will state the limitation explicitly in Section 4.2 and add the controlled experiment to the appendix. Importantly, the geometry of the released tensor remains measurable before downstream adaptation regardless of how that geometry arose.
[W2/Q2] Significance of SpanInit over even spacing.
We agree that the available data do not establish a statistically significant advantage of SpanInit over uniform log-spacing. On MambaVision-B (ImageNet-21k), the direct seed-paired difference is +0.09 points (95% CI [−0.13, +0.31]). Uniform log-spacing operates over the same architecture-defined scan window [1, L] and uses the same calibrated mean step to map timescales into A. It captures most of the gain over copying (+0.30 of +0.39). We will therefore explicitly distinguish the supported effect of calibrated timescale diversification from the unproven incremental accuracy benefit of selecting gaps relative to the pretrained bank.
[W3] “Checkpoint-aware”.
We agree that the greedy gap-filling rule is fixed, while its pretrained anchors and the calibrated mapping to initial A values are checkpoint-dependent. The absolute window [1, L] itself is defined by scan length, not learned from the checkpoint. In our evaluated set, all three wide-span MambaVision checkpoints improve (+0.39 to +1.14, including the newly evaluated MambaVision-L2), whereas four of the five near-collapsed checkpoints decrease; Vim-B is an inconclusive exception. This association motivates a pre-adaptation diagnostic for choosing SpanInit versus copying, not a validated universal predictor. We will make the distinction explicit in the introduction and label the decision figure in Section 5 as a hypothesis requiring validation on unseen checkpoints.