acceptodds
Under review as a conference paper at ICLR 2027

Pretrained Mamba-Based Vision Backbones Can Exhibit Collapsed Timescale Banks: Checkpoint-Aware State Expansion with SpanInit

Abstract

Expanding the recurrent state of a pretrained Mamba requires initializing the decay timescale of each added state dimension. Additional-scan, a parameter-efficient state-expansion method, copies a pretrained entry of the state matrix , so each added state begins at a decay scale already present in that channel's timescale bank. We show that the effect of this initialization varies with the timescale geometry inherited from pretraining. Because within-channel timescale ratios depend only on , the bank's relative span can be read directly from a checkpoint without data or a forward pass. Across released Mamba-based vision checkpoints, ImageNet-1k-only models with multiple state dimensions per channel exhibit near-collapsed timescale banks, whereas models exposed to ImageNet-21k have substantially wider spans. Motivated by this diagnosis, we propose SpanInit, which fills the largest uncovered gaps within the scan-relevant range in log-timescale space. SpanInit changes only the initial values and uses a single calibration forward pass to set absolute timescales, adding no trainable parameters or inference cost. On VTAB-1k, SpanInit improves Additional-scan on the ImageNet-21k-pretrained MambaVision-B/L checkpoint but degrades it on the ImageNet-1k-only checkpoint of the same architecture. These contrasting outcomes indicate that state-expansion initialization should account for the pretrained checkpoint's timescale geometry rather than follow a single checkpoint-agnostic rule.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, state space models, Mamba, Vision Mamba

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_y3jxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the effect of pretrained Mamba initialization varies with the timescale geometry, and how to initialize the decay rates using additional scans, and proposes a method called SpanInit to place each new state in the middle of the biggest empty gap between existing timescales. Across nine different checkpoints, ImageNet-1k models have a collapse and no spread in timescales, while ImageNet-21K MambaVision models have a wider span.

Critical strengths
  1. The main observation of SpanInit is simple and very cheap to compute. The threefold contributions are validated across multiple checkpoints.

  2. The gap between ImageNet-21k and ImageNet-1k checkpoints is well justified and holds across different sizes.

  3. The ablations and experiments are well designed, Figure 2 is informative and clearly shows the insertions.

Critical weaknesses
  1. The reason for low spread is not checked or validated. In Table 1 and Sec 4.2, the five ImageNet-1k checkpoints has median |A| of 0.99–1.01. Since A = −exp(A_log), this is exactly what you get when A_log is pushed to zero. This also happens when weight decay is applied to A_log during training in most recent libraries by default. If this is the case, the low spread comes from a training setting, not from anything the model learned.

  2. One of the major issues in the paper is that spacing the new states evenly (ignoring the pretrained values) already gives +0.30 of the +0.39 gain. The extra +0.09 from SpanInit is within seed noise, and no direct test between the two is shown. This weakness the paper's main contribution.

  3. The method is not really "checkpoint-aware" as the title mention; SpanInit is almost one fixed rule. It helps on some checkpoints and hurts on others, and the paper gives no way to know in advance which case you are in.

  4. Although the authors evaluated on multiple checkpoints, the evaluation scope itself is limited. It tests only VTAB-1k classification.

Questions for authors
  1. Is the weight decay applied to A_log when training each checkpoint? If you remove it and check again, does the low spread go away?

  2. Is the gain of SpanInit over even spacing statistically significant? Can you validate that?

  3. Why is your Vim-S baseline 0.9 points lower than the published result for Additional-scan, although the linear probe matches closely (52.72 vs. 52.75)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer y3jx (1/2)

We are glad the reviewer found the observation simple and cheap to compute, and the ablations well designed.

[W1/Q1] Weight decay on A_log.

We agree that determining whether A_log was weight-decayed requires the actual optimizer parameter-group mapping, not just the final A_log values or a model attribute. Our audit of available training artifacts indicates nonzero weight-decay settings for A_log (0.075–0.12 in the examined MambaVision ImageNet-1k runs and 0.05–0.1 in the examined Vim runs). We will provide, for each release where the evidence is available, the source artifact and an explicit A_log-to-optimizer-group mapping; where such evidence is absent, we will not claim that its training-time treatment is verified. The ImageNet-21k pretraining history cannot be established from the final fine-tuning checkpoint alone.

To address the second question, we trained MambaVision-T from scratch on a 100-class ImageNet subset for 100 epochs with and without weight decay on A_log, using the values of the released ImageNet-1k MambaVision-B and -L (0.075 and 0.12). Final within-channel span (8× at initialization):

Weight decay0.0750.12
Applied to A_log1.40×1.19×
A_log excluded23.1×18.5×

With A_log decayed, the span shrinks from 8× to 1.40× and 1.19×, close to the released ImageNet-1k checkpoints (1.22–1.67×); with A_log excluded, it widens to 18.5–23.1×.

We thank the reviewer for identifying this mechanism. Under the controlled MambaVision-T setup on the ImageNet subset, the comparison supports a causal effect of weight decay on the final within-channel span. It provides a plausible regularization-based explanation for the near-collapsed released checkpoints, but does not establish the exact cause in every original pretraining run. We will state the limitation explicitly in Section 4.2 and add the controlled experiment to the appendix. Importantly, the geometry of the released tensor remains measurable before downstream adaptation regardless of how that geometry arose.

[W2/Q2] Significance of SpanInit over even spacing.

We agree that the available data do not establish a statistically significant advantage of SpanInit over uniform log-spacing. On MambaVision-B (ImageNet-21k), the direct seed-paired difference is +0.09 points (95% CI [−0.13, +0.31]). Uniform log-spacing operates over the same architecture-defined scan window [1, L] and uses the same calibrated mean step to map timescales into A. It captures most of the gain over copying (+0.30 of +0.39). We will therefore explicitly distinguish the supported effect of calibrated timescale diversification from the unproven incremental accuracy benefit of selecting gaps relative to the pretrained bank.

[W3] “Checkpoint-aware”.

We agree that the greedy gap-filling rule is fixed, while its pretrained anchors and the calibrated mapping to initial A values are checkpoint-dependent. The absolute window [1, L] itself is defined by scan length, not learned from the checkpoint. In our evaluated set, all three wide-span MambaVision checkpoints improve (+0.39 to +1.14, including the newly evaluated MambaVision-L2), whereas four of the five near-collapsed checkpoints decrease; Vim-B is an inconclusive exception. This association motivates a pre-adaptation diagnostic for choosing SpanInit versus copying, not a validated universal predictor. We will make the distinction explicit in the introduction and label the decision figure in Section 5 as a hypothesis requiring validation on unseen checkpoints.

AuthorsOpenReview ↗

Response to Reviewer y3jx (2/2)

[W4] Evaluation beyond VTAB-1k.

We additionally evaluate on five few-shot fine-grained classification datasets following the NOAH [1] benchmark family. Mean accuracies on MambaVision-B (ImageNet-21k, N′ = 6, 3 seeds) are reported below.

DatasetAdditional-scanSpanInitSpanInit − Additional-scan
Food-10136.5237.80+1.28
Oxford Flowers93.3093.04−0.26
Stanford Cars9.889.69−0.19
Oxford Pets76.1477.11+0.97
FGVC-Aircraft12.9911.71−1.28

Consistent with the VTAB-1k results, the effect varies across datasets: SpanInit improves Food-101 and Oxford Pets, lowers FGVC-Aircraft, and changes little on the other two. Its unweighted five-dataset mean difference is approximately +0.10 points, so this experiment broadens the evaluation setting but should not be presented as evidence of consistent superiority.

[Q3] Vim-S baseline.

Thank you for pointing out this gap. Checking MambaPEFT's released configuration, we found that it zero-initializes the classification head, whereas we used PyTorch's default initialization. Re-running Additional-scan with a zero-initialized head (19 tasks × 3 seeds) closes most of the gap. We re-ran both methods on Vim-S and MambaVision-B (ImageNet-21k) with the zero-initialized head:

19 tasksAdditional-scanSpanInitSpanInit − Additional-scan [95% CI]
Vim-S
MambaPEFT (published)68.65––
Ours, default head (paper)67.7467.49−0.25 [−0.59, +0.09]
Ours, zero-initialized head68.5468.23−0.32 [−0.51, −0.13]
MambaVision-B (ImageNet-21k)
Ours, default head (paper)69.6069.99+0.39 [+0.20, +0.59]
Ours, zero-initialized head69.7170.27+0.56 [+0.38, +0.74]

The linear-probe objective over frozen features is convex in its classifier parameters, but finite-step optimization can still depend on initialization; therefore, the close linear-probe reproduction alone does not exclude a head-initialization effect. The 19-task re-run supports head initialization as an explanation for most of the gap (from −0.91 to −0.11 points). All arms within our paired comparisons use the same head initialization, and with the zero-initialized head the SpanInit − Additional-scan difference stays within 0.17 points of the paper's value on both checkpoints. We will explicitly state the head initialization in the revision.

Please let us know if our answers do not fully address your concerns.

[1] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural Prompt Search. arXiv:2206.04673, 2022.

Reviewer_y3jxOpenReview ↗

Follow up with the authors

Thank you to the authors for the detailed rebuttal and the new comparison with Vim-S and MambaVision-B; I really appreciate your transparency in reporting the numbers. With matched zero-initialized heads, SpanInit improves mean accuracy by 0.56% on MambaVision-B but reduces it by 0.32% on Vim-S. Across the five additional fine-grained datasets, the mean gain is only 0.10. Given the modest, backbone-dependent gains and the limited methodological and analytical contributions, I therefore maintain my rating of Weak Reject.

AuthorsOpenReview ↗

Further Response to Reviewer y3jx

We thank the reviewer for the continued engagement and for appreciating the transparency of our reporting.

We agree that the aggregate gains are modest; SpanInit changes only the initial A values of the added states and adds no trainable parameters or inference cost. However, we respectfully disagree that the backbone dependence weakens the contribution. It is the central claim of the paper: the effect of initializing added states depends on the checkpoint's timescale geometry, and this geometry can be read from A alone before adaptation, without data or a forward pass. The dependence appears even within a single architecture. On MambaVision-B, SpanInit raises the 19-task mean on the ImageNet-21k checkpoint (+0.39) and lowers it on the ImageNet-1k checkpoint (−0.38). Vim-S is an ImageNet-1k checkpoint with a near-collapsed bank, with every state within a factor of two of |A| = 1. A decrease on Vim-S is therefore consistent with the paper's diagnosis rather than a counterexample to it.

The zero-initialized-head re-runs make this pattern clearer. The gain on MambaVision-B increases from +0.39 to +0.56 [+0.38, +0.74], and the decrease on Vim-S, previously not distinguishable from zero (−0.25 [−0.59, +0.09]), now excludes zero (−0.32 [−0.51, −0.13]). MambaVision-L2 (ImageNet-21k), whose wide span was already reported in the submission, was evaluated on VTAB-1k only afterwards and also improves (+1.14 [+0.63, +1.66]). Across the evaluated checkpoints, all three wide-span checkpoints improve and four of the five near-collapsed checkpoints decrease. The paper therefore motivates choosing between SpanInit and copying by the bank geometry, which can be measured before adaptation, rather than applying a single fixed rule.

Reviewer_1n4KOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the decay timescales available within individual channels of pretrained vision Mamba models and uses that analysis to initialize additional recurrent states for parameter-efficient adaptation. Since the input-dependent step is shared across a channel's state dimensions, their mean-step timescale ratios depend only on the pretrained diagonal state matrix. The authors examine nine released checkpoints and find a striking separation: the multi-state ImageNet-1k-only checkpoints have narrow within-channel spans, whereas the ImageNet-21k-exposed MambaVision checkpoints have much wider spans. They correctly distinguish this relative span from coverage of a finite image scan.

Critical strengths
  1. The comparison with Additional-scan keeps the trainable structure, parameter budget, added B/C initialization, and data order paired. This makes the measured differences informative about the effect of changing the initial A values. The parameter counts and the distinction between classification-head parameters and adaptation parameters are explicit.

  2. Equations (4)-(6) support the checkpoint-only relative-span calculation, while Tables 1-2 show why a broad spectrum need not cover the actual scan window. The contrast between approximately 3,527-fold and 1.67-fold spans in MambaVision-B is substantial. The paper also avoids claiming that a narrow bank is inherently a worse representation or that pretraining data alone caused the difference.

  3. Table 8 measures a 3-10 second calibration pass, corresponding to 0.06-0.51% of the reported run time, with no additional inference computation relative to Additional-scan. The discussion openly recognizes that uniform spacing explains much of the gain, that Specialized tasks can regress, and that the checkpoint comparison is associational.

Critical weaknesses
  • Table 4(b) and Table 16 show +0.30 points for uniform log-spacing versus +0.39 for SpanInit on MambaVision-B. Thus, the distinctive bank-aware component contributes only 0.09 points in the reported aggregate comparison. The paper supplies significance tests against copying, but no direct paired interval for SpanInit versus uniform spacing. I would like to see that comparison treated as a primary result and repeated on MambaVision-L and the collapsed checkpoints. Without it, the stronger supported methodological conclusion is that calibrated, diverse timescales can help this checkpoint; the advantage of filling its particular gaps remains uncertain. The authors acknowledge this limitation, but it limits the method contribution nonetheless.

  • Table 1 identifies backbone families and pretraining categories, but does not give exact checkpoint filenames, release revisions, or hashes. Its "at init." column also assumes the stated S4D spectra, and Section 4.2 interprets the final spectra as contraction from those initial values. Please document the actual initialization configuration for each release, the extracted state-dictionary keys, and successful loading of those tensors. This matters particularly when several unrelated checkpoints concentrate near |A| = 1: a final checkpoint robustly establishes its present geometry, but establishing how pretraining changed that geometry additionally requires its actual starting configuration.

Questions for authors
  1. What is the seed-paired confidence interval for SpanInit minus uniform log-spacing? Does the small advantage in Table 16 reproduce on MambaVision-L and on either ImageNet-1k-only checkpoint?

  2. Can you provide exact checkpoint identifiers, state-dictionary keys, and the initialization configurations underlying Table 1? Were the S4D starting spectra verified for every evaluated release, rather than inferred from the generic Mamba formulation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1n4K

We are glad the reviewer found the paired comparison with Additional-scan informative.

[W1/Q1] Seed-paired interval for SpanInit − uniform; MambaVision-L and ImageNet-1k.

In seed-paired runs (N′ = 6, 19 tasks × 3 seeds), the interval for SpanInit − uniform includes zero on every checkpoint:

CheckpointSpanInit − uniform [95% CI]Uniform − Additional-scan [95% CI]
MV-B, ImageNet-21k+0.09 [−0.13, +0.31]+0.30 [+0.11, +0.49]
MV-B, ImageNet-1k+0.10 [−0.19, +0.39]−0.48 [−0.74, −0.23]
MV-L, ImageNet-21k0.00 [−0.16, +0.15]+0.59 [+0.36, +0.81]
MV-L, ImageNet-1k−0.07 [−0.33, +0.18]−0.13 [−0.43, +0.16]

We agree with the reviewer's conclusion. The scan window [1, L] is fixed by the scan horizon, while mapping its timescales to the added A values uses a task-calibrated mean step. On ImageNet-21k checkpoints, uniform log-spacing recovers most of SpanInit's gain on MambaVision-B (+0.30 of +0.39) and all of it on MambaVision-L (+0.59). On the near-collapsed ImageNet-1k checkpoints, neither initialization improves the overall mean over Additional-scan. These results support calibrated timescale diversification on the evaluated wide-span checkpoints, but do not establish a benefit specific to bank-aware gap filling. We will put the direct paired intervals in the main text and revise the contribution claims to distinguish the checkpoint diagnosis, calibrated placement principle, and particular greedy gap-filling implementation.

[W2/Q2] Checkpoint identifiers, state-dict keys and initialization.

We will include a reproducibility manifest for every checkpoint in Table 1: exact filename/identifier, release URL or revision, computed SHA-256, code commit, extracted A tensor key and shape, initialization settings, and state-dict loading checks. According to our checkpoint audit, the analyzed files come from the official releases, load into the corresponding implementation, and reproduce the measured geometry; each file also matches its published SHA-256 where one exists. The appendix will make these checks inspectable rather than relying on a general statement of verification.

At the released code/configuration level, the examined MambaVision and Vim model-construction paths initialize A_log with log(1), …, log(N); the configured state dimensions are N = 8 for MambaVision and N = 16 for Vim, yielding nominal initial spans of 8× and 16×. This verifies the expected starting spectrum of the examined implementation paths, but cannot by itself establish the exact optimizer history of each released checkpoint. We will separate code-level initialization evidence, checkpoint-tensor measurements, and training-stage regularization evidence in the revised discussion.

Please let us know if our answers do not fully address your concerns.

The discussion deadline is approaching. Could you please respond to the authors’ rebuttal? Thanks!

Your AC

Reviewer_ZbVpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes SpanInit, an initialization method for adding state dimensions to pretrained Mamba backbones. The added states are fine-tuned while the pretrained backbone remains frozen. SpanInit places the initial values to fill the largest uncovered gaps, in log-timescale space, within a scan-relevant range. Experiments on VTAB-1k show that SpanInit can improve downstream performance over the existing Additional-scan initialization.

Critical strengths

The paper provides a detailed analysis of the timescale geometry in pretrained Mamba checkpoints and uses this analysis to motivate its initialization strategy.

Critical weaknesses
  1. The practical value and broader relevance of expanding the state dimensions of pretrained Mamba models for adaptation remain unclear. The paper studies a specific adaptation approach, but does not establish how useful this setting is compared with more general or commonly used parameter-efficient adaptation methods.
  2. The improvements from SpanInit are modest on average and vary across pretrained checkpoints, with gains on some checkpoints and declines on others. This limits the strength of the evidence for SpanInit as a generally useful initialization strategy.
Questions for authors
  1. Could the authors compare Additional-scan with other methods for adapting pretrained Mamba models, such as MambaPEFT, ideally under comparable parameter and training budgets? This would help clarify when explicitly adding state dimensions is beneficial or even necessary when adapting pre-trained Mamba to hard problems.
  2. Could the authors further characterize when SpanInit is likely to help or hurt? For example, can checkpoint properties predict the downstream effect, given that the method improves results on some checkpoints but reduces them on others?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZbVp

We thank the reviewer for appreciating the detailed analysis of timescale geometry in pretrained checkpoints.

[W1/Q1] Comparison with other PEFT methods at matched budgets.

State expansion is competitive at the smaller of the two adaptation budgets tested here, without being universally superior to LoRA. In seed-paired runs on MambaVision-B (ImageNet-21k, 19 tasks × 3 seeds), we compare against the LoRA_p(X) variant considered in MambaPEFT, matching the reported adaptation-parameter counts (excluding the task head) and training schedule:

Adaptation params (excluding head)Additional-scanSpanInitLoRA_p(X)
50,688 (N′ = 6 / rank 6)69.6069.9968.95
540,672 (N′ = 64 / rank 64)70.4371.0271.73

At 50,688 adaptation parameters (about 0.05% of the backbone), SpanInit exceeds LoRA_p(X) by +1.04 [+0.79, +1.28] points; at 540,672 parameters, LoRA_p(X) leads by +0.70 [+0.45, +0.95] points. These are parameter-matched, not necessarily FLOP- or memory-matched, comparisons. We conducted an exploratory grouping of the 19 VTAB-1k tasks into nine provisionally termed “order-dependent” and ten “order-independent” tasks. An exploratory task-group breakdown suggests that much of the small-budget advantage is concentrated in one task group:

Difference from LoRA_p(X) [95% CI]Exploratory group A (9 tasks)Exploratory group B (10 tasks)
50,688 params, Additional-scan+1.26 [+0.68, +1.83]+0.10 [−0.08, +0.28]
50,688 params, SpanInit+2.43 [+1.92, +2.93]−0.21 [−0.38, −0.04]
540,672 params, Additional-scan−1.55 [−2.17, −0.93]−1.06 [−1.35, −0.77]
540,672 params, SpanInit−0.06 [−0.57, +0.45]−1.28 [−1.44, −1.12]

Within exploratory group A, the estimated small-budget margin over LoRA_p(X) is larger for SpanInit than for Additional-scan (+2.43 versus +1.26 points), while at the larger budget the corresponding margins are −0.06 and −1.55 points. The methods can also be combined: at 591,360 reported adaptation parameters, LoRA_p(X) at rank 64 plus six SpanInit states exceeds LoRA_p(X) at rank 70 by +0.72 [+0.49, +0.95] points. We will add these results to Section 6 and report the exact parameter accounting and training-cost setup in the appendix; we will avoid claiming that state expansion is necessary for PEFT.

[W2/Q2] Relation between checkpoint properties and the effect of SpanInit.

We agree that SpanInit is not a universal improvement. Across the evaluated checkpoints, the within-channel span of pretrained A is associated with the sign of the aggregate accuracy change. This span can be computed from frozen weights before downstream adaptation, although the current observational comparison does not establish an independently validated predictor or a causal role for span:

CheckpointWithin-channel spanSpanInit − Additional-scan
MV-B / L / L2†, ImageNet-21k3,527× / 3,335× / 2,590×+0.39 / +0.59 / +1.14
MV-B / L / L2†, ImageNet-1k1.67× / 1.43× / 1.37×−0.38 / −0.21 / −1.05
Vim-S / Vim-B, ImageNet-1k1.31× / 1.22×−0.25 / +0.17

† SpanInit − Additional-scan on MambaVision-L2 is from new runs (19 tasks × 3 seeds).

All three evaluated wide-span MambaVision checkpoints improve, while four of five near-collapsed checkpoints decline (the Vim-B effect is +0.17 and statistically inconclusive in the original analysis). This small, partially related checkpoint set suggests a possible checkpoint-conditioned choice between SpanInit and copying, but does not define or validate a universal span threshold. The exploratory task grouping also reveals heterogeneity within each checkpoint (e.g., +1.17 versus −0.31 points on ImageNet-21k MambaVision-B). We will visualize the observed association in Section 5 as a decision hypothesis, distinguishing “which initialization to use for expansion” from “whether to expand the model.”

[Presentation]

We agree that the paper is hard to follow in places. In the revision we will:

  • Add a diagnostic/decision-hypothesis figure at the start of Section 5 (pretrained span → candidate initialization choice → placement of added states), explicitly marked as requiring validation on unseen checkpoints.
  • Define the timescale, the within-channel span, the scan window [1, L] and its coverage together in Section 4.1, before they are used.
  • Move Algorithm 1 and the calibration cost (Table 8) from the appendix into Section 5.
  • Move the per-task Table 3 to the appendix and keep its Natural, Specialized, and Structured means in the main text.

Please let us know if our answers do not fully address your concerns.

We thank the authors for their response. Considering the manuscript and response, the proposed method shows consistent performance gains when compared with other methods that expand state dimensions in a post-training way, but the gains remain relatively small. When compared to PEFT methods, for small expansions, the performance varies (with PEFT outperforming the proposed method on some tasks and vice versa), while for large adaptation parameters, LoRA performs better. Although the proposed method introduces a new initialization technique, it does not seem to resolve the practical bottleneck of adapting a pretrained Mamba to hard problems. We therefore maintain our original rating.

AuthorsOpenReview ↗

Response to Reviewer ZbVp: Clarification of the Remaining Concerns

We thank the reviewer for the follow-up assessment. However, we respectfully disagree that the evidence presented in our rebuttal has been adequately reflected in the final evaluation. In particular, we would appreciate clarification on which specific technical concerns remain unresolved after the additional experiments addressing both original questions.

  1. The evaluation appears to conflate the scope of our contribution with the broader problem of Mamba adaptation.

SpanInit is not proposed as a universal replacement for PEFT methods, nor do we claim that state expansion must outperform LoRA at every adaptation budget. Our contribution concerns a more specific question: given that additional state dimensions are introduced into a frozen pretrained Mamba backbone, how should these states be initialized to complement the pretrained timescale distribution?

The observation that LoRA performs better at a larger parameter budget does not invalidate the contribution of a checkpoint-aware initialization strategy. The relevant evaluation should distinguish the effectiveness of the proposed initialization within state expansion from the broader competitiveness of state expansion as an adaptation mechanism.

  1. The new PEFT comparisons provide concrete evidence of practical value that is not fully reflected in the follow-up assessment.

At 50,688 adaptation parameters, SpanInit achieves 69.99% compared with 68.95% for LoRA_p(X), a difference of +1.04 percentage points (95% CI: [+0.79, +1.28]). We explicitly acknowledge that LoRA_p(X) performs better at the larger budget.

Furthermore, combining SpanInit with LoRA_p(X) improves performance over a higher-rank LoRA_p(X) baseline by +0.72 points (95% CI: [+0.49, +0.95]) under the reported parameter-matched setup. This demonstrates that state expansion can provide complementary benefits rather than merely competing with conventional PEFT.

These findings do not establish universal superiority, but they directly address the reviewer's original question concerning the practical relevance of state expansion.

  1. The checkpoint analysis directly addresses the reviewer's second question but is not discussed in the follow-up assessment.

Our additional analysis shows that SpanInit improves performance on all three evaluated wide-span MambaVision checkpoints, whereas four of five near-collapsed checkpoints exhibit declines. This identifies an observable checkpoint characteristic associated with when the proposed initialization helps or hurts.

We have explicitly acknowledged that this association is preliminary and does not establish a causal relationship or a universally validated selection rule. Nevertheless, it provides a concrete, pretrained-weight-based explanation for the performance heterogeneity raised in the original review.

We would therefore appreciate clarification on why this additional evidence is considered insufficient to address the original concern.

  1. Clarification of the basis for maintaining the original rating.

We respect the reviewer's assessment and acknowledge the limitations of the proposed method, including its modest average gains. However, the follow-up assessment largely reiterates these limitations without specifying why the new parameter-matched comparisons, complementary PEFT results, and checkpoint-conditioned analysis fail to address the original questions.

We would appreciate a more specific technical explanation of the remaining deficiencies, particularly whether the concern is insufficient empirical evidence for the initialization strategy itself or a requirement to establish the superiority of state expansion over general PEFT methods.

We believe these are distinct evaluation criteria, and clarifying this distinction is important for a fair assessment of the paper's actual contribution.

We appreciate the authors’ careful responses and their clarification of the settings in which SpanInit is effective.

Our request for a PEFT comparison reflected a broader concern: whether the paper addresses an important bottleneck in the research community and offers a meaningful solution to it. The added 50K and 540K parameters account for only about 0.05% and 0.5% of the full model, respectively, while the largest gains reported in the response come from LoRA. Based on these results, we would choose LoRA in practice. This left us unsure whether the problem, as framed around Additional-Scan, represents a significant bottleneck for Mamba post-training. We do not think further experiments are needed to address this concern.

Setting that concern aside, we can frame the main question as how SpanInit improves on Additional-Scan. SpanInit’s central contribution is to define a scan-relevant window and specify how to initially place the added states within it. Tables 4(b) and 4(c) illustrate this contribution. In this comparison, the differences from an initialization that Additional-Scan users might choose without a specific initialization strategy are marginal, at less than 0.1 on the reported metrics. We therefore remain uncertain that SpanInit offers a substantial improvement over Additional-Scan or addresses a sufficiently important problem to make a significant contribution to the research community.

Reviewer_bBshOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the decay-timescale structure of pretrained Mamba vision backbones and proposes SpanInit, which initializes additional states by filling gaps in log-timescale space. The authors observe near-collapsed banks in several ImageNet-1k checkpoints and broader banks in ImageNet-21k-pretrained MambaVision models. SpanInit improves mean VTAB-1k accuracy on ImageNet-21k-pretrained MambaVision-B/L but lowers it on their ImageNet-1k-only counterparts.

Critical strengths

The distinction between timescale span and coverage of the scan window is useful. The checkpoint analysis is interesting, and the comparison with Additional-scan is well controlled: the methods differ only in the initial values of the added AA entries. SpanInit has little calibration overhead and no additional inference cost relative to Additional-scan. I also appreciate the authors’ clear reporting of negative results and limitations.

Critical weaknesses
  1. I am not yet convinced that bank-aware gap filling adds much over simpler choices. On ImageNet-21k-pretrained MambaVision-B, uniform log-spacing improves the mean by 0.30 percentage points versus 0.39 for SpanInit, without a reported paired confidence interval for their difference. Comparisons with copying the longest-timescale entry or several different pretrained entries would help clarify the specific benefit of the proposed placement rule.

  2. The collapse observation is interesting, but its implications for adaptation remain unclear. The reported mean changes are negative for the ImageNet-1k MambaVision checkpoints, while the Vim results are statistically inconclusive. Tasks within the same checkpoint also respond differently. Although the authors acknowledge these limitations, I would still like a clearer explanation of when filling timescale gaps helps and when it hurts.

I lean toward a weak reject. The observation is promising, but its implications and the independent benefit of SpanInit need stronger support.

Questions for authors
  1. My main concern is whether bank-aware gap filling provides a meaningful benefit over simpler initialization choices. Could you report a paired confidence interval for SpanInit versus uniform log-spacing, and compare against copying the longest-timescale entry or several different pretrained entries? Evidence of a consistent advantage in these comparisons would substantially address this concern.

  2. Could you provide a more concrete explanation of when filling timescale gaps helps or hurts, perhaps through a focused comparison of dSprites-Loc and dSprites-Ori? An analysis of the added states’ learned timescales or contribution horizons could help connect the geometric observations to downstream behavior. A supported interpretation of representative cases would be useful, even without a complete causal account.

The first question is the more important one for my recommendation. I would be open to revising it if the additional comparisons establish the specific value of SpanInit, with the second analysis helping clarify its interpretation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bBsh (1/2)

We are glad the reviewer found the checkpoint analysis interesting and the comparison with Additional-scan well controlled.

[W1/Q1] Paired comparison with uniform log-spacing and copying baselines.

In seed-paired runs on MambaVision-B (ImageNet-21k, N′ = 6, 19 tasks × 3 seeds), SpanInit outperforms every copying baseline, including the two suggested by the reviewer, while its estimated advantage over uniform log-spacing is not statistically established:

SpanInit vs.Difference [95% CI]
Uniform log-spacing+0.09 [−0.13, +0.31]
Additional-scan (copy the first entry)+0.39 [+0.20, +0.59]
Copy the longest-timescale entry+0.31 [+0.06, +0.57]
Copy a middle-timescale entry+1.93 [+1.68, +2.18]
Copy the shortest-timescale entry+5.12 [+4.92, +5.32]
Copy a random entry per channel+1.32 [+0.99, +1.66]
Copy six different pretrained entries+0.90 [+0.70, +1.11]

These comparisons help separate timescale diversification from bank-aware placement. The scan-relevant window [1, L] is defined by the scan length; the checkpoint- and task-dependent mean step is used to map target timescales in that window to initial A values. Uniform log-spacing ignores the individual pretrained anchors yet recovers most of the improvement over Additional-scan (+0.30 of +0.39). Copying six different pretrained entries, about half of which lie outside [1, L], is worse than Additional-scan (−0.51 [−0.73, −0.29]). For the evaluated ImageNet-21k MambaVision-B checkpoint, these results support the value of initializing distinct decay scales inside the scan window [1, L], rather than simply copying pretrained entries. Bank-aware placement reduces near-duplicates (14% of new timescales lie within 1.5× of a pretrained timescale, versus 35% for uniform log-spacing); however, this geometric difference does not yield a statistically established accuracy advantage over uniform log-spacing.

We will add this direct comparison to Table 4(b) and distinguish three contributions in the revision: (i) diagnosing checkpoint-specific timescale geometry, (ii) calibrating diverse added-state timescales to the finite scan horizon, and (iii) implementing this principle with bank-aware gap filling. We will explicitly acknowledge that the incremental benefit of (iii) over uniform log-spacing remains unproven.

AuthorsOpenReview ↗

Response to Reviewer bBsh (2/2)

[W2/Q2] When filling gaps helps or hurts; dSprites-Loc vs. dSprites-Ori.

We conducted an exploratory grouping of the 19 VTAB-1k tasks into nine provisionally termed “order-dependent” and ten “order-independent” tasks. The labels are descriptive and should not be taken as an independently validated measure of scan-order dependence; the revision will enumerate all task assignments and the classification criterion. In the reported group averages, SpanInit tends to improve the former group and to lower the latter. The size of both effects depends on the checkpoint (a smaller gain and a larger loss on the near-collapsed ImageNet-1k checkpoints), but this does not yet support a causal explanation or an unconditional choice of initialization for an unseen checkpoint. We report the results across expansion sizes and MambaVision-B and -L below:

SpanInit − Additional-scanExploratory group A (9 tasks)Exploratory group B (10 tasks)
MV-B, ImageNet-21k, N′ = 6+1.17 [+0.79, +1.55]−0.31 [−0.48, −0.14]
MV-B, ImageNet-21k, N′ = 1–8 except 6+1.30 [+1.02, +1.59]−0.30 [−0.42, −0.17]
MV-L, ImageNet-21k, N′ = 6+1.34 [+1.02, +1.65]−0.09 [−0.27, +0.09]
MV-B, ImageNet-1k (near-collapsed), N′ = 6+0.60 [+0.17, +1.04]−1.27 [−1.63, −0.91]
MV-L, ImageNet-1k (near-collapsed), N′ = 6+0.71 [+0.24, +1.17]−1.02 [−1.31, −0.74]

For the pooled N′ = 1–8 comparison, the revised appendix will clarify how dependence across expansion settings, tasks, and paired seeds is handled when constructing uncertainty intervals.

Following the reviewer's suggestion, we examined dSprites-Loc and dSprites-Ori as a focused contrast between two tasks constructed from the same image family but with different prediction targets. With matched calibration inputs, the measured step and initial timescale assignment are the same. The reported median change in added-state timescales during adaptation is modest (×1.1–1.2), while the selective B and C terms may still change functional contributions. On the evaluated images, the reported contributions are distributed across timescale bands as follows for MambaVision-B (ImageNet-21k, 3 seeds per task):

Added statesBelow 1 tokenInside [1, L]Beyond L
Additional-scan, trained on dSprites-Loc10.2%58.1%31.7%
Additional-scan, trained on dSprites-Ori10.6%48.7%40.7%
SpanInit, trained on dSprites-Loc / -Ori0.1% / 0.7%99.7% / 98.9%0.1% / 0.4%

The appendix will state precisely how these contribution percentages are aggregated and normalized. Since SpanInit explicitly places new timescales within [1, L], concentration in this band is partly expected by construction and should not, by itself, be interpreted as evidence of task-specific causal importance.

These observations suggest that dSprites-Loc and dSprites-Ori may differ in how they utilize recurrent information across scan positions. They are consistent with task-dependent timescale requirements, but do not establish that orientation is order-independent or that any particular horizon causes the observed accuracy change. We will present this as a mechanistic hypothesis and a descriptive contribution analysis, not a causal explanation; causal horizon-ablation experiments remain a useful next step.

Please let us know if our answers do not fully address your concerns.

Thank you for the detailed response and new baselines. While the empirical evaluation is improved and your transparent reporting is appreciated, my core concern regarding the overall significance of the contribution remains:

  1. Marginal specific benefit: SpanInit is a modest modification. Uniform log-spacing already captures almost all the reported gains, and the paired difference of +0.09 fails to justify the specific value of the proposed placement rule.

  2. Descriptive analysis: The checkpoint and task analyses, while interesting, remain primarily descriptive. They do not offer sufficient insight into the connection between timescale geometry and downstream adaptation.

Given the limited methodological and analytical contributions, the paper does not meet the ICLR acceptance threshold. I therefore maintain my rating of 2: Weak Reject.

AuthorsOpenReview ↗

Further Response to Reviewer bBsh

We thank the reviewer for the continued engagement. We respond to both points, since the new analysis below also bears on the first.

  1. Specific benefit of SpanInit

We respectfully disagree that the +0.09 difference to uniform log-spacing undermines the contribution. In the initial review, the reviewer indicated that a consistent advantage over copying-based initializations would substantially address this concern. SpanInit outperforms all six copying-based initializations, including the two the reviewer suggested, with every 95% CI excluding zero (+0.31 to +5.12).

Uniform log-spacing, as evaluated, is not a pre-existing baseline. It relies on the two ingredients introduced by our analysis: the scan-relevant window [1, L] and the measured mean step that maps target timescales to initial A values. It ablates the bank-aware placement rule, not the principle. The comparisons separate the two: copying six distinct pretrained entries, which diversifies timescales without calibrating them to the scan length, is worse than Additional-scan (−0.51 [−0.73, −0.29]), while both horizon-calibrated variants improve on it (+0.30 and +0.39). Diversity alone does not help; calibration to the scan horizon does. That uniform log-spacing recovers most of the gain therefore supports our central claim rather than weakening it.

We agree that the bank-aware rule itself is not shown to be superior. In the revision, we will present horizon-calibrated initialization as the method, with uniform log-spacing and bank-aware gap filling as two instantiations, and describe bank-aware placement only as reducing near-duplicate timescales (14% vs. 35%).

  1. Connection between timescale geometry and downstream adaptation

To go beyond descriptive analysis, we ran the horizon ablation noted in our previous response. For the Additional-scan models trained on dSprites-Loc and dSprites-Ori (MambaVision-B, ImageNet-21k, 3 seeds each), we set to zero the output contribution of added states whose timescale (measured as in our contribution analysis) lies inside [1, L] or beyond L. In each block, we removed the same number of states from each band (equal to the number of states beyond L in that block, 16–17% of all added states; sampled at random within the band; mean of three draws) and evaluated on the full test set without retraining.

Accuracy drop in points (mean over 3 seeds; unablated accuracy 67.0 / 53.6):

Inside [1, L]Beyond LAll added states
dSprites-Loc37.629.859.2
dSprites-Ori4.719.535.1

The ordering holds in each of the 3 seeds. Relative to removing all added states, removing states beyond L costs a similar share on both tasks (50% and 56%), whereas removing states inside [1, L] costs 64% on dSprites-Loc but only 13% on dSprites-Ori. The contribution analysis in our previous response could not have revealed this: on dSprites-Ori, states inside [1, L] account for 48.7% of the added-state contribution, yet removing about 40% of them costs only 4.7 points. The ablation thus measures functional reliance rather than contribution magnitude.

This difference matches the downstream effect of SpanInit, which places new timescales inside [1, L]: it improves dSprites-Loc by 3.91 points and leaves dSprites-Ori unchanged (−0.02). For this pair, the benefit of horizon-calibrated initialization follows the task's reliance on timescales inside the scan length. We will add these results to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.