Causal-Role Recurrence, Geometric Variation, and Objective-Conditioned Readout Across Mamba-1 Scales
Abstract
Mechanistic comparisons across model scales can conflate three questions: whether independently reconstructed components play an operationally matched causal role, whether their native geometry is preserved, and how an objective reads them. We separate these questions in pretrained Mamba-1 models from 130M to 2.8B parameters. At 130M, prospective transport, response-independent specificity, matched necessity, and restoration establish a recurrent-state mechanism on disjoint populations. At larger scales, a frozen discover-then-confirm protocol selects one of five local planes on a discovery cohort, fixes a response-blind geometric control, and evaluates the pair on a disjoint confirmation cohort. The selected-restored versus matched-control contrast is positive under each scale's frozen confirmation, while a distinct historical 370M residual criterion remains failed. Coordinate-insensitive comparison of matched response-blind XG2/XG4 populations yields heterogeneous, intermediate cross-scale similarity (centered linear CKA: XG2 0.34–0.74; XG4 0.39–0.59), with the same qualitative pattern under cosine-RSM. Across scales, the frozen task contrasts reorganize functionally: no single geometric scalar explains the aggregate sign pattern. Holding each scale's contrast fixed, the pretrained next-token objective gives the opposed five-scale point-estimate vector. Pair-resampling intervals exclude zero in 8 of 10 scale-by-objective cells, and a 1.4B sensitivity contrast flips the pretrained-LM sign, so the objective comparison remains contrast-conditioned. Thus a scale-local interventional role can recur without shared plane identity or exact geometric invariance, while functional readout depends on both objective and the frozen native contrast. We frame this as a controlled Mamba-1 case study rather than a universal scaling law.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.