acceptodds
Under review as a conference paper at ICLR 2027

Causal-Role Recurrence, Geometric Variation, and Objective-Conditioned Readout Across Mamba-1 Scales

Abstract

Mechanistic comparisons across model scales can conflate three questions: whether independently reconstructed components play an operationally matched causal role, whether their native geometry is preserved, and how an objective reads them. We separate these questions in pretrained Mamba-1 models from 130M to 2.8B parameters. At 130M, prospective transport, response-independent specificity, matched necessity, and restoration establish a recurrent-state mechanism on disjoint populations. At larger scales, a frozen discover-then-confirm protocol selects one of five local planes on a discovery cohort, fixes a response-blind geometric control, and evaluates the pair on a disjoint confirmation cohort. The selected-restored versus matched-control contrast is positive under each scale's frozen confirmation, while a distinct historical 370M residual criterion remains failed. Coordinate-insensitive comparison of matched response-blind XG2/XG4 populations yields heterogeneous, intermediate cross-scale similarity (centered linear CKA: XG2 0.34–0.74; XG4 0.39–0.59), with the same qualitative pattern under cosine-RSM. Across scales, the frozen task contrasts reorganize functionally: no single geometric scalar explains the aggregate sign pattern. Holding each scale's contrast fixed, the pretrained next-token objective gives the opposed five-scale point-estimate vector. Pair-resampling intervals exclude zero in 8 of 10 scale-by-objective cells, and a 1.4B sensitivity contrast flips the pretrained-LM sign, so the objective comparison remains contrast-conditioned. Thus a scale-local interventional role can recur without shared plane identity or exact geometric invariance, while functional readout depends on both objective and the frozen native contrast. We frame this as a controlled Mamba-1 case study rather than a universal scaling law.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, Mamba, state-space models, causal interventions, representation similarity, cross-scale analysis, objective-conditioned representations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_YuR6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper compares the effect of interventions across five Mamba-1 models of different parameter sizes. The authors construct synthetic texts to construct two-dimensional subspaces, testing whether restoring a selected component affects the model's internal sensitivity compared to a control. They successfully discover such a component at each of the model sizes, although they are selected independently and are not compared semantically across models. Comparing them across models shows partial similarity geometrically, and gradient analyses show that the component's effect can change based on how it's evaluated.

Critical strengths

None

Critical weaknesses
  • The main experiment is not sufficiently specified. The paper doesn't clearly explain what XG2/XG4 are, what the six-cell substitution design is, or how they constructed the candidate planes.
  • Many parts of the paper, including figures, seem AI-generated with little human iteration, making it completely unreadable. For example, many parts of the abstract are cryptic and difficult to decipher: "prospective transport, response-independent specificity, matched necessity, and restoration establish a recurrent-state mechanism on disjoint populations", "Coordinate-insensitive comparison of matched response-blind XG2/XG4 populations yields heterogeneous, intermediate cross-scale similarity", "a scale-local interventional role can recur without shared plane identity or exact geometric invariance". The acronyms XG2/XG4 are only defined later in the paper as "disjoint response-blind generator families" (which doesn't really help much), making them confusing when they appear in the abstract. Furthermore, Figure 1, which typically summarizes the paper's main result or contribution in a visually appealing way, is completely unintelligible to me: what is the relationship and significance of the vague phrases "controlled contrasts, ..., task-margin consequence" labeled "native susceptibility mechanism" and "task-functional bridge", what P3/P5 notations mean, what the arbitrary plus/minus signs in section C mean. These presentation issues, which continue throughout the rest of the paper, make it very difficult for a human to understand and assess the contribution of the paper. Therefore, the paper's current presentation does not meet the threshold for ICLR and should receive a clear rejection.
Questions for authors

Could the authors please describe the main experiment concretely, including an example of an XG2/XG4 input pair, what the six cells in the design represent, and the steps used to construct the five candidate planes?

Reviewer_2obAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript studies the inner mechanism for the causal role in Mamba-1. The authors present three concepts: causal-role recurrence, geometric conservation, and objective-conditioned readout. Using five pretrained Mamba-1 models, the authors discover the inner components and mechanism, passing a held-out intervention test.

Critical strengths

The experimental design is sophisticated and controlled to obtain the clean results. The held-out intervention test using other Mamba-1 models is solid to confirm the discovery.

Critical weaknesses
  • Too many bold-face, italics, quotations, and parentheses, which is common in AI writing. Too much detail in SHA256 information (page 16) and too many implementation details with the model version (page 12). Figures are absolutely AI-generated; they contain artifacts such as “TASK_MATCHED” and are too fancy. Other evidence of AI writing includes “transcribed without figure-builder recomputation”, “the frozen causal synthesis”, “these training-report details are not inferred for it” and others. Too little reference with 13 entries, which is not common in ICLR. Too long a title, too many paragraphs in the abstract. Overall, this manuscript is far from ordinary ICLR paper and contains strange AI writing artifacts. I am absolutely sure that this manuscript is fully written in AI automation without sufficient human inspection. The use of AI writing can be acceptable in general, but this manuscript uses AI writing extensively without human effort to deliver clean academic paper.
  • The core finding is mainly on Q, the difference of squared susceptibility for two synthetic generator families. Although the authors consistently observed this, the definition is still limited to the Q term, which is not ensured to be semantic equivalence. In other words, different semantic components can still pass the test in practice.
  • Appendix A.4 says that 370M fails on the same test protocol. Is it possible to obtain consistent results with the same test protocol?
  • Regarding the control-plane coefficient transfer, the authors apply an orthonormal basis to match the norm. Nevertheless, an orthonormal basis can be arbitrarily chosen, and multiple choice is allowed, which yields different, inconsistent vectors. Can these control-plane coefficient transfers guarantee fixed direction?
  • The CKA analysis says that the geometry is not conserved. However, the used CKA formulation can increase depending on other factors such as the data dimension and structure of the Gram matrix. Independently, higher dimensions will increase the CKA, and this CKA computation should be controlled to ensure correct measurement.
  • Pages 9 and 12 say amplifying gradient twice in Tables 5 and 6, which should be explained.
  • Eq. 1 in page 13 inherently assumes linear J(w) with respect to w, which should be verified.
  • Writing should be improved.
    • “TASK_MATCHED” → should be explained.
    • “A_IDENTITY” → should be explained. Possibly AI writing.
    • c_i on page 15 should be defined explicitly.
    • “two plotted quantities” → three, right?
Questions for authors

See the weakness above.

Reviewer_cR39OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper mechanistically compares five sizes of Mamba-1 models. At each size, the manuscript appears to identify five principal planes (PkP_k/PPkPP_k) that encode semantic meaning across architectures. Using this, the effect of interventions can be compared in three "coordinate systems".

    1. the state space itself
    1. in reference to the principal planes
    1. in reference to the current task/objective (some downstream task vs. next token prediction)

It seems that the principal planes allow for a more stable interpretation across scales, i.e. effects can be better reproduced compared to looking just at the state space, and that this effect depends on the task that is studied, but that is all i can make out.

Critical strengths

None.

Critical weaknesses

Summary

I struggle to describe this manuscript as anything but unreadable. This concerns not only the language in general (which is so full of meaningless jargon and undefined technical terms that it is hard to make out the actual meaning of most sentences), but the manuscript present results based on abbreviated objects that are never defined. This renders numerical results completely meaningless.

Due to these issues, I truly struggle to understand the paper's research question.

W1: Language full of unnecessary technical Jargon

Most parts of the paper use words that bear no meaning to me without an actual technical definition. I had to read the first paragraph of the introduction multiple times and refer to the examples below.

W2: Objects that are central to the actual results are never defined

I looked everywhere, but I do not understand what XG1-XG4 are (where is XG3), what PkP_k and PPkPP_k are and where P1P_1 is, as well as, e.g. the "downstream task" and "point estimates" that are referred to frequently. The manuscript really dances around these terms but never defines any of them.

Without these definitions, how is a reader supposed to even interpret any of the numbers?

But even so, what is possible to interpret makes me seriously doubt the experimental rigour; some sign results are repeatedly reported as such, and these "sign effects" seem to be almost 0.

W3: I cannot make out the research question.

Because of these issues in the presentation, there is really no interpretation of the manuscript I can come up with that makes sense to me. I see there is some mechanistic property being studied, but it is not clear what (mathematically), why (would I separate these things), and how.

Examples of Unexplained Jargon and Phrases

What is the difference between PkP_k and PPkPP_k? What is a principal plane, mathematically?

Figure 1: What is the meaning of "supported"?

What is a "component"?

What is the "downstream task"? Why not name it?

Figure 1: What is the meaning of A? None of the phrases has meaning to me.

....

Questions for authors

Q1: Meaning and Definition of XG1-XG4

What precisely are XG1,XG2,XG3 and XG4? They are defined as "geometry generator families" and "holdout family"; neither term means anything to me. If they are some kind of data generators, how precisely are they defined? Why does XG3 not appear in the manuscript?

Q2: Meaning and Definition of PkP_k/PPkPP_k.

What is the difference between PkP_k and PPkPP_k? What is a principal plane, mathematically? Why does P1P_1 not appear in the manuscript?

Q3: Meaning and Definition of Point Estimate Signs.

What are the point-estimate signs? How are they defined mathematically? Why do we care about their signs? Why are the signs reported when Figure 4 A,B show that two of these signs are not even significantly nonzero? (absolute) Spearman Correlation is <.1< .1 for three out of 5 sizes. If confidence intervals include 0 and correlations are negligible, why report sign trends? This seems questionable.

Q4: Can you give a Layperson TL;DR of your Manuscript?

In as few sentences as possible, can you explain what the objective is in a way that tries to define all concepts that are referred to?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.