acceptodds
Under review as a conference paper at ICLR 2027

ReCompose: Diagnosing and Improving Conversion to Sparse Transformer-SSM Hybrids

Abstract

Converting a pretrained Transformer into a sparse hybrid replaces most attention modules with state-space models (SSMs), retaining only a few attention layers. We present **ReCompose**, a diagnosis-guided approach to recovering capabilities lost during conversion. We examine how replacement supervision, attention-layer selection, and joint training affect capability loss and recovery within existing conversion pipelines. The repair trains SSM replacements and their output projections to match complete teacher-layer outputs. It then jointly trains consecutive student layers to match teacher block outputs before whole-model distillation. The local and blockwise stages share a design choice: match the output passed onward while allowing the trained components to adapt within that computation. Controlled replacement experiments support the combined local repair through improved downstream accuracy. In a same-start Qwen comparison at matched post-assembly data exposure, blockwise-then-global training yields 0.6–1.0 percentage points higher MMLU accuracy, while direct global distillation yields higher Lambada accuracy at the tested endpoints. MMLU-selected layouts retain higher MMLU accuracy and weaker retrieval than retrieval-selected layouts. Subsequent training partly recovers lost capabilities with the selected attention layers held fixed. On Qwen and Llama, ReCompose recovers performance across knowledge, commonsense, and text-completion benchmarks while retaining only four attention layers. These results inform how to supervise replacements, select retained layers, and allocate training to recover the intended capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid language models, state-space models, model conversion, knowledge distillation, intermediate-state supervision, blockwise alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1DzFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at how to recover capabilities lost when most attention layers in a pretrained Transformer are replaced with SSMs. It proposes ReCompose: train each replacement to match the full teacher-layer output, jointly train groups of consecutive layers, and then distill the whole model. Its main contribution is diagnosing how local supervision, cross-layer training, and the choice of retained attention layers affect recovery. Experiments on Qwen and Llama with only four attention layers show improvements across knowledge, commonsense, and text completion tasks. In a controlled Qwen comparison, staged training improves MMLU by 0.6–1.0 points over direct distillation at matched data exposure, though direct distillation does better on Lambada. The paper also finds that knowledge and retrieval favor different attention layouts, and that harder retrieval and binding failures remain.

Critical strengths
  1. The paper connects its diagnosis to concrete training improvements. It separates readout trainability from supervision target choice in controlled local experiments, then tests their combined downstream benefit. The same-start comparison of staged versus direct distillation also helps establish when blockwise training is useful. This makes the proposed recipe more convincing than a pipeline supported only by final benchmark scores.
  2. The evaluation examines both capability recovery and its limits. Beyond standard benchmarks on Qwen and Llama, the paper tests attention-layout trade-offs, evidence position, and harder fact-binding cases. These tests show that improved aggregate scores can coexist with substantial failures, which makes the conclusions more credible and useful for deciding how to convert and evaluate hybrid models.
Critical weaknesses

(1) The benefit of retaining a few attention layers is unclear. The paper includes a pure-SSM regional training experiment, but it does not compare pure-SSM and hybrid models after the same full training process with comparable budgets. Without this baseline, it is hard to tell whether partial replacement is better than replacing all attention layers.

((2) The staged-training ablation is useful, but the results do not show a clear advantage. Compared with direct global distillation, staged training improves MMLU by only 0.6–1.0 points and consistently performs worse on Lambada. With only one run per setting and no compute-matched comparison, I am not convinced that the extra blockwise stage is necessary.

Questions for authors

(1) Why do you choose the qwen2.5 1.5B as the base model. If you want to train a small model you can use qwen3/3.5. The base model is too weak. (2) Why use staged training instead of direct global distillation? The ablation shows a small MMLU gain but lower Lambada accuracy, so the benefit of adding the blockwise stage is unclear. (3) The experiment testing readout adaptation and full-layer supervision replaces only 6 of 28 attention layers with SSMs, while the main model replaces 24. Do these two changes still improve performance when 24 layers are replaced?

2 comments

We thank the reviewer for recognizing that the paper connects its diagnosis to concrete training choices and examines both recovery and its limits. Our contribution is this diagnosis rather than a new method: each component of ReCompose has precedent (Section 2), and staged training is one of the choices we diagnose, not a step we claim is always better.

W1 (retaining attention). Assembled from the same layerwise-distilled Llamba-8B bank, keeping four attention layers raises MMLU from 34.1 with all 32 layers replaced (Appendix E.3) to 58.7 before any recovery training, and to 60.5 after recovery (Table 4); Section 4.2 now states this comparison. In the recovered model, zeroing those four attention outputs drops nine-candidate binding accuracy from 50.4% to 10.6%, the 11.1% chance level (Appendix J.2). The paper takes a small attention budget as given, as current hybrid conversions do, and studies recovery under it.

W2 / Q2 (staged training). The blockwise stage is not extra training: six blockwise rounds replace six of the nine global rounds at matched data exposure. It targets a specific mismatch: replacements are prepared on teacher inputs but receive student outputs after assembly. From the same assembly at rounds 7/8/9, staging gives MMLU 50.5/50.3/50.7 versus 49.5/49.7/50.0 and Lambada 46.8/48.7/49.8 versus 50.9/50.7/51.0 (Table 3), at about 1.2× the recovery compute. We therefore report an allocation trade-off favoring MMLU, which is itself a diagnostic finding (Section 6.2). For the single-run concern, we are training a second staged/direct pair from a fresh assembly and re-scoring the Table 3 endpoints with FP32 option scores and paired intervals.

Q1 (Qwen2.5-1.5B). We needed a pure Transformer small enough for many controlled conversions with one shared teacher. Dense Qwen3 remains a pure Transformer and a natural next teacher [1]; Qwen3.5 already interleaves Gated DeltaNet layers with attention [2], so converting it starts from a hybrid. For a stronger teacher, the Llama-3.1-8B study (teacher MMLU 68.3) raises Lambada from 26.5 to 65.2 (Table 4).

Q3 (24 replacements). We are running this comparison at the 4/28 layout: conventional (frozen readout, attention-output target) versus ReCompose local preparation for all 24 replacements, followed by identical recovery. At six replacements, the combined change raises MMLU by 1.32 points (95% CI [0.91, 1.75]) and HellaSwag by 0.49 ([0.21, 0.78]) (Table 10). We will post these results in this thread.

[1] Qwen3-4B official configuration [2] Qwen3.5-4B official model card

Thanks for the clarification. I now understand that the paper’s main contribution is diagnosing how different training choices affect recovery. My main concern is that the six-layer experiment shows a benefit from combining readout adaptation with full-layer supervision, but it’s still unclear whether that benefit carries over to the main setting, where 24 layers are replaced. The main results show that the full pipeline recovers some capabilities, but they don’t isolate the contribution of those two changes. The comparison you’re running, with the same recovery procedure in both conditions, would directly address this concern. I’ll keep my current rating for now and would reconsider it if the new results show a clear benefit in the 24-layer setting.

Reviewer_sEu9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to preserve and recover model capabilities when converting a pretrained Transformer into a hybrid with mostly state-space model (SSM) layers and only a few softmax attention layers. It proposes ReCompose, which (1) trains each SSM replacement and its output projection to match the teacher’s complete layer output, (2) trains groups of consecutive student layers to match teacher block outputs, and (3) distills the full student model from the teacher. The authors also study how the choice of retained attention layers affects different capabilities.

Critical strengths
  1. A controlled study of conversion choices that pipelines usually entangle. Rather than evaluating one monolithic pipeline, the paper separates the supervision target, readout trainability, training allocation and layer selection. Several comparisons start from the same model at matched data exposure, and many report paired intervals.. Component-level evidence of this kind is largely missing from the conversion literature, even though the individual effects here are small (see W2).
  2. Capability-specific analysis of layer selection and recovery. At four attention layers, layouts drawn from one replacement bank start with opposite profiles. Repair partly restores retrieval even from zero, and some rankings reverse during training. Layouts should therefore be judged after repair and against the intended capabilities, not by a single aggregate score.
  3. Diagnostics beyond aggregate scores, transparently reported. Hard key–value binding stays near chance after recovery (11.8% vs 11.1% chance; teacher 90.2%). All mixed-layout checkpoints fail when the evidence appears early in essay contexts (0/64). Scoring audits and limitations are reported openly. These tests show that the aggregate scores most conversion papers report can hide important failures.
Critical weaknesses
  1. More component ablations are needed. The paper does not show how much retained attention and SSM layers each contribute to downstream performance after training. Related work offers examples of useful attention/SSM diagnostics while demonstrating that softmax attention components can often make up for and hide low-performing linear components (Benfeghoul et al., 2025; Michalak and Abreu, 2025; Cabannes et al., 2025). Furthermore, retrieval changes greatly with attention placement. Appendix J partly addresses attention’s role by reducing attention outputs during binding tests, but does not test the SSM contribution to retrieval. Adding such diagnostics to your analysis may further help to differentiate and explain contributing methods.
  2. The benefit of blockwise-first training is not established. The key comparison (Tab. 3) gains 0.6-1.0 MMLU but loses 1.2-4.1 points on Lambada, from one run per arm with unmatched compute. The authors' own audit (App. I.5) finds 1.4k-2.6k exact bf16 top-score ties on MMLU in comparable checkpoints, so a difference of roughly 100 questions may reflect tie-breaking rather than the training path. The local-repair choices are likewise tested downstream only with 22/28 attention layers kept.
  3. No end-to-end comparison with established pipelines. All baselines are internal, and every four-attention-layer Qwen comparison starts from ReCompose’s local repair. No complete pipeline such as MOHAWK/Llamba or Mamba-in-the-Llama is compared at a matched budget. This matters because each component has precedent (HyLo, Zebra-Llama, LoLCATs, BRECQ, MOHAWK). Total Qwen conversion cost is not reported, and runtime is profiled only on a construction-stage Llama assembly, without end-to-end decode throughput or memory, so the trade-off between accuracy, conversion cost and inference efficiency remains open.
  4. Limited generality. The full recipe is shown only on Qwen2.5-1.5B. The Llama study reuses a Llamba bank without local repair, freezes different components and has no direct-global arm, so it does not validate the method. Other model families, scales, and more modern versions (e.g., Phi, Qwen3-4B, Llama 3.2) would show whether the findings transfer.
Questions for authors
  1. In the repaired models, what happens to retrieval and MMLU when the retained attention layers are removed entirely, or when the SSM path is ablated?
  2. Starting from the same 4/28 layout, how does conventional local preparation (frozen readout, attention-write target) compare when followed by the same blockwise and global training? How does the full ReCompose pipeline compare with an established pipeline at a matched budget?
  3. What is the total Qwen conversion cost, including local preparation? What are the decode throughput and memory of the repaired models?
  4. Can the full recipe, including local repair and the same trainable scope, be applied to Llama and other models? What is Llama's retrieval after repair?
  5. The selection benchmarks are also used for reporting. Could the authors report retrieval on fresh, held-out examples for all repaired layouts (not only the mixed one)?
1 comment

We thank the reviewer for the careful review and for recognizing the controlled study of conversion choices, the capability-specific analysis of layouts and the diagnostics beyond aggregate scores. We agree that the contribution lies in this diagnosis rather than in new components: each component of ReCompose has precedent (Section 2), and the paper's aim is to isolate which conversion choice affects which capability. We answer each point below.

W1 / Q1 (component ablations). Zeroing the four retained attention outputs (α = −1) of the recovered Llama lowers nine-candidate binding accuracy from 50.4% to 10.6%, the 11.1% chance level, on the same 500 examples (Appendix J.2).

W2 (blockwise-first training). We agree that a 0.6–1.0-point MMLU gain should not carry a claim on its own; the revision states Table 3 as an allocation trade-off (higher MMLU staged, higher Lambada direct), not as an advantage of blockwise-first training. The tie audit in Appendix I.5 concerns the shared-bank layout checkpoints, not Table 3.

W3 / Q2 / Q3 (established pipelines, cost, efficiency). As a diagnostic study, we relate to established pipelines in two ways: the Llama study recovers from an existing Llamba-8B bank (Lambada 26.5 → 65.2), and Table 19 places our models beside published conversions under their own protocols (at 4/32 attention, 60.5/77.0/70.9 MMLU/HellaSwag/WinoGrande versus 50.78/73.14/63.38 for Mamba-in-the-Llama-8B; context, not a matched comparison). Recovery costs about 205 GPU-hours staged versus 168 direct, extrapolated from recorded runs; local preparation uses about 2.54B tokens per layer (Appendices A.3, D.4). The hybrid keeps a KV cache in 4 of 28 layers: for one 32k BF16 sequence the cache is 0.94 GB for the teacher versus about 0.14 GB (computed from the architecture), and prefill is 1.06–1.54× faster for a four-attention Llama assembly (Appendix K).

W4 / Q4 (generality). The Llama-3.1-8B study carries the recovery stages to a second family at 8B scale, closing 85% of the Lambada gap to the teacher under a narrower recipe (Section 3.1, Table 4). After recovery, Llama ranks the correct value first in 58.2% of ordinary binding cases but 11.8% of hard ones (chance 11.1%; Table 6).

Q5 (fresh retrieval). At assembly, fresh 32k tests cover all three layouts and preserve their ordering (Section 6.4).

Experiments in progress. We are running experiments for the remaining points: attention/SSM branch ablations on a recovered Qwen model (W1, Q1); FP32 re-scoring of the Table 3 endpoints and a second staged/direct pair (W2); conventional versus ReCompose local preparation for all 24 replacements at the 4/28 layout (W2, Q2); prefill latency and peak memory of the Qwen hybrid (W3, Q3); and fresh retrieval for the recovered MMLU- and retrieval-selected layouts (Q5). We will post the results in this thread when they finish.

Reviewer_C4LWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how best to distil a pretrained Transformer into a sparse hybrid Transformer-SSM model, where most attention layers are replaced by SSMs and only a few (four, in the main experiments) are retained. From what I could gather, it focuses on analysing the impact of different choices in the distillation pipeline, including: (i) whether each SSM replacement should be trained to match the output of the attention mixer alone or of the complete decoder layer; (ii) whether to jointly train blocks of consecutive layers before distilling the whole model, or to distil the whole model directly; and (iii) which attention layers to retain, and how this choice trades off different model capabilities. Several experiments and ablations are reported using Qwen2.5-1.5B and Llama-3.1-8B-Instruct as teachers.

Critical strengths
  • The core question of how to vest perform hybridisation is extremely relevant
  • The authors report expansive ablation results
Critical weaknesses
  • The paper is very hard to read: the main message is unclear, buried under fragmented studies and ad-hoc terminology
Questions for authors

The paper, in its current form, is very difficult to follow, to the point that I cannot confidently identify the implemented procedure and experimental setup, let alone extract the main claims. For example, notice that the main text never specifies which SSM architecture replaces the attention layers (it is only mentioned in Appendix A.3). This is a pity, especially because I consider the topic extremely relevant. I provide a (non-exhaustive) list of action points below, covering only the first six pages. However, the issue goes beyond minor rewordings here and there: I strongly encourage you to carefully revise the whole paper, restructuring it so that

  • (i) the setup is stated upfront and in plain terms: which SSM is used, which layers are replaced, and what is trained against which target at each stage;
  • (ii) the main pipeline and its headline result are presented clearly;
  • (iii) ablations and diagnostic studies are clearly separated from the main claims, rather than spread across four interleaved studies;
  • (iv) terminology is standard, or clearly defined before it is used ("readout", "attention write", "repair", "construction", "shared bank", "splice", "blockwise round", ...).

Minor

  • L21: “In a same-start Qwen comparison at matched post-assembly data exposure, blockwise-then-global training yields”: this sentence is clunky to parse. Make it more human readable
  • L24: “MMLU-selected layouts retain … retrieval-selected layouts” same here: you haven’t even established yet what a “MMLU-selected layout” is
  • L45: “Architecture-aware pipelines also combine selection and recovery” what does this mean?
  • L47-52: This whole paragraph is a very, very convoluted way of saying “we play around with replacing different layers, using different targets for distillation, and investigating different aligning choices for the replaced layers”. Please, proofread
  • L55: “so a smaller error at the replaced attention output need not mean a smaller error at the complete layer output” I guess this depends on how the layer is being translated: regardless, expand
  • L76: “Construction experiments reveal capability trade-offs among selected layouts. Llama layer controls”: construction of what? Selected layouts of what? Llama layer controls of what?
  • L78: “Subsequent Qwen training recovers some capability losses while keeping the selected attention positions fixed” Subsequent to what? Also, why would the selected attention position ever change during training?
  • L81: “using tests that vary evidence position, introduce single-error answer alternatives” vary evidence position of what? What does “single-error answer alternatives” mean?
  • L83: “A controlled factorial” what is a controlled factorial?
  • L87: “A same-start Qwen comparison favors blockwise-first training for MMLU and direct global training for Lambada” so we have to choose between the two? No way to get good scores on both MMLU and Lambada?
  • L90: “without changing the selected layers” it’s the third time the possibility of changing selected layers in training is voiced, but it's unclear where it comes from. If it is meant to contrast with methods that adapt the layout during training, say so once, explicitly
  • L103: “Intermediate-output matching is established” too terse
  • L108: “whereas we retain full attention” ambiguous: it clashes with the scope of the paper of hybridising
  • L108: “our factorial comparison separates readout trainability from target choice” what is a factorial comparison (2x, and also further down: I won’t report this anymore)? What do you mean by readout trainability?
  • L112: “Block reconstruction for quantization supervises combined outputs” expand
  • L114: “through end-to-end distillation after selection” selection of what?
  • L114: “Super Apriel supervises layer outputs” what do you mean by supervising layer outputs? Do you mean “trains from”?
  • L116: “From one teacher input per group, ReCompose propagates student outputs through consecutive complete layers and supervises only the group’s final output” again, very complicated way of saying “we group consecutive layers and distill the whole group to match the teacher output”
  • L123: “placed using retrieval and sliding-window attention placed using MMLU” the sentence is unnecessarily obscure: you’re using a selection criterion based on retrieval / MMLU to select which layers to replace
  • L129: “We compare capability-selected layouts from a shared replacement bank at a fixed full-attention count, tracking task-specific recovery under a common repair protocol” Again, hard to parse. Also, unclear what you mean by shared replacement bank.
  • L140-143: Notation bloating, consider trimming. At the end of the day, what you need is (i) a way to define the retained attention layers, and (ii) a way to define the substituted SSM layers, both possibly obtained by combining multiple layers together.
  • Sec4.1: it is unclear to me the actual setup of the experiment. Are you distilling what into what? The original Qwen attention layer onto an SSM (which SSM?)? Onto another attention layer? Are you training just the readout operations?
  • Sec4.2: why introducing this now if you only discuss setup and results in 6.2? I find your choice of section separation very confusing in general
  • L248: “After operator and head-output alignment, our default trains the SSM” You haven’t even yet specified which SSM you’re using. What did you align, really?
  • L258: “plus a suffix” What is this suffix? An extra layer? It would be relevant to expand, since you highlight later that “the suffix trains separately”.
  • L263: “An attention-only single-layer block stays fixed” expand
  • L290: “The shared-bank” Unclear what you’re referring to (2x, not repeating this again)
  • L295: “Paired 95% intervals quantify evaluation uncertainty” expand
  • L311: “From the same locally repaired Qwen assembly, six blockwise rounds” what do you mean by locally-repaired Qwen assembly? The description in Sec4.1 does it no justice in describing what is going on. What are “blockwise rounds”?
1 comment

We thank the reviewer for the detailed reading and the concrete action points, and we agree that the submitted presentation made the paper harder to follow than it should be. The revision follows your four requests: (i) Section 3.1 states the setup before any equation: Mamba-2 replaces 24 of the 28 attention layers of Qwen2.5-1.5B, softmax attention is kept at layers {0, 9, 17, 21}, and each training step's inputs, targets and trained components are given in plain terms; (ii) the abstract and introduction state the pipeline and its headline result; (iii) Table 2 maps each study to its question and scope; (iv) terminology is unified and defined at first use, with a glossary in Appendix L.1. Your line comments directly guided these edits.

At the same time, we believe the substance of the paper holds, and that the original writing obscured it. The reviewer's summary already names the three choices we analyze (the local supervision target, blockwise versus direct training, and which attention layers to keep). Our contribution is a diagnosis of these choices, not a new method: each component of ReCompose has precedent (Section 2), and the paper isolates, through controlled comparisons from common starting points with paired intervals, how each choice affects capability when most attention layers are replaced:

  • How each replacement is supervised: training its readout improves both attention-output and complete-layer fit, whereas the target choice trades one for the other (Table 1); combined, they raise MMLU by 1.32 points (95% CI [0.91, 1.75]) in a six-replacement test (Table 10).
  • How training is allocated after assembly: from the same assembly at matched data exposure, blockwise-then-global training favors MMLU and direct distillation favors Lambada (Table 3). On L87: both schedules improve both tasks over the assembly; the trade-off is between schedules, not between tasks.
  • Which attention layers are kept: layouts built from the same replacements start with different capability profiles, and identical recovery training changes their standing (Table 5, Figure 2); behavioral tests expose failures that aggregate scores hide (Table 6).

These findings tell practitioners which choice matters for which capability, and how converted hybrids must be evaluated; Reviewers sEu9 and 1DzF likewise highlight this controlled separation as the paper's strength. We hope the revised presentation lets the reviewer reconsider the paper's informativeness and soundness on this substance. If any experiment or inference raises a concern, we would be grateful to know which one and will address it directly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers. Reviewers sEu9 and 1DzF value the controlled separation of conversion choices and the diagnostics beyond aggregate scores; Reviewer C4LW finds the question extremely relevant and asks for a clearer presentation, which the revision provides.

Our contribution is a diagnosis, not a new method. Each component of ReCompose has precedent (Section 2). What the paper adds are controlled comparisons, from common starting points and with paired intervals, that isolate how three conversion choices affect capability when most attention layers are replaced, and tests that expose failures aggregate scores hide:

  • Local supervision: training the readout improves both attention-output and complete-layer fit, whereas the target choice trades one for the other (Table 1); combined, they raise MMLU by 1.32 points (95% CI [0.91, 1.75]) in a six-replacement test (Table 10).
  • Training allocation: from the same assembly at matched data exposure, blockwise-then-global training gives 0.6–1.0 points higher MMLU and direct distillation higher Lambada (Table 3).
  • Attention layout: layouts built from the same replacements start with different capability profiles, and identical recovery training changes their standing: a 9-point 32k retrieval gap shrinks to within 3 points while an MMLU lead of about 10 points persists (Table 5, Figure 2). Hard binding and early-evidence retrieval remain near failure after recovery (Table 6).

With four attention layers, recovery closes most of the Lambada gap to the teacher (Qwen 14.1 → 49.8; teacher 61.8) but less than a fifth of the MMLU gap (48.9 → 50.7; teacher 59.1) (Table 4).

Revision. The abstract and introduction now state this contribution and the headline result. Section 3.1 gives the setup before any equation, Table 2 maps each study to its question, and terminology is unified and defined at first use. Original table and figure numbers are unchanged.

In progress. We are running 24-replacement local-preparation controls with a second staged/direct pair, FP32 re-scoring of Table 3, attention/SSM branch ablations, prefill latency and memory measurements, and fresh retrieval for all recovered layouts and for recovered Llama. We will post results in this thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.