- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper looks at how to recover capabilities lost when most attention layers in a pretrained Transformer are replaced with SSMs. It proposes ReCompose: train each replacement to match the full teacher-layer output, jointly train groups of consecutive layers, and then distill the whole model. Its main contribution is diagnosing how local supervision, cross-layer training, and the choice of retained attention layers affect recovery. Experiments on Qwen and Llama with only four attention layers show improvements across knowledge, commonsense, and text completion tasks. In a controlled Qwen comparison, staged training improves MMLU by 0.6–1.0 points over direct distillation at matched data exposure, though direct distillation does better on Lambada. The paper also finds that knowledge and retrieval favor different attention layouts, and that harder retrieval and binding failures remain.
Critical strengths
- The paper connects its diagnosis to concrete training improvements. It separates readout trainability from supervision target choice in controlled local experiments, then tests their combined downstream benefit. The same-start comparison of staged versus direct distillation also helps establish when blockwise training is useful. This makes the proposed recipe more convincing than a pipeline supported only by final benchmark scores.
- The evaluation examines both capability recovery and its limits. Beyond standard benchmarks on Qwen and Llama, the paper tests attention-layout trade-offs, evidence position, and harder fact-binding cases. These tests show that improved aggregate scores can coexist with substantial failures, which makes the conclusions more credible and useful for deciding how to convert and evaluate hybrid models.
Critical weaknesses
(1) The benefit of retaining a few attention layers is unclear. The paper includes a pure-SSM regional training experiment, but it does not compare pure-SSM and hybrid models after the same full training process with comparable budgets. Without this baseline, it is hard to tell whether partial replacement is better than replacing all attention layers.
((2) The staged-training ablation is useful, but the results do not show a clear advantage. Compared with direct global distillation, staged training improves MMLU by only 0.6–1.0 points and consistently performs worse on Lambada. With only one run per setting and no compute-matched comparison, I am not convinced that the extra blockwise stage is necessary.
Questions for authors
(1) Why do you choose the qwen2.5 1.5B as the base model. If you want to train a small model you can use qwen3/3.5. The base model is too weak. (2) Why use staged training instead of direct global distillation? The ablation shows a small MMLU gain but lower Lambada accuracy, so the benefit of adding the blockwise stage is unclear. (3) The experiment testing readout adaptation and full-layer supervision replaces only 6 of 28 attention layers with SSMs, while the main model replaces 24. Do these two changes still improve performance when 24 layers are replaced?
2 comments
Thanks for the clarification. I now understand that the paper’s main contribution is diagnosing how different training choices affect recovery. My main concern is that the six-layer experiment shows a benefit from combining readout adaptation with full-layer supervision, but it’s still unclear whether that benefit carries over to the main setting, where 24 layers are replaced. The main results show that the full pipeline recovers some capabilities, but they don’t isolate the contribution of those two changes. The comparison you’re running, with the same recovery procedure in both conditions, would directly address this concern. I’ll keep my current rating for now and would reconsider it if the new results show a clear benefit in the 24-layer setting.
We thank the reviewer for recognizing that the paper connects its diagnosis to concrete training choices and examines both recovery and its limits. Our contribution is this diagnosis rather than a new method: each component of ReCompose has precedent (Section 2), and staged training is one of the choices we diagnose, not a step we claim is always better.
W1 (retaining attention). Assembled from the same layerwise-distilled Llamba-8B bank, keeping four attention layers raises MMLU from 34.1 with all 32 layers replaced (Appendix E.3) to 58.7 before any recovery training, and to 60.5 after recovery (Table 4); Section 4.2 now states this comparison. In the recovered model, zeroing those four attention outputs drops nine-candidate binding accuracy from 50.4% to 10.6%, the 11.1% chance level (Appendix J.2). The paper takes a small attention budget as given, as current hybrid conversions do, and studies recovery under it.
W2 / Q2 (staged training). The blockwise stage is not extra training: six blockwise rounds replace six of the nine global rounds at matched data exposure. It targets a specific mismatch: replacements are prepared on teacher inputs but receive student outputs after assembly. From the same assembly at rounds 7/8/9, staging gives MMLU 50.5/50.3/50.7 versus 49.5/49.7/50.0 and Lambada 46.8/48.7/49.8 versus 50.9/50.7/51.0 (Table 3), at about 1.2× the recovery compute. We therefore report an allocation trade-off favoring MMLU, which is itself a diagnostic finding (Section 6.2). For the single-run concern, we are training a second staged/direct pair from a fresh assembly and re-scoring the Table 3 endpoints with FP32 option scores and paired intervals.
Q1 (Qwen2.5-1.5B). We needed a pure Transformer small enough for many controlled conversions with one shared teacher. Dense Qwen3 remains a pure Transformer and a natural next teacher [1]; Qwen3.5 already interleaves Gated DeltaNet layers with attention [2], so converting it starts from a hybrid. For a stronger teacher, the Llama-3.1-8B study (teacher MMLU 68.3) raises Lambada from 26.5 to 65.2 (Table 4).
Q3 (24 replacements). We are running this comparison at the 4/28 layout: conventional (frozen readout, attention-output target) versus ReCompose local preparation for all 24 replacements, followed by identical recovery. At six replacements, the combined change raises MMLU by 1.32 points (95% CI [0.91, 1.75]) and HellaSwag by 0.49 ([0.21, 0.78]) (Table 10). We will post these results in this thread.
[1] Qwen3-4B official configuration [2] Qwen3.5-4B official model card