- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies recurrent-attention hybrid language models and shows that standard supervised fine-tuning (SFT) further shifts memory use toward attention while leaving the recurrent pathway underutilized. It proposes a simple auxiliary loss, in which an auxiliary forward pass masks attention access to past context so that past information must reach the current prediction through recurrent state. This intervention improves long-context QA and agentic task performance on Qwen3.5-4B and Nemotron-H-4B, and the authors further show that the same imbalance and auxiliary-supervision benefit generalize to attention-based LMs with textual and latent memory.
Critical strengths
- The empirical analysis of pathway imbalance is clear and useful: the paper demonstrates that providing both attention and recurrent pathways does not imply balanced use, and that standard SFT can reinforce attention reliance. This is a consequential finding for hybrid LM design and fine-tuning.
- The proposed auxiliary loss is simple but effective across models and task families, including QA, TextWorld/BabyAI, failure-recovery settings, and textual/latent memory. The breadth of validation makes the intervention more than a single-benchmark artifact.
Critical weaknesses
- The comparison is selectively narrow with respect to contemporaneous 2025–2026 hybrid-LM fine-tuning interventions. The paper compares only against standard SFT and internal ablations, omitting methods on the same model family and tasks, such as S0 Tuning (+23.6 pp pass@1 on Qwen3.5-4B), selective GDN freezing, and component-aware LoRA placement. This leaves unclear whether L_rec offers an advantage over existing coordination strategies, which is consequential for the paper’s central claim that targeted supervision is needed to coordinate memory pathways.
- The contribution is primarily empirical. Masking attention in an auxiliary forward pass to force reliance on another memory pathway is close to existing masking/regularization techniques. The paper should more sharply distinguish its contribution from prior memory-coordination, retrieval-aware distillation, and component-specific fine-tuning methods, and should clarify which design choices are newly introduced versus inherited.
Young, J. (2026). S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models. arXiv. https://arxiv.org/abs/2604.01168
Questions for authors
- How does L_rec compare against S0 Tuning, selective layer freezing, and component-aware LoRA placement under an identical protocol on Qwen3.5-4B? Since S0 Tuning reports substantially larger absolute gains on the same model, what evidence supports the claim that recurrent auxiliary supervision is the preferable coordination mechanism?
- How do the gains compare at strictly equal wall-clock/FLOPs and under full fine-tuning or larger hybrid models, rather than LoRA on 4B models only?
- Could the gains be explained mainly as a regularization effect rather than improved recurrent-pathway use? What evidence isolates pathway utilization from generic regularization?
7 comments
W2:The contribution is primarily empirical. Masking attention in an auxiliary forward pass to force reliance on another memory pathway is close to existing masking/regularization techniques. The paper should more sharply distinguish its contribution from prior memory-coordination, retrieval-aware distillation, and component-specific fine-tuning methods, and should clarify which design choices are newly introduced versus inherited.
Q3: Could the gains be explained mainly as a regularization effect rather than improved recurrent-pathway use? What evidence isolates pathway utilization from generic regularization?
Thank you for your question and suggestions. Since the question raises several points, we have organized our response into bullets.
- Our Contribution
We would like to highlight that our contributions are mainly on threefold:
- In Section 4, we show that hybrid LMs still underuse the recurrent pathway after standard SFT.
- In Section 5, to use both pathways well, we introduce an auxiliary loss that encourages use of the recurrent pathway.
- In Section 6, we analyze how the auxiliary loss helps to learn from past trajectories and show that the finding generalizes to attention only models with textual and latent memory.
To our knowledge, our work is the first to show that hybrid LMs tend to underuse the recurrent pathway even after standard SFT (Section 4) and to propose a method specifically designed to address this imbalance. As the reviewer mentioned, the auxiliary loss in Section 5 builds on an existing masking mechanism, which we also mentioned in the paper as “commonly used to control information flow in LMs”. Our contribution is to identify this imbalance and design an auxiliary loss that uses this mechanism to encourage greater use of the recurrent pathway.
- Comparison with existing masking / regularization techniques
To evaluate whether generic regularization or masking can account for these gains, we conduct experiment with comparison with regularization and masking baselines:
- R-Drop [3], a regularization method that passes the same input through the model twice with different dropout samples and minimizes the symmetric KL divergence between the two output distributions.
- Input token dropout [4], a regularization method that replaces a random subset of input tokens.
- Random attention masking [5], a masking method that randomly masks attention entries.
As shown in table below, on the question answering task, all baselines improve accuracy (Full) over standard SFT, but the pathway imbalance persists, recurrent-only showing much lower performance compared to attention-only. Also, ours with auxiliary loss show highest performance (Full). These results support that specifically encouraging the use of the recurrent pathway contributes to the gain rather than generic regularization. We include more details in Appendix C.6.
| Method | Full | attention-only | recurrent-only |
|---|---|---|---|
| SFT | 34.6 | 25.0 | 5.0 |
| R-Drop | 35.3 | 26.7 | 6.5 |
| Input token dropout | 37.0 | 27.4 | 6.5 |
| Random Attention Masking | 35.9 | 30.8 | 7.1 |
| Ours | 38.5 | 23.7 | 22.8 |
- Evidence of improved recurrent pathway use
Along with our analysis of recurrent-only, we also provide evidence in Sections 5.2 and 6 that better use of the recurrent pathway contributes to improved performance.
In Section 5.2, we show that the gains are larger in longer-context settings for both QA and agentic tasks. This pattern is consistent with our analysis in Section 2 and with prior work suggesting that the recurrent pathway is particularly useful for aggregating information across long contexts.
In Section 6, we observe that the model trained with the auxiliary loss makes better use of previous trajectories, especially as more trajectories are provided. Also, it shows larger gains on multi-evidence questions (Fig. 7), for which we found the recurrent pathway to be particularly helpful (Fig. 3).
- Relation to prior lines of work (memory coordination, retrieval-aware distillation, and component-specific fine-tuning)
Retrieval-aware distillation [6] builds a hybrid by keeping only the few attention heads needed for precise retrieval and converting the rest into recurrent heads. The paper focuses on a different question from our work: what is new way of designing hybrid LMs, whereas our work focuses on making the existing hybrid LMs better use both pathways.
Component-specific fine-tuning [2] restricts which components are updated (e.g., only the attention or recurrent layers) whereas ours update all components whereas change how information flows. Thus, component-specific fine-tuning does not balance the utilization as we observed in W1.
Memory coordination: Our work focuses on memory coordination between the two pathways of hybrid LM. To our knowledge, no prior work has shown that the bias towards attention persists under standard SFT, or that correcting it to coordinate the two pathways better improves performance.
We discussed representative work on each topic above but if the reviewer had other works in mind, we would be glad to discuss them.
Q2: How do the gains compare at strictly equal wall-clock/FLOPs and under full fine-tuning or larger hybrid models, rather than LoRA on 4B models only?
- Equal wall-clock/FLOPs
As described in Section 3 (Training), each optimization step with the auxiliary loss requires additional computation. Thus, to account for this cost, we use half as many optimization steps as standard SFT.
- Full fine-tuning or larger hybrid models
Thank you for the suggestion. As suggested, we ran experiments with full fine-tuning (Full FT) on Qwen3.5-4B and LoRA fine-tuning on Qwen3.5-9B. We observe similar trends in both settings: (1) with standard SFT, the models tend to underuse the recurrent pathway; (2) adding auxiliary loss (Ours) leads to more balanced use of both pathways and improves performance. We have updated in Appendix C.7.
| Method | QA Full | QA attention-only | QA recurrent-only | TextWorld (Quest) Full | TextWorld (Quest) attention-only | TextWorld (Quest) recurrent-only |
|---|---|---|---|---|---|---|
| SFT (4B, LoRA) | 34.6 | 25.0 | 5.0 | 78.5 | 52.0 | 4.0 |
| Ours (4B, LoRA) | 38.5 | 23.7 | 22.8 | 80.5 | 53.0 | 46.5 |
| SFT (4B, Full FT) | 39.5 | 28.1 | 4.4 | 78.5 | 52.5 | 9.0 |
| Ours (4B, Full FT) | 40.4 | 23.6 | 21.0 | 82.0 | 56.0 | 39.0 |
| SFT (9B, LoRA) | 43.7 | 28.9 | 9.3 | 84.0 | 59.0 | 7.0 |
| Ours (9B, LoRA) | 45.1 | 26.7 | 23.2 | 85.0 | 58.5 | 47.0 |
[1] S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
[2] Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
[3] R-Drop: Regularized Dropout for Neural Networks
[4] Data Noising as Smoothing in Neural Network Language Models
[5] DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
[6] Retrieval-Aware Distillation for Transformer-SSM Hybrids
We thank the reviewer for their thoughtful comments and for recognizing the clarity of our analysis of the pathway imbalance, the simplicity and effectiveness of our auxiliary loss, and the breadth of our experiments.
We provide our detailed responses below.
Thank you for the suggestion. As suggested, we conduct experiments comparing with suggested baselines including S0-Tuning[1] and component-aware LoRA placement[2]. As shown in table below, we evaluate overall performance with previous context passed through both pathways (Full) and use the attention-only and recurrent-only setups as ablations, passing previous context through only the corresponding pathway. We observe that training with our auxiliary loss (Ours) shows strongest performance on both question answering and agentic tasks. Also, we observe that the baselines tend to underuse the recurrent pathway as we observe with standard SFT, whereas ours show balanced use of both pathways. We have included the results in Appendix C.3.
In more detail, we analyzed why S0-Tuning did not yield performance gains in our settings as those reported in their paper. We identified two possible explanations. First, the paper experimented over a small subset of HumanEval (48 instances) for training where they observed the model they compared with (standard LoRA) tended to overfit whereas S0-Tuning avoids this issue. Our training set is much larger (12k). Second, the dataset they used (HumanEval) is much shorter compared to our task (our QA is around 80 times longer on average). As S0-Tuning modifies only the initial recurrent state, its influence may diminish over these longer sequences.
Also, we were unable to identify the paper on selective GDN freezing mentioned in the review. We would appreciate it if the reviewer could share the title or citation so that we can evaluate the method.