acceptodds
Under review as a conference paper at ICLR 2027

Competing Learning in Hybrid Architectures

Abstract

Many modern language models have adopted hybrid architecture that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete for learning the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distribution, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid architecture, linear attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_abWSOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how RNN layers and Attention layers with softmax acquire retrieval functions in hybrid language models. They propose synthetic copying experiments with counterfactual activation patching, and freeze/unfreeze interventions to argue that the learning in early recurrent layers suppresses the attention layers from learning retrieval. The authors propose Better Layer Division (BLD) which freezes recurrent blocks during the early stage of pre-training, then trains all parameters. Experiments across six KDA model sizes and GDN/GLA variants show consistent aggregate improvements in retrieval, alongside increased reliance on full-context attention.

Critical strengths

Retrieval improvements are persistent across model sizes and recurrent architectures, under matched training budgets.

  • Tables 1 and 2
  • Appendix C specifies matched data views, token budgets, and optimizers to rule out extra training data as a confounder.

The interventions seem to provide evidence that training history affects the functionalities the models are learning.

  • changes in attention attribution (Figure 4, Appendix A.4), or retrieval performance (Figure 6, Appendix H)
Critical weaknesses
  1. The attribution score does not seem to distinguish a layer that retrieves information, from a layer which passes along the information that has been retrieved earlier.
  • Equation (6) seems to measure, how replacing a mixer output changes the final answer's logits.
  • Correct me if I am wrong, but this can contain information retrieved by earlier layers.
  • Figures 2 to 4 show changes in the predictions, but I do not think these results establish the claim that recurrent learning reduces attention's retrieval specific learning signal.
  1. Does freezing blocks isolate competition between recurrent and attention mixers?
  • The observed changes in Table 5 and 9 could arise from freezing other parts such as MLPs/normalization which could affect the retrieval learning.
  • I feel the paper demonstrates a useful training recipe, but does not establish that it works by correcting competition over retrieval learning.
Questions for authors

What are the held-out synthetic retrieval accuracy and cross-entropy trajectories around freezing and unfreezing? The attribution plots do not establish whether the observed redistribution accompanies improved task performance or occurs at comparable performance. This distinction would affect how I interpret the intervention's benefit.

3 comments
AuthorsOpenReview ↗

Response to abWS 1/2

Dear reviewer abWS,

Than you for distinguishing functional attribution, training signals, and task performance. We address these separately below.

We have updated the manuscript and highlighted the revisions in blue.

Weakness 1.a: Attribution versus information relayed from earlier layers.

Response W1.a:

You have raised a potential relay confound: a downstream layer could receive information retrieved upstream and obtain a high score when its output is replaced. Our intervention replaces only the mixer output f(xA)f(x_A) with f(xB)f(x_B), while keeping the recipient layer’s input xAx_A unchanged. The addition therefore becomes xA+f(xB)x_A + f(x_B). Information already present in xAx_A remains available through the unchanged skip connection. Information already available upstream therefore retains an unchanged path to subsequent computation; merely passing through the layer does not automatically receive attribution.

We additionally examine this concern empirically using the new layer-resolved figure (Fig. 5). In the KDA-dominant regime, L3 has a large contribution while the downstream L4 attention mixer remains negligible, despite receiving the preceding computation. In the attention-dominant regime, only L4 has a substantial contribution. Contributions coexist near the transition, where restoring recurrent updates produces seed-dependent takeovers relative to the permanent-freeze controls. The detailed result is added in our updated paper with blue fonts, Section 3.3.2.

The following table reports final mixer-output contribution scores C at training step 1,500, averaged over three seeds on Spaced Retrieval (gap 8,192). KDA blocks are frozen at the indicated step and unfrozen after step 500 in every condition. Bold marks the largest contribution in each row.

Freeze stepL1 KDAL2 AttentionL3 KDAL4 Attention
500.0000.0000.00018.684
1500.0000.00815.1731.317
4000.0000.00322.3500.012

Early freezing at step 50 yields attention dominance (L4: 18.684; L3: approximately zero), whereas late freezing at step 400 yields KDA dominance (L3: 22.350; L4: 0.012). At step 150, both retain measurable contributions (L3: 15.173; L4: 1.317), with KDA dominant at the endpoint. Because all conditions resume KDA updates at the same step, this reversal supports Competing Learning: the eventual allocation depends on the components’ preceding learning history.

The results also support the metric’s practical selectivity. In the KDA-dominant regime, downstream L4 remains near zero despite receiving L3’s computation; it does not automatically receive a large score merely because retrieval information is available upstream. Conversely, early freezing produces a large L4 contribution with negligible contributions from the other layers.

Across almost all settings, a single layer has a markedly higher score than all other layers, although its identity changes with the freezing schedule. This consistent separation provides strong empirical support for the score’s effectiveness in identifying the dominant layer contributing to retrieval (Fig. 5; §3.3.2).

Weakness 1.b: Evidence about retrieval-learning signals.

Response W1.b:

We assess retrieval through end-to-end predictions, including the representations and readout needed to produce the answer. In our synthetic task, supervision comes exclusively from retrieved answer tokens (Appendix A.2). Evidence for competition comes from the systematic redistribution of retrieval contributions: KDA dominates under standard training (Fig. 2), whereas sufficiently early KDA freezing enables attention to take over (Fig. 4, left). After unfreezing, attention either reinforces or loses its advantage depending on its preceding learning history (Fig. 4, right; Appendix A.4).

Our evidence is not limited to prediction changes. The controlled gradient experiment (Fig. 6; §3.3.3; Appendix I.1; Table 24) freezes only the KDA mixers and measures unclipped attention gradients on first-answer-token retrieval cross-entropy. It preserves full backward passes and uses the same step-specific baseline clipping coefficient before masking frozen updates. Across ten seeds, freezing KDA progressively amplifies attention gradients relative to the never-frozen baseline, while resuming KDA updates attenuates this amplification. Together with the history-dependent freeze–unfreeze results, this directly supports competition during retrieval learning. We do not interpret attribution curves alone as direct measurements of retrieval-specific gradients.

AuthorsOpenReview ↗

Response to abWS 2/2

Weakness 2: Does freezing whole blocks confound mixer competition with MLP/normalization effects?

Response W2:

We conducted component-wise freezing controls on the 240M Chinchilla configuration, using identical training settings and 16K context in both stages. Mixer-only freezes only the KDA mixers, whereas MLP-only freezes only the MLPs in the corresponding KDA blocks. Normalization remains trainable in both controls, and all parameters are trainable in the second stage.

Initial-stage freezingRecall ↑RULER ↑
Whole KDA blocks31.2726.89
KDA mixers only33.1725.47
KDA-block MLPs only24.1316.16

RULER averages the manuscript’s 57 task–length combinations at 1K–16K (§4.2; Appendix E);

Mixer-only freezing achieves retrieval performance comparable to whole-block freezing while keeping MLPs and normalization trainable. MLP-only freezing falls substantially behind both. These controls directly support the effectiveness of the mixer-targeted intervention motivated by Competing Learning: freezing MLPs or normalization is unnecessary to obtain strong retrieval performance, and MLP-only freezing cannot reproduce it.

Together with the mixer-only gradient experiment, these controls support the effectiveness of intervening on recurrent mixer learning, without requiring MLP or normalization freezing.

Question 1: Held-out synthetic retrieval accuracy and cross-entropy around freezing and unfreezing.

Response Q1:

Although recurrence-dominated and attention-dominated solutions both ultimately achieve nearly perfect performance, Spaced Retrieval serves as a controlled tool for observing how the division of labor forms. The synthetic experiments demonstrate two distinct paths through which retrieval capability can develop, with either recurrence or attention assuming the dominant role. The practical evidence that BLD improves the division of labor in hybrid models comes from its clear advantages in the subsequent LLM pretraining and downstream evaluations (Tables 1–2 in the paper), together with the attention-head analyses that support the intended specialization (§4.5; Figs. 8 and 10).

We compare the same three freezing times used in the layer-resolved analysis: steps 50, 150, and 400. All three conditions reach 100% held-out exact-match accuracy at step 1,500 in every one of the three training seeds, while developing different layer contributions.

The table 1 places retrieval performance and layer contributions side by side. The table 2 then shows how each solution develops before and after unfreezing.

Table 1: Final performance and layer contributions at step 1,500

Freeze stepFinal exact accuracy (%)Final answer CEL3 KDA contribution CL4 Attention contribution C
50100.000.0009910.00018.684
150100.000.00233315.1731.317
400100.000.00016022.3500.012

Table 2: Held-out accuracy and answer CE during training

Training stepfreeze@50 accuracy (%)freeze@50 CEfreeze@150 accuracy (%)freeze@150 CEfreeze@400 accuracy (%)freeze@400 CE
500.002.2042660.002.2042660.002.204266
1500.851.20856712.230.87007812.230.870078
40084.000.12563854.080.31401699.470.006071
50090.480.07270160.670.25913499.750.002461
50182.250.11776564.400.24010199.730.002830
51079.770.13983029.400.63112499.220.006686
70099.770.00646995.270.04374399.930.001037
1,00099.980.00159099.880.004146100.000.000210
1,500100.000.000991100.000.002333100.000.000160

All three conditions reach 100% accuracy through different paths: attention dominates with freeze@50, KDA dominates with freeze@400, and freeze@150 shows a transition ending in KDA dominance. BLD’s LLM and downstream gains (Tables 1–2 in the paper) demonstrate the practical value of steering this specialization.

If you have any remaining concerns, we would be happy to address them. If these clarifications and additional results resolve your concerns, we hope you will consider updating your score to more accurately reflect the paper’s scope and contributions.

Thank you for the response and additional experimental results the authors have brought up. They address my some of concerns.

For the first weakness, I still feel the skip-connection argument does not rule out the relay by itself. The output which has been substituted can bring forward information about the answer. I think overall, the empirical result in Figure 5 is more informative but are from a small parameter model in a synthetic setting.

For weakness 2, I have one doubt about the parameter size being 240M, which does not seem to match Table 1. BLD evaluated in Table1-2 still freezes entire blocks and I do not think these results directly test the mixer adequately.

Looking at the extensive rebuttal response from other reviewers' comments as well, I raise my score from 1 to 2. I still think this is a borderline paper and would like to see what how other reviewers assess the paper after the rebuttal.

Reviewer_G5aZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how recurrent and softmax-attention layers acquire retrieval functions in hybrid language models. The authors argue that recurrent layers often learn retrieval earlier, reducing the learning signal available to attention layers. Based on synthetic retrieval experiments, they propose Better Layer Division, which freezes recurrent blocks early in pretraining before jointly training all components. Experiments show improvements in language modeling and retrieval-related tasks across several model sizes and recurrent architectures.

Critical strengths

The paper addresses a relevant question about how hybrid models use their different components. The freezing and unfreezing experiments go beyond observing final checkpoints and provide useful evidence that training order affects specialization. BLD is simple and requires no architectural changes. The natural-language experiments also use matched data, token budgets, and optimization schedules, making the reported gains more convincing. I appreciate the evaluation across model sizes and the additional experiments with other recurrent mixers.

Critical weaknesses
  1. The attribution score does not directly establish a layer's retrieval ability or learning speed. Section 3.2 measures how replacing a mixer output changes the final answer margin. This captures the effect of that output within the current network, but the effect also depends on information supplied by earlier layers and how later layers use it. A layer may carry information retrieved upstream without performing retrieval itself.

  2. The argument separating depth sensitivity from output-format failure is not convincing. A manual audit should distinguish incorrect answers, correct answers in an unexpected format, and incomplete outputs.

  3. The evidence for the contrast between depth-sensitive and depth-tolerant recurrence is uneven. The high-level degradation is evaluated across tasks, whereas increasing low-level depth beyond training is examined only in a small mathematics probe.

Questions for authors

Please see the weakness parts and address these issues.

1 comment
AuthorsOpenReview ↗

Response to G5aZ 1/1

Dear reviewer G5aZ,

Thank you for recognizing the training-order interventions, matched natural-language comparisons, and evaluation across model sizes and recurrent mixers.

We have updated the manuscript and highlighted the revisions in blue.

Question 1: Attribution score and retrieval learning.

Response 1:

You have raised a potential relay confound: a downstream layer could receive information retrieved upstream and obtain a high score when its output is replaced. Our intervention replaces only the mixer output f(xA)f(x_A) with f(xB)f(x_B), while keeping the recipient layer’s input xAx_A unchanged. The addition therefore becomes xA+f(xB)x_A + f(x_B). Information already present in xAx_A remains available through the unchanged skip connection. Information already available upstream therefore retains an unchanged path to subsequent computation; merely passing through the layer does not automatically receive attribution.

We additionally examine this concern empirically using the new layer-resolved figure (Fig. 5). In the KDA-dominant regime, L3 has a large contribution while the downstream L4 attention mixer remains negligible, despite receiving the preceding computation. In the attention-dominant regime, only L4 has a substantial contribution. Contributions coexist near the transition, where restoring recurrent updates produces seed-dependent takeovers relative to the permanent-freeze controls. The detailed result is added in our updated paper with blue fonts, Section 3.3.2.

The following table reports final mixer-output contribution scores C at training step 1,500, averaged over three seeds on Spaced Retrieval (gap 8,192). KDA blocks are frozen at the indicated step and unfrozen after step 500 in every condition. Bold marks the largest contribution in each row.

Freeze stepL1 KDAL2 AttentionL3 KDAL4 Attention
500.0000.0000.00018.684
1500.0000.00815.1731.317
4000.0000.00322.3500.012

Early freezing at step 50 yields attention dominance (L4: 18.684; L3: approximately zero), whereas late freezing at step 400 yields KDA dominance (L3: 22.350; L4: 0.012). At step 150, both retain measurable contributions (L3: 15.173; L4: 1.317), with KDA dominant at the endpoint. Because all conditions resume KDA updates at the same step, this reversal supports Competing Learning: the eventual allocation depends on the components’ preceding learning history.

The results also support the metric’s practical selectivity. In the KDA-dominant regime, downstream L4 remains near zero despite receiving L3’s computation; it does not automatically receive a large score merely because retrieval information is available upstream. Conversely, early freezing produces a large L4 contribution with negligible contributions from the other layers.

Across almost all settings, a single layer has a markedly higher score than all other layers, although its identity changes with the freezing schedule. This consistent separation provides strong empirical support for the score’s effectiveness in identifying the dominant layer contributing to retrieval (Fig. 5; §3.3.2).

Weakness 2–3: depth sensitivity & output-format

Response 2-3:

Weaknesses 2 and 3 discuss depth sensitivity, output-format failures, and a mathematics probe, which are not examined in our paper. Could you please confirm whether these comments were intended for this submission?

If you have any remaining concerns, we would be happy to address them. If these clarifications and additional results resolve your concerns, we hope you will consider updating your score to more accurately reflect the paper’s scope and contributions.

Reviewer_1ERgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates competing retrieval learning in hybrid models, where layers that learn retrieval earlier may weaken the learning signals available to others. Synthetic tasks and freezing interventions motivate Better Layer Division (BLD), which initially freezes recurrent blocks before joint training. Experiments on 180M–1B KDA models and GDN/GLA hybrids show consistent retrieval-related gains, while head analyses suggest increased reliance on full-context attention.

Critical strengths
  1. The paper addresses how functional specialization develops in hybrid models. Freezing/unfreezing experiments show that early interventions can reshape layer contributions.

  2. BLD is simple and consistently improves retrieval-related metrics across six KDA model sizes under matched training conditions, with additional gains on GDN and GLA hybrids.

Critical weaknesses
  1. The experiments demonstrate that freezing changes layer contributions, but do not sufficiently isolate competition for retrieval-specific learning signals from alternative representation and optimization effects. The mechanistic advance over prior work on weakened optimization pressure also remains unclear.

  2. The retrieval gains support BLD’s usefulness, but do not fully establish a better division of labor between recurrent and attention layers. Evaluation at limited model scales and training budgets also leaves unclear whether BLD produces lasting capability improvements or primarily accelerates learning.

Questions for authors
  1. Mechanism and freezing controls: Since the intervention freezes entire recurrent blocks, including MLPs and normalization, can the authors compare mixer-only and MLP/normalization-only freezing at a representative scale? What evidence distinguishes competition for retrieval-specific learning signals from broader changes in representations and optimization?

  2. Attribution and mechanistic novelty: How does the attribution score distinguish retrieval performed by the current layer from processing information supplied upstream or effects of downstream readout? Beyond the weakened optimization pressure and delayed retrieval-head formation discussed by Qiao et al. (2026), what additional testable prediction or causal evidence does “Competing Learning” provide?

  3. Why favor attention over recurrence? The paper motivates assigning long-range retrieval to attention, but why should giving attention an early advantage produce a better overall model than allowing KDA to retain its retrieval advantage? Changing which component dominates retrieval does not itself establish a better division of labor. What evidence shows that BLD strengthens attention’s retrieval while preserving recurrent layers’ useful capabilities, and that this complementary specialization explains the downstream gains?

  4. Scope and practical significance: What general guidance does this study provide for practical hybrid-LLM training beyond the tested configurations? The main experiments use models trained from scratch up to 1B parameters with a fixed 3:1 recurrence-to-attention ratio. GDN and GLA broaden mixer coverage but retain a similar setup. Under what architectural and data conditions should practitioners expect recurrence to hinder attention learning and benefit from BLD?

  5. Robustness and training budget: Do the gains persist across training seeds, freezing durations, and longer token budgets at a representative scale, or does Vanilla eventually catch up? Figure 7 fits the same checkpoints and shows a small loss advantage within the existing budget; what evidence distinguishes accelerated learning from a sustained capability improvement?

4 comments
AuthorsOpenReview ↗

Response to 1ERg 1/3

Dear reviewer 1ERg,

Thank you for the careful reading and constructive suggestions. Below, we clarify the mechanistic interpretation and present additional controls and evaluations addressing the concerns about BLD’s effectiveness, scope, and robustness.

We have updated the manuscript and highlighted the revisions in blue.

Question 1.a: Comparison against mixer-only and MLP/normalization-only freezing.

Response 1.a:

We conducted component-wise freezing controls on the 240M model:

Initial-stage freezingRecall ↑RULER ↑
Whole KDA blocks31.2726.89
KDA mixers only33.1725.47
KDA-block MLPs only24.1316.16

Mixer-only freezing achieves retrieval performance comparable to whole-block freezing while keeping MLPs and normalization trainable. MLP-only freezing falls substantially behind both. These controls directly support the effectiveness of the mixer-targeted intervention motivated by Competing Learning: freezing MLPs or normalization is unnecessary to obtain strong retrieval performance, and MLP-only freezing cannot reproduce it.

Question 1.b: Evidence distinguishes competition for retrieval-specific learning signals from broader changes in representations and optimization.

Response 1.b:

We assess retrieval through end-to-end predictions, including the representations and readout needed to produce the answer. In our synthetic task, supervision comes exclusively from retrieved answer tokens (Appendix A.2).

Evidence for competition comes from the systematic redistribution of retrieval contributions: KDA dominates under standard training (Fig. 2), whereas sufficiently early KDA freezing enables attention to take over, with earlier freezing generally yielding stronger final attention contributions (Fig. 4, left). After unfreezing, attention either reinforces or loses its advantage depending on its preceding learning history (Fig. 4, right).

Across these experiments, one layer ultimately contributes substantially more to retrieval than the others. We interpret this dominant layer as the winner of the competition, further supporting our Competing Learning hypothesis.

AuthorsOpenReview ↗

Response to 1ERg 2/3

Question 2.a: How does the attribution score distinguish retrieval performed by the current layer from processing information supplied upstream or effects of downstream readout?

Response 2.a:

You have raised a potential relay confound: a downstream layer could receive information retrieved upstream and obtain a high score when its output is replaced. Our intervention replaces only the mixer output f(xA)f(x_A) with f(xB)f(x_B), while keeping the recipient layer’s input xAx_A unchanged. The addition therefore becomes xA+f(xB)x_A + f(x_B). Information already present in xAx_A remains available through the unchanged skip connection. Information already available upstream therefore retains an unchanged path to subsequent computation; merely passing through the layer does not automatically receive attribution.

We additionally examine this concern empirically using the new layer-resolved figure (Fig. 5). In the KDA-dominant regime, L3 has a large contribution while the downstream L4 attention mixer remains negligible, despite receiving the preceding computation. In the attention-dominant regime, only L4 has a substantial contribution. Contributions coexist near the transition, where restoring recurrent updates produces seed-dependent takeovers relative to the permanent-freeze controls. The detailed result is added in our updated paper with blue fonts, Section 3.3.2.

The following table reports final mixer-output contribution scores C at training step 1,500, averaged over three seeds on Spaced Retrieval (gap 8,192). KDA blocks are frozen at the indicated step and unfrozen after step 500 in every condition. Bold marks the largest contribution in each row.

Freeze stepL1 KDAL2 AttentionL3 KDAL4 Attention
500.0000.0000.00018.684
1500.0000.00815.1731.317
4000.0000.00322.3500.012

Early freezing at step 50 yields attention dominance (L4: 18.684; L3: approximately zero), whereas late freezing at step 400 yields KDA dominance (L3: 22.350; L4: 0.012). At step 150, both retain measurable contributions (L3: 15.173; L4: 1.317), with KDA dominant at the endpoint. Because all conditions resume KDA updates at the same step, this reversal supports Competing Learning: the eventual allocation depends on the components’ preceding learning history.

The results also support the metric’s practical selectivity. In the KDA-dominant regime, downstream L4 remains near zero despite receiving L3’s computation; it does not automatically receive a large score merely because retrieval information is available upstream. Conversely, early freezing produces a large L4 contribution with negligible contributions from the other layers.

Across almost all settings, a single layer has a markedly higher score than all other layers, although its identity changes with the freezing schedule. This consistent separation provides strong empirical support for the score’s effectiveness in identifying the dominant layer contributing to retrieval (Fig. 5; §3.3.2).

Question 2.b: What additional testable prediction or causal evidence does “Competing Learning” provide?

Response 2.b:

While Qiao et al. (2026)’s Large-Window Laziness analysis focuses on how SWA window size delays retrieval-head formation, we investigate competition in KDA-based recurrent–attention hybrids. Our hypothesis is bidirectional: either component can dominate retrieval if it acquires a sufficient early advantage. We test this through layer-wise analysis on Spaced Retrieval and freeze–then-unfreeze interventions.

Crucially, when KDA is frozen sufficiently early, subsequently unfreezing it further strengthens attention’s retrieval contribution relative to keeping KDA frozen; with later freezing, unfreezing instead reduces attention’s contribution (Fig. 4, right). This dependence on prior learning history supports a competition whose outcome can be reversed, extending the observation that efficient attention can delay attention learning.

This insight also motivates BLD, which gives attention an early advantage and improves downstream performance (Tables 1–2 in the paper). Furthermore, DuoAttention-style identification (Xiao et al., 2025) and head-masking experiments show that the final BLD model contains more retrieval heads than Vanilla (§4.5; Figs. 8 and 10; Appendix I).

We therefore examine both the causal redistribution of retrieval contributions during training and the resulting head specialization at the evaluated training endpoint, beyond tracking retrieval-head formation speed alone.

AuthorsOpenReview ↗

Response to 1ERg 3/3

Question 3: Why favor attention over recurrence?

Response 3:

Prior work documents the recall limitations of fixed-size recurrent states (Arora et al., 2024; Jelassi et al., 2024). This motivates hybrid architectures that combine recurrence’s efficiency with attention’s stronger long-range retrieval capacity, assigning short-distance dependencies to recurrence and long-distance dependencies to attention (Chen et al., 2026, §5.1).

Learning retrieval earlier within a limited pretraining context does not necessarily produce the best allocation for subsequent context extension. Our hypothesis is that recurrence-dominated retrieval during initial pretraining can leave attention underdeveloped, requiring additional functional reorganization when longer contexts place greater demands on recurrent memory. BLD encourages attention to acquire retrieval earlier; recurrent layers remain fully trainable throughout subsequent training.

We tested this hypothesis’s practical implication by extending both 1B models from 16K to 128K with the same approximately 5B additional tokens and training schedule, updating all parameters:

MetricVanillaBLD
RULER, 1K–32K ↑27.9834.91
Recall ↑39.3039.93
CSR ↑48.6049.60
WikiText PPL ↓21.1820.65
128K dev NLL ↓1.87341.8628

BLD retains a 6.93-point RULER advantage after matched 128K continued pretraining, alongside better general-capability and language-modeling metrics. Together with our attention-head interventions (§4.5), these results support establishing attention’s retrieval role early, with benefits persisting through long-context adaptation. They provide model-level evidence of complementarity, while direct causal verification of individual recurrent capabilities remains open.

Question 4: Scope and practical significance.

Response 4:

Our 3:1 configuration and experimental setup follow the established practices of representative frontier hybrid models (e.g. Kimi K3, Qwen3.8-Flash-Next) discussed in Related Work. Our focus is on understanding and improving training dynamics within these practical configurations; identifying optimal mixing ratios or exhaustively validating all architecture–data combinations is beyond the scope of this study.

Our findings suggest that BLD is particularly relevant when recurrence gains an early advantage in retrieval learning and suppresses attention’s learning signal, although its applicability beyond the evaluated settings remains to be established.

Question 5: Robustness and training budget.

Response 5:

Given the computational cost of our multi-scale study, repeating every scale with multiple seeds is challenging. We therefore repeated the representative 390M model across three training seeds. BLD improves training loss, Recall, RULER, and WikiText PPL in all three paired runs:

MethodTraining loss ↓CSR ↑Recall ↑RULER ↑WikiText PPL ↓
Vanilla3.1610 ± 0.003843.05 ± 1.0624.69 ± 3.5419.83 ± 3.4129.31 ± 0.43
BLD3.1418 ± 0.001944.11 ± 0.2332.16 ± 1.8528.65 ± 3.3327.93 ± 0.16

Values are mean ± sample standard deviation; RULER covers 1K–16K.

This table shows that the advantage is consistent across seeds.

Furthermore, at the 1B scale, BLD’s advantage persists after an additional 5B tokens of context-extension training to 128K, with all parameters trainable. The results are shown in the table in Response 3.

These results demonstrate persistence under substantial additional training.

If you have any remaining concerns, we would be happy to address them. If these clarifications and additional results resolve your concerns, we hope you will consider updating your score to more accurately reflect the paper’s scope and contributions.

Thank you for the explanations and the additional experiments.

However, I still have concerns regarding the underlying mechanism.

An attribution score might simply reflect a layer processing information that has already been retrieved by earlier layers, rather than the layer performing the retrieval itself. Furthermore, improved overall model performance does not, by itself, demonstrate that attention and recurrent layers have developed a superior division of labor—specifically, that attention is better suited for long-range retrieval while recurrent layers (KDA) are better for short-range processing, and that this complementarity is the source of the performance gains. It remains unclear under what other training settings BLD would be beneficial (methodological conditions for application are unclear).

Because explaining the mechanism is a core contribution of this paper, these unresolved issues are significant to my assessment. So I maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.