- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies learned routing failures in hybrid Mamba–attention models on associative recall. Its main experiment holds the number of correctly routed training steps fixed while varying when misrouting occurs. Early misrouting can cause lasting failure, whereas delaying the same deficit by about 50 steps largely preserves performance. The authors also measure attention-pathway suppression and test several routing interventions. A proposed mechanism offers a qualitative explanation, although its learning-rate scaling prediction is falsified.
Critical strengths
- The timing experiment is a useful and well-controlled contribution. Matching the amount of correct routing and including an equal-recovery control provide strong evidence that early routing history matters. This goes beyond observing that learned routers are difficult to train.
- The paper reports important negative results and limitations openly. Multi-seed experiments, failed controls, the falsified scaling prediction, and the corrected architectural artifact make the empirical investigation more informative.
Critical weaknesses
- The role of the optimization setup remains unresolved. The effect depends strongly on learning rate and occurs within the first few dozen updates, but the warmup schedule is not clearly documented. Lower learning rates substantially reduce or eliminate failure in the reported deficit experiments. These results leave open whether the phenomenon persists under a less aggressive training schedule. They also weaken the conclusion that permanent external routing information is necessary.
- The evidence does not yet establish that early lock-in explains the learned-router failures. The main timing experiments bypass the router. Similar attention-suppression trajectories are suggestive, but different routing methods may fail for different reasons. Testing learned routers under settings where the imposed deficit is harmless would help distinguish these explanations. Furthermore, Table 3 starts at step 500, after the reported sensitive interval, so it does not directly establish that useful routing signals are absent when they are needed.
Questions for authors
- Is learning-rate warmup used? If not, does adding warmup preserve the early lock-in effect?
- Do the learned routers in Table 1 still fail at the lower learning rate where the imposed deficit is harmless? If so, what evidence connects their failure to early lock-in?
- Are marginal-utility measurements available within the first 50 updates? What do they show before the attention pathway has been suppressed?
4 comments
Round-2 update for Reviewer Tmvr
Update: learned routers at the lower learning rate (Q2, W2). As promised, we report the remaining results. All runs use a constant learning rate of 1e-3.
- An imposed deficit is nearly harmless at this rate. With misrouting on steps 0–250 or 0–8000 and correct routing afterwards, 9/10 runs survive in each case (10/10 without a deficit).
- Oracle routing trains (5/5; accuracy 0.999).
- The four Table 1 routers (straight-through, marginal-utility, MoD, REINFORCE) all remain at chance (0/20; 5 seeds each), with query recall at the random level (≈0.06).
- A router trained to raise its selection probability at the labeled positions, i.e. one given an external routing signal, succeeds in 5/5 runs (accuracy 0.74; query recall 0.73, range 0.67–0.84), against 0/20 for the Table 1 routers (Fisher's exact test, p = 2×10⁻⁵). These are the runs of the label-position control described in our update to Reviewer 1i7q.
These results sharpen the paper's claim into two complementary findings. First, the task loss does not supply a usable routing signal: the Table 1 routers fail even at a learning rate at which the backbone recovers from misrouting. Second, at higher learning rates, and more readily at greater depth, misrouting additionally locks in the backbone (Sec. 6, including under warmup, and at depth 4 even at 1.5e-3). At 1e-3, a router given an external routing signal succeeds in every run, which indicates what a working router needs. At 3e-3 the same router fails (see our update to Reviewer 1i7q). We agree with the reviewer that different failures can have different causes; the abstract, TL;DR and introduction now present the two findings separately.
Thank you for the detailed rebuttal and substantial additional experiments. The new results address my main concerns and significantly strengthen the paper. In particular, the lower-learning-rate experiments are convincing. The finding that imposed misrouting is largely recoverable (9/10), while all four learned routers still fail (0/20), demonstrates that learned-router failure cannot be explained solely by early backbone lock-in. The success of the externally supervised router (5/5) further supports the distinction between insufficient routing supervision and backbone lock-in. The authors have adequately addressed my main concerns, and the revised empirical findings make a meaningful contribution. I have therefore raised my rating from Weak Reject to Weak Accept.
Thank you
Thank you for reading the new results so carefully and for raising your rating. Your questions on the learning rate led to the experiments that separate the two obstructions and the paper is clearer for it.
Response to Reviewer Tmvr
We thank the reviewer for the precise and constructive questions, which have helped sharpen the paper's central claim.
Q1, W1 (warmup and optimization setup). We thank the reviewer for pointing out that the schedule was not documented. The main protocol (Secs. 3–6) uses a constant learning rate of 3e-3 without warmup, and the depth experiments use a 2000-step linear warmup; the revision includes a per-experiment schedule table. To answer the question directly, we repeated the Sec. 6 protocol with a 2000-step linear warmup to 3e-3 (10 seeds per condition). Warmup does not remove lock-in; it moves the sensitive window to the end of warmup:
We agree that the effect depends on the learning rate, and we now state the conclusion more precisely. Under misrouting, a router trained on the task loss has only a few early steps of signal (Q3 below), which is why we argue that routing information must come from outside the task loss. The requirement that this information be permanent applies to the high-learning-rate regime in which lock-in occurs.
Q3, W2 (signal within the first 50 updates). Table 3 did not cover these early steps, and the new measurements address them directly. We measured δ at 16 points between steps 1 and 250 (5 seeds; learning rates 3e-3 and 1e-3).
Under misrouting, the signal therefore exists only within the sensitive window identified by the timing experiment, and it is lost there: at 3e-3 before the attention pathway is suppressed, at 1e-3 together with it. A label-free ranking score (the KL divergence between outputs with and without attention) shows no query signal in this period under any routing, including oracle. Table 3 and Fig. 1 now include these measurements.
Q2, W2 (learned routers at the lower learning rate). We appreciate this suggestion, which directly tests our account. We have run all four Table 1 routers at 1e-3 (5 seeds each), with an oracle-routing control that trains at this rate (5/5). Interpreting them requires knowing whether an imposed deficit is harmless at the same constant learning rate. Our earlier evidence for this used warmup, and the warmup results above show that the schedule matters. That control finishes tomorrow, and we will post the complete answer by Nov 13.
The revised PDF, with all changes marked in blue, will be uploaded by Nov 16.