acceptodds
Under review as a conference paper at ICLR 2027

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent-Depth Language Models

Abstract

Recurrent-depth language models repeatedly apply shared network blocks to refine latent representations, enabling additional test-time computation without generating explicit intermediate reasoning tokens. However, each recurrent step recomputes full self-attention over the entire context, repeating global attention routing whose cost is quadratic in context length. We study how attention routing evolves across recurrent depth and uncover a consistent separation in convergence timescales: routing-related quantities, including the attention support and the attention distribution, stabilize substantially earlier than representation-related quantities such as hidden states and attention outputs. This suggests a two-stage structure in recurrent inference, where the model first discovers a sparse working set of relevant context and then continues refining representations over largely the same routing support. Motivated by this structure, we introduce WISE (Working-set Inference with Support Exploitation), a training-free method that uses unrestricted global attention during early recurrent steps to discover a block-structured working set, and then reuses it as the routing support in later steps while keeping recurrent refinement and within-support attention computation dynamic. Controlled interventions show that discovering the working set over multiple recurrent steps, rather than from the first step alone, yields more effective working sets, and that reusing only the routing support better preserves model behavior than more restrictive forms of late attention reuse. Across multi-hop QA benchmarks, WISE largely preserves the behavior of full-attention inference, while matched context-scaling experiments reveal an increasingly favorable quality–efficiency tradeoff as the routing support becomes increasingly sparse with longer contexts. A sparse-attention implementation translates this structured sparsity into practical GPU acceleration, achieving up to a late-step attention speedup over native FlashAttention at 4K context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent depth, latent reasoning, sparse attention, efficient inference, test-time compute, working-set attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5gcyOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper has a clear motivation, which originally comes from a mechanism discovery in Figure 1. The paper shows that attention routing stabilizes before the representation refinement is complete. That means we can reduce the attention calculation in later refinement steps, as the routing converges earlier. This paper proposes WISE, the working set inference with supporting exploitation. It balances performance and efficiency based on the observation from the above paragraph. The experiments support both the mechanism and the computational implications. The proposed WISE yields up to1.76x attention speedup over native Flash Attention at 4K context, indicating the effectiveness of the mechanism dis-observation and the proposed method.

Critical strengths
  • The motivation is substantially clear in this paper. The observation acts as a very important foundation for the proposed method. The experiment shows that the proposed method has a rational, reasonable speedup for the recurrent transformer models. The contribution may be important for the recurrent-style models.
  • The experiment is well designed in the direction of proving the performance. Table 1, Table 2, and Figure 2 show the efficiency improvement gained by the proposed method compared to the other reasonable baselines.
Critical weaknesses
  • It is necessary to prove that the approach can be applied to diverse model series, instead of only using one model to be responsible for all the results. Because one model may have a coincidental combination gain for the result, different model architecture families should be tested.
  • The paragraph's length is a bit long for reading, although it is clear to understand. I think we can condense the explanations and shorten the repeated claims and proofs. Save more space for extensive experiments. For example, we can test more model families and benchmarks. That will be more useful and make the paper's novelty and contribution solid.
Questions for authors

I don't have any other questions for this paper. If the experimental flaws can be addressed, I would consider this paper a significant contribution.

3 comments

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_5gcyOpenReview ↗

Response to author's rebuttal.

Thanks to the author for their effort in working on the weaknesses.

Only one model was added for the experiments that do not match the requirements of more model families.

Although I appreciate the general direction and the novelty of this work, the experimental scope concern is still not fully addressed. And I agree with reviewer JnR1 on the concern they found. Based on these, I will show my final assessment on the rating.

Reviewer_JnR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether attention routing in recurrent-depth language models stabilizes early while representations keep refining. WISE runs 12 unrestricted recurrent steps, forms the union of B=32 block supports covering 95% attention mass over steps 9-12, and reuses that support for the remaining 20 steps while recomputing hidden states, Q/K/V and within-support weights. Two backbones support the diagnostic phenomenon; Huginn experiments provide interventions (SizeMatched, Freeze-A, Truncate) and a sparse GPU implementation reporting up to 1.758x late-step attention speedup over native FlashAttention at 4K context and 1.355x for the full T=32 attention trajectory. The question is whether continued recurrence remains valuable once repeated global routing becomes avoidable.

Critical strengths

S1. Well-targeted interventions that make the mechanism claim testable. SizeMatched exactly controls support cardinality per routing unit, Freeze-A uses identical support while freezing attention weights, and Truncate shares the early trajectory and native exit path. These controls separate which support is kept from how much is kept, and the diagnostics are replicated on a second backbone, with asymmetric-threshold reversals, right censoring, absent cross-backbone F1 gains and a non-replicating sensitivity effect reported openly.

S2. A systems evaluation that is more careful than typical sparse-attention speedup claims. It uses native FlashAttention as the baseline, includes a same-backend support control and counts one-time schedule preparation, and the block-size sweep shows that greater sparsity does not automatically yield greater speedup.

Critical weaknesses

W1. The independent contribution over closest support-reuse methods is not isolated. Kascade discovers salient indices in dense anchor layers and recomputes sparse attention in reuse layers, IndexCache directly reuses top-k routing indices, and CLSA shares routing across layers while retaining layer-specific queries and refinement. WISE's plausible distinction is the recurrent-depth temporal mechanism, training-free dense discovery and the temporal union, not support-only reuse itself. The step-1 SizeMatched control is valuable but does not separate the four-step union from a step-12 snapshot, periodic refresh, or Kascade/IndexCache-style anchor strategies adapted to the same recurrent backbone, depth, block granularity and budget; raw speedup comparisons across DSA/CLSA architectures would not answer this. The theory bridge is also incomplete: adjacent rolling-set Jaccard at step 12 is 0.9674 but J(W12,W32)=0.8987, which shows low local drift rather than exact identification; the theorem's geometric convergence and strict margins are not measured, and retained mass on the Full trajectory does not by itself guarantee quality on WISE's own sparse trajectory.

W2. Practical efficiency and quality preservation are not fully established. Table 2's 1.758x (late attention) and 1.355x (full T=32 attention trajectory) are not full-model speedups, and the systems workload uses saved supports; the paper does not separately time block-mass collection, cumulative-threshold selection and the four-step union, so inclusion of every online discovery operation is unverified. No prefill/generation wall-clock, memory or throughput on matched workloads is given. On quality, a non-significant difference is not an equivalence test: Table 9's WISE-Full intervals permit roughly 1-2 F1 points of degradation, the 4K setting loses 3.3 points, and on the second backbone WISE-SizeMatched F1 differences are +0.0083 and -0.0043 without a consistent advantage. The teacher-forced gold-answer loss is only described qualitatively on page 7 and has no table or confidence intervals. These limits are partly disclosed but should also qualify the abstract and contribution statements.

Questions for authors
  1. Does the T=32 timing include block-mass collection, 95% threshold selection/sorting, the steps 9-12 union and schedule preparation individually? Please state the timing boundary and per-operation costs.
  2. What are the quality, coverage and end-to-end costs of budget-matched step-12 snapshots, periodic refresh and Kascade/IndexCache-style anchor controls on the same backbone?
  3. Do missed newly important blocks explain the 3.3-point loss at 4K, and does WISE's own sparse trajectory retain the stability/coverage properties the theorem assumes?
  4. Can the abstract qualify the multi-step discovery benefit as evidence on Huginn, and can the teacher-forced gold-answer loss be reported numerically with paired confidence intervals?
5 comments

Part 1/2 — W1, W2, Q1

We thank the reviewer for recognizing our "well-targeted interventions" and "a systems evaluation that is more careful than typical sparse-attention speedup claims." We address both weaknesses and all four questions across two replies.

W1. Independent contribution, closest controls, and theory

We agree that support reuse itself is not novel. Our contribution is measuring recurrent-depth routing–representation timescale separation and exploiting it without training. Moreover, WISE is the first work that designs support reuse for recurrent depth models. We added a deployable step-12 Mass-95 snapshot, periodic refresh at step 20, and a budget-matched anchor-style schedule (dense steps 1–10, 20, 28), on the same Huginn backbone and B=32 blocks. These are adaptations of anchor/refresh strategies, not faithful reimplementations of Kascade, IndexCache, or CLSA.

The N=30 paired comparisons below report WISE minus each alternative (positive ΔF1 and negative ΔNLL favor WISE):

TaskControlΔF1 [95% CI]Δgold NLL [95% CI]
HotpotQARefresh+0.0206 [−0.0063,+0.0581]+0.0137 [−0.0051,+0.0309]
HotpotQAAnchor+0.0197 [−0.0063,+0.0564]+0.0591 [+0.0348,+0.0838]
HotpotQASnapshot+0.0179 [0,+0.0452]+0.0048 [−0.0168,+0.0257]
2WikiRefresh+0.0086 [0,+0.0255]−0.0260 [−0.0547,+0.0004]
2WikiAnchor+0.0086 [0,+0.0255]−0.0011 [−0.0410,+0.0364]
2WikiSnapshot−0.0121 [−0.0325,0]−0.0064 [−0.0282,+0.0135]
HotpotQA 4KRefresh+0.0020 [−0.0105,+0.0177]−0.0430 [−0.0703,−0.0190]
HotpotQA 4KAnchor+0.0054 [−0.0081,+0.0215]+0.0274 [+0.0056,+0.0511]
HotpotQA 4KSnapshot+0.0141 [−0.0070,+0.0432]−0.0448 [−0.0709,−0.0212]

These mixed findings do not establish general superiority. In separate, cardinality-matched step-12 snapshot controls on independent N=1,000 cohorts:

TaskWISE−Step-12 ΔF1 [95% CI]
HotpotQA−0.0009 [−0.0069,+0.0047]
2Wiki−0.0040 [−0.0109,+0.0027]

Thus, evidence for delayed discovery is stronger than evidence for an incremental four-step union advantage. Theorem 6.1 is conditional on geometric convergence and strict margins, which we have not empirically established. Local Jaccard stability does not prove exact support identification at step 12.

W2. Full-model efficiency and quality preservation

The original 1.758×/1.355× figures were attention-only with saved supports, not full-model acceleration. New fully online measurements include discovery, scheduling, and model computation:

MethodComplete 4K prompt forward (ms)Full / method
Native Full5058.4281.0000×
WISE5106.2590.9906×
Snapshot5015.8851.0085×
Refresh5132.4710.9856×
Anchor5074.6000.9968×

Exploratory matched N=10, batch 1, two balanced repeats, uncontended A6000; complete prompt forward, not autoregressive request latency. Four-step WISE does not achieve online prompt-forward speedup. Memory/throughput and full-generation gains are not established.

Independent larger-cohort quality results:

TaskNFull F1WISE F1ΔF1 [95% CI]ΔEM [95% CI]
HotpotQA1,0000.24960.2565+0.0069 [−0.0016,+0.0154]+0.001 [−0.007,+0.009]
2Wiki1,0000.20860.2020−0.0066 [−0.0165,+0.0030]−0.008 [−0.017,+0.001]
HotpotQA 4K5000.33710.2915−0.0456 [−0.0636,−0.0289]−0.040 [−0.058,−0.022]

Nonsignificance is not equivalence. The 4K degradation was already disclosed in Sec. 7.5. Gold NLL with paired CIs appears in Q4 (Part 2).

Q1. Timing boundary and per-operation costs

The original T=32 attention timing included schedule construction but excluded online block-mass collection and selection, because it used saved supports. We now profile all discovery operations individually (ms; three predetermined 4K inputs):

MethodCollectionSelectionUnionSchedulingSparse reuse
WISE96.5094.1680.0860.61293.163
Snapshot23.5131.044—0.61685.507
Refresh123.6215.2140.0871.22786.107
Anchor82.7173.135—1.83988.157

These are separately instrumented ranges, already included in the complete-forward numbers—not additional charges. Collection, not the union operation, dominates discovery overhead.

Part 2/2 — Q2, Q3, Q4

Q2. Snapshot, periodic-refresh, and anchor quality, coverage, and cost

We now evaluate all three on the same Huginn backbone, three matched N=30 cohorts, B=32, using online support decisions. Here Snapshot means the deployable step-12 Mass-95 method, distinct from the cardinality-matched snapshot reported in W1.

TaskMethodF1EMGold NLLΔNLL vs Full [95% CI]Changed answers
HotpotQAFull0.163804.611800/30
HotpotQAWISE0.174404.6078−0.0040 [−0.0743,+0.0619]5/30
HotpotQASnapshot0.156604.6030−0.0088 [−0.0873,+0.0673]6/30
HotpotQARefresh0.153804.5941−0.0177 [−0.0767,+0.0418]5/30
HotpotQAAnchor0.154704.5487−0.0630 [−0.1321,−0.0017]6/30
2WikiFull0.40090.3004.023500/30
2WikiWISE0.39720.3004.0065−0.0170 [−0.0549,+0.0183]2/30
2WikiSnapshot0.40920.3004.0130−0.0105 [−0.0602,+0.0408]2/30
2WikiRefresh0.38860.3004.0325+0.0090 [−0.0318,+0.0560]5/30
2WikiAnchor0.38860.3004.0077−0.0159 [−0.0554,+0.0243]4/30
Hotpot 4KFull0.28850.06675.502500/30
Hotpot 4KWISE0.27520.06675.6919+0.1894 [+0.0762,+0.3399]6/30
Hotpot 4KSnapshot0.26110.06675.7367+0.2342 [+0.1099,+0.4019]7/30
Hotpot 4KRefresh0.27320.06675.7349+0.2324 [+0.1049,+0.4084]5/30
Hotpot 4KAnchor0.26990.06675.6644+0.1619 [+0.0511,+0.3114]4/30

F1/EM are fractions; NLL is nats/gold token (lower better). All CIs use paired 10,000-resample bootstraps; no multiplicity correction. Pairwise WISE–control CIs are in W1 (Part 1).

We separately audited sparse-step block density and Full-trajectory retained attention mass:

TaskMethodDensity (%)Retained mass (%)Common steps 13–32 (%)
HotpotQAWISE44.1896.6296.62
HotpotQASnapshot40.7596.0596.05
HotpotQARefresh42.7996.3996.57
HotpotQAAnchor41.7396.2096.59
2WikiWISE49.9096.9396.93
2WikiSnapshot46.7196.3796.37
2WikiRefresh48.2796.6596.82
2WikiAnchor47.1596.4596.81
Hotpot 4KWISE36.7096.1096.10
Hotpot 4KSnapshot33.3695.5195.51
Hotpot 4KRefresh35.6695.9196.12
Hotpot 4KAnchor34.5795.7496.17

The common-window metric counts dense refresh steps at 100% coverage. Full attention never supplies online support choices. Actual compute budgets differ:

MethodDense stepsSparse stepsCollection callsSchedule builds
Full1–32 (32)000
WISE1–12 (12)13–32 (20)161
Snapshot1–12 (12)13–32 (20)41
Refresh1–12,20 (13)13–19,21–32 (19)202
Anchor1–10,20,28 (12)11–19,21–27,29–32 (20)123

The matched online prompt-forward costs are given in W2 (Part 1); no method is uniformly best in quality and cost. Equal dense-step budgets do not imply equal runtime or support cardinality.

Q3. Missed blocks and WISE's own sparse trajectory

We probed unrestricted attention from WISE's sparse hidden states, rather than from Full, at steps 16/24/32 on 21 matched 4K failure/control pairs. At step 32, retained mass was 95.7595% (failures) and 95.7645% (controls); paired difference −0.0050 percentage points [95% CI −0.0140,+0.0043]. We did not find a reliable aggregate drift/missed-mass difference. This does not rule out individual newly important blocks or recurrent error amplification. The theorem's sufficient conditions remain unverified on actual sparse trajectories.

Q4. Qualify discovery claims; numerical gold loss

We agree the abstract should distinguish early routing stabilization, delayed discovery, and the unproven incremental benefit of four-step union. We will explicitly qualify union and online-acceleration claims.

Expanded independent WISE−Full teacher-forced gold NLL results (paired 95% CIs):

TaskNFull NLLWISE NLLΔNLL [95% CI]
HotpotQA1,0004.39314.3310−0.0621 [−0.0751,−0.0493]
2Wiki1,0004.66674.6669+0.0002 [−0.0141,+0.0152]
HotpotQA 4K5004.30494.4421+0.1372 [+0.1165,+0.1595]

This indicates a native HotpotQA NLL improvement, an unresolved difference on 2Wiki, and significant 4K degradation. We thank the reviewer for prompting these direct comparisons and clearer scope statements.

Clarification on Attention-Level Speedup and Full-Model Efficiency

We would like to further clarify why WISE achieves substantial attention-level acceleration while the improvement in complete model forward latency remains limited.

1. The limited full-model speedup is largely explained by the computational characteristics of Huginn.

Huginn has a relatively wide hidden dimension (5,280) and a large feed-forward intermediate dimension (17,920). Consequently, dense projections, feed-forward transformations, and other non-attention computations dominate its recurrent execution.

Our profiling provides the following measurements:

ComponentLatency
Native Attention Kernel226.66 ms
Complete Model Forward5,058.43 ms
Approximate Attention Kernel Share4.48%

The component and complete-forward measurements use separate matched-workload profiling cohorts, so the proportion is approximate.

Under Amdahl's Law, even eliminating all attention-kernel computation would yield an upper bound of approximately 1.047× full-model speedup for this computational profile.

Therefore, the limited overall acceleration is consistent with the architecture's computational composition, rather than indicating that the sparse attention implementation is ineffective.

2. WISE achieves substantial acceleration within the computation it targets.

Measurement ScopeSpeedup
Late-step Attention Kernel vs. Native FlashAttention1.758×
Full T=32 Attention Trajectory (saved supports)1.355×
Complete Online Model Forward0.991×

WISE specifically reduces repeated global attention computation while preserving dynamic Q/K/V projections, within-support attention weights, and recurrent representation refinement.

The first two measurements demonstrate that the discovered sparsity translates into genuine wall-clock acceleration of attention computation.

The complete online measurement additionally includes support discovery and scheduling overhead, which further limits the net benefit in the current implementation.

3. Implications for the contribution of WISE.

These results distinguish two aspects of our contribution:

  • Mechanism and algorithm: WISE identifies an early stabilization of attention routing relative to representation refinement, enabling redundant global routing computation to be avoided.
  • Systems realization: Our exact block-sparse implementation achieves substantial attention-kernel acceleration over native FlashAttention, while complete-model benefits remain constrained by the backbone's computational composition and online discovery overhead.

We therefore emphasize that the demonstrated attention-level acceleration is substantial, whereas complete-model acceleration is architecture- and implementation-dependent.

We thank the reviewer for encouraging us to clarify this distinction.

Reviewer_ftXsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper takes the recurrent budget as given and asks whether every recurrent step of a recurrent-depth language model needs unrestricted global attention. It measures a separation of convergence timescales: routing quantities — the attention support and the attention distribution — stabilize substantially earlier than representation quantities such as hidden states and attention outputs. The resulting training-free method, WISE, leaves attention unrestricted for an initial discovery phase, forms a block-level working set from the last few discovery steps, and freezes it for the remaining steps while hidden states, projections and within-support weights continue to be recomputed. Validation covers convergence diagnostics on two backbones, a set of matched interventions that vary how and when the support is chosen, a custom exact block-sparse kernel measured against FlashAttention, and context scaling up to the backbone's maximum supported length.

Critical strengths
  1. The central finding is a measurement that had not been made.Prior work on recurrent-depth efficiency tracks convergence on the representation axis — how much hidden states still move between loops. Separating where the model reads from how it transforms what it reads, and timing the two independently, is new. The finding is robust in the ways that matter: it survives a sweep of all four convergence criteria, holds on every diagnostic example, and replicates on a second, independently trained backbone. It also yields a concrete switching depth rather than remaining a description, and support stability is a diagnostic other work can adopt on its own.

  2. The structured sparsity is realized as wall-clock gain.A custom exact block-sparse kernel converts the discovered structure into measured speedup over native FlashAttention, so the benefit is demonstrated in wall-clock terms rather than only in FLOP counts.

Critical weaknesses
  1. The evaluation is too narrow, and too thinly instrumented, to support the paper's unconditional quality claims.All downstream quality results come from two benchmarks of the same task family — multi-hop extractive QA with distractor paragraphs — which is the structure most favorable to freezing a routing support, so the evaluation cannot show where the method stops working. Each uses a small fixed cohort, and no full split is reported anywhere. Quality then rests on a single metric, while two metrics the paper itself declares are never reported: Appendix B states that normalized exact match is reported alongside F1, and both Section 7.1 and Appendix B declare a paired teacher-forced gold-answer loss as a continuous measure of behavioral perturbation whose direction Section 7.3 cites — yet neither appears in any table.

  2. Claims exceed the evidence, and the scope of validity is never stated.The contribution that discovering the working set over multiple recurrent steps yields more effective working sets is not supported by the only ablation isolating the width of the discovery window, which Section C.5 itself reports as failing to replicate on held-out data. There is also no limitations section. The method needs enough recurrent depth to afford both a discovery phase and a useful reuse phase, which rules out shallower released loop models — Ouro, for instance, is trained with four recurrent steps and cannot host a twelve-step discovery phase at all. It also presumes a working set that is genuinely sparse for the task. Unstated, neither condition lets a reader distinguish "inapplicable by construction" from "untested."

Questions for authors
  1. Could you report the main comparison on a full benchmark split rather than a fixed 100-example cohort? Relatedly, could you comment on the low absolute level of the matched full-attention baseline, which currently makes "largely preserves full-attention behavior" hard to interpret?
  2. Table 1 reports neither normalized exact match nor the teacher-forced gold-answer loss, although Appendix B declares both and Section 7.3 cites the latter's direction. Could both be added, with confidence intervals? Being continuous and already computed, the teacher-forced loss would resolve orderings that F1 on 100 examples cannot.
  3. Both behavioral benchmarks are multi-hop extractive QA with distractor paragraphs. Do you expect support reuse to hold for tasks requiring broad coverage, such as summarization or exhaustive listing, where the working set may not be sparse at any depth?
  4. Is there a minimum recurrent depth below which the method cannot be instantiated? A twelve-step discovery phase plus a useful reuse phase appears to rule out shallower released loop models such as Ouro. Would you state this and the sparsity requirement in a limitations section?
4 comments

Part 1/2 — Evaluation and quality metrics (W1, Q1, Q2)

We thank the reviewer for recognizing that "the central finding is a measurement that had not been made". We appreciate the detailed feedback and address both weaknesses and all four questions separately.

W1. Evaluation breadth and missing quality metrics

Response. We expanded the original N=100 evaluation to independent N=1,000 cohorts on HotpotQA and 2Wiki and N=500 at 4K, with paired 95% bootstrap CIs:

BenchmarkNFull F1WISE F1WISE−Full ΔF1 [95% CI]
HotpotQA1,0000.24960.2565+0.0069 [−0.0016, +0.0154]
2Wiki1,0000.20860.2020−0.0066 [−0.0165, +0.0030]
HotpotQA 4K5000.33710.2915−0.0456 [−0.0636, −0.0289]

We additionally report EM and gold-answer NLL, both with paired CIs (Q2), and LongBench MultiNews summarization (Q3). These address missing instrumentation and provide initial evidence beyond extractive QA. They are not full benchmark splits, and the 4K degradation is statistically significant.

W2. Temporal union and scope of validity

Response. We agree that delayed discovery and the incremental value of multi-step union should not be conflated. We added a per-routing-unit cardinality-matched step-12 snapshot:

BenchmarkNWISE−Step-12 ΔF1 [95% CI]
HotpotQA1,000−0.0009 [−0.0069, +0.0047]
2Wiki1,000−0.0040 [−0.0109, +0.0027]

The four-step union does not show a consistent F1 advantage over the snapshot. Delayed discovery is better supported than temporal union, and we will qualify the claim. Applicability also requires adequate recurrent depth and sufficiently sparse support (Q4).

Q1. Full split and low Full-attention F1

A1. The W1 table reports the new independent cohorts and paired intervals; these are not full splits. We acknowledge the relatively low absolute Full F1. Our intervention tests paired changes relative to that fixed backbone, not state-of-the-art QA accuracy. Nonsignificant F1 effects are not equivalence proofs. The significant 4K loss confirms the degradation already reported in Sec. 7.5/Fig. 2b of the original submission.

Q2. Normalized EM and teacher-forced gold-answer loss

A2. Both were already measured; we now report them explicitly on all 2,500 independent examples, with 10,000 paired example-bootstrap resamples.

BenchmarkFull EMWISE EMΔEM [95% CI]
HotpotQA0.1080.109+0.001 [−0.007, +0.009]
2Wiki0.1150.107−0.008 [−0.017, +0.001]
HotpotQA 4K0.1580.118−0.040 [−0.058, −0.022]
BenchmarkFull NLLWISE NLLΔNLL [95% CI]
HotpotQA4.39314.3310−0.0621 [−0.0751, −0.0493]
2Wiki4.66674.6669+0.0002 [−0.0141, +0.0152]
HotpotQA 4K4.30494.4421+0.1372 [+0.1165, +0.1595]

NLL is teacher-forced nats per gold token (lower is better). Thus, native HotpotQA improves in NLL, 2Wiki has an interval spanning zero, and 4K degrades in both EM and NLL.

Q3. Broad-coverage tasks

A3. We added LongBench MultiNews multi-document summarization, outside multi-hop extractive QA:

NFull ROUGE-LWISE ROUGE-LWISE−Full [95% CI]Block density
3026.711126.4716−0.2395 [−1.7409, +1.2322]50.07%

WISE retains 96.57% of future Full-attention mass. This is a preliminary cross-task result, not evidence of equivalence; exhaustive listing and highly diffuse support remain untested.

Q4. Minimum recurrent depth and sparsity

A4. Twelve discovery steps is a setting for T=32, not a universal minimum. WISE requires (i) sufficient depth for routing to stabilize, (ii) enough subsequent reuse to amortize discovery costs, and (iii) sufficiently sparse support for real speed gains. The present configuration cannot be applied directly to Ouro (T=4); shorter schedules are unvalidated. We will state these limits explicitly.

We thank the reviewer again for helping us strengthen the evaluation and narrow the claims.

Part 2/2 — Claims, generalization, and applicability (W2, Q3, Q4)

W2. Temporal-union claim and unstated limitations

We agree that delayed discovery and the incremental value of a four-step union are distinct claims. We added a step-12 snapshot baseline matching WISE's block cardinality for every routing unit.

The original matched N=100 cohorts show the effect of selecting support later:

BenchmarkFull F1Step-1 matchedStep-12 matchedWISE
HotpotQA0.22630.18360.21720.2317
2Wiki0.25160.24460.24630.2497
HotpotQA 4K0.33700.28970.30170.3037

However, on independent N=1,000 native-context cohorts, the union does not show a consistent advantage over the matched step-12 snapshot:

BenchmarkNWISE−Step-12 ΔF1 [95% CI]
HotpotQA1,000−0.0009 [−0.0069, +0.0047]
2Wiki1,000−0.0040 [−0.0109, +0.0027]

We therefore qualify the multi-step-union contribution and emphasize the better-supported finding: routing stabilizes earlier than representation, making delayed support discovery possible. We also explicitly acknowledge depth, sparsity, and amortization requirements below.

Q3. Broad-coverage tasks beyond extractive QA

We tested LongBench MultiNews, a multi-document summarization task:

NFull ROUGE-LWISE ROUGE-LWISE−Full [95% CI]Block density
3026.711126.4716−0.2395 [−1.7409, +1.2322]50.07%

WISE retains 96.57% of future Full-attention mass. This is preliminary evidence beyond distractor-based QA, not a proof of equivalence. Exhaustive listing and other tasks requiring broader coverage remain untested.

Q4. Minimum recurrent depth, attention sparsity, and convergence times

Attention sparsity is not peculiar to multi-hop QA. While softmax attention is mathematically dense, its probability mass is often concentrated on a relatively small subset of positions. Such effective sparsity is a broadly observed property of attention mechanisms, rather than an assumption specific to extractive QA.

Our MultiNews experiment also demonstrates a non-dense block working set outside the original task family. Importantly, however, attention concentration, block-level sparsity, and early support stabilization are distinct properties. We do not assume that every task or model exhibits them to the same degree.

Convergence depth is not universal. It varies with the backbone, task, context, and convergence criterion. Our diagnostics explicitly measure these differences. For example, on the independently trained Recurrent-Llama-T32 backbone:

TaskSupport convergenceAttention convergenceHidden convergence
HotpotQA5.077.008.93
GSM8K3.877.109.63

In both tasks, routing-related quantities stabilize before hidden representations, but the convergence depths differ. Consequently, the discovery schedule should be understood as model- and workload-dependent.

The 12-step discovery phase is an empirical configuration for Huginn with T=32, not a theoretical minimum required by WISE. The actual requirements are:

  1. Enough recurrent steps for routing support to become sufficiently stable.
  2. Enough subsequent reuse steps to amortize discovery and scheduling overhead.
  3. Sufficient effective block sparsity to make sparse execution beneficial.

For Ouro (T=4), our current 12-step schedule is indeed inapplicable by construction. However, this does not establish that support reuse is fundamentally impossible in shallow recurrent models. A shorter discovery schedule would need separate convergence measurements and quality/efficiency validation. We have not performed that evaluation and therefore make no claim of benefit on Ouro.

Likewise, if a task requires a broad working set, the resulting block density may eliminate the speed advantage. This is a practical limitation of exploiting sparsity, not a claim that attention sparsity itself is unique to QA.

We will explicitly state these applicability conditions and the untested shallow-depth/broad-coverage settings in the revised manuscript.

We thank the reviewer again for helping us strengthen the empirical evaluation and clarify its scope.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.

Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence for the Area Chair

Dear Area Chair,

Thank you for overseeing our submission. We would like to briefly summarize the reviewers' main concerns and our responses to facilitate your assessment.

1. Reviewer 5gcy — Generalization and Evaluation Breadth

  • Concern: Limited model diversity and evaluation scope.
  • Response: We clarified the cross-backbone validation on Recurrent-Llama-T32, where the routing–representation convergence gap consistently holds. We also expanded QA evaluation to 2,500 examples and added LongBench MultiNews summarization.

2. Reviewer ftXs — Quality, Discovery Mechanism, and Applicability

  • Concern: Small evaluation cohorts, missing metrics, insufficient evidence for temporal union, and unclear limitations.
  • Response: We added larger independent cohorts, EM and gold-answer NLL with confidence intervals. Step-12 snapshot comparisons clarify that delayed discovery is better supported than the incremental benefit of multi-step union. We also explicitly acknowledged the depth and sparsity requirements.

3. Reviewer JnR1 — Novelty and Practical Efficiency

  • Concern: Insufficient baseline controls, unaccounted discovery overhead, and uncertain quality–efficiency benefits.
  • Response: We added snapshot, refresh, and anchor-style controls, along with detailed online profiling. WISE achieves 1.758× attention-kernel acceleration.

Overall, the rebuttal strengthens the evidence for our central finding: attention routing stabilizes substantially earlier than representation refinement across recurrent-depth models. We have also clarified the distinction between this mechanistic contribution, its attention-level computational benefits, and the limitations of the current implementation.

We sincerely appreciate your consideration.

Best regards,

The Authors

Area_Chair_H9R4OpenReview ↗

reviewers 5gcy and ftXs

Dear reviewers 5gcy and ftXs please engage in the discussion process or acknowledge that all your concerns are addressed if they are

your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.