acceptodds
Under review as a conference paper at ICLR 2027

Count Is Not Enough: Inference-Time Mutation of Local–Global Attention

Abstract

Deployed hybrid attention models such as Gemma-3 interleave a small number of global-attention layers with sliding-window local layers in a fixed 5:1 ratio. We study how much of that trained pattern is actually load-bearing at inference time, by mutating layer types and window sizes of the frozen checkpoint — masking-only interventions that change no weights — and measuring exact negative log-likelihood (NLL), key–value cache memory, and decode throughput in one matched harness on A100-40GB GPUs. We re-implement NLL-guided fullattention layer selection, the strongest published training-free selection method, and compare it against uniform, positional, native-prefix, and periodic selection strategies at matched global-layer budgets. The result is a two-regime finding. At 1B parameters the trained pattern is near-optimal: NLL-guided selection improves on it by only −0.006 NLL (bootstrap 95% CI [−0.009, −0.004]), and cheap heuristics track the expensive baseline. At 4B parameters the trained pattern is beatable: NLL-guided selection finds a different set of five global layers that improves NLL by −0.085 (CI [−0.093, −0.077]) — the count of global layers is necessary but not sufficient, and which layers are global matters as models grow. A second finding is that the local window is a one-way knob: shrinking it from 512 to 128 tokens at 1B is free, while enlarging it past the trained value is catastrophic (NLL 3.66 →6.72 at 4k context). Converting global layers to local reduces KV-cache memory by up to 20× at 16k context with at most 2% decode-throughput cost at 1B, making the demoted configuration a practical long-context deployment on memory-bound cards.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid attention, sliding window, Gemma-3, inference-time mutation, KV cache, layer selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BZYwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper changes which layers use global attention and how many nearby tokens local attention can see, without retraining Gemma-3 1B or 4B. Changing the global layers lowers 4B WikiText-2 NLL from 2.9687 to 2.8836 while keeping model weights and positional encoding fixed. The experiments also measure how window size affects quality, how much cache is allocated, and how fast the 1B model decodes.

The authors keep the positional encoding (RoPE) fixed when changing which tokens can attend to each other. This makes the comparison cleaner: the measured effect is less likely to come from an accidental change to RoPE.

The paper shows a useful pitfall: letting local attention see more tokens can hurt badly. At 4K context, increasing the 1B window from 512 to 1024 raises NLL from about 3.66 to 6.72.

The paper does not yet show that the memory-saving settings work well on long inputs. Quality tests stop at 4K; the 16K test only allocates the cache and does not run the model. Decoding speed is measured only on the 1B model with eager kernels.

The baseline results need clearer labels: Table 1 calls the greedy variant the baseline, and Figure 1’s caption disagrees with its legend. The single-pass 4B results use a different input length, so the two methods’ gains cannot be compared directly. The 1B numbers in Table 1 differ by 0.0046, but the text says 0.0061.

Critical strengths

The authors keep the positional encoding (RoPE) fixed when changing which tokens can attend to each other. This makes the comparison cleaner: the measured effect is less likely to come from an accidental change to RoPE.

The paper shows a useful pitfall: letting local attention see more tokens can hurt badly. At 4K context, increasing the 1B window from 512 to 1024 raises NLL from about 3.66 to 6.72.

Critical weaknesses

The paper does not yet show that the memory-saving settings work well on long inputs. Quality tests stop at 4K; the 16K test only allocates the cache and does not run the model. Decoding speed is measured only on the 1B model with eager kernels.

The baseline results need clearer labels: Table 1 calls the greedy variant the baseline, and Figure 1’s caption disagrees with its legend. The single-pass 4B results use a different input length, so the two methods’ gains cannot be compared directly. The 1B numbers in Table 1 differ by 0.0046, but the text says 0.0061.

Questions for authors

Does the memory-saving setup keep retrieval or question-answering accuracy at 16K?

How do greedy and single-pass selection compare at the same input length?

Which 1B improvement is correct: 0.0046 or 0.0061?

Reviewer_T3y9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether the local–global attention layout of an already trained hybrid model can be improved at inference time without updating weights. Using frozen Gemma-3 checkpoints, it changes which layers use global attention and the window sizes of local layers, while preserving their trained positional embeddings. It compares layer-selection rules and measures language-model likelihood, KV-cache allocation, and decode throughput.

The main observations are that changing global-layer placement improves measured likelihood more at 4B than at 1B, enlarging local windows can substantially degrade quality, and converting global layers to local attention reduces cache requirements. The contribution is an empirical study of trained hybrids, with potential implications for deployment.

Critical strengths
  1. The study asks a useful question about already trained hybrid models. Treating the native layout as a competitor provides a meaningful reference for inference-time changes. Preserving the trained RoPE basis also addresses an important source of confounding when changing attention masks.

  2. Layer selection and evaluation use separate data. Validation-based selection, multiple resampling seeds, and paired bootstrap intervals strengthen the evidence for likelihood improvements within the tested settings. The study also includes both cache allocation and actual decode measurements.

Critical weaknesses

W1. The claimed faithful baseline differs from the published procedure

Tang et al.[1] score the benefit of enabling full attention in one layer while other layers use local attention during prefill, using answer-token NLL and emulating full-attention decode. Here, the single-pass baseline instead scores demotion from an all-global configuration using language-model NLL. These are different interventions. Moreover, the headline table labels the iterative greedy variant as the baseline, while the separate 4B single-pass result uses shorter sequences than the greedy result. Adaptation to trained hybrids is reasonable, but its attribution and comparison need to be explicit, with both procedures evaluated on identical sequences and windows.

W2. The largest memory savings lack matching quality and runtime evidence

The strongest cache reduction is measured at 16k context with all-local attention and a 128-token window. The main quality results use shorter contexts and native windows, while throughput covers only 1B at batch size one with a roughly 3k-token prompt; it does not include the all-local, 128-window configuration. Consequently, the quoted quality cost and throughput change do not validate the configuration delivering the largest savings. Allocating smaller cache tensors alone also does not establish successful long-context decoding. A paired quality–memory–runtime evaluation, including a task requiring distant information, is needed to support the deployment claim.

W3. The scale and positional-encoding interpretations are not isolated

The 1B and 4B comparisons change context length, native window, and architecture alongside parameter count. Thus, their different selection gains do not establish a scale-dependent transition. Likewise, the window-enlargement cliff is observed, but the proposed RoPE explanation is not separated from changes in the attention distribution and exposure to previously masked keys. Matched-context selection experiments and a targeted positional-encoding control would strengthen these interpretations. Otherwise, the conclusions should remain checkpoint-specific, and the universal advice to never enlarge windows should be narrowed.

[1] Tang, Qiong, et al. "NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation." arXiv preprint arXiv:2606.27791v1. 2026.

Questions for authors
  1. Which parts of the published NLL-guided procedure were retained or adapted? Please specify the background attention configuration, scoring tokens, and greedy initialization, and compare single-pass and iterative selection under identical evaluation settings.

  2. What quality and runtime does the maximum-saving configuration achieve? Please evaluate 16k, all-local attention with a 128-token window, including long-range task performance and cached decoding correctness, alongside memory and throughput.

  3. Does the two-regime finding persist under matched context lengths? What evidence distinguishes the proposed RoPE mechanism from the broader distribution shift caused by exposing additional keys?

  4. Can the strategy definitions and reported results be made consistent? Please provide exact layer sets, explain the 1B aggregate discrepancy, and report the stated Alpaca evaluation.

1 comment
Reviewer_T3y9OpenReview ↗

Response to the rebuttal

In my view, the rebuttal is heavily marked by AI-generated content and consists merely of concessions; I find the data provided unconvincing. Considering the overall quality of the paper, I will maintain my score.

Reviewer_X324OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper carries out an analysis of sliding-window attention (SWA) layer patterns in Gemma 1B and 4B checkpoints. Specifically, it asks whether the choice of which layers are SWA versus full attention (FA) matters -- more than just the number of layers which are SWA. It analyses this question using negative log-likelihood guided selection and evaluations on the Wikitext dataset and also looks at the impact of the SWA vs FA choices on memory usage and throughput, finding that while memory usage may decrease by using SWA, throughput doesn't proportionally improve.

Critical strengths

An analysis of SWA vs FA can be very useful in improving LLM inference, making it quicker and more efficient.

Critical weaknesses
  • The writing is incredibly dense in several places and with weird phrasing, impeding understanding. For example, there are confidence intervals and numbers scattered all throughout the text, making it hard to understand what the results really are.
  • The experimentation is very limited, restricted to just Gemma 1B and 4B and the wikitext2 dataset, making it very hard to assess how these findings generalise. There needs to be a lot more thorough experimentation on several datasets and models to make claims as strong as those the authors are making. The metric choice is also flawed, looking at the NLL alone is insufficient as it doesn't tell where the pass@1/k is affected, i.e., task success.
  • Some of the explanations for enlarging the SWA window need to be substantiated with experiments. In general there are either several unsubstantiated claims or those made strongly without appropriate evidence.
  • The work is not very novel compared to already-cited work such as Jolicoeur-Martineau et al. (2026), and some work such as Super Apriel by SLAM Labs et al. (2026) which is very relevant to questions of layer selection in hybrids and performs a detailed analysis/investigation on this question is not cited.
Questions for authors

The weaknesses list most but not all issues I have with the paper, but I don't believe there are specific answers that would materially affect my assessment. The paper needs a lot of work and is currently far from thorough in terms of experiments versus claims, and so while the idea/application is interesting, I'm not sure a rebuttal can convince me to change my assessment due to the volume of work and rewriting required.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.