- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper changes which layers use global attention and how many nearby tokens local attention can see, without retraining Gemma-3 1B or 4B. Changing the global layers lowers 4B WikiText-2 NLL from 2.9687 to 2.8836 while keeping model weights and positional encoding fixed. The experiments also measure how window size affects quality, how much cache is allocated, and how fast the 1B model decodes.
The authors keep the positional encoding (RoPE) fixed when changing which tokens can attend to each other. This makes the comparison cleaner: the measured effect is less likely to come from an accidental change to RoPE.
The paper shows a useful pitfall: letting local attention see more tokens can hurt badly. At 4K context, increasing the 1B window from 512 to 1024 raises NLL from about 3.66 to 6.72.
The paper does not yet show that the memory-saving settings work well on long inputs. Quality tests stop at 4K; the 16K test only allocates the cache and does not run the model. Decoding speed is measured only on the 1B model with eager kernels.
The baseline results need clearer labels: Table 1 calls the greedy variant the baseline, and Figure 1’s caption disagrees with its legend. The single-pass 4B results use a different input length, so the two methods’ gains cannot be compared directly. The 1B numbers in Table 1 differ by 0.0046, but the text says 0.0061.
Critical strengths
The authors keep the positional encoding (RoPE) fixed when changing which tokens can attend to each other. This makes the comparison cleaner: the measured effect is less likely to come from an accidental change to RoPE.
The paper shows a useful pitfall: letting local attention see more tokens can hurt badly. At 4K context, increasing the 1B window from 512 to 1024 raises NLL from about 3.66 to 6.72.
Critical weaknesses
The paper does not yet show that the memory-saving settings work well on long inputs. Quality tests stop at 4K; the 16K test only allocates the cache and does not run the model. Decoding speed is measured only on the 1B model with eager kernels.
The baseline results need clearer labels: Table 1 calls the greedy variant the baseline, and Figure 1’s caption disagrees with its legend. The single-pass 4B results use a different input length, so the two methods’ gains cannot be compared directly. The 1B numbers in Table 1 differ by 0.0046, but the text says 0.0061.
Questions for authors
Does the memory-saving setup keep retrieval or question-answering accuracy at 16K?
How do greedy and single-pass selection compare at the same input length?
Which 1B improvement is correct: 0.0046 or 0.0061?
Response to the rebuttal
In my view, the rebuttal is heavily marked by AI-generated content and consists merely of concessions; I find the data provided unconvincing. Considering the overall quality of the paper, I will maintain my score.