- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Frequency-Aware Attention. The authors observe that LLMs often struggle with rare tokens and hence decide to make token frequency a core design decision, baked into the attention mechanism itself in three ways: a frequency-dependent bias to attention logits, a rescaling of value vectors through a learned function of token rarity, and frequency-conditioned gating. The authors proceed to train a series of models ranging from 150M to 1.2 B parameters and evaluate model performance across a variety of benchmarks, showing consistent performance increases. Finally, the authors perform comprehensive ablations to isolate the effects of each component on terminal performance.
Critical strengths
First and foremost, this is objectively good science. I appreciate the multiseed evaluations, the extensive appendix, exhaustive ablations, efficiency/systems level details. No implementation details are left ambiguous. Additionally, the consistent empirical improvements, particularly lower loss per matched FLOPs, and comparisons against existing frequency-aware methods are compelling.
Critical weaknesses
I am not convinced that we can attribute the demonstrated improvements to token frequency itself.
- Table 4 in Appendix Q suggests that replacing the frequency table with a generic learnable token-specific scalar yields identical perplexity, even when initialized from constant or randomly shuffled assignment. Moreover, the vectors initialized from constant and shuffled values have near-zero correlation with frequency at the end of training, so the model achieves the same terminal performance without recovering the underlying frequency ordering. While the fixed frequency prior achieves modest improvements downstream, this suggests that much of the improvement might just come from providing additional token-specific information to attention.
- For example, one particular ablation the authors could run is rescaling the value vectors with a fixed learned scalar, instead of something frequency-dependent. If performance drops, this verifies the importance of IWVS specifically
Is there any mechanistic explanation for why this works?
- It’s not clear that attending to rare context tokens should meaningfully improve predictions of rare targets. While FAB and IWVS modify how attention should be distributed across context tokens, D0 PPL evaluates predictions of target tokens. This lacks any formal connection.
- The paper would benefit from experiments that show either that improvements in rare token prediction are specifically due to increased attention to rare tokens in context. Alternatively, some theoretical analysis/an example describing why frequency-dependent attention improves prediction quality should be sufficient.
Cross-architecture results are not convincing
- The GPT NeoX experiments show no perplexity improvement from FAB and IWVS, but FAA does improve performance. As such, we need an IG baseline to fully determine whether frequency-specific priors provide meaningful gains. If it turns out that the frequency priors don’t help, I’d be very curious as to why.
Questions for authors
I'm most concerned about attributing improvements to the token frequency prior, so if you can convince me here, I'll change my score from 2 to 3.
Thank you to the Area Chair and the reviewers for your time and feedback on our submission. After reviewing the comments, we have decided to withdraw the paper to further strengthen our work and address the points raised. We appreciate your efforts during the review process.