acceptodds
Under review as a conference paper at ICLR 2027

Frequency-Aware Attention: Tokenizer-Induced Self-Information as an Architectural Inductive Bias

Abstract

The performance of Transformer-based large language models (LLMs) is heavily influenced by the power-law token-frequency distribution induced by the tokenizer and corpus, leading to disparate learning dynamics in which LLMs struggle with long-tail tokens. Prior approaches inject token-frequency statistics through data, loss, output-logit, tokenizer, or embedding interventions; we instead make these statistics explicit as an architectural prior within Transformer attention. We introduce Frequency-Aware Attention (FAA), which injects corpus-derived self-information—defined as the negative log-probability of tokens—as an explicit prior into the attention mechanism in the form of attention-logit bias, value scaling, and gating. At the 150M scale, gate-free FAA—using only attention-logit bias and value scaling—reduces perplexity (PPL) by 3.1% and rarest-decile perplexity (D0 PPL) by 7.2% in a representative run, with only 0.043% additional parameters. Full FAA reduces PPL by 4.9% and D0 PPL by 8.9% on average across three seeds, with approximately 4.9% additional parameters. It also improves ARC-E and ARC-C accuracy by 1.89 and 2.05 percentage points, respectively. Our results suggest that making token-frequency statistics architecturally explicit yields a useful inductive bias for language modeling; PPL reductions persist at 450M and 1.2B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformers, attention mechanisms, language modeling, token frequency, inductive bias, rare-token learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QwNYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Frequency-Aware Attention. The authors observe that LLMs often struggle with rare tokens and hence decide to make token frequency a core design decision, baked into the attention mechanism itself in three ways: a frequency-dependent bias to attention logits, a rescaling of value vectors through a learned function of token rarity, and frequency-conditioned gating. The authors proceed to train a series of models ranging from 150M to 1.2 B parameters and evaluate model performance across a variety of benchmarks, showing consistent performance increases. Finally, the authors perform comprehensive ablations to isolate the effects of each component on terminal performance.

Critical strengths

First and foremost, this is objectively good science. I appreciate the multiseed evaluations, the extensive appendix, exhaustive ablations, efficiency/systems level details. No implementation details are left ambiguous. Additionally, the consistent empirical improvements, particularly lower loss per matched FLOPs, and comparisons against existing frequency-aware methods are compelling.

Critical weaknesses

I am not convinced that we can attribute the demonstrated improvements to token frequency itself.

  • Table 4 in Appendix Q suggests that replacing the frequency table with a generic learnable token-specific scalar yields identical perplexity, even when initialized from constant or randomly shuffled assignment. Moreover, the vectors initialized from constant and shuffled values have near-zero correlation with frequency at the end of training, so the model achieves the same terminal performance without recovering the underlying frequency ordering. While the fixed frequency prior achieves modest improvements downstream, this suggests that much of the improvement might just come from providing additional token-specific information to attention.
  • For example, one particular ablation the authors could run is rescaling the value vectors with a fixed learned scalar, instead of something frequency-dependent. If performance drops, this verifies the importance of IWVS specifically

Is there any mechanistic explanation for why this works?

  • It’s not clear that attending to rare context tokens should meaningfully improve predictions of rare targets. While FAB and IWVS modify how attention should be distributed across context tokens, D0 PPL evaluates predictions of target tokens. This lacks any formal connection.
  • The paper would benefit from experiments that show either that improvements in rare token prediction are specifically due to increased attention to rare tokens in context. Alternatively, some theoretical analysis/an example describing why frequency-dependent attention improves prediction quality should be sufficient.

Cross-architecture results are not convincing

  • The GPT NeoX experiments show no perplexity improvement from FAB and IWVS, but FAA does improve performance. As such, we need an IG baseline to fully determine whether frequency-specific priors provide meaningful gains. If it turns out that the frequency priors don’t help, I’d be very curious as to why.
Questions for authors

I'm most concerned about attributing improvements to the token frequency prior, so if you can convince me here, I'll change my score from 2 to 3.

Reviewer_BZGGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Frequency-Aware Attention, which incorporates corpus token-frequency statistics into attention through logit bias, value scaling, and output gating. Experiments at 150M, 450M, and 1.2B parameters show improvements in overall and rare-token perplexity, alongside gains on several downstream benchmarks.

Critical strengths
  • faa consistently improves overall and rare-token perplexity across multiple model scales. Gate-matched ablations show gains beyond the existing content gate.
  • The gate-free variant achieves meaningful improvements with only 0.043% additional parameters. Extensive ablations help assess the contribution of each component.
Critical weaknesses
  • Learned token tables and alternative prior assignments achieve nearly identical ppl, leaving the specific advantage of corpus-frequency conditioning unclear.
  • Cross-architecture and larger-scale experiments lack IG-only controls. It is therefore unclear how much of their improvement comes from frequency conditioning rather than the existing gate.
Questions for authors
  • Could you report D0 and complete decile ppl for the midpoint and shuffled-initialized learned-token controls? Given their nearly identical overall ppl and weak frequency correlations, what measurable advantage specifically depends on the corpus-frequency mapping?
  • Do you have an IG-only Neox comparison, or a gate-matched comparison at a larger scale? These results would materially affect whether I interpret the transfer results as evidence for frequency conditioning beyond the content gate.
  • Have you compared frequency-dependent IWVS against token-independent learnable per-layer/head value scales, with comparable initialization? This would help establish whether the frequency-dependent response is necessary.
  • Appendix N shows that excluding evaluation data changes the prior statistics only slightly. Have you also evaluated model ppl/D0 using the train-only prior, rather than only comparing prior distributions and decile assignments? Please distinguish inference-time replacement from training with the train-only prior.
Reviewer_m9fdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Frequency-Aware Attention (FAA), which uses corpus-derived token frequencies as an explicit prior within attention. It introduces three components: a frequency-dependent attention-logit bias (FAB), value scaling (IWVS), and output-gate modulation (FAIG). The authors evaluate models from 150M to 1.2B parameters trained on approximately 103.4B tokens and report lower perplexity, particularly for rare tokens, alongside downstream improvements.

Critical strengths
  1. The empirical gains are clear: at 150M, FAA reduces overall PPL by 4.9% and rarest-decile PPL by 8.9% across three seeds. PPL improvements also hold at 450M and 1.2B, supporting the method’s effectiveness.

  2. The paper is clearly written and easy to follow. The distinct roles of FAB, IWVS, and FAIG make the design intuitive and help readers understand how token frequency enters the attention mechanism.

Critical weaknesses
  1. The relevance to modern frontier LLMs remains unclear. The experiments cover models up to 1.2B parameters trained on about 103B tokens, leaving open whether the motivating frequency-related limitations remain significant at much larger scales. Direct validation would be expensive, but a deeper discussion of how these limitations might change with model capacity and token exposure is needed. This uncertainty about the broader impact is the main factor behind my rating.

  2. The paper does not compare against frequency-balanced data sampling. Oversampling examples containing rare tokens is a natural alternative to modifying attention. Without this comparison, it is difficult to judge the added value of the proposed architectural intervention. A comparison under matched training FLOPs would help establish whether FAA offers greater benefits than simply allocating more training exposure to rare-token examples.

  3. The explanation of why frequency conditioning works remains incomplete. The component ablations are useful, but improved PPL and increased attention to rare tokens do not establish the mechanism behind the gains. For example, an analysis of how FAB and IWVS change gradient allocation across frequency groups could connect the design to rare-token learning. Controlled experiments varying token frequency while holding the underlying prediction task fixed could also help distinguish frequency-specific benefits from general token conditioning.

Questions for authors
  1. The learned-token controls achieve nearly the same PPL as the corpus-frequency prior despite weak alignment with actual token frequencies. How do the authors interpret this result? Does the benefit come primarily from token rarity or from a more general token-conditioned modulation mechanism?

  2. IWVS alone worsens both overall and rare-token perplexity, but improves performance when combined with FAB. How do the authors explain this interaction?

Comments on the paper

1 comment

Thank you to the Area Chair and the reviewers for your time and feedback on our submission. After reviewing the comments, we have decided to withdraw the paper to further strengthen our work and address the points raised. We appreciate your efforts during the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.