acceptodds
Under review as a conference paper at ICLR 2027

Modulated Transformer

Abstract

Softmax based attention suffer from several challenges like attention sink and attention noise. A well known reason for which is that all attention scores need to sum to one. Hence, a natural remedy would be to explore alternatives to softmax. A recent advancement that replaces softmax is the DIFF attention which relies on correcting a base information carrying softmax attention map by subtracting another sequence dependent softmax attention map instead of per token-position dependent gate as in gated attention. We introduce MODULATED TRANSFORMER that computes a sequence dependent correcting mechanism by modulating the amplitude of a base softmax attention map signal by taking hadamard product with the second correcting attention map. Modulation of a base softmax signal by a suitable modulating function results in sharper and sparser final attention patterns. At 780m scale, Modulated Transformer surpasses softmax based Transformer in language modeling and achieves significant model size and token efficiency as indicated by a lower perplexity score with only 58% of the model size of 1.3B Vanilla Transformer with similar token budget. It also outperforms softmax based Transformer by 3.15 average accuracy points and by 3.24 points after pruning attention weights and is competitive with next strongest baseline DIFF on short-context language modeling tasks. With these results, the Modulated Transformer advances the abilities of existing language models and opens up a new avenue for research on developing new expressive and stable modulating functions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer, Self-Attention, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fXvmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a modulated transformer by modulating the amplitude of softmax attention map to handle the problem of attention sink and attention noise. Compared with vanilla transformer, this proposed method achieves model size and token efficiency.

Critical strengths
  1. The paper is easy to follow. The idea is simple but seems to be effective. The design is well motivated.
  2. Ablation studies are conducted and effectively demonstrate the choice of design.
  3. The AI use statement is clear. This is a model disclosure of AI involvement.
Critical weaknesses
  1. The claimed value of “model size and token efficiency” is not fully demonstrated. Actually, from the results, this cannot be derived.
  2. The central motivation, alleviating attention sink and attention noise, are not sufficiently and quantitively evaluated. In the qualitative evaluation in Figure 6, top and bottom rows are not head-to-head comparison.
  3. Figure 3 is not evidence that modulated scores are better, just showing the scores are lower.
  4. Modulated is not shown to be superior to Differential although it is built based on differential transformer.
  5. Table 2 does not have results for differential transformer.
Questions for authors

see weakness

Reviewer_iMRcOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Modulated Transformer, which uses a scaled sigmoid gate instead of the subtraction in Differential Transformer (DIFF). It shows lower perplexity and better average zero-shot accuracy than Vanilla at 130M–780M, and the results are similar to DIFF. The paper also reports that the 780M model achieves lower perplexity than a 1.3B Vanilla model, and that the accuracy gain remains after attention pruning.

Critical strengths

The method is simple, and it improves over Vanilla at all three model sizes. It also shows performance similar to DIFF with a different mechanism. The paper shows that epsilon in headwise RMSNorm can work like an implicit head gate. It also affects perplexity.

Critical weaknesses
  1. The contribution over existing methods is not clear.

The paper contrasts the method with Gated Attention (Qiu et al., 2025), which uses a per-position gate. However, Gated Attention is not compared in the experiments. In particular, the elementwise variant seems to be an important baseline. Without this comparison, it is hard to know whether the additional pairwise QK scores are really needed.

The proposed attention can be rewritten as a normalized softmax with a logit penalty and a query-dependent scale. Headwise RMSNorm removes this scale when epsilon is zero. So the improvement may come from the logit penalty, or from the head gating. The paper does not separate these two effects.

Also, the results are close to DIFF. Modulated is slightly better at 130M and 360M, and DIFF is slightly better at 780M. Since the forward throughput in Table 6 is also similar, it is not clear what multiplication offers over subtraction.

  1. The experiments do not fully support the claims.

Compared to Vanilla, Modulated also uses QK-Norm, headwise RMSNorm, and a different head/value setting. Only QK-Norm ablation is tested. I am not sure that the improvement comes from the modulation itself.

The attention sink and sparsity claims seem weak. Figure 6 shows only some selected heatmaps, and there is no quantitative result. Small attention weights do not always mean that unimportant information is removed. The small learned scale may also make the attention look sparse.

Questions for authors
  1. How does Vanilla perform with the same QK-Norm, headwise RMSNorm, and head/value dimensions? Also, could the authors compare with headwise and elementwise Gated Attention under the same setting?

  2. How does the method compare with softmax(A2 - A1) or softmax(A2 - softplus(λ - A1))? An epsilon sweep and an explicit output gate would help to separate the two effects.

  3. What is the main reason to use multiplication instead of subtraction? Since the results are close to DIFF, a comparison of training time, quality at the same compute, or variance across seeds would be helpful.

  4. Could the authors show quantitative attention sink and sparsity results across layers and heads? How much attention can be removed without performance drop?

Reviewer_jAmTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper divides attention into two groups of scores A_1 and A_2, and corrects the basic softmax attention by element wise multiplication. A_1 provides gating that relies on query key pairs and A_2 is responsible for allocating basic attention. Subsequently, apply Headwise RMSNorm to WV and combine it with QK norm. Compared to the subtraction correction of the DiT, multiplication correction is used here. Authors claim that this design can generate sharper and sparser attention, alleviate attention sink, and improve the utilization efficiency of parameters and training tokens.

Critical strengths
  1. The method is concise and has clear design differences. Gated operation is applied to each query key pair, which can change the relative weight of a single query between different keys. The expressive power is different from gating that only scales the entire attention head output. It is also a reasonable extension of the differential transformer correction approach (Subsec 2.1).

  2. The experiment covers three scales, and the improvement compared to Vanilla is relatively consistent. The average improvements in Table 1 are 1.49, 1.35, and 3.15 percentage points, respectively, supported by C4 percentile and training curve. Table 3 also indicates that even without QK norm, the modulated PPL is still superior to Vanilla, so it cannot be simply attributed to QK norm for all the benefits.

  3. Authors carefully discussed the numerical role of normalization. Sec 2.1 and Subsec A.3 explicitly state that the scale invariance of Headwise RMSNorm will be broken by \epsilon and provide ablation by removing scaling factors. This section has more analytical value than simply displaying attention heatmaps, and it also indicates that the author is aware of the connection between the method and no op head gating.

Critical weaknesses
  1. The most critical gating baseline is missing, and the independent benefits of core contributions are still unclear. Compared to Vanilla, this method also changes the pairwise token gating headwise RMSNorm, QK-norm, and the number and dimensions of attention heads. Table 3 only controls for QK norm and does not sufficiently separate other factors.

Moreover, authors don’t reply to what finer-grained query key gating brings more than simpler output gating?

  1. The prominent conclusion of parameter efficiency depends on cross-paper comparisons that do not match. Table 2 compares the 778M, 12B tokens, and PPL 14.60 used in this method with the 1339M, 13.1B tokens, and PPL 15.56 referenced from GaLore, emphasizing that only about 58% of the parameters are required. However, GaLore [1] uses a maximum sequence length of 256, while Sec 3.1 of this manuscript uses 512. The training context is already different, and the evaluation context and other details need to be unified, so the efficiency advantage of parameters cannot be strictly quantified based on this.

Reference: [1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Questions for authors
  1. After matching QK norm, headwise RMSNorm, head configuration, and tuning budget, how much profit is left compared to ordinary attention and output gated attention?

  2. Does the conclusion of 58% parameter still hold when running 1.3B Vanilla in the same 512 context and evaluation protocol?

  3. Pls provide the experiment results on larger models like 7B or 14B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.