- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GMProp, a learned importance-sampling distribution for positive random-feature attention. It fits a Gaussian mixture to each attention head while keeping the pretrained model weights frozen. An additional learned coordinate transformation preserves query–key inner products while improving approximation quality. Importance weighting maintains an unbiased kernel estimator, and the fixed sampling distribution supports linear-time prefill and constant-state decoding.
Critical strengths
-
The method adapts random-feature sampling to pretrained models for kernel approximation.
-
The paper provides a theoretical characterization of the variance-optimal proposal and motivates the Gaussian mixture approximation.
-
The proposed method consistently improves over the evaluated PRF baselines.
Critical weaknesses
-
The quality gap to exact attention remains substantial. Despite improvements over PRF baselines, the paper does not demonstrate a configuration that adequately preserves model quality.
-
Quality evaluation is limited to models up to 1B parameters and contexts of at most 512 tokens. There are no long-context retrieval or generation evaluations, leaving it unclear whether the learned proposals remain effective as context length increases.
-
The accuracy-efficiency trade-off is not practical. The cost experiments mainly use 256 features at a sequence length of 2K, where quality degradation is already substantial.
The results support an improvement in random-feature approximation, but it is not a useful attention replacement for pretrained models.
Questions for authors
please see weakness
Summary of the revision
Since our first revision, the revised PDF adds the experiments we committed to and makes these changes:
Training with GMProp (2ecf, oyUd, nsQh). New Appendix K, summarized in Section 4.2. Low-rank adapters on Pythia-410M and OLMo-1B with each estimator in place (Appendix K.1, Table 14), and the architectures of Pythia-70M and Pythia-410M trained from scratch with exact attention and with each estimator, under the same recipe (Appendix K.2, Tables 15 and 16).
Training without the importance weight (oyUd). Appendix K.3: the mixtures of Table 2 trained without the weight, with the model frozen (Table 17) and with Pythia-70M from scratch (Table 18), the kernel the estimate then converges to (Eq. 55), and its relation to KL-Transformer and DARKFormer (Table 19).
Baseline details. Section 4 corrects which FAVOR# variant its authors used in their experiments and explains why we run SDERF. Appendix G states that FAVOR# is re-solved under a stop-gradient during training, and that the FAVOR++ and FAVOR# solves leave out the attention-sink key. The caption of Table 3 states that decode costs are derived for each method as published.
Presentation. The abstract, introduction and conclusion are rewritten. The introduction now states the cost of refitting the sampling distribution at inference and in training, and summarizes the results.
The Pythia-410M run with FAVOR# is still training, and we will post its result as soon as it ends.