- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The draft studies what attention maps can be computed using a fixed number of exactly mergeable value states. Under a fixed-query, value-linear summary model, the authors show that the minimum number of retained value vectors for a positive real-rate exponential-polynomial generator equals its translation dimension. They give a moment-based construction and a matching lower bound, including when the decoder is nonlinear. They also characterize when these maps can be represented by nonnegative softmax mixtures on the same scores through a finite Hankel matrix. Based on these results, the authors introduce two-state and three-state attention maps and evaluate their estimation risk, language-modeling performance, and execution cost.
Critical strengths
-
The authors give a way to compute attention by merging summaries from different blocks. Within their specific setting, they prove the minimum number of value vectors that need to be stored, and their method uses exactly this number.
-
The authors show that their two-state Ray attention cannot be represented by a nonnegative softmax mixture on the same scores. Their synthetic experiments also show examples where it gives lower prediction error than these mixtures.
Critical weaknesses
-
The gain over learned temperature is small. In Tab 1, Ray improves NLL by only about 0.000139, 0.000057, and 0.000686 at lengths 128, 256, and 512. At length 512, the gain over Coxian is only about 0.000066. The reported confidence interval is against ordinary softmax. It is therefore unclear whether Ray is reliably better than these stronger baselines. Please report confidence intervals for these comparisons.
-
In Section 4, the authors claim the long-context FP32 implementation first finds the global maximum for each row, then makes a second pass to compute the output. Once the maximum is known, any positive h can be evaluated with one weighted value sum and a normalizer. This does not directly test the online merge in Algorithm 1. Please report the time, memory use, and numerical error of that merge, and clarify which GPU results use it.
Correct me if I'm wrong
Questions for authors
See Weaknesses
Response to Reviewer gJjx
Thank you for a careful and constructive review, and for pointing out the distinction between two-pass evaluation and the single-pass online merge of Algorithm 1.
Response: Under the calibrated protocol,
Ray⋆has the lowest mean NLL at every length; its paired 95% article intervals exclude zero againstCoxian⋆and softmax at every length, and it leadsMix-2⋆at every length (−2.18 [−3.88, −0.47]at 512; Table 2; Supp. B.2, Table S3, Figure S4).Ray⋆ − Coxian⋆Ray⋆ − Mix-2⋆Ray⋆ − SoftmaxRayandCoxian) beat every non-coupled map at every length (Table 1); under calibration, two-stateRay⋆beatsCoxian⋆at all lengths ([−3.27, −0.09]at 128,[−3.45, −0.16]at 256; 7/8 seeds at 512) with one fewer rate parameter.Ray − Temperatureis negative in mean and median across all 6 protocol × length cells (winning 6/8 seeds at 256 and 512; calibrated means−0.77, −0.96, −1.23and medians−1.15, −1.43, −1.35; default 512 mean−1.32, median−1.91; per-seed values: Table S4). All 24 logged coupled runs (Ray,Coxian) end at γ<0, excluded from temperatures (γ≡0) and positive mixtures (γ≥0,Mix-2collapsing to γ=0; Figure 4b–c), and projecting every Ray head of the seed-42 model onto its best positive temperature changes layer outputs by 1.6–4.6%, 64–160× the FP16 reconstruction error (Section 5.4).Response: Sections 4 and 5.5–5.6 label every GPU route, and our single-pass GPU implementation of Algorithm 1 (Supp. D) matches softmax's online-merge precision (2.0–5.9×10−7 vs. FP64) and decodes within 0.5–6.6% of FlashAttention-2. (Supp. C.4–C.6, Tables S9–S14, Figure S12.)
f+bat 1.13–1.17× FA2 at 4k–8k (Table S14).Thank you again for feedback that led directly to our paired baseline intervals and single-pass GPU online merge benchmarks.