- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work find that reusing indices across query positions is orthogonal to both, yet unsafe to exploit directly. And this work find that adjacent queries' index sets concide strongly, but rare query paris drift sharply. This work propose Reusable Indexing (Rix), which provides one with an O(Hd) closed-form reusability metric. The experient support that the Rix is useful.
Critical strengths
- This work proposes to use the accumulated attention mass to choose a key set, which is interesting.
- The work seems the simple and effective.
- The experiments support the effectiveness of Rix.
- This work discusses the effect of ReLU and Exp.
- This work provides the code to reproduce the results, which increases the reliability.
Critical weaknesses
- This work involves several concerps, such as von Mises–Fisher, which may make the paper requires theory background. This work should further and explain the notation for the theory.
- It seems that the work does not compare with baselines? It mainly compare with Rix variants.
- For equation 5, the softplus may make the w to be zero, so that the attention becomes uniform distribution. This work should disucss the effect of w with softplus in Equation.
Questions for authors
NA
2 comments
After the the above response, I keep the positive score.
Reply to reviewer Y23m
We thank the reviewer for the constructive comments.
Theoretical concepts. Appendix C.1 now provides a self-contained derivation from uniform spherical keys to the per-head vMF distribution and the multi-head KL bound. It defines the mean direction, concentration, mean resultant length, Bessel and hypergeometric functions, and the exp–erf approximation used for efficient evaluation.
Comparison with baselines. Rix is not an incremental modification of an existing reuse method. It introduces calibrated per-pair reuse-error estimation and optimal adaptive anchor placement, while remaining compatible with anchor construction methods and other reuse axes. Consequently, we fail to find baseline that represents the full framework. We therefore use component-wise and compositional comparisons: Table 2 compares dense attention, full indexing without reuse, fixed-period placement, the external PIVOT-reuse method, and ORS; Tables 4–5 isolate the indexer activation and training objective. Rix composes with PIVOT's anchor synthesis, and Table 2 shows that combining ORS with PIVOT-reuse further improves over fixed pooling.
We also establish target-weighted head cosine (THC), a simplified variant of the proposed gate, to isolate the contribution of the reuse predictor. THC replaces the proposed vMF divergence with directional cosines and head-weight/norm corrections. Define per-head directional cosines, target-side head shares, and head-concentration vectors as
a_h(j)=\frac{w_{j,h}\\|q_{j,h}\\|}{\sum_{h'}w_{j,h'}\\|q_{j,h'}\\|},\qquad \boldsymbol\kappa(q)=(\\|q_1\\|,\ldots,\\|q_H\\|).$$ With $\boldsymbol w(q)=(w_1,\ldots,w_H)$ and $\operatorname{cos}(u,v)=u^{\top}v/(\\|u\\|\\,\\|v\\|)$, the mixture-corrected THC distance is $$\begin{aligned} D_{\mathrm{THC}}(i,j)={}&1-\sum_h a_h(j)c_h(i,j)\\ &+\lambda_w\bigl[1-\operatorname{cos}(\boldsymbol w(q_i),\boldsymbol w(q_j))\bigr]\\ &+\lambda_\kappa\bigl[1-\operatorname{cos}(\boldsymbol\kappa(q_i),\boldsymbol\kappa(q_j))\bigr]. \end{aligned}$$ The first term emphasizes heads carrying the target query's index energy; the other two capture head-weight and concentration-profile reallocation. The coefficients $\lambda_w,\lambda_\kappa$ and the affine map $\beta\ln D_{\mathrm{THC}}+c$ are fitted with the same realized-KL calibration target as the proposed gate. THC also costs $O(Hd)$, but its central cosine term is insensitive to a uniform rescaling of all query heads. Both predictors use strictly matched training and calibration recipes and are evaluated on all 1,300 RULER-32k cases. Proxy metrics use probe offsets 1–4 and zero-slack 75% iso-reuse; generation uses top-$k$ 2048, $W=8$, and exactly 75% realized ORS reuse. | Predictor | $L_{\mathrm{mono}}$ ↓ | Rejection (%) ↓ | Decision error (%) ↓ | KL regret ↓ | idx | ORS 75% | |---|---:|---:|---:|---:|---:|---:| | THC | 0.141188 | 6.671 | 10.007 | 0.009527 | **87.23** | 77.08 | | vMF (ours) | **0.140648** | **6.482** | **9.722** | **0.008472** | 87.16 | **77.71** | The vMF gate improves every oracle-aligned metric. The 95% paired intervals for THC minus vMF are $[-0.887,0.635]$ for idx and $[-0.500,1.962]$ for ORS 75%, so the end-task scores do not significantly separate them. **Zero head weights.** This is a well-taken concern. We indeed observed zero-weight heads with ReLU during training, which directly motivated our use of softplus. In exact arithmetic, $\operatorname{softplus}(x)>0$ for every finite $x$, avoiding ReLU's absorbing zero-weight state; empirically, this change resolved the failure mode. We explain this motivation in Appendix D.4.