The main problems I found concern the novelty of the parameterization and the fairness and reliability of the BPB comparisons.
The central point of the paper is the parameterization δ(γI+P); however, it is a reparameterization of the learned identity–attention mixture used in Value-SkipInit (He et al., 2023, Eq. 7), Attn(X)=(αI+βA(X))V(X), assuming β=0. The authors do show the equivalence between the two parameterizations in Section 5, but they do not cite the related work there. Although these works are already cited in Section 2, in Section 5 it would be appropriate to cite Value-SkipInit, He et al. (2023) beside α0I+α1P at line 254 and FDAM (Chen et al., 2025) and FAiT (He et al., 2026) beside αP+β(I−P) at line 263 since they combine P and I−P.
As to the comparison between parameterizations of Section 6,
using the Laplacian I−P in every head is only one configuration of the prior method of Zhang & Papyan (2026). This means the authors are not comparing their method to the best config of Zhang & Papyan (2026), but to a configuration that is suboptimal in terms of the downstream language-model results of Zhang & Papyan (2026).
Still on this point, in Figure 4(a,b) the initialization matches the one used for Figures 2(a), 3(a), since δ(γI+P)=P+I for δ=1 and γ=1. In Figures 2(b), 3(b) γ=0 for layer 0, but in Figures 4(c,d) γ=−1 for layer 0. Is this discrepancy intentional? This also matters for Figure 5, because the authors' method uses P in layer 0 and P−I in later layers, while α0I+α1P and αP+β(I−P) start at either I+P or I−P in every layer.
As regards Section 6, the authors also report that the learning rates are different for these experiments: Appendix C gives (0.005,0.005) for the alternatives, versus (0.0025,0.02) for the proposed method. Were the learning rates (0.005,0.005) chosen through hyperparameter tuning?
Figure 5 (Section 6) shows that the authors' configuration is better, but in order to be able to attribute the improvement to the parameterization, the initializations have to match and the learning rates of the alternative parameterizations also need to be chosen through hyperparameter tuning (if they were not already).
The performance differences are also very small, and error bars of 1 standard deviation are much smaller than 95% t-confidence intervals (which would be x2.48 wider), since there are 2 degrees of freedom with 3 seeds. I think it could be useful to report CIs for the mean BPB differences between methods across matched seeds and see if they exclude 0, since I am not sure how reliable the performance differences are compared to the Laplacian baselines.