- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3
Read review
Paper summary
The paper investigates the statistical estimation costs of multiplication-free table-lookup parameters in transformer models. It shows that these parameters can substantially affect the model's predictive capacity by upper- and lower-bounding the model’s VC dimension. The analysis follows a careful accounting of how the parameters travel through and thus affect the predictions. The authors additionally analyze the effects of model precision and the location where the lookup parameters are injected in the model.
Critical strengths
- I believe the topic studied in the paper is generally interesting and well-aligned with modern practice in lookup-based computations.
- I also think the question is quite novel and important.
- I also appreciate the realistic model (with standard assumptions such as hard attention).
- To the extent that I was able to check (I am not familiar with the topic and related literature), the claims seem correct and well-posed.
Critical weaknesses
- The organization and exposition of the paper are quite difficult to follow, from the Abstract to the main theoretical passages. This was my main issue with the paper, as it made it, despite the interesting topic, hard to read and follow.
- For example, none of the quantities on Lines 18–21 in the Abstract are defined, and it is not clear what those sentences are saying.
- Similar holds for writing such as “depth weighting saturates at …” (Line 24) and “confirms the precision law” (Line 29).
- Moreover, on Lines 47–49, the authors say “they answer this”, without specifying what the question is, and it is unclear what the bound is referring to.
- Similar things occur throughout the Introduction and the paper (undefined quantities in Eq. (11) and other contribution bullet points, etc.)
- For example, none of the quantities on Lines 18–21 in the Abstract are defined, and it is not clear what those sentences are saying.
- While I don’t think there’s anything inherently wrong with rewriting the draft with AI to fix grammatical errors and improve readability, this text reads like very AI-written (with passages such as Lines 160 and 161, among many others), which I imagine is what led to many undefined quantities and unclear writing as well.
- While I was able to follow the main results behind the effects of the parameters on the VC dimension, I struggled to grasp the importance of the effects of the precision (Section 5.2). This section, and others in Section 5 as well, seem less motivated, and it’s unclear to me what purpose they serve.
- Similarly, it is unclear to me what additional value the experiments serve. They are delivered with no introduction and connection to the theoretical part, leaving the reader to infer how they augment the work.
Questions for authors
- I am quite unfamiliar with the literature on GLI, and one question I had when I first read the paper (and which I cannot formulate an answer to yet) is: why is it necessarily surprising that non-matrix-multiplicative parameters have the same capacity as other parameters in the model? Classical non-parametric -gram models similarly have a large capacity (for large enough ), so I am not too surprised that they affect the statistical properties of the model. What am I missing?
- Do the results change substantially if we allow parameter tying across injection sites?
2 comments
I thank the authors for the helpful clarifications and, importantly, improvements to the presentation. I have adjusted my score accordingly.
We thank the reviewer for recognising the novelty and practical relevance of the question, and for the frank assessment of the presentation, which we agree with. We have rewritten the affected parts.
Presentation.
Purpose of Section 5.2 and of the experiments. Section 5.2 now opens with its motivation. Models are trained in 16 or 32 bits, so an exact-arithmetic capacity result is useful only if one knows which part of it survives rounding. The per-parameter capacity of a table entry survives; the placement advantage of the ceiling does not.
Section 7 now opens by distinguishing two kinds of experiment:
Q1. Why is it surprising? This question led to a sharper statement, which is now in the introduction ("Why the answer is not obvious"). An n-gram or lookup classifier on its own has small capacity: with K keys and rows of width p, it realises at most one label per key, so its VC dimension is at most K, independent of p and depth (Proposition 33). Our lower bound shows that the same Kp entries inside a Transformer realise Θ(pLlogT) labels per key, which is the rate of an ordinary weight. The surprise is therefore not that tables have capacity, but that attention multiplies it by pLlogT.
New evidence (Section 7.5, Tables 19 and 21):
Q2. Parameter tying. Yes, and the answer is clean (new Proposition 16, proof in Appendix C). A table shared across all sites in Λ is first live at minΛ. It contributes Kp(L−minΛ+1) to Φ, and VC=Θ(Kp(L−minΛ)logT) from the table, independently of ∣Λ∣. A tied table is thus priced like one untied slice at its first site, with ∣Λ∣ times fewer parameters. The lower bound holds because the hat gates of the witness close every site other than the start layer.
We hope the revised presentation now makes the contribution easy to follow, and we thank the reviewer again for the questions, both of which led to new results in the paper.