- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how much information is lost when an in-context regression prompt is compressed into low-order moments, and how this loss limits prediction accuracy. It then shows that given raw data, affine attention models can only utilize gram information, and that a source operation can overcome this limitation.
Critical strengths
The paper studies a meaningful question: what information an attention layer must retain for accurate in-context regression prediction. It characterizes the information retained by finite-degree moments in Theorem 3.1 and establishes when these moments are loseless for estimating the regression coefficient in Theorem 3.2.
Critical weaknesses
My major concern is that the writing of this paper needs substantial improvement. The paper lacks a clear logic flow, and there are some notations and non-standard terminologies that authors used without prior definition or proper reference, which makes it difficult to follow. For example, in eq.(2), it is unclear to me what is the meaning of and . In eq.(3), I could not find the definitions of and until line 254 and line 273. In line 190 it is confusing what the author meant by "hierarchy" and "classification". Similar issues exist throughout the paper. For basic readability, the authors should at least clearly define all notations and non-standard technical terms upon first use, and refer readers to the equations where appropriate.
Besides clarity issues, the contribution of this paper appears limited. On transformer side, the major claim of this paper is that linear attention layers can only utilize Gram information of the prompt, which is not surprising; the proposed source operation seems tailored to the specific in-context regression setting rather than a general approach, and I did not see the benefits of this approach over simply providing higher-order moments to the model. Also, the connection between the theoretical results in section 4 and the experiments in section 5 is not clear to me.
Questions for authors
- Please see the weakness part. The whole paper needs a substantial revision to be readable.
- In line 285, what exactly does "source operation" refer to? Can you provide a concrete formula and explain how can it be applied in practice, and how does it relate to the experiments in Section 5?
- What is the advantage of this source operation over processing the raw data directly, such as providing higher-order moments or other features to the model? Also, can this approach be generalized to other settings beyond in-context regression?