- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when correlations among in-context examples help or hurt prediction. It derives an exact risk formula for linear attention, showing a trade-off between query relevance and context redundancy. It further extends the analysis to graph-structured prompts and SBM, deriving conditions for when structure is beneficial. The theory is validated through synthetic experiments with linear and softmax attention models.
Critical strengths
-
The paper provides a clean theoretical explanation of how query relevance and context redundancy affect ICL performance. The finite-dimensional risk formula is a useful result.
-
The graph and SBM analysis is interesting, especially the connection between graph connectivity, homophily/heterophily, and prediction risk.
Critical weaknesses
-
Limited novelty over prior work. The three key statistics are closely related to those already studied by Letey et al. (2026). The main relevance–redundancy decomposition seems largely a reformulation and extension of existing results. The graph/SBM analysis is new, but I am not fully convinced that the overall theoretical novelty is strong enough.
-
Strong assumptions and limited practical validation. The theory relies on Gaussian, isotropic features, linear regression tasks, and a one-step linear-attention predictor. These assumptions are quite restrictive for real ICL. Even the experiments on real graph topologies use synthetic Gaussian features. There is no evaluation on real LLM prompting tasks, so it is unclear how well the conclusions generalize.
Questions for authors
-
Could the authors clarify what is fundamentally new in Theorem 1 compared to the risk expressions in Letey et al. (2026), beyond the finite-dimensional corrections and change of variables?
-
How robust are the results when the Gaussian and isotropic assumptions are relaxed? Could the authors provide an experiment with real embeddings or real LLM-based ICL to support the practical relevance of the theory?