acceptodds
Under review as a conference paper at ICLR 2027

When Do Structured Prompts Help In-Context Learning? A Relevance–Redundancy Theory of In-Context Prediction

Abstract

In-context learning is commonly analyzed under independent demonstrations, yet real prompts are often structured: demonstrations can be correlated with one another and with the query. When does this structure improve in-context prediction, and when does it merely make the context redundant? We develop a relevance–redundancy account in the solvable setting of in-context linear regression, allowing an arbitrary positive-semidefinite, unit-diagonal sample-similarity kernel over the demonstrations and query. For a linear-attention predictor trained on independent prompts and deployed on a structured one, we derive an exact finite-dimensional excess-risk law. Structure decomposes into a redundancy cost among demonstrations, a relevance gain between demonstrations and the query, and a higher-order mediated interaction. In the weak-structure regime, prediction improves precisely when , where is the context-to-dimension ratio and is the label-noise variance. We then characterize which relevance–redundancy combinations arise from coherent structure. Under graph-induced covariance, the two terms become internal context connectivity and query connectivity. In a sparse stochastic block model, a single latent context-composition variable changes relevance linearly but redundancy quadratically, yielding explicit homophily- and heterophily-dependent help–hurt phase transitions. Finally, trained full linear attention closely follows the exact law, while nonlinear softmax attention often depends on the same coordinates but with architecture-dependent coefficients. Thus, relevance and redundancy describe a shared prompt geometry, while their exchange rate is model-specific.

Primary area:
learning theory
Keywords:
In-context learning, attention, learning theory, structured prompts, correlated demonstrations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_f4BEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when correlations among in-context examples help or hurt prediction. It derives an exact risk formula for linear attention, showing a trade-off between query relevance and context redundancy. It further extends the analysis to graph-structured prompts and SBM, deriving conditions for when structure is beneficial. The theory is validated through synthetic experiments with linear and softmax attention models.

Critical strengths
  • The paper provides a clean theoretical explanation of how query relevance and context redundancy affect ICL performance. The finite-dimensional risk formula is a useful result.

  • The graph and SBM analysis is interesting, especially the connection between graph connectivity, homophily/heterophily, and prediction risk.

Critical weaknesses
  • Limited novelty over prior work. The three key statistics are closely related to those already studied by Letey et al. (2026). The main relevance–redundancy decomposition seems largely a reformulation and extension of existing results. The graph/SBM analysis is new, but I am not fully convinced that the overall theoretical novelty is strong enough.

  • Strong assumptions and limited practical validation. The theory relies on Gaussian, isotropic features, linear regression tasks, and a one-step linear-attention predictor. These assumptions are quite restrictive for real ICL. Even the experiments on real graph topologies use synthetic Gaussian features. There is no evaluation on real LLM prompting tasks, so it is unclear how well the conclusions generalize.

Questions for authors
  • Could the authors clarify what is fundamentally new in Theorem 1 compared to the risk expressions in Letey et al. (2026), beyond the finite-dimensional corrections and change of variables?

  • How robust are the results when the Gaussian and isotropic assumptions are relaxed? Could the authors provide an experiment with real embeddings or real LLM-based ICL to support the practical relevance of the theory?

Reviewer_CNNtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes structured prompts in isotropic Gaussian in-context linear regression. For a linear-attention predictor trained on independent prompts, it derives a finite-dimensional deployment-risk expression in terms of within-context redundancy, query relevance, and a higher-order interaction. It then studies graph-induced covariance and stochastic block models, and examines how these coordinates describe trained linear and nonlinear attention models.

Critical strengths
  1. Keeping the trained predictor fixed while changing the deployment prompt is a meaningful approach to example selection. The finite-dimensional expression exposes the assumptions and correction terms, and the graph analysis gives an interpretable context-selection application.
  2. The paper distinguishes exact linear-model results from empirical nonlinear transfer, and reports multiple training seeds, held-out prompts, and instability in some nonlinear coefficients.
Critical weaknesses
  1. The increment over Letey et al. needs a sharper comparison. That work already defines the three covariance summaries and separates context and query correlations; its theory is not limited to the exponential kernel used experimentally. Please distinguish the new finite-dimensional and fixed-deployment results from inherited identities. This does not establish that the current theorem was already proved. (pp. 3–5; Letey et al., Section 2 and Appendices C–E)
  2. The harmful-coupling interpretation needs qualification for signed similarities. With the third item as query, S = [[1, -0.1, 0.1], [-0.1, 1, 0.1], [0.1, 0.1, 1]] is positive definite but gives M = -0.002. A positive coefficient of M does not make its realized contribution positive. Please distinguish monotonicity from the sign of the contribution; this is an interpretation issue, not a counterexample to Theorem 1. (pp. 5–6)

Reference: Letey et al. (2026), Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch. https://arxiv.org/html/2607.03660v1

Questions for authors
  1. Which finite-dimensional terms and fixed-deployment conclusions go beyond specializing or reparameterizing the population-risk analysis of Letey et al.?
  2. Under what conditions does the harmful interpretation of M apply, and how should it be read for signed PSD kernels?
  3. Do the nonlinear coefficients predict excess risk on independently generated prompt banks across covariance strengths and label-noise levels?
Reviewer_h68nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when correlated demonstrations help in context linear regression. It derives an exact finite dimensional risk law for a linear attention predictor trained on independent prompts. The analysis separates query relevance from context redundancy and includes a mediated interaction. Graph kernels connect these quantities to connectivity patterns. A stochastic block model explains how context composition changes expected risk. Experiments examine agreement with trained linear attention and explore how nonlinear architectures respond to the same prompt structure.

Critical strengths
  • The qualtiative results are nice and help to understand the paper.

  • I think the exact finite-dimensional calculation is solid. The derivations in the appendices on the population risk and the deployed predictor are explicit and can be checked directly.

  • Graphs and SBMs provide interpretable theoretical examples, allowing the authors to map abstract correlations onto concrete graph structures.

  • The reproducibility is good. The trained-model appendix includes seed ranges, failed nonlinear fits, and splits within the prompt bank. The real-graph appendix clearly states that it uses synthetic gaussian features on real topologies.

Critical weaknesses
  • I think the novelty over [1] is quite limited. I believe the paper does not sufficiently discuss how it differs from prior work. [1] already introduces a general (K) and query-correlation vectors and formulates its theory using the same three summaries. Its exponential kernel is a particular example. The current paper acknowledges these summaries but also describes the previous restriction as a single correlation-length parameter controlling all dependence. I think this overstates the difference. The finite-dimensional law, the fixed-deployment calculation, and the graph/SBM consequences should be clearly distinguished from the general covariance formulation already available in [1].

  • I noticed that Figure 4b compares fitted risk coefficients with a scalar predictor. Their agreement does not establish that the learned predictor has the assumed structure. I think examining its learned matrices would strengthen the interpretation in section 6.

  • Insufficient justification of some mathematical claims, eg the claimed DPP equivalence and the interpretation of redundancy in proposition 6 need more rigorous justification.

reference [1] Letey, M., Lu, Y. M., Pehlevan, C., & Zavatone-Veth, J. (2026). Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch. arXiv preprint arXiv:2607.03660.

Questions for authors
  • Could you expand the comparison with the Letey et al paper in section 2? I think directly comparing the covariance assumptions and prediction rules would clarify what follows specifically from fixing the deployed predictor. This would help distinguish the contribution from the shared structural summaries.

  • Section 6 and Appendix K use (S=I+0.15A), which is not always PSD under the stated parameter ranges. Please clarify whether invalid kernels were rejected or rescaled, and whether (R,Q,M) were computed after any correction.

  • Do the fitted softmax coefficients remain predictive when the covariance family changes? Keeping the coefficients fixed would help test whether they capture a transferable response to prompt structure. I think this would make the scope of the nonlinear evidence more concrete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.