acceptodds
Under review as a conference paper at ICLR 2027

Multi-layer Cross-Attention is Provably Optimal for Multi-modal In-context Learning

Abstract

Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer self-attention models fail to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context-length are large. We show that this cross-attention mechanism is provably Bayes-optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.

Primary area:
learning theory
Keywords:
Theory, in-context learning, multi-modal, attention, cross-attention, asymptotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Wy3gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies in-context regression under a multimodal latent-factor model whose input covariance varies across tasks. It proves that the specified single-layer linear and softmax attention models cannot recover the Bayes-optimal predictor, and proposes a custom deep cross-attention architecture with repeated input injection to approximate task-specific covariance inversion. For restricted linearized versions with only one or two trainable scalars, gradient flow achieves Bayes-optimal prediction asymptotically as context length and depth grow.

Critical strengths
  • Introduces a multimodal latent-factor setup with task-dependent input covariance, extending ICL analysis beyond fixed-covariance settings.
  • Establishes a clear separation between single-layer attention and a constructed deep architecture, with training guarantees showing how depth enables task-specific covariance correction.
Critical weaknesses
  • Despite the new multimodal setup, the central mechanism—using depth to implement iterative regression and approach an optimal estimator—is well established [1][2]. Prior work also demonstrates benefits of depth when covariance varies across contexts [1].
  • The model modifies standard attention with repeated input injection and fixed-input cross-attention. The theoretical analysis further fixes most weight matrices and trains only one or two scalars shared across layers. Consequently, the guarantees primarily concern tuning a preconstructed iterative solver, limiting their implications for training standard transformers.

[1] Blake Bordelon, Mary Letey, and Cengiz Pehlevan. Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time. 2025.

[2] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS, 2023.

Questions for authors
  • Can similar guarantees be established for standard transformer architectures or with unrestricted weight matrices? Which architectural modifications and parameter constraints are essential to the analysis?
  • What are the main technical advances over prior work on transformers implementing iterative regression and benefiting from depth under varying task covariances? Which aspects of the multimodal setup require genuinely new proof techniques?
3 comments

Thank you for reviewing our work and for the detailed feedback.

W1. We are grateful for the pointer to [1], which we now cite (Section 2, lines 86–90). While preparing this revision we also located Gatmiry et al. (2024a, 2024b), now cited in the same paragraph. They delineate our setting further, since the first treats a varying spectrum but only expressivity (depth lower bounds and constructions), and the second treats learnability by gradient flow but only for a fixed covariance. To our knowledge, no prior work treats both. Using depth to implement iterative regression is indeed well established [1, 2], and our one-parameter model performs in-context gradient descent (Lemmas E.1–E.2). In [1], covariances vary across contexts only through random rotations of a fixed spectrum. In our setting the spectrum itself is random, and this randomness is what makes the problem hard. In our model, if the spectrum took a fixed value with positive probability, a single attention layer could be Bayes optimal on that event (Section 4, lines 222–225, and Appendix D.4), whereas with a random spectrum every single-layer configuration fails (Theorem 4.1). The spectrum is moreover coupled to the task vector and to the noise level. Rather than assuming this setting, we show that it arises from a natural latent factor model of multimodal data (Section 3, summarized in lines 42–44), which we view as a key contribution. Within it, and unlike the constructions of [2], we prove that gradient flow learns the Bayes-optimal predictor (Theorems 6.2–6.3).

W2. From a methodological perspective, the broader point is that cross-attention need not pair two modalities. A main contribution is demonstrating that a non-standard use of attention, particularly between an evolving quantity and a fixed function of the observations, is what works in this setting. Moreover, although the parameter space is restricted for tractability, the parameters are not set by hand. Theorems 6.2 and 6.3 prove that gradient flow learns them, including in the two-parameter model whose loss is non-convex and non-coercive. Figure 3 further shows the predicted whitening when (WtS,WtV)(\mathbf W^S_t,\mathbf W^V_t) are untied, diagonal or full.

Q1. Our results establish sufficiency of the proposed architecture, not uniqueness. Extending our guarantees to unrestricted weights remains open and we mention this as a future direction (lines 538–539). In our analysis, the fixed-input cross-attention and the re-injection of X\mathbf X are shown to elicit the whitening mechanism of Eqn. 6.1 required of a Bayes-optimal predictor, but we do not preclude the possibility of other models exhibiting this whitening behavior. We note however that empirically, in ablations where we remove each of the proposed mechanisms we see that performance degrades (Appendix B and Table 1). To our knowledge, ours is the first proof that a deep attention-based model learns the Bayes-optimal predictor for a multimodal ICL task. Figure 3 shows the untied, diagonal and full weights clustering around the tied limits along training. If this clustering were established as a property of the training dynamics, the unrestricted problem would reduce layer by layer to the one we solve, which is how we view our analysis as the first step toward the general case.

Q2. The aspect of the multimodal setup that requires new techniques is the random spectrum, together with its coupling to the task vector and the noise level (see W1). It leads to three technical advances: (i) We use Gaussian exponential tilting to reduce finite-head softmax attention to linear attention in Theorem 4.1. (ii) In the one-parameter setting, we introduce epi-convergence type arguments from variational analysis (to our knowledge, this is new to ICL theory) to prove convergence to the Bayes optimal estimator. (iii) In the two parameter case, the loss is non-convex and non-coercive so gradient flow at fixed depth is only guaranteed to reach a stationary point, and the stationary points themselves change with the depth. We confine the stationary points to a region where the loss gradient is stable uniformly in depth and show that every stationary point collapses to the Bayes-optimal solution as depth grows (Theorem 6.3). These challenges arise precisely due to the randomness in the spectrum of the covariates, and do not have counterparts in the prior fixed covariance analysis. To our knowledge, Theorems 6.2–6.3 are the first training guarantees when the spectrum varies across prompts.

We hope these clarifications and the revised manuscript address the reviewer's concerns, in particular on what is new relative to [1] and [2], and we would be glad to clarify anything further.

Thanks the author for the comprehensive response and I agree with the novelty in random sprectrum. But my main concern is still about the tailored architecture since including the new residual can largly change the result. Overall, I think the paper is boarderline and I decide to keep the score.

Thank you for your reply and for agreeing that the random spectrum makes the setting new. We agree that our positive results depend on the re-injection term St\mathbf S_t (residual), but we view this as part of the finding rather than a limitation: in Appendix B, removing St\mathbf S_t leaves its error an order of magnitude above that of the full proposed model, which keeps improving. Whether other architectures can provably learn the same prompt-specific whitening remains open, as we note in the Discussion.

Reviewer_hg5MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the capacity of transformers to perform ICL on multi-modal data. They show that single-layer linear self attention and softmax self attention cannot achieve the Bayes-optimal performance in-context over their proposed class of data distributions (softmax reduces to linear in the infinite-context limit under their gaussian model). They also propose a multi-layer model with cross-attention, with SA and skip connections, that does solve the ICL problem. They further show that gradient flow on population loss converges to the Bayes-optimal ICL predictor. I would highlight that the gradient flow result concerns linearized attention with one or two trainable scalars.

Critical strengths
  1. The theory seems correct, though I have not gone through all the proofs carefully.
  2. The negative result (Theorem 4.1) is clean, and the proof is intuitive and easy to understand.
  3. The construction is also quite easy to understand.
  4. Synthetic experiments support the theory.
Critical weaknesses
  1. The title "multi-layer cross-attention is provably optimal" is much broader than what the paper actually showed. The authors prove that in this very specific, simple, multimodal regression setting, multi-layer cross-attention (with fixed key/query matrices and final readout) is sufficient to recover the bayes-optimal predictor. This is a far from what "optimal for multi-modal in-context learning" implies.
  2. The two-parameter model is simple, with some important matrices fixed to be identity, and it is unclear if the theory generalizes in any meaningful way to un-constrained full-weight training. (However, the experiment in Figure 3 do give empirical support that un-constrained weights may learn similar behavior.)
  3. I do not believe that being able to handle "different covariate distributions across tasks" is new to this paper. A number of existing work, which show that Transformers can do GD on regression (e.g. von Oswald et al), have constructions that work even when covariate distribution is task dependent (simply because GD converges, as long as the step-size is not too large). The assumption on covariate-distribution being uniform across tasks was often needed to show the stronger result: convergence of gradient flow over largely-unconstrained weights, or characterizations of optimal solutions; these are generally more difficult to show than the one/two-scalar-parameter model considered in this paper.
  4. The avMNIST experiment does not validate the paper's ICL theory (there is no context, it seems like a standard classification task).
Questions for authors

see weaknesses

3 comments

Thank you for reviewing our work and for the detailed feedback.

W1. By “optimal” we mean that the predictor attains the Bayes-optimal prediction E[yq∣m,ζ,xq]\mathbb E[y_q\mid\mathbf m,\zeta,\mathbf x_q] as the context length and then the depth grow, for the latent factor model of Section 3, which the abstract states together with the regime (lines 20–22). The title follows the convention of this literature, where “provably optimal” usually refers to optimality within a stated model, as in “One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention” (Mahankali et al., 2024). We would nevertheless be happy to add “Bayes-” to the title for the camera-ready if this is permitted.

W2. These restrictions are introduced for tractability, reducing the loss to a function of one or two scalars. In a fully parameterized model we would not expect the weights to collapse to scaled identities, but rather to learned preconditioners in the spirit of LISTA (Gregor & LeCun, 2010), whose optimum is beyond the present analysis. More expressive parametrizations contain our subfamilies and therefore retain the Bayes-optimal solution, and Figure 3 shows that the predicted whitening persists when (WtS,WtV)(\mathbf W^S_t,\mathbf W^V_t) are untied, diagonal or full. We now discuss this after Theorem 6.3 (lines 368–371) and in the Discussion (lines 538–539).

W3. Our contribution concerns learnability rather than expressivity. Constructions where transformers implement gradient descent (e.g., von Oswald et al., 2023) remain valid when the covariance varies across tasks, but they show that suitable weights exist, not that training finds them. The stronger training results the reviewer mentions, such as gradient flow over largely unconstrained weights (Zhang et al., 2024a) or characterization of the optimal solutions (Ahn et al., 2023), assume the same covariate distribution for every prompt. The same holds for the looped-transformer analysis of Gatmiry et al. (2024b), which we located after submission and now cite (lines 86–90), while its companion paper (2024a) allows a varying spectrum but proves only lower bounds and constructions, not learnability. In our setting, the spectrum of the covariance, the task vector and the noise level are all random and coupled through the multimodal latent factors. This changes the problem, since the single-layer model that is learnable and Bayes-optimal under a fixed covariance provably fails here (Theorem 4.1). To our knowledge, Theorems 6.2–6.3 are the first training guarantee in such a setting, and even our restricted family leads to a non-convex, non-coercive loss (Appendix G). Extending these guarantees to unconstrained weights is an important direction for future work. We have placed our contribution relative to these results in Section 2 (lines 86–90) and in the Discussion (lines 531–533).

W4. The avMNIST experiment indeed does not reproduce the prompt-based setting of the theory. The in-context predictions are tested directly by the synthetic experiments (Figures 2–3, Appendix B), which resample the task variables across prompts and evaluate on unseen prompts. The avMNIST experiment serves a narrower, complementary purpose, which is to show that the architecture identified by the theory remains learnable in practice once softmax attention, layer normalisation, and MLPs are restored. We now state this purpose in Section 7 (line 463). The new CosmoBench experiment (Section 7, lines 499–509 and Table 2, and Appendix L) is closer to the theory's setting. Each point cloud is a universe generated by its own unseen cosmological parameters which shift both the covariates and the prediction rule and must be inferred from the cloud, mimicking the structure of Section 3 with these parameters in the role of (m,ζ)(\mathbf m,\zeta). Appendix L provides more details on the cosmology problem and its relation to our setting.

We hope the revision addresses the concerns and would be glad to clarify anything further.

Thank you again for your thoughtful review. We wanted to ask whether our rebuttal addressed your concerns. We believe it clarifies the sense in which our predictor is optimal, how the restricted parametrization relates to unconstrained training, and the novelty of the setting considered in our paper. The revision also adds a CosmoBench experiment closer to the theory’s in-context setting, but we would be happy to address any remaining points. If the rebuttal has resolved your concerns, we would appreciate your consideration of an updated score.

I think the authors for their response. My concerns are addressed and I maintain my positive score.

Reviewer_sRemOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal in-context learning under prompt-dependent covariate shifts. It proves that single-layer self-attention cannot achieve Bayes-optimal prediction and proposes a multi-layer cross-attention architecture that can achieve Bayes optimality through prompt-specific covariance correction. The theoretical results are supported by synthetic experiments, while experiments on corrupted avMNIST provide preliminary evidence for the practical utility of the proposed cross-attention and data re-injection mechanisms.

Critical strengths
  1. The paper establishes a clear separation between single-layer attention and multi-layer cross-attention for multimodal ICL, proving that the former cannot achieve Bayes-optimal prediction while the latter can under the proposed setting. This provides a principled explanation for the role of depth and cross-attention.

  2. The synthetic experiments closely support the theoretical predictions, while the corrupted avMNIST results show that the theory-motivated cross-attention and re-injection mechanisms remain beneficial in a more practical Transformer architecture.

Critical weaknesses
  1. Limited methodological novelty. The proposed multi-layer cross-attention mechanism appears largely built upon existing cross-attention and hierarchical fusion strategies. The paper does not sufficiently clarify what fundamentally distinguishes the proposed design from closely related architectures, weakening the significance of the contribution.

  2. Insufficient experimental validation. The empirical evaluation does not fully isolate the contribution of the key components through comprehensive ablations and comparisons with strong recent baselines. Consequently, it remains unclear whether the reported improvements originate from the proposed mechanism itself or from increased architectural complexity.

Questions for authors

The Bayes-optimality guarantees rely on infinite context length and depth. How well do the theoretical conclusions extend to practically relevant finite-depth and finite-context settings?

The theory is established under a specific Gaussian latent factor model. Which assumptions are essential for the failure of single-layer attention and the optimality of multi-layer cross-attention?

The real-data validation is limited to corrupted avMNIST with relatively modest gains. How strongly should the practical effectiveness of the proposed architecture be inferred from this experiment?

2 comments

Thank you for reviewing our work and for your detailed feedback.

W1. Two things distinguish our architecture. First, the cross-attention is not between modalities, as it is in the bidirectional cross-attention baseline of Table 1 (Lu et al., 2019). At every layer the keys and values are the fixed input X\mathbf X and the queries are the evolving state Ft\mathbf F_t. Second, X\mathbf X is re-injected at every layer through St\mathbf S_t. We have made this distinction explicit in Section 5 (lines 268–270). From a methodological perspective, the broader point is that cross-attention need not pair two modalities. A main contribution is demonstrating that a non-standard use of attention, particularly between an evolving quantity and a fixed function of the observations, is what works in this setting. We believe this general idea could be of broader interest.

W2. The paper is foremost theoretical, and the avMNIST experiment (Section 7) is meant to show that the architecture is practical in that it remains learnable once the linearized model is replaced by standard transformer blocks and all parameters are freed. Table 1 already isolates the two components. Removing re-injection or cross-attention lowers the accuracy, removing both lowers it further, and the full model has fewer parameters (4.36×1044.36\times10^4) than the standard bidirectional cross-attention transformer it outperforms (6.07×1046.07\times10^4), so the gains do not come from added complexity. It is not a state-of-the-art claim and large-scale comparisons are a natural direction for follow-up work, as we now state in Section 7 (lines 481–483). We have also added a second real-data experiment, on CosmoBench's velocity task (Section 7, lines 499–509 and Table 2, and Appendix L), where our paper-inspired model improves on the benchmark's graph network by 25% in R2R^2 with half of its parameters.

Q1. While our main results are asymptotic, Appendix H gives a non-asymptotic guarantee for the one-parameter model at a fixed parameter in an explicit range. Once L>dL>d, with high probability, the excess risk is at most a bias decaying geometrically in the depth plus a noise term of order d/Ld/L (Theorem H.1). Figure 2 confirms this at moderate depth. Section 6 (lines 402–407) summarizes this guarantee, and Theorem H.1 (lines 2127–2144) states it in full. We would be glad to display the bound in the main text if the reviewer finds this helpful.

Q2. The varying spectrum of the covariance is key for establishing the failure of single-layer attention and in necessitating the deep cross-attention model developed in Section 5. Under the latent factor model, each prompt's covariance I+mm⊤\mathbf I+\mathbf m\mathbf m^\top has a top eigenvalue 1+∥m∥21+\|\mathbf m\|^2 that changes from prompt to prompt. Theorem 4.1 shows that no single attention layer can make this prompt-specific correction, since its weights, however they are trained, are shared by all prompts while the spectrum changes from prompt to prompt. If every prompt had the same spectrum, a single layer model would succeed (Section 4, lines 222–225). Multi-layer cross-attention instead re-reads the input at every layer, refining a prompt-specific correction that becomes Bayes-optimal with depth. This needs only mild conditions on m\mathbf m (Assumption 6.1). Gaussianity is needed only for the Bayes predictor to be linear in the query and for the reduction of softmax to linear attention, as we now state in Section 3 (lines 136–138). We have added a new Appendix J (lines 2222–2256) which reruns the experiment of Figure 2 with Rademacher, uniform, Laplace and skewed exponential latent factors and noise. The errors of these distributions track those of the Gaussian setting, and as the training context grows the trained parameter converges to the value predicted by our analysis under every law.

Q3. The avMNIST experiment is meant to show that the architecture remains learnable in practice, once softmax attention, nonlinearities, layer normalisation and MLPs are added and all parameters are trained on real data. The re-injection and the cross-attention to the fixed input cause no optimization difficulties in our experiments. The model trains with the same standard settings as the baselines and reaches the best accuracy in Table 1, above each ablation and above the standard bidirectional cross-attention transformer. As the gains are modest, this is not a claim of superiority but a preliminary test, which motivates larger-scale experiments in follow-up work. The new CosmoBench experiment (Section 7, Table 2, and Appendix L) adds a larger real-data result, in which our model improves considerably on the benchmark's baselines.

We hope the revision addresses the concerns and would be glad to clarify anything further.

Thank you again for your thoughtful review. We wanted to ask whether our rebuttal addressed your questions and concerns. We believe it clarifies what distinguishes the proposed cross-attention mechanism, why its gains do not come from added complexity, which assumptions the theory needs, and what it guarantees at finite depth and context. The revision also adds experiments beyond Gaussian data and a second real-data experiment on CosmoBench, but we would be happy to address any remaining points. If the rebuttal has resolved your concerns, we would appreciate your consideration of an updated score.

Comments on the paper

1 comment

We thank all three reviewers for their careful reading and constructive feedback, which has helped us improve the manuscript. We have uploaded a revised PDF with every change in blue. The revision places our contribution relative to the prior work the reviewers raised (Section 2); states which assumptions each result needs (Sections 3 and 4) and demonstrates empirically the universality of our results beyond the Gaussian data assumptions (Appendix J); adds a second real-data experiment on CosmoBench's velocity task where our model improves R2R^2 by 20–25% (relative) over the benchmark's baselines, with half the parameters of its graph network (Section 7, Table 2, Appendix L); and addresses the points raised in the individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.