- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies in-context regression under a multimodal latent-factor model whose input covariance varies across tasks. It proves that the specified single-layer linear and softmax attention models cannot recover the Bayes-optimal predictor, and proposes a custom deep cross-attention architecture with repeated input injection to approximate task-specific covariance inversion. For restricted linearized versions with only one or two trainable scalars, gradient flow achieves Bayes-optimal prediction asymptotically as context length and depth grow.
Critical strengths
- Introduces a multimodal latent-factor setup with task-dependent input covariance, extending ICL analysis beyond fixed-covariance settings.
- Establishes a clear separation between single-layer attention and a constructed deep architecture, with training guarantees showing how depth enables task-specific covariance correction.
Critical weaknesses
- Despite the new multimodal setup, the central mechanism—using depth to implement iterative regression and approach an optimal estimator—is well established [1][2]. Prior work also demonstrates benefits of depth when covariance varies across contexts [1].
- The model modifies standard attention with repeated input injection and fixed-input cross-attention. The theoretical analysis further fixes most weight matrices and trains only one or two scalars shared across layers. Consequently, the guarantees primarily concern tuning a preconstructed iterative solver, limiting their implications for training standard transformers.
[1] Blake Bordelon, Mary Letey, and Cengiz Pehlevan. Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time. 2025.
[2] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS, 2023.
Questions for authors
- Can similar guarantees be established for standard transformer architectures or with unrestricted weight matrices? Which architectural modifications and parameter constraints are essential to the analysis?
- What are the main technical advances over prior work on transformers implementing iterative regression and benefiting from depth under varying task covariances? Which aspects of the multimodal setup require genuinely new proof techniques?
3 comments
Thanks the author for the comprehensive response and I agree with the novelty in random sprectrum. But my main concern is still about the tailored architecture since including the new residual can largly change the result. Overall, I think the paper is boarderline and I decide to keep the score.
Thank you for your reply and for agreeing that the random spectrum makes the setting new. We agree that our positive results depend on the re-injection term (residual), but we view this as part of the finding rather than a limitation: in Appendix B, removing leaves its error an order of magnitude above that of the full proposed model, which keeps improving. Whether other architectures can provably learn the same prompt-specific whitening remains open, as we note in the Discussion.
Thank you for reviewing our work and for the detailed feedback.
W1. We are grateful for the pointer to [1], which we now cite (Section 2, lines 86–90). While preparing this revision we also located Gatmiry et al. (2024a, 2024b), now cited in the same paragraph. They delineate our setting further, since the first treats a varying spectrum but only expressivity (depth lower bounds and constructions), and the second treats learnability by gradient flow but only for a fixed covariance. To our knowledge, no prior work treats both. Using depth to implement iterative regression is indeed well established [1, 2], and our one-parameter model performs in-context gradient descent (Lemmas E.1–E.2). In [1], covariances vary across contexts only through random rotations of a fixed spectrum. In our setting the spectrum itself is random, and this randomness is what makes the problem hard. In our model, if the spectrum took a fixed value with positive probability, a single attention layer could be Bayes optimal on that event (Section 4, lines 222–225, and Appendix D.4), whereas with a random spectrum every single-layer configuration fails (Theorem 4.1). The spectrum is moreover coupled to the task vector and to the noise level. Rather than assuming this setting, we show that it arises from a natural latent factor model of multimodal data (Section 3, summarized in lines 42–44), which we view as a key contribution. Within it, and unlike the constructions of [2], we prove that gradient flow learns the Bayes-optimal predictor (Theorems 6.2–6.3).
W2. From a methodological perspective, the broader point is that cross-attention need not pair two modalities. A main contribution is demonstrating that a non-standard use of attention, particularly between an evolving quantity and a fixed function of the observations, is what works in this setting. Moreover, although the parameter space is restricted for tractability, the parameters are not set by hand. Theorems 6.2 and 6.3 prove that gradient flow learns them, including in the two-parameter model whose loss is non-convex and non-coercive. Figure 3 further shows the predicted whitening when (WtS,WtV) are untied, diagonal or full.
Q1. Our results establish sufficiency of the proposed architecture, not uniqueness. Extending our guarantees to unrestricted weights remains open and we mention this as a future direction (lines 538–539). In our analysis, the fixed-input cross-attention and the re-injection of X are shown to elicit the whitening mechanism of Eqn. 6.1 required of a Bayes-optimal predictor, but we do not preclude the possibility of other models exhibiting this whitening behavior. We note however that empirically, in ablations where we remove each of the proposed mechanisms we see that performance degrades (Appendix B and Table 1). To our knowledge, ours is the first proof that a deep attention-based model learns the Bayes-optimal predictor for a multimodal ICL task. Figure 3 shows the untied, diagonal and full weights clustering around the tied limits along training. If this clustering were established as a property of the training dynamics, the unrestricted problem would reduce layer by layer to the one we solve, which is how we view our analysis as the first step toward the general case.
Q2. The aspect of the multimodal setup that requires new techniques is the random spectrum, together with its coupling to the task vector and the noise level (see W1). It leads to three technical advances: (i) We use Gaussian exponential tilting to reduce finite-head softmax attention to linear attention in Theorem 4.1. (ii) In the one-parameter setting, we introduce epi-convergence type arguments from variational analysis (to our knowledge, this is new to ICL theory) to prove convergence to the Bayes optimal estimator. (iii) In the two parameter case, the loss is non-convex and non-coercive so gradient flow at fixed depth is only guaranteed to reach a stationary point, and the stationary points themselves change with the depth. We confine the stationary points to a region where the loss gradient is stable uniformly in depth and show that every stationary point collapses to the Bayes-optimal solution as depth grows (Theorem 6.3). These challenges arise precisely due to the randomness in the spectrum of the covariates, and do not have counterparts in the prior fixed covariance analysis. To our knowledge, Theorems 6.2–6.3 are the first training guarantees when the spectrum varies across prompts.
We hope these clarifications and the revised manuscript address the reviewer's concerns, in particular on what is new relative to [1] and [2], and we would be glad to clarify anything further.