acceptodds
Under review as a conference paper at ICLR 2027

When Broader Pretraining Hurts In-Context Learning: A Spectral Diagnostic from Inverse Linear Regression

Abstract

Broader pretraining can improve generalization in in-context learning, but adding source tasks can hurt performance on a fixed target family. We study this effect in ill-posed inverse linear regression (ILR), where the learner estimates a latent task vector from an underdetermined linear system. From a Bayesian viewpoint, pretraining provides a prior for this inference. For a Gaussian target prior with equal variance on a low-dimensional subspace and known rank and ridge ratio, learning the Bayes rule reduces to estimating that subspace. For block-diagonal source covariances, an eigengap determines whether additional tasks preserve the target subspace, create a tie, or displace target directions. We derive a sharp mixing threshold above which target-subspace recovery is lost. We prove this phase transition for a controlled two-stage ridge estimator and use a data-driven ridge estimator and a linear transformer to examine the same diagnostic empirically when estimation, architecture, and optimization also affect performance.

Primary area:
learning theory
Keywords:
in-context learning, inverse linear regression, pretraining, transfer learning, eigengap

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cz5DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies pretraining on a target task together with additional source tasks in in-context learning. It theoretically analyzes the possible negative effects of adding those tasks. The results show that when the added tasks have high variance in the target-task directions and low variance in other directions, the eigenvalue gap becomes larger, which the paper treats as positive transfer. Otherwise, adding the tasks can hurt performance.

Critical strengths

A key strength is that the paper explains when adding source tasks helps or hurts performance on the target tasks theoretically. It gives a precise threshold beyond which the added tasks cause the model to lose important information about the target tasks.

Critical weaknesses
  • The motivation is insufficient. The authors say that training is "often broadened by adding related source task families," but they do not cite practical papers that support this claim. They also say that the effect of adding tasks on the original task is not well understood, but they do not adequately compare this point with prior work. As written, it is unclear whether the paper studies a method that is commonly used to train real LLMs, whether that method causes a practical problem, or what is currently unknown about that problem. To make the motivation convincing, the authors should ground it more clearly in prior work, especially practical studies of real-world training methods, and explain how this paper builds on them.
  • The discussion is often abstract, and the information is not presented in a clear order, which makes the paper very difficult to read. All sections are hard to follow, but the Introduction is especially difficult. The following examples were unclear to me when I first read the paper.
    • "The Bayesian formulation makes this role explicit" (L36): Does the explanation that follows explain why the Bayesian formulation makes the role explicit?
    • "The inverse map depends on the prior only through the target subspace" (L37): What does this mean?
    • "ridge ratio" (L38): What is this of?
    • "The block-diagonal source assumption" (L42): At this point, it is unclear what this means.
    • "Pooled mLSE covariance" (L44): At this point, it is unclear what is being pooled or which covariance is meant.
    • "The covariance identity" (L72): This probably refers to Eq(9), but readers do not see what it means at this point.
    • "When off-target source variance exceeds the smallest on-target source variance" (L73): This sounds like it compares a scalar with a matrix.
    • "A controlled two-stage ridge estimator" (L74): What makes this estimator "controlled"?
    • "Under the stated dimension condition" (L75): With respect to what is the excess measured?
    • "Off-support sources" (L76): What does "support" refer to?
    • "Under the stated dimension condition" (L64): The text refers to a dimension condition, but no such condition appears at this point.

I believe substantial revisions to the entire paper are necessary for acceptance.

Questions for authors
  • The paper's main contribution seems to be that mixing in source tasks can hurt estimation when their differences from the target task are strong. However, negative transfer from source tasks that do not align with the target is a classical finding in multitask learning and transfer learning. The paper also notes that this has been already shown in ICL. Although this paper studies a different model theoretically, does it offer any new insight beyond deriving the same result for a specific ICL model?
  • In real-world LLM pretraining, what does the operation "often broadened by adding related source task families" refer to?
  • Can adding source tasks improve performance under the setting studied in this paper? Also, why is the eigenvalue gap a justified measure of positive transfer? If adding source tasks does not improve performance, why add them?
  • What practical procedure does the heuristic of selecting the top rr tasks abstract?
  • cTRE assumes that rwr_w and ρ\rho are known. Please explain in more detail why this simplification is justified.
1 comment

We thank the reviewer for reading the paper carefully and for the constructive comments, which will help us to improve the paper.

Q1 (What is new?). A mechanism with an exact threshold between positive and negative transfer, particularly a rigorous mathematical theory for this phase transition. This builds on previous empirical observations, but to our knowledge, it is the first rigorous mathematical analysis for the learning theory.

Q2 (What does "broadened" refer to in real-world LLM?). Our model does not describe language-model pretraining. Our model is motivated by scientific foundation models trained on several PDE families, operator families, or image processing families. Returning to LLMs, just as an analogy (possibly a bad one), when the original tasks are math proofs, one may use non-math literature data for broader pretraining, e.g., novels or science books.

Q3 (Can sources help? Why the gap? Why adding them when they may not help?). Yes, adding aligned sources lowers the risk bound. The gap is necessary for the pre-training to recover the low-dimensional prior, necessary to solve the underdetermined linear system.

In practice, we don't know a priori whether additional training data would help: for example, in PDE modeling or MRI image processing, we have limited pre-training data and don't have the ground-truth task prior, so we use other datasets to broaden pre-training.

Q4 (What practical procedure does the heuristic of selecting the top tasks abstract?). Learning a shared low-dimensional representation across tasks. This is a low-rank prior in the Bayesian setting or the regularization in inverse problems. Any successful pretraining must identify that low-dimensional representation.

Q5 (Why assume rwr_w and ρ\rho known?). They serve as a control: they make the excess risk exactly the subspace error, which can be rigorously analyzed. If they are to be estimated from data (i.e., as in dTRE), rigorous analysis is difficult, and their estimation error will degrade the phase transition, as shown in the numerical results of dTRE in Figure 3.

Reviewer_P4ZROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how adding source tasks during pretraining affects in-context learning performance on a fixed target distribution. The authors focus on inverse linear regression, where the goal is to recover a latent task vector from an underdetermined linear system. The target task vectors are sampled from a Gaussian distribution supported on a fixed low-dimensional subspace, while the additional source tasks may be aligned with this subspace or have components outside it. The main theoretical analysis assumes isotropic Gaussian inputs and noise, equal nonzero eigenvalues of the target covariance, and a source covariance that is block-diagonal with respect to the target subspace and its orthogonal complement.

The authors analyze a two-stage ridge regression estimator. In the first stage, the estimator computes a minimum-norm least-squares solution for each pretraining task and uses the leading eigenvectors of their covariance matrix to estimate the target subspace. In the second stage, given a new target task, it performs ridge regression restricted to the estimated subspace. Section 3 analyzes the population version, assuming the covariance is known, while Section 4 studies a finite-sample version (cTRE) that estimates it empirically. Both assume knowledge of the target subspace dimension and the optimal ridge regularization parameter.

Finally, the authors present synthetic experiments with cTRE, a data-driven ridge estimator (dTRE), and a trained linear transformer. The results support the theoretical findings for cTRE and show qualitatively similar negative-transfer behavior for dTRE and the transformer, although both exhibit smoother degradation rather than a sharp phase transition.

Critical strengths

Clear and well-presented theoretical analysis. The paper is well written and provides clear explanations of the main theoretical results and their derivations.

Critical weaknesses
  1. Unclear significance and generality of the theoretical contribution. I find it difficult to identify the main new insight and its significance for the ICL literature. Prior work, particularly Letey et al. (ICLR 2026), has already demonstrated that broader pretraining can hurt ICL generalization depending on the alignment between pretraining and test-task distributions. The current paper extends this line of work by establishing an exact finite-dimensional threshold for target-subspace recovery and finite-sample excess-risk bounds for a specific two-stage ridge estimator. However, much of the prior theoretical literature on ICL analyzes simplified versions of transformers, such as linear-attention models. In contrast, the theoretical analysis here focuses on a hand-designed estimator that explicitly performs spectral subspace estimation followed by ridge regression. It is unclear how this estimator relates to the mechanisms learned by transformers, or whether the theoretical results provide new insights that generalize beyond this particular estimator.

  2. Additional weaknesses: The assumption that the target rank and regularization parameter are known and the limited experimental validation beyond the proposed estimator.

Questions for authors
  1. What motivates the choice of the leading-eigenspace estimator? In particular, the negative-transfer threshold seems to characterize a failure of this specific estimator rather than a fundamental statistical limitation. Could the authors clarify whether the identified negative-transfer phenomenon extends to other estimators, and what makes the proposed estimator particularly relevant to ICL?

  2. Do you think that a linear transformer trained with gradient descent learns a mechanism similar to the proposed two-stage ridge estimator?

1 comment

We thank the reviewer for the careful reading and for questions that go to the heart of the paper.

Q1 (Motivation of the leading-eigenspace estimator; is the negative-transfer specific to the estimator, or does it hold for all estimators). The leading eigenspace from pretraining is necessary to solve the underdetermined linear system Y=Xw+ϵY=Xw+\epsilon from the context (X,Y)(X,Y), where X∈Rn×dX\in \mathbb{R}^{n\times d} with n<dn<d. That is, the latent task vector w∈Rdw\in\mathbb{R}^d must be estimated using the low-dimensional task prior, which is the leading eigenspace of the task-prior covariance (the target subspace).

Negative transfer (if it happens) holds for all estimators. Every accurate estimator must explicitly or implicitly extract this leading eigenspace. When we broaden the pre-training with new task vectors from a different distribution, the task prior of the augmented pretraining data shifts away from the original task prior. This shift may help or hurt the recovery of the original leading eigenspace. Thus, negative-transfer occurs when the shift hurts the recovery, and it applies to all estimators. Figure 3 verifies this point by showing that the linear transformer and the data-based two-stage estimator follow the same transition.

Q2 (does a trained linear transformer learn this mechanism?). The linear transformer approximates the Bayesian rule, but due to the approximation error, it shows the same diagnostic with a smooth degradation, as shown in Figure 3.

On the known rank and ridge ratio: This is the controlled setting so that the controlled two-stage ridge estimator (cTRE) can be rigorously analyzed. This study aims to answer whether broader pretraining can hurt ICL by analyzing the ICL-ILR setting, not to propose a practical estimator. The practical estimators (i.e., dTRE and linear transformer) relax this constraint and show a degraded phase transition in Figure 3, but they are difficult to analyze.

Reviewer_GyFEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors pose ICL as a type of implicit Bayesian inference, and seeking insights about transformer ICL they study Bayesian updates in the case of linear regression, where a subspace must be estimated in order to solve the task at hand. There are also some experiments that appear to be about evaluating linear transformer ICL on synthetic data.

Critical strengths

Understanding ICL in transformers is an important open problem, and attempting to leverage insights from Bayesian inference is a reasonable approach.

Critical weaknesses

The paper is lacking in a clear narrative arc. I suspect this is due to overreliance on LLM tooling in the research. Experimental results are not communicated effectively, whatsoever. For example it was not clear what the dataset is that is used in Sec 5, or what its relevance is to ICL in LLMs. The theoretical results are quite dense and hard to follow.

The paper focuses on TRE (in its so-called conditional and data-driven variants, cTRE and dTRE) as a way to understand ICL behavior. However I found this choice of framework to be under-motivated. I gather that Lu and Yu 2025 [https://arxiv.org/abs/2505.12138] use TRE as a baseline to compare with transformer ICL, but the submission itself does not provide a clear explanation for why this is a suitable analytic framework. Additionally, the focus on synthetic regression data adds to my concern that the results do not provide insights on realistic ICL settings (e.g. LLMs) and undercut the soundness of the submission.

Ultimately understanding Bayesian inference in linear regression may provide some useful intuitions for how transformers do ICL, I am not seeing much compelling evidence from this paper that would be useful to ICL practitioners.

Questions for authors

None

1 comment

We thank the reviewer for taking the time to review our paper and for the comments on the narrative and presentation, which will help us revise the paper and clarify the main message.

This paper does not aim to understand how transformers do ICL or ICL for LLMs. Instead, we focus on rigorously analyzing the effect of broader pretraining (using additional source data whose distribution differs from the original target distribution) in a controlled ICL setting (ICL for inverse linear regression).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.