Informativity Thresholds for In-Context Learning

When does in-context learning (ICL) succeed, and when does it fail? We give a quantitative answer by building an informativity theory of ICL on Willems' fundamental lemma: the prompt is recorded input-output data, the lemma specifies how much excitation that data must contain before the continuation is determined, and the conditioning of the data sets how error grows as information thins. Measured on this scale, a transformer trained on the task class nearly matches the analytic Bayes predictor and beats the model-free data-enabled predictor by 3-11x where only identification is possible, collapsing near the point where the prompt first determines the answer. The theory yields one universal law (a prompt that leaves the answer undetermined within the task class defeats every architecture) and one model-relative law: identification provably needs less context information than model-free interpolation, so how much is enough depends on the predictor's prior. Pretrained open-weight LLMs obey the same two laws in two regimes. On latent-system identification probes (2.7B-120B, five families) their error is flat in the prompt's information content, 11-18x above least squares regardless of scale or instruction tuning and 47-75x with a 3072-token reasoning budget: the bottleneck is the model's extraction, not the prompt. On short-memory sources, which their prior covers, error falls as context information grows and, where emissions are ambiguous, ends below both fixed-order n-gram baselines (four models, 2.7B-14B). Swapping only the training prior on one fixed transformer moves the same task between the regimes. Prompt engineering can supply information; whether a model can use it is fixed at training time.