Certifying Factual Information Surplus in Noisy Transformer States

How is information about a learned fact distributed across the parameter groups of a transformer? We study this question using small transformers trained on synthetic data. Within each experimental repetition, models are trained on all possible assignments of a fixed set of binary facts, with shared initialization and data-order randomness across assignments. We model observations of saved parameter groups by adding independent Gaussian noise, which specifies measurement precision. We derive a computable lower bound on joint information surplus: the mutual information about an individual fact in a pair of groups observed together, minus the sum of the mutual informations in their separate observations. The method combines entropies of Gaussian distributions matched to the means and covariances of the observed states with analytic Rényi divergence corrections. We establish a sufficient condition, expressed through covariances between parameter groups, for detecting positive surplus when differences between trained states are small relative to observation noise. Evaluation using Gram matrices of parameter states and rigorous interval arithmetic certifies positivity without Monte Carlo integration. In a confirmatory experiment with a protocol fixed in advance, the method establishes small positive lower bounds across independent training repetitions, whereas the tested pairwise entropy and transport bounds do not certify positivity on the same data. Thus, for the studied source and noise level, jointly observing embeddings and the output group comprising final normalization and the answer head provides more information about an individual fact than the sum available from their separate observations. This result shows that characterizing distributed factual information at a specified observation precision requires accounting for dependencies between parameter groups.