Predictive Uncertainty Geometry in Low-Rank Language Model Adaptation

Bayesian low-rank adaptation makes uncertainty estimation tractable, but the quality of predictive uncertainty depends on the geometry and coordinates of the posterior covariance, not only on the adapter dimension. For nonlinear models whose likelihood depends on an informed subspace, we derive a novel trace law for mean-field directional distortion: factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement, whereas structured Gaussian approximations preserve the relevant geometry. We further show that covariance perturbations can substantially change posterior uncertainty while leaving integrated predictive probabilities and proper scores nearly unchanged. Controlled restricted-adapter experiments in Qwen3-4B and Phi-4-mini support these geometric mechanisms using HMC references. On Qwen, only 9 of 64 generalized covariance directions have MF/HMC variance ratio above two, yet 300 frozen predictive sensitivities place 78.4% of their HMC-whitened energy in those directions. Along a fitted covariance path, measured log-odds variance responses track first-order covariance-response predictions at Spearman 0.999, while matched-model predictive scores change negligibly; no selective-answering threshold is certified at target error rates through 20%. All ten matched-seed coordinate comparisons across Qwen and Phi favor the categorical-Fisher eigenbasis.