acceptodds
Under review as a conference paper at ICLR 2027

Predictive Uncertainty Geometry in Low-Rank Language Model Adaptation

Abstract

Bayesian low-rank adaptation makes uncertainty estimation tractable, but the quality of predictive uncertainty depends on the geometry and coordinates of the posterior covariance, not only on the adapter dimension. For nonlinear models whose likelihood depends on an informed subspace, we derive a novel trace law for mean-field directional distortion: factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement, whereas structured Gaussian approximations preserve the relevant geometry. We further show that covariance perturbations can substantially change posterior uncertainty while leaving integrated predictive probabilities and proper scores nearly unchanged. Controlled restricted-adapter experiments in Qwen3-4B and Phi-4-mini support these geometric mechanisms using HMC references. On Qwen, only 9 of 64 generalized covariance directions have MF/HMC variance ratio above two, yet 300 frozen predictive sensitivities place 78.4% of their HMC-whitened energy in those directions. Along a fitted covariance path, measured log-odds variance responses track first-order covariance-response predictions at Spearman 0.999, while matched-model predictive scores change negligibly; no selective-answering threshold is certified at target error rates through 20%. All ten matched-seed coordinate comparisons across Qwen and Phi favor the categorical-Fisher eigenbasis.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian deep learning, predictive uncertainty, variational inference, low-rank adaptation, posterior geometry, covariance approximation, parameterization dependence, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EUdeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyze Bayesian low-rank adaptation. The author argue that factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement. Experiment on Qwen3-4B/Phi-4- mini show full covariance Gaussian better mimic HMC compared to MF Gaussian VI.

Critical strengths

The uncertainty quantification in large language model is important.

Critical weaknesses
  1. The problem lack clear motivation. Could the author enumerate a specific application of Bayesian low-rank adaptation?

  2. There exists some gap between the designed experiment and the theoretical analysis. It would be nice if the author can design some toy experiment visualizing the mechanics shown in the theoretical analysis

  3. Isn't the main argument " factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement. " obvious?

  4. sentences with AI pattern which are hard to understand :

  • abstract: "All ten matched-seed coordinate comparisons across Qwen and Phi favor the categorical-Fisher eigenbasis"
  • Page 2 line 99: "Fixing A removes the two-factor ambiguity in BA from a matched-posterior comparison. The adapter matrix rank, stochastic dimension p, informed dimension m, and represented covariance rank kare all distinct quantities. "
Questions for authors
  1. What does it mean by "INFORMED SUBSPACE"?

  2. I have no idea what the following sentence means:

  • Page 2 line 101 "We adopt the convention that uncertainty is conditional on the frozen backbone and any frozen adapter factors."

  • Page 2 line 103 "gz(x) denote identifiable logit contrasts." What does it mean by " logit contrast"?

  • Page 3 line 128 "Its null complement is retained exactly by Bayes"

3 comments
AuthorsOpenReview ↗

Response to Reviewer EUde (1/2): Motivation and illustrative example

We thank the reviewer for the detailed questions on motivation and clarity. These questions helped us explain the original geometric argument more clearly and make its practical consequence explicit through a two-coordinate example, an acquisition theorem, and targeted experiments.

On W1. Motivation: a specific application of Bayesian low-rank adaptation.

A concrete application is label acquisition. When a language model is adapted under a limited annotation budget, which unlabelled example should be annotated next? Bayesian active learning by disagreement (Houlsby et al., 2011), or BALD, scores a candidate by how much its label would reveal about the parameters, and this depends on how predictions vary across posterior draws. Two approximations can give the same integrated predictive probabilities, and hence the same accuracy and NLL, but different information gains. A second application, already studied in the submitted version, is selective answering, in which the model abstains on inputs with high uncertainty (Appendix C.6).

Section 4.1 makes the acquisition consequence exact. In the logistic model of Theorem 4.4, exact Bayes eventually prefers a query about a direction the data have not informed, while mean field prefers a query about a direction already learned. Their predictive means and scores agree exactly. Even so, the exact-posterior regret of the mean-field choice converges to a positive constant, and no scalar rescaling of the mean-field covariance changes its ranking. Figure 1 shows the mechanism and the reversal.

We then test the application on Phi. On the 300 MMLU-Pro prompts, Fisher coordinates reduce the error of the acquisition values by about 70%, while the top query stays the same. On ARC-Challenge, with actual acquisition and refitting, both uncertainty-based rules give lower final NLL than random selection in all five runs, with no mean final-NLL advantage from Fisher coordinates over native coordinates (Section 5.2).

On W2. Gap between theory and experiments; a toy example visualizing the mechanism.

We agree, and the revision adds such an example, built under the assumptions of the theory. Section 3 now introduces the mechanism with two coordinates before the nonlinear theorem. Under a standard normal prior, put t=(z1+z2)/2t=(z_1+z_2)/\sqrt{2} and v=(z1−z2)/2v=(z_1-z_2)/\sqrt{2}, and let a balanced logistic likelihood depend only on tt. Exact Bayes learns tt and leaves vv at its prior variance. Any diagonal Gaussian with covariance diag⁡(d1,d2)\operatorname{diag}(d_1,d_2) in (z1,z2)(z_1,z_2) has

Var⁡q(t)=Var⁡q(v)=(d1+d2)/2,\operatorname{Var}_q(t)=\operatorname{Var}_q(v)=(d_1+d_2)/2,

so it cannot make the learned sum precise while keeping the unobserved difference uncertain (Equation 4). Exact Bayes does both through dependence between z1z_1 and z2z_2.

Figure 1 visualizes this posterior and the decision theorem built on it. At n=100n=100, the information gains (nats) of the candidate probabilities σ(2t)\sigma(2t) and σ(v)\sigma(v) are:

Candidate queryExact posteriorNative mean field
Informed: σ(2t)\sigma(2t)0.018360.03400
Unobserved: σ(v)\sigma(v)0.093710.00916

Exact Bayes selects the unobserved query and mean field the informed one, while every predictive mean equals 1/21/2, so NLL and Brier scores agree for either realized label. The figure uses deterministic quadrature and the variational stationarity equations; Theorem 4.4 proves the limiting reversal, and the table illustrates it. Table 1 and Appendix C.4 keep the nonlinear support-factor and trace checks, with convergence plots. The transformer experiments then ask whether the same geometric effects appear in fitted posteriors, measured against Hamiltonian Monte Carlo (HMC).

AuthorsOpenReview ↗

Response to Reviewer EUde (2/2): Technical novelty and clarity of exposition

On W3. Is the main argument obvious?

The qualitative observation is known, and the paper says so: in both the submitted and the revised version, the contribution paragraph of the introduction opens by citing it as established (Margossian and Saul, 2023; Margossian et al., 2025; Odgers et al., 2026). What we add is the quantitative nonlinear limit and the predictive and decision statements that follow from it.

Under the fixed-profile likelihood and tail conditions of Assumption 3.1, Theorem 3.2 gives the limiting exact and mean-field informed covariances without assuming a Gaussian posterior; the proof first shows concentration of the optimal diagonal Gaussian and then expands its stationarity equations. Corollary 3.3 gives

tr⁡(R)=s,λmax⁡(R)≥s/m,\operatorname{tr}(R)=s,\qquad \lambda_{\max}(R)\ge s/m,

where ss counts the active native coordinates and mm is the informed dimension. At the same time, a supported null direction keeps exact variance one while its mean-field variance is of order 1/n1/n. For a scalar logistic likelihood, the predictive variance inflation is exactly ss to leading order, for every fixed success fraction 0<ρ<10\lt\rho\lt1 (Corollary 3.5). The text after Corollary 3.3 explains that s/ms/m is the average distortion, so individual predictive directions can still be uninflated.

Two further results concern observables and decisions. Theorem 4.2 shows that, under its nondegeneracy condition, a covariance direction can change uncertainty at first order while probability means change only at second order. Theorem 4.4 shows that identical predictive scores can coexist with a wrong information-gain ranking.

Section 5.1 now also states the coordinate comparison precisely. Native and Fisher fits each optimize pp variances. With a fixed orthogonal Fisher basis OO, their native-coordinate covariances are diag(v)\mathrm{diag}(v) and Odiag(w)O⊤O\mathrm{diag}(w)O^\top. At p=64p=64 both have 64 covariance parameters, against 2080 for an unrestricted covariance. The Fisher fit is generally dense in native coordinates, with eigenvectors fixed by the chosen basis and eigenvalues fitted, and the prior, likelihood, subspace and mean are shared. The comparison therefore isolates the choice of axes for a diagonal approximation.

On Q1. Sentences and terms that were hard to follow.

Informed subspace. These are the parameter combinations on which the likelihood depends. If the likelihood sees only t=U⊤zt=U^\top z, moving zz orthogonally to the columns of UU leaves it unchanged. Section 3 now gives this definition with the sum/difference example from point 2, before the assumptions.

Conditioning on frozen components (p. 2, l. 101). We place a distribution only on designated coordinates of the output factor and hold the pretrained network and the input factor fixed, so the reported uncertainty excludes uncertainty about those fixed quantities. Section 2 now gives the total-variance decomposition and an example in which the data identify the product aBaB, the sign aa stays uncertain, and the conditional variance of the new prediction BB vanishes while its joint variance stays positive.

Logit contrasts (p. 2, l. 103). Softmax is unchanged when the same number is added to every logit, so we use differences ηj−ηK=log⁡(p(j)/p(K))\eta_j-\eta_K=\log(p(j)/p(K)), which are identifiable log probability ratios. Section 2 gives this formula at first use.

"Its null complement is retained exactly by Bayes" (p. 3, l. 128). Write z=Ut+wz=Ut+w with U⊤w=0U^\top w=0. The standard Gaussian prior makes tt and ww independent, and the likelihood depends only on tt, so conditioning on the data leaves the law of ww at the prior. In the two-coordinate example, vv is the scalar coordinate along this null direction. Section 3 now spells this out.

Abstract sentence on coordinate comparisons. It meant that in each of five paired optimization runs on each backbone, log-odds variance estimates in the Fisher basis have smaller error against HMC than those in native coordinates. The abstract now names the quantity and the reference.

Matrix rank and the dimensions pp, mm, kk (p. 2, l. 99). Section 2 now works through a rank-one adapter: BB is a column with doutd_{\mathrm{out}} entries and AA is a row, so BABA has rank at most one even when every entry of BB is random. The dimension pp counts the coordinates we vary, mm the combinations the likelihood sees, and kk the directions a structured covariance models explicitly, with prior variance in the complement.

We hope that the clearer exposition, the illustrative example and its proved decision consequence, and the additional theoretical and empirical results address your main concerns. If so, we would be grateful if you would reconsider your assessment and score in light of the revised manuscript. We would also welcome your feedback on any concerns that remain and we remain committed to address those as well to the best of our ability.

AuthorsOpenReview ↗

Follow-up: Motivation and geometric intuition

Thank you again for your questions on motivation and clarity. The revision develops the same central argument as the submission: covariance geometry can substantially affect predictive uncertainty even when predictive scores barely change. The original nonlinear trace law and covariance-response analysis remain the foundation.

Two places give a focused route through the clarifications:

  • Mechanism and clarity — Section 3. The two-coordinate example shows why exact Bayes can make a learned sum precise while retaining uncertainty in an unobserved difference, whereas a Gaussian diagonal in the original coordinates cannot separate these variances. It introduces the mechanism quantified by the nonlinear trace law.
  • Concrete application — Section 4.1, Figure 1 and Theorem 4.4. Exact Bayes and mean field have identical predictive means, NLL and Brier scores, yet eventually select different acquisition queries. The exact-posterior regret of the mean-field choice converges to a positive constant.

These additions make the intuition and decision consequence of the original geometric findings explicit. Could you please let us know whether they address your concerns about motivation and clarity? We would be happy to clarify any specific step.

Reviewer_yAXwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the uncertainty estimation problem in low-rank adaptation of language models. The paper builds on the existing Bayesian adaptation and asks a fundamental question regarding how faithfully the current approximations are successful in estimating the true posterior, which is ultimately responsible for efficient predictive uncertainty. First, the paper establishes a nonlinear directional-distortion theorem. Next, they demonstrate how perturbations to the covariance can change the posterior without substantial changes in popular metrics. Finally, they conduct experiments with two language models to demonstrate their theoretical findings empirically.

Critical strengths
  • The paper presents fundamental work on Bayesian low-rank language model adaptation.

  • The paper provides an important finding regarding how, despite substantial covariance errors, popular metrics fail to differentiate them.

Critical weaknesses
  • Assumption question: How valid is the conclusion if A is not fixed?

  • Minor: It’s suggested to introduce full forms first before writing short forms (example: MF/HMC) to enhance reliability.

  • The work is done for uncertainty in multiclass setting, while the current uncertainty estimation demands in language models require the assumption of autoregressive generation. How can the current work support the practical demands of language models?

Questions for authors

See weakness section

3 comments
AuthorsOpenReview ↗

Response to Reviewer yAXw (1/2): Frozen factors and notation

We thank the reviewer for recognizing the main finding, and for the three questions, which led to a new example, clearer notation and a new experiment.

On W1. How valid is the conclusion if AA is not fixed?

The revision now states when the conditional result applies and gives a case where carrying it over to joint-factor uncertainty would be wrong. For a joint posterior and a scalar prediction hh,

Var(h∣D)=E[Var(h∣A,D)∣D]+Var(E[h∣A,D]∣D).\mathrm{Var}(h\mid D)=\mathbb{E}[\mathrm{Var}(h\mid A,D)\mid D]+\mathrm{Var}(\mathbb{E}[h\mid A,D]\mid D).

Freezing one fitted factor replaces the average in the first term by a single conditional variance and drops the second term. This matters for identifiable predictions, not only for the nonunique factorization BABA.

In our example, B∼N(0,1)B\sim N(0,1) and A=(a,1)A=(a,1) with a∈{−1,1}a\in\lbrace -1,1\rbrace equally likely, and we observe Yi=aB+ϵiY_i=aB+\epsilon_i at input (1,0)⊤(1,0)^\top, with independent ϵi∼N(0,1)\epsilon_i\sim N(0,1). At the new input (0,1)⊤(0,1)^\top the prediction is BB. With cn=(1+n)−1c_n=(1+n)^{-1} and mn=nYˉ/(1+n)m_n=n\bar Y/(1+n),

Var⁡(B∣a,D)=cn,Var⁡(B∣D)=cn+mn2.\operatorname{Var}(B\mid a,D)=c_n,\qquad \operatorname{Var}(B\mid D)=c_n+m_n^2.

If Yˉ→b⋆≠0\bar Y\to b_\star\neq0, the conditional variance tends to zero while the joint variance tends to b⋆2>0b_\star^2>0. Section 2 now contains this example; Proposition A.1 and Example A.2 (Appendix A.2) state it formally, with proofs in Appendix B.9.

For a given frozen A=aA=a, the nonlinear theorem applies when the conditional prior can be whitened to a standard Gaussian and the conditional likelihood satisfies the informed-subspace assumptions. When AA is fitted and then frozen, Proposition A.3 gives sufficient uniform conditions, since pointwise conditions do not cover a data-dependent factor. Joint inference over (A,B)(A,B) is a different target. The Gaussian response identities of Section 4 also apply to a joint Gaussian approximation over the vectorized (A,B)(A,B) when their smoothness conditions hold. Making one factor stochastic and the other deterministic has precedent in BLoB (Wang et al., 2024). All our experiments condition on a fixed AA, and the paper states this restriction explicitly.

On W2. Introduce full forms before abbreviations such as MF and HMC.

We expand MF and HMC at first use and clarify the other notation as follows. The introduction expands variational inference (VI) and Hamiltonian Monte Carlo (HMC). Section 1.2 expands mean field (MF), Kullback-Leibler (KL), the no-U-turn sampler (NUTS), effective sample size (ESS), negative log-likelihood (NLL), expected calibration error (ECE), evidence lower bound (ELBO) and maximum a posteriori (MAP), and says what role each plays. Section 5.2 expands Bayesian active learning by disagreement (BALD). Where we report that runs favor a basis, we now name the quantity compared: log-odds or sequence-variance error against HMC.

AuthorsOpenReview ↗

Response to Reviewer yAXw (2/2): Autoregressive theory and experiments

On W3. The analysis is multiclass, while language models generate autoregressively. How does the work address this?

The revision adds a finite-sequence extension of the theory and a full-vocabulary autoregressive experiment, including actual generation. Under the same restricted p=64p=64 Phi posterior fitted to MMLU-Pro answer choices, Fisher coordinates have lower mean error for the variance of multi-token sequence log probabilities:

TargetNative errorFisher errorDifferenceFits with lower Fisher error
Greedy (primary)0.1215710.091839-0.0297324/5
Canonical (secondary)0.0727470.036029-0.0367185/5

The error is the absolute log ratio of the estimated sequence-log-probability variance to its HMC reference, with a common 10−1210^{-12} additive offset, averaged over 32 prompts, five paired saved fits and two Gaussian Monte Carlo repeats. The reductions, 24.5% and 50.5%, are reductions in this error. One greedy fit favors native coordinates, and we report every outcome (Section 5.3 and Table 2; all paired values in Appendix C.15).

Why this is autoregressive. The prompts keep the ARC options but ask for an answer in words with a short explanation. Each greedy target is frozen before the covariance comparison; all 32 contain 18 to 32 tokens, 19 end with end-of-sequence, and 13 are horizon-limited prefix events. Every token probability is normalized over 200064 tokens. One parameter draw governs every position, so the mixture is

Pˉq(y1:T∣x)=Ez∼q[∏t=1Tpz(yt∣x,y1:t−1)],\bar P_q(y_{1:T}\mid x)=\mathbb E_{z\sim q}\left[\prod_{t=1}^{T}p_z(y_t\mid x,y_{1:t-1})\right],

rather than a product of token averages. The empirical mixture and its prefix-reweighted factorization agree within 7.11×10−157.11\times10^{-15} in log probability. We also sample 704 continuations on eight prompts from HMC and all ten Gaussian fits, holding one parameter draw fixed per trajectory and sampling the full vocabulary at temperature one.

What the theory gives. Under explicit smoothness and growth conditions, Proposition A.10 extends the Gaussian covariance-response identities to fixed finite-horizon sequence events and sequence log probabilities, and Proposition A.11 gives the Fisher decomposition over model-generated prefixes (Appendix A.7, summarized in Section 4.1). Both hold for a fixed horizon.

Reference checks. We recompute HMC diagnostics for the sequence functionals: the maximum rank-normalized/folded R^\widehat R is 1.004441 and the minimum batch-means ESS over batch lengths 25, 50 and 100 is 1915.797, so every reference gate passes. The block-50 95% interval for the primary contrast, [−0.034609,−0.024944][-0.034609,-0.024944], reflects HMC Monte Carlo error given the prompts, fits and Gaussian draws; other block lengths give similar intervals. Absolute log-mixture error moves from 0.115735 to 0.114336 for greedy targets and from 0.121632 to 0.123848 for canonical targets, so the fidelity gain in variance does not carry over uniformly to sequence probabilities.

Numerical precision. Backbone activations are in bfloat16 (BF16); adapter perturbation, normalization and scoring are in float64, and the generation head in float32. Direct and reduced float64 log probabilities agree within 5×10−145\times10^{-14} in the preflight probe. A fully native-BF16 tail differs by up to 0.043453 in token probability, above the original 0.02 tolerance, already at zero adapter perturbation; native rounding applied to the same cached zero-adapter states reproduces this difference. Before any comparative outcome was produced, we amended the validation policy: independent dense-adapter and matched-native replay checks became the blocking implementation checks, and the cross-precision comparison is kept as a reported flag, including this failure. The model, posterior, prompts, draw plan and endpoints did not change. Appendix C.15.4 documents the amendment and the probe's scope of one prompt, three tokens and three latent states.

In sum, we now evaluate autoregressive uncertainty over the full vocabulary and sample from the posterior predictive under a stated numerical model. The posterior is still trained on the answer-choice likelihood, with one fixed-factor projection random, and we make no claim about semantic accuracy.

Thank you again for your constructive feedback. We hope that the conditional versus joint-factor analysis, clearer notation, and new autoregressive theory and experiments address your questions. If so, we would appreciate your updated assessment and score in light of the revised manuscript. We would also welcome your feedback on any remaining concerns.

AuthorsOpenReview ↗

Follow-up: Frozen factors and autoregressive uncertainty

Thank you again for your questions on frozen factors and autoregressive generation. The revision extends the original uncertainty analysis in the two directions you identified:

  • Frozen factors — Section 2 and Appendix A.2. The variance decomposition makes the distinction between conditional and joint-factor uncertainty explicit. The accompanying example shows how conditional predictive variance can vanish while joint predictive variance remains positive.
  • Autoregressive evaluation — Section 5.3 and Table 2. Using the original 64-dimensional Phi posterior fitted to MMLU-Pro answer choices, we evaluate full-vocabulary, multi-token continuations on 32 ARC-Challenge prompts. Fisher coordinates reduce the mean error in sequence log-probability variance against HMC by 24.5% relative to native coordinates on the primary targets. Appendix A.7 also extends the covariance-response identities to fixed finite horizons.

The sequence experiment carries the original covariance comparison into autoregressive prediction under the same fitted posterior. Could you please let us know whether this evaluation and the conditional-versus-joint analysis address your two main questions? We would be happy to clarify any particular point.

Reviewer_eZC4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how covariance and parameter coordinates affect uncertainty estimation in Bayesian LoRA. The authors show that mean-field variational inference can overestimate or underestimate uncertainty, while these differences may have little effect on predictive metrics. Experiments on MMLU-Pro dataset with 2 models support these findings.

Critical strengths
  • The paper is clearly written and well-motivated.
  • The theoretical analysis provides useful insights into how mean-field variational inference can distort predictive uncertainty in Bayesian LoRA.
  • The paper provides a theoretical explanation for why standard predictive metrics may fail to detect errors in posterior uncertainty estimation.
Critical weaknesses
  • The experiments only use one dataset. More experiments on different tasks can expand the scope of contribution.

  • The experiments are restricted to a 64-dimensional posterior, while the full LoRA adapter contains thousands of parameters, which limits the generalizability of the findings.

  • The practical benefits are limited, as the improved covariance estimation does not lead to significant improvements in NLL, Brier score, or ECE.

Questions for authors

Could the authors extend the scope of the experiments to further demonstrate the generalizability of the findings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eZC4 (1/2): Additional tasks and larger posterior dimensions

We thank the reviewer for the positive assessment of the clarity and theory, and for the questions on scope, which led to the new experiments.

On W1. The experiments use one dataset. Can other tasks expand the scope? and Q1: extending the experimental scope

We add a second dataset and two new forms of evaluation beyond the original covariance audit:

AspectSubmitted evidenceAdded evidence
DatasetMMLU-ProARC-Challenge, official 1119/299/1172 train/validation/test split
DecisionPredictive and selective-answering auditsFour acquisition rounds with refitting, 256 to 512 labels, five paired runs
Dimensionp=64p=64 with HMC referencep=64,256,1024p=64,256,1024, five paired fits at each
OutputAnswer-choice probabilitiesFull-vocabulary sequences: 64 targets on 32 prompts, and 704 sampled continuations

ARC-Challenge (Clark et al., 2018) keeps every question in its original option order, including three- and five-choice questions. The acquisition study reveals 64 training labels per round and refits the model. A common maximum a posteriori (MAP) procedure is then trained on each acquired set, so the comparison isolates the acquisition rule, and test labels are used only after each trajectory is complete. Section 5.2 summarizes the design and results, Appendix D.7 gives every round, and Appendix C.13 gives the implementation.

The sequence study asks for an answer in words with a short explanation and normalizes each token over the full 200064-token vocabulary. The primary HMC-referenced variance error falls from 0.121571 to 0.091839, with four of five paired fit averages favoring Fisher coordinates, and we also generate continuations with one parameter draw held fixed per trajectory (Section 5.3, Table 2). The prompts keep their options, the horizon is 32 tokens, and the posterior is the original restricted Phi model trained on answer choices. Within that setting, the study extends the observable from answer-letter probabilities to whole sequences.

On W2. The 64-dimensional posterior is much smaller than a full LoRA adapter.

The new dimension study raises the stochastic dimension by a factor of 16, using all 1119 ARC training examples and five paired optimization seeds at each dimension. The table reports posterior-predictive NLL:

Dimension ppNative NLLFisher NLLFisher minus native
640.5016390.502106+0.000468
2560.5015200.502022+0.000502
10240.5015200.502032+0.000512

Entries are means over five fits; lower NLL is better, and all 15 paired differences are positive. Brier scores follow the same ordering, and accuracy is identical. Fisher coordinates give lower BALD values and a lower estimated negative ELBO. The latter indicates a better achieved reverse-KL objective, subject to Monte Carlo error; without an independent posterior reference for ARC, we cannot determine which fit estimates predictive variances more accurately. Within each dimension, native and Fisher fits share the subspace, prior and MAP mean. Across dimensions, nested subspaces with prior scale 264/p2\sqrt{64/p} keep the total output-factor prior energy fixed, so each dimension defines its own Bayesian model. Table 19 gives the paired ranges, and Appendix C.14 (Tables 9 and 10) gives every pair and the secondary metrics.

Thus the predictive study now covers p=64,256,1024p=64,256,1024. HMC-referenced fidelity and autoregressive evaluations use p=64p=64 MMLU-Pro posteriors; the ARC-trained posteriors have no HMC reference. The main nonlinear limit still keeps dimension fixed. Multilayer and joint-factor experiments remain open: every experiment adapts one final-block projection with the input factor fixed. The conclusion and Appendix C.16 distinguish these completed studies from the remaining scope.

AuthorsOpenReview ↗

Response to Reviewer eZC4 (2/2): Predictive scores and practical benefit

On W3. Improved covariance estimation does not significantly improve NLL, Brier score or ECE. What is the practical benefit?

We agree that better uncertainty fidelity alone does not establish a gain in these scores. Our theory explains why agreement in predictive scores need not certify uncertainty fidelity. Along the directions of Theorem 4.2, Remark 4.3 shows that proper scores respond to a covariance change only at second order or higher, while the uncertainty responds at first order. The score expansions concern NLL and Brier; ECE is an empirical calibration diagnostic here. Small score differences therefore leave covariance accuracy unresolved. The original Qwen comparison shows this: the structured Prediction-PC16 covariance lowers the median HMC-referenced log-variance error from 0.840 to 0.348, while NLL, Brier and ECE change by less than 10−310^{-3} (Section 5.1). The practical question is whether the uncertainty error changes a decision, and we test two steps: the selected query, and prediction after acquiring labels and refitting.

On the 300 frozen Phi MMLU-Pro prompts, Fisher coordinates reduce the root mean squared error of BALD values against HMC from 5.474×10−45.474\times10^{-4} to 1.649×10−41.649\times10^{-4} nats, about 70%. All 20 fit-repeat comparisons still choose the same top query, which is also the HMC maximizer, so the primary selected-query utility difference is exactly zero (Section 5.2, Table 17). Here a large fidelity gain leaves the action unchanged.

ARC-Challenge tests acquisition with refitting. After 512 labels, the common MAP procedure gives the following; NLL is the prespecified primary endpoint.

Acquisition ruleTest NLLBrierECE
Native-coordinate mean field0.5024510.2426820.057595
Fisher-coordinate mean field0.5024780.2426840.057220
Random0.5039010.2434030.060469

Both uncertainty-based rules have lower final NLL than random acquisition in all five paired runs. Fisher has lower final NLL than native in one run, and the mean native-minus-Fisher NLL difference is −2.753×10−5-2.753\times10^{-5}. We keep the final-round endpoint and report all five runs (Table 18; Appendix C.14). The autoregressive study shows the same pattern: the primary sequence-variance error improves by 24.5%, while canonical-target log-mixture error worsens slightly and mixture NLL per token moves only from 3.629305 to 3.628018.

We see two practical conclusions. Theorem 4.4 gives a decision problem in which equal predictive scores conceal a wrong acquisition ranking with nonvanishing regret, so score agreement cannot certify an uncertainty-based decision. On MMLU-Pro, the measured fidelity gain leaves the selected query unchanged. On ARC, both uncertainty-based rules achieve lower final NLL than random selection in all five paired runs, while Fisher coordinates give no mean final-NLL advantage over native coordinates. Without an independent reference for the ARC-trained posteriors, we do not claim a fidelity gain for those fitted targets. The abstract and conclusion distinguish the observed fidelity and predictive results.

Thank you again for your constructive feedback. We hope that the additional dataset, larger-dimensional experiments, and explicit comparison of uncertainty fidelity with predictive and decision performance address your questions. If so, we would appreciate your updated assessment and score in light of the revised manuscript. We would also welcome your feedback on any remaining concerns.

AuthorsOpenReview ↗

Follow-up: Broader experimental evidence

Thank you again for suggesting a broader experimental evaluation. The original Qwen and Phi comparisons remain the empirical foundation; the added Phi studies examine the same geometric questions across additional settings.

  • Section 5.2 adds ARC-Challenge as a second dataset, acquisition with refitting, and predictive comparisons at posterior dimensions 64, 256 and 1024. Both uncertainty-based acquisition rules achieve lower final NLL than random selection in all five runs.
  • Section 5.3 and Table 2 add full-vocabulary, multi-token evaluation under the original 64-dimensional Phi posterior. On the primary targets, Fisher coordinates reduce the mean error in sequence log-probability variance against HMC by 24.5% relative to native coordinates.

These additions extend the evidence from the original covariance audit to acquisition decisions, larger posterior dimensions and autoregressive uncertainty. Could you please let us know whether they address your request to broaden the experimental scope? We would be happy to clarify any aspect of the setup or results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of revision

We thank the three reviewers for their careful reading and constructive questions.

The central message of the submitted paper is unchanged: posterior covariance geometry can substantially affect predictive uncertainty even when predictive probabilities and standard scores barely change. The nonlinear trace law, covariance-response analysis, and original controlled Qwen3-4B and Phi-4-mini experiments remain the foundation of the paper. The revision clarifies this argument and addresses the reviewers' questions about its motivation, assumptions and empirical scope.

Reading guide to the revision

  1. Understanding the mechanism — Section 3. A two-coordinate example introduces the distinction between learned and unobserved parameter combinations before the formal results. Exact Bayes can make the learned sum precise while retaining uncertainty in the unobserved difference; a Gaussian diagonal in the original coordinates cannot separate these variances. This illustrates the mechanism quantified by the nonlinear trace law.

  2. Why the geometry matters for a decision — Section 4.1, Theorem 4.4 and Figure 1. In a logistic example, exact Bayes and mean field have identical predictive means, NLL and Brier scores, yet eventually choose different queries. The exact-posterior regret of the mean-field choice converges to a positive constant. This gives a concrete decision consequence of the paper's original distinction between uncertainty fidelity and predictive scores.

  3. What freezing an adapter factor excludes — Section 2 and Appendix A.2. A conditional-versus-joint variance decomposition identifies the uncertainty omitted by freezing a factor. An explicit example shows conditional predictive variance vanishing while joint predictive variance remains positive.

  4. Broader empirical evidence — Section 5.2. ARC-Challenge adds a second dataset, acquisition with refitting at p=64p=64, and predictive comparisons at p=64,256,1024p=64,256,1024. Both uncertainty-based acquisition rules achieve lower final NLL than random selection in all five runs; mean final NLL is essentially the same for Fisher and native coordinates. These experiments examine how the original geometric findings relate to decisions and predictive performance.

  5. Autoregressive uncertainty — Section 5.3 and Table 2. Full-vocabulary, multi-token evaluation on 32 ARC-Challenge prompts under the original p=64p=64 Phi answer-choice posterior finds 24.5% lower mean sequence-variance error against Hamiltonian Monte Carlo (HMC) for Fisher coordinates on the primary greedy targets. Sequence-probability metrics are mixed, again separating uncertainty fidelity from predictive-score performance. Appendix A.7 extends the covariance-response identities to fixed finite horizons.

Revision markings and organization. The main text remains within ten pages. Blue marks substantive additions and corrections relative to the submission; wording changes and rearrangements are unmarked. Full proofs and numerical details are in the appendices. Section, figure and table references refer to the revised manuscript; page and line numbers quoted from the reviews refer to the submission. The individual responses below address each question in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.