- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper analyze Bayesian low-rank adaptation. The author argue that factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement. Experiment on Qwen3-4B/Phi-4- mini show full covariance Gaussian better mimic HMC compared to MF Gaussian VI.
Critical strengths
The uncertainty quantification in large language model is important.
Critical weaknesses
-
The problem lack clear motivation. Could the author enumerate a specific application of Bayesian low-rank adaptation?
-
There exists some gap between the designed experiment and the theoretical analysis. It would be nice if the author can design some toy experiment visualizing the mechanics shown in the theoretical analysis
-
Isn't the main argument " factorization can inflate uncertainty along informed predictive directions while collapsing it in an uninformed complement. " obvious?
-
sentences with AI pattern which are hard to understand :
- abstract: "All ten matched-seed coordinate comparisons across Qwen and Phi favor the categorical-Fisher eigenbasis"
- Page 2 line 99: "Fixing A removes the two-factor ambiguity in BA from a matched-posterior comparison. The adapter matrix rank, stochastic dimension p, informed dimension m, and represented covariance rank kare all distinct quantities. "
Questions for authors
-
What does it mean by "INFORMED SUBSPACE"?
-
I have no idea what the following sentence means:
-
Page 2 line 101 "We adopt the convention that uncertainty is conditional on the frozen backbone and any frozen adapter factors."
-
Page 2 line 103 "gz(x) denote identifiable logit contrasts." What does it mean by " logit contrast"?
-
Page 3 line 128 "Its null complement is retained exactly by Bayes"
3 comments
Response to Reviewer EUde (2/2): Technical novelty and clarity of exposition
On W3. Is the main argument obvious?
The qualitative observation is known, and the paper says so: in both the submitted and the revised version, the contribution paragraph of the introduction opens by citing it as established (Margossian and Saul, 2023; Margossian et al., 2025; Odgers et al., 2026). What we add is the quantitative nonlinear limit and the predictive and decision statements that follow from it.
Under the fixed-profile likelihood and tail conditions of Assumption 3.1, Theorem 3.2 gives the limiting exact and mean-field informed covariances without assuming a Gaussian posterior; the proof first shows concentration of the optimal diagonal Gaussian and then expands its stationarity equations. Corollary 3.3 gives
where counts the active native coordinates and is the informed dimension. At the same time, a supported null direction keeps exact variance one while its mean-field variance is of order . For a scalar logistic likelihood, the predictive variance inflation is exactly to leading order, for every fixed success fraction (Corollary 3.5). The text after Corollary 3.3 explains that is the average distortion, so individual predictive directions can still be uninflated.
Two further results concern observables and decisions. Theorem 4.2 shows that, under its nondegeneracy condition, a covariance direction can change uncertainty at first order while probability means change only at second order. Theorem 4.4 shows that identical predictive scores can coexist with a wrong information-gain ranking.
Section 5.1 now also states the coordinate comparison precisely. Native and Fisher fits each optimize variances. With a fixed orthogonal Fisher basis , their native-coordinate covariances are and . At both have 64 covariance parameters, against 2080 for an unrestricted covariance. The Fisher fit is generally dense in native coordinates, with eigenvectors fixed by the chosen basis and eigenvalues fitted, and the prior, likelihood, subspace and mean are shared. The comparison therefore isolates the choice of axes for a diagonal approximation.
On Q1. Sentences and terms that were hard to follow.
Informed subspace. These are the parameter combinations on which the likelihood depends. If the likelihood sees only , moving orthogonally to the columns of leaves it unchanged. Section 3 now gives this definition with the sum/difference example from point 2, before the assumptions.
Conditioning on frozen components (p. 2, l. 101). We place a distribution only on designated coordinates of the output factor and hold the pretrained network and the input factor fixed, so the reported uncertainty excludes uncertainty about those fixed quantities. Section 2 now gives the total-variance decomposition and an example in which the data identify the product , the sign stays uncertain, and the conditional variance of the new prediction vanishes while its joint variance stays positive.
Logit contrasts (p. 2, l. 103). Softmax is unchanged when the same number is added to every logit, so we use differences , which are identifiable log probability ratios. Section 2 gives this formula at first use.
"Its null complement is retained exactly by Bayes" (p. 3, l. 128). Write with . The standard Gaussian prior makes and independent, and the likelihood depends only on , so conditioning on the data leaves the law of at the prior. In the two-coordinate example, is the scalar coordinate along this null direction. Section 3 now spells this out.
Abstract sentence on coordinate comparisons. It meant that in each of five paired optimization runs on each backbone, log-odds variance estimates in the Fisher basis have smaller error against HMC than those in native coordinates. The abstract now names the quantity and the reference.
Matrix rank and the dimensions , , (p. 2, l. 99). Section 2 now works through a rank-one adapter: is a column with entries and is a row, so has rank at most one even when every entry of is random. The dimension counts the coordinates we vary, the combinations the likelihood sees, and the directions a structured covariance models explicitly, with prior variance in the complement.
We hope that the clearer exposition, the illustrative example and its proved decision consequence, and the additional theoretical and empirical results address your main concerns. If so, we would be grateful if you would reconsider your assessment and score in light of the revised manuscript. We would also welcome your feedback on any concerns that remain and we remain committed to address those as well to the best of our ability.
Follow-up: Motivation and geometric intuition
Thank you again for your questions on motivation and clarity. The revision develops the same central argument as the submission: covariance geometry can substantially affect predictive uncertainty even when predictive scores barely change. The original nonlinear trace law and covariance-response analysis remain the foundation.
Two places give a focused route through the clarifications:
- Mechanism and clarity — Section 3. The two-coordinate example shows why exact Bayes can make a learned sum precise while retaining uncertainty in an unobserved difference, whereas a Gaussian diagonal in the original coordinates cannot separate these variances. It introduces the mechanism quantified by the nonlinear trace law.
- Concrete application — Section 4.1, Figure 1 and Theorem 4.4. Exact Bayes and mean field have identical predictive means, NLL and Brier scores, yet eventually select different acquisition queries. The exact-posterior regret of the mean-field choice converges to a positive constant.
These additions make the intuition and decision consequence of the original geometric findings explicit. Could you please let us know whether they address your concerns about motivation and clarity? We would be happy to clarify any specific step.
Response to Reviewer EUde (1/2): Motivation and illustrative example
We thank the reviewer for the detailed questions on motivation and clarity. These questions helped us explain the original geometric argument more clearly and make its practical consequence explicit through a two-coordinate example, an acquisition theorem, and targeted experiments.
On W1. Motivation: a specific application of Bayesian low-rank adaptation.
A concrete application is label acquisition. When a language model is adapted under a limited annotation budget, which unlabelled example should be annotated next? Bayesian active learning by disagreement (Houlsby et al., 2011), or BALD, scores a candidate by how much its label would reveal about the parameters, and this depends on how predictions vary across posterior draws. Two approximations can give the same integrated predictive probabilities, and hence the same accuracy and NLL, but different information gains. A second application, already studied in the submitted version, is selective answering, in which the model abstains on inputs with high uncertainty (Appendix C.6).
Section 4.1 makes the acquisition consequence exact. In the logistic model of Theorem 4.4, exact Bayes eventually prefers a query about a direction the data have not informed, while mean field prefers a query about a direction already learned. Their predictive means and scores agree exactly. Even so, the exact-posterior regret of the mean-field choice converges to a positive constant, and no scalar rescaling of the mean-field covariance changes its ranking. Figure 1 shows the mechanism and the reversal.
We then test the application on Phi. On the 300 MMLU-Pro prompts, Fisher coordinates reduce the error of the acquisition values by about 70%, while the top query stays the same. On ARC-Challenge, with actual acquisition and refitting, both uncertainty-based rules give lower final NLL than random selection in all five runs, with no mean final-NLL advantage from Fisher coordinates over native coordinates (Section 5.2).
On W2. Gap between theory and experiments; a toy example visualizing the mechanism.
We agree, and the revision adds such an example, built under the assumptions of the theory. Section 3 now introduces the mechanism with two coordinates before the nonlinear theorem. Under a standard normal prior, put t=(z1+z2)/2 and v=(z1−z2)/2, and let a balanced logistic likelihood depend only on t. Exact Bayes learns t and leaves v at its prior variance. Any diagonal Gaussian with covariance diag(d1,d2) in (z1,z2) has
Varq(t)=Varq(v)=(d1+d2)/2,
so it cannot make the learned sum precise while keeping the unobserved difference uncertain (Equation 4). Exact Bayes does both through dependence between z1 and z2.
Figure 1 visualizes this posterior and the decision theorem built on it. At n=100, the information gains (nats) of the candidate probabilities σ(2t) and σ(v) are:
Exact Bayes selects the unobserved query and mean field the informed one, while every predictive mean equals 1/2, so NLL and Brier scores agree for either realized label. The figure uses deterministic quadrature and the variational stationarity equations; Theorem 4.4 proves the limiting reversal, and the table illustrates it. Table 1 and Appendix C.4 keep the nonlinear support-factor and trace checks, with convergence plots. The transformer experiments then ask whether the same geometric effects appear in fitted posteriors, measured against Hamiltonian Monte Carlo (HMC).