acceptodds
Under review as a conference paper at ICLR 2027

FB-LoRA: Divergence-Free Gram Transport for Joint Bayesian Low-Rank Adaptation

Abstract

Low-rank adaptation enables parameter-efficient specialization of pretrained language models, yet reliable deployment requires calibrated uncertainty over the adapted weights. Directly applying Euclidean Langevin dynamics to low-rank factors injects isotropic noise, which disproportionately perturbs uninformative directions in the effective-weight space and degrades pretrained representations. To address this geometry mismatch, we introduce FB-LoRA, a Bayesian framework grounded in divergence-free Gram transport. We specify a proper generalized posterior over joint factor coordinates using a Gaussian prior and a bounded scoring loss. By preconditioning each factor’s drift and diffusion with a bounded spectral function of the opposite factor’s Gram matrix, the coordinate divergence of the mobility operator vanishes identically everywhere, eliminating state-dependent drift corrections without numerical approximations. This construction establishes a scale-invariant relative spectral allocation that concentrates stochastic exploration on dominant singular modes while guaranteeing localized finite-horizonconvergence. Across language understanding, mathematical reasoning, and safety benchmarks on LLaMA-3-8B, the resulting posterior ensemble maintains foundational capabilities and substantially suppresses nullspace jailbreak susceptibility. Furthermore, empirical predictive disagreement enables forward-only selective prediction and active information acquisition without test-time backpropagation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian LoRA, Langevin dynamics, Gram transport, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DPLNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FB-LoRA, a method for Bayesian sampling of LoRA factors.

Critical strengths
  1. The structure is simple. Using “dependent factors” to set div M = 0 is a clean design. It avoids costly drift corrections, and the computation involves only r×r matrices, resulting in minimal overhead
Critical weaknesses
  1. The Identity method achieved an ECE of 21.65 in Table 1 and an AURC×100 of 21.65 in Table 2; these are two different metrics but the same value. 2The ASR values for all baselines fall within the range of 74–78% but the method described in this paper achieved 14% only; Can authors provide the raw logs or code for reproducibility?
  2. There is a 62% difference in ASR between Dynamic Diagonal Gram and Dynamic Gram. What is the source of this discrepancy?
  3. Baseline is not fair enought
  4. The safety result contradicts Theorem 2
  5. please carefully check all numbers described in the paper;
Questions for authors

see above

Reviewer_J6ZAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FB-LoRA, a Langevin sampling method for LoRA. The method uses opposite-factor Gram matrices to define state-dependent mobilities for the factors in W=W0+BAW=W_0+BA. This construction has zero coordinate divergence and is designed to sample the joint generalized posterior over the LoRA factors. The paper analyzes the continuous-time invariant distribution and effective-weight covariance, and evaluates posterior ensembles through low-dimensional and LLM experiments. The reported results include improvements in performance retention, calibration, and safety-related metrics.

Critical strengths
  • The mobility construction makes direct use of LoRA’s bilinear structure. Using B⊤BB^\top B for the AA update and AA⊤AA^\top for the BB update allows state-dependent mobility while keeping the coordinate divergence zero.

  • The theoretical analysis helps explain how the proposed dynamics behave. It establishes the continuous-time invariant distribution under the stated conditions and characterizes the local covariance of the effective update X=BAX=BA. The result for balanced factors also explains the relative attenuation of smaller singular modes.

  • The LLaMA-3-8B results are promising, with improvements in MMLU retention, GSM8K, calibration, selective prediction, and safety-related metrics. The additional sampling cost is also relatively small in the reported setting.

Critical weaknesses
  • The comparison with Identity SGLD needs further examination. Its MMLU accuracy is 42.14%, compared with 65.42% for the frozen backbone. The common η=5×10−4\eta=5\times10^{-4} and MAP-based scale normalization provide a useful controlled comparison, but the same step size may not be suitable for every mobility. Although low-dimensional step-size sweeps are provided, it is unclear whether this large LLM performance gap persists after sampler-specific tuning.

  • The connection between the LLM gains and posterior approximation accuracy remains unclear. The quadrature and predictive ESS/s results are useful, but agreement in low-dimensional experiments does not establish accuracy in the LLM setting. ESS measures sampling efficiency for particular quantities and does not, by itself, confirm that the target has been adequately explored (Vehtari et al., 2021).

  • Continuous-time invariance also does not establish that the experimental step size and chain length are sufficient. The paper acknowledges that the bound involving λe−β\lambda e^{-\beta} is not a practical burn-in prescription; this does not imply slow mixing in practice. Fixed-step SGLD can also retain asymptotic bias (Vollmer et al., 2016). Different mobilities may have different stationary approximations, so longer chains alone may not resolve the differences. Further checks are needed to connect the LLM gains to reduced approximation error.

  • The safety explanation needs more direct support. The mobility ablation shows an effect on ASR, but it does not establish spectral attenuation as the cause. The covariance analysis does not show that the attenuated singular directions correspond to safety-related directions. The nullspace projection used for the posterior ensemble is also not fully clear, since each sampled adapter defines a different effective update.

  • The choice of f(s)=ϵ+s1+s/cf(s)=\epsilon+\frac{s}{1+s/c} has clear mathematical motivations: the positive floor and saturation ensure uniform ellipticity and boundedness. However, its advantage for sampling is less clear. It would help to separate the effects of the opposite-factor Gram structure, the increasing spectral function, and the particular (ϵ,c)(\epsilon,c) settings. Ablations could address this concern without requiring a general optimality theorem.

References

  • Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: An improved R^\widehat{R} for assessing convergence of MCMC (with discussion). Bayesian Analysis, 16(2), 667–718.

  • Vollmer, S. J., Zygalakis, K. C., & Teh, Y. W. (2016). Exploration of the (non-) asymptotic bias and variance of stochastic gradient Langevin dynamics. Journal of Machine Learning Research, 17(159), 1-48.

Questions for authors
  • It would be useful to compare separately tuned samplers under the same wall-clock budget, reporting MMLU, GSM8K, calibration, and effective-weight perturbations. For step-size studies, the diffusion time T=KηT=K\eta should also be considered: reducing η\eta while keeping KK fixed shortens exploration time as well.

  • Could the reference experiments include the distribution of BABA, second moments, and predictive disagreement? Reporting estimation error and Monte Carlo uncertainty alongside ESS/s would help assess whether improved efficiency is accompanied by more accurate posterior estimates.

  • It would help to examine how the LLM results change with longer chains, different initializations, and smaller step sizes. If exploration near MAP is a possible explanation, useful trajectory measurements would include
    ∣θt−θMAP∣2,∣BtAt−B^A^∣F| \theta_t - \theta_{\mathrm{MAP}} |_2, \qquad |B_t A_t - \hat{B} \hat{A} |_F

where B^=BMAP\hat{B}=B_{\mathrm{MAP}} and A^=AMAP\hat{A}=A_{\mathrm{MAP}}. These distances describe the extent of exploration, but do not establish its causal role in the gains. The statement that the method "reliably samples the intended target distribution" should also distinguish the evidence from low-dimensional experiments and from LLM experiments.

  • How sensitive are the results to (ϵ,c)(\epsilon,c) and the spectral function? Comparisons with other bounded increasing functions, while controlling the overall mobility scale, would help show which parts of the design are necessary for the reported gains.

  • How is the nullspace projection defined for the ensemble: using one adapter, separate projections for each sample, or a shared subspace? An analysis of the relationship between attenuated singular directions and safety-related directions would also be helpful. Without such evidence, the causal interpretation should be more qualified.

Reviewer_WPY5OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FB-LoRA, a Bayesian low-rank adaptation framework grounded in divergence-free Gram transport to address the geometry mismatch between factor and weight space. It defines a generalized posterior over joint factor coordinates using a Gaussian prior and a bounded scoring loss. By preconditioning each factor’s drift and diffusion with the opposite factor’s Gram matrix, the mobility operator has zero coordinate divergence, preserving the target posterior without state-dependent drift corrections. The authors characterize effective-weight covariance and establish scale-invariant relative spectral allocation at balanced factor states, prioritizing dominant singular modes. Following a MAP warm start, stochastic sampling generates adapter ensembles for Bayesian model averaging, selective prediction, and active information acquisition. Experiments on controlled benchmarks and LLaMA-3-8B demonstrate improved calibration, capability retention, and robustness against nullspace jailbreaks.

Critical strengths
  1. The opposite-factor Gram preconditioning provides an elegant way to build a divergence-free mobility operator, avoiding explicit drift corrections while preserving the target posterior in continuous time.

  2. The paper derives the effective-weight covariance operator, connecting factor-space dynamics to perturbations in the observable weight space. The tangent/normal-space interpretation and spectral analysis at balanced factor states tells how preconditioning shapes stochastic exploration.

  3. The paper establishes results on posterior invariance, entropy dissipation, effective-weight covariance, and conditional finite-horizon discretization errors with assumptions for the main theoretical guarantees.

  4. The preconditioning is computationally manageable, relying on small r x r Gram matrix. The experiments compare deterministic and Bayesian LoRA baselines and include controlled numerical validation and fixed-target mobility ablations to examine the effects of different preconditioning strategies.

Critical weaknesses
  1. The theory establishes posterior invariance in continuous time, but the LLM experiments use 450 discrete sampling steps and 5 retained adapters. I am not sure how well these samples approximate the target posterior. The choice of β = 100 and a unit Gaussian prior also seems worth discussing, given the high-dimensional factor space. Some sensitivity analysis on chain length and step size would help.

  2. The spectral allocation analysis assumes balanced factors, but I could not tell whether this condition approximately holds during sampling. Also, the mobility function saturates for large Gram eigenvalues. It would be helpful to report the Gram spectra and factor imbalance along the sampling trajectory. I would also be interested in seeing the bounded-inverse mobility included in the main LLM ablation.

  3. The improvement in attack success rate is quite large, but the underlying mechanism remains unclear. I found the nullspace projection for the adapter ensemble somewhat unclear, since different adapters may have different subspaces. The attack setup for the frozen backbone could also be clarified. Would the results remain similar under different attack budgets or other attack settings?

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.