acceptodds
Under review as a conference paper at ICLR 2027

BLoM: Bayesian Inference over Merge Operators for LoRA Model Merging

Abstract

Merging low-rank adaptation (LoRA) adapters efficiently combines task-specific capabilities, but common practice selects a discrete merge configuration specifying coefficients, calibration strengths, and ranks, using validation accuracy. Such selection can overfit when task updates interfere. We quantify this merge-selection gap and observe losses up to 5.6 accuracy points on a high-interference task pair. BLoM replaces selection among discrete configurations with posterior inference over the continuous merge operator that each configuration induces. A lossless core-space projection makes this operator compact and enables a full-covariance Gaussian approximation within a representative layer of models with up to 8 billion parameters. The posterior supports a regularized mean predictor, Bayesian model averaging (BMA), confidence-based abstention, and PAC-Bayes analysis of Gibbs risk, the expected loss of a posterior-sampled operator. Across five standard settings spanning three language models and two vision transformers, BLoM-Mean improves the NLI suite average by 6.9 points over the strongest of twelve baselines, including task arithmetic, TIES, DARE-TIES, RegMean, aligned-subspace, and LoRA-specific methods. In cross-domain merging, it gains 13.9 points over the strongest of three evaluated baselines, task arithmetic, TIES, and TSV-M. On the pilot pair, BLoM-Mean improves validation-selected TSV-M by 4.8 points and exceeds its finite-grid test oracle. Nominal balanced-design risk curves remain below one across nine verified ranks. These results recast LoRA merging as inference over plausible merge operators.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Inference, LoRA, Model Merging, Overfitting, Core Space, Low-Rank

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4ey9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies validation-selection overfitting in LoRA model merging and proposes BLoM, which performs posterior inference over a merge operator in a lossless shared core space. It refines a deterministic merge using labeled held-out data and constructs a full-covariance Gaussian approximation based on the empirical Fisher at one representative layer. The resulting posterior supports a MAP-based mean predictor, Bayesian model averaging, selective prediction, and PAC-Bayes analysis, and the method is evaluated across several language and vision benchmarks.

Critical strengths
  1. The paper identifies a relevant practical problem: selecting merge configurations using finite validation sets can lead to non-negligible test-time degradation, particularly when task updates interfere.
  2. The shared core-space formulation makes full-covariance operator inference computationally feasible for relatively large models and provides a unified framework for prediction, uncertainty estimation, and PAC-Bayes analysis. The paper is also generally clear and visually polished, and it explicitly distinguishes the Mean, BMA, and Gibbs predictors.
Critical weaknesses

[w1] Limited Novelty. BLoM combines a previously established lossless LoRA core-space representation with standard techniques such as anchor-regularized MAP estimation, an empirical-Fisher Laplace approximation, posterior sampling, Bayesian model averaging, and PAC-Bayes analysis. Crucially, the headline results come from BLoM-Mean, which does not use the posterior covariance, while BLoM-BMA does not consistently outperform it; thus, the experiments primarily demonstrate the effectiveness of supervised core-space refinement rather than a substantive benefit from the proposed Bayesian treatment.

[w2] Incomplete Ablation Analysis. Specifically, 1) sensitivity to the manually selected representative layer is not examined; and 2) the effect of the operator-fitting data budget on performance and overfitting is not systematically studied.

Questions for authors

[Q1] Could the authors compare BLoM against relevant Mix-of-LoRA approaches, such as LoRAHub-style learned composition or Mixture-of-LoRA methods, as well as the more recent ACE-Merging baseline? These comparisons would help clarify whether BLoM provides meaningful benefits over simpler learned LoRA composition and modern covariance-aware merging methods.

[Q2] Do the conclusions remain valid on more recent vision models, a broader range of vision tasks, and reasoning-intensive settings such as mathematical, or general logical reasoning? Evidence from these settings would materially strengthen the claimed generality of the method.

Reviewer_pmAUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper titled “BLOM: BAYESIAN INFERENCE OVER MERGEOPERATORS FOR LORA MODEL MERGING” address the problem of a fixed merged model whose configuration is generally selected by the tuning on validation data may overfit when task updates interfere. The authors introduce a novel merging method, BLoM, which replaces discrete selection with posterior inference over a continuous merge operator. The study focuses on LoRA-finetuned models and evaluates across a mix of vision and language benchmarks, significantly improving performance on the NLI setting.

Critical strengths

[1] The paper attempts to address the problem of overfitting to the validation split, which is an understudied problem in the model merging space.

[2] The method is evaluated across both vision and language benchmarks while evaluating over LLMs.

Critical weaknesses

[1] The performance gains are inconsistent across settings and require further analysis: BLoM underperforms the strongest baselines on ViT-B/32, ViT-L/14, and Finance-7B, while achieving comparable performance on Finance-1.5B. In contrast, it substantially outperforms both the merging baselines and the average individual-expert accuracy on NLI.

[2] Discrepancies in the NLI evaluation: Table 11 reports individual-expert accuracies of 66.0% on QNLI and 49.4% on RTE, compared with 94.49% and 89.86% in the Core Space paper [a] using the KnOTS [b] checkpoints and setup. These discrepancies are particularly concerning because these tasks account for essentially all of BLoM’s net NLI improvement over RegMean. The merging baselines also need to be reconciled with the original KnOTS results. KnOTS reports normalized accuracies of 90.2%, 90.0%, and 92.9% for TA, TIES, and KnOTS-TIES, respectively, while this study reports raw accuracies of 74.6%, 74.0%, and 74.6%. Until the expert-accuracy discrepancies and evaluation differences are explained, it is difficult to assess the reliability of the claimed NLI gains.

[3] The experimental coverage and validation-budget analysis are limited: Using at most 200 validation examples per task emphasizes a data-constrained regime, leaving unclear whether the reported merge-selection overfitting remains substantial with larger validation sets. Evaluating larger budgets would help establish the practical relevance of the motivation. Furthermore, the cross-domain benchmark overlaps substantially with NLI, reusing six of its eight tasks, and therefore provides limited independent evidence of generalization. The NLI suite itself primarily combines experts trained on closely related inference tasks across different datasets. While relevant, this offers limited evidence for merging experts with distinct capabilities. Additionally, the work does not evaluate the method across generative tasks, which would be of great interest.

References:

[a] Panariello, Aniello, et al. "Accurate and efficient low-rank model merging in core space." Advances in Neural Information Processing Systems 38 (2026): 61793-61825.

[b] Stoica, George, et al. "Model merging with svd to tie the knots." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

[1] Why is the crossdomain evaluation in Table-2 evaluated only across 3 baselines skipping evaluation across all the other baselines listed.

[2] L227 states “The center uses TSV-Core for language models and uniform core averaging for vision, with all other projections fixed at that merge”, why was this choice made? Why does the method differ depending on modality?

Reviewer_iwZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BLoM, a Bayesian framework for LoRA model merging. It first projects LoRA updates into a shared core space, optimizes the merge operator using a small amount of labeled data, and then constructs a Gaussian posterior using the empirical Fisher information. The paper motivates this approach through merge-selection overfitting, where selecting the best merging configuration on a validation set may lead to poor generalization. Experiments on language and vision tasks show promising results in some settings.

Critical strengths

The paper identifies an interesting problem in model merging. Merge-selection overfitting is a reasonable concern when many merging configurations are compared using a limited validation set.

The core-space representation provides a practical way to optimize merge operators without directly updating the full model parameters.

The paper attempts to connect model merging with Bayesian inference, providing uncertainty estimation, Bayesian model averaging, and PAC-Bayes analysis within a unified framework.

Critical weaknesses

The connection between the motivation and the proposed method is not sufficiently established. The paper identifies merge-selection overfitting as a problem, but it does not convincingly demonstrate that Bayesian inference is necessary to address it. Optimizing the merge operator in core space is a reasonable approach, but it is not clear how this specifically reduces selection overfitting rather than simply finding a better solution through additional supervised optimization. Moreover, optimizing a continuous operator can itself introduce overfitting.

The contribution of Bayesian inference remains unclear. The main accuracy results rely on BLoM-Mean, which uses the optimized posterior mean. Its gains may largely come from supervised MAP optimization with L2 regularization. The paper needs stronger comparisons with standard gradient-based optimization under the same initialization, data budget, objective, and number of update steps to isolate the contribution of Bayesian inference. The benefits of posterior sampling and Bayesian model averaging are also not consistently demonstrated.

The experimental comparison may not be sufficiently fair. BLoM optimizes the merge operator using labeled data, whereas conventional merging baselines mainly select from predefined configurations. These approaches have different optimization capabilities and use supervision differently. The improvements therefore cannot be attributed solely to the proposed Bayesian framework without stronger controls.

The evaluation of merge-selection overfitting is limited. The gap between the validation-selected configuration and the best configuration on the test set is useful as a diagnostic, but the latter is an oracle selected using test labels and is not an achievable deployment baseline. More importantly, the paper should demonstrate more directly that BLoM reduces the sensitivity of model selection to limited validation data, rather than merely outperforming a fixed set of candidate merging configurations.

The empirical benefits are not consistent across settings. Although BLoM performs well on some NLI and cross-domain experiments, it does not consistently outperform strong baselines on the finance and vision tasks. The paper should better characterize when the method works and whether its benefits are related to task interference, validation-set size, or other factors.

Questions for authors

Can the authors demonstrate that BLoM specifically mitigates merge-selection overfitting? For example, how does its performance and stability change as the validation set size and the number of candidate merging configurations vary?

How much of the improvement comes from Bayesian inference, as opposed to ordinary supervised optimization of the merge operator? Can the authors compare BLoM-Mean with standard gradient optimization and L2-regularized optimization using exactly the same data, initialization, objective, and optimization budget?

Can the authors provide a fair comparison in which conventional merging methods and BLoM have access to the same labeled data budget? This would help distinguish the benefit of the proposed framework from the benefit of additional supervised optimization.

What practical benefits does BLoM-BMA provide over BLoM-Mean? Since Bayesian model averaging does not consistently improve accuracy, can the authors demonstrate benefits in calibration, uncertainty estimation, or selective prediction?

The PAC-Bayes analysis concerns the Gibbs predictor, whereas the main deployed prediction modes are BLoM-Mean and BLoM-BMA. Can the authors clarify the practical implications of this theoretical result for the predictors used in the main experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.