- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies validation-selection overfitting in LoRA model merging and proposes BLoM, which performs posterior inference over a merge operator in a lossless shared core space. It refines a deterministic merge using labeled held-out data and constructs a full-covariance Gaussian approximation based on the empirical Fisher at one representative layer. The resulting posterior supports a MAP-based mean predictor, Bayesian model averaging, selective prediction, and PAC-Bayes analysis, and the method is evaluated across several language and vision benchmarks.
Critical strengths
- The paper identifies a relevant practical problem: selecting merge configurations using finite validation sets can lead to non-negligible test-time degradation, particularly when task updates interfere.
- The shared core-space formulation makes full-covariance operator inference computationally feasible for relatively large models and provides a unified framework for prediction, uncertainty estimation, and PAC-Bayes analysis. The paper is also generally clear and visually polished, and it explicitly distinguishes the Mean, BMA, and Gibbs predictors.
Critical weaknesses
[w1] Limited Novelty. BLoM combines a previously established lossless LoRA core-space representation with standard techniques such as anchor-regularized MAP estimation, an empirical-Fisher Laplace approximation, posterior sampling, Bayesian model averaging, and PAC-Bayes analysis. Crucially, the headline results come from BLoM-Mean, which does not use the posterior covariance, while BLoM-BMA does not consistently outperform it; thus, the experiments primarily demonstrate the effectiveness of supervised core-space refinement rather than a substantive benefit from the proposed Bayesian treatment.
[w2] Incomplete Ablation Analysis. Specifically, 1) sensitivity to the manually selected representative layer is not examined; and 2) the effect of the operator-fitting data budget on performance and overfitting is not systematically studied.
Questions for authors
[Q1] Could the authors compare BLoM against relevant Mix-of-LoRA approaches, such as LoRAHub-style learned composition or Mixture-of-LoRA methods, as well as the more recent ACE-Merging baseline? These comparisons would help clarify whether BLoM provides meaningful benefits over simpler learned LoRA composition and modern covariance-aware merging methods.
[Q2] Do the conclusions remain valid on more recent vision models, a broader range of vision tasks, and reasoning-intensive settings such as mathematical, or general logical reasoning? Evidence from these settings would materially strengthen the claimed generality of the method.