acceptodds
Withdrawn from ICLR 2027

Do LoRA Merge Diagnostics Score the Deployed Model?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

A LoRA merge diagnostic can fail at three distinct stages: the update it scores, the curvature it estimates, and the endpoint it is intended to predict. We analyze these mismatches while holding candidate construction and evaluation roles explicit. An exact factor-covariance identity separates factor aggregation from scalar-equivalent correction and rank truncation. For sequence-balanced Fisher estimation, probe-count correction removes aggregation bias but retains cross-probe variance; categorical marginalization removes label-sampling variance. A five-adapter Pythia-1.4B case study exhibits a deployed-to-reference norm ratio and task-dependent local-quadratic miscalibration. Controlled studies use twenty independently trained banks per panel, two architectures, and a fixed twenty-four-candidate library. Replacing uniform logit drift with specialist-probability weighting lowers mean worst-task KL regret from to in a nonlinear classifier and from to in causal attention. This preservation gain does not establish a native-accuracy gain: weighted-minus-uniform attention accuracy is percentage points, with paired interval . Selection for native retention using selection labels improves accuracy over KL selection by percentage points . The results support matching diagnostics to deployed updates and preservation geometry, while evaluating native utility separately.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, LoRA, parameter-efficient fine-tuning, Fisher information, task arithmetic, mergeability prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Qj1EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper works on diagnostics for merging LoRA specialists: each candidate merge is scored by a proxy for how far its predictions drift from every task’s specialist, the target being the specialist-relative KL divergence on task contexts, and the candidate with the smallest worst-task score is selected from a fixed library. In practice, merges are built by combining low-rank factors directly, masking or rescaling coordinates, or recompressing to the adapter rank, so the update that is actually deployed can differ from the weighted sum of specialist updates. The authors argue that the diagnostic can therefore fail at three distinct stages (scoring the wrong update, estimating the wrong curvature, and targeting the wrong endpoint) and that predicting the KL magnitude accurately is different from identifying its minimizer. In order to separate these errors, they decompose the deployment residual of factor-space merging into a scalar-equivalent rescaling, a cross-factor covariance term, and a separate rank-truncation term, and introduce a specialist-probability-weighted logit-drift score that isolates output-space geometry from network linearization. They show that normalizing summed token scores by the probe count removes the aggregation bias of sequence-balanced Fisher estimates but leaves a cross-probe variance that does not vanish with more probes, that label marginalization removes label-sampling variance, and that the remaining KL error splits exactly into a linearization term and a logit-path curvature term. The experiments cover one Pythia-1.4B bank of 5 specialists and controlled classifier and causal-attention models with 20 adapter banks each. Specialist weighting lowers worst-task KL regret relative to uniform logit drift, performing comparably to a marginalized Fisher quadratic, but does not improve native accuracy.

Critical strengths
  1. The experiments are rigorous. Geometry, selection, and endpoint samples are drawn independently, so every selector is judged on data it never saw. Uncertainty is quantified with paired bootstraps over 20 independently trained banks, and the native accuracy is treated as a co-primary endpoint bounded by an oracle.
  2. The theory is correct and quantitatively matches the experiments. The derivations of the factor-covariance identity, the sequence-balanced Fisher estimators, and the finite-logit KL decomposition are complete with explicitly stated assumptions. Moreover, the predicted plateau of the corrected summed-score variance, κ/m+2−3/m\kappa/m + 2 - 3/m, agrees closely with the measured values (2.73, 2.15, 2.02 at m=8,32,128m = 8, 32, 128).
  3. The controlled experiments are fully reproducible from the appendix. Seeds, draw orders, numerical precision, and per-bank selection records are all specified, so every reported number can in principle be regenerated.
  4. The paper separates predicting KL magnitude from identifying its minimizer with mechanistic evidence. The covariance-interpolation sweep changes selections progressively on identical logits, and the exact remainder decomposition shows that a score can select a good candidate while still miscalibrating KL by a large margin.
Critical weaknesses
  1. The headline regret reduction appears to reflect a preference for shrinkage rather than per-bank discrimination. In panel B, the weighted score selects 0.5F0.5F in all 20 banks, so a fixed “always deploy 0.5F0.5F” policy attains the same regret (0.0238). Since the minimax KL target rewards attenuation and only the ray and KnOTS candidates have a scale grid, any metric that favors smaller updates would win this comparison. Reporting the best fixed-candidate baseline and repeating the comparison with the half-scale candidates removed would show whether output geometry itself matters.
  2. The real-model experiment does not test the paper’s main selection and utility claims. The Pythia experiment supports only the deployment- and local-miscalibration findings, and with the single bank, point estimates, no independent endpoint, and near-floor specialists, it cannot support the selection conclusions either.
  3. The related-work discussion omits the federated LoRA literature, where the factor mismatch has been studied [1,2,3]. Citing only a concurrent federated work leaves the impression that the mismatch is recent; the deployment decomposition should instead be positioned as a generalization of these results.

[1] Improving LoRA in Privacy-preserving Federated Learning. Sun et al., ICLR 2024.

[2] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. Wang et al., NeurIPS 2024.

[3] FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models. Singhal et al., ACL 2025.

Questions for authors
  1. Does the weighted score outperform the best fixed candidate? In panel B, it selects 0.5F0.5F in every bank, so please report a fixed-candidate baseline chosen by leave-one-bank-out in both panels B and C. Could the author repeat the selector comparison after removing the half-scale candidates, or restricting the library to γ=1\gamma = 1? If the advantage over uniform drift largely disappears, the gain would be attributable to a preference for shrinkage rather than output geometry.
  2. Does scoring the nominal update UαU\alpha instead of the deployed update vv indeed lead to worse selections? The work quantifies the deployment residual but never shows its decision-level consequence. Could the authors run the same selectors on UαU\alpha and on vv over the existing library to directly test the recommendation that diagnostics should reconstruct deployed products?
  3. Does the preservation–utility conflict persist when the specialists approach their teachers? In Panel B, the specialists reach only about 52% accuracy, even though each teacher is itself a rank-16 update the specialists could in principle recover. Training them longer and tracking whether KL-selected merges still fail to improve accuracy would show whether the conflict is general or specific to weak specialists.
Reviewer_NzUjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to understand how to select the best lora merging method before deploying the model.

Critical strengths

None

Critical weaknesses
  1. The paper is hard to read and the writing style is extremely convoluted.

  2. The practical significance of the method is not clear and nor the paper provides any comparison.

  3. The paper is mostly an analytically study looking into the effect of merging in section 3 and sec 4.

  4. It is not clear if any of the analytical tool developed in sec 3 and sec 4 can help us select the best merging methods for the downstream task.

Questions for authors
  1. what is the practical use case of the method?

  2. can you use the insights in sec 3 and sec 4 to propose a new lora merging method?

Reviewer_8SAbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduce these diagnostics that aim to pick a merged LoRA update that best matches individual expert LoRA updates. It then proceeds to claim that these diagnostics fail to do so.

Critical strengths

None

Critical weaknesses
  • The problem setting is poorly introduced and motivated
  • It's unclear why the paper focus on the KL metric in equation 7. Why not downstream accuracy?
  • Also unclear why the specific diagnostics that the paper introduced are worth considering.
  • This appears an empirical paper. But the formalizations in the main paper neither fits the empirical framing nor introduces the problem setting well. Overall, the writing choice is puzzling and confusing.
Questions for authors
  • Why is the KL metric in equation 7 the focus of the paper?
  • Why does the paper considered only the diagnostic introduced by the authors. Are there no other related work that aims to predict LoRA merging performances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.