- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper works on diagnostics for merging LoRA specialists: each candidate merge is scored by a proxy for how far its predictions drift from every task’s specialist, the target being the specialist-relative KL divergence on task contexts, and the candidate with the smallest worst-task score is selected from a fixed library. In practice, merges are built by combining low-rank factors directly, masking or rescaling coordinates, or recompressing to the adapter rank, so the update that is actually deployed can differ from the weighted sum of specialist updates. The authors argue that the diagnostic can therefore fail at three distinct stages (scoring the wrong update, estimating the wrong curvature, and targeting the wrong endpoint) and that predicting the KL magnitude accurately is different from identifying its minimizer. In order to separate these errors, they decompose the deployment residual of factor-space merging into a scalar-equivalent rescaling, a cross-factor covariance term, and a separate rank-truncation term, and introduce a specialist-probability-weighted logit-drift score that isolates output-space geometry from network linearization. They show that normalizing summed token scores by the probe count removes the aggregation bias of sequence-balanced Fisher estimates but leaves a cross-probe variance that does not vanish with more probes, that label marginalization removes label-sampling variance, and that the remaining KL error splits exactly into a linearization term and a logit-path curvature term. The experiments cover one Pythia-1.4B bank of 5 specialists and controlled classifier and causal-attention models with 20 adapter banks each. Specialist weighting lowers worst-task KL regret relative to uniform logit drift, performing comparably to a marginalized Fisher quadratic, but does not improve native accuracy.
Critical strengths
- The experiments are rigorous. Geometry, selection, and endpoint samples are drawn independently, so every selector is judged on data it never saw. Uncertainty is quantified with paired bootstraps over 20 independently trained banks, and the native accuracy is treated as a co-primary endpoint bounded by an oracle.
- The theory is correct and quantitatively matches the experiments. The derivations of the factor-covariance identity, the sequence-balanced Fisher estimators, and the finite-logit KL decomposition are complete with explicitly stated assumptions. Moreover, the predicted plateau of the corrected summed-score variance, , agrees closely with the measured values (2.73, 2.15, 2.02 at ).
- The controlled experiments are fully reproducible from the appendix. Seeds, draw orders, numerical precision, and per-bank selection records are all specified, so every reported number can in principle be regenerated.
- The paper separates predicting KL magnitude from identifying its minimizer with mechanistic evidence. The covariance-interpolation sweep changes selections progressively on identical logits, and the exact remainder decomposition shows that a score can select a good candidate while still miscalibrating KL by a large margin.
Critical weaknesses
- The headline regret reduction appears to reflect a preference for shrinkage rather than per-bank discrimination. In panel B, the weighted score selects in all 20 banks, so a fixed “always deploy ” policy attains the same regret (0.0238). Since the minimax KL target rewards attenuation and only the ray and KnOTS candidates have a scale grid, any metric that favors smaller updates would win this comparison. Reporting the best fixed-candidate baseline and repeating the comparison with the half-scale candidates removed would show whether output geometry itself matters.
- The real-model experiment does not test the paper’s main selection and utility claims. The Pythia experiment supports only the deployment- and local-miscalibration findings, and with the single bank, point estimates, no independent endpoint, and near-floor specialists, it cannot support the selection conclusions either.
- The related-work discussion omits the federated LoRA literature, where the factor mismatch has been studied [1,2,3]. Citing only a concurrent federated work leaves the impression that the mismatch is recent; the deployment decomposition should instead be positioned as a generalization of these results.
[1] Improving LoRA in Privacy-preserving Federated Learning. Sun et al., ICLR 2024.
[2] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. Wang et al., NeurIPS 2024.
[3] FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models. Singhal et al., ACL 2025.
Questions for authors
- Does the weighted score outperform the best fixed candidate? In panel B, it selects in every bank, so please report a fixed-candidate baseline chosen by leave-one-bank-out in both panels B and C. Could the author repeat the selector comparison after removing the half-scale candidates, or restricting the library to ? If the advantage over uniform drift largely disappears, the gain would be attributable to a preference for shrinkage rather than output geometry.
- Does scoring the nominal update instead of the deployed update indeed lead to worse selections? The work quantifies the deployment residual but never shows its decision-level consequence. Could the authors run the same selectors on and on over the existing library to directly test the recommendation that diagnostics should reconstruct deployed products?
- Does the preservation–utility conflict persist when the specialists approach their teachers? In Panel B, the specialists reach only about 52% accuracy, even though each teacher is itself a rank-16 update the specialists could in principle recover. Training them longer and tracking whether KL-selected merges still fail to improve accuracy would show whether the conflict is general or specific to weak specialists.