acceptodds
Under review as a conference paper at ICLR 2027

Unmergeable LoRA: Proactively Protecting Adapters from Unauthorized Merging

14 views

Abstract

Public LoRA adapters enable convenient capability sharing, but also allow unauthorized reuse of proprietary capabilities through model merging. Existing proactive defenses for full-model release do not transfer to the fixed-base LoRA setting, leaving released adapters effectively unprotected against unauthorized merging. In this work, we propose LoRA Preservation-Barrier Hardening (LoRA-PBH), a release-time defense that directly protects adapters against post-release merging while preserving their standalone performance. Instead of modifying model representations that become ineffective under LoRA merging, LoRA-PBH optimizes the released adapter to make unauthorized merged models retain little defender performance without requiring changes to the public base model or knowledge of future attacker adapters. Experiments on vision, language, and diffusion models demonstrate that LoRA-PBH consistently preserves standalone performance while substantially degrading merged performance across multiple backbones, merging methods, merge coefficients, and multi-adapter settings. Code is included in the supplementary material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Merging, Low-Rank Adaptation, Lagre Language Model, Stable Diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RHX9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses proactive protection of publicly released LoRA adapters against unauthorised merging, and makes two claims.

First, that proactive defences designed for full-parameter release (PaRaMS, MergeLock, MergeBarrier) do not transfer to the fixed-base LoRA setting. The argument is structural: inside a LoRA factorisation the natural port A'=TA, B'=BT^{-1} leaves the induced update dW=(alpha/r)BA exactly unchanged (Eq. 9), so it is invisible to any merging method that consumes dW; applying the paired maps directly to dW does change the merge but breaks the standalone model, because the frozen base prevents cancellation (Eq. 26). Appendix A adds that MergeBarrier's attention and FFN components are not instantiable at the released rank (Eqs. 20, 22), and that realising a full-weight transformation with a rank-r update would require a base correction of rank at most 2r, while the correction actually induced can have rank d > 2r (A.3).

Second, it proposes LoRA-PBH, a release-time defence posed as a constrained max-min problem (Eq. 12): maximise the worst-case preservation barrier -- the increase in defender-task loss from the protected standalone model to the merged model -- subject to retaining standalone performance. Since the attacker's adapter and merge configuration are unknown, the perturbation set is approximated by probes built from defender-generated surrogate adapters (k-means pseudo-tasks warm-started from the defender adapter), the inner minimum is replaced by a differentiable SoftMin, and drift is limited by a normalised Frobenius anchor. Proposition 1 gives a conditional lower bound under a finite coverage radius and local Lipschitz assumption.

Evaluation covers ViT-B/32, B/16 and L/14 (five datasets, 20 defender-free-rider combinations each), three 7-8B LLM backbones with third-party released adapters, and qualitative SDXL results. On ViT-B/32 under task arithmetic at c=0.5, defender-task merged accuracy falls from 74.0% to 3.0% with standalone reported as +0.5%; protection transfers to six further merging methods, across c in [0.3, 0.8] and for two to five adapters. On LLMs merged accuracy falls 76.7% to 60.2%. A white-box adaptive attack recovers 3.0% to 34.4%. A runnable ViT-B/32 codebase is included.

Critical strengths
  1. The structural diagnosis is correct and the appendix does the harder half of the work. The core identity is elementary, but Appendix A goes further: MergeBarrier's two components are not merely ineffective but not representable at the released rank (Eqs. 20, 22), and A.3 gives a counterexample where the required base correction has rank d > 2r. Figure 3 confirms both routes empirically for all three defences under a controlled fixed-rank budget. This is a checkable negative result and the part of the paper I would most want the community to have.

  2. The protocol is broad and disciplined, and the released code backs it up. Each defender is protected once using defender-side data only, then evaluated against four unseen free-riders; six further merging methods are applied with no method-specific retraining; the coefficient sweep, the two-to-five adapter sweep, the alternative-probe study and the update-shift diagnostic each probe a distinct failure mode. Static inspection of src/method.py indicates agreement with Algorithm 1: the detached TA probe, the two-pass stop-gradient SoftMin weighting, the normalised anchor and the gradient surrogate are all implemented as described.

Critical weaknesses
  1. A formally published, closely related prior work is missing, and with it a key LoRA-native baseline. TRAP2 (Jang et al., "Making Models Unmergeable via Scaling-Sensitive Loss Landscape", ICML 2026, PMLR 306:50690-50717; arXiv:2601.21898, public since January 2026) states the same base-cancellation obstacle that Section 5 analyses -- "applying paired transformations solely to the adapter dW fails to cancel terms associated with the frozen base model W0" -- already uses LoRA variants of PaRaMS and Merge-Lock as baselines, and proposes a training-time adapter-native objective J = L_nominal - lambda * L_off that shapes the loss against off-nominal rescalings. Its two adaptations are not identical to this paper's two routes, and I am not claiming the contributions coincide: the formal rank analysis, the seven-method coverage and the adaptive attack here are additional. But the submission currently compares only against its own adaptations of full-model defences, which it then shows to fail, so no LoRA-native defence appears in the comparison at all. Positioning and experiments both need revisiting.

  2. Two results are framed in a way that omits the costs they carry. (a) Standalone preservation is discussed against only one of two valid baselines. The no-barrier ablation in Figure 5 -- same 100 steps, same data, Ls + Aw only -- reaches 86.4%, against 81.6% for the original adapter and 82.1% for full LoRA-PBH. Relative to the released adapter the method is +0.5, so the claim that it preserves existing utility stands; relative to simply continuing to optimise it is -4.3, which is the utility-versus-protection opportunity cost. Both numbers are in Figure 5; what is missing is any discussion of the difference. (b) The protection is blunter than the formulation suggests. Protected merged accuracies in Table 1 (0.5, 2.1, 9.3, 0.7, 2.5) sit near or below the uniform-chance reference implied by the paper's own class counts (0.51, 2.13, 10.00, 2.33, 2.22; RESISC45 the one case slightly above), and Figure 2 shows protected merged SDXL outputs as noise rather than as images that have lost the defender's style. For a defence meant to deny unauthorised capability retention, reaching chance is not itself a failure. The issue is scope: Table 10 shows the free-rider task also collapsing (74.0% to 11.7%) under TA at c = 0.5, so in the configuration the paper leads with, merging destroys both capabilities. Whether an owner can block unauthorised merging while still permitting sanctioned composition is left unanswered, and that gap belongs in the paper as a scope limitation.

Questions for authors

Items 1-3 would move my score; the rest are secondary.

  1. (Experiment request) Please position the paper against TRAP2 (ICML 2026, PMLR 306:50690-50717; arXiv:2601.21898) and compare on the ViT-B/32 protocol under matched base model, data and standalone-utility constraint. The candidate increment over a pure scaling proxy is the surrogate-task probe, so please make it measurable: LoRA-PBH with its surrogate probe, LoRA-PBH with the surrogate replaced by scalar rescaling of dW_def, and TRAP2, with hardening cost. Please also report the relative norm and cosine similarity between dW_psi and dW_def, since warm-starting does not by itself bound how far the surrogate moves. TRAP2 Section 6.4 evaluates post-merge SFT recovery; how does LoRA-PBH fare there?

  2. (Clarification) Please discuss both standalone baselines rather than one: +0.5 against the original adapter, and -4.3 against the no-barrier ablation of the same length. How should an owner weigh the opportunity cost when deciding whether to harden?

  3. (Experiment request) The coefficient sweep uses a single SHARED coefficient, but task arithmetic does not require this. What happens under asymmetric coefficients, say lambda_def = 1.0 with lambda_atk in {0.1, 0.3, 0.5}? Separately, every task used as private defender data is a public dataset, so a free-rider could fine-tune the merged model on a disjoint split of it. Does the barrier survive either attack, on ViT and on LLMs?

  4. (Clarification) Which configuration produced the reported tables? The released code differs from the paper in three places: Appendix D.1 says 30 epochs per vision adapter while config.py sets 35 / 56 / 22 / 21 / 25; the hardening coefficient is sampled from [0.3, 0.8] in code versus [0.2, 1.0] in the paper; and n_candidates=8 with keep=8 means no barrier filtering, versus R=16 filtered to J=8.

  5. (Clarification) Two framing points. (a) With M=1 the SoftMin reduces to a single barrier and beta has no effect, so the objective optimised is a stochastic estimate of the expected barrier rather than the worst case of Eq. (12). Proposition 1 may well hold on a fixed model and fixed probe set; what it does not do is establish a worst-case guarantee for the resampled-per-step procedure actually run. Would you add an M>1 ablation or restate Section 6.3? Do you have any empirical handle on the coverage radius when the attacker's adapter is unconstrained? (b) On LLMs the protected merged accuracy (60.2%) is below the base model's own (69.2%) on every backbone, which arguably strengthens your case but makes the base model the right reference point; could Table 3 be reframed that way?

  6. (Suggestion, optional) All runs use seed 0 with no variance reported, which matters given that protected standalone accuracy is reported as rising.

6 comments

We sincerely thank the reviewer for the detailed and constructive review. We particularly appreciate the suggestions for strengthening our empirical evaluation and clarifying the scope of our protection mechanism. Following these suggestions, we have conducted additional experiments and revised our presentation. We first address the three questions that the reviewer identified as most important for the evaluation, followed by additional clarifications.

Q1: Positioning against TRAP2 and the requested comparison

We thank the reviewer for pointing us to TRAP2. The revised manuscript cites and discusses TRAP2 in Sections 1 and 2, and adds the comparison below to Section 7.2.1 and Appendix D.7. The primary advantage of LoRA-PBH is its post-hoc applicability: it can protect an already-trained LoRA adapter without modifying or repeating its original training process, whereas TRAP2 incorporates its protection objective during adapter training. This makes LoRA-PBH applicable to existing adapters, including third-party adapters that have already been released.

Beyond this difference in applicability, our protection objectives also differ. TRAP2 makes an adapter sensitive to rescaling its own update, using scaling variations as a proxy for merging. In contrast, LoRA-PBH explicitly models the perturbation introduced by merging with an external adapter. Since the actual merging partner is unknown, we construct defender-generated surrogate adapters to approximate such external updates and directly optimize the preservation barrier under these perturbations. This allows LoRA-PBH to model effects beyond simple rescaling of the defender's own update while preserving standalone utility. LoRA-PBH is therefore not a modification of TRAP2: both methods shape the loss landscape around the released update, but they model different perturbations and act at different stages. TRAP2 notes that paired transformations on the adapter cannot cancel the frozen base; Section 5 and Appendix A formalize this obstacle (Eq. 26 and the rank result of Appendix A.3) and additionally cover the factor-internal route, which leaves ΔW\Delta W unchanged (Eq. 9), as the revised Appendix D.7 now states.

(1) Direct comparison with TRAP2.

We implemented TRAP2 using its official code and hyperparameters (LoRA on attention projections, r=16r=16, with early stopping), evaluating both values in its hyperparameter grid, $\lambda\in\lbrace 0.001,0.01\rbrace $. All adapters start from the same CLIP ViT-B/32 weights and are evaluated using the same pipeline, over the 20 defender–free-rider pairs of Table 1, with Task Arithmetic (TA) at c=0.5c=0.5. The scaling-probe variants replace the surrogate update with a rescaling of the defender update, using the same 100-step hardening budget. Results are averaged over five random seeds, so the 100-step LoRA-PBH row differs slightly from the seed-0 values of Table 1 (82.1, +0.5+0.5, 3.0).

MethodStandalone Acc. (Δ\Delta)Merged Defender Acc.Merged Free-rider Acc.
Unprotected81.674.074.0
TRAP2 (λ=0.001\lambda=0.001)75.7 (−5.9-5.9)4.825.7
TRAP2 (λ=0.01\lambda=0.01)71.6 (−10.0-10.0)3.623.2
LoRA-PBH (Scaling, ΔWψ=0\Delta W_\psi=0)53.2 (−28.4-28.4)3.863.8
LoRA-PBH (Scaling, ΔWψ=ΔWdef\Delta W_\psi=\Delta W_{\mathrm{def}})78.3 (−3.3-3.3)69.770.2
LoRA-PBH (Surrogate, 100 steps)82.2 (+0.6+0.6)3.112.0
LoRA-PBH (Surrogate, 400 steps)84.2 (+2.6+2.6)3.512.3

Both TRAP2 and LoRA-PBH reduce merged defender-task accuracy to approximately chance level (3.1–4.8%). However, their standalone utility differs substantially. The TRAP2 adapters end 5.9–10.0 percentage points below our unprotected adapters, whereas LoRA-PBH preserves the standalone performance of the adapters it hardens (+0.6 points with 100 steps and +2.6 points with 400 steps). On the free-rider task, the merged model retains 23.2–25.7% with TRAP2 and 12.0–12.3% with LoRA-PBH; we discuss this effect under W2(b).

(2) Why surrogate probes matter.

To isolate the contribution of surrogate probes, we replace them with two scaling-based alternatives while keeping the original adapters and 100-step hardening budget unchanged.

Neither scaling-based alternative achieves both protection and standalone utility. When ΔWψ=ΔWdef\Delta W_\psi=\Delta W_{\mathrm{def}}, standalone accuracy remains relatively close to the original (78.3%), but merged defender-task accuracy remains high (69.7%). Conversely, when ΔWψ=0\Delta W_\psi=0, merged accuracy drops to 3.8%, but standalone accuracy collapses to 53.2%. In contrast, the surrogate-based objective achieves both objectives simultaneously, retaining 82.2% standalone accuracy while reducing merged accuracy to 3.1%.

We further analyze the surrogate updates. Averaged over runs and retained surrogates, their relative norm is ∥ΔWψ∥/∥ΔWdef∥=1.37\Vert \Delta W_\psi\Vert /\Vert \Delta W_{\mathrm{def}}\Vert =1.37, with a cosine similarity of 0.43 to the defender update. In comparison, actual free-rider updates have cosine similarities of only 0.02–0.07. Thus, the surrogate updates capture directions beyond simple rescaling of the defender update. The controlled ablation confirms that these additional directions are important for achieving the protection–utility trade-off.

(3) Post-merge fine-tuning.

Following the post-merge fine-tuning evaluation of TRAP2, we evaluate whether fine-tuning can recover the protected capability. We fine-tune merged models (TA with scaling c=0.5c=0.5) for 200 AdamW steps (learning rate 10−510^{-5}), using either 64 labeled samples per task or a larger split containing 20% of each test set. We evaluate on the held-out remainder. Results are averaged over the 20 defender–free-rider pairs.

ModelBefore FT64 samples per task20% split
Unprotected merge74.067.876.2
LoRA-PBH merge3.053.762.1
Free-rider adapter only56.463.875.1
Base model without adapter48.760.871.6

Although fine-tuning lifts the protected merge out of the initial performance collapse, it does not recover an advantage from retaining our adapter. The fine-tuned protected merge remains 10.1 and 13.0 percentage points below the free-rider adapter alone under the two data settings, respectively. It is also 7.1 and 9.5 points below the fine-tuned base model. In contrast, the unprotected merge performs better than both alternatives after fine-tuning.

These results suggest that under the evaluated fine-tuning budgets an attacker would obtain better defender-task performance by fine-tuning its own adapter without our protected adapter, or even the base model alone.

(4) Hardening cost.

All timings are measured using two RTX A6000 GPUs. LoRA-PBH starts from an existing trained adapter and takes 4.1 minutes in total for 100 hardening steps, 6.0 minutes for 200 steps, and 9.3 minutes for 400 steps. For reference, standard training of our adapter takes 45.2 minutes. TRAP2 is applied during adapter training. Under its official early-stopping protocol, one run takes 148.3 minutes (24k steps) for λ=0.001\lambda=0.001 and 381.5 minutes (54k steps) for λ=0.01\lambda=0.01. Selecting λ\lambda requires one training run per candidate. Although the workflows are different, these results demonstrate the practical advantage of applying LoRA-PBH to an existing adapter without repeating its original training.

Q2: Standalone utility and opportunity cost

We thank the reviewer for pointing out the importance of considering both standalone baselines. We agree that the no-barrier ablation provides an important reference for understanding the opportunity cost of protection.

The two baselines answer different questions. The original adapter represents the owner's actual alternative, which motivates our standalone preservation constraint. The no-barrier control measures what the same optimization budget would achieve if spent exclusively on standalone utility. We therefore added a paired hardening-budget study, in which each LoRA-PBH run is compared with a no-barrier control using the same number of steps and random seed. Results are averaged over five runs.

Hardening stepsNo-barrier controlLoRA-PBHvs. originalvs. control (paired)Merged Defender Acc.Total time
10086.582.2+0.6+0.6−4.3-4.33.14.1 min
20086.782.7+1.1+1.1−4.0-4.03.76.0 min
40088.184.2+2.6+2.6−3.9-3.93.59.3 min

Protection remains at chance level at every budget (3.1–3.7%), while the standalone accuracy of LoRA-PBH remains above that of the original adapter. Compared with continued standalone optimization, the opportunity cost stays at approximately four percentage points (−4.3-4.3 to −3.9-3.9), in exchange for reducing merged defender-task accuracy from 74.0% to 3.1–3.7%.

We believe this provides a practical basis for an adapter owner to evaluate whether hardening is worthwhile. For an already-trained or third-party adapter, LoRA-PBH preserves the original utility, with only a few additional minutes of hardening. In our LLM experiments, which use third-party released adapters, hardening changes standalone accuracy by only −0.1-0.1 to +0.2+0.2 points (Table 3). For an owner who can continue training, we recommend first training the adapter to convergence and then applying LoRA-PBH. The revised manuscript now discusses both baselines and this opportunity cost (Section 7.2.1 and Appendix D.5).

Q3: Asymmetric merging coefficients and fine-tuning with disjoint defender data

We thank the reviewer for suggesting these two important attack scenarios. The original coefficient sweep uses a shared merging coefficient, whereas an attacker can independently control the contribution of each adapter. Furthermore, since the defender datasets are public, an attacker may obtain additional samples that were not used during protection. We therefore extend our evaluation to asymmetric merging and fine-tuning with disjoint public data.

(1) Asymmetric merging coefficients.

We fix the defender coefficient to λdef=1.0\lambda_{\mathrm{def}}=1.0 and vary the attacker coefficient over $\lambda_{\mathrm{fr}}\in\lbrace 0.1,0.3,0.5\rbrace $. The following table reports defender-task / free-rider-task accuracy after TA merging on ViT-B/32, averaged over 20 pairs and five runs.

(λdef,λfr)(\lambda_{\mathrm{def}},\lambda_{\mathrm{fr}})UnprotectedLoRA-PBH
(0.5, 0.5)74.0 / 74.03.1 / 12.0
(1.0, 0.1)80.2 / 53.962.7 / 13.8
(1.0, 0.3)79.8 / 65.550.1 / 25.2
(1.0, 0.5)74.6 / 68.328.6 / 19.4

Asymmetric coefficients do not give the free-rider a usable merge. With our adapter kept at full weight, the merged model moves closer to the defender's standalone configuration, so defender-task accuracy is higher than under equal coefficients. Nevertheless, compared with the unprotected merge under the same coefficients, LoRA-PBH reduces defender-task accuracy by 17.5–46.0 points and free-rider-task accuracy by 40.1–48.9 points.

More specifically, for the free-rider task, accuracy remains at 13.8–25.2% for all three attacker coefficients, substantially below the base model without any adapter (48.7%). For the defender task, accuracy decreases from 62.7% to 28.6% as the attacker coefficient increases. At λfr=0.3\lambda_{\mathrm{fr}}=0.3, it is already close to the base-model level (50.1% vs. 48.7%), and at λfr=0.5\lambda_{\mathrm{fr}}=0.5, it is 20.1 points below the base model.

We further evaluate the same asymmetric configurations on three LLM backbones. The following results are averaged over backbones and pairs. Each entry reports defender-task / free-rider-task accuracy.

(λdef,λfr)(\lambda_{\mathrm{def}},\lambda_{\mathrm{fr}})UnprotectedLoRA-PBH
(0.5, 0.5)76.7 / 76.760.2 / 68.8
(1.0, 0.1)81.3 / 70.868.2 / 65.9
(1.0, 0.3)80.2 / 73.266.3 / 67.2
(1.0, 0.5)78.6 / 75.463.5 / 68.6

With λdef=1.0\lambda_{\mathrm{def}}=1.0, both tasks remain below the average base-model accuracy (69.2%) for all three attacker coefficients. Defender-task accuracy is 63.5–68.2%, compared with 82.1% for the protected adapter used alone. Free-rider-task accuracy is 65.9–68.6%, compared with 70.8–75.4% without protection. Thus, the asymmetric attack does not produce a merge that preserves both capabilities on either ViT or LLM backbones. The revised Section 7.3 states this result, and Appendix D.4 reports both tables.

(2) Fine-tuning with disjoint public data.

We also evaluate an attacker who obtains additional labeled samples from the same public defender dataset but does not have access to the samples used during LoRA-PBH hardening. Such an attacker holds labeled defender-domain data: it could train its own adapter and has no need to free-ride, which is why the threat model in Section 4.3 excludes it. We nevertheless measured this case as requested. We use disjoint data splits for protection and attacker fine-tuning, and evaluate on held-out data.

For ViT-B/32, the fine-tuning results are reported in the table under Q1. Although fine-tuning recovers part of the defender capability, the protected merge remains inferior to fine-tuning the free-rider adapter alone or the base model without our adapter.

We further conduct the disjoint-data fine-tuning experiment on the three LLM backbones. The same trend is observed: relative to the free-rider adapter alone, fine-tuning a merge that retains our protected adapter incurs an additional performance cost of 3.1 points with 64 samples and 1.9 points with the 20% split. In contrast, including the unprotected adapter provides gains of 3.7 and 5.2 points under the respective settings.

These results indicate that, even when the attacker obtains additional public defender-domain data that does not overlap with the protection data, retaining the protected adapter does not provide a measurable advantage over fine-tuning without it. The revised Section 4.3 states the attacker's resources explicitly, and Appendix D.3 reports these results.

Reviewer_UCSXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to prevent unauthorized merging of publicly released LoRA adapters while preserving their standalone performance. The authors first examine why existing defenses designed for full-parameter models, including PaRaMS, MergeLock, and MergeBarrier, do not directly work in the fixed-base LoRA setting. They then propose LoRA Preservation-Barrier Hardening (LoRA-PBH), which uses defender-generated surrogate adapters to construct merge perturbations and optimizes the released adapter to increase the performance gap between standalone and merged models. The approach is evaluated on image classification, language understanding, and text-to-image generation, covering multiple backbones, merging methods, and attack settings. The reported results show substantial degradation in merged performance with relatively little impact on standalone utility, although several aspects of the theoretical formulation and robustness evaluation remain unclear.

Critical strengths
  1. The paper addresses an interesting problem that has received relatively limited attention in the LoRA setting. As lightweight adapters are increasingly shared and reused, unauthorized merging becomes a realistic concern. I think this is a worthwhile direction, particularly because most existing protection methods were originally designed for full-parameter models rather than publicly released adapters.

  2. One part I particularly liked is the analysis in Section 5, where authors explain why transformations applied within the LoRA factorization may preserve the effective weight update and therefore become invisible to update-space merging methods.

  3. Another positive aspect is that LoRA-PBH does not require modifying the public base model or knowing the attacker's adapter beforehand. This makes the approach more practical than methods that rely on controlling the entire model or anticipating a specific merging partner.

Critical weaknesses
  1. My main concern is with the protection objective in Section 4, the attacker is allowed to choose the merging method and its configuration, but it is not entirely clear what constraints define a meaningful unauthorized merge. For example, choosing a defender coefficient close to one and a free-rider coefficient close to zero could preserve much of the defender's capability, even though it would no longer provide a useful combination of the two tasks. More importantly, the protected adapter remains publicly usable on its own.

  2. Section 6 motivates LoRA-PBH through a max–min preservation-barrier objective, and Section 6.3 introduces SoftMin to approximate the worst sampled barrier. However, Appendix D.2 states that all reported experiments use a single perturbation per optimization step (M = 1), in which case SoftMin reduces to the individual barrier and the temperature parameter has no effect. This makes it difficult to determine whether the proposed worst-case optimization is actually responsible for the reported gains. Proposition 1 is also conditional on a coverage radius and a local Lipschitz constant that are not estimated experimentally.

  3. The comparison with PaRaMS, MergeLock, and MergeBarrier is useful, but these methods were primarily developed for full-parameter models. Their direct adaptations either leave the LoRA weight updates unchanged or substantially damage standalone performance, so their poor results are partly expected from the structural analysis. I am not fully convinced that this provides a strong comparison against alternative LoRA-specific protection strategies under comparable optimization budgets. Interestingly, Appendix A.1 also reports that PaRaMS achieves better merge suppression than LoRA-PBH under the factor-level PEM-TA interface (26.3% versus 31.5%).

  4. Section 7.4 is a welcome addition, but the evidence is not quite sufficient to support a strong robustness claim. The adaptive attack already restores defender-task merged accuracy from 3.0% to 34.4% on ViT-B/32, which is a fairly large recovery. Moreover, the evaluation considers one feature-guided optimization strategy with a specific objective and training budget. It remains unclear how the defense would behave against attackers using different optimization objectives, larger budgets, or more flexible merging coefficients.

  5. Vision results are strong, but the evidence from language and diffusion models is less complete. For language models, Table 3 reports a smaller average reduction in merged accuracy, from 76.7% to 60.2%, while the protected merged accuracy is also below the average base-model accuracy of 69.2%. In the vision experiments, free-rider-task accuracy drops sharply as well, from 74.0% to 11.7%. These results raise the question of whether the method specifically prevents capability preservation or simply makes the resulting merged model broadly ineffective. For diffusion models, Figure 2 contains only three qualitative examples without quantitative measures of concept preservation or generation quality.

  6. The single-probe training configuration, surrogate construction procedure, and adaptive attack implementation are essential for understanding the method but receive relatively little attention in the main text. The hardening experiments also appear to use a single random seed, without reporting performance variance or confidence intervals. Given that the method involves clustering, surrogate training, and stochastic optimization, it would be helpful to know how stable the results are across different runs.

Questions for authors
  1. How do the authors formally distinguish a successful unauthorized merge from a trivial configuration that largely preserves the defender adapter while suppressing the contribution of the free-rider adapter? Does the perturbation set in Eq. (12) exclude configurations approaching zero merge perturbation, and if so, what constraints are imposed?

  2. Since Appendix D.2 indicates that the experiments use M = 1, could the authors explain what practical advantage the SoftMin formulation provides in the reported implementation? A direct comparison between single-probe and multi-probe training would help determine whether the proposed worst-case objective contributes beyond ordinary stochastic barrier optimization.

  3. Could the authors report variability across independent hardening runs, at least for the main ViT-B/32 results? Also, for the SDXL experiments, are there quantitative or human-evaluation results across multiple prompts and random seeds that support the qualitative observations in Figure 2?

6 comments

We sincerely thank the reviewer for the thoughtful and constructive review. We particularly appreciate the questions concerning the definition of successful unauthorized merging, the practical role of SoftMin, and the completeness of our empirical evaluation. Following these suggestions, we have conducted additional experiments and clarified several aspects of our threat model and theoretical formulation. We first address the three questions directly, together with their corresponding weaknesses, and then respond to the remaining concerns.

Q1 & W1: Definition of successful unauthorized merging

We agree that our original presentation did not make the attacker's objective sufficiently explicit. The attacker's goal is to obtain a single merged model that preserves both the defender's proprietary capability and the free-rider's own capability. Appendix B.1 already describes this criterion, but Section 4 and Eq. (12) did not explicitly incorporate it. The revised manuscript clarifies this definition (the attacker's goal in Section 4.3, the restricted perturbation set in Section 6.1, and its formal definition in Appendix C.4).

Specifically, we introduce a minimum free-rider contribution requirement: κatk(q)=Perfatk(Wϕ~+q)−Perfatk(Wϕ~)Perfatk(Wϕatk)−Perfatk(Wϕ~),\kappa_{\mathrm{atk}}(q)=\frac{\mathrm{Perf}_{\mathrm{atk}}(W_{\tilde\phi}+q)-\mathrm{Perf}_{\mathrm{atk}}(W_{\tilde\phi})}{\mathrm{Perf}_{\mathrm{atk}}(W_{\phi_{\mathrm{atk}}})-\mathrm{Perf}_{\mathrm{atk}}(W_{\tilde\phi})},

Qρ={q: κatk(q)≥ρ},ρ∈(0,1].\mathcal Q_\rho=\lbrace q:\thinspace \kappa_{\mathrm{atk}}(q)\ge\rho\rbrace,\qquad \rho\in(0,1].

Here, κatk(q)\kappa_{\mathrm{atk}}(q) measures the fraction of the performance gap between the protected standalone model and the attacker's standalone model, on the attacker task, that the merged model recovers. The parameter ρ\rho specifies the minimum contribution required by the free-rider. A successful unauthorized merge must find some q∈Qρq\in\mathcal Q_\rho that also preserves the defender's capability.

To answer Q1 directly, the revised perturbation set excludes configurations approaching zero merge perturbation. Since κatk(0)=0\kappa_{\mathrm{atk}}(0)=0, configurations such as λdef≈1\lambda_{\mathrm{def}}\approx1 and λfr≈0\lambda_{\mathrm{fr}}\approx0 represent standalone use of the protected adapter rather than a meaningful combination of the two tasks. Our hardening probes also avoid this region in practice because each probe contains a surrogate update.

We further evaluate asymmetric merging coefficients on ViT-B/32, fixing λdef=1\lambda_{\mathrm{def}}=1 and varying $\lambda_{\mathrm{fr}}\in\lbrace 0.1,0.3,0.5\rbrace $. Results are averaged over 20 defender–free-rider pairs and five random seeds.

(λdef,λfr)(\lambda_{\mathrm{def}},\lambda_{\mathrm{fr}})UnprotectedLoRA-PBH
(0.5, 0.5)74.0 / 74.03.1 / 12.0
(1.0, 0.1)80.2 / 53.962.7 / 13.8
(1.0, 0.3)79.8 / 65.550.1 / 25.2
(1.0, 0.5)74.6 / 68.328.6 / 19.4

Defender-task / free-rider-task accuracy (%).

Even with asymmetric coefficients, the protected merges do not preserve both capabilities. Specifically, for the defender task, accuracy decreases from 62.7% to 28.6% as the attacker coefficient increases, compared with 82.2% for the protected adapter used alone. For the free-rider task, accuracy remains at 13.8–25.2%, substantially below both the corresponding unprotected merges (53.9–68.3%) and the base model without any adapter (48.7%).

Thus, none of the evaluated asymmetric configurations provides a useful combination of the two capabilities. The revised Section 7.3 states this result, and Appendix D.4 reports the full tables.

Regarding the fact that the protected adapter remains publicly usable on its own, we clarify that this is intentional. LoRA-PBH protects against unauthorized merging, not direct use of the adapter as released. Standalone use does not obscure the origin of the capability. Our threat model instead concerns a derived merged model that inherits the defender's capability while obscuring its origin.

Q2 & W2: Practical role of SoftMin and Proposition 1

We agree with the reviewer that when M=1M=1, SoftMin reduces to a single sampled barrier and the temperature parameter β\beta has no effect. Section 6.3 presents the general MM-probe objective, whereas the reported implementation uses M=1M=1, i.e., single-probe stochastic barrier optimization (Appendices C.2 and D.2). Our original presentation did not distinguish these two settings clearly, and we have revised Section 6.3 accordingly.

To answer Q2 directly, the reported protection gains come from optimizing the preservation barrier over surrogate probes resampled at every step, rather than from multi-probe SoftMin aggregation. SoftMin provides the general formulation for M>1M>1. We conducted two additional experiments on ViT-B/32 to examine the practical contribution of the barrier and the effect of increasing MM.

(1) Contribution of the preservation barrier.

We vary the barrier weight ν\nu, with five independent runs for each value. Without the barrier (ν=0\nu=0, Appendix D.5), merged defender-task accuracy remains at 74.7%, close to the unprotected level. From ν=0.3\nu=0.3 upward, merged accuracy reaches chance level in every run. These results confirm that the preservation barrier is responsible for the protection effect.

(2) Single-probe versus multi-probe optimization.

We compare M∈{1,4,8}M\in\{1,4,8\} under hardening budgets ranging from 100 to 400 steps. For protection, merged defender-task accuracy remains approximately at chance level for all configurations. For M=4M=4 and M=8M=8, it is 3.0% at 100 steps and 3.4–4.5% from 200 steps onward, compared with 3.1–3.7% for M=1M=1. For computational cost, runtime increases with M×M\timessteps. Multi-probe training provides almost no additional protection while requiring substantially more computation.

These results show that M=1M=1 is the most efficient configuration in our evaluated setting. We therefore retain single-probe optimization in the reported implementation.

(3) Scope of Proposition 1.

Proposition 1 is a conditional sufficient guarantee, as stated in Appendix C.1. When the coverage and local Lipschitz conditions hold, it connects the sampled SoftMin objective to the worst-case barrier. It should not be interpreted as an unconditional guarantee against arbitrary attacker adapters.

We distinguish this theoretical result from our empirical generalization evaluation. Each protected adapter is tested against four unseen free-riders and six additional merging methods (Tables 1 and 2). These free-riders are not close to the surrogate updates: they have cosine similarities of only 0.02–0.07 with ΔWdef\Delta W_{\mathrm{def}} and lie 1.5–1.9 times their own norm from the nearest surrogate. Nevertheless, their merged defender-task performance remains approximately at chance level.

The revised manuscript states that the theoretical proposition is conditional, while the transfer to unseen adapters is supported by empirical evaluation.

Q3 & W6: Experimental reliability, SDXL evaluation, and reproducibility

We thank the reviewer for requesting additional evidence regarding the stability of hardening and the quantitative evaluation of diffusion models. We have conducted five independent hardening runs and added a quantitative SDXL evaluation across multiple prompts and random seeds.

(1) Variability across independent hardening runs.

We re-ran the complete ViT-B/32 hardening pipeline with five independent random seeds, including clustering, surrogate training, surrogate filtering, and hardening.

Across the five runs, merged defender-task accuracy remains at chance level, with variation of only ±0.15\pm0.15 percentage points. Standalone accuracy changes vary by approximately ±1.2\pm1.2 points at both 100 and 400 hardening steps. For comparison, the no-barrier control exhibits larger variation (±1.7\pm1.7 points).

These results indicate that the protection effect is stable across independent runs. We also clarify that the +0.5+0.5-point standalone change in the original comparison should be interpreted as preservation of utility rather than a statistically established improvement, which is reported in Appendix D.5 of the revised manuscript.

(2) Quantitative SDXL evaluation.

For the three SDXL defender LoRAs in Figure 2, we generate 50 images per condition (10 prompts ×\times 5 random seeds) and compute average text-to-image alignment on the defender prompts.

ConditionAlignment
Base model, no LoRA0.246
Unprotected LoRA, standalone0.324
Unprotected LoRA, merged0.316
Protected LoRA, standalone0.322
Protected LoRA, merged0.186

The protected standalone adapter retains nearly the same alignment as the unprotected adapter (0.322 vs. 0.324), showing that the defender concept remains available when the adapter is used independently. In contrast, after merging, alignment drops to 0.186, below even the base model without any LoRA (0.246). The unprotected merge retains an alignment of 0.316.

These quantitative results support the qualitative observations in Figure 2: LoRA-PBH preserves standalone concept generation while disrupting concept preservation after merging. They are reported in Section 7.2.3 (Table R2) and Appendix D.8 of the revised manuscript.

(3) Main-text implementation details.

We agree that the single-probe configuration, surrogate construction procedure, and adaptive attack implementation are essential for understanding the method. We have moved a concise description of these components from the appendix into the main text, including the actual M=1M=1 setting, surrogate candidate generation and filtering, and the adaptive attack's optimization procedure and access assumptions.

responses to the remaining weaknesses

W3: LoRA-specific baselines and PEM-TA

We appreciate the reviewer's concern that the original comparisons mainly involve defenses designed for full-model releases. We agree that a direct comparison with LoRA-specific alternatives is important.

We therefore add a comparison with TRAP2 (ICML 2026), using its official code and hyperparameters, as well as two controlled scaling-probe variants of LoRA-PBH. The full quantitative comparison is provided in our response to Reviewer RHX9, and in Section 7.2.1 and Appendix D.7 of the revised manuscript.

Both TRAP2 and LoRA-PBH reduce merged defender-task accuracy to approximately chance level. However, TRAP2 is a training-time defense and requires retraining an existing adapter with its protection objective (24k–54k steps in our runs). Its resulting standalone accuracy is 5.9–10.0 points below the unprotected adapter. In contrast, LoRA-PBH hardens an existing adapter in 4.1 minutes while preserving its standalone accuracy (+0.6 points with 100 steps).

The controlled scaling-probe ablations further show that simply rescaling the defender update cannot achieve the same protection–utility trade-off. This supports the importance of surrogate-based perturbations in our post-hoc protection objective.

Regarding PEM-TA, the results are consistent with the different mechanisms of PaRaMS and LoRA-PBH. PaRaMS modifies the LoRA factorization while preserving the effective update ΔW\Delta W. Consequently, its protection becomes visible when the attacker merges AA and BB separately, through the cross terms in Eq. (18), but disappears when merging operates directly on ΔW\Delta W (Eq. 17). LoRA-PBH is not specifically hardened against this factor-level interface, and its merged accuracy is 31.5% under PEM-TA (80.7% without protection). However, for update-space merging methods such as TA, TIES-Merging, and KnOTS, PaRaMS provides no protection (74.0% under TA), whereas LoRA-PBH reduces merged accuracy to 3.0% under TA.

Since the attacker can choose the merging interface, at the default coefficient the relevant comparison is the worst case over the evaluated interfaces: 74.0% for PaRaMS versus 31.5% for LoRA-PBH. The revised Appendix A.1 clarifies this distinction.

W4: Robustness against stronger adaptive attacks

We thank the reviewer for emphasizing that standard merging results alone are insufficient to establish robustness against motivated attackers. We agree that the original adaptive attack, which recovers defender-task accuracy to 34.4%, requires further evaluation.

(1) Larger attack budgets.

We extend the adaptive attack in Section 7.4 to 400 steps, four times its original budget, across all 20 ViT-B/32 defender–free-rider pairs. The full accuracy-versus-budget curve is shown in Figure R1 (Appendix D.3) of the revised manuscript.

Beyond the original 100 steps (34.4%), defender-task accuracy remains between approximately 30% and 37%: it reaches 37.2% at 200 steps and 31.0% at 400 steps. Importantly, it remains below the base model without any adapter (48.7%) at every evaluated budget.

In contrast, free-rider-task accuracy increases from 11.7% to 80.5% (unprotected merge: 74.0%). Thus, the additional optimization budget primarily restores the attacker's own capability, rather than the defender's capability.

(2) Flexible merging coefficients.

The adaptive attack in Section 7.4 already jointly learns both merging coefficients (Appendix D.3), so the extended-budget experiment includes this flexibility.

We additionally evaluate fixed asymmetric merging coefficients, as discussed in Q1. With λdef=1\lambda_{\mathrm{def}}=1, free-rider-task accuracy remains at 13.8–25.2%, below the base model without any adapter (48.7%).

(3) Alternative attack objectives.

We consider three additional attack strategies.

Reviewer_FEhNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to protect publicly released LoRA adapters against unauthorized merging while preserving their standalone performance and keeping the public base model fixed.

The proposed LoRA PBH modifies an adapter to increase its defender-task loss after merging with surrogate adapters. These surrogates are constructed from clustered defender features. A standalone task loss and a normalized weight-space anchor help preserve the released adapter’s utility.

Evaluation covers several vision & language backbones. It preserves standalone performance while degrading merged performance across multiple backbones, merging methods, merge coefficients, and multi-adapter setting.

Critical strengths

First, the paper identifies a useful structural limitation of directly transferring existing defenses to LoRA. The distinction between transformations of stored factors and transformations of effective weight updates clearly explains why factor-internal reparameterizations are invisible to common merging interfaces.

Second, the reported suppression of non-adaptive merging is substantial and extends beyond the training configuration. Each protected adapter is constructed using only defender-side information and then evaluated against four unseen task adapters. Results across vision and language backbones support the empirical finding. The objective ablations also show that removing the barrier restores merged performance, while removing preservation terms harms standalone utility. These results provide meaningful evidence that the proposed objective achieves its intended tradeoff under the evaluated conditions.

Critical weaknesses

It remains unclear to me that how much protection this provides against someone who actively wants to recover the capability. Most of the evidence concerns standard merging procedures. In practice, a motivated user has the released weights and a functioning standalone model, which gives them room to modify the adapter or learn from its behavior. The adaptive experiment already recovers a substantial amount of performance, and the paper does not establish where that recovery would stop with a stronger effort. I therefore find the evidence convincing that the method disrupts common merging workflows, but less convincing that it provides a durable restriction on capability reuse. This distinction matters for the practical value of the proposed protection.

Questions for authors

What practical restriction should an owner expect after releasing a protected adapter? Is the goal to discourage straightforward merging, or to resist a motivated user who is willing to modify or retrain the released weights? The latter would make the contribution substantially stronger, but seems less established by the current results.

2 comments

We thank the reviewer for the careful and constructive review. We appreciate the opportunity to clarify what protection LoRA-PBH provides, how far an attacker can recover the protected capability, and why the considered threat model is practically relevant.

W & Q: What restriction can an owner expect, and where does recovery stop?

What protection can an owner expect.

To answer the reviewer's question directly, LoRA-PBH provides protection beyond discouraging straightforward merging. It also resists the evaluated adaptive optimization attacks, in which a motivated attacker modifies the released adapter without access to defender-domain data. Direct merging reduces defender-task accuracy to approximately chance level on ViT (Table 1) and below the base-model performance on LLMs (Table 3). Furthermore, even an adaptive attack on the released adapter remains below the base model on ViT-B/32, including when its optimization budget is increased to four times that of Section 7.4.

In our experiments, the defender's capability was recovered only by attacks that use additional defender-domain data and a separate training procedure. We therefore clarify the attacker model in the revised Section 4.3: the attacker is a resource-constrained free-rider who can access and modify the released adapter but lacks the defender-domain data and training resources needed to independently relearn the protected capability.

Where recovery stops.

The adaptive attack in Section 7.4 initially recovers defender-task accuracy to 34.4%, which is already below the base model without any adapter on the same defender tasks (48.7%). To investigate whether a larger optimization budget can further recover the capability, we extend the same attack to 400 steps, four times its original budget, across all 20 ViT-B/32 defender–free-rider pairs. The results are reported in Section 7.4 and Appendix D.3 (Figure R1) of the revised manuscript.

Increasing the optimization budget provides little additional recovery. Beyond the original 100 steps (34.4%), defender-task accuracy remains between approximately 30% and 37%, reaching 37.2% at 200 steps and 31.0% at 400 steps. More importantly, it remains below the base-model performance at every evaluated budget.

In contrast, the additional optimization primarily recovers the free-rider's own capability: free-rider-task accuracy increases from 11.7% to 80.5%, exceeding the unprotected merge (74.0%). This is expected because the attacker optimizes using its own labeled data. Thus, without defender-domain data, increasing the optimization budget primarily restores the attacker's own task rather than the defender's capability.

Modifying the adapter analytically.

We also consider AGA (Guo et al., ECCV 2026), a recent data-free attack that attempts to undo parameter-level defenses by aligning protected weights with the public base model.

AGA relies on recovering a hidden transformation applied to the full model weights. Specifically, it assumes a protected model of the form Wp=(W0+τ)P,W_p=(W_0+\tau)P, where W0W_0 is the public base model, τ\tau is the model update, and PP is a hidden transformation. By aligning the protected weights with the public base, AGA attempts to recover the inverse transformation.

AGA cannot undo LoRA-PBH. Our method leaves the public base model unchanged and protects only the released LoRA update: Wp=W0+ΔWp.W_p=W_0+\Delta W_p. There is no hidden transformation of the base weights to recover or invert. Consequently, AGA's least-squares alignment returns the public base plus at most a projection of the protected update, never the original adapter. We provide the proof sketch in our response to Reviewer UCSX and include the complete derivation in the revised appendix.

Learning from the model's behavior.

We further distinguish attackers according to whether they have access to defender-domain data.

  • Without defender-domain data. The adaptive attack in Section 7.4 already explores this scenario. It matches the features of our protected standalone model using the attacker's own data (Eq. 30), but its defender-task accuracy remains below the base model, even with an extended optimization budget. Thus, access to the protected model's behavior alone does not enable the evaluated attacker to recover the defender's capability.
  • With defender-domain data. Additional data enables recovery through independent training procedures. We evaluate two representative strategies (reported in Appendix D.3 of the revised manuscript):
    • Labeled data. Post-merge fine-tuning partially recovers defender-task performance. However, the fine-tuned protected merge remains worse than fine-tuning the free-rider adapter alone using the same data. The performance disadvantage is 10.1–13.0 points on ViT-B/32 and 1.9–3.1 points on LLMs. In contrast, the unprotected adapter provides gains of 1.1–4.0 points on ViT-B/32 and 3.7–5.2 points on LLMs. On ViT-B/32, the protected merge also remains 7.1–9.5 points below the base model fine-tuned using the same data.

    • Unlabeled data. Knowledge distillation from the protected model recovers merged defender-task accuracy to 69.3% on ViT-B/32 (compared with 74.0% for the unprotected merge, averaged over all pairs) and 74.3% on Llama-3.1-8B (compared with 76.6% for the unprotected merge).

These results establish an important limitation: LoRA-PBH does not prevent an attacker with sufficient defender-domain information from independently learning the protected capability. However, both fine-tuning and distillation require additional data and a separate training process. The attacker is no longer obtaining the capability simply by merging the released weights; instead, it must construct a new model or adapter through additional learning. The revised Section 7.4 states this boundary and limits its conclusion to adaptive attackers without defender-domain data.

Why this scope is practically relevant.

Our threat model reflects a practical use case of publicly released LoRA adapters. Adapters are widely shared through repositories such as Hugging Face, and one of their main advantages is that users can combine existing adapters through merging without collecting additional training data or performing further optimization. This makes unauthorized merging an attractive way to reuse another owner's capability at little cost.

LoRA-PBH specifically restricts this direct reuse route. An attacker cannot readily obtain a useful combination of the defender's and its own capabilities simply by merging the released adapters, even when allowed to modify the merging coefficients or optimize the merging configuration. Recovering the defender's capability instead requires additional resources, such as collecting defender-domain data and performing a separate training or distillation procedure.

We therefore do not claim to prevent all possible forms of capability recovery. Rather, our goal is to make unauthorized capability reuse through merging substantially more difficult and costly, while preserving the standalone usability of the released adapter. We believe this is a meaningful practical restriction for adapter owners who distribute LoRAs for independent use but do not intend to authorize their capabilities to be incorporated into other models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.