acceptodds
Under review as a conference paper at ICLR 2027

Towards Building Non-Fine-Tunable Foundation Models

Abstract

Open-sourcing foundation models (FMs) enables broad reuse but also exposes model trainers to economic and safety risks from unrestricted downstream fine-tuning. We address this problem by building non-fine-tunable foundation models: models that remain broadly usable in their released form while yielding limited adaptation gains under task-agnostic unauthorized fine-tuning. We propose Private Mask Pre-Training (PMP), a pre-training framework that concentrates representation learning into a sparse subnetwork identified early in training. The binary mask defining this subnetwork is kept private, and only the final dense weights are released. This forces unauthorized fine-tuning without access to the mask to update parameters misaligned with pretraining subspace, inducing an intrinsic mismatch between the fine-tuning objective and the pre-training geometry. We provide theoretical analysis showing that this mismatch destabilizes gradient-based adaptation and bounds fine-tuning gains. Empirical results on large language models demonstrating that PMP preserves base model performance while consistently degrading unauthorized fine-tuning across a wide range of downstream tasks, with the strength of non-fine-tunability controlled by the mask ratio.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Foundation Model, Non-Fine-Tunable Model, LLM Security

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RgQQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Private Mask Pre-Training (PMP) to build non-fine-tunable foundation models, which stay usable as released but gain little from unauthorized fine-tuning.

  • In a 500-step warm-up, PMP forms a global top-kk mask from absolute gradients. It selects the first mask whose IoU with the previous mask stays at or above 0.99 for 5 steps (an early-bird lottery ticket with ρ=0.7\rho = 0.7, so 70% of parameters are trainable).
  • Pre-training then updates only θM\theta_M, while the complement θMˉ\theta_{\bar M} stays at its random initialization θ(0)\theta^{(0)}.
  • Only the dense weights are released. Authorized users receive the mask and fine-tune only θM\theta_M.

Section 3 argues, through Assumption 1 and Proposition 1, that one SGD step of fine-tuning without the mask raises the expected pre-training loss by at least cη2c\eta^2. The experiments pre-train a TinyLlama-1.1B-scale model from random initialization on SlimPajama-6B (the GPT-2 setup is not described). They then fine-tune GPT-2 and TinyLlama on GLUE with LoRA and with full fine-tuning. PMP lowers the fine-tuned scores; for example, the TinyLlama full fine-tuning average is 59.55 against 68.65 without defense (Table 3). Ablations cover random masks, the mask ratio, the learning rate and the number of epochs. The appendix compares PMP with SOPHON, REPNOISE and Vaccine and reports BeaverTails harmfulness.

Critical strengths
  1. The problem is relevant, and PMP is a simple, task-agnostic way to approach it. Existing defenses such as SOPHON and model immunization need the data or objective they want to restrict (Sections 1 and 6). PMP needs neither. It masks gradients during pre-training (Eq. 5) and withholds MM at release (Section 2.3). Tying authorized fine-tuning to possession of the mask is a concrete form of controlled release that others could build on. This framing is the main reason I see value in the work.

  2. Within the tested setting, the drop in fine-tuned scores is consistent and is checked against the obvious confounds.

    • Table 1 shows lower fine-tuned scores with PMP for both GPT-2 and TinyLlama, under both LoRA and full fine-tuning, on all nine tasks.
    • Figure 3 replaces the early-bird mask with a random mask of the same ratio, and the fine-tuned scores become higher on every task. So the effect depends on which parameters are frozen, not only on how many.
    • Figure 4 varies the mask ratio, the fine-tuning learning rate (1e-5 to 4e-5) and the number of epochs (1 to 5).
    • Appendix A.2 adds SOPHON, REPNOISE, Vaccine and BeaverTails.

    How much this signal means for real foundation models depends on Weakness 2.

Critical weaknesses
  1. Proposition 1 does not follow from Assumption 1, and a corrected version would still not bound downstream gains.

    • The first-order term is not handled. The conditional optimum only gives ∇θMLpre=0\nabla_{\theta_M} \mathcal{L}_{pre} = 0. The gradient with respect to θMˉ\theta_{\bar M}, which stays at random initialization, is in general nonzero. The proof sketch notes that the first-order term "has no guaranteed sign" and then draws the conclusion from the quadratic term alone, although for small η\eta the first-order term dominates.
    • Its sign actually works against the claim. By Eq. 7, E[gMˉ]=∇θMˉLpre+∇θMˉΔ\mathbb{E}[g_{\bar M}] = \nabla_{\theta_{\bar M}}\mathcal{L}_{pre} + \nabla_{\theta_{\bar M}}\Delta. The first-order change in Lpre\mathcal{L}_{pre} is therefore −η∥∇θMˉLpre∥2−η⟨∇θMˉLpre,∇θMˉΔ⟩-\eta\|\nabla_{\theta_{\bar M}}\mathcal{L}_{pre}\|^2 - \eta\langle\nabla_{\theta_{\bar M}}\mathcal{L}_{pre}, \nabla_{\theta_{\bar M}}\Delta\rangle. With Δ=0\Delta = 0, fine-tuning without the mask is simply continued pre-training of all parameters, and a small step lowers Lpre\mathcal{L}_{pre} whenever ∇θMˉLpre≠0\nabla_{\theta_{\bar M}}\mathcal{L}_{pre} \neq 0.
    • A two-parameter example meets every stated condition and violates Eq. 8. Let θM=a\theta_M = a, θMˉ=b\theta_{\bar M} = b and Lpre=ϵ2a2+12(b−1)2\mathcal{L}_{pre} = \frac{\epsilon}{2}a^2 + \frac{1}{2}(b-1)^2 with small ϵ>0\epsilon > 0, with released point (0,0)(0, 0) and Δ=0\Delta = 0. One SGD step gives b+=ηb^+ = \eta, and Lpre\mathcal{L}_{pre} changes by −η+η2/2<0-\eta + \eta^2/2 < 0 for 0<η<20 < \eta < 2.
    • Even a corrected bound would not reach the experiments. Gain is defined on Lpre\mathcal{L}_{pre} (Section 3), while the experiments measure downstream accuracy, and a rise in Lpre\mathcal{L}_{pre} does not mean that Lft\mathcal{L}_{ft} is hard to reduce. A standard model at a local minimum of Lpre\mathcal{L}_{pre}, with positive curvature along the update, also gets the same cη2c\eta^2 increase, so Eq. 8 does not separate PMP from standard pre-training. The analysis also covers one SGD step, while the experiments use multi-step AdamW and LoRA.

    Figure 5 does not fill the gap. It plots one random direction per curve, and the masked-direction curve reaches its minimum near α≈−0.07\alpha \approx -0.07, slightly below its value at α=0\alpha = 0, which does not fit the conditional-optimum assumption. The paper presents this analysis as the mechanism behind non-fine-tunability, so the mechanism is currently unsupported.

  2. The experiments do not test capability preservation, and several signs suggest that the models are too weakly pre-trained for the fine-tuning results to be meaningful.

    • The Base scores carry no information about capability. The paper does not say how Base GLUE scores are produced before fine-tuning. The reference Base row of TinyLlama gives MRPC 31.62 (= 129/408, always "not equivalent") and RTE 52.71 (= 146/277, always "entailment"). The PMP Base row is at chance level (SST-2 50.80, QNLI 50.72, MNLI-m 32.81). The three defended baselines also have higher Base averages than the undefended model (45.77, 42.86 and 41.07 vs. 38.41, Table 4). So "preserves base model performance" compares chance with chance.
    • Fine-tuning itself often fails, and failure is scored as protection. Without PMP, GPT-2 full fine-tuning reaches 56.61 on QQP, below always predicting "not duplicate" (63.18). GPT-2 LoRA gives 68.38 on MRPC (= 279/408, a constant prediction). TinyLlama gives 68.36 on CoLA, below always predicting "acceptable" (69.13 = 721/1043). Because lower fine-tuned scores count as success, a PMP model that collapses to a constant prediction is scored as protected; for example, GPT-2 with PMP and full fine-tuning gives QNLI 50.54 (= 2761/5463) and RTE 47.29 (= 131/277).
    • Pre-training looks too short to produce a capable model. Pre-training uses about 164M tokens (20,000 updates × 32 sequences × 256 tokens) with a learning rate of 2×10−52 \times 10^{-5} (Appendix A.1). This learning rate is low for training a 1B-scale model from scratch. The tokenizer is not stated. If the loss in Figure 5 is the mean per-token cross-entropy in nats, its value at α=0\alpha = 0 (about 11.2) is above ln⁡32000≈10.37\ln 32000 \approx 10.37 and ln⁡50257≈10.82\ln 50257 \approx 10.82. In that case the model is worse than a uniform prediction over any vocabulary smaller than about 73K tokens.

    Without a language-modeling metric for matched checkpoints, and without a reference model that fine-tunes normally, the claims that PMP "preserves base model performance" and yields a non-fine-tunable foundation model are not tested. The observed suppression may not carry over to models with real pre-trained capability.

Questions for authors
  1. How are the Base GLUE scores obtained: zero-shot prompting, or a classification head with no training?

    • Please report held-out SlimPajama loss or perplexity for the PMP and W/O PMP checkpoints, and state the tokenizer and the loss plotted in Figure 5.
    • Please confirm that 2×10−52 \times 10^{-5} is the pre-training learning rate (Figure 4 suggests the same value is also the main fine-tuning learning rate), and give the fine-tuning hyperparameters (learning rate, epochs, batch size, LoRA rank, classification head), which Appendix A does not list.
    • Please also report GLUE, under the same fine-tuning protocol, for a randomly initialized model of the same architecture and for a W/O PMP checkpoint with matched pre-training loss.

    If the W/O PMP model is clearly better than the random-initialization control and PMP keeps its perplexity close to W/O PMP, my main empirical concern would be much weaker.

  2. Have you tested attackers who try to recover or approximate the mask? Three routes follow from the paper itself.

    • Frozen parameters equal θ(0)\theta^{(0)} exactly (Eq. 5). An attacker who can reproduce the initialization (Appendix A.1 fixes seed 42) can compare weights directly, and any normalization weights in the complement stay at their constant initial value.
    • The Figure 6 caption says that near-zero gradients come mostly from frozen parameters, so thresholding gradient magnitudes gives a candidate mask.
    • An attacker can fine-tune only the top-kk parameters by gradient magnitude, as PMP itself does in Eq. 2. Section 5.2 assumes that the adversary does not know the mask ratio, but the paper reports it (ρ=0.7\rho = 0.7).

    Please report the composition of the mask per tensor type, the IoU between recovered and true masks, and GLUE after fine-tuning only the recovered set. A high recovery rate would lower my assessment further, and a low one with no fine-tuning benefit would raise it.

  3. Under what additional condition does Eq. 8 hold, for example a sign condition on ⟨∇θMˉLpre,E[gMˉ]⟩\langle \nabla_{\theta_{\bar M}} \mathcal{L}_{pre}, \mathbb{E}[g_{\bar M}] \rangle? How would the result connect to Lft\mathcal{L}_{ft} or to downstream accuracy, and how would it compare PMP with a standard model? Please address the example in Weakness 1.

  4. (Clarification) In Figure 2, authorized fine-tuning of TinyLlama reaches about 65 on QNLI and 62 on MNLI-m, against 79.37 and 71.17 for full fine-tuning without PMP (Table 1). It is also slightly below unauthorized fine-tuning on SST-2 and MRPC, although Section 4.3 and the caption say that it consistently matches or outperforms. Is this loss for mask holders expected? For Table 5, please give the harmful fine-tuning data, the number of steps and the judge model, and explain why the Defended score is 0.05 for all four defenses.

Reviewer_GzNHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors explore building non-finetunable foundation models to block the unauthorized downstream use while preserving tunability for authorized ones. They proposed a three-stage pipeline: (1) sparse mask identification during early pre-training, (2) pre-training over that masked parameter subspace, (3) releasing the full model weights with and without the mask depending on the authorization. The authors validated their method with two small language models on GLUE datasets, showing that the proposed method preserves good fine-tunability with the mask while somehow blocking the fine-tunability without the mask.

Critical strengths
  • The problem -- developing selectively non-finetunable foundation models -- the authors are working on is very important and timely.
  • The writing is very straightforward and easy to follow.
  • The proposed method is intuitive and easy to reproduce, and is also backed up by the theoretical statement.
Critical weaknesses
  1. Lack of rigor in experimental design and empirical support.
  • It is known that the fine-tunability also depends on the pre-training budget (Springer et al., 2025), therefore, it is hard to claim whether the observed non-fine-tunability is purely due to the proposed method or just due to differences in the total progress in pre-training. The authors did not report any training curve / validation curve nor test varying pre-training token budgets.
  • Besides, more fundamentally, I am concerning that whether the baseline pre-training itself without mask is valid. If we see the GLUE performance of the pre-trained base model, the performance is surprisingly low (even matching random guessing or the majority-class rate in many cases, e.g., CoLA, QQP, RTE). Is your pre-training done faithfully? I would appreciate if the authors can add a random-init fine-tuning baseline for better comparison.
  1. Lack of supporting evidence. Authorized fine-tuning falls far below the dense baseline (e.g., in QNLI ~64 vs. 79.4), indicating that PMP may just produce a weaker pre-trained model rather than blocking fine-tuning from a good base model. If the learning objective mismatch were the cause, as the authors intended, it should largely recover.
  2. The mask criterion is weakly justified. A global top-k of instantaneous gradient norm differs from the cited Early-Bird criterion and resembles more the pruning-at-initialization scores. Then, why should we specifically use that instantiation instead of already established ones (Frankle et al., 2021; Su et al., 2020). It would be great if Figure 3 could include a comparison against a layer-wise shuffled mask with matched density. Moreover, recent analysis of early training dynamics (Im et al., 2026) suggests early gradients mainly reflect corpus statistics, which raises questions about the authors' 'ticket' interpretation.
  3. Missing related work weakens the novelty claim. There has already been some highly relevant attempts in the task-agnostic, key-based protection line of work: NNSplitter (Zhou etl a. 2023) and secret-key model protection (MaungMaung et al. 2021). Or Self-Destructing Models (Henderson et al. 2023) and Non-Transferable Learning (Wang et al. 2022) that share the motivation. But the authors failed to contextualize their work in this relevant line of work.

Reference

  • Springer et al. 2025, Overtrained Language Models Are Harder to Fine-Tune
  • Frankle et al. 2021, Pruning Neural Networks at Initialization: Why Are We Missing the Mark?
  • Su et al. 2020, Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
  • Im et al. 2026, How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
  • Zhou et al. 2023, NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation
  • MaungMaung et al. 2021, A Protection Method of Trained CNN Model with Secret Key from Unauthorized Access
  • Henderson et al. 2023, Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models
  • Wang et al. 2022, Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorization
Questions for authors

See the weaknesses section above.

By the way, I really appreciate the authors' high-quality writing. This is the only paper that I, as a human, can read smoothly among all the papers I am reviewing.

Reviewer_7BCROpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

After ~500 warm-up steps, the trainer selects 70% of parameters (early-bird lottery-ticket criterion) and continues pre-training updating only those, freezing the rest. The final dense weights are released; the mask stays private. The claim: fine-tuning without the mask must update "settled" and "unsettled" parameters together, which destabilizes it — GLUE fine-tuned average drops 68.65 → 59.55 on a ~1.1B model, while base performance is preserved and mask-holders fine-tune normally. Unlike Sophon/RepNoise/Vaccine, no knowledge of downstream tasks is required.

Critical strengths
  1. Clean, under-explored problem. Task-agnostic non-fine-tunability at pre-training time, clearly separated from task-specific defenses.
  2. Simple, fully specified method with the right ablations (random vs. early-bird mask, mask ratio, LR, epochs, loss landscapes).
  3. Honest appendix — admits RepNoise suppresses fine-tuning more strongly in the paper's own comparison.
Critical weaknesses

W1 (decisive) — The mechanism that powers the defense is also the leak.

The defense works because the trained 70% and the frozen 30% behave differently: the trained part is settled and barely wants to move; the frozen part is undertrained and still wants to move a lot (Assumption 1, Fig. 5, Fig. 6). This difference is easy to measure from outside:

  • An attacker with only the released model and any public web corpus pushes a few hundred batches through the model and records how much each parameter wants to change (gradient energy). The frozen parameters rank at the top — the private mask is recovered. The attacker does not need to know the 70% ratio; a cutoff sweep replaces it.
  • An even simpler attack needs no probe data: fine-tune normally and watch which parameters move. The paper's own Fig. 6 caption admits the frozen parameters carry a visible signature ("near-zero gradients predominantly correspond to..." one group).
  • §5.2 only rebuts an attacker who looks at gradients once. Averaging over batches removes exactly the noise it relies on.

The dilemma: either the two parameter groups differ measurably — then the mask is recoverable; or they do not — then updating the frozen group is harmless and the defense does nothing. There is no third option.

Scale makes this worse, not better: in the paper's barely-trained regime (~0.16B tokens) the two groups are at their most similar; on a real foundation model the trained part is more settled and the signature sharper.

"An adversary cannot effectively recover the private mask" (§5.2) is therefore unsupported. The claim requires (a) measured mask-recovery accuracy under realistic probes and (b) fine-tuning results under partially recovered masks.

Questions for authors

Major:

  1. Run the recovery attack yourselves: report mask-recovery accuracy using gradient statistics computed on public web data, and report fine-tuned performance when the attacker uses a partially (50–90%) recovered mask.
  2. Reconcile §5.2 with Fig. 6: how is the mask undetectable if your own figure shows the frozen parameters carry a distinct gradient signature?

Minor:

  1. No pre-training loss or perplexity appears anywhere in the paper. For a pre-training intervention this is the basic sanity check — report it.
  2. State the unstated assumptions: (i) no pre-mask checkpoint ever leaks — one leak reveals the mask by subtraction; (ii) distillation — the released model is fully capable by design, so copying it into a student model bypasses control entirely. The latter applies to all compared defenses equally, but it belongs in the threat model.
  3. Add a 'PMP + authorized FT' variant to Table 1, for both models and all tasks to show the effect of ~30% unused params.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.