acceptodds
Under review as a conference paper at ICLR 2027

RAISE: Post-Training Membership Inference Mitigation via Model Editing

Abstract

Membership inference attacks (MIAs) pose privacy risks to fine-tuning data. After fine-tuning, a privacy audit may reveal that particular fine-tuning records carry strong membership signals. Yet applying training-time defenses at this point would require rerunning fine-tuning. We present a novel post-training approach that formulates MIA mitigation as a model-editing problem. Given selected member records, our approach, RAISE (eference-ligned nference-ignal diting), directly edits the fine-tuned checkpoint without rerunning fine-tuning. Our analysis shows that the membership signal is concentrated in a small number of token positions with unusually low losses relative to non-member references. RAISE localizes the corresponding spans and uses constrained checkpoint editing to align their losses with ranges derived from non-member references, while limiting changes to unrelated model behavior. Across six domains and ten MIAs, RAISE reduces mean attack AUC from 0.855 to 0.576 and mean TPR at 1% FPR from 25.1% to 2.8%. It also outperforms the evaluated post-training baselines, generalizes across five model configurations, and remains effective against adaptive attacks (AUC below 0.58), while retaining substantial fine-tuning utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership Inference, Post-Training Defense, Model Editing, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9n5bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAISE is a post-training defense intended to reduce membership signals for designated fine-tuning records without rerunning the original fine-tuning procedure. It localizes spans with unusually low reference-relative loss, selects reference-quantile loss targets, and applies constrained model editing. The main experiment evaluates ten attacks across six domains and reports a mean AUC reduction from 0.855 to 0.576 and TPR at 1% FPR from 25.1% to 2.8%. Additional experiments examine localization, write layer, protection coverage, other model configurations, and trimmed-loss adaptive attacks.

Critical strengths
  1. Protecting a selected set of audited records in an existing checkpoint is a useful practical setting. The paper states that the protection is empirical and limited to the designated records, and reference-relative localization is a plausible design that receives supporting ablations.

  2. The primary before/after effect is substantial across six domains and ten attacks on the same evaluation records. Reporting per-attack/domain results, utility measurements, bootstrap variability, and additional model configurations provides more evidence than a single aggregate attack score.

Critical weaknesses
  1. The comparative evidence does not yet establish how the defense's privacy–utility advantage varies across operating points. Table 2 gives one point per baseline on Pile-CC. NPO and RAISE have nearly matched perplexity (12.58 versus 12.59), which supports RAISE at that operating point. However, baseline hyperparameters and matched-utility sweeps are insufficiently documented; other comparisons, such as Forget-CE at perplexity 11.76, also change utility cost. A retrain-without-the-protected-set reference would calibrate both residual membership and utility in the small evaluated fine-tuning setup; it need not be a deployable defense to be useful as a reference. The judge-based utility experiment also needs its judge identity, prompts, question set, and variability. Without these details and a fair operating-point comparison, the large primary effect is promising but its advantage over existing post-training interventions is uncertain.

  2. Several evaluation and specification gaps affect the broader protection claims. Table 9 includes below-chance post-edit AUCs: the Llama-3.2-3B SPV-MIA AUC 0.4063 corresponds to 0.5937 under score reversal. Applying max(AUC, 1−AUC) separately to its ten attacks gives mean 0.5441, compared with the reported one-sided mean 0.4989. This is an orientation-invariant diagnostic, not an operational attacker selected using test labels; an attacker should choose orientation using independent auxiliary data. Importantly, this issue does not overturn the main Table 1 results or by itself falsify the stated below-0.58 point estimates for the trimmed adaptive attacks, which remain below that threshold after reversal. It does require qualification of cross-model “near chance” interpretations and broader defense claims. Separately, Table 1's GitHub mean 0.625 and Table 7's selected-layer result 0.5865 need a run/configuration reconciliation, and the hidden-state objective and writeback specification need enough detail to reproduce the intervention.

Overall assessment: I currently lean against acceptance, narrowly. The main reduction is promising and the problem is useful, but fair privacy–utility comparisons, operationally calibrated attack evaluation, and reproducible intervention details are needed to assess the claimed advantage. Addressing these points could change my recommendation.

Questions for authors
  1. What privacy–utility curves do the baselines achieve under documented tuning, and how does RAISE compare at matched utility? What does a retrain-without-the-protected-records reference show?
  2. Please report both score orientations, with attack orientation/calibration chosen on independent data. Was the learned Ensemble attack trained or recalibrated on defended-model features? How does it perform as an adaptive attacker?
  3. What explains the GitHub 0.625-versus-0.5865 results? Please identify the run, evaluation population, default settings, and selected write layer for each.
  4. Given the module families specified in Appendix B.6, please identify the exact edited layer and its relationship to the target-search layer, and specify the complete hidden-state objective, request batching, preservation keys, and parameter-selection procedure. Appendix C.3 states that MIA outcomes were not used to choose the write layer; were evaluation attack outcomes used to tune other settings?
  5. Are the main evaluation non-members disjoint from the defender/reference pools and attack auxiliary data? Please provide the judge configuration, actual editing cost, and uncertainty across fine-tuning/edit seeds, beyond record bootstrap alone.
Reviewer_vXUxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAISE, a post-training method to reduce membership-inference risk for selected fine-tuning records without retraining. It observes that the member vs. non-member loss gap is concentrated in a few token spans. For each such span, it sets a target loss range taken from same-domain non-member data, then writes the change into the checkpoint with a ROME- or MEMIT-style constrained edit. On six MIMIR domains with Llama-3-8B and ten attacks, average AUC drops from 0.855 to 0.576 and TPR at 1% FPR drops from 25% to 3%. It outperforms NPO, Forget-CE, and logit noise, at some utility cost.

Critical strengths
  1. The results are strong and broad: ten attacks, six domains, five models, and an adaptive-attack section. Even strong reference-based attacks such as SPV-MIA drop to near chance.

  2. "Align to non-member behavior instead of just pushing the likelihood down" is a sensible idea, and the span ablation (vs. random and lowest-loss selection) supports it.

Critical weaknesses
  1. The novelty over unlearning is thin. "Make selected members indistinguishable from non-members under MIA" is how empirical unlearning is usually evaluated, and the baselines are unlearning methods. The idea of calibrating to non-member or retrained behavior instead of over-suppressing is already discussed in the unlearning literature. The paper does not explain clearly what problem it solves that unlearning does not. Moreover, if a record really becomes indistinguishable from a non-member, it is unclear in what sense its useful contribution is "kept."

  2. The evaluation does not fully support the claims. Each domain uses 100 protected members and 100 non-members, so TPR at 1% FPR rests on a single non-member. AUC near 0.5 is reported under a fixed score direction, so over-correction (members now look less familiar than non-members) could be hidden. The effect on unprotected members (the onion effect) is not reported separately, and there is no retrain-without-the-records reference. Utility loss is not small either: the judge score drops from 7.39 to 6.33, and perplexity is worse than Forget-CE.

Questions for authors
  1. What is the difference in problem definition between RAISE and unlearning the selected records, and why would retraining without them not be the right reference?

  2. Can you report results with larger evaluation sets, and attack AUC in both score directions?

  3. Does editing the 100 most vulnerable records make the remaining, unprotected members easier to attack?

Reviewer_mWZCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training mitigation of membership inference risks in fine-tuned LLMs. Given a set of known members to protect and non-members from the same distribution, the proposed method, RAISE, identifies spans with unusually low token losses relative to the references and sets target loss intervals based on reference quantiles. It then uses constrained model editing to adjust an MLP projection toward these targets without repeating fine-tuning. Experiments across six domains, ten MIAs, and five models show that RAISE substantially reduces attack performance on protected records while preserving utility.

Critical strengths

S1. Reducing membership risk directly at an existing checkpoint, without retraining or serving-time components, is a well-motivated problem.

S2. Casting MIA mitigation as localized model editing is an interesting idea, and the pipeline is clearly described.

S3. The experiments cover multiple domains, model families, and attack families, with ablations of the main design choices.

Critical weaknesses

W1. The evaluation does not establish robustness against stronger attacks. RAISE directly adjusts the token losses used by all evaluated attacks, so their reduced effectiveness is largely expected. The evaluation lacks attacks that exploit signals beyond the edited spans or adapt specifically to the defense. For example, comparing local losses [1] against the target's own pretrained model, using shadow models, or training a defense-aware classifier could reveal remaining membership signals. These attacks are particularly relevant because RAISE modifies only selected spans rather than removing membership signals throughout each record.

W2. RAISE adjusts protected records toward the loss distribution of other non-members, which may overshoot their expected losses had they never been used for training. Indeed, several attacks achieve AUC below 0.5, suggesting that reversing their scores could still distinguish protected members. Moreover, if the original checkpoint is available, comparing it with the edited model may reveal which records were protected.

W3. The experiments do not establish the practical advantage of post-training editing. RAISE is motivated by avoiding expensive retraining, yet experiments use relatively small LoRA fine-tuning tasks, and the editing cost is not reported. The most natural baseline, retraining without the protected records, is missing, especially given RAISE's utility loss and limited protection coverage. Existing post-training baselines are not compared at matched utility. Finally, the small evaluation sets make low-FPR attack results unreliable, since TPR at 1% FPR depends on only one or two non-member scores.

[1] Window-based Membership Inference Attacks Against Fine-tuned Large Language Models. USENIX Security 2026.

Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.