acceptodds
Withdrawn from ICLR 2027

Surprise Masking: Private Data Filtering with Model Likelihoods

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data useful for language model training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple form of perplexity filtering that uses a publicly available reference model to identify unwanted spans and suppress learning from them while preserving the rest of each example. On synthetic code benchmarks, we show SM improves the privacy-utility tradeoff over standard masking methods and PII filtering. In fine-tuning experiments, SM reduces the recovery of private values from 66.3% to 2.3% and the reproduction of stylistic coding quirks from 23.6% to 1.3%, while preserving performance on the evaluated coding tasks. SM is similarly useful in pretraining. We also evaluate SM for sanitizing datasets by directly removing unwanted content. Here, SM outperforms DePA, a perplexity-based defense for detecting poisoned dead code, while specialized detectors perform better on broader credential and PII benchmarks. Across eleven settings, we find that SM's downstream success is predictable from the separation of unwanted and useful content in the distribution; particularly, we find it particularly useful with code data. Ultimately, SM is an inexpensive, adaptable method to preserve valuable examples while reducing the learning of unwanted information.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy, perplexity, data filtering, log likelihood, data sanitization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vTqgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Surprise Masking (SM), a simple to prevent language models from learning private or unwanted content. Specifically, a public reference model is used to score token windows by their mean negative log-likelihood. Tokens with the score higher than a given threshold are masked out of the training loss but stay visible as context. Variants include global and local thresholds, prompted scoring, consensus voting, and duplicate propagation. The effectiveness of SM is empirically validated with comprehensive experiments on different datasets and tasks.

Critical strengths

The method is cheap, simple, and needs no trained detector. In the code settings it improves the privacy–utility tradeoff by a large margin, including on stylistic quirks that PII scrubbers cannot catch. The paper reports its failure cases honestly. The distributional AUROC analysis gives practitioners a concrete way to decide in advance whether SM will work.

Critical weaknesses
  • The novelty of the paper is limited. Perplexity-based token masking and filtering already exist (Wu et al., 2025; DePA; Goldfish), so the contribution is mostly a new application and a broader evaluation.
  • The experimental setup is narrow and artificial where the main results use synthetic data, a single coding library, small models or LoRA and a fairly specific prefix-completion attacker.
  • The proposed method requires a chosen threshold and window length. To better choose those parameters, a labeled, in-distribution validation set are needed, which may not be practically available and threshold transfer across domains is inconsistent. SM gives no formal privacy guarantee.

Minor issues:

  • There is a broken references (“??”) at line 272;
  • A duplicated sentence is written in the Figure 3 caption.
Questions for authors
  • How does SM differ conceptually from earlier perplexity-based token masking (Wu et al., 2025) and from DePA, beyond being applied to privacy? What part of the method do you consider new?
  • In Code PII, each private value appears in 100 chats. How does SM perform when a secret appears only once or a few times, which is more common in practice? Have you tested SM on any real (non-synthetic) codebase?
  • SM’s thresholds are tuned on a labeled, in-distribution validation set. How should practitioners build one when labels are not available? How well does SM work with a fixed default threshold or an unsupervised selection rule?
Reviewer_V9beOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Surprise Masking (SM), which uses a reference model’s conditional likelihoods to identify potentially private or unwanted spans and exclude them from the training loss. It combines this signal with window-based selection and duplicate propagation. Experiments on synthetic code show reduced private-value extraction and stylistic imitation while preserving evaluated task utility. Additional sanitization experiments illustrate both the method’s usefulness and its limitations on broader PII detection.

Critical strengths
  1. The paper addresses a practical problem: retaining useful training examples while reducing the learning of unwanted content. Evaluating downstream extraction and task performance, rather than detection alone, makes the reported improvements meaningful.

  2. The study includes useful baselines and acknowledges its limitations. In particular, the weaker results on broad PII detection help clarify where surprise-based masking may be appropriate.

Critical weaknesses
  1. The effect of context on the masking decisions could be better characterized. Since surprise is conditioned on preceding tokens, private content unseen during reference-model training may become easier to predict after related information appears in the current example. I appreciate that iterative duplicate propagation can recover matching subsequences within the same group, so a lower score for a repeated occurrence does not necessarily imply a missed mask. However, it remains unclear how well the complete method handles cases without a suitable initial match or where private context supports reconstruction without matching token subsequences. A targeted analysis would help establish whether this affects final coverage or downstream extraction.

  2. The methodological advance over prior perplexity-based masking appears incremental, making the empirical contribution particularly important. The strongest training results are concentrated in synthetic code and utility evaluation on one library, as the authors acknowledge. Appendix E.1 usefully compares interventions with a fixed SM selection, but I did not find an ablation isolating duplicate propagation’s contribution to that selection. Clearer component attribution and broader evidence of utility preservation would strengthen the contribution. These limitations leave me leaning toward weak rejection.

Questions for authors
  1. [Context dependence] Do you have results separating first and subsequent occurrences of private values, or comparing the same value under different private contexts? It would be helpful to distinguish changes in surprise scores from changes in final mask coverage after propagation, and, where available, downstream extraction.

  2. [Component attribution] Could you report the effect of removing duplicate propagation at comparable masking budgets or collateral masking rates? Appendix E.1 compares interventions using a fixed selection; my question concerns how propagation contributes to the selection itself. Please also clarify how identity/user groups would be obtained in practice.

  3. [Utility] Beyond the acknowledged single-library limitation, is there evidence that SM preserves learning of useful content unfamiliar to the reference model, such as rare APIs or project-specific conventions?

  4. [Protocol clarification] Appendix C.1 states that Code PII pretraining begins with an unmasked phase that includes Code PII tokens. What is the extraction rate before the subsequent masking phase, and how should this prior exposure affect interpretation of the results?

Reviewer_Ln3TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to reduce the learning of private information from useful training data. The main approach is to use a public reference model to score and mask token spans by negative log-likelihood, based on the intuition that private content should be unfamiliar to the reference model and therefore more surprising. The paper considers two settings: selective training, where selected spans are removed from the training loss but remain in the attention context, and direct sanitization, where selected spans are removed from the data. The evaluation of selective training focuses on memorization after training, using synthetic code datasets with planted private values or user-specific coding styles, while also measuring coding utility. The evaluation of direct sanitization focuses on privacy-oriented data filtering, testing whether surprise can identify credentials, broad PII, and poisoned code. The results show strong privacy–utility tradeoffs on the synthetic code tasks and in settings where private and non-private content are well separated by surprise.

Critical strengths
  1. The paper studies a practically relevant problem of reducing privacy leakage during model training.
  2. The method is simple and computationally lightweight.
  3. The paper provides a broad empirical evaluation across selective training, direct sanitization, and multiple privacy-related benchmarks.
Critical weaknesses
  1. The key intuition (line 064) that private information should be surprising to a public reference model is not sufficiently justified. Private information may not always consist of unfamiliar or low-likelihood tokens; it can also arise from novel combinations of common and predictable content, such as an undisclosed medical condition expressed in ordinary language. Conversely, unusual but non-private content may also receive high surprise scores (e.g., an unusual variable name in a codebase). It's unclear how reliably reference-model surprise can serve as a proxy for privacy.
  2. The selective-training evaluation in Section 3 is based on a synthetic setup that may favor SM. The experiments inject artificial private values or user-specific coding styles into normal code; these injected attributes are likely to be surprising to the reference model and contribute little to the coding utility being measured. This makes the private attributes relatively easy to separate from the useful task signal, so the strong privacy–utility tradeoff may partly reflect the dataset construction.
  3. The privacy evaluation of selective training is relatively narrow. In Appendix B.1 (line 862), the attacker “receives only a prefix of the private value, without its training document, surrounding code, or synthetic identity,” and succeeds if the generated suffix matches the hidden target. This setup mainly tests completion of highly distinctive private strings, and is less representative of natural memorization than the evaluation used by the baseline Goldfish (Hans et al., 2024), which conditions on surrounding text and tests whether the model continues the memorized training content. It would also be useful to evaluate broader privacy risks such as semantic memorization and membership inference as tested in Goldfish.
  4. The technical contribution of the core method appears limited. Selecting high-perplexity tokens with a reference model and excluding them from the training loss closely resembles Wu et al. (2025), while loss masking to prevent memorization of specific content has also been studied by Kosireddy & Lucas (2025). The proposed method appears to be a relatively incremental extension of existing token-selection and loss-masking techniques.
  5. The practical deployment of SM relies on labeled validation data, and its performance is sensitive to threshold tuning. The recommended protocol (line 434) selects the threshold on an in-distribution validation set with private-content annotations, which may be difficult to obtain in practice. Lines 324–327 also show that threshold transfer is inconsistent across distributions. This makes it unclear how reliably SM can be calibrated when representative labeled validation data are unavailable.
Questions for authors
  1. How sensitive are the selected spans and downstream privacy–utility tradeoffs to the choice of reference model? The main selective-training experiments use GPT-OSS-20B as the reference model, so it would be useful to know whether other reference models, such as Qwen or Llama, produce consistent surprise scores and masks on the same data spans.
  2. What is the benefit of retaining selected private spans in the attention context? In Table 6, masking both the loss and attention achieves similar privacy and utility to masking only the loss. Would directly redacting the same SM-selected spans before training provide a comparable or better privacy–utility tradeoff?
  3. Why are the stronger privacy detectors evaluated in the direct-sanitization setting not also compared in the selective-training experiments? Section 4 shows that SM underperforms detectors such as CredSweeper, OPF log-odds, and a fine-tuned DeBERTa model (lines 322 and 360–361). How would the privacy–utility tradeoff compare if these detectors were used to identify spans before training and generate the training masks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.