acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Accuracy Misses What Linear Probes Still Infer from Anonymized Text

Abstract

Large language models infer personal attributes from online comments with near-human accuracy, and text anonymizers rewrite the comments to prevent such inference. Anonymizers are evaluated by adversarial accuracy, the share of attributes that an LLM attacker still infers correctly. Most of them also use such an attacker while rewriting, so a lower accuracy may show that the attacker's answer has changed rather than that the attribute has left the text. We therefore examine the text with an adversary outside the rewriting loop, a linear probe trained on other authors' labeled raw text. On SynthPAI, six published anonymizers remove up to all of the attacker's advantage over a majority-class guess, but at most 15% of the probe's advantage over chance. The gap remains for retrained probes and a fine-tuned classifier, and a frozen probe finds it on real Reddit comments. Controlled edits on SynthPAI suggest why: much of the remaining signal lies in register, how a comment is written rather than what it states. To narrow the gap, Probe-Guided Register Selection (PGRS) rewrites each comment in several registers and publishes the rewrite the defender's own probe is least certain about. PGRS lowers adversarial accuracy more than most published anonymizers, at a moderate utility cost. PGRS lowers frozen-probe AUC from 0.76 to 0.51 on three probe models, two of which take no part in selection. Because it selects against a frozen probe, we also report a probe retrained on its output, which reaches 0.56-0.60 against 0.73-0.77 for the published anonymizers. We therefore recommend reporting probe AUC, frozen and retrained, alongside adversarial accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
text anonymization, personal attribute inference, privacy, linear probing, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_cgAcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that text anonymizers are mostly evaluated in terms of adversarial accuracy, the share of attributes an LLM attacker can still infer, with many of the approaches using the same attacker within their rewriting looping. A low accuracy may therefore mean that the attacker was misled and not that the attribute is removed. To test this, a linear probe on a small open model's hidden states is used to score author-level AUC. Empirical results show that the two measures diverge, i.e., the anonymizers cut attacker accuracy significantly, yet frozen-probe AUC stays close the raw text. PGRS, the proposed defense, generates 8 register-directed rewrites and publishes the one with the highest frozen-probe entropy. With this defense, AUC drops significantly.

Critical strengths
  1. Proposes a new well-motivated evaluation measure with empirical demonstrations. The authors provide convincing evidence for the gap in evaluating text anonymizers only in terms of adversarial accuracy. For example, IncogniText illustrates how adversarial accuracy can mislead: roughly half of the attacker's first guesses hit the planted wrong value, driving accuracy below the majority prior while probe AUC remains near its raw-text level. RLAA shows the same gap with minimal edits. Extending Elazar & Goldberg (2018) to text anonymization, the paper offers an actionable recommendation to report both frozen and retrained probe AUC.
Critical weaknesses
  1. PGRS's practical advantage is unclear. The authors acknowledge that frozen-probe AUC overstates protection and that retrained-probe AUC is the meaningful test. However, the ablations favor a simpler alternative: applying one fixed register to every comment achieves retrained AUC 0.630 at utility 0.868, versus PGRS's 0.582 at utility 0.758. Probe-guided selection thus gains only 0.05 AUC at a cost of 0.11 utility. The claim that PGRS improves protection "without a loss on adversarial accuracy or utility" holds only against FgAA (e.g., its utility is substantially below RLAA). Moreover, the retrained result may be optimistic: both probes are trained on the same development authors, allowing the adversary to learn from outputs selected in-sample to confuse a probe trained on those authors.
Questions for authors

Q. Given Table 3, why should practitioners prefer full PGRS over fixed-register rewriting, which is cheaper and has higher utility at a similar retrained AUC? Can you show the privacy-utility frontier under the retrained probe?

Reviewer_M5VTOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The submission's goal is to target the challenge of PII data protection from scaled inference with more revised metrics to evaluate the efficiency of anonymization methods through using a linear probe trained on raw text from other authors. The work presents an argument, stating that attribute inference is connected to writing style features, rather than explicit content leakage, supported by experiments on SynthPAI dataset, where current anonymizers only marginally reduce the probe's ability to infer attributes. In this paper, the authors introduce a new method called Probe-Guided Register Selection (PGRS), which rewrites each comment in various common register style types and then selects the version which makes the trained probe the least certain. PGRS shows strong positive results compared to other existing anonymization methods by reducing probe AUC, even when the probe is retrained on the anonymized data, covering a large variety of existing alternative anonymizers and highlighting their strengths and weaknesses with the new metric; however, the general picture is universal for all previous methods which probe AUC being low. The authors raise a valid argument, that the future assessment of anonymizer quality should also include probe AUC (either frozen or retained) in addition to the usual adversarial accuracy to provide more robust and explainable evaluation.

Critical strengths

S1. The work is well-motivated - PGRS highlights a critical gap in evaluation of anonymizer efficiency, shedding a more clear picture and how anonymization and attacker inference work, with signals hinting towards style rather than content influencing this.

S2. The idea and results and clear and well-presented in text and visual forms; PGRS impact is noticeable and justified with probe AUC usage. The register style rewriting of comments is a fresh and useful idea, covering a new aspect of deanonymization logic.

S3. Paper method and code are reproducible, the dataset is openly available, unlocking transparent analysis of existing anonymizer perofrmance.

Critical weaknesses

W1. PGRS is not highlighted enough as a contribution - it is a strong finding and deserves to be well-presented in contribution subsection.

W2. Choice of models lacks justification (using two very old options which have newer versions available, e.g. mistral and olmo) and attacker/judge model selection is also not well-ablated.

W3. The claim that inference is mainly driven by style attributes and not PII leakage is not supported enough; for example, provided example in Case 1 Appendix H seems hard even for a human to infer high income in real life, while PGRS changes mainly the register and the probability drops drastically, there is not connection between which changes in style cause this and the work would greatly benefit from exploring this area more.

Questions for authors

Q1. Did you try other datasets than SynthPAI? There have been more recent synthetic datasets which can have better writing quality (e.g. thanks to using newer and stronger models) - do you think this could play a factor?

Q2. Did you see a potential reason for LLM as attacker "efficiency" due to the fact that it infers personal attributes not only from contextual hints, but also from stereotypical assumptions about stylistic quirks matching particular personal traits? The work seems to lead into this track and can open a new interesting direction (connected to W3).

Q3. Are distributions of development and test set authors different? How different is the average representation of such sets? How many comments (not authors) in each split? In SynthPAI, different authors had different number of comments, which can skew the text proportion between dev and test.

Q4. "IncogniText shows most clearly why the two measures can disagree. It rewrites each comment toward a wrong attribute value, and about half of the attacker’s first guesses land on this planted value, which pushes adversarial accuracy below the majority-class prior" - i find this statement very insightful, can you please give some specific qualitative example for visual purposes to support the experimental details from Appendix C.3 Table 6? SPIA (Subject-level Inference for Realistic Text Anonymization Evaluation, https://arxiv.org/abs/2604.21211) partially covers that.

Major (addressing these would help raise the score from my side): W2, Q2/W3, Q3 above; I understand that W2 may require additional experiments, but addressing this point would really help solidify the experiment setup.

Minor (does not influence the score, only persona recommendations): Claude generated figures with non-symmetric features and misaligned parts; I recommend to rephrase "PGRS brings the frozen probe close to chance" naming, shows unclear wording to non-native English speakers

Reviewer_6BDqOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • Paper argues that the 'adversarial accuracy' metric commonly used in the anonymization literature is not reliable. This is because anonymizers optimize against the same kind of LLM attacker that is used to score them downstream.
  • While anonymizers fool the LLM attacker, giving a perception of privacy, there is barely any difference in linear classifier probes before and after anonymization.
  • Authors propose a novel anonymization procedure called PGRS. PGRS first rewrites each comment in 8 different tones (casual, formal, etc.) and selects the rewrite that minimizes the certainty of the defender's internally maintained linear probes.
Critical strengths

The paper's main strength is showing a blind spot in how anonymizers are evaluated in the literature.

  • Demonstration that IncogniText drops attacker accuracy from 0.75 to 0.18, while probe AUC only goes from 0.76 to 0.72 is the really important result!
  • PGRS is cheap, fully local and can be added in posthoc manner on top of any existing anonymizer. To make sure evaluation is unbiased, authors also report retrained-probe AUC for their own method.
Critical weaknesses

The paper's main weaknesses are synthetic data and utility cost.

  • Most results are on SynthPAI, which is generated by LLM personas. what is more concerning is that gains are more in synthetic data and not big in on reddit, which is real human text.
  • Utility significantly drops Utility drops to 0.76 on SynthPAI and 0.53 on Reddit.
Questions for authors
  • I view paper's main contribution as the demonstration of a blind spot, and the defense is not particularly convincing (primarily due to the experiments). Can the authors conduct more experiments using real text?
  • Is there any human evaluation of utility, especially of whether users would accept the register changes?
Reviewer_DZu9OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a key point in attribute inference: texts optimized on adversarial LLMs may still contain personal information that is identifiable by another classifier. To counteract this, the authors present PGRS, which rewrites each comment in several registers. This method lowers frozen-probe AUC, demonstrating its performance.

Critical strengths
  • The gap in evaluation makes sense.
  • The comparison is comprehensive, including five peer methods and multiple models.
Critical weaknesses
  • In the experiments, full PGRS scores 0.76 on the utility judge, while simple "Abstraction" alone scores 0.89. While the authors propose PGRS-U to balance this, there is an unavoidable tension: the more you alter the writing style to confuse the probe, the further the text drifts from the author's original voice and utility.
  • There is still room for improvement in the paper's writing. Some expressions are not easy to understand.
Questions for authors

My biggest concern with this paper is the trade-off. As the method successfully defends the attribute inference, it also undermines the utility. Does this field have some shared understanding regarding what constitutes an acceptable utility-privacy trade-off? For instance, could you provide some examples comparing the pre-rewriting and post-rewriting versions of the proposed method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.