acceptodds
Under review as a conference paper at ICLR 2027

LLM Anonymization Against Agentic Re-Identification

Abstract

Agentic LLMs with web search change the threat model for text anonymization: weak contextual cues can become cross-referenceable evidence for re-identification, yet those same details also carry downstream analytic value of the text. Existing defenses either remove explicit identifiers, perturb text for formal privacy, or test rewritten text against non-web inference models, leaving underexplored the operating region between resistance to agentic web-search re-identification and utility retention. We introduce AURA (Anonymization with Utility-Retention Adaptation), an LLM-powered mask-reconstruct framework that decouples privacy localization from utility-preserving reconstruction and selects candidates with adversarial privacy and utility-retention checks. We evaluate AURA on real-user interview transcripts using re-identification attacks carried out by web-search agents, along with a utility evaluation based on interviewee-profile facts, codebook facts, and the joint contextual utility grid. Our results show that adaptive-scope AURA yields the lowest agentic re-identification counts under each of three attacker models among the non-DP methods, and that at matched scope and backbone, AURA's mask-reconstruct design retains more contextual utility than the prior LLM anonymizer (+6.4 pp unit-grid recovery) at comparable privacy Source Code: https://anonymous.4open.science/r/AURA-2AC1.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
text anonymization, privacy, large language models, LLM agents, re-identification, attribute inference, privacy-utility tradeoff

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_JUpKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AURA, a framework for anonymizing interview transcripts against re-identification by language models with web-search access. The method separates identifying sensitive vulnerable passages from reconstructing them while preserving useful semantic information. AURA establishes a privacy scope, masks potentially identifying details, and generates alternative replacements for those passages, rather than direct application of the masks. It then selects among these candidates using checks for attribute leakage, specificity, and information loss. They also ablate heavily on the types of privacy parameters used, as well as go deep into ablations on the pareto tradeoff between privacy and utility.

The authors evaluate AURA on 53 interviews from 1,250-transcript Anthropic Interviewer dataset. They select these based on for which samples did an initial attack produced verifiable identification evidence. They compare it with iterative LLM anonymization, one-shot rewriting, entity redaction, and DP-MLM, using three web-enabled attacker models (GPT-5.1, GPT-5.4-mini, and Gemini-3-Flash). Adaptive AURA achieves the lowest re-identification counts among the evaluated non-DP methods while preserving substantial measured utility. With the model and privacy scope held fixed, AURA improves contextual semantic utility recovery over the prior iterative anonymizers by 6.4 percentage points. Human evaluations provide additional evidence about readability and factual faithfulness, although they are in the appendix.

Critical strengths

1. The method is clearly motivated and extremely well explained. I particularly loved the separation between finding identifying details and deciding how to rewrite them to preserve semantic intent. This directly addresses the difficulty that the same information can be both identifying and important to understanding an interview, this tradeoff is very nuanced. The comparison with the prior iterative anonymizers, holding the model and privacy scope fixed, provides evidence that the reconstruction stage helps preserve utility.

2. The privacy–utility analysis makes a strong case for the approach. Web-enabled attackers represent a realistic threat, and evaluating several attackers and baselines make the results more informative. I appreciated that the DP-MLM comparison shows the trade-off between stronger observed privacy protection and information loss at lower epsilons. AURA occupies a useful region of this trade-off, and the human evaluations in the Appendix provide supporting evidence beyond automated scores.

Critical weaknesses

1. Preserving factual correctness does not necessarily preserve qualitative meaning. The paper does address faithfulness: Section 5.2 reports that 86.9% of retained rewrite pairs were judged faithful, and the appendices discuss factual distortions. However, this check focuses on whether a rewrite introduces false information. A rewrite could avoid false statements while losing a qualification, relationship, or contextual semantic detail that changes how someone can interpret the participant’s views. I would like this distinction to be more closely connected to the main utility argument on tradeoffs.

2. The selected sample leaves uncertainty about the overall utility of the pipeline. The evaluation uses 53 previously identifiable interviews. I understand this choice, since evaluating only cases that an attacker already cannot identify would provide little evidence about the defense. However, this also leaves open how AURA behaves on less identifiable interviews. Does it recognize that little intervention is needed, or does it still remove useful context? Differences across interview topics and types of identifying information would also help explain how broadly the reported pareto tradeoff applies.

Questions for authors

The questions below are listed in order of importance to my assessment:

1. Can you distinguish necessary information removal from avoidable loss of meaning? Did the human evaluation identify rewrites that introduced no false information but changed the interpretation of a participant’s views? Connecting these cases to the utility scores would help clarify what those scores capture. It also helps strengthen the notion of "faithfulness". If possible can this ablation be tied into the main results also using the human evaluation results from the Appendix J.

2. How does AURA behave when the initial attacker cannot identify the participant? Does the pipeline generally leave these interviews unchanged, or still mask and reconstruct substantial passages? A small comparison of the amount of text changed and the resulting utility retention would help address the concern about unnecessary intervention.

3. Do the results vary across interview topics or types of identifying details? For example, are interviews linked through a specific source affected differently from interviews linked through several biographical clues from the text? A breakdown within the existing samples would help explain where preserving useful context is most difficult. Do the identifying details being scrubbed here destroy semantic meaning if the biographical clues are ruined?

The main theme of the questions above aim to clarify when AURA removes identifying details while preserving the meaning that makes a de-anonymization useful. Is there basically an idea of abstention here?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JUpK, Part 1 of 2: Qualitative Meaning vs. Faithfulness, and Necessary vs. Avoidable Loss

We thank Reviewer JUpK for connecting anonymization to qualitative interpretation. We address W1 and Q1 together below, and W2, Q2, and Q3 in Part 2.

W1 and Q1: Factual Faithfulness vs. Qualitative Meaning, and Necessary Removal vs. Avoidable Loss We agree with this distinction. Appendix J's human check, which found 86.9% faithfulness across 122 resolved tasks, tests whether rewrites introduce false factual claims. However, a rewrite with zero false claims can still drop qualifiers, agency, or context that shift how a researcher interprets a participant's views.

Experimental setup for segment-level interpretive shift: Human experts annotated 773 excerpts across the 53 interviews using a 13-code codebook spanning 5 interpretive dimensions: C1 Trust and Delegation, C2 Interaction, C3 Frustrations, C4 Professional Identity and Skills, and C5 Outlook and Autonomy.

  1. Span alignment and edit bins: Each original excerpt is deterministically matched to the closest 1-to-8-sentence window of $0.5\times$ to $2\times$ length by MiniLM cosine similarity in the same speaker turn of the anonymized text. Excerpts are grouped by word-level difflib similarity $r$ into Unedited with $r>0.98$, Lightly edited with $0.8<r\le 0.98$, and Heavily edited with $r\le 0.8$.
  2. Paired thematic coding: An LLM coder, deepseek-v4-flash at temperature 0, sees the codebook and anonymized transcript, without the original excerpt, and picks the primary code from 4 options: the true code and 3 mid-distance distractors. On originals, this coder matches human experts at $\kappa=0.671$, agrees with qwen3.8-flash on 85.8% of items, and achieves 0.933 repeat self-consistency. Code consistency is the share of excerpts assigned the same code before and after anonymization. A code flip flags a shifted interpretation even when the rewrite is factually faithful.
Method, 773 excerptsOverall ConsistencyUnedited, $r>0.98$Lightly EditedHeavily Edited, $r\le 0.8$Share Heavily EditedC1 Trust / C5 OutlookC4 Identity and Skills
Presidio0.920.930.910.867%0.93 / 0.970.92
Advanced Anonymizer0.910.940.910.8217%0.92 / 0.900.89
RUPTA, ACL '250.880.850.890.8744%0.90 / 0.900.85
AURA fixed 8-attr0.900.910.910.8819%0.93 / 0.970.83
AURA + adapt. privacy0.890.910.890.8520%0.92 / 0.950.83
DP-MLM, $\varepsilon=50$0.71--0.71100%0.780.57

Three findings connect interpretive fidelity to our utility tradeoff and Appendix J:

  1. Span masking localizes risk: Unedited excerpts score 0.91 to 0.94, matching the coder's 0.933 ceiling. AURA heavily edits just 20% of excerpts vs. 44% for RUPTA, and 74% of all AURA code flips occur in edited excerpts.
  2. Losses concentrate in identity-linked codes: AURA keeps non-identifying dimensions near ceiling at 0.92 on C1 and 0.95 on C5, dropping mainly in C4 Identity and Skills at 0.83, where tenure and role qualifiers act as quasi-identifiers.
  3. Tie to main utility scores: Across all 340 profile facts on $N=53$, AURA retains 0.85 of non-identifying general roles and tasks across $n=173$, while suppressing linkable tools at 0.57 across $n=54$, history and locations at 0.44 across $n=18$, and demographics at 0.29 across $n=21$. RUPTA drops to 0.58 even on general roles and tasks.

Manual audit of faithful yet altered interpretations: One author compared side by side all 23 heavily edited flips under AURA and Advanced Anonymizer, and the first 20 of 43 under RUPTA:

Cause of Heavily Edited FlipAURAAdv. Anon.RUPTA
Alignment boundary artifact from an adjacent sentence9124
Necessary entity abstraction with meaning intact, coder noise1077
Genuine meaning shift, faithful but altered interpretation4, all identity-linked4, 3 identity-linked6, stance or workflow lost
Depersonalization, first-person agency turned passive003
  • AURA and Advanced Anonymizer, 4 of 23 each: Both generalized "early in my PhD" across 2 excerpts of one passage and removed "years in professional writing" as the reason for pickiness, a necessary identity generalization. AURA's 4th generalized a specific error example; Advanced Anonymizer's, like RUPTA, erased self-employment.
  • RUPTA, 9 of 20 audited flips: Losses were mostly avoidable and unrelated to identity, e.g., reversing stance from "never give up the human factor" to "depends on how capabilities develop", erasing self-employment and losing the autonomy code, flattening trust evolution, deleting workflows, and 3 depersonalizations.
AuthorsOpenReview ↗

Response to Reviewer JUpK, Part 2 of 2: Less-Identifiable Interviews, Topic Breakdown, and Clue Types

W2 Part A and Q2: Behavior When the Initial Attacker Cannot Identify the Participant Setup: From the 1,197 public Anthropic Interviewer transcripts where our multi-round web-search protocol found no confirmed real-world identity, we drew a length-matched, domain-stratified sample of 10 unidentifiable transcripts covering 6 Science, 3 Work, and 1 Creativity. We extracted 55 validated ground-truth profile facts via the identical pipeline, ran AURA + adaptive privacy, and compared edit rates and utility retention against the 53 identifiable transcripts:

CohortMethodInitial ProbeAdded AttrsWord Edit Rate, Med / MeanUser Turns EditedProfile Retention
10 UnidentifiableAURA + adapt.0/10 identified4 / doc0.060 / 0.0670.760.72
53 IdentifiableAURA + adapt.53/53 identified0 to 4 / doc0.071 / 0.0760.790.73
53 IdentifiableAdv. Anonymizer53/53 identified-0.052 / 0.0580.790.70
53 IdentifiableRUPTA, ACL '2553/53 identified-0.091 / 0.1190.780.51
53 IdentifiablePresidio NER53/53 identified-0.003 / 0.0040.260.99
  1. Why AURA still modify rather than leaving text unchanged: All 1,250 interviews follow the same protocol eliciting projects, workflows, and tools, and being unidentified today does not guarantee safety against future web indexing or acquaintances. On all 10 unidentifiable transcripts, they still inherit concrete quasi-identifiers such as niche subfields, software stacks, employer types, and career stage, adding 4 scope categories.
  2. No excessive rewriting or extra utility loss: Because AURA masks only flagged spans, it modifies just 6.0% of words at the median, which is lower than 7.1% on the 53 identifiable cases and 9.1% for RUPTA, and retains 0.72 profile and 0.74 occupational facts, matching 0.73 and 0.76 on the 53 cases. Moreover, because Step 1 outputs an explicit confidence flag, data stewards can skip dynamic scope expansion whenever that flag is below threshold.

W2 Part B and Q3: Variation Across Interview Topics and Types of Identifying Clues

  • 1. Breakdown by interview topic across $N=53$: Below we compare our main Utility Grid, measuring joint retention of profile and AI-use codebook facts, against downstream coding retention relative to originals across 773 excerpts:
Topic, $N=53$AURA GridRUPTA GridAdv. Anon. GridAA Coding Ret.AURA Coding Ret.RUPTA Coding Ret.
Science, $n=31$0.630.410.690.960.960.90
Work, $n=17$0.780.500.610.950.960.92
Creativity, $n=5$0.930.610.800.980.980.99

Preserving profile context is hardest in Science, where grid utility is 0.63 vs. 0.78 in Work and 0.93 in Creativity, with the same pattern holding in the 10 unidentifiable transcripts. This is because a scientist's specialized research topic is their public identifier in papers and grants, whereas AURA's downstream coding retention stays consistent across topics at 0.96 to 0.98.

  • 2. Specific-source linkage vs. biographical-clue linkage: We audited 10 paired attack trajectories on the same interview and attacker, comparing a run where a baseline such as Presidio was re-identified against a run where AURA prevented re-identification. This comparison reveals two linkage pathways:

    • Specific publication or research source in 7 of 10 pairs, all in Science: Attackers combine 3 to 4 niche technical terms, such as detailed working pipeline, company product description, and art piece details etc. , and injury model, to match a paper, thesis, or lab page directly with zero named entities.
    • Specific project or product plus biographical clues in 3 of 10 pairs across Science, Work, and Creativity: Attackers query a distinctive project or product, such as an open-source tool or design-asset line, alongside role clues to locate a code repository or online store and its owner. Once AURA abstracts these anchors into category-level terms, attackers lose specific search keywords and resort to searching long verbatim quotes of 8 or more words, which accounts for 31% of queries in AURA misses vs. 10% in hits across 1,846 logged trajectories.
  • 3. Does removing these clues destroy semantic meaning? No. AURA retains 96.2% of the expert codes supported in raw transcripts, along with Advanced Anonymizer, non-identity themes remain at 0.92 to 0.95 consistency, and O*NET domain and SOC group retention stay at 0.92 to 0.94. Abstracting overly specific anchors breaks web searchability while keeping qualitative meaning intact.

Thank you for the detailed response and additional analyses. These strengthen my positive assessment.

  1. Interpretive fidelity: The thematic-coding analysis and manual audit connect the utility results more directly to qualitative meaning.

  2. Less-identifiable interviews: The additional comparison addresses my concern about unnecessary intervention. The limited word-level editing is reassuring, although it demonstrates restrained editing rather than automatic abstention.

  3. Topic differences: The breakdown helps explain where the privacy–utility trade-off is hardest, particularly when scientific details both convey useful context and enable identification.

I am maintaining my recommendation of a 3. The response strengthens the empirical case, while the small additional sample and broader generalization of interpretive fidelity remain limitations. Thank you for the careful follow-up.

Reviewer_U67aOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about LLM anonymization and introduces AURA, which decouples privacy localization from utility-preserving reconstruction and selects candidates with adversarial privacy and utility-retention checks. The author conducted extensive experiments on the privacy-utility trade-off and demonstrated promising results.

Critical strengths
  1. This paper presents the first threat modeling and optimization for agent-based web search re-identification.
  2. AURA provides a practical anonymization framework and achieves a balance between privacy preservation and utility retention by introducing mask and reconstruct modules.
Critical weaknesses
  1. The defense baseline used in the experiment is outdated; even Advanced Anonymizer is already a defense method from 2024.
  2. The AURA framework is architecturally complex, containing multi-stage and multi-agent interactions. Can the authors provide the computational overhead and API call costs?
  3. In Table 1, the dataset (consisting of 53 entries) is too small; it fails to demonstrate how the AURA method performs on general transcripts.
Questions for authors

See weaknesses detailed above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer U67a, part 1 of 2: 2025 Baseline RUPTA and Cost Breakdown

We thank Reviewer U67a for the practical feedback. Below we address W1, W2, and W3 in order.

W1: Comparison Against Newer 2025 Defense Baselines, RUPTA at ACL 2025 First, a brief citation note: Advanced Anonymizer by Staab et al. was published at ICLR 2025 following its 2024 arXiv preprint. To compare against the latest 2025 defense, we added RUPTA by Yang et al. at ACL 2025 as requested.

Setup: We ran RUPTA's official PersonalReddit reflexion code with GPT-4.1, 1 initial plus up to 5 reflection rounds while preserving interviewer questions, across all 53 transcripts. RUPTA's DB-bio mode prompts an offline LLM to guess Wikipedia celebrities from memory without web search, mismatching open-web linkage of non-celebrities. All methods are evaluated under the same three web-search attackers, GPT-5.1, GPT-5.4-mini, and Gemini-3-Flash, our profile, codebook, and grid metrics, and a new downstream coding task where deepseek-v4-flash checks whether each of 773 expert-coded excerpts still supports its expert code after anonymization, reported relative to originals:

Method, $N=53$Re-ID % $\downarrow$ 5.1 / 5.4m / Gem | AvgProfile % $\uparrow$Code % $\uparrow$Utility Grid % $\uparrow$Coding Ret. % $\uparrow$
Presidio NER49.1 / 75.5 / 50.9 | 58.598.598.197.197.7
Advanced Anonymizer, ICLR '2520.8 / 22.6 / 18.9 | 20.869.495.967.596.2
RUPTA, ACL '25, new24.5 / 34.0 / 7.5 | 22.049.788.845.691.9
AURA fixed 8-attr, GPT-4.115.1 / 26.4 / 18.9 | 20.177.495.273.296.5
AURA + adapt. privacy, GPT-4.17.5 / 11.3 / 0.0 | 6.373.596.470.496.2
AURA + adapt. hybrid, Qwen-35B3.8 / 13.2 / 3.8 | 6.976.897.575.797.1

RUPTA guards 8 coarse demographic attributes rather than specific research or project web anchors, leaving mean Re-ID at 22.0% vs. 6.3% for AURA, while whole-turn rewrites erase analytical context at 45.6% vs. 70.4% grid utility. AURA Pareto-dominates both 2025 baselines.

W2: Computational Overhead, Latency, and API Cost Breakdown AURA is an offline, one-time pre-release curation pipeline run once before publishing a dataset, replacing 30 to 60 min of manual redaction per interview, with candidate generation and evaluation parallelized. Logged per-transcript calls, tokens, median latency, and API cost across our 53 interviews averaging 1,906 tokens:

Method and BackboneLLM Calls / DocMean Tokens In / OutMedian Wall-ClockMean Cost / Doc in USD
One-Shot Rewrite, GPT-4.11.0~2.2k / 1.9k~12 s~$0.02
Advanced Anonymizer, GPT-4.1, 5 rds10.0~28.7k / 16.1k~95 s~$0.19
RUPTA, GPT-4.1, 1+5 reflexion rds50.9~85k / 22k~140 s$0.21
AURA Fixed 8-Attr, GPT-4.1, 4 init, 10 mask, 4 refill, 16 eval34.0~127k / 60k~3.5 min~$0.74
AURA + Adapt. Privacy, GPT-4.1 + GPT-5.1 probe36 to 39*~165k / 66k~5.0 min~$0.95 to $1.10
AURA + Adapt. Hybrid, Qwen3.5-35B or 27B + DeepSeek36.1~126k / 56k~8.1 to 12.0 min$0.12 to $0.18

*Adds 1 initial probe, 1 scope-expansion call, and 1 check probe, plus 1 retry on ~10% of cases.

With open-weight Qwen3.5 backbones, full adaptive AURA costs $0.12 - $0.18 per interview, or $6 - $9 for all 53, using fewer calls at 36 vs. 51 and lower cost than RUPTA at $0.21.

AuthorsOpenReview ↗

Response to Reviewer U67a, part 2 of 2: Sample Generalizability

W3: Sample Size at $N=53$, Statistical Stability, and Performance on General Transcripts

  1. Token scale and statistical stability: While $N=53$ has fewer documents than social-media benchmarks, each Anthropic Interviewer transcript is a multi-turn interview averaging 1,906 tokens across 22 turns, or 101,026 tokens in total. One interview equals ~90 comments in SynthPAI, which average 106 chars in Yukhymenko et al. at NeurIPS 2024, and ~$2\times$ the 1,000-token cap in PersonalReddit from Staab et al. at ICLR 2025. Evaluation spans 340 validated profile facts, 773 expert-coded excerpts, 9 O*NET-SOC major groups, and over 3,300 attack runs, and qualitative interview studies typically reach thematic saturation within 9 to 17 interviews, per Hennink and Kaiser in 2022.

    Stability setup: Across all 21 defense configurations, we drew 1,000 random subsamples of size $n \in {10, 20}$ and 1,000 bootstrap resamples of size $n=53$, measuring pairwise method-rank agreement on pairs with a full-sample gap $\ge 0.05$:

MetricSubsample $n=10$Subsample $n=20$Bootstrap $n=53$
Profile-Fact Retention0.950.980.99
Occupational Fact Retention0.950.970.99
Web-Search Re-ID0.910.970.98

Even at $n=10$, rankings match the full cohort 91% to 95% of the time, and 98% to 99% under $n=53$ bootstrap. Under $n=53$ bootstrap, AURA beats RUPTA in 100% of resamples on profile retention with Wilcoxon $p < 10^{-4}$. AURA ties Advanced Anonymizer in utility with a reduced Re-id from 20.8% to 6.3%. Stratified across 31 Science, 17 Work, and 5 Creativity interviews, AURA also beats RUPTA in every domain at 0.63, 0.78, and 0.93 vs. 0.41, 0.50, and 0.61 on grid utility.

  1. General unidentifiable transcripts: From the remaining 1,197 public interviews with no confirmed identity under multi-run search, we sampled 10 length-matched transcripts across 6 Science, 3 Work, and 1 Creativity, with 55 validated profile facts, and ran AURA + adaptive privacy. Because all interviews elicit work and research details under the same protocol, AURA abstracts quasi-identifiers with the same light edit rate and high utility as on the 53 identifiable cases: median word edit rate is 6.0% vs. 7.1% and profile-fact retention is 0.72 vs. 0.73.
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer U67a,

Thank you for your time and helpful comments. As the discussion period ends on Nov. 18th, we wanted to briefly follow up on the three points you raised:

  • Recent baselines: We added RUPTA (Yang et al., ACL 2025). Adaptive AURA achieves lower re-identification under all three attackers (0–7/53 vs. 4–18/53) and higher utility (72.0–76.1% vs. 44.3% grid recovery).
  • Computational cost: With open-weight backbones, full adaptive AURA costs USD 0.12–0.18 per interview, about USD 9 for all 53. That is fewer LLM calls and lower cost than RUPTA (see our earlier response).
  • Dataset size: Method rankings remain stable under subsampling (91–98% pairwise agreement even at n=10; App. M.6).

Please let us know if any concern remains. We would be glad to clarify further.

Best regards,

The Authors

Reviewer_UgtsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies text anonymization against LLM agents that combine contextual clues with web searches to re-identify individuals. The proposed AURA framework first uses a web-search-based re-identification agent to identify transcript-specific quasi-identifiers and expand the scope of sensitive information to be protected. It then applies an existing iterative LLM anonymizer to identify risky spans and selectively regenerates only the masked spans, choosing among candidates using LLM-based judges for privacy, specificity, and utility. Experiments show that AURA with the expanded privacy scope achieves lower re-identification rates than non-DP baselines.

Critical strengths

S1. The problem is timely and well motivated: agentic LLMs with web search turn contextual details in released qualitative data into a practical re-identification risk, and the evaluation uses real transcripts with verified risk.

S2. Running the web-search attack only once, separately from the rewriting loop, is a practical design, and the masked spans can serve as an interpretable risk map for data stewards.

S3. The evaluation covers several attacker models, both closed- and open-weight backbones, uncertainty estimates, and human studies of readability and faithfulness.

Critical weaknesses

W1. The utility metric rewards retaining personal attributes that the anonymization process is intended to protect. At a matched privacy scope, AURA’s utility improvement comes entirely from recovering profile facts, with no improvement in codebook recovery. Since privacy is evaluated only through binary re-identification, it is unclear whether the utility gain reflects better information preservation or greater disclosure of sensitive attributes. Moreover, the human validation does not establish whether the LLM judge can reliably detect information loss.

W2. AURA uses essentially the same re-identification attack to identify sensitive information and evaluate privacy, with held-out attackers differing mainly in the underlying LLM. This raises concerns about whether the privacy gains generalize to different attack strategies. Moreover, the prior anonymizer is not evaluated with the same adaptive privacy scope, making it difficult to separate the benefits of AURA’s method from those of the additional information supplied by the attack.

W3. AURA largely combines existing techniques: iterative LLM anonymization, best-of-N span reconstruction, and LLM-based candidate selection. The adaptive scope guides the anonymizer toward information identified by the re-identification attack, while the reconstruction phase introduces relatively limited methodological innovation. Furthermore, [1] studies the privacy-utility trade-off against LLM-based re-identification, which is not included as a baseline.

[1] Robust Utility-Preserving Text Anonymization Based on Large Language Models. ACL, 2025.

Questions for authors

Q1. What is the re-identification rate of the original, unmodified transcripts under each attacker?

Q2. Which model generates the adaptive scope for the API-powered variants, and is it one of the evaluation attackers?

Q3. How do the methods compare in attribute-level leakage, beyond binary re-identification?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ugts, Part 1 of 2: Threat Model, Judge Validation, and Adaptive Scope Ablation

We thank Reviewer Ugts for the rigorous critique. We address W1 and W2 below, and W3 plus Q1 to Q3 in Part 2.

W1: Attribute Retention vs. Sensitive Disclosure, Behavioral Utility, and Judge Reliability

  • Why retaining profile attributes is utility rather than leakage under our threat model: Attribute-level privacy and web-search Re-ID have different leakage signals:
DimensionAttribute-Level Privacy, Staab et al., ICLR '25Web-Search Re-Identification, Ours
Protected secretCoarse attributes such as age, role, or cityLink between transcript and a real person
AdversaryClosed-world LLM inferring from text aloneOpen-world LLM querying search and public web
Leakage signalAny inferable attribute counts as leakageHyper-specific anchors enabling web linkage
Utility relationErases the profile needed for analysisAbstracts anchors and keeps general role or domain

In qualitative studies such as the Anthropic Economic Index, occupational attributes are the covariates linking who participants are to how they use AI. Under web-search Re-ID, an attribute is harmful only when its specificity singles someone out online. In our experiment, retaining at least one named-entity fact raises Re-ID sixfold from 0.05 to 0.31, whereas general role and task retention barely differs between identified and safe texts at 0.88 vs. 0.74. Abstracting named firms and projects into phrases like "an organization doing interior projects" lets AURA raise profile retention while cutting GPT-5.1 Re-ID to 7.5%, vs. 20.8% for Advanced Anonymizer and 24.5% for RUPTA.

  • Excerpt-level downstream coding and judge validation: Codebook-fact recall is transcript-level and saturates at $\ge 94%$ across non-destructive methods, so we added downstream codebook coding on all 773 expert-coded excerpts across $N=53$. Given the expert code, its definition, and the original excerpt as reference, an LLM coder, deepseek-v4-flash, judges whether each aligned anonymized excerpt still expresses the same coded insight, reaching $\kappa=0.80$ against human expert labels on originals. AURA retains 96.2% of supported codes, while Advanced Anonymizer recovers 96.2% and above RUPTA at 91.9%, with a paired 95% CI for AURA minus RUPTA of 0.6 to 8.4 points, while a shuffled-excerpt null retains 6.3%. To verify that the same deepseek-v4-flash judge also detects profile information loss, we compared its labels against majority votes from 41 crowd raters across 71 resolved profile-fact tasks, where it reaches substantial agreement with $\kappa=0.777$.

W2: Attack Generalization and Adaptive Scope Controlled Ablation

  • Controlled ablation giving AURA's scope to Advanced Anonymizer: Adaptive scope discovery is part of AURA's defense, so giving it to Advanced Anonymizer forms an AURA-baseline hybrid. To isolate AURA's mask-and-refill stages from scope discovery, we ran Advanced Anonymizer with AURA's cached scope of 8 base with up to 4 web-probed attributes on all $N=53$ interviews:
Configuration, $N=53$Privacy ScopeRe-ID % GPT 5.1 / GPT 5.4mini / Gemini | AvgProfile %Grid %
Advanced AnonymizerFixed 8-attr20.8 / 22.6 / 18.9 | 20.869.467.5
Adv. Anonymizer + AURA scopeAURA adaptive, $\le 12$1.9 / 3.8 / 0.0 | 1.937.434.0
AURA, GPT-4.1Fixed 8-attr15.1 / 26.4 / 18.9 | 20.177.473.2
AURA + adapt., GPT-4.1AURA adaptive, $\le 12$7.5 / 11.3 / 0.0 | 6.373.570.4
AURA + adapt., Qwen-35BDeepSeek + Tavily, $\le 12$3.8 / 13.2 / 3.8 | 6.976.875.7
  1. Fixed 8-attr scope: AURA matches Advanced Anonymizer's mean Re-ID at 20.1% vs. 20.8% while gaining +8.0 profile and +5.7 grid points.
  2. Matched AURA scope: Advanced Anonymizer erases every flagged attribute, cutting Re-ID to 1.9% but halving utility to 34.0% vs. 70.4% on the grid, whereas AURA caps specificity to preserve profile context at 6.3% mean Re-ID.
  • Cross-attacker generalization: AURA + adapt. with Qwen-35B builds scopes via DeepSeek-V4-Flash and Tavily, sharing neither LLM nor search engine with GPT-5.1, GPT-5.4-mini, or Gemini-3-Flash, yet achieves 3.8% GPT-5.1 Re-ID and 75.7% grid utility because scopes target linkable text anchors rather than attacker phrasing. We will explicitly state that our claims are scoped to web-search Re-ID.
AuthorsOpenReview ↗

Response to Reviewer Ugts, Part 2 of 2: Methodological Novelty, RUPTA Baseline, and Q1 to Q3

W3: Methodological Novelty and Comparison with RUPTA [1] from ACL 2025

  • Threat-model difference and AURA's design: While LLM rewriting, sampling, and judging are standard primitives, AURA is built around the difference between attribute-level privacy and web-search Re-ID. Defenses like Advanced Anonymizer and RUPTA assume a closed-world guesser inferring a fixed list of demographic attributes from text alone. Web-search Re-ID is instead an open-ended infer, cross-reference, and aggregate process: the attacker infers unique, specific details from the interview, queries the web, cross-references retrieved pages, and aggregates them into one identity. Specificity drives linkage here, and interviewees overshare niche details like paper topics, client types, and tool stacks unneeded for analysis. Whole-passage rewriters built for attribute inference miss open-ended web anchors outside their fixed list while erasing broad profile context needed for analysis. That motivates AURA's three stages: a web-probed adaptive scope discovers open-ended, participant-specific clues that link to the web, diff-derived span masking localizes edits to risky spans and leaves behavioral text untouched, heavily editing 20% of coded excerpts vs. 44% for RUPTA, and specificity filtering during refill screens candidates to filter out unique linkable specificity while keeping role and task context.
  • Evaluating RUPTA on $N=53$ interviews: RUPTA's DB-bio mode prompts an offline LLM to recall Wikipedia celebrities from memory given the true name and uses one occupation label as utility, mismatching non-celebrity web linkage. We ran RUPTA's PersonalReddit reflexion code with GPT-4.1, 1 initial plus up to 5 reflection rounds over 8 attributes while preserving interviewer turns, on all 53 transcripts:
MethodRe-ID % AvgProfile %Code %Utility Grid % 95% CICoding Ret. %
AURA + adapt. privacy6.373.596.470.4, 62 to 7896.2
Advanced Anonymizer, ICLR '2520.869.495.967.5, 59 to 7696.2
RUPTA, ACL '2522.049.788.845.6, 39 to 5291.9

AURA Pareto-dominates RUPTA, with $3.5\times$ lower mean Re-ID at 6.3% vs. 22.0% and +24.8 points higher grid utility at 70.4% vs. 45.6%.

Q1: Original Re-ID Rate and Ground-Truth Reliability Because the dataset has no identity labels and single search runs are stochastic, we built our ground truth across all 1,250 raw interviews via a 6-stage verification protocol against false positives: screening all 1,250 interviews with our direct-intent GPT-5.1 web attacker for high or very-high confidence matches, filtering out dataset or organization hits to keep only named individuals, re-running each candidate up to 4 times and keeping it only if the exact same person appears in $\ge 3/4$ runs with $\ge 2$ at high or very-high confidence while discarding conflicting candidates, applying the same $\ge 3/4$ gate to dataset-hit and indirect-intent matches, and manually verifying every surviving match against public records across publications, theses, portfolios, and timelines. This protocol errs toward false negatives so every retained match is a high-certainty, reproducible public link. The 53 transcripts passed all six stages, giving a 100%, or 53 of 53, verified Re-ID baseline on raw texts under GPT-5.1.

Q2: Model Generating the Adaptive Scope in API Variants Yes, GPT-5.1 is one of the attackers: in API adaptive variants, GPT-5.1 probes the raw text via web search for candidate links and evidence, and GPT-4.1 distills that evidence into $\le 4$ added quasi-identifier categories for a total scope $\le 12$. Two controls show that privacy gains do not depend on sharing the attacker model: GPT-5.4-mini and Gemini-3-Flash never touch scope generation yet are held to 11.3% and 0.0% Re-ID, and our open-weight hybrid variants using Qwen-27B or 35B with DeepSeek-V4-Flash and Tavily hold all three attackers and their search engines out, yet achieve 3.8% to 7.5% GPT-5.1 Re-ID.

Q3: Attribute-Level Retention Beyond Binary Re-ID Splitting all 340 profile facts on $N=53$, 9 named-entity plus 331 attribute facts below, shows selective suppression of identifying attributes while keeping role and task attributes:

MethodGeneral Role or TaskNiche ExpertiseTools or MethodsHistory or LocationDemographics, Age, or Edu
Advanced Anonymizer0.730.800.740.280.19
RUPTA0.580.400.440.560.14
AURA + adapt. privacy0.850.770.570.440.29

AURA removes 71% of demographic and 56% of location/history facts, while keeping 85% of role/task facts vs. 58% for RUPTA.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Ugts,

We truly appreciate your detailed and technical review, which substantially improved the paper. As the discussion period ends on Nov. 18th, we would be grateful to know whether our revisions address your concerns:

  • W1 (utility vs. attribute disclosure): A new attribute-level analysis (App. M.5, Table 14) shows that re-identification is associated with retained named entities and tools, not demographic attributes. The coding judge agrees with experts at κ=0.80 (substantial to strong agreement).
  • W2 (matched scope): We added an Advanced Anonymizer + AURA-scope ablation (Sec. 5.3, Table 7). It reaches similar privacy with much lower utility, which isolates the contribution of mask-reconstruct.
  • W3: RUPTA (ACL 2025) is now included as a baseline throughout.
  • Q1 (original re-ID rate): All 53 original transcripts are re-identified under GPT-5.1 by construction (App. N).
  • Q2 (scope generator): GPT-5.1 runs the scope probe for the API variants, so GPT-5.4-mini and Gemini-3-Flash are held out. DeepSeek-V4-Flash runs it for the Qwen variants, so all three evaluation attackers are held out (Sec. 4.2).
  • Q3 (attribute-level leakage): See App. M.5, Table 14, which compares retention by fact type between re-identified and non-re-identified cases.

In particular, we would value your view on whether the matched-scope ablation resolves the concern about separating scope from reconstruction. We are happy to provide any further clarification.

Best regards,

The Authors

Reviewer_RLa5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the paper focuses on the problem of defending re-identification attacks based on weak contextual cues of anonymized texts, and propose a framework named Anonymization with Utility-Retention Adaptation (AURA), which aims to solve this problem while preserving the information needed for downstream use simultaneously. Specifically, this method utilize an iterative rewriting process based on LLM-based attribute inference to find candidates with potential privacy risks, and select candidate based on three scores including privacy severity, specificity, and utility. Finally, the proposed algorithm is evaluated on the 53 re-identifiable interview transcripts from the Anthropic Interviewer dataset.

Critical strengths

no substantive strength can be identified.

  • The paper is well-written and easy to follow.
  • The paper focuses on an important problem, i.e., defending re-identification attacks based on weak contextual cues of anonymized texts
  • The provided appendix is detailed, which help readers to better understand the paper.
Critical weaknesses
  • Experiments are only conducted on 53 user samples from a single dataset, making the conclusion not convincing.
  • The utility evaluation is based on measuring the recovery of interviewee-profile facts, codebook facts, and the joint contextual utility grid. However, no actual downstream analysis task is considered, making it hard to assess whether the utility is preserved.
  • This framework relies on external models and search services, which posing new potential privacy leakage risks.
  • The reliability of the 53 re-identification ground is doubtful. In Page9, it is said that re-identification ground truth works as a proxy for privacy risk and cannot locate the participants behind the transcripts without their confirmation. How can this ground truth be accessed as reliable without participant confirmation?
Questions for authors
  • Is it possible to conduct downstream tasks to verify whether utility is preserved after anonymization?
  • How to preserve the privacy in the process of using the proposed framework, specifically the LLM-based steps?
  • Could you please explain how the reliability of the 53 re-identification ground is assessed in the paper?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RLa5, Part 1 of 2: Dataset Scale, Statistical Stability, and Downstream Tasks

We thank Reviewer RLa5 for the constructive feedback. We address W1, W2, and Q1 below, and W3, Q2, W4, and Q3 in Part 2.

W1: Dataset Scale at $N=53$ and Statistical Stability While $N=53$ has fewer documents than social-media benchmarks, each Anthropic Interviewer transcript is a multi-turn interview averaging 1,906 tokens across 22 turns, or 101,026 tokens in total. One interview equals ~90 comments in SynthPAI, which average 106 chars in Yukhymenko et al. at NeurIPS 2024, and ~$2\times$ the 1,000-token cap in PersonalReddit from Staab et al. at ICLR 2025. Evaluation spans 340 validated profile facts, 773 expert-coded excerpts, 9 O*NET-SOC major groups, and over 3,300 attack runs. Qualitative interview studies typically reach thematic saturation within 9 to 17 interviews, per Hennink and Kaiser in 2022.

Stability setup: Across all 21 defense configurations, we drew 1,000 random subsamples of size $n \in {10, 20}$ and 1,000 bootstrap resamples of size $n=53$, measuring pairwise method-rank agreement on pairs with a full-sample gap $\ge 0.05$:

MetricSubsample $n=10$$n=20$$n=53$
Profile-Fact Retention0.950.980.99
Occupational Fact Retention0.950.970.99
Downstream Thematic Coding1.001.001.00
Web-Search Re-ID0.910.970.98

Even at $n=10$, rankings match the full cohort 91% to 100% of the time. Under $n=53$ bootstrap, AURA beats RUPTA in 100% of resamples on profile retention with Wilcoxon $p<10^{-4}$. AURA ties Advanced Anonymizer in utility with a reduced Re-id from 20.8% to 6.3%. Across 31 Science, 17 Work, and 5 Creativity interviews, AURA also beats RUPTA in every domain at 0.63, 0.78, and 0.93 vs. 0.41, 0.50, and 0.61 on L3 utility.

W2 and Q1: Actual Downstream Qualitative Analysis Tasks Yes. We added two downstream tasks mirroring how social scientists analyze these interviews:

  • Task A, Codebook Thematic Coding on 773 excerpts: Human researchers coded 773 excerpts across the 53 interviews using a 13-code codebook. In each anonymized transcript, we deterministically align each excerpt to the closest 1-to-8-sentence window in the same speaker turn by cosine similarity. Given the expert code with its definition and inclusion and exclusion criteria, the original excerpt as reference, and the aligned anonymized excerpt in context, an LLM coder, deepseek-v4-flash at temperature 0, judges whether the anonymized excerpt still expresses the same coded insight and cites an exact evidence quote. On original transcripts, the LLM coder reaches $\kappa=0.80$ against human expert labels, confirming 85.7% of expert codes, and two independent runs differ by at most 0.3%. We report retention relative to originals.
  • Task B, Economic Index Occupational Profiling across L1 to L3: Following the Anthropic Economic Index, which maps O*NET occupations against AI use, our crowd-validated judge verifies each interviewee's profile at three granularities. This judge matches the human crowd majority on 95.8% of 71 tasks with $\kappa=0.777$. L1 tests broad domain across 5 umbrella templates, L2 tests the plain-language O*NET-SOC major group, and L3 tests fine-grained occupation-by-AI-use pairs, which is our contextual utility grid.

Synthetic examples of statements verified at each level:

LevelParticipant AParticipant B
L1 DomainWorks in a technical or scientific field such as science, engineering, or mathWorks in a cultural or community field such as arts, design, or media
L2 SOC GroupWorks in life, physical, or social scienceWorks in arts, design, or media
L3 Utility GridPostdoc modeling groundwater flow with FEM software and using AI to debug scriptsFreelance interactive museum exhibit designer and using AI for concept sketches
Text SourceCoding $\kappa$, 95% CINorm. RetentionL1L2L3 Grid, 95% CI
Original Transcripts0.705, 0.67 to 0.741.0001.001.001.00
AURA + adapt. privacy0.694, 0.66 to 0.730.9740.920.940.70, 0.62 to 0.78
Advanced Anonymizer0.670, 0.63 to 0.710.9190.920.920.67, 0.59 to 0.76
RUPTA, ACL '250.670, 0.63 to 0.700.9180.940.960.46, 0.39 to 0.52
DP-MLM, $\varepsilon=10$0.309, 0.27 to 0.350.0730.000.000.00

AURA keeps 96.2% of the expert codes supported in originals, on par with Advanced Anonymizer with better privacy and significantly above RUPTA at 91.9%, with a paired 95% CI for AURA minus RUPTA of 0.6 to 8.4 points, while retaining far more L3 detail than RUPTA at $0.70$ vs. $0.46$ with non-overlapping CIs.

AuthorsOpenReview ↗

Response to Reviewer RLa5, Part 2 of 2: Execution Privacy and Ground-Truth Reliability

W3 and Q2: Preserving Privacy While Running AURA across LLM Steps and Web Search We separate the two external components used during pipeline execution, the LLM and the search engine:

  1. LLM-Based Steps are Fully Localizable: Every LLM stage in AURA, including scope initialization, iterative span masking, candidate reconstruction, and selection, runs cleanly with open-weight models such as Qwen3.5-27B, Qwen3.5-35B-A3B, and DeepSeek-V4-Flash. Hosting these weights on local GPUs keeps raw transcripts locally, so zero raw text leaves the data steward's machine during LLM calls, while matching cloud GPT-4.1 performance at 3.8% to 7.5% GPT-5.1 Re-ID and 74.6% to 75.7% utility grid on $N=53$.
  2. Empirical Query-Exposure Audit of Web-Search Probing: Probing open-web linkability inevitably sends search queries to a search API. To measure cumulative disclosure in a trajectory and single-query disclosure which atomic and irreducible, we audited all logged raw-transcript probe runs from our initial cohort where full search tool calls on raw text were archived, along with all 1,846 logged post-rewrite trajectories with saved query logs and per-transcript Re-ID labels:
Exposure vs. Raw TranscriptCumulative, per TranscriptSingle Query
Query volume and lengthMedian 12 queries, max 36Median 7 words, max 22
Raw words sent as verbatim $\ge$4-gramsMean 1.68%, median 0.97%, max 8.8%Mean 0.19%, max 2.2%, 64% have none
Profile facts disclosed11.1% full disclosed, 37.5% partial, 51.4% no disclose91% of 347 queries disclose 0 profile fact,
Demographic, age, or education facts0% full0 of 61 query-fact pairs
GT name, 1,662 post-rewrite trajectoriesAppears in 14% of the attempts3.3% of 47,659 queries
Before 1st GT-name query, 220 hitsMedian 9 queries, 0.34% raw wordsName-free queries: mean 0.17% raw words

Three takeaways hold at both the single-query and cumulative levels:

  • Minimal text exposure: Each single query is a ~7-word topical fragment averaging 0.19% verbatim words, with 64% containing zero verbatim 4-grams and 91% fully disclosing zero profile facts. Cumulatively across all 12 queries in a run, the search engine sees only 1.68% of raw words, 11.1% of profile facts, and 0% of demographic facts.
  • Identity is synthesized locally: For each query, only 0.6% contain a person name from the raw text. The GT name never occurs in the raw tex. It highlights only after a median of 9 queries covering 0.34% of raw words. Linking snippets to a candidate happens inside the LLM reading results, which can stay on-device.
  • Confirmation queries can be removed: A search engine sees a person's name together with interview context only in a confirmation query, which the attacker sends after it has already guessed who the person is. Such queries are 3.3% of all post-rewrite queries and appear in 1% to 11% of trajectories on AURA outputs, compared with 57% on Presidio. Blocking queries that contain person names removes this step. When we cut the 220 successful trajectories off before their first name query, the true person still ranks in the top 3 candidates in 83% of cases and in the top 5 in 95%.

W4 and Q3: Reliability of the 53 Re-Identification Ground Truths Without Contacting Participants Page 9 notes that we never contact interviewees, which IRB rules and dataset terms forbid, not that our ground truths are unverified guesses. Because single web-search runs are stochastic and common roles like "freelance 3D artist" can surface different people across runs, we built the 53 ground truths from all 1,250 public interviews via a strict 6-stage verification protocol against false positives:

  1. Confidence screening: We run our direct-intent web-search attacker across all 1,250 interviews and retain only high or very-high confidence matches.
  2. Entity-type filtering: We discard matches resolving only to the Anthropic dataset or an organization, keeping only named individuals.
  3. Multi-run consistency of at least 3 of 4: We re-run each candidate independently up to 4 times and keep it only if the exact same person appears in at least $3/4$ runs with at least $2$ at high or very-high confidence. Any interview with conflicting candidates is discarded.
  4. Dataset-hit and Stage 5 indirect-intent checks: Candidates recovered after dataset-hit filtering or via indirect-intent queries must pass the same $\ge 3/4$ multi-run filter.
  5. Manual expert verification: The authors manually verified every surviving match against public records, confirming alignment across multiple independent clues including specific publications, theses, software or portfolios, and career timelines stated in the interview.
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer RLa5,

Thank you again for your thoughtful review. As the discussion period ends on Nov. 18, we wanted to briefly check whether our response has addressed your concerns. In summary:

  • Downstream utility: We added two downstream tasks modeled on the Anthropic Economic Index: occupational profiling (L1–L3) and codebook coding of 773 expert-coded excerpts (Sec. 4.1, 5.2; App. F.1, Table 7).
  • Privacy of the pipeline itself: We added a query-exposure audit (App. M.8, Table 18). The search API sees only short fragments, and all LLM stages can run on local open-weight models.
  • Ground-truth reliability: We describe the six-stage verification protocol (Sec. 4; App. N) and a subsampling analysis showing stable method rankings even at n=10 (App. M.6).

If any of these points remain unclear, or if there are further experiments you would find convincing, we would be glad to address them before the deadline.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of New Experiments, Clarifications, and Upcoming Manuscript Revision

We thank the Area Chair and all four reviewers for their careful reading and constructive feedback. Following your suggestions, we ran extensive new experiments and analyses. Below we summarize the main additions. We respond point by point in each reviewer's thread and will upload the revised PDF with highlighted changes during the discussion period.

1. Threat Model Clarification: Web-Search Re-Identification vs. Attribute-Level Privacy The two threat models differ in their leakage signal. Attribute-level privacy, as in Advanced Anonymizer at ICLR 2025 and RUPTA at ACL 2025, treats coarse demographic attributes such as age, location, and occupation as the secret. Under that model, any retained attribute counts as leakage, even though such attributes are shared by many people. Web-search re-identification is instead linkage exposure. An open-web attacker extracts fine-grained details, queries search engines, cross-references retrieved pages, and aggregates them into a unique person. Specificity then matters in the leakage signal. Across 1,846 logged attack trajectories, linkage is driven by high-specificity anchors such as named entities, niche technical combinations matching a paper or project, and verbatim searchable phrases. Retaining at least one named entity raises the re-identification rate from 0.05 to 0.31, whereas general roles and tasks barely affect re-identification. Because qualitative research studies who uses AI and how, erasing attributes greatly impairs utility. AURA instead performs specificity minimization in the refilling stage, abstracting linkable anchors while keeping general role, task, and behavioral context.

2. Summary of New Experiments and Analyses across $N=53$ Interviews

  • ACL 2025 Baseline, RUPTA: We evaluated RUPTA across all 53 interviews under the identical three web-search attackers, GPT-5.1, GPT-5.4-mini, and Gemini-3-Flash, and the same utility benchmarks.
  • Adaptive Scope Ablation: Giving AURA's web-probed scope to Advanced Anonymizer cuts average re-identification to 1.9% but halves utility from 70.4% to 34.0%. Erasing attributes overshoots, whereas capping specificity preserves analytical value.
  • Two Downstream Qualitative Tasks: First, we ran codebook thematic coding on all 773 expert-coded interview excerpts using 4-way classification with the 13-code expert codebook. Our LLM coder matches human experts at $\kappa=0.671$ on original transcripts, and AURA preserves $\kappa=0.694$ compared with $0.705$ on originals, with a 95% CI on the paired drop of $[-0.02, 0.04]$ that includes 0. Second, we ran Economic Index occupational profiling across O*NET-SOC levels, from broad domain at L1 and SOC major group at L2 to fine-grained occupation-by-AI-use cells at L3, which corresponds to our contextual utility grid. At L3, AURA retains 70.4% vs. 45.6% for RUPTA
  • Interpretive Fidelity and Unidentifiable Transcripts: coding consistency in the segment level and a manual audit of heavily edited code changes show that AURA heavily edits half as many excerpts as RUPTA, 20% vs. 44%, and causes genuine meaning shifts in only 4 cases, all tied to identity anchors. On 10 unidentifiable transcripts sampled from the remaining 1,197 interviews, AURA edits lightly with a median word change of 6.0% compared with 7.1% on the 53 re-identifiable cases, and retains 0.72 of profile facts vs. 0.73 in the 53 cases.
  • Query Exposure, Sample Stability, Ground-Truth Protocol, and Cost: We audited search queries across 1,846 trajectories, verified bootstrap rank stability at 0.98 to 1.00, detailed our 6-stage ground-truth protocol, and logged per-transcript API calls, latency, and cost, which drops to $0.12 to $0.18 per interview according to different open-weight models.
Method, $N=53$Re-ID % $\downarrow$ 5.1 / 5.4m / GeminiL3 Utility Grid % $\uparrow$Coding Retention % $\uparrow$
Original Transcripts100.0 verified GT100.0100.0
Presidio NER49.1 / 75.5 / 50.997.197.7
Advanced Anonymizer, ICLR '2520.8 / 22.6 / 18.967.596.2
RUPTA, ACL '25, new24.5 / 34.0 / 7.545.691.9
DP-MLM, $\varepsilon=50$ | $\varepsilon=10$17.0 / 9.4 / 3.8 | 0 / 0 / 055.2 | 0.049.5 | 0.0
AURA fixed 8-attr, GPT-4.115.1 / 26.4 / 18.973.296.5
AURA + adapt. privacy, GPT-4.17.5 / 11.3 / 0.070.496.2
AURA + adapt. hybrid, Qwen-35B3.8 / 13.2 / 3.875.797.1
AuthorsOpenReview ↗

Revised transcript available for rebuttal

We thank all reviewers for their constructive feedback. All changes in the revised manuscript are highlighted. Below we briefly map each reviewer’s main concerns to the corresponding revisions.

Reviewer RLa5

  • Sample size: The 53 transcripts are all verified re-identifiable cases from the 1,250 public interviews (Sec. 4; App. N). Pairwise method rankings agree with the full cohort in 91–98% of resamples, even at n=10 (App. M.6). We also evaluate transcripts the probe could not identify (App. M.9).
  • Downstream tasks: We add occupational profiling (L1–L3) and codebook coding of 773 expert-coded excerpts, modeled on the Anthropic Economic Index (Sec. 4.1, 5.2; App. F.1, Table 7).
  • Pipeline privacy: We add a discussion (Sec. 6) and a query-exposure audit (App. M.8, Table 18). All LLM stages can run on open-weight models, and the search API sees only short fragments (median 7 words per query).
  • Ground truth: We describe a six-stage verification protocol: high-confidence screening, named individuals only, 3-of-4 run consistency, and manual checks against public records (Sec. 4; App. N). Temporal consistency is in App. M.2.

Reviewer Ugts

  • W1 (utility vs. disclosure): We clarify the threat model (Sec. 3). An attribute-level analysis (App. M.5, Table 14) shows that named entities and tools separate re-identified from non-re-identified cases, while demographic facts do not (discussed in Sec. 6). On judge reliability, the judge agrees with experts at κ=0.80, and a shuffled-excerpt null retains only 6.3% (Sec. 4.1; App. F.1, J).
  • W2 (attacker overlap, matched scope): Sec. 4.2 clarifies scope generation. GPT-5.1 runs the probe for the API variants, so GPT-5.4-mini and Gemini-3-Flash are held out. DeepSeek-V4-Flash runs it for the Qwen variants, so all three attackers are held out. We add an Advanced Anonymizer + AURA-scope ablation (Sec. 4.2, 5.3; Table 7; Fig. 2). It reaches similar privacy with much lower utility, which isolates the contribution of mask-reconstruct.
  • W3: We add RUPTA (Yang et al., ACL 2025) as a baseline throughout (Sec. 1, 2, 4.2; Tables 1, 7; App. D).
  • Q1: All 53 originals are re-identified under GPT-5.1 by construction. We did not repeat raw-text attacks with the other attackers, to avoid sending raw text to more external services (App. N).
  • Q2: See W2. Q3: See App. M.5, Table 14.

Reviewer U67a

  • Baselines: We add RUPTA (ACL 2025) (Sec. 4.2; Tables 1, 7).
  • Overhead: (This is not in the revised PDF; the full breakdown is in our earlier response.) AURA is an offline, one-time pre-release step that replaces 30–60 min of manual redaction per interview. Per-transcript costs (mean 1,906 tokens per interview):
MethodCalls/docMedian timeCost/doc (USD)
Adv. Anonymizer (GPT-4.1)10~95 s~0.19
RUPTA (GPT-4.1)50.9~140 s0.21
AURA adaptive (GPT-4.1 + GPT-5.1)36–39~5 min~0.95–1.10
AURA adaptive (Qwen3.5 + DeepSeek)36.1~8–12 min0.12–0.18

With open-weight backbones, full adaptive AURA costs about USD 9 for all 53 interviews. That is fewer calls and lower cost than RUPTA.

  • Dataset size: See our response to RLa5 (App. N, M.6, M.9).

Reviewer JUpK

  • Faithfulness vs. meaning: We add an interpretive-fidelity analysis using blind code consistency, by edit level and code dimension (App. M.7, Table 16). A manual audit separates necessary identity generalization from avoidable meaning shifts (Table 17). AURA heavily edits 20% of excerpts vs. 44% for RUPTA. All of AURA's genuine meaning shifts are identity-linked, whereas most of RUPTA's are avoidable. We connect this to the main tradeoff in Sec. 6.
  • Less identifiable interviews / abstention: On 10 transcripts the probe could not identify, AURA edits lightly: a median of 6.0% of words vs. 7.1% on identifiable ones (App. M.9, Table 19). The probe's confidence flag lets data stewards skip scope expansion below the threshold, and if no masks are produced, the transcript is returned unchanged (Sec. 3.2).
  • Topics and linkage pathways: We add coding retention by topic and a linkage audit (App. M.10, Table 20). Science interviews are linked through a specific publication or research source; other cases through a specific project or product combined with biographical clues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.