- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AURA, a framework for anonymizing interview transcripts against re-identification by language models with web-search access. The method separates identifying sensitive vulnerable passages from reconstructing them while preserving useful semantic information. AURA establishes a privacy scope, masks potentially identifying details, and generates alternative replacements for those passages, rather than direct application of the masks. It then selects among these candidates using checks for attribute leakage, specificity, and information loss. They also ablate heavily on the types of privacy parameters used, as well as go deep into ablations on the pareto tradeoff between privacy and utility.
The authors evaluate AURA on 53 interviews from 1,250-transcript Anthropic Interviewer dataset. They select these based on for which samples did an initial attack produced verifiable identification evidence. They compare it with iterative LLM anonymization, one-shot rewriting, entity redaction, and DP-MLM, using three web-enabled attacker models (GPT-5.1, GPT-5.4-mini, and Gemini-3-Flash). Adaptive AURA achieves the lowest re-identification counts among the evaluated non-DP methods while preserving substantial measured utility. With the model and privacy scope held fixed, AURA improves contextual semantic utility recovery over the prior iterative anonymizers by 6.4 percentage points. Human evaluations provide additional evidence about readability and factual faithfulness, although they are in the appendix.
Critical strengths
1. The method is clearly motivated and extremely well explained. I particularly loved the separation between finding identifying details and deciding how to rewrite them to preserve semantic intent. This directly addresses the difficulty that the same information can be both identifying and important to understanding an interview, this tradeoff is very nuanced. The comparison with the prior iterative anonymizers, holding the model and privacy scope fixed, provides evidence that the reconstruction stage helps preserve utility.
2. The privacy–utility analysis makes a strong case for the approach. Web-enabled attackers represent a realistic threat, and evaluating several attackers and baselines make the results more informative. I appreciated that the DP-MLM comparison shows the trade-off between stronger observed privacy protection and information loss at lower epsilons. AURA occupies a useful region of this trade-off, and the human evaluations in the Appendix provide supporting evidence beyond automated scores.
Critical weaknesses
1. Preserving factual correctness does not necessarily preserve qualitative meaning. The paper does address faithfulness: Section 5.2 reports that 86.9% of retained rewrite pairs were judged faithful, and the appendices discuss factual distortions. However, this check focuses on whether a rewrite introduces false information. A rewrite could avoid false statements while losing a qualification, relationship, or contextual semantic detail that changes how someone can interpret the participant’s views. I would like this distinction to be more closely connected to the main utility argument on tradeoffs.
2. The selected sample leaves uncertainty about the overall utility of the pipeline. The evaluation uses 53 previously identifiable interviews. I understand this choice, since evaluating only cases that an attacker already cannot identify would provide little evidence about the defense. However, this also leaves open how AURA behaves on less identifiable interviews. Does it recognize that little intervention is needed, or does it still remove useful context? Differences across interview topics and types of identifying information would also help explain how broadly the reported pareto tradeoff applies.
Questions for authors
The questions below are listed in order of importance to my assessment:
1. Can you distinguish necessary information removal from avoidable loss of meaning? Did the human evaluation identify rewrites that introduced no false information but changed the interpretation of a participant’s views? Connecting these cases to the utility scores would help clarify what those scores capture. It also helps strengthen the notion of "faithfulness". If possible can this ablation be tied into the main results also using the human evaluation results from the Appendix J.
2. How does AURA behave when the initial attacker cannot identify the participant? Does the pipeline generally leave these interviews unchanged, or still mask and reconstruct substantial passages? A small comparison of the amount of text changed and the resulting utility retention would help address the concern about unnecessary intervention.
3. Do the results vary across interview topics or types of identifying details? For example, are interviews linked through a specific source affected differently from interviews linked through several biographical clues from the text? A breakdown within the existing samples would help explain where preserving useful context is most difficult. Do the identifying details being scrubbed here destroy semantic meaning if the biographical clues are ruined?
The main theme of the questions above aim to clarify when AURA removes identifying details while preserving the meaning that makes a de-anonymization useful. Is there basically an idea of abstention here?
3 comments
Response to Reviewer JUpK, Part 2 of 2: Less-Identifiable Interviews, Topic Breakdown, and Clue Types
W2 Part A and Q2: Behavior When the Initial Attacker Cannot Identify the Participant Setup: From the 1,197 public Anthropic Interviewer transcripts where our multi-round web-search protocol found no confirmed real-world identity, we drew a length-matched, domain-stratified sample of 10 unidentifiable transcripts covering 6 Science, 3 Work, and 1 Creativity. We extracted 55 validated ground-truth profile facts via the identical pipeline, ran AURA + adaptive privacy, and compared edit rates and utility retention against the 53 identifiable transcripts:
| Cohort | Method | Initial Probe | Added Attrs | Word Edit Rate, Med / Mean | User Turns Edited | Profile Retention |
|---|---|---|---|---|---|---|
| 10 Unidentifiable | AURA + adapt. | 0/10 identified | 4 / doc | 0.060 / 0.067 | 0.76 | 0.72 |
| 53 Identifiable | AURA + adapt. | 53/53 identified | 0 to 4 / doc | 0.071 / 0.076 | 0.79 | 0.73 |
| 53 Identifiable | Adv. Anonymizer | 53/53 identified | - | 0.052 / 0.058 | 0.79 | 0.70 |
| 53 Identifiable | RUPTA, ACL '25 | 53/53 identified | - | 0.091 / 0.119 | 0.78 | 0.51 |
| 53 Identifiable | Presidio NER | 53/53 identified | - | 0.003 / 0.004 | 0.26 | 0.99 |
- Why AURA still modify rather than leaving text unchanged: All 1,250 interviews follow the same protocol eliciting projects, workflows, and tools, and being unidentified today does not guarantee safety against future web indexing or acquaintances. On all 10 unidentifiable transcripts, they still inherit concrete quasi-identifiers such as niche subfields, software stacks, employer types, and career stage, adding 4 scope categories.
- No excessive rewriting or extra utility loss: Because AURA masks only flagged spans, it modifies just 6.0% of words at the median, which is lower than 7.1% on the 53 identifiable cases and 9.1% for RUPTA, and retains 0.72 profile and 0.74 occupational facts, matching 0.73 and 0.76 on the 53 cases. Moreover, because Step 1 outputs an explicit confidence flag, data stewards can skip dynamic scope expansion whenever that flag is below threshold.
W2 Part B and Q3: Variation Across Interview Topics and Types of Identifying Clues
- 1. Breakdown by interview topic across $N=53$: Below we compare our main Utility Grid, measuring joint retention of profile and AI-use codebook facts, against downstream coding retention relative to originals across 773 excerpts:
| Topic, $N=53$ | AURA Grid | RUPTA Grid | Adv. Anon. Grid | AA Coding Ret. | AURA Coding Ret. | RUPTA Coding Ret. |
|---|---|---|---|---|---|---|
| Science, $n=31$ | 0.63 | 0.41 | 0.69 | 0.96 | 0.96 | 0.90 |
| Work, $n=17$ | 0.78 | 0.50 | 0.61 | 0.95 | 0.96 | 0.92 |
| Creativity, $n=5$ | 0.93 | 0.61 | 0.80 | 0.98 | 0.98 | 0.99 |
Preserving profile context is hardest in Science, where grid utility is 0.63 vs. 0.78 in Work and 0.93 in Creativity, with the same pattern holding in the 10 unidentifiable transcripts. This is because a scientist's specialized research topic is their public identifier in papers and grants, whereas AURA's downstream coding retention stays consistent across topics at 0.96 to 0.98.
-
2. Specific-source linkage vs. biographical-clue linkage: We audited 10 paired attack trajectories on the same interview and attacker, comparing a run where a baseline such as Presidio was re-identified against a run where AURA prevented re-identification. This comparison reveals two linkage pathways:
- Specific publication or research source in 7 of 10 pairs, all in Science: Attackers combine 3 to 4 niche technical terms, such as detailed working pipeline, company product description, and art piece details etc. , and injury model, to match a paper, thesis, or lab page directly with zero named entities.
- Specific project or product plus biographical clues in 3 of 10 pairs across Science, Work, and Creativity: Attackers query a distinctive project or product, such as an open-source tool or design-asset line, alongside role clues to locate a code repository or online store and its owner. Once AURA abstracts these anchors into category-level terms, attackers lose specific search keywords and resort to searching long verbatim quotes of 8 or more words, which accounts for 31% of queries in AURA misses vs. 10% in hits across 1,846 logged trajectories.
-
3. Does removing these clues destroy semantic meaning? No. AURA retains 96.2% of the expert codes supported in raw transcripts, along with Advanced Anonymizer, non-identity themes remain at 0.92 to 0.95 consistency, and O*NET domain and SOC group retention stay at 0.92 to 0.94. Abstracting overly specific anchors breaks web searchability while keeping qualitative meaning intact.
Thank you for the detailed response and additional analyses. These strengthen my positive assessment.
-
Interpretive fidelity: The thematic-coding analysis and manual audit connect the utility results more directly to qualitative meaning.
-
Less-identifiable interviews: The additional comparison addresses my concern about unnecessary intervention. The limited word-level editing is reassuring, although it demonstrates restrained editing rather than automatic abstention.
-
Topic differences: The breakdown helps explain where the privacy–utility trade-off is hardest, particularly when scientific details both convey useful context and enable identification.
I am maintaining my recommendation of a 3. The response strengthens the empirical case, while the small additional sample and broader generalization of interpretive fidelity remain limitations. Thank you for the careful follow-up.
Response to Reviewer JUpK, Part 1 of 2: Qualitative Meaning vs. Faithfulness, and Necessary vs. Avoidable Loss
We thank Reviewer JUpK for connecting anonymization to qualitative interpretation. We address W1 and Q1 together below, and W2, Q2, and Q3 in Part 2.
W1 and Q1: Factual Faithfulness vs. Qualitative Meaning, and Necessary Removal vs. Avoidable Loss We agree with this distinction. Appendix J's human check, which found 86.9% faithfulness across 122 resolved tasks, tests whether rewrites introduce false factual claims. However, a rewrite with zero false claims can still drop qualifiers, agency, or context that shift how a researcher interprets a participant's views.
Experimental setup for segment-level interpretive shift: Human experts annotated 773 excerpts across the 53 interviews using a 13-code codebook spanning 5 interpretive dimensions: C1 Trust and Delegation, C2 Interaction, C3 Frustrations, C4 Professional Identity and Skills, and C5 Outlook and Autonomy.
difflibsimilarity $r$ into Unedited with $r>0.98$, Lightly edited with $0.8<r\le 0.98$, and Heavily edited with $r\le 0.8$.deepseek-v4-flashat temperature 0, sees the codebook and anonymized transcript, without the original excerpt, and picks the primary code from 4 options: the true code and 3 mid-distance distractors. On originals, this coder matches human experts at $\kappa=0.671$, agrees withqwen3.8-flashon 85.8% of items, and achieves 0.933 repeat self-consistency. Code consistency is the share of excerpts assigned the same code before and after anonymization. A code flip flags a shifted interpretation even when the rewrite is factually faithful.Three findings connect interpretive fidelity to our utility tradeoff and Appendix J:
Manual audit of faithful yet altered interpretations: One author compared side by side all 23 heavily edited flips under AURA and Advanced Anonymizer, and the first 20 of 43 under RUPTA: