acceptodds
Under review as a conference paper at ICLR 2027

Forgetting Who, Preserving What: Speaker-Identity Unlearning in Audio-Language Models

Abstract

Audio-language models jointly encode who is speaking and what is said, creating a practical deletion request: remove a target speaker-identity association from a deployed model while preserving the linguistic value of the same recordings. Existing sequence-level unlearning objectives do not distinguish identity-bearing acoustic evidence from content-bearing evidence and can therefore trade stronger forgetting for broad representation damage. To address this challenge, we introduce CAUL, a selective model-side unlearning framework for AudioLLMs. It consists of three stages: (i) Intervention-Aligned Support Localization, which identifies acoustic channels whose neutralization suppresses target-identity evidence more than linguistic content; (ii) Cross-Utterance Evidence Aggregation, which aggregates channel-level evidence across utterances to obtain stable speaker-specific support; and (iii) Sparse Local Association Editing, which neutralizes the selected acoustic support toward a retain-derived reference and jointly optimizes target-identity unlikelihood with retain KL to preserve non-target behavior. Alongside CAUL, we establish VoxSU-Bench, a speaker-unlearning benchmark built from VoxCeleb datasets. Experimental results on VoxSU-Bench show that CAUL achieves strong speaker-identity forgetting while preserving high speech-content retention. To the best of our knowledge, this is the first study of speaker-identity association unlearning for AudioLLMs. We will release the code and benchmark in the near future.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Audio-Language Models;Machine Unlearning;Trustworthy AI;Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XdwMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates speaker-identity unlearning in AudioLLMs, aiming to remove learned voice–identity associations while preserving linguistic content. The authors propose CAUL, a three-stage method that identifies identity-relevant hidden dimensions, aggregates evidence across utterances, and selectively edits model representations. They also introduce VoxSU-Bench, a benchmark built on VoxCeleb, and evaluate the method on Qwen2-Audio and SALMONN.

Critical strengths
  • Interesting and relevant problem. The paper addresses an important and relatively unexplored problem in AudioLLMs. Promising methodology and empirical results. The authors propose both a dedicated unlearning method and a benchmark to evaluate it.

  • Experiments across different AudioLLMs, along with ablation studies, provide encouraging evidence.

  • The novelty claim (" this is the first study of speaker-identity association unlearning for AudioLLMs") is defensible to the best of my knowledge. However, the authors might want to check the latest literature, as it looks like related approaches that are worth citing are coming out (for instance, this paper from Meta)

Critical weaknesses
  • Major clarity issues. This is my main concern. Although the basic ideas are relatively intuitive, the paper is too difficult to follow. It took me a long time to understand it well enough for a review. Sections 4.2–4.4 introduce numerous equations without sufficiently explaining the intuition behind the operations, their purpose, or how the three stages interact. For example, a simple running example would clarify identity–content directional effects, cross-utterance evidence aggregation, and sparse local association editing. The paper should also introduce crucial terms more clearly, such as retain subset, retain-directed neutralization, and the identity/content queries. Also, the term channel could be confusing, as it refers to hidden-state dimensions rather than attention heads or convolutional channels. The outputs of each stage should be made explicit. I think the paper needs significant rewriting to make its contributions more accessible and help readers fully appreciate the work.
Questions for authors
  • Could the authors provide a concrete end-to-end example illustrating the three stages of CAUL?
  • In particular, what exactly are the inputs and outputs of each stage, and how are the identified dimensions used during the final editing procedure?
  • What motivates using an exponential moving average rather than a standard arithmetic average?
  • Can the authors clarify which benchmark artifacts, implementation details, and evaluation scripts are currently available to reviewers? Providing an anonymized implementation and benchmark manifests would substantially strengthen reproducibility.

Additional Suggestions:

  • The authors state that they will release the code and benchmark "in the near future." However, the current availability of these resources is unclear, preventing reviewers from independently inspecting the benchmark and implementation.
  • Figure 1 is overly complicated and contains too much text. I strongly recommend simplifying it to provide a clearer, more intuitive overview of CAUL.
  • Adding the corresponding references directly to Table 1 and briefly explaining the main differences between the baselines in Section 6.1 would improve readability. More comprehensive descriptions are already available in the appendix.
4 comments
AuthorsOpenReview ↗

Response to Reviewer XdwM (Part 1/2)

We sincerely thank you for your positive assessment and constructive suggestions. Following your feedback, we have restructured Section 4 in the revised manuscript, simplified Figure 1, expanded the related work and Table 1 baseline descriptions, and added a complete step-by-step numerical walkthrough box in Appendix A.2. Below we clarify each question (Part 1/2):

W1&Q1-2: Concept Definitions, Stage-wise I/O, and Numerical Walkthrough (Updated in Section 4 & Appendix A.2)

  • Clarification of Core Concepts:
    • Meaning of Channel: A single hidden-feature dimension d∈{1,…,D}d \in \{1,\dots,D\} in the audio-token representation matrix Z(x)=[z1,…,zT]∈RT×DZ(x)=[z_1,\dots,z_T]\in\mathbb{R}^{T\times D} at the cross-modal interface (e.g., D=4096D=4096 after LM Layer 1 in Qwen2-Audio-7B), rather than an attention head or convolutional channel.
    • Paired Identity and Content Queries: For the same utterance xx, we pair an identity query qq asking for speaker ss with a content query qctq_ {\mathrm{ct}} targeting the frozen pseudo-transcript cc (generated once before identity acquisition by clean pretrained Qwen2-Audio, Appendix E.3) to compute identity and content log-likelihoods ℓid,ℓct\ell_ {\mathrm{id}}, \ell_ {\mathrm{ct}} in Eq. (3).
    • Retain Subset Rp\mathcal{R}_p and Retain-Directed Neutralization: We randomly sample M=5M=5 fixed utterances from R\mathcal{R} to form Rp\mathcal{R}_p and average across frames and samples to obtain the neutral reference zˉret∈RD\bar{z}_ {\mathrm{ret}} \in \mathbb{R}^D. Eqs. (4)–(5) interpolate dimension dd along zt,d→zˉret,dz_ {t,d} \to \bar{z}_ {\mathrm{ret},d} and measure first-order directional effects adid,adcta_d^{\mathrm{id}}, a_d^{\mathrm{ct}}, separating identity suppression from transcription harm in a way undirected gradient norms cannot.
  • Stage-wise Inputs/Outputs and How DsD_s Is Used in Final Editing:
    • Stage I (Support Localization, Eqs. 2–6): Takes starting parameters θ0\theta_0, utterance representation Z∈RT×DZ\in\mathbb{R}^{T\times D} for target speaker ss's kk-th clip, paired queries (q,qct)(q, q_ {\mathrm{ct}}), and neutral reference zˉret\bar{z}_ {\mathrm{ret}}, applies the one-sided trade-off rd=adid−λctmax⁡(adct,0)r_d = a_d^{\mathrm{id}} - \lambda_ {\mathrm{ct}}\max(a_d^{\mathrm{ct}}, 0) to penalize only content-harming dimensions, and outputs utterance score vector r(k)∈RDr^{(k)} \in \mathbb{R}^D.
    • Stage II (Evidence Aggregation, Eqs. 7–8): Takes r(k)r^{(k)} and speaker accumulator Cs(k−1)C_s^{(k-1)}, smooths single-utterance lexical and prosodic noise via online EMA Cs(k)=μCs(k−1)+(1−μ)r(k)C_s^{(k)} = \mu C_s^{(k-1)} + (1-\mu)r^{(k)}, selects the top K=max⁡(1,round⁡(ρD))K=\max(1,\operatorname{round}(\rho D)) dimensions, and outputs sparse speaker support Ds(k)⊂{1,…,D}D_s^{(k)} \subset \{1,\dots,D\}.
    • Stage III (Local Editing, Eqs. 9–12): DsD_s serves as a coordinate-slice mask over hidden dimensions: at each audio token tt, LneuL_ {\mathrm{neu}} slices only the KK-dimensional subvector Zθ,t,Ds∈RKZ_ {\theta,t,D_s}\in\mathbb{R}^K for cosine alignment with zˉret,Ds∈RK\bar{z}_ {\mathrm{ret},D_s}\in\mathbb{R}^K (leaving all D−KD-K unselected dimensions untouched by LneuL_ {\mathrm{neu}}), while jointly optimizing LULL_ {\mathrm{UL}} and LretL_ {\mathrm{ret}} on shared LoRA θ\theta to output Mθ^M_ {\hat{\theta}}.
  • End-to-End Numerical Example: We added a full two-utterance numerical walkthrough (D=20,K=2D=20, K=2 under default hyperparameters) in Appendix A.2 (Pages 16–17, blue box) of the revised manuscript, tracing concrete vectors and coordinate-sliced neutralization for speaker id10022 across Stages I–III.
AuthorsOpenReview ↗

Response to Reviewer XdwM (Part 2/2)

Below we continue our response on EMA aggregation, code and benchmark release, and presentation updates (Part 2/2):

Q3: Exponential Moving Average vs. Arithmetic Mean

  • Design Rationale: Written recursively as rˉs(k)=μkrˉs(k−1)+(1−μk)r(k)\bar{r}_s^{(k)} = \mu_k \bar{r}_s^{(k-1)} + (1-\mu_k)r^{(k)}, an online arithmetic mean has history weight μk=k−1k→1\mu_k = \frac{k-1}{k} \to 1, which locks in early noisy estimates if the first 1–2 sampled clips contain atypical words or background noise. Constant μ=0.5\mu=0.5 smooths utterance noise while retaining online adaptability and a uniform update scale across speakers without per-speaker counters.
  • Momentum Ablation and Direct Comparison: Sweeping μ\mu on Qwen2-Audio-7B under identical checkpoints, seeds, and denominators achieves 100.00% FR and 0.00% IL across all settings:
Momentum μ\mu0.00 (Single-Utterance)0.250.50 (Default)0.750.90
Consecutive Top-KK Jaccard ↑0.1270.2190.3670.5930.803
Test CRR ↑ (%)94.9594.9595.8795.4194.95

CRR peaks at 95.87% for μ=0.50\mu=0.50 and drops back to 94.95% in the high-rigidity regime 0.75–0.900.75\text{--}0.90 approached by an arithmetic mean at larger kk. In an independent matched-sequence comparison from the same checkpoint, EMA likewise achieves higher test CRR (91.82%) than the online arithmetic mean (91.36%) at identical 100.00% FR and UFR.

Q4&AS1: Availability of Anonymous Code and Benchmark Artifacts

  • Anonymous Repository and Scope: "In the near future" in the original abstract referred to public release upon acceptance. Our double-blind anonymous repository is now live at https://anonymous.4open.science/r/CAUL-VoxSU-BB59/ , providing CAUL and all baselines across all three backbones, configurations, evaluation/verification scripts, VoxSU-Bench construction scripts, split manifests, and Qwen2-Audio-7B per-example predictions (trained adapter weights will be released upon acceptance).

S3&AS2-3: Related Work Update and Presentation Improvements

  • Recent Audio Unlearning Literature (S3): Section 2 now discusses the recent work from Meta by Liu [1], which studies unlearning private factual knowledge in spoken QA and complements our goal of erasing voice-to-identity associations while preserving linguistic content on the same unmodified waveform. Please let us know if you had another work in mind.
  • Figure 1 and Baseline Clarifications (AS2-3): We simplified text and equation annotations in Figure 1 to highlight the three-stage I/O flow, added baseline citations directly in Table 1, and summarized how the five sequence-level baselines differ from CAUL in Section 6.1 (details in Appendix B.1).

Reference

  • [1] Liu, Machine Unlearning for Speech Question Answering in Large Audio-Language Models, arXiv:2609.13195, 2026.

We sincerely thank you again for your valuable comments and appreciate the opportunity for you to reconsider the strengths and contributions of our work.

Thank you to the authors for their replies. While I think there is still room to improve clarity (the other reviewers also raised this), I confirm my current weak acceptance score.

AuthorsOpenReview ↗

Response to Reviewer XdwM

Dear Reviewer XdwM,

Thank you again for your thoughtful response and continued support. We appreciate the care you took in reading our paper and pointing out where our explanations were difficult to follow. Your feedback helped us clarify those passages in the revised manuscript.

To our knowledge, this is the first study of how audio language models can unlearn the link between a speaker’s voice and their identity while preserving their understanding of what was said. We hope this work provides a starting point for others studying speaker privacy. Our discussion with you has also helped us identify questions we would like to explore next.

Best regards,
The Authors

Reviewer_4WaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this manuscript, the authors provide a method and a benchmark to unlearn speaker identities in audio language models. The authors use a small portion of VoxCeleb 1 to create the benchmark and evaluate three audio LLMs and different adaptation methods, showing that the proposed CAUL method provides good speaker-identity forgetting in the model while preserving content. Whereas the authors consider enforcing identity retention for other speakers, the results are unclear in this respect.

Critical strengths

This study addresses a gap in current research in audio LLMs: the tension between forgetting speaker identity and preserving the linguistic content of the same recording. The proposed method, CAUL, seems to provide a good balance between keeping the content and forgetting identity.

Critical weaknesses

Considering that the authors have access to more than 1000 speakers, it is unclear why they only select 40 for the benchmark, and most of the results seem to be reported with only 10 speakers, which would limit the generalization of the proposed method. This number should be better justified or, at least, the authors should provide some evidence about the sensitivity to speaker selection.

The authors establish three speaker sets: forget (n=10), retain (n=20), and neutral (n=10). Considering the metrics proposed in section 5 and the results of section 6, it is unclear whether any of the metrics or results clearly reflect the percentage of the retain or neutral speakers that are correctly identified after unlearning.

Questions for authors

In the abstract, the authors say “we will release the code and benchmark in the near future.” It is unclear what the “near future” is. Is it after acceptance? In the reproducibility statement, the authors say “The accompanying implementation provides the experiment configurations, training and evaluation scripts, benchmark manifests, and per-example outputs needed to reproduce the reported results” but the accompanying implementation only includes that information partially.

The need for these methods could be better justified. It is unclear to which extent current audio LLMs contain information about the speaker. Does literature analyze whether results on deliberately acquired associations transfer to associations acquired during ordinary model training?

Section 2 lists works that might be relevant but there is not a critique of which ones are better/worse and why. For instance, the authors mention the work He et al. 2025 but they do not indicate which models were evaluated and to which extent public figures could be identified, which can motivate this work. A more clear explanation on how important this is in current models will be very relevant. For instance, according to appendix section E6, the authors indicate that they have to “teach” the evaluated models about the identities of the target speakers so then they can “forget” them. Is there any clear evidence that models learn identity without being explicitly taught?

Considering that VoxCeleb contains more than 1000 speakers, it is unclear why the authors only select 40 for the benchmark. Also, most results seem to be referred to the 10 speakers from the forget group. This number should be better justified or, at least, the authors should provide some evidence about the sensitivity to speaker selection.

The authors establish three speaker sets: forget, retain, and neutral. Considering the metrics proposed in section 5 and the results of section 6, it is unclear if any of the metrics or results clearly reflect the percentage of the retain or neutral speakers that are correctly identified after unlearning. In the same sense, the authors say “For the reported experiments, identity acquisition uses the 80 training clips of the 10 target speakers” in appendix E6. I understand this is related to the forget set only. Aren’t the authors evaluating the retain and neutral speakers too? It is unclear what portion of the experiments/results is related to which speaker set or how these sets are combined to provide a single result.

In line 322, the authors define TFR as “the percentage of transformed inputs correctly identified before unlearning that are no longer correctly identified after unlearning”. It is unclear what “correctly identified” means in this context. The authors should specify whether this is related to the speaker’s identity, content, both, or something else.

Some minor comments: The authors mention the usage of “frozen pseudo transcripts” (line 216 and 313). Clearly describing this term within the main text will improve the manuscript’s readability.

Not all the the notation used in equation (2) is described (D and z_t are not clearly described, for instance).

In a similar manner, the authors say that Mid is the model after identity acquisition (line 146), and then, in line 216, the authors say that the identity-acquired model is \Theta_0. Are these the same model? Something similar could be said about the unlearning model, mentioned in line 262.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4WaT (Part 1/2)

Thank you for your thorough review and constructive feedback. In the revised manuscript, we expanded the related work critique, notation and benchmark clarifications, and updated the speaker-group generalization and non-target identity retention studies in Section 6.6 (Tables 5 and 6). Below we address your questions point by point (Part 1/2):

Q1: Code and Benchmark Release

  • Anonymous Repository Now Public: "In the near future" in the original abstract referred to releasing the public repository upon acceptance. Our complete double-blind anonymous repository is now open at https://anonymous.4open.science/r/CAUL-VoxSU-BB59/ .
  • Released Artifacts: Except for trained adapter weights to be released upon acceptance, the repository provides full implementations of CAUL and all five baselines across all three backbones, configurations, evaluation and offline verification scripts, VoxSU-Bench construction pipelines, split manifests, and Qwen2-Audio-7B per-example predictions and qualitative comparisons.

Q2-3: Off-the-Shelf Speaker Identification, Literature Critique, and Controlled Acquisition

  • Evidence of Voice Identification in Off-the-Shelf Models: He et al. [1] evaluated commercial AudioLLMs such as GPT-4o after removing lexical identity cues, showing that GPT-4o identifies Donald Trump, Kamala Harris, and Barack Obama from voice alone at 100.0%, 72.1%, and 64.4% accuracy, respectively. This confirms that ordinary training spontaneously encodes voice-to-identity associations.
  • Critique of Related Paradigms in Section 2: Waveform anonymization alters input audio at the cost of signal quality without erasing internal model memory. Sequence-level unlearning (GA, NPO, SCRUB) directly suppresses output tokens and damages speech transcription because identity and lexical content share the same speech representation. Existing multimodal representation unlearning (e.g., MMUnlearner) is more stable than output suppression, yet does not exploit conditional directional differences between paired identity and content queries on the same utterance to separate acoustic channels, which is why CAUL performs substantially better.
  • Why Controlled Identity Acquisition Is Used: Public pretraining corpora expose individuals unevenly, making raw-model evaluation unable to distinguish "the model never knew the speaker" from "unlearning genuinely succeeded." Following TOFU [2], controlled acquisition establishes a uniform, verifiable starting point across methods.
  • Transferability to Ordinary Training and Dense Fine-Tuning Verification: Borisiuk et al. [3] show that native pretraining associations are more broadly distributed than fine-tuned ones and more prone to utility collapse under direct output suppression, while WMDP [4], evaluated directly on native pretrained models, shows that localized hidden-state interventions are far more stable and effective than sequence-level suppression, aligning with CAUL's design. Moreover, identity weights learned across all language-model layers are merged into the backbone before unlearning, and CAUL localizes channels solely from hidden states regardless of acquisition parameterization. Under the exact same all-LM-layer q_proj/v_proj scope and multi-epoch setup on Qwen2-Audio-7B, we compare LoRA acquisition against 1.07B non-LoRA dense fine-tuning:
Identity Acquisition ModeAcquisition ParamsPre-Unlearning ILPost-CAUL IL ↓Post-CAUL FR ↑Post-CAUL UFR ↑
Matched-Scope LoRA Control4.19M37.75% (171/453)0.22% (1/453)98.31% (58/59)100.00% (112/112)
Non-LoRA Dense Fine-Tuning1.07B28.04% (127/453)4.42% (20/453)86.36% (38/44)87.95% (73/83)

Even when identity associations are written directly into 1.07B dense parameters, CAUL reduces IL from 28.04% to 4.42%, erasing 86.36% of seen-prompt and 87.95% of unseen-prompt associations.

  • [1] He et al., The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege, AIES 2025.
  • [2] Maini et al., TOFU: A Task of Fictitious Unlearning for LLMs, COLM 2024.
  • [3] Borisiuk et al., Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning, Findings of ACL 2026.
  • [4] Li et al., The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, ICML 2024.
AuthorsOpenReview ↗

Response to Reviewer 4WaT (Part 2/2)

Below we continue on benchmark scale, speaker sets, and notation (Part 2/2):

W1&Q4: Benchmark Scale and Speaker-Selection Sensitivity (Section 6.6, Table 5)

  • Benchmark Scale: Our 40-speaker scale matches representative multimodal/speech unlearning benchmarks: MMUBench in SIU [5] covers 20 visual concepts, and zero-shot TTS unlearning [6] evaluates 1, 3, and 10 target speakers. Figure 2 also scales from 3 to 20 target speakers, CRR spans all 385 test clips across 40 speakers, and our released scripts support custom pool expansion.
  • Three Disjoint 40-Speaker Groups: On Qwen2-Audio-7B, we sample three mutually disjoint 40-speaker groups outside the original 40 speakers (120 new speakers, each 10 forget / 20 retain / 10 neutral) under Table 1's hyperparameters and video-disjoint protocol:
MethodIL ↓ (%)FR ↑ (%)UFR ↑ (%)CRR ↑ (%)TFR ↑ (%)
GA0.00 ± 0.00100.00 ± 0.00100.00 ± 0.0027.17 ± 45.29100.00 ± 0.00
GA+KL18.06 ± 3.7334.63 ± 20.1229.37 ± 20.2897.79 ± 1.0426.28 ± 18.12
NPO0.00 ± 0.00100.00 ± 0.00100.00 ± 0.0027.51 ± 45.88100.00 ± 0.00
SimNPO0.00 ± 0.00100.00 ± 0.00100.00 ± 0.0027.23 ± 46.11100.00 ± 0.00
SCRUB11.87 ± 10.3772.46 ± 28.2765.92 ± 23.4895.11 ± 1.4274.56 ± 30.84
CAUL0.83 ± 1.4498.41 ± 2.7595.67 ± 7.5095.20 ± 3.5796.61 ± 4.96

Across all three new groups, only CAUL consistently maintains strong forgetting alongside 95.20% CRR.

W2&Q5: Speaker-Set Roles and Non-Target Identity Retention (Section 6.6, Table 6)

  • Main Evaluation Roles: Table 1 evaluates identity erasure (IL, FR, UFR, TFR) on F\mathcal{F} and pools all F∪R∪N\mathcal{F}\cup\mathcal{R}\cup\mathcal{N} test clips with pre-unlearning WER <= 0.30 for global CRR (Appendix E.9, Table 8 original / Table 11 revised, reports per-split CRR).
  • 40-Speaker Joint Acquisition: Original Appendix E.6 enrolled only F\mathcal{F} to keep non-target sets as pure content controls. Per your suggestion, we jointly acquire all 40 identities across F∪R∪N\mathcal{F}\cup\mathcal{R}\cup\mathcal{N} on Qwen2-Audio-7B and unlearn only F\mathcal{F} (N\mathcal{N} unseen during unlearning), measuring retained identity hits on R\mathcal{R} and N\mathcal{N}:
MethodF\mathcal{F} FR ↑F\mathcal{F} UFR ↑R\mathcal{R} Ret. ↑N\mathcal{N} Ret. ↑
GA100.00% (21/21)100.00% (30/30)0.00% (0/29)0.00% (0/29)
GA+KL80.95% (17/21)100.00% (30/30)48.28% (14/29)0.00% (0/29)
NPO100.00% (21/21)100.00% (30/30)0.00% (0/29)0.00% (0/29)
SimNPO100.00% (21/21)100.00% (30/30)0.00% (0/29)0.00% (0/29)
SCRUB9.52% (2/21)16.67% (5/30)93.10% (27/29)100.00% (29/29)
CAUL100.00% (21/21)100.00% (30/30)55.17% (16/29)48.28% (14/29)

GA, NPO, and SimNPO drop R\mathcal{R} and N\mathcal{N} retention to 0.00%, GA+KL drops to 0.00% on unseen N\mathcal{N}, and SCRUB retains non-target identities only because target FR is 9.52%. Only CAUL achieves 100.00% FR and UFR while preserving 55.17% on R\mathcal{R} and 48.28% on unseen N\mathcal{N}.

Q6-9: Metric and Notation Clarifications

  • "Correctly Identified" in TFR (Q6): Strictly correctly identifying the target speaker's identity, not speech content: under noise, speed, or volume perturbations, TFR measures the fraction of pre-unlearning identity hits no longer outputting that target identity (CRR measures content separately).
  • Frozen Pseudo-Transcripts (Q7, Appendix E.3): Deterministically generated once across all 705 clips before acquisition using clean pretrained Qwen2-Audio and frozen throughout acquisition, unlearning, and evaluation.
  • Eq. (2) Notation (Q8): In Z(x)=[z1,…,zT]∈RT×DZ(x) = [z_1, \dots, z_T] \in \mathbb{R}^{T \times D}, TT is audio-token length, DD is hidden dimension, and zt∈RDz_t \in \mathbb{R}^D is the tt-th audio token's hidden state.
  • Model/Parameter Notation (Q9, Appendix A.1): Yes, MidM_{\mathrm{id}} and θ0\theta_0 denote the same pre-unlearning checkpoint (Mid=Mθ0M_{\mathrm{id}} = M_{\theta_0}), M0M_0 is the clean pretrained model, MθM_{\theta} is the model during unlearning, and Mθ^M_{\hat{\theta}} is the final unlearned model.

References

  • [5] Li et al., Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models, NeurIPS 2024.
  • [6] Kim et al., Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech, ICML 2025.

Thank you again for your valuable feedback, and we appreciate your reconsideration of our work.

Thanks for your response. Firstly, could the authors confirm whether they have included all these explanations and where(section, line, etc.)? It is unclear whether all the responses they provide led to a clear action in the paper. Secondly, the fact that the number of speakers considered matches the numbers of other benchmarks is not a justification for whether that number provides results that would be representative of the problem you are trying to solve.

Reviewer_vvUWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses content preserving speaker identity association unlearning in AudioLLMs for the first time, with a LoRA bsed unlearning method CAUL and a modified benchmark VoxSU-Bench from previously existing benchmark VoxCeleb. The speaker identities as id from benchmark is explicitly injected into the model for finetuning with LoRA to create association then unlearned. Results are tested on multile AudioLLMs.

Critical strengths
  1. This paper is the first to suggest identity-voice association unlearning in Audio LLMs, where unlearning target is the association of the voice's speaker characteristic to its mapping of speaker identity. They define why Audio LLM is a challenging case.

  2. A reproducible benchmark is provided for further evaluation that seems to be a contribution to a new task the paper is suggesting.

Critical weaknesses
  1. Motivation. I am not very convinced with the motivation of this work. The use case is mentioned in Section 1 as consent withdrawal, account deactivation, personalization updates.However the suggestion is to keep the content of what a speaker said. If a speaker who may have been used as a train set wish to be removed, should the content not also be removed? The paper does not justify this asymmetry.

  2. Practicality. The method utilizes a 'neutralization target' as a mean over 'five retained utterances'. However, there is no selection criterion or ablation mentioned in the paper. How are these targets selected?

  3. Continuing from above, how does this method affect the retain-speaker identification after unlearning (misattribution of forget audio to a retain speaker). A concerning case would be when an unlearned speaker A being mapped to a specific retain speaker B constantly, which poses vulnerability in traceability of this unlearning.

  4. Experiment Setup. The experiment seems to rely on specific speaker identifiers - a VoxCeleb ID string rather than matching the voice to a real identity (a sample of the voice). This seems like a critical gap, the model is simply unlearning the label associated with the dataset not the actual identity. If this is the intended scope, the practicality and robustness of the resulting unlearning are questionable.

  5. Another point in experiment setup. The experiment seems to be relying on shallow, recent, localized finetuned model. The support localization also operates on first LM layer, while a pretrained speaker identity knowledge would realistically be distributed across the encoder and LM layers. This evaluation setting is very synthetic in a way makes the problem much easier - identities are ID strings injected by 50 step LoRA and the unlearning step can be performed on these pre-localized finetuned layers.

  6. Baselines are run for fixed 50 steps with untuned hyperparameters rather than compared at matched forgetting. Figure 3 shows GA/NPO/SimNPO reach 100% FR by step 20 with higher CRR than CAUL at that point; Table 1 reports them after collapse. A fair comparison requires early stopping or a Pareto curve of FR vs. CRR per method with reasonable hyperparameters per method.

  7. For supplementary : code is provided, but no qualitative examples (e.g., model outputs before and after unlearning) are included, which limits validation of the method beyond aggregate metrics despite the reproducibility statement claiming to include per-example outputs.

Questions for authors
  1. Could the authors justify why linguistic content from recordings whose consent was withdrawn must be retained for model usage? It is currently unclear, especially considering the authors mentioned right to be forgotten in the motivation.

  2. Could the authors provide explanation on how the neutralization targets are selected - and evidence on how sensitive the results are based on the method of neutralization target selection?

  3. The paper mentiones that eight clips per forget speaker are used for unlearning, but Table 3 shows single-utterance localization performs almost identically. What is the contribution of the cross-utterance aggregation stage? Could there be an ablation for this part?

  4. In Fig 2, the authors show a forget-size varied experiment from 3 to 20 while fixing optimization budget to 50 in a simultaneous way. Few things are unclear - does this mean step size did not need to be scaled along with the increased forget set size? If so, could there be an ablation on how few of step sizes you need to sufficiently unlearn one speaker?

  5. Could the authors justify or prove that the model does not recognize an identity after unlearning - rather than the evidence being the model no longer produces ID string?For example,by testing whether two utterances from the same speaker are still matched or a map of speaker embedding or representation space state that show identities are pushed away from its original embedding/ mismatched across multiple speech inputs from a same speaker could convince my decision.

  6. Could the authors provide a search sweep experiment of hyperparameters for baselines? I believe this is important because each method poses different strength of unlearning and utility collapse, and unlearning methods are significantly affected from hyperparameter tuning - yet this paper uses fixed hyperparameters across all methods rather than specific tuning.

  7. Could the authors test this not on a LoRA-finetuned setup, and rather a controlled finetuned model? This would help isolate affects of the different coverage / exposure of celebrities in pretraining as the authors mention themselves in appendix F. The evidence that this unlearning method will scale to an actual model (that would most likely not be simply 50 steps of LoRA finetuned) is not concrete. Also - how does the pre-injection/pre-finetuned Audio models (which would be considered somewhat exactly-unlearned) actually perform on these speakers?

  8. The authors mention in L68-74, that Audio LM unlearning is especially challenging compared to previous unlearning in speech domain. However other speech domain works face the shared problem of a phonetic information being entangled with learned speech representation and the identity being possibly dependent on the content [1]. This disentanglement of speaker identity from actual speech content seems likely to be a shared problem, not something that makes AudioLLM especially challenging. Could the authors elaborate or point out the distinction of Audio LLM case?

[1] Kim et al., Speaker identity unlearning for zero-shot text-to-speech. 2025

6 comments
AuthorsOpenReview ↗

Response to Reviewer vvUW (Part 1/3)

Thank you for your thorough review and constructive feedback. In the revised manuscript, we clarified our motivation and related work in Section 1, added the per-method validation tuning and early-stopping study in Section 6.5 (Table 4), and included the speaker representation-shift visualization in Appendix C.2 (Figure 4). Below we address your concerns point by point (Part 1/3):

W1&Q1: Motivation for Content-Preserving Identity Unlearning

We apologize for the confusion from placing general consent withdrawal and the right to be forgotten alongside our setting in Section 1. When a speaker revokes consent for private training recordings as a whole, memorized training transcripts should indeed be removed via sample-level unlearning. Here, "preserving linguistic content" means fine-grained selective unlearning, removing voice-to-identity inference while keeping normal speech transcription and comprehension intact:

  • Least-Privilege Deployment and Voice Privacy: End-to-end AudioLLMs unnecessarily expose speaker identities during routine speech tasks and should follow least privilege by stripping identity inference while retaining semantic comprehension [1]. Consistent with classical voice privacy [2], we conceal speaker identity while preserving intelligibility, shifting intervention from external waveform modification to internal model updates.
  • Preventing Transcription Collapse: Fine-grained unlearning erases targeted sensitive associations with minimal utility damage [3, 4]. Because voice cues and phonetics are entangled, sequence-level baselines in Table 1 (GA, NPO, SimNPO) collapse transcription on source-video-disjoint test speech (CRR drops to about 20%) when erasing identity, so evaluating CRR ensures identity unlearning does not break general speech understanding.

W2-3&Q2: Neutralization Reference Selection, Sensitivity, and Misattribution Check

  • Selection Rule and Sensitivity (W2 & Q2): The 5 retain clips are randomly sampled directly from the retain training split under a fixed seed with no manual curation. Per Eq. (2) in Section 4.2, each clip is averaged across audio frames to remove duration differences and then across clips to form zˉret\bar{z}_ {\mathrm{ret}}. Evaluating 3 independent random draws of M=5M=5 plus M=1M=1 and M=20M=20 on Qwen2-Audio-7B yields:
zˉret\bar{z}_ {\mathrm{ret}} SamplingIL ↓ (%)FR ↑ (%)UFR ↑ (%)CRR ↑ (%)
Random draw, M=1M=1 clip0.00100.00100.0093.18
Random draw, M=5M=5 (mean ± std, 3 draws)0.00 ± 0.00100.00 ± 0.00100.00 ± 0.0092.27 ± 0.91
Random draw, M=20M=20 clips0.00100.00100.0095.00

Across random draws and M∈{1,5,20}M \in \{1,5,20\}, FR and UFR stay at 100.00% with minimal CRR variance.

  • Zero Misattribution to Specific Retain Speakers (W3): Eq. (9) aligns only 10% sparse dimensions toward a multi-speaker mean rather than an individual speaker. Across all 453 main-experiment test queries, CAUL misattributes 0.00% (0/453) of forget clips to any retain speaker on both Qwen2-Audio-7B and SALMONN-7B, avoiding speaker A-to-B traceability risks.

Q3: Role of Cross-Utterance Aggregation and Momentum Ablation

  • Eliminating Cross-Seed Residual Leakage (Appendix C.1): Stage I directional scoring separates most identity- and content-sensitive dimensions, while Stage II online EMA needs no extra forward passes and filters single-utterance lexical/background noise. In Appendix C.1 (Table 5 original / Table 8 revised), EMA raises consecutive Top-KK Jaccard overlap from 0.127 to 0.367, stabilizing three-seed FR in Table 3 from 97.22% ± 2.41% to 100.00% ± 0.00%, cutting IL from 0.15% ± 0.13% to 0.00%, and raising CRR from 91.74% to 92.51%.
  • Momentum μ\mu Ablation: Sweeping μ\mu in Eq. (7) on Qwen2-Audio-7B under a fixed seed achieves 100.00% FR and 0.00% IL at all settings:
Momentum μ\mu0.00 (Single-Utterance)0.250.50 (Default)0.750.90
Consecutive Jaccard ↑0.1270.2190.3670.5930.803
Test CRR ↑ (%)94.9594.9595.8795.4194.95

Performance is robust across μ\mu, with default μ=0.50\mu=0.50 balancing denoising and online adaptation to reach the highest CRR (95.87%).

  • [1] He et al., The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege, AIES 2025.
  • [2] Tomashenko et al., The VoicePrivacy 2020 Challenge: Results and Findings, Computer Speech & Language 2022.
  • [3] Warnecke et al., Machine Unlearning of Features and Labels, NDSS 2023.
  • [4] Huo et al., MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models, Findings of ACL 2025.
AuthorsOpenReview ↗

Response to Reviewer vvUW (Part 2/3)

Below we continue our response on the optimization step budget, internal representation shift, and per-method baseline tuning (Part 2/3):

Q4: Fixed 50-Step Budget and Minimum Steps for a Single Speaker

  • Scaling Steps at Larger Forget-Set Sizes: Fixing 50 steps does not imply a constant budget at arbitrary scale. Under round-robin speaker sampling, 50 steps give each of the 10 target speakers 5 cross-utterance updates, and as shown in Figure 3, continuing from step 20 (where FR saturates) to step 50 lets the retain regularizer recover CRR from 66.51% to 95.87%. Figure 2 fixes 50 steps to compare methods under a matched compute budget across 3–20 speakers, whereas at much larger forget sets total steps should scale accordingly.
  • Minimum Steps for Single-Speaker Unlearning: Running ∣F∣=1|\mathcal{F}|=1 single-speaker unlearning on Qwen2-Audio-7B (lr=2e-4 fixed) yields:
Single Target SpeakerStep-5 FR ↑Step-5 UFR ↑Step-5 CRR ↑Steps to 100% FR & UFR
id10377100.00% (13/13)100.00% (26/26)91.36%≤ 5 steps
id10482100.00% (3/3)100.00% (4/4)91.36%≤ 5 steps
id1056483.33% (5/6)100.00% (14/14)90.00%10 steps (6/6)

Sufficiently unlearning one speaker requires only 5–10 steps, consistent with our 50-step budget for 10 speakers.

W4&Q5: Internal Representation Visualization and Cross-Utterance Matching (Appendix C.2, Figure 4)

Section 3 uses canonical identifiers solely to prevent uneven pretraining familiarity with public figures from confounding evaluation. To verify that the model does not merely suppress surface id tokens, we extract internal speaker representations on Qwen2-Audio-7B at the shared prefix "The speaker is" strictly before any identity token is generated, and provide a representation-space visualization in Appendix C.2 (Figure 4, Pages 19–20) comparing the pretrained base model M0M_0, pre-unlearning model MidM_ {\mathrm{id}}, and CAUL-unlearned model Mθ^M_ {\hat{\theta}}:

Model StageCosine Similarity to Pre-Unlearning Identity Centroid ↓Cross-Video Same-Speaker Matching AUC ↓
Pretrained Base M0M_0 (Before Acquisition)0.4570.888
Pre-Unlearning Model MidM_ {\mathrm{id}}0.9050.920
Post-CAUL Model Mθ^M_ {\hat{\theta}}0.7430.868
  • Embedding Visualization Shows Shift Away from Original Centroids (Appendix C.2, Figure 4a): Before unlearning, held-out utterances cluster around each speaker's pre-unlearning identity star centroid (★). After CAUL, embeddings shift consistently away from their ★ centroids and lose speaker separability, dropping cosine similarity to pre-unlearning centroids from 0.905 to 0.743 (nearest-centroid match rate drops from 79.17% to 45.83%).
  • Cross-Video Same-Speaker Matching Returns to Base-Model Level (Appendix C.2, Figure 4b): Pairing held-out clips across disjoint source videos, same-speaker verification AUC from internal cosine similarity rises from 0.888 on M0M_0 to 0.920 after acquisition, whereas CAUL reduces it back to 0.868, returning to the pretrained base-model level.

W6&Q6: Per-Method Tuning, Early Stopping, and Matched-Forgetting Comparison (Section 6.5, Table 4)

  • Disjoint Validation Sweep and Early Stopping: Original baseline hyperparameters followed each method's standard setup (Appendix B.2). Because unconstrained baselines degrade past step 20 while fixed wret=5w_ {\mathrm{ret}}=5 restricts GA+KL forgetting, we construct a source-video-disjoint validation split on Qwen2-Audio-7B and sweep learning rates {5e-5, 1e-4, 2e-4} (2.5e-5 to 2e-4, 5 values for GA), wret∈{1,2,5}w_ {\mathrm{ret}} \in \{1, 2, 5\}, and steps {10, 20, 30, 40, 50} (25 candidate checkpoints per method), selecting each method's checkpoint maximizing validation CRR at validation FR >= 90%:
MethodValidation-Selected ConfigurationIL ↓ (%)FR ↑ (%)UFR ↑ (%)CRR ↑ (%)
GAlr=2e-4, wret=0w_ {\mathrm{ret}}=0, step=200.00100.00100.0079.55
GA+KLlr=2e-4, wret=1w_ {\mathrm{ret}}=1, step=500.66100.0093.7582.27
NPOlr=2e-4, wret=1w_ {\mathrm{ret}}=1, step=500.44100.0095.8380.91
SimNPOlr=2e-4, wret=1w_ {\mathrm{ret}}=1, step=500.8895.8393.7580.91
SCRUBlr=1e-4, wret=1w_ {\mathrm{ret}}=1, step=302.6591.6791.6786.82
CAULlr=2e-4, wret=5w_ {\mathrm{ret}}=5, step=400.00100.00100.0091.82

Tuning and early stopping substantially improve baselines (GA recovers to 79.55% CRR at step 20 and GA+KL reaches 100.00% FR with 82.27% CRR), yet at matched 100.00% FR and UFR, CAUL still achieves 91.82% CRR (+9.55 points over GA+KL and +12.27 over GA).

AuthorsOpenReview ↗

Response to Reviewer vvUW (Part 3/3)

Below we conclude our response regarding full-layer identity acquisition, comparison with zero-shot TTS unlearning, and qualitative outputs (Part 3/3):

W5&Q7: Full-Layer Acquisition, Base Model M0M_0 Reference, and Dense Fine-Tuning

  • Full-Layer Acquisition Without Localization Leakage (Appendix B.2 & E.6): "50 steps" is solely the unlearning budget. Identity acquisition fine-tunes all language-model layers over multiple epochs and merges weights into the backbone before unlearning, so unlearning methods have no access to where acquisition updates occurred. CAUL localizes channels at the cross-modal interface (LM Layer 1 output in Qwen2-Audio, speech projector output in SALMONN) because acoustic features have just entered the language backbone before deep mixing with text tokens, while Stage III updates span both the audio projector and all language-model layers.
  • Pre-Acquisition Base Model M0M_0 Reference: On the same evaluation manifests, unmodified Qwen2-Audio-7B (M0M_0) yields 0.00% target identity hits (and 0.00% when queried with real VoxCeleb celebrity names) while achieving WER <= 0.30 on 216/385 content clips, matching post-CAUL behavior.
  • 1.07B Non-LoRA Dense Fine-Tuning Control: Under the exact same all-LM-layer q_proj/v_proj scope and multi-epoch setup, we unfreeze 1.07B full dense weight matrices for identity acquisition and apply lightweight CAUL (4.19M parameters):
Identity Acquisition ModeAcquisition ParamsPre-Unlearning ILPost-CAUL IL ↓FR ↑UFR ↑
Matched-Scope LoRA Control4.19M37.75%0.22%98.31%100.00%
Non-LoRA Dense Fine-Tuning1.07B28.04%4.42%86.36%87.95%

Even when target identities are written via 256x larger full dense matrices across all layers, CAUL reduces IL from 28.04% to 4.42%, achieving 86.36% FR and 87.95% UFR.

Q8: Shared Challenge and Key Distinction from Zero-Shot TTS Unlearning

  • Shared Entanglement vs. Input Modality Structure: We fully agree that acoustic-phonetic entanglement is a shared fundamental challenge across zero-shot TTS unlearning [5] and our work, and apologize for our imprecise phrasing in Section 1. The key distinction is whether spoken content is supplied via a separate external text input: in zero-shot TTS [5], the model receives two separate inputs ("reference speech + target text to synthesize"), so unlearning a target speaker can ignore the reference speech entirely and synthesize the external text in a random anonymous voice without losing lexical content. In a generative AudioLLM, there is no external text input: spoken words exist only inside the target speaker's waveform itself, so indiscriminately suppressing audio representations collapses lexical comprehension (GA/NPO drop CRR to about 20% in Table 1), requiring sparse channel-level neutralization.

W7: Qualitative Examples and Per-Example Outputs

We have updated our double-blind anonymous repository at https://anonymous.4open.science/r/CAUL-VoxSU-BB59/ , adding qualitative before-and-after cross-method output comparisons (qualitative_examples.md) and full Qwen2-Audio-7B per-example predictions under benchmark/evaluation_outputs/.

  • [5] Kim et al., Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech, ICML 2025.

We sincerely thank you again for your valuable comments and appreciate the opportunity for you to reconsider the strengths and contributions of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

Thank you for the thoughtful discussion of our paper’s central question: How can an audio-language model forget who spoke while preserving its understanding of what was said in the same recording? Reviewer vvUW highlighted the novelty of the task and the reproducibility of VoxSU-Bench. Reviewer 4WaT recognized the research gap and the balance CAUL achieves between identity forgetting and content preservation. Reviewer XdwM highlighted the promise of the method and the breadth of the cross-model experiments and ablation studies. We are encouraged by these assessments.

Your suggestions guided revisions throughout the manuscript, from the explanation of the task and method to the experimental design and interpretation of the results. The revision adds three pages, one in the main text and two in the appendix. We tuned each method independently to compare content retention at matched levels of forgetting, added three cohorts covering 120 new speakers, and examined how unlearning affects non-target speakers when their identities are also enrolled. We also tested CAUL on identities learned through dense fine-tuning. During the discussion, we reported a larger experiment with 100 speakers enrolled and 40 targeted for simultaneous unlearning, along with a speaker-by-speaker analysis of model predictions.

We sincerely thank the Area Chair for coordinating the review and giving us the opportunity to engage with all three reviewers. We are especially grateful to the reviewers, whose careful questions strengthened the experiments and deepened our understanding of how speaker-identity unlearning should be evaluated. Looking ahead, we will continue to study safety, privacy, and ethics in audio-language models, with the hope of offering the community new research directions and contributing to the responsible development and use of these models.

Best regards, The Authors of Submission 18739

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.