- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates speaker-identity unlearning in AudioLLMs, aiming to remove learned voice–identity associations while preserving linguistic content. The authors propose CAUL, a three-stage method that identifies identity-relevant hidden dimensions, aggregates evidence across utterances, and selectively edits model representations. They also introduce VoxSU-Bench, a benchmark built on VoxCeleb, and evaluate the method on Qwen2-Audio and SALMONN.
Critical strengths
-
Interesting and relevant problem. The paper addresses an important and relatively unexplored problem in AudioLLMs. Promising methodology and empirical results. The authors propose both a dedicated unlearning method and a benchmark to evaluate it.
-
Experiments across different AudioLLMs, along with ablation studies, provide encouraging evidence.
-
The novelty claim (" this is the first study of speaker-identity association unlearning for AudioLLMs") is defensible to the best of my knowledge. However, the authors might want to check the latest literature, as it looks like related approaches that are worth citing are coming out (for instance, this paper from Meta)
Critical weaknesses
- Major clarity issues. This is my main concern. Although the basic ideas are relatively intuitive, the paper is too difficult to follow. It took me a long time to understand it well enough for a review. Sections 4.2–4.4 introduce numerous equations without sufficiently explaining the intuition behind the operations, their purpose, or how the three stages interact. For example, a simple running example would clarify identity–content directional effects, cross-utterance evidence aggregation, and sparse local association editing. The paper should also introduce crucial terms more clearly, such as retain subset, retain-directed neutralization, and the identity/content queries. Also, the term channel could be confusing, as it refers to hidden-state dimensions rather than attention heads or convolutional channels. The outputs of each stage should be made explicit. I think the paper needs significant rewriting to make its contributions more accessible and help readers fully appreciate the work.
Questions for authors
- Could the authors provide a concrete end-to-end example illustrating the three stages of CAUL?
- In particular, what exactly are the inputs and outputs of each stage, and how are the identified dimensions used during the final editing procedure?
- What motivates using an exponential moving average rather than a standard arithmetic average?
- Can the authors clarify which benchmark artifacts, implementation details, and evaluation scripts are currently available to reviewers? Providing an anonymized implementation and benchmark manifests would substantially strengthen reproducibility.
Additional Suggestions:
- The authors state that they will release the code and benchmark "in the near future." However, the current availability of these resources is unclear, preventing reviewers from independently inspecting the benchmark and implementation.
- Figure 1 is overly complicated and contains too much text. I strongly recommend simplifying it to provide a clearer, more intuitive overview of CAUL.
- Adding the corresponding references directly to Table 1 and briefly explaining the main differences between the baselines in Section 6.1 would improve readability. More comprehensive descriptions are already available in the appendix.
4 comments
Response to Reviewer XdwM (Part 2/2)
Below we continue our response on EMA aggregation, code and benchmark release, and presentation updates (Part 2/2):
Q3: Exponential Moving Average vs. Arithmetic Mean
- Design Rationale: Written recursively as , an online arithmetic mean has history weight , which locks in early noisy estimates if the first 1–2 sampled clips contain atypical words or background noise. Constant smooths utterance noise while retaining online adaptability and a uniform update scale across speakers without per-speaker counters.
- Momentum Ablation and Direct Comparison: Sweeping on Qwen2-Audio-7B under identical checkpoints, seeds, and denominators achieves
100.00% FRand0.00% ILacross all settings:
| Momentum | 0.00 (Single-Utterance) | 0.25 | 0.50 (Default) | 0.75 | 0.90 |
|---|---|---|---|---|---|
| Consecutive Top- Jaccard ↑ | 0.127 | 0.219 | 0.367 | 0.593 | 0.803 |
| Test CRR ↑ (%) | 94.95 | 94.95 | 95.87 | 95.41 | 94.95 |
CRR peaks at 95.87% for and drops back to 94.95% in the high-rigidity regime approached by an arithmetic mean at larger . In an independent matched-sequence comparison from the same checkpoint, EMA likewise achieves higher test CRR (91.82%) than the online arithmetic mean (91.36%) at identical 100.00% FR and UFR.
Q4&AS1: Availability of Anonymous Code and Benchmark Artifacts
- Anonymous Repository and Scope: "In the near future" in the original abstract referred to public release upon acceptance. Our double-blind anonymous repository is now live at https://anonymous.4open.science/r/CAUL-VoxSU-BB59/ , providing CAUL and all baselines across all three backbones, configurations, evaluation/verification scripts, VoxSU-Bench construction scripts, split manifests, and Qwen2-Audio-7B per-example predictions (trained adapter weights will be released upon acceptance).
S3&AS2-3: Related Work Update and Presentation Improvements
- Recent Audio Unlearning Literature (S3): Section 2 now discusses the recent work from Meta by Liu [1], which studies unlearning private factual knowledge in spoken QA and complements our goal of erasing voice-to-identity associations while preserving linguistic content on the same unmodified waveform. Please let us know if you had another work in mind.
- Figure 1 and Baseline Clarifications (AS2-3): We simplified text and equation annotations in Figure 1 to highlight the three-stage I/O flow, added baseline citations directly in Table 1, and summarized how the five sequence-level baselines differ from CAUL in Section 6.1 (details in Appendix B.1).
Reference
- [1] Liu, Machine Unlearning for Speech Question Answering in Large Audio-Language Models, arXiv:2609.13195, 2026.
We sincerely thank you again for your valuable comments and appreciate the opportunity for you to reconsider the strengths and contributions of our work.
Thank you to the authors for their replies. While I think there is still room to improve clarity (the other reviewers also raised this), I confirm my current weak acceptance score.
Response to Reviewer XdwM
Dear Reviewer XdwM,
Thank you again for your thoughtful response and continued support. We appreciate the care you took in reading our paper and pointing out where our explanations were difficult to follow. Your feedback helped us clarify those passages in the revised manuscript.
To our knowledge, this is the first study of how audio language models can unlearn the link between a speaker’s voice and their identity while preserving their understanding of what was said. We hope this work provides a starting point for others studying speaker privacy. Our discussion with you has also helped us identify questions we would like to explore next.
Best regards,
The Authors
Response to Reviewer XdwM (Part 1/2)
We sincerely thank you for your positive assessment and constructive suggestions. Following your feedback, we have restructured Section 4 in the revised manuscript, simplified Figure 1, expanded the related work and Table 1 baseline descriptions, and added a complete step-by-step numerical walkthrough box in Appendix A.2. Below we clarify each question (Part 1/2):
W1&Q1-2: Concept Definitions, Stage-wise I/O, and Numerical Walkthrough (Updated in Section 4 & Appendix A.2)
Channel: A single hidden-feature dimension d∈{1,…,D} in the audio-token representation matrix Z(x)=[z1,…,zT]∈RT×D at the cross-modal interface (e.g., D=4096 after LM Layer 1 in Qwen2-Audio-7B), rather than an attention head or convolutional channel.id10022across Stages I–III.