- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies post-hoc adversarial robustness installation for speech foundation model descendants. The setting is that a pretrained backbone has already been fine-tuned into task-specific descendants, and the defender only has the descendant weights plus clean unlabeled downstream audio — no labels, no original fine-tuning pipeline, no per-descendant adversarial training. The method has two stages: (1) learn a task-agnostic weight update Δ_rob once per backbone from unlabeled source audio, minimizing adversarial representation drift with an explicit clean-representation anchor (one-sided stop-gradient, fixed anchor); (2) install the same update additively at a fixed untuned scale α=1 into each descendant, freeze the patched encoder, and realign only the linear readout on clean unlabeled audio using pseudo-targets from the original descendant plus a geometric-margin hinge term. Evaluated with a seven-attack pool (including adaptive white-box and psychoacoustic attacks, all targeting the exact installed model) on three backbones and five tasks, the method improves pooled robust accuracy by 48.7 points on average across 12 classification descendants, recovers 79.5% of the task-specific AT gain, with a worst-task clean-accuracy loss of 7.3 points. A drift/margin analysis links reduced representation drift to robust predictions via an attack-relative sufficient margin condition. Amortized compute breaks even with task-specific AT at ~2.2 descendants per backbone.
Critical strengths
- A well-defined, practically relevant problem with a method that fits its constraints exactly. The paper identifies a real deployment scenario — robustness becoming a requirement after fine-tuning, with labels/pipeline unavailable — and the two-stage design (shared backbone update + label-free readout realignment) satisfies every stated constraint without hand-waving: no downstream labels, no downstream adversarial examples, no inference-time overhead. This is consequential because most prior work either assumes robustness is in place before specialization or requires per-target AT; showing that one amortized update can be installed into 12 heterogeneous descendants is the paper's central empirical claim, and the access-matched baselines (interpolation, TIES, FARE-style, patch-only, realign-only) isolate what is actually new.
- Evaluation rigor that is rare in this area. Every attack targets the exact installed model (including the realigned readout); the pool includes adaptive mechanism-aware attacks and query-limited SPSA; Fig. 4a shows monotonic degradation under increasing budget, directly addressing gradient-obfuscation concerns; the method is tested across three backbones, five tasks, full fine-tuning and LoRA, with seed variability and bootstrap errors reported. The complementary drift/margin analysis additionally explains why the method works rather than only that it works. Together these make the 48.7-point / 79.5%-recovery headline numbers credible.
Critical weaknesses
-
Insufficient evidence for fixed generality: Although evaluated across three backbones, five tasks, and a LoRA setting, is stress-tested only on WavLM-large/SID-250; the descriptive analysis of task-vector magnitude, source-to-downstream shift, and robustness transfer uses only four classification descendants, insufficient for a predictive rule. Systematic variation of specialization strength, task-vector norm, and source-target shift, plus failure cases, is needed.
-
Uneven clean-robustness trade-offs: SID-250 is the worst: a 7.3-point clean-accuracy drop, robust accuracy of 40.9% versus 56.8% for task-specific adversarial training, a looser margin bound, and another 7.2-point robust drop without margin regularization. The 5%/10% labeled controls only replace pseudo-labels and do not equal limited-label per-descendant adversarial adaptation; comparison with few-shot supervised/adversarial fine-tuning is needed.
-
Missing direct per-descendant unsupervised robustification baseline: For each fine-tuned descendant, directly optimize the Stage-1 drift-min objective using same-source or downstream unlabeled audio, then perform readout realignment; report performance- and compute-matched variants to quantify the amortization cost of “learn once per backbone.”
-
No evaluation under distribution-shifted readout calibration: Stage 2 assumes clean, in-distribution calibration audio and varies only the amount of same-domain sampling. In deployment, shifts in speakers, channels, noise, language, or class coverage can make teacher predictions unreliable and harm readout fitting. Controlled domain-shift and mixed-domain calibration experiments are needed.
-
Physical over-the-air robustness remains untested: Only digital waveform perturbations are evaluated; it is not shown to survive loudspeaker-microphone transfer, RIRs, background noise, or physical adversarial examples. A playback-and-re-recording study or room/device response simulation is needed.
-
Incomplete drift-margin analysis for CTC ASR: The margin is defined only at the frame-level CTC projection, with no sequence-level robustness condition and no ASR analogue of the classification Figure 3 diagnostic. Frame-level drift/margin statistics, their relationship to token/sequence errors, and failure cases involving CTC alignment changes are needed.
Questions for authors
See weaknesses
5 comments
Part 2/2
W3
We appreciate this suggestion: it directly tests the benefit of sharing one robustness update. For each WavLM-large classification descendant, we optimized the same Stage 1 objective on 100 h of unlabeled Common Voice, starting from and clean-anchoring to its own weights. Both direct variants use our label-free Stage 2 realignment and evaluation.
| Method | KWS (C/R) | IC (C/R) | SID-250 (C/R) | ER (C/R) |
|---|---|---|---|---|
| Ours (shared update) | 95.2/82.1 | 98.2/87.6 | 84.1/40.9 | 63.4/37.1 |
| Direct (4 epochs) | 96.1/85.9 | 98.0/91.7 | 86.2/45.8 | 64.8/41.9 |
| Direct (compute-matched, N=15) | 95.9/41.0 | 98.5/41.7 | 88.6/16.9 | 67.1/15.8 |
Four-epoch direct training achieves 4.4 pp higher mean RA, but costs 78.4 GPU-h across these four tasks, versus 5.8 GPU-h allocated to ours under N=15 amortization (13.5×). The 5.8 GPU-h is an amortized allocation, not the standalone cost of learning the shared patch for just four descendants. At matched amortized compute, direct training has much lower RA on every task. Thus, full-budget direct training can attain higher RA, whereas our reusable update provides substantial robustness at far lower amortized cost (§6, lines 381–387; Appendix F, lines 969–985).
W4
Stage 2 explicitly assumes clean, in-distribution unlabeled audio (§3.1, lines 154–157; §3.3, lines 225–230). Appendix C.1 specifies 45 minutes of held-out in-task calibration (lines 755–761); Appendix H varies calibration quantity, not domain (lines 1054–1063). We did not claim shift-invariant realignment.
To test sensitivity, we held calibration duration (45 minutes), α=1, and Stage 2 fixed, applying 20 dB Gaussian noise, RIR convolution, or 50/50 clean/RIR only to calibration audio. The original descendant generated pseudo-labels on these inputs. We evaluated on the unchanged clean test split and regenerated 37 dB attacks against each realigned model. Entries are ΔCA/ΔRA (pp) relative to standard calibration.
| Task | Clean calib. (C/R) | Gaussian 20 dB | RIR | 50/50 clean/RIR |
|---|---|---|---|---|
| KWS | 95.2/82.1 | −1.0/−1.7 | −2.1/−4.0 | −0.9/−1.6 |
| IC | 98.2/87.6 | −0.4/−0.8 | −0.8/−1.8 | −0.2/−0.5 |
| SID-250 | 84.1/40.9 | −3.5/−4.7 | −5.1/−7.1 | −2.4/−3.2 |
| ER | 63.4/37.1 | −2.3/−2.8 | −3.6/−5.1 | −1.7/−3.0 |
| Mean | — | −1.8/−2.5 | −2.9/−4.5 | −1.3/−2.1 |
The largest RIR-induced RA decrease is on SID-250 (40.9→33.8%), while mixed calibration retains 37.7%. We also varied speaker coverage at fixed duration: restricting its 45-minute pool from 250/250 to 125/250 speakers yields 77.0/35.0% CA/RA versus 84.1/40.9%, with all speakers retained at test time and speaker IDs used only to construct subsets. These results quantify sensitivity to calibration shifts and show that mixing clean and shifted audio mitigates acoustic mismatch.
W5
The manuscript explicitly evaluates digital waveform perturbations (§3.1, lines 184–188) and identifies physical playback/re-recording as a separate, unevaluated setting in §9 (lines 469–475).
To probe beyond that scope, we tested WavLM-large/SID-250 with simulated RIR, with or without 20 dB noise, and no channel-specific training or realignment. Adaptive attacks optimize through the channel at 37 dB source-side SNR.
| Condition | Original (C/R) | Ours (C/R) | ΔRA (pp) |
|---|---|---|---|
| Digital | 91.4/3.2 | 84.1/40.9 | +37.7 |
| RIR only | 85.5/5.7 | 78.0/42.6 | +36.9 |
| RIR + noise (20 dB) | 81.3/5.4 | 74.0/39.5 | +34.1 |
Against channel-aware white-box attacks optimized through simulated RIR and RIR+noise, our method retains RA gains of 36.9 and 34.1 pp, respectively. These tests probe reverberation and noise relevant to physical deployment and over-the-air attacks.
W6
We appreciate the request for CTC alignment analysis. The manuscript explicitly limits the CTC margin argument to frame-level projections, not sequence-level guarantees (§4.2, lines 301–302; Appendix C.2, lines 839–841); sequence robustness is evaluated using adversarial WER. We added WavLM-large ASR diagnostics analogous to Fig. 3, using each model's highest-WER valid seven-attack candidate for both diagnostics and error measurement.
| Model | Frame drift ↓ | Nonblank margin ↑ | Adv. CER (%) ↓ | Adv. WER (%) ↓ |
|---|---|---|---|---|
| Original | 1.00 | 1.00 | 42.1 | 91.5 |
| Patch only | 0.35 | 0.96 | 13.5 | 37.2 |
| Ours | 0.37 | 1.14 | 9.8 | 30.7 |
Frame drift measures mean RMS representation change; margin averages clean geometric margins on frames assigned nonblank tokens by the original descendant. Both indices are separately normalized to the original (=1), so they cannot be compared using the margin > drift inequality. Patching sharply reduces drift; realignment raises margins and lowers CER/WER. A representative defended-model failure changes “nightfall the yellow lamps” to “nightfal the yelo las” under attack, consistent with CTC sensitivity to blank separation, repeated symbols, and missing token spikes. It illustrates how localized alignment changes can affect several words despite low average drift.
Note: We also plan to add the new results and discussion to the paper before the rebuttal deadline.
Follow-up on Our Rebuttal Response
Dear Reviewer GuGn,
Thank you again for your thoughtful and detailed review.
We have posted a two-part, point-by-point response addressing all six concerns raised in your review, including additional experiments on installation-scale sensitivity and robustness transfer under stronger downstream specialization and source-data shifts, comparisons with few-shot adversarial training and direct per-descendant unsupervised robustification, further experiments on readout calibration under distribution shifts, robustness under simulated acoustic channels, and CTC drift–margin analysis.
As the discussion period approaches its conclusion, we would greatly appreciate knowing whether these additional results and clarifications have addressed your main concerns or whether any questions remain. We would be happy to provide further clarification.
Thank you again for your time and consideration.
Best regards,
The Authors
Part 1/2
We thank the reviewer for recognizing the practical value of our post-specialization setting, its adherence to deployment constraints, and the breadth and rigor of our evaluation across backbones, tasks, adaptive attacks, and drift–margin analyses. We appreciate the thoughtful questions, which give us an opportunity to clarify the paper's stated scope and provide additional evidence on the limits of robustness transfer.
Note: All manuscript references below refer to the version on which this review was based. Unless stated otherwise, classification results are WavLM-large clean/pooled seven-attack robust accuracy (CA/RA, %).
W1
As stated in §3.3 (lines 205–208), α=1 is fixed and untuned; we do not claim it is optimal for every descendant. Fig. 4b originally focused on SID-250, which had the least favorable observed clean–robustness trade-off (lines 419–424, 441–444). To examine generality more directly, we extended the α sweep to all four WavLM-large classification tasks, repeating the same 45-minute label-free readout realignment at each setting. Tuned α maximizes validation RA; the table reports test results.
Tuning increases mean RA by 3.2 pp but reduces mean CA by 3.4 pp. On SID-250, α=1.25 adds only 0.4 pp RA while losing 4.0 pp CA. The same pattern holds on a second backbone: wav2vec2-large/SID-250 achieves 81.6/42.1 at α=1 versus 77.6/43.9 at tuned α=1.25. Thus, fixed α=1 offers a favorable untuned trade-off. These sweeps assess tuning sensitivity, deployment retains fixed α=1 without adversarial validation or per-descendant tuning.
Beyond α sensitivity, we tested stronger specialization and changes in Stage 1 source audio. As reported to reviewer SpNU (W1 & Q1), after 12 fine-tuning epochs, the same fixed patch retains RA gains of +60.6 pp on KWS and +32.5 pp on SID-250 (global relative task-vector norms: 2.13% and 2.81%, respectively), although SID-250 incurs a larger clean-accuracy cost. Learning Stage 1 instead from 100 h of LibriSpeech or VoxCeleb2, with installation and seven-attack evaluation unchanged, reduces RA by 2.4 pp on average across eight source/task combinations (two sources × four tasks: KWS, IC, SID-250, ER). The largest decrease is 6.5 pp (ER/VoxCeleb2: 37.1→30.6% RA), yet RA remains 25.3 pp above the original ER descendant (5.3% RA).
Additionally, the manuscript already characterizes the four-descendant analysis as descriptive, not predictive (§4.1, lines 256–263; Appendix G, lines 1031–1034). The additional experiments further characterize the boundaries of robustness transfer without claiming invariance or a general predictive rule.
W2
We agree that SID-250 presents our most challenging clean–robustness trade-off. Its 7.3 pp CA loss is explicitly highlighted as the worst observed case in Fig. 4 (lines 441–444). Nevertheless, label-free installation improves RA from 3.2% to 40.9% (+37.7 pp), while retaining 84.1% CA. For comparison, four-epoch task-specific AT achieves 79.3/56.8% CA/RA (Table 2). Importantly, §5 explicitly identifies AT as a supervised reference, not an access-matched post-hoc baseline (lines 315–320). We address the reviewer's request for a limited-label adversarial adaptation comparison below.
The margin observation also deserves clarification. The manuscript defines an attack-relative sufficient, not necessary, condition (§4.2, lines 285–299). Therefore, SA=33.1% being below measured RA=40.9% is expected: the bound is conservative, not a failure of the analysis. Moreover, removing margin regularization lowers RA to 33.7% (−7.2 pp; Table 3), indicating its importance.
Finally, the reviewer's distinction between labeled readout fitting and few-shot adversarial adaptation is useful. Table 3 explicitly presents the 5%/10% controls as readout-only supervision references (lines 446–460), not as few-shot AT. We therefore added the requested comparison, starting from the unpatched SID-250 descendant and updating encoder and readout for four epochs with 10-step PGD at 37 dB. Both the 5% and 10% labeled subsets include all 250 speakers.
Our method exceeds these fixed-four-epoch few-shot AT baselines by 20.5/12.2 pp RA. The 0.14 GPU-h is the incremental installation cost; the shared Stage 1 cost is accounted for separately (Table 12). These fixed-four-epoch results do not rule out improvements from longer or differently tuned few-shot AT.