acceptodds
Under review as a conference paper at ICLR 2027

Robustness After Specialization: Installing Adversarial Robustness in Speech Foundation Model Descendants

Abstract

Speech foundation models are often specialized into task-specific descendants before adversarial robustness becomes a deployment requirement. We study *post-hoc robustness installation*: can robustness learned once at a foundation backbone be installed into an already fine-tuned descendant without downstream labels, the original fine-tuning pipeline, per-descendant adversarial training, or additional inference-time computation? From unlabeled source audio, we learn a task-agnostic backbone update . It reduces adversarial representation drift while explicitly minimizing clean-representation changes. For each descendant, we add the same update at a fixed, untuned scale. We then freeze the patched encoder and realign its linear readout on clean unlabeled audio. We evaluate three backbones on five speech tasks, producing 15 descendants through full encoder fine-tuning. With one shared update per backbone, seven-attack robust accuracy improves by 48.7 percentage points on average across the 12 classification descendants. The worst-task clean-accuracy loss is 7.3 percentage points. The method recovers 79.5% of the task-specific classification AT gain. Evaluation attacks the exact installed model, including with adaptive white-box attacks. Our analysis measures transferred representation stability and relates the remaining adversarial drift to downstream decision margins via an attack-relative sufficient criterion.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
speech foundation models, adversarial robustness, post-hoc robustness, robustness transfer, adversarial training, self-supervised speech representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GuGnOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-hoc adversarial robustness installation for speech foundation model descendants. The setting is that a pretrained backbone has already been fine-tuned into task-specific descendants, and the defender only has the descendant weights plus clean unlabeled downstream audio — no labels, no original fine-tuning pipeline, no per-descendant adversarial training. The method has two stages: (1) learn a task-agnostic weight update Δ_rob once per backbone from unlabeled source audio, minimizing adversarial representation drift with an explicit clean-representation anchor (one-sided stop-gradient, fixed anchor); (2) install the same update additively at a fixed untuned scale α=1 into each descendant, freeze the patched encoder, and realign only the linear readout on clean unlabeled audio using pseudo-targets from the original descendant plus a geometric-margin hinge term. Evaluated with a seven-attack pool (including adaptive white-box and psychoacoustic attacks, all targeting the exact installed model) on three backbones and five tasks, the method improves pooled robust accuracy by 48.7 points on average across 12 classification descendants, recovers 79.5% of the task-specific AT gain, with a worst-task clean-accuracy loss of 7.3 points. A drift/margin analysis links reduced representation drift to robust predictions via an attack-relative sufficient margin condition. Amortized compute breaks even with task-specific AT at ~2.2 descendants per backbone.

Critical strengths
  1. A well-defined, practically relevant problem with a method that fits its constraints exactly. The paper identifies a real deployment scenario — robustness becoming a requirement after fine-tuning, with labels/pipeline unavailable — and the two-stage design (shared backbone update + label-free readout realignment) satisfies every stated constraint without hand-waving: no downstream labels, no downstream adversarial examples, no inference-time overhead. This is consequential because most prior work either assumes robustness is in place before specialization or requires per-target AT; showing that one amortized update can be installed into 12 heterogeneous descendants is the paper's central empirical claim, and the access-matched baselines (interpolation, TIES, FARE-style, patch-only, realign-only) isolate what is actually new.
  2. Evaluation rigor that is rare in this area. Every attack targets the exact installed model (including the realigned readout); the pool includes adaptive mechanism-aware attacks and query-limited SPSA; Fig. 4a shows monotonic degradation under increasing budget, directly addressing gradient-obfuscation concerns; the method is tested across three backbones, five tasks, full fine-tuning and LoRA, with seed variability and bootstrap errors reported. The complementary drift/margin analysis additionally explains why the method works rather than only that it works. Together these make the 48.7-point / 79.5%-recovery headline numbers credible.
Critical weaknesses
  1. Insufficient evidence for fixed α=1\alpha=1 generality: Although evaluated across three backbones, five tasks, and a LoRA setting, α=1\alpha=1 is stress-tested only on WavLM-large/SID-250; the descriptive analysis of task-vector magnitude, source-to-downstream shift, and robustness transfer uses only four classification descendants, insufficient for a predictive rule. Systematic variation of specialization strength, task-vector norm, and source-target shift, plus failure cases, is needed.

  2. Uneven clean-robustness trade-offs: SID-250 is the worst: a 7.3-point clean-accuracy drop, robust accuracy of 40.9% versus 56.8% for task-specific adversarial training, a looser margin bound, and another 7.2-point robust drop without margin regularization. The 5%/10% labeled controls only replace pseudo-labels and do not equal limited-label per-descendant adversarial adaptation; comparison with few-shot supervised/adversarial fine-tuning is needed.

  3. Missing direct per-descendant unsupervised robustification baseline: For each fine-tuned descendant, directly optimize the Stage-1 drift-min objective using same-source or downstream unlabeled audio, then perform readout realignment; report performance- and compute-matched variants to quantify the amortization cost of “learn once per backbone.”

  4. No evaluation under distribution-shifted readout calibration: Stage 2 assumes clean, in-distribution calibration audio and varies only the amount of same-domain sampling. In deployment, shifts in speakers, channels, noise, language, or class coverage can make teacher predictions unreliable and harm readout fitting. Controlled domain-shift and mixed-domain calibration experiments are needed.

  5. Physical over-the-air robustness remains untested: Only digital waveform perturbations are evaluated; it is not shown to survive loudspeaker-microphone transfer, RIRs, background noise, or physical adversarial examples. A playback-and-re-recording study or room/device response simulation is needed.

  6. Incomplete drift-margin analysis for CTC ASR: The margin is defined only at the frame-level CTC projection, with no sequence-level robustness condition and no ASR analogue of the classification Figure 3 diagnostic. Frame-level drift/margin statistics, their relationship to token/sequence errors, and failure cases involving CTC alignment changes are needed.

Questions for authors

See weaknesses

5 comments

We thank the reviewer for recognizing the practical value of our post-specialization setting, its adherence to deployment constraints, and the breadth and rigor of our evaluation across backbones, tasks, adaptive attacks, and drift–margin analyses. We appreciate the thoughtful questions, which give us an opportunity to clarify the paper's stated scope and provide additional evidence on the limits of robustness transfer.

Note: All manuscript references below refer to the version on which this review was based. Unless stated otherwise, classification results are WavLM-large clean/pooled seven-attack robust accuracy (CA/RA, %).

W1

As stated in §3.3 (lines 205–208), α=1 is fixed and untuned; we do not claim it is optimal for every descendant. Fig. 4b originally focused on SID-250, which had the least favorable observed clean–robustness trade-off (lines 419–424, 441–444). To examine generality more directly, we extended the α sweep to all four WavLM-large classification tasks, repeating the same 45-minute label-free readout realignment at each setting. Tuned α maximizes validation RA; the table reports test results.

TaskFixed α=1 (C/R)Tuned (C/R)ΔCA/ΔRA (pp)
KWS95.2/82.193.4/86.2−1.8/+4.1
IC98.2/87.693.5/91.5−4.7/+3.9
SID-25084.1/40.980.1/41.3−4.0/+0.4
ER63.4/37.160.2/41.6−3.2/+4.5

Tuning increases mean RA by 3.2 pp but reduces mean CA by 3.4 pp. On SID-250, α=1.25 adds only 0.4 pp RA while losing 4.0 pp CA. The same pattern holds on a second backbone: wav2vec2-large/SID-250 achieves 81.6/42.1 at α=1 versus 77.6/43.9 at tuned α=1.25. Thus, fixed α=1 offers a favorable untuned trade-off. These sweeps assess tuning sensitivity, deployment retains fixed α=1 without adversarial validation or per-descendant tuning.

Beyond α sensitivity, we tested stronger specialization and changes in Stage 1 source audio. As reported to reviewer SpNU (W1 & Q1), after 12 fine-tuning epochs, the same fixed patch retains RA gains of +60.6 pp on KWS and +32.5 pp on SID-250 (global relative task-vector norms: 2.13% and 2.81%, respectively), although SID-250 incurs a larger clean-accuracy cost. Learning Stage 1 instead from 100 h of LibriSpeech or VoxCeleb2, with installation and seven-attack evaluation unchanged, reduces RA by 2.4 pp on average across eight source/task combinations (two sources × four tasks: KWS, IC, SID-250, ER). The largest decrease is 6.5 pp (ER/VoxCeleb2: 37.1→30.6% RA), yet RA remains 25.3 pp above the original ER descendant (5.3% RA).

Additionally, the manuscript already characterizes the four-descendant analysis as descriptive, not predictive (§4.1, lines 256–263; Appendix G, lines 1031–1034). The additional experiments further characterize the boundaries of robustness transfer without claiming invariance or a general predictive rule.

W2

We agree that SID-250 presents our most challenging clean–robustness trade-off. Its 7.3 pp CA loss is explicitly highlighted as the worst observed case in Fig. 4 (lines 441–444). Nevertheless, label-free installation improves RA from 3.2% to 40.9% (+37.7 pp), while retaining 84.1% CA. For comparison, four-epoch task-specific AT achieves 79.3/56.8% CA/RA (Table 2). Importantly, §5 explicitly identifies AT as a supervised reference, not an access-matched post-hoc baseline (lines 315–320). We address the reviewer's request for a limited-label adversarial adaptation comparison below.

The margin observation also deserves clarification. The manuscript defines an attack-relative sufficient, not necessary, condition (§4.2, lines 285–299). Therefore, SA=33.1% being below measured RA=40.9% is expected: the bound is conservative, not a failure of the analysis. Moreover, removing margin regularization lowers RA to 33.7% (−7.2 pp; Table 3), indicating its importance.

Finally, the reviewer's distinction between labeled readout fitting and few-shot adversarial adaptation is useful. Table 3 explicitly presents the 5%/10% controls as readout-only supervision references (lines 446–460), not as few-shot AT. We therefore added the requested comparison, starting from the unpatched SID-250 descendant and updating encoder and readout for four epochs with 10-step PGD at 37 dB. Both the 5% and 10% labeled subsets include all 250 speakers.

MethodLabelsCA/RA (%)Adaptation GPU-h
Ours (installation)0%84.1/40.90.14
Few-shot AT5%83.5/20.40.92
Few-shot AT10%82.3/28.71.71

Our method exceeds these fixed-four-epoch few-shot AT baselines by 20.5/12.2 pp RA. The 0.14 GPU-h is the incremental installation cost; the shared Stage 1 cost is accounted for separately (Table 12). These fixed-four-epoch results do not rule out improvements from longer or differently tuned few-shot AT.

W3

We appreciate this suggestion: it directly tests the benefit of sharing one robustness update. For each WavLM-large classification descendant, we optimized the same Stage 1 objective on 100 h of unlabeled Common Voice, starting from and clean-anchoring to its own weights. Both direct variants use our label-free Stage 2 realignment and evaluation.

MethodKWS (C/R)IC (C/R)SID-250 (C/R)ER (C/R)
Ours (shared update)95.2/82.198.2/87.684.1/40.963.4/37.1
Direct (4 epochs)96.1/85.998.0/91.786.2/45.864.8/41.9
Direct (compute-matched, N=15)95.9/41.098.5/41.788.6/16.967.1/15.8

Four-epoch direct training achieves 4.4 pp higher mean RA, but costs 78.4 GPU-h across these four tasks, versus 5.8 GPU-h allocated to ours under N=15 amortization (13.5×). The 5.8 GPU-h is an amortized allocation, not the standalone cost of learning the shared patch for just four descendants. At matched amortized compute, direct training has much lower RA on every task. Thus, full-budget direct training can attain higher RA, whereas our reusable update provides substantial robustness at far lower amortized cost (§6, lines 381–387; Appendix F, lines 969–985).

W4

Stage 2 explicitly assumes clean, in-distribution unlabeled audio (§3.1, lines 154–157; §3.3, lines 225–230). Appendix C.1 specifies 45 minutes of held-out in-task calibration (lines 755–761); Appendix H varies calibration quantity, not domain (lines 1054–1063). We did not claim shift-invariant realignment.

To test sensitivity, we held calibration duration (45 minutes), α=1, and Stage 2 fixed, applying 20 dB Gaussian noise, RIR convolution, or 50/50 clean/RIR only to calibration audio. The original descendant generated pseudo-labels on these inputs. We evaluated on the unchanged clean test split and regenerated 37 dB attacks against each realigned model. Entries are ΔCA/ΔRA (pp) relative to standard calibration.

TaskClean calib. (C/R)Gaussian 20 dBRIR50/50 clean/RIR
KWS95.2/82.1−1.0/−1.7−2.1/−4.0−0.9/−1.6
IC98.2/87.6−0.4/−0.8−0.8/−1.8−0.2/−0.5
SID-25084.1/40.9−3.5/−4.7−5.1/−7.1−2.4/−3.2
ER63.4/37.1−2.3/−2.8−3.6/−5.1−1.7/−3.0
Mean—−1.8/−2.5−2.9/−4.5−1.3/−2.1

The largest RIR-induced RA decrease is on SID-250 (40.9→33.8%), while mixed calibration retains 37.7%. We also varied speaker coverage at fixed duration: restricting its 45-minute pool from 250/250 to 125/250 speakers yields 77.0/35.0% CA/RA versus 84.1/40.9%, with all speakers retained at test time and speaker IDs used only to construct subsets. These results quantify sensitivity to calibration shifts and show that mixing clean and shifted audio mitigates acoustic mismatch.

W5

The manuscript explicitly evaluates digital waveform perturbations (§3.1, lines 184–188) and identifies physical playback/re-recording as a separate, unevaluated setting in §9 (lines 469–475).

To probe beyond that scope, we tested WavLM-large/SID-250 with simulated RIR, with or without 20 dB noise, and no channel-specific training or realignment. Adaptive attacks optimize through the channel at 37 dB source-side SNR.

ConditionOriginal (C/R)Ours (C/R)ΔRA (pp)
Digital91.4/3.284.1/40.9+37.7
RIR only85.5/5.778.0/42.6+36.9
RIR + noise (20 dB)81.3/5.474.0/39.5+34.1

Against channel-aware white-box attacks optimized through simulated RIR and RIR+noise, our method retains RA gains of 36.9 and 34.1 pp, respectively. These tests probe reverberation and noise relevant to physical deployment and over-the-air attacks.

W6

We appreciate the request for CTC alignment analysis. The manuscript explicitly limits the CTC margin argument to frame-level projections, not sequence-level guarantees (§4.2, lines 301–302; Appendix C.2, lines 839–841); sequence robustness is evaluated using adversarial WER. We added WavLM-large ASR diagnostics analogous to Fig. 3, using each model's highest-WER valid seven-attack candidate for both diagnostics and error measurement.

ModelFrame drift ↓Nonblank margin ↑Adv. CER (%) ↓Adv. WER (%) ↓
Original1.001.0042.191.5
Patch only0.350.9613.537.2
Ours0.371.149.830.7

Frame drift measures mean RMS representation change; margin averages clean geometric margins on frames assigned nonblank tokens by the original descendant. Both indices are separately normalized to the original (=1), so they cannot be compared using the margin > drift inequality. Patching sharply reduces drift; realignment raises margins and lowers CER/WER. A representative defended-model failure changes “nightfall the yellow lamps” to “nightfal the yelo las” under attack, consistent with CTC sensitivity to blank separation, repeated symbols, and missing token spikes. It illustrates how localized alignment changes can affect several words despite low average drift.

Note: We also plan to add the new results and discussion to the paper before the rebuttal deadline.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer GuGn,

Thank you again for your thoughtful and detailed review.

We have posted a two-part, point-by-point response addressing all six concerns raised in your review, including additional experiments on installation-scale sensitivity and robustness transfer under stronger downstream specialization and source-data shifts, comparisons with few-shot adversarial training and direct per-descendant unsupervised robustification, further experiments on readout calibration under distribution shifts, robustness under simulated acoustic channels, and CTC drift–margin analysis.

As the discussion period approaches its conclusion, we would greatly appreciate knowing whether these additional results and clarifications have addressed your main concerns or whether any questions remain. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_r42aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a post-hoc adversarial robustness enhancement method for already fine-tuned speech foundation models. It first learns a task-agnostic robustness update from unlabeled audio and then installs the update into specialized models, followed by pseudo-label-based readout realignment. Experiments across three foundation models and five speech tasks demonstrate substantial robustness improvements, achieving an average 48.7-percentage-point increase in robust accuracy across 12 classification models while largely preserving clean performance.

Critical strengths

The paper addresses an important deployment scenario where adversarial robustness must be introduced after model specialization without downstream labels or access to the original training pipeline. The ability to reuse a single robustness update across multiple descendants offers practical advantages in computational efficiency.

The paper demonstrates substantial robustness improvements across different speech models and tasks. The inclusion of multiple adaptive white-box attacks, relevant baselines, and component ablations provides reasonably convincing empirical support for the proposed approach.

Critical weaknesses

The proposed framework combines several established techniques, including adversarial representation regularization, clean-feature preservation, additive weight merging, pseudo-label distillation, and margin optimisation. Although the ablations show that individual components contribute to performance, it remains unclear whether this relatively elaborate two-stage design is necessary or offers a fundamental methodological advantage over simpler alternatives. The contribution appears primarily to be an effective integration of existing techniques rather than a substantially new learning methodology.

While the paper evaluates FARE-style training, checkpoint interpolation, and TIES-Merging, more recent approaches such as LORE, which explicitly optimises the robustness–clean-accuracy trade-off, are discussed but not experimentally compared. This leaves uncertainty about whether the proposed method offers a genuine advantage over stronger alternative robustness objectives. Moreover, a systematic robustness–accuracy Pareto comparison is missing, making it difficult to assess whether the added complexity provides a superior trade-off.

Questions for authors

Could the authors clarify why the separate robustness-update and readout-realignment stages are necessary? Would a simpler alternative, such as a single constrained robustness objective with clean-representation preservation, achieve comparable results? Can the authors provide an experimental comparison with LORE or other recent robustness-learning objectives under the same post-specialization installation protocol? Can the authors provide Pareto comparisons against the strongest baselines at matched clean-accuracy levels, particularly for speaker identification and emotion recognition, where clean-performance degradation is noticeable?

4 comments

We thank the reviewer for recognizing the practical value and computational efficiency of our approach and our evaluation across speech models and tasks, including adaptive attacks and ablations.

Note: All manuscript references below refer to the version on which this review was based.

W1 — Clarifying our contribution

Our novelty lies in the new post-specialization robustness setting, the successful transfer of a single shared task-agnostic robustness weight update across specialized models, and the substantial computational savings from reusing that update. To our knowledge, we are the first to learn one task-agnostic adversarial robustness update per backbone and install the same update across independently fine-tuned descendants using only their weights and clean unlabeled audio downstream. Robust-first transfer introduces robustness before downstream adaptation [1]. Our post-specialization setting requires transferring it into encoders that have already changed through fine-tuning, while retaining their task-specific weight updates (lines 72–95). We adapt existing robustness objectives and weight-merging methods as baselines within this setting (§6, lines 373–379); their original formulations did not introduce or evaluate this specific post-specialization installation protocol.

Furthermore, FARE matches adversarial features to the original backbone’s clean features. Our Stage 1 objective separates adversarial stability around the updated backbone’s clean features (with one-sided stop-gradient) from preservation of the original backbone’s clean features (Eq. 2; Appendix A, lines 659–671). With all other settings fixed, our objective recovers 79.5% of the task-specific adversarial training (AT) gain across all 12 classification descendants, compared with 61.4% for FARE-style training (Table 2).

Across 12 classification descendants, the same update per backbone at fixed α=1\alpha = 1 raises pooled robust accuracy by 48.7 points on average over the original models (Table 2). Reusing the update across 15 descendants per backbone requires only 16% of task-specific AT compute (Table 12).

W1 & Q1 — Why are separate robustness-update and readout-realignment stages necessary?

The two stages address complementary requirements of robust predictions. As established in §4.2 (lines 267–300), a sufficient condition for preserving a clean-correct prediction against the evaluated attack pool is that the clean geometric margin exceeds adversarial representation drift: m(x)>∥rθ⋆(x)∥2m(x) > \lVert r_\theta^\star(x) \rVert_2. Stage 1 reduces adversarial drift through a shared robustness update while preserving clean representations (Eq. 2). However, installing this update into an already-specialized encoder can shift its clean features relative to the original readout, reducing decision margins despite improved representation stability (Figure 3). Stage 2 therefore realigns the readout to the patched features and encourages larger geometric margins (Eq. 4), with the encoder frozen.

The ablations across the four WavLM-large classification tasks demonstrate why neither stage alone achieves the same performance as the full method (Table 3):

  • Stage 1 alone (patch only). The patch increases mean robust accuracy by 42.4 points over the original models, but reduces mean clean accuracy by 13.0 points.
  • Stage 2 alone (realignment only). Refitting the readout without the patch improves mean robust accuracy by only 2.0 points.
  • Both stages together. With the same patched encoder, realignment recovers 9.0 of the 13.0 lost clean-accuracy points and adds another 8.4 points of robust accuracy over patching alone.

The component ablations further support this design (Table 3). Removing the Stage 1 clean anchor reduces mean clean accuracy by 20.6 points, demonstrating its importance for preserving clean features. Removing the Stage 2 geometric-margin term reduces mean robust accuracy by 5.4 points, including 7.2 points on speaker identification (SID-250) and 5.9 points on emotion recognition (ER), while clean accuracy on those tasks changes by at most 0.2 points. Thus, Stage 1 supplies adversarial representation stability, while Stage 2 restores descendant-specific decision alignment and improves robustness to residual drift. Both stages are therefore needed to achieve substantial robustness gains while recovering most of the clean accuracy lost during patching.

W1 & Q2 — Could a single constrained robustness objective achieve comparable results?

Not under our evaluated post-specialization installation protocol. Our LORE results demonstrate that a constrained robustness objective alone does not achieve comparable performance without readout realignment. Preserving clean features relative to the original backbone does not guarantee their alignment with an already-specialized descendant's readout. Adding Stage 2 to the same constrained robustness patch improves clean/robust accuracy by 9.4/9.7 points on SID-250 and 13.3/11.8 points on ER (see the section below). Alternatively, optimizing a constrained objective directly on each descendant would require a separate robustness-training run per model, sacrificing the computational advantage of learning one shared update per backbone.

W2 & Q3 — How does LORE compare under the same post-specialization installation protocol?

We conducted a new comparison with LORE [3] under the same post-specialization installation protocol. On WavLM-large, we adapted LORE's constrained robustness objective to learn a shared backbone patch, swept its clean-embedding constraint, and evaluated the patch with and without readout realignment. We used the same source data, threat model, and fixed installation scale (α=1\alpha = 1) as our method (§3.3, lines 205–229).

Entries show clean/pooled robust accuracy (C/R, %).

MethodSID-250ER
FARE-style [2] + realignment76.0 / 28.655.2 / 26.9
LORE patch only75.2 / 27.148.9 / 20.9
LORE + realignment84.6 / 36.862.2 / 32.7
Ours (patch + realignment)84.1 / 40.963.4 / 37.1

Adding realignment to the same LORE patch improves clean/robust accuracy by 9.4/9.7 points on SID-250 and 13.3/11.8 points on ER, confirming that a constrained robustness objective does not eliminate the need for descendant-specific readout realignment. With the same realignment procedure, our patch achieves 4.1 and 4.4 points higher robust accuracy than LORE on SID-250 and ER, respectively. LORE patch training also requires 1.58× our Stage 1 GPU-hours.

Importantly, our central contribution is not the Stage 1 objective in isolation, but the reusable installation of robustness into already-specialized descendants. The LORE comparison further demonstrates that this installation benefits from descendant-specific realignment, while our Stage 1 objective achieves higher robustness at lower computational cost in the evaluated setting.

W2 & Q4 — How do the methods compare at matched clean accuracy?

At the reported Table 2 settings, our method already achieves higher clean and robust accuracy than checkpoint interpolation, TIES, and FARE-style training on both SID-250 and ER. Thus, our robustness advantage over these baselines does not come at the expense of clean accuracy. To further assess the clean–robustness trade-off, we evaluated the strongest baselines (TIES, FARE, and LORE) at comparable clean accuracy. For each baseline, we selected the configuration whose clean validation accuracy was closest to our method's, without using adversarial results for selection.

The table reports the resulting test-set clean/pooled robust accuracy (C/R, %), with our method's test clean accuracy shown as the reference target.

TaskTarget clean (%)Ours (C/R)TIES (C/R)FARE (C/R)LORE (C/R)
SID-25084.184.1 / 40.985.0 / 28.584.5 / 21.584.6 / 36.8
ER63.463.4 / 37.162.3 / 29.462.9 / 19.263.5 / 31.1

At comparable clean accuracies, our method achieves 4.1 points higher robust accuracy on SID-250 and 6.0 points higher on ER than LORE, the strongest comparator at both targets. The LORE configuration for ER differs from the preceding comparison because it was selected to match clean validation accuracy.

Note: We also plan to add the new results and discussion to the paper before the rebuttal deadline.

References

  • [1] Hua et al. Initialization Matters for Adversarial Transfer Learning. CVPR 2024.

  • [2] Schlarmann et al. Robust CLIP. ICML 2024.

  • [3] Khodabandeh et al. LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders. NeurIPS 2025.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer r42a,

Thank you again for your thoughtful and detailed review.

We have posted a two-part, point-by-point response addressing your concerns, including additional analyses of the complementary roles of our two stages, new comparisons with LORE under the same post-specialization installation protocol, and matched-clean-accuracy evaluations against TIES, FARE, and LORE.

As the discussion period approaches its conclusion, we would greatly appreciate knowing whether these additional results and clarifications have addressed your main concerns or whether any questions remain. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

Dear Reviewer r42a,

Thank you for your positive response and for taking the time to review our additional experiments and clarifications. We are pleased that our response has addressed most of your concerns regarding novelty and comparisons with the state of the art.

We appreciate your thoughtful feedback and reconsideration of your assessment.

Best regards,
The Authors

Reviewer_SpNUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-hoc installation of adversarial robustness into specialized speech foundation models. The authors first learn a task-agnostic robustness update once for each pretrained backbone using unlabeled source audio and adversarial perturbations, with an objective that reduces adversarial representation drift while preserving clean representations. They then add this update to independently fine-tuned downstream models without modifying the specialization update, freeze the patched encoder, and realign only the task readout using clean, unlabeled downstream audio and pseudo-labels from the original model. The method is evaluated across models and benchmarks, with large improvements in robust accuracy over the original fine-tuned models while retaining most of the clean-task performance, and the paper also analyzes representation drift, decision margins, specialization regimes, attack strength, and calibration-data efficiency.

Critical strengths

The paper proposes a practical post-hoc robustness setting where they learn one robustness update per speech backbone and install it into already specialized descendants without downstream labels, downstream adversarial examples, or the original fine-tuning pipeline. This is a useful setting that is different from standard robust-first training or task-specific adversarial training, and it is the main reason I lean towards acceptance because in practice the defender often only has the descendant weights and clean unlabeled audio. Under these constraints, the method makes a big improvement in robust accuracy on average across 12 classification descendants, with a worst-task clean accuracy loss of 7.3 points. The paper is well written and easy to follow. In addition, the author also provides comprehensive evaluation across the model backbones and downstream tasks under different attack methods, including adaptive attacks that target the method itself, with detailed ablation studies of the method.

Critical weaknesses

For me, the main weakness is that the relative task-vector norms are only around 1%, so the robustness update may transfer mainly because the descendants stay close to the backbone, and it is unclear whether this still holds for descendants that are fine-tuned longer or move further from the backbone.

The second weakness is that the individual components have close precedents, where the Stage 1 objective is a variant of FARE, the installation is task arithmetic, and the readout realignment is self-distillation with a margin term, so the contribution is mainly the setting and the combination. In addition, the task-specific adversarial training reference is only trained for four epochs from the backbone and has lower clean accuracy than the proposed method on several tasks, so the 79.5% recovery of the adversarial training gain may be overstated if this reference is under-tuned.

Questions for authors

In current setting, the task-vector norms are only around 1%, so the update may transfer mainly because the descendants stay close to the backbone. Does the robustness update still transfer when the descendant moves further from the backbone?

6 comments

We thank the reviewer for recognizing the practical value of our setting and the breadth of our evaluation, including adaptive attacks and detailed ablations.

W1 & Q1 — Does robustness still transfer as descendants move farther from the backbone?

Yes. Even after 12 epochs of downstream fine-tuning, our method improves WavLM-large robust accuracy by 60.6 points on KWS (2.13% relative norm) and 32.5 on SID-250 (2.81%).

Additionally, global relative task-vector norms of around 1% are consistent with those reported in standard fine-tuning of pretrained models. Zeng et al. [1] report relative ℓ2\ell_2 norms of 0.33–0.65% across eight vision tasks and two OpenCLIP checkpoints. Dong et al. [2] report a relative ℓ2\ell_2 norm of about 2.2% after standard RoBERTa fine-tuning on IMDb under a 20-epoch protocol. Also, these ratios are global and can obscure differences in weight changes across layers. Li and Zhang [3] measured relative ℓ2\ell_2 parameter changes during fine-tuning of pretrained ResNet-18 and found substantially larger changes in later layers.

As discussed in Section 4.1 (lines 256–263), robustness transfer may depend on downstream specialization. Table 13 reports norms of 0.92–1.70% across four different tasks (lines 1036–1042). We tested longer fine-tuning within each task. All six checkpoints received the original update at the same untuned scale (α = 1), followed by readout realignment on clean unlabeled audio.

Each pair shows the original descendant → our method at the same epoch.

TaskFine-tuning epochsRelative task-vector normClean acc. (%)Robust acc. (%)Robust gain (pp)
KWS41.02%97.8→95.218.3→82.1+63.8
KWS81.75%99.0→96.520.6→83.4+62.8
KWS122.13%97.2→92.716.9→77.5+60.6
SID-25041.48%91.4→84.13.2→40.9+37.7
SID-25082.34%91.2→83.03.3→37.1+33.8
SID-250122.81%93.5→81.62.4→34.9+32.5

On KWS, the distance from the backbone more than doubles, while the robustness gain declines only by 3.2 points. On SID-250, the task with the least favourable clean–robust accuracy trade-off, the robust gain remains 32.5 points at 12 epochs, while the clean-accuracy cost increases from 7.3 to 11.9 points. Even after tripling the fine-tuning duration, the same update delivers substantial robustness gains on both tasks.

W2.1 — Where does the novelty lie given the precedents for each component?

Our contributions are the new post-specialization robustness setting, the transfer of a task-agnostic robustness weight update across specialized models, and the combination that makes transfer effective and substantially reduces the compute cost per model. To our knowledge, we are the first to learn one task-agnostic adversarial robustness update per backbone and install the same update across independently fine-tuned descendants using only their weights and clean unlabeled audio downstream (lines 72–95).

Additionally, FARE matches adversarial features to the original encoder’s clean features. Our objective separates adversarial stability around updated clean features (with one-sided stop-gradient) from preservation of original clean features. With all other settings fixed, it recovers 81.1% of the WavLM-large AT gain versus 62.3% for FARE-style training (Table 2). Readout realignment repairs the feature–readout mismatch, recovering 9.0 of the 13.0 clean-accuracy points lost by patching alone and adding 8.4 robust-accuracy points (Table 3). Reusing the update across 15 descendants per backbone requires only 16% of task-specific AT compute (Table 12).

W2.2 — Does four-epoch AT overstate the fraction of the robustness gain recovered?

The four-epoch AT reference matched the original specialization duration (lines 783–787 and 807–812). We extended task-specific AT to 8 and 12 epochs on all four WavLM-large classification tasks. Entries give clean / robust accuracy (%), with each task’s best robust accuracy in bold.

Task4 epochs8 epochs12 epochs
KWS94.5/90.194.9/91.393.7/86.4
IC96.6/93.896.7/93.596.5/94.1
SID-25079.3/56.879.4/60.278.4/61.8
ER60.2/48.659.1/50.658.0/44.5

Extending AT to 8 and 12 epochs provides limited robustness gains overall. Against the AT checkpoint with the highest robust accuracy for each task, WavLM-large still recovers 78.4% of the AT gain, versus 81.1% under the four-epoch AT reference. AT’s lower clean accuracy in Table 2 reflects the clean–robust trade-off and does not indicate insufficient training.

Note: We also plan to add the new results and discussion to the paper before the rebuttal deadline.

References

[1] Zeng et al. Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic. TMLR 2026.

[2] Dong et al. How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness? NeurIPS 2021.

[3] Li and Zhang. Improved Regularization and Robustness for Fine-tuning in Neural Networks. NeurIPS 2021.

Dear Reviewer SpNU,

Thank you again for your thoughtful and constructive review.

We have posted a detailed point-by-point response addressing your concerns, including additional experiments on robustness transfer under stronger downstream specialization, longer task-specific adversarial training, and clarifications regarding the novelty and contributions of our approach.

As the discussion period approaches its conclusion, we would greatly appreciate knowing whether our response has addressed your main concerns or whether any questions remain. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_r42aOpenReview ↗

thanks

I thank the author for the careful revision, most of my concerns around novelty and SOTA are addressed; thus I have increased my score to 3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.