acceptodds
Under review as a conference paper at ICLR 2027

Learning to Watermark Speech Synthesis Against Model-Driven Reconstruction

Abstract

Modern TTS systems increasingly generate synthetic speech at scale for diverse users. This setting calls for content-level provenance that can verify the origin of released speech and attribute it to the requesting user, which generative watermarking can support by embedding multi-bit identifiers directly into synthesized speech. Once released, however, speech may undergo heterogeneous learned transformations during distribution and editing, with reconstruction objectives that can preserve speech utility while affecting watermark recoverability differently. We find that no single watermark carrier remains consistently reliable across reconstruction models, as its survival depends jointly on the embedded structure, reconstruction mechanism, and observation representation. To this end, we propose Thrive, a multi-bit generative speech watermarking framework for modern autoregressive TTS, covering both discrete-token and continuous-representation generation under reconstruction attacks. Specifically, Rise synchronizes watermark injection into intermediate representations with its continued integration into subsequent generation, while Care combines waveform and spectral experts using bit-wise reliability selection. Experiments on both autoregressive paradigms show that Thrive preserves synthesis fidelity, achieves 87.6% average recovery accuracy under reconstruction attacks, and supports source attribution over candidate sets of up to 10,000 identities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Generative Speech Watermarking, Representation–Generation Synchronization, Cross-Representation Recovery, Reconstruction Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6cPqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Thrive, which is a multi-bit generative watermark for autoregressive TTS model designed to survive model-driven reconstruction. This method has two main parts, RISE injects a message into intermediate representations and modulates later generation with the same message, whereas CARE decodes with a waveform expert and a spectral expert, then uses a calibrated per bit reliability selector to choose between their logits. The experiments show that Thrive achieves high bit accuracy under reconstruction attacks and covers source attribution over up to 10000 identities.

Critical strengths
  1. The main finding is that robustness profiles cross between backbones and methods. Dots.tts is good at codecs, while qwen3 is stronger on vocoders and enhancers. This finding is informative and gives the community a reusable reconstruction benchmark.
  2. I like how the authors present their figures. The drawing is nice and clear to me.
Critical weaknesses
  1. The novelty is overstated. RISE modifies only the codec decoder in qwen3 or the AudioVAE decoder for Dots.tts and leaves the autoregressive model and its tokens or latents untouched. As a result, Thrive is a message conditioned decoder watermark, which is close to HFGw and RIWF instead of a new watermarking method for autoregressive TTS.
Questions for authors
  1. How is TPR@1%FPR computed for the 16 bit payload? Is the threshold the number of matching bits against the claimed message or the best match over all registered users?
  2. How is the generation quality for the unwatermarked outputs from qwen3 and dots.tts? So that we know how the watermarks affect the quality of the generated output.
  3. Why Dots.tts STOI has a std 0.197 and 0.314 on the OOD sets? Is it catastrophic failures on some utterances?
  4. What is the deployment cost? e.g. inference latency and runtime compared to other methods.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6cPq (Part 1/3)

Dear Reviewer 6cPq, We sincerely appreciate the reviewer's careful evaluation and constructive feedback. We address each concern in detail below and have incorporated corresponding revisions into the manuscript, with all changes highlighted for clarity.

W1: Novelty

We would like to clarify the distinct contributions of Thrive relative to prior work. In Thrive, Representation-Generation Synchronized Embedding (RISE) coordinates the Representation Injection Path and the Generation Integration Path through a shared watermark message while keeping whole autoregressive model frozen. It injects watermark residuals into intermediate representations and conditions subsequent synthesis on the same watermark. This design enables different watermarks to be embedded at inference time without retraining.

In contrast, HFGw first pretrains TBWM [1] and then uses its decoder to guide full-parameter fine-tuning of the vocoder, enabling it to generate speech carrying a fixed watermark. RIWF first learns parameter masks to identify embedding locations within convolutional kernels, then replaces the selected kernel elements with watermark encodings and normalizes the modified weights. The key distinction is that RISE jointly learns watermark formation in intermediate representations and its continued integration into synthesis, rather than realizing the watermark solely through vocoder-weight modification.

In a nutshell, by operating on synthesis representations and transformations, RISE provides a shared embedding mechanism for both discrete-token and continuous-representation autoregressive TTS. Building on this mechanism, Thrive combines our Cross-Representation Adaptive Recovery with a training strategy that separates capability learning from reliability calibration, achieving strong robustness against reconstruction attacks and scalable multi-user attribution.

Q1: TPR@1%FPR Calculation and Threshold

We compute TPR empirically from per-speech verification outcomes. Following VINE [2], we decode each audio and count the matching bits between the recovered message w^\hat{\mathbf{w}} and its pre-specified claimed 16-bit watermark w\mathbf{w}. The threshold applies to this claimed message, not the best match over all registered users. Under the theoretical assumption that bit matches are independent with probability 0.5 for unwatermarked audio, the matching count follows Binomial⁡(16,0.5)\operatorname{Binomial}(16,0.5). The smallest integer threshold with theoretical FPR at most 1% is 14 matches, corresponding to an FPR of 0.209% because the matching count is discrete.

For the empirical evaluation, we calibrate the operating threshold using real unwatermarked outputs, consistent with the recommendation in SoK [3] to complement theoretical analysis with empirical measurements. These negative samples are generated with watermark injection disabled and evaluated using the same extraction and keyed decoding procedure against messages assigned before extraction. For each detector, we select the smallest integer threshold whose false-positive fraction on the calibration set is at most 1%. Calibration and test sets have no source-content overlap.

We then freeze the calibrated threshold for clean and all attacks. For each condition, TPR is the number of watermarked speech samples whose matching count meets or exceeds this threshold, divided by the total number of watermarked test samples. Each sample therefore contributes one observed detection decision. We provide the complete calculation procedure in App. D.2.

References:

[1] Liu C, et al. Detecting voice cloning attacks via timbre watermarking. NDSS 2024.

[2] Lu S, et al. Robust watermarking using generative priors against image editing: From benchmarking to advances. ICLR 2025.

[3] Zhao X, et al. Sok: Watermarking for ai-generated content. IEEE S&P 2025.

AuthorsOpenReview ↗

Response to Reviewer 6cPq (Part 2/3)

Q2: Unwatermarked Generation Quality

To better distinguish the intrinsic quality of the underlying generators from the impact of watermark embedding, we have extended Table I to include unwatermarked outputs from Qwen3-TTS and Dots.tts across all three datasets. We report DNSMOS and NISQA because both assess perceptual quality without a reference. These comparisons use the same evaluation samples and synthesis conditions, with watermark embedding disabled or enabled. The unwatermarked outputs also serve as references for the paired fidelity metrics, for which self-comparison would provide no additional quality information.

Table I. Fidelity.

Method (bps)STOI ↑PESQ ↑SSIM ↑MCD ↓DNSMOS ↑NISQA ↑VGGish ↓CLAP ↓ACC ↑TPR ↑
LJSpeech (In-Distribution Dataset)
Qwen3-TTS----2.711 ± 0.4782.991 ± 1.173----
Ours [Qwen3-TTS 0.6B] (16)0.958 ± 0.0733.565 ± 0.7680.850 ± 0.0871.754 ± 0.1362.638 ± 0.4292.807 ± 1.1651.3550.0860.9991.000
Dots.tts----3.099 ± 0.3034.185 ± 0.540----
Ours [Dots.tts 2B] (16)0.999 ± 0.0034.210 ± 0.2590.980 ± 0.0040.777 ± 0.1033.066 ± 0.3174.094 ± 0.5480.6040.0370.9991.000
LibriTTS (OOD Dataset)
Qwen3-TTS----2.794 ± 0.4143.224 ± 1.015----
Ours [Qwen3-TTS] (16)0.969 ± 0.0503.747 ± 0.5660.858 ± 0.0751.781 ± 0.1572.271 ± 0.4313.159 ± 1.0071.5390.0890.9991.000
Dots.tts----3.163 ± 0.2243.936 ± 0.812----
Ours [Dots.tts] (16)0.995 ± 0.0244.045 ± 0.4790.980 ± 0.0040.739 ± 0.1353.105 ± 0.2713.827 ± 0.8120.4120.0520.9990.999
AISHELL-3 (OOD Dataset)
Qwen3-TTS----3.102 ± 0.2503.437 ± 0.808----
Ours [Qwen3-TTS] (16)0.980 ± 0.0233.820 ± 0.4390.874 ± 0.0481.763 ± 0.1462.667 ± 0.3533.291 ± 0.7981.2430.0830.9991.000
Dots.tts----3.176 ± 0.2193.481 ± 1.058----
Ours [Dots.tts] (16)0.997 ± 0.0104.069 ± 0.4710.979 ± 0.0040.764 ± 0.1383.084 ± 0.2943.399 ± 1.0450.5510.0670.9991.000

Across all three datasets, Thrive achieves near-perfect clean recovery on both backbones, with PESQ scores of 3.565-4.210 and NISQA remaining close to the unwatermarked baselines. These results demonstrate reliable multi-bit watermarking with high fidelity across discrete-token and continuous-representation autoregressive TTS. We have updated Table 1 and the corresponding discussion in Section 5.2 of the manuscript, with the revisions highlighted in blue.

Q3: High STOI Variance on OOD Datasets

We would like to clarify that the reported variability primarily reflects STOI's handling of short evaluation clips.

Within fixed one-second windows, speech duration and onset vary, leaving some unwatermarked references with insufficient frames after silence removal. In these cases, pystoi returns a near-zero invalid marker, which inflates the standard deviation when counted as a valid score. The same references produce this return even when compared with themselves, confirming that this statistical anomaly reflects insufficient temporal support for STOI, rather than watermark-induced intelligibility collapse. STOI statistics exclude clips for which fewer than 30 STFT frames remain after silence removal, causing pystoi to return the invalid value 10−510^{-5}. We compute the reported mean and standard deviation over the remaining valid clips. We have corrected the STOI statistics in Table 1 (Sec. 5.2 of the manuscript) and highlighted the updated values.

PESQ and SSIM do not share this frame requirement and measure similarity between clean and watermarked outputs. Short audio pairs can remain highly similar even when STOI is non-computable, so their higher scores are consistent with this explanation.

AuthorsOpenReview ↗

Response to Reviewer 6cPq (Part 3/3)

Q4: Inference Latency and Deployment Cost

We have added an efficiency analysis in Sec. 5.6, with detailed comparisons in Table II. The results show that Thrive introduces small generation overhead and completes recovery from one-second speech in approximately 4.5 ms.

All methods are benchmarked on the same GPU using FP32, batch size one, and 16 bps. Generation overhead is relative to each native host: Thrive covers text-to-waveform synthesis, while vocoder baselines start from mel spectrograms. Recovery includes each method's full extraction pipeline on identical one-second speech.

Table II. Computational efficiency.

MethodParams. (M) ↓WM Size (MiB) ↓GFLOPs ↓Latency (ms) ↓GPU (GiB) ↓Throughput ↑RTF ↓
Generation
HFGw13.936 / 053.16252.813 / 0 (0.00%)4.401 / 0.003 (0.06%)0.122226.7590.004
RIWF0 / 0.1150.438346.116 / 0.001 (0.0003%)37.377 / 3.869 (10.35%)0.09524.4880.041
Groot0 / 227.063866.177346.116 / 0.454 (0.13%)38.916 / 0.977 (2.51%)1.36025.3200.039
Ours-Q0 / 5.88922.450456.763 / 0.826 (0.18%)1013.676 / 6.499 (0.64%)4.3560.9411.063
Ours-D0 / 12.19646.52411838.964 / 24.306 (0.21%)971.546 / 30.920 (3.18%)9.4350.9981.002
Recovery
HFGw0.3061.16926.7131.3130.060761.6250.001
RIWF4.35316.6049.20810.4470.04595.7250.010
Groot23.13188.2390.3920.8440.1011185.0180.0008
Ours-Q2.83910.8294.8424.6240.088216.2480.005
Ours-D2.83910.8294.8424.4460.088224.9430.004

The results highlight three advantages: 1. Parameter-efficient adaptation. RISE freezes the TTS host and trains only 5.889M/12.196M watermark parameters on Qwen/Dots, 97.4%/94.6% fewer than Groot, keeping the adaptation compact. 2. Low generation overhead. Coordinating representation injection with generation modulation adds little overhead, increasing GFLOPs by only 0.18%/0.21% and end-to-end latency by 0.64%/3.18%. 3. Efficient dual-expert recovery: Full CARE remains efficient with both experts and selection included, processing one-second speech in 4.446-4.624 ms with approximately 0.088 GiB peak GPU memory and 56%-57% lower latency than RIWF.

Reviewer_i2ApOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-bit speech watermarking framework, THRIVE, to resist model reconstruction attacks. RISE combines feature injection with watermark conditional LoRA, and CARE adaptively fuses waveform and spectral expert predictions. On two TTS backbones and three datasets, the watermark recovery rate is near perfect without attacks, while the average bit accuracy is 87.6% under reconstruction attacks.

Critical strengths

Controlled experiments reveal the impact of reconstruction models and observational representations on watermark survival rates.

Identifier replacement is possible without retraining, and TTS is compatible with both discrete and continuous representations; cross-dataset testing and testing with unseen reconstruction models enhance the evidence for practical applicability.

Ablation experiments support the effectiveness of both representation injection and reliability selection components.

Critical weaknesses

The construction of negative samples and threshold calibration for TPR@1%FPR are unclear; source tracing lacks rejection evaluation for out-of-candidate samples. With 10,000 candidate identities, the reconstructed source tracing accuracy is only 39.4%/26.6%.

Baseline loadings are inconsistent, and there is insufficient explanation regarding the fairness of backbone adaptation and reconstructed perception training; source tracing is only compared with RIWF, providing limited evidence.

Questions for authors

Is the TPR a measured result or inferred from bit accuracy? Please explain the negative sample rate, threshold calibration, and the false assignment rate during multi-identity retrieval.

Under consistent backbone, payload, data augmentation, and audio quality conditions, can the robustness advantage compared to a strong multi-bit baseline be verified?

4 comments
AuthorsOpenReview ↗

Response to Reviewer i2Ap (Part 1/2)

Dear Reviewer i2Ap, We greatly appreciate the reviewer's insightful comments and valuable suggestions. In response, we have provided point-by-point clarifications and strengthened the relevant discussions in the revised manuscript, with all modifications highlighted for easy reference.

W1&Q1: TPR Computation and Open-Set Attribution Evaluation

Thank you for these suggestions, which helped us clarify our evaluation and strengthen the paper. We address TPR computation and threshold calibration first, followed by additional open-set attribution experiments. We have also added the detailed calculations to the App. D.2 and the new experiment to App. D.8.

TPR computation and threshold calibration. We compute TPR empirically from per-utterance verification outcomes, rather than infer it from mean bit accuracy. Following VINE [1], verification counts matching bits against a pre-specified claimed message, rather than searching across registered identities. Under independent random matching, Binomial(16,0.5)\mathrm{Binomial}(16,0.5) gives a minimum threshold of 14 matches for the 1% FPR target, yielding a theoretical FPR of 0.209%.

We complement this analysis with empirical calibration [2]. Negatives are real outputs generated with watermark injection disabled and evaluated against pre-assigned messages. Using content-disjoint calibration and test sets, each detector selects the smallest integer threshold with calibration FPR at most 1%. We then freeze this threshold for clean and attacked speech and compute TPR as the fraction of watermarked utterances passing verification.

Open-set attribution and false-attribution evaluation. To evaluate false attribution across multiple identities, we calibrate the maximum mean bitwise log-likelihood over the full registry on unwatermarked clips at a 1% FPR target for each method and candidate-set size. With the threshold fixed, we attribute a sample to the unique highest-scoring identity only if its score meets the threshold. Otherwise, we return unknown. Correct attribution requires both acceptance and identification of the true registered user. This rate is computed over all in-pool test utterances, so unsuccessful attribution includes both incorrect identity assignments and rejections. We then evaluate unwatermarked speech and watermarked speech from held-out identities excluded from both the registry and calibration. Table I reports the observed test FPR and false-attribution rates under clean conditions.

Table I. Open-set attribution and false-attribution rates under clean conditions.

Candidates NNMethodCorrect Attribution ↑Unwatermarked FPR ↓Out-of-pool FAR ↓In-pool FAR ↓
10Ours-Q1.00000.01300.00000.0000
Ours-D1.00000.01040.00000.0000
RIWF1.00000.00780.16640.0000
100Ours-Q1.00000.01300.00080.0000
Ours-D1.00000.00520.00200.0000
RIWF0.99400.01040.02160.0040
1,000Ours-Q0.99960.00000.00040.0000
Ours-D1.00000.01040.00160.0000
RIWF0.89480.00520.00160.0031
10,000Ours-Q0.99960.00520.00040.0004
Ours-D1.00000.01300.00600.0000
RIWF0.86360.00520.04160.0231

At N=10,000N=10{,}000, Thrive on Qwen3-TTS (Ours-Q) and RIWF have the same observed unwatermarked FPR of 0.0052, while Ours-Q achieves higher correct attribution (0.9996 vs. 0.8636) and lower out-of-pool FAR (0.0004 vs. 0.0416). Thrive on Dots.tts (Ours-D) achieves 1.0000 correct attribution and 0.0060 out-of-pool FAR at an observed unwatermarked FPR of 0.0130. These results demonstrate accurate identification of registered users alongside rejection of unregistered watermarks under clean conditions.

References:

[1] Lu S, et al. Robust watermarking using generative priors against image editing: From benchmarking to advances. ICLR 2025.

[2] Zhao X, et al. Sok: Watermarking for ai-generated content. IEEE S&P 2025.

AuthorsOpenReview ↗

Response to Reviewer i2Ap (Part 2/2)

W2&Q2: Fair Comparison with Strong Multi-bit Baselines under Matched Conditions

We retrained Groot and HFGw on LJSpeech at 16 bps using the same augmentation as Thrive. Because their embedding mechanisms depend on their synthesis architectures, we retain their native backbones and training objectives. Tables II and III report speech quality and reconstruction robustness under these settings.

Table II. Fidelity.

MethodPESQ ↑STOI ↑NISQA ↑
Groot3.375 ± 0.2910.958 ± 0.0403.916 ± 0.627
HFGw3.591 ± 0.4750.986 ± 0.0363.126 ± 0.753
Ours-Q3.565 ± 0.7680.958 ± 0.0732.807 ± 1.165
Ours-D4.210 ± 0.2590.999 ± 0.0034.094 ± 0.548

Table III. Robustness against reconstruction attacks.

Reconstruction attackGrootHFGwOurs-QOurs-D
Neural codec regeneration
EnCodec0.9820.5650.8910.967
FACodec0.7320.4860.7880.842
STCodec0.9770.4710.7620.818
HiFiCodec0.7070.4940.8540.943
VoiceCraft0.9880.5510.7960.837
Vocoder resynthesis
WaveGlow0.7550.6220.9330.892
HiFi-GAN0.7240.5260.8600.769
BigVGAN0.7460.5130.8990.882
Vocos0.7440.5400.8810.863
DiffWave0.7550.5630.9250.868
PriorGrad0.7550.5630.9260.870
WaveFM0.7470.5370.9200.965
Enhancement reconstruction
FRCRN0.9970.9980.9020.805
Demucs0.9940.9990.9760.895

Across the 14 reconstruction attacks, Ours-Q/D achieve mean bit ACC of 0.880/0.873, compared with 0.829/0.602 for Groot/HFGw. Both Thrive variants outperform both baselines on all seven vocoder attacks, with mean ACC of 0.906/0.873 versus 0.747/0.552. The comparison between Ours-Q and HFGw further demonstrates this advantage at similar PESQ (3.565 vs. 3.591), while Ours-D achieves the highest mean scores across all three quality metrics. For attribution, Groot and HFGw use the fixed watermark and require retraining to change identities. We therefore compare against RIWF, which supports changing identifiers within a single trained model.

These results show that Thrive's overall reconstruction-robustness gains persist after matching payload and augmentation. Together with our ablations, they support the coordinated embedding and recovery design of RISE and CARE across both discrete-token and continuous-representation autoregressive TTS.

Thank you for your clarification and supplementary experiment; after considering the rebuttals, I maintain my original rating of 3.

Dear Reviewer i2Ap,

Thank you very much for taking the time to review our rebuttal. We are glad that our responses helped address your questions, and sincerely appreciate your recognition and continued support for our work.

Best regards,

The Authors

Reviewer_mRrQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces THRIVE, a multi-bit generative speech watermarking framework designed to protect modern autoregressive speech synthesis against learned model-driven reconstruction attacks. Through controlled spectro-temporal analysis, the authors show that no single acoustic carrier or observation space remains reliably invariant across diverse neural codecs, vocoders, and enhancement models. To overcome this vulnerability, the framework employs a dual-path embedding scheme that synchronizes intermediate representation residuals with low-rank generation modulation, enabling dynamic inference-time watermarking without model retraining. For extraction, a dual-expert recovery module evaluates complementary waveform and spectral representations, using a calibrated reliability selector to dynamically choose the most dependable expert for each bit. Evaluated across discrete-token and continuous-representation backbones, the method preserves synthesis fidelity and common distortion robustness, though it exhibits substantial performance drops under aggressive neural codec regeneration.

Critical strengths

The paper provides a clear diagnostic showing that pattern survival is an outcome of the joint interaction between the embedded acoustic structure, the reconstruction mechanism, and the observation domain. This provides a clear conceptual motivation for multi-representation recovery.

Unlike prior work restricted to two-stage acoustic-vocoder pipelines or zero-bit sampling steering, THRIVE addresses both discrete-token and continuous-representation autoregressive models, supporting inference-time embedding of distinct identifiers without fine-tuning the base generator.

Decoupling capability learning from reliability calibration ensures that the expert selector is trained on stable, fixed expert behaviors, preventing convergence instability.

The evaluation covers an extensive set of 14 reconstruction models across neural codecs, vocoders, and enhancement systems, alongside out-of-distribution multi-speaker and cross-lingual evaluations.

Critical weaknesses

The paper’s central point is achieving robustness against model-driven reconstruction. However, under neural codec regeneration, the recovery drops substantially. On Qwen3-TTS, the TPR@1% FPR drops to 23.0% on STCodec, 30.9% on FACodec, 33.8% on VoiceCraft, and 58.0% on HiFiCodec. On Dots.tts, performance also drops sharply on STCodec (42.2%) and VoiceCraft (50.4%). If the watermark fails to verify on 50% to 75% of samples under common codecs, the core claim of robust reconstruction survival is not adequately met.

The framework introduces considerable engineering complexity: On the generation side, RISE requires LoRA modulation across 57 modules in Qwen3-TTS and 48 modules in Dots.tts, along with multi-stage residual convolutional injection.

On the recovery side, CARE requires executing two full deep neural networks (a 1D dilated convolutional waveform model and a 2D multi-scale spectral model), extracting segment and global logits, computing statistical descriptors, and evaluating an MLP selector. The paper provides no measurements of inference latency, GPU memory usage, throughput, or RTF for either generation or recovery, making it difficult to assess deployability.

The evaluation is limited to a 16-bit payload. In practical applications, 16 bits is barely sufficient for user indexing and leaves no budget for timestamps, session metadata, or ECC. Even at 16 bits, when bit accuracy drops under neural codec regeneration, attribution accuracy falls to 26.6%–39.4% over 10,000 candidate identities. The paper does not demonstrate whether the approach can scale to standard 32-bit or 64-bit payloads without degrading audio fidelity.

The threat model evaluates only off-the-shelf, non-targeted reconstruction models. A motivated adversary aware of the dual-expert mechanism could introduce simple adversarial perturbations or optimize lossy filters targeting the agreement descriptors of the selector to force incorrect expert choices.

Questions for authors

How robust is the reliability selector against targeted manipulations, such as adversarial noise designed to maximize disagreement between the segment and global heads?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mRrQ (Part 1/3)

Dear Reviewer mRrQ,

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully addressed each comment below and revised the manuscript accordingly, with all changes highlighted for ease of reference.

W1: Robustness Under Neural Codec Reconstruction

We agree that verification remains challenging under some neural codecs, but Table 2 (Sec. 5.3 of the manuscript) shows gains under neural codec and other reconstruction attacks. Our controlled analysis demonstrated that reconstruction affects watermark structures and their observability differently, leaving no single carrier uniformly reliable across attacks. Therefore, the key contribution of our Thrive lies in identifying an embedding space that provides stable and balanced resistance to diverse regeneration attacks, thereby improving overall watermark robustness.

At the same 16 bps, Thrive on Qwen3-TTS and Dots.tts (Ours-Q/D) achieves codec-average TPR@1%FPR of 44.1%/67.6%, versus AudioSeal's 0.3% and RIWF's 21.3%. ACC, common in multi-bit watermarking, captures bit recovery even when verification fails. Ours-Q/D retain codec ACC of 76.2%-89.1% / 81.8%-96.7% and exceed RIWF on 13 of 14 models spanning codecs, vocoders, and enhancement systems. Although some attacks remain challenging, Thrive achieves more balanced robustness across codec regeneration, vocoder resynthesis, and enhancement reconstruction. The framework improves resistance to reconstruction attacks on both discrete-token and continuous-representation autoregressive TTS.

W2: Computational Complexity and Deployment Cost of RISE and CARE

We have added an efficiency analysis in Sec. 5.6, with detailed comparisons in Table I. The results show that Thrive introduces small generation overhead and completes recovery from one-second speech in approximately 4.5 ms.

All methods are benchmarked on the same GPU using FP32, batch size one, and 16 bps. Generation overhead is relative to each native host: Thrive covers text-to-waveform synthesis, while vocoder baselines start from mel spectrograms. Recovery includes each method's full extraction pipeline on identical one-second speech.

Table I. Computational efficiency.

MethodParams. (M) ↓WM Size (MiB) ↓GFLOPs ↓Latency (ms) ↓GPU (GiB) ↓Throughput ↑RTF ↓
Generation
HFGw13.936 / 053.16252.813 / 0 (0.00%)4.401 / 0.003 (0.06%)0.122226.7590.004
RIWF0 / 0.1150.438346.116 / 0.001 (0.0003%)37.377 / 3.869 (10.35%)0.09524.4880.041
Groot0 / 227.063866.177346.116 / 0.454 (0.13%)38.916 / 0.977 (2.51%)1.36025.3200.039
Ours-Q0 / 5.88922.450456.763 / 0.826 (0.18%)1013.676 / 6.499 (0.64%)4.3560.9411.063
Ours-D0 / 12.19646.52411838.964 / 24.306 (0.21%)971.546 / 30.920 (3.18%)9.4350.9981.002
Recovery
HFGw0.3061.16926.7131.3130.060761.6250.001
RIWF4.35316.6049.20810.4470.04595.7250.010
Groot23.13188.2390.3920.8440.1011185.0180.0008
Ours-Q2.83910.8294.8424.6240.088216.2480.005
Ours-D2.83910.8294.8424.4460.088224.9430.004

The results highlight three advantages: 1. Parameter-efficient adaptation. RISE freezes the TTS host and trains only 5.889M/12.196M watermark parameters on Qwen/Dots, 97.4%/94.6% fewer than Groot, keeping the adaptation compact. 2. Low generation overhead. Coordinating representation injection with generation modulation adds little overhead, increasing GFLOPs by only 0.18%/0.21% and end-to-end latency by 0.64%/3.18%. 3. Efficient dual-expert recovery: Full CARE remains efficient with both experts and selection included, processing one-second speech in 4.446-4.624 ms with approximately 0.088 GiB peak GPU memory and 56%-57% lower latency than RIWF.

AuthorsOpenReview ↗

Response to Reviewer mRrQ (Part 2/3)

W3: Capacity Scalability

We thank the reviewer for highlighting capacity. We have extended our evaluation to 20- and 32-bit payloads on Qwen3-TTS and Dots.tts (Ours-Q/D), assessing fidelity, robustness, and multi-user attribution on LJSpeech. More detailed results, including out-of-distribution evaluations, are provided in App. D.9.

Table II. Fidelity and robustness.

MetricOurs-Q (20-bit)Ours-Q (32-bit)Ours-D (20-bit)Ours-D (32-bit)
Fidelity
STOI ↑0.948 ± 0.0880.955 ± 0.0780.999 ± 0.0020.993 ± 0.006
PESQ ↑3.118 ± 0.6973.188 ± 0.7354.211 ± 0.3473.834 ± 0.454
SSIM ↑0.836 ± 0.0930.839 ± 0.0780.970 ± 0.0040.929 ± 0.012
MCD ↓2.154 ± 0.1392.369 ± 0.1420.961 ± 0.1182.159 ± 0.198
DNSMOS-SIG ↑2.105 ± 0.4412.325 ± 0.5543.038 ± 0.3163.003 ± 0.298
NISQA ↑2.732 ± 1.1242.747 ± 1.1514.122 ± 0.6103.591 ± 0.743
FAD-VGGish ↓2.3402.5260.5622.346
FAD-CLAP ↓0.1270.1750.0450.200
Robustness: ACC / TPR ↑
Common distortions0.992 / 0.9870.960 / 0.9950.993 / 0.9890.998 / 1.000
Neural codec regeneration0.805 / 0.6030.660 / 0.2480.789 / 0.5600.765 / 0.553
Vocoder resynthesis0.888 / 0.8500.745 / 0.5420.846 / 0.7590.700 / 0.387
Enhancement reconstruction0.781 / 0.5400.926 / 0.9510.827 / 0.6600.586 / 0.155

Table III. Multi-user Attribution.

ConditionCandidates (N)Ours-Q (20-bit)Ours-Q (32-bit)Ours-D (20-bit)Ours-D (32-bit)
Clean101.0001.0001.0001.000
Clean1001.0001.0001.0001.000
Clean1,0000.9991.0001.0001.000
Clean10,0000.9991.0001.0001.000
Common distortions100.9881.0000.9951.000
Common distortions1000.9730.9900.9871.000
Common distortions1,0000.9610.9850.9801.000
Common distortions10,0000.9480.9760.9730.999

Table II shows that, at 20 bits, Ours-Q/D achieve PESQ of 3.118/4.211 while retaining codec ACC of 80.46%/78.93% and vocoder ACC of 88.76%/84.63%. Table III further demonstrates effective higher-capacity attribution: at 10,000 candidates, both 32-bit models achieve 100% under clean conditions and 97.63%/99.91% under common distortions. The lower codec and vocoder recovery at 32 bits shows that increasing capacity makes it harder to maintain robustness across reconstruction attacks.

We selected 20 bits to match HFGw's default payload and 32 bits to test doubling the main payload. Our main 16-bit setting is consistent with AudioSeal and RIWF and provides 65,536 possible identifiers, covering the 10,000-candidate identity evaluation used to assess our core designs.

Maintaining reconstruction robustness at higher payloads remains a limitation of the current configuration. We thank the reviewer for motivating this analysis and highlighting future work on higher-capacity watermarking with stronger robustness and support for richer metadata and error correction. We have revised App. E accordingly.

AuthorsOpenReview ↗

Response to Reviewer mRrQ (Part 3/3)

W4: Robustness Against Adaptive Attacks

We evaluated an agreement-targeted attack designed to manipulate the reliability estimates of Cross-Representation Adaptive Recovery (CARE) and induce incorrect expert selection. With a perturbation bound of α=0.001\alpha=0.001 and budgets up to 1,000 steps, Qwen/Dots retain mean bit ACC of 0.966/0.975 and empirical TPR of 0.920/1.000, respectively.

We adopted AudioMarkBench's white-box optimization loop [1] with AudioSeal's perturbation parameterization δ=αtanh⁡(v)\delta=\alpha\tanh(v) [2]. Given the known watermark message, we targeted bits where the experts disagreed but CARE selected correctly, optimizing selector scores to favor the incorrect expert. Gradients flowed only through the agreement-descriptor path. If no such bits existed, we used a generic watermark-removal objective under the same attack budget. CARE remained frozen, while only vv was optimized using Adam (learning rate 0.1) on 50 utterances per model.

Table IV. Robustness under agreement-targeted attacks.

ModelαStep budgetBit ACC ↑TPR@1%FPR ↑PESQ ↑STOI ↑
Qwen0.0011000.974 ± 0.0490.9403.3870.931
Qwen0.0015000.971 ± 0.0490.9403.3830.931
Qwen0.0011,0000.966 ± 0.0540.9203.3840.930
Dots0.0011000.983 ± 0.0341.0003.9270.996
Dots0.0015000.978 ± 0.0391.0003.9210.996
Dots0.0011,0000.975 ± 0.0421.0003.9200.996

As shown in Table IV, recovery declines only modestly as the budget increases. At 1,000 steps, Qwen/Dots retain bit ACC of 0.966/0.975 and TPR@1%FPR of 0.920/1.000, respectively. This behavior is consistent with CARE's joint reliability decision, which assesses agreement alongside confidence, segment strength, and temporal stability when choosing between complementary waveform and spectral experts. The limited routing damage and high recovery on both backbones support CARE's design for sustaining recovery under agreement-targeted perturbations. We have added this experiment to App. D.7.

Q1: Robustness of the Reliability Selector Against Targeted Attacks

We evaluated a targeted attack designed to maximize disagreement between each expert's segment and global predictions and disrupt CARE's reliability estimates. At α=0.001\alpha=0.001, the attack increases this disagreement, while Qwen/Dots retain mean bit ACC of at least 0.995/0.996 and TPR@1%FPR of 1.000 across budgets of 100, 500, and 1,000 steps.

To target segment-global disagreement, we used each expert's inverted global bit predictions as segment targets at each iteration and minimized binary cross-entropy averaged across both experts. We optimized this CARE-specific objective using AudioMarkBench's white-box loop [1], with the bounded perturbation δ=αtanh⁡(v)\delta=\alpha\tanh(v) adopted from AudioSeal [2]. CARE remained frozen, and we evaluated 50 utterances per model with α=0.001\alpha=0.001, Adam (lr=0.1), and budgets up to 1,000 steps. For each step budget, results correspond to the saved attack iterate with the lowest ACC.

Table V. Robustness under disagreement-targeted attacks.

ModelαStep budgetBit ACC ↑TPR@1%FPR ↑Δ Disagreement ↑PESQ ↑STOI ↑
Qwen0.0011000.996 ± 0.0151.0000.0153.2420.946
Qwen0.0015000.995 ± 0.0171.0000.0173.2200.947
Qwen0.0011,0000.995 ± 0.0171.0000.0183.2160.948
Dots0.0011000.996 ± 0.0201.0000.0143.8370.997
Dots0.0015000.996 ± 0.0201.0000.0163.8130.997
Dots0.0011,0000.996 ± 0.0201.0000.0163.8080.997

The positive disagreement increments confirm that the attack disrupted its intended cue, while recovery changed little with additional budget. This is consistent with CARE's decision structure. Segment predictions supply reliability cues, whereas recovered bits come from the selected expert's global predictions. Greater segment-global disagreement can therefore coexist with correct global predictions. CARE also combines agreement with confidence, segment strength, and temporal stability when comparing experts. These results support using complementary waveform-spectral predictions and joint reliability estimation to sustain recovery when internal consistency cues are deliberately perturbed at this budget. We have added this experiment to App. D.7.

References:

[1] Liu H, et al. AudioMarkBench: Benchmarking Robustness of Audio Watermarking. NeurIPS 2024.

[2] Roman R S, et al. Proactive Detection of Voice Cloning with Localized Watermarking. ICML 2024.

Dear Reviewer mRrQ,

As the discussion period draws to a close, we would be grateful for your feedback on our responses and additional experiments. If any questions remain, we would be happy to clarify them. We sincerely appreciate the time and care you have devoted to reviewing our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Responses and Revision Summary

We sincerely appreciate all reviewers for their constructive feedback, which has helped us further clarify and demonstrate the strengths of our method. Following their suggestions, we have provided point-by-point responses and revised the manuscript accordingly, with changes highlighted in blue. To facilitate review, we summarize the main concerns, corresponding revisions, and key findings below.

  1. Method advantages: In response to the three reviewers' comments, we clarified how Thrive leverages RISE to support multi-user attribution across both discrete-token and continuous-representation autoregressive TTS while achieving balanced resistance to diverse reconstruction attacks. We have also conducted additional comparisons under matched payload capacities and training augmentation according to i2AP. Thrive consistently maintains its overall robustness advantage under these controlled settings, further validating the effectiveness of our embedding and recovery design.

  2. Computational efficiency: Following reviewers mRrQ and 6cPq, we added an efficiency analysis in Sec. 5.6 of the manuscript. The results show that RISE enables parameter-efficient adaptation with small overhead, while the complete CARE pipeline recovers watermarks from one-second speech in approximately 4.5 ms.

  3. Robustness against adaptive attacks: Following reviewer mRrQ's suggestion, we have added two white-box attacks targeting expert selection and segment-global disagreement in App. D.7. Both backbones maintain high recovery accuracy under the evaluated perturbation budget, further supporting CARE's reliability against targeted manipulation.

  4. Verification protocol and open-set attribution: Addressing reviewers i2Ap and 6cPq, we detailed TPR computation and threshold calibration in App. D.2 and added clean open-set attribution experiments in App. D.8. With up to 10,000 candidates, Thrive accurately identifies registered users while maintaining low false-attribution rates for unregistered identities.

We thank all reviewers again for their time and constructive feedback. We hope that our responses and revisions adequately address their concerns.

AuthorsOpenReview ↗

Final Rebuttal Summary and Reviewer Feedback

Dear AC,

We sincerely appreciate the time and effort you and the reviewers have devoted to our submission. We thank all reviewers for their constructive feedback, which has helped us clarify our contributions and strengthen the experimental evidence.

In this paper, motivated by real-world speech distribution and editing, we investigate how model-driven reconstruction can preserve speech utility while disrupting watermarks. To understand this vulnerability, we conduct controlled analysis and establish a reusable reconstruction benchmark, revealing how watermark survival depends on reconstruction mechanisms and recovery representations. These insights motivate Thrive, a first multi-bit generative watermarking framework for both discrete-token and continuous-representation autoregressive TTS, built on Representation-Generation Synchronized Embedding and Cross-Representation Adaptive Recovery. Experiments demonstrate improved robustness across heterogeneous reconstruction attacks and preserved synthesis fidelity, supporting provenance verification and multi-user attribution without retraining for new watermark.

  1. Scientific Insights. Our controlled analysis shows that no single carrier or recovery representation is uniformly reliable across reconstruction attacks, motivating coordinated embedding and adaptive recovery. Reviewers mRrQ and i2Ap highlighted these diagnostic insights, while 6cPq recognized the informative robustness differences across backbones and the reusable reconstruction benchmark for the community.

  2. Methodological Strengths. Guided by these insights, Thrive combines Representation-Generation Synchronized Embedding (RISE) and Cross-Representation Adaptive Recovery (CARE) with two-stage training that separates capability learning from reliability calibration. Reviewer mRrQ highlighted the stability of this training strategy, while i2Ap recognized the contributions of both embedding and recovery components.

  3. Robustness and Generalization. Our experiments demonstrate synthesis fidelity, robustness across diverse reconstruction attacks, and multi-user attribution, supported by generalization studies and component ablations. Reviewer mRrQ recognized the breadth of the evaluation, while i2Ap highlighted its practical relevance and the ablation evidence supporting our design.

  4. Practical Applicability. Thrive supports user-specific identifiers at inference time without retraining across both discrete-token and continuous-representation autoregressive TTS, enabling one trained model to serve multiple users with distinct watermarks. This flexibility was explicitly recognized by reviewers mRrQ and i2Ap.

Furthermore, we summarize the main revisions below.

  1. Computational efficiency. Following reviewers mRrQ and 6cPq, we added generation and recovery measurements in Sec. 5.6 (Table 4). Thrive adds little generation overhead, with full CARE recovery in ~4.5 ms per one-second speech.

  2. Verification and open-set attribution. In response to the comments from reviewers i2Ap and 6cPq, we clarified empirical TPR computation and threshold calibration in App. D.2 and added open-set attribution experiments in App. D.8. Under clean conditions, Thrive accurately identifies registered users among 10,000 candidates while maintaining low false-attribution rates for unregistered identities.

  3. Robustness against adaptive attacks. Following reviewer mRrQ, we added two white-box attacks targeting expert selection and segment-global disagreement in App. D.7. Both backbones maintain high recovery accuracy under the evaluated perturbation budget and up to 1,000 optimization steps, further supporting CARE's reliability against targeted manipulation.

  4. Higher capacity and fair comparisons. To address feedback from reviewers mRrQ and i2Ap, we extended capacity evaluations to 20 and 32 bps and retrained Groot and HFGw at 16 bits using Thrive's configuration. The results demonstrate accurate multi-user attribution at higher capacities, while confirming Thrive's robustness advantages in matched comparisons.

In a nutshell, we sincerely thank reviewers 6cPq and i2Ap for reviewing our responses, confirming that their concerns were well addressed, and giving positive assessments before the close of the author-reviewer discussion period. We also sincerely thank reviewer mRrQ for the constructive feedback and have responded to each point. Our efficiency measurements addressed the deployment costs that both mRrQ and 6cPq asked about, while our matched baseline comparisons and open-set evaluation addressed i2Ap's related questions about reconstruction robustness and multi-user attribution.

We are deeply grateful to all reviewers and the AC for the time, care, and effort devoted to evaluating our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.