acceptodds
Under review as a conference paper at ICLR 2027

Self-Rewarding GRPO for Instruction-Guided Zero-Shot Speech Synthesis

Abstract

Following natural-language instructions while preserving an unseen speaker's identity remains challenging for zero-shot text-to-speech. We present ACE-Instruct-TTS and Self-Rewarding Group Relative Policy Optimization (SR-GRPO), a post-training method that derives instruction rewards from the TTS model itself rather than an external audio understanding model. SR-GRPO uses a frozen supervised fine-tuning (SFT) checkpoint to score the same generated speech tokens with and without the instruction, holding the target text and speaker conditioning fixed. Inspired by conditional pointwise mutual information, the length-normalized log-likelihood ratio provides a continuous measure of relative compatibility with the instruction. We combine this signal with speaker-similarity and duration rewards. We also introduce ACE-Instruct-TTS-Eval-ZH, a 650-sample Chinese benchmark evaluating instruction following, speaker preservation, and intelligibility across six categories, including compositional and conflicting conditions. SR-GRPO improves instruction-following rate from 67.08% to 75.23%, the highest among evaluated systems, and increases mean human-rated instruction-following scores from 3.64 to 3.90. Reward ablations favor the proposed likelihood contrast over conditional-only, rank-based, and external binary rewards. Speaker similarity remains above all evaluated external baselines but decreases relative to SFT, while character error rate remains comparable. Beyond this benchmark, SR-GRPO improves the Chinese InstructTTSEval macro-average score from 79.2% to 81.2% and improves joint success across three SpeechEditBench tasks under an adapted re-synthesis protocol. On a second backbone, CosyVoice3, SR-GRPO improves instruction-following rate by 5.54 percentage points, with a similar speaker-preservation trade-off. We train instruction following only in Mandarin and leave multilingual instruction following to future work.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-shot TTS, controllable speech synthesis, group relative policy optimization, self-rewarding, conditional PMI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_if65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SR-GRPO for instruction-guided zero-shot TTS. The method uses a frozen SFT model to score the same generated speech tokens with and without the instruction, using the difference in their length-normalized log-likelihoods as an instruction reward. This reward is combined with speaker-similarity and duration rewards for policy optimization. The authors also introduce a 650-sample Chinese benchmark covering six categories of instructions. SR-GRPO improves instruction-following rate, while speaker similarity decreases. Reward ablations, comparisons with classifier-free guidance and Best-of-N decoding, evaluations on external benchmarks, and experiments on CosyVoice3 provide further evidence for the approach.

Critical strengths
  1. The likelihood-contrast reward is simple to implement and avoids an external instruction evaluator. The reward ablations provide useful evidence for the proposed formulation.

  2. The evaluation covers multiple baselines, instruction categories, and reward configurations. The CosyVoice3 experiment provides additional evidence beyond the authors' own backbone.

  3. The benchmark evaluates instruction following alongside speaker similarity and intelligibility, making the trade-off between controllability and speaker preservation explicit.

Critical weaknesses
  1. The instruction reward is an indirect measure of adherence. A higher conditional likelihood relative to the instruction-free baseline does not necessarily mean that all requested attributes are satisfied. The ranking results provide some support, but the paper does not directly establish how well the reward correlates with human judgments across instruction categories.

  2. The perceptual cost of reduced speaker similarity remains unclear. SIM drops from 0.756 to 0.733 after RL, but the human study evaluates instruction following only and excludes speaker timbre and general audio quality. Reference-aware human evaluations would help determine whether the gain in adherence comes at a meaningful cost to speaker identity or naturalness.

  3. The robustness of some gains is uncertain. SR-GRPO improves IFR by only 0.92 percentage points over Best-of-8. The paper does not report paired confidence intervals for system differences. In addition, Best-of-N uses CPMI alone, whereas RL optimizes speaker similarity and duration as well. A comparison using the same combined rewards would better isolate the benefit of policy optimization.

Questions for authors
  1. How strongly does the likelihood-contrast reward correlate with human-rated instruction adherence, particularly for compositional instructions where several attributes must be satisfied simultaneously?

  2. Could you provide reference-aware human evaluations of speaker similarity and naturalness for SFT versus SR-GRPO? Is the SIM evaluation independent of the WavLM-based representation used for speaker conditioning and the timbre reward?

  3. Could you report paired uncertainty estimates for the main comparisons and compare SR-GRPO against Best-of-N selection using the same combined reward signals?

5 comments
AuthorsOpenReview ↗

W1&Q1: Correlation Between Likelihood-Contrast Reward and Human-Rated Instruction Adherence

We thank the reviewer for raising this important point. We agree that a higher likelihood contrast does not guarantee that every requested attribute is satisfied. To assess whether CPMI nevertheless provides a meaningful adherence signal, we examine its alignment with human judgments and external evaluation.

(1) Direct comparison with human-rated instruction adherence. Three raters performed A/B comparisons on 120 category-stratified inputs. Importantly, raters were asked to choose the output that better followed the instruction, not the output with better audio quality or greater overall appeal. These judgments therefore assess relative instruction adherence rather than general preference.

Among the 87 pairs with a clear majority preference, CPMI agrees with the human majority on 70.1% of pairs (95% CI: 59.8–79.3%), with Cohen's κ=0.404\kappa = 0.404. The signed CPMI difference also correlates positively with the proportion of raters preferring A for instruction adherence (Spearman's ρ=0.374\rho = 0.374, CI: 0.180–0.539). These results indicate moderate positive alignment with human-rated instruction adherence: CPMI is informative, but imperfect.

(2) Complementary evidence from external evaluation. We do not use CPMI itself as an evaluation metric. Under IFR evaluation by Gemini-2.5-Pro, an external evaluator independent of the policy model, IFR decreases monotonically from 71.23% for Rank-0 to 63.08% for Rank-4. CPMI-based Best-of-8 selection also increases IFR from 67.08% to 74.31%. These findings complement the human results by supporting the usefulness of CPMI for selecting outputs with better instruction adherence.

Full results are provided in Appendix A.5.

AuthorsOpenReview ↗

W2&Q2: Perceptual Trade-offs and Independence of Speaker Similarity Evaluation

Thank you for highlighting this limitation. We have added human evaluations of speaker similarity and overall audio quality, clarified SIM's dependence on the WavLM-based pipeline, and assessed the robustness of the speaker-similarity findings across additional encoders.

Reference-aware human evaluation

We evaluated SFT and SR-GRPO using speaker-similarity MOS (SMOS) and overall audio-quality MOS (QMOS). For SMOS, raters listened to the corresponding reference audio to assess speaker identity. All nine raters evaluated all 60 clips for each metric.

MetricSFTSR-GRPO
SMOS4.01 (0.20)3.75 (0.25)
QMOS3.79 (0.14)3.83 (0.08)

Values are means; parentheses indicate 95% confidence-interval half-widths. Higher is better.

The lower SMOS after RL is consistent with the SIM decrease from 0.756 to 0.733, suggesting a trade-off between instruction adherence and perceived speaker similarity. Mean QMOS remained similar. Details are provided in Appendix A.7.

SIM dependence and cross-encoder robustness

SIM is not fully independent of the WavLM-based representation used for speaker conditioning and the timbre reward.

To assess the robustness of the speaker-similarity findings across encoders and reduce reliance on the WavLM-based evaluator, we additionally evaluated SFT and SR-GRPO using three WeSpeaker encoders:

Speaker encoderSFTSR-GRPO
ResNet340.86330.8473
CAM++0.87760.8657
ECAPA1024_LM0.86400.8543

All three encoders show lower speaker similarity for SR-GRPO than for SFT, consistent with SIM and SMOS. These results and the discussion of SIM dependence are included in the appendix subsection "Additional Speaker-Encoder Evaluation."

Overall, the human ratings and cross-encoder results consistently suggest a speaker-similarity cost alongside improved adherence, while mean overall audio quality remains similar.

AuthorsOpenReview ↗

W3&Q3: Statistical Robustness and Reward-Matched Comparison with Best-of-N

Thank you for raising these concerns. We have added paired uncertainty estimates and combined-reward Best-of-NN results, and clarified the inference-cost trade-off.

1. Paired uncertainty estimates

For automatic IFR, we use all 650 matched inputs, 10,000 paired bootstrap resamples for 95% confidence intervals, and two-sided McNemar tests with Holm correction across six comparisons. Confidence intervals are pointwise.

BaselineSR-GRPO improvement (pp)95% paired CI (pp)Holm-adjusted pp
CosyVoice3+8.92+8.92[+5.38,+12.46][+5.38,+12.46]4.972×10−64.972\times10^{-6}
VoxCPM2+15.08+15.08[+11.38,+18.92][+11.38,+18.92]8.880×10−148.880\times10^{-14}
Qwen-Audio-3.0-TTS-Plus+2.92+2.92[−0.31,+6.15][-0.31,+6.15]0.19130.1913
MiMo-V2.5-TTS+23.54+23.54[+19.54,+27.69][+19.54,+27.69]1.377×10−261.377\times10^{-26}
ACE-Instruct-TTS-SFT+8.15+8.15[+4.77,+11.54][+4.77,+11.54]1.246×10−51.246\times10^{-5}
Best-of-8, CPMI-only+0.92+0.92[−2.31,+4.15][-2.31,+4.15]0.63380.6338

SR-GRPO significantly outperforms SFT and three external baselines after correction. Its differences from CPMI-only Best-of-8 and Qwen-Audio-3.0-TTS-Plus are not statistically significant, and we have qualified our claims accordingly. Details are provided in Appendix A.10.

For subjective scores, paired analyses on 30 matched inputs per metric yield:

MetricSR-GRPO minus SFT95% paired bootstrap CI
IFMOS+0.256+0.256[+0.022,+0.500][+0.022,+0.500]
SMOS−0.259-0.259[−0.513,−0.035][-0.513,-0.035]
QMOS+0.048+0.048[−0.085,+0.204][-0.085,+0.204]

These results support improved adherence with an average perceptual similarity cost, but no statistically detectable quality difference. Evidence for the SMOS decrease is sensitive to the test used. Details are provided in Appendix A.9.

2. Combined-reward Best-of-N

We added CPMI+timbre selection using the training weights of 0.6 and 0.3, respectively. Each reward is normalized separately by subtracting its within-input candidate-group mean and dividing by its group variance before combination.

Overall results (650 inputs). Bold indicates the best value in each row.

ScoringMetricN=1N=1N=2N=2N=4N=4N=8N=8N=16N=16
CPMI-onlyIFR (%) ↑\uparrow67.0868.9271.2374.3171.08
CPMI-onlySIM ↑\uparrow0.7560.7390.7310.7200.722
CPMI+timbreIFR (%) ↑\uparrow67.0868.4670.0072.6170.77
CPMI+timbreSIM ↑\uparrow0.7560.7500.7420.7350.732

Both scoring schemes attain their highest overall IFR at N=8, with no further improvement at N=16. Adding timbre reward improves SIM at every N>1, but yields lower IFR. At N=8, SIM increases from 0.720 to 0.735, while IFR decreases by 1.70 percentage points.

SR-GRPO achieves 75.23% IFR, numerically exceeding CPMI-only and combined-reward Best-of-8 by 0.92 and 2.62 percentage points, respectively. Details are provided in Appendix A.5.

This comparison is partially reward-matched: the length/duration reward cannot be computed at inference time and is excluded from BoN.

3. Inference cost

Best-of-8 generates eight candidates and performs reward scoring, whereas SR-GRPO generates one candidate without reward-based selection. Assuming comparable output lengths, BoN-8 requires approximately eight times the candidate-generation work, plus CPMI scoring and, for the combined variant, timbre scoring.

Our conclusion is therefore narrower: SR-GRPO significantly improves over SFT and avoids BoN-8’s sampling and scoring overhead, but does not establish statistically significant IFR superiority over CPMI-only BoN-8.

Reviewer_vgzLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Rewarded GRPO, a special method for improving the instruction-following rate of IF-TTS. The method uses the frozen SFT predecessor as the scorer to score the same generated tokens with or without instruction. They show that the IF rate is successfully improved and the method is generalizable to another model (CosyVoice 3).

Critical strengths

The idea is interesting and seems to be reasonable

The experiments are complete and the writing is easy to follow

Critical weaknesses

Concern regarding the practicality

As the authors also mention, the proposed idea is similar to using classifier-free guidance (CFG) during inference. In Figure 2, although the performances are not exactly matched, the benefits of CFG is that no training is required. Based on this, I am questioning the practicality of the paper.

Questions for authors

Is SR-GRPO complementary to CFG?

Can we further improve the performance of a model trained with SR-GRPO by applying CFG during inference? In other words, are the improvements from SR-GRPO and CFG additive?

Furthermore, it would be valuable to investigate whether similar gains can be observed when combining CFG with other RL fine-tuning methods. Such comparisons would help clarify the unique benefits of SR-GRPO and whether its improvements are complementary to those achieved through CFG.

2 comments
AuthorsOpenReview ↗

Practicality of SR-GRPO and Complementarity with CFG

We thank the reviewer for these related questions. We agree that CFG is an attractive training-free baseline. To assess the practical benefits of SR-GRPO beyond CFG tuning and its complementarity with CFG, we evaluated SFT, SR-GRPO, and SR-GSPO across eight CFG scales from 0.50 to 4.00.

For SR-GSPO, we used --ppo_epochs 2 and --clip_epsilon 0.05; all other settings were identical to those of SR-GRPO. The complete results are summarized below.

Table 16. CFG sweep for SFT, SR-GRPO, and SR-GSPO. IFR and CER are reported as percentages. Higher IFR and SIM are better; lower CER is better.

CFGSFT IFR ↑SFT SIM ↑SFT CER ↓SR-GRPO IFR ↑SR-GRPO SIM ↑SR-GRPO CER ↓SR-GSPO IFR ↑SR-GSPO SIM ↑SR-GSPO CER ↓
0.5059.540.7791.4266.770.7531.8965.850.7431.51
1.0067.080.7562.0175.230.7331.9773.850.7142.38
1.5070.920.7332.0775.230.7092.5975.690.7022.06
2.0073.690.7152.9676.150.6962.6676.460.6852.43
2.5076.000.7002.8677.230.6813.3778.150.6722.77
3.0077.380.6803.9178.620.6682.9979.230.6543.05
3.5078.310.6634.7480.460.6583.6979.850.6443.40
4.0080.000.6524.4881.230.6503.7480.620.6423.61

1. Practicality: CFG tuning alone does not fully reproduce SR-GRPO’s quality trade-offs.

The practical comparison should consider instruction following together with speaker and content preservation, rather than IFR alone. At exactly the same speaker similarity (SIM=0.733\mathrm{SIM} = 0.733), SFT with CFG=1.50\mathrm{CFG} = 1.50 achieves 70.92% IFR and 2.07% CER, whereas SR-GRPO with CFG=1.00\mathrm{CFG} = 1.00 achieves 75.23% IFR and 1.97% CER. Thus, SR-GRPO improves IFR by 4.31 percentage points without reducing SIM or increasing CER.

The advantage also holds at a high-IFR operating point: SFT with CFG=4.00\mathrm{CFG} = 4.00 achieves 80.00% IFR, 0.652 SIM, and 4.48% CER, while SR-GRPO with CFG=3.50\mathrm{CFG} = 3.50 achieves 80.46% IFR, 0.658 SIM, and 3.69% CER, improving all three metrics.

We acknowledge the upfront training cost of SR-GRPO. However, it introduces no additional inference module or decoding steps relative to SFT under the same decoding configuration. Its practical value is therefore to improve attainable quality trade-offs without adding inference-time components, with the training cost amortized over subsequent use.

2. Complementarity: CFG further improves instruction following after SR-GRPO.

Increasing CFG from 1.00 to 4.00 raises SR-GRPO’s IFR from 75.23% to 81.23%, providing a further 6.00-percentage-point gain. SR-GRPO also achieves higher IFR than SFT at every tested CFG scale. These results directly support their empirical complementarity.

3. Comparison with another RL method: CFG complementarity is shared.

SR-GSPO also benefits from CFG, with IFR increasing from 73.85% at CFG=1.00\mathrm{CFG} = 1.00 to 80.62% at CFG=4.00\mathrm{CFG} = 4.00. Therefore, we do not claim that complementarity with CFG is unique to SR-GRPO.

Instead, the comparison highlights differences in the resulting quality trade-offs. At CFG=1.00\mathrm{CFG} = 1.00, SR-GRPO outperforms SR-GSPO on all three metrics: 75.23% vs. 73.85% IFR, 0.733 vs. 0.714 SIM, and 1.97% vs. 2.38% CER. Moreover, SR-GRPO maintains higher SIM at every tested CFG scale. SR-GSPO remains competitive in IFR and CER at several scales.

In summary, SR-GRPO enables a better balance between instruction adherence and speech fidelity than CFG tuning alone, without introducing additional inference-time components. CFG can further improve instruction adherence, complementing the gains achieved by SR-GRPO.

We provide extended CFG sweeps for SFT, SR-GRPO, and SR-GSPO in Appendix A.12, together with an analysis of their quality trade-offs and complementarity with CFG.

Dear Reviewer vgzL,

Thank you for your time and effort in reviewing our manuscript and for your valuable and constructive comments.

As the discussion period is drawing to a close, we would greatly appreciate it if you could let us know whether our rebuttal and revised manuscript have adequately addressed your concerns. We remain ready to provide additional clarifications or answer any further questions you may have.

Best regards, The Authors

Reviewer_cWdyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates instruction-guided zero-shot text-to-speech (TTS), with three main contributions:

  1. Self-Rewarding Group Relative Policy Optimization (SR-GRPO): A CPMI-inspired self-rewarding mechanism that uses a frozen SFT model to compute the log-likelihood difference of generated speech with and without instructions. This difference serves as a continuous instruction reward, which is combined with speaker-similarity and duration rewards for policy optimization.
  2. ACE-Instruct-TTS: An instruction-controllable zero-shot TTS system built on a decoder-only LLM initialized from Qwen3-1.7B-Instruct. It integrates discrete speech tokens from DualCodec, speaker embeddings, and a flow-matching speech decoder, enabling instruction control through SFT and SR-GRPO.
  3. ACE-Instruct-TTS-Eval-ZH: An evaluation benchmark comprising 650 Chinese samples covering emotion, speaking style, dialects and accents, acoustic controls, compositional conditions, and conflicting conditions. It jointly evaluates instruction following, speaker similarity, and speech intelligibility.

Experimental results show that SR-GRPO improves the Instruction-Following Rate (IFR) of ACE-Instruct-TTS from 67.08% to 75.23% and IFMOS from 3.64 to 3.90, while Speaker Similarity (SIM) decreases from 0.756 to 0.733. The method also achieves a 5.54 percentage-point improvement in IFR on CosyVoice3. Reward ablations, Best-of-N experiments, and comparisons with Classifier-Free Guidance (CFG) further support the effectiveness of the self-rewarding mechanism.

Overall, the experimental evaluation is relatively comprehensive and provides substantial empirical support for the proposed method. Deriving instruction rewards without relying on an external audio understanding model also offers practical value.

Critical strengths
  1. Improving instruction following while preserving speaker identity is an important challenge in instruction-guided zero-shot TTS. The paper incorporates both instruction adherence and speaker fidelity into the optimization objective, addressing a meaningful research problem.
  2. SR-GRPO constructs continuous rewards by comparing the log-likelihoods of the same generated speech with and without instructions under a frozen SFT model, without requiring an additional audio understanding model for instruction evaluation. This design provides an approach for TTS models to optimize instruction following using their own probability distributions.
  3. The paper provides a relatively comprehensive analysis of the reward mechanism. In Table 5, the CPMI-based reward achieves an IFR of 75.23%, outperforming Conditional-only (68.77%), Ranked Scalar (69.38%), and Ext-RM Binary (67.69%). The paper also examines the trade-off between instruction adherence and speaker similarity through reward weight ablations, Best-of-N experiments, and comparisons with Classifier-Free Guidance (CFG).
  4. The evaluation covers complex instructions and includes cross-model validation. ACE-Instruct-TTS-Eval-ZH includes compositional conditions and conflicting conditions. Beyond the main model, SR-GRPO improves the IFR of CosyVoice3 from 66.31% to 71.85%, providing preliminary evidence of the method's applicability across different TTS architectures.
Critical weaknesses
  1. The relationship between self-reward signals and actual instruction adherence remains insufficiently validated. The likelihood contrast measures changes in generation probability rather than directly assessing whether the requested attributes are realized. Although Figure 1 shows a relationship between CPMI ranking and IFR, the paper lacks direct correlation analysis with human judgments and control experiments using mismatched instructions.
  2. The comparison with external reward models does not fully isolate the advantages of self-rewarding. In Table 5, CPMI achieves 75.23% IFR, compared with 67.69% for Ext-RM Binary. However, both the reward source and format differ, making it difficult to determine whether the improvement comes from self-rewarding or continuous feedback.
  3. Improved instruction adherence comes at the cost of speaker similarity. In Table 2, SR-GRPO increases IFR from 67.08% to 75.23%, while SIM decreases from 0.756 to 0.733. Increasing the timbre reward weight restores SIM to 0.755 but reduces IFR to 70.31% (Table 5), indicating a persistent trade-off between the two objectives.
  4. The subjective evaluation is limited in scale, and the statistical significance of the improvements remains unclear. IFMOS is evaluated on only 30 of the 650 samples, with five per category. In Table 2, SR-GRPO and SFT achieve IFMOS scores of 3.90 ± 0.18 and 3.64 ± 0.23, respectively, with overlapping confidence intervals. The paper provides neither paired significance tests nor results across multiple training runs.
Questions for authors
  1. To what extent does the decrease in speaker similarity after SR-GRPO reflect an actual loss of speaker identity, rather than the sensitivity of speaker similarity metrics to instruction-induced changes in pitch, emotion, and speaking style?
  2. In Figure 1(b) and Appendix A.5, why does Best-of-16 achieve a lower IFR (71.08%) than Best-of-8 (74.31%) despite selecting candidates using the highest CPMI scores? Is this related to reward misalignment or sampling variability?
  3. In the SpeechEditBench evaluation (Table 4), could the authors provide a more detailed breakdown of speaker preservation and speech content fidelity under the adapted re-synthesis protocol, beyond the reported Target Success and Joint Success rates?
  4. Could the authors provide a fully reproducible experiment using publicly available models and training data to further validate the effectiveness of SR-GRPO?
8 comments
AuthorsOpenReview ↗

W1: Insufficient Validation of Self-Reward Signals for Instruction Adherence

We thank the reviewer for raising this important point. We address the three concerns below.

(1) Likelihood vs. attribute realization. We agree that CPMI is a likelihood-based proxy. Therefore, we use it only for ranking and never for evaluation. All results are evaluated with IFR, which is judged by an external evaluator (Gemini-2.5-Pro) that is independent of the policy model. Under this evaluator, IFR decreases monotonically from 71.23% (Rank-0) to 63.08% (Rank-4). CPMI-based Best-of-8 also raises IFR from 67.08% to 74.31%. Higher CPMI thus corresponds to better attribute realization, not merely higher generation probability.

(2) Human judgments. Three raters performed A/B preference tests on 120 category-stratified inputs, yielding 87 pairs with a clear majority preference. CPMI agrees with the human majority on 70.1% of these pairs (95% CI: 59.8–79.3%), with Cohen’s κ=0.404\kappa = 0.404 between CPMI and the majority choice. The signed CPMI difference also correlates with the proportion of raters preferring A (Spearman ρ=0.374\rho = 0.374, CI: [0.180, 0.539]). All intervals exclude chance level.

(3) Mismatched-instruction control. With a semantically opposite instruction as the contrast condition, IFR still decreases monotonically from 70.00% to 61.23%. The Rank-0 to Rank-4 gap (8.77 pp) is comparable to that of the standard setting (8.15 pp). This indicates that the signal is instruction-specific rather than a generic likelihood difference.

Table 8(a) Overall IFR (%) by descending CPMI rank among five SFT candidates per input, under the standard setting and an opposite-instruction control

SubsetRank-0Rank-1Rank-2Rank-3Rank-4
Standard CPMI71.2367.8564.6264.3163.08
Opposite-instruction70.0068.4666.0065.5461.23

Full results are provided in Appendix A.5.

AuthorsOpenReview ↗

W2: Confounding Reward Source and Format in External Reward Comparisons

Thank you for highlighting this confound. We agree that the direct CPMI–Ext-RM Binary comparison cannot attribute the full improvement to self-rewarding alone. To examine reward source and format separately, we added CPMI Binary and Ranked Scalar (mean-split rank) to Table 5, keeping the SFT initialization, training data, optimization steps, decoding settings, and reward weights unchanged.

Table 5. Reward ablations on ACE-Instruct-TTS-Eval-ZH. Bold marks the best value per metric.

Model / ConfigurationIFR ↑CER ↓SIM ↑
ACE-Instruct-TTS-SFT67.082.010.756
ACE-Instruct-TTS-RL
 Ext-RM Binary67.691.930.745
 CPMI Binary71.081.880.738
 Ranked Scalar (ordinal rank)69.381.870.745
 Ranked Scalar (mean-split rank)73.852.220.717
 Conditional-only68.771.690.744
 CPMI75.231.970.733

First, CPMI Binary achieves 71.08% IFR versus 67.69% for Ext-RM Binary, a gain of 3.39 percentage points under a matched binary format. This supports the effectiveness of CPMI-based scoring without continuous feedback.

Second, with the scoring source fixed, continuous CPMI improves over CPMI Binary by 4.15 points. The ranking controls further explain this benefit: mean-split ranking achieves 73.85% IFR, outperforming ordinal ranking (69.38%) and CPMI Binary (71.08%). These results suggest the usefulness of incorporating the group-mean boundary into reward spacing and retaining within-partition ordering. Continuous CPMI further improves IFR to 75.23%, supporting the value of preserving score differences beyond rank-based spacing.

Finally, CPMI exceeds Conditional-only by 6.46 IFR points (75.23% vs. 68.77%). Comparing the same speech’s likelihoods with and without instructions helps distinguish instruction-dependent likelihood gains from generally likely speech, supporting the contribution of the instruction-free baseline.

Overall, Table 5 supports contributions from both CPMI-based scoring and feedback granularity, rather than self-rewarding alone. Detailed results and analysis are provided under “Reward ablations” in Section 5.2 (Results).

AuthorsOpenReview ↗

W3&Q1: Reduced Speaker Similarity: Identity Loss or Sensitivity to Instruction-Induced Changes?

We agree that the results reveal a trade-off. To examine this distinction, we combined automatic metrics with a nine-rater human evaluation of 30 matched pairs, verifying that both evaluations used the same outputs. Mean SMOS decreased from 4.011 to 3.752 (paired difference −0.259; 95% bootstrap CI [−0.513, −0.035]). The paired t-test (p=0.0487p = 0.0487) and bootstrap test (p=0.0330p = 0.0330) supported a mean decrease, whereas Wilcoxon did not reach significance (p=0.1034p = 0.1034). We therefore acknowledge an average perceptual similarity cost, while noting sensitivity to the statistical method.

Nevertheless, automatic and perceptual changes were not identical. Across the 30 pairs, ΔSIM\Delta\mathrm{SIM} and ΔSMOS\Delta\mathrm{SMOS} were positively associated (Spearman ρ=0.476\rho = 0.476; 95% bootstrap CI [0.101, 0.764]), indicating that SIM captures some perceptually relevant variation. Yet a screaming example showed SIM decreasing from 0.513 to 0.307 while SMOS remained at 2.61; a high-pitched example showed SIM decreasing from 0.634 to 0.509 while SMOS changed from 3.00 to 3.11. Conversely, some stylized examples declined substantially on both metrics. These examples illustrate imperfect correspondence, not proof of perceptual equivalence or preserved identity.

Thus, the decrease cannot be attributed entirely to automatic-metric sensitivity, but the current evaluation cannot quantify the respective contributions of identity degradation and expressive variation. Future work will investigate the underlying relationship between instruction following and speaker similarity in TTS, examining how instruction-induced expressive changes affect perceived similarity and speaker-identity preservation, and whether the observed trade-off can be mitigated.

Comments on the paper

1 comment

Summary of Rebuttal and Revisions

We thank all reviewers and the AC for their time and constructive feedback. Reviewers recognized the self-rewarding design that avoids an external instruction evaluator (cWdy, if65), which vgzL also found interesting and reasonable; the comprehensiveness of the experiments (cWdy, vgzL, if65); and the cross-model validation on CosyVoice3 (cWdy, if65). Below we summarize how we addressed the main concerns. All additions are included in the revised manuscript.


1. Validity of the CPMI reward (cWdy-W1, if65-W1/Q1)

  • Human alignment: In A/B tests on 120 category-stratified inputs (87 pairs with a clear majority), CPMI agrees with the human majority on 70.1% of pairs (95% CI: 59.8–79.3%; Cohen's κ = 0.404). The signed CPMI difference correlates with human preference (Spearman ρ = 0.374, 95% CI: [0.180, 0.539]).
  • Mismatched-instruction control: With an opposite instruction as the contrast, IFR still decreases monotonically with CPMI rank (70.00% → 61.23%). The Rank-0 to Rank-4 gap is 8.77 pp, comparable to 8.15 pp under the standard setting. This indicates the signal is instruction-specific. (Appendix A.5)

2. Disentangling reward source and format (cWdy-W2)

  • Same binary format, different source: CPMI Binary outperforms Ext-RM Binary (71.08% vs. 67.69%, +3.39 pp).
  • Same source, different format: Continuous CPMI adds a further +4.15 pp (75.23%).
  • Instruction-free baseline: CPMI outperforms Conditional-only by 6.46 pp.
  • Both scoring source and feedback granularity therefore contribute. (Table 5, Section 5.2)

3. Practicality and complementarity with CFG (vgzL)

  • CFG sweep: We swept eight CFG scales (0.5–4.0) for SFT, SR-GRPO, and SR-GSPO.
  • Matched SIM (0.733): SR-GRPO achieves +4.31 pp IFR with lower CER than CFG-tuned SFT.
  • High-IFR operating point: SR-GRPO (CFG 3.5) surpasses SFT (CFG 4.0) on all three metrics: 80.46% vs. 80.00% IFR, 0.658 vs. 0.652 SIM, and 3.69% vs. 4.48% CER.
  • Complementarity: SR-GRPO achieves higher IFR than SFT at every CFG scale, and increasing CFG from 1.0 to 4.0 adds a further +6.00 pp on top of SR-GRPO (75.23% → 81.23%).
  • Another RL optimizer (SR-GSPO): Replacing GRPO with GSPO under the same self-reward also improves IFR over SFT at CFG 1.0 (73.85% vs. 67.08%) and likewise benefits from CFG (+6.77 pp from CFG 1.0 to 4.0, 73.85% → 80.62%). Complementarity with CFG is therefore not unique to SR-GRPO, and the self-reward remains effective across optimizers. SR-GRPO maintains higher SIM than SR-GSPO at every CFG scale, while SR-GSPO is competitive in IFR and CER at several scales.
  • Inference cost: SR-GRPO adds no inference-time components. (Appendix A.12)

4. Speaker-similarity trade-off (cWdy-W3/Q1, if65-W2/Q2)

  • Acknowledged cost: We explicitly acknowledge this trade-off. Reference-aware SMOS (9 raters) decreases from 4.01 to 3.75 (paired diff. −0.259, 95% CI [−0.513, −0.035]; significant under the paired t-test and bootstrap, but not under Wilcoxon).
  • Overall quality: QMOS shows no detectable difference (3.79 vs. 3.83).
  • Encoder robustness: Three independent WeSpeaker encoders (ResNet34, CAM++, ECAPA) consistently confirm the SIM trend.
  • We have qualified our claims accordingly. (Appendix A.7)

5. Statistical significance (cWdy-W4, if65-W3)

  • IFMOS: +0.256, paired 95% CI [0.022, 0.500]. The t-test, Wilcoxon test, and bootstrap test are all significant at p < 0.05.
  • IFR: Using a paired bootstrap and McNemar tests with Holm correction, SR-GRPO significantly outperforms SFT and three external baselines.
  • Non-significant differences: Differences from CPMI-only Best-of-8 and Qwen-Audio-3.0-TTS-Plus are not significant, and we have revised our claims to reflect this. (Appendices A.9, A.10)

6. Best-of-N comparison (if65-W3/Q3, cWdy-Q2)

  • Combined-reward selection: We added Best-of-N with CPMI+timbre scores. SR-GRPO (75.23%) numerically exceeds both CPMI-only (74.31%) and combined-reward (72.61%) Best-of-8.
  • Inference cost: Assuming comparable output lengths, Best-of-8 requires approximately eight times the candidate-generation work, plus reward scoring.
  • Non-monotonic N=16 result: This is consistent with known reward-overoptimization effects in Best-of-N selection. (Appendix A.5)

7. Additional results and reproducibility (cWdy-Q3/Q4)

  • SIM and CER breakdown on SpeechEditBench. (Appendix A.11)
  • SR-GRPO training code on the public CosyVoice3 model, with 10,000 training examples, provided in the Supplementary Material.

We believe these revisions address the reviewers' main concerns, and we have made our claims more precise where the evidence is limited. We sincerely appreciate the AC's consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.