- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the quality of caption supervision for instruction-following text-to-speech and proposes IterCap, a self-annotating framework. The caption model generates structured speech descriptions, which are filtered through acoustic-consistency checks and cross-model verification. Controlled audio editing supplements underrepresented emotions and delivery styles, and the model undergoes multiple rounds of supervised fine-tuning on the accumulated data. Finally, a fixed set of 3,000 multiple-choice questions constructed from 200 human reference captions provides a GSPO objective for improving attribute-level judgments.
The authors evaluate caption accuracy on 325 human-annotated audio clips and use the final model to re-annotate 300K TTS training examples. With the downstream Qwen3-TTS model, audio-text data, and training settings held fixed, IterCap captions improve average InstructTTSEval accuracy from 84.5 to 88.3.
Critical strengths
1. The caption-source comparison provides relatively direct evidence of downstream utility.
Table 2 holds Qwen3-TTS, the 300K audio-text corpus, and the training settings fixed while varying only the caption source. IterCap improves the average score from 84.5 with the native captions to 88.3, with gains in all six evaluation columns. It also improves over captions generated by the same Qwen3-Omni backbone in the zero-shot setting, from 82.9 to 88.3. These results support the practical value of improving caption supervision for downstream instruction following, beyond improvements in caption-evaluation scores alone.
2. The framework addresses both annotation reliability and coverage, with a well-motivated overall design.
The two-stage verifier aims to reduce errors in self-generated labels, while targeted audio editing supplements delivery combinations missing from natural corpora. The fixed-answer MCQ objective provides an additional attribute-level learning signal. The paper explicitly separates the 200 exam references, 100 development clips, and 325 test clips, and reports both fluctuations across rounds and regressions in some attributes after GSPO. These details make the overall results easier to interpret.
Critical weaknesses
1. The target fields introduced during expansion lack independent validation of their reliability.
Section 3.3 and Appendix F explain that the target field of an edited clip is overwritten with its prescribed value and exempted from both verification stages. Requesting an "angry" rendition, for example, does not establish that the resulting audio expresses anger, nor does it guarantee preservation of the transcript and speaker identity.
The appendix reports that over 95% of accepted annotations are performance-consistent, but does not specify how this check covers edited clips and their target fields, or separately report edit success and content/speaker preservation. The available evidence therefore does not establish the reliability of this new supervision source, even though it is central to the framework's ability to introduce missing capabilities.
2. The independent benefits of multiple iterations and GSPO are not sufficiently isolated.
In Table 1, the first round improves caption accuracy from 69.6 to 80.5, while the subsequent four rounds together raise it to 82.9. The authors acknowledge that the first round combines selection and expansion, and the acceptance threshold is lowered in rounds 4 and 5. The per-round scores therefore do not isolate the benefit of iteration itself. A budget-matched single-round versus multi-round comparison is missing, and the caption-model training sources, per-round accepted sample counts, and proportions of constructed data are not sufficiently specified, limiting attribution and reproducibility.
GSPO adds 0.5 percentage points to the average while reducing fluency accuracy from 93.3 to 87.8. Its contribution is not separately evaluated in downstream TTS, and no uncertainty estimates are reported. The results therefore support the combined pipeline's utility more clearly than they establish how much multiple iterations and RL each contribute.
Questions for authors
1. How reliable are the target labels assigned to edited audio?
Please clarify whether the human consistency result of over 95% in Appendix F covers edited clips and their directly assigned target fields, and report the relevant sample sizes. Independent checks of whether the target emotion or style is realized, and whether the transcript and speaker identity are preserved, would help establish the reliability of the expansion supervision.
2. How much do multiple iterations and GSPO each contribute?
Please specify the caption-model training sources, the numbers of accepted and constructed examples in each round, and the threshold changes. A budget-matched single-round versus multi-round comparison, together with a downstream comparison of iter5 and iter5+GSPO captions under fixed TTS training settings, would help isolate these stages' contributions. Repeated-run results or uncertainty estimates would also help assess the stability of the smaller gains.
3 comments
4. Uncertainty estimates and attribute-level changes
We performed a paired bootstrap analysis on the same 325 test clips. In each resample, we drew 325 clips with replacement and used identical resampling indices for both models. We repeated this procedure 10,000 times and report the 95% confidence interval of the macro-average difference.
| Comparison | Difference (latter − former, percentage points) | Paired bootstrap 95% CI (percentage points) |
|---|---|---|
| IterCap (iter1) vs. IterCap (iter5) | +2.38 | [+1.90, +2.90] |
| IterCap (iter5) vs. IterCap (iter5+GSPO) | +0.46 | [−0.03, +0.95] |
The overall improvement from iter1 to iter5 remains positive under test-set resampling. The macro-average improvement from GSPO is smaller, and its confidence interval includes zero. This analysis characterizes test-set resampling uncertainty rather than variation across repeated training runs. The confidence interval for +2.38 percentage points corresponds to iter1 versus iter5, not the matched single-round baseline versus iter5; the downstream improvement of 1.0 percentage point is also not covered by these caption-level confidence intervals.
Regarding the fluency regression, our inspection revealed limited diversity in some fields of the GSPO training data: approximately 80% of the human fluency annotations are fluent, whereas other fields, particularly emotion and style, are considerably more diverse. This imbalance may be one of the main contributors to the fluency regression; we present it as a possible explanation based on data inspection.
Importantly, GSPO's net macro-average change of +0.46 percentage points reflects offsetting improvements and regressions, rather than uniformly small changes in every attribute. All changes and paired counts below compare IterCap (iter5) with IterCap (iter5+GSPO).
| Attribute | Change (percentage points) | Correct → incorrect | Incorrect → correct | Exact McNemar p |
|---|---|---|---|---|
| Emotion | +1.85 | 1 | 7 | 0.0703 |
| Timbre | +1.85 | 2 | 8 | 0.1094 |
| Fluency | −5.54 | 21 | 3 | 0.00028 |
We therefore view GSPO as an attribute-level calibration step: accuracy improves for some controllable attributes, alongside a clear fluency trade-off. The positive changes in emotion and timbre do not reach statistical significance in these tests, and the macro-average confidence interval also includes zero. Meanwhile, the 1.0-percentage-point improvement in downstream TTS under fixed settings provides complementary evidence of this stage's practical utility.
Regarding the substantial fluency drop observed after applying GSPO in Table~1, we conducted an additional experiment. We rebalanced the GSPO training-data distribution and reran the GSPO experiment. Under the rebalanced distribution, the severe fluency degradation disappeared and the overall metrics showed an upward trend, with only modest decreases in a small number of dimensions. This result further supports the data-distribution analysis presented in our rebuttal.
| Model | Gen | Age | Pit | Cla | Acn | Tim | Emo | Rat | Vol | Flu | Int | Cha | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IterCap (iter5) | 99.1 | 72.9 | 71.1 | 97.6 | 98.6 | 70.6 | 80.4 | 79.4 | 84.5 | 93.3 | 69.0 | 78.6 | 82.9 |
| IterCap (iter5+GSPO) | 99.4 | 73.8 | 73.6 | 96.3 | 97.2 | 72.4 | 82.2 | 84.1 | 85.7 | 87.8 | 69.3 | 78.4 | 83.4 |
| IterCap (iter5+GSPO, balanced) | 99.1 | 73.5 | 71.7 | 97.3 | 98.6 | 71.7 | 82.2 | 82.4 | 85.9 | 93.5 | 69.0 | 79.1 | 83.7 |
We also conducted a small-scale human evaluation to assess the consistency of the Gemini-based evaluation. The original InstructTTSEval experiments already included a human-consistency evaluation. For the caption-quality evaluation, we randomly sampled 100 annotations from the data used in the corresponding Table1 experiment and manually judged field-level consistency. We then cross-checked these judgments against the evaluations produced by Gemini-3-flash-preview. Human judgments and Gemini-3-flash-preview agreed on 89 out of 100 samples. This agreement rate provides supporting evidence for the reliability of the metrics reported in Table1.
Response to Reviewer fVxg
Thank you for your specific questions about the reliability of editing-based supervision, the composition of the training data across rounds, and the respective contributions of iteration and GSPO. We provide an independent human evaluation of edited samples, additional data-composition details, a budget-matched comparison, and uncertainty estimates below.
1. Reliability of target labels for edited clips
First, the human agreement rate of over 95% reported in the paper does not include edited clips.
We assess the reliability of expansion supervision through both the editing model's benchmark performance and the edited samples actually used for training. First, we evaluated the editing model separately on DOTE-Bench and reproduced metrics broadly consistent with those reported in the reference paper.
In addition, we sampled 60 examples from the edit-expansion portion of the actual training data for a subjective evaluation by one human evaluator. The evaluation separately assessed whether the target emotion/style was realized, whether the content was preserved, and whether the speaker identity was preserved. Scores were 0 (inconsistent), 1 (uncertain), and 2 (consistent).
This audit and the over-95% agreement result for accepted annotations are two separate evaluations. The edited-sample results provide human evidence for target-attribute realization, content preservation, and speaker preservation, supplementing our validation of the reliability of the expansion data.
2. Caption-model training data and round-wise thresholds
The caption-model training data are sampled from a 20,000-hour subset of our internally constructed dataset. In rounds involving constructed examples, the number of constructed examples equals the number of accepted examples. Accepted data in iter1 account for approximately 10% of the total training data; iter2 adds approximately 6%, and the newly added proportion remains around 3% per round in iter3–5.
As described in Appendix F, the cross-model agreement thresholds are as follows.
3. Respective contributions of iteration and GSPO
We have added a matched single-round baseline. We keep the final iter5 experimental configuration, the amounts of selected and expanded data, and the training compute budget fixed, but generate and filter all selection and expansion data using the iter1 rules, without using caption models updated in later rounds. Specifically, the two settings use the same number of GPUs, the same amount of training data, and the same number of training steps.
Under matched final data size and training budgets, the full iterative framework outperforms the single-round baseline by 3.9 percentage points. Its benefit therefore cannot be explained simply by a larger final training set. Updated caption models provide better annotation capabilities for subsequent rounds, supporting the design of alternating selection and expansion to improve supervision.
Following your suggestion, we have also added a downstream comparison isolating GSPO under fixed TTS training settings. The TTS backbone and the 300K audio-text corpus remain unchanged, with only the caption source varied.
GSPO improves the average downstream score by 1.0 percentage point. The following uncertainty analysis further distinguishes its net caption-level change from attribute-specific gains and regressions.