acceptodds
Under review as a conference paper at ICLR 2027

IterCap: A Self-Annotating Captioning Framework for Instruction-Following Text-to-Speech

Abstract

Instruction-following text-to-speech (instruct-TTS) learns acoustic controllability from captions, making their quality and coverage fundamental to what the model can learn: an attribute that a caption fails to specify or ground receives little supervision, regardless of the synthesizer's capacity. Existing methods either take captions as given, whether human-written, generated zero-shot by an external audio-language model, or produced by a handcrafted pipeline, or train a caption model on such captions once through supervised fine-tuning or a single round of reinforcement learning. In all cases, the supervision originatand is fixed before the model trains on it. We instead ask whether a caption model can annotate its own supervision, formulating instruct-TTS captioning as a self-ann an asymmetry: verifying the attributes asserted by a caption is easier than generating a correct open-ended caption. IterCap closes this loop through three opeenerated captions with lightweight acoustic-consistency checks and independent caption models; expansion adds examples of underrepresented delivery conditions; and optl correctness into a reference-anchored, verifiable objective through a multiple-choice exam and Group Sequence Policy Optimization (GSPO). On a 325-clip human12 controllable dimensions, IterCap raises macro-averaged per-dimension caption accuracy to 83.4%, compared with 75.3% for the strongest open-source caption modesfers to the downstream system: with the 300K-sample corpus and Qwen3-TTS backbone held fixed, IterCap captions raise instruction-following accuracy to 88.3%, compars's native captions and 82.9% using the zero-shot caption model from which IterCap is initialized.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
TTS, audio caption, audio understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fVxgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the quality of caption supervision for instruction-following text-to-speech and proposes IterCap, a self-annotating framework. The caption model generates structured speech descriptions, which are filtered through acoustic-consistency checks and cross-model verification. Controlled audio editing supplements underrepresented emotions and delivery styles, and the model undergoes multiple rounds of supervised fine-tuning on the accumulated data. Finally, a fixed set of 3,000 multiple-choice questions constructed from 200 human reference captions provides a GSPO objective for improving attribute-level judgments.

The authors evaluate caption accuracy on 325 human-annotated audio clips and use the final model to re-annotate 300K TTS training examples. With the downstream Qwen3-TTS model, audio-text data, and training settings held fixed, IterCap captions improve average InstructTTSEval accuracy from 84.5 to 88.3.

Critical strengths

1. The caption-source comparison provides relatively direct evidence of downstream utility.

Table 2 holds Qwen3-TTS, the 300K audio-text corpus, and the training settings fixed while varying only the caption source. IterCap improves the average score from 84.5 with the native captions to 88.3, with gains in all six evaluation columns. It also improves over captions generated by the same Qwen3-Omni backbone in the zero-shot setting, from 82.9 to 88.3. These results support the practical value of improving caption supervision for downstream instruction following, beyond improvements in caption-evaluation scores alone.

2. The framework addresses both annotation reliability and coverage, with a well-motivated overall design.

The two-stage verifier aims to reduce errors in self-generated labels, while targeted audio editing supplements delivery combinations missing from natural corpora. The fixed-answer MCQ objective provides an additional attribute-level learning signal. The paper explicitly separates the 200 exam references, 100 development clips, and 325 test clips, and reports both fluctuations across rounds and regressions in some attributes after GSPO. These details make the overall results easier to interpret.

Critical weaknesses

1. The target fields introduced during expansion lack independent validation of their reliability.

Section 3.3 and Appendix F explain that the target field of an edited clip is overwritten with its prescribed value and exempted from both verification stages. Requesting an "angry" rendition, for example, does not establish that the resulting audio expresses anger, nor does it guarantee preservation of the transcript and speaker identity.

The appendix reports that over 95% of accepted annotations are performance-consistent, but does not specify how this check covers edited clips and their target fields, or separately report edit success and content/speaker preservation. The available evidence therefore does not establish the reliability of this new supervision source, even though it is central to the framework's ability to introduce missing capabilities.

2. The independent benefits of multiple iterations and GSPO are not sufficiently isolated.

In Table 1, the first round improves caption accuracy from 69.6 to 80.5, while the subsequent four rounds together raise it to 82.9. The authors acknowledge that the first round combines selection and expansion, and the acceptance threshold is lowered in rounds 4 and 5. The per-round scores therefore do not isolate the benefit of iteration itself. A budget-matched single-round versus multi-round comparison is missing, and the caption-model training sources, per-round accepted sample counts, and proportions of constructed data are not sufficiently specified, limiting attribution and reproducibility.

GSPO adds 0.5 percentage points to the average while reducing fluency accuracy from 93.3 to 87.8. Its contribution is not separately evaluated in downstream TTS, and no uncertainty estimates are reported. The results therefore support the combined pipeline's utility more clearly than they establish how much multiple iterations and RL each contribute.

Questions for authors

1. How reliable are the target labels assigned to edited audio?

Please clarify whether the human consistency result of over 95% in Appendix F covers edited clips and their directly assigned target fields, and report the relevant sample sizes. Independent checks of whether the target emotion or style is realized, and whether the transcript and speaker identity are preserved, would help establish the reliability of the expansion supervision.

2. How much do multiple iterations and GSPO each contribute?

Please specify the caption-model training sources, the numbers of accepted and constructed examples in each round, and the threshold changes. A budget-matched single-round versus multi-round comparison, together with a downstream comparison of iter5 and iter5+GSPO captions under fixed TTS training settings, would help isolate these stages' contributions. Repeated-run results or uncertainty estimates would also help assess the stability of the smaller gains.

3 comments

Response to Reviewer fVxg

Thank you for your specific questions about the reliability of editing-based supervision, the composition of the training data across rounds, and the respective contributions of iteration and GSPO. We provide an independent human evaluation of edited samples, additional data-composition details, a budget-matched comparison, and uncertainty estimates below.

1. Reliability of target labels for edited clips

First, the human agreement rate of over 95% reported in the paper does not include edited clips.

We assess the reliability of expansion supervision through both the editing model's benchmark performance and the edited samples actually used for training. First, we evaluated the editing model separately on DOTE-Bench and reproduced metrics broadly consistent with those reported in the reference paper.

In addition, we sampled 60 examples from the edit-expansion portion of the actual training data for a subjective evaluation by one human evaluator. The evaluation separately assessed whether the target emotion/style was realized, whether the content was preserved, and whether the speaker identity was preserved. Scores were 0 (inconsistent), 1 (uncertain), and 2 (consistent).

Evaluation dimensionMean score (0–2)
Emotion/style consistency1.92
Content preservation2.00
Speaker preservation1.83

This audit and the over-95% agreement result for accepted annotations are two separate evaluations. The edited-sample results provide human evidence for target-attribute realization, content preservation, and speaker preservation, supplementing our validation of the reliability of the expansion data.

2. Caption-model training data and round-wise thresholds

The caption-model training data are sampled from a 20,000-hour subset of our internally constructed dataset. In rounds involving constructed examples, the number of constructed examples equals the number of accepted examples. Accepted data in iter1 account for approximately 10% of the total training data; iter2 adds approximately 6%, and the newly added proportion remains around 3% per round in iter3–5.

As described in Appendix F, the cross-model agreement thresholds are as follows.

RoundsCross-model agreement acceptance criterion
Rounds 1–3Scores from all verifier comparisons must be ≥4
Rounds 4–5Scores from all verifier comparisons must be ≥3

3. Respective contributions of iteration and GSPO

We have added a matched single-round baseline. We keep the final iter5 experimental configuration, the amounts of selected and expanded data, and the training compute budget fixed, but generate and filter all selection and expansion data using the iter1 rules, without using caption models updated in later rounds. Specifically, the two settings use the same number of GPUs, the same amount of training data, and the same number of training steps.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
Qwen3-Omni (iter0)98.868.955.796.990.247.457.866.873.575.756.646.569.6
Single-round98.869.864.697.594.865.280.078.276.988.061.572.679.0
IterCap (iter5)99.172.971.197.698.670.680.479.484.593.369.078.682.9

Under matched final data size and training budgets, the full iterative framework outperforms the single-round baseline by 3.9 percentage points. Its benefit therefore cannot be explained simply by a larger final training set. Updated caption models provide better annotation capabilities for subsequent rounds, supporting the design of alternating selection and expansion to improve supervision.

Following your suggestion, we have also added a downstream comparison isolating GSPO under fixed TTS training settings. The TTS backbone and the 300K audio-text corpus remain unchanged, with only the caption source varied.

Caption sourceZH APSZH DSDZH RPEN APSEN DSDEN RPAvg
IterCap (w/o GSPO)95.295.072.294.893.673.187.3
IterCap (ours)95.495.574.296.193.574.988.3

GSPO improves the average downstream score by 1.0 percentage point. The following uncertainty analysis further distinguishes its net caption-level change from attribute-specific gains and regressions.

4. Uncertainty estimates and attribute-level changes

We performed a paired bootstrap analysis on the same 325 test clips. In each resample, we drew 325 clips with replacement and used identical resampling indices for both models. We repeated this procedure 10,000 times and report the 95% confidence interval of the macro-average difference.

ComparisonDifference (latter − former, percentage points)Paired bootstrap 95% CI (percentage points)
IterCap (iter1) vs. IterCap (iter5)+2.38[+1.90, +2.90]
IterCap (iter5) vs. IterCap (iter5+GSPO)+0.46[−0.03, +0.95]

The overall improvement from iter1 to iter5 remains positive under test-set resampling. The macro-average improvement from GSPO is smaller, and its confidence interval includes zero. This analysis characterizes test-set resampling uncertainty rather than variation across repeated training runs. The confidence interval for +2.38 percentage points corresponds to iter1 versus iter5, not the matched single-round baseline versus iter5; the downstream improvement of 1.0 percentage point is also not covered by these caption-level confidence intervals.

Regarding the fluency regression, our inspection revealed limited diversity in some fields of the GSPO training data: approximately 80% of the human fluency annotations are fluent, whereas other fields, particularly emotion and style, are considerably more diverse. This imbalance may be one of the main contributors to the fluency regression; we present it as a possible explanation based on data inspection.

Importantly, GSPO's net macro-average change of +0.46 percentage points reflects offsetting improvements and regressions, rather than uniformly small changes in every attribute. All changes and paired counts below compare IterCap (iter5) with IterCap (iter5+GSPO).

AttributeChange (percentage points)Correct → incorrectIncorrect → correctExact McNemar p
Emotion+1.85170.0703
Timbre+1.85280.1094
Fluency−5.542130.00028

We therefore view GSPO as an attribute-level calibration step: accuracy improves for some controllable attributes, alongside a clear fluency trade-off. The positive changes in emotion and timbre do not reach statistical significance in these tests, and the macro-average confidence interval also includes zero. Meanwhile, the 1.0-percentage-point improvement in downstream TTS under fixed settings provides complementary evidence of this stage's practical utility.

Regarding the substantial fluency drop observed after applying GSPO in Table~1, we conducted an additional experiment. We rebalanced the GSPO training-data distribution and reran the GSPO experiment. Under the rebalanced distribution, the severe fluency degradation disappeared and the overall metrics showed an upward trend, with only modest decreases in a small number of dimensions. This result further supports the data-distribution analysis presented in our rebuttal.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
IterCap (iter5)99.172.971.197.698.670.680.479.484.593.369.078.682.9
IterCap (iter5+GSPO)99.473.873.696.397.272.482.284.185.787.869.378.483.4
IterCap (iter5+GSPO, balanced)99.173.571.797.398.671.782.282.485.993.569.079.183.7

We also conducted a small-scale human evaluation to assess the consistency of the Gemini-based evaluation. The original InstructTTSEval experiments already included a human-consistency evaluation. For the caption-quality evaluation, we randomly sampled 100 annotations from the data used in the corresponding Table1 experiment and manually judged field-level consistency. We then cross-checked these judgments against the evaluations produced by Gemini-3-flash-preview. Human judgments and Gemini-3-flash-preview agreed on 89 out of 100 samples. This agreement rate provides supporting evidence for the reliability of the metrics reported in Table1.

Reviewer_U5z3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes IterCap, a framework for improving speech captions used to train instruction-following TTS systems. It combines iterative self-training with acoustic and cross-model consistency filtering, controlled audio editing to expand attribute coverage, and multiple-choice-based GSPO training. The resulting caption model re-annotates a 300K-sample corpus for training Qwen3-TTS. The paper reports improvements in both caption accuracy and downstream instruction following.

Critical strengths
  1. The paper addresses a practically relevant problem: inaccurate or incomplete descriptions of speech attributes can limit the controllability of instruction-following TTS systems.

  2. The controlled caption-source comparison is valuable. With the downstream architecture, audio-text pairs, and training protocol held fixed, replacing the original captions with IterCap captions improves instruction-following accuracy from 84.5% to 88.3%. This provides evidence that the resulting annotations are useful.

Critical weaknesses
  1. The methodological novelty is limited. The framework combines established techniques: rejection-based self-training, cross-model consistency filtering, controlled audio editing, and multiple-choice-based reinforcement learning. The proposed workflow is a relatively straightforward combination of data filtering, augmentation, and retraining. Although this combination is practically useful, the paper does not sufficiently establish a new methodological insight beyond adapting these techniques to speech captioning.

  2. The experiments do not sufficiently isolate the contribution of the proposed design. Caption accuracy improves from 69.6% to 80.5% in the first round, which already combines selection and expansion, while four additional rounds increase it to 82.9%. Without a comparison under matched data and compute budgets, it is difficult to distinguish the benefit of iteration from that of accumulating more training data. The paper also lacks controlled ablations separating acoustic filtering, cross-model verification, and audio expansion. Finally, GSPO improves average caption accuracy by only 0.5 percentage points, with regressions in some dimensions, and its downstream TTS contribution is not separately evaluated.

Questions for authors
  1. How does the full iterative framework compare with a single-round filtering and fine-tuning baseline under matched training-data and compute budgets? Does iteration provide gains beyond those obtained from a larger fixed training set?

  2. What are the individual contributions of acoustic filtering, cross-model verification, and audio expansion? In particular, how much of the improvement remains when controlled audio editing is removed?

3 comments

Response to Reviewer U5z3

Thank you for recognizing the practical relevance of the problem and the value of the controlled downstream comparison. We address your concerns about the contribution, the benefit of iteration, and the roles of individual components through a matched single-round comparison and component ablations.

1. Positioning of the contribution

Our contribution is not a new standalone filtering or RL algorithm. Rather, it is a framework for instruct-TTS caption supervision that jointly addresses label reliability and limited capability coverage, while iteratively updating the caption model to produce supervision for subsequent rounds. The additional experiments evaluate the value of this integrated design rather than presenting the existing components themselves as new algorithms.

2. Does iteration help under matched data and training budgets?

To examine whether the benefit of iteration comes merely from a larger final training set, we constructed a matched single-round baseline. We keep the final experimental configuration and the amounts of selected and expanded data the same as for iter5. All selection and expansion data in the single-round baseline are generated and filtered using the original iter1 rules, without using caption models updated in later rounds. The two settings use the same number of GPUs, the same amount of training data, and the same number of training steps, with the other training parameters also held fixed.

The main difference is therefore not the final training-set size, but whether the supervision is produced through subsequent iterations with updated caption models. The results are as follows.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
Qwen3-Omni (iter0)98.868.955.796.990.247.457.866.873.575.756.646.569.6
Single-round98.869.864.697.594.865.280.078.276.988.061.572.679.0
IterCap (iter5)99.172.971.197.698.670.680.479.484.593.369.078.682.9

Under matched final data size and training budgets, the single-round baseline achieves 79.0% accuracy, compared with 82.9% for iter5, a difference of 3.9 percentage points. This shows that the benefit of iter5 cannot be explained simply by an increase in training-set size. It also supports the design of alternating selection and expansion: an improved caption model can provide better annotations in subsequent rounds, thereby improving the supervision.

3. Roles of acoustic filtering and cross-model verification

Following the original iter1 experimental setup, we replace each filtering module with random selection to examine its role in first-round selection. The results are shown below.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
w/o cross-model verification (iter1)98.868.967.195.792.059.772.678.278.586.562.270.877.6
w/o acoustic filtering (iter1)98.868.965.295.193.261.275.171.776.388.067.172.677.8
IterCap (iter1)98.870.567.497.794.464.980.280.178.990.165.976.880.5

Acoustic filtering performs coarse screening based on the broad acoustic characteristics of several fields; it cannot finely verify specific field values. Replacing it with random selection affects multiple dimensions. Given the same number of candidate samples for filtering, the proportion of high-quality samples decreases, and cross-model verification consequently retains fewer valid samples. This reduces the amount of valid training data and leads to lower overall performance than the full iter1 model.

Without random selection to control the candidate volume, directly removing acoustic filtering would increase the cost of cross-model verification by approximately fourfold, substantially increasing the computational burden. Acoustic filtering thus serves both as a low-cost pre-screen and as a means of improving the quality of candidates passed to subsequent verification.

Cross-model verification is a key component of selection: it determines which audio clips are retained and added to the high-quality training set. Removing it has relatively little effect on fields such as pitch and volume, which can be screened using coarse acoustic checks, but produces larger declines in several other dimensions, particularly timbre, emotion, and character. This is consistent with its role in verifying fine-grained attribute descriptions.

4. Role of audio expansion through controlled editing

We have added three-round experiments removing either expansion through controlled audio editing or selection. The table reports per-attribute accuracy and its macro average (%).

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
Qwen3-Omni (iter0)98.868.955.796.990.247.457.866.873.575.756.646.569.6
w/o Expansion (iter1)98.870.266.596.692.651.766.872.982.876.658.851.173.8
w/o Expansion (iter2)98.869.267.796.089.549.261.574.281.877.257.848.972.7
w/o Expansion (iter3)98.567.767.196.986.247.763.472.983.175.755.746.571.8
w/o Selection (iter1)98.870.557.296.095.165.271.766.575.488.066.571.776.9
w/o Selection (iter2)98.870.859.195.495.766.271.167.470.889.264.374.276.9
w/o Selection (iter3)98.869.556.096.395.465.872.064.668.389.564.972.676.2

Expansion is one of the main sources of the first-round capability improvement: removing it reduces iter1 accuracy from 80.5% to 73.8%. However, expansion alone, represented by w/o Selection, does not yield sustained improvement over the three tested rounds. Under the current setting and tested number of rounds, expanding coverage alone is therefore insufficient to match the full framework. These results support the complementary roles of expansion in introducing capability coverage and selection in ensuring supervision reliability.

5. Separate downstream contribution of GSPO

To isolate GSPO's downstream contribution, we hold the TTS backbone, the 300K audio-text corpus, and the training settings fixed, changing only the caption source.

Caption sourceZH APSZH DSDZH RPEN APSEN DSDEN RPAvg
IterCap (w/o GSPO)95.295.072.294.893.673.187.3
IterCap (ours)95.495.574.296.193.574.988.3

Adding GSPO improves the average downstream score by 1.0 percentage point. At the caption level, 10,000 paired bootstrap resamples on the same 325 test clips yield a macro-average difference of +0.46 percentage points, with a 95% confidence interval of [−0.03, +0.95]. This net change includes offsetting attribute-level improvements and regressions.

AttributeChange from iter5 to iter5+GSPO (percentage points)Correct → incorrectIncorrect → correctExact McNemar p
Emotion+1.85170.0703
Timbre+1.85280.1094
Fluency−5.542130.00028

We view GSPO as an attribute-level calibration step: it improves the accuracy of some controllable attributes and downstream performance, while introducing a clear fluency trade-off. Our inspection shows that approximately 80% of the samples are labeled as fluent, whereas other fields, particularly emotion and style, are considerably more diverse. This imbalance may be one of the main contributors to the fluency regression; we present it as a possible explanation based on data inspection.

Regarding the substantial fluency drop observed after applying GSPO in Table~1, we conducted an additional experiment. We rebalanced the GSPO training-data distribution and reran the GSPO experiment. Under the rebalanced distribution, the severe fluency degradation disappeared and the overall metrics showed an upward trend, with only modest decreases in a small number of dimensions. This result further supports the data-distribution analysis presented in our rebuttal.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
IterCap (iter5)99.172.971.197.698.670.680.479.484.593.369.078.682.9
IterCap (iter5+GSPO)99.473.873.696.397.272.482.284.185.787.869.378.483.4
IterCap (iter5+GSPO, balanced)99.173.571.797.398.671.782.282.485.993.569.079.183.7
Reviewer_PuYqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IterCap, a self-annotating framework for improving speech captions used to train instruction-following TTS models. It combines caption filtering through acoustic and cross-model verification, controlled audio editing to expand coverage, and MCQ-based GSPO for improving attribute-level accuracy. Experiments show that IterCap improves caption accuracy from 69.6% to 83.4% and downstream instruction-following accuracy from 84.5% to 88.3% using the same TTS backbone and training corpus.

Critical strengths
  1. The paper addresses an important problem. The quality of speech descriptions is often overlooked, despite its direct impact on the controllability of TTS models.

  2. The overall framework is well motivated. The three components target different limitations of self-training, and combining caption verification with controlled audio editing is a reasonable approach.

  3. The downstream results are encouraging. Keeping the TTS backbone and training setup fixed provides useful evidence that better captions can improve instruction following.

Critical weaknesses
  1. The contribution of each component is not sufficiently clear. Most of the caption accuracy improvement comes from iterative training, while GSPO adds only 0.5 percentage points. More ablations are needed to understand the individual contributions of selection, expansion, and optimization.

  2. The verification pipeline may still reinforce shared model biases. Agreement between multiple models does not necessarily mean a caption is correct, especially for subjective attributes such as emotion and timbre. A human evaluation of the accepted and rejected captions would help establish the reliability of the filtering process.

  3. The GSPO results are somewhat mixed. Although the overall accuracy improves, fluency drops from 93.3% to 87.8%. It would be helpful to understand this trade-off and whether the gains transfer to open-ended caption generation.

  4. The evaluation relies heavily on automatic judges. A small-scale human evaluation would make the claims about caption quality and downstream improvements more convincing.

Questions for authors
  1. Could the authors provide ablations to better isolate the contributions of selection, expansion, and GSPO?

  2. How well do the two verification stages agree with human judgments, particularly for emotion and timbre?

  3. Why does GSPO lead to a substantial drop in fluency, and do its gains generalize to open-ended caption generation?

4 comments

Response to Reviewer PuYq

Thank you for recognizing the motivation of our work, the overall framework, and the value of the controlled downstream comparison. We address your concerns about component contributions, verification reliability, and the fluency trade-off introduced by GSPO with the following additional experiments and clarifications.

1. Contributions of selection, expansion, and GSPO

We have added ablations of selection and expansion, together with a separate downstream comparison isolating GSPO. The table below reports per-attribute accuracy and its macro average (%).

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
Qwen3-Omni (iter0)98.868.955.796.990.247.457.866.873.575.756.646.569.6
w/o Expansion (iter1)98.870.266.596.692.651.766.872.982.876.658.851.173.8
w/o Expansion (iter2)98.869.267.796.089.549.261.574.281.877.257.848.972.7
w/o Expansion (iter3)98.567.767.196.986.247.763.472.983.175.755.746.571.8
w/o Selection (iter1)98.870.557.296.095.165.271.766.575.488.066.571.776.9
w/o Selection (iter2)98.870.859.195.495.766.271.167.470.889.264.374.276.9
w/o Selection (iter3)98.869.556.096.395.465.872.064.668.389.564.972.676.2

Both ablations underperform the full IterCap model, which achieves 80.5% in the first round. Expansion is an important source of the initial improvement, but expansion alone does not yield further gains over the subsequent rounds tested. With selection alone, reliability filtering improves performance in the first round, followed by a decline over later iterations. One possible explanation is the accumulation of shared model biases during self-training. Combining the two yields better overall performance, supporting our design in which label reliability and capability coverage complement each other.

The separate downstream comparison for GSPO is provided in Section 3 of this response.

2. Agreement between the two verification stages and human judgments

Using the iter1 filtering configuration discussed in the appendix F, we manually audited 100 samples drawn from the accepted annotations. With the strict thresholds used by the coarse filter, more than 95% of the audited samples agreed with human judgments on the dimensions checked by that filter.

This design prioritizes the reliability of accepted labels rather than maximizing retention. Accordingly, this conservative strategy allows some usable samples to be discarded, and the rejected set may still contain many usable samples. This is consistent with our design principle: selection strengthens capabilities already present in the model rather than relying on uncertain annotations beyond its current capabilities. This audit of the accepted set is not a quantitative estimate of the false-rejection rate.

We additionally audited 100 audio clips sampled from the annotations accepted by cross-model verification under the iter1 data-construction configuration. Human agreement with and without multi-model verification is reported below.

AttributeHuman agreement without multi-model verificationHuman agreement with multi-model verification
Timbre48%72%
Emotion56%88%

These results support the practical filtering value of multi-model verification for subjective attributes such as timbre and emotion. We do not equate model agreement with correctness; instead, the human audits assess the reliability of the retained labels.

3. GSPO: fluency trade-off, downstream utility, and transfer to open-ended generation

We agree that the decline in fluency is an important trade-off. GSPO uses MCQ rewards constructed from human reference captions. Our inspection of these data revealed substantially less diversity in fluency than in the other fields: approximately 80% of the samples are labeled as fluent, whereas other fields, particularly emotion and style, are considerably more diverse. This imbalance may be one of the main contributors to the fluency regression; we present it as a possible explanation based on data inspection.

We have also added a downstream comparison isolating GSPO. The TTS backbone, the 300K audio-text corpus, and the training settings are held fixed, with only the caption source changed.

Caption sourceZH APSZH DSDZH RPEN APSEN DSDEN RPAvg
IterCap (w/o GSPO)95.295.072.294.893.673.187.3
IterCap (ours)95.495.574.296.193.574.988.3

Adding GSPO improves the average downstream score by approximately 1.0 percentage point. This is consistent with the improvements in controllable attributes such as timbre and emotion.

To make the caption-level gains and regressions explicit, we report attribute-level changes and paired correctness counts on the same 325 test clips. All comparisons below are from iter5 to iter5+GSPO.

AttributeChange (percentage points)Correct → incorrectIncorrect → correctExact McNemar p
Emotion+1.85170.0703
Timbre+1.85280.1094
Fluency−5.542130.00028

The net macro-average change of +0.46 percentage points reflects offsetting improvements and regressions across attributes. We performed 10,000 paired bootstrap resamples at the audio-clip level, drawing 325 clips with replacement in each resample and using identical resampling indices for both models. The resulting 95% confidence interval is [−0.03, +0.95] percentage points. We therefore view GSPO as an attribute-level calibration step with a clear fluency trade-off; the downstream improvement of 1.0 percentage point under fixed training settings demonstrates its practical task utility.

Regarding open-ended generation, the statement in §4.2 that “each caption model is queried separately for every dimension” is imprecise and may suggest separate generation for each dimension. In the actual evaluation, the model generates a complete caption from the audio, and an LLM judge then evaluates each of the twelve dimensions against the human reference annotation. The evaluation does not measure MCQ answer accuracy. We will correct this description in the revised manuscript. Thus, Table 1 already evaluates the transfer of MCQ-based GSPO to open-ended caption generation, with the attribute-level gains and fluency trade-off reported above.

Regarding the substantial fluency drop observed after applying GSPO in Table~1, we conducted an additional experiment. We rebalanced the GSPO training-data distribution and reran the GSPO experiment. Under the rebalanced distribution, the severe fluency degradation disappeared and the overall metrics showed an upward trend, with only modest decreases in a small number of dimensions. This result further supports the data-distribution analysis presented in our rebuttal.

ModelGenAgePitClaAcnTimEmoRatVolFluIntChaAvg
IterCap (iter5)99.172.971.197.698.670.680.479.484.593.369.078.682.9
IterCap (iter5+GSPO)99.473.873.696.397.272.482.284.185.787.869.378.483.4
IterCap (iter5+GSPO, balanced)99.173.571.797.398.671.782.282.485.993.569.079.183.7

We also conducted a small-scale human evaluation to assess the consistency of the Gemini-based evaluation. The original InstructTTSEval experiments already included a human-consistency evaluation. For the caption-quality evaluation, we randomly sampled 100 annotations from the data used in the corresponding Table1 experiment and manually judged field-level consistency. We then cross-checked these judgments against the evaluations produced by Gemini-3-flash-preview. Human judgments and Gemini-3-flash-preview agreed on 89 out of 100 samples. This agreement rate provides supporting evidence for the reliability of the metrics reported in Table1.

Thank you for the authors’ careful and detailed response. After reviewing the rebuttal, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.