acceptodds
Under review as a conference paper at ICLR 2027

InstructTTS-OPD: On-Policy Distillation for Multi-Objective Instruction-Following TTS

Abstract

Instruction-following text-to-speech (Instruct-TTS) renders a target transcript in the manner described by a free-form natural-language instruction, and a good rendition must succeed on three fronts at once: following the instruction, pro- nouncing the transcript correctly, and sounding natural. Reinforcement learning has become the standard way to post-train such generative models toward human preferences, and TTS systems increasingly adopt it. When the three objectives are optimized together under a single scalar reward, typically with group relative pol- icy optimization (GRPO), training tends to be unstable: the objectives compete, and the shared update drifts toward whichever one is easiest to improve, producing a seesaw among them. We instead adapt on-policy distillation (OPD), which fuses several single-objective teachers by matching their velocity fields on the student’s own rollouts. Realizing this for Instruct-TTS first calls for a sound reward on each axis, which we analyze carefully: a distilled Qwen-Omni proxy for the costly Gemini judge, dialect-free data for the WER reward, and DNSMOS in place of the language-biased UTMOS. We then train three single-objective teachers with GRPO and fuse them with OPD, routing each rollout to the instruction-following or WER teacher by its data domain and letting every rollout additionally follow the MOS teacher for quality. This lets instruction following and intelligibility im- prove together without a trade-off, but the quality target conflicts with instruction following, so following it additively reintroduces the seesaw. We therefore pro- pose conflict-aware velocity fusion, which projects the quality target off the direc- tion that opposes the primary objective before fusing, and show that it markedly reduces the instruction-following/quality trade-off. The fused model improves over scalar-reward GRPO on all three objectives at once.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
InstructTTS, RL, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gizjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InstructTTS-OPD, a framework for improving instruction-following TTS through on-policy distillation. It trains separate GRPO teachers for instruction following, intelligibility, and speech quality, then distills their outputs into a student using dense velocity-field supervision. Domain routing and conflict-aware velocity fusion are introduced to reduce interference between objectives.

Critical strengths
  1. The paper addresses a practical challenge in instruction-following TTS: balancing instruction adherence, intelligibility, and speech quality during post-training.
  2. On-policy distillation provides a reasonable way to transfer knowledge from specialized teachers without directly combining potentially conflicting rewards.
  3. The reward analysis is thoughtful. The dialect experiment shows a concrete failure mode: optimizing Mandarin ASR scores can encourage the model to ignore the requested dialect.
Critical weaknesses
  1. The human study in Appendix D compares only against the base model. It therefore does not establish a perceptual advantage over GRPO, particularly given the small DNSMOS improvement.
  2. The demonstrated scope is limited to one DiTAR-based model. Although Table 4 shows IF improvements across six subsets, the joint gains in IF, intelligibility, and quality are evaluated mainly on zh_DSD. Whether these gains hold more broadly remains unclear.
  3. The IF reward model is validated through checkpoint-level correlation with Gemini. This does not establish the reliability of its individual judgments, especially when distinguishing outputs generated for the same instruction.
Questions for authors
  1. Could you report the number of prompts and listeners for the human study in Appendix D? Could you also include scalar-reward GRPO and evaluate instruction adherence and naturalness separately?
  2. Could you report the IF reward model’s utterance-level agreement with Gemini and accuracy against human annotations on an independent test set, with results broken down by instruction category?
  3. Does the GRPO baseline use equivalent domain-specific reward handling? How do the tuning budget and total training cost compare with the full teacher-training and distillation pipeline?
Reviewer_uTb2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InstructTTS-OPD, a multi-teacher on-policy distillation framework with conflict-aware velocity fusion for instruction-following TTS. The method jointly optimizes instruction following, intelligibility, and speech quality. The results are promising, and the analysis of objective conflicts is informative. However, several concerns remain regarding novelty and experimental validation.

Critical strengths
  • The paper addresses an important problem in multi-objective optimization for instruction-following TTS.

  • The experiments provide useful analyses of reward design, objective conflicts, and the contributions of different components.

Critical weaknesses
  • Multi-teacher on-policy distillation, domain routing, and velocity-field matching have already been explored in Flow-OPD[1], while the proposed conflict projection is closely related to PCGrad[2]. The authors should better clarify the unique contribution of conflict-aware velocity fusion beyond these existing techniques and provide stronger comparisons.

[1] Flow-opd: On-policy distillation for flow matching models

[2] Gradient surgery for multi-task learning (NeurIPS 2020)

  • Appendix C suggests that domain routing is a major factor in preserving instruction-following performance. However, the paper does not compare against multi-objective GRPO using the same domain-routing strategy. It is therefore difficult to determine whether the improvements mainly come from on-policy distillation or objective routing. The computational cost of training three teachers should also be considered.

  • Appendix D only compares the final model with the supervised base model. There is no direct human evaluation against multi-reward GRPO or the variant without conflict-aware fusion. Such comparisons, along with the number of raters, sample size, and statistical significance, would provide stronger evidence.

Questions for authors

see Weaknesses

Reviewer_5AekOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies joint optimization of instruction following, intelligibility, and perceptual quality in TTS. It trains three specialist GRPO teachers and distills them along student-generated flow trajectories. Domain routing selects the primary teacher, while an asymmetric projection removes the component of the quality teacher’s correction that opposes the primary direction. Experiments report improvements over the base model and multi-reward GRPO, supported by routing and fusion ablations, independent automatic metrics, and a human comparison against the base model. Validation is limited to TTS.

Critical strengths
  1. I found the reward analysis useful. Mandarin ASR can penalize correctly rendered dialect speech, while quality predictors can penalize irregularities that are intentional in emotional or expressive speech. These examples show how optimizing standard rewards can work against the requested behavior and give a concrete reason to treat the primary objectives separately.

  2. The projection ablation is encouraging. In Table 7, projected fusion achieves an IF score of 77.9 versus 75.8 for additive fusion at the same reported DNSMOS of 3.31, although the quality weights differ. I find this comparison more convincing than a gain at a single fixed weight, since it suggests that projection improves instruction following at a comparable level of predicted speech quality.

Critical weaknesses
  1. I could not find an audio-demo link in the paper. Given the claims about preserving expressive speech and dialects while improving quality, listening examples are important for assessing whether the gains are perceptually meaningful.

  2. I am not yet convinced that the experiments isolate the benefit over closely related approaches. Flow-OPD already combines specialist teachers, task routing, and an auxiliary quality teacher, and DiffusionOPD uses direct transition-mean matching. Appendix C suggests that routing accounts for an important part of the IF gain, but does GRPO use the same domain-dependent reward masking and receive a comparable search over reward weights? I would also like to see GRPO and additive OPD included in the Gemini and human evaluations. Without these comparisons, it is hard to judge whether the gains hold beyond the training reward metrics.

  3. The claims around Eq. 7 seem stronger than the projection supports. Removing an opposing component between teacher residuals is an output-space operation; it does not guarantee that the parameter update preserves the primary objective, since the student Jacobian changes the relevant geometry. Matching the primary teacher’s output also does not necessarily preserve instruction-following behavior? The empirical gains are encouraging, but I'm not convinced the formulation alone establish this protection.

Questions for authors

Does the multi-reward GRPO baseline use the same domain-dependent reward masking and dialect filtering as OPD? How were its reward weights selected?

What exactly is the protection claimed for Eq. 7 intended to cover? Is the claim limited to the local distillation target, or is there evidence that it protects the primary objective during training?

Do the improvements over GRPO and additive OPD persist under the Gemini judge?

OOC, have you evaluated the projected fusion itself outside TTS, for example in the image-generation setting of Flow-OPD? Any existing evidence would help assess the breadth of the contribution.

Are audio samples available, either on a demo page or in the supplementary material? I could not find a link in the paper. I would particularly like to hear matched examples from GRPO, additive OPD, and the proposed method, including expressive speech and dialect instructions. Convincing listening examples showing better instruction following while preserving naturalness could change my assessment and score.

====== post rebuttal =====

I understand that the intended contribution is the speech-specific reward analysis and conflict-aware velocity fusion. The reward analysis is useful, but I remain somewhat uncertain about the strength of the incremental contribution over existing OPD approaches.

Comments on the paper

1 comment
Area_Chair_NAgwOpenReview ↗

Please start the discussion on the rebuttals

Dear Reviewers,

Thanks a lot for your help reviewing this submission! We have received the rebuttals of the authors, so I would like to hear everyone's opinion on the authors' rebuttal to help me form my final meta-review and recommendation for this paper.

Can I ask everyone to share a little bit about their post-rebuttal feelings about the paper by replying to this message thread, and update their original review/rating of the paper accordingly at your earliest convenience? Thanks a lot in advance!

Best, your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.