acceptodds
Under review as a conference paper at ICLR 2027

Timestep-Adaptive Condition Modulation for Flow Matching Text-to-Speech

Abstract

In the field of text-to-speech based on flow matching (FM-based TTS), a vector field (VF) predictor is learned to transport the latent state from noise to a target acoustic representation across multiple timesteps under text and reference speech conditions. Current FM-based TTS methods often concatenate the conditions with the latent state at each timestep and feed it into shared fully connected (FC) layers to obtain a mixed result, which is used to learn the VF predictor for estimating the transport path. For those methods, if the mixing process using FC layers is regarded as the guidance of conditions on the latent state, then this guidance is identical at any timestep. However, the latent state changes dynamically with timesteps. Therefore, unlike previous methods, we suppose that the guidance of conditions on the latent state also changes dynamically with timesteps. To this end, we consider text, reference speech and latent states as conditional distributions with respect to timesteps to optimize the VF predictor for explicitly analyzing whether the guidance changes with timesteps. After deduction, the dependency of the guidance on the timestep is established. Based on this fact, we propose a Timestep-Adaptive Condition Modulation (TACM) block, which uses the timestep embedding to assign two modulation factors for the text and reference speech conditions at each timestep. In this way, the TACM block enables the conditions to adaptively provide appropriate information at different timesteps for guiding the generation of the target acoustic representation. Experimental results show the superiority of our TACM block and validate our hypothesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-speech, speech generation, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gFnEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Timestep-Adaptive Condition Modulation (TACM) block for flow-matching-based text-to-speech synthesis. Motivated by an analysis of timestep-dependent conditioning, TACM maps timestep embeddings to two separate modulation factors that adjust text and reference-speech features before their fusion with the latent state. The module is integrated into F5-TTS and ZipVoice and evaluated in a zero-shot synthesis setting. The reported results show improvements in speech intelligibility and speaker similarity, together with a small computational overhead.

Critical strengths
  1. The paper investigates a useful architectural question: whether explicitly adjusting the contributions of text and reference speech across generation timesteps can improve speech synthesis. Treating these two conditioning sources separately is a reasonable design choice, as they provide different information to the synthesis process. The proposed module offers a straightforward way to introduce this flexibility without substantially modifying the underlying model architecture.

  2. The paired comparisons with F5-TTS and ZipVoice provide encouraging empirical evidence.

  3. The proposed module is lightweight, and the paper explicitly evaluates its computational overhead.

Critical weaknesses
  1. The central claim that existing Concat-FC architectures provide identical conditional guidance at every timestep is not sufficiently established. Equations (2)–(3) show only that the additive condition-dependent term at the initial linear projection is timestep-independent. However, Figure 2 includes a downstream vector-field predictor that receives information about both the latent state and the timestep. Nonlinear processing in this predictor could therefore produce timestep-dependent effects of the conditions even when the initial projection weights are shared. Consequently, a time-independent input projection does not imply time-independent conditioning in the overall model. The paper should distinguish these two notions and provide evidence that the existing architecture fails to capture the relevant timestep-dependent interactions, rather than drawing this conclusion solely from the initial mixing layer.

  2. The connection between the theoretical analysis and the specific TACM design remains relatively weak. The derivation in Section 3.2 establishes a timestep-dependent coefficient under a one-dimensional Gaussian setting, with text and reference speech represented jointly as a single condition. However, it does not directly justify using two separate scalar modulation factors or their particular bounded ranges. Section 3.3 itself acknowledges that the analytical coefficient cannot be directly applied to the actual TTS setting. Moreover, the implemented mechanism is essentially a timestep-conditioned scaling operation, and the existing ablations do not establish whether this particular design is preferable to other lightweight alternatives, such as timestep-conditioned scale-and-shift modulation or simple predetermined modulation schedules.

  3. The analysis does not sufficiently substantiate the claimed mechanism of timestep-dependent guidance. Figure 5 shows that the learned modulation factors vary with time, but this alone does not demonstrate that they recover an underlying optimal guidance schedule or that the baseline lacks timestep-dependent conditioning. Similarly, Figure 4 shows that WER improves earlier while speaker similarity continues improving later, but these observations do not isolate the causal contributions of text and reference speech at different stages.

  4. The experimental scope is relatively narrow. Although two backbones are evaluated, the reported synthesis evaluation is conducted on the same LibriSpeech-PC test-clean subset, consisting of 1,127 utterances with lengths of 4–10 seconds. This leaves uncertainty about whether the improvements extend to longer utterances, different speaking styles, noisy reference speech, or other data distributions.

  5. The statistical reliability of some reported improvements is unclear. Table 1 lists only one training seed for each backbone, and the main result tables do not report uncertainty estimates. While the improvements in WER and speaker similarity are encouraging, some speech-quality gains are quite small. For example, F5-TTS improves from 3.877 to 3.887 in UTMOS and from −0.04 to +0.01 in CMOS.

Questions for authors

Please see the above weaknesses.

4 comments

We thank the reviewer for recognizing the value of timestep-dependent condition modulation, the effectiveness of TACM across both backbones, and its low computational overhead. Below we respond to each of your concerns.

Weaknesses:

Response to W1: Our comparison of Eqs. (3) and (9) focuses on the condition mixing process in Concat-FC rather than the entire VF Predictor. In Eq. (3), the mixed result of Concat-FC is

\underbrace{c_{text}W_{text}+c_{ref}W_{ref}}_{Condition Part} +b,\quad k\in\\{i,j\\},\\ t_i\neq t_j.$$ Here, the condition part is **timestep-independent**, while $x_{t_k}$ **changes dynamically with $t$**. As illustrated in Fig. 1, this observation leads us to hypothesize that the guidance of conditions on the latent state should also change dynamically with timesteps in the Concat-FC mixing process. However, Eq. 3 does not indicate how the guidance should change with timesteps. Since $h_{t_k}^{\text{Concat-FC}}$ is then combined with the timestep embedding and fed into the predictor, we further derive the optimal VF to analyze whether the part related to the conditions is timestep-dependent. Under the Gaussian assumptions, we obtain $$v_t^*(x_t,t,c)=a_x(t)x_t+\underbrace{a_c(t)\mu_c}_{\text{Condition Part}}, \quad a_c(t)=\frac{(1-t)\sigma_0^2}{(1-t)^2\sigma_0^2+t^2\sigma_1^2}.$$ Similar to Eq. 3, the optimal VF contains a part related to the latent state and a part related to the condition. Here, the part related to the condition is **timestep-dependent**, as $a_c(t)$ varies with $t$. This result supports our hypothesis and provides the motivation for **explicitly modulating the guidance of conditions** in the Concat-FC mixing process. Accordingly, we propose the TACM block, which uses the timestep embedding to assign two modulation factors $\alpha(t)$ and $\beta(t)$ to $c_{text}$ and $c_{ref}$ before mixing them with $x_{t}$. This comparison does not rule out the VF Predictor's ability to implicitly model timestep-dependent condition guidance. We have revised the Abstract, Introduction, Sec. 4.4, and Conclusion to clarify that the timestep-dependent guidance of conditions on the latent state refers to the Concat-FC mixing process. Additionally, Secs. 3.2 and 3.3 have been revised to clarify the connection between the Concat-FC mixing process, the optimal VF analysis, and the design of TACM. **Response to W2:** The optimal VF analysis in Eq. 9 shows that the guidance of conditions on the latent state is timestep-dependent. Based on this result and the **different roles of text and reference speech** in TTS, we design TACM with two separate learnable modulation factors. The results in Tabs. 4 and 7 and Fig. 4 show that text modulation mainly improves WER, particularly at early timesteps, while reference speech modulation mainly improves SIM-o and UTMOS, particularly at later timesteps. Furthermore, Tab. 3 shows that using separate modulation factors for text and reference speech outperforms using a shared factor. We also compare comparison between bounded and unbounded modulation. Fig. 6 and Tab. 10 show that **unbounded modulation produces excessive scaling** of the text condition, substantially degrading SIM and UTMOS despite slightly reducing WER on both datasets. As explained in App. C, text affects both prompt and target regions, while reference speech affects only the prompt region, suggesting a restricted range for text and a wider range for reference speech. We then evaluate these range combinations in Tab. 5 and select [0.85,1.15] and [0.00,2.00] based onTab. 6. Although the timestep-dependent $a_c(t)$ in Eq. 9 cannot be directly applied to TTS, we retain its scaling form to adjust the guidance of conditions. With Scaling, the mixed result in Concat-FC is $$h_t^{\text{TACM}}=x_tW_x+\underbrace{\alpha(t)c_{\text{text}}W_{\text{text}}+\beta(t)c_{\text{ref}}W_{\text{ref}}}_{\text{Guidance}}+b.$$ In contrast, Scale-and-Shift gives $$h_t^{{S\\&S}}=h_t^{TACM}+\underbrace{\delta_{text}(t)W_{text}+\delta_{ref}(t)W_{ref}}_{Additional Shift}.$$ where $\delta_{text}(t)$ and $\delta_{ref}(t)$ are timestep-dependent shifts. Since the timestep embedding $e_t$ is added after Concat-FC, the input to the VF Predictor becomes $$\begin{aligned} h_t^{S\\&S}+e_t &=h_t^{TACM}+\underbrace{(e_t+\delta_{text}(t)W_{text}+\delta_{ref}(t)W_{ref})}_{\tilde e_t}\\\\ &=h_t^{TACM}+\tilde e_t. \end{aligned}$$ The additional shifts can be **combined with the timestep embedding** as $\tilde e_t$, without directly modulating the part related to the conditions. Thus, Scaling explicitly modulates condition guidance without additional shifts. Furthermore, we have added experiments comparing TACM with Linear and Cosine Modulation using **predetermined modulation factors** in Tab. 3. The results show that TACM outperforms both methods, supporting learnable timestep-dependent modulation factors over predetermined ones.

Response to W3: We have revised the conclusion of Fig. 5 to state that the variation of α(t)\alpha(t) and β(t)\beta(t) with tt is consistent with the theoretical result in Eq. 9, without claiming that they recover the optimal guidance schedule. The previous description did not clearly state that the lack of explicit timestep-dependent guidance refers to the Concat-FC mixing process rather than the entire VF Predictor, leading to ambiguity.

Fig. 4 analyzes acoustic structures and objective metrics across timesteps without isolating the two conditions. To examine their respective effects, we conduct individual condition modulation ablations in Tab. 4 and early and late timestep ablations in Tab. 7. On both F5-TTS and ZipVoice, disabling early text modulation causes larger WER increases than disabling late text modulation, while disabling late reference speech modulation leads to larger decreases in SIM-o and UTMOS than disabling early reference speech modulation. These results support the different roles of text and reference speech modulation across timesteps.

Response to W4: LibriSpeech test-clean and its LibriSpeech-PC subset are commonly used for zero-shot TTS evaluation, as in NaturalSpeech3, E2-TTS, VoiceStar, DiFlow-TTS, and FlexiCodec-TTS (Ju et al., 2024; Eskimez et al., 2024; Peng et al., 2026; Nguyen et al., 2026; Li et al., 2026a). To further evaluate TACM on different datasets, we have added experiments on Matcha-TTS using LJSpeech (single-speaker) and VCTK (multi-speaker) for training and testing. The results show improvements in SIM, WER, and UTMOS on both datasets (see Tab. 12).

For conditioning settings, zero-shot TTS typically uses both text and reference speech conditions, while the newly added seen-speaker TTS experiments do not require reference speech conditioning. These experiments further support the use of TACM across different datasets and TTS settings. We leave the evaluation of longer utterances, different speaking styles, and noisy reference speech for future work.

Response to W5: We use the default training seed specified in each official implementation, given the substantial computational cost of these TTS backbones. We have added 95% confidence intervals for both objective and subjective metrics in the main result tables, estimated using 10,000 bootstrap resamples of test utterances (see App. D). We acknowledge that the UTMOS and CMOS improvements on F5-TTS are small. However, the overall results in Tabs. 2 and 12 support the effectiveness of TACM across different backbones and datasets.

Thanks for the reviewers' careful response. My major concerns have been addressed; thus, I would like to increase my rating to 3.

Thank you very much for your thoughtful questions and for increasing your rating. We sincerely appreciate your careful consideration and would be happy to address any further questions.

Reviewer_Pge7OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that in flow-matching TTS, the guidance from text and reference speech should change across timesteps, and supports this with a simplified Gaussian derivation. It proposes TACM, a lightweight block that uses the timestep embedding to scale the text and reference speech conditions before the input projection. TACM is added to F5-TTS and ZipVoice trained on LibriTTS. It improves WER, SIM-o, and subjective scores on LibriSpeech-PC test-clean with negligible computational overhead.

Critical strengths
  • The authors propose a simple conditioning idea that meanigfully improves F5-TTS and ZipVoice. TACM only rescales the text and reference speech conditions over timestpes, adding very few parameters and almost no inference cost. Yet it consistently improves WER, SIM-o, and subjective scores on both F5-TTS and ZipVoice.
Critical weaknesses
  • The proposed method has limited generality. TACM targets models that concatenate conditions at the input, such as F5-TTS and ZipVoice. It seems closer to an inductive bias for this specific design than a general method. It is unclear whether the gains carry over to recent TTS systems with stronger conditioning.
  • I could not find audio samples, which makes it hard to verify the reported perceptual improvements. In addition, the method is a simple conditioning change tailored to a specific architecture. While practically useful, I feel the contribution may not yet be sufficient.
Questions for authors
  • Can TACM be applied to TTS models with other conditioing mechanisms, larger-scale dataset, or stronger conditioning? Any such evidence would help show that the method is general.
5 comments

We thank the reviewer for highlighting the simplicity and effectiveness of TACM, its consistent improvements across both backbones, and its negligible computational overhead. Below, we address each of your concerns.

Weaknesses:

Response to W1: Concat-FC is adopted in FM-based NAR-TTS methods such as Matcha-TTS, E2-TTS, F5-TTS, ZipVoice, ProsodyFM, Rectifying the Emotional Flow, and DMOSpeech2 (Mehta et al., 2024; Eskimez et al., 2024; Chen et al., 2025; Zhu et al., 2025; He et al., 2025; Feng et al., 2026; Li et al., 2026b), as discussed in the Introduction. F5-TTS and ZipVoice remain widely used and competitive backbones, with recent studies building upon F5-TTS (Eom et al., 2026; Varadhan et al., 2026; Xu et al., 2026; Liu et al., 2026) and ZipVoice (Zhu et al., 2026).

As shown in the table below, we have added experiments on Matcha-TTS under the seen-speaker setting using both LJSpeech (single-speaker) and VCTK (multi-speaker), where TACM improves SIM, WER, and UTMOS over Matcha-TTS. Further experimental details are provided in App. E.1. Combined with the results on F5-TTS and ZipVoice, these experiments support the applicability of the TACM block across different VF Predictors and conditioning settings.

ModelLJSpeech SIM ↑LJSpeech WER ↓LJSpeech UTMOS ↑VCTK SIM ↑VCTK WER ↓VCTK UTMOS ↑
Ground Truth1.000±0.0001.000_{\pm 0.000}1.779±0.3281.779_{\pm 0.328}4.380±0.0164.380_{\pm 0.016}1.000±0.0001.000_{\pm 0.000}1.557±0.3301.557_{\pm 0.330}3.999±0.0253.999_{\pm 0.025}
Matcha-TTS0.971±0.0010.971_{\pm 0.001}1.613±0.3121.613_{\pm 0.312}4.186±0.0324.186_{\pm 0.032}0.921±0.0050.921_{\pm 0.005}1.534±0.3161.534_{\pm 0.316}4.008±0.0374.008_{\pm 0.037}
Matcha-TTS (SFM)0.973±0.0020.973_{\pm 0.002}1.715±0.2821.715_{\pm 0.282}4.257±0.0304.257_{\pm 0.030}0.918±0.0100.918_{\pm 0.010}0.886±0.313\mathbf{0.886}_{\pm 0.313}4.106±0.0424.106_{\pm 0.042}
Matcha-TTS + TACM0.975±0.002\mathbf{0.975}_{\pm 0.002}1.422±0.271\mathbf{1.422}_{\pm 0.271}4.263±0.030\mathbf{4.263}_{\pm 0.030}0.924±0.005\mathbf{0.924}_{\pm 0.005}0.895±0.3010.895_{\pm 0.301}4.112±0.040\mathbf{4.112}_{\pm 0.040}

Response to W2: Audio comparisons are available in our Audio Demo.

Concat-FC is widely adopted in both zero-shot and seen-speaker FM-based NAR-TTS methods with different VF Predictors, including Transformer-based E2-TTS, DiT-based F5-TTS, Zipformer-based ZipVoice (Eskimez et al., 2024; Chen et al., 2025; Zhu et al., 2025), and U-Net-based Matcha-TTS (Mehta et al., 2024). We analyze the timestep-dependent guidance of conditions by deriving the optimal VF in Sec. 3.2. Based on this analysis and the different roles of text and reference speech in TTS, we design TACM as a lightweight plug-in module in Sec. 3.3. We then validate its effectiveness across different FM-based TTS backbones in Sec. 4.2 and App. E.1, and further analyze how TACM affects speech generation at different timesteps through ablation studies in Secs. 4.3 and 4.4.

Questions:

Response to Q1: Our theoretical analysis based on the FM objective shows that timestep-dependent condition guidance is not limited to a specific TTS backbone. We have also added experiments on Matcha-TTS using LJSpeech and VCTK under the seen-speaker setting. The improvements on both datasets further support the applicability of TACM to FM-based TTS models with different conditioning settings.

Experiments on LJSpeech and VCTK have been added to complement the existing LibriTTS experiments. Together, these experiments cover three widely used TTS datasets. In addition, 95% confidence intervals for objective and subjective metrics are reported in the main result tables (Tabs. 2 and 12). Developing sota zero-shot TTS models often involves much larger datasets (10k+ hours), which is beyond the scope of this work and requires more computational resources. We believe our current setup provides a strong and fair evaluation of TACM across different FM-based NAR-TTS backbones.

Although we have not evaluated TACM on models with stronger conditioning mechanisms, the consistent improvements across these backbones and datasets support its applicability to different FM-based TTS architectures and conditioning settings.

References

[1] Eom et al. (2026). Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioning. arXiv:2606.20266.

[2] Varadhan et al. (2026). IN-F5: Adapting an English TTS Foundation Model for Multilingual and Zero-Resource Indian Speech Synthesis. Interspeech 2026.

[3] Xu et al. (2026). X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning. arXiv:2605.05611.

[4] Liu et al. (2026). Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning. arXiv:2609.15184.

[5] Zhu et al. (2026). ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching. Findings of ACL 2026.

Thank you for the additional experiments and clarifications.

The clarification in Appendix B.5 addresses my concern that the original manuscript overstated the limitations of conventional models regarding timestep conditioning. The authors now acknowledge that existing models can also learn timestep-dependent interactions implicitly. The additional experiments with linear and cosine modulation, along with the modulation range analysis, provide further evidence for the effectiveness of TACM. The experiments on Matcha-TTS using LJSpeech and VCTK also partially address my previous concern about the limited generality of the method. Thank you for providing the audio demo as well. I think these additions make the experimental results more convincing.

I still have two suggestions:

  1. It would be interesting to compare TACM with a simple alternative that concatenates each condition with the timestep embedding and passes them through an MLP. This could help determine whether the improvements come from TACM's specific modulation mechanism or more generally from learning timestep-dependent condition transformations.
  2. The additional analysis helps explain the choice of modulation ranges, but these are still manually defined based on TTS-specific assumptions. It would be useful to explore whether these ranges could be determined automatically, without relying on such assumptions, to make the approach more general.

Overall, the revisions address several of my previous concerns and make the experimental results more convincing. While I still have some concerns about the novelty and generality of the method, I will increase my rating to 2 based on the current revisions.

Thank you for acknowledging the effectiveness of TACM and for the valuable suggestions. We have conducted additional experiments based on these suggestions and provided further explanations on the novelty and generality of TACM.

Suggestion 1: We have added a comparison experiment with Condition-Time Linear (CT-Linear) on Matcha-TTS using LJSpeech and VCTK in App. E.1 of the revised manuscript. CT-Linear concatenates the text condition with the timestep and processes the concatenated feature through an FC layer before mixing it with the latent state. Based on Eq. (54), the two methods differ in how they use the timestep:

CT-Linear:ht=xtWx+ctextWtext∗⏟Condition Part+etWe+b∗,\text{CT-Linear:}\quad h_t = x_tW_x + \underbrace{c_{\text{text}} W^*_{\text{text}}}_{\text{Condition Part}} + e_tW_e + b^ *,

TACM:ht=xtWx+α(t)ctextWtext⏟Condition Part+b.\text{TACM:}\quad h_t = x_tW_x + \underbrace{\alpha(t)c_{\text{text}}W_{\text{text}}}_{\text{Condition Part}} + b.

CT-Linear uses separate terms for the text condition and timestep, while TACM explicitly adjusts condition guidance through α(t)\alpha(t), consistent with the theoretical analysis in Sec. 3.2. As shown in Tabs. 12 and 13, CT-Linear performs similarly to Matcha-TTS on both datasets, while TACM achieves better performance with fewer additional parameters and lower computational overhead. Those results show that TACM's improvements come from its explicit modulation mechanism rather than from a general timestep-dependent condition transformation.

Suggestion 2: We have added an experiment with Direct FC in App. C.1 of the revised manuscript. Direct FC predicts the modulation factor as α(t)=FC(et)\alpha(t)=\mathrm{FC}(e_t) without predefined range constraints. As shown in Fig. 6 and Tab. 10, both Direct FC and Unbounded modulation cause uncontrolled amplification or suppression of the text condition, resulting in worse performance. Those results show the importance of learning modulation factors within a bounded range to prevent excessive amplification or suppression of condition guidance.

Generality and novelty of the method: Our theoretical analysis is based on the FM objective, where the conditions are not limited to text and reference speech. The novelty of TACM lies in deriving the dependency of the guidance of conditions on the timestep from the optimal VF. Based on this fact, TACM explicitly adjusts the guidance of text and reference speech conditions on the latent state through separate timestep-dependent modulation factors. TACM adopts a lightweight modulation mechanism that improves performance with low additional inference overhead, as shown by the RTF results.

We evaluate TACM on different FM-based NAR-TTS backbones, as summarized below:

FM-based NAR-TTS   VF Predictor   Setting   Dataset
F5-TTSDiTZero-shotLibriTTS
ZipVoiceZipformerZero-shotLibriTTS
Matcha-TTSU-NetSeen-speakerLJSpeech, VCTK

The consistent improvements across these backbones provide convincing evidence for the generality of TACM in FM-based NAR-TTS. Furthermore, our theoretical analysis motivates extending TACM to FM-based AR-TTS (Du et al., 2024; 2025) and diffusion-based TTS (Wang et al., 2026; Zhou et al., 2026) that also use Concat-FC to provide condition guidance to the latent state. These extensions have not been tested experimentally, and we discuss this limitation in App. A (Limitations) and leave further evaluation for future work.

We argue that our analysis for selecting modulation ranges is general. Specifically, we first examine the roles of conditions (text and reference speech in TTS); then compare bounded and unbounded modulation to determine whether range constraints are needed (App. C.1); next, determine the relative range widths based on how the conditions affect the latent state (text affects both prompt and target regions, motivating a restricted range, while reference speech affects only the prompt region, allowing a wider range; App. C.2); finally, select the specific ranges through experiments (text: [0.85,1.15][0.85,1.15], reference speech: [0,2][0,2]; Sec. 4.3). The specific ranges depend on the roles of conditions in each task.

We would be happy to discuss any further questions about the novelty and generality of our method.

References

[1] Du et al. (2024). CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. arXiv:2412.10117.

[2] Du et al. (2025). CosyVoice 3: Towards In-the-Wild Speech Generation via Scaling-Up and Post-Training. arXiv:2505.17589.

[3] Wang et al. (2026). M3-TTS: Native Two-Stream Multi-Modal Diffusion Transformer for Zero-Shot Speech Synthesis. EMNLP 2026.

[4] Zhou et al. (2026). VoxCPM2 Technical Report. arXiv:2606.06928.

Reviewer_xSeUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TACM, which applies separate timestep-dependent scalars to text and reference-audio conditions in flow-matching TTS. A Gaussian example motivates varying conditioning strength over time. Experiments on F5-TTS and ZipVoice report better speaker similarity and intelligibility with little computational overhead.

Critical strengths

TACM is a small, practical addition that improves both evaluated backbones under matched training settings. Table 2 shows consistent gains in speaker similarity and WER, and Table 4 shows complementary benefits from text and reference modulation. The parameter and runtime overhead is small, with useful results at lower sampling budgets (Tables 7–8).

Critical weaknesses

The backbone can vary its response to the conditions over time even with a fixed input projection. Comparing the input layer in Eq. 3 with the optimal velocity field in Eq. 9 therefore provides limited support for the claimed architectural limitation. The experimental explanation also needs stronger controls: Var. Mod. is insufficiently specified, the shared-factor range is unclear, and Figure 4 does not isolate the effects of text and reference audio. The listening study uses only ten utterances per system, and its small score differences are reported without uncertainty estimates.

Questions for authors

How does the argument in Sections 3.1–3.2 account for timestep conditioning inside the backbone? Would ablations restricted to early or late sampling steps support the proposed roles of the two conditions? Please also report uncertainty for the listening scores and clarify how the modulation ranges were selected.

2 comments

We thank the reviewer for recognizing the practicality of TACM, its consistent improvements on F5-TTS and ZipVoice, and its low computational overhead. Below, we address each of your questions.

Weaknesses:

The backbone can vary its response to the conditions over time even with a fixed input projection. Comparing the input layer in Eq. 3 with the optimal velocity field in Eq. 9 therefore provides limited support for the claimed architectural limitation.

Response: Our comparison of Eqs. (3) and (9) focuses on the condition mixing process in Concat-FC rather than the entire VF Predictor. In Eq. (3), the mixed result of Concat-FC is

\underbrace{c_{\text{text}}W_{\text{text}}+c_{\text{ref}}W_{\text{ref}}}_{\text{Condition Part}} +b,\quad k\in\\{i,j\\},\\ t_i\neq t_j.$$ Here, the condition part is **timestep-independent**, while $x_{t_k}$ **changes dynamically with $t$**. As illustrated in Fig. 1, this observation leads us to hypothesize that the guidance of conditions on the latent state should also change dynamically with timesteps in the Concat-FC mixing process. However, Eq. 3 does not indicate how the guidance should change with timesteps. Since $h_{t_k}^{\text{Concat-FC}}$ is then combined with the timestep embedding and fed into the predictor, we further derive the optimal VF to analyze whether the part related to the conditions is timestep-dependent. Under the Gaussian assumptions, we obtain $$v_t^*(x_t,t,c)=a_x(t)x_t+\underbrace{a_c(t)\mu_c}_{\text{Condition Part}}, \quad a_c(t)=\frac{(1-t)\sigma_0^2}{(1-t)^2\sigma_0^2+t^2\sigma_1^2}.$$ Similar to Eq. 3, the optimal VF contains a part related to the latent state and a part related to the condition. Here, the part related to the condition is **timestep-dependent**, as $a_c(t)$ varies with $t$. This result supports our hypothesis and provides the motivation for **explicitly modulating the guidance of conditions** in the Concat-FC mixing process. Accordingly, we propose the TACM block, which uses the timestep embedding to assign two modulation factors $\alpha(t)$ and $\beta(t)$ to $c_{text}$ and $c_{ref}$ before mixing them with $x_{t}$. This comparison does not rule out the VF Predictor's ability to implicitly model timestep-dependent condition guidance. We have revised the Abstract, Introduction, Sec. 4.4, and Conclusion to clarify that the timestep-dependent guidance of conditions on the latent state refers to the Concat-FC mixing process. Additionally, Secs. 3.2 and 3.3 have been revised to clarify the connection between the Concat-FC mixing process, the optimal VF analysis, and the design of TACM. **The experimental explanation also needs stronger controls: Var. Mod. is insufficiently specified, the shared-factor range is unclear, and Figure 4 does not isolate the effects of text and reference audio.** **Response:** TACM w/ 1F uses a shared factor with a range of **[0.00,2.00]**, which covers both the text and reference speech modulation ranges. Var. Mod. uses **two zero-initialized learnable parameter vectors** of length $K$ (NFE), implemented with `nn.Parameter`, to compute separate modulation factors for text and reference speech at each timestep. We have added detailed modulation definitions in App. E.2. Individual condition modulation ablations are reported in Tab. 4 and early/late timestep ablations have been added in Tab. 7, with $t_s=0.25$ based on Fig. 4(b). The results show that text modulation mainly improves WER at early timesteps, while reference speech modulation mainly improves SIM-o and UTMOS at later timesteps, supporting their different roles across timesteps. **The listening study uses only ten utterances per system, and its small score differences are reported without uncertainty estimates.** **Response:** We have added **95\% confidence intervals** for both objective and subjective metrics in Tabs. 2 and 12, estimated by resampling test utterances 10,000 times (see App. D).

Questions:

How does the argument in Sections 3.1–3.2 account for timestep conditioning inside the backbone?

Response: In Eq.3, the condition term ctextWtext+crefWrefc_{\text{text}}W_{\text{text}}+c_{\text{ref}}W_{\text{ref}} in Concat-FC is timestep-independent. However, the timestep embedding is subsequently added to the mixed result and processed by the VF Predictor, implicitly affecting the guidance of conditions. To analyze this effect within the VF Predictor, we have added a local Taylor approximation in App. B.5 and obtain Ac(t,xt;θ)[c]=Fθ′(b+t)[cWc]+Fθ′′(b+t)[xtWx,cWc],\mathcal{A}_c(t,x_t;\theta)[c]=F^{\prime}_{\theta}(b+t)[cW_c]+F^{\prime\prime}_{\theta}(b+t)[x_tW_x,cW_c],

where cWc=ctextWtext+crefWrefcW_c=c_{\text{text}}W_{\text{text}}+c_{\text{ref}}W_{\text{ref}}. This result indicates that timestep conditioning within the backbone implicitly affects the conditions through the VF Predictor.

Would ablations restricted to early or late sampling steps support the proposed roles of the two conditions?

Response: Individual condition modulation ablations are reported in Tab. 4 and early/late timestep ablations have been added in Tab. 7, with ts=0.25t_s=0.25 based on Fig. 4(b). The results show that text modulation mainly improves WER at early timesteps, while reference speech modulation mainly improves SIM-o and UTMOS at later timesteps, supporting their different roles across timesteps.

Please also report uncertainty for the listening scores and clarify how the modulation ranges were selected.

Response: We have added 95% confidence intervals for both objective and subjective metrics in Tabs. 2 and 12, estimated by resampling test utterances 10,000 times (see App. D).

Regarding the selection of modulation ranges, we have added a comparison between bounded and unbounded modulation. The results in Fig. 6 and Tab. 10 show that unbounded modulation produces excessive scaling of the text condition, substantially degrading SIM and UTMOS despite slightly reducing WER on both datasets. As explained in App. C, text affects both prompt and target regions, while reference speech affects only the prompt region, suggesting a restricted range for text and a wider range for reference speech. We then evaluate these range combinations in Tab. 5 and select the specific ranges of [0.85,1.15] and [0.00,2.00] based on Tab. 6.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciation to the Reviewers

Dear Reviewers,

We thank all three reviewers (R1=xSeU, R2=Pge7, R3=gFnE) for their positive feedback. They all recognized TACM's simple and practical design, consistent performance improvements, and negligible computational overhead.

We have revised the manuscript based on the reviewers' comments, with all changes marked in blue. The main changes and new results are summarized below.

1. Clarifying the Scope of TACM: The Condition Mixing Process (R1, R3)

We clarify that TACM explicitly modulates condition guidance across timesteps in the mixing process, rather than claiming that the VF Predictor cannot model timestep-dependent effects of conditions. This distinction is clarified in the Abstract, Introduction, Sec. 4.4, and Conclusion, with additional analysis in App. B.5.

2. Clarifying the Purpose of Deriving the Optimal VF (R1, R3)

Eq. (3) only describes the Concat-FC mixing process and cannot reveal how condition guidance should change. Since the mixed result is subsequently combined with timestep and fed into the VF Predictor, we further derive the optimal VF in Eq. (9) (Sec. 3.2). The condition part in Eq. (9) exhibits timestep dependence, providing a theoretical basis for explicit modulation in the mixing process.

Eq. (3):htkConcat-FC=xtkWx+ctextWtext+crefWref⏟Condition Part+b.\text{Eq. (3):}\quad h_{t_k}^{\text{Concat-FC}}=x_{t_k}W_x+\underbrace{c_{\text{text}}W_{\text{text}}+c_{\text{ref}}W_{\text{ref}}}_{\text{Condition Part}}+b.

Eq. (9):vt∗=ax(t)xt+ac(t)μc⏟Condition Part.\text{Eq. (9):}\quad v_t^{\ast}=a_x(t)x_t+\underbrace{a_c(t)\mu_c}_{\text{Condition Part}}.

3. TACM Design and Modulation Range Selection (R1, R3)

Motivated by the timestep-dependent scaling in the optimal VF and considering the different roles of text and reference speech in TTS, TACM uses timestep embeddings to generate two separate learnable modulation factors for the two conditions.

For the modulation ranges, we first compare Direct FC, Unbounded, and Bounded variants to examine the need for range constraints. We then select a restricted range for text and a wider range for reference speech based on their different effective regions and determine the specific ranges experimentally (Sec. 3.3, App. C, Tabs. 5, 6, and 10).

4. Additional Ablations and Experiments on Different Models (R1, R2, R3)

Single-condition and shared-factor ablations support separate modulation, while early/late timestep ablations reveal the different roles of conditions across timesteps (Tabs. 3, 4, and 7). Comparisons with predetermined Linear and Cosine Modulation further support the use of learnable timestep-dependent modulation (Tab. 3 and App. E.2).

We also add a CT-Linear baseline and conduct Matcha-TTS experiments on LJSpeech and VCTK, along with analyses of model parameters and inference efficiency (App. E.1; Tabs. 12 and 13). These results further support TACM's effectiveness and lightweight design.

We also report 95% confidence intervals (App. D), provide audio demonstrations, and evaluate TACM under different NFE settings (Tab. 8).

5. Novelty (R2)

The novelty of TACM lies in deriving the timestep dependence of condition guidance from the optimal VF and using this fact to design an explicit timestep-dependent condition modulation mechanism.

6. Generality (R2, R3)

Consistent improvements across different backbones support the generality of TACM in FM-based NAR-TTS. Our theoretical analysis is based on the FM objective and is not limited to text and reference speech conditions. The analysis also motivates extending TACM to FM-based AR-TTS and Diffusion-based TTS, although these extensions remain untested and are left for future work (App. A).

We thank the reviewers again for their valuable feedback. These additions further strengthen the theoretical motivation, the TACM block design, and the experimental evidence supporting its effectiveness.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.