- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a Timestep-Adaptive Condition Modulation (TACM) block for flow-matching-based text-to-speech synthesis. Motivated by an analysis of timestep-dependent conditioning, TACM maps timestep embeddings to two separate modulation factors that adjust text and reference-speech features before their fusion with the latent state. The module is integrated into F5-TTS and ZipVoice and evaluated in a zero-shot synthesis setting. The reported results show improvements in speech intelligibility and speaker similarity, together with a small computational overhead.
Critical strengths
-
The paper investigates a useful architectural question: whether explicitly adjusting the contributions of text and reference speech across generation timesteps can improve speech synthesis. Treating these two conditioning sources separately is a reasonable design choice, as they provide different information to the synthesis process. The proposed module offers a straightforward way to introduce this flexibility without substantially modifying the underlying model architecture.
-
The paired comparisons with F5-TTS and ZipVoice provide encouraging empirical evidence.
-
The proposed module is lightweight, and the paper explicitly evaluates its computational overhead.
Critical weaknesses
-
The central claim that existing Concat-FC architectures provide identical conditional guidance at every timestep is not sufficiently established. Equations (2)–(3) show only that the additive condition-dependent term at the initial linear projection is timestep-independent. However, Figure 2 includes a downstream vector-field predictor that receives information about both the latent state and the timestep. Nonlinear processing in this predictor could therefore produce timestep-dependent effects of the conditions even when the initial projection weights are shared. Consequently, a time-independent input projection does not imply time-independent conditioning in the overall model. The paper should distinguish these two notions and provide evidence that the existing architecture fails to capture the relevant timestep-dependent interactions, rather than drawing this conclusion solely from the initial mixing layer.
-
The connection between the theoretical analysis and the specific TACM design remains relatively weak. The derivation in Section 3.2 establishes a timestep-dependent coefficient under a one-dimensional Gaussian setting, with text and reference speech represented jointly as a single condition. However, it does not directly justify using two separate scalar modulation factors or their particular bounded ranges. Section 3.3 itself acknowledges that the analytical coefficient cannot be directly applied to the actual TTS setting. Moreover, the implemented mechanism is essentially a timestep-conditioned scaling operation, and the existing ablations do not establish whether this particular design is preferable to other lightweight alternatives, such as timestep-conditioned scale-and-shift modulation or simple predetermined modulation schedules.
-
The analysis does not sufficiently substantiate the claimed mechanism of timestep-dependent guidance. Figure 5 shows that the learned modulation factors vary with time, but this alone does not demonstrate that they recover an underlying optimal guidance schedule or that the baseline lacks timestep-dependent conditioning. Similarly, Figure 4 shows that WER improves earlier while speaker similarity continues improving later, but these observations do not isolate the causal contributions of text and reference speech at different stages.
-
The experimental scope is relatively narrow. Although two backbones are evaluated, the reported synthesis evaluation is conducted on the same LibriSpeech-PC test-clean subset, consisting of 1,127 utterances with lengths of 4–10 seconds. This leaves uncertainty about whether the improvements extend to longer utterances, different speaking styles, noisy reference speech, or other data distributions.
-
The statistical reliability of some reported improvements is unclear. Table 1 lists only one training seed for each backbone, and the main result tables do not report uncertainty estimates. While the improvements in WER and speaker similarity are encouraging, some speech-quality gains are quite small. For example, F5-TTS improves from 3.877 to 3.887 in UTMOS and from −0.04 to +0.01 in CMOS.
Questions for authors
Please see the above weaknesses.
4 comments
Response to W3: We have revised the conclusion of Fig. 5 to state that the variation of and with is consistent with the theoretical result in Eq. 9, without claiming that they recover the optimal guidance schedule. The previous description did not clearly state that the lack of explicit timestep-dependent guidance refers to the Concat-FC mixing process rather than the entire VF Predictor, leading to ambiguity.
Fig. 4 analyzes acoustic structures and objective metrics across timesteps without isolating the two conditions. To examine their respective effects, we conduct individual condition modulation ablations in Tab. 4 and early and late timestep ablations in Tab. 7. On both F5-TTS and ZipVoice, disabling early text modulation causes larger WER increases than disabling late text modulation, while disabling late reference speech modulation leads to larger decreases in SIM-o and UTMOS than disabling early reference speech modulation. These results support the different roles of text and reference speech modulation across timesteps.
Response to W4: LibriSpeech test-clean and its LibriSpeech-PC subset are commonly used for zero-shot TTS evaluation, as in NaturalSpeech3, E2-TTS, VoiceStar, DiFlow-TTS, and FlexiCodec-TTS (Ju et al., 2024; Eskimez et al., 2024; Peng et al., 2026; Nguyen et al., 2026; Li et al., 2026a). To further evaluate TACM on different datasets, we have added experiments on Matcha-TTS using LJSpeech (single-speaker) and VCTK (multi-speaker) for training and testing. The results show improvements in SIM, WER, and UTMOS on both datasets (see Tab. 12).
For conditioning settings, zero-shot TTS typically uses both text and reference speech conditions, while the newly added seen-speaker TTS experiments do not require reference speech conditioning. These experiments further support the use of TACM across different datasets and TTS settings. We leave the evaluation of longer utterances, different speaking styles, and noisy reference speech for future work.
Response to W5: We use the default training seed specified in each official implementation, given the substantial computational cost of these TTS backbones. We have added 95% confidence intervals for both objective and subjective metrics in the main result tables, estimated using 10,000 bootstrap resamples of test utterances (see App. D). We acknowledge that the UTMOS and CMOS improvements on F5-TTS are small. However, the overall results in Tabs. 2 and 12 support the effectiveness of TACM across different backbones and datasets.
Thanks for the reviewers' careful response. My major concerns have been addressed; thus, I would like to increase my rating to 3.
Thank you very much for your thoughtful questions and for increasing your rating. We sincerely appreciate your careful consideration and would be happy to address any further questions.
We thank the reviewer for recognizing the value of timestep-dependent condition modulation, the effectiveness of TACM across both backbones, and its low computational overhead. Below we respond to each of your concerns.
Weaknesses:
Response to W1: Our comparison of Eqs. (3) and (9) focuses on the condition mixing process in Concat-FC rather than the entire VF Predictor. In Eq. (3), the mixed result of Concat-FC is
\underbrace{c_{text}W_{text}+c_{ref}W_{ref}}_{Condition Part} +b,\quad k\in\\{i,j\\},\\ t_i\neq t_j.$$ Here, the condition part is **timestep-independent**, while $x_{t_k}$ **changes dynamically with $t$**. As illustrated in Fig. 1, this observation leads us to hypothesize that the guidance of conditions on the latent state should also change dynamically with timesteps in the Concat-FC mixing process. However, Eq. 3 does not indicate how the guidance should change with timesteps. Since $h_{t_k}^{\text{Concat-FC}}$ is then combined with the timestep embedding and fed into the predictor, we further derive the optimal VF to analyze whether the part related to the conditions is timestep-dependent. Under the Gaussian assumptions, we obtain $$v_t^*(x_t,t,c)=a_x(t)x_t+\underbrace{a_c(t)\mu_c}_{\text{Condition Part}}, \quad a_c(t)=\frac{(1-t)\sigma_0^2}{(1-t)^2\sigma_0^2+t^2\sigma_1^2}.$$ Similar to Eq. 3, the optimal VF contains a part related to the latent state and a part related to the condition. Here, the part related to the condition is **timestep-dependent**, as $a_c(t)$ varies with $t$. This result supports our hypothesis and provides the motivation for **explicitly modulating the guidance of conditions** in the Concat-FC mixing process. Accordingly, we propose the TACM block, which uses the timestep embedding to assign two modulation factors $\alpha(t)$ and $\beta(t)$ to $c_{text}$ and $c_{ref}$ before mixing them with $x_{t}$. This comparison does not rule out the VF Predictor's ability to implicitly model timestep-dependent condition guidance. We have revised the Abstract, Introduction, Sec. 4.4, and Conclusion to clarify that the timestep-dependent guidance of conditions on the latent state refers to the Concat-FC mixing process. Additionally, Secs. 3.2 and 3.3 have been revised to clarify the connection between the Concat-FC mixing process, the optimal VF analysis, and the design of TACM. **Response to W2:** The optimal VF analysis in Eq. 9 shows that the guidance of conditions on the latent state is timestep-dependent. Based on this result and the **different roles of text and reference speech** in TTS, we design TACM with two separate learnable modulation factors. The results in Tabs. 4 and 7 and Fig. 4 show that text modulation mainly improves WER, particularly at early timesteps, while reference speech modulation mainly improves SIM-o and UTMOS, particularly at later timesteps. Furthermore, Tab. 3 shows that using separate modulation factors for text and reference speech outperforms using a shared factor. We also compare comparison between bounded and unbounded modulation. Fig. 6 and Tab. 10 show that **unbounded modulation produces excessive scaling** of the text condition, substantially degrading SIM and UTMOS despite slightly reducing WER on both datasets. As explained in App. C, text affects both prompt and target regions, while reference speech affects only the prompt region, suggesting a restricted range for text and a wider range for reference speech. We then evaluate these range combinations in Tab. 5 and select [0.85,1.15] and [0.00,2.00] based onTab. 6. Although the timestep-dependent $a_c(t)$ in Eq. 9 cannot be directly applied to TTS, we retain its scaling form to adjust the guidance of conditions. With Scaling, the mixed result in Concat-FC is $$h_t^{\text{TACM}}=x_tW_x+\underbrace{\alpha(t)c_{\text{text}}W_{\text{text}}+\beta(t)c_{\text{ref}}W_{\text{ref}}}_{\text{Guidance}}+b.$$ In contrast, Scale-and-Shift gives $$h_t^{{S\\&S}}=h_t^{TACM}+\underbrace{\delta_{text}(t)W_{text}+\delta_{ref}(t)W_{ref}}_{Additional Shift}.$$ where $\delta_{text}(t)$ and $\delta_{ref}(t)$ are timestep-dependent shifts. Since the timestep embedding $e_t$ is added after Concat-FC, the input to the VF Predictor becomes $$\begin{aligned} h_t^{S\\&S}+e_t &=h_t^{TACM}+\underbrace{(e_t+\delta_{text}(t)W_{text}+\delta_{ref}(t)W_{ref})}_{\tilde e_t}\\\\ &=h_t^{TACM}+\tilde e_t. \end{aligned}$$ The additional shifts can be **combined with the timestep embedding** as $\tilde e_t$, without directly modulating the part related to the conditions. Thus, Scaling explicitly modulates condition guidance without additional shifts. Furthermore, we have added experiments comparing TACM with Linear and Cosine Modulation using **predetermined modulation factors** in Tab. 3. The results show that TACM outperforms both methods, supporting learnable timestep-dependent modulation factors over predetermined ones.