- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AdaConv, an adaptive convolutional feed-forward network (FFN) designed to mitigate over-smoothing and inter-band spectral similarity in flow-matching TTS vector-field predictors. AdaConv combines CPKA, which fuses learned kernels in the discrete Fourier transform domain, with AGKM, which adaptively scales the fused kernel using a pooled and gated representation of the input. Experiments on Matcha-TTS and ZipVoice across three datasets show consistent improvements in word error rate (WER) while maintaining reasonable real-time factors (RTF).
Critical strengths
• The paper tackles over-smoothing directly within the vector-field predictor, offering a potentially complementary approach to existing loss-based and post-processing methods.
• . The proposed method achieves lower WER across multiple backbones, datasets, and zero-shot evaluation settings compared with static Conv1D and dynamic convolution (DConv) baselines.
• The ablation study in Table 4 examines the contributions of CPKA and AGKM, while Table 6 reports CPU and GPU RTFs to assess computational efficiency.
Critical weaknesses
• WER is the only metric that demonstrates consistent and substantial improvements. The reported results provide limited evidence of corresponding gains in subjective or objective perceptual quality, such as UTMOS and SMOS, making it difficult to establish that the proposed method improves speech quality more broadly.
• The ground-truth (GT) speech exhibits a higher WER than several generated outputs. This unexpected result complicates the interpretation of WER improvements and weakens the claim that lower WER directly demonstrates better recovery of spectral details, since ASR-based metrics can be affected by factors beyond spectral fidelity and perceptual quality.
• The claim that AdaConv reduces inter-band similarity is supported by only four utterances from a single model pair in Fig. 5(a), which is insufficient to establish the generality of the observed effect.
• Inter-band similarity statistics are not reported across the full test set. Furthermore, the analysis does not cover alternative backbones (Conformer and Zipformer), the DConv baseline, or the ablation variants in Table 4. Consequently, the evidence does not establish whether the proposed method consistently reduces inter-band similarity or whether this effect explains the observed WER improvements.
• The motivation for AdaConv relies on changes in the vector-field input across flow-matching integration steps (Section 3.1.2). However, the paper does not empirically demonstrate how the learned gates in Eq. (5) or the kernel modulation tensor in Eq. (7) evolve over these steps.
• Since the FFN inputs already incorporate time embeddings, it remains unclear whether the proposed input-dependent modulation provides benefits beyond conventional time-conditioning mechanisms. The paper would be strengthened by comparisons with simpler alternatives, such as adaptive layer normalization (adaLN), commonly used in diffusion transformer architectures.
• CPKA operates in the frequency domain and aggregates information across sequence positions, potentially changing the temporal receptive field and effective modeling capacity relative to standard Conv1D layers. The paper does not clearly specify whether the Conv1D baseline uses a matched kernel size or otherwise controls for differences in temporal modeling capacity. Therefore, it is difficult to determine how much of the observed improvement stems from adaptivity rather than increased or altered temporal context.
• CPKA does not contain input-dependent terms and therefore appears amenable to reparameterization as a single static kernel at inference time. The paper does not clarify whether this reparameterization is performed or quantify its implications for inference efficiency. In addition, the output dimensionality of the fully connected layer in Eq. (7) is not specified, and Eq. (9) lacks sufficient detail regarding padding and channel-wise summation.
Questions for authors
What is the theoretical or empirical relationship between inter-band spectral similarity and WER? Could the authors report the correlation between these metrics across the full test set? How do they explain the observation that the ground-truth audio has a higher WER than several generated outputs, and what implications does this have for interpreting WER as evidence of improved spectral fidelity?
Could the authors report inter-band similarity statistics, including the mean and standard deviation of the distance to GT, across the entire test set? Please include all evaluated backbones, the ablation variants in Table 4, the DConv baseline, and parameter-matched baselines to establish whether the observed reduction is consistent and attributable to the proposed method.
Do the gating values in Eq. (5) and the kernel modulation tensor in Eq. (7) change systematically across vector-field ODE integration steps? Could the authors quantify this behavior and compare AdaConv empirically with a baseline employing direct time-step modulation, such as adaLN, to isolate the benefits of the proposed adaptive mechanism?
3 comments
: I acknowledge the authors’ clarification regarding the distinction between WER and perceptual metrics (UTMOS, CMOS, and SMOS). My remaining concern is the interpretation of WER relative to the ground-truth (GT) speech. If natural pauses and pronunciation variations in the GT are responsible for its higher WER compared with some synthesized speech, this suggests that WER may be strongly influenced by ASR-specific factors rather than reconstruction fidelity. An analysis of where the ASR errors occur (e.g., particular phonemes or temporal regions), or verification using an alternative ASR model, would help clarify this issue and support a more careful interpretation of the reported WER improvements.
: Resolved. The additional full-test-set analysis substantially addresses my concern regarding the generality of the inter-band similarity results.
: My original question was intended to clarify the role of AGKM specifically within the flow-matching (FM) process. I understand that AGKM performs input-dependent modulation rather than direct timestep conditioning. However, the response still does not fully explain "why adapting the input and convolutional kernel according to the evolving vector-field state is particularly beneficial for flow-matching-based generation", beyond demonstrating an empirical WER improvement. A clearer explanation of the connection between the evolving FM trajectory and the proposed adaptive mechanism, and why conventional convolution or standard timestep conditioning may be insufficient, would strengthen the motivation for AGKM.
: Resolved. The clarification that CPKA operates on the kernel weights without increasing the temporal receptive field, together with the additional implementation details and efficiency discussion, addresses my main concerns.
: I appreciate the authors’ efforts to address my comments and provide additional analyses. Several of my original concerns, particularly those regarding inter-band similarity and the implementation of CPKA, have been adequately addressed. My main remaining concern is the interpretation and justification of the proposed method. While AdaConv consistently improves WER and the additional experiments provide stronger evidence that it reduces inter-band similarity, the connection between these observations and the flow-matching generation process remains insufficiently established. In particular, it would be helpful to explain more clearly why input-dependent adaptation of the convolutional kernels is expected to benefit the evolving vector field during ODE integration, rather than simply demonstrating that the architectural modification improves the final WER.
We appreciate the reviewer's acknowledgment of our previous clarifications. To further address the remaining concerns, we provide additional analyses in Apps. D.4 and D.5. The latest manuscript has been revised accordingly.
Response to W1: To further verify the WER improvements, we evaluate the generated speech using Canary-1b-v2 in addition to Whisper-large-v3 (App. D.4, WER Evaluation with Different ASR Models). AdaConv achieves the lowest WER for both Matcha-TTS and Matcha-TTS (SFM) under both ASR models, although the absolute WER values differ. GT speech also has higher WER than all generated outputs in both evaluations, suggesting that ASR-related factors can affect WER. Those results support AdaConv's improvements in speech content accuracy across different ASR models, while indicating that WER does not directly measure reconstruction fidelity.
Response to W3: In FM, the VF predictor estimates how to move the current state toward the target speech. As the predicted VFs are integrated over ODE steps, the generated state gradually contains more speech information, changing the input features of the VF predictor. As shown in App. D.5, the mean input gating value decreases while its standard deviation increases, suggesting more selective feature processing. The mean absolute changes between adjacent steps also decrease, indicating more stable gating in later steps. Meanwhile, the mean kernel modulation coefficient decreases from 0.204 to 0.150. Those results suggest that AGKM adjusts input gating and convolutional kernels as the input evolves, allowing the VF predictor to adapt to the current input features beyond timestep conditioning alone.
We appreciate the reviewer's recognition of our approach, experimental results, and computational efficiency analysis. The concerns and questions are addressed below.
Weaknesses:
Response to W1: WER measures content fidelity, while UTMOS, CMOS, and SMOS evaluate perceptual quality or speaker similarity. As shown in Table 2, AdaConv consistently reduces WER while generally maintaining comparable perceptual quality. We do not directly equate lower WER with better perceptual quality or spectral fidelity.
GT speech contains natural pauses and pronunciation variations, which affect ASR recognition. In contrast, generated speech often has clearer and more consistent pronunciation, resulting in lower WER. Therefore, lower WER does not necessarily indicate better perceptual quality or spectral fidelity than GT.
Response to W2: We conducted additional inter-band similarity analysis across the full test sets, as detailed in App. D.3. Figs. 8 and 9 show that AdaConv achieves the lowest mean inter-band similarity and distance to GT across Transformer, Conformer, and Zipformer backbones, compared with Conv1D, DConv, and the ablation variants.
Response to W3: Our motivation differs from direct timestep conditioning. AdaConv aims to adapt the convolutional kernel weights to the current input rather than directly to the timestep. As the VF input changes across ODE integration steps, AGKM adjusts the input and kernel weights accordingly. As shown in Tab. 4, AGKM improves WER while keeping the original timestep conditioning unchanged.
Response to W4: CPKA operates on frequency-domain kernel weights rather than the input sequence. AdaConv uses the same kernel size as Conv1D (k=3, padding=1), so CPKA does not increase the local temporal receptive field. CPKA is input-independent and can be precomputed at inference time, while AGKM remains input-dependent. Table 6 reports the inference efficiency of AdaConv, but the speedup from CPKA reparameterization has not been separately measured. We have clarified the FC output dimension and broadcast axes in Eq. (7), as well as padding and input-channel summation in Eq. (9), in the updated Secs. 3.1.1 and 3.1.2.
Questions:
Response to Q1: Lower inter-band similarity is associated with clearer frequency details in Mel-spectrograms, making speech content more distinguishable, as illustrated in Figs. 2 and 13. GT speech contains natural pauses and pronunciation variations that affect ASR recognition, while generated speech often has clearer pronunciation, resulting in lower WER.
Response to Q2: As shown in Figs. 8 and 9 of App. D.3, we report the mean and standard deviation of inter-band similarity and distance to GT over the full test sets, covering all three backbones, Conv1D, DConv, the ablation variants, and the parameter-matched baseline. AdaConv consistently achieves the lowest values across all three backbones.
Response to Q3: Our motivation differs from direct timestep modulation. The gating coefficients in Eq. (5) and kernel modulation coefficients in Eq. (7) are computed from the current input at each ODE integration step, rather than directly from timestep embeddings. Unlike adaLN, AGKM adapts both the input and convolutional kernel weights based on the current input features. As shown in Tab. 4, AGKM improves WER while keeping the original timestep conditioning unchanged, demonstrating the benefits of input-dependent modulation.