- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RePlan-VC, a controllable voice conversion framework that regenerates target-side semantic and prosodic trajectories instead of directly reusing source representations. The framework consists of a factorized state encoder, a hierarchical target state planner, and a conditional acoustic generator. The planner generates semantic states before prosodic states, incorporates target speaker and emotion conditions, and employs a sequence-level linguistic objective to preserve linguistic content. The model is trained using content-matched source–target speech pairs constructed from multiple speech datasets. Experiments evaluate timbre conversion, emotion conversion, and joint conversion using objective and subjective metrics. The authors also provide component ablations and analyses of the replanned intermediate states.
Critical strengths
-
The paper separately replans semantic and prosodic trajectories through hierarchical autoregressive generation while incorporating target speaker and emotion conditions. The sequence-level linguistic objective provides a mechanism for preserving linguistic content without directly reproducing the source semantic states.
-
RePlan-VC achieves competitive performance across timbre, emotion, and joint conversion. In particular, it obtains the lowest WER and highest EmoSim under joint conversion among the reported baselines. The ablations also show substantial WER degradation when key planner components are removed, supporting their practical importance.
Critical weaknesses
-
The paper does not sufficiently characterize a central supervision requirement of the proposed method. Obtaining large-scale parallel speech with matched linguistic content across different speakers and emotions is a fundamental challenge in voice conversion. However, RePlan-VC trains its hierarchical planner using content-matched source–target state sequences, and Appendix B reports 1,796,625 training pairs constructed from 791,173 utterances. Although the paper describes four pair-construction categories, it does not provide sufficient information about the origin and composition of these pairs, including the number of unique matched transcripts, their distribution across datasets and conversion types, and whether any synthetic or pseudo-parallel acoustic targets are involved. These details are essential for understanding the scalability and supervision requirements of the proposed training paradigm. The evaluation also relies on custom conversion sets constructed from VCTK and ESD, without reporting results under an explicitly identified standardized public evaluation protocol. Although the authors describe the training-pair categories and state that evaluation speakers are excluded from training, the paper does not fully specify the evaluation-pair selection criteria or whether the training and evaluation pairs follow comparable content-matching procedures. If synthetic targets or restricted parallel subsets are used, their influence on generalization and the comparability with baselines trained under different data conditions should be carefully examined. Without further clarification or evaluation under an independent protocol, it is difficult to determine how representative the reported results are of general non-parallel voice conversion.
-
Baseline configurations and preservation of non-target attributes are insufficiently characterized. The baseline comparison lacks important configuration details. Vevo provides distinct task-specific inference configurations for timbre, style, and joint voice conversion. However, Table 1 reports results simply as “Vevo” and “Vevo2,” without specifying the inference configurations, checkpoints, or reference settings used for each evaluation task. Since these choices directly affect speaker and emotion conversion, this ambiguity makes it difficult to assess the fairness and reproducibility of the comparisons. Furthermore, the paper does not sufficiently evaluate the preservation of non-target speaking-style attributes under timbre-only conversion. RePlan-VC explicitly allows the generated sequence length to differ from the source, but does not report duration consistency, speaking-rate preservation, or pause alignment metrics. Table 3 also shows that replanning improves target speaker similarity while substantially reducing source emotion preservation. These results suggest a trade-off between target-speaker adaptation and source-style preservation, which is not fully characterized by the current evaluation.
Questions for authors
-
How exactly are the 1,796,625 content-matched training pairs constructed? Please report the distribution across P1/P2/E1/E2, datasets, and unique transcripts. Are all source and target utterances original recordings, or are any synthetic or pseudo-parallel targets used? If synthetic speech is involved, what is its proportion and how is it generated?
-
The paper evaluates on custom conversion pairs constructed from VCTK and ESD. How are these pairs selected, and are the same content-matching criteria used as in training? How are the evaluation pairs selected, and do they follow the same content-matching criteria as the training pairs? Could the authors clarify the speaker-disjoint split protocol and provide the evaluation pair lists or results under an established public evaluation protocol? Could the authors provide the evaluation pair lists or results under an established public evaluation protocol to facilitate independent comparison?
-
Which Vevo and Vevo2 inference configurations, checkpoints, and reference settings were used for each conversion task? Could the authors clarify how task-specific settings were selected to ensure comparable evaluation conditions?