acceptodds
Under review as a conference paper at ICLR 2027

RePlan-VC: Target-Side Semantic and Prosodic Replanning for Controllable Voice Conversion

Abstract

Controllable voice conversion aims to modify speaker identity and expressive attributes while preserving linguistic content. Existing methods either retain source-derived content anchors while replanning selected factors or generate target-side states without explicitly modeling dependencies among functionally distinct states, limiting coordinated multi-state replanning. We propose RePlan-VC, a target-side semantic and prosodic replanning framework that regenerates semantic and prosodic trajectories under target speaker and emotion conditions. RePlan-VC employs a hierarchical autoregressive planner that generates semantic states before prosodic states, explicitly modeling their generative dependency, while target-condition modulation directly guides state prediction. To preserve linguistic content without enforcing source state reproduction, a frozen semantic to text mapper provides sequence level supervision over the replanned semantic trajectory. Experiments on timbre, emotion, and joint conversion demonstrate effective target attribute adaptation while preserving linguistic content. Under joint conversion, RePlan-VC achieves 7.68% WER, 0.5046 SECS, 0.6070 EmoSim, and 52.5% EmoAcc.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Controllable Speech Generation, Voice Conversion, Speech Large Language Models, Target-State Replanning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NnvDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RePlan-VC, a controllable voice conversion framework that regenerates target-side semantic and prosodic trajectories instead of directly reusing source representations. The framework consists of a factorized state encoder, a hierarchical target state planner, and a conditional acoustic generator. The planner generates semantic states before prosodic states, incorporates target speaker and emotion conditions, and employs a sequence-level linguistic objective to preserve linguistic content. The model is trained using content-matched source–target speech pairs constructed from multiple speech datasets. Experiments evaluate timbre conversion, emotion conversion, and joint conversion using objective and subjective metrics. The authors also provide component ablations and analyses of the replanned intermediate states.

Critical strengths
  1. The paper separately replans semantic and prosodic trajectories through hierarchical autoregressive generation while incorporating target speaker and emotion conditions. The sequence-level linguistic objective provides a mechanism for preserving linguistic content without directly reproducing the source semantic states.

  2. RePlan-VC achieves competitive performance across timbre, emotion, and joint conversion. In particular, it obtains the lowest WER and highest EmoSim under joint conversion among the reported baselines. The ablations also show substantial WER degradation when key planner components are removed, supporting their practical importance.

Critical weaknesses
  1. The paper does not sufficiently characterize a central supervision requirement of the proposed method. Obtaining large-scale parallel speech with matched linguistic content across different speakers and emotions is a fundamental challenge in voice conversion. However, RePlan-VC trains its hierarchical planner using content-matched source–target state sequences, and Appendix B reports 1,796,625 training pairs constructed from 791,173 utterances. Although the paper describes four pair-construction categories, it does not provide sufficient information about the origin and composition of these pairs, including the number of unique matched transcripts, their distribution across datasets and conversion types, and whether any synthetic or pseudo-parallel acoustic targets are involved. These details are essential for understanding the scalability and supervision requirements of the proposed training paradigm. The evaluation also relies on custom conversion sets constructed from VCTK and ESD, without reporting results under an explicitly identified standardized public evaluation protocol. Although the authors describe the training-pair categories and state that evaluation speakers are excluded from training, the paper does not fully specify the evaluation-pair selection criteria or whether the training and evaluation pairs follow comparable content-matching procedures. If synthetic targets or restricted parallel subsets are used, their influence on generalization and the comparability with baselines trained under different data conditions should be carefully examined. Without further clarification or evaluation under an independent protocol, it is difficult to determine how representative the reported results are of general non-parallel voice conversion.

  2. Baseline configurations and preservation of non-target attributes are insufficiently characterized. The baseline comparison lacks important configuration details. Vevo provides distinct task-specific inference configurations for timbre, style, and joint voice conversion. However, Table 1 reports results simply as “Vevo” and “Vevo2,” without specifying the inference configurations, checkpoints, or reference settings used for each evaluation task. Since these choices directly affect speaker and emotion conversion, this ambiguity makes it difficult to assess the fairness and reproducibility of the comparisons. Furthermore, the paper does not sufficiently evaluate the preservation of non-target speaking-style attributes under timbre-only conversion. RePlan-VC explicitly allows the generated sequence length to differ from the source, but does not report duration consistency, speaking-rate preservation, or pause alignment metrics. Table 3 also shows that replanning improves target speaker similarity while substantially reducing source emotion preservation. These results suggest a trade-off between target-speaker adaptation and source-style preservation, which is not fully characterized by the current evaluation.

Questions for authors
  1. How exactly are the 1,796,625 content-matched training pairs constructed? Please report the distribution across P1/P2/E1/E2, datasets, and unique transcripts. Are all source and target utterances original recordings, or are any synthetic or pseudo-parallel targets used? If synthetic speech is involved, what is its proportion and how is it generated?

  2. The paper evaluates on custom conversion pairs constructed from VCTK and ESD. How are these pairs selected, and are the same content-matching criteria used as in training? How are the evaluation pairs selected, and do they follow the same content-matching criteria as the training pairs? Could the authors clarify the speaker-disjoint split protocol and provide the evaluation pair lists or results under an established public evaluation protocol? Could the authors provide the evaluation pair lists or results under an established public evaluation protocol to facilitate independent comparison?

  3. Which Vevo and Vevo2 inference configurations, checkpoints, and reference settings were used for each conversion task? Could the authors clarify how task-specific settings were selected to ensure comparable evaluation conditions?

Reviewer_4My5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RePlan-VC, which regenerates target-side semantic and prosodic states for voice conversion involving timbre and emotion transfer. It introduces a hierarchical autoregressive planner that predicts the source speaker's semantic states and generates prosodic states based on them, along with a sequence-level CTC loss to preserve linguistic content. To evaluate the model, experiments are conducted on timbre, emotion, and joint conversion.

Critical strengths
  • This paper proposes a hierarchical planner that separates and sequentially generates semantic and prosodic states, explicitly modeling the generative dependency between the two representations.

  • This model is trained to preserve linguistic content using a sequence-level CTC loss with a frozen semantic-to-text mapper, without requiring direct token-level alignment with the source semantic states. The effectiveness of this approach is validated through ablation studies.

Critical weaknesses
  • Vevo and Vevo2 have already proposed generating target content-style representations conditioned on source content. This paper proposes a structure that separates semantic and prosodic states, generates semantic states first, and then predicts prosodic states conditioned on them. However, the paper does not explain why existing fused representations are insufficient to model the dependency between semantic and prosodic states. In addition, the ablation in Table 3 predicts the two states independently, so it is not a direct comparison with the fused target-side generation shown in Figure 1.

  • The contribution of semantic replanning remains unclear. Although the proposed method regenerates both semantic and prosodic states, the paper does not examine how much each process contributes to performance. In particular, there is no comparison with a setting that keeps the source semantic states and only generates target prosodic states. Without this comparison, it is difficult to tell whether semantic replanning provides additional benefits for speaker and emotion adaptation. The current w/o Replanning ablation removes both processes together, so it does not establish the need for semantic replanning.

  • Linguistic consistency and attribute preservation remain concerns. In Table 2, the A2T-Err of the replanned semantic states increases substantially, particularly in emotion conversion (from 6.18% to 39.75%). While the final speech has a relatively low WER, it is unclear how much linguistic information is preserved in the semantic states themselves. Also, in emotion conversion, the improvement in emotion similarity comes with a large drop in speaker similarity, suggesting that the replanning process does not fully address interference between different attributes.

  • The experimental setup and subjective evaluation need more details. RePlan-VC uses explicit target emotion labels, while Vevo and Vevo2 mainly control style through reference speech. However, it is unclear what conditions were provided to each baseline and whether the models were compared under settings that allow a similar level of emotion control. For the MOS, the paper only mentions that 15 participants were involved, without specifying the number of speech samples and how the scores were aggregated. Table 1 also reports only the average MOS scores, without confidence intervals.

  • RePlan-VC shows weaker performance than existing baselines. The reported results are generally worse than those of prior methods, and the demo samples in the supplementary material also sound noticeably worse than the baselines. While controllability is an important aspect of VC, as the authors emphasize, the main goal is to naturally convert speech into the target speaker's voice while preserving the original linguistic content. However, the provided samples show a loss of linguistic content and a decline in overall audio quality, making it difficult to see the practical advantages of the proposed method over existing approaches.

Questions for authors

See the weaknesses section.

Reviewer_mBppOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RePlan-VC, a controllable voice conversion framework that regenerates target-side semantic and prosodic trajectories instead of directly reusing source states. Its hierarchical planner predicts a semantic state followed by a prosodic state at each frame, with explicit speaker/emotion modulation. A frozen semantic-to-text CTC mapper provides sequence-level linguistic supervision. Experiments cover timbre, emotion, and joint conversion, with competitive results particularly on joint conversion and emotion adaptation.

Critical strengths
  1. The paper presents a coherent and unified framework for timbre, emotion, and joint conversion. Its competitive performance on joint conversion demonstrates a promising balance between linguistic-content preservation and controllable attribute adaptation.

  2. The sequence-level linguistic constraint is well motivated and empirically effective. The ablation results show that it substantially improves content preservation across all conversion settings, supporting the importance of explicit linguistic supervision when semantic trajectories are regenerated.

Critical weaknesses
  1. The experiments do not convincingly establish that the proposed semantic-to-prosodic dependency is the source of the improvement. Replacing it with parallel prediction increases WER by about 30 percentage points, yet the parallel model achieves slightly better SECS, EmoSim, and EmoAcc in joint conversion. This unusually large discrepancy may reflect optimization or architectural differences rather than the claimed dependency. A parameter-matched prosody-to-semantic alternative and a clearer analysis are needed.

  2. The evaluation has potential dependence and statistical-validity issues. Emotion2vec is used both to construct/filter training labels and to compute the main emotion metrics, which may favor models optimized toward the same representation. Moreover, the subjective evaluation reports only 15 listeners without confidence intervals or significance tests; for example, 52.5% versus 52.0% EmoAcc on 200 samples differs by only one prediction.

Questions for authors
  1. Can the emotion results be evaluated with an independent emotion recognizer and accompanied by confidence intervals or significance tests for both objective and MOS results?

  2. Table 2 shows that replanned semantic states have substantially higher probe error than source states. Can the authors provide semantic-only/prosody-only decoding or state-swapping experiments to rule out linguistic-content leakage through the prosodic stream?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.