acceptodds
Under review as a conference paper at ICLR 2027

EnvTTS: Towards Independent Speaker and Environment Cloning in Text-to-Speech

Abstract

Zero-shot text-to-speech (TTS) has made substantial progress in cloning unseen speakers from reference speech. However, the reference speech also captures acoustic characteristics of its recording environment, making speaker identity and environment difficult to control independently. Environment-controllable TTS (EnvTTS) uses separate speaker and environment references to synthesize a target speaker in a specified acoustic environment. We characterize the acoustic environment by two components: additive background sound and acoustic effects associated with reverberation. Evaluating EnvTTS requires assessing both speaker preservation and environment reproduction, yet systematic evaluation remains challenging due to the lack of a dedicated benchmark and metrics that separately characterize background and reverberation similarity. To address these gaps, we introduce EnvTTS-Bench, a benchmark of 3,000 evaluation instances with independently specified speaker and environment references drawn from real recordings, and EnvSim, an environment-similarity metric that separately measures background and reverberation similarity. We also establish two baselines: a training-free cascaded approach and EnvTTS-DiT, a flow-matching TTS model conditioned on separate speaker and environment references. Objective and human evaluations show that EnvSim aligns strongly with human judgments for separately measuring background and reverberation similarity. System evaluations on EnvTTS-Bench further show that the cascaded approach provides a practical baseline for EnvTTS, while EnvTTS-DiT achieves stronger overall performance; together, they provide useful reference systems for future EnvTTS research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Text-to-Speech, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1QSMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies environment-controllable text-to-speech (EnvTTS), which aims to independently control speaker identity and the recording environment. The paper makes three main contributions:

  1. EnvTTS-Bench: An evaluation benchmark comprising 3,000 instances constructed from four real-world meeting corpora, covering different speakers, microphone positions, and recording devices.
  2. EnvSim: An environment similarity metric based on factorized contrastive learning that separately measures background sound and reverberation similarity.
  3. EnvTTS Systems: A training-free cascaded baseline and EnvTTS-DiT, a flow-matching model conditioned on separate speaker and environment references.

Experimental results show that EnvSim outperforms general-purpose audio representations in environment similarity assessment, while EnvTTS-DiT achieves better environment similarity and speech intelligibility than the cascaded baseline.

However, this paper has several limitations and ambiguities regarding the scope of evaluation, experimental setup, and conclusions drawn from the results. These issues are detailed in the Critical Weaknesses section. The paper also contains minor notational ambiguities in Section 3.2, where z1z_1 and qq are introduced as encoder outputs in Equation (9) but subsequently used to represent concatenated multi-reference sequences in Equations (11) and (12), without explicitly clarifying the relationship between these definitions.

Critical strengths
  1. Separating speaker identity from the recording environment has clear practical value. The paper further distinguishes between background sound and reverberation, providing a more fine-grained task formulation for environment control and evaluation.
  2. EnvTTS-Bench covers four real-world meeting corpora and constructs two evaluation tiers with different levels of difficulty based on microphone positions and recording devices, providing a relatively consistent evaluation framework for future research.
  3. By independently varying background sound and room impulse responses (RIRs) when constructing training samples, EnvSim learns representations of different environmental factors. In objective evaluations, it achieves EERs of 0.022 and 0.096 for background sound and RIR verification, respectively, and Spearman correlation coefficients of 0.621 and 0.639 with human ratings, respectively.
Critical weaknesses
  1. The paper models background sound and RIR as two separate dimensions of environment similarity, but does not establish whether this distinction offers advantages that cannot be achieved with a unified environment representation. Although EnvSim achieves strong results on both dimensions, the paper does not further validate whether this design improves the evaluation, selection, or environment control capabilities of TTS systems. Therefore, the practical value of this core contribution remains insufficiently justified.
  2. The Cascade baseline relies on SAM-Audio to extract background sound from the environment reference and estimates the SNR before mixing the extracted background with the speaker reference (Appendix C.3). Potential errors in source separation and SNR estimation may affect the resulting speech prompt, but their impact is not independently evaluated. Meanwhile, Cascade uses a frozen TTS backbone, whereas EnvTTS-DiT undergoes additional fine-tuning. These differences make it difficult to determine how much of the performance improvement in Table 4 can be attributed specifically to the proposed multi-reference conditioning mechanism.
  3. The paper separately evaluates background sound and reverberation similarity through human listening tests when validating EnvSim (Table 3 and Appendix A.1). However, the subjective evaluation of EnvTTS-DiT in Table 4 only reports an overall environment similarity score (MOSEnv), without separately assessing these two factors. Although Table 4 reports EnvSimBkg and EnvSimRIR as objective metrics, the absence of corresponding subjective evaluations limits the validation of EnvTTS-DiT's ability to reproduce background sound and reverberation independently.
  4. EnvTTS-DiT aims to reproduce both environmental factors while preserving speaker identity, but the paper does not investigate whether extending environment reproduction beyond a single factor improves or degrades speaker similarity and the fidelity of individual environmental factors. Table 4 compares EnvTTS-DiT with a background-transfer Cascade and two single-reference baselines, without direct comparisons with existing environment-controllable TTS methods or controlled evaluations under background-only and reverberation-only conditions. Consequently, the incremental benefits and potential performance trade-offs of reproducing both environmental factors remain unclear.
  5. The task definition only includes the target text and two audio references, whereas the actual inference process additionally requires transcripts of both reference recordings and uses the duration of the ground-truth target recording to determine the generation length (Appendix C.3 and Appendix C.4). In particular, the latter relies on prior information that is generally unavailable in practical applications, which may affect the conclusions drawn from WER and generation quality evaluations. The paper also does not clarify how the reference transcripts are obtained in practical scenarios or evaluate the impact of transcription errors, particularly when strong background noise in the environment reference makes its speech difficult to recognize.
  6. The paper considers background sound an important component of environment control, but its training and evaluation mainly rely on meeting recordings and background sound conditions constructed using MUSAN, without specific evaluations of non-stationary sound events such as dog barking and bird chirping. Furthermore, in the final case presented on the project demo page, the generated audio fails to preserve the clearly audible dog barking in the environment reference. This raises concerns about whether the model can effectively transfer distinctive background sound events and whether EnvSim can accurately assess the absence of such events.
Questions for authors
  1. Can the authors conduct experiments to verify whether speaker identity and recording environment can be independently controlled by fixing the speaker reference while varying the environment reference, and vice versa? How does the model perform across different sessions and rooms?
  2. Does the proposed EnvTTS framework generalize to cross-room reference conditioning, where the speaker reference and environment reference are recorded in different physical spaces?
  3. Why does EnvTTS-DiT achieve substantially lower speaker similarity than Clone-Spk when conditioned on both speaker and environment references?
  4. Could the authors evaluate the responses of EnvSimBkg and EnvSimRIR to targeted acoustic modifications, such as applying denoising and dereverberation separately to the same generated speech, to assess whether each metric is primarily sensitive to its corresponding environmental factor?
  5. In Equation (2), the paper defines n(t)n(t) as additive background sound, while other sections use the term background noise. Does this formulation explicitly include salient non-stationary sound events, such as animal vocalizations, or does the intended environment reproduction primarily focus on general ambient noise characteristics?
Reviewer_W3NiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies environment-controllable TTS (EnvTTS), where a system must clone a speaker from a speaker reference while reproducing the environment of a separate environment reference. Specifically, the environment is decomposed into additive background sound and RIR-induced reverberation. The authors contribute (i) EnvTTS-Bench, a benchmark drawn from AMI, AliMeeting, ICSI and NOTSOFAR-1 with independently speaker/environment references, organized into close-talk (Tier A) and far-field cross-device (Tier B) tiers; (ii) EnvSim, a metric with separate noise and RIR heads trained by masked InfoNCE on a noise×RIR factorial grid; and (iii) two systems, a training-free cascade (SAM-Audio separation + SNR-matched mixing + frozen zero-shot TTS) and EnvTTS-DiT, a flow-matching DiT conditioned on separate environment and speaker streams. Experiments show that EnvSim outperforms Dasheng/CLAP/AES-PQ and correlates with human ratings, and that EnvTTS-DiT improves EnvSim, WER and MOS-env over the cascade with comparable speaker similarity and naturalness

Critical strengths
  1. EnvSim is a simple but well-validated factorized metric. Using separate noise/RIR heads, EnvSim presents large margins over Dasheng/CLAP/AES-PQ (e.g., EER 0.022/0.096) and human correlations (ρ≈0.62/0.64) provide convincing evidence of axis separation.
Critical weaknesses
  1. Although EnvTTS introduces reverberation to the background sound to form a more comprehensive environment context modeling. There lacks a solid validation to verify that simply manipulating the {noise, reverberation} condition pair could form a complete environment control for TTS synthesis.
  2. Both the Tier A of the constructed evaluation benchmark EnvTTS-Bench and the real half of the EnvTTS-DiT training data are based on the AMI and AliMeeting datasets. This not only hurts the trustfulness of the evaluation results, but also hinders to assess the generalization ability of EnvTTS-DiT.
  3. Only a constructed cascade baseline is included for comparison on the EnvTTS-Bench. Despite the fact that the proposed EnvTTS-Bench contains 2 separate tiers targeting different scenarios, these is no detailed tier-wise evaluation results presented in the paper, only the overall results are reported.
Questions for authors
  1. What does each axis of the “masked InfoNCE per axis” diagram in Figure 2a mean?
  2. In single reference cloning (CLONE-ENV and CLONE-SPK), how is the input sequence assembled?
  3. Could the tradeoff in single reference cloning be cause by the bias in training data construction? (i) the env reference always comes from a different speaker; (ii) the speaker reference always got a different environment condition.
  4. Is there any tier-wise detailed evaluation results on Env-Bench?
Reviewer_joouOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies text-to-speech synthesis with separate speaker and environment references. It introduces EnvTTS-Bench, a 3,000-instance benchmark constructed from four meeting corpora, and EnvSim, a learned metric with separate embeddings for background sound and reverberation. The paper also presents a training-free cascaded baseline and EnvTTS-DiT, which adapts LongCat AudioDiT to accept both references directly. Experiments show that EnvSim correlates better with listener judgments than the evaluated general-purpose representations, and that EnvTTS-DiT improves environment similarity and intelligibility over the cascaded baseline.

Critical strengths

The paper addresses a concrete gap in evaluation. Background sound and reverberation can fail independently, and a single environment-similarity score makes these failures difficult to distinguish. Reporting them separately is useful for understanding system behavior, especially when the reference and generated speech contain different speakers and linguistic content.

Critical weaknesses

The experiments compare EnvTTS-DiT primarily with a cascaded baseline and single-reference anchors, without a direct comparison against closely related environment-aware TTS systems such as DAIEN-TTS. A comparison under matched conditions would help clarify the benefit of directly conditioning on unseparated references.

Questions for authors

Could the authors discuss the differences from DAIEN-TTS and, if feasible, provide a comparison under matched evaluation conditions?

Comments on the paper

1 comment

Thank the authors for their reply. Most of my concerns have been resolved. I will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.