- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies environment-controllable text-to-speech (EnvTTS), which aims to independently control speaker identity and the recording environment. The paper makes three main contributions:
- EnvTTS-Bench: An evaluation benchmark comprising 3,000 instances constructed from four real-world meeting corpora, covering different speakers, microphone positions, and recording devices.
- EnvSim: An environment similarity metric based on factorized contrastive learning that separately measures background sound and reverberation similarity.
- EnvTTS Systems: A training-free cascaded baseline and EnvTTS-DiT, a flow-matching model conditioned on separate speaker and environment references.
Experimental results show that EnvSim outperforms general-purpose audio representations in environment similarity assessment, while EnvTTS-DiT achieves better environment similarity and speech intelligibility than the cascaded baseline.
However, this paper has several limitations and ambiguities regarding the scope of evaluation, experimental setup, and conclusions drawn from the results. These issues are detailed in the Critical Weaknesses section. The paper also contains minor notational ambiguities in Section 3.2, where and are introduced as encoder outputs in Equation (9) but subsequently used to represent concatenated multi-reference sequences in Equations (11) and (12), without explicitly clarifying the relationship between these definitions.
Critical strengths
- Separating speaker identity from the recording environment has clear practical value. The paper further distinguishes between background sound and reverberation, providing a more fine-grained task formulation for environment control and evaluation.
- EnvTTS-Bench covers four real-world meeting corpora and constructs two evaluation tiers with different levels of difficulty based on microphone positions and recording devices, providing a relatively consistent evaluation framework for future research.
- By independently varying background sound and room impulse responses (RIRs) when constructing training samples, EnvSim learns representations of different environmental factors. In objective evaluations, it achieves EERs of 0.022 and 0.096 for background sound and RIR verification, respectively, and Spearman correlation coefficients of 0.621 and 0.639 with human ratings, respectively.
Critical weaknesses
- The paper models background sound and RIR as two separate dimensions of environment similarity, but does not establish whether this distinction offers advantages that cannot be achieved with a unified environment representation. Although EnvSim achieves strong results on both dimensions, the paper does not further validate whether this design improves the evaluation, selection, or environment control capabilities of TTS systems. Therefore, the practical value of this core contribution remains insufficiently justified.
- The Cascade baseline relies on SAM-Audio to extract background sound from the environment reference and estimates the SNR before mixing the extracted background with the speaker reference (Appendix C.3). Potential errors in source separation and SNR estimation may affect the resulting speech prompt, but their impact is not independently evaluated. Meanwhile, Cascade uses a frozen TTS backbone, whereas EnvTTS-DiT undergoes additional fine-tuning. These differences make it difficult to determine how much of the performance improvement in Table 4 can be attributed specifically to the proposed multi-reference conditioning mechanism.
- The paper separately evaluates background sound and reverberation similarity through human listening tests when validating EnvSim (Table 3 and Appendix A.1). However, the subjective evaluation of EnvTTS-DiT in Table 4 only reports an overall environment similarity score (MOSEnv), without separately assessing these two factors. Although Table 4 reports EnvSimBkg and EnvSimRIR as objective metrics, the absence of corresponding subjective evaluations limits the validation of EnvTTS-DiT's ability to reproduce background sound and reverberation independently.
- EnvTTS-DiT aims to reproduce both environmental factors while preserving speaker identity, but the paper does not investigate whether extending environment reproduction beyond a single factor improves or degrades speaker similarity and the fidelity of individual environmental factors. Table 4 compares EnvTTS-DiT with a background-transfer Cascade and two single-reference baselines, without direct comparisons with existing environment-controllable TTS methods or controlled evaluations under background-only and reverberation-only conditions. Consequently, the incremental benefits and potential performance trade-offs of reproducing both environmental factors remain unclear.
- The task definition only includes the target text and two audio references, whereas the actual inference process additionally requires transcripts of both reference recordings and uses the duration of the ground-truth target recording to determine the generation length (Appendix C.3 and Appendix C.4). In particular, the latter relies on prior information that is generally unavailable in practical applications, which may affect the conclusions drawn from WER and generation quality evaluations. The paper also does not clarify how the reference transcripts are obtained in practical scenarios or evaluate the impact of transcription errors, particularly when strong background noise in the environment reference makes its speech difficult to recognize.
- The paper considers background sound an important component of environment control, but its training and evaluation mainly rely on meeting recordings and background sound conditions constructed using MUSAN, without specific evaluations of non-stationary sound events such as dog barking and bird chirping. Furthermore, in the final case presented on the project demo page, the generated audio fails to preserve the clearly audible dog barking in the environment reference. This raises concerns about whether the model can effectively transfer distinctive background sound events and whether EnvSim can accurately assess the absence of such events.
Questions for authors
- Can the authors conduct experiments to verify whether speaker identity and recording environment can be independently controlled by fixing the speaker reference while varying the environment reference, and vice versa? How does the model perform across different sessions and rooms?
- Does the proposed EnvTTS framework generalize to cross-room reference conditioning, where the speaker reference and environment reference are recorded in different physical spaces?
- Why does EnvTTS-DiT achieve substantially lower speaker similarity than Clone-Spk when conditioned on both speaker and environment references?
- Could the authors evaluate the responses of EnvSimBkg and EnvSimRIR to targeted acoustic modifications, such as applying denoising and dereverberation separately to the same generated speech, to assess whether each metric is primarily sensitive to its corresponding environmental factor?
- In Equation (2), the paper defines as additive background sound, while other sections use the term background noise. Does this formulation explicitly include salient non-stationary sound events, such as animal vocalizations, or does the intended environment reproduction primarily focus on general ambient noise characteristics?
Thank the authors for their reply. Most of my concerns have been resolved. I will maintain my current score.