- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ImageTTS repurposes the FLUX.2-klein-base-4B image-generation pipeline for zero-shot and instruction-guided speech synthesis. It converts log-mel spectrograms into pseudo-RGB images, encodes them using a frozen image VAE, and adapts the pretrained Diffusion Transformer through conditional latent flow matching. The frozen Qwen3 encoder provides transcript and instruction representations, while a VAE decoder and BigVGAN-v2 produce speech. The paper separately investigates speech reconstruction through the image VAE, the effect of image-pretrained DiT initialization, and downstream performance after scaling to approximately 100K hours of speech. Its controlled experiments support improved adaptation from pretrained initialization, and the scaled models achieve competitive linguistic accuracy and style control, with weaker speaker similarity in some settings.
Critical strengths
-
The paper goes beyond architectural reuse and directly tests pretrained initialization. Section 4.2 compares pretrained and randomly initialized DiTs under the same architecture, representation, data, and optimization settings. At 100K updates, pretraining reduces WER from 4.2% to 2.7%. Appendix E additionally trains the random model to 150K updates, where its WER remains 3.8% despite reaching a comparable training loss. These experiments provide meaningful evidence of a useful initialization effect.
-
The representation analysis includes informative controls and acknowledges trade-offs. Table 1 compares several frequency resolutions against a vocoder-only baseline and evaluates Riffusion’s VAE under both original and matched preprocessing. Table 2 further crosses pretrained/random initialization with image-latent/direct-mel generation. The results show that pretraining improves WER in both spaces, while image-latent modeling favors DNSMOS and direct-mel modeling favors speaker similarity. This is more informative than evaluating the complete system alone.
-
The framework demonstrates useful downstream capabilities under a relatively transparent comparison protocol. The authors re-evaluate baselines using official implementations and a unified evaluation protocol, distinguish systems trained on comparable Emilia data from larger-data systems, and evaluate both zero-shot and instruction-guided synthesis. ImageTTS obtains the lowest text error rates among the Emilia-based systems in Table 3 and improves aggregate style accuracy over the CapTTS baselines in Table 4. The paper also explicitly acknowledges its speaker-similarity limitation.
Critical weaknesses
-
The relationship to closely related image-to-audio transfer work is insufficiently discussed. In particular, Auffusion [1], which is not cited in the submitted manuscript, already uses RGB-like mel-spectrograms with a pretrained image VAE and diffusion backbone, and evaluates the effect of image pretraining against training from scratch (Sections 3–4 and Table 1). Although ImageTTS addresses distinct challenges in transcript-faithful and reference-conditioned speech synthesis, the paper should clarify which contributions extend beyond this established transfer approach. My concern is the positioning of the contribution, rather than that Auffusion already solves the same TTS task.
-
The downstream comparisons do not establish a matched-resource advantage, and performance gains are task-dependent. Tables 3–4 organize comparisons mainly by speech-data volume, while ImageTTS uses a roughly 3.9B-parameter denoiser plus a frozen language encoder, image VAE, and vocoder. The paper does not provide an accompanying comparison of total model size, training compute, or end-to-end inference latency and memory. The English WER improvement over F5-TTS is small (1.12% versus 1.17%), whereas Chinese speaker similarity is lower (0.72 versus 0.76/0.77 for F5-TTS/MaskGCT). The results demonstrate competitive synthesis, but leave the quality–cost trade-off unresolved.
Questions for authors
Why is repurposing a pretrained image-generation model for speech synthesis a necessarily superior method? Can we just simply apply an audio/speech-pretrained alternative instead? I am curious about the advantages this repurposing offers over more direct speech-oriented approaches, rather than just comparing it with random initialization.
5 comments
Author Response to Reviewer arDv [2/2]
[Q1] Image-Based vs. Speech-Specific training for TTS
[R1] Thank you for raising this important question. Our goal is to explore a new approach to building TTS systems by assessing the feasibility of transferring image-pretrained weights and quantifying the benefits of this initialization under matched training conditions. We consider the relative merits of image and speech pretraining an important research question and leave a strictly matched comparison for future work.
We offer two preliminary observations regarding the potential benefits of this repurposing. (i) The same image VAE can be applied to different acoustic representations. Besides mel spectrograms, we also tested STFT reconstruction, as shown below. (ii) In the log-mel setting, the image VAE shows a potential advantage on intrusive metrics such as PESQ and STOI, which require reference speech for evaluation. The speech-specific VAE (VibeVoice Speech Tokenizer) scores higher on UTMOS, a non-intrusive metric that evaluates the reconstructed speech alone.
| Method | LT-P↑ | LT-S↑ | LT-SIM↑ | LT-D↑ | LT-U↑ | LS-P↑ | LS-S↑ | LS-SIM↑ | LS-D↑ | LS-U↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| VibeVoice Speech Tokenizer | 3.34 | 0.97 | 0.90 | 3.72 | 4.21 | 3.24 | 0.95 | 0.94 | 3.91 | 4.28 |
| STFT in FLUX.2, | 3.26 | 0.97 | 0.98 | 3.57 | 3.50 | 3.27 | 0.97 | 0.98 | 3.71 | 3.56 |
| FLUX.2, | 3.86 | 0.99 | 0.95 | 3.72 | 3.58 | 3.79 | 0.98 | 0.96 | 3.86 | 3.55 |
Dear Reviewer arDv,
Thank you for the time and effort you have devoted to reviewing our paper and for your constructive feedback.
As the paper discussion phase is approaching an end, we would be grateful to hear whether our response and the revised manuscript address your concerns. We would be happy to answer any further questions or clarify any remaining points.
Best regards,
The Authors
Author Response to Reviewer arDv [1/2]
Dear Reviewer arDv,
Thank you for the valuable comments! We appreciate your recognition of our contributions. Please find our response to the specific points below.
[W1] Novelty and TTS-Specific Insights Relative to Auffusion
[R1] Thank you for pointing out Auffusion. We acknowledge its contribution and now discuss it directly, repositioning our contribution from a TTS perspective.
TTS-specific setting. Auffusion generates sound events from text, whereas ImageTTS must render a specified transcript while preserving speaker identity from reference speech or following a natural-language style instruction. To meet these requirements, Section 3.2 introduces a masked-flow conditional generation framework with reference-speech conditioning and separate guidance scales for the transcript and reference speech.
TTS-specific findings. Our experiments provide three findings beyond Auffusion:
Table W1. Speech reconstruction results. LT/LS denote LibriTTS (24 kHz)/LibriSpeech (16 kHz); P/S/D/U denote PESQ/STOI/DNSMOS/UTMOS. All metrics are higher-is-better (↑). The best VAE results are in bold.
† Newly added Auffusion reconstruction results. Riffusion-n also serves as Auffusion-n because the two systems share the same image VAE and use the same ImageTTS preprocessing and BigVGAN-v2 vocoder in this configuration.
[W2] Inference Cost and Synthesis Quality
[R2] We benchmarked F5-TTS, MaskGCT, and ImageTTS-base on the first 100 Seed-TTS test-en samples using a single 40 GB NVIDIA A100. After warm-up, we measured inference time from transcript and reference speech input to waveform output, excluding disk writes. Peak allocated GPU memory was 1.94 GB for F5-TTS, 8.71 GB for MaskGCT, and 23.35 GB for ImageTTS-base. We added the inference results to Appendix H (Table 6).
At the 50-NFE setting used in the main experiments, ImageTTS-base runs slower than real time. Its 3.9B-parameter DiT and longer latent sequences contribute to the inference cost. Reducing NFE from 50 to 5 lowers the RTF from 5.01 to 0.56, while WER rises from 0.55% to 1.20%.
We additionally evaluated ImageTTS-base, F5-TTS, and MaskGCT on the Chinese portion of MiniMaxAI/TTS-Multilingual-Test-Set. ImageTTS achieved the lowest CER. Its speaker similarity was close to F5-TTS’s, and MaskGCT scored highest. We conducted a listening test with 10 utterances per model to assess speech quality (MOS) and speaker similarity (SMOS). Each listening test was conducted in a quiet lab with 10 listeners wearing headphones. The listeners’ mean age was 24 years. The table below reports MOS with 95% confidence intervals and SMOS.