acceptodds
Under review as a conference paper at ICLR 2027

ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis

Abstract

Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2% to 2.7% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Speech synthesis, cross-modal transfer, pretrained image generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_arDvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ImageTTS repurposes the FLUX.2-klein-base-4B image-generation pipeline for zero-shot and instruction-guided speech synthesis. It converts log-mel spectrograms into pseudo-RGB images, encodes them using a frozen image VAE, and adapts the pretrained Diffusion Transformer through conditional latent flow matching. The frozen Qwen3 encoder provides transcript and instruction representations, while a VAE decoder and BigVGAN-v2 produce speech. The paper separately investigates speech reconstruction through the image VAE, the effect of image-pretrained DiT initialization, and downstream performance after scaling to approximately 100K hours of speech. Its controlled experiments support improved adaptation from pretrained initialization, and the scaled models achieve competitive linguistic accuracy and style control, with weaker speaker similarity in some settings.

Critical strengths
  1. The paper goes beyond architectural reuse and directly tests pretrained initialization. Section 4.2 compares pretrained and randomly initialized DiTs under the same architecture, representation, data, and optimization settings. At 100K updates, pretraining reduces WER from 4.2% to 2.7%. Appendix E additionally trains the random model to 150K updates, where its WER remains 3.8% despite reaching a comparable training loss. These experiments provide meaningful evidence of a useful initialization effect.

  2. The representation analysis includes informative controls and acknowledges trade-offs. Table 1 compares several frequency resolutions against a vocoder-only baseline and evaluates Riffusion’s VAE under both original and matched preprocessing. Table 2 further crosses pretrained/random initialization with image-latent/direct-mel generation. The results show that pretraining improves WER in both spaces, while image-latent modeling favors DNSMOS and direct-mel modeling favors speaker similarity. This is more informative than evaluating the complete system alone.

  3. The framework demonstrates useful downstream capabilities under a relatively transparent comparison protocol. The authors re-evaluate baselines using official implementations and a unified evaluation protocol, distinguish systems trained on comparable Emilia data from larger-data systems, and evaluate both zero-shot and instruction-guided synthesis. ImageTTS obtains the lowest text error rates among the Emilia-based systems in Table 3 and improves aggregate style accuracy over the CapTTS baselines in Table 4. The paper also explicitly acknowledges its speaker-similarity limitation.

Critical weaknesses
  1. The relationship to closely related image-to-audio transfer work is insufficiently discussed. In particular, Auffusion [1], which is not cited in the submitted manuscript, already uses RGB-like mel-spectrograms with a pretrained image VAE and diffusion backbone, and evaluates the effect of image pretraining against training from scratch (Sections 3–4 and Table 1). Although ImageTTS addresses distinct challenges in transcript-faithful and reference-conditioned speech synthesis, the paper should clarify which contributions extend beyond this established transfer approach. My concern is the positioning of the contribution, rather than that Auffusion already solves the same TTS task.

  2. The downstream comparisons do not establish a matched-resource advantage, and performance gains are task-dependent. Tables 3–4 organize comparisons mainly by speech-data volume, while ImageTTS uses a roughly 3.9B-parameter denoiser plus a frozen language encoder, image VAE, and vocoder. The paper does not provide an accompanying comparison of total model size, training compute, or end-to-end inference latency and memory. The English WER improvement over F5-TTS is small (1.12% versus 1.17%), whereas Chinese speaker similarity is lower (0.72 versus 0.76/0.77 for F5-TTS/MaskGCT). The results demonstrate competitive synthesis, but leave the quality–cost trade-off unresolved.

[1] https://arxiv.org/abs/2401.01044

Questions for authors

Why is repurposing a pretrained image-generation model for speech synthesis a necessarily superior method? Can we just simply apply an audio/speech-pretrained alternative instead? I am curious about the advantages this repurposing offers over more direct speech-oriented approaches, rather than just comparing it with random initialization.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer arDv [1/2]

Dear Reviewer arDv,

Thank you for the valuable comments! We appreciate your recognition of our contributions. Please find our response to the specific points below.

[W1] Novelty and TTS-Specific Insights Relative to Auffusion

[R1] Thank you for pointing out Auffusion. We acknowledge its contribution and now discuss it directly, repositioning our contribution from a TTS perspective.

TTS-specific setting. Auffusion generates sound events from text, whereas ImageTTS must render a specified transcript while preserving speaker identity from reference speech or following a natural-language style instruction. To meet these requirements, Section 3.2 introduces a masked-flow conditional generation framework with reference-speech conditioning and separate guidance scales for the transcript and reference speech.

TTS-specific findings. Our experiments provide three findings beyond Auffusion:

  1. Transcript accuracy. Under matched settings, image pretraining reduces WER from 4.2% to 2.7% at 100k updates; random initialization reaches 3.8% at 150k despite comparable training loss (Section 4.2, Fig. 4, Appendix E).
  2. Initialization versus generation space. With direct mel modeling, pretraining still lowers WER from 4.2% to 3.1%. Image latents improve DNSMOS, whereas mel modeling improves speaker similarity (Section 4.3, Table 2).
  3. Direct reconstruction comparison. We compare Auffusion’s native HiFi-GAN and BigVGAN-v2-matched pipelines (Table W1). The results show substantial loss from native vocoding and sensitivity to preprocessing and frequency resolution, clarifying the pipeline’s limitations for speech reconstruction.

Table W1. Speech reconstruction results. LT/LS denote LibriTTS (24 kHz)/LibriSpeech (16 kHz); P/S/D/U denote PESQ/STOI/DNSMOS/UTMOS. All metrics are higher-is-better (↑). The best VAE results are in bold.

MethodLT-P↑LT-S↑LT-SIM↑LT-D↑LT-U↑LS-P↑LS-S↑LS-SIM↑LS-D↑LS-U↑
Original waveform4.641.001.003.744.014.641.001.003.934.13
Vocoder only4.431.000.983.753.994.360.990.983.893.99
Riffusion-o1.400.820.723.341.401.420.810.793.461.33
Auffusion (HiFi-GAN)†1.700.850.733.291.571.670.840.773.391.54
Auffusion-o†, F=2561.930.880.833.441.821.980.880.863.581.74
Riffusion-n, F=5122.450.920.863.562.412.510.920.893.672.28
FLUX.2 (ours), F=5123.860.990.953.723.583.790.980.963.863.55

† Newly added Auffusion reconstruction results. Riffusion-n also serves as Auffusion-n because the two systems share the same image VAE and use the same ImageTTS preprocessing and BigVGAN-v2 vocoder in this configuration.


[W2] Inference Cost and Synthesis Quality

[R2] We benchmarked F5-TTS, MaskGCT, and ImageTTS-base on the first 100 Seed-TTS test-en samples using a single 40 GB NVIDIA A100. After warm-up, we measured inference time from transcript and reference speech input to waveform output, excluding disk writes. Peak allocated GPU memory was 1.94 GB for F5-TTS, 8.71 GB for MaskGCT, and 23.35 GB for ImageTTS-base. We added the inference results to Appendix H (Table 6).

ModelNFERTF ↓SIM ↑WER (%) ↓
F5-TTS320.250.660.73
MaskGCT—0.980.691.46
ImageTTS-base50 (paper setting)5.010.650.55
ImageTTS-base202.080.650.77
ImageTTS-base101.070.630.91
ImageTTS-base50.560.661.20

At the 50-NFE setting used in the main experiments, ImageTTS-base runs slower than real time. Its 3.9B-parameter DiT and longer latent sequences contribute to the inference cost. Reducing NFE from 50 to 5 lowers the RTF from 5.01 to 0.56, while WER rises from 0.55% to 1.20%.

We additionally evaluated ImageTTS-base, F5-TTS, and MaskGCT on the Chinese portion of MiniMaxAI/TTS-Multilingual-Test-Set. ImageTTS achieved the lowest CER. Its speaker similarity was close to F5-TTS’s, and MaskGCT scored highest. We conducted a listening test with 10 utterances per model to assess speech quality (MOS) and speaker similarity (SMOS). Each listening test was conducted in a quiet lab with 10 listeners wearing headphones. The listeners’ mean age was 24 years. The table below reports MOS with 95% confidence intervals and SMOS.

ModelCER ↓SIM ↑MOS ↑SMOS ↑
F5-TTS2.50%0.7493.85 ± 0.274.08 ± 0.13
MaskGCT4.24%0.8283.77 ± 0.194.32 ± 0.20
ImageTTS-base2.40%0.7514.08 ± 0.224.11 ± 0.17
AuthorsOpenReview ↗

Author Response to Reviewer arDv [2/2]

[Q1] Image-Based vs. Speech-Specific training for TTS

[R1] Thank you for raising this important question. Our goal is to explore a new approach to building TTS systems by assessing the feasibility of transferring image-pretrained weights and quantifying the benefits of this initialization under matched training conditions. We consider the relative merits of image and speech pretraining an important research question and leave a strictly matched comparison for future work.

We offer two preliminary observations regarding the potential benefits of this repurposing. (i) The same image VAE can be applied to different acoustic representations. Besides mel spectrograms, we also tested STFT reconstruction, as shown below. (ii) In the log-mel setting, the image VAE shows a potential advantage on intrusive metrics such as PESQ and STOI, which require reference speech for evaluation. The speech-specific VAE (VibeVoice Speech Tokenizer) scores higher on UTMOS, a non-intrusive metric that evaluates the reconstructed speech alone.

MethodLT-P↑LT-S↑LT-SIM↑LT-D↑LT-U↑LS-P↑LS-S↑LS-SIM↑LS-D↑LS-U↑
VibeVoice Speech Tokenizer3.340.970.903.724.213.240.950.943.914.28
STFT in FLUX.2, F=512F=5123.260.970.983.573.503.270.970.983.713.56
FLUX.2, F=512F=5123.860.990.953.723.583.790.980.963.863.55

Dear Reviewer arDv,

Thank you for the time and effort you have devoted to reviewing our paper and for your constructive feedback.

As the paper discussion phase is approaching an end, we would be grateful to hear whether our response and the revised manuscript address your concerns. We would be happy to answer any further questions or clarify any remaining points.

Best regards,

The Authors

Reviewer_7CCmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ImageTTS, which adapts the pretrained FLUX.2 image-generation model for speech synthesis. Mel-spectrograms are converted into pseudo-RGB images and encoded using a frozen image VAE, while the pretrained Diffusion Transformer is fine-tuned using conditional flow matching. It shows that a pretrained image-generation model can be adapted to speech synthesis. The paper examines speech reconstruction, the effect of image-pretrained initialization, and performance on zero-shot and instruction-guided TTS. However, the experiments do not fully distinguish image-specific knowledge transfer from general pretraining benefits. The advantage of the image VAE over speech-specific representations and the practical computational cost also remain unclear. Despite these limitations, the experimental results support a weak acceptance.

Critical strengths
  • The paper proposes a different approach to speech synthesis by repurposing a pretrained image-generation model. It converts mel-spectrograms into pseudo-RGB images, uses the frozen FLUX.2 VAE for speech representation, and fine-tunes the pretrained DiT for speech generation. This approach explores the transfer of both representations and generative priors from images to speech.
  • The speech reconstruction results show that the frozen FLUX.2 VAE preserves speech intelligibility and speaker information. At the selected resolution, it achieves 0.99 STOI and 0.95 speaker similarity on LibriTTS, outperforming the Riffusion VAE under the evaluated settings.
  • The initialization results show that image-pretrained DiT weights improve convergence and linguistic accuracy compared with random initialization. At 100K updates, WER decreases from 4.2% to 2.7%. The improvement is also observed when using direct mel modeling, suggesting that the benefit is not limited to the image-VAE latent space.
  • The zero-shot TTS results show competitive performance against speech-native models trained on similar amounts of data. ImageTTS achieves 1.12% WER and 4.08 MOS on Seed-TTS test-en, compared with 1.17% WER and 3.33 MOS for F5-TTS. The instruction-guided results further demonstrate the model's ability to generate speech conditioned on natural-language style descriptions.
Critical weaknesses
  • The contribution of image-specific pretraining is not fully isolated. The comparison with random initialization shows that FLUX.2 weights improve speech generation. However, it does not establish whether the improvement comes from knowledge learned during image generation or from more general benefits of pretrained Transformer weights. A comparison with another pretrained DiT would help clarify this point.
  • The advantage of the image VAE over speech-specific representations remains unclear. The generation results show that image-VAE latent modeling achieves lower WER and higher DNSMOS than direct mel modeling, but direct mel modeling achieves higher speaker similarity. The two approaches also differ in their input processing and sequence lengths, so the differences cannot be attributed entirely to the VAE. A comparison with a speech-specific autoencoder under matched conditions would better establish the benefit of the image VAE.
  • The initialization results are limited to a relatively small training setting. The controlled comparison uses approximately 585 hours of LibriTTS data and evaluates 200 LibriSpeech utterances. Although the final model is trained on 100K hours, the paper does not provide a randomly initialized baseline at that scale. It is therefore unclear whether the same advantage persists with substantially more speech training data.
  • The computational cost is not sufficiently evaluated. ImageTTS uses a 4B-parameter DiT and iterative flow-matching inference. The paper does not report matched inference latency, real-time factor, or peak GPU memory against speech-native baselines such as F5-TTS and MaskGCT. These measurements would help assess the practical cost of the approach.
  • Speaker preservation and instruction-following evaluation remain limited. The zero-shot TTS results show lower speaker similarity than MaskGCT, particularly on the Chinese test set. In addition, the instruction-guided evaluation uses 200 CapSpeech examples and mainly automatic metrics. ImageTTS achieves a Style-ACC of 0.66, below VoxCPM2's 0.69. Human listening tests and more complex style instructions would help establish the model's controllability and speech quality.
Questions for authors
  • How does the frozen image VAE compare with a speech-specific autoencoder under matched compression and training conditions?
  • Does the improvement from image-pretrained initialization persist when training on the full 100K-hour speech dataset?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 7CCm [1/2]

Dear Reviewer 7CCm,

Thank you for the valuable comments and encouraging feedback! We appreciate your recognition of our contributions. Please find our response to the specific points below.


[W1] Clarifying the Scope of the Initialization Study

[R1] Thank you for raising this valuable concern. Rigorously disentangling image-specific knowledge from the general benefits of pretrained Transformer weights would require a model with the same architecture and scale as FLUX.2, but pretrained on another modality and then continually trained under the same speech setup. We have not found such a matched checkpoint. We will therefore narrow our conclusion to state that FLUX.2 initialization improves convergence and speech generation relative to random initialization, without attributing the gain specifically to image knowledge. We will consider the suggested comparison as important future work.


[W2 & Q1] Comparison with speech-specific autoencoder

[R2] Thank you for this valuable comment. Our claim is to demonstrate the feasibility of repurposing visually pretrained models for TTS, rather than to claim that image VAEs outperform speech-specific representations. We agree that the current comparison is confounded by different token budgets: direct Mel modeling uses 11.7 frames/s (approximately 293 tokens/s), whereas our image-VAE space uses 5.8 frames/s (approximately 188 tokens/s). Therefore, the lower WER and higher DNSMOS cannot be attributed solely to either the VAE representation or the sequence capacity. We leave a strictly matched comparison for future work.

We additionally provide two preliminary observations regarding the potential benefits of image VAEs: (i) the same image VAE can be applied to different acoustic representations. Besides Mel spectrograms, we also tested STFT reconstruction, as shown below. (ii) Compared with the VibeVoice Speech Tokenizer, the image VAE achieves higher reference-based PESQ/STOI, but lower no-reference UTMOS, where the speech-specific VAE remains stronger.

MethodLT-P↑LT-S↑LT-SIM↑LT-D↑LT-U↑LS-P↑LS-S↑LS-SIM↑LS-D↑LS-U↑
VibeVoice Speech Tokenizer3.340.970.903.724.213.240.950.943.914.28
STFT in FLUX.2, F=512F=5123.260.970.983.573.503.270.970.983.713.56
FLUX.2, F=512F=5123.860.990.953.723.583.790.980.963.863.55

[W3 & Q2] Initialization Comparison at Scale

[R3] Thank you for this valuable suggestion. Following it, we trained a randomly initialized 3.9B DiT on approximately 100k hours of Emilia speech. The two models use the same architecture, frozen image VAE, Qwen3 text encoder, training data, objective, optimization settings, and inference budget. Only the DiT initialization differs.

The randomly initialized model was trained for 110k updates. Full training curves and intermediate results are provided in Appendix F (Figure 6) of the revised manuscript. Consistent with our previous LibriTTS observations, image-pretrained initialization mainly accelerates early convergence, while the training-loss difference becomes small at later stages. Nevertheless, substantial downstream gaps remain at the same 110k-update budget. For content accuracy, the image-pretrained model achieves 0.9% WER, compared with 1.7% for random initialization. Moreover, the speaker-similarity gap is more pronounced at scale: the image-pretrained model achieves 0.62 SIM, compared with 0.56 for random initialization. These results show that the benefit of image-pretrained initialization persists when model capacity, data scale, and training budget are matched.

For clarity, we summarize the downstream results at all evaluated training stages below.

UpdatesRandom WER ↓Random SIM ↑Pretrained WER ↓Pretrained SIM ↑
30k11.120.504.700.55
50k6.770.513.120.57
70k4.890.541.930.60
90k2.510.551.230.61
110k1.730.560.920.62
AuthorsOpenReview ↗

Author Response to Reviewer 7CCm [2/2]

[W4] Inference Cost

[R4] Thank you for this suggestion. We tested F5-TTS, MaskGCT, and ImageTTS-base on the first 100 samples of Seed-TTS test-en using a single 40 GB NVIDIA A100 GPU. After loading and warming up each model, we measured the elapsed time from when the model received the target transcript and reference speech until it produced the synthesized waveform, before the audio was saved to disk. We reset PyTorch’s peak memory statistics after warm-up and recorded the peak allocated GPU memory during inference. The measured peaks were 1.94 GB for F5-TTS, 8.71 GB for MaskGCT, and 23.35 GB for ImageTTS-base.

The table below reports the real-time factor (RTF), speaker similarity (SIM), and word error rate (WER), including results for ImageTTS-base at different numbers of function evaluations (NFE). We have added this analysis to Appendix H of the revised manuscript (see Table 6).

ModelNFERTF ↓SIM ↑WER (%) ↓
F5-TTS320.250.660.73
MaskGCT—0.980.691.46
ImageTTS-base50 (paper setting)5.010.650.55
ImageTTS-base202.080.650.77
ImageTTS-base101.070.630.91
ImageTTS-base50.560.661.20

At the 50-NFE setting used in the main experiments, ImageTTS-base runs slower than real time. Its 3.9B-parameter DiT and longer latent sequences contribute to the inference cost. Reducing NFE from 50 to 5 lowers the RTF from 5.01 to 0.56, while WER rises from 0.55% to 1.20%.


[W5] Additional Evaluation

[R5]Speaker preservation. We additionally evaluated ImageTTS-base, F5-TTS, and MaskGCT on the Chinese portion of MiniMaxAI/TTS-Multilingual-Test-Set. ImageTTS achieved the lowest CER. Its speaker similarity was close to F5-TTS’s, and MaskGCT scored highest.

We also conducted a listening test with 10 utterances per model to assess speech quality (MOS) and speaker similarity (SMOS). Each listening test was conducted in a quiet lab with 10 listeners wearing headphones. The listeners’ mean age was 24 years. The table below reports MOS with 95% confidence intervals and SMOS.

ModelCER ↓SIM ↑MOS ↑SMOS ↑
F5-TTS2.50%0.7493.85 ± 0.274.08 ± 0.13
MaskGCT4.24%0.8283.77 ± 0.194.32 ± 0.20
ImageTTS-base2.40%0.7514.08 ± 0.224.11 ± 0.17

Instruction following. The evaluated CapSpeech subset already includes prompts that combine attributes such as gender, age, and emotion in a single instruction. For example, “A young woman, brimming with energy and enthusiasm, delivers her speech at a moderate pace. Her voice, pitched moderately, rises and falls in a very expressive and animated manner, making her words resonate with warmth and passion.”

We sampled five utterances per model for the listening test and evaluated speech quality (MOS) and instruction adherence (IMOS). The table shows the listening-test results alongside the automatic metrics from Table 4.

ModelInstruction dataStyle-ACC ↑CLSP-Score ↑MOS ↑IMOS ↑
PromptTTS~0.3K h0.320.454.39 ± 0.481.72 ± 0.31
VoxInstruct2.6K h0.600.504.17 ± 0.403.33 ± 0.35
FireRedTTS3-Instruct330K h0.540.484.78 ± 0.264.31 ± 0.19
Qwen3-TTS-VDN/A0.540.504.53 ± 0.314.44 ± 0.25
VoxCPM2~10K h0.690.484.39 ± 0.463.44 ± 0.39
CapTTS-AR33.6K h0.520.454.01 ± 0.422.44 ± 0.30
CapTTS-NAR33.6K h0.620.493.19 ± 0.633.31 ± 0.52
ImageTTS-Instruct9K h0.660.494.18 ± 0.554.03 ± 0.31

ImageTTS-Instruct ranked third in mean IMOS, ahead of VoxCPM2. We will report the listening-test results in the Appendix.

Dear Reviewer 7CCm,

Thank you for the time and effort you have devoted to reviewing our paper and for your constructive feedback.

As the paper discussion phase is approaching an end, we would be grateful to hear whether our response and the revised manuscript address your concerns. We would be happy to answer any further questions or clarify any remaining points.

Best regards,
The Authors

Hi Reviewer 7CCm,

Could you please check the author's replies and new results?

Thank you. AC

Reviewer_goVyOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether a pretrained text-to-image model transfers to speech synthesis beyond architecture reuse. Mels are rendered as pseudo-RGB images, encoded by the frozen FLUX image VAE, and the pretrained DiT is fine-tuned with flow matching.

Critical strengths

The initialization comparison holds architecture, latent space, data, and schedule fixed. The image prior mainly buys linguistic accuracy and convergence speed, while the frozen VAE caps speaker fidelity. The failure analysis of tokenization-induced Chinese pronunciation errors in the appendix is motivating.

Critical weaknesses

The novelty claim overlooks directly relevant prior art. Auffusion [1] already initialized a text-to-audio latent diffusion model from pretrained text-to-image (Stable Diffusion) weights and showed that this initialization substantially improves generation quality and data efficiency, which precisely the prior transfer question this paper poses as open. The contribution should be repositioned against Auffusion, and the paper needs to articulate what the TTS setting reveals that the text-to-audio setting did not.

[1] Xue J, Deng Y, Gao Y, et al. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 4700-4712.

The scaled results confound model size with the image prior. ImageTTS-base is a 3.9B DiT while the comparable data baselines F5-TTS (~0.3B) is an order smaller. The pretrained-vs-random control exists only at the 585h LibriTTS scale, where a 4B model from scratch is guaranteed to be undertrained. So the one regime where pretraining must win, and the from-scratch run appears to inherit the fine-tuning optimizer settings, which could further inflate the gap.

Any evidence separating model size from the image prior at scale — a random-init run at 100k hours even with a reduced budget, or the LibriTTS comparison repeated with a smaller DiT?

Questions for authors

mentioned above

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer goVy [1/2]

Dear Reviewer goVy,

Thank you for the valuable comments! We appreciate your recognition of our contributions. Please find our response to the specific points below.

[W1] Novelty and TTS-Specific Insights Relative to Auffusion

[R1] Thank you for pointing out Auffusion. We acknowledge its contribution and now discuss it directly, repositioning our contribution from a TTS perspective in the revised manuscript.

TTS-specific setting. Auffusion generates sound events from text, whereas ImageTTS must render a specified transcript while preserving speaker identity from reference speech or following a natural-language style instruction. To meet these requirements, Section 3.2 introduces a masked-flow conditional generation framework with reference-speech conditioning and separate guidance scales for the transcript and reference speech.

TTS-specific findings. Our experiments provide three findings beyond Auffusion:

  1. Transcript accuracy. Under matched settings, image pretraining reduces WER from 4.2% to 2.7% at 100k updates; random initialization reaches 3.8% at 150k updates despite comparable training loss (Section 4.2, Fig. 4, Appendix E).
  2. Initialization versus generation space. With direct mel modeling, pretraining still lowers WER from 4.2% to 3.1%. Image latents improve DNSMOS, whereas mel modeling improves speaker similarity (Section 4.3, Table 2).
  3. Reconstruction comparison. We compare Auffusion’s native HiFi-GAN and BigVGAN-v2-matched pipelines (Table W1). The results show substantial loss from native vocoding and sensitivity to preprocessing and frequency resolution, clarifying the pipeline’s limitations for speech reconstruction.

Table W1. Speech reconstruction results. LT/LS denote LibriTTS (24 kHz)/LibriSpeech (16 kHz); P/S/D/U denote PESQ/STOI/DNSMOS/UTMOS. All metrics are higher-is-better (↑). The best VAE results are in bold.

MethodLT-P↑LT-S↑LT-SIM↑LT-D↑LT-U↑LS-P↑LS-S↑LS-SIM↑LS-D↑LS-U↑
Original waveform4.641.001.003.744.014.641.001.003.934.13
Vocoder only4.431.000.983.753.994.360.990.983.893.99
Riffusion-o1.400.820.723.341.401.420.810.793.461.33
Auffusion (HiFi-GAN)†1.700.850.733.291.571.670.840.773.391.54
Auffusion-o†, F=2561.930.880.833.441.821.980.880.863.581.74
Riffusion-n, F=5122.450.920.863.562.412.510.920.893.672.28
FLUX.2 (ours), F=5123.860.990.953.723.583.790.980.963.863.55

† Newly added Auffusion reconstruction results. Riffusion-n also serves as Auffusion-n because the two systems share the same image VAE and use the same ImageTTS preprocessing and BigVGAN-v2 vocoder in this configuration.

[W2] Initialization Comparison at Scale

[R2] Thank you for this valuable suggestion. Following it, we trained a randomly initialized 3.9B DiT on approximately 100k hours of Emilia speech. The two models use the same architecture, frozen image VAE, Qwen3 text encoder, training data, objective, optimization settings, and inference budget. Only the DiT initialization differs.

The randomly initialized model was trained for 110k updates. Full training curves and intermediate results are provided in Appendix F (Figure 6) of the revised manuscript. Consistent with our previous LibriTTS observations, image-pretrained initialization mainly accelerates early convergence, while the training-loss difference becomes small at later stages. Nevertheless, substantial downstream gaps remain at the same 110k-update budget. For content accuracy, the image-pretrained model achieves 0.9% WER, compared with 1.7% for random initialization. Moreover, the speaker-similarity gap is more pronounced at scale: the image-pretrained model achieves 0.62 SIM, compared with 0.56 for random initialization. These results show that the benefit of image-pretrained initialization persists when model capacity, data scale, and training budget are matched.

For clarity, we summarize the downstream results at all evaluated training stages below.

UpdatesRandom WER (%) ↓Random SIM ↑Pretrained WER (%) ↓Pretrained SIM ↑
30k11.120.504.700.55
50k6.770.513.120.57
70k4.890.541.930.60
90k2.510.551.230.61
110k1.730.560.920.62
AuthorsOpenReview ↗

Author Response to Reviewer goVy [2/2]

[R2, continued] Model-Capacity Advantage over F5-TTS. We agree that ImageTTS-base has an advantage in model capacity over F5-TTS, although we selected the smallest available model in the FLUX.2-Klein family. We therefore do not claim that its advantage over F5-TTS is caused solely by image-pretrained initialization. The comparison with F5-TTS reflects the combined effects of model capacity and pretraining.

Optimizer Hyperparameters We agree with the this concern about optimization settings. We did not tune the optimizer separately for the pretrained and randomly initialized models, but followed the AdamW hyperparameter defaults of the Diffusers FLUX.2 training implementation. AdamW is initialized from scratch in both runs, and the pretrained model does not inherit any optimizer state.

We understand that separately tuning the learning rate, scheduler, and other training hyperparameters may affect and potentially improve both models. However, a fair sweep would require multiple additional 4B-model training runs, and we currently do not have sufficient GPU resources for such a sweep.We therefore leave separate hyperparameter tuning for each initialization to future work.

To provide further evidence, we include additional results from controlled runs conducted during our initial experiments, in which we varied the warm-up configuration, learning rate, effective batch size, and timestep/noise sampling while keeping the other settings fixed. Specifically, we shifted the logit-normal timestep distribution from logit_mean=-0.5 to +0.5, with logit_std=1.0. The downstream TTS metrics differ only slightly across these settings, and the differences become even smaller after 60K updates. Although these experiments do not replace dedicated optimizer tuning, they suggest that downstream performance is relatively stable and shows limited sensitivity to the warm-up, batchsize, and timestep sampling variations tested here.

Thank you for the detailed response and additional experiments. I acknowledge that the new large-scale comparison provides stronger evidence for the benefits of FLUX initialization.

However, my main concern regarding the significance of this approach remains. ImageTTS requires a substantially larger model and approximately 20× the inference time of F5-TTS, while achieving comparable synthesis quality and slightly lower speaker similarity.

While these experiments establish the feasibility of transferring image-pretrained weights to TTS, they do not demonstrate a compelling advantage over existing speech-native approaches in performance, efficiency, or practical utility.

Given prior work such as Auffusion, the paper needs to demonstrate additional benefits (beyond better intelligibility, such as more expressiveness or high reconstruction quality compared with vocoder) beyond the feasibility of image-to-speech transfer, which remain unclear in the current results.

Overall, I would maintain my original rating.

Comments on the paper

2 comments

Hi Reviewers,

The authors have posted their rebuttal along with new results. Please review their responses, engage with the authors to discuss any remaining concerns.

Best, Area Chair

AuthorsOpenReview ↗

A General Response by Authors

We sincerely thank all three reviewers for their thoughtful feedback and the Area Chair for facilitating the discussion.

We appreciate the reviewers’ recognition of the three main contributions of ImageTTS.

  • The controlled comparison of FLUX.2 and random initialization shows faster adaptation and improved transcript accuracy (goVy, 7CCm, arDv).
  • The reconstruction and representation analyses show that a frozen image VAE can preserve speech intelligibility and speaker information and characterize the trade-offs between image-latent and direct mel generation (7CCm, arDv).
  • ImageTTS achieves competitive results on zero-shot and instruction-guided speech synthesis (7CCm, arDv).

In our rebuttal, we provide point-by-point responses and present additional experiments. Specifically, we

  • clarify the relationship to Auffusion (goVy W1, arDv W1). We acknowledge Auffusion’s demonstration of image-to-audio transfer and clarify ImageTTS’s contributions by highlighting its TTS-specific requirements, design, and findings. Both reviewers acknowledge the results, and arDv explicitly confirms that our response fully addresses W1.
  • test FLUX.2 initialization at scale (goVy W2, 7CCm W3 and Q2). We add a matched comparison against a randomly initialized 3.9B DiT on approximately 100k hours of Emilia. Reviewer goVy acknowledges that this comparison provides stronger evidence for the benefits of FLUX.2 initialization.
  • clarify the benefits of image-to-speech transfer relative to speech-native approaches (7CCm W2 and Q1, arDv Q1, goVy follow-up). ImageTTS shows greater training data efficiency, achieving a slightly lower WER than F5-TTS-small (2.70% vs. 2.71%) with substantially less cumulative exposure to the same speech data (36k vs. 273k hours). For reconstruction, the same image VAE handles both log-mel and STFT spectrograms. Its log-mel pipeline scores higher than the VibeVoice Speech Tokenizer on reference-based metrics (PESQ and STOI), and VibeVoice scores higher on UTMOS.
  • extend the evaluation (7CCm W5, arDv W2, goVy follow-up). We add Chinese speech synthesis results and present listening tests that assess speech quality, speaker similarity, and instruction adherence.
  • report inference cost (7CCm W4, arDv W2, goVy follow-up). To address the reviewers’ questions, we present additional experiments measuring runtime and peak GPU memory for ImageTTS, F5-TTS, and MaskGCT. In response to concerns about RTF, we clarify two points. First, to test the feasibility of image-to-speech transfer, we use the smallest available image backbone whose VAE can faithfully reconstruct speech from spectrograms. Its 3.9B-parameter denoiser is roughly 10× the size of F5-TTS’s and 2.7× that of MaskGCT’s. Backbone size substantially affects inference cost. Optimizing the backbone for faster inference is beyond the scope of this study. Second, all our evaluations focus on offline TTS, where synthesis quality is the primary criterion. Meeting real-time latency targets is a separate research question.

The revised manuscript incorporates the additional results and discussions summarized above. We thank the reviewers again for their suggestions and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.