- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes TPAE, an audio codec for streaming generation. TPAE uses a frozen block-causal ASR teacher to guide the acoustic latent through soft directional alignment. After that, nested channel prefixes (32~128 channels) are trained with a shared encoder and decoder. During inference, a DiTAR-style model uses the selected prefix as both prediction target and autoregressive history, without an external semantic encoder. TPAE is evaluated on a Chinese speech generation task on Seed-TTS test-zh and speech and music reconstruction.
Critical strengths
TPAE is intuitive and supporting generation/reconstruction without a semantic encoder during inference, which has practical values.
Critical weaknesses
-
The paper is difficult to follow, and I would suggest authors to significantly improve the presentation. For example, TPAE is used before its definitions.
-
Baselines are not clearly defined and more information is needed. Currently, authors just list them as "VAE", "semantic-fused VAE" etc., while critical configurations like architecture, frame rate etc. are not reported.
-
TPAE is not clearly outperforming the baselines. E.g., the VAE-64 baseline achieves the best CER performance, and that TPAE's improvements are not that big: 3.159 vs 3.097 UTMOS. Furthermore, there is no ablations to isolate the value of the alignment.
-
Experiments are finished with authors internal training datasets so the reproducibility is unknown.
-
The evaluation is also weak, main comparisons are only made on the single Chinese dataset Seed-TTS test-zh.
-
Authors claimed streaming, but there is no evaluations are done in this setting.
Questions for authors
- Could you describe more about the baselines?
- line 1083: What is the Flash-3B initialization model?
- You claim streaming in the paper, have you evaluated latency and RTF of TPAE in streaming scenarios?
4 comments
Response to Reviewer pbVB (Weaknesses)
We thank the reviewer for the comments. Each answer quotes the weakness first.
Weakness 1: The paper is difficult to follow; for example, TPAE is used before its definitions.
Answer 1: Thank you for pointing this out. The updated PDF will define TPAE at first mention in Section 1 and re-check the first-use ordering of all terms and abbreviations.
Weakness 2: Baselines are not clearly defined, and critical configurations like architecture and frame rate are not reported.
Answer 2: Addressed in Question 1 of our response to the questions; a self-contained baseline description table (sample rate, frame rate, objectives, generator pairing) will be added to the updated PDF.
Weakness 3: TPAE is not clearly outperforming the baselines: the VAE-64 baseline achieves the best CER, and the TPAE improvements are not that big (3.159 vs 3.097 UTMOS). Furthermore, there are no ablations to isolate the value of the alignment.
Answer 3: We respond to both points in turn.
First, on the comparison. The figures cited are from the step-7,000 snapshot; the updated evaluation gives TPAE-64 at step 7,400: UTMOS 3.232 / CER 1.407 (vs. VAE-64 3.097 / 1.331), and the human pairwise test (5 screened raters, 500 comparisons) found no statistically significant preference (p = 0.616; protocol under Reviewer 2XqG (W4 & Q5)). Our claim is that removing the external semantic encoder does not cost perceptual quality, not that the metrics dominate.
Second, on the ablations. The paper already contains ablations on every design axis:
- Alignment strength: sweeps of the constraint strength, with and without the radius penalty (Table 2)
- Decoder input: semantic features as decoder input vs. as supervision only (Table 2 and Appendix A.1)
- Frame rate: latent-rate exploration (Table 3)
- Generator capacity and normalization: shared-checkpoint capacity comparison and normalization ablation (Table 6)
- Prefix length: 32/48/64/96/128-channel prefixes on speech and music (Tables 8-9)
- Channel weighting: uniform vs. weighted flow-matching loss, with learning curves (Table 1 and Figure 4)
- Sampling steps and guidance scale (Table 10)
The capacity-matched VAE-64 generator run is in progress and will be reported as soon as it completes (under Reviewer 2XqG (Q1)). What does not yet exist is a codec trained entirely without semantic supervision, which requires weeks of codec-level retraining; the cost analysis and conditional commitment are detailed under Reviewer 2XqG (W3) and uDHT (Q2 & W2).
Weakness 4: Experiments are finished with the authors internal training datasets, so the reproducibility is unknown.
Answer 4: We disclose extensive implementation details in the appendix (losses, schedules, receptive fields, hyperparameters). The remaining limitation is that the training corpus is internal and cannot be released; we will state this in the updated PDF. Training code and model weights, together with the evaluation scripts and configuration files, will be released progressively within institutional permission (see also Reviewer uDHT (T2-T4)).
Weakness 5: The evaluation is weak; main comparisons are only made on the single Chinese dataset Seed-TTS test-zh.
Answer 5: These evaluations have now been added: Seed-TTS test-en (1,088 utterances) and test-hard (400) for all three comparable systems; an expanded music evaluation on 400 clips led by FD (OpenL3); and a pairwise human listening test vs. VAE-64. Together with the streaming measurements (Question 3 above), they support the system being usable and flexible with an architecturally simple design. Full tables are given under Reviewer uDHT (T1), uDHT (O2-O4), and 2XqG (W4 & Q5). An updated PDF will include all of these.
Thank for providing the extra details. They are helpful to better understand the paper and I updated my rating (1 -> 2) accordingly.
Response to Reviewer pbVB
Dear Reviewer pbVB,
Thank you very much for the follow-up and for updating your rating. We are glad the additional details were helpful. Your review motivated a further round of experiments, which we posted today; in brief:
- Streaming evaluation (your Q3). They are non-streaming; we are streaming: on matched long-form inputs (19 s to 140 s), TPAE-64's streaming RTF (0.75) is on par with or slightly better than the non-streaming VAE-64's offline processing RTF (0.78), with first audio in 181 ms.
- Matched-capacity comparison. We retrained VAE-64 with the identical 12x1536 generator, data, and budget: enlarging its capacity improves UTMOS (3.097 to 3.224) but the conclusion is unchanged, the two systems are on par (CER 1.30 vs 1.407, SIM 0.715 vs 0.706), which directly addresses the attribution concern.
- Long-form quality. On paired long-form inputs (up to 120 s), TPAE-64 wins in every duration bucket: both VAE capacities degrade with duration, while TPAE-64 keeps speaker similarity essentially flat (0.876 to 0.886).
In the revision, we will follow your suggestions to improve the presentation so that the paper is easier to follow.
Thank you again for taking the time to re-read our work; we hope the updated materials address your remaining concerns, and we are happy to answer any further questions.
Best regards, Authors
Response to Reviewer pbVB (Questions)
We thank the reviewer for the questions. Each answer quotes the question first; full evidence and protocols are given under the other reviews.
Question 1: Could you describe more about the baselines?
Answer 1: The VAE baselines are KL-regularized CNN codecs operating at 24 kHz with a latent stride of 2,048 samples, i.e., an 11.72 Hz latent rate, trained without any semantic or SSL supervision (verified from their training objective). We note that this latent rate is denser than the 10 Hz rate of TPAE, so the frame-rate factor favors the baselines in reconstruction comparisons. The generator configurations are listed in Table 1; all systems share the same 450k-hour internal corpus (independently packed). Full configuration details are given under Reviewer uDHT (Q1 & W1).
Question 2: Line 1083: What is the Flash-3B initialization model?
Answer 2: Flash-3B is our internal LLM backbone that provides the autoregressive skeleton of the generation system. Its role as the shared initialization, together with the full generator configuration (12-layer DiT with width 1,536, global batch size 128, and the optimizer and schedule in Table 12), is already given in Appendix E.3. A short description and citation of the Flash-3B model itself will be added in the updated PDF.
Question 3: You claim streaming in the paper; have you evaluated latency and RTF of TPAE in streaming scenarios?
Answer 3: First-chunk median latency 179 ms (p90 217 ms), per-chunk RTF median 0.772 (p90 0.828), and gapless playback for 99/100 utterances under a standard 200 ms jitter buffer, on a single 4-GPU H800 group (TPAE-96 weighted, 100 utterances). The full protocol and per-chunk distribution are given under Reviewer 2XqG (W2).