acceptodds
Under review as a conference paper at ICLR 2027

Learning to Organize Continuous Audio Representations for Streaming Generation

Abstract

Streaming speech generation poses a representation challenge: acoustic latents must be compact and easy to predict incrementally, yet retain the detail needed to reconstruct natural speech and music. Systems that use an external semantic encoder must maintain a relationship between acoustic and semantic representations during generation. We propose a low-frame-rate continuous audio representation that incorporates semantic structure directly into reconstructable acoustic latents. The same representation serves as a generation target and supports autoregressive audio-history modeling, without requiring an external semantic encoder at inference. Block-causal processing makes the representation suitable for streaming, while soft semantic alignment provides a content prior without removing acoustic flexibility. We further organize channels into expandable ordered prefixes, allowing a single audio encoder–decoder to provide representations at different capacities and enabling the generator to adjust its learning priorities at a fixed capacity. Experiments on Seed-TTS Chinese speech generation and speech and music reconstruction show that this design offers a unified representation for streaming generation with semantic guidance, acoustic fidelity, and adjustable capacity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
continuous audio representations, audio autoencoders, latent-space generative modeling, semantic alignment, ordered representation learning, channel-prefix training, streaming speech synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pbVBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes TPAE, an audio codec for streaming generation. TPAE uses a frozen block-causal ASR teacher to guide the acoustic latent through soft directional alignment. After that, nested channel prefixes (32~128 channels) are trained with a shared encoder and decoder. During inference, a DiTAR-style model uses the selected prefix as both prediction target and autoregressive history, without an external semantic encoder. TPAE is evaluated on a Chinese speech generation task on Seed-TTS test-zh and speech and music reconstruction.

Critical strengths

TPAE is intuitive and supporting generation/reconstruction without a semantic encoder during inference, which has practical values.

Critical weaknesses
  • The paper is difficult to follow, and I would suggest authors to significantly improve the presentation. For example, TPAE is used before its definitions.

  • Baselines are not clearly defined and more information is needed. Currently, authors just list them as "VAE", "semantic-fused VAE" etc., while critical configurations like architecture, frame rate etc. are not reported.

  • TPAE is not clearly outperforming the baselines. E.g., the VAE-64 baseline achieves the best CER performance, and that TPAE's improvements are not that big: 3.159 vs 3.097 UTMOS. Furthermore, there is no ablations to isolate the value of the alignment.

  • Experiments are finished with authors internal training datasets so the reproducibility is unknown.

  • The evaluation is also weak, main comparisons are only made on the single Chinese dataset Seed-TTS test-zh.

  • Authors claimed streaming, but there is no evaluations are done in this setting.

Questions for authors
  • Could you describe more about the baselines?
  • line 1083: What is the Flash-3B initialization model?
  • You claim streaming in the paper, have you evaluated latency and RTF of TPAE in streaming scenarios?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pbVB (Questions)

We thank the reviewer for the questions. Each answer quotes the question first; full evidence and protocols are given under the other reviews.


Question 1: Could you describe more about the baselines?

Answer 1: The VAE baselines are KL-regularized CNN codecs operating at 24 kHz with a latent stride of 2,048 samples, i.e., an 11.72 Hz latent rate, trained without any semantic or SSL supervision (verified from their training objective). We note that this latent rate is denser than the 10 Hz rate of TPAE, so the frame-rate factor favors the baselines in reconstruction comparisons. The generator configurations are listed in Table 1; all systems share the same 450k-hour internal corpus (independently packed). Full configuration details are given under Reviewer uDHT (Q1 & W1).


Question 2: Line 1083: What is the Flash-3B initialization model?

Answer 2: Flash-3B is our internal LLM backbone that provides the autoregressive skeleton of the generation system. Its role as the shared initialization, together with the full generator configuration (12-layer DiT with width 1,536, global batch size 128, and the optimizer and schedule in Table 12), is already given in Appendix E.3. A short description and citation of the Flash-3B model itself will be added in the updated PDF.


Question 3: You claim streaming in the paper; have you evaluated latency and RTF of TPAE in streaming scenarios?

Answer 3: First-chunk median latency 179 ms (p90 217 ms), per-chunk RTF median 0.772 (p90 0.828), and gapless playback for 99/100 utterances under a standard 200 ms jitter buffer, on a single 4-GPU H800 group (TPAE-96 weighted, 100 utterances). The full protocol and per-chunk distribution are given under Reviewer 2XqG (W2).

AuthorsOpenReview ↗

Response to Reviewer pbVB (Weaknesses)

We thank the reviewer for the comments. Each answer quotes the weakness first.


Weakness 1: The paper is difficult to follow; for example, TPAE is used before its definitions.

Answer 1: Thank you for pointing this out. The updated PDF will define TPAE at first mention in Section 1 and re-check the first-use ordering of all terms and abbreviations.


Weakness 2: Baselines are not clearly defined, and critical configurations like architecture and frame rate are not reported.

Answer 2: Addressed in Question 1 of our response to the questions; a self-contained baseline description table (sample rate, frame rate, objectives, generator pairing) will be added to the updated PDF.


Weakness 3: TPAE is not clearly outperforming the baselines: the VAE-64 baseline achieves the best CER, and the TPAE improvements are not that big (3.159 vs 3.097 UTMOS). Furthermore, there are no ablations to isolate the value of the alignment.

Answer 3: We respond to both points in turn.

First, on the comparison. The figures cited are from the step-7,000 snapshot; the updated evaluation gives TPAE-64 at step 7,400: UTMOS 3.232 / CER 1.407 (vs. VAE-64 3.097 / 1.331), and the human pairwise test (5 screened raters, 500 comparisons) found no statistically significant preference (p = 0.616; protocol under Reviewer 2XqG (W4 & Q5)). Our claim is that removing the external semantic encoder does not cost perceptual quality, not that the metrics dominate.

Second, on the ablations. The paper already contains ablations on every design axis:

  • Alignment strength: sweeps of the constraint strength, with and without the radius penalty (Table 2)
  • Decoder input: semantic features as decoder input vs. as supervision only (Table 2 and Appendix A.1)
  • Frame rate: latent-rate exploration (Table 3)
  • Generator capacity and normalization: shared-checkpoint capacity comparison and normalization ablation (Table 6)
  • Prefix length: 32/48/64/96/128-channel prefixes on speech and music (Tables 8-9)
  • Channel weighting: uniform vs. weighted flow-matching loss, with learning curves (Table 1 and Figure 4)
  • Sampling steps and guidance scale (Table 10)

The capacity-matched VAE-64 generator run is in progress and will be reported as soon as it completes (under Reviewer 2XqG (Q1)). What does not yet exist is a codec trained entirely without semantic supervision, which requires weeks of codec-level retraining; the cost analysis and conditional commitment are detailed under Reviewer 2XqG (W3) and uDHT (Q2 & W2).


Weakness 4: Experiments are finished with the authors internal training datasets, so the reproducibility is unknown.

Answer 4: We disclose extensive implementation details in the appendix (losses, schedules, receptive fields, hyperparameters). The remaining limitation is that the training corpus is internal and cannot be released; we will state this in the updated PDF. Training code and model weights, together with the evaluation scripts and configuration files, will be released progressively within institutional permission (see also Reviewer uDHT (T2-T4)).


Weakness 5: The evaluation is weak; main comparisons are only made on the single Chinese dataset Seed-TTS test-zh.

Answer 5: These evaluations have now been added: Seed-TTS test-en (1,088 utterances) and test-hard (400) for all three comparable systems; an expanded music evaluation on 400 clips led by FD (OpenL3); and a pairwise human listening test vs. VAE-64. Together with the streaming measurements (Question 3 above), they support the system being usable and flexible with an architecturally simple design. Full tables are given under Reviewer uDHT (T1), uDHT (O2-O4), and 2XqG (W4 & Q5). An updated PDF will include all of these.

Thank for providing the extra details. They are helpful to better understand the paper and I updated my rating (1 -> 2) accordingly.

AuthorsOpenReview ↗

Response to Reviewer pbVB

Dear Reviewer pbVB,

Thank you very much for the follow-up and for updating your rating. We are glad the additional details were helpful. Your review motivated a further round of experiments, which we posted today; in brief:

  1. Streaming evaluation (your Q3). They are non-streaming; we are streaming: on matched long-form inputs (19 s to 140 s), TPAE-64's streaming RTF (0.75) is on par with or slightly better than the non-streaming VAE-64's offline processing RTF (0.78), with first audio in 181 ms.
  2. Matched-capacity comparison. We retrained VAE-64 with the identical 12x1536 generator, data, and budget: enlarging its capacity improves UTMOS (3.097 to 3.224) but the conclusion is unchanged, the two systems are on par (CER 1.30 vs 1.407, SIM 0.715 vs 0.706), which directly addresses the attribution concern.
  3. Long-form quality. On paired long-form inputs (up to 120 s), TPAE-64 wins in every duration bucket: both VAE capacities degrade with duration, while TPAE-64 keeps speaker similarity essentially flat (0.876 to 0.886).

In the revision, we will follow your suggestions to improve the presentation so that the paper is easier to follow.

Thank you again for taking the time to re-read our work; we hope the updated materials address your remaining concerns, and we are happy to answer any further questions.

Best regards, Authors

Reviewer_uDHTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TPAE, a 10 Hz, 128-channel continuous audio autoencoder for streaming speech generation. Two stages are proposed for the training: In the base stage, a frozen block-causal CTC-trained ASR teacher with an RMS-normalized bottleneck guides the acoustic latent through a directional loss and a log-radius penalty, and the decoder only sees the acoustic latent. In the following stage, nested-prefix training (32/48/64/96/128 channels) is combined with domain-dependent prefix sampling, an auxiliary semantic prediction head on speech, and a multi-term channel-scale regularizer. A follow-up DiTAR-style model uses the selected prefix as both the DiT target and the AR history. A notable position here is that no semantic encoder is needed at inference. The paper also proposes group-wise channel weighting of the flow-matching loss. Evaluation are conducted Seed-TTS test-zh generation and speech/music reconstruction with reasonable improvments.

Critical strengths
  • The design question is clear: Can one latent serve as both DiT target and AR history? Removing the semantic encoder at inference is a practical simplification
  • Soft directional alignment with a separate radius penalty is shown to be effective over the experiments. Interesting facts are shared in Appendix A.1 that feeding semantic features into the decoder hurts reconstruction and alignment, bu using them only as supervision works better here.
  • Reconstruction improves steadily from 32 to 128 channels for both speech and music (shown in tabel 8 and 9)
Critical weaknesses
  • Baselines are not clear and not matched here. The VAE baselines are internal and not described. DiT sizes and training lengths differ across rows. So gains cannot be tied to the representation here until further aligned configraution are presented. Each component has close precedents: Semantic-VAE, Matryoshka, STAR-VAE, DC-AE 1.5, and DiTAR. Combining them is fine. But then direct comparisons with Semantic-VAE-style alignment and STAR-VAE-style truncation are needed.
  • There is no generation result for the same model without semantic alignment. Table 2 covers only early reconstruction runs.
  • The paper argues that longer prefixes add useful capacity. But TPAE-96 is worse than TPAE-64 on CER and UTMOS. In this case, the reviewer would suggest authors either add statement that llonger prefixes help reconstruction but not generation at this budget; or use more evidences from experiments to support the statement.
  • It turns out that channel weighting mainly helps early training. The CER gain is large at step 2k. It shrinks to 0.4 pp at 3k and nearly vanishes by 6k. This should be framed as faster convergence, or shown to hold at convergence.

Other issues:

  • Seed-TTS test-en, test-hard, and subjective tests are missing, which could be better if more tests are included
  • Music eval is weak here. Only 100 clips are used, with SI-SDR, MCD, and MR-STFT. SI-SDR says little about perceptual quality for GAN codecs.
  • All data are internal. “Flash-3B” is not described or cited. No code release is mentioned. With many tuned constants, reproduction would be hard.
  • “TPAE” appears in Figure 1 before it is defined.
Questions for authors
  • What are the VAE baselines? How do they relate to Base TPAE without alignment?
  • Can you report generation without alignment, with a direct 64-channel codec, and with VAE-64 on the larger DiT?
  • Does TPAE-96 overtake TPAE-64 with longer training?
7 comments
AuthorsOpenReview ↗

Response to Reviewer uDHT (Q1)

Question 1: What are the VAE baselines? How do they relate to Base TPAE without alignment?

Weakness 1 (condensed): The VAE baselines are internal and not described; DiT sizes and training lengths differ across rows; direct comparisons with Semantic-VAE-style alignment and STAR-VAE-style truncation are needed.

Answer 1: Thank you for the question. We clarify in four parts:

1. The VAE baselines and the VAE family.

  • KL-regularized continuous audio VAEs with a CNN encoder-decoder, operating at 24 kHz with a latent stride of 2,048 samples, i.e., an 11.72 Hz frame rate (denser than the 10 Hz rate of TPAE).
  • VAE-64 is acoustic-only: no SSL features, no semantic distillation, and no ASR-teacher loss; the training objective is reconstruction (mel/stft/waveform) plus adversarial loss plus KL.
  • For generation, VAE-64 is paired with a 6x1024 DiT; the VAE-128 variants use 6x1024 without a semantic encoder, or 12x1536 with a semantic encoder or semantic-fused latents (full configurations in Table 1).
  • All systems are trained on the same 450k-hour internal speech corpus.

2. Relation to Base TPAE without alignment.

  • An alignment-free Base TPAE does not exist in our work: the codec is co-designed with the ASR-teacher alignment, since unifying semantic and acoustic information in one latent is the design goal.
  • The closest existing comparison is therefore TPAE-64 vs. VAE-64, which differ in two factors: the transformer vs. CNN skeleton, and the ASR-teacher semantic shaping.
  • The skeleton factor is quantified in Table 4: under the same LibriTTS test-clean protocol, the transformer backbone reaches a directional semantic cosine of 0.496 vs. 0.345 for the CNN-based PAE, with better reconstruction.

3. Comparison with Semantic-VAE-style alignment.

  • Table 1 already contains two systems of this type: the semantic-encoder row (UTMOS 2.837) and the semantic-fused row (2.385) feed semantic features into the latent/decoder path and require the external semantic encoder at inference.
  • Both underperform the acoustic-only VAE-64 (3.097) despite larger budgets, consistent with Appendix A.1, where semantic features hurt reconstruction and alignment when used as decoder input.

4. Comparison with STAR-VAE-style truncation.

  • A truncation-based codec baseline does not exist in our work. Our design differs by construction: channels are ordered by importance during training, and low-capacity representations are obtained by selecting the ordered prefix rather than post-hoc truncation, so the selected channels stay balanced in scale and directly usable for generation.
  • We have not run a direct effectiveness comparison against post-hoc truncation; if the reviewer considers it necessary, we can add this baseline, conditional on acceptance.

These clarifications, together with the full codec-level configurations, will be incorporated in the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer uDHT (Q2)

Question 2: Can you report generation without alignment, with a direct 64-channel codec, and with VAE-64 on the larger DiT?

Weakness 2 (condensed): There is no generation result for the same model without semantic alignment; Table 2 covers only early reconstruction runs.

Answer 2: We answer each part with what exists and what does not.

1. Generation without alignment. You are right that Table 2 covers only early fixed-stage reconstruction runs; we do not claim they establish converged generation behavior. What they do show, together with Table 1, is a consistent pattern: semantic features as decoder input hurt both reconstruction and alignment (Appendix A.1), and semantic inputs never improved generation over the acoustic-only row within the VAE family. The same-architecture codec retrained without the ASR-teacher loss does not exist, and the rebuttal window does not allow the retraining (roughly 600 H800 GPU-days per cell at our measured throughput: codec retraining about 480, a 7,000-step generator run about 112, plus full evaluation). Should the paper be accepted, we will include this ablation.

2. A separately trained standalone 64-channel codec does not exist either. TPAE-64 is the 64-channel prefix of the ordered codec, and we state the trade-off rather than claiming equivalence to an independently trained codec (full-dimensional ordered vs. Base TPAE: PESQ -0.072, SI-SDR -1.091 dB on speech, Section 4.4).

3. A VAE-64 run on the 12x1536 DiT is in progress. The nearest matched evidence: both 12x1536 VAE rows (semantic encoder 2.837, fused 2.385) underperform acoustic-only VAE-64 on the smaller DiT (3.097) despite a larger budget (~2 vs. ~1 epoch), so giving the semantic-input VAE a larger DiT did not close the gap. We have launched the capacity-matched VAE-64 run on the 12x1536 DiT and will report the result as soon as it is available.

AuthorsOpenReview ↗

Response to Reviewer uDHT (Q3)

Question 3: Does TPAE-96 overtake TPAE-64 with longer training?

Weakness 4 (condensed): Channel weighting mainly helps early training; either frame it as faster convergence or show it holds at convergence.

Answer 3: You offered two paths and we took both.

New runs. We continued the weighted 96-channel arm from step 7,000 to 9,000 and re-evaluated the full Seed-TTS test-zh (2,020 utterances) -- the step 8,000/9,000 rows below are new runs conducted for this response:

ArmStep (~epoch)CER (%)SIMUTMOS
TPAE-96 weighted7,000 (~1.03)1.6990.7223.159
TPAE-96 weighted8,000 (~1.18)1.7630.7283.146
TPAE-96 weighted9,000 (~1.33)1.5570.7293.199
TPAE-96 uniform9,000 (~1.33)1.6470.7323.116
TPAE-647,400 (~1.09)1.4070.7213.232

Finding. From step 7,000 to 9,000, weighted 96-channel improves CER 1.699 to 1.557 and UTMOS 3.159 to 3.199, and its SIM (0.729) exceeds TPAE-64's (0.721). However, UTMOS at step 9,000 (3.199) does not yet overtake TPAE-64 at step 7,400 (3.232). We state the finding as: at this budget, longer prefixes add reconstruction capacity and speaker similarity, while generation-quality parity with the 64-channel prefix has not yet been reached.

On convergence. The learning curves have not converged at step 9,000. Full convergence of the 2-epoch schedule (13,574 steps) is not reachable within the response window; the 7,000-to-9,000 trend nevertheless answers the direction of your question: CER and UTMOS improve overall between the two points (with checkpoint-level fluctuation in between), and SIM has already overtaken.

CER gap magnitude. The gap you point to (1.699 vs. 1.457 at the paper's step 7,000) is 0.24 pp, about 5 utterances out of 2,020, of the same magnitude as the weighted arm's own fluctuation between adjacent checkpoints (1.699 to 1.763 to 1.557). Combined with SIM overtaking and the 0.03 UTMOS gap on a non-converged curve, our reading is that this gap is within checkpoint-to-checkpoint variation rather than a systematic disadvantage of the 96-channel prefix; the final judgment belongs to the converged point.

Channel weighting. Your description matches our own wording -- the contribution statement reads "better early content accuracy and UTMOS from tiered weighting at 96 channels." We also ran the same-step control beyond early training: weighted and uniform 96-channel arms share the codec, generator, data, and budget and differ only in the loss weights. At step 9,000, weighted leads uniform on UTMOS 3.199 vs. 3.116 and CER 1.557 vs. 1.647; on SIM uniform is slightly higher (0.732 vs. 0.729), which we report as measured. So the gain does not vanish beyond early training; we report it as faster early convergence plus a persistent advantage at this step count, with the caveat that training has not converged and we do not claim a converged-endpoint advantage.

Curves. The learning-curve figure in the paper will be extended through the new checkpoints in the updated PDF, so the weighting comparison can be read beyond early training.

Reviewer_2XqGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Transformer Prior-Aligned Autoencoder (TPAE), a continuous audio representation framework designed for streaming audio generation. The method first incorporates semantic supervision from a pretrained ASR teacher through directional alignment, encouraging the acoustic latent representation to preserve linguistic information while maintaining acoustic reconstruction capabilities. It then applies ordered prefix training, allowing a shared encoder-decoder to support multiple latent dimensions. Finally, channel-weighted flow matching is introduced to prioritize different latent channel groups during downstream speech generation. The authors evaluate the proposed framework on speech generation and speech/music reconstruction tasks, reporting competitive generation performance and improved early training results through channel weighting.

Critical strengths
  1. The proposed framework combines semantic supervision and ordered latent representations in a unified codec. Supporting multiple latent dimensions through a shared encoder-decoder is a potentially useful feature for applications with different computational requirements.
  2. The paper provides evaluations covering speech generation, speech/music reconstruction, and channel-weighting strategies. The results suggest that channel weighting can improve early generation performance under the evaluated training settings.
Critical weaknesses
  1. The main motivation of this work is to develop a continuous audio representation that is easier for generative models to learn. However, the current experiments do not provide a sufficiently controlled comparison to support this claim. In Table 1, TPAE-64 achieves a slightly higher UTMOS than VAE-64 (3.140 vs. 3.097), but a worse CER (1.457 vs. 1.331) and nearly identical speaker similarity. More importantly, TPAE-64 uses a 12-layer DiT with a hidden dimension of 1536, whereas VAE-64 uses a 6-layer DiT with a hidden dimension of 1024. The training configurations also differ across baselines. These differences make it difficult to determine whether the observed results come from the proposed representation or from generator capacity and training settings.

  2. Streaming generation is emphasized in the title and motivation. The proposed codec uses block-causal processing, limited decoder lookahead, and patch-based autoregressive generation, making the architecture compatible with incremental generation. However, the paper does not report real-time factor, first-chunk latency, per-chunk generation latency, or end-to-end inference throughput. Furthermore, the proposed TPAE generation system uses a substantially larger DiT than some VAE baselines. Although the proposed representation supports compact latent prefixes and avoids an external semantic encoder at inference, these architectural properties do not necessarily translate into lower generation latency.

  3. Appendix A.1 compares different semantic fusion and alignment configurations, but it does not include a directly comparable acoustic-only codec without semantic supervision. In particular, the configuration with zero directional alignment still uses semantic features as part of the decoder input.Therefore, these experiments do not establish whether semantic alignment itself improves acoustic reconstruction or downstream generation. A direct ablation removing semantic supervision while keeping the codec architecture and training configuration unchanged would be necessary to validate this design. Additionally, since the second training stage already includes an auxiliary semantic prediction objective, it would be useful to investigate whether the initial directional alignment provides additional benefits beyond this auxiliary supervision.

  4. The reported generation improvements are relatively modest and are primarily reflected in UTMOS. For example, TPAE-64 improves UTMOS over VAE-64 by only 0.043, while TPAE-96 with channel weighting achieves an improvement of 0.062. Meanwhile, the proposed models do not consistently outperform the baselines in CER or speaker similarity. Since UTMOS is an automatically predicted quality score rather than a direct human judgment, these small differences may not necessarily correspond to meaningful perceptual improvements. Human listening tests, particularly pairwise preference evaluations against comparable baselines, would provide stronger evidence of improved generation quality.

  5. The proposed ordered prefix training allows a shared codec to reconstruct audio using different numbers of latent channels. While this is a useful property, the reconstruction experiments mainly demonstrate that longer prefixes provide better reconstruction quality, which is expected from increased representation capacity. Furthermore, the full-dimensional Ordered TPAE achieves lower reconstruction SI-SDR than Base TPAE, suggesting that prefix training introduces a trade-off in reconstruction fidelity. It remains unclear whether ordered prefix learning improves downstream generation compared with independently trained codecs using the same latent dimensions.

Questions for authors
  1. Can the authors provide a fair comparison between TPAE and conventional VAEs using the same generator architecture, latent dimensions, and training budget?

  2. What are the practical benefits of the proposed codec for streaming generation?

  3. How much does semantic alignment contribute to the final performance? Can the authors provide an ablation without semantic supervision while keeping the codec architecture and training settings unchanged?

  4. Does ordered prefix training offer better generation performance than independently trained codecs with the same latent dimensions?

  5. Can the authors provide human evaluation to validate the improvements in generation quality?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 2XqG (Q1)

Question 1: Can the authors provide a fair comparison between TPAE and conventional VAEs using the same generator architecture, latent dimensions, and training budget?

Weakness 1 (condensed): TPAE-64 uses a larger DiT than VAE-64 and training configurations differ across rows, so gains cannot be tied to the representation.

Answer 1: We thank the reviewer for pointing out that Table 1 is not rigorous on this dimension: the baselines use smaller generators, and the training configurations differ across rows, so Table 1 alone cannot fully attribute the observed results to the proposed representation rather than to generator capacity and training settings. We agree with this concern, and we would like to clarify the following:

1. Semantically structuring the acoustic latent was a deliberate, representation-level design decision. Our goal is generation without an external semantic encoder at inference while the representation retains both semantic and acoustic potential, i.e., the acoustic latent itself is internally structured by semantics, which is what makes it easy for a generative model to learn. Evidence, stated under separate protocols: under the same LibriTTS test-clean protocol (directional cosine, Table 4), representations from a CNN-based acoustic codec score 0.345 while transformer-based semantic-aware representations score 0.496-0.500, indicating that transformer-based representations align more readily with semantic targets than CNN-based acoustic latents; and in an earlier internal audit under a different (pooled pairwise) protocol, unshaped VAE latents scored about 0.03 while the semantic target itself scored 0.68.

2. The comparisons already in Table 1 provide evidence at three levels. All generation numbers use the paper protocol, the full test-zh set (2,020 utterances); the paper reported step 7,000, and the step 8,000/9,000 results are new runs conducted for this response.

(1) Isolating latent dimension (same generator, same budget): VAE-64 and VAE-128 share the same 6x1024 DiT and roughly the same one-epoch budget. VAE-128 is worse across the board on generation (CER 1.805% vs. 1.331%, UTMOS 2.263 vs. 3.097), so generation performance tracks latent quality rather than dimension count.

(2) Same DiT, larger budget: VAE with an external semantic encoder (UTMOS 2.837) and the semantic-fused variant (2.385), both using the 12x1536 DiT with more training, still fall below TPAE-64 s7400 (3.232), with comparable CER (1.447% vs. 1.407%). This is not a single-variable capacity test. Within the VAE family, the acoustic-only 64-channel variant achieves the best UTMOS with the smallest generator: adding dimensions, semantic inputs, or capacity has never improved it.

(3) Same generator, same budget, strict control (new runs this round): within the TPAE family, channel-weighted vs. uniform 96-channel training at matched checkpoints gives UTMOS 3.146 vs. 3.054 at step 8,000 and 3.199 vs. 3.116 at step 9,000.

3. Bound on the stakes. The paired listening test shows no perceptual preference between TPAE-64 and VAE-64 (500 valid pairs after screening, two-sided binomial p=0.616p = 0.616). Under either attribution of the small UTMOS difference, generating without an external semantic encoder does not degrade perceived quality, and we do not claim perceptual superiority over the acoustic-only baseline.

4. Ongoing experiment and revision. We have launched the supplementary experiment the reviewer asks for: retraining VAE-64 with the same 12x1536 DiT as TPAE-64 and the same framework and configuration as our existing TPAE training, which gives the acoustic-only baseline the same generator capacity and isolates the representation. The run is in progress and will reach the comparison point in one to two days; we will report the result as soon as it is available. In the updated Table 1, every row will also list DiT size, training length, and guidance setting, with the two unmatched configurations explicitly marked.

AuthorsOpenReview ↗

Response to Reviewer 2XqG (Q2)

Question 2: What are the practical benefits of the proposed codec for streaming generation?

Weakness 2 (condensed): Streaming is emphasized, but no RTF, first-chunk latency, per-chunk latency, or throughput is reported; a larger DiT does not necessarily translate into lower generation latency.

Answer 2: We measured chunked streaming generation of TPAE-96 weighted on a single 4-GPU H800 group (one stream, no batching, NFE 10, CFG 2.0) on 100 Seed-TTS test-zh utterances, 200 ms chunks:

MetricMedianp90
First-chunk latency (text input to first audio chunk ready)179 ms217 ms
Subsequent chunk inter-arrival (200 ms chunks)150 ms166 ms
Streaming RTF0.7720.828

Gapless playback. With a standard 200 ms jitter buffer, playback is gapless for 99/100 utterances (2,881 streamed chunks); the single exception is one transient stall event.

Why DiT size does not change the latency structure. At 10 Hz with patch length 2, each chunk requires a fixed number of DiT evaluations (10 sampling steps, doubled by CFG) regardless of utterance length, so per-chunk latency is constant and predictable; the measured inter-arrival distribution reflects this. The absence of an external semantic encoder also removes a per-chunk dependency that encoder-conditioned systems incur at inference.

Scope. Research-prototype measurement (no batching, no distillation or inference-compilation optimizations): evidence that the block-causal design supports streaming at interactive latencies, not a production-throughput claim. We do not compare latency against the VAE baseline, which likewise uses no external encoder at inference; the structural benefit is relative to systems that require per-chunk semantic encoding. The measurement protocol and the full latency distribution will be added to the revision.

AuthorsOpenReview ↗

Response to Reviewer 2XqG (Q3)

Question 3: How much does semantic alignment contribute? Can you provide an ablation without semantic supervision under unchanged architecture and training settings?

Weakness 3 (condensed): No directly comparable acoustic-only codec without semantic supervision; the zero-alignment configuration still feeds semantic features to the decoder; the contribution of directional alignment is not established.

Answer 3: We agree that the ablation isolating the semantic contribution is the right test and that it is missing: the same architecture, data, and budget, with the directional loss disabled and the auxiliary semantic head removed.

What existing results already suggest:

(i) This design has precedent and has proven effective: Semantic-VAE, CALM, Ming-UniAudio, and SemBridge (Section 2.1), among others, all supervise an acoustic latent with semantic targets to give it content structure, which makes the representation easier for a generative model to learn. Where we differ from these systems is how semantics is used: soft directional alignment keeps semantic features as a training-time signal only, so the semantic structure is internalized in the acoustic latent rather than carried by an external encoder, and no semantic encoder is needed at inference.

(ii) Appendix A.1 shows that removing semantic features from the decoder input improves both reconstruction and alignment (PESQ 2.991 to 3.114, SI-SDR 2.656 to 4.367 dB, directional cosine 0.617 to 0.658): in our setting, semantic information serves best as supervision rather than as decoder content.

(iii) Within the VAE family in Table 1, semantic inputs never improved generation over the acoustic-only row.

Why this ablation is not included in this reply. We would like to run it, but the cost is prohibitive within the response window: the codec alone trains for more than 30,000 steps (roughly 480 H800 GPU-days per codec run at our measured throughput), and each ablation cell additionally requires a 7,000-step downstream generator run plus full evaluation (about 112 more H800 GPU-days); the decisive comparison needs at least two cells, over 1,100 H800 GPU-days in total. Should the paper be accepted, we will include this ablation, together with the interaction between the stage-2 auxiliary semantic head and the stage-1 directional alignment (a 2x2 grid over the two objectives), in the camera-ready.

Comments on the paper

2 comments
Area_Chair_Cbn8OpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

AuthorsOpenReview ↗

Summary of Changes to the Revised Manuscript

Thank you again for the constructive feedback throughout the discussion. We have uploaded a revised version of the manuscript; all changes relative to the original submission are highlighted in blue, and we will remove the color in the camera-ready version. The main changes are:

  1. Capacity-matched control (Section 4.2, Table 1): we retrained VAE-64 with the identical 12×1536 DiT generator, data, and budget as TPAE. At matched capacity the two systems are on par (CER 1.30 vs 1.407, SIM 0.715 vs 0.706, UTMOS 3.224 vs 3.232), showing that the earlier gap was not a capacity artifact.
  2. Long-form generation (new Section 4.3, Table 2): on paired inputs up to 120 s, both VAE capacities degrade sharply with duration while TPAE-64 is better in every duration bucket and keeps speaker similarity essentially flat (0.876–0.886).
  3. Streaming efficiency (new Section 4.4, Table 3): on matched long-form inputs, TPAE-64 achieves a median streaming RTF of 0.75 versus 0.78 for offline VAE-64, with first audio at 181 ms; measurement protocols are detailed in Appendix E.4.
  4. Expanded appendix evaluations: FAD-based music evaluation on 400 clips (Appendix C.1, protocol in F.2), Seed-TTS test-hard and test-en results (C.2), a human pairwise listening test under ITU-T P.808 (C.3), full generation-system configurations (E.1), and streaming/long-form measurement protocols (E.4).
  5. Presentation: we added references in the abstract and related work, cited the SI-SDR metric, and made minor clarifying edits throughout. Following the reviewers' suggestions, we also clarified the TPAE definition at first mention and added a self-contained baseline configuration table (Appendix E.1).

Detailed per-reviewer responses with the full numbers are posted in the individual threads above. We hope the revision and these responses address the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.