- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DashengTokenizer which combines frozen semantic audio features with a small, trainable mel-spec projection. The two branches are added together, and a jointly trained waveform decoder reconstructs audio from the resulting continuous features. The same representation is evaluated for audio understanding and used as a target for conditional generation. Experiments cover speech, music, and environmental sound, including reconstruction, understanding tasks, flow-matching generation, and continuous autoregressive generation.
Critical strengths
-
The design is simple and addresses a useful problem. Adding acoustic information to an existing understanding encoder offers a direct way to make its features decodable. The method preserves the original feature dimensions and avoids a separate quantization stage.
-
The experiments demonstrate several uses of the representation. Beyond reconstructing existing audio, the paper shows that generators can predict these features and decode them into new audio.
-
The branch and loss analyses reveal meaningful trade-offs. The paper compares semantic, acoustic, and fused features, and examines how the alignment loss affects reconstruction and understanding. It also explicitly reports cases where fusion hurts performance, rather than presenting the representation as uniformly better.
Critical weaknesses
-
The benefit of additive fusion remains unclear. Fusion lowers understanding performance on 16 of 22 tasks compared with the semantic branch. Acoustic-only input also gives better reconstruction on all six metrics with the fixed decoder. These results motivate comparisons with concatenation, learned fusion, and acoustic-only generation targets.
-
The generation comparison changes several factors at once. Dashengtokenizer uses 1280-dim features, while the VAE uses 128-dim latents and a different decoder. Using the same DiT backbone therefore does not isolate the benefit of semantic features.
-
Training data and settings need more detail. The paper does not specify the composition and splits of the 282k-hour adaptation dataset. The main generation experiments also rely on a private 77k-hour dataset and an unpublished companion manuscript. This limits reproducibility and makes evaluation overlap difficult to assess.
Questions for authors
NA
6 comments
SisZ-W2: Generation controls for dimension and decoder
SisZ-W2: Generation controls for dimension and decoder
The equal-width controls show that dimensionality alone does not explain the generation gains. Mel-only, Semantic-only and DashengTokenizer all use 1280D/25Hz targets and the same Vocos decoder family, with a decoder trained for each representation. Stable-VAE provides the 128D acoustic reference:
| Representation type | Model | Dim. | AudioCaps · FAD↓ | AudioCaps · CLAP↑ | AudioCaps · GLAP↑ | MusicCaps · FAD↓ | MusicCaps · CLAP↑ | MusicCaps · GLAP↑ | LibriTTS · WER↓ (%) | LibriTTS · UTMOS-v2↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Acoustic | Stable-VAE | 128 | 4.382 | 0.323 | 5.264 | 2.904 | 0.464 | 7.621 | 33.40 | 1.488 |
| Acoustic | Mel-only | 1280 | 6.262 | 0.278 | 1.018 | 5.019 | 0.428 | 4.934 | 35.61 | 2.827 |
| Semantic | Semantic-only | 1280 | 3.284 | 0.402 | 5.940 | 2.151 | 0.513 | 8.466 | 27.90 | 2.949 |
| Joint | DashengTokenizer | 1280 | 3.201 | 0.420 | 6.582 | 1.391 | 0.499 | 12.553 | 11.04 | 3.078 |
At 1280D, Semantic-only generates better than Mel-only across the listed metrics. The joint target further improves FAD, GLAP and speech WER/UTMOS-v2; MusicCaps CLAP remains higher for Semantic-only. Stable-VAE has lower FAD and WER and higher MusicCaps CLAP than Mel-only, whereas Mel-only has higher speech UTMOS-v2. These controls show that representation content matters beyond target width. (See Tables 4–5 in the revised manuscript; Table G1 in the General Response.)
The 1280D Whisper + Mel / MingTok Unified comparison in General G1 provides a complementary generation comparison across construction routes. General R1 and U1–U3 give their full reconstruction and understanding results.
SisZ-W3: Training data and reproducibility
The approximately 282k-hour mixture is used for tokenizer adaptation. The separate 77k-hour ACAVCaps extension trains the DiT generators. The original-paper adaptation sources and sampling ratios are:
| Tokenizer adaptation sources | Reported sampling ratio |
|---|---|
| Music: Million Song Dataset; MTG-Jamendo | 21% |
| English speech: Emilia; YODAS; LibriLight; Common Voice 15 | 21% |
| Chinese speech: AISHELL-1/2/3; Emilia | 40% |
| Other-language speech: Multilingual Emilia | 10% |
| General sound: AudioSet; FSD50K; AudioCaps; CochlScene; ACAVCaps | 26% |
These are relative sampling weights, rather than each corpus's share of the total duration. Recovered supplementary training configurations select the training partitions of Emilia, YODAS, Common Voice, AISHELL-1/2/3, AudioSet, FSD50K, CochlScene, AudioCaps and ACAVCaps, and use LibriSpeech dev-clean for validation. Appendix A.3 documents these verified selections separately from the original mixture; the recovered configurations do not establish corpus-wide cross-source overlap removal.
The 77k-hour dataset extends ACAVCaps with the same conditioning format and content organization. The main DashengTokenizer and Stable-VAE generation comparisons use this same mixture. The extension is private; its format follows public ACAVCaps.
Public ACAVCaps alone supports effective conditional generation. Its training corpus is about one sixth the size of the extension. Mixed-scene alignment and speech performance remain close:
| MECAT Mixed metric | Public ACAVCaps (1×) | Extended corpus (≈6×) |
|---|---|---|
| FAD₄↓ | 2.909 | 2.279 |
| CLAP↑ | 0.480 | 0.478 |
| GLAP↑ | 9.788 | 10.214 |
| MultiFLAM↑ | 0.801 | 0.788 |
| WER↓ (%) | 22.18 | 22.03 |
| UTMOS↑ | 2.664 | 2.670 |
The public-data model also reaches AudioCaps FAD 2.992 (extension: 3.201) and LibriTTS WER 9.30% (11.04%). MusicCaps favors the extended setting in FAD (1.391 versus 5.097) and CLAP (0.499 versus 0.439). These results provide a public-data DiT-training route and show where the larger training setting remains advantageous. Section 4.4 summarizes this finding; Appendix C.3, Table C3 reports the complete comparison, including both framewise alignment measures and speech quality.
ACAVCaps is the training split and MECAT the held-out test split, as described in MiDashengLM §3.1 and the generation-data description. Appendix A.3 now specifies the original sources and sampling weights and the recovered supplementary train/validation selections. (See Section 4.4 and Table A4 in Appendix A.3 of the revised manuscript; Table G2 in the General Response.) Appendix C.3 documents the public-data generation comparison independently of the companion manuscript.
SisZ-W1: Joint representation and additive fusion
Our contribution is a joint semantic–acoustic representation that supports audio understanding and conditional generation while remaining decodable to waveform. Addition constructs this shared target without increasing the semantic feature width. We assess its utility against independently trained acoustic-only and semantic-only models.
Mel-only uses learned Mel features; Semantic-only uses frozen MiDashengLM features; Joint combines both and denotes DashengTokenizer. Each model has its own Vocos decoder. (See revised Section 3.1.)
Reconstruction uses Seed-TTS (EN); AC/MC denote AudioCaps/MusicCaps. The understanding columns are the two XARES-LLM task-group means. (See Tables 2–4 in the revised manuscript; Tables R1, U1 and G1b in the General Response.)
The joint representation approaches Mel-only in speech reconstruction, remains close to Semantic-only in adapted understanding, and improves MusicCaps FAD and LibriTTS WER over either independent target. Mel-only reconstructs better than Semantic-only, yet generates worse. MusicCaps CLAP is higher for Semantic-only (0.513 versus 0.499); the joint target improves FAD and speech results.
All D3 rows use one jointly trained DashengTokenizer model and the same fixed decoder, with a, s, or s+a as input. The models above instead have separately trained decoders. (See Table D3 in Appendix D.2 and Figure 2 in Appendix B.2 of the revised manuscript.)
X-ARES uses task-specific readouts; XARES-LLM tests a learned projector and LLM LoRA. Both retain frozen audio features, providing complementary evidence about downstream access to the representation.
The original task-specific X-ARES scores show changes after Dasheng fusion. In the matched LLM comparison, Whisper and WavLM remain close to their semantic references. Together with the Dasheng learning trajectory, this supports encoder- and readout-dependent effects.
We chose addition to preserve feature width; concatenation would increase it or require a projection. The present comparisons establish the utility of additive fusion against standalone targets, rather than its superiority over alternative fusion schemes. The complete reconstruction, understanding and generation results are in General R1, U1–U3 and G1a–G1b.