acceptodds
Under review as a conference paper at ICLR 2027

DashengTokenizer: Acoustically Augmented Semantic Representations for Audio Understanding and Generation

Abstract

Audio understanding and generation place different demands on representations: accurate reconstruction does not ensure useful understanding features, while semantic features may not directly support sound synthesis. DashengTokenizer connects these capabilities by reusing continuous, high-dimensional features from a frozen audio encoder and injecting acoustics through a parallel lightweight Mel projection. The unquantized fusion preserves the original feature dimension. The projection and waveform decoder are jointly trained in one adaptation stage; understanding tasks read the fused features, while conditional generators predict the same type of representation before decoding. Experiments demonstrate competitive reconstruction across speech, music, and environmental sound and advantages over compared encoders and tokenizers on several sound and music understanding tasks. Full-sequence flow matching and continuous autoregression demonstrate use as a generation target. In multi-domain comparisons with the same DiT backbone configuration, the representation and decoder outperform an acoustic VAE scheme on text-to-audio, music, and speech metrics. Acoustic adaptation can thus extend pretrained understanding features into a continuous representation connecting reconstruction, understanding, and conditional generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
continuous audio tokenization, audio representation learning, acoustic-semantic fusion, audio understanding, audio generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SisZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DashengTokenizer which combines frozen semantic audio features with a small, trainable mel-spec projection. The two branches are added together, and a jointly trained waveform decoder reconstructs audio from the resulting continuous features. The same representation is evaluated for audio understanding and used as a target for conditional generation. Experiments cover speech, music, and environmental sound, including reconstruction, understanding tasks, flow-matching generation, and continuous autoregressive generation.

Critical strengths
  1. The design is simple and addresses a useful problem. Adding acoustic information to an existing understanding encoder offers a direct way to make its features decodable. The method preserves the original feature dimensions and avoids a separate quantization stage.

  2. The experiments demonstrate several uses of the representation. Beyond reconstructing existing audio, the paper shows that generators can predict these features and decode them into new audio.

  3. The branch and loss analyses reveal meaningful trade-offs. The paper compares semantic, acoustic, and fused features, and examines how the alignment loss affects reconstruction and understanding. It also explicitly reports cases where fusion hurts performance, rather than presenting the representation as uniformly better.

Critical weaknesses
  1. The benefit of additive fusion remains unclear. Fusion lowers understanding performance on 16 of 22 tasks compared with the semantic branch. Acoustic-only input also gives better reconstruction on all six metrics with the fixed decoder. These results motivate comparisons with concatenation, learned fusion, and acoustic-only generation targets.

  2. The generation comparison changes several factors at once. Dashengtokenizer uses 1280-dim features, while the VAE uses 128-dim latents and a different decoder. Using the same DiT backbone therefore does not isolate the benefit of semantic features.

  3. Training data and settings need more detail. The paper does not specify the composition and splits of the 282k-hour adaptation dataset. The main generation experiments also rely on a private 77k-hour dataset and an unpublished companion manuscript. This limits reproducibility and makes evaluation overlap difficult to assess.

Questions for authors

NA

6 comments
AuthorsOpenReview ↗

SisZ-W1: Joint representation and additive fusion

Our contribution is a joint semantic–acoustic representation that supports audio understanding and conditional generation while remaining decodable to waveform. Addition constructs this shared target without increasing the semantic feature width. We assess its utility against independently trained acoustic-only and semantic-only models.

Mel-only uses learned Mel features; Semantic-only uses frozen MiDashengLM features; Joint combines both and denotes DashengTokenizer. Each model has its own Vocos decoder. (See revised Section 3.1.)

RepresentationReconstruction · EN PESQ↑Reconstruction · EN STOI↑Understanding · Classification↑Understanding · ASR/captioning↑Generation · AC FAD↓Generation · MC FAD↓Generation · LibriTTS WER↓ (%)Generation · LibriTTS UTMOS-v2↑
Mel-only4.0000.9860.5380.2386.2625.01935.612.827
Semantic-only2.2080.9070.8130.6703.2842.15127.902.949
DashengTokenizer4.1250.9870.7850.6483.2011.39111.043.078

Reconstruction uses Seed-TTS (EN); AC/MC denote AudioCaps/MusicCaps. The understanding columns are the two XARES-LLM task-group means. (See Tables 2–4 in the revised manuscript; Tables R1, U1 and G1b in the General Response.)

The joint representation approaches Mel-only in speech reconstruction, remains close to Semantic-only in adapted understanding, and improves MusicCaps FAD and LibriTTS WER over either independent target. Mel-only reconstructs better than Semantic-only, yet generates worse. MusicCaps CLAP is higher for Semantic-only (0.513 versus 0.499); the joint target improves FAD and speech results.

All D3 rows use one jointly trained DashengTokenizer model and the same fixed decoder, with a, s, or s+a as input. The models above instead have separately trained decoders. (See Table D3 in Appendix D.2 and Figure 2 in Appendix B.2 of the revised manuscript.)

X-ARES uses task-specific readouts; XARES-LLM tests a learned projector and LLM LoRA. Both retain frozen audio features, providing complementary evidence about downstream access to the representation.

The original task-specific X-ARES scores show changes after Dasheng fusion. In the matched LLM comparison, Whisper and WavLM remain close to their semantic references. Together with the Dasheng learning trajectory, this supports encoder- and readout-dependent effects.

We chose addition to preserve feature width; concatenation would increase it or require a projection. The present comparisons establish the utility of additive fusion against standalone targets, rather than its superiority over alternative fusion schemes. The complete reconstruction, understanding and generation results are in General R1, U1–U3 and G1a–G1b.

AuthorsOpenReview ↗

SisZ-W2: Generation controls for dimension and decoder

SisZ-W2: Generation controls for dimension and decoder

The equal-width controls show that dimensionality alone does not explain the generation gains. Mel-only, Semantic-only and DashengTokenizer all use 1280D/25Hz targets and the same Vocos decoder family, with a decoder trained for each representation. Stable-VAE provides the 128D acoustic reference:

Representation typeModelDim.AudioCaps · FAD↓AudioCaps · CLAP↑AudioCaps · GLAP↑MusicCaps · FAD↓MusicCaps · CLAP↑MusicCaps · GLAP↑LibriTTS · WER↓ (%)LibriTTS · UTMOS-v2↑
AcousticStable-VAE1284.3820.3235.2642.9040.4647.62133.401.488
AcousticMel-only12806.2620.2781.0185.0190.4284.93435.612.827
SemanticSemantic-only12803.2840.4025.9402.1510.5138.46627.902.949
JointDashengTokenizer12803.2010.4206.5821.3910.49912.55311.043.078

At 1280D, Semantic-only generates better than Mel-only across the listed metrics. The joint target further improves FAD, GLAP and speech WER/UTMOS-v2; MusicCaps CLAP remains higher for Semantic-only. Stable-VAE has lower FAD and WER and higher MusicCaps CLAP than Mel-only, whereas Mel-only has higher speech UTMOS-v2. These controls show that representation content matters beyond target width. (See Tables 4–5 in the revised manuscript; Table G1 in the General Response.)

The 1280D Whisper + Mel / MingTok Unified comparison in General G1 provides a complementary generation comparison across construction routes. General R1 and U1–U3 give their full reconstruction and understanding results.

AuthorsOpenReview ↗

SisZ-W3: Training data and reproducibility

The approximately 282k-hour mixture is used for tokenizer adaptation. The separate 77k-hour ACAVCaps extension trains the DiT generators. The original-paper adaptation sources and sampling ratios are:

Tokenizer adaptation sourcesReported sampling ratio
Music: Million Song Dataset; MTG-Jamendo21%
English speech: Emilia; YODAS; LibriLight; Common Voice 1521%
Chinese speech: AISHELL-1/2/3; Emilia40%
Other-language speech: Multilingual Emilia10%
General sound: AudioSet; FSD50K; AudioCaps; CochlScene; ACAVCaps26%

These are relative sampling weights, rather than each corpus's share of the total duration. Recovered supplementary training configurations select the training partitions of Emilia, YODAS, Common Voice, AISHELL-1/2/3, AudioSet, FSD50K, CochlScene, AudioCaps and ACAVCaps, and use LibriSpeech dev-clean for validation. Appendix A.3 documents these verified selections separately from the original mixture; the recovered configurations do not establish corpus-wide cross-source overlap removal.

The 77k-hour dataset extends ACAVCaps with the same conditioning format and content organization. The main DashengTokenizer and Stable-VAE generation comparisons use this same mixture. The extension is private; its format follows public ACAVCaps.

Public ACAVCaps alone supports effective conditional generation. Its training corpus is about one sixth the size of the extension. Mixed-scene alignment and speech performance remain close:

MECAT Mixed metricPublic ACAVCaps (1×)Extended corpus (≈6×)
FAD₄↓2.9092.279
CLAP↑0.4800.478
GLAP↑9.78810.214
MultiFLAM↑0.8010.788
WER↓ (%)22.1822.03
UTMOS↑2.6642.670

The public-data model also reaches AudioCaps FAD 2.992 (extension: 3.201) and LibriTTS WER 9.30% (11.04%). MusicCaps favors the extended setting in FAD (1.391 versus 5.097) and CLAP (0.499 versus 0.439). These results provide a public-data DiT-training route and show where the larger training setting remains advantageous. Section 4.4 summarizes this finding; Appendix C.3, Table C3 reports the complete comparison, including both framewise alignment measures and speech quality.

ACAVCaps is the training split and MECAT the held-out test split, as described in MiDashengLM §3.1 and the generation-data description. Appendix A.3 now specifies the original sources and sampling weights and the recovered supplementary train/validation selections. (See Section 4.4 and Table A4 in Appendix A.3 of the revised manuscript; Table G2 in the General Response.) Appendix C.3 documents the public-data generation comparison independently of the companion manuscript.

Reviewer_StYBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DashengTokenizer, a continuous audio representation intended to support understanding, reconstruction, and generation. It keeps a pretrained audio encoder (from MiDashengLM-7B) frozen and adds a lightweight, trainable Mel-spectrogram projection to the encoder features. The fused 1280-d, 25 Hz features are decoded to waveforms by a Vocos decoder trained jointly with the projection; an alignment loss pulls the acoustic branch toward the semantic features. Validation covers reconstruction on speech, sound, and music; linear-probe understanding on the 22 X-ARES tasks; and text-to-audio, music, and speech generation, where a DiT trained on these features outperforms an acoustic-VAE baseline with the same configuration.

Critical strengths
  1. Tackles an important problem. Audio understanding and generation still rely on separate representations. A shared continuous representation would simplify unified audio and omni models, so the direction is well motivated.

  2. Simple, broad-coverage design. Keeping the encoder frozen and adding a 0.66M-parameter Mel branch makes the approach cheap and easy to reproduce. It is evaluated across speech, sound, and music on reconstruction, understanding, and generation.

  3. Candid trade-off analysis. The alignment-weight sweep and the decoder-input diagnostic openly expose the tension between understanding and reconstruction. This is useful evidence for the community, although the paper does not draw the corresponding conclusion.

Critical weaknesses
  1. Understanding and generation are not unified. Reconstruction comes almost entirely from the Mel branch: acoustic-only input decodes better than the fused features, while semantic-only input fails with a PESQ of 1.04. Adding that branch degrades 16 of 22 understanding tasks. The conflict persists rather than being resolved.

  2. Understanding gains are inherited from the frozen encoder. The external wins on ESC, FMA, and MAESTRO are already achieved by the frozen MiDashengLM encoder. The main comparison uses Dasheng 0.6B instead of this encoder, which misattributes the advantage to the tokenizer.

  3. Speech understanding degrades most. ASR drops by 11.6 points and intent classification by 14.9 points relative to the frozen encoder. Speech is the most important modality for unified omni models, so this undermines the paper's motivation.

  4. Not a tokenizer in the usual sense. There is no quantization and no reported bitrate, so it cannot serve as discrete tokens for an LLM. The autoregressive experiment uses a different 768-dimensional variant whose reconstruction and understanding are never evaluated.

Questions for authors
  1. Semantic-only decoder. What reconstruction quality is achieved by a decoder trained on the frozen semantic features alone, in the RAE style? This would show whether the semantic features carry any reconstruction information.

  2. Isolating the generation gain. Does generation still beat the acoustic VAE when only the semantic features or only the acoustic features are used as the target? This would show whether the gain comes from the semantic features or simply from a higher-dimensional latent.

7 comments
AuthorsOpenReview ↗

StYB-W1: Shared representation and three-task trade-offs

Our contribution is a joint semantic–acoustic representation that supports audio understanding and conditional generation while remaining decodable to waveform. The same continuous feature sequence serves all three uses. We assess this combination directly with independently trained acoustic, semantic and joint models.

Mel-only uses learned Mel features; Semantic-only uses frozen MiDashengLM features; Joint combines both and denotes DashengTokenizer. Each model has its own Vocos decoder. (See revised Section 3.1.)

RepresentationReconstruction · EN PESQ↑Reconstruction · EN STOI↑Understanding · Classification↑Understanding · ASR/captioning↑Generation · AC FAD↓Generation · MC FAD↓Generation · LibriTTS WER↓ (%)Generation · LibriTTS UTMOS-v2↑
Mel-only4.0000.9860.5380.2386.2625.01935.612.827
Semantic-only2.2080.9070.8130.6703.2842.15127.902.949
DashengTokenizer4.1250.9870.7850.6483.2011.39111.043.078

Reconstruction uses Seed-TTS (EN); AC/MC denote AudioCaps/MusicCaps. The understanding columns are the two XARES-LLM task-group means. (See Tables 2–4 in the revised manuscript; Tables R1, U1 and G1b in the General Response.)

DashengTokenizer approaches Mel-only in speech reconstruction and remains close to Semantic-only in adapted understanding. As a generation target, it lowers MusicCaps FAD and LibriTTS WER relative to either independent target. The full generation comparison also retains a trade-off: MusicCaps CLAP is 0.513 for Semantic-only versus 0.499 for the joint representation.

The original D3 result uses one jointly trained DashengTokenizer model, fixes its decoder, and supplies a, s, or s+a; it shows that this decoder relies strongly on the acoustic input. The independently trained controls above test the representations themselves. (See Table D3 in Appendix D.2 of the revised manuscript.) X-ARES and the adapted LLM evaluate task-specific readout and language-model use, respectively. W3 examines this distinction across Dasheng, Whisper and WavLM; Q1–Q2 give the requested reconstruction and generation controls.

AuthorsOpenReview ↗

StYB-W2: Pretrained semantic reference and joint contribution

The pretrained encoder supplies the semantic capability; our method contributes a joint semantic–acoustic representation for understanding and generation. We distinguish inherited understanding from the incremental effects of the joint construction, using the actual frozen MiDashengLM features as the Semantic-only reference.

The direct comparison shows what the joint construction adds:

CapabilityTest / metricSemantic-onlyDashengTokenizer
ReconstructionSeed-TTS (EN) · PESQ↑2.2084.125
ReconstructionSeed-TTS (EN) · STOI↑0.9070.987
Understandingclassification mean↑0.8130.785
UnderstandingASR and captioning mean↑0.6700.648
GenerationAudioCaps · FAD↓3.2843.201
GenerationMusicCaps · FAD↓2.1511.391
GenerationLibriTTS · WER↓27.90%11.04%

The joint model improves speech and spectral reconstruction and conditional generation over Semantic-only while keeping adapted understanding close. MusicCaps FAD decreases from 2.151 to 1.391, and LibriTTS WER from 27.90% to 11.04% with transcript-plus-description input. This combination is the contribution demonstrated by the controls. (See Tables 2–4 in the revised manuscript; Tables R1, U1 and G1 in the General Response.)

The reconstruction–generation contrast motivates introducing semantic information. Our route retains frozen semantic features and learns an acoustic complement; MingTok-Audio introduces semantics through distillation into a reconstruction-based model. (See Section 4.6 and Table 5 in the revised manuscript.)

AuthorsOpenReview ↗

StYB-W3: Speech understanding and language-model adaptation

Within the same XARES-LLM runs, longer projector/LoRA adaptation narrows the speech-task gaps between DashengTokenizer and Semantic-only.

X-ARES uses task-specific readouts; XARES-LLM tests a learned projector and LLM LoRA. Both retain frozen audio features, providing complementary evidence about downstream access to the representation.

With longer LLM adaptation, the joint/semantic task-group ratios are 99.0% for classification and 99.3% for ASR/captioning. For the DashengTokenizer family, Semantic-only uses frozen MiDashengLM features and Joint denotes the full model. The corresponding speech tasks are:

Speech taskSemantic-onlyJoint
Intent accuracy↑ (%)99.5098.92
English WER↓ (%)10.5811.70
Chinese CER↓ (%)13.5116.13

Intent accuracy differs by 0.58 percentage points; the joint representation has English WER and Chinese CER higher by 1.12 and 2.62 points, respectively.

We then extend the construction to Whisper and WavLM. The common XARES-LLM comparison averages 15 classification and five ASR/captioning test sets (scores: 0–1):

Model familyTaskSemantic-onlyJointJoint / semantic
DashengTokenizerClassification0.8130.78596.5%
ASR/captioning0.6700.64896.7%
WhisperClassification0.7510.74799.6%
ASR/captioning0.6650.666100.1%
WavLMClassification0.6840.698102.1%
ASR/captioning0.5780.57499.3%

The joint representations remain close to their semantic references, with a smaller early gap for Whisper and WavLM than for DashengTokenizer. The original DashengTokenizer X-ARES semantic/joint means are 74.55/74.06. These task-specific results complement the common-setting LLM comparison.

The encoder extensions show that acoustic supplementation does not uniformly reduce understanding. The narrowing DashengTokenizer gap with LLM adaptation is consistent with increased difficulty in using its joint features, while Whisper and WavLM show smaller differences. General U2–U3 retain the individual common-setting task scores. (See common-setting tasks in Table 3 of the revised manuscript; longer group trends in Section 4.7 and Figure 2 of Appendix B.2.)

Reviewer_4BSiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DashengTokenizer, a continuous audio representation with a good balance achieving over audio understanding and generation. The main body of the tokenizer include: frozen features from MiDashengLM-7B audio encoder at 25Hz; a parallel acoustic branch that projects non-overlapping Mel patches into feature space; a Vocos decoder with joint training. Evaluation was conducted on reconsturction quality (speech, audio, msic); linear probing on understanding tasks; text-conditioned geneeration and speech enhancement.

Critical strengths
  • The idea is rather simple and easy to reproduce at the architecture level.
  • The evaluation is comprehensive and covered the stated domains.
  • THe degradation results are reported with fair discussion (e.g., 16/22 task degration after feature fusion; wrose wer in single domain TTS etc.)
  • The alignment-weight scan in appendix D.1 shows the reconstruction-understanding tradeoff in a clean way, which is notably a good analysis.
Critical weaknesses
  • a key statemtn in the paper is that the model is shown to be a "acoustically agumented segmentic representations", however, the claim is not well supported. From acoustic perspective, the acoustic branch is essentially a linear embedding of the Mel spectrogram, and the decoder relies on it alone. Quite a few evidences supports that the Mel spectrogram is the key: (1) the branch is quite light with a single linear projection+layernorm; (2) table D3 confirsm that the decoder does not need the semantic features. Acoustic-only input reconstructs better than fusion on all six metrics. Semantic-only input gives PESQ to around 1.04. A Vocos model trained on the same 128-bin Mel is the obvious missing baseline. I expect it to match Tables 3–4.
  • The comparison with codecs and VAEs ignores the information budget. The representation is with no compression. Comparing PESQ/STOI against Mimi, XY-Tokenizer, SNAC, and DAC, or against VAEs with 64 to 128 dimensional latents, is not a like-for-like comparison. Tables 3–4 should report bitrate or latent values per second for every system. The text should drop claims of “competitive reconstruction” against bottlenecked codecs. Calling the system a “tokenizer” is also an extended definition that needs claim, given there is no quantization or compression.
  • The understanding gains belong to the frozen encoder, and fusion mostly hurts. The honest conclusion is that adding the Mel embedding costs understanding performance. The abstract and contributions should say this.
  • The generation comparison rests on a weak, under-specified baseline. Notably, “acoustic VAE” is not described: architecture, frame rate, training data, decoder/vocoder, and reconstruction quality are all missing. However, this is the only baseline for the main generation claim. The paper also never tests the main question its framing raises: does the semantic part of z help generation?
Questions for authors
  • Main issues explained in weakness, would be great if the author could provide a reasonable rebuttal to the above issues.
13 comments
AuthorsOpenReview ↗

4BSi-W1: Reconstruction: independent Mel and semantic controls

We added the requested Vocos baseline trained on the same 128-bin Mel input. It is a strong reconstruction baseline; the joint representation's added value must be assessed with understanding and generation as well. Semantic-only trains its decoder on frozen MiDashengLM audio-encoder features; DashengTokenizer is the semantic–acoustic joint model. The three models have independently trained Vocos decoders. (See Table 2 in the revised manuscript; Table R1 in the General Response.)

ModelFrame rate (Hz)Speech · ZH PESQ↑Speech · ZH STOI↑Speech · EN PESQ↑Speech · EN STOI↑Music · Mel↓Music · STFT↓Sound · Mel↓Sound · STFT↓
Mel-only254.0310.9874.0000.9860.2341.5500.2371.656
Semantic-only252.3380.9032.2080.9071.1552.8951.1502.815
DashengTokenizer254.1630.9884.1250.9870.2931.4580.3201.585

Speech: Seed-TTS (ZH/EN); music: MUSDB18; sound: AudioSet.

The table above shows that DashengTokenizer improves all eight reconstruction measures over the independently decoded Semantic-only model. Relative to Mel-only, speech PESQ is slightly higher and STOI is close (0.988/0.987 versus 0.987/0.986); Mel-only has lower Mel distances, while DashengTokenizer has lower STFT distances on both MUSDB18 and AudioSet.

D3 uses one model trained jointly with s+a and feeds its fixed decoder with a, s, or s+a; it shows that this decoder relies mainly on its acoustic input. (See Table D3 in Appendix D.2 of the revised manuscript.) This is different from asking whether frozen semantic features are decodable with their own trained decoder, as tested above. W3 and W4 examine the adapted-understanding and conditional-generation value of the same selected representations.

AuthorsOpenReview ↗

4BSi-W2: Continuous tokenization and reconstruction comparison

Original Tables 3–4, combined as revised Table 2, compare reconstruction quality at each system's native configuration. They do not establish an advantage at matched compression rates. The revision distinguishes reconstruction quality from compression efficiency. (See Tables A2 and 2 in the revised manuscript; Table R1 in the General Response.) The continuous representations use the following dimensions and frame rates:

ModelDim.Frame rate (Hz)
Stable-VAE12850
MingTok-VAE6450
MingTok Unified128050
DashengTokenizer128025

“Tokenizer” denotes a continuous audio encoding–decoding interface here. Its vector sequence can be read by downstream models or predicted as a generation target. This usage also appears in VibeVoice and MingTok-Audio, and does not require vector quantization. (See Section 3.1 of the revised manuscript.)

AuthorsOpenReview ↗

4BSi-W3: Understanding of the joint representation

The semantic capability comes from the pretrained encoder; our contribution is a joint semantic–acoustic representation for downstream understanding and generation.

X-ARES uses task-specific readouts; XARES-LLM tests a learned projector and LLM LoRA. Both retain frozen audio features, providing complementary evidence about downstream access to the representation.

For the DashengTokenizer family, Semantic-only uses frozen MiDashengLM features and Joint denotes the full model. The X-ARES macro scores (0–100) equally weight 22 tasks. (See Table B3 in Appendix B.1 of the revised manuscript.)

Model familySemantic-onlyJoint
DashengTokenizer74.5574.06

The joint representation remains close to its semantic reference overall, with task-dependent differences. We then extend the construction to Whisper and WavLM and compare all three families through the same learned LLM interface.

XARES-LLM averages 15 classification and five ASR/captioning test sets (scores: 0–1). (See task scores in Table 3 of the revised manuscript; group means and task scores in Tables U1–U3 of the General Response.)

Model familyTaskSemantic-onlyJointJoint / semantic
DashengTokenizerClassification0.8130.78596.5%
ASR/captioning0.6700.64896.7%
WhisperClassification0.7510.74799.6%
ASR/captioning0.6650.666100.1%
WavLMClassification0.6840.698102.1%
ASR/captioning0.5780.57499.3%

DashengTokenizer reaches 96.5%/96.7% of its semantic reference in the shared setting. With longer LLM adaptation, its absolute group scores reach 0.834/0.675, compared with 0.842/0.679 for Semantic-only—ratios of 99.0%/99.3%. Whisper and WavLM are already close to their respective semantic references in the common comparison.

The narrowing DashengTokenizer–Semantic-only gap during LLM adaptation, together with the close Whisper/WavLM pairs, suggests that some task-level differences reflect how readily the downstream model can use the joint features. The revised abstract and contributions describe encoder- and readout-dependent understanding, alongside the evidence for a joint representation serving understanding and generation. Section 4.3 and Appendix B provide the detailed results and learning analysis. (See Section 4.7 and Figure 2 in Appendix B.2 of the revised manuscript.)

Comments on the paper

8 comments
AuthorsOpenReview ↗

General Response: Revision Summary

We thank the reviewers for their constructive comments. The paper's central objective and method remain unchanged: DashengTokenizer constructs a continuous semantic–acoustic representation for audio understanding and generation, combining a frozen semantic encoder with a learned Mel branch and waveform decoder.

The experimental main line remains reconstruction → understanding → generation. The revisions add controlled comparisons and analysis of this same construction; encoder extensions and the comparison with semantic distillation examine its applicability.

What changed in the manuscript

FocusRevisionLocation
Framing and discussionClarify the joint-representation objective and connect the findings across the three uses.Abstract; §§1, 3, 4.7 and 5
ReconstructionAdd independently trained Mel-only and Semantic-only controls; distinguish them from fixed-decoder branch inputs.§4.2, Table 2; Appendix D.2, Table D3
UnderstandingPresent complete XARES-LLM classification, ASR and captioning results alongside X-ARES profiles and learning-dynamics analysis.§4.3, Table 3; Appendix B
GenerationCompare joint and independent targets with Stable-VAE and MingTok references on MECAT and external benchmarks.§4.4, Table 4; Appendix C.1–C.2
Encoder and route comparisonsExtend the construction to Whisper and WavLM; compare Whisper + Mel with MingTok Unified across reconstruction, understanding and generation.§§4.5–4.6; Tables 2, 3 and 5
Data and public-data trainingSpecify sources and sampling ratios; compare training on public ACAVCaps and its extended corpus.§§4.1, 4.4; Appendix A.3, Table A4; Appendix C.3, Table C3

Common questions across the reviews

Shared questionResponse focusReviewers
What does the joint representation add?Assess acoustic, semantic and joint models through reconstruction, understanding and generation together.4BSi, StYB, SisZ
How does acoustic supplementation affect understanding?Separate the pretrained semantic reference from downstream feature access; use both readout settings and learning trends.4BSi, StYB, SisZ
Are the generation comparisons informative?Detail the VAE baseline, add equal-width controls, and compare acoustic supplementation with semantic distillation.4BSi, StYB, SisZ
What does continuous tokenizer mean?Clarify the continuous encoding–decoding interface and the scope of reconstruction comparisons.4BSi, StYB
How reproducible is the training setting?Disclose training sources and sampling, and provide generation evidence from public-data training.4BSi, SisZ

Key evidence and where to find it

UseSelected evidenceManuscript
ReconstructionMel-only vs Joint: English PESQ 4.000 vs 4.125; STOI 0.986 vs 0.987.(Table 2 in the revised manuscript)
UnderstandingSemantic-only vs Joint: classification 81.32 vs 78.49; ASR 85.71 vs 81.66; captioning 54.61 vs 53.64.(Table 3 in the revised manuscript)
GenerationSemantic-only vs Joint: MusicCaps FAD 2.151 vs 1.391; LibriTTS WER 27.90% vs 11.04%.(Table 4 in the revised manuscript)

Detailed evidence follows in the individual replies. The main text presents the central comparisons and their interpretation; the appendices provide complete task/subset results, learning dynamics and implementation details.

AuthorsOpenReview ↗

General Response: Models and comparison setup

We thank the reviewers for asking what the joint representation contributes beyond a pretrained semantic encoder or an acoustic reconstruction model. We organize the additional evidence around reconstruction, understanding and conditional generation. The Dasheng controls distinguish acoustic, semantic and joint targets; Whisper and WavLM test transfer to other encoders; and Whisper + Mel versus MingTok Unified compares acoustic supplementation with semantic distillation.

The revised manuscript develops these analyses in the following locations:

RevisionWhere to find it
Independent acoustic, semantic and joint controlsSections 3 and 4.2; Table 2
Understanding and learned-readout dynamicsSection 4.3; Table 3; Appendix B
Generation, encoder extensions and distillation comparisonSections 4.4–4.6; Tables 4–5; Appendix C
Generation trained on public ACAVCapsSection 4.4; Appendix C.3, Table C3
Alignment objectives and fixed-decoder diagnosticAppendix D

Models compared

Stable-VAE is the released waveform VAE used by UniFlow-Audio, with its native waveform encoder and decoder. It is our acoustic VAE baseline.

MingTok-VAE and MingTok Unified are two outputs of the released MingTok-Audio model: the 64-dimensional VAE latent and the 1280-dimensional unified representation, respectively. MingTok introduces semantics through distillation from Whisper features.

Whisper-only uses the frozen Whisper encoder features. Whisper + Mel adds a learned Mel branch to those features using our construction. Similarly, WavLM-only uses the frozen WavLM encoder features, and WavLM + Mel applies the same acoustic supplementation to that encoder.

The DashengTokenizer group contains three independently trained models: Mel-only uses a learned Mel-patch projection without a semantic encoder; Semantic-only uses frozen MiDashengLM audio-encoder features; Joint combines both and denotes the full DashengTokenizer. Each model has its own Vocos decoder.

The encoder-only and Mel-augmented controls for Whisper and WavLM also have independently trained Vocos decoders. In the augmented models, the encoder remains frozen while the Mel projection and decoder train jointly. D3 instead uses one such jointly trained DashengTokenizer model and supplies different branch inputs to the same fixed decoder.

Bold indicates the best displayed value within each stated comparison group. WER is reported in percent; other scores retain their native scales.

AuthorsOpenReview ↗

General Response: Reconstruction and understanding overview

General Response: Reconstruction and understanding overview

Table R1: Reconstruction across speech, music and sound

(See Table 2 in the revised manuscript.)

The independently trained controls test whether the joint representation is decodable without confusing them with the original fixed-decoder branch diagnostic.

GroupModelDim.HzSpeech · ZH PESQ↑Speech · ZH STOI↑Speech · EN PESQ↑Speech · EN STOI↑Music · Mel↓Music · STFT↓Sound · Mel↓Sound · STFT↓
BaselineStable-VAE128504.0480.9903.8580.9920.2321.1290.3211.385
MingTokMingTok-VAE64504.2120.9834.0280.9830.4081.7010.4341.858
MingTokMingTok Unified1280504.2100.9804.0400.9800.4321.6160.4621.768
WhisperWhisper-only1280501.3270.8071.4310.8392.1244.4461.5393.862
WhisperWhisper + Mel1280503.8800.9763.7980.9750.4951.9010.3591.835
WavLMWavLM-only1024501.3340.8371.4060.8512.0114.9931.8013.871
WavLMWavLM + Mel1024504.1120.9874.0580.9860.2411.5550.2431.664
DashengTokenizerMel-only1280254.0310.9874.0000.9860.2341.5500.2371.656
DashengTokenizerSemantic-only1280252.3380.9032.2080.9071.1552.8951.1502.815
DashengTokenizerJoint1280254.1630.9884.1250.9870.2931.4580.3201.585

Speech: Seed-TTS (ZH/EN); music: MUSDB18; sound: AudioSet. DashengTokenizer retains the reconstruction results from the original paper; Stable-VAE and MingTok Unified also retain their originally reported values; entries absent from the original paper are existing supplementary evaluations.

DashengTokenizer reconstructs substantially better than Semantic-only and approaches Mel-only in speech reconstruction. Compared with Mel-only, its speech PESQ is slightly higher and STOI is close; Mel-only has lower Mel distances, while DashengTokenizer has lower STFT distances on music and sound. Adding the acoustic branch also substantially improves reconstruction for Whisper and WavLM. These observations motivate evaluating understanding and generation together, rather than ranking representations by reconstruction alone.

Table U1: XARES-LLM: common-setting overview

(See Table 3 and learning dynamics in Appendix B.2 of the revised manuscript.)

We keep audio features frozen and adapt the same SmolLM2-135M readout with a projector and LoRA. For each representation, classification and ASR/captioning use two separately trained multi-task projector/LoRA models. The cross-model comparison uses a common LLM adaptation budget (200K updates). Classification averages 15 test sets; ASR/captioning averages five. U2–U3 give every component score.

GroupRepresentationTask 1 · Classification↑Task 2 · ASR/captioning↑
MingTokMingTok-VAE0.4630.220
MingTokMingTok Unified0.6500.625
WhisperWhisper-only0.7510.665
WhisperWhisper + Mel0.7470.666
WavLMWavLM-only0.6840.578
WavLMWavLM + Mel0.6980.574
DashengTokenizerMel-only0.5380.238
DashengTokenizerSemantic-only0.8130.670
DashengTokenizerJoint0.7850.648

For Dasheng, the joint/semantic ratios are 96.5% and 96.7%. Extending the construction to Whisper and WavLM gives 99.6%/100.1% and 102.1%/99.3%, respectively. These paired outcomes suggest that the effect of acoustic supplementation depends on the encoder and downstream adaptation. Whisper + Mel retains task-group scores close to Whisper-only, while exceeding MingTok Unified in this common setting; the reconstruction and generation tables show their complementary strengths.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.