acceptodds
Under review as a conference paper at ICLR 2027

Empowering Unified Audio Language Modeling with Hierarchical Semantic Representations

Abstract

We present SL-Audio, an audio-language modeling framework that supports audio captioning, text-to-audio generation, and text-guided audio separation through hierarchical semantic representations. Unlike existing approaches that typically address these capabilities independently or rely on a single semantic granularity, SL-Audio organizes audio into multiple semantic layers, ranging from high-level semantic concepts to progressively finer acoustic representations. This hierarchical design enables different tasks to flexibly access representations at appropriate levels of abstraction, where higher-level representations provide semantic guidance and finer representations capture progressively richer acoustic details. Building on this hierarchy, SL-Audio enables a single LLM backbone to support audio captioning, text-to-audio generation, and text-guided audio separation within a unified framework, allowing different tasks to leverage different levels of the hierarchy according to their representation requirements. Experiments on these representative tasks demonstrate that the proposed hierarchical representations effectively support diverse audio-language tasks within a unified framework, providing an initial step toward unified audio-language modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Audio Language Modeling, Hierarchical Representation Learning, Unified Audio Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GozXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SL-Audio, a unified ALLM for genreral audio understanding, generation, and text-guided editing. The core of SL-Audio is to build the ALLM upon hierarchical audio representations (features from different CLAP layers), forming a shared semantic interface that facilitates both understanding and generation tasks. Experiments show that SL-Audio achieves superior performance on audio captioning, text-to-audio (TTA) and text-guided audio separation.

Critical strengths
  1. SL-Audio unifies audio understanding and generation into a single ALLM framework and then adapts the same framework to an editing task.
  2. SL-Audio demonstrate how shared hierarchical audio representations can benerit tasks with different semantic and acoustic demands.
Critical weaknesses

SL-Audio is designed and evaluated on general audio tasks, leaving speech tasks with richer semantic information unexplored.

Questions for authors
  1. Ambiguous experimental settings. For the systems in Tables 4 and 7, does the audio input contain all layers (f0,f1,f2f_0,f_1,f_2), or only the indicated fif_i? Was each system re-trained with the selected layers, or only the inference process is modified? If the systems were trained with different representation sets, I doubt if understanding and generation are truly unified within a single model. If the difference arises only at inference time, it is unclear whether the performance gap between Flatten-Audio and SL-Audio on captioning and editing tasks in Table 5 stem from training-inference mismatch or the distinction between coarse and fine representations.

  2. De-tokenizer input. The SL-Audio-Tokenizer de-tokenizer is trained with three input combinations: f0f_0, f0+f1f_0+f_1, and f0+f1+f2f_0+f_1+f_2. How is Flatten-Audio (f2f_2) in Table 7 evaluated when it appears to provide only f2f_2 to the de-tokenizer?

  3. Choice of hierarchical representations. Have you considered using f0f_0, f1f_1, and f2f_2 separately, instead of f0f_0, f0+f1f_0+f_1, and f0+f1+f2f_0+f_1+f_2, as the multi-layer representations for the tokenizer and the audio-language model? I understand that the hierarchical design aims to support understanding, generation, and editing within one model. However, it is unclear whether the higher-level representations (f0f_0 and f1f_1) help with audio editing, particularly on the output side.

  4. Effectiveness of pretraining on audio separation Multimodal Instruct Tuning for text-guided audio seperation uses 1.4 million triplets of training data, nearly a quarter of the pretraining data amount (6.2 M pairs). Have you validated the performance of a model trained from scratch on 1.4 million triplets instruct tuning data only? This could help identify whether the task-specific performance comes from pertraining or merely from instruct tuning.

Reviewer_s6kFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SL-Audio, a unified framework for audio understanding and generation, including three tasks: audio captioning, text-to-audio generation, and text-guided audio separation. SL-Audio uses CLAP to extract features from multiple semantic layers, which the authors claim allows the model to flexibly select features from different layers for different tasks. The authars train a latent diffusion module using CLAP's multi-layer features and combine it with a VAE decoder to build a de-tokenizer. Then use an LLM as backbone to predict multi-layer feature, which are converted back into audio through the de-tokenizer, achieving a unified architecture for audio understanding and generation.

Critical strengths

1.They use CLAP to extract features from multiple semantic layers and it allows the model to capture both high-level semantic information and low-level acoustic details. 2.When predicting audio tokens, the LLM generates tokens within each layer in parallel, while different layers are generated autoregressively.

Critical weaknesses

1.There are already several unified audio understanding and generation models, such as UALM, UniAudio2, and Qwen2.5-Omni. However, the authors only compare SL-Audio with task-specific models on the three tasks and have not verified the advantages of SL-Audio over other frameworks 2.The baselines used for the three tasks are mostly much smaller models, while SL-Audio is large, with a 13B LLM as backbone. The authors should include comparisons with models of similar sizes. 3.The effectiveness of the multi-layer is not fully demonstrated, as some experiments lack ablation studies on the different layers.

Questions for authors

1.Did authors compare SL-Audio with more new audio tokenizer works, as most of the current baselines are relatively old? Also, since SL-Audio uses a continuous tokenizer, AudioLDM2-Tokenizer is the only continuous tokenizer included in the baseline. You should compare newer continuous tokenizers, such as Dasheng Tokenizer and LoSATok. 2.For t2a generation task, SL-Audio achieves a relatively high CLAP score, which is somewhat expected since CLAP is also used as encoder. However, the FAD and FD scores are worse than those older models, such as AudioGen. It is also worth noting that the reported results for AudioLDM2 differ from those in the original paper 3.Authors claim that SL-Audio is a unified framework for audio understanding and generation, but on the understanding side, they only evaluate audio captioning and do not include any audio reasoning tasks. I suggest adding benchmarks such as MMAU and MMAR, which include more fine-grained tasks like counting and comparison. These tasks would provide a better way to evaluate whether the multi-layer tokenizer could work. More importantly, rather than simply reporting an overall score, you could progressively add the three layers and examine whether each additional layer improves performance on fine-grained reasoning questions.

Reviewer_nj4DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sl-audio, an audio-language model that unifies audio captioning, text-to-audio (TTA) generation, and text-queried audio separation using continuous, multi-scale representations from a pretrained CLAP encoder and introduce a next-layer prediction mechanism to model these representations in a coarse-to-fine manner. The authors argue that different tasks benefit from different levels of semantic and acoustic granularity.

Critical strengths

The idea of exposing different levels of semantic and acoustic detail to an LLM is intuitive and potentially useful.

The separation experiments (Table 4) also provide useful evidence that progressively finer representations can improve signal-level reconstruction.

Authors evaluate the framework across three distinct audio-language tasks, demonstrating its flexibility beyond a single application.

Critical weaknesses

The entire premise of the paper is that the LLM generates audio via coarse-to-fine next layer prediction. In Table 7, when using the proposed f_0 →f_1 → f_2 next-layer prediction for TTA generation, performance collapses catastrophically - CLAP-T: 0.53 to 0.14, Inception Score (IS): 10.14 to 3.92, FD: 24.21 to 68.49, and FAD: 4.06 to 7.93. This means for text-to-audio generation, the LLM is merely regressing a single 768-d global CLAP vector (f_0), and all acoustic synthesis is actually done by the pretrained Stable Audio Open DiT de-tokenizer. It is unclear how much of the reported generation performance can be attributed to the proposed next-layer prediction mechanism.

The wording around the unified model needs to be clarified. The paper claims a unified model, but the headline results in Tables 2, 3, and 4 come from three fine-tuned checkpoints: SL-Audio-gen, SL-Audio-cap, and SL-Audio-sep. It remains unclear whether a single checkpoint can simultaneously perform all three tasks without degrading its captioning and generation capabilities.

Table 1 compares SL-Audio-Tokenizer (which passes 273 continuous 768-dimensional float vectors per 10s = 273 X 768 X 16 ~ 335 kbps) against discrete codecs: EnCodec (0.75 kbps), SemantiCodec (1.4 kbps), and DAC (1.5 kbps). Using 768-d continuous embeddings "tokens" and comparing token counts against quantized discrete indices without reporting bitrates is misleading. It would help to supplement these results with human evaluations.

Also, Table 1 and Table 2 use CLAP-A and CLAP-T respectively for evaluating semantic preservation and TTA alignment, while SL-audio tokenizer itself uses CLAP encoder. I am afraid this directly rewards the model for reconstructing its own conditioning space. Table 2, SL-Audio-gen has worse FAD (2.82) and FD (16.71) than Make-an-Audio 2 (FAD 1.07, FD 12.52) and AudioGen (FAD 1.69, FD 13.79).

There are several design choices introduced simultaneously: multiscale CLAP features, layer-specific projections, hierarchical attention masks, progressive next-layer prediction. However it does not clearly establish whether the gains arise from the hierarchical attention mechanism or simply from providing multiple pretrained CLAP feature scales. A useful baseline would be to concatenate all three feature scales and predict them using a nonhierarchical architecture with matched parameters and training data.

Questions for authors

Ln 277: “For each layer, representations from the previous layer are up-sampled and projected to initialize the conditioning inputs for the current layer corresponding to ˆf0→1, ˆf1→2 in Fig. 2.” How is 1 token (f_0) upsampled to 16 tokens f_1 and 256 tokens f_2?

Please specify the actual weights for alpha_i in Eq.3

On AudioCaps test set, SL-Audio using f_0 is reported as FAD 3.35, FD 17.30, IS 12.39 in Table 2, but SL-Audio (best) in Table 7 (500-clip subset) is FAD 4.06, FD 24.21, IS 10.14 Please clarify the exact model configurations underlying these results. Which results correspond to the pretrained SL-Audio checkpoint versus the task-specific SL-Audio-gen and SL-Audio-cap checkpoints?

Can the authors report bitrates for SL-Audio-Tokenizer and clarify whether the reconstruction advantages are retained under compression budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.