- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SL-Audio, a unified ALLM for genreral audio understanding, generation, and text-guided editing. The core of SL-Audio is to build the ALLM upon hierarchical audio representations (features from different CLAP layers), forming a shared semantic interface that facilitates both understanding and generation tasks. Experiments show that SL-Audio achieves superior performance on audio captioning, text-to-audio (TTA) and text-guided audio separation.
Critical strengths
- SL-Audio unifies audio understanding and generation into a single ALLM framework and then adapts the same framework to an editing task.
- SL-Audio demonstrate how shared hierarchical audio representations can benerit tasks with different semantic and acoustic demands.
Critical weaknesses
SL-Audio is designed and evaluated on general audio tasks, leaving speech tasks with richer semantic information unexplored.
Questions for authors
-
Ambiguous experimental settings. For the systems in Tables 4 and 7, does the audio input contain all layers (), or only the indicated ? Was each system re-trained with the selected layers, or only the inference process is modified? If the systems were trained with different representation sets, I doubt if understanding and generation are truly unified within a single model. If the difference arises only at inference time, it is unclear whether the performance gap between Flatten-Audio and SL-Audio on captioning and editing tasks in Table 5 stem from training-inference mismatch or the distinction between coarse and fine representations.
-
De-tokenizer input. The SL-Audio-Tokenizer de-tokenizer is trained with three input combinations: , , and . How is Flatten-Audio () in Table 7 evaluated when it appears to provide only to the de-tokenizer?
-
Choice of hierarchical representations. Have you considered using , , and separately, instead of , , and , as the multi-layer representations for the tokenizer and the audio-language model? I understand that the hierarchical design aims to support understanding, generation, and editing within one model. However, it is unclear whether the higher-level representations ( and ) help with audio editing, particularly on the output side.
-
Effectiveness of pretraining on audio separation Multimodal Instruct Tuning for text-guided audio seperation uses 1.4 million triplets of training data, nearly a quarter of the pretraining data amount (6.2 M pairs). Have you validated the performance of a model trained from scratch on 1.4 million triplets instruct tuning data only? This could help identify whether the task-specific performance comes from pertraining or merely from instruct tuning.