- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MOSS-Audio aims to provide unified understanding of speech, environmental sounds, and music by jointly capturing semantic, acoustic, and temporal information. The paper combines a dedicated audio encoder, DeepStack-based multi-layer feature injection, and staged training to develop 4B/8B models supporting tasks such as question answering, transcription, and captioning.
Critical strengths
-
The research objective is practically relevant. Jointly addressing semantic content and acoustic information across speech, environmental sounds, and music is valuable for general-purpose audio understanding.
-
The paper presents substantial system integration work. It brings together data construction, a dedicated audio encoder, multi-layer feature injection, and staged training within a unified framework, offering engineering and system-level value.
Critical weaknesses
- The paper lacks controlled comparisons showing that DeepStack outperforms simple multi-layer fusion.
- Table 5 compares only final-layer features with DeepStack, leaving it unclear whether the gains come from access to multiple encoder layers or from the layer-wise injection mechanism. Using the same encoder layers, comparable parameter counts, and identical training budgets, the authors should compare input-level feature concatenation, summation of projected layer features, and DeepStack. The motivation also relies on different encoder layers capturing complementary acoustic and semantic information, which requires supporting evidence.
- The paper lacks main-model results on public ASR and captioning benchmarks.
- The authors should report CER/WER for the main MOSS-Audio-4B/8B models on public ASR datasets, such as AISHELL-1 and LibriSpeech, and standard evaluation results on public audio captioning datasets. Comparisons with representative models should follow consistent evaluation protocols. Internal ASR results and speech-attribute descriptions demonstrate task support but do not establish the main models’ relative performance on these standard tasks.
- The calculation and validity of the speech-caption evaluation metric are insufficiently explained.
- The speech-caption reference-matching evaluation in Tables 2 and 6 needs a more concrete description: which steps involve humans, which involve models, how agreement between generated and reference descriptions is scored, and how scores are aggregated across fields. The authors should also validate the scoring accuracy and its ability to distinguish accurate, partially accurate, and incorrect descriptions. Without this evidence, it is difficult to determine whether differences in the reported scores reflect meaningful differences in description quality.
- The paper lacks a domain-specific evaluation of the main models across speech, sound, and music.
- Although the paper emphasizes unified understanding across these three domains, the aggregate results in Table 1 do not clearly reveal performance in each domain. The authors should report separate main-model results for speech, sound, and music, and include evaluations on established sound-event and music tasks. The existing component ablations do not sufficiently establish how the complete models perform on sound and music.
- Broken cross-references hinder reading and verification.
- Table or appendix references around L367 and L454–457 are unresolved
Questions for authors
See weekness
7 comments
W2: Public ASR and captioning benchmarks
The accompanying table reports ASR results for our final 4B/8B-Instruct models across 29 test conditions, including AISHELL-1/2/4, AliMeeting, WenetSpeech, and KeSpeech. MOSS-Audio-8B achieves an overall macro-average error rate of 11.29, corresponding to reductions of 20.0%, and 24.9% relative to Kimi-Audio-7B-Instruct and Qwen2.5-Omni-7B, respectively. Our 4B model also outperforms all three baselines overall, scoring 11.58.
| Dataset / Test condition | Kimi-Audio-7B-Instruct | Qwen2.5-Omni-7B | MOSS-Audio-4B-Instruct | MOSS-Audio-8B-Instruct |
|---|---|---|---|---|
| AISHELL-1 | 0.79 | 1.16 | 2.26 | 1.82 |
| AISHELL-2 (Android / iOS / Mic) | 2.91 / 3.03 / 2.88 | 2.88 / 2.77 / 2.73 | 3.22 / 3.33 / 3.20 | 2.97 / 2.95 / 2.91 |
| THCHS-30 | 1.39 | 3.06 | 3.53 | 2.82 |
| MAGICDATA-READ | 2.15 | 3.16 | 3.72 | 3.20 |
| AISHELL6-Whisper (normal / whisper) | 0.69 / 4.63 | 0.71 / 6.57 | 0.73 / 5.86 | 0.69 / 4.80 |
| AliMeeting (far / near) | 28.22 / 13.82 | 32.03 / 18.73 | 27.29 / 9.68 | 36.82 / 11.25 |
| AISHELL-4 | 20.61 | 21.01 | 20.33 | 24.36 |
| SeniorTalk (sentence) | 19.70 | 19.96 | 16.93 | 17.42 |
| ChildMandarin | 13.79 | 12.29 | 13.25 | 13.10 |
| AISHELL-6A (mild / moderate) | 7.00 / 9.34 | 7.27 / 10.94 | 6.36 / 9.77 | 5.84 / 8.94 |
| AISHELL-6A (severe / StutteringSpeech) | 12.56 / 10.75 | 12.92 / 10.53 | 12.68 / 10.28 | 11.52 / 9.72 |
| AISHELL-6B (LRDWWS / Uncontrol) | 44.44 / 42.57 | 51.99 / 49.45 | 43.35 / 44.25 | 39.76 / 39.27 |
| WenetSpeech (test-meeting) | 7.15 | 8.43 | 8.17 | 7.86 |
| Fleurs (cmn_hans_cn) | 5.10 | 5.13 | 8.13 | 7.52 |
| CS-Dialogue | 14.56 | 14.02 | 9.14 | 9.07 |
| TALCS | 12.74 | 10.46 | 8.37 | 8.22 |
| ASCEND | 21.83 | 14.42 | 12.83 | 13.26 |
| KeSpeech | 5.51 | 6.40 | 14.65 | 9.18 |
| WSYue-ASR-eval (short) | 53.17 | 57.43 | 9.04 | 8.33 |
| MIR-1K | 38.35 | 42.62 | 18.47 | 17.24 |
| open-cpop | 5.17 | 2.75 | 3.10 | 2.39 |
| MNV_17 | 4.68 | 4.56 | 4.01 | 4.31 |
| Overall (macro-average over 29 conditions) | 14.12 | 15.05 | 11.58 | 11.30 |
The advantages are particularly pronounced on diverse speech inputs. On WSYue-ASR-eval, our 8B model achieves 8.33, reducing errors by 76.0%–85.5% relative to these three baselines. On MIR-1K, it achieves 17.24, a 55.0%–59.5% reduction; on CS-Dialogue, it scores 9.07, versus 14.56/30.52/14.02. These results demonstrate strong final-model transcription capabilities, with substantial gains on Cantonese speech, singing, and conversational speech.
Regarding captioning, we prioritized broad audio understanding and fine-grained speech description. Our evaluations on MMAU, MMAU-Pro, MMAR, and MMSU already provide evidence of understanding and reasoning across speech, environmental sounds, and music. Our dedicated speech-captioning evaluation further examines vocal attributes such as accent, texture, clarity, and emotion, using 2,000 clips with human-written and reviewed references. Both model sizes outperform all evaluated baselines on the aggregate reference-matching score: 3.711/3.725, compared with 3.599 for the strongest baseline.
We appreciate the suggestion to broaden the captioning evaluation. We will supplement these results with evaluations of our final models on public audio-captioning benchmarks, including AudioCaps and Clotho.
W3: calculation and validity of speech-captioning scores.
Thank you for requesting a more explicit description. Humans establish the audio-grounded reference descriptions by listening to the clips and writing and reviewing the caption fields. The evaluated audio-language models generate predictions in the same 13-field schema. A separate text-only judge, Qwen3-235B with temperature 0, receives the field name and definition, the human reference, and the model prediction. It receives neither the audio nor the identity of the evaluated model and returns an integer score with a short justification.
Scoring rubric. The 1–5 rubric explicitly distinguishes different levels of correctness: 5, fully consistent and sufficiently specific; 4, correct main attribute with a minor omission or reduced specificity; 3, partially correct, with an incorrect or missing salient qualifier or mixed correct/incorrect cues; 2, largely incorrect with only weak or generic overlap; and 1, contradictory, off-topic, empty, unknown, or hallucinated. Clear synonyms and paraphrases are accepted, whereas lexical overlap alone is insufficient.
Aggregation. For each field, we average the scores over the examples included for that field. Macro-13 in Table 2 is their unweighted mean across all 13 fields; Macro-10 equally averages the remaining ten fields after excluding Gender, Age, and Personality. The scores remain on the original 1–5 scale.
Validity evidence. We conducted a calibration evaluation across seven open-ended dimensions, with multiple annotators following the same 1–5 scoring rubric. Our judge achieved a Spearman correlation of 0.79 with human judgments and a mean absolute deviation of 0.40 points. We additionally assessed semantic sensitivity using paraphrases, unrelated descriptions, and contradictory descriptions, which were expected to receive scores of at least 4, at most 2, and 1, respectively. The judge satisfied these thresholds in 88% of cases. These results support the scoring protocol’s sensitivity to semantic correctness and its agreement with human judgments across the calibrated dimensions.
These findings provide supporting evidence for semantic sensitivity and agreement with human judgments within our speech caption's evaluated setting.
W4
Evaluation across speech, sound, and music. Our evaluation covers all three domains using the final MOSS-Audio models. In particular, MMAU, MMAU-Pro, and MMAR include music- and sound-related questions that directly assess understanding in these domains. We additionally evaluate speech through ASR and speech captioning. Music and environmental sound are therefore already part of our main-model evaluation, beyond the component-level ablations.
We acknowledge that reporting benchmark-level aggregate scores in Table 1 makes the performance within each domain less explicit. We appreciate the reviewer’s suggestion to provide a finer-grained breakdown, which would help readers better assess domain-specific strengths and weaknesses. The proposed specialized sound-event and music tasks would provide complementary evidence and broaden the scope of our existing evaluation.
W5
Broken cross-references. Thank you for pointing out the unresolved references around Lines 367 and 454–457. We apologize for the difficulty this creates in locating and verifying the supporting experimental details. We will correct the affected references and ensure that all referenced tables and appendix sections are included in the revised PDF. We will also state the essential comparison settings and results explicitly in the main text so that the experimental evidence is easier to follow and assess.
Thank you for your careful review and constructive feedback. We appreciate your suggestions on our architectural design, evaluation coverage, and scoring methodology. Below, we address your concerns point by point and provide additional results and clarifications.
W1 Rationale for adopting DeepStack
Thank you for suggesting these comparisons. We would like to clarify the role of DeepStack in our paper: we adopt an existing cross-layer conditioning principle, explicitly credited to Meng et al. (2024), as a design choice within MOSS-Audio.
Our choice is motivated by prior empirical evidence and its interface properties. The original DeepStack study discusses sequence-wise and feature-dimension concatenation in Section 3.2 and compares these alternatives in Table 7. On its seven visual benchmarks, DeepStack achieves an average score of 51.1 versus 49.4 for Dimension Concat, with both configurations using 576 input visual tokens. This provides a concrete empirical precedent for adopting the approach.
In MOSS-Audio, we adapt this principle by retaining the final-layer audio prefix and injecting temporally aligned intermediate encoder features into early decoder layers. The effectiveness demonstrated in the original work provides a strong motivation for exploring this principle in audio. Similarly, given the depth-dependent nature of acoustic representations, incorporating intermediate features is a well-motivated design choice. To further account for the modality-specific characteristics of audio, we will supplement our existing experiments with targeted validation at the 0.6B scale.
Our Table 5 evaluates the selected configuration against the original final-layer-only interface under matched training and evaluation settings. It shows gains of 0.0222 and 0.0240 in pure-music and pure-sound DATE. These results provide direct empirical support for the benefits of the adopted configuration in non-speech captioning and characterize its domain-dependent trade-offs at the tested 0.6B scale.
We agree that comparisons using the same acoustic layers through concatenation, summation, and DeepStack would further characterize the alternatives. Retraining multiple alternatives at the full 4B/8B scales is beyond our available compute budget. We have therefore conducted controlled experiments at the 0.6B scale to examine the adopted design against the original interface, and will use this tractable scale for further audio-specific validation. By combining established evidence from prior work with controlled experiments in our setting, we aim to substantiate this architectural choice as thoroughly as our available resources permit.
DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs, 2024.