acceptodds
Under review as a conference paper at ICLR 2027

MOSS-Audio: Unified Audio Understanding Across Semantics, Acoustics, and Time

Abstract

Unified audio understanding requires integrating semantic content with fine-grained acoustic cues and temporal structure. We present MOSS-Audio, a family of audio-language models for unified understanding and reasoning across speech, environmental sound, and music. An audio encoder trained from scratch and DeepStack cross-layer feature injection provide the language decoder with acoustic features at multiple abstraction levels. Explicit time markers anchor features to elapsed time for timestamped transcription and time-aware question answering through autoregressive generation. Our annotation pipeline segments recordings at event boundaries and merges specialist outputs into unified captions that preserve acoustic detail and event chronology, while retaining intermediate annotations for task-specific instruction tuning. Large-scale pretraining and staged post-training yield instruction-following (Instruct) and reasoning (Thinking) variants. Across evaluation, MOSS-Audio achieves the highest average accuracy among evaluated open-source systems, including 30B-scale models. MOSS-Audio also leads evaluated open and proprietary systems in average speech-captioning score and achieves more accurate timestamp alignment than evaluated omni-model baselines. We will release model weights, training data, code, and complete training recipes to support further research in unified audio understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified audio understanding, Audio-language models, Acoustic representation learning, Temporal grounding, Audio-grounded reasoning.

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_iRMSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSS-Audio aims to provide unified understanding of speech, environmental sounds, and music by jointly capturing semantic, acoustic, and temporal information. The paper combines a dedicated audio encoder, DeepStack-based multi-layer feature injection, and staged training to develop 4B/8B models supporting tasks such as question answering, transcription, and captioning.

Critical strengths
  1. The research objective is practically relevant. Jointly addressing semantic content and acoustic information across speech, environmental sounds, and music is valuable for general-purpose audio understanding.

  2. The paper presents substantial system integration work. It brings together data construction, a dedicated audio encoder, multi-layer feature injection, and staged training within a unified framework, offering engineering and system-level value.

Critical weaknesses
  1. The paper lacks controlled comparisons showing that DeepStack outperforms simple multi-layer fusion.
  • Table 5 compares only final-layer features with DeepStack, leaving it unclear whether the gains come from access to multiple encoder layers or from the layer-wise injection mechanism. Using the same encoder layers, comparable parameter counts, and identical training budgets, the authors should compare input-level feature concatenation, summation of projected layer features, and DeepStack. The motivation also relies on different encoder layers capturing complementary acoustic and semantic information, which requires supporting evidence.
  1. The paper lacks main-model results on public ASR and captioning benchmarks.
  • The authors should report CER/WER for the main MOSS-Audio-4B/8B models on public ASR datasets, such as AISHELL-1 and LibriSpeech, and standard evaluation results on public audio captioning datasets. Comparisons with representative models should follow consistent evaluation protocols. Internal ASR results and speech-attribute descriptions demonstrate task support but do not establish the main models’ relative performance on these standard tasks.
  1. The calculation and validity of the speech-caption evaluation metric are insufficiently explained.
  • The speech-caption reference-matching evaluation in Tables 2 and 6 needs a more concrete description: which steps involve humans, which involve models, how agreement between generated and reference descriptions is scored, and how scores are aggregated across fields. The authors should also validate the scoring accuracy and its ability to distinguish accurate, partially accurate, and incorrect descriptions. Without this evidence, it is difficult to determine whether differences in the reported scores reflect meaningful differences in description quality.
  1. The paper lacks a domain-specific evaluation of the main models across speech, sound, and music.
  • Although the paper emphasizes unified understanding across these three domains, the aggregate results in Table 1 do not clearly reveal performance in each domain. The authors should report separate main-model results for speech, sound, and music, and include evaluations on established sound-event and music tasks. The existing component ablations do not sufficiently establish how the complete models perform on sound and music.
  1. Broken cross-references hinder reading and verification.
  • Table or appendix references around L367 and L454–457 are unresolved
Questions for authors

See weekness

7 comments

Thank you for your careful review and constructive feedback. We appreciate your suggestions on our architectural design, evaluation coverage, and scoring methodology. Below, we address your concerns point by point and provide additional results and clarifications.

W1 Rationale for adopting DeepStack

Thank you for suggesting these comparisons. We would like to clarify the role of DeepStack in our paper: we adopt an existing cross-layer conditioning principle, explicitly credited to Meng et al. (2024), as a design choice within MOSS-Audio.

Our choice is motivated by prior empirical evidence and its interface properties. The original DeepStack study discusses sequence-wise and feature-dimension concatenation in Section 3.2 and compares these alternatives in Table 7. On its seven visual benchmarks, DeepStack achieves an average score of 51.1 versus 49.4 for Dimension Concat, with both configurations using 576 input visual tokens. This provides a concrete empirical precedent for adopting the approach.

In MOSS-Audio, we adapt this principle by retaining the final-layer audio prefix and injecting temporally aligned intermediate encoder features into early decoder layers. The effectiveness demonstrated in the original work provides a strong motivation for exploring this principle in audio. Similarly, given the depth-dependent nature of acoustic representations, incorporating intermediate features is a well-motivated design choice. To further account for the modality-specific characteristics of audio, we will supplement our existing experiments with targeted validation at the 0.6B scale.

Our Table 5 evaluates the selected configuration against the original final-layer-only interface under matched training and evaluation settings. It shows gains of 0.0222 and 0.0240 in pure-music and pure-sound DATE. These results provide direct empirical support for the benefits of the adopted configuration in non-speech captioning and characterize its domain-dependent trade-offs at the tested 0.6B scale.

We agree that comparisons using the same acoustic layers through concatenation, summation, and DeepStack would further characterize the alternatives. Retraining multiple alternatives at the full 4B/8B scales is beyond our available compute budget. We have therefore conducted controlled experiments at the 0.6B scale to examine the adopted design against the original interface, and will use this tractable scale for further audio-specific validation. By combining established evidence from prior work with controlled experiments in our setting, we aim to substantiate this architectural choice as thoroughly as our available resources permit.

DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs, 2024.

W2: Public ASR and captioning benchmarks

The accompanying table reports ASR results for our final 4B/8B-Instruct models across 29 test conditions, including AISHELL-1/2/4, AliMeeting, WenetSpeech, and KeSpeech. MOSS-Audio-8B achieves an overall macro-average error rate of 11.29, corresponding to reductions of 20.0%, and 24.9% relative to Kimi-Audio-7B-Instruct and Qwen2.5-Omni-7B, respectively. Our 4B model also outperforms all three baselines overall, scoring 11.58.

Dataset / Test conditionKimi-Audio-7B-InstructQwen2.5-Omni-7BMOSS-Audio-4B-InstructMOSS-Audio-8B-Instruct
AISHELL-10.791.162.261.82
AISHELL-2 (Android / iOS / Mic)2.91 / 3.03 / 2.882.88 / 2.77 / 2.733.22 / 3.33 / 3.202.97 / 2.95 / 2.91
THCHS-301.393.063.532.82
MAGICDATA-READ2.153.163.723.20
AISHELL6-Whisper (normal / whisper)0.69 / 4.630.71 / 6.570.73 / 5.860.69 / 4.80
AliMeeting (far / near)28.22 / 13.8232.03 / 18.7327.29 / 9.6836.82 / 11.25
AISHELL-420.6121.0120.3324.36
SeniorTalk (sentence)19.7019.9616.9317.42
ChildMandarin13.7912.2913.2513.10
AISHELL-6A (mild / moderate)7.00 / 9.347.27 / 10.946.36 / 9.775.84 / 8.94
AISHELL-6A (severe / StutteringSpeech)12.56 / 10.7512.92 / 10.5312.68 / 10.2811.52 / 9.72
AISHELL-6B (LRDWWS / Uncontrol)44.44 / 42.5751.99 / 49.4543.35 / 44.2539.76 / 39.27
WenetSpeech (test-meeting)7.158.438.177.86
Fleurs (cmn_hans_cn)5.105.138.137.52
CS-Dialogue14.5614.029.149.07
TALCS12.7410.468.378.22
ASCEND21.8314.4212.8313.26
KeSpeech5.516.4014.659.18
WSYue-ASR-eval (short)53.1757.439.048.33
MIR-1K38.3542.6218.4717.24
open-cpop5.172.753.102.39
MNV_174.684.564.014.31
Overall (macro-average over 29 conditions)14.1215.0511.5811.30

The advantages are particularly pronounced on diverse speech inputs. On WSYue-ASR-eval, our 8B model achieves 8.33, reducing errors by 76.0%–85.5% relative to these three baselines. On MIR-1K, it achieves 17.24, a 55.0%–59.5% reduction; on CS-Dialogue, it scores 9.07, versus 14.56/30.52/14.02. These results demonstrate strong final-model transcription capabilities, with substantial gains on Cantonese speech, singing, and conversational speech.

Regarding captioning, we prioritized broad audio understanding and fine-grained speech description. Our evaluations on MMAU, MMAU-Pro, MMAR, and MMSU already provide evidence of understanding and reasoning across speech, environmental sounds, and music. Our dedicated speech-captioning evaluation further examines vocal attributes such as accent, texture, clarity, and emotion, using 2,000 clips with human-written and reviewed references. Both model sizes outperform all evaluated baselines on the aggregate reference-matching score: 3.711/3.725, compared with 3.599 for the strongest baseline.

We appreciate the suggestion to broaden the captioning evaluation. We will supplement these results with evaluations of our final models on public audio-captioning benchmarks, including AudioCaps and Clotho.

W3: calculation and validity of speech-captioning scores.

Thank you for requesting a more explicit description. Humans establish the audio-grounded reference descriptions by listening to the clips and writing and reviewing the caption fields. The evaluated audio-language models generate predictions in the same 13-field schema. A separate text-only judge, Qwen3-235B with temperature 0, receives the field name and definition, the human reference, and the model prediction. It receives neither the audio nor the identity of the evaluated model and returns an integer score with a short justification.

Scoring rubric. The 1–5 rubric explicitly distinguishes different levels of correctness: 5, fully consistent and sufficiently specific; 4, correct main attribute with a minor omission or reduced specificity; 3, partially correct, with an incorrect or missing salient qualifier or mixed correct/incorrect cues; 2, largely incorrect with only weak or generic overlap; and 1, contradictory, off-topic, empty, unknown, or hallucinated. Clear synonyms and paraphrases are accepted, whereas lexical overlap alone is insufficient.

Aggregation. For each field, we average the scores over the examples included for that field. Macro-13 in Table 2 is their unweighted mean across all 13 fields; Macro-10 equally averages the remaining ten fields after excluding Gender, Age, and Personality. The scores remain on the original 1–5 scale.

Validity evidence. We conducted a calibration evaluation across seven open-ended dimensions, with multiple annotators following the same 1–5 scoring rubric. Our judge achieved a Spearman correlation of 0.79 with human judgments and a mean absolute deviation of 0.40 points. We additionally assessed semantic sensitivity using paraphrases, unrelated descriptions, and contradictory descriptions, which were expected to receive scores of at least 4, at most 2, and 1, respectively. The judge satisfied these thresholds in 88% of cases. These results support the scoring protocol’s sensitivity to semantic correctness and its agreement with human judgments across the calibrated dimensions.

These findings provide supporting evidence for semantic sensitivity and agreement with human judgments within our speech caption's evaluated setting.

W4

Evaluation across speech, sound, and music. Our evaluation covers all three domains using the final MOSS-Audio models. In particular, MMAU, MMAU-Pro, and MMAR include music- and sound-related questions that directly assess understanding in these domains. We additionally evaluate speech through ASR and speech captioning. Music and environmental sound are therefore already part of our main-model evaluation, beyond the component-level ablations.

We acknowledge that reporting benchmark-level aggregate scores in Table 1 makes the performance within each domain less explicit. We appreciate the reviewer’s suggestion to provide a finer-grained breakdown, which would help readers better assess domain-specific strengths and weaknesses. The proposed specialized sound-event and music tasks would provide complementary evidence and broaden the scope of our existing evaluation.

W5

Broken cross-references. Thank you for pointing out the unresolved references around Lines 367 and 454–457. We apologize for the difficulty this creates in locating and verifying the supporting experimental details. We will correct the affected references and ensure that all referenced tables and appendix sections are included in the revised PDF. We will also state the essential comparison settings and results explicitly in the main text so that the experimental evidence is easier to follow and assess.

Reviewer_u63gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MOSS-Audio, a family of audio-language models for unified understanding of speech, environmental sounds, and music. The system combines an event-oriented data pipeline, a dedicated audio encoder, DeepStack-based cross-layer feature injection, and explicit time markers. A multi-stage training procedure produces Instruct and Thinking variants at two model scales. Experiments cover general audio understanding, fine-grained speech captioning, speech recognition, and timestamp prediction, with competitive results across these tasks.

Critical strengths
  1. Substantial system-building effort with strong overall performance: The paper develops a comprehensive audio understanding system with competitive results. The event-oriented data pipeline is particularly valuable for preserving event context and integrating complementary supervision.

  2. Detailed description of the system design: The paper provides substantial implementation detail across data processing, model architecture, and training stages, offering useful guidance for researchers building similar systems.

  3. A relevant architectural adaptation and broad evaluation: Adapting DeepStack to expose intermediate acoustic features to the decoder is a sensible design choice. The evaluation covers diverse tasks, widely used benchmarks, and representative baselines.

Critical weaknesses
  1. Limited methodological novelty and scientific insight: The contribution primarily lies in integrating established techniques into a strong system. The paper offers limited new insight into why this combination works or which design choices are essential.

  2. The ablation study does not sufficiently explain the contributions of the proposed components: The DeepStack ablation shows mixed results but does not investigate which acoustic attributes or failure cases it improves, or whether the findings extend to the final model scales. The data pipeline and explicit time markers also lack isolated ablations.

  3. Presentation problems obscure important experimental details: Multiple unresolved appendix and table references (“??”) on pages 7–9 make supporting details difficult to locate. These references should be corrected, and the essential ablation settings, results, and interpretations should appear in the main text.

Questions for authors

All questions are addressed in the Weaknesses section.

2 comments

Thank you for recognizing the substantial system-building effort, the value of the event-oriented data pipeline, and the breadth of our evaluation. We address your three concerns below.

W1. Methodological novelty and scientific insight.

Our contribution is the design and empirical characterization of a unified audio-understanding system spanning semantic content, acoustic detail, and temporal localization. The event-oriented pipeline connects segmentation, specialist annotation, and supervision construction: fused descriptions provide integrated supervision, while retained branch-specific annotations support focused task instructions. We explicitly credit DeepStack as an existing principle adapted to this audio setting.

The component studies provide empirical findings alongside the complete-system evaluation. In Section 4.4, the encoder comparison uses a common Qwen3-1.7B decoder with matched downstream data, a 100k-step training budget, optimization, decoding, and scoring. The MOSS encoder demonstrates strong performance across a range of test conditions, supporting its use under the evaluated protocol. The DeepStack comparison further shows gains in pure-music and pure-sound captioning, as discussed below. We will sharpen the presentation of these findings and their implications for unified audio modeling.

W2. Component contributions, acoustic attributes, and generalization across model scales.

Controlled DeepStack comparison. Both configurations start from the same MOSS encoder and Qwen3-0.6B-base weights, with matched ASR alignment, MECAT-Caption fine-tuning, preprocessing, and scoring. The table below summarizes the three pure categories from Table 5. The additional “Overall (pure categories only)” row is computed as their equal-weight mean and excludes mixed-speech examples.

CategoryChange in DATE
Pure music+0.0222
Pure sound+0.0240
Pure speech−0.0070
Overall (pure categories only; macro-average)+0.0131

The pure-category macro-average DATE increases from 0.4527 to 0.4658, driven by improvements in pure-music and pure-sound captioning. These gains suggest that DeepStack helps the model exploit intermediate acoustic information relevant to music and sound in the tested 0.6B setting. Pure-speech performance decreases, indicating that the benefits vary across content types.

Event-guided segmentation and time markers. Our pipeline addresses a concrete mismatch between speech-based segmentation and unified audio supervision. VAD detects speech activity, but speech boundaries do not necessarily coincide with the boundaries of non-speech events. Segmentation based solely on speech activity can cut through ongoing sound events and separate them from relevant context. For example, a sound event may begin before an utterance and continue after speech stops. Preserving this context matters because environmental sounds, music, and their relationships with speech are themselves important supervision signals.

We therefore use detected acoustic events to guide segmentation and annotation routing, then combine specialist outputs into temporally coherent descriptions. This design aims to retain event continuity and context within the clip-duration constraints. Time markers complement this supervision by providing elapsed-time references, which work together with timestamp targets to associate acoustic content with positions on the recording’s timeline.

Evidence from the final 4B/8B models. Complementing the controlled 0.6B study, Tables 1–3 evaluate the complete models on the capabilities motivating these design choices. Table 1 covers understanding across speech, sound, and music. Table 2 provides attribute-level evaluation: both Instruct models achieve higher Accent, Texture, and Clarity reference-matching scores than the evaluated baselines. Table 3 shows that 8B-Instruct achieves the lowest reported timestamp error among the evaluated models on both AISHELL-1 and LibriSpeech, with AAS values of 35.77 ms and 131.61 ms, respectively. These results demonstrate competitive general audio understanding and comparative strengths in fine-grained speech description and speech timestamp localization under the reported protocols.

We will make the relationship between design and evidence clearer in the revision, distinguishing the controlled findings on DeepStack at 0.6B from the end-to-end capabilities demonstrated by the final 4B/8B systems.

W3. Unresolved references and accessibility of experimental details.

We apologize for the unresolved references on pages 7–9 and the supporting material omitted from the submitted PDF. We will restore the existing material, correct the affected references, and make the essential encoder-comparison and DeepStack settings, results, and interpretations self-contained in the main text. We will verify the exported PDF to ensure that every referenced table and appendix is present and correctly linked.

Thank you for your response, but I do not see an updated version of the paper. While I acknowledge that the MOSS-Audio system performs very well overall, I still do not believe it offers sufficient research innovation or new insights. I will maintain my original score.

Reviewer_PvhyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MOSS-Audio, a family of audio-language models for unified understanding and reasoning across speech, environmental sound, and music. It combines event-oriented data construction, a dedicated audio encoder, DeepStack-based cross-layer conditioning, and explicit time tokens, with multi-stage training producing Instruct and Thinking variants.

Critical strengths

S1. Broad evaluation and consistent improvements from reasoning-oriented post-training

The paper provides a broad evaluation of unified audio understanding across speech, environmental sound, and music, extending beyond semantic understanding to fine-grained speech captioning, transcription, and timestamp prediction. In particular, the Thinking variants consistently outperform their Instruct counterparts on all four audio understanding benchmarks at both model sizes. These results support the effectiveness of the proposed model family across diverse audio understanding tasks.

Critical weaknesses

W1. Limited analysis of individual component contributions

The overall experimental results demonstrate the competitiveness of the proposed system, but the analysis of the individual contributions of some key design choices remains limited. This makes it difficult to assess the specific benefits of these components separately from the performance of the complete system.

Questions for authors

Q1. Clarification on encoder-layer selection and decoder injection positions

Could the authors clarify the rationale for selecting audio encoder block outputs 8, 16, and 24 and adding their projected features to the outputs of the first three decoder blocks? When adapting the DeepStack principle to audio, were alternative encoder-layer combinations or decoder injection positions considered? Could the authors share any existing analyses or observations supporting the chosen configuration?

Q2. Clarification on the contributions of reasoning post-training stages

The comparison between Instruct and Thinking includes both reasoning cold start and reinforcement learning. Could the authors clarify the performance changes associated with each stage?

4 comments

Thank you for recognizing the broad evaluation and the consistent improvements of the Thinking variants. We address the component analysis and both questions below.

W1. Limited analysis of individual component contributions

We understand the reviewer’s interest in identifying the benefits of individual design choices. The existing component comparisons and the stage-wise results summarized in Q2 provide the following findings:

  • The dedicated encoder improves recognition under a shared downstream configuration. Using the same Qwen3-1.7B decoder, training data, 100k-step budget, and matched optimization and evaluation settings, the MOSS Audio Encoder demonstrates stronger overall speech recognition performance than AuT across a diverse evaluation suite. This comparison supports the benefit of the encoder choice while holding the downstream decoder and training recipe fixed.

  • Intermediate-feature conditioning improves sound and music description. In the controlled Qwen3-0.6B-base comparison, DeepStack improves pure-music DATE from 0.4705 to 0.4927 and pure-sound DATE from 0.4015 to 0.4255. These results support the usefulness of intermediate encoder features for non-speech captioning in the tested setting. The effects vary across content types: speech-related scores decrease, leaving the overall score nearly unchanged. We discuss the configuration rationale and supporting observations in Q1.

  • The stage-wise comparison distinguishes the changes following cold start and RL. For the 8B model, Overall accuracy progresses from 66.32 for Instruct, to 67.36 after cold start, and to 71.08 after RL. The corresponding average gains are 1.04 and 3.71 percentage points, calculated before rounding the averages. Q2 provides the benchmark-level breakdown for both transitions.

We appreciate the reviewer’s interest in extending this analysis to each design choice in the final 4B/8B models. We would have liked to provide such a comprehensive analysis. However, the architectural choices are incorporated during pretraining and learned jointly with the rest of the model, so meaningful full-scale comparisons would require additional pretraining runs followed by subsequent training under matched budgets. We do not currently have the compute resources to undertake these runs during rebuttal. We therefore address the specific questions below using the available results and a more detailed explanation of the design choices.

Q1. Rationale for encoder-layer selection and decoder injection positions

  • Why encoder layers 8, 16, and 24? These are zero-based indices in our 32-block encoder, corresponding to the ninth, seventeenth, and twenty-fifth blocks. They provide approximately evenly spaced intermediate representations alongside the final-layer audio prefix. This depth-spanning configuration is motivated by prior observations that acoustic and linguistic information vary across encoder depth, allowing the decoder to access representations from multiple abstraction levels.

  • Why the first three decoder blocks? Each selected representation is independently projected through a GatedMLP and added at aligned audio-token positions to the output of one of the first three decoder blocks. Introducing these auxiliary features early allows subsequent decoder blocks to process them while preserving temporal correspondence and audio-prefix length. Marker and instruction positions receive no direct addition. The three adapter pathways add parameters and computation without extending the input sequence.

  • Were alternatives evaluated, and what observations support this configuration? We do not have a controlled comparison of alternative encoder-layer combinations or decoder injection positions to report. The relevant evidence is Table 5, which compares the selected DeepStack configuration with final-layer conditioning. Its pure-music and pure-sound DATE improvements correspond to relative gains of approximately 4.7% and 6.0%, respectively. These observations support the value of the selected intermediate-feature conditioning pathway for sound and music understanding in the evaluated captioning setting.

Q2. Separate contributions of reasoning cold start and reinforcement learning

For the 8B model, we summarize three checkpoints: Instruct, after instruction tuning; Cold Start, after supervised reasoning initialization but before RL; and Thinking, after DAPO-based reinforcement learning. Cold start uses supervised audio-centered and text-only reasoning examples to initialize multi-step response patterns, while RL subsequently optimizes sampled responses using task rewards.

BenchmarkInstructCold StartThinking (after RL)Δ Cold StartΔ RL
MMSU66.3667.2475.52+0.88+8.28
MMAU (test-mini)77.0378.2677.33+1.23−0.93
MMAU-Pro57.4862.7664.92+5.28+2.16
MMAR64.4261.1966.53−3.23+5.34
Overall66.3267.3671.08+1.04+3.71

Δ Cold Start is Cold Start minus Instruct; Δ RL is Thinking minus Cold Start, both in percentage points.

Cold start increases Overall accuracy by 1.04 percentage points, with its largest gain on MMAU-Pro (+5.28), while MMAR decreases by 3.23 points. RL yields a further 3.71-point average improvement, with gains on MMSU (+8.28), MMAU-Pro (+2.16), and MMAR (+5.34), while MMAU decreases by 0.93 points. This breakdown separates the observed performance changes associated with the two stages and shows that both contribute to the aggregate improvement.

Reviewer_PvhyOpenReview ↗

Official Comment by Reviewer Pvhy

Thank you for the detailed responses and the additional stage-wise results. The clarification of the configuration is helpful, and the new results provide a clearer understanding of the performance changes following reasoning cold start and reinforcement learning.

To clarify my original concern in W1, I was particularly interested in the individual contributions of event-oriented data construction and explicit time markers. For the former, a comparison with conventional segmentation strategies, such as fixed-duration or VAD-based segmentation, or an analysis of event-context preservation and annotation quality would have been helpful. For the latter, a comparison with and without input time markers while retaining the same timestamp supervision would help clarify their additional contribution.

I have also reviewed the authors' responses to the other reviewers, which clarify the motivation for these design choices and the computational constraints associated with isolated ablations. I sincerely appreciate the authors' detailed clarifications and maintain my positive assessment of the paper.

Reviewer_JRQwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MOSS-Audio, a family of unified audio-language models designed to understand speech, environmental sounds, and music while preserving fine-grained acoustic and temporal information. The framework combines an event-oriented data construction pipeline, a dedicated audio encoder trained from scratch, DeepStack-based injection of intermediate acoustic features into the language decoder, and explicit time markers for temporal understanding. The models are trained through multiple stages, including audio-language alignment, joint pretraining, instruction tuning, reasoning cold start, and reinforcement learning, producing Instruct and Thinking variants at 4B and 8B scales.

Experiments on MMAU, MMAU-Pro, MMAR, and MMSU demonstrate competitive audio understanding performance, with MOSS-Audio-8B-Thinking achieving the highest average score among the evaluated open-source models. Additional evaluations show improvements in speech captioning, transcription, and timestamp accuracy, while controlled experiments investigate the contributions of the audio encoder and DeepStack conditioning.

Critical strengths
  • Strong performance across diverse audio understanding tasks, given the relatively small model size and broad task coverage.
  • A comprehensive approach to preserving acoustic and temporal information. The improvements in fine-grained speech captioning and timestamp accuracy demonstrate the practical value of learning acoustic properties beyond spoken content.
Critical weaknesses
  • Insufficient ablation studies for the proposed data pipeline and temporal modeling. The paper does not provide controlled comparisons between event-guided segmentation and conventional segmentation, or between models trained with and without explicit time markers. Without these experiments, it is difficult to determine how much these components contribute to the reported performance improvements.
  • Limited evidence for the overall effectiveness of DeepStack conditioning. Table 5 shows improvements in pure sound and music captioning, but the overall DATE score increases only from 0.4823 to 0.4831, while both speech-related scores decrease. Additionally, the ablation uses a 0.6B decoder, leaving it unclear whether the same improvements hold for the full 4B and 8B models.
  • Evaluation reliability and temporal baselines. If the paper can show variability across repeated benchmark evaluations and provide more details about the text-based judge used for the internal speech-captioning evaluation. Comparisons against specialized temporal audio-language approaches, such as TimeAudio and TEMPO, would also help establish the advantages of the proposed temporal modeling.
Questions for authors

See weaknesses.

3 comments

Thank you for the constructive review and for recognizing MOSS-Audio’s strong performance across diverse audio tasks. Our contribution brings together event-oriented data construction, acoustic feature conditioning, and temporal supervision in a unified audio-language model. The results across audio understanding, detailed captioning, and timestamped transcription demonstrate the effectiveness of this overall design. We clarify the motivation and supporting evidence for each point below.

W1. Event-oriented data construction and temporal modeling.

Our event-oriented pipeline addresses a concrete problem we observed during data construction. VAD identifies speech activity; it does not explicitly identify acoustic-event or scene boundaries. In the audio examples we inspected, VAD-based cuts fragmented most sound events, separating parts of an event from the surrounding acoustic context. This matters for unified audio understanding, where environmental sounds, music, and their relationships with speech are themselves important supervision signals.

Our pipeline introduces an event-oriented organization of audio training data. It uses detected acoustic events to guide segmentation, merges overlapping intervals, and incorporates surrounding context before routing clips to complementary annotation specialists. Their outputs are then fused into coherent descriptions that retain acoustic details and event chronology. The benefit of this design is to preserve information that speech-activity-based cuts can discard or fragment. Our inspection of VAD-segmented examples provides a qualitative basis for this design choice.

Explicit time markers serve a complementary role: they provide elapsed-time references within the audio sequence, while word- and sentence-level timestamp targets teach finer temporal alignment. As described in Section 3.3, the two-second marker spacing does not constrain the precision of predicted boundaries. The resulting 8B-Instruct system achieves AAS of 35.77 ms on AISHELL-1 and 131.61 ms on LibriSpeech, achieving the lowest reported AAS in Table 3 on both datasets. These results support the practical effectiveness of our approach to temporal modeling within a unified system.

We agree that a more detailed quantitative ablation of the data-construction choices and temporal components would be valuable. Given the compute and time required for matched retraining, our current component analysis is not exhaustive. We plan to extend these controlled studies in future work as resources permit.

W2. DeepStack gains and the scale of the component study.

Table 5 provides controlled evidence for the value of intermediate acoustic features in non-speech description. DeepStack improves pure-music DATE from 0.4705 to 0.4927 and pure-sound DATE from 0.4015 to 0.4255, corresponding to relative improvements of approximately 4.7% and 6.0%. Both conditions use the same encoder and Qwen3-0.6B-base decoder initialization with matched training data and procedures.

The category-level results highlight an important benefit for unified audio understanding: DeepStack improves pure-music and pure-sound captioning by +0.0222 and +0.0240, respectively. These gains directly support our goal of strengthening acoustic understanding beyond speech semantics. Examining the individual categories therefore reveals the added value of intermediate acoustic features for sound and music description, providing concrete support for the design motivation discussed in Section 4.4.

The controlled study uses a 0.6B decoder, while the 4B/8B evaluations assess the complete model family. These experiments provide complementary evidence: the former characterizes the benefits and trade-offs of the conditioning interface, and the latter demonstrates the capabilities of the resulting unified models. Extending the controlled study to larger decoders is a worthwhile direction for the more detailed architectural analysis discussed above.

W3. Evaluation reliability and temporal baselines.

Repeated evaluations. We evaluate every model five times under the same task-specific scoring protocol and report the mean across the five runs. Each benchmark score in Table 1 is therefore a five-run average, and the overall score is the equal-weight mean across the four benchmarks. Under this protocol, MOSS-Audio-8B-Thinking achieves the highest reported average among the evaluated open models 71.08. We will make the five-run evaluation and aggregation procedure explicit in the revised manuscript.

Speech-captioning judge. For the 2,000-clip benchmark, human annotators listen to the audio and write and review the reference descriptions. Evaluated models produce predictions in the same 13-field schema. A separate text-only judge, Qwen3-235B with temperature 0, receives the field name and definition, human reference, and model prediction. It receives neither the audio nor the evaluated model’s identity and returns an integer score with a short justification.

Scoring rubric. The 1–5 rubric distinguishes levels of correctness: 5, fully consistent and sufficiently specific; 4, correct main attribute with a minor omission or reduced specificity; 3, partially correct, with an incorrect or missing salient qualifier or mixed correct/incorrect cues; 2, largely incorrect with only weak or generic overlap; and 1, contradictory, off-topic, empty, unknown, or hallucinated. Clear synonyms and paraphrases are accepted; lexical overlap alone is insufficient.

Aggregation. We first average scores over the examples included for each field. Macro-13 is the unweighted mean of the resulting 13 field scores; Macro-10 equally averages the remaining ten after excluding Gender, Age, and Personality. Both remain on the original 1–5 scale.

Validity evidence. We conducted a calibration evaluation across seven open-ended dimensions, with multiple annotators using the same rubric. The judge achieved a Spearman correlation of 0.79 with human judgments and a mean absolute deviation of 0.40 points. We also tested semantic sensitivity using meaning-preserving paraphrases, unrelated descriptions, and contradictory descriptions, expected to receive scores of at least 4, at most 2, and 1, respectively. Across these checks, 88% of evaluated cases met the expected threshold for their perturbation type. These results support the protocol’s sensitivity to semantic correctness and its agreement with human judgments across the calibrated dimensions.

TimeAudio and TEMPO. We appreciate these suggestions for specialized temporal comparisons. Table 3 evaluates timestamped transcription on AISHELL-1 and LibriSpeech against human gold-standard labels using shared AAS scoring. Comparisons with TimeAudio and TEMPO under matched tasks, audio, output granularity, and scoring rules would broaden this evaluation, and we plan to explore them in future work.

The broad task results, the controlled improvements in sound and music captioning, and the calibration evidence for our speech-captioning evaluation provide complementary support for MOSS-Audio’s unified design and its assessment. We thank the reviewer for the suggestions to further strengthen the component analysis and evaluation reporting.

Reviewer_ABUbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSS-Audio is a family of audio-language models for unified understanding of speech, environmental sounds, and music. It combines an event-oriented data pipeline with a dedicated audio encoder and Qwen3-based language decoders. The pipeline segments recordings around acoustic events, integrates specialist annotations into temporally coherent captions, and retains branch-specific supervision for instruction tuning. The architecture incorporates DeepStack-style intermediate-feature injection and explicit time markers to support acoustic detail recognition and temporal grounding. A five-stage training process produces Instruct and Thinking variants. Experiments show competitive audio understanding, strong speech captioning, and accurate timestamp prediction. However, component-level evidence remains limited: DeepStack improves pure-music and pure-sound captioning but slightly reduces speech-captioning performance, while the independent contributions of event-guided segmentation and time markers are not isolated.

Critical strengths
  1. Competitive unified audio understanding with a relatively compact language backbone. MOSS-Audio-8B-Thinking achieves the highest average score among the evaluated open models across four public benchmarks (71.08 versus 70.67 for Step-Audio-R1), while the Instruct variant also demonstrates strong speech-captioning and timestamping results. Although the average lead is modest, this breadth supports the practical value of a shared model for semantic, acoustic, and temporal understanding.

  2. A coherent approach to preserving acoustic information throughout data construction and modeling. The event-oriented pipeline connects segmentation with complementary specialist annotations, while cross-layer conditioning exposes intermediate acoustic representations to the language decoder. Controlled experiments show improved pure-music and pure-sound captioning with DeepStack. Although these gains do not extend to every category, they provide concrete evidence that intermediate features can benefit non-speech understanding, making the approach a useful contribution beyond transcription-focused modeling.

Critical weaknesses
  1. Insufficient evidence for the central methodological contributions. The paper does not isolate the effects of event-guided segmentation or explicit time markers. Its DeepStack ablation uses only a 0.6B decoder and shows mixed results: improvements in pure-music and pure-sound captioning accompany declines in speech categories, leaving the overall score nearly unchanged (+0.0008). Consequently, the experiments do not establish whether the proposed components materially explain the strong full-system performance or generalize to the main 4B/8B models.

  2. Important gaps in training and evaluation transparency limit reproducibility. The encoder is described as trained from scratch, but its initial training procedure is insufficiently specified, while the first documented stage already freezes it. The reasoning post-training description also lacks concrete task-reward definitions. Evaluation reports repeated-run averages without specifying repetition counts or uncertainty estimates, making the narrow advantage over the strongest baseline difficult to assess. These omissions prevent a confident assessment of both reproducibility and the robustness of the claimed improvements.

Questions for authors
  1. Do you have controlled ablations isolating event-guided segmentation and explicit time markers, with training data volume and compute held comparable? These would clarify whether the proposed components explain the reported gains.

  2. Have you evaluated DeepStack against final-layer conditioning at the main 4B/8B scales? The 0.6B experiment shows nearly unchanged overall performance and declines in speech categories, leaving its contribution to the main models unclear.

4 comments

Thank you for the careful review and for recognizing the practical value of unified audio understanding with a relatively compact language backbone. Your comments highlight two areas that require clarification: the scope of our component-level evidence and the reproducibility of training and evaluation. We address each point below.

For Weakness 1 and Question 1:

We agree that controlled comparisons with comparable data volume and compute would help quantify the independent contributions of event-guided segmentation and explicit time markers. Such comparisons require substantial data-processing and training resources. In particular, changing the segmentation strategy would entail reprocessing the audio corpus and reconstructing the corresponding training examples, while isolating the effect of time markers would require an additional matched training run. These experiments were beyond our available resource budget, and we have not conducted these isolated ablations. We therefore clarify the rationale for both designs and the scope of the existing evidence below.

Event-guided segmentation aims to preserve acoustic events and their surrounding context. It merges overlapping event intervals and preferentially places segment boundaries in gaps between events, with a hard-cut fallback at the duration cap. VAD-based segmentation identifies speech activity but does not explicitly account for non-speech event boundaries. Our approach considers these boundaries directly, reducing the risk of fragmenting sound or music events and separating them from related speech. The duration cap means that event integrity cannot always be preserved, but event boundaries guide segmentation whenever feasible.

Explicit time markers provide elapsed-time anchors every 25 acoustic features, corresponding to two seconds, while word- and sentence-level timestamp targets provide output-side temporal supervision. The timestamp results in Table 3 evaluate this combined design: 8B-Instruct achieves AAS values of 35.77 ms on AISHELL-1 and 131.61 ms on LibriSpeech. These results characterize the complete system's temporal localization capability without isolating the contribution of time markers. We will clarify this scope in the methodological discussion. The DeepStack component study is discussed under Question 2 below.

For Weakness 2:

Encoder pretraining and its relationship to Stage 1.

The encoder undergoes standalone pretraining on ASR and audio captioning before the five-stage audio-language training described in the submission. Stage 1 therefore starts from an already pretrained encoder and freezes it while training the audio interfaces. Stage 2 subsequently updates the encoder jointly with the adapters and language decoder. We apologize for not making this sequence explicit.

Specifically, MOSS Audio Encoder was pretrained from scratch with autoregressive ASR and audio-captioning objectives on approximately 2 million hours of audio, comprising 577 million audio-text pairs with transcription or caption supervision from diverse audio corpora, conversational recordings, and online audio. Audio segments average approximately 12.5 seconds. The encoder converts 16 kHz audio into 128-dimensional log-Mel features with a 10 ms frame shift. Three convolutional downsampling layers and a 32-layer Transformer produce 1280-dimensional representations at 12.5 Hz. The encoder has approximately 644 million parameters and uses sinusoidal positional embeddings with approximately eight-second bidirectional attention windows per layer.

During this standalone pretraining, an eight-layer Whisper-style decoder attends to the encoder representations and predicts the target transcript or caption tokens through teacher forcing. Both components are jointly optimized with token-level cross-entropy. We use the Qwen3 tokenizer without loading pretrained Qwen language-model weights. Pretraining spans approximately two epochs and 56k optimizer updates, with an effective global batch size of 20,480 examples. We use AdamW with a peak learning rate of 4 × 10⁻⁴, 3% warmup followed by cosine decay, and weight decay of 0.01.

Reasoning post-training and reward design.

Following instruction tuning, we perform reasoning cold start using a mixture of audio-centered and text-only reasoning data with teacher-provided reasoning traces. Audio-centered examples train the model to connect its answers to perceptual evidence, including spoken content, speaker attributes, prosody, emotion, acoustic events, temporal relations, and musical characteristics. Text-only examples help transfer general multi-step reasoning patterns to audio-language modeling.

We then apply DAPO-based reinforcement learning across speech understanding, paralinguistic analysis, environmental sound understanding, music understanding, audio question answering, and temporal reasoning. Our reward design uses two forms of feedback. Answer correctness is evaluated with a binary reward: a correct answer receives 1, and an incorrect answer receives 0. For response quality and format compliance, an LLM judge evaluates each response against a fixed rubric, and we use the judge's score directly.

For each prompt, the policy generates 16 candidate responses. DAPO uses their relative rewards to increase the probability of higher-reward responses. We dynamically filter groups whose reward standard deviation is near zero, since they provide little within-group advantage signal, and perform additional sampling to replenish the training batch. This focuses optimization on prompts for which the sampled responses receive differing rewards.

Evaluation repetitions and averaging.

Each evaluation was repeated five times by regenerating the model outputs. These are inference-time repetitions; the models were not retrained between runs. Baseline results were also obtained through our own evaluations. For each benchmark, we report the mean score across the five generation runs. The “Avg.” column in Table 1 then takes the equally weighted mean of the four benchmark scores: MMAU, MMAU-Pro, MMAR, and MMSU.

For Question 2:

Our controlled comparison between final-layer-only conditioning and DeepStack uses Qwen3-0.6B-base; we have not conducted the corresponding comparison at 4B or 8B. Both conditions start from the same MOSS audio encoder and language-backbone weights, with matched training data, ASR alignment, MECAT-Caption fine-tuning, preprocessing, and scoring.

Within this controlled setting, Table 5 provides evidence for the value of incorporating intermediate acoustic features through DeepStack in non-speech description. Pure-music DATE improves from 0.4705 to 0.4927, and pure-sound DATE improves from 0.4015 to 0.4255, corresponding to relative improvements of approximately 4.7% and 6.0%, respectively. These category-level gains support our design motivation in Section 4.4: strengthening acoustic understanding beyond speech semantics.

The 0.6B component study and the 4B/8B evaluations provide complementary evidence: the former characterizes the benefits and trade-offs of the conditioning interface, while the latter demonstrates the capabilities of the complete unified models. Extending the controlled comparison to both 4B and 8B would require substantial additional training compute, as each scale requires separately trained final-layer-only and DeepStack variants under matched conditions. These experiments exceed our current computational budget. The existing study provides concrete support for our design motivation through improved sound and music description, while directly quantifying DeepStack's contribution at larger scales remains an important direction for future work.

Thank you for your detailed response. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.