- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MixCardBench, a benchmark for structured understanding of complex audio scenes. What’s new: It asks models to detect different sound sources and generate structured descriptions for speech, music and sound effects.
Critical strengths
- Fine-grained diagnostic evaluation. Instead of reducing audio understanding to a single captioning metric, the benchmark separately evaluates source detection/localization and multiple structured fields. This makes it possible to identify where different LALMs fail and provides substantially more diagnostic information than conventional audio-captioning evaluation.
- Extensive analysis. The paper provides detailed analyses across speech, music, and sound effects, individual MixCard fields, and different levels of polyphony. These experiments reveal meaningful differences between models and help identify source detection/localization as an important bottleneck.
Critical weaknesses
-
The MixCard representation is predominantly source-centric. While individual sources are described using detailed semantic and acoustic attributes, the representation contains relatively little information about the overall acoustic scene or relationships between sources, such as spatial relationships, foreground/background structure, acoustic environment, or semantic/causal interactions between events. Therefore, a set of individually correct MixCards may not necessarily constitute a complete structured understanding of the overall sound scene.
-
Since AudioSet originates from YouTube videos, it is unclear why the visual modality is entirely discarded during benchmark construction. An audio-only benchmark does not necessarily need visual inputs at inference time, but visual evidence during benchmark construction could provide an additional independent signal for validating sounding objects/actions and scene context, particularly given that ecological validity is an important motivation of the paper.
-
The benchmark construction depends substantially on SAM-Audio separation of the original AudioSet-SL recordings. In particular, speech and music are extracted sequentially, while the remaining signal is treated as sound effects. Errors or leakage in these upstream separation stages may therefore propagate into source retrieval/replacement, relative loudness estimation, soundscape synthesis, and ultimately benchmark annotations. This concern may be particularly important for heterogeneous sound effects, for which the residual signal may contain imperfectly removed speech or music components. A more direct analysis of separation errors and their downstream impact would strengthen the benchmark validation.
-
The construction pipeline relies on a relatively long chain of learned models and specialized tools for source separation, source retrieval, speech resynthesis, event matching, attribute prediction, filtering, and annotation generation. Although the role of each component is described, there is limited analysis of why these particular components were selected or how sensitive the resulting benchmark is to these choices. Comparisons with alternative components, or at least component-level quality analyses, would make it easier to assess how much benchmark quality depends on specific upstream models.
-
Qwen3-Omni is used to generate several semantic and perceptual attributes in the reference MixCards, while Qwen3-Omni itself is also evaluated and substantially outperforms the other evaluated LALMs. The human-corrected subset and the results on signal-processing-derived fields partially mitigate this concern, but it remains unclear how much of Qwen3-Omni’s advantage reflects genuine audio-understanding capability versus alignment with the annotation style produced by the same model family. A reference annotation pipeline independent of the evaluated models would provide stronger evidence.
-
Although AudioSet-SL provides diverse real-world recordings, the synthesis pipeline removes many difficult examples. In particular, non-English speech, overlapping/multiple speakers, and several challenging speech categories are filtered, and only a fraction of the speech examples survive the quality-control pipeline. Consequently, while AudioSet-SL provides the ecological structure used for synthesis, the resulting benchmark may represent a considerably cleaner and narrower distribution than real-world polyphonic audio. It would be useful to quantify how the final benchmark distribution differs from the original AudioSet-SL distribution beyond the current acoustic distribution metrics.
-
A substantial fraction of the reference MixCard fields are generated by LALMs or specialized prediction models rather than being directly annotated or measured. The authors perform human validation, which is valuable, but this covers only a subset of the benchmark. Moreover, a non-negligible fraction of the validated cards required human edits, particularly for music and sound effects. This raises the question of whether the remaining automatically generated annotations are sufficiently reliable to serve as benchmark ground truth. A larger human-annotated subset or more detailed field-level human agreement analysis would strengthen this aspect of the benchmark.
-
Given that the central motivation is structured understanding of polyphonic and multi-source audio scenes, it is surprising that 52.5% of the soundscapes contain only a single source, while only 27.7% contain at least three sources. 8659_MixCards_Evaluating_Struc.pdf This is particularly relevant because the paper itself shows that source-detection performance decreases as polyphony increases. Therefore, the aggregate benchmark score may be substantially influenced by relatively easy single-source examples and may understate the difficulty of the intended multi-source setting. Reporting a multi-source-only aggregate score could help clarify this issue.
-
The interpretation of Stage-2 performance under increasing polyphony may be confounded by the two-stage evaluation procedure. If Stage 2 evaluates descriptions only for sources that are successfully detected and matched in Stage 1, then increasing polyphony may cause the harder sources to be missed before they reach Stage 2. The remaining sources evaluated in Stage 2 may therefore constitute an increasingly selected, easier subset. Consequently, relatively stable Stage-2 performance as polyphony increases does not necessarily demonstrate that structured-description ability itself is robust to polyphony. An additional oracle-source/oracle-event experiment, where ground-truth source identities and temporal locations are provided before MixCard generation, would help disentangle source-detection errors from description capability.
Questions for authors
- Is Stage 2 only evaluated on sources successfully detected in Stage 1? If so, could you also report Stage-2 results using oracle source labels/timestamps?
- How reliable are the automatically generated MixCards outside the human-validated subset?
- Since Qwen3-Omni is used for MixCard generation and also performs the best in evaluation, could this introduce some bias?
- Since more than half of the benchmark is single-source, could you report the main results separately for single-source and multi-source cases?
6 comments
Continuing our response to Reviewer 8yxe
W5 & Q3 “Since Qwen3-Omni is used for MixCard generation and also performs the best in evaluation, could this introduce some bias?”
We acknowledge that using Qwen3-Omni for benchmark construction may introduce bias. We selected it as the strongest available open-source model to maximize annotation quality. However, Qwen3-Omni also performs strongly on numerical fields for which it was not used during generation, and remains competitive with Gemini Pro on the full benchmark (see Table B below). While these results suggest that its performance is not solely attributable to its role in benchmark construction, we cannot fully disentangle the two effects. Importantly, even Qwen3-Omni leaves substantial room for improvement.
W7 & Q2 “How reliable are the automatically generated MixCards outside the human-validated subset?”
We address this point in our response to Reviewer ACzt, showing that our validated set correlates with the full set. Absolute score deviations are small: across the 6 models and 29 card fields, the median absolute difference is 0.018, and the 95% CI of the difference contains 0 in 86.2% of cases. There is also strong agreement in model rankings, with a mean per-field Spearman ρ of 0.96 and Kendall τ_b of 0.92. Only 17 of 435 model pairs (3.9%) swap order across all fields, all within bootstrap noise, and the best and worst models are the same in 28 of 29 fields. This indicates that the fully generated mixcards are reliable, and we opt for a fully synthetic pipeline in lieu of manual validation, which is expensive and hard to scale.
W8 & Q4: Single-source vs. multi-source scenes
We thank the reviewer for raising this point. We clarify that our analysis considers source multiplicity within a scene, rather than exclusively temporal overlap.
We conducted an additional single-source vs. multi-source evaluation, finding that model rankings and our main findings remain consistent. However, the dataset exhibits a strong relationship between source count and category composition: 98.5% of single-source scenes contain music only, while approximately 97% of multi-source scenes contain sound effects. Moreover, only 31 and 7 scenes contain isolated speech and SFX, respectively, preventing meaningful conclusions for these categories in isolation.
This is particularly relevant to our polyphony analysis: since sound effects are substantially harder to detect than music, the decline in detection F1 with increasing source count may partly reflect changes in source composition, rather than source multiplicity alone. We will clarify this limitation, include the additional breakdown, and distinguish source multiplicity from temporal overlap in the revised paper.
W9 & Q1: Oracle source and event baselines
We thank the reviewer for this suggestion. We evaluated mixcard generation when the models receive the oracle tags and timestamps.
| Model | Stage 1 F1 (event / segment) | Stage 2, mixcards conitioned on predicted tags | Stage 2, mixcards conditioned on oracle tags | Change |
|---|---|---|---|---|
| Kimi | 0.290 / 0.428 | 0.375 | 0.352 | -0.023 |
| AF3 | 0.294 / 0.564 | 0.065 | 0.070 | +0.005 |
| AF-Next | 0.270 / 0.543 | 0.360 | 0.369 | +0.009 |
| Music Flamingo | 0.153 / 0.437 | 0.327 | 0.292 | -0.035 |
| Qwen2.5-Omni | 0.358 / 0.588 | 0.475 | 0.488 | +0.013 |
| Qwen3-Omni | 0.535 / 0.648 | 0.633 | 0.619 | -0.014 |
| gpt-audio-mini | 0.227 / 0.582 | - | 0.563 | - |
| Gemini 3.1 Pro | 0.434 / 0.701 | - | 0.609 | - |
Table B: Model performace on MixCardBench using predicted tags and oracle tags.
With oracle tags and timestamps, Stage 2 scores change by at most 0.035, and three of the six models score slightly lower. Even when every model receives the same oracle tags, Stage 2 scores still range from 0.070 (AF3) to 0.619 (Qwen3-Omni), close to their scores with predicted tags. Source detection remains a major challenge, as the low Stage 1 F1 shows (event F1 from 0.15 to 0.54). However, correct tags alone do not lead to better mixcards. This suggests that models also struggle to understand and describe source attributes from the audio, beyond the challenges of sound event detection. We will add these results to Figure 4 to disentangle source detection ability from mixcard generation ability.
Continuing our response to Reviewer 8yxe
W6: The benchmark is cleaner than real-world audio
Our synthesis and filtering pipeline is designed to ensure annotation correctness and coverage, which are essential for reliable evaluation. These stricter quality-control criteria inevitably exclude some difficult examples. We agree that the resulting benchmark represents a cleaner and more constrained subset of AudioSet-SL, while retaining real-world acoustic context. While a broader distributional comparison could help characterize the filtering effects, we consider it beyond the central objective of this work, which is reliable evaluation rather than reproducing the full AudioSet-SL distribution. Even under these more constrained conditions, existing models exhibit substantial limitations, demonstrating that the benchmark offers meaningful challenges and room for improvement.
We thank the reviewer for the detailed feedback. We address each point below.
W1: Relationships between sources
Our task setup indeed does not model the relationships between sources in an acoustic scene. We see source-specific mixcards as a necessary first step toward structured scene descriptions, and they already allow a finer analysis of model limitations than previous benchmarks. Current models also leave large headroom on our benchmark, suggesting the need to improve on per-source description before evaluating on more complex scene-level structured understanding tasks.
W2: Why not use the original AudioSet-SL videos
While AudioSet-SL clips do come with videos, our pipeline changes the audio substantially: it preprocesses the annotations, clones voices, maps music to new tracks and matches sfx by metadata. The synthesized sources therefore do not have fine temporal alignment with the original video, limiting its utility for verifying the mixture or generating mixcards.
W3: Effect of separation errors
SAM Audio generates N candidate separations and keeps the one with the highest ranking score. In the released configuration, this score is a weighted sum of a CLAP text-audio similarity score and the SAM Audio Judge score. We set N = 8, which gives reasonable source estimates for speech and music. Sfx separation is less reliable, but we match sfx sources by metadata similarity, not audio similarity, and we only use the sfx residual (the audio left after removing speech and music) for loudness estimation, which limits the impact of separation errors. We do not ablate the effect of separation artifacts on benchmark performance. However, we filter sources with SAM Audio Judge, PE-AV and speech quality metrics. These filters remove low-quality sources before soundscape synthesis, so they never reach the mixcard generation pipeline.
W4: Ablations of pipeline components
The huge design space of possible components and their model choices in our pipeline makes ablating each one infeasible. We did however run preliminary experiments to guide our main model choices. We summarize them below.
Speech separation. We compared ClearerVoice-Studio (MossFormer2_SE_48K), a strong open-source speech separation model, and SAM Audio, by the DNSMOS quality of their separated speech, on the speech sources of the 4,884 released scenes.
Table A: Comparison of speech separation models on the MixCardBench subset of AudioSet-SL eval
SAM Audio scored higher on signal quality and overall quality, and about the same on background quality. We therefore use SAM Audio for both speech and music separation, which keeps the pipeline to a single separation model.
Music replacement. We replace the music stems separated from AudioSet-SL with publicly available tracks from FMA. To do this, we extract embeddings for each separated stem and each FMA track, and match each stem to its nearest FMA track. We compared MULE and MusicFM as the embedding model. In our preliminary analysis, MULE retrieved closer matches and better preserved the mood of the original music, which also matched our findings in prior work. We use MULE in the final pipeline.
Instrument labels in music mixcards. We first tried dedicated instrument classifiers: the pretrained Essentia MTG-Jamendo instrument tagger, and a probe on PaSST embeddings trained on OpenMIC-2018. Both restrict the set of labels we can assign. OpenMIC-2018 has only 20 classes, so our probe cannot label electric guitar or drum machine, which appear in 14% and 13% of our music cards (Table 12). The 40 MTG-Jamendo classes cover only 30 of the 53 instrument labels in our music mixcards. The Essentia tagger was also less accurate than the LALM, achieving micro F1 of 0.26, against 0.36 for Qwen3-Omni with an open-vocabulary prompt on OpenMIC-2018 test clips. Thus we use Qwen3-Omni predicted instrument labels in MixCardBench.
Our general rule was to use a dedicated model when a validated estimator exists for the exact quantity (loudness, F0, tempo, intelligibility, emotion), and an LALM otherwise. We also benchmarked LALMs on music captioning datasets such as Song Describer. Qwen3-Omni and Qwen2.5-Omni performed best for music understanding. We chose Qwen3-Omni because its descriptions were richer and it had the lowest rate of false claims in our listening audit. We prompt Qwen3-Omni on the isolated music stem with a fixed vocabulary of 58 instruments (Appendix I.1.3).