acceptodds
Under review as a conference paper at ICLR 2027

MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes

Abstract

Audio captioning benchmarks typically evaluate a single unstructured free-text description of an audio scene, providing limited insight into how a model fails to understand polyphonic audio. A plausible caption may omit a source, or conflate multiple sources into one, or assign an attribute of one source to another, and conventional captioning evaluation cannot locate these errors. We introduce MixCardsBench, a benchmark for structured captioning of polyphonic audio, which evaluates models in two stages: first, identifying and temporally localizing the sources in a scene, and second, generating a structured MixCard describing the semantic, temporal, and acoustic properties of each source. The benchmark contains 4884 ecologically plausible synthetic soundscapes with up to eight overlapping speech, music and sound effects sources built from high-quality recordings and source-level annotations, where each source has a reference MixCard. We evaluate six state-of-the-art large audio language models and find substantial limitations in current models, which struggle considerably with speech and sound effects timing and detection, with the strongest model reaching only around 60% mean card-field score. Performance varies significantly across attributes, exposing capability differences that are hidden by aggregate caption scores. We assess the quality and ecological validity of MixCardBench through human validation, distribution matching and a listening study, and shows that human corrections have little effect on model rankings, and our synthesized soundscapes are closer to in-the-wild YouTube videos than source guided but otherwise random synthesis. https://mixcards-anon.github.io/mixcards-demo/

Primary area:
datasets and benchmarks
Keywords:
structured descriptions, audio language models, structured captioning benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8yxeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MixCardBench, a benchmark for structured understanding of complex audio scenes. What’s new: It asks models to detect different sound sources and generate structured descriptions for speech, music and sound effects.

Critical strengths
  1. Fine-grained diagnostic evaluation. Instead of reducing audio understanding to a single captioning metric, the benchmark separately evaluates source detection/localization and multiple structured fields. This makes it possible to identify where different LALMs fail and provides substantially more diagnostic information than conventional audio-captioning evaluation.
  2. Extensive analysis. The paper provides detailed analyses across speech, music, and sound effects, individual MixCard fields, and different levels of polyphony. These experiments reveal meaningful differences between models and help identify source detection/localization as an important bottleneck.
Critical weaknesses
  1. The MixCard representation is predominantly source-centric. While individual sources are described using detailed semantic and acoustic attributes, the representation contains relatively little information about the overall acoustic scene or relationships between sources, such as spatial relationships, foreground/background structure, acoustic environment, or semantic/causal interactions between events. Therefore, a set of individually correct MixCards may not necessarily constitute a complete structured understanding of the overall sound scene.

  2. Since AudioSet originates from YouTube videos, it is unclear why the visual modality is entirely discarded during benchmark construction. An audio-only benchmark does not necessarily need visual inputs at inference time, but visual evidence during benchmark construction could provide an additional independent signal for validating sounding objects/actions and scene context, particularly given that ecological validity is an important motivation of the paper.

  3. The benchmark construction depends substantially on SAM-Audio separation of the original AudioSet-SL recordings. In particular, speech and music are extracted sequentially, while the remaining signal is treated as sound effects. Errors or leakage in these upstream separation stages may therefore propagate into source retrieval/replacement, relative loudness estimation, soundscape synthesis, and ultimately benchmark annotations. This concern may be particularly important for heterogeneous sound effects, for which the residual signal may contain imperfectly removed speech or music components. A more direct analysis of separation errors and their downstream impact would strengthen the benchmark validation.

  4. The construction pipeline relies on a relatively long chain of learned models and specialized tools for source separation, source retrieval, speech resynthesis, event matching, attribute prediction, filtering, and annotation generation. Although the role of each component is described, there is limited analysis of why these particular components were selected or how sensitive the resulting benchmark is to these choices. Comparisons with alternative components, or at least component-level quality analyses, would make it easier to assess how much benchmark quality depends on specific upstream models.

  5. Qwen3-Omni is used to generate several semantic and perceptual attributes in the reference MixCards, while Qwen3-Omni itself is also evaluated and substantially outperforms the other evaluated LALMs. The human-corrected subset and the results on signal-processing-derived fields partially mitigate this concern, but it remains unclear how much of Qwen3-Omni’s advantage reflects genuine audio-understanding capability versus alignment with the annotation style produced by the same model family. A reference annotation pipeline independent of the evaluated models would provide stronger evidence.

  6. Although AudioSet-SL provides diverse real-world recordings, the synthesis pipeline removes many difficult examples. In particular, non-English speech, overlapping/multiple speakers, and several challenging speech categories are filtered, and only a fraction of the speech examples survive the quality-control pipeline. Consequently, while AudioSet-SL provides the ecological structure used for synthesis, the resulting benchmark may represent a considerably cleaner and narrower distribution than real-world polyphonic audio. It would be useful to quantify how the final benchmark distribution differs from the original AudioSet-SL distribution beyond the current acoustic distribution metrics.

  7. A substantial fraction of the reference MixCard fields are generated by LALMs or specialized prediction models rather than being directly annotated or measured. The authors perform human validation, which is valuable, but this covers only a subset of the benchmark. Moreover, a non-negligible fraction of the validated cards required human edits, particularly for music and sound effects. This raises the question of whether the remaining automatically generated annotations are sufficiently reliable to serve as benchmark ground truth. A larger human-annotated subset or more detailed field-level human agreement analysis would strengthen this aspect of the benchmark.

  8. Given that the central motivation is structured understanding of polyphonic and multi-source audio scenes, it is surprising that 52.5% of the soundscapes contain only a single source, while only 27.7% contain at least three sources. 8659_MixCards_Evaluating_Struc.pdf This is particularly relevant because the paper itself shows that source-detection performance decreases as polyphony increases. Therefore, the aggregate benchmark score may be substantially influenced by relatively easy single-source examples and may understate the difficulty of the intended multi-source setting. Reporting a multi-source-only aggregate score could help clarify this issue.

  9. The interpretation of Stage-2 performance under increasing polyphony may be confounded by the two-stage evaluation procedure. If Stage 2 evaluates descriptions only for sources that are successfully detected and matched in Stage 1, then increasing polyphony may cause the harder sources to be missed before they reach Stage 2. The remaining sources evaluated in Stage 2 may therefore constitute an increasingly selected, easier subset. Consequently, relatively stable Stage-2 performance as polyphony increases does not necessarily demonstrate that structured-description ability itself is robust to polyphony. An additional oracle-source/oracle-event experiment, where ground-truth source identities and temporal locations are provided before MixCard generation, would help disentangle source-detection errors from description capability.

Questions for authors
  1. Is Stage 2 only evaluated on sources successfully detected in Stage 1? If so, could you also report Stage-2 results using oracle source labels/timestamps?
  2. How reliable are the automatically generated MixCards outside the human-validated subset?
  3. Since Qwen3-Omni is used for MixCard generation and also performs the best in evaluation, could this introduce some bias?
  4. Since more than half of the benchmark is single-source, could you report the main results separately for single-source and multi-source cases?
6 comments

We thank the reviewer for the detailed feedback. We address each point below.

W1: Relationships between sources

Our task setup indeed does not model the relationships between sources in an acoustic scene. We see source-specific mixcards as a necessary first step toward structured scene descriptions, and they already allow a finer analysis of model limitations than previous benchmarks. Current models also leave large headroom on our benchmark, suggesting the need to improve on per-source description before evaluating on more complex scene-level structured understanding tasks.

W2: Why not use the original AudioSet-SL videos

While AudioSet-SL clips do come with videos, our pipeline changes the audio substantially: it preprocesses the annotations, clones voices, maps music to new tracks and matches sfx by metadata. The synthesized sources therefore do not have fine temporal alignment with the original video, limiting its utility for verifying the mixture or generating mixcards.

W3: Effect of separation errors

SAM Audio generates N candidate separations and keeps the one with the highest ranking score. In the released configuration, this score is a weighted sum of a CLAP text-audio similarity score and the SAM Audio Judge score. We set N = 8, which gives reasonable source estimates for speech and music. Sfx separation is less reliable, but we match sfx sources by metadata similarity, not audio similarity, and we only use the sfx residual (the audio left after removing speech and music) for loudness estimation, which limits the impact of separation errors. We do not ablate the effect of separation artifacts on benchmark performance. However, we filter sources with SAM Audio Judge, PE-AV and speech quality metrics. These filters remove low-quality sources before soundscape synthesis, so they never reach the mixcard generation pipeline.

W4: Ablations of pipeline components

The huge design space of possible components and their model choices in our pipeline makes ablating each one infeasible. We did however run preliminary experiments to guide our main model choices. We summarize them below.

Speech separation. We compared ClearerVoice-Studio (MossFormer2_SE_48K), a strong open-source speech separation model, and SAM Audio, by the DNSMOS quality of their separated speech, on the speech sources of the 4,884 released scenes.

Separation modelDNSMOS SIGDNSMOS BAKDNSMOS OVL
MossFormer2_SE_48K3.2793.9072.952
SAM Audio3.3563.8833.004

Table A: Comparison of speech separation models on the MixCardBench subset of AudioSet-SL eval

SAM Audio scored higher on signal quality and overall quality, and about the same on background quality. We therefore use SAM Audio for both speech and music separation, which keeps the pipeline to a single separation model.

Music replacement. We replace the music stems separated from AudioSet-SL with publicly available tracks from FMA. To do this, we extract embeddings for each separated stem and each FMA track, and match each stem to its nearest FMA track. We compared MULE and MusicFM as the embedding model. In our preliminary analysis, MULE retrieved closer matches and better preserved the mood of the original music, which also matched our findings in prior work. We use MULE in the final pipeline.

Instrument labels in music mixcards. We first tried dedicated instrument classifiers: the pretrained Essentia MTG-Jamendo instrument tagger, and a probe on PaSST embeddings trained on OpenMIC-2018. Both restrict the set of labels we can assign. OpenMIC-2018 has only 20 classes, so our probe cannot label electric guitar or drum machine, which appear in 14% and 13% of our music cards (Table 12). The 40 MTG-Jamendo classes cover only 30 of the 53 instrument labels in our music mixcards. The Essentia tagger was also less accurate than the LALM, achieving micro F1 of 0.26, against 0.36 for Qwen3-Omni with an open-vocabulary prompt on OpenMIC-2018 test clips. Thus we use Qwen3-Omni predicted instrument labels in MixCardBench.

Our general rule was to use a dedicated model when a validated estimator exists for the exact quantity (loudness, F0, tempo, intelligibility, emotion), and an LALM otherwise. We also benchmarked LALMs on music captioning datasets such as Song Describer. Qwen3-Omni and Qwen2.5-Omni performed best for music understanding. We chose Qwen3-Omni because its descriptions were richer and it had the lowest rate of false claims in our listening audit. We prompt Qwen3-Omni on the isolated music stem with a fixed vocabulary of 58 instruments (Appendix I.1.3).

AuthorsOpenReview ↗

Continuing our response to Reviewer 8yxe

W5 & Q3 “Since Qwen3-Omni is used for MixCard generation and also performs the best in evaluation, could this introduce some bias?”

We acknowledge that using Qwen3-Omni for benchmark construction may introduce bias. We selected it as the strongest available open-source model to maximize annotation quality. However, Qwen3-Omni also performs strongly on numerical fields for which it was not used during generation, and remains competitive with Gemini Pro on the full benchmark (see Table B below). While these results suggest that its performance is not solely attributable to its role in benchmark construction, we cannot fully disentangle the two effects. Importantly, even Qwen3-Omni leaves substantial room for improvement.

W7 & Q2 “How reliable are the automatically generated MixCards outside the human-validated subset?”

We address this point in our response to Reviewer ACzt, showing that our validated set correlates with the full set. Absolute score deviations are small: across the 6 models and 29 card fields, the median absolute difference is 0.018, and the 95% CI of the difference contains 0 in 86.2% of cases. There is also strong agreement in model rankings, with a mean per-field Spearman ρ of 0.96 and Kendall τ_b of 0.92. Only 17 of 435 model pairs (3.9%) swap order across all fields, all within bootstrap noise, and the best and worst models are the same in 28 of 29 fields. This indicates that the fully generated mixcards are reliable, and we opt for a fully synthetic pipeline in lieu of manual validation, which is expensive and hard to scale.

W8 & Q4: Single-source vs. multi-source scenes

We thank the reviewer for raising this point. We clarify that our analysis considers source multiplicity within a scene, rather than exclusively temporal overlap.

We conducted an additional single-source vs. multi-source evaluation, finding that model rankings and our main findings remain consistent. However, the dataset exhibits a strong relationship between source count and category composition: 98.5% of single-source scenes contain music only, while approximately 97% of multi-source scenes contain sound effects. Moreover, only 31 and 7 scenes contain isolated speech and SFX, respectively, preventing meaningful conclusions for these categories in isolation.

This is particularly relevant to our polyphony analysis: since sound effects are substantially harder to detect than music, the decline in detection F1 with increasing source count may partly reflect changes in source composition, rather than source multiplicity alone. We will clarify this limitation, include the additional breakdown, and distinguish source multiplicity from temporal overlap in the revised paper.

W9 & Q1: Oracle source and event baselines

We thank the reviewer for this suggestion. We evaluated mixcard generation when the models receive the oracle tags and timestamps.

ModelStage 1 F1 (event / segment)Stage 2, mixcards conitioned on predicted tagsStage 2, mixcards conditioned on oracle tagsChange
Kimi0.290 / 0.4280.3750.352-0.023
AF30.294 / 0.5640.0650.070+0.005
AF-Next0.270 / 0.5430.3600.369+0.009
Music Flamingo0.153 / 0.4370.3270.292-0.035
Qwen2.5-Omni0.358 / 0.5880.4750.488+0.013
Qwen3-Omni0.535 / 0.6480.6330.619-0.014
gpt-audio-mini0.227 / 0.582-0.563-
Gemini 3.1 Pro0.434 / 0.701-0.609-

Table B: Model performace on MixCardBench using predicted tags and oracle tags.

With oracle tags and timestamps, Stage 2 scores change by at most 0.035, and three of the six models score slightly lower. Even when every model receives the same oracle tags, Stage 2 scores still range from 0.070 (AF3) to 0.619 (Qwen3-Omni), close to their scores with predicted tags. Source detection remains a major challenge, as the low Stage 1 F1 shows (event F1 from 0.15 to 0.54). However, correct tags alone do not lead to better mixcards. This suggests that models also struggle to understand and describe source attributes from the audio, beyond the challenges of sound event detection. We will add these results to Figure 4 to disentangle source detection ability from mixcard generation ability.

AuthorsOpenReview ↗

Continuing our response to Reviewer 8yxe

W6: The benchmark is cleaner than real-world audio

Our synthesis and filtering pipeline is designed to ensure annotation correctness and coverage, which are essential for reliable evaluation. These stricter quality-control criteria inevitably exclude some difficult examples. We agree that the resulting benchmark represents a cleaner and more constrained subset of AudioSet-SL, while retaining real-world acoustic context. While a broader distributional comparison could help characterize the filtering effects, we consider it beyond the central objective of this work, which is reliable evaluation rather than reproducing the full AudioSet-SL distribution. Even under these more constrained conditions, existing models exhibit substantial limitations, demonstrating that the benchmark offers meaningful challenges and room for improvement.

Reviewer_M6v3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors present MixCardBench, a benchmark for evaluating structured descriptions of audio scenes with overlapping speech, music, and SFX. It includes around 4884 synthetic audio scenes built from real-world event patterns, with reference MixCards describing semantic, temporal, and acoustic properties. The evaluation separates source detection and localization from the quality of field-level description. Tests with six large audio-language models show that detecting sound effects and identifying sources becomes harder as the number of overlapping sounds increases. Human annotation checks and a listening study assess reference quality and the realism of the audio scenes.

Critical strengths
  1. The paper identifies a clear and real gap in how audio captioning is evaluated for scenes with several overlapping sounds.
  2. The authors propose a reliable synthesis pipeline to rebuild AudioSet-SL's timing, co-occurrence, and relative loudness from open stems. This approach helps address AudioSet’s redistribution and drift issues.
  3. Furthermore, to validate the results, they use several methods, including human review, comparisons with AudioSet-SL using standard metrics, listening studies, and independent checks by language model judges.
Critical weaknesses
  1. The main critical weakness of the paper is that it does not include simple baselines. The paper does not establish how much the reported performance depends on the audio itself. Without it, it's hard to tell whether the reported scores reflect real audio understanding or just good guessing. I think this matters especially for numerical fields. According to Equation 6, a prediction off by one median absolute deviation still gets a score of 0.75. This means a model that always predicts the dataset's typical value could score about 0.75 without hearing anything. As a result, it is difficult to interpret the finding that models perform best on numerical fields. I suggest adding simple baselines: always predict the median for numerical fields like loudness, pitch, and tempo; always predict the most common label for ordinal and categorical fields such as loudness level or plurality; always predict a fixed set of frequent tags for semantic fields like mood or sound descriptors; and run the models with the same prompts on silent audio to see how much of their score comes from language and dataset priors.
  2. Furthermore, the paper uses detection metrics and expands an existing structured-caption evaluator. It needs to show how using both together differs from using each metric on its own.
Questions for authors
  1. I am curious how the benchmark handles attribute binding when there are several instances of the same source type, since it assigns one card for each speech or music category and each sound-effect class?
3 comments
AuthorsOpenReview ↗

Response to Reviewer M6v3: part 1

We thank the reviewer for their comments. We address each weakness and question in turn.

W1: Random reference mixcards baseline

We evaluated random reference MixCards using oracle source categories and timestamps, providing the same information to the evaluated models. Oracle refers to ground truth stage 1 tags conditioning model's stage 2 predictions. For each tagged source, it predicts a mixcard drawn at random from all dataset mixcards of the same type (speech, music or sfx). We report the mean over 10 random seeds. The standard deviation is at most 0.011 on every field.

FieldRandom reference mixcardsQwen3-Omni (difference)Gemini 3.1 Pro (difference)
Numerical or quantized fields
Speech loudness0.766+0.057+0.039
Music loudness0.732+0.051+0.082
Sfx loudness0.695+0.097+0.109
Speech median F00.735+0.114+0.104
Speech frequency profile0.825-0.091-0.031
Music frequency profile0.747-0.021+0.050
Sfx frequency profile0.684+0.038+0.056
Speech intelligibility0.861-0.326-0.099
Music tempo0.475-0.019+0.073
Sfx plurality0.824+0.042+0.048
Open-ended fields
Speech emotion0.659+0.049+0.048
Speech manner0.512+0.003-0.032
Speech perceived audio quality0.712-0.098-0.194
Music perceived audio quality0.549+0.122-0.012
Sfx perceived audio quality0.580+0.064-0.105
Music genre0.295+0.162+0.143
Music mood0.426+0.281+0.235
Music instruments0.363+0.296+0.283
Music descriptors0.380+0.204+0.150
Sfx sounding object0.275+0.297+0.294
Sfx sounding action0.415+0.243+0.245
Sfx sound descriptors0.394+0.135+0.088
Sfx temporal descriptors0.588+0.118+0.072
Speech short caption0.646-0.245-0.137
Music short caption0.205+0.159+0.267
Sfx short caption0.067+0.217+0.294
Speech descriptive caption0.533+0.138+0.009
Music descriptive caption0.361+0.317+0.212
Sfx descriptive caption0.155+0.274+0.256
Card score
Speech0.694-0.044-0.032
Music0.453+0.155+0.149
Sfx0.469+0.151+0.134

The results show that some numerical and quantized fields achieve relatively high scores from dataset priors alone. This likely reflects distributional biases introduced by our data construction and filtering pipeline, which prioritizes high-quality, ecologically plausible soundscapes. Such priors may inflate absolute scores on certain fields, particularly in speech, where the random baseline achieves a higher aggregate score than the evaluated models.

Importantly, the strongest models consistently outperform this baseline on open-ended semantic and descriptive fields, suggesting that these aspects of the benchmark capture information beyond distributional priors. Moreover, the baseline assumes oracle source identification and localization, bypassing a central challenge evaluated by MixCardsBench.

We will include these results and explicitly discuss how dataset priors affect the interpretation of individual field scores. We believe this additional analysis strengthens the benchmark by identifying which fields are most discriminative and where further improvements to data diversity are needed. Random reference mixcards outperform most models on numerical fields and fields with quantized outputs, such as median F0, frequency profile, intelligibility, tempo and plurality. On open-ended fields, such as sound descriptors, sounding action, instruments and descriptive caption, the strongest models beat this baseline. Qwen3-Omni and Gemini 3.1 Pro beat it on every open-ended field, and Qwen2.5-Omni on all but the speech descriptive caption. No model beats random reference mixcards on the overall speech card score (0.694, against 0.662 for the best model).

AuthorsOpenReview ↗

Response to Reviewer M6v3: part 2

We continue addressing reviewer comments below.

W1: Noise input baseline

We also evaluate the models on MixCardBench by replacing audio with noise in stage 2 and providing oracle stage 1 tags. We compute the score improvement for each model m by adding audio, i.e. sm,audio−sm,noises_{m, audio} - s_{m,noise} in the table below:

FieldQwen3-OmniQwen2.5-OmniAF3Music FlamingoAF-NextKimi-Audio
Numerical or quantized fields
Speech loudness+0.823-0.053+0.003+0.095-0.024+0.258
Music loudness+0.432+0.060+0.059+0.147-0.057+0.098
Sfx loudness+0.295+0.086-0.076-0.074-0.118-0.021
Speech median F0+0.849+0.347+0.002+0.007+0.431-0.001
Speech frequency profile+0.734+0.019+0.000+0.007-0.113-0.013
Music frequency profile+0.198+0.143+0.003+0.001-0.015-0.039
Sfx frequency profile+0.103+0.012+0.009+0.004-0.042+0.018
Speech intelligibility+0.330-0.290+0.029+0.382+0.073-0.042
Music tempo+0.388+0.036+0.008-0.154-0.069-0.088
Sfx plurality+0.239+0.037+0.179+0.079-0.215+0.052
Open-ended fields
Speech emotion+0.478+0.012+0.031-0.004+0.242-0.164
Speech manner+0.515-0.027+0.017-0.002-0.221+0.013
Speech perceived audio quality+0.612+0.082+0.013-0.248+0.105+0.013
Music perceived audio quality+0.177+0.006+0.035-0.221-0.077-0.029
Sfx perceived audio quality+0.048+0.008+0.058-0.135-0.108-0.011
Music genre+0.136-0.012+0.091+0.014+0.186+0.012
Music mood+0.438+0.031+0.105+0.009-0.082+0.039
Music instruments+0.653+0.244+0.040-0.064+0.042+0.032
Music descriptors+0.522+0.040+0.025-0.137-0.013-0.018
Sfx sounding object+0.246+0.033+0.115-0.074-0.022-0.001
Sfx sounding action+0.266-0.021+0.123-0.101-0.102-0.002
Sfx sound descriptors+0.168+0.030+0.091-0.107-0.132+0.018
Sfx temporal descriptors+0.174-0.036+0.071-0.135-0.150+0.077
Speech short caption+0.401+0.046+0.005-0.039-0.113+0.014
Music short caption+0.300+0.074+0.012+0.012-0.030+0.012
Sfx short caption+0.165+0.027+0.009-0.018-0.009-0.007
Speech descriptive caption+0.668+0.104+0.011+0.069-0.178-0.005
Music descriptive caption+0.562+0.195+0.000-0.004-0.065-0.019
Sfx descriptive caption+0.224+0.012+0.019-0.040-0.032-0.012
Card score
Speech+0.601+0.026+0.012+0.030+0.022+0.008
Music+0.380+0.082+0.037-0.040-0.018+0.000
Sfx+0.193+0.019+0.059-0.060-0.094+0.011

The results show that the strongest model relies on the audio for every field. Qwen3-Omni scores higher with real audio on all 29 fields, and its card scores drop by 0.601, 0.380 and 0.193 for speech, music and sfx when the audio is replaced. The largest drops are on signal-derived fields such as median F0, loudness and frequency profile, and on detailed descriptions such as instruments and descriptive captions. In contrast, the other five models change little: their card scores move by less than 0.1, and Music Flamingo, AF-Next and Kimi-Audio score higher with noise on more than half of the fields. This likely reflects that these models generate their stage 2 predictions mainly from the oracle tags and dataset priors, which is consistent with their card scores being close to or below the random reference MixCards baseline.

Overall, the effect of the audio is largest for speech and smallest for sfx. The oracle sfx tag already names the sound class, so parts of an sfx MixCard can be guessed without listening, whereas speech fields such as median F0, loudness and perceived audio quality cannot. Moreover, the highest scores on MixCardBench come from the model that uses the audio, while models that do not use it score close to or below the random reference mixcards baseline.

We will update Fig. 4 in the paper with these baseline results.

Q1: Multiple instances of the same source type

Our mixcards do annotate multiple non-overlapping instances of the same sound class. Each instance has its own timestamps, and signal fields such as loudness, frequency profile and median F0 are given per instance. We also indicate plurality within a given MixCard. We leave instance-level non-signal attributes, such as manner of speech and sound descriptors, and time-varying field values to future work.

Thank you for performing additional experiments. The noise-input baseline further supports the paper's findings. Two new baselines address my primary concern that a high score does not necessarily indicate genuine understanding. However, the results of random mixcard are particularly interesting and somewhat support my original concern; notably, it scores high in certain fields, such as speech/music loudness (0.766/0.732), and in some fields marginally better. That's why, in my opinion, it is useful to evaluate the performance with fixed answers. I understand that a random baseline partially addresses this; still, it can select both frequent and less frequent values. Nevertheless, I appreciate the authors effort in including these additional results. I maintain my weak accept recommendation. Best of luck.

Reviewer_ACztOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Propose a benchmark for evaluating multi-source (speech, music, and sound effects) structured captioning of audio, and a data-generation pipeline for soundscapes. They evaluate 6 large audio language models, with the strongest model reaching ~60% mean score.

Critical strengths

The proposed benchmark is constructed in a fine-grained manner, covering multiple audio sources and emphasizing fine-grained attributes, which is beneficial to the community.

Critical weaknesses

The evaluation coverage seems limited, covering only three model series: Qwen, Kimi, and Audio Flamingo. Other model series, such as GPT and Gemini, could be included, especially considering that the Qwen series is involved in benchmark construction.

Questions for authors

Has the consistency between the results on the full benchmark and those on the 774-card validated subset been checked, for example, in terms of score differences and Spearman's rank correlation?

1 comment

We thank the reviewer for their feedback and address their concerns below.

Q1: Consistency between the full benchmark and the validated cards

For each of the 6 models and each of the 29 card fields (174 model × field cells), we compare the score on the full benchmark (4,884 scenes, automatic references) with the score on the validated subset (235 scenes, human-edited references), with 95% confidence intervals from a 2,000-draw scene bootstrap.

MetricSpeechMusicSFXAll
Fields (cells = 6 models x fields)9 (54)10 (60)10 (60)29 (174)
Median |Δ| (max)0.021 (0.102)0.013 (0.117)0.018 (0.089)0.018 (0.117)
Δ 95% CI contains 081.50%86.70%90.00%86.20%
Per-field Spearman ρ, median (min)0.943 (0.886)1.000 (0.943)0.943 (0.771)1.000 (0.771)
Per-field Kendall τ_b, median (min)0.867 (0.733)1.000 (0.867)0.867 (0.600)1.000 (0.600)

Score differences. The difference between the two scores (delta) is small: its median absolute value is 0.018 (maximum 0.117), whereas the median gap between two models on the same field is 0.216, and every |delta| is below its field's median model gap. The 95% CI of delta contains 0 in 86.2% of cells.

Rank agreement. The validated subset ranks the models the same way as the full benchmark. Across the 29 fields, the mean Spearman ρ between the two rankings is 0.96 (Kendall τ_b 0.92), and the ranking is identical in 15 fields. The best and the worst model are the same in 28 of 29 fields. In 12 of the 14 fields that differ, the difference is a single swap of two neighboring models; the other two have two (speech manner) and three (sfx sounding action) swapped pairs. Of these 17 swapped pairs (out of 435), 11 are among Music Flamingo, AF Next and Kimi-Audio, and all 17 are within bootstrap noise on the validated subset.

W1: Evaluation of closed-source models

We evaluated GPT-Audio-Mini, Gemini 3.8 Live, and Gemini Pro on our validated benchmark, and present the results below. Oracle tags refers to ground truth stage 1 tags conditioning model's stage 2 predictions. Predicted tags refers to the model's own stage 1 predictions conditioning stage 2 predictions.

Results on MixCardBench-val (565)Stage 1 (event/ segment)Stage 2, mixcards conditioned on oracle tags (speech/music/sfx)Stage 2, mixcards conditioned on predicted tags (speech/music/sfx)
Qwen3-Omni-Instruct0.416/0.6800.607 (0.655/0.579/0.598)0.616 (0.647/0.599/0.608)
GPT-Audio-Mini0.159/0.5440.550 (0.658/0.485/0.541)0.555 (0.673/0.481/0.545)
Gemini 3.8 Live0.214/0.5920.573 (0.658/0.565/0.502)0.581 (0.678/0.573/0.499)
Gemini Pro0.305/0.6560.601 (0.646/0.616/0.571)0.603 (0.658/0.625/0.562)

We find that Gemini Pro is similar to Qwen3-Omni in mixcard generation (stage 2) but performs slightly worse in sound event detection (stage 1). On the other hand, smaller closed-source models are not as competitive on stage 2 and even less so on stage 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Uploaded a revised version of the paper

Thank you for your valuable feedback during this discussion phase. Please see our updated paper, which highlights changes in blue text in the main body and in captions for new or updated tables and figures. The changes are mainly in Section 5 and Appendix H.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.