- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents AudioWeb-365M, a dataset of 365 million audio-text pairs curated from 1.47 billion web and dataset audio samples through a pipeline of cleaning, two-tier deduplication (acoustic fingerprint matching and semantic near-duplicate screening), benchmark quarantine, taxonomy-based selection, and automatic captioning. It is the largest publicly released resource of its kind. Using this dataset, the authors train five CLAP model tiers (XXS–L) and conduct a systematic scaling study. The central conclusion is that, under a fixed compute budget, "training smaller models on more samples is preferable to increasing parameter count": each doubling of compute reduces the MAEB-30 error rate by approximately 4.1%, and low-compute runs can be extrapolated to accurately predict the compute-Pareto frontier models. In particular, the 153M-parameter M model reaches 52.3% on MAEB-30 (best among CLAP-style models, fourth on the official leaderboard), rivaling multimodal foundation models with billions of parameters. The paper commits to releasing the dataset indices, captions, model checkpoints, and code, positioning the work as reusable infrastructure for contrastive language-audio pretraining.
Critical strengths
-
The paper distills 365 million audio-text pairs (AudioWeb-365M) from 1.47 billion web and dataset audio samples — an order of magnitude larger than the previous largest public audio-text collection (approximately 50 million pairs). The pipeline spans signal checks, two-tier deduplication, benchmark quarantine, selection over a 4,416-class sound taxonomy, and automatic captioning, and is described as "fully documented," with a commitment to release indices, captions, metadata, and code.
-
The experiments cover five model tiers from 27.3M to 340.1M parameters across three orders of magnitude of data scale, yielding quantitative conclusions such as "each doubling of compute reduces the MAEB-30 error rate by about 4.1% (α ≈ -0.063)" and "training smaller models on more samples is more effective than increasing parameter count by more than 3×." Runs below 10¹¹ GFLOPs accurately extrapolate the compute-Pareto frontier (mean absolute error of 0.5 points).
-
Evaluation spans three major benchmarks — MAEB-30 (30 tasks), HEAR (17 tasks), and MSEB (26 tasks) — with head-to-head comparisons against more than ten audio-text and multimodal models: 52.3% on MAEB-30 (best among CLAP-style models, fourth on the official leaderboard), 81.4–82.0 on HEAR (best among audio-text models), and 29.1 on MSEB (second place, surpassing all multimodal models with billions of parameters), all achieved with a 153M-parameter model comparable to billion-parameter counterparts.
Critical weaknesses
-
Captions lack per-sample validation or filtering, and the human evaluation agreement is low, undermining the reliability of the dataset's core asset.
-
The extrapolation range of the scaling conclusions is limited, and the generality of the central scientific claims has not been sufficiently validated.
Questions for authors
-
Among the 470 million generated captions, is there any sampled estimate of the "factual accuracy" or hallucination rate? Beyond the 50-sample human listening study, is there a larger-scale (e.g., ≥1,000 samples) automatic or human faithfulness audit? If so, please report the accuracy and confidence intervals; if not, please explain why per-sample quality filtering (e.g., thresholding on LLM scores) was not performed.
-
The three annotators agreed on the top choice for only 12 out of 50 samples (Kendall τ = 0.248), and the LLM–human correlation was only τ = 0.345. Given such low agreement, how robust is the selection conclusion that "Qwen3-Omni + MOSS-Audio is the best choice"? Were any alternative evaluation protocols with higher annotator agreement attempted (e.g., pairwise comparisons, anchoring-based calibration)?
-
Benchmark quarantine applied only three-tier matching (identifier / hash / waveform) against the evaluation audio of MAEB, HEAR, and MSEB. However, caption training may indirectly include evaluation content (e.g., YouTube subtitles or surrounding web text). Has the risk of overlap between the text side (captions or web context) and evaluation task labels/descriptions been assessed?
-
The fit of L(C) = A_c·(C + B_c)^(−α_c) + E_c only covers up to 340.1M parameters, a single dataset, and a single architecture. Is there any evidence (e.g., small-scale cross-validation, controls varying the architecture or data distribution) supporting the generalization of α ≈ −0.063 and the "more data beats more parameters" conclusion to larger models or other audio pretraining objectives?
-
The conclusion states that the models can serve as drop-in replacements for CLAP models in downstream applications (including text-to-audio generation), yet no generative downstream experiments appear in the main text. Are there unreported downstream generation or fine-tuning results that support this claim?
3 comments
The conclusion states that the models can serve as drop-in replacements for CLAP models in downstream applications (including text-to-audio generation), yet no generative downstream experiments appear in the main text. Are there unreported downstream generation or fine-tuning results that support this claim?
We thank the reviewer for pointing out the missing downstream experiment to support our statement. To fill this gap, we trained two downstream models (1) MusicLDM and (2) Audio Flamingo 2 (AF2).
(1) Text-to-music generation: We train MusicLDM [2] to test whether our model's embeddings benefit a generative model. We replace CLAP in the MusicLDM architecture with our model and condition the latent diffusion UNet on its frozen embeddings. We train on 10,000 music tracks and test on 1,000 held-out tracks. We keep all settings the same and only swap the frozen conditioner between MusicLDM's original LAION-CLAP and ours-M.
As shown in the table below, conditioning with our model outperforms the LAION-CLAP-based version in Fréchet Distance (PANNs), Fréchet Audio Distance (VGGish), and KL divergence (KL). While our version loses on inception score (IS), IS rewards confident AudioSet sound-event predictions and does not measure fidelity to the reference music.
| Conditioner (frozen) | FD(PANN) ↓ | FAD(VGGish) ↓ | KL ↓ | IS ↑ |
|---|---|---|---|---|
| LAION-CLAP (retrained) | 15.81 | 3.55 | 4.16 | 2.16 |
| Ours-M | 13.07 | 1.33 | 3.78 | 1.83 |
(2) Audio-language model: We replaced the audio encoder of Audio Flamingo 2 (AF2) [3]. AF2 originally uses its own purpose-built and much larger CLAP model (656M parameters in total), designed specifically for AF2. We therefore regard it as an upper reference rather than a general-purpose CLAP model. In our experiment, we keep the CLAP model and the language model frozen for all variants and train AF2’s adapter layers, plus a linear projection that feeds the frame-level tokens into AF2, with an identical recipe and data for every encoder.
We evaluate on a total of 11 benchmarks spanning question answering (QA) and classification. Our model M outperforms slightly larger CLAP alternatives on average, by +1.3 points on QA benchmarks and +5.2 points on classification tasks. Especially noticeable are the low ESC-50 scores where AF2 based on other general purpose CLAP models struggles to answer unfamiliar open questions reliably (ours 31.6 vs at most 9.2 points) often failing to name any sound. When only considering answers containing a sound, we still outperform other general purpose models (ours 31.6 vs at most 14.6 points). The native AF2 CLAP ties our model M on QA, even though it is more than four times larger. It outperforms our model M by +8 points on classification, which we attribute to its tight integration into the AF2 architecture, custom training data and the difference in model size.
| Model (QA) | MMAU-mini | MMAR | MMSU | MuChoMusic | SAKURA | Mean |
|---|---|---|---|---|---|---|
| AF-CLAP (ref.) | 56.6 | 37.7 | 36.4 | 44.8 | 43.4 | 43.8 |
| LAION larger CLAP | 53.1 | 38.1 | 37.0 | 41.4 | 39.1 | 41.7 |
| M2D-CLAP | 51.7 | 37.8 | 35.9 | 41.4 | 37.9 | 40.9 |
| TP-CLAP | 53.8 | 37.0 | 36.0 | 41.3 | 43.2 | 42.3 |
| Ours-M | 56.0 | 38.9 | 36.1 | 45.0 | 42.2 | 43.6 |
| Model (Classification) | VocalSound | CREMA-D | VoxCeleb gender | ESC-50 | NSynth | GTZAN | Mean |
|---|---|---|---|---|---|---|---|
| AF-CLAP (ref.) | 88.7 | 15.7 | 99.7 | 49.2 | 84.2 | 45.6 | 63.9 |
| LAION larger CLAP | 73.4 | 24.0 | 98.8 | 5.0 | 51.2 | 37.2 | 48.3 |
| M2D-CLAP | 84.9 | 15.8 | 98.8 | 4.5 | 36.9 | 34.8 | 45.9 |
| TP-CLAP | 89.3 | 17.4 | 99.6 | 9.2 | 52.3 | 36.6 | 50.7 |
| Ours-M | 91.7 | 19.3 | 97.1 | 31.6 | 64.9 | 31.0 | 55.9 |
*Audio Flamingo 2 with Qwen2.5-1.5B, accuracy in %. Please note that the CLAP models’ training data was not screened against these benchmarks and partially overlaps some of them. The results should therefore be seen as indicative rather than conclusive. We utilize AF2's own evaluation protocol using open question format and mapping the answers to a label with LAION larger CLAP.
We included both experiments in the revised paper in Appendix G.
We thank the reviewer for their interesting and useful questions. We would love to hear about any other concerns raised.
[2] K. Chen, et al. "MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies." ICASSP 2024.
[3] S. Ghosh, et al. "Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities." ICML 2025.
Dear Reviewer kdBE,
As the discussion period is ending soon, we wanted to check if our response has addressed your concerns and if there are any remaining questions. Happy to clarify anything before the deadline.
We would like to thank the reviewer for their thorough feedback on our paper. We are pleased that the reviewer recognizes multiple strengths in our work, including the scale of our dataset, larger than the previous largest public audio-text collection, and its documented curation pipeline. We appreciate that the reviewer notes our scaling experiments across five model sizes and three orders of magnitude of data, as well as the broad evaluation on 3 benchmarks MAEB-30, HEAR and MSEB, where our models are comparable to much larger models.
We thank the reviewer for bringing up the points on captioning validation and annotator agreement.
For per-sample validation, 700k captions that failed on first try were retried, 250k audios with an empty or half-filled caption were captioned again, until all were filled. In total, the prompt changed about 25 times each time checking its audio captions, before it was settled. Before training our CLAP models, on a random sample of 20,997 final captions, a Gemini 3.6 Flash judge rated only 5.6% [5.3, 5.9] as incorrect. Separately, we also verified subsentences (i.e. a man speaks calmly → a man speaks + he speaks calmly) of 600 clips before submission, only 10.6% [9.2, 12.0] (MOSS) and 13.0% [11.5, 14.7] (Qwen3-Omni) of single subsentences were not found in the audio by the Gemini 3.6 Flash judge.
Next to the studies in the paper (where Qwen3-Omni and MOSS ranked top 3 on 8,997 clips with a Gemini 3.6 Flash judge and on 50 clips with 2 human annotators ordering all 7 models, τ=0.295 between annotators and τ=0.345 with the LLM judge), we did a new pairwise study as the reviewer suggested, where three raters comparing caption pairs on 200 clips moderately agree on single clips (τ=0.19–0.32) and had Qwen3-Omni and both MOSS models as their top 3 best performing models. Separately, we also evaluated with 3 foundation models with audio capability (MiMo-V2.6-Pro, Gemini 4 Carbon, and Inkling-975B), and gave them 2200 samples, and again this showed, MOSS thinking off and on took first and second place with 64.8% and 61.8% win rate (pairwise win rate against the other 6 models in the 7-model ranking) respectively, and Qwen3-Omni came third with 60.5% win rate.
Thank you for this question on the generality of our scaling fits. To the best of our knowledge, our work covers the largest data scale (1M to 1,024M) and the most model sizes (5) out of all audio-text scaling studies. A full study covering multiple datasets, architectures, and objectives would be a standalone publication [1], which we leave to future work. Within the scope of this publication, a scaling function fitted on a smaller compute range (up to 1011 GFLOPs) accurately predicts five larger held-out runs with a mean absolute error of 0.64 MAEB-30 points (Figure 3, 13 in new version). In addition, we added a subsection to the main paper on compute-optimal allocation (Nopt∝C0.405, Dopt∝C0.583), which shows that data should grow about 1.4$\times$ faster than model size within our compute range. This is consistent with the discrete audio-language model SODA (C0.367, C0.579, i.e. about 1.58$\times$), which uses a different training objective. We did not train models beyond 340M parameters because our fits indicate that larger models would require significantly more unique data than our 365M-sample dataset provides.
We appreciate the reviewer raising this concern on text-side overlap.
We compared all captions across our 365M clips (both original and generated captions) against all 6.65M distinct texts from MAEB, MSEB, and HEAR, and found that only 235 captions (136 distinct texts) out of 365M clips have an exact match (or 563 captions across 326 distinct texts when ignoring case and punctuation), where almost all of them are either single or a couple of words long, or are general terms like "people are talking and walking" or "the sound of children playing".
[1] Nezhurina, Marianna, et al. "Scaling laws for robust comparison of open foundation language-vision models and datasets." Advances in Neural Information Processing Systems 38 (2026): 153979-154018.