acceptodds
Under review as a conference paper at ICLR 2027

AudioWeb-365M: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining

Abstract

Recent advances in contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.47 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process it with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our models outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. As a result, our models can serve as a drop-in replacement for CLAP in downstream applications. We will publicly release our dataset, captions, model checkpoints, and code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
contrastive language-audio pretraining, audio-text representation learning, audio-text retrieval, scaling laws, data curation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kdBEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents AudioWeb-365M, a dataset of 365 million audio-text pairs curated from 1.47 billion web and dataset audio samples through a pipeline of cleaning, two-tier deduplication (acoustic fingerprint matching and semantic near-duplicate screening), benchmark quarantine, taxonomy-based selection, and automatic captioning. It is the largest publicly released resource of its kind. Using this dataset, the authors train five CLAP model tiers (XXS–L) and conduct a systematic scaling study. The central conclusion is that, under a fixed compute budget, "training smaller models on more samples is preferable to increasing parameter count": each doubling of compute reduces the MAEB-30 error rate by approximately 4.1%, and low-compute runs can be extrapolated to accurately predict the compute-Pareto frontier models. In particular, the 153M-parameter M model reaches 52.3% on MAEB-30 (best among CLAP-style models, fourth on the official leaderboard), rivaling multimodal foundation models with billions of parameters. The paper commits to releasing the dataset indices, captions, model checkpoints, and code, positioning the work as reusable infrastructure for contrastive language-audio pretraining.

Critical strengths
  • The paper distills 365 million audio-text pairs (AudioWeb-365M) from 1.47 billion web and dataset audio samples — an order of magnitude larger than the previous largest public audio-text collection (approximately 50 million pairs). The pipeline spans signal checks, two-tier deduplication, benchmark quarantine, selection over a 4,416-class sound taxonomy, and automatic captioning, and is described as "fully documented," with a commitment to release indices, captions, metadata, and code.

  • The experiments cover five model tiers from 27.3M to 340.1M parameters across three orders of magnitude of data scale, yielding quantitative conclusions such as "each doubling of compute reduces the MAEB-30 error rate by about 4.1% (α ≈ -0.063)" and "training smaller models on more samples is more effective than increasing parameter count by more than 3×." Runs below 10¹¹ GFLOPs accurately extrapolate the compute-Pareto frontier (mean absolute error of 0.5 points).

  • Evaluation spans three major benchmarks — MAEB-30 (30 tasks), HEAR (17 tasks), and MSEB (26 tasks) — with head-to-head comparisons against more than ten audio-text and multimodal models: 52.3% on MAEB-30 (best among CLAP-style models, fourth on the official leaderboard), 81.4–82.0 on HEAR (best among audio-text models), and 29.1 on MSEB (second place, surpassing all multimodal models with billions of parameters), all achieved with a 153M-parameter model comparable to billion-parameter counterparts.

Critical weaknesses
  • Captions lack per-sample validation or filtering, and the human evaluation agreement is low, undermining the reliability of the dataset's core asset.

  • The extrapolation range of the scaling conclusions is limited, and the generality of the central scientific claims has not been sufficiently validated.

Questions for authors
  • Among the 470 million generated captions, is there any sampled estimate of the "factual accuracy" or hallucination rate? Beyond the 50-sample human listening study, is there a larger-scale (e.g., ≥1,000 samples) automatic or human faithfulness audit? If so, please report the accuracy and confidence intervals; if not, please explain why per-sample quality filtering (e.g., thresholding on LLM scores) was not performed.

  • The three annotators agreed on the top choice for only 12 out of 50 samples (Kendall τ = 0.248), and the LLM–human correlation was only τ = 0.345. Given such low agreement, how robust is the selection conclusion that "Qwen3-Omni + MOSS-Audio is the best choice"? Were any alternative evaluation protocols with higher annotator agreement attempted (e.g., pairwise comparisons, anchoring-based calibration)?

  • Benchmark quarantine applied only three-tier matching (identifier / hash / waveform) against the evaluation audio of MAEB, HEAR, and MSEB. However, caption training may indirectly include evaluation content (e.g., YouTube subtitles or surrounding web text). Has the risk of overlap between the text side (captions or web context) and evaluation task labels/descriptions been assessed?

  • The fit of L(C) = A_c·(C + B_c)^(−α_c) + E_c only covers up to 340.1M parameters, a single dataset, and a single architecture. Is there any evidence (e.g., small-scale cross-validation, controls varying the architecture or data distribution) supporting the generalization of α ≈ −0.063 and the "more data beats more parameters" conclusion to larger models or other audio pretraining objectives?

  • The conclusion states that the models can serve as drop-in replacements for CLAP models in downstream applications (including text-to-audio generation), yet no generative downstream experiments appear in the main text. Are there unreported downstream generation or fine-tuning results that support this claim?

3 comments
  1. The paper distills 365 million audio-text pairs (AudioWeb-365M) […]
  2. The experiments cover five model tiers from 27.3M to 340.1M parameters […]
  3. Evaluation spans three major benchmarks […]

We would like to thank the reviewer for their thorough feedback on our paper. We are pleased that the reviewer recognizes multiple strengths in our work, including the scale of our dataset, larger than the previous largest public audio-text collection, and its documented curation pipeline. We appreciate that the reviewer notes our scaling experiments across five model sizes and three orders of magnitude of data, as well as the broad evaluation on 3 benchmarks MAEB-30, HEAR and MSEB, where our models are comparable to much larger models.

Captions lack per-sample validation or filtering, and the human evaluation agreement is low […]

[…] is there any sampled estimate of the "factual accuracy" or hallucination rate? […]

The three annotators agreed on the top choice for only 12 out of 50 samples […]

We thank the reviewer for bringing up the points on captioning validation and annotator agreement.

For per-sample validation, 700k captions that failed on first try were retried, 250k audios with an empty or half-filled caption were captioned again, until all were filled. In total, the prompt changed about 25 times each time checking its audio captions, before it was settled. Before training our CLAP models, on a random sample of 20,997 final captions, a Gemini 3.6 Flash judge rated only 5.6% [5.3, 5.9] as incorrect. Separately, we also verified subsentences (i.e. a man speaks calmly →\rightarrow a man speaks + he speaks calmly) of 600 clips before submission, only 10.6% [9.2, 12.0] (MOSS) and 13.0% [11.5, 14.7] (Qwen3-Omni) of single subsentences were not found in the audio by the Gemini 3.6 Flash judge.

Next to the studies in the paper (where Qwen3-Omni and MOSS ranked top 3 on 8,997 clips with a Gemini 3.6 Flash judge and on 50 clips with 2 human annotators ordering all 7 models, τ=0.295\tau = 0.295 between annotators and τ=0.345\tau = 0.345 with the LLM judge), we did a new pairwise study as the reviewer suggested, where three raters comparing caption pairs on 200 clips moderately agree on single clips (τ=0.19–0.32\tau = 0.19\text{--}0.32) and had Qwen3-Omni and both MOSS models as their top 3 best performing models. Separately, we also evaluated with 3 foundation models with audio capability (MiMo-V2.6-Pro, Gemini 4 Carbon, and Inkling-975B), and gave them 2200 samples, and again this showed, MOSS thinking off and on took first and second place with 64.8% and 61.8% win rate (pairwise win rate against the other 6 models in the 7-model ranking) respectively, and Qwen3-Omni came third with 60.5% win rate.

The extrapolation range of the scaling conclusions is limited […]

The fit […] only covers up to 340.1M parameters, a single dataset, and a single architecture. […]

Thank you for this question on the generality of our scaling fits. To the best of our knowledge, our work covers the largest data scale (1M to 1,024M) and the most model sizes (5) out of all audio-text scaling studies. A full study covering multiple datasets, architectures, and objectives would be a standalone publication [1], which we leave to future work. Within the scope of this publication, a scaling function fitted on a smaller compute range (up to 101110^{11} GFLOPs) accurately predicts five larger held-out runs with a mean absolute error of 0.64 MAEB-30 points (Figure 3, 13 in new version). In addition, we added a subsection to the main paper on compute-optimal allocation (Nopt∝C0.405N_{\text{opt}} \propto C^{0.405}, Dopt∝C0.583D_{\text{opt}} \propto C^{0.583}), which shows that data should grow about 1.4$\times$ faster than model size within our compute range. This is consistent with the discrete audio-language model SODA (C0.367C^{0.367}, C0.579C^{0.579}, i.e. about 1.58$\times$), which uses a different training objective. We did not train models beyond 340M parameters because our fits indicate that larger models would require significantly more unique data than our 365M-sample dataset provides.

[…] Has the risk of overlap between the text side […] and evaluation task labels/descriptions been assessed?

We appreciate the reviewer raising this concern on text-side overlap.

We compared all captions across our 365M clips (both original and generated captions) against all 6.65M distinct texts from MAEB, MSEB, and HEAR, and found that only 235 captions (136 distinct texts) out of 365M clips have an exact match (or 563 captions across 326 distinct texts when ignoring case and punctuation), where almost all of them are either single or a couple of words long, or are general terms like "people are talking and walking" or "the sound of children playing".

[1] Nezhurina, Marianna, et al. "Scaling laws for robust comparison of open foundation language-vision models and datasets." Advances in Neural Information Processing Systems 38 (2026): 153979-154018.

The conclusion states that the models can serve as drop-in replacements for CLAP models in downstream applications (including text-to-audio generation), yet no generative downstream experiments appear in the main text. Are there unreported downstream generation or fine-tuning results that support this claim?

We thank the reviewer for pointing out the missing downstream experiment to support our statement. To fill this gap, we trained two downstream models (1) MusicLDM and (2) Audio Flamingo 2 (AF2).

(1) Text-to-music generation: We train MusicLDM [2] to test whether our model's embeddings benefit a generative model. We replace CLAP in the MusicLDM architecture with our model and condition the latent diffusion UNet on its frozen embeddings. We train on 10,000 music tracks and test on 1,000 held-out tracks. We keep all settings the same and only swap the frozen conditioner between MusicLDM's original LAION-CLAP and ours-M.

As shown in the table below, conditioning with our model outperforms the LAION-CLAP-based version in Fréchet Distance (PANNs), Fréchet Audio Distance (VGGish), and KL divergence (KL). While our version loses on inception score (IS), IS rewards confident AudioSet sound-event predictions and does not measure fidelity to the reference music.

Conditioner (frozen)FD(PANN) ↓FAD(VGGish) ↓KL ↓IS ↑
LAION-CLAP (retrained)15.813.554.162.16
Ours-M13.071.333.781.83

(2) Audio-language model: We replaced the audio encoder of Audio Flamingo 2 (AF2) [3]. AF2 originally uses its own purpose-built and much larger CLAP model (656M parameters in total), designed specifically for AF2. We therefore regard it as an upper reference rather than a general-purpose CLAP model. In our experiment, we keep the CLAP model and the language model frozen for all variants and train AF2’s adapter layers, plus a linear projection that feeds the frame-level tokens into AF2, with an identical recipe and data for every encoder.

We evaluate on a total of 11 benchmarks spanning question answering (QA) and classification. Our model M outperforms slightly larger CLAP alternatives on average, by +1.3 points on QA benchmarks and +5.2 points on classification tasks. Especially noticeable are the low ESC-50 scores where AF2 based on other general purpose CLAP models struggles to answer unfamiliar open questions reliably (ours 31.6 vs at most 9.2 points) often failing to name any sound. When only considering answers containing a sound, we still outperform other general purpose models (ours 31.6 vs at most 14.6 points). The native AF2 CLAP ties our model M on QA, even though it is more than four times larger. It outperforms our model M by +8 points on classification, which we attribute to its tight integration into the AF2 architecture, custom training data and the difference in model size.

Model (QA)MMAU-miniMMARMMSUMuChoMusicSAKURAMean
AF-CLAP (ref.)56.637.736.444.843.443.8
LAION larger CLAP53.138.137.041.439.141.7
M2D-CLAP51.737.835.941.437.940.9
TP-CLAP53.837.036.041.343.242.3
Ours-M56.038.936.145.042.243.6
Model (Classification)VocalSoundCREMA-DVoxCeleb genderESC-50NSynthGTZANMean
AF-CLAP (ref.)88.715.799.749.284.245.663.9
LAION larger CLAP73.424.098.85.051.237.248.3
M2D-CLAP84.915.898.84.536.934.845.9
TP-CLAP89.317.499.69.252.336.650.7
Ours-M91.719.397.131.664.931.055.9

*Audio Flamingo 2 with Qwen2.5-1.5B, accuracy in %. Please note that the CLAP models’ training data was not screened against these benchmarks and partially overlaps some of them. The results should therefore be seen as indicative rather than conclusive. We utilize AF2's own evaluation protocol using open question format and mapping the answers to a label with LAION larger CLAP.

We included both experiments in the revised paper in Appendix G.

We thank the reviewer for their interesting and useful questions. We would love to hear about any other concerns raised.

[2] K. Chen, et al. "MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies." ICASSP 2024.

[3] S. Ghosh, et al. "Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities." ICML 2025.

Dear Reviewer kdBE,

As the discussion period is ending soon, we wanted to check if our response has addressed your concerns and if there are any remaining questions. Happy to clarify anything before the deadline.

Reviewer_4fUDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper builds an audio–text dataset by collecting existing datasets and web audio, applying deduplication and selection, and generating long/short captions with audio-language models (Qwen3-30B and Moss-8B). It then trains contrastive models at different sizes and budgets to study scaling and practical model–data trade-offs.

Critical strengths
  1. The paper is well-written, and the dataset creation pipeline is mostly sound and clear. The appendix clearly documents data sources, processing steps, and selection rules. The complementary acoustic and semantic deduplication stages are sensible quality-control measures, making the dataset construction more transparent and credible.

  2. The model-size and training-budget comparisons provide useful guidance on allocating a fixed compute budget.

Critical weaknesses
  1. My major concern is the caption reliability. The paper employed two large audio-language models (Qwen3-30B and Moss-8B) as automatic captioner. I believe that it requires at least verifying the two captioner's accuracy on several established captioning benchmarks before using them as reliable large-scale automatic labeler (Section 3.4 should have described this information but unfortunately not.)

Appendix C primarily reports "relative rankings" rather than "factual accuracy" or acceptance rates, with human evaluation covering only 50 clips. Qwen3, which captions half of the automatically annotated data, ranks fifth in human evaluation. These results do not adequately establish annotation reliability, particularly for event counts, temporal ordering, and unsupported details introduced through predicted metadata.

Moreover, the auto-generated captions do not need to be uniformly incorrect for this to be a serious concern: systematic errors can be amplified at scale. For example, recurring mistakes about emotion and speaker identity (in my experience, Moss-8B makes a lot of such mistakes), could propagate into downstream audio captioning and understanding research.

  1. The comparison with previous dataset-building and scaling studies is insufficient. Tseng et al. (2026) already aggregate diverse audio–text corpora and investigate scaling, yet is absent from Table 1 and receives only a brief discussion. The paper does not clearly establish which limitations of previous work it addresses or which experimental conclusions are new, beyond reproducing related analyses at a larger scale.
Questions for authors
  1. Figure 3 (right) is crowded, making the model-size trade-off and possible saturation difficult to read. Please clarify what the curves imply about the benefits of further data scaling.

  2. For speech clips, are captions intended to describe spoken content, paralinguistic attributes, or both? The intended supervision would be clearer with the actual instructions and representative short/long captions.

  3. When predicted metadata conflicts with the audio, are captioners instructed to prioritize the audio? This is unclear from the current prompt description.

4 comments
  1. The paper is well-written, and the dataset creation pipeline is mostly sound and clear. The appendix clearly documents data sources, processing steps, and selection rules. The complementary acoustic and semantic deduplication stages are sensible quality-control measures, making the dataset construction more transparent and credible.
  2. The model-size and training-budget comparisons provide useful guidance on allocating a fixed compute budget.

We would like to thank the reviewer for their thorough feedback on our paper. We appreciate that the reviewer recognizes several strengths in our work, including the data curation pipeline, with extensive steps documented in the appendix, noting the deduplication stages and our transparent data construction pipeline.

My major concern is the caption reliability. The paper employed two large audio-language models (Qwen3-30B and Moss-8B) as automatic captioner. I believe that it requires at least verifying the two captioner's accuracy on several established captioning benchmarks before using them as reliable large-scale automatic labeler (Section 3.4 should have described this information but unfortunately not.)

Appendix C primarily reports "relative rankings" rather than "factual accuracy" or acceptance rates, with human evaluation covering only 50 clips. Qwen3, which captions half of the automatically annotated data, ranks fifth in human evaluation. These results do not adequately establish annotation reliability, particularly for event counts, temporal ordering, and unsupported details introduced through predicted metadata.

Moreover, the auto-generated captions do not need to be uniformly incorrect for this to be a serious concern: systematic errors can be amplified at scale. For example, recurring mistakes about emotion and speaker identity (in my experience, Moss-8B makes a lot of such mistakes), could propagate into downstream audio captioning and understanding research.

We thank the reviewer for bringing up the point on captioning reliability. Our main two captioning models were chosen with two notions in mind: accuracy and speed of inference. Our initial argument to include MOSS-Audio-8B-Thinking and Qwen3-Omni-30B-A3B was that at the time of annotation, they scored the best on the most popular general open-source audio understanding benchmarks: they were first and second on MMAU Pro and MMSU, and third and first respectively on MMAR.

Before training our CLAP models, on a random sample of 20,997 final captions, a Gemini 3.6 Flash judge rated only 5.6% [5.3, 5.9] as incorrect. Separately, we also verified subsentences (i.e. a man speaks calmly →\rightarrow a man speaks + he speaks calmly) of 600 clips before submission, where only 10.6% [9.2, 12.0] (MOSS) and 13.0% [11.5, 14.7] (Qwen3-Omni) of single subsentences were not found in the audio by the Gemini 3.6 Flash judge. Next to the studies in the paper, we did a new pairwise study during the rebuttal phase, where three raters comparing caption pairs on 200 clips had Qwen3-Omni and both MOSS models as their top 3 best performing models, and we evaluated with 3 foundation models with audio capability (MiMo-V2.6-Pro, Gemini 4 Carbon, and Inkling-975B) on 2200 samples:

CaptionerSpeed (clips/s) ↑Vocab @400k ↑Unsupported Subsentences (600 clips) ↓Paper: Human Rank (50 clips) ↓Paper: LLM Rank (8,997 clips) ↓Rebuttal: Human Win Rate (200 clips) ↑Rebuttal: 3-Judge Win Rate (2,200 clips) ↑
MOSS-Audio-8B-Thinking (used)22.916,61310.6%2.912.7653.5%61.8%
MOSS-Audio-8B-Thinking (think-OFF)19.215,76813.6%3.242.6163.7%64.8%
Qwen3-Omni-30B-A3B (used)75.96,61913.0%4.323.4355.3%60.5%
Qwen2.5-Omni-7B23.08,66015.6%4.104.4351.6%44.8%
AudioFlamingoNext55.59,85824.0%4.264.5138.0%47.2%
Nemotron-3-Nano-Omni96.57,07825.4%4.575.1644.2%38.5%
Audex-30B-A3B36.59,69733.9%4.605.1043.4%32.5%

Based on all this empirical analysis, we believe these were really the best models for our task at that time of annotation and data collection.

The comparison with previous dataset-building and scaling studies is insufficient. Tseng et al. (2026) already aggregate diverse audio–text corpora and investigate scaling, yet is absent from Table 1 and receives only a brief discussion. The paper does not clearly establish which limitations of previous work it addresses or which experimental conclusions are new, beyond reproducing related analyses at a larger scale.

We are grateful to the reviewer for mentioning Tseng et al. (2026), it was included into Section 2 (Related Work) but is now also included into Table 1. While the work of Tseng et al. (2026) is interesting and useful research, their focus is on how contrastive and captioning models are scaled over 4 different dataset sizes: 0.4M, 1M, 4M and 10M. Our work focuses on a fixed compute budget between model size and samples seen, from which a compute optimum dataset and model can be calculated. It works scaling out over five different model sizes, over 11 different samples-seen scales (samples doubled between 1M to 1024M), showing that both increase in parameters and increase in dataset improves performance and predicting scaling at higher compute. Our work particularly shows sound understanding performance saturating sooner than speech and music.

Figure 3 (right) is crowded, making the model-size trade-off and possible saturation difficult to read. Please clarify what the curves imply about the benefits of further data scaling.

Thank you for the helpful feedback on Figure 3. The figure shows that the initially compute-optimal models XXS and XS saturate, while models M and L perform well at larger compute budgets. We redesigned the figure to make it easier to understand and adapted the related text in the caption and main body.

For speech clips, are captions intended to describe spoken content, paralinguistic attributes, or both? The intended supervision would be clearer with the actual instructions and representative short/long captions.

We appreciate this question on the captioning instructions. We kept the instructions for the captioning models relatively general and did not specify whether spoken content should be transcribed or emotions captured. We only tell the model to “cover ... the audible character of speech or music when present”. The synthetically generated captions describe either just the spoken content, spoken content with paralinguistic attributes, or just the paralinguistic attributes. We have added the full captioning prompt to the appendix, including a sample of a short and long caption for the categories of music, sound, and speech.

When predicted metadata conflicts with the audio, are captioners instructed to prioritize the audio? This is unclear from the current prompt description.

Thank you for asking about this. Yes, the captioners are instructed to use the metadata only as supporting context. The prompt describes that "Automated audio analysis may be imperfect" and that it should "use only as supporting context" and to "describe what you actually hear rather than restating these labels". The prompt also says: "Never invent details you cannot hear".

[Not a weakness in paper quality, just a reminder before dataset release] Please clarify whether the unverified source licenses noted in lines 1800–1807 will be checked before release, and how sensitive, violent, or other content potentially unsuitable for public release will be screened and handled.

Thank you for raising this concern regarding screening of the data. For sources without a verified redistribution license (such as Common Crawl and YouTube), we do not redistribute the audio and only release the links, captions, and metadata. To screen and remove unsuitable content, we checked all 365M clips against the University of Toulouse (UT1) domain blacklist (5.8M domains across 69 risk categories) and trained an audio safety classifier predicting both explicit audio and gore/real violence on top of our largest CLAP audio embeddings, calibrated by manually labelling $\sim$1,250 clips (reaching 95.1% accuracy on a held-out audio test set and 0.84 AUROC on gore). Using this pipeline, we removed 26,701 clips (83.7 hours) from the released dataset: 26,192 explicit audio clips and 509 gore and real-violence clips.

We thank the reviewer for their interesting and useful questions. We would love to hear about any other concerns raised.

Dear Reviewer 4fUD,

As the discussion period is ending soon, we wanted to check if our response has addressed your concerns and if there are any remaining questions. Happy to clarify anything before the deadline.

The author's response addressed my major concern on caption reliability. I will raise my score.

Reviewer_ixurOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds an audio–text corpus of roughly 365M short clips from web sources. Processing includes filtering, deduplication, captioning, and benchmark quarantine. The authors train five contrastive dual-encoder sizes and study scaling. Planned releases include indices, captions, metadata, checkpoints, and code.

Critical strengths
  1. The 365M corpus offers substantial resource value if its index, captions, and processing records are fully released.

  2. Five model sizes and multiple training budgets provide systematic evidence.

Critical weaknesses
  1. The component methods have precedents.

  2. Final corpus counts conflict: 364,955,979 versus 366,104,195 clips. Applied semantic removals are also far fewer than proposed removals. This obscures the identity of the training resource.

Questions for authors
  1. Good data work. I don't have any other question.
2 comments

We would like to thank the reviewer for their thorough feedback on our paper.

  1. The 365M corpus offers substantial resource value if its index, captions, and processing records are fully released.
  2. Five model sizes and multiple training budgets provide systematic evidence.

We are pleased that the reviewer recognized several strengths in our work, noting our substantial dataset that is fully open as a valuable resource, as well as our variety of model sizes providing comprehensive evidence.

The component methods have precedents.

We appreciate the reviewer's observation. We agree that individual components in our data pipeline and model implementation are established. However, several methods in the data pipeline were used in limited form but never on this scale, same for the model implementation. Since these methods are already used in the field, we know what to expect from their effect, which helps us focus on the truly novel work in our paper, including: (1) the corpus of around 365M audio-text pairs from 467 sources, including 90% (1.3M h) newly collected data from YouTube and web crawls (Table 16, 19 in new version) not part of any existing academic literature. (2) how to split a specific compute budget between model size and training samples for contrastive audio-text models. (3) Training fully from scratch on AudioWeb-365M, our CLAP model based loosely on NaFlex and ModernBERT performs amongst the highest compared to all current embedding models, where our 5 models score on par with embedding models 8 to 32 times their size on MAEB-30.

Final corpus counts conflict: 364,955,979 versus 366,104,195 clips. Applied semantic removals are also far fewer than proposed removals. This obscures the identity of the training resource.

Thank you for noticing. The two counts come from different versions of the dataset catalogue, the correct dataset contains roughly 366M samples. We will fix this count in the revised version. We called the dataset AudioWeb-365M to make an analogy to the amount of days in a year. On the semantic removals, the SemDeDup algorithm initially had 114M candidates, and ultimately removed 18.2M individual samples matched.

Appendix H reports that about 12% of sources lack verified license declarations. Audio rights need clarification, as do the release scope and safeguards for inferred age, gender, and emotion.

We are happy to clarify both points. This 12% counts each source equally. By samples, the sources that do not contain a verified license hold only 1.6% of AudioWeb-365M, and 84% of those samples are from web crawls. Current laws allow using them only for research purposes. This part of the data is also shared online and contains links, captions, metadata and no audio.

Regarding the safeguards for inferred age, gender and emotion, they are outputs of open-source classifiers and may be incorrect. They were used only as context for the captioning of the audio, and are not used to identify or profile speakers.

Good data work. I don't have any other question.

We thank the reviewer for their interesting and useful questions. We would love to hear about any other concerns raised.

I will maintain my score, looking forward to the open-sourcing.

Comments on the paper

1 comment

We thank all the reviewers for their valuable feedback, allowing us to further strengthen the experimental setup in addition to the depth and clarity of our manuscript. We are happy that the reviewers think highly of the open dataset and the well-documented curation pipeline we provide. We also appreciate the positive feedback on our scaling study and its ability to guide future compute allocations.

To further strengthen our results and scaling study, we additionally trained our largest model (L) on 1,024 million samples, standardizing the data range of all our scaling runs to 1M–1,024M. This led to our largest model (340 million parameters) improving from 52.2% to 53.6% on MAEB-30, matching LCO-Embedding-Omni-7B (53.5%), the next best model in our comparison, with 8.9 billion parameters. We also re-evaluated some of the baselines to allow for a fairer comparison.

The reviewers' feedback allowed us to add the following major improvements to our manuscript:

  • the corrected corpus count and CaptionStew in Table 1 (Section 3.3, Table 1)
  • a more detailed caption validation (Section 3.4, Appendix C.1, Table 4)
  • a redesign of Figure 3 and a compute-optimal allocation subsection (Section 5, Figures 3 and 5)
  • the full captioning prompt with example captions (Appendix C, Figures 9 and 10)
  • additional details on the dataset screening before release (Ethics statement)
  • downstream experiments with MusicLDM and Audio Flamingo 2 (Section 6, Appendix G)

We uploaded a revised version of our manuscript with all edited sections marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.