acceptodds
Under review as a conference paper at ICLR 2027

Woosh-Mimic: Conditioning the Woosh Sound Effect Foundation Model with Vocal Mimicry

Abstract

Text-to-audio generation enables semantic control over generated sounds through natural language. Adding vocal mimicry could provide an intuitive, nonverbal way to convey fine-grained temporal variations in timing, intensity, pitch, rhythm, and spectral characteristics, without describing them in words or manually specifying numerical control trajectories. However, vocal mimics and the sound effects (SFXs) they represent belong to substantially different acoustic domains, and learning the correspondence between them remains underexplored. We propose Woosh-Mimic, a vocal-mimic-conditioned SFX generation method built on the pretrained Woosh framework. Woosh-Mimic combines semantic conditioning from production SFX metadata with temporal acoustic conditioning from vocal mimics and learns from paired vocal-mimic and SFX recordings. To examine the effect of the domain mismatch, we compare a Mimic Model trained with acoustic conditions extracted from vocal mimics against a Reference Model trained with conditions extracted from the corresponding reference SFXs. The Mimic Model achieves lower Fréchet Audio Distance across all seven evaluation splits, whereas the Reference Model more strongly reflects the characteristics of the input vocal mimics in both objective and subjective evaluations. This contrast suggests that matching the conditioning domains between training and inference improves generation quality but does not necessarily improve the reflection of individual vocal mimics. Our findings highlight the potential of vocal mimicry as an expressive complement to language-based semantic control while identifying faithful control across acoustic domains as a central challenge.

Primary area:
generative models
Keywords:
Sound Effect Synthesis, Vocal Mimic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WocNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Woosh-Mimic combines sound-effect metadata and vocal-mimic conditioning within the pretrained Woosh framework, training additional modules on paired vocal mimics and sound effects. Through a comparison of two training-condition sources, objective evaluations, and a listening study, the paper finds that training with vocal mimics improves FAD, while training with reference-SFX features yields better vocal-mimic fidelity.

Critical strengths
  1. The paper addresses a practical problem. Vocal mimicry offers an intuitive way to express rhythm, intensity, and timbral changes that can be difficult to specify precisely through text.

  2. The comparison between the two training approaches is informative, and the objective evaluation and listening study provide complementary evidence. The paper openly reports their different effects on distributional similarity and mimic fidelity, offering a useful observation about learning across the vocal-mimic and sound-effect domains.

Critical weaknesses
  1. The evaluation of generation quality relies primarily on FAD. This metric measures similarity between generated and reference audio distributions in a feature space, but is insufficient on its own to establish an improvement in perceptual audio quality. I suggest adding text–audio alignment measures such as CLAP similarity, perceptual or aesthetic metrics appropriate for sound effects, and a separate listening evaluation of audio quality. The current listening study only assesses how well the outputs reflect the vocal mimics, so it does not establish whether lower FAD corresponds to better perceived quality.
  2. The reported Corr and DTW metrics compare acoustic trajectories from the input vocal mimic and the generated sound effect. This is appropriate for measuring how closely the output follows the mimic, but the relationship to the paired reference SFX remains unclear. I suggest also reporting these metrics between the generated and reference SFXs, together with the corresponding results between the input mimic and reference SFX as context. This additional analysis could help distinguish differences in control fidelity from discrepancies already present between the human imitation and its intended sound effect.
Questions for authors
  1. The experiments primarily compare two training approaches within the same Woosh framework, leaving the comparison with existing methods limited. I suggest including at least one related mimic- or acoustic-trajectory-conditioned method under comparable input conditions.
  2. The lack of qualitative audio examples makes it difficult to understand the practical value of the task, particularly what vocal-mimic queries offer beyond text descriptions.
Reviewer_tVJgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Woosh-Mimic, a vocal-mimic-conditioned sound effect (SFX) generation framework built upon the pretrained Woosh model. The method combines semantic conditioning from SFX metadata with temporal acoustic conditioning extracted from vocal imitations. Specifically, the authors introduce a Mimic Encoder that transforms 15-dimensional acoustic feature sequences into conditioning tokens, which are integrated into the frozen Woosh-Flow model through additional modality blocks.

The paper investigates the effect of the training–inference conditioning-domain gap by comparing two variants: a Mimic Model trained with features extracted from vocal imitations and a Reference Model trained with features extracted from the target SFXs. Experiments show that the Mimic Model consistently achieves better FAD scores, while the Reference Model demonstrates stronger adherence to vocal-mimic characteristics in both objective and subjective evaluations.

Critical strengths
  1. Meaningful problem setting. Vocal mimicry provides an intuitive way to specify temporal and acoustic characteristics that are difficult to communicate through text alone. Combining semantic and acoustic control is a practically relevant direction for sound effect generation.
  2. Reasonable technical design. The proposed framework effectively extends a pretrained SFX foundation model with an additional conditioning modality while keeping the original generation backbone frozen.
  3. Interesting empirical findings. The contrasting results between generation quality and vocal-mimic reflection provide useful insights into the challenges of learning cross-domain acoustic conditioning. The paper also includes both objective evaluations and a reasonably designed subjective listening study.
Critical weaknesses
  1. Insufficient comparisons with existing vocal-mimic-conditioned generation methods.

The experimental evaluation only compares the proposed Mimic Model against a Reference Model with the same architecture but a different source of training conditions. While this comparison helps investigate the conditioning-domain gap, it does not establish the effectiveness of Woosh-Mimic relative to existing approaches. Several closely related methods, including Sketch2Sound, FlashFoley, T-Foley, and PronounSE, are discussed in the related work but are not included in the experimental comparisons. Comparisons with these methods, where evaluation settings can be reasonably aligned, would substantially strengthen the paper.

  1. Limited evidence for the effectiveness of the proposed vocal-mimic conditioning.

A central motivation of the paper is to learn correspondences between vocal imitations and target SFXs through paired training. However, the proposed Mimic Model performs consistently worse than the Reference Model in vocal-mimic reflection, including both objective acoustic similarity metrics and subjective listening evaluations. Although the Mimic Model achieves better FAD scores, FAD primarily measures distribution-level generation quality and does not demonstrate whether the generated audio faithfully follows the input vocal mimic. Consequently, the current results do not clearly establish the benefits of the proposed paired mimic-to-SFX training strategy for controllable generation. It would be valuable to investigate whether the model genuinely responds to variations in vocal-mimic inputs rather than primarily relying on semantic metadata.

  1. Missing ablations on semantic and acoustic conditioning.

The proposed framework combines two conditioning modalities, but their individual contributions and interactions are not adequately evaluated. In particular, the paper lacks comparisons with a metadata-only Woosh baseline and an acoustic-only variant. Such experiments are important for determining whether vocal-mimic conditioning provides meaningful improvements over the pretrained generation model. Additionally, controlled experiments that vary the vocal mimic while keeping the semantic condition fixed (and vice versa) would provide more convincing evidence of the model's controllability and the complementary roles of the two conditioning modalities.

Questions for authors

Please see weaknesses.

Reviewer_ZDHmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed an adaptation of a pre-trained text-to-SFX model, Whoosh, to condition the Timing and other characteristics of the output based on vocal imitations, resulting in the whoosh-mimic model. The main proposal is to train on paired data of target SFX and actual vocal imitations of these SFX as the input, as opposed to previous work, which extracted the condition signal from the target SFX directly without using vocal imitations during training. They compare the proposed approach to a reference version, which extracts the conditioning signal from the target SFX directly without using vocal imitations during training. Quantitative evaluation shows Ushmimic obtains lower FAD, but the reference baseline shows better adherence to the input condition. A user preference study shows that the majority of users prefer the reference baseline.

Critical strengths

The paper examines an interesting question: Is it better to train on paired data of SFX and their corresponding vocal imitations, as opposed to what prior work has done (where the conditioning signal is extracted from the target SFX during training without using vocal imitations during training).

Critical weaknesses

Usefulness of the method. The results show that the reference actually adheres better to the condition and is also preferred overall by listeners. As such, the benefit of the proposed approach is not clear. Obtaining lower FAD is a good directional signal, but it is only a proxy for quality. Ultimately, I would have expected to see a user preference for the quality of the proposed method in the listening study, which doesn't materialize. Coupled with the fact that the proposed method requires paired training data, which is harder to obtain compared to prior work that didn't require paired training data, the added value of the proposed method is unclear to me. While I commend the authors for submitting a paper with mostly negative results, I do not believe it provides sufficient, well substantiated, insight.

Weak evaluation. There is only a comparison between the proposed method, Whoosh-Mimic, and a baseline reference, which is using the same input features and architecture proposed by the authors, just extracting the condition features from the target SFX instead of the paired vocal imitations. I would have expected to see a comparison to other baselines from the literature, and in particular, the Sketch2Sound paper from which this paper draws inspiration. Without a comparison to prior baselines, it is unclear how either of the methods evaluated in the paper compare to prior work and whether they are any better. In particular, a key difference compared to prior work, as I understand it, is that in the methods evaluated in the paper, the conditioning features are pooled into a fixed dimensionality representation. Whereas in prior work, the condition signal is temporally aligned with the output, which is meant to make it easier for the model to adhere temporally to the input condition. The decision to move away from this design choice is not clearly motivated in the paper. I would have expected to at least see an ablation in the evaluation section to motivate this design change proposed by the authors in this paper. No such evaluation is conducted, and thus this change remains unmotivated and unsupported.

Questions for authors
  • What motivates pooling the condition signals over time into a fixed dimensionality, as opposed to following prior literature and using frame-aligned conditioning signals?
  • Why is there no comparison to baselines from prior literature?
  • Did you collect what criteria subjects in the listening study used to determine their preference? In particular, there are two independent criteria they could be assessing:
  1. The overall audio quality of the output
  2. The adherence to the input condition (i.e., the vocal imitation)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.