acceptodds
Withdrawn from ICLR 2027

AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Audio generation models can translate natural-language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression, and their outputs do not readily support direct edits to notes, timbral parameters, or modulation relationships. We present AutoSynth, which represents MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence for a synthesizer, and learns their dependencies with an audio-conditioned autoregressive model. A single model supports both tasks. Given reference audio, the model directly predicts a synthesizer program; given text, it uses a pretrained audio generation model and converts the generated audio into a program. Training consists of two stages: supervised learning on large-scale audio–program pairs automatically constructed from a small set of native presets, followed by group-relative policy optimization with a mixed reward combining semantic similarity, pitch-related features, acoustic similarity, and sound usefulness. The pipeline requires neither paired text–target-program annotations nor a differentiable synthesizer. Experiments show that AutoSynth produces complete, editable synthesizer programs and achieves competitive results in both synthesizer inversion and text-driven generation. Audio demos and source code are available at https://auto-synth.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
synthesizer inversion, synthesizer program generation, text-to-audio generation, editable audio generation, autoregressive modeling, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wHVcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes AutoSynth, an audio-conditioned autoregressive model that jointly models MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence. Training proceeds in two stages: supervised fine-tuning (SFT) on large-scale audio–program pairs, followed by GRPO post-training. The authors also show that by cascading an off-the-shelf text-to-audio model (Stable Audio 3), the system can generate synthesizer programs whose sounds match a text description. They evaluate on a cross-generator Gemini–Woosh set and on FSD50K against Synth Permutations, DDSynth-RL, and CTAG, and run a 10-person expert listening study. They report better semantic alignment and content quality than the synthesizer-program baselines.

Critical strengths
  1. The proposed method is well motivated, and jointly modeling MIDI, synthesizer parameters, and modulation routes with an autoregressive Transformer is novel.
  2. The code is released and the appendices are detailed, which supports reproducibility.
  3. The authors construct a cross-generator Gemini–Woosh test set to avoid leakage from the generators used to build the training data.
Critical weaknesses
  1. Although the method is novel, the evaluation is not fully convincing. The baselines use different synthesizers (Surge XT, Dexed, and SynthAX), so performance differences may reflect the synthesizer rather than the method. In addition, most of the evaluation metrics (CLAP, wMFCC, SOT, MSS, RMS, and AudioBox CE and CU) are also used as rewards during GRPO. This makes the comparison unfair, since these metrics are susceptible to reward hacking and may yield higher scores without real improvement.
  2. There is no ablation study on the joint modeling, and the model’s editability is not evaluated. For example, MIDI accuracy (e.g., note-level F1) should be reported, as well as whether editing the MIDI preserves the intended timbre.
  3. Since the subjective evaluation is important, its rating criteria should be described in the main text rather than only in the appendix. Figures 3 and 4 report “Alignment” and “Content quality” scores without explaining the rating scale or criteria.
Questions for authors
  1. Can you provide a baseline (e.g., Synth Permutations or DDSynth-RL) retrained on Helm’s fixed parameters, so that all methods share the same synthesizer?
  2. How was Synth Permutations retrained (training data and number of steps)? Does it reproduce the in-domain results reported in the original paper?
  3. Can you report MIDI accuracy (e.g., note-level F1) and program statistics, such as the number of active routes and the fraction of parameters set to extreme values, before and after GRPO?
Reviewer_WJbLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Instead of generating waveforms (typical of generative text-to-audio models), authors build encoder-decoder architectures for generating synthesizer programs (MIDI performance events, fixed params, modulation routes as a single sequence) which can be executed via Helm synthesizer. This allows for higher production quality audio generation (which is otherwise limited by audio codec quality) and directly editable generation (helpful for musicians and audio engineers). Authors train models using SFT and GRPO with synthetic data, obtaining strong audio quality scores.

Critical strengths

(1) Unique and pragmatic idea: Authors conceive of a clear application-based idea, generating synthesizer program components (MIDI performance events and 133 parameters and variable-length modulation routes within the same sequence) instead of direct audio. The benefit is dual--artifact-free rendering and enabling easy human interaction.

(2) Clever technique for RL: Authors use GRPO to align model with label-free audios (e.g., Stable Audio 3 generations). Using RL as a substitute for supervised learning in cases of unobtainable labels is an effective solution.

(3) Strong generation metrics: Authors demonstrate strong audio metrics for synthesizer inversion (e.g., CLAP) and also get strong production quality for text-to-sound generation (e.g., AudioBox metrics). Further, they find improvements for metrics which were not included as part of the reward signal (e.g., LanguageBind and AudioBox PQ).

Critical weaknesses

(1) Framing of paper as reducing audio compression constraints in text-to-audio generation: Authors claim that the paper is intended to tackle the audio quality reductions caused by use of audio codecs. However, the provided text-to-audio pipeline uses Stable Audio 3 for audio generation (which is later converted to synthesizer programs) which utilizes audio codecs. I therefore believe that their framing is overstated (for the text-to-audio generation case). The authors perform RL using reward signals from Stable Audio 3 generations (using distance metrics like CLAP), hence likely limiting their own synthesizer generations by audio codecs. Adding to concerns, while their method improves on PQ scores, CU, CE, KL, and FAD decline, limiting the expected improvements from removal of audio codecs.

(2) Limited methodological novelty: Authors use standard supervised learning and RL methods. The novel part is the program representation itself, i.e. jointly modelling MIDI, synthesizer parameters, and modulation routes, which is better described as a representational contribution than a new technique.

(3) Low text CLAP scores: Authors obtain lower alignment with text in Table 2. Importantly, the scores decline from Stable Audio 3’s baseline (especially on FSD50K), indicating that the additional treatment of audio to convert to synthesizer programs destroys the semantic alignment to text.

(4) Six of the eight reward signals are included as evaluated metrics: CLAP, wMFCC, SOT, MSS, RMS, and AudioBox CE and CU were used in GRPO as reward signal and included as evaluation metrics. Authors do hold out LanguageBind and PQ and both improve, demonstrating some generalization. However, these are only two metrics and I would recommend inclusion of further evaluation benchmarks to support claims.

Questions for authors

(1) How are GRPO metric weights identified? Were ablations performed to select the correct weighting? (2) Can you ablate the output space within Helm (params only, +MIDI, +routes) to measure the contribution of the joint representation? (3) Why is text not provided to the program decoder as an additional conditioning stream, given it is available at inference? (4) Were significance tests performed on the listening study? Several CIs in Figures 3 and 4 appear to overlap.

Reviewer_aUAyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoSynth uses an audio-conditioned autoregressive model to generate executable Helm programs, including MIDI events, synthesizer parameters, and modulation routes. The model is first trained on synthesized audio–program pairs using supervised learning, followed by group-relative policy optimization with rewards based on rendered audio. For text-to-sound generation, the system first generates reference audio using Stable Audio 3 and then applies the same model to recover the corresponding synthesizer program.

Critical strengths
  1. The output representation is useful and flexible. Generating MIDI events, synthesizer parameters, and modulation routes together provides more flexibility for sound editing than generating waveforms or fixed parameter vectors. I also think modeling variable-length modulation structures is an interesting direction.
  2. The evaluation is relatively comprehensive. Besides the metrics used for RL rewards, the paper also evaluates LanguageBind and AudioBox PQ and includes a listening study with sound-creation professionals. Although these evaluations have some limitations, they provide additional evidence beyond the optimized CLAP score.
  3. The human evaluation supports the main results. For Gemini–Woosh inversion, AutoSynth achieves higher alignment (3.54 vs. 2.62) and content quality (3.86 vs. 2.84) than DDSynth-RL. Although the listening study is relatively small, these results provide useful evidence for the effectiveness of the method.
Critical weaknesses
  • My main concern is that AutoSynth and the baselines use different synthesizers (Helm, Surge XT, Dexed, and SynthAX Voice) and different training data and configurations. This makes it difficult to tell whether the improvements in Tables 1–2 and the listening study come from the proposed method or simply from differences in the sound engines and training data. I think a baseline trained on the same Helm data, ideally with a comparable model architecture and training setup, would make the comparison more convincing. The SFT-only model already shows the benefit of RL training, but it does not validate the proposed representation and architecture.

  • The paper does not directly show how much joint modeling of MIDI, synthesizer parameters, and modulation routes contributes to the final performance. Appendix D only compares token ordering using training loss, without evaluating the actual generated audio. I think ablations that remove variable-length modulation routes or predict different components separately would help demonstrate the benefits of the proposed representation. It would also be useful to compare different token orderings based on rendered audio rather than just prediction loss.

  • Five of the six inversion metrics are also used as RL rewards, so improvements on these metrics are expected. While LanguageBind and human evaluation provide additional evidence, Table C2 shows that GRPO improves some quality metrics (CE/CU/PQ) while making fidelity metrics such as CLAP distance, LanguageBind distance, and KL divergence worse. Table B1 also suggests a trade-off between audio quality and fidelity. I think an ablation without the CE/CU rewards would help clarify this trade-off. It would also be helpful to explain how the reward weights were selected.

  • The paper highlights editability as a major advantage, but the evidence mainly comes from an LLM-based demonstration rather than a systematic evaluation. I think a small editing benchmark measuring whether edits are successful while preserving unrelated sound properties would make this claim more convincing.

Questions for authors
  1. How does AutoSynth compare with a structural or modular baseline using the same Helm engine and training data?
  2. How much do variable-length modulation routes and joint modeling contribute to the final audio quality? Do different token orderings affect the generated audio?
  3. Can the authors provide a quantitative evaluation of editing performance? How are the training and test presets separated?
  4. How many unique clips were used in the listening study? Can the authors provide confidence intervals, training costs, and results across different random seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.