acceptodds
Under review as a conference paper at ICLR 2027

ProCraft: Instruction-Driven Music Production from Free-Form Text

Abstract

Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoing and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
music editing, music generation, audio effects, audio codec language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8LsWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProCraft, a 0.5B autoregressive codec language model designed for seven distinct music-editing tasks, ranging from stem extraction to audio effects. To tackle long codec sequences, ProCraft incorporates two key designs: a persistent edit head that injects 32 compressed instruction-audio queries across all decoder layers via gated cross-attention, and an intermediate representation alignment (at layer 4) guided by MERT, pitch/instrument tags, and StemFX embeddings. To address the scarcity of compound editing data, the authors synthesize training triplets on the fly from Slakh2100 using Qwen3-30B tool calls paired with MultiAFX.

Critical strengths
  1. The persistent edit head directly targets the causal context dilution issue over long sequence generation (2,250 tokens). The two-pass training scheme is a thoughtful design that ensures training matches inference-time condition caching.

  2. Transferring REPA-style alignment (layer 4 aligned with MERT, pitch/instrument labels, and effect embeddings) to codec LMs is sound and empirically proves beneficial for out-of-domain stem extraction and removal.

  3. The paper is honest about its bottlenecks—specifically isolating that 7.3 dB of the SI-SDR gap is attributable to the codec reconstruction itself, and designing evaluation metrics that actively penalize trivial identity shortcuts.

Critical weaknesses
  1. While the model claims to unify seven operations, the full breakdown in Appendix B (Tables 3 & 4) shows that ProCraft fails to consistently win across specific tasks. Dedicated baselines outperform it significantly: separators dominate SI-SDR, MusicGen-Stem wins on accompaniment instrument matching, piano-cover models better track source melody, and InstructFX2FX is closer on audio effects. The only significant gains are against weak, general-purpose free-text editors. In production, a modular pipeline chaining specialized tools (e.g., separator + FX engine) would likely offer far better performance, undermining the practical motivation for this unified generative model.

  2. The training and evaluation pipelines rely entirely on synthetic mixes (Slakh2100) and instructions generated by a single LLM (Qwen3-30B) via templates. The paper does not evaluate robustness against authentic human prompts (which are often ambiguous, musically naive, or noisy), leaving its generalization in real user-facing scenarios unverified.

Questions for authors
  1. The edit rate on real-world recordings (MoisesDB removal) remains still low at 38.0%. Could you provide a qualitative or quantitative breakdown of the remaining 62% failure cases? Specifically:
  1. Are these failures caused by instruction/stem misidentification, partial/insufficient suppression, or artifact corruption?
  2. Is this primarily a limitation of the acoustic modeling capacity or the low-bitrate codec representation?
  1. Both the training and evaluation instructions are synthesized by Qwen3-30B using predefined motivation templates. Have you evaluated the model on authentic, user-generated instructions that may contain informal slang, ambiguous terminology, or musically naive descriptions? How robust is the persistent edit head when prompt wording deviates from the LLM template distribution?
7 comments
AuthorsOpenReview ↗

Response for Reviewer 8LsW (1/2)

We clarify that ProCraft is not meant to replace dedicated systems on each operation, but rather demonstrate that a single model can infer what to edit jointly from both the instruction and the audio. It reads the instruction and the source audio in one sequence, so every output code can attend to both, and the benefit of this joint conditioning shows in the comparison with Instruct-MusicGen retrained on our data, which conditions audio and text on the two separate paths and trailed ProCraft on nearly every metric of every operation. Section 5.2 reports:

It rarely isolated the requested stem on extraction, with an edit rate of 0.16 on Slakh and 0.19 on MoisesDB against 0.79 and 0.82 for ProCraft, or rendered the requested instrument on instrument transfer, with an instrument match of 0.31 on both sets against 0.64 and 0.61. [...] Whereas Instruct-MusicGen conditions on the instruction and source audio through separate paths, ProCraft reads both in one sequence, so every output code can attend to them together.

Such joint reasoning matters in music production, where the same request can call for different edits depending on the song: "tighten the mix" might mean compressing the drums in one song and cutting the low end of the guitars in another. A modular pipeline would need a separate component to make this decision from the instruction and the audio before calling its tools, which is the problem ProCraft addresses within one model. Although ProCraft still trailed deterministic task-specific systems such as source separators, it was competitive with generative task-specific systems, with a 0.5B-parameter model trained only on Slakh2100, and we believe this joint approach offers a simple and general basis for instruction-driven music production.

We thank you for the suggestion of performing an error analysis. We note that ProCraft removed the requested stem in 38.0% of MoisesDB removal items, and most of the other 62% left the requested stem in place or made it only partly quieter. To label these failures, we separated the source, the target, and the output with HTDemucs-6s, as for edit rate, and measured how much the level of each stem changed from the source to the output. In 33 items (35.5%), the requested stem changed by less than 2 dB and the output stayed close to the source after encoding and decoding (copy-through). In 26 items (28.0%), the requested stem became 2 to 6 dB quieter, which is not enough to pass the threshold (partial suppression). In 15 items (16.1%), another stem was removed instead (wrong stem); in 9 of these, the instruction used the phrasing "pull the X out", and ProCraft isolated the stem instead of removing it. The remaining 19 items (20.4%) were nearly silent or cut short, damaged stems that should have stayed, or could not be judged reliably by the test. By requested stem, guitar and bass account for 74 of the 93 failures (each about 75%, against 38% for drums and 36% for piano). Neither the codec nor loudness explains this: the codec-decoded target passes the test for every guitar item and 85% of bass items, and the requested stem is as loud in failed as in successful items. Bass removal fails in 29% of Slakh items but 75% of MoisesDB items, which points to the gap between rendered and recorded instrument sounds, while guitar is hard in both domains (78% on Slakh). Section 5.1 summarizes this analysis, and Appendix J (Table 13) gives the full breakdown:

We found that most of ProCraft's failures left the requested stem in place or reduced it only partly. These failures concentrated on guitar and bass stems, and bass removal failed far more often than on Slakh, which points to the gap between rendered and recorded instrument sounds.

AuthorsOpenReview ↗

Response for Reviewer 8LsW (2/2)

We are also grateful for the suggestion to test ProCraft on authentic human-written instructions. To that end, we evaluated it on the SonicMaster test set, whose prompts were written by a music expert for amateur musicians in plain wording, and none of which comes from our language model (new Section 5.5, Table 4). We selected 200 clips from its test split, 50 in each of four groups (EQ, dynamics, reverb, and clipping), with 75 distinct prompts in total. Each prompt asks to correct a degradation, such as fixing the EQ, removing reverb, or declipping, a task that is absent from our training data, so this set is out-of-domain in both wording and task. We also measured how far its prompts are from ours using Fréchet BERT Distance (Section 5.5), noting that:

These prompts differ from ours: their Fréchet BERT Distance to our audio-effect test prompts is 3.74, against 0.87 between two sets of our own audio-effect prompts of the same size (75), and 70% of the words in the SonicMaster prompts don't exist in ours.

We compared ProCraft with Instruct-MusicGen retrained on our data and the audio-effect baselines of the paper. FAD uses the MUSDB18-HQ full-mix reference set, and AFx-Rep compares each output with the clean target (see Table 4 and General Comments).

We found that ProCraft had lower FAD than IMG (retrained), FxSearcher, and Text2FX in every group, and than InstructFX2FX on EQ and reverb, so it handled the unseen prompts without degrading the audio. On AFx-Rep, InstructFX2FX was closest to the target in every group, and ProCraft ranked second. These show the robustness of our model despite being trained only on synthetic instructions. Section 5.5 puts this in context:

InstructFX2FX, FxSearcher, and Text2FX only apply audio effects, which is all this test set asks for, whereas ProCraft is a general music editor that also removes, extracts, and generates stems from the same model. Overall, ProCraft generalized to human-written prompts and remained competitive with these dedicated audio-effect baselines, while InstructFX2FX was best on AFx-Rep.

Thanks — the error analysis answers my Question 2, and it's a good one: since the codec-decoded targets pass the removal test, the codec is not the problem. The model just isn't carrying out the edit on real recordings. But that doesn't fix Weakness 1. Most failures are the model doing nothing (35.5% copy-through) or half the job (28% partial suppression) — exactly the failure modes a simple tool chain doesn't have. On the same MoisesDB removal items, AudioSep and Banquet get edit rates of 0.61 and 0.71 against 0.38. And the chained baseline is still not run: your benchmark requests are literal ("cut the bass and add reverb"), and your own data pipeline already does the modular thing (LLM tool calls + MultiAFX), so the "tighten the mix" argument doesn't really apply to the comparison I asked for. Also, the Instruct-MusicGen gap still mixes up the conditioning design with the backbone (Section 5.2 admits this), and the claimed benefit of joint reasoning is never actually tested on ambiguous instructions. So Question 2 is answered; Questions 1, 3, and 4 look open. Keeping my score.

Reviewer_QogcOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ProCraft, a 0.5B autoregressive codec language model that edits 30s music excerpts from free-form text instructions. It supports seven operations across eight instrument families. The model is initialized from InspireMusic and adds two components: a persistent edit head that summarizes the instruction and source audio through learnable queries and is injected into every decoder layer via gated cross-attention, and an intermediate-layer alignment loss that aligns an early layer with MERT, pitch, instrument, and StemFX targets. Training data is produced by an on-the-fly synthesis pipeline: a tool-calling LLM writes instructions and specifies audio effects as executable function calls, which are rendered on Slakh2100 stems.

Critical strengths
  1. The task formulation is novel and practically relevant. Real music production workflows request several changes at once, and no existing system covers free-form phrasing, stem-level operations, and audio effects in one model. The paper also evaluates editing and preservation as two separate axes, which is the right way to measure this task.

  2. The on-the-fly editing-pair synthesis is a clean solution to the lack of compound editing triplets: because effects are executed as function calls rather than only described in text, the target matches the instruction by construction. This pipeline is itself a useful contribution to the community.

I also listened to the audio samples on the demo page, and the editing quality is impressive, especially given that the model has only 0.5B parameters. The paper is well written and a pleasure to read, from the motivation and method design to the engineering details.

Critical weaknesses
  1. The ablation does not separate the two claimed contributions. Section 5.2 only compares the full model with ProCraft-plain, which removes the gated cross-attention and the alignment losses together. If I understand the design correctly, the two mechanisms address different failure modes, so their effects should be measurable separately.

  2. The training data is somewhat limited in scale: the model is trained only on Slakh2100 and a few open-source datasets. That said, this is understandable given that this is one of the earliest works on free-form music editing.

Questions for authors
  1. It would be helpful to report the training compute in the paper, such as the GPU type and GPU count. Will you release the synthesized training dataset? Both are important for reproducibility.

  2. Could you provide two separate ablations, i.e., removing only the gated cross-attention and removing only the alignment losses, with both editing and preservation metrics? If the individual contributions are clarified, I am willing to raise my score.

2 comments
AuthorsOpenReview ↗

Response for Reviewer Qogc

We thank you for the suggestion of performing a more detailed ablation study to clarify the separate contributions of alignment and cross-attention. We trained ProCraft without the gated cross-attention, without the alignment losses, and without both, all for 20K steps on the same data. Table 2 now reports the editing metrics of this 2x2 ablation, and Appendix D (Tables 9–10) gives all Editing and Preservation metrics on both datasets, in addition to results from ablating each alignment target separately.

On real recordings, removing only the cross-attention lowered the removal edit rate from 0.38 to 0.26 and the extraction edit rate from 0.82 to 0.70, and raised the extraction mel-L1 to the target from 1.49 to 1.77. Removing only the alignment losses lowered the removal edit rate to 0.27 but did not lower extraction (0.85), and removing both lowered removal further to 0.24. As the reviewer anticipated, the two mechanisms therefore help in different places (updated Section 5.2). Alignment grounds an early layer in the content of the source, so it matters most where that content must be kept: instrument transfer, which must keep every note of a track while changing its instrument, suffered most without it. Cross-attention keeps the instruction within reach of every output code, so it matters most where the output depends on following the request: extraction suffered most without it because it must find the exact stem that the instruction names. Removal needs both, as the model must find the named stem and keep everything else intact. Section 5.2 now opens this analysis with:

The two mechanisms serve different needs of an edit. Alignment grounds an early layer in the content of the source, such as which instruments play which notes, so it should matter most where the output must keep that content. Cross-attention keeps the instruction within reach of every output code, so it should matter most where the output depends on following the request.

ProCraft was trained on one NVIDIA RTX 5090 (32 GB) in about 40 hours, with an effective batch of 16 sequences for 20K steps and a second RTX 5090 serving the instruction language model for on-the-fly synthesis. The revised paper reports these settings in Appendix E:

ProCraft trains on one NVIDIA RTX 5090 (32 GB) in about 40 hours, with a second RTX 5090 serving the instruction language model for on-the-fly synthesis.

Upon acceptance, we will release the model weights, the training data, and the code for the model, training, and data generation (footnote in Section 1).

We agree that the training data is limited in scale, and the revised Conclusion now states this limitation:

Our training data is also limited to Slakh2100; as the synthesis pipeline only needs multitrack stems, larger multitrack corpora such as SPDMX are a natural next step.

As the synthesis pipeline only needs multitrack stems, it extends directly to larger multitrack corpora, and the MoisesDB results already show that the model transfers to real recordings without training on them.

Thank you for your detailed supplementary experiments and response. Considering that the promised release of models and data will improve reproducibility, and my concerns about ablation experiments have been addressed, I will increase my score.

Reviewer_55brOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProCraft, an autoregressive model for compound music editing that edits 30-second music excerpts end-to-end from free-form instructions. To enable this, the authors fine-tune the pre-trained InspireMusic-0.5B model with additional intermediate-layer alignment losses and gated cross-attention, which they present as the core contributions. The authors also propose an on-the-fly editing-pair synthesis pipeline that generates compound (source, instruction, target) triplets. Experiments against a range of baselines, together with a listening study, show that the proposed approach performs competitively with or better than existing methods across diverse editing operations.

Critical strengths
  • Well-designed pipeline. The overall pipeline is reasonably designed for comprehensive music editing. In particular, the on-the-fly editing-pair synthesis is critical for the diversity of the training set and, by extension, for scalability.
  • Thorough evaluation. The paper evaluates the proposed model against a wide range of baselines across a diverse set of metrics. The listening study is also well designed and well executed, and it complements the objective evaluation.
Critical weaknesses
  1. Technical novelty is limited, and the ablation cannot attribute gains to the proposed mechanisms.
  • The two mechanisms presented as technical contributions combine existing components: BLIP-2-style learnable queries, Flamingo-style tanh-gated cross-attention, and REPA.
  • Combining existing methods can still be a valid contribution, but the paper should then show which component actually helps. The only ablation, ProCraft-plain, removes gated cross-attention and all four alignment losses at once. It is also not stated whether the learnable queries remain in its input sequence.
  • Consequently, the claim in Section 1 that “both reduce leakage” is not supported. None of the following is tested: queries without cross-attention, cross-attention without the gate (plain cross-attention), the number of queries K, or each alignment target in isolation. This point is central to my assessment. Without these ablations, the technical contribution reduces to a system and dataset contribution.
  1. The reported benefit of the mechanisms is inconsistent across metrics and settings.
  • The advantage of ProCraft over ProCraft-plain is concentrated in editing-specific metrics.
  • For removal, edit rate and edit strength improve. However, the intrusive metrics in Table 3 (SI-SDR, mel-L1, and MRSTFT), which are important for a source-separation-like task, show no significant difference.
  • For audio effects, ProCraft-plain achieves higher AFx-Rep similarity on Slakh (Table 3), even though the StemFX alignment target is intended specifically for effects.
Questions for authors

Questions

  1. Learnable queries: Does ProCraft-plain keep the 32 learnable queries in its input sequence? Is there evidence that summarizing the edit into queries is necessary? How was K = 32 chosen?
  2. Gate values: What are the learned per-layer gate values tanh(α) after training? Did you compare against ungated cross-attention?
  3. FAD/KAD reference set:
  • The FAD/KAD reference set (L265–267) appears to be fixed across all tasks and to consist of full mixes. For extraction, accompaniment, and cover, where the expected output is a single instrument, this would favor outputs that fail to isolate the target part.
  • For instance, AudioSep obtains the best FAD on MoisesDB extraction, yet according to edit rate and edit strength it rarely isolates the requested stem. When I listened to the AudioSep results on the provided sample page, its output was nearly the unseparated mixture itself.
  • Is this indeed the case? Does the MoisesDB portion of the reference pool overlap with the MoisesDB test sources?
  1. Generation length. How often does generation fail to produce an end-of-sequence token, or stop early? When computing SI-SDR, mel-L1, and MRSTFT, how are outputs of different lengths aligned with the target?

Suggestions

  1. Ablate each alignment target individually (MERT, pitch, instrument, StemFX). StemFX deserves particular attention given the audio-effects results.
  2. Report per-item coherence metrics for accompaniment, such as COCOLA [1] or PHALAR [2], to complement the existing evaluation.

Minor

  • L138: “each part of the sequence f” should presumably read “s”.
  • L131: “.Instruction” should read “. Instruction”.

References [1] R. Ciranni et al., “COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations,” ICASSP, 2025. [2] D. Marincione et al., “PHALAR: Phasors for Learned Musical Audio Representations,” ICML, 2026.

2 comments
AuthorsOpenReview ↗

Response for Reviewer 55br (1/2)

We trained ProCraft without alignment loss and gated cross-attention separately and report the resulting 2x2 ablation in the new Table 2, with the removal of each alignment target among the four in Appendix D. On real recordings (MoisesDB), the removal edit rate was 0.38 with both mechanisms, 0.26–0.27 with either one removed, and 0.24 with neither. As this supports an improvement in editing success rather than a reduction in leakage specifically, we replaced our claim that both mechanisms "reduce leakage" with the claim that each raises editing success on real recordings. The contribution list in Section 1 now reads:

A persistent edit head keeps the instruction accessible to every decoder layer, and intermediate-layer music alignment preserves stem-level structure. Ablation study shows that both improve editing success on real recordings.

Please note that the learnable queries follow BLIP-2 and the gated cross-attention follows Flamingo; Our contribution lies in how they are combined to successfully perform music editing: in a codec language model, the instruction is followed by more than 2,000 audio tokens, and the queries and cross-attention act as a shortcut that keeps the instruction within reach of every output code.

We also clarify that all ablation variants, including ProCraft-plain, keep the 32 learnable queries in their input sequence, and only the cross-attention path that reads them is removed. The variant without cross-attention thus tests the mechanism alone, and it lowered the edit rate on real recordings by 0.12 on both removal and extraction, indicating that the queries help mainly through the cross-attention path. The number of queries, K = 32, follows the Q-Former of BLIP-2 and was not tuned. Section 5.2 and Appendix D now state this explicitly:

Every variant keeps the 32 learnable queries in its input sequence, follows the training recipe of Appendix E, and is scored on the same test sets and metrics as ProCraft.

After training, the per-layer gate values |tanh α| ranged from 0.0004 to 0.031. Although these values are small, the cross-attention still affects the output strongly: setting the gates to zero at inference changed the final hidden states by 37% (relative L2 distance over 100 test items). The gate is initialized at zero, as in Flamingo, so that the model starts from the pretrained backbone; In the orignal Flamingo paper, the author reports that this gating improved both training stability and final performance.

On removal, the intrusive metrics (SI-SDR, mel-L1 and MRSTFT) differed little between ProCraft and its variants, because for a codec language model they are dominated by the codec rather than by the edit. Encoding and decoding the ground-truth target with WavTokenizer already yields only +1.05 dB SI-SDR on Slakh removal (Section 5.1), and at most +5.1 dB across removal, extraction and denoising on both test sets. The differences between our variants are much smaller than this codec error. For this reason, we measure Editing with edit rate and edit strength, which test whether the requested stem was actually removed or isolated. Section 5.1 reports this codec ceiling:

To explain this gap, we encoded and decoded the ground-truth targets with WavTokenizer, which yielded +1.05 dB, so 7.3 dB of this 11.3 dB gap stems from the low-bitrate (0.9 kbps) single-codebook codec rather than the model.

AuthorsOpenReview ↗

Response for Reviewer 55br (2/2)

It was indeed the case that the submitted FAD and KAD scores used a single reference set of 390 full-mix crops from MoisesDB and MUSDB18, which favoured outputs that leave the mix unseparated and overlapped with our MoisesDB test songs. We thank you for pointing this out, and we fixed this error now with reference sets drawn from vocal-excluded MUSDB18-HQ only, which shares no song with our test sets, and matched each set to the output an operation should produce: full mixes for removal and denoising, isolated stems for extraction and accompaniment, the "other" stem for cover, and the bass and "other" stems for instrument transfer. With these new reference sets, the MoisesDB extraction FAD of AudioSep changed from 0.34 to 0.31 and that of ProCraft from 0.52 to 0.14, so AudioSep is no longer best. All FAD and KAD values in the revised paper use the new reference sets, which Appendix F describes as follows:

Each reference set matches the output an operation should produce: 419 full-mix crops for removal and denoising; 567 isolated-stem crops (drums, bass, and MUSDB's "other" stem, which holds guitar and piano) for extraction and accompaniment; the 282 "other" crops for cover; and the 423 bass and "other" crops for instrument transfer, whose output is one re-rendered stem.

Generation rarely stopped early, as output length is capped at the source length of 2,250 codes. Over 14,400 outputs, 0.9% ended more than 1 s before the source length. When output and target differ in length, both are trimmed to the shorter one. Appendix A now adds:

Generation stops at the first audio end-of-sequence token, with a fallback maximum equal to the source-code length: 2,250 decoding steps for a 30 s clip at 75 Hz. Generation rarely stops early: about 1% of outputs end more than 1 s before the source length.

Regarding ablation of the alignment targets, removing the StemFX target did not lower the quality of audio effects, possibly because the other alignment targets also carry information about the effects applied to the target. More generally, no ablation variant lowered effects quality significantly: AFx-Rep cosine stayed within 0.393–0.405 on Slakh and 0.190–0.196 on MoisesDB, including the variant without any alignment (AFx-Rep rows of Tables 9 and 10). This is consistent with the 2x2 ablation, in which both mechanisms mattered most for removal, extraction and instrument transfer, where the model must find the named stem or keep the content of the source.

We also thank you for suggesting COCOLA and PHALAR as Preservation metrics to evaluate how well an accompaniment fits its source per item. These are now added in Tables 5–6 and 9–10 and to the accompaniment panels of Figure 3 (definition in Appendix F). Both average a learned similarity between the generated part and the source over aligned 5 s windows. Please note that Instruct-MusicGen returns a full mix on accompaniment, so its added part is first separated with HTDemucs-6s. We report that with COCOLA, ProCraft fit its source best on Slakh and second best on MoisesDB, whereas PHALAR ranked the dedicated accompaniment models (MusicGen-Stem, MSDM and the released Instruct-MusicGen) above ProCraft:

Slakh / MoisesDB (bold = best, italic = second best)COCOLAPHALAR
ProCraft62.24 / 62.104.01 / 2.65
ProCraft-plain61.46 / 61.843.29 / 2.58
MusicGen-Stem60.58 / 64.294.79 / 4.87
MSDM60.17 / 59.974.51 / 3.81
IMG (released)59.63 / 61.624.51 / 3.50
IMG (retrained)51.65 / 53.651.28 / 0.26

The two metrics differ in what they compare: PHALAR uses STFT features that keep phase, whereas COCOLA uses magnitude spectrograms. Our codec does not guarantee phase consistency with the source, which may explain why ProCraft ranks lower under PHALAR than under COCOLA. The revised Section 5.1 notes this.

We also corrected the two typos ("each part of the sequence ss" and ". Instruction").

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response for Reviewers (1/2)

We sincerely thank the three reviewers for their time and effort on helping us improve our submission. The main concern raised by all reviews is that our ablation was unconclusive, as it removed both the REPA-style embedding alignment and cross-attention. Indeed, the two mechanisms solve different problems: alignment teaches an early layer the musical content of the source (which instruments play, what notes) in order for the model to know what is in the mix that it must edit, while cross-attention carries the instruction to every layer to ensure the request is not diluted over the ensuing sequence of audio tokens. We clarified the role of these two mechanisms by performing a much more rigourous ablation as follows:

  • a 2x2 ablation of cross-attention × alignment (ProCraft, no cross-attention, no alignment, neither);
  • ProCraft with one alignment target removed (MERT, pitch, instrument, StemFX).

Please also note that ProCraft-plain has been retrained for the 2x2 and replaced in all tables as we found a bug where alignment only removed MERT instead of all four embeddings. The new Table 2 reports the editing metrics of the 2x2 and Appendix D (Tables 9-10) now includes the results with each alignment target ablated. Here is the new Table 2 (S = Slakh, M = MoisesDB; bold = best, italic = second best in column):

SystemRemoval edit rate (S / M)Removal strength, dB (S / M)Extraction edit rate (S / M)Extraction strength, dB (S / M)Instr. match (S / M)Frame-F1 (S)
ProCraft (ours)0.527 / 0.38010.15 / 7.370.787 / 0.82012.18 / 12.510.640 / 0.6070.558
−alignment0.453 / 0.2737.57 / 5.250.760 / 0.85312.12 / 13.250.460 / 0.5130.525
−cross-attention0.527 / 0.2628.41 / 4.890.693 / 0.7008.87 / 7.490.620 / 0.5730.523
ProCraft-plain (neither)0.460 / 0.2406.83 / 4.820.640 / 0.7808.80 / 12.030.447 / 0.5200.526

In short, this new ablation study revealed that

  • Removal needs both alignment and cross-attention, as the model must find the named stem (instruction) and keep everything else intact (content). On real recordings (MoisesDB), ablating either mechanism lowered removal edit rate from 0.38 to 0.26–0.27, and removing both lowered it to 0.24.
  • Cross-attention affected extraction the most: extraction must find the exact stem the instruction names while dropping the rest of the content, so it depended more on following the request. Without cross-attention, edit rate dropped by 0.12 (MoisesDB) and 0.09 (Slakh), whereas removing alignment did not lower it on MoisesDB (0.85).
  • Alignment affected instrument transfer the most: instrument transfer must keep every note of a track while changing its instrument, so it depends on knowing the content of the source. Without alignment, instrument match drops from 0.64 to 0.46 on Slakh.

The revised Section 5.2 states this split as follows:

Instrument transfer must keep every note of a track while changing its instrument, and it suffered most without alignment: instrument match fell from 0.64 to 0.46 on Slakh and from 0.61 to 0.51 on MoisesDB, against 0.62 and 0.57 without cross-attention. Extraction must find the exact stem that the instruction names while dropping the rest of the content, and it suffered most without cross-attention: on MoisesDB its edit rate fell from 0.82 to 0.70 and its edit strength from 12.5 to 7.5 dB, whereas without alignment it stayed at 0.85. Removal needs both, as the model must find the named stem and keep everything else intact: on MoisesDB its edit rate fell from 0.38 to 0.27 without alignment and to 0.26 without cross-attention, and to 0.24 without both.

In addition, we performed an error analysis to better understand the failure modes of our model to stem removal on real music in MoisesDB in Appendix J. These were categorised and tallied in terms of Copy-through, Partial suppression, Wrong stem, and Others failures, and broken down by the stem to be removed in Table 13. We summarise our findings in Section 5.1 as follows:

We found that most of ProCraft's failures left the requested stem in place or reduced it only partly. These failures concentrated on guitar and bass stems, and bass removal failed far more often than on Slakh, which points to the gap between rendered and recorded instrument sounds.

AuthorsOpenReview ↗

General Response for Reviewers (2/2)

We further performed an out-of-domain evaluation (new Section 5.5, Table 4) on audio effects editing with 200 human-written prompts from the SonicMaster test set. We found that our On-the-fly Editing-pair Synthesis allowed the model to generalize robustly: ProCraft achieved competitive performance amongst exclusive audio-effect editing models, despite being a general music editor that also removes, extracts, and generates stems from the same model:

FAD ↓ / AFx-Rep ↑ (bold = best, italic = second best)EQdynamicsreverbclipping
ProCraft0.241 / 0.2970.290 / 0.3170.305 / 0.2990.260 / 0.293
InstructFX2FX0.253 / 0.4080.233 / 0.7250.399 / 0.3330.225 / 0.372
IMG (retrained)0.536 / 0.2830.593 / 0.2740.603 / 0.2720.535 / 0.291
FxSearcher0.397 / 0.1490.519 / 0.1530.665 / 0.1260.510 / 0.118
Text2FX0.714 / 0.1440.713 / 0.1360.860 / 0.1410.764 / 0.108

We also fixed the FAD/KAD reference set (per-task pools from MUSDB18-HQ, no overlap with our test songs), added COCOLA and PHALAR as additional metrics in all result tables and in Figure 3.

Once again, we thank each reviewer for your detailed feedback. We hope that we have already addressed all concerns satisfactorily, but please feel free to suggest how we could improve our submission further before the end of the discussion period so that our work could be of a meaningful contribution to the community.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates for the Area Chair (1/2)

Dear Area Chair,

We thank the three reviewers for their constructive feedback, which we believe have substantially improved the quality and contribution of our submission. Here, we summarise their concerns and our responses.

A unified framework for instruction-driven music editing

ProCraft takes a music recording and a text instruction and generates the edited recording with a single 0.5B codec language model. It handles 30-second recordings, eight instrument families, and seven operations: removal, extraction, accompaniment, cover, instrument transfer, denoising, and audio effects, as well as compound requests combining arrangement edits with effects. Our contribution is to bring these operations into one instruction-driven framework, supported by an on-the-fly pipeline that generates instructions paired with executable edits and their audio targets. The model uses a persistent edit head to keep the instruction and input-audio information accessible throughout generation, and stem-level alignment objectives to learn the musical structure needed for editing.

Reviewer Qogc recognised the novelty and practical relevance of the task, the synthesis pipeline, and the editing quality of the small model, while Reviewer 55br highlighted the pipeline and thorough evaluation, and Reviewer 8LsW recognized the motivation for the persistent edit head and acknowledged the insights learnt from their proposed error analysis. Following our additional experiments, Reviewer Qogc confirmed that the ablation concerns were addressed and raised the rating to 4 (Clear acceptance).

Addressing each reviewer's concerns

1. Separating the contributions of the two mechanisms [Reviewers 55br, Qogc]. The original ablation removed cross-attention and alignment together, preventing attribution of their effects. We added a 2×2 ablation with the backbone and learnable queries fixed, plus individual ablations of all four alignment targets (Appendix G, Tables 14–15). These ablations successfully distinguished each mechanism's role: cross-attention helps extraction, alignment helps instrument transfer, and removing either reduces removal success on real recordings. We also corrected and reran the original ProCraft-plain ablation after identifying an alignment-removal bug. These results support distinct, task-dependent benefits, addressing the attribution concern that Reviewer Qogc accepted.

2. Comparing with specialist tools and examining joint reasoning [Reviewer 8LsW]. The reviewer asked whether an LLM could interpret the instruction and then call existing specialists, without the need for a unified model that we proposed. To address this, we built a chain modular baseline in which Qwen3-30B-A3B-Instruct reads the instruction and track, then calls specialist models for the selected operations. We executed these calls on the same single-operation test items as ProCraft and added the baseline to Appendix C (Tables 7–9). Results show that our model outperformed this chaining baseline on accompaniment, instrument transfer, and extraction. Examining the chaining LLM's accuracy in calling the correct specialist or track further revealed a substantial drop in performance to ambiguous instructions, which constrain the strength of each specialist model.

We have also clarified the scope of the architectural evidence in Section 5.2: The fixed-backbone ablation in point 1 supports keeping the joint instruction–audio queries accessible during generation, but does not isolate joint versus separate conditioning. Likewise, the comparison with Instruct-MusicGen cannot isolate joint conditioning because its backbone differs.

3. Testing transfer to human-written requests [Reviewer 8LsW]. The reviewer proposed a clarification question of whether training and testing with instructions from one language model establishes robustness to human wording. To that end, we evaluated additional success metrics on the SonicMaster dataset, as well as introduced a new test on the Mix Evaluation dataset (Tables 4, 10, and 11).

Examining edit success rate and edit application rate on SonicMaster revealed that InstructFX2FX's lead on AFx-Rep was likely largely driven by leaving the input unedited or by making very small edits. By contrast, ProCraft edited all clips and maintained audio quality while preserving the source content more than the remaining models.

To further evaluate model performance on vague, human-written prompts, we tested each model on listening comments in Mix Evaluation, where the appropriate operation must be inferred. While our model scored highest on AFx-Rep and FAD, no model (including ours) came closer to the best-rated mix than the unedited source. We therefore suggest that such comment-driven inferences remain an open problem and tone down our claims on robustness to user-written instructions.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates for the Area Chair (2/2)

4. Explaining removal failures and correcting evaluation references [Reviewers 55br, 8LsW]. Reviewer 8LsW asked why removal success was low on real recordings and whether the codec--rather than the editing model--was responsible. In response, we carried out an error analysis where we categorised removal failures into unchanged outputs, partial suppression, wrong-stem edits, and other failures in Appendix M (Table 18). We also passed the correct target audio through the codec: the reconstructed targets passed the removal test for every guitar item and 85% of bass items, to show that codec reconstruction alone does not explain the unsuccessful edits. 8LsW explicitly confirmed that this error analysis answered their question (follow-up).

On the other hand, Reviewer 55br pointed out that there was overlap in the MUSDB18-HQ data for FAD/KAD metrics and the test set. We have now corrected this, so that the reference pools use task-matched MUSDB18-HQ audio with no overlap with test songs (Appendix I). We have also added COCOLA and PHALAR as complementary accompaniment measures in Tables 5–6 and 14–15.

Summary

To summarise, the discussion period has strengthened our paper in four ways. First, we showed that ProCraft outperformed a chain of specialist tools on extraction, accompaniment, and instrument transfer. Second, the 2×2 ablation clarified the respective contributions of the gated cross-attention and representation alignment. Third, the error analysis and the SonicMaster and Mix Evaluation tests identified ProCraft's failure modes and limitations. Finally, the evaluation now includes unedited references and explicit success criteria, which together with our multiple metrics give a realistic picture of model performance. We believe these new insights strengthen our central conceptual advancement of a single generative model that performs seven music-editing operations from free-text instructions, which represents a solid contribution to the conference.

We would be most grateful if you would consider our responses favourably in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.