- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ProCraft, a 0.5B autoregressive codec language model designed for seven distinct music-editing tasks, ranging from stem extraction to audio effects. To tackle long codec sequences, ProCraft incorporates two key designs: a persistent edit head that injects 32 compressed instruction-audio queries across all decoder layers via gated cross-attention, and an intermediate representation alignment (at layer 4) guided by MERT, pitch/instrument tags, and StemFX embeddings. To address the scarcity of compound editing data, the authors synthesize training triplets on the fly from Slakh2100 using Qwen3-30B tool calls paired with MultiAFX.
Critical strengths
-
The persistent edit head directly targets the causal context dilution issue over long sequence generation (2,250 tokens). The two-pass training scheme is a thoughtful design that ensures training matches inference-time condition caching.
-
Transferring REPA-style alignment (layer 4 aligned with MERT, pitch/instrument labels, and effect embeddings) to codec LMs is sound and empirically proves beneficial for out-of-domain stem extraction and removal.
-
The paper is honest about its bottlenecks—specifically isolating that 7.3 dB of the SI-SDR gap is attributable to the codec reconstruction itself, and designing evaluation metrics that actively penalize trivial identity shortcuts.
Critical weaknesses
-
While the model claims to unify seven operations, the full breakdown in Appendix B (Tables 3 & 4) shows that ProCraft fails to consistently win across specific tasks. Dedicated baselines outperform it significantly: separators dominate SI-SDR, MusicGen-Stem wins on accompaniment instrument matching, piano-cover models better track source melody, and InstructFX2FX is closer on audio effects. The only significant gains are against weak, general-purpose free-text editors. In production, a modular pipeline chaining specialized tools (e.g., separator + FX engine) would likely offer far better performance, undermining the practical motivation for this unified generative model.
-
The training and evaluation pipelines rely entirely on synthetic mixes (Slakh2100) and instructions generated by a single LLM (Qwen3-30B) via templates. The paper does not evaluate robustness against authentic human prompts (which are often ambiguous, musically naive, or noisy), leaving its generalization in real user-facing scenarios unverified.
Questions for authors
- The edit rate on real-world recordings (MoisesDB removal) remains still low at 38.0%. Could you provide a qualitative or quantitative breakdown of the remaining 62% failure cases? Specifically:
- Are these failures caused by instruction/stem misidentification, partial/insufficient suppression, or artifact corruption?
- Is this primarily a limitation of the acoustic modeling capacity or the low-bitrate codec representation?
- Both the training and evaluation instructions are synthesized by Qwen3-30B using predefined motivation templates. Have you evaluated the model on authentic, user-generated instructions that may contain informal slang, ambiguous terminology, or musically naive descriptions? How robust is the persistent edit head when prompt wording deviates from the LLM template distribution?
7 comments
Response for Reviewer 8LsW (2/2)
We are also grateful for the suggestion to test ProCraft on authentic human-written instructions. To that end, we evaluated it on the SonicMaster test set, whose prompts were written by a music expert for amateur musicians in plain wording, and none of which comes from our language model (new Section 5.5, Table 4). We selected 200 clips from its test split, 50 in each of four groups (EQ, dynamics, reverb, and clipping), with 75 distinct prompts in total. Each prompt asks to correct a degradation, such as fixing the EQ, removing reverb, or declipping, a task that is absent from our training data, so this set is out-of-domain in both wording and task. We also measured how far its prompts are from ours using Fréchet BERT Distance (Section 5.5), noting that:
These prompts differ from ours: their Fréchet BERT Distance to our audio-effect test prompts is 3.74, against 0.87 between two sets of our own audio-effect prompts of the same size (75), and 70% of the words in the SonicMaster prompts don't exist in ours.
We compared ProCraft with Instruct-MusicGen retrained on our data and the audio-effect baselines of the paper. FAD uses the MUSDB18-HQ full-mix reference set, and AFx-Rep compares each output with the clean target (see Table 4 and General Comments).
We found that ProCraft had lower FAD than IMG (retrained), FxSearcher, and Text2FX in every group, and than InstructFX2FX on EQ and reverb, so it handled the unseen prompts without degrading the audio. On AFx-Rep, InstructFX2FX was closest to the target in every group, and ProCraft ranked second. These show the robustness of our model despite being trained only on synthetic instructions. Section 5.5 puts this in context:
InstructFX2FX, FxSearcher, and Text2FX only apply audio effects, which is all this test set asks for, whereas ProCraft is a general music editor that also removes, extracts, and generates stems from the same model. Overall, ProCraft generalized to human-written prompts and remained competitive with these dedicated audio-effect baselines, while InstructFX2FX was best on AFx-Rep.
Thanks — the error analysis answers my Question 2, and it's a good one: since the codec-decoded targets pass the removal test, the codec is not the problem. The model just isn't carrying out the edit on real recordings. But that doesn't fix Weakness 1. Most failures are the model doing nothing (35.5% copy-through) or half the job (28% partial suppression) — exactly the failure modes a simple tool chain doesn't have. On the same MoisesDB removal items, AudioSep and Banquet get edit rates of 0.61 and 0.71 against 0.38. And the chained baseline is still not run: your benchmark requests are literal ("cut the bass and add reverb"), and your own data pipeline already does the modular thing (LLM tool calls + MultiAFX), so the "tighten the mix" argument doesn't really apply to the comparison I asked for. Also, the Instruct-MusicGen gap still mixes up the conditioning design with the backbone (Section 5.2 admits this), and the claimed benefit of joint reasoning is never actually tested on ambiguous instructions. So Question 2 is answered; Questions 1, 3, and 4 look open. Keeping my score.
Response for Reviewer 8LsW (1/2)
We clarify that ProCraft is not meant to replace dedicated systems on each operation, but rather demonstrate that a single model can infer what to edit jointly from both the instruction and the audio. It reads the instruction and the source audio in one sequence, so every output code can attend to both, and the benefit of this joint conditioning shows in the comparison with Instruct-MusicGen retrained on our data, which conditions audio and text on the two separate paths and trailed ProCraft on nearly every metric of every operation. Section 5.2 reports:
Such joint reasoning matters in music production, where the same request can call for different edits depending on the song: "tighten the mix" might mean compressing the drums in one song and cutting the low end of the guitars in another. A modular pipeline would need a separate component to make this decision from the instruction and the audio before calling its tools, which is the problem ProCraft addresses within one model. Although ProCraft still trailed deterministic task-specific systems such as source separators, it was competitive with generative task-specific systems, with a 0.5B-parameter model trained only on Slakh2100, and we believe this joint approach offers a simple and general basis for instruction-driven music production.
We thank you for the suggestion of performing an error analysis. We note that ProCraft removed the requested stem in 38.0% of MoisesDB removal items, and most of the other 62% left the requested stem in place or made it only partly quieter. To label these failures, we separated the source, the target, and the output with HTDemucs-6s, as for edit rate, and measured how much the level of each stem changed from the source to the output. In 33 items (35.5%), the requested stem changed by less than 2 dB and the output stayed close to the source after encoding and decoding (copy-through). In 26 items (28.0%), the requested stem became 2 to 6 dB quieter, which is not enough to pass the threshold (partial suppression). In 15 items (16.1%), another stem was removed instead (wrong stem); in 9 of these, the instruction used the phrasing "pull the X out", and ProCraft isolated the stem instead of removing it. The remaining 19 items (20.4%) were nearly silent or cut short, damaged stems that should have stayed, or could not be judged reliably by the test. By requested stem, guitar and bass account for 74 of the 93 failures (each about 75%, against 38% for drums and 36% for piano). Neither the codec nor loudness explains this: the codec-decoded target passes the test for every guitar item and 85% of bass items, and the requested stem is as loud in failed as in successful items. Bass removal fails in 29% of Slakh items but 75% of MoisesDB items, which points to the gap between rendered and recorded instrument sounds, while guitar is hard in both domains (78% on Slakh). Section 5.1 summarizes this analysis, and Appendix J (Table 13) gives the full breakdown: