- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces RIME, a framework that generates realistic paired edit-instruction data from any baseline music dataset. RIME is realized by POEMS, a toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. The authors use RIME to build a 15K benchmark from MusicCaps with instructions at three abstraction levels, evaluate zero-shot multimodal agents on it, and fine tune Gemma 3N on RIME generated data. Validation includes a human listening study (N=10) confirming instruction-audio correspondence. The authors also present audio embedding and edit graph recovery metrics, where they show agents struggle as instructions become more abstract, with SFT helping mainly on explicit instructions.
Critical strengths
-
A well motivated, musician centered task with clear practical relevance. I really like that the paper focuses on iterative post production, meaning tweaking specific parts of an existing mix instead of one shot generation. That's much closer to how producers actually work. I particularly appreciate the distinction between instruction abstraction levels (AL0-2) which captures how production intentions can be communicated with varying degrees of granularity.
-
I also like the framework design, representing post production workflows as executable, stem aware edit graphs does make sense and I can see the authors have put much effort in designing the framework including recipes and parameter priors.
Critical weaknesses
-
The evaluation of AL2 level does not clearly establish editing success. At AL2, instructions deliberately omit effect names, order, and parameters, but agents are evaluated against the single reference graph used to generate them? Many different edit chains can achieve the same perceptual request, for example, an instruction like "give the bass a distant and spacious feel" could be achieved through room reverb, but also by a short delay, chorus/ensemble effects, and many more. The evaluation however primarily measures whether the agent recovers the exact effects and parameter values in the reference graph rather than whether the resulting audio successfully fulfills the instruction. This effectively treats a problem with multiple valid solutions as one with a single correct answer, which potentially penalizes valid alternative approaches and discouraging diversity in editing strategies. Consequently, the sharp drop in graph and parameter recovery may partly reflect non-identifiability rather than model failure. and I believe this could undermine the interpretation of the paper's central benchmark results, especially its conclusions about agents' limitations in handling abstract production requests.
-
The contribution of RIME's specific design choices is not isolated in the SFT experiments. The SFT experiment shows that training on RIME generated data improves performance relative to the untuned baseline (particularly at AL0). However there is no comparison against simpler synthetic data generation approaches, such as randomly sampled effect chains paired with explicit descriptions, under a matched training budget. The training set is also constructed to cover every recipe used in evaluation. As a result it is unclear whether the improvements come from RIME's production-oriented recipes, priors, and constraints, or simply from exposure to (any) synthetic examples and familiarity with a fixed tool grammar. Since the paper's central claim is that RIME provides useful training data, comparing only against an untuned model does not strongly support whether RIME's carefully designed data generation process is necessary or offers meaningful advantages over simpler alternatives.
Questions for authors
I appreciate the motivation and effort behind the work. I am open to raising my score if the authors can address the following questions, particularly those concerning evaluation validity and the contribution of RIME's design.
- Evaluation validity at AL2:
At AL2, instructions deliberately omit effect names, order, and parameters, yet agents are evaluated against the single reference graph used to generate them. Since multiple edit chains could satisfy the same perceptual request, I wonder whether the sharp decline in graph and parameter recovery reflects model limitations or the evaluation methodology itself.
Could the authors provide additional evidence to help distinguish these possibilities? For example, human listeners could directly evaluate whether the agent's output fulfills the given instruction (either by listening study asking the edited audio fulfills the instruction, or by a study w/ experienced producers asking whether the resulting edits musically make sense given instruction).
I am also curious about the audio similarity results. In Appendix G.1 three of the four models have lower output-target MERT similarity than the input-target similarity (i.e., leaving the audio unchanged). How should these results be interpreted alongside the main results in Table 1? Could this be because the edits are too subtle for MERT to capture reliably, or because the models struggle to produce the intended audio? A human evaluation of the agent outputs could be helpful in clarifying this distinction as well.
- Contribution of RIME's design to SFT improvements
The SFT experiments show that training on RIME generated data improves performance relative to the untuned baseline (particularly at AL0), however I am curious how much of this improvement can be attributed specifically to RIME's production informed design. Could the authors provide an internal ablation comparing RIME against simpler synthetic data generation approaches, such as random sampled effect chains without the proposed recipes, parameter priors, or constraints, while keeping the training data size and budget fixed? I believe this could help clarify whether RIME's carefully designed components provide additional benefits beyond training on miscellaneous synthetic examples.
- Coverage/diversity of production practices
RIME's pipeline favors conventional production practices through a fixed set of recipes and parameter constraints that exclude extreme settings. This seems reasonable for an initial benchmark, but I also wonder whether it might limit the diversity of editing approaches including unconventional or experimental techniques that are also used in music production.
Also could training against a single reference chain for each abstract instruction encourage agents to converge toward relatively homogeneous editing strategies, even when multiple valid approaches exist? How do the authors view this tradeoff between production realism and creative flexibility? Could RIME be extended to suport multiple valid realizations of the same instruction or less conventional processing techniques?
- Potential for real world applications
I would appreciate a short discussion, perhaps in the appendix, elaborating on how RIME And POEMS could be applied or extended to real world production environments. I could see a bit of possibilities by reading the introduction but would be interesting to see more concrete examples of potential applications an dfuture directions.
For example, could agents trained using RIME and POEMS transfer to actual DAW environments through integrations such as an Ableton MCP server? Also different DAWs and third-party plugins often expose different parameter ranges, scales, and naming conventions for nominally equivalent effects. how might the proposed framework generalize across these tools?
- Writing and presentation
I have a minor suggestion regarding the framing of prior work. I would encourage the authors to emphasize methodological differences and the contributions of RIME, rather than primarily highlighting the limitations of existing approaches.
For example,
- Line 174: I find the phrasing "we found it produced poor results" somewhat subjective and unnecessarily negative. Could the authors use more neutral, objective language and clarify the specific limitations they observed when using SAM Audio in this setting, ideally with supporting evidence or examples?
- Line 127-128: The comparison with LLM2Fx-Tools is relevant, but could the authors consider framing it more around the methodological differences and additional capabilities introduced by RIME, rather than focusing primarily on what the earlier work lacks?
Also few minor things:
- Line 329 is missing reference
- Line 453-457: The text discusses operator F1 as a function of graph complexity but Figure 5 displays MERT audio similarity?
6 comments
Author Response 2/3
As a result it is unclear whether the improvements come from RIME's production-oriented recipes, priors, and constraints, or simply from exposure to (any) synthetic examples and familiarity with a fixed tool grammar… Could the authors provide an internal ablation comparing RIME against simpler synthetic data generation approaches, such as random sampled effect chains without the proposed recipes, parameter priors, or constraints, while keeping the training data size and budget fixed?
We have run a new SFT experiment on 3,000 random effect plans with the identical training protocol. When evaluated on a matched random subset of 10% of the evaluation set, we found the following results. We will add a full comparison in the revision, but this may take longer than the rebuttal period.
Results below are presented as “Regular SFT / Random SFT” for Gemma 3n in all result columns.
| Metric | Overall (Reg./Rand.) | AL 0 | AL 1 | AL 2 |
|---|---|---|---|---|
| Completion (%) ↑ | 99.4 / 95.4 | 99.6 / 100.0 | 99.8 / 98.8 | 98.8 / 87.4 |
| FAD (MERT) ↓ | 0.132 / 0.147 | 0.132 / 0.168 | 0.228 / 0.205 | 0.270 / 0.275 |
| KAD (MERT) ↓ [IGNORE, ALL SIMILAR] | -0.018 / 0.005 | -0.101 / -0.093 | -0.077 / -0.098 | -0.048 / -0.018 |
| Output-target similarity: MERT ↑ | 0.971 / 0.972 | 0.983 / 0.979 | 0.969 / 0.973 | 0.961 / 0.963 |
| Output-target similarity: CLAP ↑ | 0.874 / 0.868 | 0.915 / 0.901 | 0.872 / 0.876 | 0.836 / 0.826 |
| Edit similarity: MERT ↑ | 0.613 / 0.593 | 0.731 / 0.699 | 0.640 / 0.629 | 0.467 / 0.450 |
| Edit similarity: CLAP ↑ | 0.731 / 0.705 | 0.814 / 0.791 | 0.747 / 0.731 | 0.633 / 0.593 |
| Graph F1 ↑ | 0.846 / 0.786 | 0.929 / 0.908 | 0.888 / 0.867 | 0.721 / 0.582 |
| Parameter score ↑ | 0.467 / 0.449 | 0.871 / 0.870 | 0.375 / 0.375 | 0.154 / 0.102 |
| Parameters within 10% (%) ↑ | 34.3 / 33.8 | 81.7 / 80.9 | 15.1 / 16.5 | 6.0 / 4.0 |
| Parameters within 25% (%) ↑ | 41.6 / 40.5 | 85.0 / 84.7 | 28.3 / 29.4 | 11.5 / 7.4 |
simpler synthetic data generation approaches, such as randomly sampled effects…including unconventional or experimental techniques that are also used in music production.
Part of the training/evaluation sets contain randomly sampled effect chains. We have added a paragraph to the data generation part of section 4 to explain this better. Additionally, we include results on just the random subset of our evaluation data below. However, we note that these random chains are often more complex and thus have a higher baseline difficulty.
Results below are presented as “Priors-Based / Random-Constrained” in all result columns.
| Model | $N$ | FAD $\downarrow$ | KAD $\downarrow$ | $\Delta \text{sim}_a \uparrow$ | Graph F1 $\uparrow$ | Params $\uparrow$ |
|---|---|---|---|---|---|---|
| Gemini | 14295 / 705 | 0.04 / 0.59 | 0.02 / 0.63 | 0.54 / 0.38 | 0.65 / 0.38 | 0.33 / 0.07 |
| GPT | 14295 / 705 | 0.03 / 0.42 | 0.02 / 0.13 | 0.65 / 0.41 | 0.83 / 0.63 | 0.45 / 0.29 |
| Gemma | 14295 / 705 | 0.09 / 0.43 | 0.11 / 0.06 | 0.61 / 0.53 | 0.86 / 0.82 | 0.38 / 0.32 |
| Gemma SFT | 14295 / 705 | 0.03 / 0.44 | 0.02 / 0.15 | 0.62 / 0.43 | 0.86 / 0.74 | 0.49 / 0.33 |
In Appendix G.1 three of the four models have lower output-target MERT similarity than the input-target similarity (i.e., leaving the audio unchanged). How should these results be interpreted alongside the main results in Table 1? Could this be because the edits are too subtle for MERT to capture reliably
We believe this is likely, and so we provide the MERT results for transparency, but do not consider the resolution here meaningful enough to interpret them.
A human evaluation of the agent outputs could be helpful in clarifying this distinction as well
We unfortunately cannot scale an appropriate human evaluation for this within the rebuttal period, since this requires (1) properly-trained listeners (e.g. engineers who are accustomed to hearing fine differences in production), (2) enough of them to have statistically meaningful results, (3) a new IRB protocol. We will explore whether this is feasible, however we believe the primary results do not depend on it.
converge toward relatively homogeneous editing strategies
This is a known problem with SFT, where probability mass is centered around a known, correct solution. It is possible that reinforcement learning with audio-based rewards could mitigate this; however, we leave that for future work
Author Response 3/3
For example, could agents trained using RIME and POEMS transfer to actual DAW environments through integrations such as an Ableton MCP server? Also different DAWs and third-party plugins often expose different parameter ranges, scales, and naming conventions for nominally equivalent effects. how might the proposed framework generalize across these tools?
This is an excellent suggestion, and we have included this discussion in Appendix A in our revised draft.
Line 127-128: The comparison with LLM2Fx-Tools is relevant, but could the authors consider framing it more around the methodological differences and additional capabilities introduced by RIME, rather than focusing primarily on what the earlier work lacks?
We have updated this sentence to be a reflection of the previously unexplored areas in agentic production that our system tackles. We appreciate this feedback, as this is what we really intended to address with this sentence.
Line 174: I find the phrasing "we found it produced poor results" somewhat subjective and unnecessarily negative. Could the authors use more neutral, objective language and clarify the specific limitations they observed when using SAM Audio in this setting, ideally with supporting evidence or examples?
This has been updated to reflect the specific limitations of SAM Audio as it pertains to music separation and a few justifications as to why we instead opted for Demucs.
Line 329 is missing reference
Fixed, thanks
Line 453-457: The text discusses operator F1 as a function of graph complexity but Figure 5 displays MERT audio similarity?
Graph complexity is the x-axis (number of operations) and MERT audio similarity is the y-axis.
Author Response 1/3
Thank you for your thoughtful and detailed review. We very much appreciate your positive comments about the motivation, effort involved, practical relevance, and framework design. Please find our responses to your points below.
To summarize our responses to your five questions (with more detail below):
—
This is a valid point, however, we have several methods of guarding against this. We provide audio-based metrics which only compare the output audio to the ground-truth audio. Our listener study also validates that generated instructions satisfy their corresponding RIME edits more than other edits. In the graph metrics themselves, we provide tolerance scores (specifically for parameterization) with multiple thresholds to guard against judging based on exact match (which is not realistic for real-world post production. We also include FAD/KAD to the reference audio below and refer to Appendix H.1 (Tables 11 and 12) for the output-target audio similarity metrics.
We account for this in the audio metrics, where we look at edit similarity in the MERT embedding space, FAD/KAD between agent output and ground truth, embedding directions, and CLAP-based analysis against the text. The graph-based analysis is primarily used to explain why agents fail, however, the trends we see with abstraction level performance decay are generally consistent in audio-based metrics as well.