acceptodds
Under review as a conference paper at ICLR 2027

RIME: Enabling Large-Scale Agentic Music Post-Production

Abstract

Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully formed from a musician's mind. Despite the promise of music generation models for one-shot output, such fine-grained *iterative refinement* workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as *agentic post-production*, wherein individual aspects of a track are targeted, refined, and combined into a final version. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We observe that there is a language for modifying recorded music that is dense, consistent, and learnable, and to leverage this we introduce the **R**ule-based **I**nstructions for **M**usic **E**diting (**RIME**) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use POEMS and RIME to generate 15,000 pairs of edit instructions and ground-truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning using synthetic data generation. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering. We include all code for RIME at: [https://anonymous.4open.science/r/RIME-0116](https://anonymous.4open.science/r/RIME-0116)

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Music, audio, agents, benchmark, post-training, post-production, editing, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TzdSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces RIME, a framework that generates realistic paired edit-instruction data from any baseline music dataset. RIME is realized by POEMS, a toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. The authors use RIME to build a 15K benchmark from MusicCaps with instructions at three abstraction levels, evaluate zero-shot multimodal agents on it, and fine tune Gemma 3N on RIME generated data. Validation includes a human listening study (N=10) confirming instruction-audio correspondence. The authors also present audio embedding and edit graph recovery metrics, where they show agents struggle as instructions become more abstract, with SFT helping mainly on explicit instructions.

Critical strengths
  1. A well motivated, musician centered task with clear practical relevance. I really like that the paper focuses on iterative post production, meaning tweaking specific parts of an existing mix instead of one shot generation. That's much closer to how producers actually work. I particularly appreciate the distinction between instruction abstraction levels (AL0-2) which captures how production intentions can be communicated with varying degrees of granularity.

  2. I also like the framework design, representing post production workflows as executable, stem aware edit graphs does make sense and I can see the authors have put much effort in designing the framework including recipes and parameter priors.

Critical weaknesses
  1. The evaluation of AL2 level does not clearly establish editing success. At AL2, instructions deliberately omit effect names, order, and parameters, but agents are evaluated against the single reference graph used to generate them? Many different edit chains can achieve the same perceptual request, for example, an instruction like "give the bass a distant and spacious feel" could be achieved through room reverb, but also by a short delay, chorus/ensemble effects, and many more. The evaluation however primarily measures whether the agent recovers the exact effects and parameter values in the reference graph rather than whether the resulting audio successfully fulfills the instruction. This effectively treats a problem with multiple valid solutions as one with a single correct answer, which potentially penalizes valid alternative approaches and discouraging diversity in editing strategies. Consequently, the sharp drop in graph and parameter recovery may partly reflect non-identifiability rather than model failure. and I believe this could undermine the interpretation of the paper's central benchmark results, especially its conclusions about agents' limitations in handling abstract production requests.

  2. The contribution of RIME's specific design choices is not isolated in the SFT experiments. The SFT experiment shows that training on RIME generated data improves performance relative to the untuned baseline (particularly at AL0). However there is no comparison against simpler synthetic data generation approaches, such as randomly sampled effect chains paired with explicit descriptions, under a matched training budget. The training set is also constructed to cover every recipe used in evaluation. As a result it is unclear whether the improvements come from RIME's production-oriented recipes, priors, and constraints, or simply from exposure to (any) synthetic examples and familiarity with a fixed tool grammar. Since the paper's central claim is that RIME provides useful training data, comparing only against an untuned model does not strongly support whether RIME's carefully designed data generation process is necessary or offers meaningful advantages over simpler alternatives.

Questions for authors

I appreciate the motivation and effort behind the work. I am open to raising my score if the authors can address the following questions, particularly those concerning evaluation validity and the contribution of RIME's design.

  1. Evaluation validity at AL2:

At AL2, instructions deliberately omit effect names, order, and parameters, yet agents are evaluated against the single reference graph used to generate them. Since multiple edit chains could satisfy the same perceptual request, I wonder whether the sharp decline in graph and parameter recovery reflects model limitations or the evaluation methodology itself.

Could the authors provide additional evidence to help distinguish these possibilities? For example, human listeners could directly evaluate whether the agent's output fulfills the given instruction (either by listening study asking the edited audio fulfills the instruction, or by a study w/ experienced producers asking whether the resulting edits musically make sense given instruction).

I am also curious about the audio similarity results. In Appendix G.1 three of the four models have lower output-target MERT similarity than the input-target similarity (i.e., leaving the audio unchanged). How should these results be interpreted alongside the main results in Table 1? Could this be because the edits are too subtle for MERT to capture reliably, or because the models struggle to produce the intended audio? A human evaluation of the agent outputs could be helpful in clarifying this distinction as well.

  1. Contribution of RIME's design to SFT improvements

The SFT experiments show that training on RIME generated data improves performance relative to the untuned baseline (particularly at AL0), however I am curious how much of this improvement can be attributed specifically to RIME's production informed design. Could the authors provide an internal ablation comparing RIME against simpler synthetic data generation approaches, such as random sampled effect chains without the proposed recipes, parameter priors, or constraints, while keeping the training data size and budget fixed? I believe this could help clarify whether RIME's carefully designed components provide additional benefits beyond training on miscellaneous synthetic examples.

  1. Coverage/diversity of production practices

RIME's pipeline favors conventional production practices through a fixed set of recipes and parameter constraints that exclude extreme settings. This seems reasonable for an initial benchmark, but I also wonder whether it might limit the diversity of editing approaches including unconventional or experimental techniques that are also used in music production.

Also could training against a single reference chain for each abstract instruction encourage agents to converge toward relatively homogeneous editing strategies, even when multiple valid approaches exist? How do the authors view this tradeoff between production realism and creative flexibility? Could RIME be extended to suport multiple valid realizations of the same instruction or less conventional processing techniques?

  1. Potential for real world applications

I would appreciate a short discussion, perhaps in the appendix, elaborating on how RIME And POEMS could be applied or extended to real world production environments. I could see a bit of possibilities by reading the introduction but would be interesting to see more concrete examples of potential applications an dfuture directions.

For example, could agents trained using RIME and POEMS transfer to actual DAW environments through integrations such as an Ableton MCP server? Also different DAWs and third-party plugins often expose different parameter ranges, scales, and naming conventions for nominally equivalent effects. how might the proposed framework generalize across these tools?

  1. Writing and presentation

I have a minor suggestion regarding the framing of prior work. I would encourage the authors to emphasize methodological differences and the contributions of RIME, rather than primarily highlighting the limitations of existing approaches.

For example,

  • Line 174: I find the phrasing "we found it produced poor results" somewhat subjective and unnecessarily negative. Could the authors use more neutral, objective language and clarify the specific limitations they observed when using SAM Audio in this setting, ideally with supporting evidence or examples?
  • Line 127-128: The comparison with LLM2Fx-Tools is relevant, but could the authors consider framing it more around the methodological differences and additional capabilities introduced by RIME, rather than focusing primarily on what the earlier work lacks?

Also few minor things:

  • Line 329 is missing reference
  • Line 453-457: The text discusses operator F1 as a function of graph complexity but Figure 5 displays MERT audio similarity?
6 comments
AuthorsOpenReview ↗

Author Response 1/3

Thank you for your thoughtful and detailed review. We very much appreciate your positive comments about the motivation, effort involved, practical relevance, and framework design. Please find our responses to your points below.

To summarize our responses to your five questions (with more detail below):

  • Q1/W1: This is a valid point and common in conditional generation problems (which we have now explicitly acknowledged in the paper’s conclusion), but we make extensive efforts to guard against this through audio metrics, comparisons to reference audio, and tolerance scores for graph-based metrics
  • Q2/W2: We include an additional SFT experiment where we train Gemma 3n on randomly sampled effect chains with the same training protocol as our previous SFT experiments. We show that, in general, SFT with RIME priors outperforms SFT with random constrained effects. We also note that our SFT protocol is a first baseline, and we would expect greater improvements in future work using more sophisticated post-training setups.
  • Q3: We include analysis of agent performance on fully random chains as a proxy for unconventional effect chains (we factorize this below). We also discuss the limitations of SFT for agentic production and potential future work that may address this.
  • Q4: We have added Appendix A to the paper, discussing the potential uses of RIME and POEMS in real post-production workflows. We appreciate this thoughtful suggestion and are glad you are thinking about the potential impact of this work for musicians.
  • Q5: We have updated the paper addressing each writing suggestion, and appreciate this feedback to make the paper better

—

Many different edit chains can achieve the same perceptual request
…measures whether the agent recovers the exact effects and parameter values in the reference graph rather than whether the resulting audio successfully fulfills the instruction.

This is a valid point, however, we have several methods of guarding against this. We provide audio-based metrics which only compare the output audio to the ground-truth audio. Our listener study also validates that generated instructions satisfy their corresponding RIME edits more than other edits. In the graph metrics themselves, we provide tolerance scores (specifically for parameterization) with multiple thresholds to guard against judging based on exact match (which is not realistic for real-world post production. We also include FAD/KAD to the reference audio below and refer to Appendix H.1 (Tables 11 and 12) for the output-target audio similarity metrics.

Model / baselineFAD ↓KAD ↓
Untouched input0.10910.1694
GPT Audio Mini0.02550.0157
Gemini 3 Flash0.04050.0236
Gemma 3n0.08540.0998
Gemma 3n SFT0.03280.0222

We account for this in the audio metrics, where we look at edit similarity in the MERT embedding space, FAD/KAD between agent output and ground truth, embedding directions, and CLAP-based analysis against the text. The graph-based analysis is primarily used to explain why agents fail, however, the trends we see with abstraction level performance decay are generally consistent in audio-based metrics as well.

AuthorsOpenReview ↗

Author Response 2/3

As a result it is unclear whether the improvements come from RIME's production-oriented recipes, priors, and constraints, or simply from exposure to (any) synthetic examples and familiarity with a fixed tool grammar… Could the authors provide an internal ablation comparing RIME against simpler synthetic data generation approaches, such as random sampled effect chains without the proposed recipes, parameter priors, or constraints, while keeping the training data size and budget fixed?

We have run a new SFT experiment on 3,000 random effect plans with the identical training protocol. When evaluated on a matched random subset of 10% of the evaluation set, we found the following results. We will add a full comparison in the revision, but this may take longer than the rebuttal period.

Results below are presented as “Regular SFT / Random SFT” for Gemma 3n in all result columns.

MetricOverall (Reg./Rand.)AL 0AL 1AL 2
Completion (%) ↑99.4 / 95.499.6 / 100.099.8 / 98.898.8 / 87.4
FAD (MERT) ↓0.132 / 0.1470.132 / 0.1680.228 / 0.2050.270 / 0.275
KAD (MERT) ↓ [IGNORE, ALL SIMILAR]-0.018 / 0.005-0.101 / -0.093-0.077 / -0.098-0.048 / -0.018
Output-target similarity: MERT ↑0.971 / 0.9720.983 / 0.9790.969 / 0.9730.961 / 0.963
Output-target similarity: CLAP ↑0.874 / 0.8680.915 / 0.9010.872 / 0.8760.836 / 0.826
Edit similarity: MERT ↑0.613 / 0.5930.731 / 0.6990.640 / 0.6290.467 / 0.450
Edit similarity: CLAP ↑0.731 / 0.7050.814 / 0.7910.747 / 0.7310.633 / 0.593
Graph F1 ↑0.846 / 0.7860.929 / 0.9080.888 / 0.8670.721 / 0.582
Parameter score ↑0.467 / 0.4490.871 / 0.8700.375 / 0.3750.154 / 0.102
Parameters within 10% (%) ↑34.3 / 33.881.7 / 80.915.1 / 16.56.0 / 4.0
Parameters within 25% (%) ↑41.6 / 40.585.0 / 84.728.3 / 29.411.5 / 7.4

simpler synthetic data generation approaches, such as randomly sampled effects…including unconventional or experimental techniques that are also used in music production.

Part of the training/evaluation sets contain randomly sampled effect chains. We have added a paragraph to the data generation part of section 4 to explain this better. Additionally, we include results on just the random subset of our evaluation data below. However, we note that these random chains are often more complex and thus have a higher baseline difficulty.

Results below are presented as “Priors-Based / Random-Constrained” in all result columns.

Model$N$FAD $\downarrow$KAD $\downarrow$$\Delta \text{sim}_a \uparrow$Graph F1 $\uparrow$Params $\uparrow$
Gemini14295 / 7050.04 / 0.590.02 / 0.630.54 / 0.380.65 / 0.380.33 / 0.07
GPT14295 / 7050.03 / 0.420.02 / 0.130.65 / 0.410.83 / 0.630.45 / 0.29
Gemma14295 / 7050.09 / 0.430.11 / 0.060.61 / 0.530.86 / 0.820.38 / 0.32
Gemma SFT14295 / 7050.03 / 0.440.02 / 0.150.62 / 0.430.86 / 0.740.49 / 0.33

In Appendix G.1 three of the four models have lower output-target MERT similarity than the input-target similarity (i.e., leaving the audio unchanged). How should these results be interpreted alongside the main results in Table 1? Could this be because the edits are too subtle for MERT to capture reliably

We believe this is likely, and so we provide the MERT results for transparency, but do not consider the resolution here meaningful enough to interpret them.

A human evaluation of the agent outputs could be helpful in clarifying this distinction as well

We unfortunately cannot scale an appropriate human evaluation for this within the rebuttal period, since this requires (1) properly-trained listeners (e.g. engineers who are accustomed to hearing fine differences in production), (2) enough of them to have statistically meaningful results, (3) a new IRB protocol. We will explore whether this is feasible, however we believe the primary results do not depend on it.

converge toward relatively homogeneous editing strategies

This is a known problem with SFT, where probability mass is centered around a known, correct solution. It is possible that reinforcement learning with audio-based rewards could mitigate this; however, we leave that for future work

AuthorsOpenReview ↗

Author Response 3/3

For example, could agents trained using RIME and POEMS transfer to actual DAW environments through integrations such as an Ableton MCP server? Also different DAWs and third-party plugins often expose different parameter ranges, scales, and naming conventions for nominally equivalent effects. how might the proposed framework generalize across these tools?

This is an excellent suggestion, and we have included this discussion in Appendix A in our revised draft.

Line 127-128: The comparison with LLM2Fx-Tools is relevant, but could the authors consider framing it more around the methodological differences and additional capabilities introduced by RIME, rather than focusing primarily on what the earlier work lacks?

We have updated this sentence to be a reflection of the previously unexplored areas in agentic production that our system tackles. We appreciate this feedback, as this is what we really intended to address with this sentence.

Line 174: I find the phrasing "we found it produced poor results" somewhat subjective and unnecessarily negative. Could the authors use more neutral, objective language and clarify the specific limitations they observed when using SAM Audio in this setting, ideally with supporting evidence or examples?

This has been updated to reflect the specific limitations of SAM Audio as it pertains to music separation and a few justifications as to why we instead opted for Demucs.

Line 329 is missing reference

Fixed, thanks

Line 453-457: The text discusses operator F1 as a function of graph complexity but Figure 5 displays MERT audio similarity?

Graph complexity is the x-axis (number of operations) and MERT audio similarity is the y-axis.

Reviewer_gYnPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets agentic music post-production, where an agent isolates a stem, applies an effect chain and mixes it back. It introduces POEMS, a toolkit of about 20 audio operators exposed through MCP, and RIME, a rule-based generator that produces edit graphs from recipes, parameter priors (fit to MixParams and SocialFX) and ordering constraints, then writes instructions at three abstraction levels (AL0 to AL2). The authors build a 15,000-triple benchmark on MusicCaps, run a 10-person listening study, evaluate three multimodal LLMs as zero-shot tool-calling agents, and fine-tune Gemma 3n on 3,000 generated examples. Zero-shot agents pick operators well when instructions are explicit but fail to recover parameters from abstract instructions. SFT helps, mostly at the explicit level.

Critical strengths
  1. The data pipeline is carefully built. Priors come from real mixing data, effect ordering is referenced, an engineer audited the recipes, and an LLM judge screens the instructions. The listening study (n=10) confirms that matched instructions score above distractors at all three levels. This is more validation than most synthetic editing datasets get, and it is why I think the benchmark will be used.
  2. Separating operator recovery from parameter recovery is the most useful result. Operator F1 stays above 0.9 for long chains at AL0, while parameter accuracy (25% tolerance) drops to 3.6-8.7% at AL2. That points to one specific bottleneck.
Critical weaknesses
  1. The audio metrics have no reference row, and the one number that could serve as one (Table 11) is already at the ceiling. Input-to-target MERT similarity is 0.972, i.e. what an agent gets by returning the input untouched; the four agents score 0.971, 0.979, 0.965 and 0.971. Failed runs are also scored as the input. Table 1 and Section 5 report FAD and KAD and say GPT-Audio-Mini has the best values, but FAD/KAD for the untouched input are never given, so as it stands the audio metrics do not show whether any agent beats doing nothing.
  2. The SFT gain is mostly where the instruction already contains the answer. At AL0 every effect, value and order is in the text; at AL2 the SFT model has the lowest edit similarity (0.465) and the lowest head-to-head win rate (36.2%). The abstract does not qualify the claim. The SFT data are also described in two different ways, as audio triples in Section 4 and as plans built without real data in Appendix D.2.
Questions for authors
  1. What are FAD, KAD and output-to-target similarity for the untouched input, and how do the results change when only completed runs are counted? What is edit similarity when the output equals the input?
  2. What audio do the 3,000 SFT triples use, and is there any clip or graph overlap with the evaluation set?
  3. 25,000 - 7,908 = 17,092, but Section 4 reports 15,335 valid clips, and the training set shows the same kind of gap (20,144 vs 20,098). I may be missing a filtering step here, but what removes the rest?
  4. At AL1 and AL2 many graphs satisfy the same instruction. How should operator F1 and parameter accuracy be read at these levels?
  5. Could LLM2Fx-Tools be run on the single-stem subset as an external baseline?
  6. Did the listening study have ethics approval, and how were participants compensated? Also, Section 4 has an unresolved reference ("as described in ??").
3 comments

Thank you for your thoughtful review; we greatly appreciate your positive feedback on the careful design of the RIME pipeline and the usefulness of the parameterization results. We respond to your questions below:

The audio metrics have no reference row…What are FAD, KAD and output-to-target similarity for the untouched input, and how do the results change when only completed runs are counted? What is edit similarity when the output equals the input?

Below are FAD/KAD including the untouched inputs. The output-target audio similarity numbers are provided in Appendix H.1 (Tables 11 and 12)

Model / baselineFAD ↓KAD ↓
Untouched input0.10910.1694
GPT Audio Mini0.02550.0157
Gemini 3 Flash0.04050.0236
Gemma 3n0.08540.0998
Gemma 3n SFT0.03280.0222

The abstract does not qualify the claim

This is a valid point. We have updated the abstract to highlight that SFT mostly improves parameterization, which is a primary failure mode of post-production agents. This holds throughout all abstraction levels and better describes the results outlined later in the paper.

The SFT data are also described in two different ways, as audio triples in Section 4 and as plans built without real data in Appendix D.2. … What audio do the 3,000 SFT triples use

Thank you for catching this; this is a typo, there is no audio here. We have updated the paper to describe these as tuples (input audio, instruction) and now better specify this distinction in Section 4.

25,000 - 7,908 = 17,092, but Section 4 reports 15,335 valid clips, and the training set shows the same kind of gap (20,144 vs 20,098)

We have updated Section 4 and the Appendix to directly address this. There were 1,757 plans which failed to produce valid output audio because Demucs failed to successfully identify the target instrument. We identify this via RMS thresholding of the source estimate from Demucs.

At AL1 and AL2 many graphs satisfy the same instruction. How should operator F1 and parameter accuracy be read at these levels?

We report parameters within variable tolerance for this reason, allowing for clear relative comparisons between models (which is the goal; the absolute of 100% may not be required, as it is not on many other benchmarks). While “many graphs satisfy the same instruction” is potentially true, some prompts are specific enough that the satisfying graph is quite specific.

This is why we include (1) the user study, and (2) several metrics to evaluate the results holistically without over-relying on any one (e.g. graph accuracy).

Could LLM2Fx-Tools be run on the single-stem subset as an external baseline?

LLM2Fx-Tools only contains 9 available tools, containing mostly EQ, Leveling and Time-based effects. POEMS contains over 20. It is difficult to get a comparable baseline given the disparity in number/kind of tools.

Did the listening study have ethics approval, and how were participants compensated?

Yes, the listening study was approved by our IRB. Participants were compensated at \15/hr,butbecausethemediancompletiontimewaslowerthananticipated,theactualratewas15/hr, but because the median completion time was lower than anticipated, the actual rate was \\25.74/hr on average.

Also, Section 4 has an unresolved reference ("as described in ??").

Fixed, thanks for pointing it out!

Hi reviewer gYnP,

We are reaching out in advance of the deadline to make sure we have answered all of your questions and to see if you have any more feedback.

We also wanted to discuss two changes we have just made to the paper.

  1. In regard to the discussion of AL1 and AL2 corresponding to potentially multiple edit graphs, we have added Section 6 (Limitations and Future Work), which expands upon how we guard against this limitation and how we envision future work can improve upon it.

  2. We have also added Appendix A, which discusses potential real-world uses of RIME and POEMS with DAW integration and feedback from musicians. We hope you will also take this into consideration in your final assessment.

We would appreciate any additional feedback on these revisions.

Thank you for your response.

The FAD and KAD values ​you provided for the original input data have addressed my primary concerns. I will maintain the score of 3.

Good work.

Reviewer_Uj66OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RIME, a rule-based framework for generating synthetic music post-production data, and POEMS, a toolkit of audio-processing operators for multimodal agents. RIME constructs editing workflows using predefined recipes, parameter priors, and constraints, and generates editing instructions at three abstraction levels. The authors generate 15,000 evaluation triplets and 3,000 training triplets to benchmark multimodal agents. A small-scale listening study validates instruction–audio correspondence, while experiments show that agents struggle with abstract instructions and complex effect chains. Supervised fine-tuning of Gemma 3n improves certain editing metrics.

Critical strengths

The paper addresses an interesting application of multimodal agents to music post-production. Integrating source separation, audio effects, and mixing into a tool-calling workflow is practically meaningful. The three instruction abstraction levels also reflect different ways musicians and engineers communicate editing intentions.

RIME offers a structured approach to generating instruction-conditioned editing data. The evaluation across different abstraction levels and editing complexities provides useful preliminary evidence of current agents' limitations.

Critical weaknesses

The framework mainly integrates existing audio-processing tools, heuristic recipes, and LLM-generated instructions, without introducing substantial innovations in machine learning methodology or agent learning. The supervised fine-tuning approach is relatively standard. Moreover, the experimental findings are largely descriptive. Performance degradation with increasingly abstract instructions or longer effect chains is intuitive, but the paper does not sufficiently investigate its underlying causes, such as audio perception, language grounding, planning, or parameter estimation. Consequently, the work appears closer to a preliminary agentic music-production system demonstration than a sufficiently developed ML methodology or application contribution for ICLR.

The baseline selection requires stronger justification. In particular, Gemma 3n is not a state-of-the-art music-specialized understanding model, potentially limiting agent performance. The Qwen-based filtering and evaluation components also require validation for music-specific perceptual judgments. Without stronger music-aware baselines, it is difficult to assess whether the reported limitations arise from the proposed task or the selected models.

The paper also lacks essential ablation studies on recipe constraints, parameter priors, instruction generation, and filtering. Furthermore, training and evaluation share predefined recipe families, making generalization to unseen workflows uncertain.

Although motivated by professional music production, the synthetic edits and limited listening study do not adequately establish practical usefulness or correspondence with real-world production decisions.

Lastly, the paper is generally clear, but its scientific narrative lacks focus. Listing seven contributions makes the work appear fragmented, combining toolkit development, dataset construction, benchmarking, and fine-tuning without establishing a central methodological contribution. The presentation emphasizes implementation and workflow descriptions more than scientific hypotheses and experimental insights. Overall, it reads more like a computer music workshop demonstration than a focused ICLR research paper.

Questions for authors
  1. Baseline selection: Why were these models selected, particularly Gemma 3n? Would stronger music-specialized audio understanding models substantially change the reported results?

  2. Ablation studies: What are the individual contributions of recipe constraints, parameter priors, and LLM-based filtering? How does RIME compare with simpler random or constrained effect-chain generation?

  3. Generalization: Can the authors demonstrate performance on unseen recipe families or real editing instructions collected from professional producers?

  4. Scientific insights: Can the authors distinguish failures in audio understanding, language grounding, planning, and parameter estimation to explain the observed degradation across abstraction levels?

4 comments
AuthorsOpenReview ↗

Author Response 1/3

We appreciate that you regard this work as interesting, practically meaningful, reflecting communication of editing intents, and offering useful evidence of current limitations.

We have several substantial disagreements with this review, however. We believe that the overall score rests on 5 issues, which we address below. To summarize:

  1. Contribution to agent learning: we do not claim a fundamental algorithmic contribution. This is an applied contribution, opening up the possibility of post-training agents for post-production, with initial baselines for this task
  2. Baseline selection: music-specific models are not suited to agent workflows. We pick models that have the core necessary multimodal and agentic capabilities
  3. Ablation studies: we believe this concern is underspecified. We have several different analyses of our results, including e.g. sensitivity analyses, multiple embeddings, several metrics, and different data splits (e.g. abstraction level). We add some more below. Beyond this, ablations on the data generating pipeline would reduce the validity of the data without a clear path to evaluating this
  4. Scientific insights: we include a large collection of experiments that we believe yield useful scientific insights for this application, giving analyses of
    1. Parameterization across accuracy thresholds (Table 2)
    2. Graph accuracy across different components (Figure 4)
    3. Abstraction level (Table 1, Figure 4, Figure 5, Table 2)
    4. Chain length (Figure 5)
    5. Trade-offs between different metrics (e.g. graph vs. audio) (Section 5 discussion)
    6. Human listener study showing perceptual separability of recipes (Section 5.1, Figure 3)
    7. Cross-prior correlations (Figure 10, Figure 11)
    8. Robustness of empirical priors under resampling (Figure 12)
  5. Generalization: We show generalization via performance on randomly sampled effect chains, providing further experiments and results for this. We argue that showing generalization to real-world production decisions is outside the scope of this work, and constitutes a separate HCI study.
AuthorsOpenReview ↗

Author Response 2/3

without introducing substantial innovations in machine learning methodology or agent learning

We do not claim advancements in agent learning. There does not exist any data to make substantial contributions to agent learning for Agentic Post-Production. The contribution of this paper is providing that data and demonstrating its validity through listener studies, downstream evaluation on multiple orchestrators, and preliminary SFT.

the paper does not sufficiently investigate its underlying causes, such as audio perception, language grounding, planning, or parameter estimation

We evaluate language grounding through abstraction levels, demonstrating that the underlying cause of most agent failure is the lack of specification in instructions. We furthermore evaluate parameter estimation and show that the primary failure mode at AL1 and AL2 is parameter estimation, where every model recovers less than 20% of parameters within a 10% accuracy threshold. We show via plan SFT that planning can improve parameter recovery, but only to an extent.

it is difficult to assess whether the reported limitations arise from the proposed task or the selected models

We are confused by this point, because this is a limitation of all machine learning benchmarks. We address this to the best of our ability by analyzing two frontier LLMs with audio capability (GPT and Gemini) as well as an open-source multimodal LLM (Gemma 3n). Could you specify how you would like to see these limitations decoupled?

The paper also lacks essential ablation studies on recipe constraints, parameter priors, instruction generation, and filtering.

It is unclear both what ablations you are asking for and what they would indicate as far as results. Are these ablations to show how different components of the data generation pipeline affect agent performance? We already have parameter-level analysis and could provide recipe-level analysis as well.

Would ablations for instruction generation/filtering mean using different methods of instruction generation and filtering? Or using different models? It is really unclear what exactly you are asking for here.

The synthetic edits and limited listening study do not adequately establish practical usefulness or correspondence with real-world production decisions.

We already include several forms of human validation within the paper. We have a:

  1. A listener study confirming instructions faithfully correspond to generated edits
  2. Auditing by a professional engineer
  3. Interfaces (see Appendix F) for users to validate parameter priors, edit chains, and instructions
  4. In addition, we ground many of the priors in human-designed FX chains (Appendix C.4)

Can you explain what specific further criteria would satisfy practical usefulness or correspondence with real-world production decisions? If, for instance, you mean testing it with producers in studios, we would argue this constitutes a new HCI contribution (and would encourage the community to pursue it).

The presentation emphasizes implementation and workflow descriptions more than scientific hypotheses and experimental insights

The main contribution of RIME is a data generation pipeline for post-production data. We highlight the workflow descriptions as this is the key component of the reproducible artifacts from this work. We include several downstream experiments (comparing agent performance by model, abstraction level, instruction length) whose results could not be fully interpreted without a holistic understanding of the data it is operating on.

Would stronger music-specialized audio understanding models substantially change the reported results?

Can you please list which music-understanding models you believe would be able to perform agentic post-production? Music-specialized audio understanding models such as MusicFlamingo are designed to describe music and are not suitable for orchestrating agentic workflows.

Ablation studies: What are the individual contributions of recipe constraints, parameter priors, and LLM-based filtering?

It’s unclear to us what you mean here. Ablating components of the data generating pipeline would result in different data distributions. Would you want to see, e.g. plots of the data distribution under ablations? Or impacts on model performance? It’s unclear that the latter has a clear interpretation, since testing the model on invalid chains would simply yield nonrepresentative results.

However, we can split the evaluation by random constrained vs. recipe-based chains, for instance (we show this below in response to other questions).

AuthorsOpenReview ↗

Author Response 3/3

How does RIME compare with simpler random or constrained effect-chain generation?

Part of the training/evaluation sets contain randomly sampled effect chains. We have added a paragraph to the data generation part of section 4 to explain this better. Additionally, we include results on just the random subset of our evaluation data below. However, we note that these random chains are often more complex and thus have a higher baseline difficulty.

Results below are presented as “Priors-Based / Random-Constrained” in all result columns.

Model$N$FAD $\downarrow$KAD $\downarrow$$\Delta \text{sim}_a \uparrow$Graph F1 $\uparrow$Params $\uparrow$
Gemini14295 / 7050.04 / 0.590.02 / 0.630.54 / 0.380.65 / 0.380.33 / 0.07
GPT14295 / 7050.03 / 0.420.02 / 0.130.65 / 0.410.83 / 0.630.45 / 0.29
Gemma14295 / 7050.09 / 0.430.11 / 0.060.61 / 0.530.86 / 0.820.38 / 0.32
Gemma SFT14295 / 7050.03 / 0.440.02 / 0.150.62 / 0.430.86 / 0.740.49 / 0.33

We have also run a new SFT experiment on 3,000 random effect plans with the identical training protocol. When evaluated on a matched random subset of 10% of the evaluation set, we found the following results. We will add a full comparison in the revision, but this may take longer than the rebuttal period.

Results below are presented as “Regular SFT / Random SFT” for Gemma 3n in all result columns.

MetricOverall (Reg./Rand.)AL 0AL 1AL 2
Completion (%) ↑99.4 / 95.499.6 / 100.099.8 / 98.898.8 / 87.4
FAD (MERT) ↓0.132 / 0.1470.132 / 0.1680.228 / 0.2050.270 / 0.275
KAD (MERT) ↓ [IGNORE, ALL SIMILAR]-0.018 / 0.005-0.101 / -0.093-0.077 / -0.098-0.048 / -0.018
Output-target similarity: MERT ↑0.971 / 0.9720.983 / 0.9790.969 / 0.9730.961 / 0.963
Output-target similarity: CLAP ↑0.874 / 0.8680.915 / 0.9010.872 / 0.8760.836 / 0.826
Edit similarity: MERT ↑0.613 / 0.5930.731 / 0.6990.640 / 0.6290.467 / 0.450
Edit similarity: CLAP ↑0.731 / 0.7050.814 / 0.7910.747 / 0.7310.633 / 0.593
Graph F1 ↑0.846 / 0.7860.929 / 0.9080.888 / 0.8670.721 / 0.582
Parameter score ↑0.467 / 0.4490.871 / 0.8700.375 / 0.3750.154 / 0.102
Parameters within 10% (%) ↑34.3 / 33.881.7 / 80.915.1 / 16.56.0 / 4.0
Parameters within 25% (%) ↑41.6 / 40.585.0 / 84.728.3 / 29.411.5 / 7.4

Can the authors demonstrate performance on unseen recipe families or real editing instructions collected from professional producers?

The random-constrained experiments above are effectively unseen, however collecting real editing instructions from professional producers is a very different task; producers are a small, rare population. We have a professional producer in the loop on the design of all of these recipes, and many aspects are sourced from existing production data too.

Can the authors distinguish failures in audio understanding, language grounding, planning, and parameter estimation

We do distinguish between parameter estimation and other failure modes precisely in Section 5 (Figure 4 and Table 2)

Thank you for the detailed experiment, which partially address my concerns about baseline selection, ablations, and generalization. However, my main concerns regarding limited methodological novelty, insufficient validation against real-world music production workflows, and the lack of deeper scientific insights remain. While I acknowledge the value of the proposed framework as an applied contribution, I am not yet fully convinced that the current experimental evidence meets the expectations of ICLR. But in recognition of the new evidence, I am raising my rating from clear rejection to weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.