acceptodds
Under review as a conference paper at ICLR 2027

MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent

Abstract

Text-to-music systems produce increasingly convincing audio, yet evaluation reveals little about whether the result matches user intent. A global text–audio relevance score can overlook the implicit intent in underspecified prompts and mask failures in specific requirements—instrumentation, structure, rhythm, or mood progression. To bridge this gap, we formulate text-to-music intent alignment as satisfying a per-request rubric of independently verifiable items covering both a request's explicit requirements and its implied musical intent. Scoring items individually makes evaluation diagnostic—by intent source and musical dimension—rather than a single opaque score. We instantiate this as MuRA-Bench, a benchmark of real-world platform requests curated by music experts. We further propose MIRA (Musical Intent Refinement Agent), a test-time agent that first grounds a request's intent into rubrics, then searches over prompt revisions for a black-box generator under a bounded budget—iteratively generating music, verifying it against the rubrics, and using this feedback to guide a trajectory-aware tree search. Experiments across open-source and commercial backends show that MIRA improves intent alignment, enabling an open-source generator to achieve performance comparable to representative commercial systems(e.g. Suno and Mureka).Project Page:https://mirareview.github.io

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-music generation, musical intent alignment, rubric-based evaluation, music generation benchmarks, agentic refinement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A3HtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames text-to-music intent alignment as satisfying a per-request rubric of atomic, audibly checkable items, split into explicit requirements (Base) and traits inferred from references and context (Completion). It contributes two things:

  • MuRA-Bench: 100 requests derived from Mureka platform seeds, with 1,566 items. An LLM drafts each rubric and one of three music experts revises it. Items are scored by MusicFlamingo audio QA using soft yes/no probabilities.
  • MIRA: a test-time agent wrapped around a black-box generator. It grounds the request with tools, builds an online rubric, and runs a shallow, budgeted tree search over prompts (B = 3 or 9 generations) guided by MusicFlamingo, with branch-specific memory.

Across ACE-Step, SongGeneration2 and YuE2-3B, MIRA improves Overall by 5.6-11.5 points over direct prompting, and gains persist under two other evaluators. The headline claim is that YuE2-3B + MIRA (82.1) reaches the highest Overall score among all evaluated systems, ahead of Suno v5.5 (81.9).

Critical strengths
  1. The problem framing is useful and well executed. Splitting a request into explicit and inferred requirements, with dimension tags, turns a single opaque relevance score into something diagnostic. The evaluator is chosen empirically: eight scoring methods are compared against 1,536 expert item ratings, and MF-AQA leads on every metric (item SRCC 0.690 vs 0.509 for the next best, CLAPScore 0.245). This gives the benchmark a credible footing that most text-to-music relevance metrics lack.

  2. The search gains are not only a selection effect. In the equal-budget comparison, Best-of-9 also selects with MusicFlamingo, yet MIRA still improves Overall by 3.6 points on ACE-Step and 2.4 on YuE2-3B. The ablation in Table 3 attributes most of the gain to tool grounding (Completion 0.719 → 0.748 on ACE-Step) and to adaptive search (0.748 → 0.781). Gains also hold when outputs are rescored by MOSS-Music and Qwen3-Omni (Table 8).

Critical weaknesses
  1. The search optimizes the same judge that produces the headline numbers, and the headline claim does not survive a change of judge. MIRA searches with MusicFlamingo AQA and selects with it, and Table 2 evaluates with it. The online rubric R(x, E) and the gold rubric R*(x) are different, but both are decomposed by an LLM from the same request, and the paper reports neither their overlap nor how much the experts changed the drafts.

The claim that YuE2-3B + MIRA beats every commercial system holds only under MusicFlamingo, and only by 0.2 points with no variance reported. Under the other evaluators it does not hold:

  • MOSS-Music: 78.4, which ties MiniMax 3.0 but is below Suno v5, v5.5 and v6 (79.3, 80.0, 79.7).
  • Qwen3-Omni: 70.6, against 72.5 for MiniMax 3.0 and 75.0 for Suno v5.5.

The human study does not test YuE2 + MIRA at all, and ACE-Step + MIRA is rated below Suno v5.5 on all three axes (e.g. 4.04 vs 4.26 overall). "Comparable to Mureka and MiniMax" is defensible; "highest among evaluated systems" is not. The cross-evaluator check only partly addresses circularity: all three judges use the same AQA format, and the two alternatives agree less with experts (Table 1).

  1. The commercial comparison and the statistics are both thin. The two sides are not run under the same conditions:
  • MIRA gets nine generations, tool grounding and verifier selection.
  • Commercial systems get a single direct-prompt call.
  • MIRA is presented as backend-agnostic, yet it is not applied to any commercial system, not even as Best-of-N.

On the statistics:

  • No table reports seeds or variance, even though the paper stresses that generation is stochastic and the Direct baseline is a single sample.
  • The memory component adds 0.010 on ACE-Step and 0.005 on YuE2 over 20 requests, which is within plausible noise.
  • The selection of that 20-request subset is not described, and it looks easier than the full set (Direct Overall 0.759 vs 0.676 on ACE-Step).
  • The human comparison has no paired test and one rating per clip.
  1. The benchmark's own validity and side effects are not measured. Each request was revised by a single expert, so there is no agreement estimate for the rubrics themselves. This matters most for Completion items, which are interpretive; some items, such as "open-plains biome atmosphere", are hard to judge atomically by ear. The prompts come from Mureka's user distribution, and the ranking of commercial systems changes with the evaluator: Mureka V9.5 is above MiniMax 3.0 under MusicFlamingo and below it under both alternatives. Audio quality is declared out of scope (C.4). Without a quality metric (FAD, Audiobox Aesthetics, SongEval), there is no evidence that steering prompts toward rubric wording leaves musicality intact.
Questions for authors
  1. Rubric overlap. How much do the online rubric R(x, E) and the gold rubric R*(x) overlap, measured by semantic matching? Which LLM drafted the gold rubrics, and what fraction of items did experts add, delete or rewrite?

  2. Decoupled judge. Can you run search with one judge and evaluation with another, for example Qwen3-Omni for search and MusicFlamingo for evaluation?

  3. Variance and significance. What are the means and standard deviations of Table 2 over at least three seeds, and is the YuE2 + MIRA vs Suno v5.5 gap significant?

  4. Commercial backends. How do commercial systems score with Best-of-9 or MIRA applied, or with the tool-augmented prompt alone? For the 32 vocal requests, were the same lyrics given to all systems?

  5. Missing details. How was the 20-request subset chosen? What is the average size m of the online rubric, and what does each request cost in verifier queries (roughly m × B), planner calls and wall-clock time? For the human study, were the "intent criteria" shown to raters the gold rubric?

  6. Ablations and side effects. SongGeneration2 uses stochastic sampling while the other backends use fixed seeds, and it is excluded from the equal-budget comparison. How much of its +11.5 gain comes from sampling diversity? How sensitive are results to η = 0.60 and ε = 0.03? How do audio-quality metrics change under MIRA?

  7. Minor. "Dim-Macro" and "D-Macro" are both used. Figure 3 uses "inferred" in the caption and "completed" in panel (c); using one term throughout would help. The contribution list has a redundant phrase ("three generation generators").

9 comments

We sincerely thank the reviewer for the careful review and specific suggestions on evaluation, statistical evidence, and reproducibility.

W1: The search optimizes the same judge that produces the headline numbers, and the headline claim does not survive a change of judge.

We thank the reviewer for raising this concern. We separately address the relationship between judges, rubric construction, and commercial-system rankings, and add experiments using different models for search and final evaluation.

First, MIRA's improvements to the generation backends persist when the evaluator changes. The existing cross-evaluator results are shown below; all scores are Overall:

EvaluatorACE-Step DirectACE-Step + MIRAYuE2 DirectYuE2 + MIRASuno v5.5
MusicFlamingo67.6476.3176.4782.0681.88
MOSS-Music64.7271.8573.8678.3879.97
Qwen3-Omni59.567.1266.7670.5874.99

All three evaluators observe improvements after applying MIRA to ACE-Step and YuE2. We use a common AQA format to make each evaluation target explicit and enable direct comparison of judgments across models. The expert-agreement experiment in Table 1 supports this choice. We therefore use MusicFlamingo as the main evaluator while retaining results from the other models to show performance under different judges.

Second, we add a decoupled experiment using Qwen3-Omni for search and MusicFlamingo for final evaluation. On the same 20 ACE-Step requests, Qwen3-Omni provides search feedback and selects candidates, while MusicFlamingo evaluates the selected outputs against the expert gold rubrics.

Experiment sourceSearch judgeBudgetOverallBaseCompletionD-Macro
New experimentQwen3-Omni376.82878.896573.582575.7265
New experimentQwen3-Omni979.326481.639175.464378.242
Existing reference with the same configurationMusicFlamingo981.074184.802976.269979.4977

When different models perform search and final evaluation, increasing the budget from 3 to 9 improves all four final-evaluation metrics, with Overall increasing by 2.4984 points; at the same budget, MusicFlamingo-guided search achieves a higher score.

Regarding rubric sources and overlap, the gold rubrics were drafted by GPT-5.6-sol and revised by experts who received common training, while the online rubrics are generated by Kimi 2.6. The gold drafts contained 1,078 Base and 335 Completion items, and the expert-revised versions contained 996 and 550 items, respectively. Additional model-assisted semantic matching across 100 requests shows that the online rubrics fully cover 83.13% of gold Base items. For Completion, full coverage is 4.04% and partial coverage is 63.33%.

Finally, we will revise the wording of the commercial-system ranking. Across the original experiment and two additional seeds, YuE2 + MIRA achieves Overall 81.7693±0.2839, with no statistically significant difference from Suno v5.5's 81.8779. We will therefore replace "outperforms all commercial systems" with "reaches the intent-alignment performance of the latest commercial systems." The existing human study covers ACE-Step, with overall ratings of 4.04 for MIRA and 3.84 for Direct, compared with 4.26 for Suno v5.5. Given the time required for human evaluation, we will include a corresponding YuE2 human study in subsequent work.

W2: The commercial comparison and the statistics are both thin.

We thank the reviewer for noting that the commercial comparison conditions and statistical evidence need further explanation. We have added commercial-backend controls on Suno v6, experiments with two additional generation seeds, memory ablations, and re-rating by an additional expert.

Commercial-backend controls. We have applied MIRA to Suno v6 and compared Direct, Best-of-9, and MIRA on the same 20 requests:

SettingOverallBaseCompletionD-Macro
Direct (historical audio, reaggregated on the same subset)85.314191.515777.950185.5241
Best-of-9 (new)86.340492.400679.653886.0720
MIRA B=3 (new)86.733091.454780.985686.2085
MIRA B=9 (new)87.578692.010082.070687.1559

MIRA B=9 improves Overall by 2.2645 points over Direct. Relative to Best-of-9, it improves Overall and Completion by 1.24 and 2.42 points, respectively. These results add empirical evidence for MIRA on a commercial black-box backend.

Repeated runs and variance. The original experiment and two additional runs are shown below. Each condition covers 100 requests, and scores are Overall under MusicFlamingo evaluation. Means and sample standard deviations are computed over the two additional runs only; the original experiment is listed separately as a reference.

BackendSettingOriginal experimentAdditional seed 1Additional seed 2Mean±sample SD over two additional runs
ACE-StepDirect67.635667.762567.657167.7098±0.0745
ACE-StepMIRA B=374.190174.406274.344774.3754±0.0435
ACE-StepMIRA B=976.314476.078875.740875.9098±0.2390
YuE2Direct76.468875.817677.298176.5579±1.0468
YuE2MIRA B=380.482179.078379.824779.4515±0.5277
YuE2MIRA B=982.061381.752381.494381.6233±0.1824

Memory ablations. Each condition uses the same 20 requests with nine candidates per request. Full scores and means and sample standard deviations across the two additional runs are shown below:

BackendExperimentWithout memoryWith memoryWith minus without memory
ACE-StepOriginal experiment82.618383.6067+0.9885
ACE-StepAdditional seed 180.363382.1374+1.7741
ACE-StepAdditional seed 282.032382.9497+0.9174
ACE-StepMean±SD over two additional runs81.1978±1.180282.5436±0.5744+1.3458±0.6058
YuE2Original experiment86.675787.1537+0.4779
YuE2Additional seed 185.183585.6295+0.4460
YuE2Additional seed 286.640386.9820+0.3418
YuE2Mean±SD over two additional runs85.9119±1.030186.3057±0.9564+0.3939±0.0737

Differences are computed from unrounded scores. Both backends obtain positive memory gains in both additional runs.

Subset selection. The 20 requests were randomly sampled and then fixed; all controls share the same request list.

Human ratings. We added an expert to re-rate 12 requests, 48 audio clips, and 780 rubric items from the original human-evaluation set. Item SRCC, Kendall τb, and Pearson correlation with the original expert ratings are 0.4311, 0.3322, and 0.4139, respectively. Agreement between the evaluators and the combined ratings is reported in by1a-W3.

W3: The benchmark's own validity and side effects are not measured.

We thank the reviewer for the suggestions. We clarify the gold-rubric revision process and the basis for selecting the main evaluator, and report additional audio-quality results.

Gold-rubric construction. Before revision, the three experts received common training and followed the same guidelines: retain explicit user constraints, check contextual support for Completion requirements, ensure that every item is audibly judgeable and corresponds to a single requirement, and remove or merge redundant content. Experts independently revised their assigned requests following these guidelines. This process aimed to align revision standards. The original study did not assign requests for repeated revision, so cross-expert agreement for the rubrics themselves was not computed. For wording such as "open-plains biome atmosphere," the judgment concerns the fit between the musical atmosphere and the scene, rather than identifying a literal geographic environment from audio. We will clarify this judgment target.

Evaluator selection and commercial rankings. Yes, different evaluators produce different commercial-system rankings. Our main evaluator was selected based on agreement with expert judgments rather than the ranking of any particular commercial system:

EvaluatorMureka V9.5MiniMax 3.0Item-level Spearman with human ratingsClip-level Pearson with human ratings
MusicFlamingo81.233277.04950.68960.8169
MOSS-Music76.517278.40490.50210.5912
Qwen3-Omni70.838672.52990.50940.6318

Commercial-system scores come from the full benchmark, while human-rating correlations come from the validation set in Table 1. MusicFlamingo achieves the highest agreement in the available human validation, so we use it as the main evaluator; its results have more direct support from human judgments. Results from the two alternative evaluators are retained to show how rankings depend on the evaluator.

Audio quality. We have additionally scored the historically selected audio using Audiobox Aesthetics and SongEval, without reselecting candidates based on quality metrics:

BackendMetricDirectMIRA B=9Difference
ACE-StepProduction quality PQ7.97537.9118−0.0635
ACE-StepContent enjoyment CE7.10796.9563−0.1516
ACE-StepVocal musicality3.96764.01710.0495
YuE2Production quality PQ8.16778.1244−0.0433
YuE2Content enjoyment CE7.45767.4248−0.0327
YuE2Vocal musicality4.03383.9912−0.0426
SongGeneration2Production quality PQ8.19658.1389−0.0576
SongGeneration2Content enjoyment CE7.3157.1954−0.1196
SongGeneration2Vocal musicality3.89353.8609−0.0326

PQ/CE are from Audiobox Aesthetics, while vocal musicality is assessed by SongEval only for audio containing vocals. Differences are B=9 minus Direct, and metrics retain their original scales. Under Audiobox Aesthetics and SongEval evaluation, some production-quality and content-enjoyment metrics decrease slightly, while vocal musicality does not show a clear direction of change.

Reviewer_U4dxOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes using rubrics to improve the prompt alignment performance of text-to-music generation systems. They show that rubrics can be used to both evaluate the quality of text-to-music systems (through MuraBench) and to improve their performance at test time through agentic refinement (via the proposed MIRA).

Critical strengths

This work is quite strong overall. Strengths include:

  • Simple yet timely and effective method. Rubrics are an emerging technique for evaluation across AI domains. However, they have received little attention in the music generation literature. In music gen, they seem particularly useful for addressing two key issues with evaluation: subjectivity and opaqueness of single-score metrics.
  • Proposed MuRA-Bench dataset appears to be a useful and principled evaluation for measuring the prompt-audio alignment of text-to-music generation systems
  • Proposed MIRA method shows strong performance across multiple music AI models
Critical weaknesses

The weaknesses here are not too notable, and some are likely easily rectified with additional experiments:

  • Baselines / ablations in Table 2. The proposed MIRA method has numerous bells and whistles. The results would be much improved by a proper ablation of each component in Table 2. If a systematic set of ablations is intractable, at least a holistic comparison to much simpler strategies, e.g., best-of-B-candidates using autogenerated rubrics alone or MusicFlamingo yes-no logit score for "Does this audio match this prompt"?
  • Additional verification insights / baselines. In Table 1, it would be useful to break down verification performance by rubric item type (style, instrument, mood, etc.). Easy to imagine certain models are better at certain types than others, and that performance could be improved by picking the best verifier for the rubric item type. Additionally, would be useful to see performance of other models like Gemini 3.8 Flash or Gemini 3.8 Flash, Qwen3-Omni-30b-A3B-Captioner (better at music understanding tasks than the main qwen3-omni-30b checkpoint in my experience)
  • Rubrics are not novel. They have been explored in several other fields. Still, their application within music is fairly novel, and the proposed MIRA method for leveraging them for test-time refinement offers a new perspective.
Questions for authors
  • In Table 1, are the 1,536 expert ratings for rubric items part of the MuraBench dataset? Will they be released upon publication? These verified audio/rubric/{yes,no} pairs could be particularly useful for further research into rubric verification
  • One concern with the proposed method is that it relies on very detailed prompts from users, which may be an unrealistic assumption. How many rubric items can we reliably extract from typical user prompts? The prompts in the MuraBench dataset seem to be unusually detailed - more so than the prompts I'm used to seeing from users of commercial music AI tools. I might suggest running experiments by artificially restricting the level of detail in prompts and measuring the efficacy of an approach like MIRA.
  • Do the proposed rubrics work as post-training rewards as well as test-time scaling? Likely beyond the scope of the work here, but would be a really interesting direction for future work.
4 comments

We sincerely thank the reviewer for the thoughtful assessment of our work and the constructive suggestions on baselines, verification, and future applications.

W1: Baselines / ablations in Table 2.

We thank the reviewer for the suggestion. The proposed strategy of "selecting the best of B candidates using automatically generated rubrics" corresponds to the Best-of-9 baseline in Table 3 of the original paper. This method independently generates nine candidates from the original, unmodified request, scores them item by item with MusicFlamingo using an online rubric automatically constructed from that request, and selects the best candidate. Expert gold rubrics are used only for final evaluation.

Table 3 further adds tool grounding, feedback-driven search, and branch memory step by step. We also add a tool grounding + nine independent prompt rewrites + verifier reranking control: nine distinct prompts are generated upfront, without candidate-audio feedback or search memory, and the same verifier scores and selects the best candidate.

All conditions below use the same 20 requests with nine candidates per request, and are evaluated against the same expert gold rubrics. Scores are on a 0–100 scale. Apart from the independent-rewriting conditions marked "new," all results are from the original paper.

ACE-Step

SettingOverallBaseComp.D-Macro
Original request + Best-of-980.087.471.979.5
Tools + Best-of-981.587.174.880.7
Tools + independent rewrites (new)81.486.775.279.8
Tools + feedback-driven search, no memory82.686.278.181.4
Full MIRA83.688.078.282.6

YuE2

SettingOverallBaseComp.D-Macro
Original request + Best-of-984.891.077.585.0
Tools + Best-of-986.091.180.085.2
Tools + independent rewrites (new)85.690.779.785.0
Tools + feedback-driven search, no memory86.791.380.886.3
Full MIRA87.291.182.186.8

Compared with the new tool-grounded independent-rewriting control, Full MIRA scores 2.3 and 1.5 points higher in Overall on ACE-Step and YuE2, respectively, and is higher on all four metrics. In the original component ablation, adding feedback-driven search on top of tool grounding and then adding branch memory progressively improves Overall for both backends. We will clarify these baseline settings in the main text and incorporate the new independent-rewriting control into Figure 4.

W2: Additional verification insights / baselines.

We thank the reviewer for the suggestion. We have added verification-performance analyses across the seven musical dimensions and Base/Completion, along with Gemini 3.8 Flash and Qwen3-Omni-30B-A3B-Captioner comparisons.

We use the expert annotations from Table 1, covering 25 requests and 100 audio clips. All methods below are compared on the same 1,525 valid audio–rubric pairs, using Spearman correlation with expert ratings.

Verification performance by dimension. Results for the original eight evaluators are:

GroupItemsCLAPMF-PromptingMF-CascadeMF-AQAQwen2.5 AQAAnyAudio AQAMOSS AQAQwen3 AQA
Overall1,5250.2440.5510.4370.6890.4670.4620.5020.511
Style1190.3530.5380.4810.5940.4960.5380.4780.61
Instrumentation/vocals2480.2270.7140.560.7710.5650.5560.5620.616
Mood2400.1240.4980.3560.6370.4390.3830.4840.454
Rhythm2890.320.5390.4040.7410.4340.4760.5280.503
Harmony/melody890.3260.520.4610.6930.4680.4340.4670.497
Structure/energy1600.1690.4890.2290.640.4060.450.4530.455
Production/texture3800.2310.5060.4740.6650.4460.4120.4810.522
Base9910.2360.5450.4330.6670.4740.4680.4680.505
Completion5340.2330.5620.4370.7280.4420.440.5680.515

Qwen3 AQA uses Qwen3-Omni-30B-A3B-Instruct. MF-AQA achieves the highest correlation in six of the seven dimensions; Qwen3-Instruct achieves the highest correlation for style. This shows that aggregate metrics can obscure differences across musical attributes, and supports the reviewer's suggestion to select verifiers according to rubric-item type.

Additional evaluators. The Gemini 3.8 Flash interface used in this experiment does not provide the yes/no logits required by our AQA implementation; Qwen3-Omni Captioner natively accepts only audio and generates descriptions, rather than accepting rubric questions. We therefore use the following two scoring approaches and set up MusicFlamingo controls under the same respective protocols:

Direct scoring: The full audio and rubric are provided to the audio model, which directly assigns a score of 1–5. This is essentially the same as the Prompting setting.

Caption-then-score: The audio model generates descriptions of consecutive segments covering the full audio. All descriptions are then provided in temporal order to Kimi K2.6, which assigns a score of 1–5 based on the rubric. This is essentially the same as the Cascade setting.

Scoring protocolAudio modelOverallBaseCompletion
Direct scoringGemini 3.8 Flash0.3560.3320.384
Captions scored by KimiQwen3-Omni Captioner0.4660.4380.515

W3: Rubrics are not novel.

We thank the reviewer for recognizing this work, especially the value of rubric-based evaluation in music and the perspective offered by MIRA's test-time iterative refinement.

Q1: In Table 1, are the 1,536 expert ratings for rubric items part of the MuraBench dataset? Will they be released upon publication? These verified audio/rubric/{yes,no} pairs could be particularly useful for further research into rubric verification

We thank the reviewer for the suggestion. Yes, the 1,536 expert ratings in Table 1 are part of MuRA-Bench. Upon publication, we will release the corresponding audio, rubric items, and expert annotations together.

Q2: One concern with the proposed method is that it relies on very detailed prompts from users, which may be an unrealistic assumption. How many rubric items can we reliably extract from typical user prompts? The prompts in the MuraBench dataset seem to be unusually detailed - more so than the prompts I'm used to seeing from users of commercial music AI tools. I might suggest running experiments by artificially restricting the level of detail in prompts and measuring the efficacy of an approach like MIRA.

We thank the reviewer for pointing this out. The relatively detailed requests in MuRA-Bench mainly result from screening during benchmark construction. We excluded requests that were overly simple, vague, and lacked judgeable musical requirements, retaining samples that provide clear evaluation criteria to support requirement-level diagnosis of generated outputs.

This choice was motivated by the evaluability of intent alignment. For example, for "generate a nice piece of music," many styles, instruments, and arrangements could be reasonable, making it difficult to establish reliable criteria for whether a piece of audio satisfies the user's intent. Specifying those attributes ourselves would risk evaluating whether a system follows preferences introduced by the evaluator. Requests with concrete musical requirements or traceable reference information are better suited to constructing audibly checkable rubrics.

Q3: Do the proposed rubrics work as post-training rewards as well as test-time scaling? Likely beyond the scope of the work here, but would be a really interesting direction for future work.

We thank the reviewer for suggesting this direction, which we also see as naturally connected to MIRA's objectives. Rubric-level scores identify which musical requirements a generated output satisfies well and which still need improvement, offering the potential for more specific training signals. For example, individual requirement-satisfaction scores could be combined into a reward, or rubric scores for candidate audio generated from the same request could be used to construct preference pairs for generator post-training.

We are particularly interested in whether models can learn improvements that currently require multiple rounds of search through this feedback, reducing candidate sampling and feedback rounds at inference time. The candidates and evaluation records accumulated during MIRA search could therefore also provide useful data connecting test-time optimization with post-training.

Thanks for the response. I appreciate the additional information and experiments, which further improve the work. I will maintain my positive score.

Reviewer_by1aOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper targets a gap in text-to-music evaluation: global text–audio relevance scores do not reveal which specific requirements of a request (instrumentation, structure, mood progression, implied style of a referenced artist) are met. Its core definition formulates text-to-music intent alignment as prompt-specific musical rubric satisfaction: each request is paired with a rubric of binary, independently audible items, split into "base" items stated in the prompt and "completion" items inferred from references and context. This is instantiated as MuRA-Bench, 100 requests derived from anonymized intent seeds collected on the Mureka platform, with LLM-drafted rubrics revised by three music experts, scored by asking MusicFlamingo yes/no questions and using the soft yes-probability. The second contribution, MIRA, is a test-time agent that builds its own online rubric via tool lookups, then runs a budgeted tree search over prompt rewrites for a black-box generator, using MusicFlamingo item scores as feedback and a branch-specific memory of improvements/regressions. Validation consists of an evaluator-agreement study against expert ratings (Tab. 1), benchmark results on three open-source backends plus ten commercial systems (Tab. 2), a blinded 400-clip human study on one backend (Fig. 3), an equal-budget comparison against Best-of-9 (Fig. 4), a component ablation on 20 requests (Tab. 3), and cross-evaluator scoring with two other audio-language models (Tab. 8).

Critical strengths
  1. MuRA-Bench provides a useful diagnostic benchmark for music intent alignment. Its expert-revised, platform-derived requests, base/completion split, and dimension-level evaluation help identify specific failures beyond a single relevance score. The agreement study also shows that MF-AQA outperforms seven alternative graders, including CLAPScore and a strong general-purpose omni model (Table 1). Together, these features make the benchmark a valuable resource beyond this paper.

  2. MIRA demonstrates consistent improvements across generators and evaluators. Its backend-agnostic, training-free, three-stage workflow operates through prompts to black-box generators and improves Overall, D-Macro, Base, and Completion across all three backends, both budgets, and all three evaluators (Tables 2 and 8). It also outperforms budget-matched Best-of-9 (Figure 4). Although the gain varies by evaluator, this consistency supports the method’s broader usefulness and reduces concern that it merely exploits the evaluator used for selection.

Critical weaknesses

1. MIRA is a well-designed three-stage workflow, but its framing as an agent and the added value of its tool and memory components need stronger support.

Anthropic's "Building effective agents" (2024; https://www.anthropic.com/research/building-effective-agents) distinguishes workflows, "systems where LLMs and tools are orchestrated through predefined code paths", from agents, "systems where LLMs dynamically direct their own processes and tool usage", and recommends agents for "open-ended problems where it's difficult or impossible to predict the required number of steps, and where you can't hardcode a fixed path."

The task here is not of that kind: for each request, the objective (a rubric fixed before search), the action space (rewriting a single text prompt), the number of steps (a fixed generation budget), and the evaluation criterion (MusicFlamingo item scores) are all specified in advance, and MIRA's stages correspond to the article's predefined workflow patterns (routing, prompt chaining, parallelization, and an evaluator-optimizer loop), followed by reranking by argmax over verifier scores. The one decision the Agent makes per request is which tools to call; beyond that, the process is fixed: tool grounding is performed once before candidate generation, and the tools are not revisited for requirements that keep failing; the search schedule is set by the budget and the Agent does not decide when to stop or which sub-goal to pursue next, so every query follows the same budget schedule.

  • Agent framing: MIRA uses a predefined sequence of rubric construction, prompt search, and verifier-based selection under a fixed generation budget. Although the LLM selects tools, it does not control stopping, revisit tools in response to persistent failures, or dynamically choose subsequent sub-goals. This more closely resembles the workflow patterns than an agent that directs its own process. The paper should clarify what the agent framing adds beyond an evaluator-optimizer workflow.

  • Tool contribution: Tools + Best-of-9 improves Completion scores (Table 3), but also changes the rubric used for selection. Without a control where the same planner rewrites requests without tools, these gains cannot be attributed specifically to retrieved evidence. Tool-call statistics and representative traces would also help establish whether routing is appropriate for each request.

  • Memory contribution: Memory is reset between requests and covers at most nine candidates, making it a within-request feedback history rather than accumulated experience. Its reported Overall gains are modest—1.0 and 0.5 points on ACE-Step and YuE2—and measured on only 20 requests without uncertainty estimates (Table 3). Moreover, B = 3 already captures 50–76% of the B = 9 gain despite offering little history (Table 2).

Overall, a budget-matched comparison against tool-free LLM rewriting plus verifier reranking, together with repeated-run memory ablations, is needed to establish whether the additional components justify their complexity.

2. Rubric validity in real-world use

In a real-world scenario (no golden rubric for every user query), MIRA must automatically generate a rubric for each new user query and optimize the generated music against that rubric, without access to expert revision. The validity of these online rubrics is therefore critical: an incomplete or inaccurate rubric could guide optimization toward higher verifier scores while missing the user’s actual intent. However, the paper does not measure the gap between automatically generated rubrics and expert-revised gold rubrics; Table 1 validates grading on gold items only. The authors should evaluate this gap on unseen queries, examining omitted requirements, unsupported additions, and changes in meaning, and compare retrieval-free online-rubric optimization with gold-rubric optimization under the same budget. This would clarify whether MIRA’s optimization remains effective when the rubric itself must be generated automatically.

3. Human evidence is thin relative to its role: single rater per item, no inter-rater agreement, no significance tests, and ablations on 20 requests without variance.

Tab. 1 uses one expert per rubric item; Fig. 3 uses one expert per clip, and its CIs explicitly do not capture rater disagreement; the 20-request ablation and equal-budget results come from a single run, with no repeated runs or variance reported, so the roughly one-point contributions of search and memory (Tab. 3) cannot be distinguished from noise. This is consequential because the method's novelty lies precisely in the search and memory components, and the benchmark's credibility rests on human agreement that is currently unquantified.

Questions for authors
  1. If MIRA generates N candidates from varied prompts and then uses the verifier as a reranker to select the sample best aligned with the query and rubric, in what sense is it an agent rather than a generate-and-rerank workflow?

  2. The equal-budget comparison (Fig. 4) contrasts MIRA only with Best-of-9 on the unchanged request, which differs from MIRA in both prompt diversity and feedback. Could you add a control that generates nine independent, diverse LLM rewrites of the request (with the same tool grounding but no verifier feedback, search, or memory) and reranks them with the same verifier? If this control closes most of the gap to full MIRA, the gains come from prompt diversity plus reranking rather than from agentic behavior, and the framing should be revised accordingly.

  3. How were the gold rubrics constructed in practice? The paper states that "an instruction-following LLM" drafts the base and completion requirements, but neither the model nor the drafting prompt is reported. Please specify (i) which LLM(s) drafted the rubrics and with what prompt; (ii) one or two before/after examples of an LLM draft vs. the expert-revised rubric (Tab. 5 shows only the final version); and (iii) revision statistics, i.e., the fraction of drafted items kept, edited, merged, removed, or newly added by experts, and the number of seeds marked unusable. If the same model (Kimi 2.6) also constructs MIRA's online rubric, how much do online and gold rubrics overlap before and after expert revision? Since the three experts revised disjoint subsets, is there any measure of consistency across experts?

  4. Can you report inter-rater agreement (e.g., have a second expert re-rate a sample of the 1,536 item ratings and of the 400 clip ratings)? Without a human–human ceiling, the 0.69 item-level SRCC in Tab. 1 is hard to interpret.

  5. On tool calling: (i) what backs the three tools (music_reference_lookup, scene_to_music_lookup, track_audio_evidence)? (ii) How often is each tool called across the 100 requests, how often does the evidence gate admit its output, and do gains differ between reference-based and explicitly specified requests? (iii) Could you add an "LLM rewrite without tools + Best-of-9" control to Tab. 3 and show a few example traces? These determine reproducibility, whether tools are invoked according to the query, and whether the Comp. gains come from external evidence or from the planner LLM's own knowledge.

  6. MuRA-Bench is built from Mureka platform data "with platform permission", and Mureka V9/V9.5 are evaluated as commercial baselines (Tab. 2). Please state whether any author is affiliated with Mureka and whether the seed requests could overlap with Mureka's training or tuning data.

14 comments

We sincerely thank the reviewer for carefully reading our paper and for the detailed suggestions on MIRA's design, evaluation, and reproducibility.

W1: MIRA is a well-designed three-stage workflow, but its framing as an agent and the added value of its tool and memory components need stronger support.

We thank the reviewer for these suggestions. We clarify MIRA's agentic decision-making mechanism and add independent rewriting controls, tool-use analysis, and repeated-seed memory ablations.

Agent framing

MIRA's language-model decisions include both tool selection and constructing subsequent prompts from requirement-level feedback on actual generated audio and the corresponding branch history. Verification results influence both which branches are expanded and how subsequent candidates are constructed. Subsequent generation actions therefore depend on observations from prior execution. This feedback-driven interaction follows the idea in ReAct of using execution observations to guide subsequent reasoning and actions (ReAct, 2023).

A fixed rubric provides a stable search objective, and a fixed budget supports resource-matched comparisons; specific revisions vary with audio feedback. The Anthropic article cited by the reviewer also identifies a maximum number of iterations as a possible stopping condition for agents (Anthropic, 2024). MIRA organizes requirement-level feedback into branch search, preserving different revision directions and their histories for further exploration. Final reranking selects the output; the verifier has already influenced candidate construction throughout the preceding process.

Tool contribution and independent rewriting controls

We have added tool-free and tool-assisted independent rewriting baselines using the same planner. On the same 20 requests, the tool-free baselines achieve Overall scores of 80.5 on ACE-Step and 85.3 on YuE2; the tool-assisted baselines achieve 81.4 and 85.6, respectively. Full MIRA in the original paper achieves 83.6 and 87.2, respectively. The comparison across all four metrics and its analysis are provided in Q2.

We also examine tool traces for all 100 requests: 69 requests invoke tools, and 63 ultimately receive additional information. Tool-use details are provided in Q5. We will incorporate the new independent rewriting baselines into the revised paper.

Memory contribution

Two additional generation seeds compare memory-enabled and memory-disabled search under the same nine-candidate budget. Results are shown alongside the original experiments below. Each condition covers 20 requests; scores are Overall.

BackendExperimentWithout memoryWith memoryWith minus without memory
ACE-StepOriginal experiment82.618383.6067+0.9885
ACE-StepAdditional seed 180.363382.1374+1.7741
ACE-StepAdditional seed 282.032382.9497+0.9174
YuE2Original experiment86.675787.1537+0.4779
YuE2Additional seed 185.183585.6295+0.4460
YuE2Additional seed 286.640386.982+0.3418

Across the two additional runs, the mean memory gain and sample standard deviation are 1.3458±0.6058 for ACE-Step and 0.3939±0.0737 for YuE2.

W2: Rubric validity in real-world use

We thank the reviewer for highlighting the importance of online rubric validity. We have added a rubric substitution experiment and a semantic matching analysis between online and expert gold rubrics to examine how rubric choice affects optimization and which expert requirements are covered by automatically constructed rubrics.

Rubric substitution experiment

We use ACE-Step on the same existing 20-request ablation subset, fixing the initial prompts, root audio, lyrics, generation seeds, tree search, and verifier, while changing only the rubric used to guide search. The three conditions use tool-assisted online rubrics, online rubrics constructed from the original request alone, and expert gold rubrics. Each optimization condition uses nine candidates per request, and all selected outputs are evaluated against the same 309 expert rubric items.

Experiment sourceRubric used for searchSearch budgetOverallBaseCompletionD-Macro
Reused shared starting pointRoot audio176.640978.917973.426375.4699
Existing referenceTool-assisted online rubric981.074184.802976.269979.4977
New experimentQuery-only online rubric980.615884.301275.753979.2222
New experimentExpert gold rubric981.920383.476379.559480.7927

Neither automatic-rubric condition accesses the expert gold rubric during search. Their Overall scores under gold-rubric evaluation improve by 4.43 and 3.97 points over the shared root audio. These results show that automatically constructed online rubrics provide useful optimization guidance and improve intent alignment as measured by expert criteria. The condition that directly uses the expert gold rubric during search serves as an oracle reference, achieving an Overall score of 81.9203.

Semantic coverage of expert rubrics

We additionally use GPT-6.1-sol for model-assisted semantic matching between the actual online rubrics saved for all 100 requests and the expert gold rubrics. Full coverage requires the complete musical meaning and important qualifiers of an item to be expressed; partial coverage indicates that some of its musical features are captured.

Item groupOnline itemsGold itemsGold items fully covered by online rubricsGold items partially covered by online rubrics
Base1,15199683.13%9.64%
Completion2605704.04%63.33%

These results show that automatically constructed online rubrics preserve most explicit requirements and capture musical features associated with a majority of expert completion requirements. Together with the rubric substitution experiment, where both automatic-rubric conditions improve scores under expert evaluation, these findings support the usefulness of online rubrics for guiding musical intent refinement.

W3: Human evidence is thin relative to its role: single rater per item, no inter-rater agreement, no significance tests, and ablations on 20 requests without variance.

We thank the reviewer for the suggestion. We have added an evaluation by an additional rater and memory ablations with two additional generation seeds.

(i) Additional-rater evaluation and evaluator agreement. An additional expert re-rated 48 audio clips and 780 rubric items from the original human-evaluation audio set. Agreement with the original expert ratings is 0.4311 in Item SRCC, 0.3322 in Kendall τb, and 0.4139 in Pearson correlation.

For the same items, we retain both the original and additional ratings without averaging them first, yielding 1,560 rating records. Agreement between each evaluator and the combined ratings is as follows:

EvaluatorItem SRCCItem τbClip LCCClip SRCCClip τbPair Acc.
MusicFlamingo AQA0.5880.4460.7180.7130.52674.65%
Qwen3-Omni0.40.290.6270.5980.43470.42%
Qwen2.5-Omni0.3590.2590.4390.4730.33566.90%
MOSS-Music0.4120.3030.3530.390.27166.90%
AnyAudio-Judge0.3260.2340.4090.3790.26866.20%
MF-Prompting0.430.3560.6550.6180.45473.94%
MF-Cascade0.3370.2680.4710.5110.34959.15%
CLAPScore0.2420.1730.40.4530.356.34%

MusicFlamingo AQA remains the highest on all six agreement metrics under the combined ratings.

(ii) Repeated runs and memory ablations. Each condition uses the same 20 requests with nine candidates per request; scores are Overall. Means and sample standard deviations are computed over the two additional runs only, with the original experiment listed separately.

BackendExperimentWithout memoryWith memoryWith minus without memory
ACE-StepOriginal experiment82.618383.6067+0.9885
ACE-StepAdditional seed 180.363382.1374+1.7741
ACE-StepAdditional seed 282.032382.9497+0.9174
ACE-StepMean±SD over two additional runs81.1978±1.180282.5436±0.5744+1.3458±0.6058
YuE2Original experiment86.675787.1537+0.4779
YuE2Additional seed 185.183585.6295+0.4460
YuE2Additional seed 286.640386.982+0.3418
YuE2Mean±SD over two additional runs85.9119±1.030186.3057±0.9564+0.3939±0.0737

Both backends obtain positive memory gains in both additional runs, with greater variation across runs for ACE-Step. We will add these re-rating and repeated-run results to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.