- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper frames text-to-music intent alignment as satisfying a per-request rubric of atomic, audibly checkable items, split into explicit requirements (Base) and traits inferred from references and context (Completion). It contributes two things:
- MuRA-Bench: 100 requests derived from Mureka platform seeds, with 1,566 items. An LLM drafts each rubric and one of three music experts revises it. Items are scored by MusicFlamingo audio QA using soft yes/no probabilities.
- MIRA: a test-time agent wrapped around a black-box generator. It grounds the request with tools, builds an online rubric, and runs a shallow, budgeted tree search over prompts (B = 3 or 9 generations) guided by MusicFlamingo, with branch-specific memory.
Across ACE-Step, SongGeneration2 and YuE2-3B, MIRA improves Overall by 5.6-11.5 points over direct prompting, and gains persist under two other evaluators. The headline claim is that YuE2-3B + MIRA (82.1) reaches the highest Overall score among all evaluated systems, ahead of Suno v5.5 (81.9).
Critical strengths
-
The problem framing is useful and well executed. Splitting a request into explicit and inferred requirements, with dimension tags, turns a single opaque relevance score into something diagnostic. The evaluator is chosen empirically: eight scoring methods are compared against 1,536 expert item ratings, and MF-AQA leads on every metric (item SRCC 0.690 vs 0.509 for the next best, CLAPScore 0.245). This gives the benchmark a credible footing that most text-to-music relevance metrics lack.
-
The search gains are not only a selection effect. In the equal-budget comparison, Best-of-9 also selects with MusicFlamingo, yet MIRA still improves Overall by 3.6 points on ACE-Step and 2.4 on YuE2-3B. The ablation in Table 3 attributes most of the gain to tool grounding (Completion 0.719 → 0.748 on ACE-Step) and to adaptive search (0.748 → 0.781). Gains also hold when outputs are rescored by MOSS-Music and Qwen3-Omni (Table 8).
Critical weaknesses
- The search optimizes the same judge that produces the headline numbers, and the headline claim does not survive a change of judge. MIRA searches with MusicFlamingo AQA and selects with it, and Table 2 evaluates with it. The online rubric R(x, E) and the gold rubric R*(x) are different, but both are decomposed by an LLM from the same request, and the paper reports neither their overlap nor how much the experts changed the drafts.
The claim that YuE2-3B + MIRA beats every commercial system holds only under MusicFlamingo, and only by 0.2 points with no variance reported. Under the other evaluators it does not hold:
- MOSS-Music: 78.4, which ties MiniMax 3.0 but is below Suno v5, v5.5 and v6 (79.3, 80.0, 79.7).
- Qwen3-Omni: 70.6, against 72.5 for MiniMax 3.0 and 75.0 for Suno v5.5.
The human study does not test YuE2 + MIRA at all, and ACE-Step + MIRA is rated below Suno v5.5 on all three axes (e.g. 4.04 vs 4.26 overall). "Comparable to Mureka and MiniMax" is defensible; "highest among evaluated systems" is not. The cross-evaluator check only partly addresses circularity: all three judges use the same AQA format, and the two alternatives agree less with experts (Table 1).
- The commercial comparison and the statistics are both thin. The two sides are not run under the same conditions:
- MIRA gets nine generations, tool grounding and verifier selection.
- Commercial systems get a single direct-prompt call.
- MIRA is presented as backend-agnostic, yet it is not applied to any commercial system, not even as Best-of-N.
On the statistics:
- No table reports seeds or variance, even though the paper stresses that generation is stochastic and the Direct baseline is a single sample.
- The memory component adds 0.010 on ACE-Step and 0.005 on YuE2 over 20 requests, which is within plausible noise.
- The selection of that 20-request subset is not described, and it looks easier than the full set (Direct Overall 0.759 vs 0.676 on ACE-Step).
- The human comparison has no paired test and one rating per clip.
- The benchmark's own validity and side effects are not measured. Each request was revised by a single expert, so there is no agreement estimate for the rubrics themselves. This matters most for Completion items, which are interpretive; some items, such as "open-plains biome atmosphere", are hard to judge atomically by ear. The prompts come from Mureka's user distribution, and the ranking of commercial systems changes with the evaluator: Mureka V9.5 is above MiniMax 3.0 under MusicFlamingo and below it under both alternatives. Audio quality is declared out of scope (C.4). Without a quality metric (FAD, Audiobox Aesthetics, SongEval), there is no evidence that steering prompts toward rubric wording leaves musicality intact.
Questions for authors
-
Rubric overlap. How much do the online rubric R(x, E) and the gold rubric R*(x) overlap, measured by semantic matching? Which LLM drafted the gold rubrics, and what fraction of items did experts add, delete or rewrite?
-
Decoupled judge. Can you run search with one judge and evaluation with another, for example Qwen3-Omni for search and MusicFlamingo for evaluation?
-
Variance and significance. What are the means and standard deviations of Table 2 over at least three seeds, and is the YuE2 + MIRA vs Suno v5.5 gap significant?
-
Commercial backends. How do commercial systems score with Best-of-9 or MIRA applied, or with the tool-augmented prompt alone? For the 32 vocal requests, were the same lyrics given to all systems?
-
Missing details. How was the 20-request subset chosen? What is the average size m of the online rubric, and what does each request cost in verifier queries (roughly m × B), planner calls and wall-clock time? For the human study, were the "intent criteria" shown to raters the gold rubric?
-
Ablations and side effects. SongGeneration2 uses stochastic sampling while the other backends use fixed seeds, and it is excluded from the equal-budget comparison. How much of its +11.5 gain comes from sampling diversity? How sensitive are results to η = 0.60 and ε = 0.03? How do audio-quality metrics change under MIRA?
-
Minor. "Dim-Macro" and "D-Macro" are both used. Figure 3 uses "inferred" in the caption and "completed" in panel (c); using one term throughout would help. The contribution list has a redundant phrase ("three generation generators").
9 comments
W2: The commercial comparison and the statistics are both thin.
We thank the reviewer for noting that the commercial comparison conditions and statistical evidence need further explanation. We have added commercial-backend controls on Suno v6, experiments with two additional generation seeds, memory ablations, and re-rating by an additional expert.
Commercial-backend controls. We have applied MIRA to Suno v6 and compared Direct, Best-of-9, and MIRA on the same 20 requests:
| Setting | Overall | Base | Completion | D-Macro |
|---|---|---|---|---|
| Direct (historical audio, reaggregated on the same subset) | 85.3141 | 91.5157 | 77.9501 | 85.5241 |
| Best-of-9 (new) | 86.3404 | 92.4006 | 79.6538 | 86.0720 |
| MIRA B=3 (new) | 86.7330 | 91.4547 | 80.9856 | 86.2085 |
| MIRA B=9 (new) | 87.5786 | 92.0100 | 82.0706 | 87.1559 |
MIRA B=9 improves Overall by 2.2645 points over Direct. Relative to Best-of-9, it improves Overall and Completion by 1.24 and 2.42 points, respectively. These results add empirical evidence for MIRA on a commercial black-box backend.
Repeated runs and variance. The original experiment and two additional runs are shown below. Each condition covers 100 requests, and scores are Overall under MusicFlamingo evaluation. Means and sample standard deviations are computed over the two additional runs only; the original experiment is listed separately as a reference.
| Backend | Setting | Original experiment | Additional seed 1 | Additional seed 2 | Mean±sample SD over two additional runs |
|---|---|---|---|---|---|
| ACE-Step | Direct | 67.6356 | 67.7625 | 67.6571 | 67.7098±0.0745 |
| ACE-Step | MIRA B=3 | 74.1901 | 74.4062 | 74.3447 | 74.3754±0.0435 |
| ACE-Step | MIRA B=9 | 76.3144 | 76.0788 | 75.7408 | 75.9098±0.2390 |
| YuE2 | Direct | 76.4688 | 75.8176 | 77.2981 | 76.5579±1.0468 |
| YuE2 | MIRA B=3 | 80.4821 | 79.0783 | 79.8247 | 79.4515±0.5277 |
| YuE2 | MIRA B=9 | 82.0613 | 81.7523 | 81.4943 | 81.6233±0.1824 |
Memory ablations. Each condition uses the same 20 requests with nine candidates per request. Full scores and means and sample standard deviations across the two additional runs are shown below:
| Backend | Experiment | Without memory | With memory | With minus without memory |
|---|---|---|---|---|
| ACE-Step | Original experiment | 82.6183 | 83.6067 | +0.9885 |
| ACE-Step | Additional seed 1 | 80.3633 | 82.1374 | +1.7741 |
| ACE-Step | Additional seed 2 | 82.0323 | 82.9497 | +0.9174 |
| ACE-Step | Mean±SD over two additional runs | 81.1978±1.1802 | 82.5436±0.5744 | +1.3458±0.6058 |
| YuE2 | Original experiment | 86.6757 | 87.1537 | +0.4779 |
| YuE2 | Additional seed 1 | 85.1835 | 85.6295 | +0.4460 |
| YuE2 | Additional seed 2 | 86.6403 | 86.9820 | +0.3418 |
| YuE2 | Mean±SD over two additional runs | 85.9119±1.0301 | 86.3057±0.9564 | +0.3939±0.0737 |
Differences are computed from unrounded scores. Both backends obtain positive memory gains in both additional runs.
Subset selection. The 20 requests were randomly sampled and then fixed; all controls share the same request list.
Human ratings. We added an expert to re-rate 12 requests, 48 audio clips, and 780 rubric items from the original human-evaluation set. Item SRCC, Kendall τb, and Pearson correlation with the original expert ratings are 0.4311, 0.3322, and 0.4139, respectively. Agreement between the evaluators and the combined ratings is reported in by1a-W3.
W3: The benchmark's own validity and side effects are not measured.
We thank the reviewer for the suggestions. We clarify the gold-rubric revision process and the basis for selecting the main evaluator, and report additional audio-quality results.
Gold-rubric construction. Before revision, the three experts received common training and followed the same guidelines: retain explicit user constraints, check contextual support for Completion requirements, ensure that every item is audibly judgeable and corresponds to a single requirement, and remove or merge redundant content. Experts independently revised their assigned requests following these guidelines. This process aimed to align revision standards. The original study did not assign requests for repeated revision, so cross-expert agreement for the rubrics themselves was not computed. For wording such as "open-plains biome atmosphere," the judgment concerns the fit between the musical atmosphere and the scene, rather than identifying a literal geographic environment from audio. We will clarify this judgment target.
Evaluator selection and commercial rankings. Yes, different evaluators produce different commercial-system rankings. Our main evaluator was selected based on agreement with expert judgments rather than the ranking of any particular commercial system:
| Evaluator | Mureka V9.5 | MiniMax 3.0 | Item-level Spearman with human ratings | Clip-level Pearson with human ratings |
|---|---|---|---|---|
| MusicFlamingo | 81.2332 | 77.0495 | 0.6896 | 0.8169 |
| MOSS-Music | 76.5172 | 78.4049 | 0.5021 | 0.5912 |
| Qwen3-Omni | 70.8386 | 72.5299 | 0.5094 | 0.6318 |
Commercial-system scores come from the full benchmark, while human-rating correlations come from the validation set in Table 1. MusicFlamingo achieves the highest agreement in the available human validation, so we use it as the main evaluator; its results have more direct support from human judgments. Results from the two alternative evaluators are retained to show how rankings depend on the evaluator.
Audio quality. We have additionally scored the historically selected audio using Audiobox Aesthetics and SongEval, without reselecting candidates based on quality metrics:
| Backend | Metric | Direct | MIRA B=9 | Difference |
|---|---|---|---|---|
| ACE-Step | Production quality PQ | 7.9753 | 7.9118 | −0.0635 |
| ACE-Step | Content enjoyment CE | 7.1079 | 6.9563 | −0.1516 |
| ACE-Step | Vocal musicality | 3.9676 | 4.0171 | 0.0495 |
| YuE2 | Production quality PQ | 8.1677 | 8.1244 | −0.0433 |
| YuE2 | Content enjoyment CE | 7.4576 | 7.4248 | −0.0327 |
| YuE2 | Vocal musicality | 4.0338 | 3.9912 | −0.0426 |
| SongGeneration2 | Production quality PQ | 8.1965 | 8.1389 | −0.0576 |
| SongGeneration2 | Content enjoyment CE | 7.315 | 7.1954 | −0.1196 |
| SongGeneration2 | Vocal musicality | 3.8935 | 3.8609 | −0.0326 |
PQ/CE are from Audiobox Aesthetics, while vocal musicality is assessed by SongEval only for audio containing vocals. Differences are B=9 minus Direct, and metrics retain their original scales. Under Audiobox Aesthetics and SongEval evaluation, some production-quality and content-enjoyment metrics decrease slightly, while vocal musicality does not show a clear direction of change.
We sincerely thank the reviewer for the careful review and specific suggestions on evaluation, statistical evidence, and reproducibility.
We thank the reviewer for raising this concern. We separately address the relationship between judges, rubric construction, and commercial-system rankings, and add experiments using different models for search and final evaluation.
First, MIRA's improvements to the generation backends persist when the evaluator changes. The existing cross-evaluator results are shown below; all scores are Overall:
All three evaluators observe improvements after applying MIRA to ACE-Step and YuE2. We use a common AQA format to make each evaluation target explicit and enable direct comparison of judgments across models. The expert-agreement experiment in Table 1 supports this choice. We therefore use MusicFlamingo as the main evaluator while retaining results from the other models to show performance under different judges.
Second, we add a decoupled experiment using Qwen3-Omni for search and MusicFlamingo for final evaluation. On the same 20 ACE-Step requests, Qwen3-Omni provides search feedback and selects candidates, while MusicFlamingo evaluates the selected outputs against the expert gold rubrics.
When different models perform search and final evaluation, increasing the budget from 3 to 9 improves all four final-evaluation metrics, with Overall increasing by 2.4984 points; at the same budget, MusicFlamingo-guided search achieves a higher score.
Regarding rubric sources and overlap, the gold rubrics were drafted by GPT-5.6-sol and revised by experts who received common training, while the online rubrics are generated by Kimi 2.6. The gold drafts contained 1,078 Base and 335 Completion items, and the expert-revised versions contained 996 and 550 items, respectively. Additional model-assisted semantic matching across 100 requests shows that the online rubrics fully cover 83.13% of gold Base items. For Completion, full coverage is 4.04% and partial coverage is 63.33%.
Finally, we will revise the wording of the commercial-system ranking. Across the original experiment and two additional seeds, YuE2 + MIRA achieves Overall 81.7693±0.2839, with no statistically significant difference from Suno v5.5's 81.8779. We will therefore replace "outperforms all commercial systems" with "reaches the intent-alignment performance of the latest commercial systems." The existing human study covers ACE-Step, with overall ratings of 4.04 for MIRA and 3.84 for Direct, compared with 4.26 for Suno v5.5. Given the time required for human evaluation, we will include a corresponding YuE2 human study in subsequent work.