acceptodds
Under review as a conference paper at ICLR 2027

TASTE: Taxonomy-Aware Symbolic Piano Generation from Textual Description

Abstract

Text-conditioned symbolic music generation has attracted growing interest with the development of generative models. Yet two questions remain underexplored: what aspects of music the conditioning text should describe, and how the corresponding data should be collected and trained to improve controllability and text–music alignment. We address these challenges with TASTE, a taxonomy-aware framework for symbolic music generation with a main focus on piano pieces. First, we introduce a taxonomy of musical elements that specifies how a piece can be captured through textual description in a musically grounded way. The taxonomy comprises a set of metadata tags (genre, emotion, section plan, etc.) together with a free-form description of music imagery. This structure decomposes textual input to match the generation goal more coherently and precisely. Second, we build a data pipeline for collecting, annotating, and retrieving public symbolic piano resources, yielding a corpus of 432K MIDI files labeled at multiple levels of granularity. Finally, we design a staged training procedure that leverages this multi-granularity data across two model families: an autoregressive model and a non-autoregressive diffusion model. Both objective and subjective evaluations show that our models achieve superior musical-element control, text-description alignment, and overall music quality. We will release the full pipeline, all annotated datasets, and the model checkpoints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Music Generation, Symbolic Music Generation, Controllable Music Generation, Musical Taxonomy, Multi-Granularity Annotation, Staged Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MTZwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TASTE, a symbolic text-to-MIDI (TTM) generation framework that accepts a curated text taxonomy. The taxonomy comprises three parts: 1) tags for objective, categorical attributes; 2) a structure annotation indicating temporal recurrence; and 3) a free-form caption. The paper begins with curating a TTM dataset with text annotations under the proposed taxonomy. A total of 432K MIDI-text pairs are curated into three tiers: 1) high-quality and 2) curated, which contain fewer samples but more reliable labels sourced from real metadata, and 3) a larger-scale tier whose labels are estimated by annotators trained on the first two tiers. The curated data is used to train a Transformer and a DiT model for text-to-music generation. Experiments compare both models to baseline TTM models regarding text-to-music alignment, tag accuracy, structure adherence, and general music quality. An additional subjective listening study shows higher preference for the proposed model than the baselines.

Critical strengths
  • The proposed taxonomy for text-to-MIDI generation is of great value. Music perception is highly subjective, meaning some aspects can be described clearly in text, while others can only be captured vaguely. To make TTM models useful, researchers must discuss what makes a text description of music effective and frame their model-design rationales accordingly. This paper's discussion is a good start and may offer insights for follow-up TTM research.
Critical weaknesses

However, the technical discussions in the paper appear to lack sufficient rationale, and the experiments appear insufficient.

  • Line 257 "Imagery retrieval and rewriting". This retrieval + rewriting method sounds less well motivated or validated. I understand the two smaller tiers' existing captions may be higher quality, and it may be beneficial to rewrite from them. But this may also potentially limit the diversity. It could be more convincing to have an experiment comparing two setups: 1) the proposed retrieval + rewriting strategy, and 2) using an audio captioning model + LLM.

  • The evaluation uses transcribed MIDI from AI-generated piano performances, the rationale of which is not fully validated. Although piano transcription has been quite accurate today, it can still make octave errors and imprecise durations. For AI-generated audio input, transcription models could be more error-prone and thus may bias the evaluation results.

  • The section label control does not appear to be effective. On the demo page, no obvious recurrent patterns are found in the sections of the same label.

  • The 5s granularity for section boundary accuracy appears to be too coarse. In music structure literature, 0.5s and 3s are typically both reported.

  • There lacks an ablation study for the proposed Restricted Backward Consolidation.

Questions for authors
  • Line 244 "Where source metadata provide a genre, we map and keep it instead of the probe prediction." There appears to be insufficient context for this statement. To the reviewer's understanding, the pruning process may alter certain validation pieces' tagged label when the probe's confidence for that label is low, but for the genre field, for certain reason, the tagged label is never altered. What is the motivation for this setup?

  • Line 247. Should "0.60 validation precision" be "0.60 confidence/likelihood"?

4 comments

Thank you for the detailed review and the concrete suggestions. We ran the comparison and the ablation you asked for and added the boundary tolerances you recommended. All of it will be added to the paper.

1. Imagery retrieval and rewriting versus an audio LLM

Retrieval and rewriting yielded the highest Vendi diversity score and the strongest match to the source music in our comparison with direct audio-LLM captioning. For this comparison, we used an open audio LLM (Qwen3-Omni-30B) to listen to the 10,000 large-tier pieces and wrote the imagery directly, under the same writing rules as our rewrite step.

Source of the descriptionsDiversity (Vendi ↑)Near-duplicates ↓Matches original music ↑
Ours (retrieval + rewriting)24.87.0%58.2%
Audio LLM, recommended sampling4.994.1%53.8%
Audio LLM with our diversity instructions19.11.4%52.4%

The Vendi score is the effective number of distinct descriptions in a text-embedding space. The last column asks whether CLaMP 3 places a piece closer to its own description than to that of another piece. And we interpret our pipeline and the experiment as follows:

  • Retrieval does not limit diversity. Our descriptions are also more diverse than the two tiers they are retrieved from (17.5 and 19.4), because the rewrite changes the concrete scenery seeds.
  • The audio LLM alone collapses onto a few scenes. For one model without rewriting, its sampling space is natively not unlimited. For the Qwen variant we use, 67% of its descriptions mention rain, against 4% of ours. Our diversity instructions repair most of this.
  • Retrieval keeps the description tied to the music. The initial retrieval is based on music embeddings, so the overall mood and vibe are transferred. In experiments, our pipelines also match their source music best. The audio LLM becomes less tied to the music because, first, most audio LLMs remain poor at non-lyrical music understanding, and, further, it is pushed to be more diverse to avoid collapse.

All these results motivate the design of retrieval for vibe grounding, and rewriting for diversity.

2. Transcription of AI-generated performances

All systems generate MIDI directly, so transcription does not introduce system-specific errors in their outputs. Also, the transcribed references behave like clean piano performances. Their audio distance to MAESTRO (Disklavier recordings) is 0.497, lower than every external baseline (0.745 to 2.049), and they receive the highest listener rating (3.85 out of 5). Frequent octave or duration errors would clearly destroy the listening experience and pull these numbers down. Finally, we use Transkun, which is the state-of-the-art transcriber for piano, while existing symbolic datasets are mostly transcribed from even earlier systems, which have a higher likelihood of such transcription errors. Nonetheless, we agree and will list this as a limitation.

3. Section control: recurrence and boundary tolerance

The DiT approaches reference-level recurrence and achieves comparable boundary F1 at 3 s, while the AR model shows weaker recurrence and boundary control. Boundary F1 falls at 0.5 s for both models and for the references.

Recurrence. We now quantify it without the probe by using training-free metrics that ask, within each rendered piece, are two sections with the same letter more similar in the chroma self-similarity matrix than two non-adjacent sections with different letters? The AUC is 0.901 for the reference performances, 0.876 for the DiT, and 0.653 for the AR model. Thus, the AR model can achieve chroma and harmony similarity, yet exact motif, melody, texture replication or explicit variation remain limitations, and we will state the AR limitation in the paper.

Boundary tolerance. As you suggested, we now report 0.5 s and 3 s next to 5 s. The value in parentheses is a control that scores the same detected boundaries against the plan of a different piece, as the reference value.

SystemF1 at 0.5 sF1 at 3 sF1 at 5 s (paper)
Reference0.256 (0.074)0.661 (0.211)0.896 (0.353)
TASTE (DiT)0.197 (0.057)0.707 (0.225)0.868 (0.340)
TASTE (AR)0.109 (0.055)0.344 (0.190)0.517 (0.320)

At 3 s, the DiT is already comparable to the reference and three times the control, so its boundaries follow the plan it was given. For AR, our model is also roughly two times larger than the control in all tolerances, which supports the AR model’s section boundary enforcement. At 0.5 s, all rows drop, including the real performances, and it’s a limitation on label precision. Our plans are natively written on a 5-second grid, far coarser than this tolerance.

4. Ablation of the restricted backward consolidation

Restricted backward consolidation raises mean tag agreement by 0.029, with the largest gain in accompaniment. Production quality and imagery alignment are unchanged.

We retrained the curated and high-quality stages on the labels as they were before consolidation, with everything else unchanged. A second seed of the consolidated recipe measures run-to-run variation. The table follows Table 3.

100 prompts, 8 samples eachWith consolidationWith consolidation (second seed)Without consolidation
Key0.7910.7660.738
Tempo0.7630.7690.760
Emotion0.5430.5940.539
Genre0.5210.5150.489
Accompaniment0.3050.3110.235
Performance0.3000.2930.286
Mean over the six tags0.5370.5410.508
CLaMP 3 (full prompt)0.1400.1390.138
CLAP (full prompt)0.3410.3410.341
Production quality (Audiobox PQ)7.9487.9437.950

Removing consolidation lowers the mean tag agreement by 0.029 (95% CI 0.017 to 0.042), about seven times the difference between the two consolidated runs. The loss is concentrated in accompaniment, one of the two fields that consolidation may revise substantially on the curated set. Production quality and imagery alignment are unchanged, so consolidation gives a modest and reliable gain in tag control at no cost in quality.

Questions

Line 244, genre from source metadata. We apologize for the unclear sentence. Pruning never alters a label. When the source dataset already provides a genre for a large-tier piece (276,948 of 371,053), we map it to our vocabulary and use it, because metadata from the source is stronger evidence than a prediction from the notes, typically coming from the original data source’s genre. For example, POP909 is clearly pop music, and for those sourced from YouTube, genres can be deterministically inferred from video titles and descriptions in the dataset construction process. The probe predicts genre only for the remaining pieces.

Line 247, "0.60 validation precision". Precision is the intended word. For each field, we choose the lowest confidence threshold at which the predictions above it are correct at least 60% of the time on the validation split. We will rewrite both sentences for better clarification.

Thank you for your response to my questions.

My remaining concerns regard the demo. Even for the DiT model, there does not appear to be a salient structure in the generated MIDI. I also have the impression that the MIDI is overly free-flowing, without a consistent sense of meter, and thus sounds unstable.

Reviewer_xdXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TASTE is a framework for text-conditioned symbolic piano generation that separates musical intent into attribute tags, temporal section plans, and free-form imagery descriptions. The authors construct a three-tier corpus of approximately 432K annotated MIDI files and develop staged training procedures for autoregressive and latent diffusion models. Experiments report improved attribute agreement and text–music alignment over the evaluated baselines, with the autoregressive model also receiving higher listener-rated music quality.

Critical strengths
  1. A coherent formulation connecting intent representation, supervision, and generation. The distinction between discrete attributes, temporal structure, and open-ended imagery is well motivated, and each channel receives an appropriate annotation strategy, conditioning mechanism, and evaluation target. Its implementation across a substantial annotated corpus and two model families gives the formulation practical substance and provides a potentially useful foundation for controllable symbolic music generation.

  2. The autoregressive model shows improvements supported by both automatic and human evaluation. Emotion agreement reaches 58.5%, compared with 27.6%/28.5% for text2midi/MIDI-LLM. Listener imagery-matching accuracy improves to 80.3% from 54.5%/51.5%, while mean quality ratings increase to 3.26 from 2.71/2.82 (Tables 3 and 6). These consistent improvements provide evidence of practical gains in controllability and perceived quality, beyond a single automatic metric.

Critical weaknesses
  1. The experiments do not isolate the contribution of taxonomy-aware conditioning. Comparisons with existing systems confound the conditioning design with differences in training data, architecture, and training procedure. The stage-wise and weak-data ablations are informative, but do not compare separate conditioning channels against a unified caption containing the same information under matched data, backbone, and training budget. Consequently, the results establish the effectiveness of the complete system but leave the central methodological contribution insufficiently substantiated.

  2. The evaluation does not independently establish the claimed imagery and structural alignment. The imagery listening test does not sufficiently specify distractor construction or report comparisons controlling for emotion and energy, leaving unclear whether performance reflects imagery beyond coarse affective cues. Structural evaluation also relies on a probe involved in constructing reference annotations; the authors acknowledge that reference boundary F1 is partly determined by this process. The complementary novelty metric supports changes near boundaries but does not independently validate recurrence or overall form. These limitations affect the two capabilities that most clearly distinguish the proposed taxonomy from conventional attribute control.

Questions for authors

N/A

2 comments

Thank you for the clear and constructive review. And we made substantial experiments about your concerns, and we will add these experiment results into the paper.

1. Separate conditioning channels versus a unified caption

In a matched ablation, separate channels improve key agreement after full training and mean tag agreement after large-tier-only training, which implies stronger label noise robustness. After all three stages, a unified caption achieves similar overall alignment. The caption model matches our AR model in backbone, data, training stages, and text encoder, and only the conditioning is changed to one templated caption.

100 prompts, 8 samples eachTaxonomy, large tier onlyCaption, large tier onlyTaxonomy, finalTaxonomy, final (second seed)Caption, final
Key0.6090.5760.7910.7660.724
Tempo0.7600.7280.7630.7690.758
Emotion0.3960.3990.5430.5940.621
Genre0.4690.4340.5210.5150.546
Accompaniment0.3160.2790.3050.3110.299
Performance0.2800.2800.3000.2930.295
Mean over the six tags0.4720.4490.5370.5410.540
CLaMP 3 (full prompt)0.0870.0880.1400.1390.139
CLAP (full prompt)0.3460.3480.3410.3410.340
Production quality (Audiobox PQ)7.7967.7837.9487.9437.949

After all three stages, a single caption reaches similar alignment to taxonomy input. Its higher emotion and genre values are not significant compared with the second seed (p ≥ 0.10). Separate channels, however, lead in key agreement by 4 to 7 points (p ≤ 0.02) and perform better under weak supervision: after the large-tier stage alone, mean agreement is 0.472 versus 0.449 (p = 0.006). These results illustrate that separate inputs add a measured benefit for exact conditions, especially under weak supervision.

Why the taxonomy is still needed. The caption is a template filled with our taxonomy fields, so the ablation changes only how the taxonomy is serialized. Emotion provides concrete support for this taxonomy: the caption model itself reaches an emotion agreement of 0.62, compared with 0.28 and 0.29 for existing caption-conditioned systems. Our claim concerns the whole pipeline: attributes, form, and imagery remain distinct so that each can be annotated, learned, and evaluated on its own terms.

  • Annotation and Evaluation. Each channel has its own evidence and reliability control. Dropping a low-confidence tag or revising one field is either undefined or extremely complex with an undivided caption. Our consolidation ablation shows that such correction matters (mean agreement +0.03). Furthermore, the taxonomy, implemented as separate input channels, enables evaluation per field, whereas typical captions support only overall music-text alignment without detailing which parts align poorly.
  • Control. Separate channels let a user fix some fields while changing one. Our strongest structure control also comes from the DiT, where the plan is a time-aligned channel that a sentence cannot carry.

2. Imagery evaluation with emotion and energy control

We will clarify that the listening-study distractor came from a different emotion class. Listeners still performed above chance when energy was matched, and swapping imagery with emotion and tempo held fixed had a modest but reliable effect on music–text alignment.

Re-analysis. We group the emotion labels by energy (angry, anxious and happy are high, calm and sad are low). When both options share the same level, listeners still identify the TASTE description in 68.8% of trials (22/32, p = 0.025), against 78.0% overall. Thus, energy carries part of the signal but not all of it. The subset is small.

New test with emotion and tempo fixed. For each prompt, we generate 8 pieces with its own imagery and 8 with the imagery of another prompt of the same emotion and tempo class. Pieces generated from the own imagery are closer to that text than the others in 53.4% of pairs under CLaMP 3 (p = 0.009) and in 54.5% under LAION-CLAP (p = 0.004).

Although the effect is small, imagery indeed changes the music beyond emotion and tempo. Our revised claim is that imagery adds modest but reliable control beyond the tags, and we view imagery as a minor “correction” beyond discrete labels for specific sentiments.

3. Structure evaluated independently of the probe

Two training-free analyses of rendered audio show that the DiT approaches reference-level section recurrence, while the AR model remains weaker. Among conditional systems, TASTE is closest to the reference in overall form, as other systems don’t support and show any section in generated music.

  • Recurrence. Within each piece, are two sections with the same letter more similar in the chroma self-similarity matrix than two non-adjacent sections with different letters (AUC, chance 0.5)?
  • Overall form. The Structureness Indicator (Wu and Yang, 2020) measures how much of a piece is explained by repetition at 15 s and longer. It needs no plan, so it can also be applied to models with no structure conditions.
SystemRecurrence AUCStructurenessMean length
Reference0.9010.312145 s
TASTE (DiT)0.8760.201135 s
TASTE (AR)0.6530.185136 s
text2midin/a0.02953 s
MIDI-LLMn/a0.04443 s
Aria (unconditional)n/a0.384279 s

The DiT realizes recurrence at the level of the reference performances. The AR model is above chance but remain below the reference, and we will state this limitation.

Among the conditional systems, TASTE is the closest to the reference in overall form. The metric grows with piece length, and within the first 45 seconds our models do not repeat more than the baselines. It shows that TASTE sustains repetition over long pieces, not a ranking at equal length. Aria scores even above real performances from high-quality music with sections and recurrences, which suggests literal looping.

Reference: Shih-Lun Wu & Yi-Hsuan Yang. The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures. ISMIR 2020

Reviewer_P8v5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies text-conditioned piano MIDI generation. The main idea is to split the input into musical tags, a section plan, and an imagery description, and to use different annotation methods for each. The authors build a roughly 432K-piece corpus and train autoregressive and diffusion models through progressively more reliable data tiers. Both models improve over the evaluated caption-based baselines on several measures of control and alignment. The listening results are more mixed: AR improves perceived quality, but DiT receives ratings close to the weaker baselines despite its strong automatic scores.

Critical strengths
  1. The strongest part of the paper is the work on the data. Different musical properties require different evidence, and the pipeline takes this seriously. The authors also report the weaker parts of their annotations rather than treating all 432K pieces as equally reliable. This is a useful contribution for a task where paired data are limited.
  2. There is also evidence that the resulting system works better than the evaluated caption-conditioned baselines. The AR model improves both attribute agreement and listener-rated quality. Its quality score is 3.26, compared with 2.71 for text2midi and 2.82 for MIDI-LLM. The imagery-choice results are encouraging too, although I have a question about the design of that task.
Critical weaknesses
  1. My main concern is the treatment of the 100 held-out examples. Appendix B.1 says the probe benchmark includes “all high-quality pairs,” and B.4 says the production probes are refit on the full benchmark. It is not clear whether the generation-test examples are excluded from these steps, or from the imagery retrieval pool and projection fitting. This would not necessarily mean the generator directly sees test examples, but test information could still affect the labels used to train it. The evaluation tag critics are explicitly trained on the 9,849 training pieces, so that part appears separate. I would like the same clarity for the rest of the pipeline. There is a related issue with structure: the probe helps refine the reference boundaries and is then used for evaluation. The Foote metric provides some independent evidence, but it does not verify section recurrence.
  2. I am also not yet convinced that the experiments establish the benefit of the proposed separation into channels. The full system changes the data, training, and conditioning design together. The large-tier ablation helps assess the data recipe, but it does not tell us whether a single caption containing the same information would work as well. For imagery specifically, the listening study leaves an important detail unstated: how was the incorrect description chosen? If the two options differ in emotion, the task may mostly measure emotional matching. That would still be useful, but it would provide weaker support for the claim that imagery adds control beyond the tags.
Questions for authors

see weaknesses

2 comments

Thank you for the careful reading and the constructive questions. We audited the pipeline for your first concern and ran new experiments for the others. All of this will be added to the paper.

1. The 100 held-out examples

The 100 test pieces were held out from the generator and the evaluation critics, while label construction used test-set material. We found no extra gain from large-tier training on prompts whose descriptions seeded rewritten training labels.

  • Tag probes (B.1, B.4). The 100 pieces are among the 61,129 used to fit them (0.16%).
  • Imagery retrieval (B.6). 52 test descriptions were retrieved among these 100 and seeded 506 of the 371,053 large-tier descriptions (0.14%). The model trains on an LLM rewrite, and no rewrite equals a test description.
  • Structure probe (B.5). 94 of the 100 pieces were in its training split.

If the model had learned from the 52 retrieved descriptions, large-tier training should help more on those prompts than on the other 48. We measure this gain against our ablation that skips the large tier and so never saw any derived description.

Gain from large-tier training52 retrieved prompts48 other promptsDifference [95% CI]
Imagery–music similarity (CLaMP 3)+0.003+0.006−0.003 [−0.010, +0.004]
Mean agreement over the six tags+0.014+0.015−0.001 [−0.029, +0.027]
Section boundary F1+0.058+0.066−0.009 [−0.066, +0.051]

No metric shows an extra gain on the exposed prompts, and the effect of 100 labels is negligible based on experiments, so we did not refit the probes with new isolation. We will state this inclusion in Appendix B.

2. Recurrence measured without the probe

Two training-free analyses of rendered audio show that the DiT approaches reference-level section recurrence, while the AR model remains weaker. Among conditional systems, TASTE is closest to the reference in overall form, as other systems don’t support and show any section in generated music.

  • Recurrence. Within each piece, are two sections with the same letter more similar in the chroma self-similarity matrix than two non-adjacent sections with different letters (AUC, chance 0.5)?
  • Overall form. The Structureness Indicator (Wu and Yang, 2020) measures how much of a piece is explained by repetition at 15 s and longer. It needs no plan, so it can also be applied to models with no structure conditions.
SystemRecurrence AUCStructurenessMean length
Reference0.9010.312145 s
TASTE (DiT)0.8760.201135 s
TASTE (AR)0.6530.185136 s
text2midin/a0.02953 s
MIDI-LLMn/a0.04443 s
Aria (unconditional)n/a0.384279 s

The DiT realizes recurrence at the level of the reference performances. The AR model is above chance and clearly below the reference, and we will state this limitation.

Among the conditional systems, TASTE is the closest to the reference in overall form. The metric grows with piece length, and within the first 45 seconds our models do not repeat more than the baselines. It shows that TASTE sustains repetition over long pieces, not a ranking at equal length. Aria scores even above real performances from high-quality music with sections and recurrences, which suggests literal looping.

3. Taxonomy (Separate channels) versus a single caption

In a matched ablation, separate channels improve key agreement after full training and mean tag agreement after large-tier-only training, which implies stronger label noise robustness. After all three stages, a unified caption achieves similar overall alignment. The caption model matches our AR model in backbone, data, training stages, and text encoder, and only the conditioning is changed to one templated caption.

100 prompts, 8 samples eachTaxonomy, large tier onlyCaption, large tier onlyTaxonomy, finalTaxonomy, final (second seed)Caption, final
Key0.6090.5760.7910.7660.724
Tempo0.7600.7280.7630.7690.758
Emotion0.3960.3990.5430.5940.621
Genre0.4690.4340.5210.5150.546
Accompaniment0.3160.2790.3050.3110.299
Performance0.2800.2800.3000.2930.295
Mean over the six tags0.4720.4490.5370.5410.540
CLaMP 3 (full prompt)0.0870.0880.1400.1390.139
CLAP (full prompt)0.3460.3480.3410.3410.340
Production quality (Audiobox PQ)7.7967.7837.9487.9437.949

After all three stages, a single caption reaches similar alignment to taxonomy input. Its higher emotion and genre values are not significant compared with the second seed (p ≥ 0.10). Separate channels, however, lead in key agreement by 4 to 7 points (p ≤ 0.02) and perform better under weak supervision: after the large-tier stage alone, mean agreement is 0.472 versus 0.449 (p = 0.006). These results illustrate that separate iputs add a measured benefit for exact conditions, especially under weak supervision.

Why the taxonomy is still needed. The caption is a template filled with our taxonomy fields, so the ablation changes only how the taxonomy is serialized. Emotion provides concrete support for this taxonomy: the caption model itself reaches an emotion agreement of 0.62, compared with 0.28 and 0.29 for existing caption-conditioned systems. Our claim concerns the whole pipeline: attributes, form, and imagery remain distinct so that each can be annotated, learned, and evaluated on its own terms.

  • Annotation and Evaluation. Each channel has its own evidence and reliability control. Dropping a low-confidence tag or revising one field is either undefined or extremely complex with an undivided caption. Our consolidation ablation shows that such correction matters (mean agreement +0.03). Furthermore, the taxonomy, implemented as separate input channels, enables evaluation per field, whereas typical captions support only overall music-text alignment without detailing which parts align poorly.
  • Control. Separate channels let a user fix some fields while changing one. Our strongest structure control also comes from the DiT, where the plan is a time-aligned channel that a sentence cannot carry.

4. The description in the listening study

The listening-study distractor came from a different emotion class, and listeners still performed above chance when the options shared the same energy level. A separate test holding the tags fixed found a modest but reliable effect of swapping imagery within the same emotion and tempo class.

Re-analysis. When both options share the same energy, listeners still identify the TASTE description in 68.8% of trials (22/32, p = 0.025), compared with 78.0% overall. Therefore, energy carries part of the signal but not all of it.

New test with all tags fixed. For each prompt, we generate 8 pieces with its own imagery and 8 with the imagery of another prompt of the same emotion and tempo class. Pieces generated from the own imagery are closer to that text than the others in 53.4% of pairs under CLaMP 3 (p = 0.009) and in 54.5% under LAION-CLAP (p = 0.004, chance 50%).

Although the effect is small, imagery indeed changes the music beyond emotion and tempo. Our revised claim is that imagery adds modest but reliable control beyond the tags, and we view imagery as a minor “correction” beyond discrete labels for specific sentiments.

Reference: Shih-Lun Wu & Yi-Hsuan Yang. The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures. ISMIR 2020

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.