- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TASTE, a symbolic text-to-MIDI (TTM) generation framework that accepts a curated text taxonomy. The taxonomy comprises three parts: 1) tags for objective, categorical attributes; 2) a structure annotation indicating temporal recurrence; and 3) a free-form caption. The paper begins with curating a TTM dataset with text annotations under the proposed taxonomy. A total of 432K MIDI-text pairs are curated into three tiers: 1) high-quality and 2) curated, which contain fewer samples but more reliable labels sourced from real metadata, and 3) a larger-scale tier whose labels are estimated by annotators trained on the first two tiers. The curated data is used to train a Transformer and a DiT model for text-to-music generation. Experiments compare both models to baseline TTM models regarding text-to-music alignment, tag accuracy, structure adherence, and general music quality. An additional subjective listening study shows higher preference for the proposed model than the baselines.
Critical strengths
- The proposed taxonomy for text-to-MIDI generation is of great value. Music perception is highly subjective, meaning some aspects can be described clearly in text, while others can only be captured vaguely. To make TTM models useful, researchers must discuss what makes a text description of music effective and frame their model-design rationales accordingly. This paper's discussion is a good start and may offer insights for follow-up TTM research.
Critical weaknesses
However, the technical discussions in the paper appear to lack sufficient rationale, and the experiments appear insufficient.
-
Line 257 "Imagery retrieval and rewriting". This retrieval + rewriting method sounds less well motivated or validated. I understand the two smaller tiers' existing captions may be higher quality, and it may be beneficial to rewrite from them. But this may also potentially limit the diversity. It could be more convincing to have an experiment comparing two setups: 1) the proposed retrieval + rewriting strategy, and 2) using an audio captioning model + LLM.
-
The evaluation uses transcribed MIDI from AI-generated piano performances, the rationale of which is not fully validated. Although piano transcription has been quite accurate today, it can still make octave errors and imprecise durations. For AI-generated audio input, transcription models could be more error-prone and thus may bias the evaluation results.
-
The section label control does not appear to be effective. On the demo page, no obvious recurrent patterns are found in the sections of the same label.
-
The 5s granularity for section boundary accuracy appears to be too coarse. In music structure literature, 0.5s and 3s are typically both reported.
-
There lacks an ablation study for the proposed Restricted Backward Consolidation.
Questions for authors
-
Line 244 "Where source metadata provide a genre, we map and keep it instead of the probe prediction." There appears to be insufficient context for this statement. To the reviewer's understanding, the pruning process may alter certain validation pieces' tagged label when the probe's confidence for that label is low, but for the genre field, for certain reason, the tagged label is never altered. What is the motivation for this setup?
-
Line 247. Should "0.60 validation precision" be "0.60 confidence/likelihood"?
4 comments
3. Section control: recurrence and boundary tolerance
The DiT approaches reference-level recurrence and achieves comparable boundary F1 at 3 s, while the AR model shows weaker recurrence and boundary control. Boundary F1 falls at 0.5 s for both models and for the references.
Recurrence. We now quantify it without the probe by using training-free metrics that ask, within each rendered piece, are two sections with the same letter more similar in the chroma self-similarity matrix than two non-adjacent sections with different letters? The AUC is 0.901 for the reference performances, 0.876 for the DiT, and 0.653 for the AR model. Thus, the AR model can achieve chroma and harmony similarity, yet exact motif, melody, texture replication or explicit variation remain limitations, and we will state the AR limitation in the paper.
Boundary tolerance. As you suggested, we now report 0.5 s and 3 s next to 5 s. The value in parentheses is a control that scores the same detected boundaries against the plan of a different piece, as the reference value.
| System | F1 at 0.5 s | F1 at 3 s | F1 at 5 s (paper) |
|---|---|---|---|
| Reference | 0.256 (0.074) | 0.661 (0.211) | 0.896 (0.353) |
| TASTE (DiT) | 0.197 (0.057) | 0.707 (0.225) | 0.868 (0.340) |
| TASTE (AR) | 0.109 (0.055) | 0.344 (0.190) | 0.517 (0.320) |
At 3 s, the DiT is already comparable to the reference and three times the control, so its boundaries follow the plan it was given. For AR, our model is also roughly two times larger than the control in all tolerances, which supports the AR model’s section boundary enforcement. At 0.5 s, all rows drop, including the real performances, and it’s a limitation on label precision. Our plans are natively written on a 5-second grid, far coarser than this tolerance.
4. Ablation of the restricted backward consolidation
Restricted backward consolidation raises mean tag agreement by 0.029, with the largest gain in accompaniment. Production quality and imagery alignment are unchanged.
We retrained the curated and high-quality stages on the labels as they were before consolidation, with everything else unchanged. A second seed of the consolidated recipe measures run-to-run variation. The table follows Table 3.
| 100 prompts, 8 samples each | With consolidation | With consolidation (second seed) | Without consolidation |
|---|---|---|---|
| Key | 0.791 | 0.766 | 0.738 |
| Tempo | 0.763 | 0.769 | 0.760 |
| Emotion | 0.543 | 0.594 | 0.539 |
| Genre | 0.521 | 0.515 | 0.489 |
| Accompaniment | 0.305 | 0.311 | 0.235 |
| Performance | 0.300 | 0.293 | 0.286 |
| Mean over the six tags | 0.537 | 0.541 | 0.508 |
| CLaMP 3 (full prompt) | 0.140 | 0.139 | 0.138 |
| CLAP (full prompt) | 0.341 | 0.341 | 0.341 |
| Production quality (Audiobox PQ) | 7.948 | 7.943 | 7.950 |
Removing consolidation lowers the mean tag agreement by 0.029 (95% CI 0.017 to 0.042), about seven times the difference between the two consolidated runs. The loss is concentrated in accompaniment, one of the two fields that consolidation may revise substantially on the curated set. Production quality and imagery alignment are unchanged, so consolidation gives a modest and reliable gain in tag control at no cost in quality.
Questions
Line 244, genre from source metadata. We apologize for the unclear sentence. Pruning never alters a label. When the source dataset already provides a genre for a large-tier piece (276,948 of 371,053), we map it to our vocabulary and use it, because metadata from the source is stronger evidence than a prediction from the notes, typically coming from the original data source’s genre. For example, POP909 is clearly pop music, and for those sourced from YouTube, genres can be deterministically inferred from video titles and descriptions in the dataset construction process. The probe predicts genre only for the remaining pieces.
Line 247, "0.60 validation precision". Precision is the intended word. For each field, we choose the lowest confidence threshold at which the predictions above it are correct at least 60% of the time on the validation split. We will rewrite both sentences for better clarification.
Thank you for your response to my questions.
My remaining concerns regard the demo. Even for the DiT model, there does not appear to be a salient structure in the generated MIDI. I also have the impression that the MIDI is overly free-flowing, without a consistent sense of meter, and thus sounds unstable.
Thank you for the detailed review and the concrete suggestions. We ran the comparison and the ablation you asked for and added the boundary tolerances you recommended. All of it will be added to the paper.
1. Imagery retrieval and rewriting versus an audio LLM
Retrieval and rewriting yielded the highest Vendi diversity score and the strongest match to the source music in our comparison with direct audio-LLM captioning. For this comparison, we used an open audio LLM (Qwen3-Omni-30B) to listen to the 10,000 large-tier pieces and wrote the imagery directly, under the same writing rules as our rewrite step.
The Vendi score is the effective number of distinct descriptions in a text-embedding space. The last column asks whether CLaMP 3 places a piece closer to its own description than to that of another piece. And we interpret our pipeline and the experiment as follows:
All these results motivate the design of retrieval for vibe grounding, and rewriting for diversity.
2. Transcription of AI-generated performances
All systems generate MIDI directly, so transcription does not introduce system-specific errors in their outputs. Also, the transcribed references behave like clean piano performances. Their audio distance to MAESTRO (Disklavier recordings) is 0.497, lower than every external baseline (0.745 to 2.049), and they receive the highest listener rating (3.85 out of 5). Frequent octave or duration errors would clearly destroy the listening experience and pull these numbers down. Finally, we use Transkun, which is the state-of-the-art transcriber for piano, while existing symbolic datasets are mostly transcribed from even earlier systems, which have a higher likelihood of such transcription errors. Nonetheless, we agree and will list this as a limitation.