- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LyriCo frames controllable Chinese lyric generation as supervised fine-tuning of decoder-only LMs on lyrics interleaved with control tokens. The tokens encode section layout, line count, per-line character count, per-line word-group (prosodic-phrase) lengths and rhyme class; theme and genre are given as text. Training data are ~190k Mandarin songs. They are annotated by a Pop/gufeng classifier, LLM-extracted theme labels, a TTS-prosody-based segmenter for word groups, and audio structure analysis aligned to LRC timestamps. Two mechanisms are added on top of standard SFT: an up-weighted loss on control tokens (λ), and a fixed, position-growing additive attention bias from content tokens to the current line's rhyme, length and word-format tokens (β). Evaluation uses 500 briefs (Pop/gufeng) with per-constraint accuracies. It includes seven 1.5B (λ, β) configurations, 14B and GLM-9B models compared with prompted GPT-5.5 and DeepSeek-V4-Pro, 506 ChipSong cases compared against published SongNet/ChipSong numbers, and a GPT-5.6 judge for theme and genre adherence.
Critical strengths
- The paper tackles a practically relevant and unusually fine-grained control problem: section layout, line count, Chinese-character count, internal group lengths, rhyme, genre, and theme in one representation. The 14B model achieves about 99% or better on every reported structural metric and is much more reliable than prompted API models on long word-group constraints. This is useful evidence that supervised adaptation can deliver precise structural compliance.
- The experimental scope and metric caveats are stated clearly. The paper distinguishes per-constraint from whole-brief success, acknowledges the near-ceiling content judge, and labels the ChipSong numbers as nonsample-matched published references. The annotation pipeline also reports explicit audit denominators. The TTS-inspired grouping is a promising representation, although singability itself is not validated.
Critical weaknesses
- Evidence for the two proposed mechanisms is not strong enough to separate their contribution from run and sampling variation. Standard SFT (L1A0) already reaches 95.19% section accuracy, 97.58% line-length accuracy, and 97.96% word-format accuracy. L3A3 adds 3.40 points in line count, 3.61 in section accuracy, 0.76 in line length, 0.80 in rhyme, and 0.56 in word format.
- All ablation values appear to come from one training run per configuration and one stochastic sample per brief (temperature 0.9, top-p 0.95).
- No training seeds, repeated generations, confidence intervals, or paired significance tests are reported.
- The pattern is non-monotonic: L1A5 is not consistently better than L1A3, and L3A0 exceeds L3A3 on line-count accuracy.
- The mechanisms are not isolated at 14B or on GLM-9B; those comparisons only show complete fine-tuning versus unfine-tuned backbones. The robust conclusion is that fine-tuning on the structured representation greatly improves compliance. The incremental value of loss weighting and attention bias remains uncertain, which limits methodological novelty.
- The evaluation does not establish overall lyric quality, strict joint success, or benchmark/data independence.
- There is no human or reference-based evaluation of coherence, originality, artistic quality, or singability. The single GPT-5.6 content judge is near ceiling and measures only genre/theme consistency.
- Appendix A.4 excludes parse failures from sample-level accuracies, scores only overlapping requested/generated lines, and exempts undefined "center sentences" from some checks. The paper does not report strict all-constraints success per brief with every failure counted.
- Natural strong alternatives are missing: checker-and-resample or best-of-k prompting, grammar/constrained decoding, a 14B SFT-only ablation, and specialized systems retrained on the same data.
- The 500-brief benchmark construction and its overlap with training songs, themes, or templates are not described. The ChipSong table uses different test sets, denominators, word-format aggregation, and rhyme classes, so bold cross-row maxima are not a controlled comparison.
- The source, rights, and license terms for roughly 250k songs with lyrics, audio, and LRC data are unspecified. This affects reproducibility and raises a copyright/terms-of-use concern.
Questions for authors
Clarifications that could change my assessment:
- Can you repeat the Table 4 ablations across training seeds and repeated generations, report paired uncertainty, and test L3A3 versus L1A0? Does the configuration ordering persist?
- Please report strict per-brief success with parse failures and missing/extra lines counted as failures. What are "center sentences," and how many units does each system exclude from each metric?
- What do a 14B SFT-only model and checker-and-resample or constrained-decoding baselines achieve under matched compute and inputs?
- How were the 500 briefs constructed, and what overlap exists with training songs, themes, artists, or structural templates?
- What are the sources and license or permission terms for the lyrics, audio, and LRC files? What data, code, segmenter, prompts, outputs, and checkpoints will be released? Future-work suggestion: add blinded human evaluation of lyric quality and singability, and compare TTS-inspired grouping with length-matched or lexical alternatives.
3 comments
Thanks for the rebuttal that addresses several of my concerns, especially by adding strict whole-lyric results, explaining the benchmark construction, and evaluating forced control tokens.
However, the main concern about the contribution of the proposed mechanisms remains. In particular, L3A3 performs worse than L3A0 on strict whole-lyric success (31.6% vs. 34.0%), and the confidence interval includes zero. The lack of repeated training runs, a matched 14B SFT comparison, and clear information about data licensing also remains.
Therefore, I would keep my score unchanged.
Thank you for your follow-up. For attention’s contribution, our additional results provide complementary evidence on constraint coverage and lyric quality. Strict whole-lyric success treats any violation as a failure; the threshold analysis additionally distinguishes outputs satisfying a high proportion of the requirements, which is relevant to practical usability.
These comparisons use the matched 1.5B checkpoint (6200), with repetition penalty 1.15. The supplementary exploratory analysis below reports lyric-level pass rates across all 500 briefs. Each threshold requires correct overall structure and the specified proportion of lines to satisfy character-count, rhyme, and word-group constraints simultaneously:
| Configuration | ≥90% compliant lines | ≥85% compliant lines | ≥80% compliant lines |
|---|---|---|---|
| Weighted loss only | 79.0% | 86.2% | 91.4% |
| Weighted loss + attention | 79.0% | 88.4% | 91.6% |
Adding attention yields 11 additional lyrics meeting the 85% threshold, alongside higher observed scores on all five individual constraint metrics.
The LLM-as-judge assessment on 100 shared briefs also reports higher mean ratings across all four reported dimensions:
| Configuration | Fluency | Theme relevance | Style | Expression |
|---|---|---|---|---|
| Weighted loss only | 2.70 | 3.91 | 3.66 | 2.46 |
| Weighted loss + attention | 2.74 | 3.96 | 3.74 | 2.53 |
These results characterize attention’s observed effects on individual constraints, high-compliance outputs, and quality ratings. We will make this specific contribution explicit in the revised paper.
Thank you for identifying the evaluation details needed to assess the contribution. We re-scored existing outputs under a strict whole-lyric criterion, added paired uncertainty estimates, and report the ablations separately from the human comparison. The principal result: the complete configuration improves strict compliance over ordinary SFT by 5.6 percentage points on the same 500 briefs.
1. Strict whole-lyric compliance and matched ablations
L1A0 denotes ordinary SFT, L3A0 weighted loss only, and L3A3 weighted loss plus attention. We compare existing 1.5B outputs on all 500 briefs at checkpoint 6200 (the last commonly saved checkpoint, i.e., the same training progress) with repetition penalty 1.15.
A successful lyric must be completely parseable and satisfy the requested total line count, complete section-type sequence and section lengths, every line's Chinese-character count and rhyme class, and all word-group lengths. Missing or extra lines/groups, malformed output, and unclosed sections fail. Theme and perceived genre are separate content-quality dimensions.
With 10,000 paired bootstrap resamples stratified by Pop/gufeng, weighted loss improves strict compliance over SFT by 8.0 points (95% interval: +2.8 to +13.4), and the complete configuration by 5.6 points (+0.6 to +10.6). The L3A3–L3A0 difference is −2.4 points ([−7.8, +3.2]). These intervals quantify variation across briefs for the saved models; no additional training or decoding was performed.
Adding attention to weighted loss raises all five per-constraint metrics: line count 95.60→96.40, section accuracy 98.22→98.80, line length 97.88→98.16, rhyme 94.78→94.92, and word format 98.10→98.46 (%). Because strict success requires all constraints simultaneously, per-constraint gains need not raise the whole-lyric pass rate.
Failure handling. The original parser extracts recognized content without requiring well-formed markers, hence its 0/500 sample-level errors. A complete-format check finds at least one violation (e.g., mismatched section start/end types) in 95/500 (19.0%), 17/500 (3.4%), and 21/500 (4.2%) outputs; all count as failures above. These measure format validity, not missing lyrics. The original per-metric excluded-unit counts have not been fully reconstructed; the strict evaluation includes every brief.
Center-sentence rule. This rule does not affect the 500-brief benchmark: no prompt supplies a center sentence, so the branch is never activated, and all generated lines are checked against their constraints. Appendix A.4 documents it for an optional LyriCo feature in which a user-supplied sentence must be retained; that requirement then takes priority over conflicting length, rhyme, or grouping requirements for that line. We will clarify this in Appendix A.4.
2. SFT and stronger inference baselines
L1A0 is the ordinary-SFT control with the same 1.5B backbone, representation, and training recipe; changing only the loss weight and attention coefficient isolates the added mechanisms. The 14B and GLM experiments evaluate the complete systems and have a different scope. A 14B SFT-only ablation and matched-compute comparisons with checker-based resampling, best-of-k selection, or constrained decoding are outside the current experiments; we will make the baseline scope explicit and discuss these methods in the revision.
3. Benchmark construction and overlap
The 500 briefs are programmatically generated with random sampling and coverage rules; the generator does not read training lyrics, model outputs, or scores. The set contains 250 Pop and 250 gufeng briefs, 34 themes, total line counts 10–36, 35 section-type/section-length combinations, 16 rhyme classes, and word-group lengths 1–7; complete prompts are unique and identical across L1A0, L3A0, and L3A3. As generated specifications rather than sampled songs, song- or artist-level splitting does not apply. Shared themes or structural patterns may still occur in training; the benchmark evaluates control coverage rather than a dedicated split of unseen combinations.
4. Reproducibility and release plan
We will release the LyriCo model weights, evaluation code, test prompts, and all generated test outputs from LyriCo and the directly evaluated baselines, allowing researchers to recompute the objective metrics and benchmark new systems.
The annotation pipeline uses open-source components for audio-level musical structure analysis and prosodic grouping, and an in-house classifier for Pop/gufeng labels; audio boundaries are mapped to lyric lines via LRC timestamps. We will explain each component's inputs, outputs, and role.