- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PianoPoet, a single-checkpoint autoregressive causal Transformer framework that unifies piano transcription (audio-to-MIDI) and piano rendering (MIDI-to-audio). While prior unified models (e.g., U-MusT) required separate checkpoints and lagged behind specialized methods, PianoPoet achieves competitive performance across both tasks within a single 113M-parameter model. The core technical innovation lies in representation decoupling: 1) A shared MIDI tokenizer encoding paired pitch-velocity, 10 ms hybrid timestamps, and pedal tokens. 2) Continuous semantic audio frames (50 Hz, frozen Conformer) for fine-grained transcription. 3) A custom piano-specific neural audio tokenizer (PianoPoet DAC) employing 64-bit binary quantization (1-bit FSQ), 48 kHz stereo support, mid-side Mel loss, and semantic distillation, operating at 3.2 kbps.
Validation is performed on MAESTRO v3 and MAPS. For transcription, PianoPoet sets a new state-of-the-art among generative baselines (97.94% Onset F1, 91.50% Onset & Offset F1) and approaches top discriminative models. For rendering, it outperforms prior models on distribution metrics (FD, MMD), LPAPS, and subjective listening quality (MUSHRA 79.4, rendering quality 4.15/5), while exhibiting negligible negative transfer compared to its single-task counterparts.
Critical strengths
- Effective Representation Decoupling for Bidirectional Unification: Prior unified models struggled because they forced symmetric audio tokenization across both directions. PianoPoet's realization that transcription demands continuous semantic frames while rendering requires compact discrete acoustic tokens—while sharing the symbolic MIDI tokenizer—effectively resolves this bottleneck. This enables a single 113M causal Transformer to achieve state-of-the-art generative transcription while maintaining top-tier rendering without cross-task interference.
- High-Fidelity Stereo Codec and Controllable Acoustic Rendering: The design of PianoPoet DAC (64-bit binary quantizer, 48 kHz stereo, mid-side Mel loss) combined with 5-second acoustic prompt conditioning is practically impactful. It achieves superior human MUSHRA ratings (79.4) and enables zero-shot stereo panning and acoustic environment control, addressing a major limitation in previous mono-centric piano synthesis systems.
Critical weaknesses
- Substantial Deficit in Rendered Note Onset Accuracy: While PianoPoet excels in perceptual and distribution metrics (FD, MMD, LPAPS), its rendered note-level Onset F1 is 70.47% (Table 3), trailing specialized baselines like P-MUSE (90.84%) and MIDI-VALLE (81.19%) by a wide margin. In acoustic rendering, precise rhythmic attack and onset crispness are paramount to professional piano timbre; this gap indicates that discrete autoregressive token prediction introduces timing jitter or smeared transients that are not fully captured by distribution metrics.
- Narrow Domain Scope and Missing Inference Complexity Profiling: The framework is exclusively tuned to solo piano music, leveraging piano-specific tokenizers and datasets (MAESTRO). The paper does not explore whether this decoupling paradigm generalizes to multi-instrument or vocal-instrumental music. Furthermore, the paper omits runtime efficiency benchmarks (Real-Time Factor, GPU VRAM usage) for generating long-form 48 kHz stereo audio via autoregressive token sampling.
Questions for authors
- What is the primary cause of the low note-level Onset F1 (70.47% vs. 90.84% for P-MUSE) in Table 3? Is this due to the 50 Hz (20 ms) time resolution of the neural audio tokenizer, or due to exposure bias/sampling temperature during autoregressive token generation?
- What is the generation speed (Real-Time Factor) and GPU memory consumption when rendering a full 10-to-20 minute track on standard hardware (e.g., an NVIDIA RTX 4090 or A100)? Does the sliding-window mechanism accumulate drift or audible seams across window boundaries?
Responses to Reviewer xA2L
Thank you for your positive review of our paper. We address your questions in order:
As we mention in the related work (Line128), P-MUSE renders MIDI into audio by directly constructing spectrograms with a diffusion model. This side-steps using a neural audio codec, as in U-MUST, MIDI-VALLE, and PianoPoet.
We conjecture that due to the information bottleneck introduced from the neural audio codec (approximately 3kbps), some timing information is harder to learn for PianoPoet.
By setting the tolerance from 50ms to 100ms following Reviewer PQdW’s advice, PianoPoet gets on par with MIDI-VALLE, on top of its advantages in other metrics.
This presents a trade-off: P-MUSE and MIDI-VALLE maintain a strong onset F1 score but perform worse in other metrics including onset+offset F1 score and subjective quality.
As this point was mentioned by all other reviewers, we have performed followup analysis on this point in the general response to all reviewers. We hope this satisfies your concerns on this point.
We measured rendering speed and GPU memory usage on a NVIDIA H100 using a 24-minute track, excluding file I/O and model weight loading. RTF is generation time divided by audio duration.
PianoPoet achieves the lowest RTF and GPU memory usage among the evaluated autoregressive model. We could not evaluate P-MUSE under the same conditions because its implementation is not publicly available and the inference results were kindly provided by P-MUSE team.
Our sliding-window inference mechanism was designed to avoid audible seams when rendering long performances. The onset F1 scores in Table 3 provide quantitative evidence of timing alignment, while the supplementary audio samples allow assessment of continuity across window boundaries for 30s excerpts (due to size limitation, we chose to upload 30s excerpts). *Please see our general response for further details of the inference algorithm.
Thank you for pointing this out. We would like to expand similar investigations toward multi-instrument and vocal-instrument music in our future works.