acceptodds
Under review as a conference paper at ICLR 2027

PianoPoet: Unifying Piano Transcription and Rendering via Autoregressive Modeling

Abstract

Piano transcription and rendering connect the acoustic and symbolic domains of music in opposite directions; however, unified models that can perform both tasks lag behind task-specific methods. One approach to unification is to formulate both tasks as inverse sequence-to-sequence problems which share a token representation space; however, audio input and output formats appropriate for the tasks differ significantly. In this paper, we investigate this challenge and present PianoPoet, a single-checkpoint autoregressive framework that unifies both tasks with a shared transformer backbone. PianoPoet shares a common MIDI tokenizer for both directions while decoupling audio representations: continuous semantic audio frames for fine-grained transcription, and a piano-specific discrete audio tokenizer for realistic piano rendering. PianoPoet outperforms prior unified models, achieves the best results among the evaluated generative transcription methods, and outperforms specialized rendering methods in most evaluated metrics. It supports full-band stereo rendering, enabling stereo-position control through acoustic prompts. Ablations show that the proposed PianoPoet framework allows both directions to share a common backbone without quality degradation, maintaining performance comparable to its single-task variants.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
piano transcription, piano rendering, MIDI-to-audio, auto-regressive audio-symbolic modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xA2LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PianoPoet, a single-checkpoint autoregressive causal Transformer framework that unifies piano transcription (audio-to-MIDI) and piano rendering (MIDI-to-audio). While prior unified models (e.g., U-MusT) required separate checkpoints and lagged behind specialized methods, PianoPoet achieves competitive performance across both tasks within a single 113M-parameter model. The core technical innovation lies in representation decoupling: 1) A shared MIDI tokenizer encoding paired pitch-velocity, 10 ms hybrid timestamps, and pedal tokens. 2) Continuous semantic audio frames (50 Hz, frozen Conformer) for fine-grained transcription. 3) A custom piano-specific neural audio tokenizer (PianoPoet DAC) employing 64-bit binary quantization (1-bit FSQ), 48 kHz stereo support, mid-side Mel loss, and semantic distillation, operating at 3.2 kbps.

Validation is performed on MAESTRO v3 and MAPS. For transcription, PianoPoet sets a new state-of-the-art among generative baselines (97.94% Onset F1, 91.50% Onset & Offset F1) and approaches top discriminative models. For rendering, it outperforms prior models on distribution metrics (FD, MMD), LPAPS, and subjective listening quality (MUSHRA 79.4, rendering quality 4.15/5), while exhibiting negligible negative transfer compared to its single-task counterparts.

Critical strengths
  1. Effective Representation Decoupling for Bidirectional Unification: Prior unified models struggled because they forced symmetric audio tokenization across both directions. PianoPoet's realization that transcription demands continuous semantic frames while rendering requires compact discrete acoustic tokens—while sharing the symbolic MIDI tokenizer—effectively resolves this bottleneck. This enables a single 113M causal Transformer to achieve state-of-the-art generative transcription while maintaining top-tier rendering without cross-task interference.
  2. High-Fidelity Stereo Codec and Controllable Acoustic Rendering: The design of PianoPoet DAC (64-bit binary quantizer, 48 kHz stereo, mid-side Mel loss) combined with 5-second acoustic prompt conditioning is practically impactful. It achieves superior human MUSHRA ratings (79.4) and enables zero-shot stereo panning and acoustic environment control, addressing a major limitation in previous mono-centric piano synthesis systems.
Critical weaknesses
  1. Substantial Deficit in Rendered Note Onset Accuracy: While PianoPoet excels in perceptual and distribution metrics (FD, MMD, LPAPS), its rendered note-level Onset F1 is 70.47% (Table 3), trailing specialized baselines like P-MUSE (90.84%) and MIDI-VALLE (81.19%) by a wide margin. In acoustic rendering, precise rhythmic attack and onset crispness are paramount to professional piano timbre; this gap indicates that discrete autoregressive token prediction introduces timing jitter or smeared transients that are not fully captured by distribution metrics.
  2. Narrow Domain Scope and Missing Inference Complexity Profiling: The framework is exclusively tuned to solo piano music, leveraging piano-specific tokenizers and datasets (MAESTRO). The paper does not explore whether this decoupling paradigm generalizes to multi-instrument or vocal-instrumental music. Furthermore, the paper omits runtime efficiency benchmarks (Real-Time Factor, GPU VRAM usage) for generating long-form 48 kHz stereo audio via autoregressive token sampling.
Questions for authors
  1. What is the primary cause of the low note-level Onset F1 (70.47% vs. 90.84% for P-MUSE) in Table 3? Is this due to the 50 Hz (20 ms) time resolution of the neural audio tokenizer, or due to exposure bias/sampling temperature during autoregressive token generation?
  2. What is the generation speed (Real-Time Factor) and GPU memory consumption when rendering a full 10-to-20 minute track on standard hardware (e.g., an NVIDIA RTX 4090 or A100)? Does the sliding-window mechanism accumulate drift or audible seams across window boundaries?
1 comment
AuthorsOpenReview ↗

Responses to Reviewer xA2L

Thank you for your positive review of our paper. We address your questions in order:

W1&Q1: What is the primary cause of the low note-level Onset F1 (70.47% vs. 90.84% for P-MUSE) in Table 3? Is this due to the 50 Hz (20 ms) time resolution of the neural audio tokenizer, or due to exposure bias/sampling temperature during autoregressive token generation?

As we mention in the related work (Line128), P-MUSE renders MIDI into audio by directly constructing spectrograms with a diffusion model. This side-steps using a neural audio codec, as in U-MUST, MIDI-VALLE, and PianoPoet.

We conjecture that due to the information bottleneck introduced from the neural audio codec (approximately 3kbps), some timing information is harder to learn for PianoPoet.

By setting the tolerance from 50ms to 100ms following Reviewer PQdW’s advice, PianoPoet gets on par with MIDI-VALLE, on top of its advantages in other metrics.

This presents a trade-off: P-MUSE and MIDI-VALLE maintain a strong onset F1 score but perform worse in other metrics including onset+offset F1 score and subjective quality.

As this point was mentioned by all other reviewers, we have performed followup analysis on this point in the general response to all reviewers. We hope this satisfies your concerns on this point.

Q2: What is the generation speed (Real-Time Factor) and GPU memory consumption when rendering a full 10-to-20 minute track on standard hardware (e.g., an NVIDIA RTX 4090 or A100)? Does the sliding-window mechanism accumulate drift or audible seams across window boundaries?

We measured rendering speed and GPU memory usage on a NVIDIA H100 using a 24-minute track, excluding file I/O and model weight loading. RTF is generation time divided by audio duration.

ModelParametersToken rateRTFGPU memory (MB)
U-MusT350M86 Hz1.5444,630
MIDI-VALLE350M50 Hz0.7735,350
PianoPoet113M50 Hz0.5951,830
PianoPoet (render-only, large)300M50 Hz0.7402,534

PianoPoet achieves the lowest RTF and GPU memory usage among the evaluated autoregressive model. We could not evaluate P-MUSE under the same conditions because its implementation is not publicly available and the inference results were kindly provided by P-MUSE team.

Our sliding-window inference mechanism was designed to avoid audible seams when rendering long performances. The onset F1 scores in Table 3 provide quantitative evidence of timing alignment, while the supplementary audio samples allow assessment of continuity across window boundaries for 30s excerpts (due to size limitation, we chose to upload 30s excerpts). *Please see our general response for further details of the inference algorithm.

W2: The framework is exclusively tuned to solo piano music, leveraging piano-specific tokenizers and datasets (MAESTRO). The paper does not explore whether this decoupling paradigm generalizes to multi-instrument or vocal-instrumental music.

Thank you for pointing this out. We would like to expand similar investigations toward multi-instrument and vocal-instrument music in our future works.

Reviewer_GbcdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PianoPoet, a single-checkpoint autoregressive framework for both audio-to-MIDI transcription and performance MIDI-to-audio rendering. PianoPoet DAC is proposed to support stereo piano rendering. Experimental results demonstrate competitive performance on both tasks, with particularly strong transcription results among the evaluated generative baselines.

Critical strengths
  1. The framework supports both piano transcription and rendering with a shared autoregressive backbone. The transcription ablations show that the unified model maintains performance close to its transcription-only counterpart.

  2. PianoPoet DAC is specifically designed for stereo piano audio and demonstrates promising perceptual quality at a low bitrate. It could be useful for future piano rendering systems.

  3. The evaluation covers a broad range of objective metrics, subjective quality assessments, and ablations of the audio representations and rendering components.

Critical weaknesses
  1. The full-track rendering procedure is not clearly specified. The authors state that PianoPoet renders complete tracks rather than short clips, but training uses a maximum sequence length of 3072. Please explain how longer sequences are handled during inference and how continuity across segments is maintained if chunked generation is used.

  2. Rendering onset F1 (70.47, Table 3) is substantially below P-MUSE’s 90.84. The discussion of Transkun’s preference for mechanical renderings does not sufficiently explain this gap. The codec’s reconstruction onset F1 of 87.36 (Table 5) and the render-only model’s score of 73.41 (Table 6) suggest possible contributions from codec distortion and joint training, but these effects are not clearly separated.

  3. The transcription comparisons use different training data. PianoPoet uses 590 additional hours of synthetic audio and a pretrained Conformer encoder. Table 4 shows that the Mel variant without synthetic data performs worse than MT3-Piano. The paper does not evaluate the final model with both encoder pretraining and model training restricted to MAESTRO. It is therefore unclear how much of the improvement comes from the architecture rather than additional data.

Minor suggestion

Figure 1 could more clearly convey the sequence formulation. Placing inputs and outputs below and above the backbone may suggest an encoder–decoder architecture. Showing them as a single concatenated sequence for each task, with conditioning and prediction regions clearly marked, would better illustrate the causal Transformer design.

Questions for authors
  1. How does PianoPoet render full tracks beyond the training sequence length?
  2. What explains the substantially lower rendering onset F1 compared with P-MUSE?
  3. How does the final model perform when trained only on MAESTRO, including semantic encoder pretraining?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer Gbcd

Thank you for your positive review of our paper. We address your questions and concerns in order:

W1 & Q1: How does PianoPoet render full tracks beyond the training sequence length?

Our model renders and transcribes 15-second windows. For longer recordings, we advance the window by 5 seconds at each step, conditioning on previously generated output in the overlapping region. We plan to update the appendix in the camera ready. We encourage the reviewer to visit our general response to reviewers where we provide more detail and examples both full-track transcription and rendering.

W2 & Q2: What explains the substantially lower rendering onset F1 compared with P-MUSE?

As we mention in the related work (Line128), P-MUSE renders MIDI into audio by directly constructing spectrograms with a diffusion model. This side-steps using a neural audio codec, as in U-MUST, MIDI-VALLE, and PianoPoet.

We conjecture that due to the information bottleneck introduced from the neural audio codec (approximately 3kbps), some timing information is harder to learn for PianoPoet.

Nevertheless, by setting the tolerance from 50ms to 100ms, PianoPoet’s onset F score gets on par with MIDI-VALLE, on top of its advantages in other metrics.

This presents a trade-off: P-MUSE and MIDI-VALLE maintain a strong onset F1 score but perform worse in other metrics including onset+offset F1 score and subjective quality.

As this point was mentioned by all other reviewers, we have performed followup analysis on this point in the general response to all reviewers. We hope this satisfies your concerns on this point.

W3&Q3: How does the final model perform when trained only on MAESTRO, including semantic encoder pretraining? The reviewer had the following concern: “Table 4 shows that the Mel variant without synthetic data performs worse than MT3-Piano. The paper does not evaluate the final model with both encoder pretraining and model training restricted to MAESTRO. It is therefore unclear how much of the improvement comes from the architecture rather than additional data.”.

To address this we have trained a transcription model in the same setup as PianoPoet, but on the MAESTRO dataset without extra data (including for semantic encoder pretraining). The results are presented in the following table:

ModelData aug.SynthOnset F1Onset & offset F1Onset & offset & velocity F1
MT3-Pianon/aNo96.0183.9482.75
Mel (MAESTRO-only)YesNo95.9881.1578.92
Frozen pretrained encoder (MAESTRO-only)YesNo96.7787.1185.70

This shows that even when restricted to only the MAESTRO dataset, our final PianoPoet setup performs better than MT3-Piano, however we note that it still lags behind our model trained with additional synthetic data. We hope this addresses your concern.

suggestion on Figure1

We sincerely appreciate your advice. We would like to reflect your opinion and make a better version in the camera-ready.

Thank you for the detailed response. The clarification of the full-track inference procedure, the additional onset analysis, and the MAESTRO-only experiment address my concerns. I will keep my score.

Thank you, Reviewer Gbcd, to confirm the most concerns have been addressed. We appreciate your supports.

Reviewer_PQdWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

PianoPoet uses a single autoregressive Transformer for both piano transcription and MIDI-to-audio rendering. The model shares its MIDI representation while separating the audio representations used in the two directions: continuous features from a frozen Conformer for transcription, and a purpose-built discrete stereo codec for rendering. The evaluation covers transcription, full-track rendering, listening tests, historical recording restoration, and stereo panning control.

Critical strengths
  1. The separation of audio representations is well supported. In Table 4, transcription with discrete acoustic tokens gives an onset F1 of 79.00%, compared with 97.94% using continuous features.
  2. The piano-specific stereo codec is valuable in its own right. Mid-side losses, binary quantization, and semantic distillation support high-quality reconstruction at a low bitrate. At 3.2 kbps, the codec achieves a MUSHRA score of 79.4, outperforming the general-purpose codecs included in the comparison.
  3. The evaluation covers several dimensions of quality. Beyond standard transcription metrics, the paper reports perceptual distance, note and pedal accuracy, dynamic correlation, and listening-test results.
Critical weaknesses
  1. Joint training does not yet show a clear benefit to either task. Transcription onset F1 is 97.94% for the unified model and 97.99% for the single-task model; rendering onset F1 is 70.47% versus 73.41%. These results support the feasibility of parameter sharing, but not a performance benefit from joint training.
  2. The handling of long recordings is insufficiently explained. Training uses a maximum sequence length of 3,072 tokens, while full-track evaluation includes pieces lasting more than 20 minutes. The segmentation, stitching, prompt updates, and context management need a clearer description.
Questions for authors
  1. Could the low onset F1 be caused partly by local timing drift? How do the results change with a wider onset tolerance than 50 ms, or after DTW alignment?
  2. Are the Conformer used for transcription and the semantic-distillation encoder the same model? What pretraining data were used, and did they include MAESTRO or synthetic audio generated from Aria-MIDI?
2 comments
AuthorsOpenReview ↗

Responses to Reviewer PQdW (Part 1)

Thank you for your positive review of our paper.We address your questions and concerns in order:

W1: Joint training does not yet show a clear benefit to either task… These results support the feasibility of parameter sharing, but not a performance benefit from joint training.

This observation is consistent with our findings, and we hope to investigate synergy in future works.

We scope our contributions as unifying the transcription and rendering framework within a single modeling framework (Line 77-80) without comprising the performance of each task.

W2: The handling of long recordings is insufficiently explained.

Our model renders and transcribes 15-second windows. For longer recordings, we advance the window by 5 seconds at each step, conditioning on previously generated output in the overlapping region. We plan to update the appendix in the camera ready version.

We encourage the reviewer to visit our general response to reviewers where we provide more detail and examples both full-track transcription and rendering.

AuthorsOpenReview ↗

Responses to Reviewer PQdW (Part 2)

Q1: Could the low onset F1 be caused partly by local timing drift? How do the results change with a wider onset tolerance than 50 ms, or after DTW alignment?

We have re-computed our note onset and note onset-and-offset F1 scores for compared rendering models with 100ms tolerance. We present the results in the following table:

Onset F-score (%) at different onset tolerances.

Onset tolerancePianoPoet (unified model in the paper)MIDI-VALLEP-MUSE
25ms20.6178.5390.13
50ms (default, used in the paper)70.4781.1990.84
100ms81.0682.1190.98
200ms82.5682.9791.07

This is consistent with timing jitter being partially responsible for the degraded note-onset F1. We additionally note that PianoPoet is best-in-class for note onset-and-offset accuracy as in Table 3. Please see the general response to all reviewers for more details on our investigation.

Q2: Are the Conformer used for transcription and the semantic-distillation encoder the same model? What pretraining data were used, and did they include MAESTRO or synthetic audio generated from Aria-MIDI?

The Conformer checkpoint used for the semantic-distillation codec is an earlier checkpoint of the one used in transcription, as the codec training took a longer time. In the manuscript both the transcription models and the Conformer were trained on MAESTRO as well as synthetic audio from Aria-MIDI.

To isolate the effect of the synthetic data we have trained a transcription model in the same setup as PianoPoet, but on the MAESTRO dataset alone (including for semantic encoder pretraining). The results are presented in the following table:

ModelData aug.SynthOnset F1Onset & offset F1Onset & offset & velocity F1
MT3-Pianon/aNo96.0183.9482.75
Mel (MAESTRO-only)YesNo95.9881.1578.92
Frozen pretrained encoder (MAESTRO-only)YesNo96.7787.1185.70

We hope this adds useful context to the impact of synthetic data.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to common questions and weaknesses for all reviewers (Part 1)

We thank the reviewers for their time, as well as for their positive and constructive comments about our submission. In this general response we will respond to questions that the reviewers had in common.

Q1: More information about the inference algorithm for long recordings.

We provide an overview of the full-track inference algorithm for MIDI-like tokens in Appendix A lines 793-798 of the original manuscript. In the camera-ready version, we will update the wording for clarity.

Our model renders and transcribes 15-second windows. For longer recordings, we use the overlapping-window strategy described in Appendix A: we advance the window by 5 seconds at each step, conditioning on previously generated output in the overlapping region. Interior windows generate the central 5 seconds, using 5 seconds of context on either side. The first window generates the initial 10 seconds, and the final window generates the remaining 10 seconds.

The timelines below illustrate this procedure. All intervals are in seconds and refer to positions in the full recording, whose duration is T.

Transcription

Window   Audio input       MIDI context      Generated MIDI    Appended MIDI
0        [0:15]            —                 [0:10]            [0:10]
1        [5:20]            [5:10]            [10:15]           [10:15]
2        [10:25]           [10:15]           [15:20]           [15:20]
...
Last     [T−15:T]          [T−15:T−10]       [T−10:T]          [T−10:T]

Rendering

Window   MIDI input        Codec context     Generated audio   Appended audio
0        [0:15]            —                 [0:10]            [0:10]
1        [5:20]            [5:10]            [10:15]           [10:15]
2        [10:25]           [10:15]           [15:20]           [15:20]
...
Last     [T−15:T]          [T−15:T−10]       [T−10:T]          [T−10:T]

For rendering, codec context consists of previously generated audio codec tokens. The generated codec tokens are decoded into audio. Note that we do not change the acoustic prompt during full-track rendering, providing consistent acoustic conditioning to the model.

AuthorsOpenReview ↗

General response to common questions and weaknesses for all reviewers (Part 2)

Q2: In the rendering benchmarks in Table 3, why are the render-transcribe benchmarks lower for our model compared to two specific models?

Although the results for PianoPoet on note onset F1 score is below MIDI-VALLE and P-MUSE, we would like to remind reviewers that its other scores, including note onset+offset F1 score, are better. This presents a trade-off: P-MUSE and MIDI-VALLE maintain a strong onset F1 score but perform worse in other metrics including onset+offset F1 score and subjective quality.

Having said this, reviewers are correct to point out this disparity in the onset F1 score. We have investigated this during the rebuttal phase in two ways:

  1. We investigate if the note-onset F1 metric improves with model scale.
    • P-MUSE and MIDI-VALLE are 481M and 350M parameters respectively, while PianoPoet had 113M parameters.
    • We have trained a larger rendering model with ~300M parameters on the MAESTRO dataset, keeping other training conditions the same as our rendering ablations.
  2. Following the suggestion of reviewer PQdW, we have re-run our MIR-eval metrics for a larger onset accuracy tolerance of 100ms (compared to the 50ms default for MIR-eval). Here we investigate whether the note-onset F1 metric is due to missing / incorrect notes, or due to timing jitter.

We summarize the results in the following table:

EvaluationSystemDefault 50ms: onset F1Default 50ms: onset+offset F1Relaxed 100ms: onset F1Relaxed 100ms: onset+offset F1
RenderingMIDI-VALLE81.1933.7982.1141.38
RenderingP-MUSE90.8432.4590.9740.32
RenderingPianoPoet70.4740.7981.0657.08
RenderingPianoPoet (larger, render/MAESTRO-only)75.0845.0883.3660.07

These results show that:

  • PianoPoet’s onset F1 score benefits from a relaxed tolerance more than the two compared models, indicating a timing jitter in its output.
  • Increasing the model size to 300m improved performance. At 100ms tolerance, our 100M model is close to MIDI-VALLE, the 300M model outperforms MIDI-VALLE. However, at 50ms tolerance, it is still behind MIDI-VALLE.
  • PianoPoet always leads in the onset+offset F1 score, which is also important.

Audio formats. P-MUSE produces audio at 24kHz mono, MIDI-VALLE produces audio at 32kHz mono, but PianoPoet works at 48kHz stereo. In the future, we would like to investigate methods including reinforcement learning to improve onset clarity without sacrificing other metrics.

Information bottleneck. Moreover, P-MUSE does not use a neural audio codec, instead constructing spectrograms directly, and therefore avoids the information bottleneck introduced of approximately 3 kbps. This may help it preserve precise onset information. Future work could investigate this hypothesis by using higher codec bitrates.

Summary: although PianoPoet has lower onset-only F1 at the default tolerance, it achieves the highest onset-and-offset F1 among the compared rendering models and leads on all other metrics reported in Table 3. At a 100 ms onset tolerance, its note-onset F1 also approaches that of MIDI-VALLE which is the only other competitive model that uses a neural audio codec.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.