acceptodds
Under review as a conference paper at ICLR 2027

Streaming Audio-to-Score Transcription

Abstract

We present the first streaming music transcription system that converts live audio directly into human-readable sheet music. Our system is comprised of a cache-aware Conformer–RNN-T that emits a playback-ordered Intervals-and-Moments (InterMo) notation sequence in real-time, with an incremental sheet music rendering system for low latency. Our model emits InterMo tokens as solo piano, solo violin, and violin–piano duet performances unfold, directly producing time-aligned scores spanning up to three staves, without intermediate stages, updated at each musical moment. We benchmark streaming transcription on different latency configurations between 0.16 and 2.24 seconds, and show that our streaming model performs competitively with conventional offline audio-to-MIDI-to-score cascades. We additionally introduce a musically motivated *conductor* buffer, and show that a startup delay of only 5 seconds substantially increases streaming quality, especially in piano music. By bringing metrically structured, polyphonic notation into the streaming setting, our approach moves real-time music transcription toward human-facing applications in which live performances can be immediately captured, read, and communicated as conventional musical scores.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
automatic music transcription, streaming transcription, audio-to-score transcription, sequence transduction, symbolic music

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p2NyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes the first streaming audio-to-score transcription system. The system reuses the InterMo notation proposed in Rubato and adopts a Conformer–RNN-T to generate it in real time with low latency. The model is trained and evaluated on solo piano, solo violin, and violin–piano duets. The system is benchmarked at different latencies and shows competitive results compared to offline systems. The paper also proposes a buffer that delays the start of transcription, which improves streaming quality.

Critical strengths
  1. Novel and under-explored task. To my knowledge, transcribing audio into a human-readable score online has not been done before. Prior online work stops at MIDI events or beat positions, so formulating the task, and showing that multi-staff notation can be produced live, is a useful contribution.
  2. Simple and effective conductor buffer. The conductor buffer is easy to implement and clearly effective. Without it, many low-latency outputs cannot be converted into a valid score (e.g., 18 of 34 URMP files at 0.16 s; Table 3), and nearly all of these failures contain no bar lines. With the buffer, conversion failures essentially disappear, and OMR-NED improves on piano at nearly every lookahead.
Critical weaknesses
  1. Missing background on Rubato. The proposed system builds heavily on Rubato, but the paper never explains it properly. InterMo gets one sentence (l. 155–158), IntervalPiece is never defined, and the synthesis pipeline is only cited (l. 262–263). A preliminary section introducing Rubato and these components would make the paper self-contained.
  2. No streaming baseline. Although this task may not have been explored before, a cascade of online MIDI transcription (e.g., Wei et al., 2024; Kusaka & Maezawa, 2024) followed by online beat tracking (e.g., Meier et al., 2024) and quantization could serve as a baseline. Alternatively, running Rubato on short sliding windows at matched latency would be a valuable comparison. Without such a baseline, it is unclear whether the end-to-end approach outperforms a streaming cascade.
  3. Limited technical novelty. The streaming Conformer and cache-aware inference come from prior work, and InterMo, IntervalPiece and the synthesis pipeline come from Rubato. The new contributions are mainly the integration, the conductor-buffer heuristic and the incremental renderer.
Questions for authors
  1. For comparison with Rubato, do you use the same training data as the proposed system?
  2. The score updates “after every completed InterMo moment”, but some tokens merge an interval with the start of the next moment. How does the system detect that a moment is complete? If it waits for the next interval token, the display would likely to lag by up to one gap between notes.
  3. A large portion of training data are synthetic, would it be better to train the model with MAESTRO as in Rubato?
2 comments

We thank the reviewer. The revision (changes in blue) keeps the architecture and changes the training: the three separately trained models of the submission are replaced by one, trained offline and then adapted to streaming with the conductor buffer sampled between 0 and 30 s. This single checkpoint serves every lookahead and offline decoding, improves on the submitted results (pooled OMR-NED 89.8→87.5 at 0.16 s, 83.9→80.2 offline), and works with or without the buffer (without it: 1 invalid score of 1,701 at 0.16 s, vs 66 for the model trained without a buffer).

1. Background. The revision adds a paragraph on the components we take from Rubato (Section 3.1): in InterMo, a score is a sequence of records, each a metric interval followed by the notes that start or stop at that moment; IntervalPiece is a BPE vocabulary over InterMo whose merges respect the record structure, adapted from BoundlessBPE.

2. Streaming baseline. Online MIDI transcription plus online beat tracking is not a score transcriber: producing notation would also need online quantization, key, time-signature, staff and voice estimation and an incremental score converter, none of which exists. More importantly, such a cascade cannot be expected to outperform its own offline version, which has future context at every stage. We already compare against offline cascades of this kind (a 1.3B-parameter transcriber plus trained converters), and our streaming model matches or beats them. Building the missing online components is interesting future work, for which our system can serve as the streaming baseline (OMR-NED↓):

ATEPPASAPURMPDuet
Ours, 0.16 s87.884.479.980.0
Ours, 2.24 s82.976.479.071.5
MuScriptor → M2ST87.684.8*95.192.4
MuScriptor → APT-EPR87.883.8*92.990.6

*The cascades' converters were trained on ASAP test data (point 5).

We also ran the reviewer's sliding-window suggestion: Rubato on non-overlapping 2.24 s windows (our longest lookahead) is worse than our 2.24 s model on all six sets we ran (OMR-NED, lower is better, Rubato vs. ours: ASAP 78.8 vs 76.4, KRAISLER piano 79.8 vs 77.0, duet 91.9 vs 71.5, violin 95.9 vs 83.1, URMP 92.3 vs 79.0, Bach10 93.6 vs 80.1).

3. Novelty. The encoder, cache-aware inference, InterMo, IntervalPiece and the synthesis come from prior work. Our contributions: (i) the task and a complete system: a transducer emits multi-staff notation directly in real time, with no note-detection, beat-tracking, quantization, staff-separation, key or meter stages, several of which have no online version; (ii) the conductor buffer, which the reviewer calls a heuristic. In the original submission it was a fixed inference setting; in the revision it is sampled during training, and with this training one model also serves offline decoding and still works without the buffer; (iii) incremental engraving, whose final score equals the offline conversion in 205/206 recordings. That a standard speech-recognition model family transcribes polyphonic multi-staff music in real time is itself a result, with implications for joint speech–music conversational AI without music-specific machinery.

4. When is a moment complete? The reviewer's reading is right: 1/8 PL: a-3 c4 opens a new record and thereby closes the previous one. But the display does not wait: each note is drawn in gray when its own token is emitted, 0.21 s after it is played (median at 0.16 s); only the written duration waits, becoming final 0.24 s after the note ends; no causal system can fix a duration before the note ends. Section 3.3 is rewritten accordingly.

5. ASAP overlap. The training data of the cascades' MIDI-to-score converters (M2ST, APT-EPR) include performance–score pairs from the ASAP test set, which favours them on ASAP; neither Rubato nor our model has this overlap. Our model still matches the cascades at 0.16 s and leads from 0.56 s.

6. Training data and Rubato. Same data? No. Rubato is trained on piano only, with more real piano audio, including MAESTRO through an auxiliary MIDI-transcription task. We use 48 h of real audio and also transcribe violin and violin–piano duets, which is why we also compare with the multi-instrument MuScriptor. Rubato is an offline attention encoder–decoder with a decoder that has 12× as many parameters as ours (75.9M vs 6.3M); we are 4.5× faster on GPU and over 6× faster on CPU. As in speech recognition, transducers serve streaming and on-device use, and attention decoders serve offline use. Within that trade-off we exceed Rubato on spelled pitch and staff (81.9 vs 79.5; 76.4 vs 74.8; higher is better), match it on exact durations (34.4 vs 34.6) and beat it by 8–12 points on all four violin/duet sets even at 0.16 s. We did not retrain either model on the other's data: Rubato's MAESTRO supervision is MIDI, which our transducer can use only with a multitask extension we have not explored, future work that may narrow the piano gap.

Reviewer_p2NyOpenReview ↗

I raised the score

I appreciate the authors' reply, it has resolved my questions, I will raise my score.

Reviewer_nzdXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a streaming audio-to-score transcription system that uses a cache-aware Conformer–RNN-T to generate InterMo representations for incremental score rendering. Experiments on piano, violin, and violin–piano performances demonstrate promising results compared with offline transcription systems.

Critical strengths
  1. The paper introduces a novel and challenging setting: streaming audio-to-score transcription, which requires recovering meter, rhythm, voices, measures, and staff structure from audio in real time.

  2. The evaluation covers multiple instruments and several real-performance datasets. The paper is also transparent about conversion failures, dataset-specific behavior, and the remaining gap between streaming and offline systems.

Critical weaknesses
  1. The paper provides neither a demo nor sufficient qualitative examples of generated scores. Readability is an important aspect of score transcription but is difficult to assess using OMR-NED alone. The paper would benefit from representative successful and failure cases, preferably showing how the displayed score evolves over time.

  2. As acknowledged in the limitations, the paper reports only theoretical latency rather than actual end-to-end latency. Since real-time transcription is the paper’s core contribution, the authors should measure the full latency, including decoding, score conversion, and rendering.

  3. It would be better to include decomposed sub-metrics, such as pitch, onset, duration, etc., as in [1,2] to help better understand different aspect of transcription performance.

Reference

[1] Nakamura, Eita, Emmanouil Benetos, Kazuyoshi Yoshii, and Simon Dixon. "Towards complete polyphonic music transcription: Integrating multi-pitch detection and rhythm quantization." In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 101-105. IEEE, 2018.

[2] Hiramatsu, Y., Nakamura, E. and Yoshii, K., 2021, November. Joint Estimation of Note Values and Voices for Audio-to-Score Piano Transcription. In ISMIR (pp. 278-284).

Questions for authors
  1. What application scenarios motivate the proposed streaming system, and what latency would be acceptable in those scenarios?

  2. How readable are the generated scores in practice? Could the authors provide representative predicted scores and discuss pitch, rhythm, meter, voice-assignment, and layout errors separately?

2 comments

We thank the reviewer. One change affects all numbers below: instead of three separately trained models, the revision (changes in blue) uses a new training procedure that yields a single model of the same architecture: it is trained offline first, then adapted to streaming with the conductor buffer sampled between 0 and 30 s. It serves every lookahead and offline decoding, and it emits notes sooner than the submitted model (point 2). Italic rows are the submitted model.

1. Applications and acceptable latency. Our primary motivating application is real-time score transcription for music education and accessibility. For example, during an online music class, a student could follow the written notation as music is performed, without requiring a reference score in advance. A longer-term application is interactive music interfaces, where live transcription provides a symbolic representation that can subsequently be used for arrangement, editing, or conversational interaction; we deliberately use a conventional streaming speech-recognition architecture to facilitate future integrated music–speech conversational AI. We have not yet conducted a user study to establish an acceptable latency threshold. In the online class, we envision the transcription being ready shortly after the teacher finishes playing and begins speaking, suggesting that a delay on the order of one second is acceptable; at 1.20 s our model is already ahead of every offline cascade on every pooled metric. During continuous performance, a short and predictable latency matters most; the lookahead is chosen at run time, down to 0.16 s. We note that applications that offer anything close to real-time musical transcription to a score are not yet widely available commercially.

2. Measured end-to-end latency (102 real ASAP recordings, 0.16 s lookahead, 5 s buffer; medians, [95th percentile]):

played → engravedemissionchunk waitdecoderender
Ours0.21 s [0.31]0.12 s0.04 s0.02 s0.01 s
Submitted0.27 s [0.39]0.17 s0.04 s0.02 s0.01 s

At 0.16 s lookahead, a played note reaches the screen 0.21 s later: i.e. about 50 ms beyond the 0.16 s lookahead. The revision reports this detail.

  • Decoding on one L40S takes 15–18 ms per 160 ms update (p99 ≤ 26.5 ms). The 8-bit model on four server-CPU threads takes 43.4 ms (p95 62.6 ms), with OMR-NED unchanged.
  • In a browser on a consumer laptop (Apple M4 Pro, Microsoft Edge), decoding takes a median of 46.4 ms per update (p95 58.7 ms; 0.07 % over 160 ms) and rendering 7.4 ms.
  • Rendering keeps up; the final incremental score equals the offline conversion in 205/206 recordings (all sets except ATEPP). A note is shown in gray as soon as it is emitted (0.21 s after it is played); its written duration becomes final 0.24 s after the note ends.
  • The largest term is the model's own emission delay, which the streaming adaptation shortened by about 55 ms. The revision reports these measurements alongside the lookahead-based latency.

3. Decomposed metrics. In the spirit of [1, 2], after a pitch alignment that ignores bar lines, we report a series of increasingly stringent F1 scores for pitch, spelling, staff, exact written duration (floating note) and position in the measure (in-bar note) — onset and duration in score time. At 2.24 s: 80.8 / 75.1 / 33.2 / 12.0 % (spelled / staff / floating / in-bar), above all offline MuScriptor cascades (≤78.9 / 71.8 / 27.9 / 4.9 %). Replacing our alignment with MV2H's changes F1 by a median 0.2 points. Full table: our response to Reviewer PJ2z.

4. Readability and examples. In most outputs, pitches, spelling and left/right hand assignment are correct; the errors a reader notices are note values and bar lines. A frequent case is a piece written in a related meter (e.g. 2/4 instead of 4/4): the notes are correct, but every bar line is shifted. In the worst outputs the bar grid breaks down, with far too few bar lines (e.g. 4 bars for a 27-bar piece) or far too many; on solo violin, fewer bar lines correlate with worse scores. Voices and layout are set by the engraving stage from the model's output, identically in streaming and offline. The supplementary material (also at https://streaming-audio-to-score.pages.dev/examples) shows both: for each test set, one representative recording (reference, ours, Rubato and both cascades side by side), successful and failure cases, and videos with audio of the score as it is written in real time. Live demo: https://streaming-audio-to-score.pages.dev.

Thank you for the detailed response. The new latency measurements and decomposed metrics address my concerns, and I found the demo quite convincing. I will keep my score.

Reviewer_PJ2zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an RNN-T-based model for streaming audio-to-score transcription. As the authors note, this is the first model to support streaming audio-to-score transcription, which is a significant contribution to the field. RNN-T is a suitable choice given the nature of the task, in which multiple output events can occur at a single timestep. Through cache-aware streaming with a conductor buffer, the model supports a range of inference configurations. Notably, it runs in real time on a consumer CPU. It is trained on both real and synthetic data. The experiments show that the model performs competitively against offline baselines, and that the conductor buffer improves OMR-NED.

Critical strengths
  • First streaming audio-to-score model, with an appropriate architectural choice. No streaming audio-to-score model existed before this work, and building one requires an architecture suited to the task. The choice of RNN-T appears well justified.
  • Informative ablations. The ablation studies on the conductor buffer and the lookahead help clarify which components drive the model’s performance.
Critical weaknesses
  • High absolute error and limited evaluation. The proposed model is competitive with state-of-the-art offline baselines, but the absolute OMR-NED values remain quite high. Complementary metrics such as note-level F1, or qualitative examples of model outputs, would help readers judge the practical quality of the transcriptions.
  • Limited relevance of Section 3.3. Section 3.3 describes the rendering pipeline, which involves no learning-related content and is therefore of limited relevance to ICLR. This section could be condensed considerably.
Questions for authors
  • How was the V=1800 no-MIDI IntervalPiece vocabulary constructed? Could this vocabulary design adversely affect model performance?
  • If the model mispredicts the time signature, note pitches and timings may still be correct, yet OMR-NED could degrade severely because notes are placed in the wrong measures. Could the authors discuss this issue and also report a note-level metric, such as note F1, to disentangle these effects?
1 comment

We thank the reviewer. Since submission, we improved the training curriculum. The submission reported training three separate models (one offline, two streaming with a fixed 0 s or 5 s conductor buffer). The revision (changes in blue) trains one: after offline training, the same model is adapted to streaming while the buffer is sampled between 0 and 30 s at every step (30 s is close to offline). This single checkpoint serves every lookahead and offline decoding and improves OMR-NED at every lookahead and offline. Bold rows: this checkpoint; italic rows: the submitted ones.

1. Meter-free F1 and the effect of time-signature errors. We introduce new evaluation measures. First, we align prediction and reference with an exact order-preserving alignment (Needleman–Wunsch) of simultaneous note groups by MIDI number, so bar lines, meter and spelling cannot affect the coarse scores. We then compute a series of F1 scores that proceed from simplest to hardest/more stringent: Frequency (MIDI number) → spelled pitch (letter, accidental, octave) → pitch with correct staff assignment → note with exact written duration (floating note) → note with a correct position in the measure (in-bar note). Per recording (ATEPP, all systems), OMR-NED correlates with the duration levels (Spearman 0.67 floating, 0.66 in-bar) far more than with the pitch levels (0.40–0.42), so it mainly measures rhythm and bar structure in this setting.

SystemAligned freq.↑Spelled pitch↑Staff assigned pitch↑Floating notes↑In-bar notes↑OMR-NED↓
Ours, 0.16 s78.475.368.324.06.187.5
Ours, 2.24 s83.580.875.133.212.082.4
Ours, offline84.181.976.434.414.580.2
Submitted, 0.16 s78.976.067.623.64.089.8
Submitted, offline82.279.872.929.88.883.9
Rubato (offline)84.579.574.834.621.575.1
MuScriptor → M2ST (offline)82.578.871.826.34.687.4
MuScriptor → APT-EPR (offline)82.878.971.227.94.987.5

F1 (%) is pooled over all seven test sets (1,701 recordings); unreadable outputs count every reference note as missed. Offline rows of our model use the ±21 s sliding window of offline training.

  • Pitch is mostly right; metrical placement is the hard part, for every system. At 2.24 s, spelled-pitch F1 is 80.8 and staff 75.1, higher than every offline baseline; requiring the exact duration drops it to 33.2 and the bar position to 12.0. The cascades fall the same way (to 26–28, then 4.6–4.9).
  • Streaming matches the offline cascades at 0.16 s and beats them from 0.24 s: OMR-NED 87.5 vs 87.4–87.5 at 0.16 s; paired difference −0.8 (95 % CI [−1.3, −0.3]) at 0.24 s; ahead on every column from 1.20 s.

Time signature. The data are consistent with the reviewer's hypothesis. We split the recordings by whether the first predicted time signature is exactly right (22% of recordings at 2.24 s). Those reach OMR-NED 75.2, the rest 84.3 (remember that higher is worse). In the second group the notes are largely correct, but a wrong meter shifts all bar lines, and OMR-NED, which compares the scores bar by bar, penalizes every shifted bar. Typical confusions share a pulse (2/4↔4/4, 3/4↔3/8). The submitted model defaulted to 4/4 (79 % of ATEPP recordings); after the new training the model chooses among many meters (e.g. BWV 873 in its correct 12/16).

Examples. The supplementary material (also at https://streaming-audio-to-score.pages.dev/examples) adds side-by-side excerpts of all systems for one representative recording per test set, successful and failure cases, and real-time videos; live demo: https://streaming-audio-to-score.pages.dev.

2. The V = 1800 IntervalPiece vocabulary. IntervalPiece is BPE with BoundlessBPE's completion-gated merging (Schmidt et al., 2025) extended to the nested InterMo grammar: BoundlessBPE lets merges cross pretoken boundaries once both adjacent pretokens are complete, and we apply this rule at every level of the grammar. Frequent combinations of complete musical units therefore become larger tokens, while incomplete units never cross their boundaries; no token extends beyond one metric interval and the staff content it introduces. "No-MIDI" excludes performance tokens (velocity, pedal). Tokenization is lossless, so the vocabulary affects performance mainly through sequence length; V = 1800 is the size at which 30 s segments fit the T × U × V transducer lattice in GPU memory. We preferred it to UnigramLM, which compresses out-of-domain data worse (a larger problem with several instruments), and to plain BPE, whose merges cross note boundaries: it learns tokens such as 4 E-4, which joins the octave digit ending one note with the next note, so a token no longer stands for a musical unit.

3. Section 3.3. Condensed in the revision to the mechanism (provisional notes in gray, finalised by later output); the rendering details are in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.