- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes the first streaming audio-to-score transcription system. The system reuses the InterMo notation proposed in Rubato and adopts a Conformer–RNN-T to generate it in real time with low latency. The model is trained and evaluated on solo piano, solo violin, and violin–piano duets. The system is benchmarked at different latencies and shows competitive results compared to offline systems. The paper also proposes a buffer that delays the start of transcription, which improves streaming quality.
Critical strengths
- Novel and under-explored task. To my knowledge, transcribing audio into a human-readable score online has not been done before. Prior online work stops at MIDI events or beat positions, so formulating the task, and showing that multi-staff notation can be produced live, is a useful contribution.
- Simple and effective conductor buffer. The conductor buffer is easy to implement and clearly effective. Without it, many low-latency outputs cannot be converted into a valid score (e.g., 18 of 34 URMP files at 0.16 s; Table 3), and nearly all of these failures contain no bar lines. With the buffer, conversion failures essentially disappear, and OMR-NED improves on piano at nearly every lookahead.
Critical weaknesses
- Missing background on Rubato. The proposed system builds heavily on Rubato, but the paper never explains it properly. InterMo gets one sentence (l. 155–158), IntervalPiece is never defined, and the synthesis pipeline is only cited (l. 262–263). A preliminary section introducing Rubato and these components would make the paper self-contained.
- No streaming baseline. Although this task may not have been explored before, a cascade of online MIDI transcription (e.g., Wei et al., 2024; Kusaka & Maezawa, 2024) followed by online beat tracking (e.g., Meier et al., 2024) and quantization could serve as a baseline. Alternatively, running Rubato on short sliding windows at matched latency would be a valuable comparison. Without such a baseline, it is unclear whether the end-to-end approach outperforms a streaming cascade.
- Limited technical novelty. The streaming Conformer and cache-aware inference come from prior work, and InterMo, IntervalPiece and the synthesis pipeline come from Rubato. The new contributions are mainly the integration, the conductor-buffer heuristic and the incremental renderer.
Questions for authors
- For comparison with Rubato, do you use the same training data as the proposed system?
- The score updates “after every completed InterMo moment”, but some tokens merge an interval with the start of the next moment. How does the system detect that a moment is complete? If it waits for the next interval token, the display would likely to lag by up to one gap between notes.
- A large portion of training data are synthetic, would it be better to train the model with MAESTRO as in Rubato?
2 comments
I raised the score
I appreciate the authors' reply, it has resolved my questions, I will raise my score.
We thank the reviewer. The revision (changes in blue) keeps the architecture and changes the training: the three separately trained models of the submission are replaced by one, trained offline and then adapted to streaming with the conductor buffer sampled between 0 and 30 s. This single checkpoint serves every lookahead and offline decoding, improves on the submitted results (pooled OMR-NED 89.8→87.5 at 0.16 s, 83.9→80.2 offline), and works with or without the buffer (without it: 1 invalid score of 1,701 at 0.16 s, vs 66 for the model trained without a buffer).
1. Background. The revision adds a paragraph on the components we take from Rubato (Section 3.1): in InterMo, a score is a sequence of records, each a metric interval followed by the notes that start or stop at that moment; IntervalPiece is a BPE vocabulary over InterMo whose merges respect the record structure, adapted from BoundlessBPE.
2. Streaming baseline. Online MIDI transcription plus online beat tracking is not a score transcriber: producing notation would also need online quantization, key, time-signature, staff and voice estimation and an incremental score converter, none of which exists. More importantly, such a cascade cannot be expected to outperform its own offline version, which has future context at every stage. We already compare against offline cascades of this kind (a 1.3B-parameter transcriber plus trained converters), and our streaming model matches or beats them. Building the missing online components is interesting future work, for which our system can serve as the streaming baseline (OMR-NED↓):
*The cascades' converters were trained on ASAP test data (point 5).
We also ran the reviewer's sliding-window suggestion: Rubato on non-overlapping 2.24 s windows (our longest lookahead) is worse than our 2.24 s model on all six sets we ran (OMR-NED, lower is better, Rubato vs. ours: ASAP 78.8 vs 76.4, KRAISLER piano 79.8 vs 77.0, duet 91.9 vs 71.5, violin 95.9 vs 83.1, URMP 92.3 vs 79.0, Bach10 93.6 vs 80.1).
3. Novelty. The encoder, cache-aware inference, InterMo, IntervalPiece and the synthesis come from prior work. Our contributions: (i) the task and a complete system: a transducer emits multi-staff notation directly in real time, with no note-detection, beat-tracking, quantization, staff-separation, key or meter stages, several of which have no online version; (ii) the conductor buffer, which the reviewer calls a heuristic. In the original submission it was a fixed inference setting; in the revision it is sampled during training, and with this training one model also serves offline decoding and still works without the buffer; (iii) incremental engraving, whose final score equals the offline conversion in 205/206 recordings. That a standard speech-recognition model family transcribes polyphonic multi-staff music in real time is itself a result, with implications for joint speech–music conversational AI without music-specific machinery.
4. When is a moment complete? The reviewer's reading is right:
1/8 PL: a-3 c4opens a new record and thereby closes the previous one. But the display does not wait: each note is drawn in gray when its own token is emitted, 0.21 s after it is played (median at 0.16 s); only the written duration waits, becoming final 0.24 s after the note ends; no causal system can fix a duration before the note ends. Section 3.3 is rewritten accordingly.5. ASAP overlap. The training data of the cascades' MIDI-to-score converters (M2ST, APT-EPR) include performance–score pairs from the ASAP test set, which favours them on ASAP; neither Rubato nor our model has this overlap. Our model still matches the cascades at 0.16 s and leads from 0.56 s.
6. Training data and Rubato. Same data? No. Rubato is trained on piano only, with more real piano audio, including MAESTRO through an auxiliary MIDI-transcription task. We use 48 h of real audio and also transcribe violin and violin–piano duets, which is why we also compare with the multi-instrument MuScriptor. Rubato is an offline attention encoder–decoder with a decoder that has 12× as many parameters as ours (75.9M vs 6.3M); we are 4.5× faster on GPU and over 6× faster on CPU. As in speech recognition, transducers serve streaming and on-device use, and attention decoders serve offline use. Within that trade-off we exceed Rubato on spelled pitch and staff (81.9 vs 79.5; 76.4 vs 74.8; higher is better), match it on exact durations (34.4 vs 34.6) and beat it by 8–12 points on all four violin/duet sets even at 0.16 s. We did not retrain either model on the other's data: Rubato's MAESTRO supervision is MIDI, which our transducer can use only with a multitask extension we have not explored, future work that may narrow the piano gap.