- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the task of adding prosodic variation to sports commentary. They assume input of streaming audio, visual and textual commentary, and learn a model that determines suitable prosody. On football and basketball domains, they demonstrate that the resulting model outperforms pre-existing models, using automatic metrics that measure distributional qualities of the speech output and a human preference evaluation.
Critical strengths
(S1) The paper structure, writing quality and motivation is outstanding. The authors meaningfully motivate the need for more work in this direction and clearly define their problem statement (the delta between context-agnostic and context-informed synthesized commentary).
(S2) I expect this work will stimulate further research on the use of prosodic information in order to make synthesized commentary more engaging. This is a practical and valuable contribution.
(S3) The authors clearly demonstrate the successes and limitations of their model, using both automatic and human metrics. It is convincing that they outperform the stated baselines.
Critical weaknesses
(W1) Am I correct that the model takes as input audio without commentary, and this audio was obtained by automatically removing commentary from the gold audio track? If so, I have serious concerns about imperfections in this step and hence the leakage of the gold prosody information.
(W2) I would have enjoyed a more precise statement of the authors hypotheses for these experiments. As it stands, I understand the hypothesis is that adding learnt prosody information can aid automatic sports commentary. However, this is rather too abstract for my liking. The closest test of something more concrete is the final row of the ablation column, when the authors remove the prosody information from otherwise the same learned model. However, the reported numbers make me concerned about the methodological setup for this experiment: the automatic metrics are worse by enormous margins. The margins are much larger than the authors' baselines, yet I would expect this model to benefit over the baselines from the ~200 hours of training data. Is this because the speech synthesis model relies on prosodic predictions, which are random here? If so, the paper lacks a fair comparison against a system that benefits from all of the same training and inputs, but without explicit prosodic feature prediction.
(W3) The model relies on hand-crafted features. Can you provide an experiment which relies only on neural audio features (e.g. Emotion2Vec, as the authors use for evaluation) that are learnt at training time, and predicted and included in speech synthesis inference at test time?
Questions for authors
See above.
2 comments
Thank you for your response. Please ensure that these are addressed in the final version of your paper, in particular W1.
W1: clarification of model's audio input
Yes, the model takes only environmental sounds as input.
We took several steps to prevent residual commentary from leaking into the input. Because some words can remain after a single pass, we apply speech removal in multiple rounds and then run an automatic speech recognition (ASR) model on the result to detect any remaining intelligible words. We discard utterances whose environmental audio is of poor quality. We also manually inspected random samples from the test set and found that the commentary was successfully removed in most cases.
In addition, our ablation gives evidence against leakage. If the model were relying on residual ground-truth prosody in the audio, when only train with audio as input it should give a competitive score as the full model. However, we see there is a quite large margin in performance.
W2: clarification of w/o prosody injection
We would like to clarify that the final row of ablation is not a model with the prosody information removed. It is the same model, trained on the same data with the same inputs, in which prosody is serialized as explicit tokens in the sequence and generated sequentially. This is the design where our injection mechanism replaces, so the row tests the injection mechanism, not the value of prosody. The decoder is never fed random prosody.
The reason why the margin is so large is that the serialized variant does not converge well under the same training budget. The pretrained backbone has only seen plain word sequences. Inserting 6 prosody tokens after every word makes the sequence about 7 times longer and shifts the input away from the pretraining distribution, so the positional and sequential patterns the backbone learned no longer hold. Relearning them requires far more data than is available for this task, which explains the low scores. Prosody injection keeps the original sequence length and structure, so the model can reuse its pretrained knowledge.
W3: clarification of prosody features
Thank you for the suggestion. We would first like to clarify that all perceptual inputs to ProsodyMLM (vision, environmental audio, and text) are processed by learned neural encoders. The concern therefore applies to the prosody representation the model predicts and conditions on for previously generated words. We adopt the word-level prosody tokens of ProsodyLM (Qian, Kaizhi, et al.'25). They are computed from the outputs of StyleTTS2’s neural alignment and acoustic front-end, summarized at the word level into duration, pitch, and energy statistics, and quantized into discrete tokens, from which the neural StyleTTS2 decoder synthesizes speech. ProsodyLM has shown that such tokens enable language models to capture emotion for expressive speech generation, and they are word-aligned, interpretable, and directly controllable, which a learned latent representation does not provide. We agree that a fully learned representation such as Emotion2Vec is a legitimate alternative that may also work. However, Emotion2Vec is designed for speech analysis, and no existing decoder synthesizes speech from it, so testing this pipeline would require training a new synthesizer, which we could not complete within the rebuttal period. We will discuss it as an alternative design and a direction for future work in the final paper.