acceptodds
Under review as a conference paper at ICLR 2027

Prosodic Video Narration from Streaming Audio-Visual Events

Abstract

Compelling spoken narration requires not only the right words, but also prosody that reflects the significance of unfolding events. Existing streaming video models focus on what to say, while text-conditioned speech models lack the perceptual context needed to determine how to say it. We introduce Prosodic Video Narration, a task of continuously generating spoken descriptions whose content and prosody are grounded in streaming audio-visual events. To support this task, we construct a sports commentary dataset with aligned video, environmental audio, commentary text, and word-level prosodic annotations. We further propose ProsodyMLM, a streaming multimodal language model that models discrete word-level prosody from audio-visual context. Our design enables efficient autoregressive prosody prediction through lightweight adapters and prosody injection, while a two-stage training strategy transfers expressive speech capabilities before grounding them in multimodal events. Across quantitative metrics and a user study, ProsodyMLM generates narration that better aligns with reference speech and underlying events than existing baselines, improving pitch, speaking rate, and emotional alignment while maintaining competitive energy alignment, with particularly strong gains during highly expressive moments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
prosody, speech, multimodal, language models, audio-visual alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p2CrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the task of adding prosodic variation to sports commentary. They assume input of streaming audio, visual and textual commentary, and learn a model that determines suitable prosody. On football and basketball domains, they demonstrate that the resulting model outperforms pre-existing models, using automatic metrics that measure distributional qualities of the speech output and a human preference evaluation.

Critical strengths

(S1) The paper structure, writing quality and motivation is outstanding. The authors meaningfully motivate the need for more work in this direction and clearly define their problem statement (the delta between context-agnostic and context-informed synthesized commentary).

(S2) I expect this work will stimulate further research on the use of prosodic information in order to make synthesized commentary more engaging. This is a practical and valuable contribution.

(S3) The authors clearly demonstrate the successes and limitations of their model, using both automatic and human metrics. It is convincing that they outperform the stated baselines.

Critical weaknesses

(W1) Am I correct that the model takes as input audio without commentary, and this audio was obtained by automatically removing commentary from the gold audio track? If so, I have serious concerns about imperfections in this step and hence the leakage of the gold prosody information.

(W2) I would have enjoyed a more precise statement of the authors hypotheses for these experiments. As it stands, I understand the hypothesis is that adding learnt prosody information can aid automatic sports commentary. However, this is rather too abstract for my liking. The closest test of something more concrete is the final row of the ablation column, when the authors remove the prosody information from otherwise the same learned model. However, the reported numbers make me concerned about the methodological setup for this experiment: the automatic metrics are worse by enormous margins. The margins are much larger than the authors' baselines, yet I would expect this model to benefit over the baselines from the ~200 hours of training data. Is this because the speech synthesis model relies on prosodic predictions, which are random here? If so, the paper lacks a fair comparison against a system that benefits from all of the same training and inputs, but without explicit prosodic feature prediction.

(W3) The model relies on hand-crafted features. Can you provide an experiment which relies only on neural audio features (e.g. Emotion2Vec, as the authors use for evaluation) that are learnt at training time, and predicted and included in speech synthesis inference at test time?

Questions for authors

See above.

2 comments

W1: clarification of model's audio input

Yes, the model takes only environmental sounds as input.

We took several steps to prevent residual commentary from leaking into the input. Because some words can remain after a single pass, we apply speech removal in multiple rounds and then run an automatic speech recognition (ASR) model on the result to detect any remaining intelligible words. We discard utterances whose environmental audio is of poor quality. We also manually inspected random samples from the test set and found that the commentary was successfully removed in most cases.

In addition, our ablation gives evidence against leakage. If the model were relying on residual ground-truth prosody in the audio, when only train with audio as input it should give a competitive score as the full model. However, we see there is a quite large margin in performance.

W2: clarification of w/o prosody injection

We would like to clarify that the final row of ablation is not a model with the prosody information removed. It is the same model, trained on the same data with the same inputs, in which prosody is serialized as explicit tokens in the sequence and generated sequentially. This is the design where our injection mechanism replaces, so the row tests the injection mechanism, not the value of prosody. The decoder is never fed random prosody.

The reason why the margin is so large is that the serialized variant does not converge well under the same training budget. The pretrained backbone has only seen plain word sequences. Inserting 6 prosody tokens after every word makes the sequence about 7 times longer and shifts the input away from the pretraining distribution, so the positional and sequential patterns the backbone learned no longer hold. Relearning them requires far more data than is available for this task, which explains the low scores. Prosody injection keeps the original sequence length and structure, so the model can reuse its pretrained knowledge.

W3: clarification of prosody features

Thank you for the suggestion. We would first like to clarify that all perceptual inputs to ProsodyMLM (vision, environmental audio, and text) are processed by learned neural encoders. The concern therefore applies to the prosody representation the model predicts and conditions on for previously generated words. We adopt the word-level prosody tokens of ProsodyLM (Qian, Kaizhi, et al.'25). They are computed from the outputs of StyleTTS2’s neural alignment and acoustic front-end, summarized at the word level into duration, pitch, and energy statistics, and quantized into discrete tokens, from which the neural StyleTTS2 decoder synthesizes speech. ProsodyLM has shown that such tokens enable language models to capture emotion for expressive speech generation, and they are word-aligned, interpretable, and directly controllable, which a learned latent representation does not provide. We agree that a fully learned representation such as Emotion2Vec is a legitimate alternative that may also work. However, Emotion2Vec is designed for speech analysis, and no existing decoder synthesizes speech from it, so testing this pipeline would require training a new synthesizer, which we could not complete within the rebuttal period. We will discuss it as an alternative design and a direction for future work in the final paper.

Thank you for your response. Please ensure that these are addressed in the final version of your paper, in particular W1.

Reviewer_kPjtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Prosodic Video Narration, the task of generating spoken commentary whose prosody is grounded in streaming video and environmental audio; in practice, it predicts discrete word-level prosody for given commentary text. The authors build INF-STREAM-PROSODY (~420 hours of FIFA/NBA broadcasts with word-level prosodic annotations) and an out-of-domain test set, FIFA2026. ProsodyMLM adapts a frozen StreamingVLM with LoRA, an audio projector, and prosody embeddings added to word representations rather than inserted as extra tokens. It is trained by distillation from ProsodyLM followed by multimodal fine-tuning, with an optional pitch-balanced fine-tuning stage (Ours-bal). With ground-truth text as input, it leads Qwen3-TTS and ProsodyLM on most utterance-level prosody metrics, though not on in-domain energy, and is preferred over both in a user study.

Critical strengths

Well-defined problem with clear motivation. The paper accurately identifies the disconnect between what to say and how to say it, and formulates audio-visually grounded prosody generation as a clear streaming task. The two text-only baselines show opposite biases: Qwen3-TTS is too flat, while ProsodyLM is too exaggerated. This indirectly supports the motivation. Potentially valuable dataset. The ~420 hours of data align video, environmental audio, and word-level prosody. If released, it could serve as a benchmark for this task, and its value does not depend on whether the proposed method is effective. Simple and low-cost adaptation. Prosody is injected via additive embeddings without inserting extra tokens. As a result, word-level prosody prediction can be added to a frozen backbone by training only LoRA and a few new modules.

Critical weaknesses

The multimodal contribution is not isolated. The baselines in the main tables are zero-shot, text-only models, whereas ProsodyMLM is trained on in-domain data. The ablations also lack a variant with both vision and audio removed. It is therefore unclear whether the gains come from multimodal grounding or from in-domain training. Limited overall gains without statistical support. Against the best baseline for each metric, each of the two variants wins only 8 of the 12 results across the three test sets. Both variants also trail ProsodyLM on in-domain energy. No variance or significance tests are reported, and key details of the user study are missing, such as the number of participants and which variant was evaluated. The advantage on highly expressive moments is partly by construction. The high-excitement subset and the training subset for Ours-bal are selected by the same pitch criterion (P3). Meanwhile, rebalancing actually increases the full-set F0 difference from 32.30 to 46.06. The advantage of prosody injection is not validated. The only controlled comparison is against a serialized-token variant that, by the authors’ own account, fails to converge well. The paper also emphasizes efficiency but reports no latency measurements.

Questions for authors

Plz refer to the Weaknesses.

2 comments

W1: new results with a ProsodyLM backbone and in-domain adaptation

Thank you for the suggestion.

Our goal is to develop a system where prosody modeling can be added to any language model, like StreamingVLM's (Xu, Ruyi, et al.'26) backbone. ProsodyLM (Qian, Kaizhi, et al.'25) is a general-purpose model trained on a much larger dataset, so we used it directly as a zero-shot baseline. Comparing against a fine-tuned ProsodyLM would not be a fair comparison either, because even with our distillation stage, our model is trained with far less computation.

To isolate the contribution of the prior knowledge, we ran an additional controlled experiment with a second variant of our model, which applies our multimodal information to the ProsodyLM backbone. We also redesigned the prosody injection pipeline for this backbone: after the prosody tokens are removed, every remaining token keeps its original position index, so the relative positions match those the backbone was pretrained on.

We fine-tuned two models from the same ProsodyLM backbone on our training set and evaluated them on the full set:

  • Text-tuned: fine-tuned with the in-domain text data only.
  • Ours-new: fine-tuned with audio and visual input and with prosody injection.
ModelF0RateEnergyEmotion
ProsodyLM (zero-shot)64.422.6270.5520.941
Text-tuned29.591.8650.5730.954
Ours-new23.531.2580.6280.957

The two fine-tuned models share the same backbone, training data, and generation scheme, so we can get rid of the impact of domain shifts. The model trained with our mechanism, F0 and Rate are much better than the text-tuned only model, while energy and emotion remain comparable.

We have a more concrete description of this new variant setting in the revised paper's appendix. We will have a full analysis of it in the final paper.

W2: multimodal contribution

We reevaluate an additional text-only variant on our ablation set by removing both audio and vision. Compared with the full model, removing both modalities worsened all four metrics. The joint removal also produced the highest rate and energy errors among the ablation settings, especially the energy metrics. It also has worse emotional similarity.

SettingF0RateEnergyEmotion
Full model31.472.3610.6780.944
Without vision36.462.4170.6510.942
Without audio34.202.6000.6890.937
Without audio or vision36.382.7210.7880.933

To test whether the model benefits from aligned audiovisual cues, we held the commentary text fixed and replaced the corresponding audio and visual inputs with randomly selected segments (Random-AV). All four metrics worsened on both evaluation sets, with larger degradation on the excited subset. This suggests that aligned audiovisual cues are especially important for excited delivery.

Evaluation setSettingF0RateEnergyEmotion
Full setRandom-AV34.182.2840.8470.948
Full setOurs32.302.1350.6430.951
Excited subsetRandom-AV49.852.4460.9140.943
Excited subsetOurs44.812.2460.6770.948

W3: clarification of energy gap

We use ProsodyLM's (Qian, Kaizhi, et al.'25) StyleTTS2 decoder to synthesize speech from word-level prosody sequences. The decoder was trained on ProsodyLM's data, and differences between that training distribution and our in-domain audio may lead to this gap in energy performance. Even so, our energy scores remain competitive to ProsodyLM's.

W4: user study details

Our user study had 22 participants. Each participant answered 20 pairwise questions, 10 against Qwen3-TTS and 10 against ProsodyLM, giving 220 responses per comparison. All comparisons used the balanced variant of our model (Ours-bal).

To test significance, we ran a two-sided exact sign test for each comparison. We excluded ties and tested the null hypothesis that our model and the baseline are equally likely to be preferred. That is the number of responses favoring our model follows Binomial(n, 0.5), where n is the number of non-tied responses. Our model is preferred over Qwen3-TTS with p = 0.035 and over ProsodyLM with p = 6 × 10⁻¹¹. Both differences are significant at the 0.05 level.

W5: clarification of the rebalancing trade-off and new excited subset results

First, our advantage does not depend on rebalancing. On both test sets, our model without rebalancing still outperforms the baselines on most metrics.

Second, we rebuild a high-excitement subset selected by an independent criterion of P3. We use Jev, TypeSafe’s SystemOne model, which answers a question about a structured input with a probability between 0 and 1. For each reference utterance, we give Jev the transcript and the utterance-level averages of five prosodic features (duration, log-F0 range, log-F0 median, log-F0 slope and log-energy), each with a brief description, and ask whether the speaker sounds excited. The returned probability is used as the excitement score. The table below reports results on the top 30% of utterances ranked by this score:

SystemF0 ↓Rate ↓Energy ↓Emotion ↑
Ours32.401.9920.5710.954
Ours-bal34.001.9030.5670.955
Qwen3-TTS35.972.1061.0790.942
ProsodyLM62.352.5180.5050.947

Both of our models outperform Qwen3-TTS and ProsodyLM on F0, speaking rate and emotion. Our advantage on highly expressive segments therefore holds even when the subset is selected by a criterion that played no part in building the training data.

Finally, we agree that rebalancing increases the F0 difference, both on the full test set and, slightly, on this subset. This is expected: Ours-bal is biased toward high-excitement speech and is not meant to be our best general-purpose model. From the new experiment results we can see that its F0 gap to Ours-full shrinks from 13.76 on the full Inf-Stream-Prosody set to only 1.60 on this subset, while the other systems’ F0 errors stay close to their full-set values. This indicates that rebalancing specifically benefits highly excited utterances. The purpose of Ours-bal is to show that the overall excitement level of the generated commentary can be controlled through the composition of the fine-tuning data. This kind of control could support more personalized commentary, such as matching a preferred commentator style or energy level.

W6: clarification of prosody injection mechanism We would like to clarify that the poor convergence of the serialized-token variant is not an artifact of our setup: it is the direct consequence of serialization, and it is the problem that prosody injection was designed to solve.

The pretrained backbone has only seen plain word sequences. Serializing prosody inserts 6 prosody tokens after every word, which changes the input in two ways:

  • Sequence length: the sequence becomes about 7 times longer.
  • Sequence structure: consecutive words are now separated by 6 prosody tokens, so the positional and sequential patterns the backbone learned in pretraining no longer hold. Especially RoPE, as it depends on relative position.

The model must therefore relearn these patterns before it can generate correct prosody tokens, which requires far more data than we have for this task. Under the same data and training budget, the serialized variant does not converge well.

Prosody injection avoids this mismatch. It adds the prosody information to the word embeddings, so the sequence keeps its original length and structure, and the input stays close to the distribution the backbone was pretrained on. The model can then reuse its pretrained knowledge and converge much faster.

W7: efficiency experiment results

During inference, because the prosody tokens are injected into the word tokens, the sequence length is shorter. This reduces the KV cache size, since we no longer need to store cache entries for the prosody tokens as their information is already carried by the word tokens. The shorter context also helps speeding up inference.

To assess inference latency and real-time performance, we benchmarked our method with and without injection on a single NVIDIA A5000 GPU using a randomly generated word-prosody sequence of 2,048 tokens. The table below reports decoding latency per word and KV cache memory usage.

SettingLatency (ms/word)KV cache (MB)
With injection276112
Without injection338721
Reviewer_zxeyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Prosodic Video Narration, a task of continuously generating spoken descriptions whose content and prosody are grounded in streaming audio-visual events. To support this task, the authors construct a sports commentary dataset with aligned video, environmental audio, commentary text, and word-level prosodic annotations. They also propose ProsodyMLM, which extends a pretrained streaming VLM with lightweight adapters and word-level prosody injection. For the training, a two stage training strategy is used. it first learns text–prosody relationships and then aligns prosody with audio-visual events. Experiments show better pitch, speaking rate, and emotional alignment than existing baselines.

Critical strengths
  1. The authors construct a 420 hours dataset with aligned visual content, environmental audio, spoken commentary, and word-level prosodic annotations.

  2. The paper proposes an effective prosody generation module named word level prosody injection. It preserves the original text sequence length and the experimental results show improvements in pitch, speaking rate, and emotional alignment.

Critical weaknesses
  1. The task is to generate prosodic for streaming video, but the experiments are mainly conducted on short video clips of 5–30 seconds. Are there any more evaluation of inference latency, real-time performance, or long-term prosodic coherence? It will make the paper more convincing.

  2. The proposed model predicts prosody using ground truth commentary text. I'm confused that in a real time streaming system, the commentary text would also need to be generated from audio visual observations. So the generated commentary may contain errors or temporal misalignment. Will the proposed method still perform well when conditioned with generated commentary text that may contain errors? Because it's very important for real-time systems.

Questions for authors
  1. Could the proposed method still perform well when conditioned with real time generated commentary text that may contain errors?

  2. Could the model still perform well over long video streams more than 30 seconds length?

1 comment

W1 & Q2: longer video results

To assess inference latency and real-time performance, we benchmarked our method with and without injection on a single NVIDIA A5000 GPU using a randomly generated word-prosody sequence of 2,048 tokens. The table below reports decoding latency per word and KV cache memory usage.

SettingLatency (ms/word)KV cache (MB)
With injection276112
Without injection338721

For the longer video, as we maintain a bounded history, the total context length will be kept during the streaming process, so the performance should be similar in long versions. Per suggestion and to further demonstrate this on longer inputs, we randomly sampled 50 clips, each more than two minutes long. The results are shown below.

We have also added two longer demos on our anonymous demo page (https://prosodymlm.github.io/).

MethodF0RateEnergyEmotion
Qwen-TTS33.122.1321.2080.940
ProsodyLM70.532.5680.5630.943
Ours30.731.9700.6140.953

W2 & Q1: clarification of robustness

Thank you for raising this point. We agree that robustness to imperfect text input is essential for real-time deployment. This work mainly focuses on aligning prosody with the audiovisual context given the commentary content. We use ground-truth text so that prosody modeling can be evaluated separately from errors in text generation.

Generally, our model takes the audio and visual streams as inputs alongside the text. Many cues come directly from these streams rather than from the words, such as rising excitement before a goal, crowd noise and fast-moving play. When the generated text contains small errors, the model can therefore still infer the intended intensity and pacing from the audiovisual context.

To test this, we ran a new experiment that simulates errors in the preceding commentary. We randomly sampled 500 utterances from the excited subsets, where incorrect context is most likely to have an effect. In each utterance, we randomly replaced one third of the words in the preceding commentary (2–3 words on average) along with their prosody tokens. As shown below, performance changes slightly relative to the unmodified baseline, which suggests that the model is robust to errors in the prior generated text.

SettingF0RateEnergyEmotion
Unmodified prior commentary45.932.2970.6820.947
Corrupted prior commentary46.332.3430.6650.947

Comments on the paper

1 comment

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found the task well motivated (kPjt, p2Cr), the dataset valuable (zxey, kPjt), and the prosody injection design simple and effective (zxey, kPjt), and that all reviewers recommended acceptance.

The main concerns were (1) whether the gains come from our system design or simply from in-domain data adaptation, and (2) the efficiency of prosody injection. To address them, we added the following results:

  • Source of gains (kPjt): A controlled comparison on the ProsodyLM backbone between a text-only model and one with multimodal information and prosody injection, both fine-tuned on the same in-domain data.

  • Efficiency (zxey, kPjt): Latency and KV-cache measurements showing that prosody injection decodes faster and uses substantially less memory because it produces shorter sequences.

These results and the related clarifications are included in the revised paper and highlighted in blue, and we will add a more detailed analysis in the final version. We respond to each reviewer’s questions individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.