acceptodds
Under review as a conference paper at ICLR 2027

CASTSCRIPT: ALIGNING “WHO” AND “WHAT” IN MULTILINGUAL AUDIO-VISUAL CAPTIONING

Abstract

Speaker-attributed transcription, i.e., jointly determining who speaks and what is said, is a pivotal yet under-explored capability in audio-visual captioning, held back by two gaps. First, existing dialogue-centric benchmarks are largely limited to English and Chinese and lack scene and acoustic annotations for difficulty-stratified diagnosis. Second, speaker attribution and utterance transcription are heterogeneous competencies, yet existing methods optimize them under a single aggregate objective, yielding supervision that is neither dense nor dimension-specific. To close these gaps, we introduce CastScriptBench and CastScript. CastScriptBench comprises 508 in-the-wild conversational videos over 13 languages, spanning 1,356 shots, 4,236 utterances, and 1,218 speakers, annotated with 8 scene and acoustic attributes for difficulty-stratified analysis and evaluated by a verifiable protocol that combines edit-distance-based transcription evaluation with checklist-based speaker attribution. CastScript is the first to adapt multi-teacher on-policy distillation (MOPD) to captioning: unlike conventional MOPD that assigns experts by task, we route by output dimension within a single sequence, supervising tag tokens with an attribution expert and text tokens with a transcription expert, replacing the sequence-level scalar reward with per-token distribution matching. On CastScriptBench, both proprietary and open-source models mis-attribute and mistranscribe, worsening sharply as speaker count and overlap rise. CastScript surpasses even the transcription expert in fidelity, attains the best open-source performance, and establishes MOPD as an effective recipe for multi-dimensional captioning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speaker-attributed transcription, audio-visual captioning, multilingual benchmark, multi-teacher on-policy distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_6jBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CastScriptBench, a benchmark of 508 multilingual (13 languages) in-the-wild clips for speaker-attributed transcription. It has structured references, 8 clip-level attributes and an LLM-judged checklist metric for attribution (Acc), plus Levenshtein-based fidelity (Fid). It also proposes CastScript, a Qwen2.5-Omni-7B model trained with SFT, two single-objective GRPO experts, and multi-teacher on-policy distillation (MOPD) that routes supervision by token type: tag tokens go to the attribution expert and utterance tokens to the transcription expert. CastScript is reported as the best open-source model, matching the attribution expert and slightly exceeding the transcription expert on fidelity metrics.

Critical strengths
  • The task is relevant, and the benchmark’s multilingual coverage and attribute annotations enable stratified diagnosis.
  • The benchmark quality is verified by a human-agreement study, and the ablation compares SFT, experts, mix-RL and MOPD with and without routing.
  • Token-type routing is simple and intuitive, and the paper is clearly organized
Critical weaknesses
  • The key claims rest on tiny single-run differences: Fid 0.814 vs 0.808 against the transcription expert, Acc 0.650 vs 0.651 against the attribution expert, and +0.013 Acc / +0.017 Fid for routing (Table 4). With 508 clips, no seeds and no confidence intervals, these may be noise.
  • Acc counts an item only if both the transcription and description are also correct (Eq. 1), so it does not isolate attribution. It also relies on matching free-text descriptions to Gemini-drafted references. The human study is small (100 clips, one annotator), and standard metrics (cpWER/cpCER) are not reported.
  • The robustness claims are stronger than the reported results in Table 2. Gemini is much better on the Hard tier except in the hardest setting, where CastScript beats it on both Acc and fidelity metrics.
Questions for authors
  • Q1: Which model provides the attribution reward during training, and is it the evaluation judge? Please show results with a held-out judge, and add a decoupled attribution metric and cpWER/cpCER.
  • Q2: Can you add reward-routed GRPO and expert-merging baselines, and a test showing that the benchmark requires visual grounding (audio-only or visual ablation)?
2 comments

W1. Additional stability results.

We would like to clarify the comparisons. Experts are MOPD intermediates; our main baselines are Mix-RL/main-table models. No-routing tests dimension-specific supervision. Fixed SFT initialization/teachers; four-seed mean ± sample SD:

MethodAccFid95% CI (Acc; Fid)
D-ORCA0.5320.584[0.498, 0.566]; [0.556, 0.614]
Qwen3-Omni0.5860.676[0.556, 0.616]; [0.643, 0.708]
Mix-RL0.6200.791[0.593, 0.647]; [0.774, 0.809]
CastScript0.648 ± 0.0030.808 ± 0.004[0.622, 0.674]; [0.793, 0.824]

Across all four seeds, CastScript outperforms Mix-RL and MOPD without routing on both metrics and retains the best open-source performance in the main table.

CIs: 10,000 clip bootstraps (n=508).

Mean Acc/Fid (0.648/0.808) are comparable to the respective experts' scores (0.651/0.808), and both exceed those of MOPD without routing. We have narrowed the corresponding main-text claim to "matching each expert on its respective dimension".

W2/Q1. Attribution evaluation.

We address attribution–transcription coupling in four points:

1. Acc uses semantic matching. The appendix prompt allows equivalent same-language wording and ignores irrelevant punctuation, case and minor transcription differences. We have corrected the potentially misleading wording in Eq. (1) to clarify that Acc uses same-language semantic matching.

2. Evidence that Acc and Fid distinguish different capabilities. Fid measures transcription quality, while Acc is not solely determined by it: D-ORCA has lower Fid but much higher Acc than AVoCaDO, supporting Acc's focus on correct speaker attribution.

3. Impact of transcription differences on Acc. Our inspection found that ASR errors caused only 4% of uncredited cases, mainly utterance substitutions, omitted final words, and missing interjections. For example, "Drew." became "True?"; "Shh, they can't know I was gonna sneak." became "They can't know what's gonna see." We therefore also assess attribution after matching, to reduce these effects.

4. Attribution evaluation after matching. For three representative models, LCS dynamic programming with character edit similarity matches predicted/reference dialogue; Qwen3.8-Max judges speaker correspondence for conditional attribution accuracy. CastScript outperforms Qwen3-Omni at all thresholds (C.7).

LCS matching thresholdGemini 3.5 Flash\color{gray}{\text{Gemini 3.5 Flash}}Qwen3-OmniCastScript (Ours)
>0.60.852\color{gray}{0.852}0.6720.741
>0.70.859\color{gray}{0.859}0.6700.750
>0.80.860\color{gray}{0.860}0.6760.753
>0.90.858\color{gray}{0.858}0.6760.752

References. Gemini drafts are manually corrected and independently verified (B.1). Descriptions are matched by speaker identity.

Reward/judges. Qwen3.8-27B provides rewards and primary evaluation. Held-out Kimi-K3/Qwen3.8-Max: CastScript Acc 0.638/0.573, best open-source (C.3).

Human validation. During rebuttal, a second reviewer rescored the same 100 clips: agreement/κ 0.952/0.886 with three-judge majority, 0.978/0.945 with reviewer 1.

cpWER/cpCER.

ModelcpWER ↓cpCER ↓
Gemini 3.5 Flash\color{gray}{\text{Gemini 3.5 Flash}}0.260\color{gray}{0.260}0.158\color{gray}{0.158}
Qwen3-Omni1.2830.932
CastScript0.4520.390
  • Qwen3-Omni: Repetition causes unusually high cpWER/cpCER.
  • Gemini: Substitutions lead word errors; character insertions, deletions and substitutions are balanced.
  • CastScript: Word errors are mixed; character errors mainly involve insertions/deletions, without Qwen3-Omni's insertion dominance.

Results and analysis are in C.8.

W3. Scope of robustness.

We agree our claim was too broad. Gemini leads on Hard and has higher Acc in 8/9 Table 2 groups. Our Acc/Fid lead is limited to ≥4 speakers with heavy overlap (0.422/0.602 vs. 0.402/0.554). CastScript beats Qwen3-Omni on both metrics in all nine.

Q2. Additional baselines and audio-only results.

1. Baselines.

GRPO separately normalizes attribution/transcription rewards, routing advantages to description/tag and transcript tokens, respectively. Expert merging averages teacher LM/projection parameters. CastScript: 4-seed mean ± SD.

MethodAccFid
Reward-routed GRPO0.5780.787
Expert merging0.6170.794
CastScript0.648 ± 0.0030.808 ± 0.004

Results support token-wise expert-distribution supervision.

2. Audio-only comparison.

Both inputs use the same anonymous-speaker metric: per-clip optimal one-to-one ID mapping, weighted LCS (character edit similarity >0.6), and adjacent same-speaker merging.

ModelAudio-onlyAudiovisual
D-ORCA0.3330.509

Audiovisual input improves accuracy by 0.176.

Thank you again for your constructive feedback. We have revised the relevant main text and marked changes in blue. Please raise any further questions; we will respond promptly.

Dear Reviewer 6jBK,

Thank you once again for the time and effort you dedicated to reviewing our paper. Your insightful comments have helped us significantly improve our work.

We fully understand that you have a busy schedule, so we just wanted to gently follow up and see if our responses and revisions have adequately addressed your concerns. If there are any remaining questions, or if we have misunderstood any of your points, please do let us know—we are more than happy to provide further clarification.

If you feel that your concerns have been satisfactorily resolved, we would be sincerely grateful if you would kindly consider re-evaluating our work.

Warm regards,

The Authors

Reviewer_5QW5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces speaker-attributed transcription in multilingual audio–visual videos: identifying “who speaks” and “what is said” while producing visually grounded speaker descriptions and dialogue records in chronological order. To address the limitation of training these two distinct abilities with a single objective, the authors introduce CastScriptBench and CastScript:

  • CastScriptBench includes 508 conversational videos across 13 languages and 7 writing systems. Each clip is annotated with 8 scene and acoustic attributes, allowing performance to be examined under conditions such as multiple speakers, overlapping speech, and background noise.
  • CastScript builds on Qwen2.5-Omni-7B. After supervised fine-tuning, the authors train separate experts for speaker attribution and transcription, then distill both into one model. Supervision is routed by token type: the attribution expert guides speaker descriptions and speaker IDs, while the transcription expert guides the dialogue text. The resulting model scores 0.650 on attribution accuracy and 0.814 on transcription fidelity, outperforming Qwen3-Omni and the mixed-reward RL baseline on both measures.
Critical strengths
  1. The paper presents speaker descriptions and dialogue records separately. This makes the output structure clear and links each line of dialogue to a visible person, rather than an anonymous speaker label. This is a useful way to study audio–visual speaker grounding.
  2. CastScriptBench covers multiple languages and labels several challenging conditions, including overlapping speech, background noise, music, shot changes, speech rate, and speaker visibility. The authors also examine how speaker count and overlap affect results. This gives a more detailed picture than an overall score alone.
  3. The method assigns different experts to speaker labels and dialogue text. That fits the two parts of the task. During on-policy training, the experts receive prefixes generated by the student. Speaker-ID tokens are also given extra weight, since they make up only a small part of the output.
  4. The experiments compare several approaches, starting from the same SFT model. These include continued SFT, two specialist RL models, Mix-RL, and distillation without routing. Among these tested methods, the full routed-distillation approach achieves the strongest balance between attribution accuracy (Acc) and transcription fidelity (Fid).
Critical weaknesses
  1. Acc combines transcription and speaker attribution. A response receives credit only when the utterance is matched in the original language and assigned to the right speaker. So a transcription error can lower Acc even when the speaker is identified correctly. The paper could report attribution accuracy separately and break down the main types of errors.
  2. The gains may come partly from weighting and extra training. Mix-RL uses equal reward weights, but the final method gives extra weight to speaker-ID tokens and adds two expert models plus a distillation stage. In Table 8, increasing the tag weight from 1 to 10 raises Acc from 0.632 to 0.650. More controlled comparisons would help show how much improvement comes from token routing itself.
  3. The multilingual annotations need more validation. The paper identifies four graduate-student annotators, but does not describe their proficiency across the 13 languages or whether native speakers were involved. Gemini also generates the training references and RL/MOPD ground truth, but the paper does not describe a comparable human review process for these annotations. Their accuracy should be checked, especially for languages where Gemini may be less reliable.
  4. The difficulty levels may mix different factors. Non-Latin writing is treated as an adverse factor, but the paper does not show that the script itself makes speaker attribution harder. The language distribution is also uneven: the dataset has 154 English and 138 Mandarin clips, but only four Arabic clips. Per-language results and analyses that account for other factors would make the multilingual claims stronger.
Questions for authors
  1. Since Acc requires both correct transcription and speaker assignment, could you report a separate metric for speaker attribution?
  2. Could you compare different tag and reward weights, as well as baselines with matched training compute? This would help show whether the gains come from routing or from the additional training and weighting.
  3. Could you clarify the annotators’ language expertise and whether native speakers were involved? How accurate are the Gemini-generated training references?
  4. Could you compare the primary LLM judge with human reviewers? Also, does LLM-based dialogue extraction affect the baseline scores?
1 comment

W1/Q1. Speaker attribution evaluation and error analysis

Thank you for these suggestions. We respond in five points:

1. Acc uses semantic matching. The appendix prompt permits equivalent expressions in the same original language, without identical wording, and ignores irrelevant punctuation, capitalization, and minor transcription differences. We have corrected the potentially misleading wording in Eq. (1) to clarify that Acc uses same-language semantic matching.

2. Evidence that Acc and Fid distinguish different capabilities. Fid measures transcription quality, while Acc is not solely determined by it: D-ORCA has lower Fid but much higher Acc than AVoCaDO, supporting Acc's focus on correct speaker attribution.

3. Impact of transcription differences on Acc. Our inspection found that ASR errors caused only 4% of uncredited cases, mainly utterance substitutions, omitted final words, and missing interjections. For example, "Drew." became "True?"; "Shh, they can't know I was gonna sneak." became "They can't know what's gonna see.", omitting "Shh" and misrecognizing the latter part. We therefore also assess attribution after matching, to reduce these effects.

4. Attribution evaluation after matching. We evaluate three representative models using LCS dynamic programming with character edit similarity to match predicted and reference dialogue. Qwen3.8-Max then judges speaker correspondence to compute conditional attribution accuracy. CastScript outperforms Qwen3-Omni at all thresholds. These results are in Appendix C.7.

LCS matching thresholdGemini 3.5 Flash\color{gray}{\text{Gemini 3.5 Flash}}Qwen3-OmniCastScript (Ours)
>0.60.852\color{gray}{0.852}0.6720.741
>0.70.859\color{gray}{0.859}0.6700.750
>0.80.860\color{gray}{0.860}0.6760.753
>0.90.858\color{gray}{0.858}0.6760.752

5. Major error types. Spot checks of uncredited cases at >0.6 reveal attribution errors and conflicting speaker descriptions. Qwen3-Omni often assigns speech to another person; CastScript merges different people under one ID; Gemini also confuses on-screen and off-screen speakers.

W2/Q2. Tag/reward weights and longer training

Table 8 tests tag weights 1/3/10/20. Even at 1, Acc/Fid (0.632/0.808) exceed Mix-RL's 0.620/0.791; both peak at 10 (0.650/0.814).

Reward ratios below are attribution:transcription.

SettingAcc ↑Fid ↑
Mix-RL (1:1)0.6200.791
Mix-RL (1:9)0.5840.778
Mix-RL (9:1)0.6160.725
Mix-RL (1:1, 2 epochs)0.5100.680
CastScript (Ours)0.6500.814

Neither alternative reward ratio improves on equal-weight Mix-RL. As our teachers are first trained on the same data, we extend Mix-RL to 2 epochs for fairness; repetitive outputs become more frequent, and performance declines.

W3/Q3. Multilingual annotations and training references

The benchmark's four core annotators collectively cover Mandarin, Cantonese, English, and Spanish. Three rotate between correcting model-generated drafts and cross-review; the fourth, senior annotator resolves disagreements.

For the remaining nine languages, members of our university's language school with the relevant language proficiency checked approximately 20% of each language's annotations: 37 videos and 231 utterances in total. They found speaker-alignment errors in 2 utterances and minor utterance-final omissions in 3. All were corrected.

For training references, we manually checked Gemini-generated annotations for 200 training samples across multiple languages. About 20% of samples contained speaker-attribution errors, with no clear language-specific pattern.

W4. Language results

In Table 3, the three selected models' mean Acc/Fid drops by 0.067/0.141 from Latin to non-Latin scripts, possibly reflecting Latin-script-heavy pretraining data in the base models.

Table 9 (C.6) covers all 13 languages. Results suggest that possible underrepresentation of less widely spoken languages in pretraining (the language long tail) has less impact than, e.g., overlap and noise. Frequent overlap helps explain why all three models perform worse on Mandarin than on some less widely spoken languages. Portuguese has frequent noise and music, with attribution accuracy below each model's overall average; Korean has little noise or overlap and performs well on attribution.

Q4. Human agreement and dialogue extraction

The primary judge, Qwen3.8-27B, achieves 94.0% agreement with human judgments (Cohen's κ=0.865). Appendix C.3 also reports human agreement with the three-judge majority vote.

Dialogue extraction is used only for models such as AVoCaDO that output only overall captions. During rebuttal, manual extraction from a random 10% of each model's outputs yielded the same Fid as LLM extraction.

Thank you again for your constructive feedback. We have revised the relevant main text and marked changes in blue. Please raise any further questions; we will respond promptly.

Reviewer_YF4GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the recognition of speaker identities and the speech transcripts responding to each speaker in video understanding and captioning. The authors present a benchmark and a training strategy for the task. The two-teacher architecture and token-routed MOPD decouple the problem into two smaller and more independent tasks. Experimental results in many aspects show the strong performance of the proposed method.

Critical strengths
  1. The task is practical and crucial. Fine-grained understanding and captioning of speech in video are under increasing demand for high-quality data pair construction.
  2. The authors have a comprehensive consideration of the task. They proposed both a benchmark and a full training pipeline to tackle the problem.
  3. The proposed methods are reasonable and effective. The decoupled design of the two-expert structure and token-routed MOPD clearly decomposes the complex task into small ones.
Critical weaknesses
  1. The evaluation is not really decoupled for the recognition of the speaker identity and the responding transcripts. Acc depends on both the correctness of the speaker identity and accuracy of the responding transcripts, so it is not clear whether the improvement is from transcripts accuracy or identity recognition.
  2. The core design of the paper, token-routed MOPD, still lacks detailed comparisons and ablations. Current ablation only compares token-routed MOPD with other post-training methods, w/o routing MOPD setting and different divergence objectives settings. It would be clearer if the author provide more experimental results about the method, especially results about the actual effect of different token types with different routing stretegies (e.g. How's the performance of the model if the cast description is routed to the attribution teacher, and the remaining parts use uniform teacher assignment? The description has an important effect on the following recognition).
Questions for authors

Please see the Weakness section for more details. The evaluation of Acc and Fid is still not enough to demonstrate the actual performance of the model, while the token-routed MOPD design still lacks detailed comparision. It would be more clear if the author provide qualitative or quantitative explanation on the two problems.

3 comments

W1/Q1. Evaluation of transcription and speaker attribution

Thank you for suggesting that we disentangle transcription and attribution evaluation. We respond in four points:

1. Acc uses semantic matching. The evaluation prompt in the appendix explicitly allows matches or equivalent expressions in the same original language, without requiring identical wording, and ignores irrelevant punctuation, capitalization, and minor transcription differences. We have corrected the potentially misleading wording in Eq. (1) to clarify that Acc uses same-language semantic matching.

2. Evidence that Acc and Fid distinguish different capabilities. Fid primarily reflects ASR transcription quality, whereas Acc is not determined solely by transcription quality. For example, D-ORCA has lower Fid than AVoCaDO but considerably higher Acc, illustrating that Acc focuses more on whether an utterance is matched to the correct speaker.

3. Impact of transcription differences on Acc. We examined ID-ASR matching failures caused by ASR errors and found that these cases do occur but account for only 4% of cases receiving no credit, mainly involving utterance substitutions, omitted words at utterance endings, and missing interjections. For example, the reference "Drew." was transcribed as "True?"; for "Shh, they can't know I was gonna sneak.", the model produced "They can't know what's gonna see.", omitting the interjection "Shh" and misrecognizing the latter part. We therefore also assess attribution after matching, to reduce these effects.

4. Supplementary attribution evaluation after matching. We selected three representative models for supplementary evaluation. We match predicted and reference dialogue using LCS dynamic programming based on character edit similarity in the original text, then use Qwen3.8-Max to judge whether the speakers in the matched segments correspond and compute conditional attribution accuracy. Our method still outperforms Qwen3-Omni across different matching thresholds. We have added these results to Appendix C.7.

LCS matching thresholdGemini 3.5 Flash\color{gray}{\text{Gemini 3.5 Flash}}Qwen3-OmniCastScript (Ours)
>0.60.852\color{gray}{0.852}0.6720.741
>0.70.859\color{gray}{0.859}0.6700.750
>0.80.860\color{gray}{0.860}0.6760.753
>0.90.858\color{gray}{0.858}0.6760.752

W2/Q2. Ablation of routing by token type

Thank you for suggesting that we distinguish the contribution of routing cast descriptions from that of routing the remaining tokens. Following your suggestion, we add a description-only routing control: cast descriptions receive supervision exclusively from the attribution expert, while all remaining tokens receive an equal-weight combination of the two teachers' KL losses. Student initialization, teacher models, training data, and hyperparameters, including token-type loss weights, are held fixed between this control and full routing. The results are summarized below:

MethodAcc ↑Fid ↑
Mix-RL0.6200.791
MOPD without routing0.6370.797
MOPD with description-only routing0.6380.796
MOPD with token routing (Ours)0.6500.814

Full token routing achieves higher Acc and Fid than the other configurations.

Thank you again for your constructive feedback. We have revised the relevant main text and marked changes in blue. Please raise any further questions; we will respond promptly.

I thank the authors for their detailed response. The additional experiments regarding disentangled evaluation and token-type routing ablations have addressed my concerns. I maintain my recommendation to accept the paper.

Dear Reviewer YF4G,

Thank you so much for your positive assessment! Your insightful comments and constructive feedback have greatly helped us strengthen our work. We sincerely thank you again for your valuable time and encouraging support.

Best regards,

The Authors

Reviewer_mVC3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CastScriptBench, with 508 videos in 13 languages, for speaker attribution and transcription. CastScript trains two RL specialists from a shared SFT model and distills them into a 7B student by output token type. It improves over the evaluated open models and training controls, with additional gains on external QA mediated by generated descriptions.

Critical strengths

The method integrates complementary specialists. A shared SFT starting point, continued-SFT and no-routing controls support practical training value, with additional external QA evidence.

Critical weaknesses

The novelty of this paper is limited, as the core contribution primarily lies in applying an existing methodology to a relatively niche domain. The technical redesign or adaptation of the method appears minimal (the data construction pipeline, modifications to MOPD and others).

Questions for authors

How did four annotators verify all 13 languages? Please provide more details about it. In Stage 2, based on my understanding, although the reward only evaluates the transcription outcome, the model is still prompted to generate the initial Cast Description during the rollout. Consequently, the sequence-level reward inevitably propagates back to these preceding Cast Description tokens, which seems to be supported by Eq. (7) on Page 6. If this is the case, wouldn't such a transcription reward inadvertently penalize or spuriously reinforce the Cast Description generated in the first step?

2 comments

W1. Limited novelty and contribution in a niche domain

Thank you for the comment. We would like to clarify two points.

1. Importance and difficulty of the task. ASR and speaker-attributed transcription (ID-ASR) have long been important and challenging problems in audio-visual understanding and generation. These capabilities receive substantial attention from both omni and video/audio generation foundation model teams. For example, Qwen3-Omni, Sec. 7 lists multi-speaker ASR as future work; Kling 3.0 explicitly highlights matching dialogue to the correct characters in multi-character scenes, while Seedance 2.0, Sec. 2.3.6 evaluates audio-visual synchronization in multi-person dialogue. These works illustrate the practical need to link speech content to characters in audio-visual generation. Second, this task is challenging: as our benchmark evaluation shows, even strong models such as Gemini and Qwen3-Omni struggle to achieve satisfactory performance in this domain. In addition, Reviewers YF4G and 6jBK both acknowledge the value of this research direction.

2. Novelty and contributions. Progress in a research area depends on benchmarks that reveal model capabilities and limitations. We therefore introduce CastScriptBench, comprising 508 in-the-wild videos and 4,236 dialogue turns across 13 languages, with annotations for eight scene and acoustic attributes. Compared with related benchmarks, it enables a finer-grained analysis of differences in model capabilities and the specific conditions under which they fail.

We further investigate the effectiveness of different post-training strategies for this task and propose token-routed MOPD based on its characteristics. To our knowledge, this is the first successful application of MOPD to captioning. Our experiments also show that our method is better suited than Mix-RL to learning the different capability dimensions required for audio-visual captioning (Table 4). We therefore believe that our work makes substantial contributions and offers clear novelty.

Q1. Verification of multilingual annotations

Our four core annotators collectively cover Mandarin, Cantonese, English, and Spanish. Annotation combines model-assisted drafting, manual correction, and cross-review. Gemini 3.5 Flash generates drafts in the original language and script, with translated copies to aid comprehension. For uncertain utterances, annotators recheck using contextual audiovisual question answering and compare TTS-generated audio with the original speech. Three annotators alternate between correction and review; the fourth, senior annotator adjudicates unresolved cases.

During rebuttal, we invited members of our university's language school proficient in the remaining nine languages to spot-check approximately 20% of each language's annotations. This additional language-quality audit covered 37 videos and 231 utterances. It identified speaker-alignment errors in 2 utterances and minor omissions at utterance endings in 3 utterances. All identified errors have been corrected.

Q2. Effect of transcription rewards on cast descriptions

Thank you for your insightful comment. Eq. (5) defines the transcription expert's objective:

\mathbb{E}_{X,\hat{y}\sim\pi}\\!\left[R_{\text{trans}}(\hat{y},y)\right].$$ This expert generates full sequences, including cast descriptions, so its sequence-level reward can affect those descriptions. However, this stage aims to strengthen transcription. In Stage 3, Eq. (8) routes supervision by token type: $$\pi_{\text{teacher}}^{(t)}(\cdot)= \begin{cases} \pi_{\theta_{\text{attr}}}(\cdot\mid\hat{y}_{ < t},X), & t\in\mathcal{I}_{\text{desc}}\cup\mathcal{I}_{\text{tag}},\\\\ \pi_{\theta_{\text{trans}}}(\cdot\mid\hat{y}_{ < t},X), & t\in\mathcal{I}_{\text{tran}}. \end{cases}$$ Here, $\mathcal{I}_{\text{desc}}$, $\mathcal{I}_{\text{tag}}$, and $\mathcal{I}_{\text{tran}}$ index descriptions and structural tokens, speaker ID digits, and transcripts, respectively. Both teachers condition on the student prefix $\hat{y}_{ < t}$; the student applies token-wise forward KL distillation (Eq. 9). Thus, descriptions receive attribution-expert supervision, avoiding direct distillation of the transcription expert's distribution at those positions. Our goal is to integrate each expert's capabilities in its corresponding dimension. In Table 4, CastScript achieves Acc/Fid of 0.650/0.814, both higher than Mix-RL's 0.620/0.791. Thank you again for your constructive feedback. We have revised the relevant main text and marked changes in blue. Please raise any further questions; we will respond promptly.

Dear Reviewer mVC3,

Thank you once again for the time and effort you dedicated to reviewing our paper. Your insightful comments have helped us significantly improve our work.

We fully understand that you have a busy schedule, so we just wanted to gently follow up and see if our responses and revisions have adequately addressed your concerns. If there are any remaining questions, or if we have misunderstood any of your points, please do let us know—we are more than happy to provide further clarification.

If you feel that your concerns have been satisfactorily resolved, we would be sincerely grateful if you would kindly consider re-evaluating our work.

Warm regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all reviewers for their careful reading and constructive suggestions. We summarize the main issues addressed in our responses and the additional experimental evidence provided.

Our work focuses on ID-ASR, an important task in audiovisual captioning that combines accurate speech transcription with visually grounded speaker attribution. This topic has attracted broad interest. The recent Kling-MLLM report identifies stronger audiovisual integration as an open challenge for captioning, calling for accurate alignment of acoustic and visual events and explicit association of speakers with their utterances. Kling-MLLM, §3.6 The Qwen3.8-Omni report also highlights the challenges posed by turn-taking, overlapping speech, and changing identities in multi-participant meetings. It emphasizes combining audio and video for speaker segmentation, content transcription, and identity alignment, using visual information to resolve references and entity ambiguity in audio. Qwen3.8-Omni, §5.1

All reviewers acknowledged specific strengths of our work. YF4G described the task as “practical and crucial” and the methods as “reasonable and effective.” 5QW5 recognized the value of linking dialogue to visible people, the benchmark’s detailed analysis, and the strongest attribution–transcription balance achieved by full routed distillation among the tested methods. 6jBK noted the benchmark’s value for stratified diagnosis, while mVC3 acknowledged the practical training value of integrating complementary specialists.

The reviewers’ suggestions led to the following clarifications and additions:

Attribution evaluation. YF4G, 5QW5, and 6jBK raised concerns about whether Acc conflates transcription and attribution. We responded from four angles: the semantic matching rule, empirical evidence that Acc and Fid capture different capabilities, the impact of transcription errors on scoring, and attribution accuracy after utterance matching. We revised the corresponding wording in the paper and additionally provided error analysis, cpWER/cpCER, human assessments, and held-out judge results.

Method comparisons. To address 5QW5’s concerns about reward weighting, training budget, and the role of routing, we added Mix-RL comparisons with different reward weights and extended training. We also added the description-only routing ablation suggested by YF4G and the reward-routed GRPO and expert-merging baselines requested by 6jBK. These comparisons further examine the value of assigning the appropriate expert supervision to different output components.

Annotation and language analysis. We clarified annotators’ language expertise, added annotation checks by personnel from the university’s language school and a human audit of training references, and reported per-language results.

Our contributions are twofold. CastScriptBench combines visually grounded speaker descriptions with dialogue records and annotations across multiple audiovisual dimensions, providing a challenging, diagnostically useful benchmark for ID-ASR. It reveals differences in model capabilities and specific failure conditions across languages and complex acoustic and visual settings. Token-routed MOPD assigns expert supervision according to the capabilities required by different parts of a caption to alleviate the attribution–transcription trade-off in Mix-RL. Additional experiments further show that, for this captioning task with multiple output dimensions, our method balances attribution and transcription better than Mix-RL, reward-routed GRPO, and expert merging.

After reviewing our response, YF4G confirmed that the additional disentangled evaluation and token-type routing ablations addressed their concerns and maintained their recommendation to accept the paper. Regrettably, we have not yet received follow-up responses from mVC3, 5QW5, and 6jBK. We remain available for further discussion and once again thank all reviewers and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.