- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CastScriptBench, a benchmark of 508 multilingual (13 languages) in-the-wild clips for speaker-attributed transcription. It has structured references, 8 clip-level attributes and an LLM-judged checklist metric for attribution (Acc), plus Levenshtein-based fidelity (Fid). It also proposes CastScript, a Qwen2.5-Omni-7B model trained with SFT, two single-objective GRPO experts, and multi-teacher on-policy distillation (MOPD) that routes supervision by token type: tag tokens go to the attribution expert and utterance tokens to the transcription expert. CastScript is reported as the best open-source model, matching the attribution expert and slightly exceeding the transcription expert on fidelity metrics.
Critical strengths
- The task is relevant, and the benchmark’s multilingual coverage and attribute annotations enable stratified diagnosis.
- The benchmark quality is verified by a human-agreement study, and the ablation compares SFT, experts, mix-RL and MOPD with and without routing.
- Token-type routing is simple and intuitive, and the paper is clearly organized
Critical weaknesses
- The key claims rest on tiny single-run differences: Fid 0.814 vs 0.808 against the transcription expert, Acc 0.650 vs 0.651 against the attribution expert, and +0.013 Acc / +0.017 Fid for routing (Table 4). With 508 clips, no seeds and no confidence intervals, these may be noise.
- Acc counts an item only if both the transcription and description are also correct (Eq. 1), so it does not isolate attribution. It also relies on matching free-text descriptions to Gemini-drafted references. The human study is small (100 clips, one annotator), and standard metrics (cpWER/cpCER) are not reported.
- The robustness claims are stronger than the reported results in Table 2. Gemini is much better on the Hard tier except in the hardest setting, where CastScript beats it on both Acc and fidelity metrics.
Questions for authors
- Q1: Which model provides the attribution reward during training, and is it the evaluation judge? Please show results with a held-out judge, and add a decoupled attribution metric and cpWER/cpCER.
- Q2: Can you add reward-routed GRPO and expert-merging baselines, and a test showing that the benchmark requires visual grounding (audio-only or visual ablation)?
2 comments
Dear Reviewer 6jBK,
Thank you once again for the time and effort you dedicated to reviewing our paper. Your insightful comments have helped us significantly improve our work.
We fully understand that you have a busy schedule, so we just wanted to gently follow up and see if our responses and revisions have adequately addressed your concerns. If there are any remaining questions, or if we have misunderstood any of your points, please do let us know—we are more than happy to provide further clarification.
If you feel that your concerns have been satisfactorily resolved, we would be sincerely grateful if you would kindly consider re-evaluating our work.
Warm regards,
The Authors
W1. Additional stability results.
We would like to clarify the comparisons. Experts are MOPD intermediates; our main baselines are Mix-RL/main-table models. No-routing tests dimension-specific supervision. Fixed SFT initialization/teachers; four-seed mean ± sample SD:
Across all four seeds, CastScript outperforms Mix-RL and MOPD without routing on both metrics and retains the best open-source performance in the main table.
CIs: 10,000 clip bootstraps (n=508).
Mean Acc/Fid (0.648/0.808) are comparable to the respective experts' scores (0.651/0.808), and both exceed those of MOPD without routing. We have narrowed the corresponding main-text claim to "matching each expert on its respective dimension".
W2/Q1. Attribution evaluation.
We address attribution–transcription coupling in four points:
1. Acc uses semantic matching. The appendix prompt allows equivalent same-language wording and ignores irrelevant punctuation, case and minor transcription differences. We have corrected the potentially misleading wording in Eq. (1) to clarify that Acc uses same-language semantic matching.
2. Evidence that Acc and Fid distinguish different capabilities. Fid measures transcription quality, while Acc is not solely determined by it: D-ORCA has lower Fid but much higher Acc than AVoCaDO, supporting Acc's focus on correct speaker attribution.
3. Impact of transcription differences on Acc. Our inspection found that ASR errors caused only 4% of uncredited cases, mainly utterance substitutions, omitted final words, and missing interjections. For example, "Drew." became "True?"; "Shh, they can't know I was gonna sneak." became "They can't know what's gonna see." We therefore also assess attribution after matching, to reduce these effects.
4. Attribution evaluation after matching. For three representative models, LCS dynamic programming with character edit similarity matches predicted/reference dialogue; Qwen3.8-Max judges speaker correspondence for conditional attribution accuracy. CastScript outperforms Qwen3-Omni at all thresholds (C.7).
References. Gemini drafts are manually corrected and independently verified (B.1). Descriptions are matched by speaker identity.
Reward/judges. Qwen3.8-27B provides rewards and primary evaluation. Held-out Kimi-K3/Qwen3.8-Max: CastScript Acc 0.638/0.573, best open-source (C.3).
Human validation. During rebuttal, a second reviewer rescored the same 100 clips: agreement/κ 0.952/0.886 with three-judge majority, 0.978/0.945 with reviewer 1.
cpWER/cpCER.
Results and analysis are in C.8.
W3. Scope of robustness.
We agree our claim was too broad. Gemini leads on Hard and has higher Acc in 8/9 Table 2 groups. Our Acc/Fid lead is limited to ≥4 speakers with heavy overlap (0.422/0.602 vs. 0.402/0.554). CastScript beats Qwen3-Omni on both metrics in all nine.
Q2. Additional baselines and audio-only results.
1. Baselines.
GRPO separately normalizes attribution/transcription rewards, routing advantages to description/tag and transcript tokens, respectively. Expert merging averages teacher LM/projection parameters. CastScript: 4-seed mean ± SD.
Results support token-wise expert-distribution supervision.
2. Audio-only comparison.
Both inputs use the same anonymous-speaker metric: per-clip optimal one-to-one ID mapping, weighted LCS (character edit similarity >0.6), and adjacent same-speaker merging.
Audiovisual input improves accuracy by 0.176.
Thank you again for your constructive feedback. We have revised the relevant main text and marked changes in blue. Please raise any further questions; we will respond promptly.