- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses named-speaker transcription in long-form TV dramas: given raw video and audio, a cast roster, and character portraits, the system outputs the start and end times, transcript, and speaking character of each utterance. The proposed pipeline, AutoCAST, first uses a fine-tuned Qwen3-Omni with interleaved integer-second markers to transcribe and segment the audio. It then builds per-character voice reference banks automatically, keeping an utterance only if its predicted speaker is detected by face recognition nearby. Finally, voiceprint label propagation adopted from DramaSR-LRM yields candidate speakers with similarity scores, and a fine-tuned Qwen3-VL-32B combines these scores with face-annotated frames and dialogue context to decide the speaker. Trained on two Chinese dramas from DramaSR-532K, AutoCAST achieves 91.70% and 87.27% attribution accuracy on seven held-out Chinese dramas and three English series, versus 83.06% and 83.74% for the best general-purpose baselines. Ablations show that automatic segmentation and bank construction come within about 0.6 points of their manually annotated counterparts.
Critical strengths
Controlled ablations directly test the main claim that the manual inputs required by DramaSR-LRM can be replaced. With the other components fixed, the automatically built bank reaches 91.70% versus 92.27% for a manually labeled bank (Table 3), and predicted segmentation reaches 92.27% versus 92.92% for GT boundaries (Table 2b). The evaluation covers about 400 hours from ten held-out series in two languages, with a strict end-to-end metric and alignment and scoring rules specified in Appendix A. On this data, the best general-purpose baselines reach 83.06% (Chinese) and 83.74% (English), compared with 91.70% and 87.27% for AutoCAST.
Critical weaknesses
W1. No downstream evaluation. The introduction motivates the task with story understanding and video QA (citing MICap and TVQA), but no experiment tests whether AutoCAST transcripts improve these or other downstream tasks, so the practical benefit of the outputs is not quantified. W2. Single evaluation domain. All test data are professionally produced TV dramas; no other recording conditions (e.g., meetings, interviews, user-generated video) are evaluated. With GT segmentation and a manually labeled bank, removing voiceprint scores lowers accuracy from 92.92% to 76.55%, and removing images lowers it to 91.49% (Table 4). How voiceprint propagation behaves under noisy or far-field audio is not examined. W3. The Table 1 baselines are general-purpose models with no task-specific adaptation described, whereas AutoCAST is fine-tuned on two DramaSR-532K dramas and pools voice references across each entire drama, including later clips (Appendix C.1).
Questions for authors
Plz refer to the Weaknesses.
2 comments
Thank you again for your thoughtful comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could take a moment to review our responses.
We would be happy to provide further clarification or additional evidence if any of our responses remain unclear. Thank you for your time and consideration!
We thank the reviewer for the careful assessment and constructive comments. We provide additional experiments and clarifications on downstream utility, cross-domain evaluation, and baseline comparisons below.
Q1.Downstream utility W1. To directly evaluate the downstream benefit of AutoCAST, We evaluated story understanding using 896 QA pairs constructed from the seven held-out Chinese dramas, with Qwen3-VL-8B throughout (Table 1). Adding AutoCAST-generated named-speaker transcripts, with automatic segmentation and reference-bank construction, improves accuracy from 39.06% to 66.07% +27.01percentagepoints. This is 0.56 points below the ground-truth-speaker condition 66.63. These results provide direct evidence that AutoCAST outputs improve story QA performance.
Q2.Evaluation beyond TV dramas W2. Without meeting-specific fine-tuning, AutoCAST transfers directly from TV dramas to AMI meeting recordings, achieving 25.40% DER (Table 2). This slightly outperforms TFAF 25.62 and substantially outperforms the evaluated general-purpose models 60.78, demonstrating transfer beyond the training domain.
Q3.Task adaptation and comparison fairness W3. We fine-tuned Qwen3-Omni on the same training data as AutoCAST. On Minning Town, AutoCAST achieves 90.97% attribution accuracy versus 72.18% for the adapted baseline +18.79points;Table3.
We also compared against CineSRD and Character-aware under matched inputs and evaluation protocols, giving both methods in each comparison the same full-drama access, including later clips. With GT segmentation and automatically clustered reference banks, AutoCAST achieves 88.07% versus CineSRD's 60.89%; in the end-to-end setting, it achieves 90.97% versus Character-aware's 42.32%.
Table 1. Story QA results on 896 DramaVQA questions from seven held-out Chinese dramas. All conditions use Qwen3-VL-8B.
Table 2. Speaker diarization results and error decomposition on the AMI meeting dataset.
Error components are contributions to DER in percentage points. Their sum may differ slightly from the reported DER due to rounding.
Table 3. End-to-end comparison with a task-specifically fine-tuned baseline on Minning Town.