- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes CAST, a framework for grounded multi-speaker audio-visual reasoning. CAST combines: (1) Speaker-Aware Audio Encoding (SAE), which inserts explicit speaker-turn delimiters into the audio representation; (2) Audio-Conditioned Spatial Modulation (ACSM), which aligns audio segments with visual patches and produces a speaker heatmap prior; and (3) structured perception reasoning with explicit <timestamp>, <speaker>, and <asr> fields before producing the final answer. The model is first trained with supervised fine-tuning and then optimized with GSPO using a heterogeneous reward that combines temporal grounding, ASR quality, speaker-description matching, factuality, and naturalness.
The paper also introduces MSR-QA, a large-scale multilingual dataset with approximately 8,000+ hours of speech video and 11 million QA pairs covering ten multi-speaker reasoning tasks. The held-out MSR-QA Bench contains 3,677 manually verified examples and includes substantial speaker overlap.
CAST achieves strong performance across AV-SpeakerBench, OmniMMI-SI, Omni-MMSI, and MSR-QA, and the component/reward ablations show that SAE, ACSM, SFT, RL, and the individual reward terms each make meaningful contributions.
Critical strengths
-
Important and well-motivated problem.
The paper targets a fundamental weakness of current AVLLMs: binding speech to the correct visible speaker in crowded multi-speaker scenes. The decomposition into who / when / what / where is intuitive and directly connected to the failure modes of existing models. -
Reusable architectural contribution through Speaker-Aware Audio Encoding (SAE).
SAE introduces an explicit turn-aware audio representation rather than treating multi-speaker audio as a single entangled stream. This component is potentially useful beyond CAST and can serve as a building block for future multi-speaker AV models. Its importance is supported by the ablation: removing SAE causes a large drop in speaker attribution and overall performance. -
Explicit cross-modal grounding through ACSM.
Rather than relying only on implicit multimodal fusion, ACSM explicitly aligns audio segments with visual patches and uses the resulting spatial prior to modulate visual representations. The visual-only and audio-shuffled controls provide useful evidence that the gains arise from meaningful audio-visual correspondence rather than static visual saliency. -
Strong training objective and informative ablations.
The heterogeneous reward is carefully designed around the task: temporal IoU supervises when, WER supervises what, key-description matching supervises who, and judge-based rewards supervise factuality and naturalness. The component and reward ablations show large, interpretable degradations when individual pieces are removed, making the contribution of each design choice clear. -
Substantial dataset and reproducibility contribution.
MSR-QA itself is a significant contribution, with approximately 8,000+ hours of multilingual video and 11 million QA pairs, together with a held-out manually verified benchmark containing multi-speaker and overlapping-speech examples.
Critical weaknesses
I do not see a critical weakness that would prevent acceptance. The following are mainly limitations or areas that could further strengthen the work.
-
Overlapping speech remains challenging.
SAE provides useful turn structure but does not explicitly separate simultaneous acoustic sources. The supplementary evaluation reports 73.42 Turn Recall but only 6.25% Segment Count Accuracy, suggesting that dense multi-speaker segmentation remains difficult. Since overlapping speech is an important part of the motivation, a breakdown by overlap severity would further clarify the current limitation. -
Structured annotation quality is less directly validated than final QA quality.
MSR-QA is constructed largely through an automatic pipeline, and the authors correctly acknowledge that registration errors can propagate to speaker associations. The manual inspection of 20,000 QA pairs reports 96% answer correctness, but this does not independently certify timestamps, bounding boxes, voice-face matches, or transcripts. More direct quality estimates for these structured fields would strengthen the dataset contribution. -
The main MSR-QA answer score relies on an LLM judge.
Overall answer accuracy is judged by GPT-4o. The authors provide a human validation study on 500 examples and report a correlation of 0.64, which is useful, but this does not directly establish judge accuracy or system-wise agreement. Reporting per-system human-vs-judge agreement or accuracy would further strengthen the evaluation.
Questions for authors
- Can the authors provide a performance breakdown as the amount/severity of overlapping speech increases?
- Could the authors report direct annotation-quality statistics for timestamps, speaker attribution / voice-face association, and transcripts in MSR-QA, in addition to final QA correctness?
- For the GPT-4o evaluation, can the authors provide per-system human-vs-judge agreement or accuracy rather than only pooled correlation?
- Do the authors envision extending SAE/CAST beyond speech-centric settings, for example to music, environmental sounds, or more general audio-visual source binding where the notion of a discrete speaker turn does not directly apply?
- What exact base model/checkpoint is CAST initialized from? From Table 1, CAST appears to build on Qwen3-Omni, but this is not stated explicitly. Please also clarify whether the
Qwen3-Omni + MSR-QAbaseline uses the identical backbone and training data.
4 comments
Response to Reviewer vgu8
Response to Concern 4: We thank the reviewer for suggesting an extension beyond speech. In music, instruments may play continuously or overlap, so discrete speaker turns cannot directly describe their activity. For a preliminary experiment, we took 1,706 instrument-labeled videos from the MUSIC-AVQA classification subset, concatenated the audio from three or four clips at a time, and used their clip-level labels as targets, e.g., <guitar><violin><piano>. We split the resulting concatenated samples into 434 training and 49 test examples, ensuring that source videos and their synthetic variants do not cross splits.
Some source clips contain multiple simultaneously sounding instruments and lack per-instrument onset and offset annotations. The concatenated labels therefore cannot reliably describe the timing of every source. In the SAE-inspired first stage, we update only the audio encoder and projector and evaluate instrument-label F1 without considering order or timing. As shown in Table R4, this stage achieves 51.5% F1, compared with 43.3% for the original Qwen3-Omni.
In the second stage, we fine-tune on the original videos underlying the training audio and their official QA pairs, updating the projector and language model with LoRA. Because the dataset lacks detailed per-instrument timestamps and bounding boxes, constructing reliable source-level audio-visual correspondences to evaluate ACSM is difficult. We therefore proceed directly from the first-stage audio encoder to video QA fine-tuning. Table R5 reports direct matches against official answers for 281 QA pairs on the 172 original videos underlying the held-out audio examples.
Table R4: Instrument recognition on concatenated audio
| Metric | Original Qwen3-Omni | SAE-inspired stage |
|---|---|---|
| Instrument-label F1 | 43.3% | 51.5% |
Table R5: Original-video QA
| Question type | Original Qwen3-Omni | Full model |
|---|---|---|
| Overall | 226/281 (80.4%) | 249/281 (88.6%) |
| Audio | 14/19 | 16/19 |
| Visual | 88/112 | 103/112 |
| Audio-visual | 124/150 | 130/150 |
These results provide a preliminary test of the two-stage approach in music.
Response to Concern 5: We thank the reviewer for this question and confirm that CAST is initialized from Qwen3-Omni-30B-Instruct, while the Qwen3-Omni + MSR-QA baseline uses the same backbone and training data, allowing us to assess the additional contributions of the CAST framework.
Thank you the authors for their detailed explanation. I am satisfied with the authors' rebuttal and looking forward to their released model weight and code as every modules (i.e, encoders and decoders) can potentially be applied to other audiovisual fields such as ASR, Active Speaker Detection, Speech Separation, etc.
Therefore, I keep my positive score and support the authors for acceptance!
We sincerely thank the reviewer for the positive feedback and continued support. We greatly appreciate your recognition of our work and its potential applications to broader audiovisual tasks. We look forward to releasing our model weights, data and code to facilitate future research.
Response to Reviewer vgu8
Response to Concern 1: We thank the reviewer and agree that overlapping speech remains challenging. SAE provides useful turn structure but does not explicitly separate simultaneous acoustic sources; this is a limitation of the SAE module, not a claim that CAST cannot handle overlapping speech. The supplementary SAE metrics are structural diagnostics of the audio encoder, not the final QA performance of CAST: low Segment Count Accuracy indicates that exactly matching the number of speaker turns is difficult in dense multi-speaker audio, while CAST further relies on ACSM, the original visual stream, and turn-aware audio representation for active-speaker grounding.
Following the reviewer’s suggestion, we add a breakdown by overlap severity on MSR-QA Bench, using the bins already defined in Supplementary Table 8.
Table R1: CAST performance on MSR-QA Bench stratified by audio-overlap ratio.
As shown in Table R1, CAST is relatively stable under mild overlap (≤25%) and degrades clearly as overlap increases. We will add this table and discussion to the revision.
**Response to Concern 2:**We thank the reviewer for this important concern. We therefore expanded human verification beyond the original 20,000-QA sample: 23 annotators worked for 5 days and verified 33,104 training QA pairs plus the full 3,677 QA pairs of MSR-QA Bench with about 1,600 QA pairs per annotator. As shown in Table R2, this yields 96.2% answer accuracy and, more importantly, direct field-level error estimates for the structured annotations.
Table R2: Field-level annotation error rates on the expanded verification set.
These field-level errors are not additive and can overlap; upstream field errors do not necessarily flip the final answer, so structured-field error rates being higher than the QA answer error rate is expected. We will add this table, the sampling strategy, and the annotation protocol to Appendix E.6.
Response to Concern 3: We thank the reviewer for this clarification. The original consistency analysis was indeed restricted to a 500-example subset. We have now manually reviewed all 3,677 QA pairs in MSR-QA Bench and repeated the Pearson correlation between LLM judge decisions and human correctness scores, for GPT-4o, Qwen3.5-397B and Gemini-3.5-Flash. As shown in Table R3, all three judges exhibit strong positive correlation with human review; GPT-4o’s full-bench correlation is consistent with its original 500-sample result, and the two additional judges show comparable consistency. This indicates that the judge–human consistency is not specific to GPT-4o. We will add these results to Appendix E.4.
Table R3: Human–judge consistency on MSR-QA Bench.
The metric is the Pearson correlation between binary judge decisions and continuous human correctness scores. The original GPT-4o result used 500 sampled examples; all new results use the full 3,677-QA benchmark.