acceptodds
Under review as a conference paper at ICLR 2027

Binding Voices to Characters: Disentangled Cross-Modal Alignment for Multi-Speaker PercepTion

Abstract

Humans effortlessly perceive crowded multi-speaker scenes, resolving *who* is speaking, *when*, *where*, and *what* they say in real time. Replicating this capability in machines, however, remains a fundamental open challenge rooted in cross-modal speaker ambiguity that existing end-to-end models fail to resolve. A natural remedy is to scale training through data engineering. However, scaling training data alone may not sufficiently constrain the correspondence between speech segments and visible speakers, motivating explicit structural supervision for speaker binding. To address these limitations, we propose **CAST** (disentangled **C**ross-modal **A**lignment for multi-**S**peaker percep**T**ion), a framework that endows model with genuine multi-speaker perception capability through two complementary structural interventions. First, Speaker-aware Audio Encoding (SAE) restructures the audio encoder to explicitly segment the audio stream by speaker identity. Second, an Audio-Conditioned Spatial Modulation (ACSM) module performs audio-visual contrastive alignment to generate a Speaker Heatmap Prior, whose spatial tokens are superimposed onto the visual tokens to produce speaker-conditioned visual representations. CAST is then trained end-to-end via supervised fine-tuning followed by reinforcement learning to align perception with structured reasoning. We further release **MSR-QA** (Multi-Speaker Reasoning QA), a large-scale multilingual dataset comprising over **8000+ hours** of effective speech video and **11 million** QA pairs across movies, TV dramas, variety shows, and interview programs. Extensive experiments demonstrate that CAST significantly outperforms existing approaches by up to **34.30** percentage points on cross-speaker temporal and spatial reasoning, while providing transparent and verifiable reasoning processes. **Web Demo:** [https://cast-iclr.github.io/CAST](https://cast-iclr.github.io/CAST)

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multispeaker, Omni

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vgu8OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CAST, a framework for grounded multi-speaker audio-visual reasoning. CAST combines: (1) Speaker-Aware Audio Encoding (SAE), which inserts explicit speaker-turn delimiters into the audio representation; (2) Audio-Conditioned Spatial Modulation (ACSM), which aligns audio segments with visual patches and produces a speaker heatmap prior; and (3) structured perception reasoning with explicit <timestamp>, <speaker>, and <asr> fields before producing the final answer. The model is first trained with supervised fine-tuning and then optimized with GSPO using a heterogeneous reward that combines temporal grounding, ASR quality, speaker-description matching, factuality, and naturalness.

The paper also introduces MSR-QA, a large-scale multilingual dataset with approximately 8,000+ hours of speech video and 11 million QA pairs covering ten multi-speaker reasoning tasks. The held-out MSR-QA Bench contains 3,677 manually verified examples and includes substantial speaker overlap.

CAST achieves strong performance across AV-SpeakerBench, OmniMMI-SI, Omni-MMSI, and MSR-QA, and the component/reward ablations show that SAE, ACSM, SFT, RL, and the individual reward terms each make meaningful contributions.

Critical strengths
  1. Important and well-motivated problem.
    The paper targets a fundamental weakness of current AVLLMs: binding speech to the correct visible speaker in crowded multi-speaker scenes. The decomposition into who / when / what / where is intuitive and directly connected to the failure modes of existing models.

  2. Reusable architectural contribution through Speaker-Aware Audio Encoding (SAE).
    SAE introduces an explicit turn-aware audio representation rather than treating multi-speaker audio as a single entangled stream. This component is potentially useful beyond CAST and can serve as a building block for future multi-speaker AV models. Its importance is supported by the ablation: removing SAE causes a large drop in speaker attribution and overall performance.

  3. Explicit cross-modal grounding through ACSM.
    Rather than relying only on implicit multimodal fusion, ACSM explicitly aligns audio segments with visual patches and uses the resulting spatial prior to modulate visual representations. The visual-only and audio-shuffled controls provide useful evidence that the gains arise from meaningful audio-visual correspondence rather than static visual saliency.

  4. Strong training objective and informative ablations.
    The heterogeneous reward is carefully designed around the task: temporal IoU supervises when, WER supervises what, key-description matching supervises who, and judge-based rewards supervise factuality and naturalness. The component and reward ablations show large, interpretable degradations when individual pieces are removed, making the contribution of each design choice clear.

  5. Substantial dataset and reproducibility contribution.
    MSR-QA itself is a significant contribution, with approximately 8,000+ hours of multilingual video and 11 million QA pairs, together with a held-out manually verified benchmark containing multi-speaker and overlapping-speech examples.

Critical weaknesses

I do not see a critical weakness that would prevent acceptance. The following are mainly limitations or areas that could further strengthen the work.

  1. Overlapping speech remains challenging.
    SAE provides useful turn structure but does not explicitly separate simultaneous acoustic sources. The supplementary evaluation reports 73.42 Turn Recall but only 6.25% Segment Count Accuracy, suggesting that dense multi-speaker segmentation remains difficult. Since overlapping speech is an important part of the motivation, a breakdown by overlap severity would further clarify the current limitation.

  2. Structured annotation quality is less directly validated than final QA quality.
    MSR-QA is constructed largely through an automatic pipeline, and the authors correctly acknowledge that registration errors can propagate to speaker associations. The manual inspection of 20,000 QA pairs reports 96% answer correctness, but this does not independently certify timestamps, bounding boxes, voice-face matches, or transcripts. More direct quality estimates for these structured fields would strengthen the dataset contribution.

  3. The main MSR-QA answer score relies on an LLM judge.
    Overall answer accuracy is judged by GPT-4o. The authors provide a human validation study on 500 examples and report a correlation of 0.64, which is useful, but this does not directly establish judge accuracy or system-wise agreement. Reporting per-system human-vs-judge agreement or accuracy would further strengthen the evaluation.

Questions for authors
  1. Can the authors provide a performance breakdown as the amount/severity of overlapping speech increases?
  2. Could the authors report direct annotation-quality statistics for timestamps, speaker attribution / voice-face association, and transcripts in MSR-QA, in addition to final QA correctness?
  3. For the GPT-4o evaluation, can the authors provide per-system human-vs-judge agreement or accuracy rather than only pooled correlation?
  4. Do the authors envision extending SAE/CAST beyond speech-centric settings, for example to music, environmental sounds, or more general audio-visual source binding where the notion of a discrete speaker turn does not directly apply?
  5. What exact base model/checkpoint is CAST initialized from? From Table 1, CAST appears to build on Qwen3-Omni, but this is not stated explicitly. Please also clarify whether the Qwen3-Omni + MSR-QA baseline uses the identical backbone and training data.
4 comments
AuthorsOpenReview ↗

Response to Reviewer vgu8

Response to Concern 1: We thank the reviewer and agree that overlapping speech remains challenging. SAE provides useful turn structure but does not explicitly separate simultaneous acoustic sources; this is a limitation of the SAE module, not a claim that CAST cannot handle overlapping speech. The supplementary SAE metrics are structural diagnostics of the audio encoder, not the final QA performance of CAST: low Segment Count Accuracy indicates that exactly matching the number of speaker turns is difficult in dense multi-speaker audio, while CAST further relies on ACSM, the original visual stream, and turn-aware audio representation for active-speaker grounding.

Following the reviewer’s suggestion, we add a breakdown by overlap severity on MSR-QA Bench, using the bins already defined in Supplementary Table 8.

Table R1: CAST performance on MSR-QA Bench stratified by audio-overlap ratio.

Overlap ratio#ClipsOverall ↑SAA ↑tIoU ↑WER ↓
0%2,08969.1768.046.043.5
(0, 10%]71668.9967.645.544.7
(10%, 25%]32867.9965.543.646.3
(25%, 50%]27865.1162.738.350.9
(50%, 75%]14955.7053.329.861.1
(75%, 100%]11743.5941.816.175.6

As shown in Table R1, CAST is relatively stable under mild overlap (≤25%) and degrades clearly as overlap increases. We will add this table and discussion to the revision.

**Response to Concern 2:**We thank the reviewer for this important concern. We therefore expanded human verification beyond the original 20,000-QA sample: 23 annotators worked for 5 days and verified 33,104 training QA pairs plus the full 3,677 QA pairs of MSR-QA Bench with about 1,600 QA pairs per annotator. As shown in Table R2, this yields 96.2% answer accuracy and, more importantly, direct field-level error estimates for the structured annotations.

Table R2: Field-level annotation error rates on the expanded verification set.

Field#ErrorsError rate ↓Main source
ASR transcript1,5084.1%overlapping speech, music/noise, accented speech
Speaker-turn / timestamp boundary1,7294.7%short backchannels, interruptions, overlapping turns
Speaker identity / voice–face match1,9495.3%occlusion, off-screen speech, dubbing, appearance change
QA answer1,3993.8%propagated upstream errors or ambiguous questions

These field-level errors are not additive and can overlap; upstream field errors do not necessarily flip the final answer, so structured-field error rates being higher than the QA answer error rate is expected. We will add this table, the sampling strategy, and the annotation protocol to Appendix E.6.

Response to Concern 3: We thank the reviewer for this clarification. The original consistency analysis was indeed restricted to a 500-example subset. We have now manually reviewed all 3,677 QA pairs in MSR-QA Bench and repeated the Pearson correlation between LLM judge decisions and human correctness scores, for GPT-4o, Qwen3.5-397B and Gemini-3.5-Flash. As shown in Table R3, all three judges exhibit strong positive correlation with human review; GPT-4o’s full-bench correlation is consistent with its original 500-sample result, and the two additional judges show comparable consistency. This indicates that the judge–human consistency is not specific to GPT-4o. We will add these results to Appendix E.4.

Table R3: Human–judge consistency on MSR-QA Bench.

The metric is the Pearson correlation between binary judge decisions and continuous human correctness scores. The original GPT-4o result used 500 sampled examples; all new results use the full 3,677-QA benchmark.

JudgeEvaluation setPearson r ↑
GPT-4o (original)500 sampled examples0.643
GPT-4o (full)3,677 QA pairs0.650
Qwen3.5-397B3,677 QA pairs0.638
Gemini-3.5-Flash3,677 QA pairs0.644
AuthorsOpenReview ↗

Response to Reviewer vgu8

Response to Concern 4: We thank the reviewer for suggesting an extension beyond speech. In music, instruments may play continuously or overlap, so discrete speaker turns cannot directly describe their activity. For a preliminary experiment, we took 1,706 instrument-labeled videos from the MUSIC-AVQA classification subset, concatenated the audio from three or four clips at a time, and used their clip-level labels as targets, e.g., <guitar><violin><piano>. We split the resulting concatenated samples into 434 training and 49 test examples, ensuring that source videos and their synthetic variants do not cross splits.

Some source clips contain multiple simultaneously sounding instruments and lack per-instrument onset and offset annotations. The concatenated labels therefore cannot reliably describe the timing of every source. In the SAE-inspired first stage, we update only the audio encoder and projector and evaluate instrument-label F1 without considering order or timing. As shown in Table R4, this stage achieves 51.5% F1, compared with 43.3% for the original Qwen3-Omni.

In the second stage, we fine-tune on the original videos underlying the training audio and their official QA pairs, updating the projector and language model with LoRA. Because the dataset lacks detailed per-instrument timestamps and bounding boxes, constructing reliable source-level audio-visual correspondences to evaluate ACSM is difficult. We therefore proceed directly from the first-stage audio encoder to video QA fine-tuning. Table R5 reports direct matches against official answers for 281 QA pairs on the 172 original videos underlying the held-out audio examples.

Table R4: Instrument recognition on concatenated audio

MetricOriginal Qwen3-OmniSAE-inspired stage
Instrument-label F143.3%51.5%

Table R5: Original-video QA

Question typeOriginal Qwen3-OmniFull model
Overall226/281 (80.4%)249/281 (88.6%)
Audio14/1916/19
Visual88/112103/112
Audio-visual124/150130/150

These results provide a preliminary test of the two-stage approach in music.

Response to Concern 5: We thank the reviewer for this question and confirm that CAST is initialized from Qwen3-Omni-30B-Instruct, while the Qwen3-Omni + MSR-QA baseline uses the same backbone and training data, allowing us to assess the additional contributions of the CAST framework.

Thank you the authors for their detailed explanation. I am satisfied with the authors' rebuttal and looking forward to their released model weight and code as every modules (i.e, encoders and decoders) can potentially be applied to other audiovisual fields such as ASR, Active Speaker Detection, Speech Separation, etc.

Therefore, I keep my positive score and support the authors for acceptance!

We sincerely thank the reviewer for the positive feedback and continued support. We greatly appreciate your recognition of our work and its potential applications to broader audiovisual tasks. We look forward to releasing our model weights, data and code to facilitate future research.

Reviewer_2ic5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces CAST, a disentangled cross-modal alignment framework for multi-speaker video QA to resolve speaker ambiguity, opaque reasoning and dataset shortages. CAST uses SAE for speaker-based audio segmentation and ACSM to produce speaker heatmap priors via audio-visual contrastive alignment. It is trained with structural chain-of-thought SFT and reinforcement learning using hybrid automatic and LLM rewards. The authors release MSR-QA, a large multilingual dataset with 8,000+ hours of videos and 11M QA pairs. CAST achieves up to 34.30% gains on four benchmarks; ablations and qualitative examples confirm reduced speaker misattribution and hallucination.

Critical strengths
  1. The authors present MSR-QA, a large-scale multilingual dataset with over 8,000 hours of audiovisual footage and 11 million QA pairs. Curated from movies, TV series, variety shows and interviews, it features structured annotations along temporal, spatial and conversational axes and targets multi-speaker perception. The SAE and ACSM modules convert audiovisual alignment from opaque fusion into an interpretable spatial prior, and the pipeline outputs three independently verifiable evidence types: timestamps, character descriptions and ASR transcripts. Distinct from vanilla text-based chain-of-thought, this transparent, debuggable inference design is a strong innovation for multimodal video VQA.

  2. The CAST framework revises reinforcement learning supervision beyond final-answer accuracy alone. It combines automatically computable metrics (tIoU, WER, character attribute matching) and LLM-based factuality and naturalness rewards. By regularizing both intermediate evidence and end predictions, it mitigates breakdowns in speaker attribution, timestamping and transcription, and prevents the undesirable shortcut of correct answers supported by invalid reasoning. The empirical evaluation is thorough, covering component ablations, reward ablations and heatmap analysis. CAST outperforms both general omni models (Qwen3-Omni, Gemini, MiniCPM-o) and AV-specialized models, with consistent gains across four benchmarks instead of overfitting to one dataset.

Critical weaknesses

The paper exhibits clear strengths and weaknesses. Please analyze the weakness concerning dataset quality: the models employed for annotation and QA generation are insufficiently capable. Additionally, analyze CAST with respect to its video length boundary, overlapping speech among multiple speakers, character entry and exit from frames, and the ambiguity of attribute descriptions.

Moreover, analyze the non-objectivity of evaluation. Although this issue is hard to avoid for open-domain tasks, please investigate whether the results would change drastically if GPT-4o were replaced with an alternative judge model.

Questions for authors

1、The Overall Score of MSR-QA adopts GPT-4o as the judge. Open-ended VQA does not have a single ground-truth answer, and LLM judging introduces subjectivity and preference bias. The judge model itself is prone to hallucinations, and no consistency validation between human-annotated subsets and the LLM judge has been performed.

2、SAE only produces speaker turn boundaries <|spk|>. The paper explicitly states that SAE does not handle mixed overlapping waveforms. For overlapping speech where multiple people speak simultaneously, SAE cannot separate concurrent voices. Overlapping dialogues are prevalent in real-world movies, TV dramas and interviews. The model exhibits limited capability under this scenario, and the paper lacks dedicated experiments and analysis for overlapping speech.

3、CAST appears incapable of tracking the same speaker across long time spans, especially with the adopted heatmap design. Please analyze the time-duration boundary that CAST can handle. Additionally, discuss its robustness when characters enter or exit the frame and when character appearances change.

4、MSR-QA is a large-scale automatically constructed dataset, presumably built with ASR, speaker diarization and MLLM annotations. Specifically, Qwen2.5-VL-7B is used for captioning, Gemini-2.5-Flash-Lite-Preview for QA generation, and Qwen3-Omni-30B-A3B for filtering. The reliability and accuracy of these relatively weak models raise concerns. Although structured annotations are designed, it is hard to fully eliminate diarization errors, ASR mistakes and character identity labeling errors given the massive data volume. The paper lacks statistical analysis on dataset annotation error rates and human-verified subsets.

5、The model leverages attribute descriptions such as “the woman with red hair wearing a blue coat” for speaker identification. Visual attribute descriptions are prone to confusion when there are lookalike characters in the frame, when characters face away from the camera, or when facial regions are blurry. Please provide corresponding analysis.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2ic5

Response to Concern 1: We thank the reviewer for this clarification. The original consistency analysis was indeed limited to a 500-example subset. We have now manually reviewed all 3,677 QA pairs in MSR-QA Bench and repeated the Pearson correlation analysis between LLM judge decisions and human correctness scores for GPT-4o and two additional judges, Qwen3.5-397B and Gemini-3.5-Flash. As shown in Table R1, all three judges show strong positive correlation with human review; GPT-4o's full-bench correlation is consistent with its original 500-sample result, and the two additional judges show comparable consistency. This indicates that the judge–human consistency is not specific to GPT-4o. We will add these results to Appendix E.4.

Table R1: Human–judge consistency on MSR-QA Bench.

The metric is the Pearson correlation LLM judge decisions and human correctness scores. The original GPT-4o result used 500 sampled examples; all new results use the full 3,677-QA benchmark.

JudgeEvaluation setPearson r ↑
GPT-4o (original)500 sampled examples0.643
GPT-4o (full)3,677 QA pairs0.650
Qwen3.5-397B3,677 QA pairs0.638
Gemini-3.5-Flash3,677 QA pairs0.644

Response to Concern 2: We thank the reviewer for this comment. We clarify that “SAE does not separate mixed waveforms” is a limitation of the SAE module itself, not a claim that CAST cannot handle overlapping speech. SAE only provides coarse turn boundaries and a structured audio interface; boundary markers alone are insufficient for multi-speaker attribution, especially under overlap. CAST therefore introduces ACSM, which uses audio-visual contrastive alignment to produce a Speaker Heatmap Prior and fuses it with the original visual stream and turn-aware audio representation, assisting active-speaker localization at the multimodal semantic level. SAE and ACSM are progressive and complementary. We will state this explicitly and add a stratified overlap analysis on MSR-QA Bench, using the overlap bins already defined in Supplementary Table 8.

Table R2: CAST performance on MSR-QA Bench stratified by audio-overlap ratio.

Overlap ratio#ClipsOverall ↑SAA ↑tIoU ↑WER ↓
0%2,08969.1768.046.043.5
(0, 10%]71668.9967.645.544.7
(10%, 25%]32867.9965.543.646.3
(25%, 50%]27865.1162.738.350.9
(50%, 75%]14955.7053.329.861.1
(75%, 100%]11743.5941.816.175.6

Under mild overlap (≤25%), CAST remains stable: Overall drops only from 69.17 at 0% to 67.99 at (10%, 25%], with limited changes in SAA, tIoU, and WER, showing robustness to light overlapping speech. As the overlap ratio increases, accuracy declines.

Response to Concern 3: We thank the reviewer. We clarify that CAST is a spatial-understanding-based multi-speaker grounding framework, not a speaker-tracking or ReID model with long-term identity memory. Its goal is to build a verifiable evidence chain of “who, when, what, and where” within the current temporal window, rather than maintaining persistent identity across long videos. When a character enters the frame and speaks in the current window, ACSM provides spatial cues; when the character leaves the frame or speaks off-screen, the heatmap no longer provides valid localization, and the model mainly relies on turn structure, ASR, and context. Under appearance changes, the <speaker> attribute description and the original visual stream partially mitigate the issue but cannot replace ReID. The paper already states that the heatmap does not assign a persistent identity to each speaker, and Table 5 shows that ACSM relies on audio-visual alignment rather than persistent visual tracking. Long-term identity consistency and tracking involve persistent memory and ReID, which is beyond the scope of this paper; we will clarify this scope in the revision and list it as future work.

AuthorsOpenReview ↗

Response to Reviewer 2ic5

Response to Concern 4: We thank the reviewer for this important concern. We agree that annotation error rates and a human-verified subset should be analysised, and we will add them to the revision. We first clarify that MSR-QA is not generated end-to-end by a single weak model. It is built with a fine-grained staged pipeline: Qwen2.5-VL-7B is restricted to visible attribute description, not long-term identity inference; Gemini-2.5-Flash-Lite-Preview generates QA conditioned on structured evidence with self-correction and consistency checks; Qwen3-Omni-30B-A3B is used in the filtering stage with full text and multimodal evidence, where its task is reduced to consistency verification rather than generation. The filter uses a conservative threshold: it prefers rejecting correct samples over admitting incorrect ones, so it mainly controls false positives rather than guaranteeing recall.

For quality control, the original Appendix E.6 sampled 20,000 QA pairs and found 96.0% answer-level correctness. Following the reviewer’s suggestion, we expanded human verification: 23 annotators worked for 5 days and verified 33,104 training QA pairs plus the full 3,677 QA pairs of MSR-QA Bench. The per-annotator load is about 1,600 QA pairs over five days, about 320 QA pairs per annotator per day. The expanded verification yields 96.2% answer accuracy. Field-level error rates are reported separately; they are not additive and can overlap, since a single QA error may be caused by an upstream ASR, diarization, or identity error.

Table R3: Field-level annotation error rates on the expanded verification set.

Field#ErrorsError rate ↓Main source
ASR transcript1,5084.1%overlapping speech, music/noise, accented speech
Speaker-turn / diarization boundary1,7294.7%short backchannels, interruptions, overlapping turns
Speaker identity / description1,9495.3%occlusion, off-screen speech, dubbing, appearance change
QA answer1,3993.8%propagated upstream errors or ambiguous questions

We will add the sampling strategy, annotation protocol, error-rate statistics, and limitations to the revised Appendix E.6.

Response to Concern 5: We thank the reviewer for this important point. We clarify three aspects. First, samples with severe face/person blur that make the <speaker> description unverifiable against the video frame were removed during data quality filtering and do not enter MSR-QA; the dataset release page will include the construction instructions and examples, and the revision will add the filtering criteria and the filtered proportion. Second, facing away from the camera does not mean the speaker is unidentifiable: <speaker> is a verifiable visible-attribute description, not a face-recognition label. When a character faces away, CAST uses non-facial cues such as clothing, hairstyle, body shape, standing/seating position, relative location, audio-visual spatial alignment, and dialogue context. The data construction does not simply discard all back-facing samples; they are retained as long as non-facial cues and audio evidence support a checkable description, and the revision will add stratified statistics on such samples. Third, when lookalike characters appear in the same frame, we require the speaker description to include spatial position, e.g., “the red-haired woman on the left, in the foreground, sitting on the purple sofa,” and we cross-check it with the ACSM Speaker Heatmap Prior, the original visual stream, and dialogue context.

Reviewer_2ic5OpenReview ↗

Response to Rebuttal

I have read the response from the authors. Most of my concerns have been addressed. I choose to keep my rate.

We sincerely thank the reviewer for taking the time to review our response. We are glad that we were able to address most of your concerns, and we appreciate your positive assessment and support for our work.

Reviewer_XGypOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-speaker audio-visual question answering, where a model must jointly infer who is speaking, when and where they speak, what they say, and how the resulting conversational evidence supports an answer. The authors introduce CAST, a three-stage framework built around: (1) Speaker-aware Audio Encoding (SAE), which inserts speaker-turn delimiter tokens into the audio sequence; (2) Audio-Conditioned Spatial Modulation (ACSM), which aligns audio segments with visual patches and uses a Speaker Heatmap Prior to modulate visual features; and (3) post-training with supervised structured perception traces followed by reinforcement learning under a heterogeneous reward that combines temporal IoU, ASR quality, speaker-description matching, factuality, and naturalness. The paper additionally presents MSR-QA, claimed to contain more than 8,000 hours of effective speech video and 11 million multilingual QA pairs. The task is important and the reported results are consistently stronger than the listed baselines. However, my main concern is that the paper introduces a relatively elaborate stack of architectural modules, structured generation, multi-stage training, metric-specific rewards, and LLM judges without establishing that this complexity is necessary. In particular, the paper lacks a carefully controlled, data-centric baseline in which the same backbone is trained on the same structured turn/speaker/timestamp/transcript annotations using ordinary multi-task SFT. Moreover, despite the title, the proposed representation does not appear to retain an explicit per-turn voice-to-character assignment. The current experiments do not sufficiently isolate the gains of the proposed architectural and post-training components from those of structured data and denser supervision. A controlled simple baseline is needed to establish whether the additional system complexity is warranted. Separately, the claims of ‘disentangled’ and ‘causal’ alignment would benefit from more direct intervention-based validation. Taken together, the mismatch between the central claims and the actual representation, the lack of a controlled simple baseline, and the absence of essential reproducibility information make the paper not ready for acceptance; I therefore recommend rejection in its current form.

Critical strengths
  1. Multi-speaker video understanding is substantially harder than generic audio-visual QA because speaker attribution, temporal grounding, transcription, spatial localization, and conversational reasoning must be solved jointly. The paper identifies meaningful failure modes, including assigning speech to a visually salient but silent person.
  2. Requiring timestamp, speaker, and ASR fields produces a more inspectable evidence interface than unconstrained free-form chain-of-thought. These fields can be evaluated separately, which is valuable for debugging and safety-critical applications.
  3. The paper evaluates on four benchmarks and reports both component and reward ablations. The ablations exhibit interpretable trends: removing SAE most strongly damages speaker attribution, removing ACSM strongly damages temporal grounding, and removing the corresponding reward terms primarily affects their target metrics.
  4. If the construction, quality control, licensing, and release are sufficiently documented, the scale and structured annotations of MSR-QA could be useful to the community independently of the proposed model.
  5. The paper notes that the shared speaker delimiter is not a persistent identity token, that SAE does not separate overlapping waveforms, that the heatmap is not a calibrated speaking probability, and that the generated evidence trace does not expose every internal computation. These are important and intellectually honest caveats.
Critical weaknesses
  1. The added complexity is not justified by a sufficiently strong simple baseline. CAST combines SAE, ACSM, a dual visual pathway, structured trace generation, SFT, group-based RL, several hand-designed rewards, attribute parsing, and LLM judges. The most informative baseline would use the same backbone, the same MSR-QA samples, the same temporal and speaker annotations, and the same training budget, but simply serialize each example as a structured list of turns with persistent character/track IDs and train with standard multi-task SFT. Although the paper reports “Qwen3-Omni + MSR-QA,” it is unclear whether this baseline receives the same structured supervision, target format, data mixture, and compute. Thus the paper does not isolate the benefit of algorithmic complexity from the benefit of better data and denser labels.

  2. The core voice-to-character binding remains partly implicit. A shared <|spk|> delimiter is reasonable if the goal is merely to mark turn boundaries and avoid arbitrary speaker-label permutations. However, because exactly the same token is inserted before every segment, SAE explicitly represents when a turn changes but not who the new speaker is. The same person returning in non-contiguous turns receives no stable identity, and simultaneous speakers are not represented. Thus, describing SAE as segmenting the stream ‘by speaker identity’ appears stronger than the representation supports: it solves speaker-change detection, not persistent speaker attribution. More importantly, ACSM defines a single patch score h_j = max_i s_ij, collapsing the audio-segment index i into one heatmap. Consequently, the representation no longer records which audio segment activated which visual person. In a scene where character A speaks one turn and character B speaks another, the heatmap may highlight both characters without preserving the assignments turn 1 -> A and turn 2 -> B. The final mapping is therefore still left to the downstream LLM. A per-turn heatmap H_k, an audio-turn-to-face-track assignment matrix, or another persistent identity representation would align more directly with the title and central claim.

  3. The structured trace is inspectable but not demonstrated to be faithful. The factorization p(Y_think, Y_ans | C) still allows the answer to condition directly on the raw multimodal context C. The model can therefore bypass the generated evidence or produce it as a post-hoc explanation. The current results do not establish that the final answer causally depends on the timestamp/speaker/ASR fields. Similarly, the use of the term “causal alignment” is stronger than the evidence provided: an audio-shuffling control establishes dependence on audio, but not causal identification of speaker identity.

  4. Reproducibility and evaluation details are insufficient in the provided manuscript. This is particularly serious because the paper repeatedly defers essential material to Appendix B (alignment and modulation interfaces), Appendix C (the representation interface), Appendix D (reward and evaluation details), and Appendix E (dataset details), yet the provided 12-page submission contains no appendix: pages 10-12 are references. Consequently, essential information remains unavailable, including the exact CAST backbone and parameter count, training schedules and compute, reward weights, positive-pair assignment for the contrastive loss, judge models and prompts, parsing rules, dataset construction and splits, and metric definitions such as SDM, SAA, and TAJ. Model-specific parsers and native prompting formats may be reasonable, but they also introduce degrees of freedom, particularly when several baselines obtain near-zero scores. No confidence intervals, repeated runs, or statistical tests are reported. The missing appendices prevent a complete assessment of both methodological validity and reproducibility.

  5. The absolute perception performance remains limited and hard cases are under-analyzed. CAST reports WER 46.32, active-speaker selection accuracy 38.43, pointing accuracy 35.65, and temporal IoU 43.54. These are improvements, but they do not yet support language suggesting reliable or genuine multi-speaker perception. Results are not stratified by overlapping speech, off-screen speakers, rapid shot changes, occlusion, repeated turns from the same character, visually similar characters, or silent but salient people. SAE explicitly does not separate overlapping waveforms, which is central to the motivating setting.

Questions for authors
  1. Can the authors provide a controlled “structured data only” baseline using the exact same backbone, MSR-QA split, structured annotations, and training budget, but no SAE, no ACSM, and no LLM-judge RL? For example, could standard SFT over serialized turn_id, timestamp, persistent speaker/face-track ID, transcript, and evidence-turn fields achieve comparable performance?
  2. How exactly is the “up to 34.30 percentage points” improvement in the abstract calculated? Which benchmark, metric, and comparison model does it refer to, and why is the strongest reported baseline not used for that comparison?
  3. What is the exact LLM/omni-modal backbone, parameter count, training compute, number of stages/updates, and inference latency of CAST? Please report an accuracy-versus-compute comparison with Qwen3-Omni + MSR-QA.
  4. How are the positive correspondences i+ and j+ in the audio-visual contrastive objective constructed? Do they rely on ground-truth face boxes, active-speaker labels, face tracks, or temporal co-occurrence? How is ambiguity handled when several face patches share the same temporal window?
  5. Why is the audio index removed through h_j = max_i s_ij? How does the model preserve the assignment between a specific turn and a specific character after this max operation? Could the authors evaluate a per-turn heatmap or explicit turn-to-track assignment matrix?
  6. How does CAST handle overlapping speakers, off-screen speakers, and the same speaker appearing in multiple non-contiguous turns? Please provide stratified results for these cases and clarify what SAE predicts at inference time when no speaker-attributed segmentation is provided.
  7. Can the authors perform faithfulness interventions on the generated evidence? For example, if the predicted speaker field is swapped while the raw context is held fixed, does the final answer change accordingly? What happens if the answer decoder is restricted to the structured evidence rather than retaining direct access to C?
  8. Which LLMs, prompts, sampling settings, and reward weights are used for the factuality and naturalness judges? Is GPT-4o also involved in training-time rewards? How sensitive are the conclusions to using a different judge, and how well do the automatic scores correlate with blinded human evaluation?
  9. Please provide full MSR-QA data statistics and governance details: source and language distributions, automatic versus human annotation proportions, quality-control protocol, inter-annotator or audit accuracy, train/validation/test splitting, deduplication and leakage checks, licensing, and release plan.
  10. Can the authors report results over multiple random seeds with confidence intervals, as well as ablations on external benchmarks rather than only MSR-QA? It would also be valuable to compare ACSM against established active-speaker detection or visual sound-localization modules under the same data and backbone.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Area Chair,

Thank you for considering our manuscript. We have addressed the reviewers’ concerns and incorporated the corresponding revisions into the paper. We briefly summarize the three main updates below:

  1. We added an analysis of the correlation between LLM judge decisions and human correctness scores in Section 4 (Table 1 and Lines 344–347).
  2. We added field-level manual verification results for 33,104 training samples in Section 4 (Table 2 and Lines 349–353).
  3. We added an analysis of CAST’s performance across different speech-overlap ratios in Section 5.2 (Table 5 and Lines 427–431).

We also corrected typos and formatting issues and revised unclear or inaccurate wording throughout the manuscript.

We appreciate your time and consideration, and we hope that our revised manuscript will be favorably considered for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.