acceptodds
Under review as a conference paper at ICLR 2027

RASA: Role-specific Acoustic–Semantic Alignment for Telecom Fraud Detection

Abstract

Distinguishing fraudulent calls from legitimate conversations requires interpreting callers' claims and requests within the surrounding conversational context. Acoustic behaviors such as disfluency, speech rate acceleration, and frequent interruptions can supplies crucial acoustic evidence when assessing deceptive intent. However, transcript-based large language models may overlook these cues, while direct audio input into large audio-language models (LALMs) does not guarantee their adequate preservation by recognition-oriented encoders. So that we propose Role-specific Acoustic–Semantic Alignment (RASA), which combines a recognition-oriented speech encoder with a reconstruction-oriented neural audio codec. Separate Fixed-Length Query Adapters aggregate their representations into compact soft-token sequences for joint reasoning within an LLM. Using transcript- and script-informed teacher supervision from ParaScam, our synthesized call dataset, role-specific alignment establishes a semantic anchor and trains the acoustic pathway to contribute acoustic evidence. The aligned model supports downstream adaptation using only audio and task labels, with no transcripts required at inference. On TeleAnti, RASA achieves 98.13% fraud-detection weighted F1 and 88.61% average weighted F1 across fraud detection, fraud-type identification, and scene classification. Under a 500-call downstream adaptation setting, RASA improves fraud-type F1 by 27.37 percentage points over its semantic-only variant, supporting the value of explicitly aligned acoustic evidence for fine-grained fraud recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Telecom Fraud Detection, Large Audio-Language Models, Semantic-Acoustic Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_N4gyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RASA, which combines a Whisper-based semantic branch and a WavTokenizer-based acoustic branch through staged alignment training for telecom fraud detection and fine-grained classification. The paper addresses non-lexical acoustic information that may be lost in transcription, a practically relevant problem. Table 2 also shows that the proposed training procedure improves fraud-type identification with 500 downstream training examples. However, the methodological novelty, fairness of the baseline comparisons, and empirical support for the core mechanisms remain insufficient. The current results primarily demonstrate the effectiveness of the proposed system on a specific data split, rather than adequately supporting the stronger claims regarding the use of acoustic evidence, representation disentanglement, and overcoming the performance bottleneck of single-encoder architectures.

Critical strengths

The paper addresses non-lexical acoustic information that is often lost in transcription, providing clear motivation and practical relevance. The dual-branch semantic–acoustic architecture and staged alignment strategy are clearly presented. The ablation results provide preliminary evidence that the approach improves fine-grained fraud-type identification with limited downstream labeled data.S

Critical weaknesses
  1. The main components of the proposed method include dual encoders, fixed-length adapters based on learnable queries, and staged alignment that learns semantic information before acoustic information. As acknowledged in the paper’s related work, SALMONN and WavLLM already employ dual-encoder architectures, query-based compression interfaces have established precedents, and BLSP-Emo follows a training strategy that first establishes semantic competence before learning supplementary information.

  2. Table 1 includes text-based models, direct audio models, and models adapted to the target task. However, the results for key strong baselines, including AntiFraud-Qwen2-Audio and the SAFE series, are taken from their original publications, whereas Appendix D states that this study selects 16,060 records from TeleAnti and constructs its own training, validation, and test splits. The paper does not sufficiently clarify whether the cited results use the same test samples, category definitions, and metric computation protocols. Consequently, the current table does not provide a reliable basis for establishing RASA’s advantages over existing methods.

  3. In Table 2, the full model achieves a fraud-type F1 of 81.00%, compared with 53.63% for the semantic-only variant, providing important evidence of gains in fine-grained classification. However, for binary fraud detection, the acoustic-only variant achieves 99.69% F1, exceeding the full model’s 96.91%; the dual-branch variant without acoustic alignment also achieves 97.49%. These results indicate that the complete design does not yield consistent benefits across tasks. The authors should explain why the semantic branch and alignment training improve fraud-type identification but not binary fraud detection.

  4. The paper repeatedly motivates its approach through behaviors such as hesitation, accelerated speech, and frequent interruptions, but provides neither real audio examples with timestamps nor statistics on the distribution of these phenomena across classes and the model’s corresponding performance. It therefore remains unclear whether the model actually uses these acoustic cues or instead exploits additional semantic information, speaker differences, recording conditions, or data-source characteristics. I recommend presenting cases where the semantic-only variant fails but the full model succeeds, alongside failure cases, and matching the model’s acoustic explanations to manually annotated events in the audio. More convincing validation would hold the textual content fixed while varying acoustic properties such as speech rate or pauses, or introduce controlled substitutions of the acoustic-branch input to assess whether predictions change as expected. Classification F1 alone is insufficient to establish that semantic and acoustic representations have been disentangled or that the model performs the acoustic-evidence-based reasoning described in the paper.

Questions for authors

See weakness

Reviewer_wi1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RASA combines Whisper and WavTokenizer through separate fixed-length adapters. Transcript-based alignment establishes a semantic anchor; script-informed alignment trains the acoustic branch. Frozen adapters and downstream LoRA are evaluated on a TeleAnti subset, including a 500-label setting with a large fraud-type gain over the semantic-only ablation.

Critical strengths

Within the train-500 ablations, full RASA improves fraud-type F1 from 53.63 to 81.00 over the semantic-only variant (Table 2), providing meaningful evidence that the second aligned pathway helps. The frozen-adapter design supports adaptation using audio and labels only. Sequential and unaligned-branch controls make the alignment contribution partially testable (§5.3).

Critical weaknesses

Table 1 mixes published baseline scores with results on a new 4,818-call test split and eight fraud types (Appendix D). The cited benchmark reports a different split and seven types. Without common evaluation, the claimed overall lead is not established. The ablations establish predictive benefit, not that it comes from fraud-relevant delivery rather than lexical or synthesis cues. Scripts add fillers and other markers; no matched-content prosody intervention or source-disjoint acoustic test is reported (Appendix B). The 500-label claim also relies on thousands of teacher-labeled alignment calls.

Questions for authors

Can the strongest adapted baselines be evaluated on exactly the same split, label mapping, and scoring code, with uncertainty across training seeds? What are ParaScam's source calls and their overlap with TeleAnti partitions? What evidence separates delivery cues from transcript, speaker, and synthesis cues, using baselines given equal alignment data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.