acceptodds
Under review as a conference paper at ICLR 2027

HIM-CoT: Reasoning over the Human-Instrument-Music Chain for Musical Performance Deepfake Detection

Abstract

Audio-visual deepfake detection increasingly exploits cross-modal correspondence, yet existing methods predominantly target talking or singing heads, where detection relies on speech–lip synchronization and facial artifacts. However, this formulation breaks down for instrumental performances. Sound production is mediated by physical interaction with an instrument. Human motion drives instrument state, which in turn generates music. This structured causal chain imposes constraints that face-centric detectors cannot capture. Therefore, the core challenge is whether the observed performance provides a physically plausible explanation for the accompanying sound. We formalize this as an evidence-grounded reasoning problem: given structured measurements of music, motion, and performer–instrument contact, can a model determine which modality violates the causal production process? To address this question, we first construct HIM-DF, a benchmark with bidirectional forgeries that isolate audio and visual inconsistencies within the same Human–Instrument–Music structure. We then propose HIM-CoT, a reasoning framework that converts structured performance measurements into queries over pretrained multimodal representations. By explicitly representing measurement uncertainty, modality-specific evidence, and cross-modal temporal discrepancies, HIM-CoT grounds multistage chain-of-thought generation in forensic cues rather than end-to-end feature fusion. Experiments demonstrate that structured evidence retrieval and explicit discrepancy modeling improve both detection accuracy and interpretability over joint embedding baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
music-video deepfake detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R35rOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HIM-DF, a bidirectional deepfake benchmark for instrumental performances. The authors propose HIM-CoT, a detection framework that exploits the physical constraints of the human-instrument-music production chain. Extensive experiments show that HIM-CoT significantly outperforms baselines in detection accuracy, cross-generator generalization, and reasoning fidelity.

Critical strengths
  • The concept of production-chain consistency shifts the detection paradigm from superficial artifact hunting to causal reasoning. By leveraging the strict kinematic and acoustic constraints of human-instrument interactions, it provides a highly intuitive mechanism to expose cross-modal forgeries.

  • HIM-DF fills a critical gap in the literature as the first bidirectional benchmark tailored for instrumental performances.

Critical weaknesses
  • The framework's reliability is strictly bottlenecked by external measurement tools. The current observation mask handles missing data but lacks confidence-aware gating, causing severe false positives when low-confidence measurements are misinterpreted as forensic evidence.

  • The HIM-DF excludes samples where both modalities are simultaneously synthesized, leaving the method's efficacy against the most threatening, cross-modally conditioned real-world attacks unverified.

Questions for authors
  • If an attacker generates highly synchronized joint audio-video forgeries for instrumental performances, does the "production-chain consistency" still provide sufficient discriminative cues to expose the forgery?

  • What are the end-to-end computational cost and inference latency of the entire pipeline, and is this architecture feasible for near-real-time deployment?

I would be inclined to raise my score if the authors can address these concerns.

Reviewer_7aBtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HIM-CoT for deepfake detection and forgery-modality attribution in instrumental performance videos, using structured pitch, hand-motion, contact, and audio–visual cues with cross-modal discrepancy modeling and three-stage reasoning. It builds HIM-DF (~7,000 samples, 12 instruments) and achieves 89.3% accuracy and 94.2% AUC, +15.0 points over the best baseline; validation is mainly limited to short solo clips.

Critical strengths
  1. Valuable task and benchmark. HIM-DF studies instrumental performance forgeries through bidirectional audio/video manipulation and modality attribution, extending detection beyond facial and speech cues.
  2. Evidence for structured measurements. Reported accuracy falls from 89.3% to 66.3% without structured evidence, while tests on unseen generators retain an advantage. These results support domain measurements as useful complements to pretrained multimodal representations.
Critical weaknesses
  1. Limited evidence for production-chain reasoning. The structured features are largely clip-level statistics, and only 22.9% of within-recording pairs are flagged as fake after a 4-second audio shift. Existing controls demonstrate some cross-modal sensitivity but do not clearly separate physical consistency from shared synthesis artifacts, limiting support for the proposed mechanism.
  2. Insufficient controls to isolate the methodological gains. Despite same-backbone and fusion baselines, the evaluation lacks strong music-synthesis and general AI-video detectors, alongside clearly specified, capacity-matched fusion alternatives using identical measurements. The specific benefit of the proposed retrieval and discrepancy mechanisms therefore remains uncertain. Missing repeated-run results and source-grouped confidence intervals further limit assessment of robustness.
Questions for authors
  1. Does the model truly use action–sound correspondence? A 4-second within-recording audio shift leaves most samples undetected. Which evidence shows the model checks performance–sound matching rather than generation artifacts? This is central to the paper’s claim.
  2. Is the gain from extra measurements or the proposed design? With the same pitch, hand-motion, and contact measurements, would a simpler fusion model under identical training still clearly underperform? Current comparisons do not answer this.

Suggestions for Future Work

  1. Model note- or event-level cross-modal alignment (e.g., pitch/onset vs. key position/strike timing) rather than clip-level statistics.
  2. Test multiple generators, both-modality forgery, ensemble settings, and weaker artifact cues.
Reviewer_fDuXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates deepfake detection in musical performances by modeling the relationships among human motion, instrument interaction, and music. The authors introduce HIM-DF, a benchmark containing 6,998 samples across 12 instrument categories, including both synthesized video and synthesized music. Based on this benchmark, the authors propose HIM-CoT, a framework that integrates structured acoustic, motion, and contact measurements with pretrained multimodal representations. The framework uses cross-modal discrepancy modeling and three-stage chain-of-thought reasoning to detect forged content and identify the manipulated modality. Experiments show that HIM-CoT achieves 89.3% accuracy on HIM-DF, outperforming existing unimodal, multimodal, and CoT-based baselines. The results are promising, although the extent to which these improvements arise from physical production-chain reasoning remains unclear.

Critical strengths
  1. The paper investigates an interesting and relatively underexplored problem in multimodal deepfake detection. The proposed HIM-DF benchmark includes both audio and visual forgeries across 12 instrument categories, providing a useful resource for studying the consistency between musical performances and their corresponding sounds.

  2. The experimental evaluation is relatively comprehensive, and the reported improvements are substantial. HIM-CoT substantially outperforms existing baselines on HIM-DF (Table 1). The ablation studies, cross-generator experiments, and explanation evaluations provide additional evidence supporting the effectiveness of the proposed method.

Critical weaknesses
  1. The CoT annotations used for training are generated with access to the ground-truth authenticity labels and forgery types. This may encourage the model to generate explanations that support predetermined conclusions rather than independently derive them from the evidence. Although Appendix C.3 provides an observation-only reasoning experiment, the authors should further include a reasoning intervention experiment, such as masking or replacing individual reasoning stages, to verify whether the generated reasoning actually contributes to the final classification decisions.

  2. The proposed method relies on the quality of structured feature extraction, but the robustness of its missing-evidence handling mechanism is not sufficiently evaluated. Appendix A.8 indicates that all observation masks are valid in HIM-DF, while Appendix C.10 reports that 32 out of 75 classification errors are related to measurement uncertainty. This raises concerns about the reliability of the framework when feature extraction becomes inaccurate or incomplete.

  3. The model shows relatively limited sensitivity to temporal inconsistencies within the same recording. In Appendix C.7, shifting the audio by 4 seconds increases the fake prediction rate from 9.4% to only 22.9%. Although the authors acknowledge this limitation, it suggests that fine-grained temporal correspondence remains challenging for the proposed framework.

Questions for authors
  1. In Appendix A.7, the CoT annotations are generated with access to the ground-truth labels. During training, does the classification head use hidden states from teacher-forced reference reasoning or from the model's own generated reasoning?

  2. Since all observation masks are valid in the current dataset, could the authors provide a simple feature-masking experiment to evaluate the robustness of the proposed mechanism when some structured measurements are unavailable?

  3. In Appendix C.7, the fake prediction rate reaches only 22.9% when the audio is shifted by 4 seconds. Could the authors clarify whether this behavior varies across different instrument categories, particularly between instruments with distinct note onsets and those producing sustained sounds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.