acceptodds
Under review as a conference paper at ICLR 2027

Learning Text-Conditioned State-Space Recurrence for Multimodal Medical Logical Deepfake Verification

Abstract

Generative models can produce biomedical images that are physically plausible yet contradict their accompanying text. We call such inconsistent image–text pairs multimodal medical logical deepfakes and study their detection as a defensive consistency-verification task. Image-only detectors cannot assess agreement with an unseen caption, while global visual self-attention incurs quadratic token interactions. We propose the Multimodal Bilateral Consistency Network (MM-BCNet), whose Text-Conditioned Vision Mamba (TC-VMamba) generates the selective state-space parameters , , and from each local visual token and a global text representation. This establishes direct text-to-parameter conditioning within the selective scan, enabling linear-complexity cross-modal recurrence across high-resolution visual sequences without token expansion. A single-query Semantic-Guided Affinity module supplies a spatial prior, and a Seeing Before Reasoning schedule learns visual forensic cues before multimodal fine-tuning. We construct a synthetic benchmark covering Semantic Misalignment and Visual Inpainting, with source-grouped splits and generation masks for edited-region localization, a task distinct from establishing semantic contradiction. Against baselines, MM-BCNet improves consistency classification, localization, and unseen-generator accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, medical image forensics, deepfake detection, state-space models, vision-language consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ShJPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MM-BCNet for detecting inconsistencies between biomedical images and accompanying text. Its main component conditions Mamba’s selective state-space parameters on visual features and a global text representation. A semantic-affinity module and two-stage training support consistency classification and edited-region localization. Evaluation uses a synthetic benchmark with modified captions and inpainted images.

Critical strengths
  • The paper addresses a meaningful problem that image-only biomedical forensic detectors cannot solve. It explicitly distinguishes image manipulation from image–text contradiction and distinguishes generation masks from contradiction evidence.
  • Directly conditioning the selective recurrence on text is a plausible fusion design. The paper appropriately limits its linear-complexity claim to the fusion component and acknowledges the visual encoder’s quadratic attention cost.
Critical weaknesses
  • Incomplete positioning against closely related work. ReSCIND/Integscan (AAAI 2026), which combines biomedical forgery synthesis with language-guided state-space localization, is omitted. BioTamperNet (ICLR 2026) is cited briefly, but its architectural relationship to the proposed method deserves clarification. Please explain the differences in affinity construction, semantic fusion, and localization, and establish the specific contribution of directly conditioning recurrence parameters on text.
  • The validity of the contradiction labels remains insufficiently established. Editing an image or changing a caption entity does not necessarily create a medically verifiable contradiction. Particularly concerning is Section 4.3, where images generated from the source caption are composited into the original image and labeled inconsistent. Manual review is mentioned, but the main paper does not establish the contradiction-validation criteria, expert agreement, or label-error rate.
  • The contrastive objective permits a text-copying shortcut. Equation 14 aligns text with a representation that already receives that same text through fusion. The claim that simultaneous classification and mask losses strictly prevent this shortcut is unsupported: different feature components can support the supervised tasks while the contrastive projection reproduces text information. Showing that the classifier uses images would not, by itself, resolve this concern.
  • The evaluation reporting needs clarification. Table 1 describes pooled micro-evaluation, yet its overall AUC and mAP values appear to be arithmetic averages of the two scenario scores. These generally differ from metrics recomputed over pooled predictions. The expert-adjudicated Integrity-100 evaluation is also mentioned without numerical results in the main paper, limiting assessment of transfer beyond synthetic construction.
Questions for authors
  • How were semantic contradictions verified, especially for caption-conditioned generated composites? What were the reviewer qualifications, agreement, and rejection rates?
  • Does the benchmark include edited but semantically consistent pairs? Without these cases, how is consistency reasoning distinguished from manipulation detection?
  • What evidence specifically rules out text copying in the contrastive objective, beyond demonstrating image dependence of the final classifier?
  • Were overall AUC and mAP computed from pooled predictions or averaged across scenarios? Please clarify the aggregation.
  • What accuracy and uncertainty were obtained on Integrity-100, and how were its labels adjudicated?
Reviewer_3Si9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a deep learning model called Multimodal Bilateral Consistency Network (MM-BCNet) to detect misleading medical image–text combinations. Unlike traditional deepfake detection methods that focus only on image manipulation, this model examines both medical images and their descriptions to identify contradictions. It uses Text-Conditioned Vision Mamba (TC-VMamba) and a Semantic-Guided Affinity module to combine visual and textual information effectively. The model follows a two-stage training approach called Seeing Before Reasoning, where it first learns to detect image manipulations and then checks image–text consistency. Experimental results show an overall detection accuracy of 87%, outperforming several existing methods. The study highlights the potential of multimodal AI in detecting medical misinformation and improving the reliability of biomedical information.

Critical strengths
  1. ⁠Multimodal nature of the inconsistency score, where text and image reasoning is fine-tuned through multistage fine-tuning.
Critical weaknesses

1.⁠ ⁠The entire caption is represented into one vector, which can lead to loss of granularity, which will be critical for finding out inconsistency. 2.⁠ ⁠Insufficient ablations to establish how the fine-tuning, TC-VMamba, SGA all affect the outcome. 3. Presentation needs to be improved.

Questions for authors
  1. How can a vision inconsistency prediction implementation compete with a multimodal one, isn't the first one boun to 50% from the beginning?
  2. Why is there no variation where you compare the results against a vision (image only model) which is achievable through a 0 offset from T_cls?
  3. The image backbone (vision encoder) fine tuned on BioFors never sees a diffusion in painting, or any other genrative images, how can it then learn to differentiate between actual image and an AI generated one?
Reviewer_F56POpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a defensive forensic task for identifying semantic contradictions between biomedical images and their accompanying textual descriptions. Existing detectors only spot visual artifacts and cannot assess image‑text semantic consistency, while conventional vision‑language models relying on global self‑attention incur quadratic computational cost and may over‑depend on language priors and overlook subtle local visual evidence. To tackle these limitations, the authors present MM‑BCNet. Its key module TC‑VMamba generates selective state‑space model parameters conditioned on both local visual tokens and global text embeddings, realizing linear‑complexity cross‑modal recurrence without expanding visual token sequences. A lightweight SGA module produces text‑driven spatial priors to re‑weight visual features. A two‑stage training strategy first pre‑trains the model to capture visual forensic artifacts, then conducts multimodal fine‑tuning for consistency reasoning to reduce linguistic shortcut learning. The authors build a synthetic benchmark derived from Open‑PMC‑18M with two deepfake categories. Comprehensive experiments are conducted against unimodal forensic baselines and multimodal large vision‑language models. MM‑BCNet achieves state‑of‑the‑art results on consistency classification, pixel‑level manipulation localization, and zero‑shot generalization over unseen image generators. Ablation studies confirm the effectiveness of SGA, TC‑VMamba, and the two‑stage training scheme. This work validates that text‑conditioned selective state‑space models can combine low‑level visual forensic cues and high‑level semantic reasoning to safeguard biomedical multimodal data integrity.

Critical strengths
  1. This paper formalizes multimodal medical logical deepfake verification, targeting semantic contradictions between biomedical images and paired text. It clearly distinguishes this task from conventional unimodal visual forgery detection, which cannot identify text‑image semantic inconsistency even for artifact‑free authentic images.
  2. The proposed TC‑VMamba injects text conditioning directly into selective state‑space model parameters, enabling text‑guided cross‑modal recurrence without inserting extra text tokens into visual sequences. Combined with the lightweight single‑query Semantic‑Guided Affinity (SGA) module, it produces text‑aware spatial priors while preserving original 2‑D visual patch layout.
  3. The Seeing Before Reasoning schedule first pre‑trains on visual forensic artifacts before multimodal fine‑tuning. This design mitigates harmful linguistic shortcut learning, forcing the model to ground consistency decisions on visual evidence rather than textual priors alone.
  4. A synthetic benchmark derived from Open‑PMC‑18M covers two practical failure modes: Semantic Misalignment (text‑only edits) and Visual Inpainting, with pixel‑level ground‑truth masks for tampering localization. It supplies valuable test resources for medical multimodal forensics research.
  5. Extensive comparisons are performed against strong unimodal forensic detectors and multimodal large vision‑language model baselines. Evaluations cover consistency classification, pixel‑level tampering localization, and cross‑generator zero‑shot generalization. Ablation studies quantitatively validate the contribution of each core component, and supplementary stress tests rule out trivial linguistic shortcut solutions.
Critical weaknesses

1.The main evaluation dataset is synthetically constructed via LLM text rewriting and diffusion‑based image inpainting. Though a small external probe set Integrity‑100 is used, large‑scale real‑world human‑curated medical logical deepfake samples are absent. Synthetic patterns may not fully replicate authentic inconsistencies seen in real‑world biomedical publications or clinical records, raising domain‑shift risks for practical deployment. 2. While TC‑VMamba achieves linear‑complexity selective scanning for visual token sequences, the full pipeline still includes a standard ViT visual encoder with quadratic self‑attention cost. The manuscript does not provide thorough wall‑clock latency and peak‑memory benchmarks against competitive cross‑attention multimodal baselines, making practical efficiency gains hard to quantify. 3. Important hyper‑parameters including SGA initial scale λ, SSM state dimension N, loss weighting coefficients, and Mamba scan‑direction settings lack systematic ablations. Justification for these choices mostly depends on fixed empirical settings without sensitivity analysis. 4.The paper lacks detailed qualitative and quantitative breakdown of failure cases. It remains unclear whether prediction errors stem from hard semantic ambiguities, low‑quality manipulated visual patches, text‑entity parsing failures, or model architectural limitations.

Questions for authors

1.Since the main dataset is constructed synthetically via LLM text modification and diffusion‑model inpainting, what are the major differences between your synthetic logical‑deepfake samples and real‑world inconsistent image‑caption pairs from actual biomedical publications? Could you provide quantitative statistics or manual human‑review analysis to characterize synthetic‑artifact bias within your dataset? How do you expect performance to degrade when transferring to fully real‑world medical multimodal data? 2. You state that TC‑VMamba selective scan has linear complexity with respect to visual tokens, but the full pipeline still contains a standard ViT encoder with quadratic self‑attention. Could you provide wall‑clock inference latency and peak GPU memory comparisons between your full MM‑BCNet and cross‑attention multimodal baselines under identical input resolutions and hardware conditions, to quantify practical efficiency improvements? 3.Could you break down your model’s misclassified test cases quantitatively? For example, how many failures are caused by ambiguous clinical semantics, poor‑quality synthetic image inpainting artifacts, incorrect medical entity understanding, or architectural limitations? Are there consistent patterns among false‑positive / false‑negative samples? 4. Your primary experiments are conducted on your Open‑PMC‑18M derived benchmark. How does MM‑BCNet perform on fully independent hospital‑sourced clinical multimodal datasets? Additionally, how does the model behave under more complex mixed forgery scenarios (e.g., simultaneous text semantic modification plus unrelated visual inpainting)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.