acceptodds
Withdrawn from ICLR 2027

Can MLLMs Assist Forensic Sketch Identification via Semantic Reasoning?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When an eyewitness describes a suspect, the description is conveyed through language rather than pixels. Forensic face recognition, however, has largely been studied as a purely visual task, and the semantic content of verbal, written, and spoken facial descriptions has received little attention. We ask whether Multimodal Large Language Models (MLLMs) can bridge language and vision by jointly reasoning over facial sketches, textual descriptions, and spoken biometric narratives for forensic identity recognition. To study this, we introduce MS-Forensics, the first multimodal benchmark for sketch-based forensic identity recognition, which pairs facial sketches with textual and audio-based biometric descriptions built from five public face and sketch datasets. Through hypothesis-driven evaluation, we show that recognition degrades as the candidate set grows, and that the saturation of description-based recognition is explained by a collision model over attribute combinations. Fusing sketches with textual or audio descriptions outperforms unimodal inputs in most settings, but MLLMs struggle under fine-grained identity similarity and rely heavily on demographic cues; once demographics are removed, the eyes and forehead emerge as the most discriminative attributes. MS-Forensics provides a benchmark for evaluating MLLMs in AI-assisted forensic analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Forensic Sketch Recognition, Face Recognition, Multimodal understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uSABOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MS-Forensics, a benchmark for studying MLLM-based forensic sketch identification using sketches, textual facial descriptions, and audio descriptions. The benchmark combines five existing face or face-sketch datasets and evaluates several MLLMs under candidate-set scaling, multimodal fusion, pose variation, visually similar candidates, missing facial cues, and memory-based sketches. The experiments suggest that combining sketches with descriptions often improves closed-set identification, while performance degrades with larger or more similar candidate sets. The paper also analyzes the facial attributes used by MLLMs and reports a strong dependence on demographic cues.

Critical strengths

The paper studies a practically relevant but relatively underexplored problem and provides a reasonably broad evaluation protocol covering sketches, text, audio, pose changes, hard negatives, and incomplete observations. The analysis of candidate-set size and facial attributes is useful for understanding the limitations of current MLLMs rather than reporting only aggregate recognition accuracy.

The observation that MLLMs rely heavily on demographic attributes and become substantially less reliable for visually similar identities is also important for assessing whether these models are suitable for forensic applications.

Critical weaknesses

The main concern is the validity of the multimodal benchmark construction. In most experiments, facial descriptions are generated directly from the same facial photograph corresponding to the target identity, and that photograph also appears in the candidate set. The audio modality is largely obtained by converting these generated descriptions to speech. Therefore, the additional modalities contain privileged information derived from the gallery image rather than independent eyewitness evidence. The authors acknowledge this issue and explicitly treat the setting as an upper bound, but the proposed cross-instance protocol is left for future work. Moreover, the expert study reports that the generated descriptions only partially resemble actual witness descriptions, while the genuinely eyewitness-based evaluation contains only three cases. These issues make it difficult to conclude that the reported multimodal gains reflect realistic forensic semantic reasoning.

The experimental evidence is also not strong enough for the broader claims made by the paper. Most evaluations are small closed-set identification tasks, often with four candidates, and candidate-set scaling only reaches 12 identities. The comparison with established recognition methods is limited, with insufficient evaluation against strong sketch-to-photo recognition or multimodal biometric baselines. The paper also provides little analysis of variance across candidate sampling, prompts, or decoding settings. More fundamentally, several main findings, such as decreasing accuracy with more candidates and improved performance after adding complementary information, are expected and provide limited new insight at the level expected for ICLR. A more realistic independently collected description protocol, stronger baselines, repeated sampling with uncertainty estimates, and substantially larger candidate galleries would be needed to establish the contribution convincingly.

Questions for authors

See weakness.

Reviewer_v5a6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the use of MLLMs for forensic sketch-based face identification. The authors introduce MS-Forensics, a benchmark combining existing face-sketch datasets with generated facial descriptions and audio. Several MLLMs are evaluated under different modality combinations, candidate set sizes, pose variations, and facial attribute settings. The results suggest that multimodal inputs often improve recognition, while performance decreases under more challenging identification conditions.

Critical strengths
  • The paper studies a practical problem and evaluates several MLLMs across multiple datasets.
  • The experiments cover different recognition conditions, including candidate set size, pose variations, and missing facial attributes.
  • The analysis of model dependence on demographic and facial attributes is potentially useful.
Critical weaknesses
  1. Limited novelty. The work mainly combines existing models and modalities without introducing a new learning method. Most findings are already expected from previous multimodal recognition research.

  2. Concerns about synthetic data. The sketches and descriptions are partly generated from the ground-truth face images. Such data may not reflect the uncertainty and errors of real eyewitness evidence. This also raises concerns about whether the reported improvements represent genuine forensic reasoning.

  3. Limited benchmark contribution. MS-Forensics largely builds on existing datasets and automatically generated annotations. Its contribution beyond reorganizing existing resources is unclear.

  4. Insufficient evidence of semantic reasoning. The experiments show accuracy improvements but do not clearly demonstrate that these improvements come from the MLLMs' semantic reasoning ability. Stronger multimodal baselines and more controlled experiments are needed.

Questions for authors
  1. How can the authors ensure that the generated sketches and descriptions reflect the uncertainty of real forensic evidence, rather than preserving information from the ground-truth images?

  2. Can the authors demonstrate that the improvements come from MLLM-based semantic reasoning rather than simply providing more information through additional modalities?

  3. What are the main new insights provided by MS-Forensics that cannot be obtained from existing multimodal face recognition benchmarks?

Reviewer_PPGYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MS-Forensics, a benchmark for evaluating multimodal large language models on sketch-based identity identification. It combines sketches, photograph-derived facial descriptions, synthetic speech, and human-recorded narratives across five source datasets. The evaluation examines candidate-set size, modality combinations, attribute removal, pose variation, and visually similar distractors. Results show that combining descriptions with sketches often improves closed-set accuracy, while larger candidate sets and fine-grained identity similarity remain challenging. Additional expert assessments and three eyewitness-driven cases explore forensic relevance.

Critical strengths
  1. The benchmark studies an interesting combination of evidence sources that is underexplored in sketch recognition. Its evaluation extends beyond aggregate accuracy to candidate-set scaling, similar-looking distractors, missing attributes, and pose changes. These protocols reveal useful failure modes and make the benchmark potentially valuable for studying the limitations of multimodal identification.

  2. The paper includes several useful checks on its synthetic description pipeline. Cross-generator evaluations address generator–evaluator overlap, while memory-based sketches, human-recorded speech, and forensic expert assessments broaden the evaluation. The authors also acknowledge that photograph-derived descriptions represent unusually favorable evidence and explicitly discuss demographic dependence and deployment limitations.

Critical weaknesses
  1. The evidence for realistic forensic utility remains limited. Most descriptions are generated from the exact photograph included among the candidates, providing appearance information unavailable to an eyewitness. Appendix G acknowledges this limitation, but cross-generator tests and synthetic pose changes do not remove the shared-image origin. The three eyewitness-driven cases and expert judgments establish plausibility rather than identification reliability. This distinction matters because adding descriptions actually reduces FaceLLM accuracy on both IIIT-D sketch settings in Table 2.

  2. The collision analysis does not yet establish the proposed explanation for candidate-set degradation. Equation 3 applies an independent-collision formula using a probability already averaged across probe identities. With heterogeneous attribute-group sizes, the calculation should condition on the probe group before averaging and account for the candidate-sampling procedure. Moreover, the intersection between collision probability and accuracy has no established explanatory significance. The analysis needs empirical validation against observed collisions before supporting the abstract’s explanatory claim.

Questions for authors
  1. Are any existing experiments based on descriptions obtained independently of the gallery photograph, beyond the three eyewitness-driven cases? Please distinguish evidence for same-image description matching from evidence for identity recognition under independently acquired descriptions.

  2. How are the 361 attribute combinations defined from free-form descriptions, and how are distractors sampled? Does a probe-conditioned collision calculation agree with the observed frequency of collisions and recognition errors across candidate-set sizes?

  3. Appendix E compares full-gallery embedding retrieval with small-candidate MLLM identification. What results are obtained under identical candidate sets and evidence modalities? The current comparison does not isolate the contribution of semantic reasoning from differences in retrieval difficulty.

  4. The ethics statement says that no biometric data is transmitted to third-party services, whereas Section 3 describes GPT-4o reconstruction from sketches. Please clarify which images were processed, whether an external service was used, and how this processing complied with the source datasets’ permissions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.