- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MS-Forensics, a benchmark for studying MLLM-based forensic sketch identification using sketches, textual facial descriptions, and audio descriptions. The benchmark combines five existing face or face-sketch datasets and evaluates several MLLMs under candidate-set scaling, multimodal fusion, pose variation, visually similar candidates, missing facial cues, and memory-based sketches. The experiments suggest that combining sketches with descriptions often improves closed-set identification, while performance degrades with larger or more similar candidate sets. The paper also analyzes the facial attributes used by MLLMs and reports a strong dependence on demographic cues.
Critical strengths
The paper studies a practically relevant but relatively underexplored problem and provides a reasonably broad evaluation protocol covering sketches, text, audio, pose changes, hard negatives, and incomplete observations. The analysis of candidate-set size and facial attributes is useful for understanding the limitations of current MLLMs rather than reporting only aggregate recognition accuracy.
The observation that MLLMs rely heavily on demographic attributes and become substantially less reliable for visually similar identities is also important for assessing whether these models are suitable for forensic applications.
Critical weaknesses
The main concern is the validity of the multimodal benchmark construction. In most experiments, facial descriptions are generated directly from the same facial photograph corresponding to the target identity, and that photograph also appears in the candidate set. The audio modality is largely obtained by converting these generated descriptions to speech. Therefore, the additional modalities contain privileged information derived from the gallery image rather than independent eyewitness evidence. The authors acknowledge this issue and explicitly treat the setting as an upper bound, but the proposed cross-instance protocol is left for future work. Moreover, the expert study reports that the generated descriptions only partially resemble actual witness descriptions, while the genuinely eyewitness-based evaluation contains only three cases. These issues make it difficult to conclude that the reported multimodal gains reflect realistic forensic semantic reasoning.
The experimental evidence is also not strong enough for the broader claims made by the paper. Most evaluations are small closed-set identification tasks, often with four candidates, and candidate-set scaling only reaches 12 identities. The comparison with established recognition methods is limited, with insufficient evaluation against strong sketch-to-photo recognition or multimodal biometric baselines. The paper also provides little analysis of variance across candidate sampling, prompts, or decoding settings. More fundamentally, several main findings, such as decreasing accuracy with more candidates and improved performance after adding complementary information, are expected and provide limited new insight at the level expected for ICLR. A more realistic independently collected description protocol, stronger baselines, repeated sampling with uncertainty estimates, and substantially larger candidate galleries would be needed to establish the contribution convincingly.
Questions for authors
See weakness.