Head Similarity: Modeling Structured Whole-Head Appearance Beyond Face Recognition

Many vision applications require consistency in both identity and appearance beyond biometric verification, particularly across viewpoints or when facial cues are limited. Conventional face recognition models prioritize identity discrimination and are not explicitly trained to distinguish appearance states within the same identity. We introduce Head Similarity, a whole-head matching task that determines whether two images depict the same person with a consistent head appearance. We formulate this as exact identity–appearance matching, where same-identity, same-state pairs are matches and all other pairs are non-matches. To retain identity discrimination beyond this binary objective, our method additionally encourages same-identity, different-state pairs to score above different-identity pairs. We construct a benchmark from long-form videos with human-verified identity and appearance-state annotations for both training and testing, covering diverse viewpoints, occlusions, and temporal appearance changes. We develop a Dual-CLS framework combining identity distillation, hierarchical similarity learning, global score supervision, and training-only patch-pair auxiliary learning. Experiments show improved identity–appearance matching over the evaluated baselines while retaining strong identity verification performance.