acceptodds
Withdrawn from ICLR 2027

Head Similarity: Modeling Structured Whole-Head Appearance Beyond Face Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many vision applications require consistency in both identity and appearance beyond biometric verification, particularly across viewpoints or when facial cues are limited. Conventional face recognition models prioritize identity discrimination and are not explicitly trained to distinguish appearance states within the same identity. We introduce Head Similarity, a whole-head matching task that determines whether two images depict the same person with a consistent head appearance. We formulate this as exact identity–appearance matching, where same-identity, same-state pairs are matches and all other pairs are non-matches. To retain identity discrimination beyond this binary objective, our method additionally encourages same-identity, different-state pairs to score above different-identity pairs. We construct a benchmark from long-form videos with human-verified identity and appearance-state annotations for both training and testing, covering diverse viewpoints, occlusions, and temporal appearance changes. We develop a Dual-CLS framework combining identity distillation, hierarchical similarity learning, global score supervision, and training-only patch-pair auxiliary learning. Experiments show improved identity–appearance matching over the evaluated baselines while retaining strong identity verification performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Head Similarity, Identity-Appearance Matching, Whole-Head Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ynwGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper authors introduced using head similarity to compare the person with the head appearance instead of just face appearace for recognizing a person. Authors argue that there area cases where the face images are occluded or even invisible, also there are some area where the difference are captured at different state, which make the comparison on face not 100% feasible. Authors in this paper introduced 1) a dataset and benchmark for comparing the whole head instead of just face for a new head-level recognition task, and 2) a head level comparison for comparing the head-level losses and patch-level losses for reconstruction and identify the identity in the image. Authors showed that for the head-level identification it is able

Critical strengths
  1. The overall method and proposal is simple and easy to follow. Authors introduced a new benchmark/problem, a new dataset and a new method to address the limitation in face recogntion for person (re)identification, which includes other information from hair and other area information to provide supplementary information for recognizing who the person is.

  2. Authors have conducted experiments on both the face-based re-id sets as well as the newly-introduced head-level dataset. Authors shown that with more information than jsut the face, along with the new model, they are able to boost the perofmrance of the recognition.

Critical weaknesses
  1. We need to numerically distinguish the head-level similarity along with the clothes level info used in the person-(re)identification. From Figure 8, w/ same backgorund, we can clearly see the similar clothes patterns for the same background setting. Authors have provided some good visualization for background changing on the right half of the figure, but such experiment config is not reflected in the numerical results. Instead of just viusalizations and to understand how much improvement it indeed comes from the head recognition instead of the clothes, authors can include 1) heatmap for where the model is looking at, and 2) results comparison for same and different background across the face-centered model and head-centered model to reveal the real gain of the method.

  2. The results are pretty interesting as it indeed shows having the whole head as recogntion is able to provide better TAR@FAR x when x is small. The major difference might come from the hair style/color, ability to recognize the person head (or face) better from different view point, etc. Wonder if it is possible for authors to conduct some further ablation to analyze which part has actually contributed to this improvement (since there are mutliple factors taht can contribute to the final performance, as what authors shown in Figure 2).

  3. [Minor and discussion] Based on the experiment setting and visualization results (please correct me if it is not correct) it seems the hair color is proviing a strong hint for TAR@FAR x when x is small. In the face recognition scenario, where the face shape are of least changes usually, hair style, color or even the length of the hair can change on a daily base. For such a in-the-wild recognition cases, I am a little bit concern how much we can gain from face recognition.

  4. [Minor] For Head Similarity Survey, even being told to ignore the background and clothing, such information is still a strong hint for recognize the two person for area beyond head. It would be great if authors provide these survey with their "different background" setting, or even make the whole background to a same color for conducting such survey comparison.

Questions for authors

Please see above. The overall results are interesting but please provide more analyze for understanding where these improvmeent come from. Also please include more details and description of the dataset (whether their hair style etc changes, how many are of pose difference etc).

Reviewer_6SKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Head Similarity, a task that checks whether two head images show the same person with the same appearance. The authors build a video benchmark with human-verified identity and appearance labels. Their Dual-CLS model improves this matching while keeping strong identity recognition.

Critical strengths
  1. Good Motivation: identity-invariant face representation may be bad in VideoGen applications, so preserving appearance features is also important.

  2. Useful Benchmark: The paper builds a video-based benchmark with human-verified identity and appearance-state labels. It covers diverse poses, including many profile and back views, and uses disjoint identities for training and testing.

Critical weaknesses
  1. Poor Presentation: (1) Too many unnecessary terms "appearance-state", "identity–appearance". (2) Bad quality Figure1,2,5,8; Tiny content hard to read.

  2. Unfair Comparison: In Tables 2 and 3, most baselines are either applied directly to whole-head images or never trained on HeadSim, so their failure is expected. Against the only adapted baseline, Dist. AdaFace, the method is actually worse on identity verification in Table 2. Fine-tuning the face model on whole-head images with standard identity losses is not compared, so the need for distillation is not justified.

  3. Motivation Not Validated: The paper is motivated by video generation and face-invisible scenarios, but neither is tested. There is no experiment on generated content, and no per-pose results show gains on profile or back views.

Questions for authors

Good Motivation, but need some improvement. Check the weaknesses.

Reviewer_yaSmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • A whole-head matching task that determines whether two images show the same person in the same appearance state (hairstyle, accessories, etc.).
  • Only R1 (same identity, same state) is positive; R2 and R3 are negative, with an auxiliary R1 > R2 > R3 ordering.
  • HeadSim-Head is built from long-form videos via shot segmentation, tracking, identity clustering, and human verification.
  • A Dual-CLS ViT initialized from AdaFace, trained with identity distillation, hierarchical similarity, global score, and patch-pair losses.
  • TAR@FAR=10⁻³ of 0.919 on HeadSim-Head (AdaFace 0.222); whole-head verification 0.890 (distillation-only 0.908).
Critical strengths
  • The problem formulation is good. The observation that face recognition deliberately discards information that generation and editing need is correct, and the R1/R2/R3 definitions and the auxiliary ordering are cleanly separated.
  • The ablations are thorough. Table 6 shows that adding the R2 > R3 term raises P(R2 > R3) from 0.815 to 0.864 and complete-order accuracy from 0.788 to 0.837, which directly checks whether the intended structure emerges.
Critical weaknesses
  • My main concern is that R1 pairs always come from the same shot. In Sec. 4.4, track segments are produced within a shot, R1 pairs are drawn from the same segment, and R2 pairs from different segments. R1 pairs therefore share not only head appearance but every condition of that shot. Sec. 3 treats illumination, background, and minor pose as nuisance factors, but since R1 never crosses a shot, invariance to them is not measured. Background replacement with COCO changes only the background. Human verification confirms labels; it does not change this construction. Without a cross-shot R1 evaluation, it is not possible to tell whether the numbers in Table 3 reflect “same head” or “same shot.”
  • The comparisons in Table 3 cover face recognition models, TransReID, and the authors’ own variants. Since this task rewards appearance-sensitive features, general visual representations such as DINOv2 or CLIP are the natural baseline, and they are absent.
Questions for authors

Q1) Can a single appearance state span multiple videos or shots? In Table 1, videos per identity (8.4, 13.3) and average states (8, 13) are nearly identical. If cross-shot R1 pairs exist, could you report TAR@FAR on that subset alone?

Q2) What is the source and licensing of the videos, and will the dataset be released? The paper only says “long-form videos.”

Q3) The ViT size, input resolution, and the hard negative mining procedure in A.1 are not specified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.