- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FAMILIAR, inspired by research in cognitive science, which suggests that familiarity with a face stems from learning how it varies across repeated encounters, with different faces exhibiting distinct dimensions of variation. Unlike existing methods that rely on predefined scenarios for multi-dimensional face synthesis, FAMILIAR models a virtual identity as a "distribution of occasions." This approach maps an identity to a unique "occasion space," moving beyond the simple addition of arbitrary perturbations to a base identity. Extensive experiments validate two applications: first, sampling scenarios for new identities to achieve state-of-the-art (SOTA) performance in synthetic face recognition; and second, performing de-identification by swapping identities while keeping the source scenario constant, thereby minimizing identity leakage while preserving attribute utility comparable to the original images.
Critical strengths
- The motivation behind this paper is compelling. Drawing inspiration from cognitive science theories, it moves beyond fixed scenario settings. Instead, it posits that the form of the scenario itself should vary by individual and be learned rather than prescribed, which is an approach that yields valuable insights regarding identity and de-identification.
- The experimental comparisons in the paper are extensive and rigorous. For instance, with the generator fixed, comparisons are made against an empirical pool, unconditional flows, diagonal/low-rank Gaussians, mixture models, conditional flows, and mismatched-identity flows.
Critical weaknesses
- Evidence regarding the distribution of the generated data and the differences in how various identities undergo transformation remains unclear. The paper’s central argument is that each virtual identity possesses a unique set of appearance combinations, with distinct dimensions of variation for each identity. However, the paper fails to specify the exact content of these appearance combinations and lacks concrete statistical or qualitative results. Furthermore, the qualitative results presented appear to lack diversity, for instance, solely of frontal-view images with only slight angular deviations.
- Unequal training data confounds the state-of-the-art claim. FAMILIAR's generator is trained on WebFace-21M (20.7M images, 1.1M identities), whereas most baselines in Table 1 train on CASIA-WebFace (~0.5M images, ~10K identities), so the 0.6 – 1.1% margins cannot be attributed to the learned occasion prior rather than to roughly 40× more real training data. A FAMILIAR variant trained only on CASIA-WebFace is needed to support the claim.
Questions for authors
As weakness mentioned, I would like to provide qualitative results and scene distribution statistics to demonstrate diversity and that different identities have their own suitable scenarios. Also, please provide FAMILIAR test results on the same training dataset. If the authors provide the attribute statistics and interpretable-space evidence requested in W1, together with a same-data comparison for W2, I am open to raising my score.
4 comments
Response to Reviewer TMQJ [2/2]: diversity of the generated data
W1 (continued). Diversity of the generated data.
The mostly frontal views in our figures reflect how we selected display examples. The generated data are as varied in pose as real photographs. In our released 0.5M dataset, 21.3% of images exceed an absolute yaw of 30° and 8.7% exceed 45°, against 21.5% and 8.3% for WebFace-21M photographs. We measured all datasets with the same estimators, and the table below compares their pose statistics:
| Dataset | Abs. yaw > 30° | Abs. yaw > 45° | Yaw IQR | Pitch IQR |
|---|---|---|---|---|
| Real (WebFace-21M) | 21.5% | 8.3% | 22.2° | 9.2° |
| FAMILIAR | 21.3% | 8.7% | 22.0° | 9.0° |
| MorphFace | 25.9% | 13.0% | 33.4° | 12.2° |
| SteerFace | 17.0% | 7.6% | 25.7° | 8.3° |
| UIFace | 17.4% | 6.8% | 24.6° | 8.3° |
| IDperturb | 16.4% | 6.0% | 25.7° | 7.8° |
| Vec2Face (HSFace10K) | 26.5% | 13.1% | 28.7° | 8.6° |
| Vec2Face+ | 23.9% | 15.0% | 28.0° | 11.3° |
| DCFace | 13.6% | 5.3% | 21.0° | 7.6° |
| CemiFace | 12.8% | 4.2% | 22.2° | 6.8° |
| DigiFace | 21.8% | 7.9% | 28.2° | 8.8° |
In the table, all synthetic datasets except HSFace10K and DigiFace-1M are the datasets of Table 1(b). From each, we use the first 2,000 identities with at least 20 usable images (up to 50 images each). For real photographs, we use 2,000 WebFace-21M identities with 20 images each. Image shares pool all of these images, whereas within-identity statistics, such as yaw and pitch IQR, use 20 random images per identity, averaged over identities.
These results show that FAMILIAR's pose statistics match those of real photographs. It is the only synthetic dataset within 4.8% (relative) of the real value on all four. Seven of the nine other datasets have a wider within-identity yaw IQR than real photographs (24.6° to 33.4°, against 22.2°), yet the five of them that we evaluated under one recognition recipe (MorphFace, SteerFace, UIFace, IDperturb, and Vec2Face+) reach 92.73% to 93.60% average accuracy at 0.5M (Table 1(b)), against 94.79% for FAMILIAR, so a wider pose spread alone does not make better training data.
FAMILIAR's expression distribution is also the closest to that of real photographs, as shown in the table below:
| Dataset | Neutral | Happy | Sad | Surprise | Fear | Disgust | Anger | TV to real |
|---|---|---|---|---|---|---|---|---|
| Real (WebFace-21M) | 28.3 | 39.3 | 4.4 | 14.6 | 1.1 | 2.2 | 10.2 | — |
| FAMILIAR | 28.2 | 36.1 | 6.8 | 14.3 | 1.9 | 2.7 | 10.0 | 3.7 |
| MorphFace | 24.3 | 38.2 | 7.8 | 10.6 | 3.4 | 3.0 | 12.6 | 8.9 |
| SteerFace | 32.6 | 35.7 | 6.1 | 9.5 | 2.6 | 1.5 | 12.1 | 9.4 |
| UIFace | 33.8 | 34.5 | 7.2 | 8.3 | 1.8 | 1.9 | 12.5 | 11.3 |
| IDperturb | 32.6 | 33.1 | 7.0 | 8.7 | 2.0 | 1.9 | 14.7 | 12.3 |
| Vec2Face (HSFace10K) | 41.3 | 38.6 | 3.2 | 8.9 | 0.2 | 0.8 | 7.0 | 13.0 |
| Vec2Face+ | 43.0 | 37.8 | 2.2 | 8.8 | 0.2 | 0.6 | 7.4 | 14.7 |
| DCFace | 36.7 | 47.1 | 2.8 | 5.0 | 0.6 | 0.7 | 7.2 | 16.2 |
| CemiFace | 40.6 | 44.0 | 2.1 | 5.0 | 0.8 | 0.5 | 7.0 | 17.1 |
| DigiFace | 21.5 | 15.0 | 12.1 | 18.8 | 16.5 | 4.6 | 11.7 | 31.1 |
In the table, values are percentages of images, and TV is the total variation distance to the real distribution, computed before rounding. The revision adds both tables to Appendix B.13.
In the table above, every expression class of FAMILIAR is within 3.2 points of its real rate, and its TV of 3.7 is the lowest of all datasets (at least 8.9 for the others). Within each identity, FAMILIAR's spread is also close to that of real photographs, e.g., an expression entropy of 1.53 versus 1.54 bits and an age SD of 5.50 versus 5.51 years.
For qualitative evidence, Figure 5(a) in the main text shows estimated yaw from −40° to +35°, and the new Figure 7 (Appendix B.13) shows the range that the occasion space can render for nine released identities, with most occasion codes edited toward a target pose, expression, or lighting.
W2 and Q1. Unequal training data.
We agree. We retrained FAMILIAR on the CASIA-WebFace (CASIA) and compared it with the six CASIA-trained methods that are publicly available, under the same recognition recipe and protocol. FAMILIAR still leads on every benchmark:
| Method | Generator training data | LFW | CFP-FP | CPLFW | AgeDB | CALFW | Avg. | IJB-C | IJB-C |
|---|---|---|---|---|---|---|---|---|---|
| DCFace | CASIA | 98.75 | 87.60 | 82.83 | 89.23 | 90.79 | 89.84 | 69.16 | 82.11 |
| CemiFace | CASIA | 99.02 | 90.19 | 87.71 | 91.31 | 92.17 | 92.08 | N/A | N/A |
| MorphFace | CASIA | 99.28 | 93.97 | 88.89 | 92.26 | 92.52 | 93.38 | 82.64 | 88.76 |
| SteerFace | CASIA | 98.98 | 94.44 | 88.71 | 92.43 | 92.98 | 93.51 | 81.51 | 89.59 |
| UIFace | CASIA | 99.26 | 93.23 | 88.23 | 90.74 | 92.21 | 92.73 | 81.40 | 87.66 |
| IDperturb | CASIA | 99.01 | 93.44 | 88.48 | 93.28 | 93.53 | 93.55 | 85.21 | 89.94 |
| FAMILIAR | CASIA | 99.32 | 94.98 | 89.82 | 93.78 | 94.11 | 94.40 | 87.62 | 91.90 |
In the table, every synthetic dataset has 0.5M images, and all rows use our IR-50/ArcFace recipe and the IJB-C template protocol. N/A: the recognizer trained on CemiFace's 0.5M dataset maps low-quality IJB-C frames to near-identical features, so TAR is near zero. The revision summarizes these results in Table 1(b) and gives the per-benchmark results and training details in Appendix B.11.
We would be glad to know whether the attribute statistics, the interpretable-space evidence (the new Figure 5), and this same-data comparison address your question.
Thanks for the detailed response. Most of my concerns have been addressed, so I will keep my score.
Thanks for your positive comments and insightful feedback, which have greatly strengthened our paper.
Response to Reviewer TMQJ [1/2]: content of occasions
We thank Reviewer TMQJ for the positive assessment and for concrete suggestions that led to new results.
We agree that what we call an occasion corresponds to the review's "scenario" or "scene". In FAMILIAR, an occasion is a learned code that captures the image-specific appearance of one photograph, without attribute labels (§3.1), and the revised §3.2 now makes its relation to identity explicit: "All identities share one occasion space, which spans photographic scenarios such as pose, expression, and illumination. Each identity has its own distribution over this space, so identities can differ in the occasions they favor."
W1 and Q1. Content of occasions and identities' own scenarios.
The revision adds the new Figure 5 to the main paper, and more results about attribute statistics and per-identity grids to Appendix B.13.
What occasions contain. An occasion is what the generator needs, besides identity, to reproduce a photograph (§3.1). Its content is interpretable. Linear probes on the occasion codes of held-out real photographs recover every measured attribute better than probes on the identity embedding, as shown in the table below:
In the table, we report R2 on 5,000 photographs of 200 unseen people, with probes fitted on 2,000 training identities. For expression, we report 7-class balanced accuracy (chance 0.14), and lighting direction is the left-right brightness asymmetry.
These results show that pose, expression, lighting, color, and sharpness are carried mainly by the occasion code (0.62 to 0.98, against at most 0.23 for the identity embedding). Only age is also predicted well from identity (0.49), as apparent age partly follows the person. Figure 5(a) in our revised paper gives the qualitative counterpart. Ten occasion codes selected from the identity-blind prior (one distribution shared by all identities), each rendered for four virtual identities, keep their pose (including large yaw), expression, and lighting down each column, while each row keeps its identity.
Identities' own occasions. Under the conditional prior, each identity has its own occasion statistics. In Figure 5(b) of the revised paper, among released identities selected for extreme statistics, one is predicted happy in all 50 of its samples and another in none. Across 2,000 identities, the table below tests whether each statistic is consistent within an identity:
In the table, each entry is a split-half r, the correlation across identities between each identity's statistic in two disjoint halves of its 20 images.
In the table above, each identity keeps its own tendencies under the conditional prior (r 0.50 to 0.99, except brightness SD), at least as consistently as real people on seven of nine statistics. Under the identity-blind prior, r stays below 0.18 except for age, which follows the rendered face.
These tendencies are also predictable for unseen people. Conditioned on five photographs of a person, the prior predicts the statistics of 20 other photographs of that person, as shown in the table below:
In the table, the prior is conditioned on five photographs of the same person (Own person) or of a different person (Another person). We report Spearman ρ over 200 unseen people, and the ceiling is the square root of the Spearman-Brown-corrected split-half reliability of the actual statistic. Renders use each person's own embedding, so age correlates under every prior.
These results show that the assigned tendencies follow the individual, even for people unseen in training. The person's own photographs give the highest correlation for every attribute, most clearly for happy rate (0.57 against at most 0.12).
Each identity's own occasions also serve it better. With the generator and identities fixed, another person's occasions lower accuracy from 94.29% to 92.86% despite wider pose spread (Table 2). For unseen real people, occasions predicted from one photograph raise rank-1 accuracy from 72.66% to 77.21%, against 73.78% with another person's occasions (Table 8).