acceptodds
Under review as a conference paper at ICLR 2027

Query-Free Adversarial Face Generation via Reference-Conditioned Composition

Abstract

Face recognition is widely deployed for identity verification in security-sensitive applications. Verification accepts faces whose representations lie sufficiently close to a reference, defining an acceptance region in a space learned to discriminate identities. Attribute information remains encoded alongside identity even though this objective neither specifies it nor explicitly constrains its variation within the accepted region. Motivated by this coexistence, we study query-free adversarial face generation using a public gallery and accessible recognition models as surrogates for an unseen target model. The goal is to find a representation that yields a face with a chosen race or gender different from the reference, while the target model accepts it as the reference identity. A global approximation of the chosen attribute class produces a candidate representation by projecting the reference representation onto it, but fitting the approximation without the reference obscures the class representations nearest to that reference. We instead form a new representation through reference-conditioned composition, ranking the chosen-class gallery by similarity to the reference and composing the nearest representations. Across independently trained models, gallery representations ranked near the reference tend to remain near it, as do their compositions; aggregating surrogate rankings further improves transfer. Our attack reaches success rates up to 94.3% on AWS Rekognition and 95.0% on Tencent Cloud at default thresholds; at the stricter settings recommended by AWS for law-enforcement use and documented by Tencent at a false match rate of , it retains rates up to 55.0% and 80.0%, respectively.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Face Recognition, Adversarial Face Generation, Query-Free Attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5vwPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a query-free attack that combines gallery retrieval, embedding averaging, and inversion to generate a face with a chosen demographic label while preserving acceptance as a reference identity. A rank-consensus variant aims to improve transfer across recognizers.

Critical strengths
  1. A meaningful zero-query setting. Once the reference image and public resources are available, the attack requires no queries to the deployed verifier. Transfer to two commercial APIs shows a relevant offline attack surface.
  2. Broad empirical coverage. The paper covers several operating points, inverse models, galleries, surrogate subsets, and commercial services. Testing against another image of the same identity is especially useful because it reveals an important limitation.
Critical weaknesses
  1. The theory and implementation do not fully match the stated method. Coverage is defined as E∥B⊤E(x)∥2\mathbb{E}\|B^\top E(x)\|_2, but PCA optimizes the expected squared norm, so the claimed PCA optimality does not follow. The analysis also omits the decoder and target-model re-encoding used by the actual attack.
  2. The evaluation does not yet show target-agnostic, identity-level impersonation. The main test verifies against the same image used to build the attack. With another image of the same identity, Ours-C drops from 96.1–98.1% to 54.1–57.1% at FMR 10−410^{-4}, and from 88.2–92.2% to 37.4–38.9% at 10−510^{-5}. Some “held-out” targets also inform the analyses or share training data with surrogates.
Questions for authors
  1. Which quantity was plotted in Figures 1 and 6: E∥B⊤E(x)∥2\mathbb{E}\|B^\top E(x)\|_2, E∥B⊤E(x)∥22\mathbb{E}\|B^\top E(x)\|_2^2, or the square root of the latter? Please revise the PCA argument accordingly.
  2. Can the authors remove all exact, near-duplicate, and same-identity gallery items related to LFW, CFP-FP, and AgeDB, then rerun the attack using a separate enrollment image for verification?
1 comment

Thank you for examining the formulation and evaluation. We correct the PCA-optimality inference identified in your review. Our controlled evaluations support reference-conditioned composition and surrogate consensus for the stated joint objective. To explain how the evidence supports these claims, we clarify the intended roles of construction analysis under available models, target verification against the supplied reference, and supplementary cross-photo evaluation.

W1/Q1. Coverage and the corrected inference. Submitted Figures 1(a) and 6 plot E[∥B⊤E(x)∥2]\mathbb{E}[\|B^\top E(x)\|_2], the mean projection norm. This is mean cosine similarity to the nearest subsphere point.

PCA's squared-norm optimality does not establish optimality for this unsquared measure. We removed the resulting claims of maximal coverage and no room for improvement. Section 3.2 now confines the conclusion to the observed coverage–validity trade-off within the evaluated PCA family. The k/d\sqrt{k/d} reference is the RMS projection norm for uniform unit vectors and an upper bound on their mean norm.

Direct gallery selection and averaging do not rely on this optimality inference.

W1. Decoder and target re-encoding. Submitted validity V=Pr⁡[a(D(z))=c]V=\Pr[a(D(z))=c] includes decoding, and joint ASR evaluates chosen-class assignment and target acceptance of generated images. Section 4.2 separately studies cross-model gallery selection.

We connect these stages on 1,500 reference–class cases, reporting mean cosine to the corresponding reference:

MethodSource compositionGenerated image/sourceGenerated image/TransFace-L
NAFG0.5680.3990.330
Ours0.8030.5550.457
Ours-C0.7390.5500.496

Consensus increases target similarity despite lower source-composition similarity and comparable generated-image source similarity. Section 5.2 also shows that coefficient fitting increases target identity acceptance but decreases chosen-class assignment and joint success. Appendix B.2 tests decoded validity under projected inputs and alternative models.

W2. Additional cross-photo generalization. Our primary objective is to generate a chosen-attribute image accepted against the supplied reference (Section 2), which the main evaluation directly tests. Acceptance against another photograph of the same identity is an additional generalization property.

The submission included a supplementary cross-photo evaluation to characterize this additional capability. Following your suggestion, we expanded it from 773 cases to all 57,658 eligible reference–class cases with benchmark-labeled same-identity partners from LFW, CFP-FP, and AgeDB.

Separate-photo joint ASR (%), FMR 10−410^{-4} / 10−510^{-5}NAFGOursOurs-C
TransFace-L10.5 / 2.528.5 / 13.238.2 / 19.6
ViT-KPRPE9.0 / 1.634.1 / 15.739.6 / 19.1

On this population, Ours-C's construction-reference ASRs at 10−410^{-4} are 89.5% and 93.9%, respectively. The decrease quantifies the extent of additional cross-photo generalization. Ours-C remains highest under both verifiers and operating points, providing additional evidence of acceptance across photographs (Appendix D.1).

Q2. Sensitivity to reference–gallery overlap. We screened galleries using ArcFace scores at the threshold calibrated to FMR 10−610^{-6}. Potential same-identity matches are flagged for 501 of 115,390 reference–class cases (0.43%). This is a score-based sensitivity control rather than exhaustive identity-disjoint cleaning and regeneration.

For Ours and Ours-C, excluding cases whose selections contain any flagged gallery image changes ASR by less than 1.0 percentage point at FMRs 10−410^{-4} and 10−510^{-5} (Appendix A.2). Applying the screen to separate-photo verification leaves 54,694 common cases for all three methods. Across both verifiers and FMRs, the maximum absolute ASR change is 0.28 percentage points. Ours-C remains highest, followed by Ours and NAFG (Appendix D.1).

W2. Target roles and broader evaluation. TransFace-L and ViT-KPRPE are excluded from selection and construction but appear in evaluation analyses. Table 3 discloses their shared training datasets with TopoFR and AdaFace; their target role does not imply independent training or development.

The submitted evaluation also covers AWS and Tencent Cloud, whose architectures and training data are undisclosed. Ours-C exceeds NAFG in every chosen class at both service thresholds (Table 1).

We additionally evaluate Glint360K-trained TopoFR-R100 and TransFace-L across all 115,390 cases (Appendix D.7). Glint360K includes MS1MV2, so this broadens checkpoint coverage without establishing training-disjoint transfer. At FMRs 10−410^{-4} / 10−510^{-5}, Ours-C achieves 91.2% / 64.5% and 60.1% / 34.4%, respectively, exceeding both Ours and NAFG across all evaluated classes at both operating points.

Reviewer_8g14OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies query-free adversarial face generation using a public gallery and accessible recognition models as surrogates for an unseen target model. The goal is to find a representation that yields a face with a chosen race or gender different from the reference, while the target model accepts it as the reference identity. A global approximation of the chosen attribute class produces a candidate representation by projecting the reference representation onto it, but fitting the approximation without the reference obscures the class representations nearest to that reference. Instead authors form a new representation through reference-conditioned composition, ranking the chosen-class gallery by similarity to the reference and composing the nearest representations.

Critical strengths
  • The paper proposes a new method for query-free adversarial face generation focusing on face attributes.
  • The proposed method achieves remarkable performance in the experiments.
Critical weaknesses
  • The presentation in the paper can be improved. A blockdiagram can better help to understand the approach and threat model.
  • The paper needs to motivate why such attack is important in real world. There is only one baseline (NAFG) in this study, which shows limited work and interest on this topic.
  • How face generation method can affect the performance? NBNet is relatively old (2018) and there are other state-of-the-art methods. See for example [1].
  • Experiments are limited to gender and race. CelebFaces Attributes Dataset (CelebA) has a larger list of attributes that can be studied in the experiments.
  • More experiments is needed in the paper. See my questions.

[1] Vulnerability of state-of-the-art face recognition models to template inversion attack, IEEE Trans. on Information Forensics and Security, 2024

Questions for authors
  • Face generation step is not clearly explained in the paper. Based on examples, it seems authors have used Arc2Face, Vec2Face and NBNet (relatively old). Can you explain the face generation process and how it affect the performance?
  • Authors have studied attribute manifold, but what about manifold of face embeddings?
  • Can masterface be also considered as baseline in this study?
  • Can other face attributes be used for adversarial face generation? Authors can use CelebFaces Attributes Dataset (CelebA) to expand their study.
2 comments

We appreciate your assessment of our method and your suggestions on clarity and evaluation.

Presentation and practical motivation. We did not clearly explain how the objective, construction, and evaluation connect; we restate this connection below.

In selfie-based profile verification, a platform compares a video selfie with profile photographs and displays a verification badge after approval. An attacker could upload a photograph with deliberately altered demographic appearance while submitting an unmodified video selfie. If the photograph is accepted as matching that selfie, the badge could lend credibility to the misleading appearance. This motivates our joint objective: generating an image with a chosen attribute different from the reference's while matching that reference without target access or queries. Figures 1 and 3 illustrate the threat model and construction; Section 1 explains the practical motivation.

Baseline coverage. The submission included image-modifying attacks alongside NAFG; we should have made this scope clearer. Appendix D.2 evaluates Sibling-Attack at two perturbation budgets and Salar et al.'s diffusion attack, using single and ensemble surrogates on the same 1,500 reference–class pairs. Both start from chosen-class images and are assessed under the same joint attribute–identity criterion. Ours and Ours-C exceed these image-modifying baselines in every comparison column.

Q1. Generation and inverse-model dependence. Table 2 uses the modified NbNet of Paik et al. (IEEE TDSC, 2026), not the original 2018 network, as identified in submitted Section 5.2. Arc2Face is the default in Table 1; Vec2Face (2025) was also evaluated in submitted Appendix D.7, now D.8.

Generation passes the constructed unit representation to the fixed inverse model paired with the composing encoder. Arc2Face uses a fixed four-step sampler; modified NbNet is deterministic (Appendix A.1). No per-reference image-space or generator-latent optimization is required.

Following your TIFS 2024 suggestion, we added a DSCasConv-family inverse model trained on FFHQ for our ArcFace representation. This tests another inverse model with the encoder fixed. Vec2Face tests a different recognition–inverse pair, with all constructions recomputed in its aligned space. Absolute joint ASR varies across configurations; Ours-C leads in all Vec2Face cells and fifteen of sixteen DSCasConv cells. See Appendix D.8.

Beyond these evaluated configurations, adapter-based inversion offers another way to obtain the recognition–inverse pairing required by our construction. Shahreza et al. (2025) train adapters on unlabeled public images to map recognition embeddings into a fixed generator’s conditioning space, allowing the pairing without retraining the full inverse model. Appendix D.8 discusses this complementary route.

Q2. Broader face-embedding geometry. Our analysis studies an attribute-conditioned subset of the face-embedding space and can therefore also be viewed as an analysis of face-embedding geometry. We focus on this subset because our objective requires generating a face with a chosen attribute while preserving acceptance against a reference outside that class. Accordingly, we examine both the structure within the chosen class and its relationship to the reference in the shared embedding space.

The broader question is also relevant beyond attribute conditioning. Our ordering analysis in Section 4.2 begins with an attribute-unfiltered gallery before examining class-restricted selection and composition. We agree that further studying face-embedding geometry without attribute conditioning, and how its properties relate to verification security, is an interesting direction extending beyond the specific construction studied here.

Q3. MasterFace as a baseline. MasterFace is a related but distinct problem: it seeks broad acceptance across enrolled identities. Our criterion requires acceptance against a specified reference and assignment to a chosen attribute different from the reference's. Broad identity coverage does not establish this reference-specific joint success, so standard MasterFace is not a directly matched baseline. Appendix D.2 explains this distinction.

Q4. Other attributes. We agree that extending beyond race and gender helps assess the construction's applicability. Following your suggestion, Appendix D.9 adds CelebA examples using 185,684 annotation-partitioned images and 256-image neighborhoods. The examples cover eyeglasses, gray hair, bald, goatee, mustache, hat, blond hair, and heavy makeup.

Attribute changes are illustrated qualitatively, and reference acceptance is measured under five recognition models; all eight displayed outputs exceed their thresholds at FMR 10−510^{-5}. These examples suggest that reference-conditioned composition can extend beyond race and gender to other facial attributes, motivating broader evaluation across attribute families.

Thanks authors for their reply which addressed most of my concerns. I increased my rating.

Reviewer_qizUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies query-free construction of synthetic faces that receive a chosen classifier label while being accepted by a face verifier as a reference identity. It replaces a global approximation with reference-conditioned composition and combines rankings from accessible models to improve transfer to unseen verifiers.

Critical strengths

The task is interesting, and the paper reports substantial API-level improvements over NAFG with controlled generator and threshold settings. The selection/composition ablations, held-out-model evaluation, alternative inverse models, and disclosure of the independent-photo performance drop are useful.

Critical weaknesses

The methodological and analytical contribution is insufficiently established against the broader literature.

The relevant comparison is not limited to NAFG. Prior work already examines attribute-dependent face-embedding geometry (Attributes Shape the Embedding Space of Face Recognition Models), interpretable feature directions (Discovering Interpretable Feature Directions), cross-model geometric compatibility (Canonical Face Embeddings; Compatibility of Face Embeddings Across Deep Neural Networks), semantic explanations linked to verification behavior (EXPL-FR), and adversarial face generation or transfer (NAFG, Adv-CPG, and DiffAttack). These works address different settings, but collectively establish the context against which the paper's claimed analytical contribution must be judged.

The authors should identify precisely what their PCA information-loss diagnosis, reference-conditioned composition, and rank-consensus analysis explain or predict that these existing findings and analytical approaches do not. The current decomposition, neighborhood composition, and observed rank relationships do not convincingly establish a sufficiently substantial new methodological or explanatory contribution for ICLR. In particular, it is unclear whether the analysis provides new understanding of attribute-restricted, post-composition transfer and its failure conditions, rather than a task-specific illustration of already familiar geometry and transfer behavior.

Figure 2, Tables 14–16/22, and Appendix C.4 provide useful ablations, attack/generator comparisons, and prototype/genuine-utility analysis. However, these do not by themselves resolve the broader contribution question. The paper needs a clear comparison of the questions, assumptions, analytical tools, and resulting insights across the relevant literature, together with task-matched controls that isolate the distinctive value of its construction and aggregation. Direct numerical comparisons should respect differences in target access, source constraints, generators, and budgets. The concern is insufficient additional insight, not merely missing citations or simplicity of implementation.

Overall assessment. Weak rejection (2/4). The concept is interesting and the controlled API-level results are useful. However, the paper does not convincingly establish that its construction and analysis add sufficiently substantial new knowledge beyond the broader literature. At the present level of methodological and analytical insight, the contribution is too weak for ICLR acceptance.

Questions for authors
  1. Across the relevant prior studies of attribute geometry, interpretable directions, cross-model compatibility, semantic verification analysis, and adversarial generation, what precise new question does this paper answer? Please compare the analytical methods and resulting insights, rather than limiting the novelty discussion to NAFG.
  2. What new explanatory or predictive result follows from the PCA diagnostic and rank-consensus analysis? Which transfer behavior or failure condition is not already explained by the existing literature, and which task-matched control isolates that difference?
1 comment

We thank you for your thoughtful review and constructive feedback. We recognize that the submission did not sufficiently connect our analyses, method, and transfer findings. We have revised the paper and restate our contributions, their significance, and supporting evidence below.

Q1. What does the work contribute beyond prior studies?

We propose a method specifically designed for zero-query adversarial face generation and transfer. The objective combines a chosen attribute with acceptance as a specified reference.

NAFG includes zero-query transfer alongside target-score-assisted reconstruction. Our approach develops reference-conditioned selection, uniform composition, and surrogate rank consensus specifically for construction and transfer without target scores. The analyses explain what information should guide construction, how composition serves the joint objective, and which cross-model relationships support transfer.

The broader literature establishes attribute geometry and cross-model compatibility. Our contribution addresses their use for reference-specific construction and transfer:

  • Attribute geometry and semantic interpretation. Leroy et al. characterize attribute-dependent geometry; Plesh et al. identify directions for decoded attribute editing; EXPL-FR links semantics to verification. We ask how to use class representations to construct an attack satisfying both attribute and identity criteria. Our analyses show why class-level approximation or closer reconstruction alone is insufficient to assess joint success.
  • Cross-model compatibility. McNeely-White et al. and Rubab et al. fit mappings between paired embeddings. We study whether the reference-conditioned selections used by our construction transfer across models. We use gallery-ranking agreement for surrogate consensus without fitting a source-to-target map.
  • Adversarial generation. Adv-CPG and face-recognition DiffAttack use conditioning and optimization for adversarial generation. Our construction selects observed class-gallery representations and uniformly combines them before fixed-model inversion. We isolate selection and coefficient effects and examine the transfer of this composition.

Q2. What do the analyses establish, and why does it matter?

1. Global approximation and reference-conditioned construction require different assessments. We ask whether approximating the attribute class supplies the information needed to construct a match to a particular reference. Section 3 distinguishes class-level coverage from decoded attribute preservation and shows how projection excludes a component of reference–gallery similarity.

Appendix B.3 changes only the selection score, with original-representation averaging and the inverse model fixed. Full similarity increases generated-image ArcFace similarity by 0.111–0.174 across five classes; chosen-class assignment drops most in White and Male. The excluded information therefore affects construction, while its value must be assessed against both objectives. This motivates using the original reference–gallery relationships directly instead of treating class approximation quality as sufficient guidance.

2. Reference-conditioned composition addresses the joint objective. We select class images by reference similarity and decode the normalized mean of their representations. This uses the reference to guide identity similarity while drawing the composition from observed class representations.

The coefficient control tests whether closer reconstruction serves this objective. With selected images fixed, fitting increases identity acceptance but reduces chosen-class assignment from 95.1% to 50.3%. Joint ASR at FMR 10−410^{-4} falls from 79.1% to 48.5% on TransFace-L and from 89.9% to 49.9% on ViT-KPRPE (Section 5.2). This supports uniform composition over reconstruction fitting for the joint objective.

3. Reference–gallery relationships provide an empirical account of selection transfer. Because our method selects by reference similarity, we investigate whether source-selected images remain similar to the reference under other models and whether their normalized means within each model exhibit corresponding similarity. We assess these relationships and generated-image success separately.

Ordering agreement tracks joint ASR across seven targets using the same 1,500 ArcFace-only outputs: Spearman correlations are 0.964 and 0.893 at FMRs 10−410^{-4} and 10−510^{-5} (Appendix C.2).

This evidence supports cross-model consistency of reference-conditioned relationships as a basis for transfer. It motivates consensus across surrogate rankings to reduce dependence on one model’s ordering. With averaging and the inverse model fixed, consensus improves generated-image joint ASR on both targets (Figure 5(c), Appendix D.5). This connects a method-specific account of transfer to an experimentally validated improvement requiring no target feedback.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful and constructive feedback. Your comments helped us clarify our contributions, correct analytical claims, and strengthen the experimental evidence.

The revision preserves the paper’s central contributions. We recognize, however, that the original presentation left important connections between our analyses, method, and transfer findings implicit, making it harder to assess what the work adds beyond prior approaches. We take responsibility for this lack of clarity. To address it, we restate below the problem we seek to solve and our contributions and have revised the manuscript to make both clearer.

Our study addresses a joint objective: generating faces with a chosen attribute class different from the reference image’s while being accepted as matches to that reference by target recognition models, without accessing or querying those models. We investigate how to use an attribute-labeled gallery to achieve this objective and improve cross-model transfer. Our contributions are as follows:

  • Analysis of attribute-restricted construction. We show that, for PCA-based class approximations, geometric coverage does not determine the fraction of decoded images assigned to the chosen class, and that projection reorders class-gallery images relative to a reference. Both findings motivate selecting on the original similarities.

  • Reference-conditioned composition. We propose selecting gallery images from the chosen attribute class by their similarity to the reference and decoding the normalized mean of their representations. Controlled comparisons examine selection and coefficient fitting. With the selected images fixed, reconstruction fitting increases identity acceptance but reduces chosen-class assignment and joint attack success, supporting uniform averaging for the joint objective.

  • Cross-model transfer analysis and improvement. We analyze agreement in reference-conditioned gallery rankings across recognition models and whether selected images and their compositions remain similar to the reference. These findings motivate surrogate rank consensus, which improves generated-image joint attack success on the evaluated targets with source-space averaging and the inverse model fixed.

  • Empirical validation. We demonstrate higher joint attack success than NAFG across open-source and commercial verifiers. Evaluations vary galleries, surrogate pools, and inverse models to test robustness, while verification against another image of the same identity assesses dependence on the construction reference.

The revision adds clearer model roles and overview figures, corrects the PCA-optimality inference, and includes additional target and inverse models and qualitative CelebA examples. It also expands verification against another image of the same identity and applies identity-overlap screening to that evaluation. Our individual responses explain how each comment informed these changes and provide the corresponding evidence and manuscript locations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.