acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Face Edit Reliable? Benchmarking Reliability in Text-Guided Face Editing

Abstract

Text-guided face editing requires fulfilling editing instructions while preserving the subject’s identity. However, existing evaluation practices primarily emphasize edit effectiveness, leaving identity preservation insufficiently examined and potentially favoring models that follow instructions at the expense of identity. To address this gap, we introduce FaceEdit-Bench, a benchmark that evaluates identity preser- vation and edit effectiveness as complementary dimensions of editing reliability. FaceEdit-Bench comprises approximately 110K annotated face edits generated by 15 editing models across 20 instruction types. We assess identity preservation by aggregating scores from multiple face recognition systems. To evaluate edit effectiveness, we develop an instruction-conditioned agent that decomposes each instruction into verifiable requirements and assesses their fulfillment using tool-derived visual evidence. Our analysis reveals divergent model rankings across the two dimensions: stronger instruction fulfillment does not necessarily imply better identity preservation, highlighting the limitations of instruction fulfillment alone as a measure of editing reliability. We further introduce FaceEditEval, a vision-language model that consolidates supervision from the recognition systems and evaluation agent into a single evaluator, enabling efficient prediction of identity preservation and edit effectiveness as separate scores. FaceEditEval achieves higher correlations with the automated reference scores than existing quality assessment methods. By accounting for both the requested changes and the preservation of identity, FaceEdit-Bench and FaceEditEval enable more comprehensive comparisons of editing models and support research on reliable text-guided face editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-Guided Face Editing, Image Editing Evaluation, Benchmark, Multimodal Evaluation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zV8TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that existing evaluations of text-guided face editing insufficiently examine identity preservation and proposes FaceEdit-Bench to assess identity preservation and edit effectiveness separately. Identity preservation is measured by averaging similarity scores from nine face recognition models, while edit effectiveness is assessed by an annotation agent that decomposes editing instructions into individual requirements and evaluates their fulfillment. Using these scores, the authors analyze 15 editing models and argue that neither dimension alone adequately characterizes editing reliability. They further introduce FaceEditEval, a vision-language model fine-tuned to predict the benchmark’s automated reference scores. The authors report higher correlations with these reference scores than the evaluated baselines, as well as higher agreement with human preferences in a study of 200 selected pairs of edited images.

Critical strengths

The benchmark compares 15 editing models using a shared set of source images and editing instructions across six categories. This controlled setup supports systematic comparisons of model behavior across different editing tasks, with identity preservation and instruction fulfillment assessed separately. I consider this experimental organization a useful contribution, although its practical value depends on the validity of the proposed scoring procedures.

Critical weaknesses

Insufficient empirical validation of the reference scores. The paper does not directly validate the reference scores defined in Section 3.3 against human judgments or assess the effects of recognizer selection, instruction decomposition, and equal-weight aggregation. Since these scores determine benchmark rankings and are used to train and evaluate FaceEditEval, this limits confidence in the benchmark analysis and the learned evaluator.

Limited evidence that FaceEditEval is a more accurate evaluator. Table 1 compares FaceEditEval, which was trained to predict the proposed scores, with baselines that were not trained on those scores. Higher agreement with these targets alone does not establish better judgment of identity preservation or instruction fulfillment. Table 2 adds evidence from human judgments, but basic details such as the number of annotators and how their judgments were combined are missing. This makes the strength of the human evidence difficult to assess.

Questions for authors
  1. Rationale for the identity preservation score. What justifies using a simple, equally weighted average of nine face-recognition similarity scores as a reliable measure of identity preservation? Does averaging these scores improve agreement with human judgments compared with individual recognizers?

  2. Validity of expression decomposition. What justifies evaluating an “embarrassed expression” through a single combination of visual cues, as illustrated in Figure 3? How does the method account for other valid ways of expressing embarrassment?

  3. Edit effectiveness score stability and equal weighting. Are the resulting scores stable when the same instruction is rephrased or decomposed differently? What justifies assigning equal weight to all requirements in Eq. (2)?

  4. Human-study protocol. For Table 2, please report the number of participants, judgments per pair, how individual judgments were combined, inter-annotator agreement, and confidence intervals. Please also clarify how the 200 gMAD pairs were selected and distributed across comparisons between methods.

  5. Interpretation of Figure 5 and ambiguous identity comparisons. Figure 5 is not discussed in the main text. Could the authors clarify what these examples are intended to demonstrate and how they support the claim that FaceEditEval reliably evaluates identity preservation? In particular, I find it difficult to judge which edit better preserves identity in the upper-right and lower-left examples. How strongly did the annotators agree on the reported preferences for these examples?

Reviewer_DGB8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a large-scale benchmark for evaluating text-guided face editing along two dimensions, namely FaceEdit-Bench. The benchmark evaluates the task in two aspects, i.e., identity preservation and edit effectiveness. About 110K edited images are generated by 15 editing models across 20 instruction types. To access the model ability, identity preservation is measured by an ensemble of face recognition models. Meanwhile, edit effectiveness is annotated by agents that decompose editing instructions into visual requirements and evaluate their fulfillment using visual tools. Finally, equipped with the labels, this paper trains FaceEditEval to predict both dimensions, which can be used as a tool for future research.

Critical strengths
  • S1. Separately evaluating identity preservation and edit effectiveness is very important since some trivial solutions (e.g., do nothing) can achieve good results.
  • S2. The proposed benchmark covers a wide range of editing instructions and recent editing models such as FLUX.1 Kontext and Qwen-Image-Edit, as well as both closed-source and open-source models.
Critical weaknesses
  • W1. According to the first two paragraphs of the abstract, this paper aims to study the problem that current evaluations on face editing leaves identity preservation insufficiently examined. However, this is not a new problem. The observation that stronger editing performance does not necessarily imply better identity preservation has already been well-established. For example, [A] explicitly discusses how editing metrics may favor stronger edits despite identity degradation. Similar tradeoffs have also been examined before in makeup transfer [B]. As such, the position of this paper is questionable.

[A] Hanhui Wang, Yihua Zhang, Ruizheng Bai, Yue Zhao, Sijia Liu, and Zhengzhong Tu. Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing. CVPR, 2025.

[B] Nefeli Andreou, Angel Martínez-González, Sabine Sternig, Matthieu Guillaumin, Epameinondas Antonakos, and Michael Opitz. CVPRW, 2026.

  • W2. Two important related works are missing. [C] separately evaluates identity preservation and attribute editing effectiveness. More importantly, [D] has already evaluated identity preservation, instruction fidelity, and overediting in a closely related setting. However, these two works are not cited and compared. It is suggested to clarify what new scientific insights FaceEdit-Bench provides beyond these established findings and evaluation practices, rather than presenting the separation of identity preservation and edit effectiveness as a novel observation.

[C] Lilian Bour, Sébastien Bougleux, Christophe Charrier, and Olivier Lézoray. 2026. An evaluation framework for generative face-editing methods: Quality, identity and disentanglement. Image Commun., 2026.

[D] Rahul Nair, Saurav Pandit, and Hannah Kerner. A Large-scale Evaluation of Text-guided Models for Facial Editing. ACM MM, 2026.

  • W3. The annotation process in Figure 3 raises concerns about whether decomposed visual requirements accurately represent the original editing intent. For example, the instruction "Add an embarrassed expression" is decomposed into "wide eyes," "forced smile," and "blush on cheeks." However, these attributes are not necessarily required to express embarrassment, and satisfying them does not guarantee that the resulting face appears embarrassed. In this case, the example receives a perfect edit effectiveness score of 1.0. This illustrates a potential mismatch between satisfying agent-generated requirements and fulfilling the original semantic instruction. Since these annotations serve as supervision for FaceEditEval, systematic errors in instruction decomposition could directly affect the reliability of both the benchmark and the learned evaluator.

  • W4. This paper does not compare against direct source-to-edit face embedding similarity for identity preservation, such as ArcFace cosine similarity or the recognition ensemble used to construct the reference labels. The proposed FaceEditEval is trained on about 80K annotated edits, whereas the VLM baselines are evaluated without comparable task-specific fine-tuning. Therefore, the reported improvements may largely reflect access to supervised training data instead of a better evaluation approach.

  • W5. The details of the user study are missing, e.g., the number of the users, background of users, study platform.

Questions for authors
  • Q1. It is suggested to differentiate with the existing benchmarks, e.g., [C, D] in weakness, and position this paper regarding the evaluation dimensions and the insights obtained
  • Q2. It is questionable if this paper validates that satisfying all decomposed visual requirements implies successful fulfillment of the original editing instruction (see W3).
  • Q3. It is suggested to compare the proposed VLM with direct source-aware face recognition similarity and VLM baselines fine-tuned using the same supervision.
  • Q4. The details of the user study are desirable.
Reviewer_u633OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Face editing should be scored on two axes: did the identity survive, and did the edit happen. FaceEdit-Bench annotates ~110K edits over 10K source images, 15 editors (10 open, 5 commercial) and 20 instruction types. Identity is the mean cosine of nine recognizers (three InsightFace, six DeepFace); effectiveness is the fraction of atomic requirements an agent marks satisfied, using BiSeNet parsing, MediaPipe landmark ratios, CLIP region-text similarity and color/crop statistics as evidence and Qwen3-VL-32B as judge. FaceEditEval distills both into four LoRA evaluators on Qwen3-VL-8B: 0.763 SRCC / 0.772 PLCC against 0.391 / 0.412 for the best VLM baseline, plus a 200-pair gMAD study (84.0% identity, 76.0% effectiveness). Rankings do not carry across axes — GPT-Image-2 keeps identity best, Seedream 5.0 Lite edits best; style edits trade identity for effect, attribute edits the reverse.

Critical strengths
  1. Two axes with a clean OOD protocol. Instruction-only metrics let a model pass by substituting a different face, so identity as its own axis is the right fix; ~110K annotations, 15 editors, and a split making the commercial subset an unseen-editor test by construction. This is the part of the benchmark I would trust and reuse.

  2. Evidence-based labels, actionable findings. Decomposing the instruction and treating tool evidence as advisory is far more careful than asking a VLM for a 1–5, and what comes out is usable: style edits trade identity for effect, attribute edits the reverse.

Critical weaknesses
  1. Labels come from nine recognizers plus a Qwen3-VL-32B agent; FaceEditEval trains on those labels; it is then scored by correlation with those same labels. A model trained on a target correlates with that target, so what is measured is distillation fidelity, not evaluation quality. Nothing establishes that the labels are right — the human study validates FaceEditEval, not sids_{id} or sedits_{edit}.

  2. The baseline that matters most is missing. The identity reference is the mean of nine recognizer cosines, so a single ArcFace cosine would likely land near 0.9 SRCC. That number is absent, so 0.861 cannot be read as an advance — an 8B VLM with LoRA may be an expensive approximation of a millisecond cosine; CLIP-I, DINO, LPIPS and DreamSim are also unused. Meanwhile IQA and FIQA see only the edited image, with no source and no instruction, while identity is defined over the pair — they cannot succeed by construction. Against a VLM that sees the same inputs, the effectiveness gap narrows to 0.726 vs 0.763 PLCC on the commercial subset.

Questions for authors
  1. (Clarification) Validate the reference labels directly: have humans judge source-edit-instruction triplets on identity and instruction satisfaction, then correlate s∗ids*_{id} and s∗edits*_{edit} against that. Everything rests on the labels being right.
  2. (Clarification) Report SRCC/PLCC for a single-model FR cosine (ArcFace or buffalo_l) on identity, and add CLIP-I / DINO / LPIPS / DreamSim. If one cosine is near 0.86, the contribution on that axis needs restating.
  3. (Clarification) What is the normalization for s~j\tilde{s}_j? Reconcile [0,1][0,1] in the text with [0,5][0,5] in Appendix C, and say how much cosine range survives it. Was per-recognizer calibration before averaging tried?
  4. (Clarification) Is the split identity-disjoint? How many unique instructions — 20 templates or a paraphrase set? Please also state the license, release plan and what filtering was applied to the LAION-Face pool.
  5. (Clarification) Does a judge from a different model family change the conclusions? The judge is Qwen3-VL-32B and the distilled backbone Qwen3-VL-8B, so part of the success may be family self-consistency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.