- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that existing evaluations of text-guided face editing insufficiently examine identity preservation and proposes FaceEdit-Bench to assess identity preservation and edit effectiveness separately. Identity preservation is measured by averaging similarity scores from nine face recognition models, while edit effectiveness is assessed by an annotation agent that decomposes editing instructions into individual requirements and evaluates their fulfillment. Using these scores, the authors analyze 15 editing models and argue that neither dimension alone adequately characterizes editing reliability. They further introduce FaceEditEval, a vision-language model fine-tuned to predict the benchmark’s automated reference scores. The authors report higher correlations with these reference scores than the evaluated baselines, as well as higher agreement with human preferences in a study of 200 selected pairs of edited images.
Critical strengths
The benchmark compares 15 editing models using a shared set of source images and editing instructions across six categories. This controlled setup supports systematic comparisons of model behavior across different editing tasks, with identity preservation and instruction fulfillment assessed separately. I consider this experimental organization a useful contribution, although its practical value depends on the validity of the proposed scoring procedures.
Critical weaknesses
Insufficient empirical validation of the reference scores. The paper does not directly validate the reference scores defined in Section 3.3 against human judgments or assess the effects of recognizer selection, instruction decomposition, and equal-weight aggregation. Since these scores determine benchmark rankings and are used to train and evaluate FaceEditEval, this limits confidence in the benchmark analysis and the learned evaluator.
Limited evidence that FaceEditEval is a more accurate evaluator. Table 1 compares FaceEditEval, which was trained to predict the proposed scores, with baselines that were not trained on those scores. Higher agreement with these targets alone does not establish better judgment of identity preservation or instruction fulfillment. Table 2 adds evidence from human judgments, but basic details such as the number of annotators and how their judgments were combined are missing. This makes the strength of the human evidence difficult to assess.
Questions for authors
-
Rationale for the identity preservation score. What justifies using a simple, equally weighted average of nine face-recognition similarity scores as a reliable measure of identity preservation? Does averaging these scores improve agreement with human judgments compared with individual recognizers?
-
Validity of expression decomposition. What justifies evaluating an “embarrassed expression” through a single combination of visual cues, as illustrated in Figure 3? How does the method account for other valid ways of expressing embarrassment?
-
Edit effectiveness score stability and equal weighting. Are the resulting scores stable when the same instruction is rephrased or decomposed differently? What justifies assigning equal weight to all requirements in Eq. (2)?
-
Human-study protocol. For Table 2, please report the number of participants, judgments per pair, how individual judgments were combined, inter-annotator agreement, and confidence intervals. Please also clarify how the 200 gMAD pairs were selected and distributed across comparisons between methods.
-
Interpretation of Figure 5 and ambiguous identity comparisons. Figure 5 is not discussed in the main text. Could the authors clarify what these examples are intended to demonstrate and how they support the claim that FaceEditEval reliably evaluates identity preservation? In particular, I find it difficult to judge which edit better preserves identity in the upper-right and lower-left examples. How strongly did the annotators agree on the reported preferences for these examples?