acceptodds
Under review as a conference paper at ICLR 2027

PixelSmile: Toward Fine-Grained Facial Expression Editing

Abstract

Fine-grained facial expression editing is limited by intrinsic semantic overlap. To address this, we construct the (FFE) dataset with continuous affective annotations and establish to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose , a diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and identity preservation, enabling fine-grained continuous expression editing and smooth expression blending.

Primary area:
generative models
Keywords:
Facial Expression Editing, Diffusion Models, Continuous Intensity Control, Identity Preservation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wjnZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PixelSmile, a method for fine-grained facial expression editing that preserves identity. The authors build FFE, a dataset of approximately 60,000 real and anime images covering 12 expressions, with continuous intensity annotations generated by multimodal models. PixelSmile fine-tunes Qwen-Image-Edit with symmetric expression-separation and identity-preservation constraints, and controlling expression intensity through text-embedding interpolation.

Critical strengths

The authors show that similar expressions are often confused and train the model to separate them. Ablation studies confirm that this helps reduce confusion.

Existing datasets either lack multiple expressions of the same identity or use only three intensity levels. FFE provides continuous intensity scores for both real and anime faces, with multiple expressions of the same identity.

Critical weaknesses

Gemini 3 Pro is used for both training labels and evaluation, so the results may not reflect actual improvements in expression editing.

FFE is generated using Nano Banana Pro, which is also a main baseline. PixelSmile’s gains over it may come from better prompts and data filtering rather than the proposed method.

Human validation checks only the reliability of intensity scores. The user study compares just three methods on smoothness and identity preservation, without independently validating expression confusion or editing accuracy.

The continuous-control baselines (SliderEdit and K-Slider) use FLUX.1 Kontext, while PixelSmile uses Qwen-Image-Edit. This makes it hard to tell whether the gains come from the method or the base model.

The paper does not specify whether baselines use equally detailed prompts. The authors also note that Nano Banana Pro produces vague edits when given only expression names.

Questions for authors

see Critical Weaknesses

The anime domain is evaluated only through visual examples, with no quantitative results. Can the authors provide numerical evaluations? Since the anime model does not use an identity constraint, how is identity preservation measured?

Will the dataset and code be released? The anime data comes from 207 commercial works. How will copyright issues be handled upon release?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wjnZ

Thank you for recognizing the design that separates similar expressions and its ablation validation, and the value of FFE's continuous intensity annotations. We address your points below.

📝 W1 / W3: Evaluation independence and human validation.

💡A: Regarding evaluation independence, we add human category annotations and evaluation with GPT-6 Astra and Claude Opus 5.5, directly validating editing accuracy and structural confusion. Human annotations and the three VLMs give a consistent method ordering on Acc-6 and mSCR. The protocol, results, and analysis are in the general response and Appendix H.

📝 W2: Training data versus training method.

💡A: We agree that high-quality training data contributes substantially, but ablations show that the data alone is not enough to resolve expression confusion. Keeping the same training data and removing the contrastive loss gives Acc-6 of 0.6471 and mSCR of 0.2725, versus 0.8627 and 0.0550 for the full model. Thus, even with carefully prompted and filtered data, the model still cannot reliably separate similar expressions without our training constraints. Data quality and the training method jointly produce the improvement, and the fixed-data ablation directly supports the method's contribution.

The dataset's value is also supported by an ablation: with the same architecture and optimization settings, training on MEAD gives Acc-6 of 0.7647 and mSCR of 0.2125, while training on FFE gives 0.8627 and 0.0550. This supports FFE's training value, and the fixed-FFE component ablation shows our training method is essential.

📝 W4: Base model differences.

💡A: We acknowledge Qwen and FLUX differ in base capability, and Qwen's stronger editing ability gives a good start. Qwen with the control module already shows some linear control, and our training improves it. We therefore compare the same Qwen base before and after training to separate the base model's existing ability from our training's gains. Comparisons across base models reflect the full system, while the same-base-model and component ablations support our training.

📝 W5: Are the evaluation prompts equally detailed?

💡A: We clarify the difference between training-data construction and evaluation. During training-data construction we use detailed local facial descriptions to improve the semantic accuracy of generated expressions. During evaluation, all models use the same simple template that only specifies the target expression word, without extra facial or action descriptions. PixelSmile therefore receives no more detailed prompts than the baselines at test time, and NBP is evaluated under the same unified condition.

📝 Q1: Anime identity preservation and quantitative evaluation.

💡A: Anime and real faces are identified differently. Real identity relies on fine facial features, while anime characters are highly simplified and can be recognized from a few semantic elements such as silhouette, signature hat, or hairstyle. We therefore believe the CLIP feature supervision used in training is sufficient to constrain these identity-bearing semantic elements and support anime identity preservation, without introducing an additional real-face identity loss. DanbooruCLIP only adapts the features to the anime domain.

For anime quantitative evaluation, we observe that existing methods produce stable edits only on a few expression categories, with large variation on complex expressions. Our method improves on most anime expressions after training, but instability remains. Meanwhile, VLM recognition of complex anime expressions is not reliable enough to separate editing failures from evaluation errors. We therefore cannot yet give a reliable quantitative method comparison for anime, and the anime results mainly demonstrate the method's applicability and current capability.

📝 Q2 / Ethics: Release plan and data sources.

💡A: Data sources and usage are already described in the paper's Ethics statement, Data Sources and Usage paragraph: all data are collected from public sources and used under their respective licenses, with the real subset from two MIT-licensed datasets and the anime subset of fictional characters from public productions. We add our release plan here: we will release the code, the data construction pipeline, and the annotations; images that may be redistributed will follow their original licenses and retain source information; images whose redistribution rights are unclear will be provided as a source index with annotations, without distributing the image files; and for the real people and commercial anime characters shown in the paper, we will verify their use and replace or remove examples without confirmed permission.

AuthorsOpenReview ↗

Kindly request for response

Dear Reviewer wjnZ,

Thank you again for your detailed feedback. We would like to draw your attention to the following points in our rebuttal:

  • Evaluation reliability: Appendix H adds independent human annotations and two additional VLM evaluators, with a consistent method ordering on Acc-6 and mSCR. Fig. 17 examines the larger variation in NBP accuracy.
  • Training versus data and base-model effects: we highlight the existing fixed-data component ablations and comparisons using the same Qwen base, which isolate the gains from our training. We also clarify that all methods receive the same simple expression-prompt template during evaluation.
  • Anime evaluation and data release: our response explains the identity supervision, the limitations of anime quantitative evaluation, and the release plan with source-specific redistribution conditions.

We would appreciate your thoughts on whether these responses address your concerns, and would be happy to clarify any remaining questions.

Best regards,
The authors

AuthorsOpenReview ↗

Final follow-up before the discussion deadline

Dear Reviewer wjnZ,

As the discussion period closes in approximately three hours, we would appreciate hearing whether our responses on evaluation reliability, the contribution of training, and the other points you raised have addressed your concerns. Please let us know if anything remains unclear so that we can respond before the deadline.

Thank you again for your careful review and time.

Reviewer_zEhMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PixelSmile, a diffusion-based framework for fine-grained facial expression editing with continuous intensity control and identity preservation. The authors construct the Flex Facial Expression (FFE) dataset containing 60K synthetic images across real-world and anime domains, annotated with continuous scores for 12 expressions. They also introduce FFE-Bench to evaluate expression confusion, editing accuracy, controllability, and identity preservation. The proposed method combines textual latent interpolation with fully symmetric contrastive training and identity supervision. Experiments against several image-editing and continuous-control baselines show improvements in expression disentanglement and control linearity. Ablation studies and a user study further support the effectiveness of the proposed components.

Critical strengths
  1. The introduction of FFE and FFE-Bench is a valuable contribution, providing continuous annotations for 12 facial expressions across real-world and anime domains. The benchmark evaluates multiple complementary aspects of expression editing, including semantic confusion, intensity control, and identity preservation.
  2. PixelSmile achieves improvements over competitive baselines, particularly in structural confusion and continuous controllability. The ablation studies demonstrate the contributions of symmetric contrastive training and identity supervision, while the qualitative results and user study provide additional evidence of editing quality.
Critical weaknesses
  1. The FFE dataset is generated using Nano Banana Pro and annotated using Gemini 3 Pro, which is also used for benchmark evaluation. This raises concerns about potential model-specific biases and the independence of the evaluation. Although human validation is provided for a subset, broader evaluation using independent human annotations or alternative recognition models would strengthen the conclusions.
  2. Textual embedding interpolation, contrastive learning, and identity-preserving losses are established techniques. The proposed framework mainly combines these components through symmetric training. Furthermore, the evaluation primarily uses the newly introduced synthetic dataset, leaving uncertainty about generalization to challenging real-world facial expressions and unseen data distributions.
Questions for authors
  1. Since Gemini 3 Pro is used for both FFE annotation and benchmark evaluation, could the authors provide results using an independent facial expression recognition model or a larger human evaluation to verify that the improvements are not evaluator-specific?
  2. How well does PixelSmile generalize to real, non-synthetic facial expression datasets such as AffectNet or other in-the-wild facial datasets, particularly under challenging lighting, pose, and occlusion conditions?
  3. Could the authors further isolate the contributions of continuous intensity supervision and symmetric contrastive training? In particular, how does training with continuous annotations compare against discrete expression labels under the same architecture and dataset?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zEhM

Thank you for recognizing the value of FFE/FFE-Bench as continuous annotation and evaluation resources, and for your appreciation of the ablation and user-study evidence. We address your points below.

📝 W1 / Q1: Independent evaluation and more extensive human validation.

💡A: Regarding independent evaluation, we add human category annotations and introduce GPT-6 Astra and Claude Opus 5.5 to directly validate accuracy and structural confusion. Human annotations and the three VLMs consistently order PixelSmile > NBP > Qwen on Acc-6 and mSCR. The full protocol, results, and analysis are in the general response and Appendix H.

📝 W2: Method contribution.

💡A: Our method contribution centers on bidirectional symmetric training for confusable expressions: from the same input we generate two similar expressions, and the two branches exchange the roles of positive sample and hard negative sample for the target images, thereby constraining expression separation in both directions simultaneously. Continuous intensity supervision further aligns the training condition with the text interpolation strength used at inference. We adopt existing interpolation, contrastive, and identity-feature tools and design how their supervision is organized for fine-grained expression control.

The asymmetric-training ablation directly tests the bidirectional design: the full model achieves mSCR of 0.0550 and Acc-6 of 0.8627, versus 0.1350 and 0.7843 after removing the symmetric framework. FFE and FFE-Bench also provide continuous expression supervision and complementary evaluations of structural confusion and control linearity.

📝 W2 / Q2: Generalization to real-world and unseen distributions.

💡A: For the real-face part of FFE, real portraits are used as editing inputs, but the target expression images are constructed by a generative model, so this still differs from validating on an independent real distribution such as AffectNet. AffectNet requires a signed academic license, and we have not obtained access within this discussion period. To provide evidence on real, non-synthetic data, we conduct a qualitative generalization study on three real datasets outside the training distribution: WFLW (varied pose), CelebA (varied lighting), and COFW (occlusion), editing four target expressions (happy, angry, surprised, sad) under each condition. As shown in Appendix G, Fig. 16, PixelSmile produces clear, target-consistent edits while preserving content and identity overall. We observe that a few edits introduce subtle lighting changes or slightly adjust head pose; one plausible explanation is that strong expression changes in training examples frequently co-occur with head-pose changes, so the model may partly convey emotion through pose variation.

📝 Q3: Attributing continuous supervision and symmetric training.

💡A: In early experiments before contrastive learning was introduced, we qualitatively compared discrete and continuous supervision on the same FFE data: when the strength coefficient was fixed to 1 and only discrete expression prompts were used, the model did not learn the expected linear control, whereas continuous intensity supervision improved it. This is an important reason for our design.

The paper also provides a dataset-level ablation: with the same architecture and optimization settings, discrete categories with three intensity levels from MEAD give CLS-6 / CLS-12 of 0.7047 / 0.6187, while FFE gives 0.8078 / 0.7305, supporting FFE's value for continuous control. The independent effect of symmetric training is supported by the component ablation: removing the symmetric framework raises mSCR from 0.0550 to 0.1350. The two forms of supervision respectively support fine-grained intensity control and separation of similar expressions, and they are complementary in the full method.

AuthorsOpenReview ↗

Kindly request for response

Dear Reviewer zEhM,

Thank you again for your constructive suggestions. They helped us strengthen the validation and clarify the roles of our training components:

  • Independent validation: Appendix H adds human annotations and two additional VLM evaluators. All evaluators consistently rank PixelSmile > NBP > Qwen on Acc-6 and mSCR.
  • Real-world generalization: Appendix G (Fig. 16) adds qualitative results on WFLW, CelebA, and COFW under varied pose, lighting, and occlusion, together with a discussion of the remaining limitations.
  • Continuous supervision and symmetric training: our response describes the early qualitative comparison of discrete and continuous supervision on the same FFE data, alongside the existing MEAD and symmetric-training ablations.

We hope these additions are helpful in addressing your questions. We would appreciate any further feedback and would be happy to provide clarification.

Best regards,
The authors

AuthorsOpenReview ↗

Final follow-up before the discussion deadline

Dear Reviewer zEhM,

With approximately three hours left in the discussion period, we would be grateful for your thoughts on the additional independent validation and real-world generalization results provided in response to your suggestions. We would welcome the opportunity to clarify any remaining questions before the deadline.

Thank you again for your constructive feedback and time.

Reviewer_An9sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PixelSmile, a framework for fine-grained facial expression editing with continuous intensity control. The authors introduce the FFE dataset with 60K images and continuous expression annotations, along with FFE-Bench for evaluation. The method combines textual embedding interpolation, symmetric contrastive learning, and identity supervision to reduce expression confusion while preserving identity. Experiments show improved editing accuracy and continuous control.

Critical strengths
  1. Clear motivation. The paper addresses the challenge of fine-grained expression editing, especially the confusion between semantically similar expressions.

  2. Strong empirical results. The method achieves promising improvements in expression editing and continuous control, supported by ablation studies.

  3. Dataset and benchmark contributions. The proposed FFE dataset and FFE-Bench provide useful resources for training and evaluating fine-grained expression editing.

Critical weaknesses
  1. Potential evaluation bias.

Gemini 3 Pro is used for both dataset annotation and benchmark evaluation, raising concerns about evaluation bias. Independent evaluation using other VLMs or human judgments would strengthen the results.

  1. Limited methodological novelty.

The method primarily combines existing techniques, including embedding interpolation, contrastive learning, identity loss, and LoRA fine-tuning. While effective, the methodological contribution is relatively incremental.

Questions for authors

Please see Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer An9s

Thank you for recognizing the motivation, the empirical results, and the FFE and FFE-Bench dataset and benchmark contributions. We address your points below.

📝 W1: Is the evaluation biased toward Gemini?

💡A: Regarding evaluation independence, we add human category annotations and evaluation with GPT-6 Astra and Claude Opus 5.5, directly validating editing accuracy and structural confusion. Human annotations and the three VLMs give a consistent method ordering on Acc-6 and mSCR. The protocol, results, and analysis are in the general response and Appendix H.

📝 W2: Method novelty.

💡A: We do build on existing techniques, but our contribution addresses continuous expression control and confusion among highly similar expressions, which these techniques have not adequately solved, by proposing a new control mechanism, a training paradigm, and a data construction method. Describing them merely as a combination of existing components does not fully reflect the novelty.

  1. Continuous expression control. We are inspired by CFG and text-embedding interpolation and combine the two ideas into a new control mechanism that is applied consistently in both training and inference. The model learns fine-grained expression semantics under continuous intensity supervision, producing smoother and more linear control. The key step is that we turn interpolation in the text space into an intensity-control ability learned by the model, rather than only modifying the text representation at inference. This is why "embedding interpolation" alone does not capture the contribution.

  2. Symmetric contrastive loss and fully symmetric joint training. We find that a CLIP-style contrastive constraint or an ordinary triplet contrastive loss is still insufficient for separating highly similar expressions. We therefore design a symmetric contrastive loss and a fully symmetric joint training scheme: the two expression branches generated from the same input exchange their roles as positive and hard-negative samples, reducing the interference of shared identity and appearance so the model focuses on semantic differences between similar expressions. Removing the symmetric framework raises mSCR from 0.0550 to 0.1350 and lowers Acc-6 from 0.8627 to 0.7843. The contribution is how contrastive learning is made to separate strongly similar expressions, not merely the use of a contrastive loss.

  3. Continuous multi-dimensional expression data. We build FFE, the first dataset for same-identity fine-grained expression editing with continuous multi-dimensional expression annotations, through a triple validation combining generation targets, image-based expression prediction, and human verification. Expression overlap and intensity variation that discrete labels cannot express are preserved in this supervision, so the model can learn fine differences between expressions and support smoother linear control. We emphasize that FFE is not only a new dataset; it provides training information that discrete supervision could not provide for expression separation and continuous control.

AuthorsOpenReview ↗

Kindly request for response

Dear Reviewer An9s,

Thank you again for your thoughtful review. We would appreciate your feedback on our responses to your two main concerns:

  • Evaluation independence: we added independent human annotations and two additional VLM evaluators in Appendix H. All evaluators retain the same method ordering on Acc-6 and mSCR; Fig. 17 further examines the variation in NBP accuracy.
  • Method novelty: our response clarifies the contributions of the learned continuous control mechanism, symmetric contrastive loss with fully symmetric joint training, and continuously annotated FFE data. We also highlight the ablation evidence for the symmetric training design.

We hope these additions and clarifications address your concerns. Please let us know if any questions remain; we would welcome the opportunity to clarify them.

Best regards,
The authors

AuthorsOpenReview ↗

Final follow-up before the discussion deadline

Dear Reviewer An9s,

With approximately three hours remaining before the discussion deadline, we would greatly appreciate your feedback on whether our additional evaluation and clarification of the methodological contributions address your concerns. If any questions remain, we are available to respond promptly.

Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

Thank you to all reviewers for your thoughtful and constructive feedback. We ran the requested evaluations, added two appendix sections, and uploaded a revised manuscript. The method and all original results remain unchanged; each reviewer's points are answered in the separate responses below, and the shared concern about evaluation independence is addressed here.


📋 Revision Summary

  1. New Appendix H: Evaluation reliability validation: five independent blind annotators and two additional VLMs (GPT-6 Astra and Claude Opus 5.5, alongside the original Gemini evaluator) cross-validate editing accuracy and structural confusion.
  2. New Appendix G: Generalization to real-world conditions: qualitative PixelSmile edits on WFLW (varied pose), CelebA (varied lighting), and COFW (occlusion), all outside the training distribution.
  3. The main results and method are unchanged.

🔍 Evaluation Reliability Validation

Shared concern across reviewers: Gemini is used for both training-data annotation and benchmark evaluation, which could introduce bias.

What we already had in original version. Appendix F.4 already validates the intensity scores against human judgments on 360 images covering 12 expressions, with Pearson and Spearman correlations of 0.79 and 0.81.

What we add in revised version. We now add independent human annotations and evaluation by GPT-6 Astra and Claude Opus 5.5 to verify accuracy, confusion, and the robustness of the method comparisons across evaluators.

Human protocol

  • Five independent blind annotators label the same 300 images — 60 per method for accuracy and 40 per method for confusion — with results averaged across annotators.
  • These 300 images form a subset of FFE-Bench selected for this validation; all annotators evaluate the same images.

VLM protocol

  • Gemini is the original benchmark evaluator; GPT-6 Astra and Claude Opus 5.5 are newly added.
  • All three VLMs use the same scoring prompt as Gemini and do not receive the target expression, method name, or other evaluators' answers.
  • To reduce sampling noise, the VLM accuracy set is expanded to 100 images per method.

Cross-evaluator results (↑ higher is better; ↓ lower is better; bold = best within each evaluator)

EvaluatorMethodAcc-6 ↑Acc-12 ↑mSCR ↓
👥 HumanPixelSmile0.88890.65560.0167
NBP0.71110.63890.1167
Qwen0.51110.38890.1583
🤖 Gemini 3 ProPixelSmile0.88000.60000.0750
NBP0.86000.62630.1250
Qwen0.54000.35000.2250
🤖 GPT-6 AstraPixelSmile0.98000.67000.0500
NBP0.90000.63640.1500
Qwen0.50000.33000.2000
🤖 Claude Opus 5.5PixelSmile0.88000.61000.0250
NBP0.66000.49490.2250
Qwen0.50000.30000.2500

📊 Consistent ranking across evaluators

Human annotations and all three VLMs consistently order PixelSmile > NBP > Qwen on Acc-6 and mSCR. Although the absolute scores differ, the relative ordering is consistent, showing that PixelSmile's advantage does not depend on the Gemini evaluator.

👥 Agreement among human annotators

On the six basic expressions, the annotators reach a Fleiss' κ of 0.62 (0.82 for PixelSmile, 0.54 for NBP, 0.47 for Qwen), with the highest agreement on PixelSmile outputs. PixelSmile's clearer expressions make it easier for annotators to agree on the category, whereas Qwen's weak expressions lead to greater disagreement among annotators.

📈 Variation in NBP accuracy across evaluators

  • Different recognition thresholds. In our inspection, GPT-6 Astra recognizes weaker expressions, whereas Claude Opus 5.5 requires more pronounced expressions, suggesting a lower effective threshold for GPT and a higher one for Claude.
  • Support from score distributions. Averaging three VLMs' target scores per image (Appendix H, Fig. 17) shows that NBP concentrates in the intermediate range, supporting its greater threshold sensitivity. PixelSmile and Qwen have fewer samples there and are less affected.
  • Similar variation among annotators. Those who recognize weak expressions give NBP an Acc-6 around 0.90, versus 0.60 for those who find them harder to recognize; the mean is around 0.70.

Since expression editing is intended for human viewers, aggregate human judgments are our primary reference, with cross-VLM evaluation providing complementary evidence.

AuthorsOpenReview ↗

Request for follow-up on our rebuttal

Dear AC, SAC, PC, and Reviewers,

Thank you for your time and constructive feedback. We have provided point-by-point responses and revised the manuscript with two additional evaluations:

  • Evaluation reliability (Appendix H): independent human annotations and two additional VLM evaluators, GPT-6 Astra and Claude Opus 5.5. Human annotators and all three VLMs consistently rank PixelSmile > NBP > Qwen on Acc-6 and mSCR. We also added a score-distribution analysis (Fig. 17) to examine the larger variation in NBP accuracy across evaluators.
  • Real-world generalization (Appendix G, Fig. 16): qualitative results on WFLW, CelebA, and COFW under varied pose, lighting, and occlusion.

We would appreciate the reviewers' follow-up on whether these additions and our individual responses address their concerns. We would also be grateful for the AC's help in encouraging further discussion, so that we can clarify any remaining questions during the discussion period.

Thank you again for your time and consideration.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.