acceptodds
Withdrawn from ICLR 2027

AffectEdit: Diagnosing and Fixing Evaluation for Affective Image Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image editors can now change the emotion an image conveys, yet it is unclear whether the metrics used to evaluate such affective edits agree with human judgment. To test them, we collect AffectEdit, a benchmark of 1,406 edits, each rated by about five annotators for whether it conveys the target emotion while preserving the scene. We find that the metrics in current use for affective editing fall short because they assess these two requirements separately. Emotion-only metrics can reward an edit that replaces the entire scene, scene-only metrics reward an edit that keeps the scene but leaves the emotion unchanged, and the one metric in current use that scores both adds the two terms, so either can mask a failure of the other. Replacing its addition with a multiplication, on the metric's own terms, makes it reject unchanged images and significantly raises its correlation with human ratings over the whole benchmark. We therefore propose PAEF (Perceived Affective Edit Fidelity), a parameter-free metric that multiplies an edit's emotion shift by its scene preservation, so that an edit scores highly only when both requirements hold. PAEF agrees with human ratings better than every metric in current use for affective edits, and holds that level on a held-out set of edits that informed none of its design choices. The strongest multimodal large language models (MLLMs), prompted or fine-tuned as judges, agree better still, but PAEF runs 3.7 to 61 times faster than an MLLM judge on the same GPU and needs no rated data. We release the benchmark, the ratings and our code (supplementary material).

Primary area:
datasets and benchmarks
Keywords:
affective editing, image editing, affective benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JeQWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether affective image-editing metrics align with human judgments of emotional change and scene preservation. It introduces AffectEdit, a benchmark of 1,406 edits from five editors with approximately five human ratings per edit. The authors identify limitations of existing metrics and propose PAEF, which combines classifier-based and valence–arousal emotion shifts with a multiplicative DINOv2 preservation gate. Ablations and diagnostic probes support preservation gating, while comparisons with MLLM judges and learned evaluators show that PAEF offers substantially lower computational cost at the expense of weaker human agreement than the strongest judges.

Critical strengths

The paper provides a useful empirical audit for affect assessment problem. It evaluates metrics against joint human judgments to reduce the reliance on emotion classifier scores alone.

Critical weaknesses

W1. It is unclear which types of modifications receive higher scores. Appendix A.4 restricts lighting-only and color-based edits when generating editing instructions, whereas Appendix D.1 describes the highest rating as “atmospheric only”. I consider lighting and color adjustments to be forms of atmospheric modification. This ambiguity limits confidence that the benchmark applies consistent criteria across different types of edits.

W2. The ablations do not sufficiently isolate the sources of improvement. Adding DA introduces both an additional V–A predictor and a target-direction projection, leaving their contributions entangled. Moreover, in Table 1, the transition from B to C simultaneously replaces the emotion and preservation terms. More controlled ablations are needed to clarify whether the gains arise from the proposed mechanism, the component choices, or their interaction.

Questions for authors

See Weaknesses.

4 comments

Thank you for two precise points; both are fair and fixable. In short: the top anchor does favour atmospheric edits and minor details, which we will quote and state with its limits, and the controlled ablations you ask for show that the gains come mainly from the component choices, with a conditional role for the product.

W1 (which edits score higher; R1 vs anchor 4). Your reading is essentially right. Raters saw anchor 4 as "Excellent, seamless: strongly [emotion], scene almost identical, only mood/lighting/atmosphere/minor details" (HIT screenshot, App. D), under the question "How well did this edit make the image feel more [emotion] while keeping the same scene?" Lighting and colour changes are atmospheric and can earn a 4, and so can a small added or removed detail; by the anchor's wording, an edit that adds or removes a prominent object cannot, and fits anchor 3 at best ("Good edit, some changes: clearly more [emotion], same place, changes make sense for the emotion"). The criteria are the same for every edit, but they are not neutral across kinds of edit: the top of the scale goes to the edits that change the scene least. This follows our definition of success, maximal emotion impact with minimal scene change. Our App. D.1 paraphrase "(atmospheric only)" dropped "scene almost identical" and "minor details"; we will quote the screenshot and state this preference.

R1 limited only the instructions we generated for FLUX.1 Kontext and Qwen-Image-Edit, because filter-only instructions gave weak edits in early runs, and so steered them toward content changes that the rubric rates 3 unless they are minor. Their edits still receive 4s (about 16% of their ratings), but the benchmark does not record how many of these are minor details rather than prominent objects. EmoEdit and EmoEditor receive only the target emotion, and AIF-Word applies filters only, so filter-only edits come almost only from AIF-Word, the weakest editor, and AffectEdit cannot say how raters judge a strong filter-only edit. We will state both as limitations. The probe results rest on anchor 0 (no emotional change, or a different scene), which holds under any reading of anchor 4.

W2 (isolating the gains). Agreed on both counts. We ran both controlled ablations you ask for and will replace Table 1 with them (paired intervals per wave in an appendix). All values are τ_b against the human mean, on all pairs.

(a) DA: V/A predictor vs target direction. Same V/A predictor and gate C throughout:

V/A term (gated by C)aloneaveraged with ES
none–0.184 (ES·C)
shift magnitude ‖(ΔV, ΔA)‖: predictor, no direction0.1740.213
DA: projection onto the target direction (DA's clip at zero, part of PAEF since submission, changes nothing)0.2040.219 (PAEF)
  • Inside PAEF, the predictor carries most of DA's contribution. Adding DA to ES·C raises τ_b from 0.184 to 0.219 (in over 99% of resamples), and the direction-blind magnitude gets most of the way (0.213; the direction adds +0.006 [−0.000, 0.013]). At equal weights ES dominates the average: it varies about six times as much as DA.
  • The direction matters when DA stands alone: +0.030 [0.012, 0.048] over the magnitude, though not significant on the held-out wave alone.
  • The weight. We also swept ((1−α)·ES + α·DA)·C, choosing α on the first wave and testing it on the held-out wave (protocol fixed in advance). Every mix from α = 0.3 to 0.9 scores above either signal alone, and agreement is flat from 0.6 to 0.95, so the two signals are complementary. The chosen α = 0.8 adds +0.018 [0.003, 0.033] on the held-out wave but loosens the top fifth (3.2% failures vs 2.1%); both probes pass at every weight. As with the zero-shot classifier (our reply to NF9t), we will report it as a variant rather than redefine PAEF.

(b) Table 1's B → C step, as a full 2×2×2 factorial (each part changed with the other two fixed; S_emo and S_str from ESMI's official code):

Emotion termPreservation termOperatorτ_bContent swaps acceptedUnchanged copiesTop-fifth failures
S_emoS_straverage (= ESMI)0.08631%rejected12.1%
S_emoS_strproduct (old B)0.08029%rejected12.4%
S_emoCaverage0.1240%rejected7.4%
S_emoCproduct0.1290%rejected7.1%
ESS_straverage0.11831%rejected5.3%
ESS_strproduct0.16432%rejected6.0%
ESCaverage0.1540%ranked above most genuine edits3.2%
ESCproduct (old C)0.1840%rejected2.1%

From old B, changing only the preservation term gives 0.129 and rejects every swap; changing only the emotion term gives 0.164 but still accepts 32% of swaps; changing both gives old C (0.184).

Mechanism, components, or interaction?

  • Component choices carry most of the gain. The preservation term alone decides whether swaps are rejected (every S_str row accepts 29–32%, every C row none); re-weighting ESMI's two terms does not change this, since any weight on S_str lowers agreement and swaps pass at every weight. The emotion term brings the larger agreement gain, which comes from the EmoEdit classifier, not the form of ES (the same shift with ESMI's ResNet-18 reaches 0.120 with C).
  • The mechanism (product vs average) has a conditional effect. None with S_emo; with ES, +0.047 alongside S_str and +0.031 alongside C (the latter not significant on the held-out wave alone); with PAEF's full emotion term, averaging lowers τ_b from 0.219 to 0.148 (Table 1, last row). Its clearest job is unchanged copies: with ES and C, the average ranks them above most genuine edits, while the product scores them zero.
  • Interaction. Replacing S_str by C adds more agreement with S_emo (+0.050) than with ES (+0.020), the operator matters only with ES, and DA's direction matters mainly without ES.

Our submitted introduction therefore overstated the operator; the revision will present the component choices as the main source of the gain and the product as the safeguard against unchanged copies.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Based on the authors’ clarification of the rating criteria and their preference for scene-preserving edits, I have an important concern regarding benchmark fairness.

  • The rubric favors "the edits that change the scene least", while prominent object additions or removals cannot receive the highest score, even if they effectively convey the target emotion.

  • Different methods face different editing constraints: FLUX.1 Kontext and Qwen-Image-Edit use different edits with AIF-Word. Thus, scores could reflect editing constraints as well as editing capabilities.

  • The reported results (16% for FLUX.1 Kontext and Qwen-Image-Edit) does not resolve this concern, as the authors cannot distinguish minor-detail edits from prominent-object modifications.

Thus, I maintain my overall rating given the remaining concern regarding benchmark fairness across different editing types.

Reviewer_5B58OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript presents AffectEdit, a benchmark for evaluating affective image editing, and proposes PAEF, a training-free evaluation metric that combines emotion shift and scene preservation through multiplicative gating. The problem of reliably evaluating affective image editing is relevant, and the benchmark provides useful empirical insights into the limitations of existing metrics. However, the methodological novelty is limited, as PAEF primarily combines established emotion estimation, feature similarity, and multiplicative gating techniques. The distinct technical contribution beyond this combination is not sufficiently established.

Critical strengths
  • The paper addresses a meaningful evaluation problem in affective image editing, highlighting the limitations of measuring emotion changes and scene preservation independently or through additive combinations.
  • The AffectEdit benchmark provides human-rated evaluations across multiple editing models and target emotions, with reported inter-rater reliability supporting the validity of the collected annotations.
Critical weaknesses
  • The methodological novelty of PAEF is limited. Its core formulation combines existing emotion classification shifts, valence–arousal representations, and DINOv2 feature similarity through a simple multiplicative operation. Although the combination is intuitive, it does not introduce a substantially new evaluation mechanism beyond established components.
  • The central idea of multiplicative gating is not fundamentally new. Related work such as FED-Bench has already employed multiplicative fidelity constraints for image editing evaluation. Similarly, categorical emotion shifts and valence–arousal-based measurements have been explored in prior studies. Extending these existing concepts to scene-level affective editing does not, by itself, establish a strong methodological contribution.
  • The paper emphasizes multiplicative composition as a key contribution, but the ablation results suggest that replacing the underlying evaluation components accounts for much of the overall improvement. Moreover, alternative aggregation operators achieve comparable or better results under certain settings. The specific novelty and necessity of the proposed formulation therefore remain insufficiently justified.
Questions for authors

Given that multiplicative gating, emotion probability shifts, and valence arousal-based evaluation have already been explored in prior work, what specific methodological contribution distinguishes PAEF from a straightforward combination of these established techniques?

3 comments

Thank you for crediting the benchmark and its reliability.

W1/W2 (known parts; FED-Bench). We agree that PAEF's parts are known, including the product, and we do not claim a new mechanism; our submitted Related Work already credits FED-Bench for multiplicative gating. The two works share the operator, not the problem. FED-Bench scores a displayed facial expression against a ground-truth target image, with a face-identity term, a pixel-level background term and MLLM judgments. A scene-level affective edit has no single correct target, and it is meant to change light and colour across the whole image, so pixel-level preservation points the wrong way: on AffectEdit, SSIM, PSNR and LPIPS agree with raters at or below zero. What such a metric needs therefore had to be measured rather than transferred, which is what our audit and factorial do. Our introduction did present the composition rule as the first contribution, which overstated the operator's role; we will rewrite it.

Q1 (what distinguishes PAEF). What distinguishes PAEF is not a new component or operator but evidence: the straightforward combinations of these techniques each miss one of the two ways an affective edit fails, and controlled tests on human ratings show which choice avoids which failure.

What we ask you to weigh is the part your review names as a strength, "the limitations of measuring emotion changes and scene preservation independently or through additive combinations":

  1. AffectEdit: 1,406 edits from five editors, each rated by about five people on an absolute scale, collected in two waves; every PAEF choice was fixed on the first wave and tested on the second. To our knowledge these are the first absolute human ratings of affective edits with reported reliability.
  2. Two probes and an audit. An affective edit fails if it replaces the scene (62 content swaps that raters rejected) or leaves the image unchanged (200 copies). Emotion-only metrics, including the emotion-probability shift and valence–arousal shifts, accept 26–39% of the swaps; scene-only metrics accept every unchanged copy; ESMI, recomputed with its official code, accepts 31% of the swaps. Seven of eight MLLM judges also separate swaps from genuine edits less well than PAEF.
  3. PAEF, which needs no rated data, rejects both probes, and agrees with raters better than the metrics affective-editing papers report (τ_b 0.219; the best of them, Emo-S, 0.172, is lower in every resample). The strongest MLLM judges and EditReward agree better still, as the paper states; PAEF is a cheap, deterministic screen, 3.7–61× faster than the judges.

W3 (components vs operator). You are right that the components carry most of the gain. Table 1 changed one thing at a time, so we ran a 2×2×2 factorial, changing each part with the other two fixed (protocol fixed before the run), and will replace Table 1 with it:

Emotion termPreservation termOperatorτ_bContent swaps acceptedUnchanged copiesTop-fifth failures
S_emo (ESMI)S_str (ESMI)average (= ESMI)0.08631%rejected12.1%
S_emoS_strproduct0.08029%rejected12.4%
S_emoC (DINOv2)average0.1240%rejected7.4%
S_emoCproduct0.1290%rejected7.1%
ESS_straverage0.11831%rejected5.3%
ESS_strproduct0.16432%rejected6.0%
ESCaverage0.1540%ranked above most genuine edits3.2%
ESCproduct0.1840%rejected2.1%
½(DA + ES)Cproduct (PAEF)0.2190%rejected2.1%

τ_b against the human mean on all 1,406 pairs; probes at each score's matched bar; S_emo and S_str from ESMI's official code.

  • The preservation term decides the swaps: every S_str row accepts 29–32% of them, every C row none. ESMI's term measures where an edit changes the image, not whether the scene survives.
  • The emotion term brings the largest gain in agreement, and it comes from the classifier. Yet the lead over ESMI is not just the classifier: with the same classifier inside both metrics, PAEF's design beats the better of ESMI's sum and product for each of three classifiers, by 0.066 to 0.085 in τ_b, and only PAEF's rejects the swaps (three runs, two pre-registered).
  • The operator's effect depends on the emotion term: none with S_emo, a modest gain with ES (with C, not significant on the held-out wave alone); with PAEF's full emotion term, averaging lowers τ_b from 0.219 to 0.148 (Table 1, last row). Its clearest job is unchanged copies: with ES and C, the average ranks them above most genuine edits, while the product scores them zero.
  • DA adds a second, continuous emotion signal. On its own, its target direction helps; inside PAEF, the V/A predictor does most of the work (see our reply to JeQW).

One part of the submitted composition claim does not survive: that switching ESMI's own terms to a product makes it reject identity edits. It rested on our reimplementation of ESMI, which we have withdrawn (see our reply to NF9t); with the official code, ESMI already rejects identity edits, and its operator makes no difference.

W3 (other operators). As our operator sweep reports (Table 14 of the submission; Table 20 of the revision), after min–max rescaling the minimum, harmonic mean and Hamacher product score significantly higher than the product on that scale (none above 0.220), while addition is among the bottom three under every scaling. As the submission says, we keep the product because it needs no common scale for two signals from different instruments, not because it wins.

What will change. The revision will lead with the benchmark and the audit, present PAEF as a simple metric built on their findings, and report the factorial and the same-classifier comparison. We hope this answers your question.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_NF9tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the evaluation of instruction-based affective image editing and introduces AffectEdit, a human-annotated benchmark of 1,406 source-edit pairs across five editing systems rated by 20 annotators. To address reference-free evaluation failures, the authors propose PAEF (PAEF = E * C), a reference-aware metric that multiplies signed affective delivery E = clip((DA + ES)/2, -1, 1) by structural consistency C = clip((Delta_CLIP_struct + SSIM)/2, 0, 1), alongside a LoRA-tuned Qwen2.5-VL-7B evaluator.

Critical strengths

(1) Well-motivated diagnosis of reference-free evaluation failures and a valuable human benchmark: The paper clearly illustrates how reference-free metrics fail under identity copies, scene replacement, and direction-agnostic scoring, and releases a 1,406-pair human-rated benchmark across five editors with transparent inter-annotator agreement analyses (Krippendorff's alpha = 0.562, Appendix D.3).

(2) Interpretable reference-aware decomposition and strong fine-tuned VLM alignment: Decomposing evaluation into signed affective delivery E and content preservation C forces unmodified copies to score zero (PAEF = 0) and lowers top-quintile human-rated failures from 13.8% (ESMI) and 5.3% (DA * C) to 2.1% (Table 16), while LoRA-tuning Qwen2.5-VL-7B further improves correlation with human ratings.

Critical weaknesses

(1) Mismatched characterization and implementation of the primary baseline (ESMI): Section 3 (p. 4) and Appendix A.1 (p. 17) claim that ESMI (Lin et al., ICCV 2025) includes a Canny edge-map term in its structural score S_str that yields S_str(I, I) = 1, ESMI(I, I) >= 0.5, and a 100% identity-edit acceptance rate (Id-Acc = 100% in Tables 1, 2, and 19). However, in both the published ICCV 2025 paper (Sec. 4.3, p. 6) and its official code (metric_ESMI.py), S_str = L_emo / (L_full + 1e-10) has no edge-map term and outputs 0 on identical images (Id-Acc = 0%), meaning the headline no-op failure attributed to ESMI comes from a custom reimplementation rather than the official baseline.

(2) Sign-reversed content scaling when E < 0 and circular classifier bias in ES: Combining signed classifier shift ES (in [-1, 1]) with non-negative direction alignment DA (clipped at 0, empirically < 0.3) makes E = clip((DA + ES)/2, -1, 1) negative whenever classifier noise produces ES < -DA; multiplying E < 0 by content consistency C > 0 then penalizes faithful edits with higher C more heavily than unedited copies (PAEF = 0), which aligns with Figure 9 where 22 of the 30 human-approved edits in PAEF's bottom quintile have ES < 0 on high-initial-probability sources. Moreover, because ES reuses the exact EmoSet-trained CLIP classifier (2024-02-23-best.pth) used to filter EmoEdit's training set (Appendix A.2), adding ES over DA * C yields no statistically significant overall tau_b gain (0.219 vs. 0.204, bootstrap win share P(PAEF > DA*C) = 0.82 with 95% CI [-0.017, +0.047] in Tables 1 and 13) while degrading system-level editor ranking from Spearman 1.00 (DA * C) to 0.60 by promoting EmoEdit to #1 (Table 12).

(3) Weak within-editor discrimination on modern editors and rubric-prompt inconsistencies: Within the two strongest modern editors (FLUX.1 Kontext and Qwen-Image-Edit, which make up 63.4% of AffectEdit), PAEF achieves low correlation with humans (tau_b = 0.082 and 0.090 in Table 7; 0.080 and 0.090 in Table 8; mean within-editor tau_b = 0.088), performing comparably to the reference-free classifier Emo-S (0.080 / 0.070) and well below EditReward (0.172 / 0.185). Additionally, instruction generation rule R1 (Appendix A.4) forbids lighting- or atmosphere-only edits for FLUX and Qwen whereas human rubric Anchor 4 (Appendix D.1) defines the top rating of 4 as "Excellent / seamless (atmospheric only)", and the 5-fold cross-validation for Qwen2.5-VL-7B (LoRA) (Appendix A.5) is stratified by emotion rather than grouped by source image.

Questions for authors
  1. Why do Section 3 and Appendix A.1 define ESMI's structural term S_str with a Canny edge-map component yielding S_str(I, I) = 1, when Lin et al. (ICCV 2025, p. 6) and the official metric_ESMI.py define S_str = L_emo / (L_full + 1e-10) and output ESMI(I, I) = 0?

  2. How do you address the monotonicity reversal of multiplying E * C when E < 0 (which penalizes higher content preservation C and places 22 human-approved edits with ES < 0 into the bottom quintile), and how does system ranking (Table 12) change when ES uses an independent classifier not used to build EmoEdit?

  3. How do you reconcile instruction rule R1 (forbidding atmospheric-only edits for FLUX and Qwen) with human rubric Anchor 4 (reserving the top score of 4 for atmospheric-only edits), and what is the LoRA judge's cross-validation tau_b when folds are grouped by source image?

4 comments

In short. Thank you for checking our code against the official ESMI implementation.

  • W1: You are right. Our "ESMI" was a reimplementation, and we withdraw the three claims that depended on it. ESMI as published still agrees with raters far less than PAEF and still accepts content swaps.
  • W2: As you suspected, the classifier causes PAEF's preference for EmoEdit; ES from zero-shot CLIP removes it and agrees with raters better. Clipping negative E changes no result, so PAEF stays as defined.
  • W3: PAEF does rank edits weakly within FLUX and Qwen, and we narrow the claim. You are right that anchor 4 favours atmospheric edits and minor details; we will state how this interacts with R1. Grouping folds by source image leaves the judge result essentially unchanged.

Section and table numbers refer to the submitted paper unless marked otherwise. The revised PDF is uploaded, with every change in blue; it contains the changes we describe below.

W1 / Q1 (ESMI). Three things in our ESMI were wrong.

  • Description. Our App. A.1 shows “... over Grad-CAM-selected emotion-neutral regions plus an edge-map term …” but our code computed no edge term. It differed from metric_ESMI.py in two places which are classifier and guard line bug.
  • Classifier. Our S_emo used the EmoEdit classifier (the one behind PAEF's ES) instead of ESMI's ResNet-18. We meant to put both metrics on one classifier, but did not say so and reported the result as ESMI.
  • One guard line. Our S_str equals the official one on all 1,406 pairs, but not on an unchanged image:
official:  S_str = L_emo / (L_full + 1e-10)                            # unchanged image -> 0
ours:      S_str = L_emo / (L_full + 1e-10) if L_full > 1e-10 else 1   # unchanged image -> 1

The guard did no numerical work (the 1e-10 already prevents division by zero). It matches Sec. 3's reading of S_str as a preservation score ("an identity edit with perfect preservation scores α"), which we also withdraw: S_str is the share of the change that falls in emotion regions, so an unchanged image scores 0. This line alone made our version accept every identity edit.

ESMI as published (official code, every pair, both probes):

ESMI as publishedPAEF
τ_b with raters0.086 [0.048, 0.122]0.219 [0.184, 0.252]
identity edits accepted0%0%
content swaps accepted31% [19, 44]0%
rated failures in the top fifth12.1%2.1%

PAEF is higher in all 2,000 bootstrap resamples, on both waves. We withdraw the three claims that held only for our reimplementation: that ESMI is not significant, that it accepts identity edits, and that multiplying its terms fixes both (as published, the product scores 0.080, no better than the sum).

What still holds. ESMI accepts content swaps because S_str measures where an edit's change falls, not whether the scene survives. With the same classifier in both metrics, PAEF's composition beats the better of ESMI's sum and product for each of three classifiers, by 0.066 to 0.085 in τ_b, on both waves (three runs, two pre-registered; the summary is post hoc), and only PAEF's composition rejects content swaps (ESMI's versions accept 21–40%). We will add these results, with every ESMI number from the official code.

W2 / Q2. 1. Negative E. For the 109 pairs (7.8%) with E < 0, better preservation lowers the score, as you note. Clipping E at zero, max(E, 0)·C, removes this and changes no result: τ_b stays 0.219, top-fifth failures stay 2.1%, and the editor ranking is unchanged. We therefore keep PAEF as defined and report the clipped form as a sensitivity check. It does not explain Fig. 9: only 14 of those 30 edits have E < 0, and all 30 stay in the bottom fifth after clipping. They come from ES saturating on sources the classifier already rates high; Sec. 4.6 traces both kinds of error to ES.

2. Classifier and editor ranking. As you suspected, PAEF's preference for EmoEdit comes from the classifier. We ran the test you asked for under a protocol fixed in advance: ES from zero-shot CLIP, which has no EmoSet lineage.

ES fromτ_beditor ranking vs raters (ρ, full benchmark)EmoEdit ranked firsttop-fifth failures
EmoEdit classifier (PAEF)0.2190.6in 80% of resamples2.1%
zero-shot CLIP0.2531.0in 0%5.0%

The zero-shot version agrees better (+0.035 [0.006, 0.064]; 0.267 vs 0.229 on the held-out wave) and rejects both probes. With ESMI's ResNet-18, also EmoSet-trained, PAEF again ranks EmoEdit first: the bias follows the training data. Under the decision rule we fixed with the test, we will report the zero-shot form as a classifier-independent variant rather than redefine PAEF, whose design was frozen before the held-out wave.

3. DA·C. Our claim for ES (Sec. 4.3) is that it keeps the top fifth clean: without ES, agreement is level, but top-fifth failures rise from 2.1% to 5.3%. The zero-shot ES does not give this (5.0%), so the classifier trades a clean top fifth against an unbiased editor ranking; we will state this trade-off.

W3 (within-editor ranking). Agreed. Within FLUX.1 Kontext and Qwen-Image-Edit, PAEF ranks edits only weakly (τ_b 0.08–0.09), and EditReward, trained on human edit preferences, does about twice as well (0.17–0.19). We will state the scope plainly: PAEF is a cheap, deterministic screen that rejects content swaps and unchanged copies and keeps its top fifth clean, not a fine ranker among strong edits.

  • Raters also agree less on these two editors (reliability of the five-rater mean 0.73 vs 0.87 on the other three), but this does not explain PAEF's larger drop.
  • Our "within editor" column was a poor summary: it removes each editor's mean but still compares pairs across editors. We will relabel it and add every method's τ_b within each editor; averaged that way, EditReward stays ahead of PAEF (0.159 vs 0.138), not significantly.
  • EditReward fails differently: it scores unchanged copies almost as high as genuine edits (Table 2, footnote), and its top fifth holds twice PAEF's share of rated failures.
  • The zero-shot variant (W2) narrows the gap within FLUX and Qwen but does not close it (post hoc).

Q3a (R1 vs anchor 4). You are right that anchor 4 favours atmospheric edits, though not only them. Raters saw it as "Excellent, seamless: strongly [emotion], scene almost identical, only mood/lighting/atmosphere/minor details" (HIT screenshot, App. D); our App. D.1 paraphrase dropped "scene almost identical" and "minor details", and we will quote the screenshot. A small added or removed detail can earn a 4; by the anchor's wording, a prominent added or removed object cannot, and fits anchor 3 at best ("same place, changes make sense for the emotion"). This follows our definition of success, maximal emotion impact with minimal scene change, but it means R1, which applied only to the instructions we generated for FLUX and Qwen, steered those editors toward content changes that the rubric rates 3 unless they are minor. Their edits still receive 4s (about 16% of their ratings), but the benchmark does not record how many of these are minor details rather than prominent objects. We will state this, and that filter-only edits come almost only from AIF-Word, the weakest editor, so AffectEdit cannot say how raters judge a strong filter-only edit. The probe results rest on anchor 0 (no emotional change, or a different scene), which holds under any reading of anchor 4.

Q3b (grouped folds). Fair point: in the original folds most test pairs shared a source image with training. Grouped by source image (protocol fixed before the run), the tuned judge reaches τ_b 0.338 (0.332 before) against PAEF's 0.219 on the same folds, essentially unchanged. We will report both.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_NF9tOpenReview ↗

Response to Authors

Thank you for checking the code, withdrawing the unsupported ESMI claims, and running all requested experiments. While your response resolves the technical ambiguities, several core limitations remain in the revised results:

  1. ESMI baseline and the ESES trade-off in PAEF: Official ESMI already rejects 100% of identity edits (0% acceptance), and replacing its sum with a product gives no gain (τb=0.080\tau_b = 0.080 vs. 0.0860.086). Moreover, PAEF's ESES term faces an unresolved trade-off: the EmoSet classifier reduces top-fifth failures (2.1%2.1\% vs. 5.3%5.3\% for DA⋅CDA \cdot C) at the cost of training-data circularity that distorts system ranking (ρ=0.60\rho = 0.60), whereas zero-shot CLIP restores ρ=1.00\rho = 1.00 but loses the top-fifth filtering benefit (5.0%5.0\% failures, matching simple DA⋅CDA \cdot C at 5.3%5.3\%).
  2. Weak discrimination within modern editors: Within FLUX.1 Kontext and Qwen-Image-Edit (63.4% of AffectEdit), PAEF correlates weakly with humans (τb=0.08–0.09\tau_b = 0.08\text{--}0.09), and under the corrected per-editor average, the existing baseline EditReward (0.1590.159) outperforms PAEF (0.1380.138). Re-scoping PAEF to a coarse pre-screen leaves limited advantage over simple DA⋅CDA \cdot C, which already rejects 100% of identity and content-swap probes with ρ=1.00\rho = 1.00.
  3. Structural gap in the benchmark (R1 vs. Anchor 4): Rubric Anchor 4 reserves the top score of 4 for mood/lighting/atmosphere or minor detail edits (capping prominent object changes at 3), yet rule R1 forbade atmospheric-only prompts for FLUX and Qwen. As noted in your reply, AffectEdit cannot evaluate how raters judge strong atmospheric-only edits, which is a dataset-level gap that text caveats alone cannot resolve.

Given these remaining limitations in both the proposed metric and the benchmark coverage, I maintain my rating of 2 (Weak rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.