- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether affective image-editing metrics align with human judgments of emotional change and scene preservation. It introduces AffectEdit, a benchmark of 1,406 edits from five editors with approximately five human ratings per edit. The authors identify limitations of existing metrics and propose PAEF, which combines classifier-based and valence–arousal emotion shifts with a multiplicative DINOv2 preservation gate. Ablations and diagnostic probes support preservation gating, while comparisons with MLLM judges and learned evaluators show that PAEF offers substantially lower computational cost at the expense of weaker human agreement than the strongest judges.
Critical strengths
The paper provides a useful empirical audit for affect assessment problem. It evaluates metrics against joint human judgments to reduce the reliance on emotion classifier scores alone.
Critical weaknesses
W1. It is unclear which types of modifications receive higher scores. Appendix A.4 restricts lighting-only and color-based edits when generating editing instructions, whereas Appendix D.1 describes the highest rating as “atmospheric only”. I consider lighting and color adjustments to be forms of atmospheric modification. This ambiguity limits confidence that the benchmark applies consistent criteria across different types of edits.
W2. The ablations do not sufficiently isolate the sources of improvement. Adding DA introduces both an additional V–A predictor and a target-direction projection, leaving their contributions entangled. Moreover, in Table 1, the transition from B to C simultaneously replaces the emotion and preservation terms. More controlled ablations are needed to clarify whether the gains arise from the proposed mechanism, the component choices, or their interaction.
Questions for authors
See Weaknesses.
4 comments
W2 (isolating the gains). Agreed on both counts. We ran both controlled ablations you ask for and will replace Table 1 with them (paired intervals per wave in an appendix). All values are τ_b against the human mean, on all pairs.
(a) DA: V/A predictor vs target direction. Same V/A predictor and gate C throughout:
| V/A term (gated by C) | alone | averaged with ES |
|---|---|---|
| none | – | 0.184 (ES·C) |
| shift magnitude ‖(ΔV, ΔA)‖: predictor, no direction | 0.174 | 0.213 |
| DA: projection onto the target direction (DA's clip at zero, part of PAEF since submission, changes nothing) | 0.204 | 0.219 (PAEF) |
- Inside PAEF, the predictor carries most of DA's contribution. Adding DA to ES·C raises τ_b from 0.184 to 0.219 (in over 99% of resamples), and the direction-blind magnitude gets most of the way (0.213; the direction adds +0.006 [−0.000, 0.013]). At equal weights ES dominates the average: it varies about six times as much as DA.
- The direction matters when DA stands alone: +0.030 [0.012, 0.048] over the magnitude, though not significant on the held-out wave alone.
- The weight. We also swept ((1−α)·ES + α·DA)·C, choosing α on the first wave and testing it on the held-out wave (protocol fixed in advance). Every mix from α = 0.3 to 0.9 scores above either signal alone, and agreement is flat from 0.6 to 0.95, so the two signals are complementary. The chosen α = 0.8 adds +0.018 [0.003, 0.033] on the held-out wave but loosens the top fifth (3.2% failures vs 2.1%); both probes pass at every weight. As with the zero-shot classifier (our reply to NF9t), we will report it as a variant rather than redefine PAEF.
(b) Table 1's B → C step, as a full 2×2×2 factorial (each part changed with the other two fixed; S_emo and S_str from ESMI's official code):
| Emotion term | Preservation term | Operator | τ_b | Content swaps accepted | Unchanged copies | Top-fifth failures |
|---|---|---|---|---|---|---|
| S_emo | S_str | average (= ESMI) | 0.086 | 31% | rejected | 12.1% |
| S_emo | S_str | product (old B) | 0.080 | 29% | rejected | 12.4% |
| S_emo | C | average | 0.124 | 0% | rejected | 7.4% |
| S_emo | C | product | 0.129 | 0% | rejected | 7.1% |
| ES | S_str | average | 0.118 | 31% | rejected | 5.3% |
| ES | S_str | product | 0.164 | 32% | rejected | 6.0% |
| ES | C | average | 0.154 | 0% | ranked above most genuine edits | 3.2% |
| ES | C | product (old C) | 0.184 | 0% | rejected | 2.1% |
From old B, changing only the preservation term gives 0.129 and rejects every swap; changing only the emotion term gives 0.164 but still accepts 32% of swaps; changing both gives old C (0.184).
Mechanism, components, or interaction?
- Component choices carry most of the gain. The preservation term alone decides whether swaps are rejected (every S_str row accepts 29–32%, every C row none); re-weighting ESMI's two terms does not change this, since any weight on S_str lowers agreement and swaps pass at every weight. The emotion term brings the larger agreement gain, which comes from the EmoEdit classifier, not the form of ES (the same shift with ESMI's ResNet-18 reaches 0.120 with C).
- The mechanism (product vs average) has a conditional effect. None with S_emo; with ES, +0.047 alongside S_str and +0.031 alongside C (the latter not significant on the held-out wave alone); with PAEF's full emotion term, averaging lowers τ_b from 0.219 to 0.148 (Table 1, last row). Its clearest job is unchanged copies: with ES and C, the average ranks them above most genuine edits, while the product scores them zero.
- Interaction. Replacing S_str by C adds more agreement with S_emo (+0.050) than with ES (+0.020), the operator matters only with ES, and DA's direction matters mainly without ES.
Our submitted introduction therefore overstated the operator; the revision will present the component choices as the main source of the gain and the product as the safeguard against unchanged copies.
Dear Reviewers,
The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!
Best, AC
Based on the authors’ clarification of the rating criteria and their preference for scene-preserving edits, I have an important concern regarding benchmark fairness.
-
The rubric favors "the edits that change the scene least", while prominent object additions or removals cannot receive the highest score, even if they effectively convey the target emotion.
-
Different methods face different editing constraints: FLUX.1 Kontext and Qwen-Image-Edit use different edits with AIF-Word. Thus, scores could reflect editing constraints as well as editing capabilities.
-
The reported results (16% for FLUX.1 Kontext and Qwen-Image-Edit) does not resolve this concern, as the authors cannot distinguish minor-detail edits from prominent-object modifications.
Thus, I maintain my overall rating given the remaining concern regarding benchmark fairness across different editing types.
Thank you for two precise points; both are fair and fixable. In short: the top anchor does favour atmospheric edits and minor details, which we will quote and state with its limits, and the controlled ablations you ask for show that the gains come mainly from the component choices, with a conditional role for the product.
W1 (which edits score higher; R1 vs anchor 4). Your reading is essentially right. Raters saw anchor 4 as "Excellent, seamless: strongly [emotion], scene almost identical, only mood/lighting/atmosphere/minor details" (HIT screenshot, App. D), under the question "How well did this edit make the image feel more [emotion] while keeping the same scene?" Lighting and colour changes are atmospheric and can earn a 4, and so can a small added or removed detail; by the anchor's wording, an edit that adds or removes a prominent object cannot, and fits anchor 3 at best ("Good edit, some changes: clearly more [emotion], same place, changes make sense for the emotion"). The criteria are the same for every edit, but they are not neutral across kinds of edit: the top of the scale goes to the edits that change the scene least. This follows our definition of success, maximal emotion impact with minimal scene change. Our App. D.1 paraphrase "(atmospheric only)" dropped "scene almost identical" and "minor details"; we will quote the screenshot and state this preference.
R1 limited only the instructions we generated for FLUX.1 Kontext and Qwen-Image-Edit, because filter-only instructions gave weak edits in early runs, and so steered them toward content changes that the rubric rates 3 unless they are minor. Their edits still receive 4s (about 16% of their ratings), but the benchmark does not record how many of these are minor details rather than prominent objects. EmoEdit and EmoEditor receive only the target emotion, and AIF-Word applies filters only, so filter-only edits come almost only from AIF-Word, the weakest editor, and AffectEdit cannot say how raters judge a strong filter-only edit. We will state both as limitations. The probe results rest on anchor 0 (no emotional change, or a different scene), which holds under any reading of anchor 4.