acceptodds
Withdrawn from ICLR 2027

AffectAgent: Grounded Planning and Memory Calibration for Affective Image Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Affective image editing aims to align visual content with a target affect while preserving its underlying semantics. The same affective target can be realized through multiple visual modifications. However, existing end-to-end editors jointly perform affective reasoning and image generation, leaving editing strategies implicit and insufficiently grounded in affective semantics. Moreover, current optimization methods often apply task-agnostic supervision, failing to adapt learning strength to varying affective transitions. To address these limitations, we introduce AffectAgent, a planner–executor–evaluator framework that combines grounded planning with memory calibration for controllable affective image editing. Specifically, we propose Knowledge-Grounded Multi-Plan Exploration (KGME), which retrieves affective knowledge to explicitly construct multiple semantically meaningful editing plans and optimize the planner over a grounded strategy space. We further introduce Memory-Calibrated Preference Optimization (MCPO) to organize historical outcomes into an episodic memory and calibrate each sample’s optimization strength against the performance distribution of tasks with similar affective transitions. Extensive experiments show that AffectAgent consistently reduces valence and arousal errors compared with state-of-the-art methods while maintaining competitive semantic consistency, demonstrating more accurate target-affect control.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Affective Image Editing, Valence-Arousal Modeling, Knowledge-Grounded Planning, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_KQXzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript presents AffectAgent, a planner–executor–evaluator framework for affective image editing. It introduces Knowledge-Grounded Multi-Plan Exploration (KGME), which constructs multiple editing plans using an external Emotional Knowledge Bank, and Memory-Calibrated Preference Optimization (MCPO), which adjusts preference supervision based on historical outcomes from similar affective transitions. Experiments demonstrate improved valence and arousal accuracy compared with existing image editing methods. The problem of controlling image affect while preserving semantic content is relevant, and the reported improvements in affective accuracy are promising. However, the methodological novelty is limited, as the proposed framework primarily integrates established knowledge retrieval, multi-plan generation, and preference optimization techniques. The distinct technical contributions beyond these existing components are not sufficiently established.

Critical strengths
  • The paper addresses a relevant problem in affective image editing by explicitly exploring multiple editing strategies and considering differences across affective transitions.
  • The experiments cover three datasets and include comparisons with existing editing methods, alternative optimization strategies, ablation studies, and human evaluations. The results demonstrate consistent improvements in target-affect accuracy.
Critical weaknesses
  • The methodological novelty of KGME is limited. Its core procedure combines existing vision-language models, semantic clustering, heuristic knowledge retrieval, and prompt-based editing plan generation. Although this provides a structured exploration process, the individual components are largely established techniques, and their integration does not introduce a substantially new planning or learning mechanism.
  • The proposed framework has considerable conceptual overlap with prior work. In particular, EmoAgent has already explored planning, editing, and evaluation agents with emotion-factor knowledge retrieval and strategy exploration. The manuscript does not sufficiently clarify how its knowledge-grounded planning differs fundamentally from such existing approaches.
  • The methodological contribution of MCPO is relatively incremental. Its formulation relies on nearest-neighbor retrieval of similar affective transitions, statistical normalization of evaluation scores, covariance-based scaling, and preference distribution matching. These operations are closely related to established normalization and preference optimization techniques.
Questions for authors

MCPO mainly combines transition-based memory retrieval with statistical normalization and adaptive preference scaling. What makes this formulation fundamentally different from applying task-conditioned reward normalization to existing preference optimization methods, and which individual calibration components are essential to its effectiveness?

Reviewer_Epm3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies affective image editing: transforming the perceived affect of a source image toward a target valence–arousal (V–A) state while preserving its core semantic content. The proposed framework, AffectAgent, decomposes the task into a trainable planner, a fixed image-editing executor, and fixed evaluators.

The framework introduces two components. Knowledge-Grounded Multi-Plan Exploration (KGME) constructs a hierarchical element–attribute–V–A knowledge bank and retrieves editing cues according to source-image compatibility, target-affect relevance, and editing difficulty. Memory-Calibrated Preference Optimization (MCPO) retrieves historical tasks with similar affective transitions and uses their local outcome statistics to construct soft preference targets for planner optimization.

At inference, the optimized planner generates a single editing instruction without knowledge retrieval or episodic memory. Experiments on EmoSet, EMOTIC, and FindingEmo report improvements in valence and arousal accuracy, while a human study provides complementary evidence of better perceptual affective alignment.

Critical strengths
  1. Well-motivated decomposition of affective editing.

An abstract affective target can be realized through multiple visual modifications. Separating strategy selection from image generation makes this ambiguity explicit and provides an inspectable intermediate representation. This is a meaningful design choice for improving the interpretability and controllability of affective editing.

  1. Knowledge retrieval accounts for edit feasibility.

KGME incorporates source-image compatibility and editing difficulty rather than relying solely on semantic retrieval. The component ablations support these design choices. For example, removing compatibility filtering reduces CLIP-I from 0.7883 to 0.7562 and increases LPIPS from 0.6522 to 0.7029 on EmoSet, alongside higher affective errors.

  1. Consistent improvements in affective accuracy across datasets.

The reported gains are directionally consistent across three datasets. On EmoSet, planner optimization reduces V-MAE from 1.8786 to 1.4603 and A-MAE from 1.7718 to 1.3910. These results support the empirical usefulness of the overall approach, although the training protocol requires clarification before the results can be interpreted as zero-shot cross-dataset transfer.

  1. Evaluation extends beyond the training reward models.

The authors distinguish their training-time evaluators from the models used for final evaluation and supplement automatic metrics with a blind, randomized human study involving 60 tasks and 30 participants. The lower human-rated V-MAE and A-MAE strengthen the evidence that the improvements are perceptually meaningful rather than merely specific to a training evaluator.

  1. Limited additional inference overhead.

Multi-plan exploration and memory calibration are confined to training. Inference requires only one planning pass followed by one editing pass, avoiding repeated candidate generation and evaluation at deployment. This makes the framework practically appealing, although the full training and annotation costs remain insufficiently documented.

Critical weaknesses
  1. The incremental benefit of covariance normalization is unclear.

MCPO combines metric-wise standardization with covariance-based normalization, but the ablations in Section 4.3 do not isolate the contribution of the latter. Comparing the full method with a variant using only metric-wise standardization would clarify whether covariance normalization provides additional gains.

  1. Comparisons with relevant affective editing methods are limited.

EmoAgent and Moodifier are cited in the introduction but are not included in Table 1, and Section 2 offers little discussion of their methodological differences from AffectAgent. A focused comparison of the planning, knowledge-grounding, and optimization mechanisms, together with experiments under comparable task settings, would help establish the distinct contribution of KGME and MCPO.

  1. Candidate-generation differences confound the GRPO comparison.

Section 4.2 uses KGME-generated candidates for SFT, DPO, and MCPO, whereas GRPO samples candidates from the current policy. The comparison therefore changes candidate generation and optimization simultaneously, making it difficult to attribute the performance gap specifically to memory calibration. A KGME-assisted GRPO baseline with comparable knowledge access and matched generation/evaluation budgets would provide a more controlled test.

  1. The claimed affect–preservation trade-off is not fully established.

On EmoSet, AffectAgent improves affective accuracy but performs worse than EmoEditor on CLIP-I (0.7883 vs. 0.8513) and LPIPS (0.6522 vs. 0.1776). Its higher Overall score is obtained with a fixed affective weight of 0.7 (Appendix B.4). The claim of a better overall balance requires weight-sensitivity or matched-preservation comparisons, alongside task-level paired confidence intervals for the human-study preservation and quality differences.

Questions for authors
  1. Evidence for MCPO’s contribution: What evidence isolates the benefit of covariance normalization beyond metric-wise standardization? Are results available for a covariance-free variant, and how does it compare with the full method?

  2. Clarification of novelty: What are the substantive methodological differences between AffectAgent and EmoAgent/Moodifier, particularly regarding knowledge-grounded planning and planner optimization? How do these differences support the claimed contribution beyond existing affective editing frameworks?

  3. Clarification of the comparison protocol: Do GRPO and MCPO receive comparable access to retrieved knowledge, and are their image-generation, evaluation, and tuning budgets matched? What evidence separates the effect of memory calibration from the difference in candidate-generation procedures?

  4. Evidence for the claimed trade-off: Does AffectAgent’s advantage persist under alternative Overall weights or matched preservation constraints? Are the human-study differences in semantic preservation and image quality statistically distinguishable in a task-level paired analysis?

Reviewer_1skXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on affective image editing, a specialized image editing task that aims at changing the emotions an image evokes in viewers to specific targets. This work targets two dimensions: valence and arousal scores (V/A scores). The proposed editing framework, AffectAgent, takes the before/target V/A scores and the input image, and the framework works during training by producing editing plans, sending the editing plans to an executor (an image editor), and using reward models to evaluate. At inference, the trained planner produces a single plan without retrieval, and the editor executes it. To acquire a good editing plan, the paper builds an emotional knowledge bank, which annotates a series of elements/attributes and how they would contribute to the V/A scores. Thus, during training, the planner can retrieve the relevant elements to formulate good plans. The paper uses MCPO, a variant of preference optimization, to use historical task outcomes to calibrate preference supervision and improve the planner performance. The paper conducts experiments on various benchmarks and shows competitive performance of AffectAgent.

Critical strengths
  1. The proposed EKB is a large-scale knowledge bank and the paper proposes a detailed and reasonable way to prepare the affective editing attributes bank. It could serve as a good source for proposing elements for specific affections in the images.
  2. The training process enables the planner to effectively learn from historical tasks to improve its performance.
Critical weaknesses
  1. Task definition: In this paper, the affective editing goal is to have an image that changes the source valence / arousal score of the images. These seem have two problems: (1) Too narrow: It limits the changes of the images to these two dimensions. On the contrary, the prior work like EmoAgent uses different phrases to describe the target changing, which seems more various. (2) Indirect: the numeric scores seem to an indirect target. This essentially requires the planner model to understand continuous score meanings. Moreover, the planner outputs a plan, which is not continuous changing with respect to the scores. I doubt if we can see a continuous changes for subtle score difference like 6 -> 6.2 -> 6.4 -> … 7. Figure 4 demonstrates a continuous change in a large scale, which is good, but this actually support we can use wordings like largely/lightly rather than using numerical values. While these are used in early works as referenced in the paper, these paper are pre-LLM works and I am not sure if using free-form text is a better solution for today’s model. Essentially I am wondering if there could be more discussion regarding the trade-off of this selection and why this work sticks with valence / arousal scores.
  2. Related work - there is one related work in 2025, EmoAgent [1], seems very similar to the idea of this proposed work. Both work propose to use an agent to perform affective image editing. How is the work different from EmoAgent is not discussed, and this is not compared.
  3. Missing some important details: for example, what is the image editor model used in this framework? I think the Qwen-Image and Flux are the baseline editing models and I am not sure which model is used in this work. Also, the paper should consider ablate the effects of using different image editors - since strong image editors may directly be able to perform affective image editing without too complicated plans.
  4. When you creating the EKB, it seems finally you require the human to annotate the effects of textual attributes rather than visual elements. Why is this the case? For example, a textual attribute like “golden morning” could correspond to very different visual elements with different valence and arousal scores?

[1] EmoAgent: A Multi-Agent Framework for Diverse Affective Image Manipulation

Questions for authors
  1. For weakness 1 - I am wondering if this two-dimension is really narrow, and I am wondering if the current model can really understand subtle difference of numeric scores.
  2. Please discuss how relevant EmoAgent is, and if applicable, compared with this related work.
  3. I am wondering if the details of the image editor is missing or if I misread the text.
  4. Please discuss the choice of annotation and why annotating on text rather than visual attributes.
Reviewer_pZEzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AffectAgent, a planner–executor–evaluator framework for affective image editing. A VLM planner writes editing instructions, a frozen editor executes them, and two in-house evaluators score the results to build preference data for training the planner. The paper has two components. KGME retrieves cues from an Emotional Knowledge Bank to construct multiple candidate plans. MCPO calibrates preference strength using historical tasks with similar affective transitions. Experiments on three datasets show lower V–A errors and higher Overall scores than the baselines.

Critical strengths
  1. Making the edit plan an explicit step is a reasonable design, and it makes affective editing more interpretable and controllable.
  2. The evaluation includes a user study and cross-dataset experiments.
Critical weaknesses
  1. The preference data comes entirely from two in-house evaluators. The training-time affect evaluator and the test-time V–A estimator are trained on similar affective data, so the planner may learn to exploit their shared biases rather than change perceived affect. This casts doubt on generalization and reproducibility.
  2. In Table 1, the method is substantially worse on CLIP-I and LPIPS. The gap shrinks in the Overall score only because the authors' own weighting (Eq. 8) favors affective accuracy.
Questions for authors

Which model does your executor use?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.