acceptodds
Under review as a conference paper at ICLR 2027

Editor's Choice: Evaluating Abstract Intent in Image Editing via Atomic Entity Analysis

Abstract

Humans naturally communicate through abstract concepts like “mood”. However, current image editing benchmarks focus primarily on explicit, literal commands, leaving abstract instructions largely underexplored. In this work, we first formalize the definition and taxonomy of abstract image editing. To measure instruction-following in this challenging domain, we introduce **Entity-Rubrics**, a framework that breaks down abstract edits into individual, entity-level assessments and achieves strong correlation with human judgment. Alongside this framework, we contribute **AbstractEdit**, the first benchmark dedicated to abstract image editing across diverse real-world scenes. Evaluating 11 leading models on this dataset reveals a fundamental challenge: standard architectures struggle to balance intent and preservation, commonly defaulting to under-editing or over-editing. Our analysis demonstrates that driving meaningful improvements relies heavily on integrating advanced LLM text encoders and iterative thinking. Looking forward, our entity-based paradigm can generalize beyond assessment to serve as a reward model, enable models to correctly interpret abstract communication, or highlight specific failures in test-time critique loops. Ultimately, we hope this work serves as a stepping stone toward seamless multimodal interaction, closing the gap between rigid machine execution and the natural, open-ended way humans communicate.

Primary area:
datasets and benchmarks
Keywords:
image editing, evaluation, instruction-following

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_aEnROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an important and underexplored problem in image editing: understanding and evaluating abstract user intent. I find the task formulation clear and meaningful, and the proposed dataset represents a potentially valuable contribution to the community. The entity-level evaluation framework also provides a promising approach to diagnosing model behavior beyond conventional image-level metrics. These contributions make the work relevant to the ICLR community.

I lean toward Weak Accept because of the importance of the problem, the clarity of the presentation, and the potential value of the benchmark as an initial step toward systematic evaluation of abstract-intent image editing. Nevertheless, I have significant concerns about the validity of the abstract–explicit instruction pairs and the general reliability of the proposed evaluator. These are not minor issues: if the paired instructions do not faithfully represent the intended abstract edits, or if the evaluator does not reliably measure editing success, they could substantially weaken the paper's central claims and the conclusions drawn from the benchmark. The limited test-set size also makes stronger evidence of statistical robustness desirable.

I therefore strongly encourage the authors to address these concerns through more systematic validation of instruction-pair consistency, broader evaluation against human judgments, and appropriate statistical robustness analyses. My tentative positive assessment reflects the potential significance of the task and dataset, but the strength of the paper's claims should be qualified by the extent to which these validation issues can be resolved.

Critical strengths
  1. Timely and meaningful research problem. The paper studies abstract-intent image editing, an important yet underexplored setting in which users specify a desired change without fully specifying the target entity or the exact operation. The distinction between identification (what to edit) and specificity (how to edit) provides a clear and useful framework for characterizing abstract editing instructions.

  2. Clear problem formulation and presentation. The paper is generally well organized, and the task setup is easy to understand. The distinction between abstract instructions and their more explicit counterparts provides an intuitive way to study how current image-editing models handle underspecified user intent. The entity-level analysis also offers a more fine-grained perspective than evaluating editing outcomes solely with holistic image-level metrics.

  3. Dataset construction is a valuable contribution. Collecting and organizing examples of abstract editing instructions is challenging, given the ambiguity and context dependence inherent in such instructions. The proposed benchmark offers a useful starting point for evaluating model behavior in this setting and may facilitate future research on reasoning-driven and ambiguity-aware image editing.

Critical weaknesses
  1. The dataset is relatively small, limiting the robustness of the empirical conclusions. The test set contains 470 samples associated with 257 unique context images. While this may be sufficient for introducing an initial benchmark, it is relatively limited for drawing broad conclusions across different domains and model families. Moreover, multiple samples may share the same context image, meaning that the effective number of independent visual contexts is smaller than the number of test examples. The paper would benefit from uncertainty estimates and statistical analyses that account for this dependence, such as paired comparisons and bootstrap confidence intervals grouped by context image. A more detailed analysis of performance across categories and domains would also help establish whether the reported trends are robust.

  2. The validity of the abstract–explicit instruction pairs is not sufficiently established. This is my main concern because the quality of these pairs is central to the benchmark's validity. The paper constructs abstract instructions and corresponding explicit instructions, with the latter serving as concrete representatives of the intended edits. However, an abstract instruction can admit multiple reasonable interpretations, and a single explicit instruction may not faithfully represent the full range of valid interpretations. Consequently, a performance difference between abstract and explicit instructions may reflect not only abstraction, but also differences in instruction length, the number of specified operations, or editing difficulty. The paper reports a manual quality audit of the training data, which is helpful, but the current validation does not fully establish the semantic consistency and contextual appropriateness of the paired instructions. I would like to see a more systematic, preferably independent human evaluation of whether each explicit instruction is a valid realization of its paired abstract instruction, along with clearer annotation criteria and inter-annotator agreement. Controlled comparisons that better disentangle abstraction from prompt complexity would further strengthen the conclusions.

  3. The current evaluation may not capture all aspects of abstract-intent fulfillment. Entity-level analysis is interpretable and useful, but successful execution of an abstract instruction can depend on relationships among entities, interactions between objects, and changes to sub-entity attributes. The paper itself acknowledges limitations in handling cross-entity relations and sub-entity components. As a result, the proposed framework may miss cases in which individual entities appear plausible but the overall edit does not satisfy the user's intent. It would be useful to discuss more explicitly how these limitations affect the benchmark's conclusions and, if feasible, to include examples or analyses illustrating such failure cases.

Questions for authors

Please refer to "Weaknesses" above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aEnR - Part 1 of 3

We thank the reviewer for their time, effort, and positive feedback. We are especially glad you found the research problem timely, the presentation clear, and the dataset construction valuable.

Regarding the Robustness of the Empirical Conclusions (Q1):

We appreciate this important insight. To properly account for instructions sharing the same context image, we re-analyzed the judge scores for all 14 editors by treating each context image (rather than each instruction) as an independent unit. We now report all results with 95% bootstrap confidence intervals grouped by context image, alongside Holm-corrected paired comparisons (91 pairs) between editors. Specifically, we resampled the 257 images with replacement (including their associated instructions and scores) 2,000 times.

We have expanded our analysis to demonstrate the following:

1. Shared context images have minimal effect on variance and ranking. A mixed-effects model analyzing score variance attributes only ~1% of the variance to the context image; the vast majority stems from the editor and the instruction itself. Consequently, image-grouped confidence intervals are only ~2.5% wider than standard ones. Furthermore, editor ranking is robust to image reuse: assigning equal weight to every image yields the exact same ranking. Keeping only a single random instruction per image (repeated 1,000 times) reproduces the ranking almost perfectly (median Kendall's =0.98). Finally, because no image is reused within the same category, repeated images always evaluate distinct skills.

2. The benchmark successfully separates editors and model families. With image-grouped intervals, each editor's average score is precise to within 0.1–0.3 points (on a 1–10 scale). We compared all 91 pairs of the 14 editors using average per-instruction score differences to control for instruction difficulty. Even after applying the strict Holm correction for multiple testing, 79 of the 91 pairs differ significantly. The remaining ties only occur between highly similar models (e.g., GPT-Image-2 and Gemini 3.1 Flash). At the family level, the conclusion is clear-cut: every closed-source editor significantly outperforms every open-source editor (48 of 48 pairs, with a minimum gap of 1.1 points).

3. Trends hold robustly across categories and domains. Editor rankings remain highly consistent across the 12 categories and 4 domains (Kendall's W=0.88 for categories, 0.93 for domains, where 1 indicates identical rankings). Variations between categories primarily manifest in the size of performance gaps, which reveals informative, model-specific weaknesses. For example, Physical-Size is universally challenging, and both Step1X variants struggle with POV/Composition instructions. Because individual categories contain fewer instructions, their confidence intervals are naturally wider (0.6–1.4 points); therefore, we restrict category-level conclusions to clearly separated groups of editors.

We will add the full per-category and per-domain results, complete with confidence intervals, to the final version of the paper.

AuthorsOpenReview ↗

Response to Reviewer aEnR - Part 2 of 3

Regarding the Validity of the Abstract–Explicit Pairs (Q2):

We thank the reviewer for highlighting the importance of the abstract–explicit pairs. To address this, we conducted rigorous independent human validation and controlled analyses to disentangle abstraction from prompt complexity and to validate rigorously explicit instructions.

Independent Validation Criteria and Results: We conducted a new, exhaustive annotation of 75 pairs (covering all categories) by an external independent annotator and an author. To establish validity, annotators evaluated the pairs against strict annotation criteria. First, they labeled every individual operation within the explicit instruction as necessary, acceptable, or unnecessary. Second, they assessed the pair across four criteria: semantic completeness (checking for critical gaps), faithfulness (preserving the original meaning), contextual appropriateness to the image (on a 1–5 scale), and interpretation naturalness (reasonable alternative). A pair was classified as valid only if it strictly met all success criteria: no critical semantic gaps, perfect faithfulness, and a contextual appropriateness score of ≥4/5\ge 4/5.

Under these criteria, the external annotator judged 90.7% of pairs as valid (98.7% had no critical gap, 96.0% were faithful, and average appropriateness was 4.77/5). For open-ended social and emotional prompts, annotators occasionally noted extra elements the explicit instruction could include, but these were almost exclusively classified as acceptable minor gaps. The author demonstrated strong inter-annotator agreement (e.g., Gwet's AC1 = 0.97 for critical gaps and 0.90 for faithfulness). Crucially, restricting our analysis to only these criteria-validated pairs does not change the sign or magnitude of any editor's abstract–explicit performance gap (e.g., Bagel shifts slightly from +2.07 to +2.26, and GPT-Image from −0.88 to −0.72).

Operation Count Analysis: We will start by being clear that abstraction and length are inherently confounded in our benchmark, as an abstract instruction is concise by nature, whereas making it explicit requires adding concrete details. Because this confound cannot be removed by construction, we address it statistically to determine whether the performance gap is driven by abstraction or mere prompt length. To definitively isolate abstraction from prompt length and operation count (expanding on Appendix B), we analyzed all the samples across 6 editors. We counted the explicit operations required for each explicit instruction (mean 6.6, range 1–16) and used fixed-effects models, controlling for editor and category, to estimate score changes per additional operation.

We found that the number of explicit operations does not predict the abstract score (p=.78) or the abstract–explicit gap (p=.19). Instead, higher operation counts slightly lower the explicit score (p=.049), meaning length works against the explicit condition, not for it. This drop is driven almost entirely by Step1X-Think (=-0.28 per operation); for all other editors, the effect is negligible (||0.08). Splitting items into operation-count tertiles (5, 6–8, 9) confirms this. The performance gap remains stable across tertiles, proving that models with large gaps maintain their behavior even on the shortest explicit prompts.

Additionally, operations marked as "unnecessary" were primarily global "look" adjustments added to the requested change (e.g., "make this look like a lavish gourmet dish" →\rightarrow "desaturate by 15%"). While these adjustments account for 53–59% of all unnecessary operations (and 25% of operations overall), the performance gap remains completely unrelated to their frequency (p=.51p = .51). Finally, to control for raw editing difficulty, we isolated instances where an editor executed the explicit instruction nearly perfectly (score ≥9/10\ge 9/10). On these demonstrably executable edits, open-source editors still fail the abstract version in 19–56% of cases, compared to only 3% for proprietary editors.

Together, these controlled comparisons confirm that the performance gap is driven by abstract reasoning, not by prompt length, operation count, or editing difficulty. We will detail this validation study and all evaluation criteria in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer aEnR - Part 3 of 3

Regarding Cross-entity Relations and Sub-entity Components: (Q3)

We thank the reviewer for this insightful feedback. Both points address important design choices, which we initially discussed in the appendix: why pairwise relations are not explicit evaluation units (Appendix A.2) and how deeply to decompose entities (Appendix C).

Cross-Entity Relations:

We agree that cases where "individual entities appear plausible but the overall edit does not satisfy the user's intent" represent a valid limitation. Such cases are challenging to analyze systematically and primarily surface in the Social and Emotional categories, where intent relies on subjective notions like a "vibe" or social dynamic. Our user study (3 annotators × 360 edits) confirms this: annotator agreement is notably lower for these categories compared to others (Krippendorff's =0.47 for Social and 0.50 for Emotional, vs. 0.56 overall).

We will add qualitative examples of these failure cases to the appendix and explicitly discuss this limitation in the main text. For instance, given the instruction "Infuse the scene with empathy and a deep bond with animals" applied to a roping photo (FLUX.1 [dev]), the model removes the lasso and makes the roper smile. However, the calf is still fleeing, and there is no gesture of care between the two; the scene still reads as a chase. Our judge, seeing the individual entity changes satisfied, scores it 9/10, while human annotators give a 6/10. Listing only these instruction-implied relations as additional evaluation units, rather than all O(n2)O(n^2) pairs, is the tractable extension we propose in Appendix C, and we will expand and explicitly discuss it in the main paper's final version.

Sub-Entity Components and Granularity:

We agree that determining the correct level of granularity requires careful reasoning. EntityRubrics dynamically handles this in its evaluation pipeline, as a fixed hierarchy would limit the scope of evaluable instructions. In practice, this dynamic granularity works well: the judge splits entities into sub-parts in 43% of evaluated edits. This occurs most frequently where parts are critical (Pose 55%, Social-Role 54%, Emotional 50%) and least in POV/Composition (25%). Furthermore, sub-parts are penalized more strictly than whole entities (labeled as failing 17.8% vs. 12.9% of the time).

For example, given "Make her look absolutely excited" (Bagel), an edit adds a wide smile but leaves the eyes static, resulting in an awkward expression. The judge successfully captures this by splitting the face into woman_mouth (scored 10/10) and woman_eyes (scored 5/10), resulting in an overall 7/10 because "the eyes and eyebrows remain static, which makes the expression feel pasted on."

However, we acknowledge that part decomposition can struggle when required components are absent from the original context image. For instance, instructed to "Make him look like extremely wealthy royalty" (Bagel-Think), an edit might add a crown but leave the man's casual clothing unchanged. While the judge catches this specific failure (man_clothing scored 4/10), it is less proactive at anticipating entirely new parts.

Benchmark-wide, 44% of edits insert a new entity, and 5% insert a new accessory. The judge's expectation step successfully anticipates 99% of changes to existing entities, but only 48% of newly inserted entities. When an anticipated insertion is missing, the judge reliably flags it (91% failure rate, e.g., "The requested accessibility ramp was not added"). Unanticipated insertions (like a scepter or throne) are discovered post-edit and correctly credited as creative enhancements (93% labeled correct), but they are not strictly required by the initial rubric.

Thank the authors for their detailed rebuttal and additional clarifications. These additions have addressed all of my previous concerns. In particular, the further analyses of empirical robustness, together with the independent human validation and controlled experiments concerning the validity of abstract–explicit instruction pairs, substantially strengthen the credibility of the empirical results and provide compelling support for the paper's central conclusions regarding differences in abstract reasoning capabilities. Furthermore, the expanded discussion of limitations concerning cross-entity relations and sub-entity components makes the paper more complete. I am satisfied with the authors' responses and therefore improve my rating to 4.

Reviewer_EBf8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper formalizes abstract image editing through identification and specificity axes, introduces the Entity-Rubrics VLM-based evaluator, and contributes AbstractEdit, containing 470 test tasks over 257 context images together with a larger training set. Entity-Rubrics decomposes an edit into entities, predicts which should change or be preserved, evaluates execution, and produces a final score. The paper benchmarks open- and closed-source image editors and validates the metric on a 60-prompt human study.

Critical strengths
  1. Pairing abstract prompts with explicit realizations for the same context image is a useful benchmark design. It helps separate difficulty in interpreting high-level intent from difficulty executing a concrete edit.
  2. Entity-Rubrics provides more interpretable diagnostics than a single holistic VLM score. The human study includes 1,080 ratings, and the reported Spearman correlation of 0.66 exceeds VIE and CLIP. The two-stage evaluator ablation also reduces human correlation substantially, supporting the proposed decomposition.
Critical weaknesses
  1. The central evaluator is not independently validated enough to support the reported rankings. Entity granularity has no stable stopping rule, cross-entity relations are omitted, and the final 1–10 score remains a holistic VLM judgment. Per-entity agreement is only κ=0.41\kappa=0.41, image-quality agreement is κ=0.28\kappa=0.28, and the reported ρ=0.66\rho=0.66 has no confidence interval or within-model analysis. Gemini 3 Flash judges all systems while Gemini-family editors rank highest. Critique and training improvements are then evaluated using the same metric that supplied their feedback or objective, creating circularity without independent human confirmation.
  2. The benchmark's abstractness and generalization are not established independently. Test prompts are generated by Gemini 2.5 Pro from fixed categories, personas, and few-shot examples, while "human verified" appears to mean author review rather than independent annotation. The training audit already identifies substantial non-abstract or weakly grounded examples. The test set contains only 257 unique context images, and no near-duplicate, contamination, or human-authored benchmark comparison is reported. Claims about text encoders and iterative reasoning are also confounded by comparisons across different proprietary model families.
Questions for authors
  1. Do the complete model rankings survive independent human pairwise evaluation with prompt-clustered confidence intervals?
  2. Do results replicate with multiple non-Gemini judges? In particular, are Gemini-family models still ranked highest under independent open and closed-source evaluators?
  3. Can the authors provide independently annotated entity lists and relation-aware rubrics to measure entity detection, expected-transformation classification, and execution accuracy separately?
  4. On a fully human-authored test set, do annotators agree that prompts satisfy the proposed abstractness definition, and do rankings persist after image deduplication and contamination checks?
  5. Were the metric prompts or aggregation rules tuned using the same 60 samples used for human correlation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer EBf8 - Part 1 of 4

We thank the reviewer for the thorough reading and constructive feedback. We are glad you appreciated our paired-benchmark design, the interpretability of Entity-Rubrics, and the ablation support.

Regarding Independent Human Pairwise Evaluation with Prompt-Clustered CIs (Q1):

To assess whether complete model rankings hold under independent human evaluation, our user study evaluated six representative editors across the full leaderboard on the same 60 prompts (1,080 total judgments). Because all models share identical prompts, we constructed direct pairwise comparisons for all 62=15 editor pairs. All reported 95% confidence intervals use a cluster bootstrap resampling prompts (2,000 iterations). As shown below, Entity-Rubrics perfectly predicts human preferences on all statistically separable pairs, and correctly behaves as a tie-breaker or mirrors the tie on indistinguishable pairs:

Editor Pair Category (15 Total)Human Evaluation (Prompt-Clustered CI)Entity-Rubrics Evaluation
12 Separable PairsStatistically separated (95% CI excludes 0)Matches exact human ordering (12/12 pairs)
Tie 1: Gemini 3.1 vs. GPT-ImageStatistical tie (=-0.01 [-0.16,+0.14])Both ranked at the very top (statistical tie)
Tie 2: Step1X vs. FLUX.2Statistical tie (=-0.27 [-0.60,+0.07])Statistical tie on this subset (=+0.02 [-1.03,+1.05])

Note: On the full benchmark, the metric correctly ranks FLUX.2 above Step1X, matching the human point-estimate direction.

Global Agreement Statistics: The table below summarizes the high overall correlation. Metrics are unweighted (reweighting to the true category distribution yields identical conclusions, e.g., reweighted =0.73).

MetricAgreement Score [95% CI]
Editor Ranking Agreement (Kendall τ\tau)0.73 [0.60, 1.00]
Per-Prompt Win Agreement (Same winner as humans)85% [82%,88%82\%, 88\%]
Item-Level Correlation (Spearman ρ\rho)0.66 [0.59, 0.71]

These results confirm that the benchmark ranking robustly reflects human consensus under rigorous statistical bounds.

Regarding Results Replication with Non-Gemini Judges and Same-Family Bias (Q2):

Yes, the results fully replicate, and Gemini-family models remain the highest-ranked under both independent open-source and closed-source evaluators. To address this, we conducted a comprehensive judge-replacement study:

1. Evaluation with an Independent Open-Source Judge We re-scored all six human-annotated editors on the full benchmark with GLM-4.1V-9B-Thinking, an open-source reasoning VLM completely independent of any evaluated editor. We used identical prompts and the two-call Entity-Rubrics protocol, swapping only the underlying model. Mirroring the Gemini judge, the GLM judge places Gemini 3.1 Flash first and GPT-Image second, with FLUX.1-Kontext and Bagel last. The only change is that FLUX.2 and Step1X-Think-Reflect swap places 3 and 4, demonstrating strong overall rank agreement (Kendall =0.87). Furthermore, we quantified how much more the Gemini judge favors the Gemini editor compared to the independent GLM judge, relative to GPT-Image (the other closed-source editor in the same quality tier). The difference is -0.10 points on the 1–10 scale (95% CI [-0.36,+0.15]), which is statistically indistinguishable from zero with a tight bound, confirming no measurable same-family effect.

2. Evaluation with an Additional Proprietary Judge To evaluate with an additional proprietary judge, we tested a category-proportional sample of 300 instances using Claude Sonnet 5 with medium-level reasoning due to budget constraints. Claude produces an editor ranking identical to Gemini's, with an item-level Spearman correlation of 0.77. In terms of same-family bias, the difference relative to Claude is -0.15 points (95% CI [-0.36,+0.06]), further demonstrating that the Gemini judge does not artificially favor the Gemini editor. Finally, as reported below on the shared user-study subset, Claude Sonnet 5 demonstrates high agreement with human raters (87% win agreement, Kendall =0.87), verifying that our evaluation findings are robust across distinct frontier judges.

MetricGemini-3-Flash (paper)GLM-4.1V-ThinkingClaude Sonnet 5
Item-level Spearman with human instruction-following0.66 [0.59, 0.71]0.46 [0.36, 0.54]0.68 [0.60, 0.74]
Same head-to-head winner as humans85% [82, 88]79% [73, 85]87% [84, 90]
Kendall , editor ranking vs. human ranking0.730.730.87
AuthorsOpenReview ↗

Response to Reviewer EBf8 - Part 2 of 4

Regarding Independent Annotations for Entity Detection and Classification Stages (Q3):

Yes, we can provide this exact breakdown. Our user study was specifically designed to allow us to measure entity detection, expected transformation, and execution accuracy as separate stages. Our user study collected independent, human entity-level annotations across all 360 study edits (60 items × 6 editors). Three MTurk annotators evaluated every entity proposed by the judge (Correct Change / Failed Change / Correctly Preserved / Failed Preservation / Unsure) and could freely introduce entities the judge missed (Fig. 15, Appendix D). This allows us to evaluate each stage of the judge against humans across 2,551 entities, benchmarked against a held-out human annotator evaluated under a leave-one-out agreement protocol.

StageHeld-out HumanVLM JudgeΔ (VLM − Human)
Entity detection (entities found evaluable by humans)—99.6%—
Expected transformation (change vs. preserve)85.6%82.1%−3.5% [−5.9, −1.3]
Execution evaluation (correct vs. failed)85.3%86.5%+1.2% [−1.4, +3.8]

Coverage and Error Profiles. Annotators added an average of 1.3 entities per edit, but rarely the same ones (only 0.03 entities per edit were added by 2 annotators), demonstrating that the judge's detected entity set comprehensively captures what humans consistently deem salient. Furthermore, the judge is significantly less prone to unwarranted penalties than human annotators: false-negative errors—such as labeling a should-change entity as "preserve" (penalizing a valid edit)—occur in only 5.6% of judge calls compared to 12.7% for humans. Similarly, marking a correct change as failed occurs in only 1.6% of judge calls versus 5.9% for humans.

Leniency Bias and Impact on Benchmark Conclusions. Consistent with known properties of VLM evaluators, the judge exhibits a mild leniency bias rather than harshness, occasionally overlooking subtle failures: it accepts 41% of changes that humans flag as failed (vs. 10% for humans) and 29% of preservation violations (vs. 19% for humans). In the 26% of edits containing a lenient label, this introduces a slight upward score shift (judge−human gap of +0.06 vs. −0.10 on a 0–1 scale), lowering local per-edit correlation (Spearman 0.53 vs. 0.69). Crucially, this effect is uniform across editors: filtering out all edits with lenient labels leaves the global editor ranking completely unchanged (Kendall =0.73 against human ranking in both cases, with identical editor ordering). Thus, while minor leniency introduces localized per-edit noise, it does not alter any of the paper's comparative findings.

We will incorporate this granular breakdown in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer EBf8 - Part 3 of 4

Regarding Prompt Abstractness and Independent Human Verification (Q4):

During the initial dataset curation, all test prompts were manually verified by the authors to ensure they strictly satisfy the two axes of our abstractness definition.

As requested, to provide independent validation, we ran a new human annotation study with external raters. Five independent annotators were trained on our definition (Section 3) and rated a stratified sample of 103 test prompts (covering all 12 categories and including the 60 user-study items).

Given the rebuttal timeline, we prioritized covering every prompt with independent judgments over adding more annotators per prompt. 88 prompts were rated by two annotators and 15 by three, yielding 221 total judgments. Annotators evaluated each prompt-image pair along both the WHAT (Identification) and HOW (Specificity) axes, classifying each as Clear (explicit; one valid realization), Partly Abstract (a few edit options), or Abstract (many options). Per our definition, a prompt is abstract if either axis is open to multiple realizations. All rates below are reweighted to the true benchmark category distribution (unweighted values in parentheses):

Benchmark prompts (n = 103)
Judged abstract by every annotator81% (77%)
Judged abstract by at least one annotator97% (95%)
Inter-annotator agreement (abstract vs. not)84%

As presented, independent annotators strongly agree that the benchmark prompts satisfy our abstractness definition. The study also reveals that most abstraction occurs on the HOW axis (Specificity). 79% (75%) of prompts are abstract in HOW, compared to 24% (22%) in WHAT. Abstraction in the target (WHAT) is concentrated in categories requiring scene-level inference or unnamed objects (e.g., Season, Mood, Insertion-Goal). Conversely, categories like Pose and Size clearly name the target entity but leave the execution open.

Analysis of Disagreements: Disagreements were rare and highly systematic: Only 5 prompts were judged "Clear" by every annotator, and all 5 belonged to the Physical-Size category (e.g., "Make their boxing gloves comically oversized"). Because these fix the target and direction (leaving only magnitude open), annotators treated the magnitude variation as negligible, placing these at the boundary of our abstract definition (accounting for just 4.9% of the benchmark). In the 19 remaining split cases, raters agreed on the target (WHAT) but debated the execution (HOW), specifically, whether varying geometric attributes like "top-down" angles constituted "a few" or "many" realizations. We will include the full annotation study, guidelines, and per-category breakdown in the revised paper.

Regarding Image Deduplication and Context Grouping (Q4):

Yes, the complete model rankings persist perfectly after controlling for shared context images. To ensure that the 257 unique context images do not bias the results, we re-analyzed the existing judge scores for all 14 editors so that each context image, rather than each instruction, is treated as an independent unit.

  • Variance Analysis: A mixed-effects model (splitting score variation into editor, context image, instruction, and noise) reveals that only ~1% of the score variation is attributable to the context image. Most variance stems from the editors and the instructions themselves.
  • Robustness to Deduplication: The editor ranking is completely invariant to image reuse. Giving every unique image equal weight yields the exact same editor ranking. Furthermore, if we strictly deduplicate by keeping only a single random instruction per image (257 items) and repeat this sampling 1,000 times, it reproduces the full ranking almost perfectly (median Kendall’s =0.98).
  • Category Separation: Finally, we ensure there is no intra-category contamination: no image is ever reused within the same category, meaning any repeated image is strictly paired with a conceptually different abstract instruction.

We now report all results with 95% cluster-bootstrap confidence intervals grouped by context image, and we will include this deduplication analysis in the final version to confirm the robustness of the rankings.

Reviewer_amnYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ENTITY-RUBRICS, a VLM-based automatic evaluation framework for instruction following in image editing. The framework treats image entities as atomic units. Using only the context image and the instruction, it first judges each entity's expected transformation state, one of three: expected change, optional change, or expected preservation. It then compares the edited image against those expectations to determine how each was executed, and finally the VLM synthesizes a score from 1 to 10. The paper also releases the ABSTRACTEDIT benchmark, comprising 470 human-verified test samples and a 4k training set.

Critical strengths
  1. The paper gives a clear definition and taxonomy of abstract image editing. It breaks down how abstract an instruction is into two questions: what to change, i.e., identification, and how to change it, i.e., specificity. If a majority of people fail to agree on either question, the instruction counts as abstract. Abstractness also depends on the image-text pairing: the same instruction applied to a different context image may lead to a different conclusion.
  2. The paper proposes ENTITY-RUBRICS, which decomposes abstract edits into per-entity assessments, treating a single image entity as the atomic unit of evaluation, including things, stuff, and global features.
  3. The paper contributes ABSTRACTEDIT, the first benchmark dedicated to abstract image editing. The benchmark contains 470 human-verified test samples and a 4k training set, spans four domains (physical, logical, emotional, social), and offers a basis for experiments in subsequent work.
  4. The paper's analysis identifies a path for improving abstract editing. The authors argue that meaningful improvement relies heavily on incorporating advanced LLM text encoders and iterative thinking, indicating a direction for follow-up experiments.
Critical weaknesses
  1. The judge belongs to the same family as some of the evaluated models, and the paper neither discloses nor controls for this. The judge is Gemini 3 Flash, and among the evaluated models are Gemini image models, with the highest-scoring Gemini 3.1 Flash being the closest same-generation model to the judge. The paper acknowledges in §A.1 that VLM judges carry architectural biases, yet provides no experiment showing how large the same-family effect is.
  2. The randomness of the judge is not addressed. VLM generation is stochastic, and the paper only analyzes the generation seeds of the evaluated models to some extent, without addressing the judge. Both the entity labels and the score are produced directly by the VLM, and no repeated experiments were run to study the impact of this randomness. The regression in §H.2 only shows a stable correlation between entity labels and scores, but does not discuss whether the VLM produces the same entity labels each time.
  3. For some models, the resolution of the generated images differs from that of the original images, and the paper does not explain this. The image examples in the appendix visibly have aspect ratios different from the original images, and the paper does not explain why those resolutions were chosen rather than the originals, while in the main text the aspect ratios appear to match the originals. If the resolution differs from the original, does this affect the experimental results and lead to different evaluation outcomes, which would undermine confidence in the results?
  4. The framework relies entirely on the VLM's classification, and the paper neither validates the reliability of that classification nor discusses the consequences of misclassification. Entity recognition, the three expectation states, and the six execution states are all produced by VLM classification, and these labels are the direct source of the score. The human study examines the correlation between the final score and human judgment, but does not report the accuracy of the intermediate labels. Misclassification flips the verdict: if an entity that should change is labeled as should-be-preserved, a correct edit is recorded as a violation and penalized, while a missed edit is recorded as correct. The paper does not discuss this risk.
Questions for authors
  1. How large is the same-family effect? The judge Gemini 3 Flash is in the same family as the evaluated Gemini 3.1 Flash, Gemini 3 Pro, and Gemini 2.5 Flash, and Gemini 3.1 Flash is precisely the highest-scoring model.
  2. How is the judge's randomness controlled? Were repeated evaluations run and run-to-run variance reported?
  3. What resolution is used for the images during evaluation? When the aspect ratio differs from the original, does the judge produce different results?
  4. How large is the impact of VLM classification failure on the results? Should there be an experiment showing that classification failure does not materially affect the results?
  5. It is also suggested to add a set of re-evaluations with a different judge, using GPT, Claude, or DeepSeek, and reporting how the results change across judges.
2 comments
AuthorsOpenReview ↗

Response to Reviewer amnY - Part 1 of 2

We thank the reviewer for the time and effort dedicated to this review. We are encouraged by your positive feedback, and we are particularly glad that you found the taxonomy and definitions clear, and that you saw the novelty and analysis of the AbstractEdit benchmark as paving the way for future work in abstract image editing.

Regarding the Gemini Judge–Evaluated Model Family Similarity (Q1):

We conducted a judge-replacement study by re-scoring all six human-annotated editors on the full benchmark with GLM-4.1V-9B-Thinking, an open-source reasoning VLM completely independent of any evaluated editor. We used identical prompts and the two-call Entity-Rubrics protocol, swapping only the underlying model. Mirroring the Gemini judge, the GLM judge places Gemini 3.1 Flash first and GPT-Image second, with FLUX.1-Kontext and Bagel last. The only change is that FLUX.2 and Step1X-Think-Reflect swap places 3 and 4, demonstrating strong overall rank agreement (Kendall =0.87).

Furthermore, we quantified how much more the Gemini judge favors the Gemini editor compared to the independent GLM judge, relative to GPT-Image (the other closed-source editor in the same quality tier). The difference is -0.10 points on the 1–10 scale (95% CI [-0.36,+0.15]), which is statistically indistinguishable from zero with a tight bound, confirming no measurable same-family effect.

Regarding Additional Proprietary Judges (Q5):

To evaluate with an additional proprietary judge, we tested a category-proportional sample of 300 instances using Claude Sonnet 5 with medium-level reasoning due to budget constraints. Claude produces an editor ranking identical to Gemini's, with an item-level Spearman correlation of 0.77. In terms of same-family bias, the difference relative to Claude is -0.15 points (95% CI [-0.36,+0.06]), further demonstrating that the Gemini judge does not artificially favor the Gemini editor.

Finally, as reported below on the shared user-study subset, Claude Sonnet 5 demonstrates high agreement with human raters (87% win agreement, Kendall =0.87), verifying that our evaluation findings are robust across distinct frontier judges.

Gemini-3-Flash (paper)GLM-4.1V-ThinkingClaude Sonnet 5
Item-level Spearman with human instruction-following ratings0.66 [0.59, 0.71]0.46 [0.36, 0.54]0.68 [0.60, 0.74]
Same head-to-head winner as humans85% [82, 88]79% [73, 85]87% [84, 90]
Kendall τ, editor ranking vs human ranking0.730.730.87

Regarding VLM (Judge) Stochasticity (Q2):

To quantify judge stochasticity, we re-evaluated the six editors from our human study using the unchanged Entity-Rubrics pipeline under deterministic settings (Gemini-3-Flash, temperature 0, fixed random seed, identical prompts). Due to API budget constraints, we evaluated two targeted subsets: (i) the full pipeline run twice on 60 items (5 per category; 360 item-editor pairs); and (ii) fixing Stage 1 (entity extraction) while running Stage 2 (labeling and scoring) three times on 123 items (10 per category; 738 pairs).

  • Score Stability: Scores are highly reliable. The test–retest Intraclass Correlation Coefficient (ICC) is 0.89 for the full pipeline and 0.94 with entities fixed. Exactly 78.6% / 79.5% of scores are identical across runs, and 95.3% / 96.0% remain within 1 point. Judge noise accounts for only 10–11% of total score variance. By comparison, inter-annotator agreement among humans on the same items yields an ICC of 0.50.

  • Entity Labels: Run-to-run variation stems primarily from how scene elements are segmented or named (e.g., lockers vs. metal_lockers, or a group vs. individuals), yielding an entity-set Jaccard similarity of 0.78 after resolving surface wording differences. However, entities matched across runs receive near-identical annotations: expected change at 95–96% (=0.93), entity group at 99.5%+ (=0.996), and edit action, necessity, and change-occurrence at 88–95% (=0.85–0.88). Crucially, these naming differences do not propagate to the outcome: in 93.4% / 94.0% of pairs, final scores agree within 1 point even when the extracted entity sets differ.

  • Rankings: The resulting editor ranking is identical across all runs (Kendall =1.0, both unweighted and reweighted to the true benchmark category distribution). Run-to-run variation in editor means (SD=0.01–0.20) remains substantially smaller than the performance margins separating editors and the underlying item-sampling uncertainty.

We will add this repeated-judge study to the final version of the paper.

AuthorsOpenReview ↗

Response to Reviewer amnY - Part 2 of 2

Regarding Image Resolution and Aspect Ratio (Q3):

Resolution Policy. Each editor was run at its default, officially recommended output setting. The judge evaluates every edited image at its native output resolution without modification (no resizing, cropping, or padding), alongside the source image at its original resolution. Most editors preserve the input aspect ratio (AR) (Bagel, Step1X, Qwen-Image-Edit, SeedDream, and Nano-Banana match the AR on 99–100% of items). GPT-Image-1.5 dynamically selects one of three canvases per call (size=auto). FLUX.1 Kontext, FLUX.2 (1024×1024), and HiDream-E1 (768×768) use a fixed square canvas, resulting in an AR mismatch on 91% of the items (which are predominantly 4:3 or 3:2).

To evaluate whether AR differences bias the judge, we conducted two complementary analyses:

To isolate this effect, we normalized each score by subtracting the item's mean score across AR-preserving editors, removing item difficulty. For each editor, we defined the penalty as its mean score on items where the AR changed minus its mean score on items where it was preserved (<0 would indicate a penalty for AR changes). For editors that never alter the AR, baseline variance yields shifts between −0.47 and +0.59, representing expected random variation.

  1. Analysis of Existing Scores (all 470 benchmark samples). Comparing each editor against itself across AR-matched versus AR-mismatched items (with item difficulty controlled), we find no systematic penalty. Effect signs are mixed (e.g., FLUX.2: −0.47 [−1.32, +0.46]; FLUX.1 Kontext: +1.81 [+0.69, +2.78]), and larger AR distortion does not correlate with lower scores (||0.05, all p>0.3).

  2. Controlled Squashing Test (60 user-study items × 3 AR-preserving editors). We squashed the outputs of three AR-preserving editors to 1024×1024 (altering geometry while preserving identical content) and re-evaluated them. The shift was computed as the score of the squashed image minus the mean score of the original image across three independent runs (to isolate the AR effect from judge variance, as noted in Q2). The mean score shift is statistically negligible at −0.08 [−0.20, +0.06], with 91% of scores changing by at most 1 point. Crucially, the editor ranking remains identical (inter-editor gaps are 1.3 points vs. shifts 0.18).

We will include the full resolution analyses in the revised appendix.

Regarding VLM Classification and Error Modes (Q4):

Our user study collected independent, human entity-level annotations across all 360 study edits (60 items × 6 editors). Three MTurk annotators evaluated every entity proposed by the judge (Correct Change / Failed Change / Correctly Preserved / Failed Preservation / Unsure) and could freely introduce entities the judge missed (Fig. 15, Appendix D). This allows us to evaluate each stage of the judge against humans across 2,551 entities, benchmarked against a held-out human annotator evaluated under a leave-one-out agreement protocol.

StageHeld-out HumanVLM JudgeΔ (VLM − Human)
Entity detection (entities found evaluable by humans)—99.6%—
Expected transformation (change vs. preserve)85.6%82.1%−3.5% [−5.9, −1.3]
Execution evaluation (correct vs. failed)85.3%86.5%+1.2% [−1.4, +3.8]

Coverage and Error Profiles. Annotators added an average of 1.3 entities per edit, but rarely the same ones (only 0.03 entities per edit were added by 2 annotators), demonstrating that the judge's detected entity set comprehensively captures what humans consistently deem salient. Furthermore, the judge is significantly less prone to unwarranted penalties than human annotators: false-negative errors, such as labeling a should-change entity as "preserve" (penalizing a valid edit), occur in only 5.6% of judge calls compared to 12.7% for humans. Similarly, marking a correct change as failed occurs in only 1.6% of judge calls versus 5.9% for humans.

Leniency Bias and Impact on Benchmark Conclusions. Consistent with known properties of VLM evaluators, the judge exhibits a mild leniency bias rather than harshness, occasionally overlooking subtle failures: it accepts 41% of changes that humans flag as failed (vs. 10% for humans) and 29% of preservation violations (vs. 19% for humans). In the 26% of edits containing a lenient label, this introduces a slight upward score shift (judge−human gap of +0.06 vs. −0.10 on a 0–1 scale), lowering local per-edit correlation (Spearman 0.53 vs. 0.69). Crucially, this effect is uniform across editors: filtering out all edits with lenient labels leaves the global editor ranking completely unchanged (Kendall =0.73 against human ranking in both cases, with identical editor ordering). Thus, while minor leniency introduces localized per-edit noise, it does not alter any of the paper's comparative findings.

We will incorporate this granular breakdown in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Invitation for Further Discussion

Dear Reviewers and Area Chair,

Thank you for your valuable feedback. Special thanks to Reviewer aEnR for your active engagement and for raising your score to a 4; your feedback significantly strengthened our work.

We invite Reviewers amnY and EBf8 to review the detailed updates in our replies. We hope these resolve your concerns and welcome any further questions before the discussion period ends.

For your convenience, here is a summary of the further analyses and validations we have already provided in our individual responses:

  • Trends robustness, abstract-explicit human validation, and sub-entity and cross-entity relations (aEnR)
  • Gemini judge similarity, judge stochasticity, and additional judges (EBf8, amnY)
  • Prompt-abstractness human verification (EBf8), VLM classification and error modes (EBf8, amnY)
  • Image resolution (amnY), and context image grouping (EBf8, amnY)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.