- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies an important and underexplored problem in image editing: understanding and evaluating abstract user intent. I find the task formulation clear and meaningful, and the proposed dataset represents a potentially valuable contribution to the community. The entity-level evaluation framework also provides a promising approach to diagnosing model behavior beyond conventional image-level metrics. These contributions make the work relevant to the ICLR community.
I lean toward Weak Accept because of the importance of the problem, the clarity of the presentation, and the potential value of the benchmark as an initial step toward systematic evaluation of abstract-intent image editing. Nevertheless, I have significant concerns about the validity of the abstract–explicit instruction pairs and the general reliability of the proposed evaluator. These are not minor issues: if the paired instructions do not faithfully represent the intended abstract edits, or if the evaluator does not reliably measure editing success, they could substantially weaken the paper's central claims and the conclusions drawn from the benchmark. The limited test-set size also makes stronger evidence of statistical robustness desirable.
I therefore strongly encourage the authors to address these concerns through more systematic validation of instruction-pair consistency, broader evaluation against human judgments, and appropriate statistical robustness analyses. My tentative positive assessment reflects the potential significance of the task and dataset, but the strength of the paper's claims should be qualified by the extent to which these validation issues can be resolved.
Critical strengths
-
Timely and meaningful research problem. The paper studies abstract-intent image editing, an important yet underexplored setting in which users specify a desired change without fully specifying the target entity or the exact operation. The distinction between identification (what to edit) and specificity (how to edit) provides a clear and useful framework for characterizing abstract editing instructions.
-
Clear problem formulation and presentation. The paper is generally well organized, and the task setup is easy to understand. The distinction between abstract instructions and their more explicit counterparts provides an intuitive way to study how current image-editing models handle underspecified user intent. The entity-level analysis also offers a more fine-grained perspective than evaluating editing outcomes solely with holistic image-level metrics.
-
Dataset construction is a valuable contribution. Collecting and organizing examples of abstract editing instructions is challenging, given the ambiguity and context dependence inherent in such instructions. The proposed benchmark offers a useful starting point for evaluating model behavior in this setting and may facilitate future research on reasoning-driven and ambiguity-aware image editing.
Critical weaknesses
-
The dataset is relatively small, limiting the robustness of the empirical conclusions. The test set contains 470 samples associated with 257 unique context images. While this may be sufficient for introducing an initial benchmark, it is relatively limited for drawing broad conclusions across different domains and model families. Moreover, multiple samples may share the same context image, meaning that the effective number of independent visual contexts is smaller than the number of test examples. The paper would benefit from uncertainty estimates and statistical analyses that account for this dependence, such as paired comparisons and bootstrap confidence intervals grouped by context image. A more detailed analysis of performance across categories and domains would also help establish whether the reported trends are robust.
-
The validity of the abstract–explicit instruction pairs is not sufficiently established. This is my main concern because the quality of these pairs is central to the benchmark's validity. The paper constructs abstract instructions and corresponding explicit instructions, with the latter serving as concrete representatives of the intended edits. However, an abstract instruction can admit multiple reasonable interpretations, and a single explicit instruction may not faithfully represent the full range of valid interpretations. Consequently, a performance difference between abstract and explicit instructions may reflect not only abstraction, but also differences in instruction length, the number of specified operations, or editing difficulty. The paper reports a manual quality audit of the training data, which is helpful, but the current validation does not fully establish the semantic consistency and contextual appropriateness of the paired instructions. I would like to see a more systematic, preferably independent human evaluation of whether each explicit instruction is a valid realization of its paired abstract instruction, along with clearer annotation criteria and inter-annotator agreement. Controlled comparisons that better disentangle abstraction from prompt complexity would further strengthen the conclusions.
-
The current evaluation may not capture all aspects of abstract-intent fulfillment. Entity-level analysis is interpretable and useful, but successful execution of an abstract instruction can depend on relationships among entities, interactions between objects, and changes to sub-entity attributes. The paper itself acknowledges limitations in handling cross-entity relations and sub-entity components. As a result, the proposed framework may miss cases in which individual entities appear plausible but the overall edit does not satisfy the user's intent. It would be useful to discuss more explicitly how these limitations affect the benchmark's conclusions and, if feasible, to include examples or analyses illustrating such failure cases.
Questions for authors
Please refer to "Weaknesses" above.
4 comments
Response to Reviewer aEnR - Part 2 of 3
Regarding the Validity of the Abstract–Explicit Pairs (Q2):
We thank the reviewer for highlighting the importance of the abstract–explicit pairs. To address this, we conducted rigorous independent human validation and controlled analyses to disentangle abstraction from prompt complexity and to validate rigorously explicit instructions.
Independent Validation Criteria and Results: We conducted a new, exhaustive annotation of 75 pairs (covering all categories) by an external independent annotator and an author. To establish validity, annotators evaluated the pairs against strict annotation criteria. First, they labeled every individual operation within the explicit instruction as necessary, acceptable, or unnecessary. Second, they assessed the pair across four criteria: semantic completeness (checking for critical gaps), faithfulness (preserving the original meaning), contextual appropriateness to the image (on a 1–5 scale), and interpretation naturalness (reasonable alternative). A pair was classified as valid only if it strictly met all success criteria: no critical semantic gaps, perfect faithfulness, and a contextual appropriateness score of .
Under these criteria, the external annotator judged 90.7% of pairs as valid (98.7% had no critical gap, 96.0% were faithful, and average appropriateness was 4.77/5). For open-ended social and emotional prompts, annotators occasionally noted extra elements the explicit instruction could include, but these were almost exclusively classified as acceptable minor gaps. The author demonstrated strong inter-annotator agreement (e.g., Gwet's AC1 = 0.97 for critical gaps and 0.90 for faithfulness). Crucially, restricting our analysis to only these criteria-validated pairs does not change the sign or magnitude of any editor's abstract–explicit performance gap (e.g., Bagel shifts slightly from +2.07 to +2.26, and GPT-Image from −0.88 to −0.72).
Operation Count Analysis: We will start by being clear that abstraction and length are inherently confounded in our benchmark, as an abstract instruction is concise by nature, whereas making it explicit requires adding concrete details. Because this confound cannot be removed by construction, we address it statistically to determine whether the performance gap is driven by abstraction or mere prompt length. To definitively isolate abstraction from prompt length and operation count (expanding on Appendix B), we analyzed all the samples across 6 editors. We counted the explicit operations required for each explicit instruction (mean 6.6, range 1–16) and used fixed-effects models, controlling for editor and category, to estimate score changes per additional operation.
We found that the number of explicit operations does not predict the abstract score (p=.78) or the abstract–explicit gap (p=.19). Instead, higher operation counts slightly lower the explicit score (p=.049), meaning length works against the explicit condition, not for it. This drop is driven almost entirely by Step1X-Think (=-0.28 per operation); for all other editors, the effect is negligible (||0.08). Splitting items into operation-count tertiles (5, 6–8, 9) confirms this. The performance gap remains stable across tertiles, proving that models with large gaps maintain their behavior even on the shortest explicit prompts.
Additionally, operations marked as "unnecessary" were primarily global "look" adjustments added to the requested change (e.g., "make this look like a lavish gourmet dish" "desaturate by 15%"). While these adjustments account for 53–59% of all unnecessary operations (and 25% of operations overall), the performance gap remains completely unrelated to their frequency (). Finally, to control for raw editing difficulty, we isolated instances where an editor executed the explicit instruction nearly perfectly (score ). On these demonstrably executable edits, open-source editors still fail the abstract version in 19–56% of cases, compared to only 3% for proprietary editors.
Together, these controlled comparisons confirm that the performance gap is driven by abstract reasoning, not by prompt length, operation count, or editing difficulty. We will detail this validation study and all evaluation criteria in the revised paper.
Response to Reviewer aEnR - Part 3 of 3
Regarding Cross-entity Relations and Sub-entity Components: (Q3)
We thank the reviewer for this insightful feedback. Both points address important design choices, which we initially discussed in the appendix: why pairwise relations are not explicit evaluation units (Appendix A.2) and how deeply to decompose entities (Appendix C).
Cross-Entity Relations:
We agree that cases where "individual entities appear plausible but the overall edit does not satisfy the user's intent" represent a valid limitation. Such cases are challenging to analyze systematically and primarily surface in the Social and Emotional categories, where intent relies on subjective notions like a "vibe" or social dynamic. Our user study (3 annotators × 360 edits) confirms this: annotator agreement is notably lower for these categories compared to others (Krippendorff's =0.47 for Social and 0.50 for Emotional, vs. 0.56 overall).
We will add qualitative examples of these failure cases to the appendix and explicitly discuss this limitation in the main text. For instance, given the instruction "Infuse the scene with empathy and a deep bond with animals" applied to a roping photo (FLUX.1 [dev]), the model removes the lasso and makes the roper smile. However, the calf is still fleeing, and there is no gesture of care between the two; the scene still reads as a chase. Our judge, seeing the individual entity changes satisfied, scores it 9/10, while human annotators give a 6/10. Listing only these instruction-implied relations as additional evaluation units, rather than all pairs, is the tractable extension we propose in Appendix C, and we will expand and explicitly discuss it in the main paper's final version.
Sub-Entity Components and Granularity:
We agree that determining the correct level of granularity requires careful reasoning. EntityRubrics dynamically handles this in its evaluation pipeline, as a fixed hierarchy would limit the scope of evaluable instructions. In practice, this dynamic granularity works well: the judge splits entities into sub-parts in 43% of evaluated edits. This occurs most frequently where parts are critical (Pose 55%, Social-Role 54%, Emotional 50%) and least in POV/Composition (25%). Furthermore, sub-parts are penalized more strictly than whole entities (labeled as failing 17.8% vs. 12.9% of the time).
For example, given "Make her look absolutely excited" (Bagel), an edit adds a wide smile but leaves the eyes static, resulting in an awkward expression. The judge successfully captures this by splitting the face into woman_mouth (scored 10/10) and woman_eyes (scored 5/10), resulting in an overall 7/10 because "the eyes and eyebrows remain static, which makes the expression feel pasted on."
However, we acknowledge that part decomposition can struggle when required components are absent from the original context image. For instance, instructed to "Make him look like extremely wealthy royalty" (Bagel-Think), an edit might add a crown but leave the man's casual clothing unchanged. While the judge catches this specific failure (man_clothing scored 4/10), it is less proactive at anticipating entirely new parts.
Benchmark-wide, 44% of edits insert a new entity, and 5% insert a new accessory. The judge's expectation step successfully anticipates 99% of changes to existing entities, but only 48% of newly inserted entities. When an anticipated insertion is missing, the judge reliably flags it (91% failure rate, e.g., "The requested accessibility ramp was not added"). Unanticipated insertions (like a scepter or throne) are discovered post-edit and correctly credited as creative enhancements (93% labeled correct), but they are not strictly required by the initial rubric.
Thank the authors for their detailed rebuttal and additional clarifications. These additions have addressed all of my previous concerns. In particular, the further analyses of empirical robustness, together with the independent human validation and controlled experiments concerning the validity of abstract–explicit instruction pairs, substantially strengthen the credibility of the empirical results and provide compelling support for the paper's central conclusions regarding differences in abstract reasoning capabilities. Furthermore, the expanded discussion of limitations concerning cross-entity relations and sub-entity components makes the paper more complete. I am satisfied with the authors' responses and therefore improve my rating to 4.
Response to Reviewer aEnR - Part 1 of 3
We thank the reviewer for their time, effort, and positive feedback. We are especially glad you found the research problem timely, the presentation clear, and the dataset construction valuable.
Regarding the Robustness of the Empirical Conclusions (Q1):
We appreciate this important insight. To properly account for instructions sharing the same context image, we re-analyzed the judge scores for all 14 editors by treating each context image (rather than each instruction) as an independent unit. We now report all results with 95% bootstrap confidence intervals grouped by context image, alongside Holm-corrected paired comparisons (91 pairs) between editors. Specifically, we resampled the 257 images with replacement (including their associated instructions and scores) 2,000 times.
We have expanded our analysis to demonstrate the following:
1. Shared context images have minimal effect on variance and ranking. A mixed-effects model analyzing score variance attributes only ~1% of the variance to the context image; the vast majority stems from the editor and the instruction itself. Consequently, image-grouped confidence intervals are only ~2.5% wider than standard ones. Furthermore, editor ranking is robust to image reuse: assigning equal weight to every image yields the exact same ranking. Keeping only a single random instruction per image (repeated 1,000 times) reproduces the ranking almost perfectly (median Kendall's =0.98). Finally, because no image is reused within the same category, repeated images always evaluate distinct skills.
2. The benchmark successfully separates editors and model families. With image-grouped intervals, each editor's average score is precise to within 0.1–0.3 points (on a 1–10 scale). We compared all 91 pairs of the 14 editors using average per-instruction score differences to control for instruction difficulty. Even after applying the strict Holm correction for multiple testing, 79 of the 91 pairs differ significantly. The remaining ties only occur between highly similar models (e.g., GPT-Image-2 and Gemini 3.1 Flash). At the family level, the conclusion is clear-cut: every closed-source editor significantly outperforms every open-source editor (48 of 48 pairs, with a minimum gap of 1.1 points).
3. Trends hold robustly across categories and domains. Editor rankings remain highly consistent across the 12 categories and 4 domains (Kendall's W=0.88 for categories, 0.93 for domains, where 1 indicates identical rankings). Variations between categories primarily manifest in the size of performance gaps, which reveals informative, model-specific weaknesses. For example, Physical-Size is universally challenging, and both Step1X variants struggle with POV/Composition instructions. Because individual categories contain fewer instructions, their confidence intervals are naturally wider (0.6–1.4 points); therefore, we restrict category-level conclusions to clearly separated groups of editors.
We will add the full per-category and per-domain results, complete with confidence intervals, to the final version of the paper.