acceptodds
Under review as a conference paper at ICLR 2027

Domain-Edit: A Benchmark for Domain-Unbiased Evaluation of Generative Models

Abstract

As text-to-image models advance, many of the prominent compositional generation benchmarks are saturating. Yet, modern generation models have been shown to be limited in their generation diversity. This makes it difficult to assert whether the generative models have developed general abilities, or their abilities have become specialized. We observe that these models are also biased in domain (like style, layout, object properties), a phenomenon we dub generator-domain bias. Existing compositional benchmarks do not control for generator-domain bias, so models are free to fall back on their preferred domain. We argue that editing tasks serve an appropriate medium for studying this: unlike pure generation, image editing requires the model to compose within a source image it did not choose. To this end, we introduce generator-domain bias, a benchmark that pairs compositional editing instructions with source images generated across diverse styles. We evaluate eight open-source models on three subsets of increasing visual diversity, spanning plain textures and five styles. We find that every model falls below its own generation accuracy once the domain is fixed, that performance varies substantially with the source style, and that models often regenerate the background rather than edit it. Based on these findings, we argue that input-domain diversity should be a core dimension of compositional evaluation.

Primary area:
generative models
Keywords:
Image generation, Image editing, Domain bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KXoqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Domain-Edit to test whether compositional abilities hold across different visual domains. It evaluates editing ability of eight open-source models on Texture, Landscape, and Scene subsets. The experiments measure instruction following, background preservation, and scene integration, and show that model performance and rankings can change across domains.

Critical strengths

The paper raises a valuable question. Strong generation or photo-editing scores may not reflect performance in other visual domains. Testing the same editing instructions across styles provides useful evidence. Measuring background preservation also captures failures that instruction-following scores miss.

Critical weaknesses
  1. The generation-to-editing gap does not provide solid evidence of generator-domain bias. Editing also requires source-image understanding and content preservation. Training differences may affect the results as well. In Table 3, the generation model is Qwen-Image, while the editing model is Qwen-Image-Edit. These are different models with different inputs. Their score gap does not isolate generator-domain bias. The within-editing comparisons support the benchmark’s motivation. However, the claims about the generation-to-editing gap should be revised.

  2. The benchmark covers a narrow range of visual contexts. Landscape contains only five underlying scenes. Each scene appears in five styles. This supports controlled style comparisons, but adds little diversity in content or layout. The findings may depend on these particular backgrounds.

  3. The human evaluation is limited and incompletely reported. The context-score study uses one annotator. It reports agreement on model rankings, rather than individual images. The authors also mention human verification of 6,300 images for Soft-TIFA. However, they do not report agreement statistics or results by style. Some cross-style differences could reflect judge errors rather than editor performance.

Questions for authors
  1. Can authors provide more detailed analysis of the generation-to-editing gap and its relationship to generator-domain bias?
  2. Are the main trends consistent across more individual Landscape scenes?
  3. Can authors report the human verification results for Soft-TIFA, separately for each style?
3 comments

We appreciate the reviewer for the constructive feedback.

Weakness 1

The generation-to-editing gap does not provide solid evidence of generator-domain bias. Editing also requires source-image understanding and content preservation. Training differences may affect the results as well. In Table 3, the generation model is Qwen-Image, while the editing model is Qwen-Image-Edit. These are different models with different inputs. Their score gap does not isolate generator-domain bias. The within-editing comparisons support the benchmark’s motivation. However, the claims about the generation-to-editing gap should be revised.

We agree. We only present them as a controlled comparison. To us, Qwen-Image serves as a reference for the generation accuracy of the model family because Qwen-Image-Edit does not support image-free generation. We will state this explicitly in the paper. We elaborate on the relation between editing and generation in the respective question (please see our response below).

Weakness 2

The benchmark covers a narrow range of visual contexts. Landscape contains only five underlying scenes. Each scene appears in five styles. This supports controlled style comparisons, but adds little diversity in content or layout. The findings may depend on these particular backgrounds.

We agree that the number of underlying scenes in Landscape is relatively small. However, since the Scene subset already covers a broader range of object-centric scenes, Landscape is intentionally designed to investigate how models compose objects within a given spatial layout when foreground objects are absent. Collecting suitable real-world background-only images across different styles is also challenging.

Moreover, evaluating five scenes across five styles, 60 compositional instructions, and three random seeds already results in 4,500 editing samples per model (36,000 across eight models). We therefore consider this a reasonable trade-off between source-image diversity and evaluation depth in a controlled setting.

While expanding the diversity of underlying landscapes would be valuable, we emphasize that our findings regarding domain sensitivity are not based on Landscape alone.

Weakness 3

The human evaluation is limited and incompletely reported. The context-score study uses one annotator. It reports agreement on model rankings, rather than individual images. The authors also mention human verification of 6,300 images for Soft-TIFA. However, they do not report agreement statistics or results by style. Some cross-style differences could reflect judge errors rather than editor performance.

We have now run a per-sample study with three annotators, who rated scene fit (0-10) for 240 Landscape edits; 16 images where the added object was absent by majority vote were excluded.

Agreement (Spearman rho)Value
Annotator vs. mean of the other two annotators0.62
Judge context score vs. human mean, per image0.10 (p = 0.16)
Judge vs. human, model ranking (8 models)0.67

Humans often disagree with each other on scene fit, so it is partly a matter of taste. Per sample, the judge does not match the human ratings well. Per model, it matches well: the judge and the humans rank the models almost the same way. This is also the only way we use the context score in the paper (Q4, Q5), and we will say so explicitly. Thank you for this remark.

Also, our main results do not depend on it: they use Soft-TIFA and background MSE.

Question 1

Can authors provide more detailed analysis of the generation-to-editing gap and its relationship to generator-domain bias?

We agree that editing can be inherently more challenging than generation, independently of generator-domain bias, so we provide an additional analysis. To make our point clearer, we propose a complementary way to test models against generator-domain bias. For that, we suggest a continuum of increasingly specific prompts. We then argue that editing is the extreme point of this continuum: there, the source image specifies the context completely.

Specifically, we add context to the GenEval prompts step by step. Spectrum 1-3 are pure text-to-image generation, without any image conditioning:

  • Spectrum 1: the original GenEval prompt.
  • Spectrum 2: Spectrum 1 + an explicit style: "The image is in a {style} style." (5 styles: real, cartoon, painting, grayscale, sketch).
  • Spectrum 3: Spectrum 2 + a detailed description of the scene, matching our 5 Landscape scenes, e.g. "The scene is a valley with tall leafy trees filling the left half, a steep wooded slope on the right, ..." (5 scenes x 5 styles). For Texture, we instead describe the background, e.g. "The background is a flat, close-up texture of a brick wall filling the entire frame." (12 textures; no style step).
  • Spectrum 4: editing on the source image itself, i.e., our original paper's results (Tables 2 and 3).

We report Soft-TIFA results:

ModelS1: GenEval promptS2: + styleS3: + style + sceneS3 (Texture): + texture descriptionS4: edit on Landscape imageS4 (Texture): edit on Texture image
BAGEL93.392.080.689.537.371.1
Qwen-Image*97.096.692.296.275.177.7
OmniGen288.887.478.083.166.363.7
SenseNova-SFT98.096.491.497.383.187.3
SenseNova-RL98.596.994.597.387.692.0
FLUX.2 Klein95.291.767.593.461.360.9

* Spectrum 4 uses Qwen-Image-Edit.

In short: accuracy goes down with every step of specification, for every model (e.g., FLUX.2: 95.2 -> 91.7 -> 67.5 -> 61.3). Editing is the most specific form of this continuum, which is why we use the image as the final form in DOMAIN-EDIT. This supports that the generator-domain bias depends on how much we specify context. We believe that this, together with a fully specified context through image editing, strengthens our results. We will include this analysis in the paper.

Question 2

Are the main trends consistent across more individual Landscape scenes?

Yes, the main trends hold within each of the five scenes.

Note that in Landscape, each scene is fixed and only its style changes (5 scenes x 5 styles). That models do better on some sources than others already shows that this bias exists. What matters, we believe, is whether our conclusions hold within each scene, and they do. Here we illustrate results across styles and measure accuracies across them.

ModelGenerationcartoongrayscalepaintingphotosketch
SenseNova-RL98.590.386.686.787.586.7
SenseNova-SFT98.086.581.481.585.780.3
Qwen-Image-Edit97.0*81.971.770.379.272.4
OmniGen288.867.056.566.272.269.5
Step1X-Edit-60.160.857.968.863.2
FLUX.295.265.460.963.156.460.5
FLUX.1 Kontext-67.736.545.743.148.5
BAGEL93.370.833.120.637.724.4

Soft-TIFA per scene (scenes named by their original style), averaged over the 5 styles of each scene (Appendix Table 8). * Qwen-Image.

In every scene, every model edits worse than its generation's, and accuracy changes with style. The top three models are the same, in the same order, in every scene. Because the content was fixed and only the style changed, we believe this supports the idea of generator-domain bias. Though we agree that the rankings themselves may change across content and we will not claim otherwise.

Question 3

Can authors report the human verification results for Soft-TIFA, separately for each style?

We apologize for omitting these results from the paper. Single onnotator labeled each of the 6,300 edited images, and we measure how well Soft-TIFA, our metric of choice in the paper, agrees with these labels (AUROC), per style and per task:

SourceStyleSingle objectTwo objectsCountingColorsPositionColor attr.All tasks
SceneoriginalNA0.9460.9790.9900.9690.9540.964
blur1.0000.9981.000NA0.9570.9150.982
mosaic1.0000.9361.0001.0000.9980.9690.979
gameNA0.9851.0001.0000.9640.8510.978
cartoon1.0000.9650.994NA0.9710.8570.967
grayscale1.0000.9781.0001.0000.9520.8880.967
sketch1.0000.9290.9681.0000.9650.9310.963
green channelNA0.9610.9640.8980.9720.9610.960
paintingNA0.9400.9980.9690.8920.8690.953
Van Gogh1.0000.8771.0001.0000.9430.8750.952
red channel1.0000.9501.0000.9800.9170.8710.946
oil paintingNA0.9420.8781.0000.9820.9500.944
All 12 styles1.0000.9460.9840.9890.9570.9140.963
Texturebrick1.0000.9981.0001.0001.0000.9510.995
paper1.0000.9951.000NA0.9970.9620.990
metal1.0000.9811.0000.8790.9920.9620.986
marble1.0000.9520.995NA0.9460.9660.972
slate1.0000.9730.986NA0.9700.8460.972
concrete1.0000.9490.9631.0000.9540.9860.964
fabric0.9790.9391.0001.0000.9460.8330.960
wood0.8980.9370.989NA0.9770.9250.958
cracked earth0.9950.8930.9700.9930.9550.8770.951
All 9 styles0.9870.9590.9900.9590.9720.9340.973
All 21 styles0.9960.9510.9870.9780.9650.9240.968

The scores report AUROC of Soft-TIFA (Qwen3-VL-8B) against binary human labels on BAGEL edits, per source style x GenEval task (50 images per cell and 6,300 images in total). By NA we denote that all edits in the cell succeeded, so AUROC is undefined (since the model didn't make mistakes).

Soft-TIFA agrees closely with humans: AUROC is 0.968 overall and at least 0.944 for every style, so evaluator errors are small and do not concentrate on particular styles. The weakest task is color attribution (0.924), and we will note this in the paper. We will add the full table and the annotation protocol to the appendix.

To go beyond a single annotator and BAGEL, 3 annotators labeled 360 outputs from all 8 models (Krippendorff's α = 0.84).

SourcenImage AUROCAtom AUROC
Plain4885.7 [74.0, 95.1]94.7
Texture4890.3 [80.9, 97.4]90.8
Real4888.2 [78.5, 96.1]92.6
Cartoon4887.3 [76.2, 95.4]93.6
Painting4891.4 [82.6, 98.1]94.2
Grayscale4887.6 [75.6, 96.1]91.3
Sketch4892.3 [84.4, 98.4]96.8
Text-only (S2)2491.7 [77.1, 100]93.6
All36088.7 [85.4, 91.6]93.3

Agreement is high on every source and does not depend on style (p = 0.95). Judge and humans rank the 8 models identically (Kendall's τ = 1.00), so our style comparisons and rankings hold under human labels. Brackets: 95% bootstrap CIs. Both studies and protocols will be added to the appendix.

Reviewer_SSPTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Generator-Domain Bias and proposes DOMAIN-EDIT, a benchmark for evaluating compositional image editing across different visual domains. The benchmark consists of three subsets, Texture, Landscape, and Scene, which cover source images ranging from plain backgrounds to complex real-world scenes. By keeping editing instructions fixed while varying source images, the authors evaluate eight open-source models in terms of semantic alignment, source preservation, and contextual integration. The experiments reveal substantial performance variations across visual domains, including background regeneration and inconsistencies between semantic accuracy and contextual quality.

Critical strengths
  1. Relevant problem formulation. The paper highlights an important limitation of existing compositional generation benchmarks: strong performance on text-conditioned generation does not necessarily translate into reliable editing under externally specified visual contexts.

  2. A structured benchmark design. DOMAIN-EDIT covers three levels of visual complexity, from texture backgrounds to landscapes and object-rich scenes. Evaluating semantic alignment alongside source preservation also provides a more informative assessment of editing quality than checking object presence alone.

  3. Useful empirical observations. The experiments reveal domain-dependent performance variations, background regeneration, and discrepancies between semantic accuracy and contextual integration. These findings illustrate why conventional generation and editing scores may not fully characterize compositional editing capabilities.

Critical weaknesses
  1. The factors underlying Generator-Domain Bias are not sufficiently disentangled. Although DOMAIN-EDIT covers Texture, Landscape, and Scene, these subsets differ in background characteristics, scene complexity, and the presence of existing objects. Consequently, the observed performance differences cannot be clearly attributed to specific factors. More controlled experiments that vary style, texture, and scene complexity independently are needed to establish which factors primarily account for the performance degradation.

  2. The effect of style transfer on the evaluation remains unclear. The Landscape and Scene subsets use Nano Banana Pro to render source images in different styles. However, style transfer may also alter object appearance, spatial relationships, or scene details. Without sufficiently verifying the preservation of source content and layout, it is difficult to determine whether the observed degradation reflects the models' domain sensitivity or artifacts introduced during source-image transformation.

  3. The evidence for the generality of the findings is limited. Although the benchmark evaluates eight models, the Landscape subset contains only a small number of underlying scenes, and the evaluation metrics differ across subsets. In particular, the use of Soft-TIFA and background MSE for Texture and Landscape, but VIEScore for Scene, complicates a unified interpretation of performance changes across the three settings. Broader source-image coverage and more consistent evaluation would strengthen the conclusions about Generator-Domain Bias.

Questions for authors
  1. Since style, texture, and scene complexity are not independently controlled across the three subsets, how can the authors determine which factors are primarily responsible for the observed performance degradation?

  2. How do the authors verify that Nano Banana Pro preserves the original content and spatial layout during style transfer, ensuring that the measured performance differences reflect domain sensitivity rather than transformation artifacts?

2 comments

We thank the reviewer for the thorough review.

Weakness 1, Question 1

The factors underlying Generator-Domain Bias are not sufficiently disentangled. Although DOMAIN-EDIT covers Texture, Landscape, and Scene, these subsets differ in background characteristics, scene complexity, and the presence of existing objects.

We clarify that our benchmark explicitly controls for style within subsets, rather than across the three subsets. Therefore, the mentioned background characteristics, scene complexity, and the presence of existing objects are controlled.

SubsetControlled factorsWhat we investigate
TextureNo layout constraintsSensitivity to background appearance and texture
LandscapeSame background-only scene and spatial layout across stylesStyle sensitivity and global contextual integration (e.g., placement, scale, and perspective)
SceneSame underlying scene content, objects, and spatial layout across stylesStyle sensitivity in object-centric editing and local object manipulation

Importantly, in both Landscape and Scene, we explicitly vary the source style while keeping the underlying scene content, and spatial layout. This allows us to investigate the effect of style independently of the underlying scene complexity.

Our primary goal is to demonstrate that compositional ability is sensitive to "input domains" and that this sensitivity varies across models.

Weakness 2, Question 2

The effect of style transfer on the evaluation remains unclear.

How do the authors verify that Nano Banana Pro preserves the original content and spatial layout during style transfer, ensuring that the measured performance differences reflect domain sensitivity rather than transformation artifacts?

We visually inspected the style-transferred images and found that the original content and spatial layout were generally well preserved, as can be seen in Appendix C.

To further investigate whether style transfer itself negatively affects editing performance, we compare the average accuracy on original inputs with that on style-transformed inputs on the Landscape subset, as shown below.

Effect of style transformation on Landscape. Soft-TIFA accuracy (↑) averaged over original and style-transformed inputs. Δ denotes the difference in percentage points (pp).

ModelOriginalTransformedΔ (pp)
BAGEL30.2839.09+8.81
Step-1X-Edit66.5461.07-5.48
FLUX.1 Kontext44.0849.37+5.29
Qwen-Image-Edit76.0874.86-1.23
OmniGen265.1866.56+1.38
SenseNova-SFT80.8983.66+2.77
SenseNova-RL87.5187.56+0.05
FLUX259.3261.76+2.44
Average63.7465.49+1.75

Six out of eight models perform better on style-transformed inputs than on the original inputs. The results suggest that style transfer itself does not systematically degrade editing performance.

Weakness 3

The evidence for the generality of the findings is limited. Although the benchmark evaluates eight models, the Landscape subset contains only a small number of underlying scenes, and the evaluation metrics differ across subsets.

While Texture and Landscape have relatively few source images, they are intentionally designed as controlled experimental settings. Each source image is evaluated using the same 60 compositional instructions across six task categories, with five random seeds for Texture and three for Landscape. This results in 3,600 and 4,500 editing samples per model.

Importantly, our evaluation is not limited to these controlled settings. As detailed in Appendix Table 5, the Scene subset contains 205 distinct underlying images across five styles, resulting in 1,025 source images. This complements Texture and Landscape by investigating domain sensitivity across a substantially broader range of scene content and editing instructions.

Regarding the evaluation metrics, we will discuss them in more detail below. We follow the established evaluation protocol of each respective benchmark. Our findings do not rely on comparing absolute scores across different metrics, but on observing performance variations within each subset.

In particular, the use of Soft-TIFA and background MSE for Texture and Landscape, but VIEScore for Scene, complicates a unified interpretation of performance changes across the three settings. Broader source-image coverage and more consistent evaluation would strengthen the conclusions about Generator-Domain Bias.

We intentionally follow the established evaluation protocols of the respective benchmarks to maintain consistency with prior work.

  • Texture and Landscape: Both subsets are derived from a compositional generation benchmark. We adopt Soft-TIFA from GenEval2 [1] to evaluate compositional accuracy and additionally use background MSE as a simple measure of source preservation. The context score for Landscape is introduced as a complementary metric to analyze contextual integration, rather than as a primary measure of compositional accuracy.
  • Scene: This subset is derived from GEdit-Bench [2], and we therefore retain its original evaluation protocol.

Our conclusions regarding Generator-Domain Bias do not rely on directly comparing absolute scores across different evaluation metrics. Instead, we analyze performance differences within each subset under the same evaluation protocol while varying the source domain.

Regarding source-image coverage, we agree that including more diverse source images would be valuable. However, there is a trade-off between source-image diversity and evaluation depth under a fixed computational budget. Since we evaluate each source image with 60 compositional instructions and multiple random seeds, increasing the number of source images would substantially increase the computational cost. We therefore prioritize a thorough evaluation across diverse compositional tasks and random seeds rather than expanding source-image coverage at the expense of evaluation robustness.

[1] GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation (2025)

[2] Step1X-Edit: A Practical Framework for General Image Editing (2025)

Reviewer_wTyKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether strong compositional text-to-image generation performance transfers to editing within a given visual context. It formalizes “generator-domain bias” as concentration in visual attributes left unspecified by a prompt and introduces DOMAIN-EDIT, comprising Texture, Landscape, and Scene subsets. These combine GenEval-derived insertion instructions or GEdit-Bench editing instructions with source images spanning different textures and styles. Eight open-source model variants are evaluated using semantic alignment, background preservation, and scene integration metrics. The experiments show substantial generation-to-editing performance gaps, sensitivity to source style, background regeneration, and changes in model rankings. Validation includes qualitative examples, a prompt-phrasing ablation, reported human verification of 6,300 images, and a limited human comparison for the context score.

Critical strengths
  1. The benchmark examines a consequential dimension of compositional evaluation. Holding editing instructions fixed while varying source styles tests whether model capabilities transfer across visual contexts. The Scene experiments are particularly useful because they compare the same editing task across original and restyled sources, exposing limitations hidden by photograph-dominated evaluation.
  2. The evaluation distinguishes complementary failure modes. Measuring semantic correctness alongside background preservation and contextual integration demonstrates why a high instruction-following score alone is insufficient. For example, the SenseNova SFT/RL comparison on white backgrounds shows similar semantic accuracy but substantially different preservation errors (Table 2). Such findings provide actionable information beyond a single leaderboard score.
Critical weaknesses
  1. The experiments do not sufficiently isolate generator-domain bias from general editing limitations. Moving from generation to editing changes both the conditioning interface and the task requirements; a performance gap can therefore arise without demonstrating that concentrated generation domains explain it. The Qwen comparison additionally uses different generation and editing checkpoints. The prompt ablation in Table 10 establishes substantial sensitivity to wording, but does not resolve this attribution problem. Likewise, observing different rankings across different tasks does not establish that existing benchmarks rank models incorrectly. The evidence supports sensitivity to source context more directly than the stronger explanation advanced throughout the paper.

  2. The benchmark’s measurement validity and breadth remain insufficiently established. Appendix A.4 states that humans verified atomic criteria on 6,300 images, but does not report agreement, judge accuracy, or errors broken down by style and task. These results are essential because style-dependent evaluator errors could affect the central comparisons. The context-score validation uses one annotator and reports correlation between aggregate model rankings, which does not establish sample-level reliability. Moreover, Scene uses perceptual quality as a preservation-related measure, although an attractive regenerated image can score well without preserving the source. Landscape contains only five underlying scenes; additional prompts and seeds do not establish generalization across scene content. These limitations weaken confidence in the resulting comparisons and rankings.

Questions for authors

The following questions request clarification or evidence needed to assess the current claims:

  1. What evidence distinguishes generator-domain bias from general editing difficulty or instruction sensitivity? Do models with greater measured generation-domain concentration exhibit larger editing gaps after accounting for generation performance?
  2. Please provide the results of the reported human verification on 6,300 images, including annotation protocol, agreement, and judge performance by style and atomic task. Do the main style comparisons and rankings persist under human labels?
  3. How was content and layout preservation verified after source-image restyling, especially for Scene instructions that depend on specific attributes or spatial relations? How were the 205 original Scene images selected?
  4. Please clarify the population used for Table 3’s context scores and report the detector-failure exclusion rates mentioned in Appendix A.4.
5 comments

We thank the reviewer for the valuable feedback.

Weakness 1

The Qwen comparison additionally uses different generation and editing checkpoints.

We agree. We only present them as a controlled comparison. To us, Qwen-Image serves as a reference for the generation accuracy of the model family because Qwen-Image-Edit does not support image-free generation. We will state this explicitly in the paper.

The prompt ablation in Table 10 establishes substantial sensitivity to wording, but does not resolve this attribution problem.

We'd like to note that we were not aiming to provide attribtuion. Its purpose is to explain why we use "Add X" instead of GenEval's "A photo of X" for editing. With "A photo of X", accuracy goes up for every model, but so does background MSE: the models tend to ignore the source image and instead generate their own preferred backgorunds (which is what generator-domain bias studies). Therefore, prompts like "Add X" measures editing performance. We will make this purpose clear in the paper.

Likewise, observing different rankings across different tasks does not establish that existing benchmarks rank models incorrectly. The evidence supports sensitivity to source context more directly than the stronger explanation advanced throughout the paper.

We agree that different rankings alone do not show that existing benchmarks rank models incorrectly. Our argument is that generation benchmarks let each model answer within the domain it prefers, so their scores reflect performance in that domain, which is exemplified in the analyses we perform on generator-domain bias. Our text-only experiments (please refer to Q1 for results) supports this: as the prompt contain more context information, accuracy drops for every model. We argue that editing fixes the domain, and once it is fixed, the rankings change. Our Q5 title, "Do existing benchmarks rank models correctly?", suggests that existing benchmarks are wrong, which is not what we show. We will rephrase it to ask whether their rankings still hold once the domain is fixed externally through either text detail or image conditioning.

Weakness 2

The context-score validation uses one annotator and reports correlation between aggregate model rankings, which does not establish sample-level reliability.

We agree. We have now run a per-sample study with three annotators, who rated scene fit (0-10) for 240 Landscape edits; 16 images where the added object was absent by majority vote were excluded.

Agreement (Spearman rho)Value
Annotator vs. mean of the other two annotators0.62
Judge context score vs. human mean, per image0.10 (p = 0.16)
Judge vs. human, model ranking (8 models)0.67

Per sample, the judge does not match the human ratings well, and per model the agreement is moderate (ρ = 0.67). We therefore treat the context score only as a complementary, model-level signal (Q4, Q5) and will state this explicitly. Our main results do not depend on it: they use Soft-TIFA and background MSE. Thank you for this remark.

Our main results do not depend on it: they use Soft-TIFA and background MSE.

Moreover, Scene uses perceptual quality as a preservation-related measure, although an attractive regenerated image can score well without preserving the source.

We believe it was a typo. We agree that PQ does not measure preservation: it rates how natural the image looks, not whether the source was kept. We will rename it to perceptual quality. None of our conclusions rely on it; we report it only because GEdit-Bench does.

Landscape contains only five underlying scenes; additional prompts and seeds do not establish generalization across scene content. These limitations weaken confidence in the resulting comparisons and rankings.

We agree that five scenes give limited content diversity, and we will state this in the paper.

However, Landscape is a controlled setup: each scene is kept fixed and only its style changes (5 scenes x 5 styles). That models do better on some sources than others already shows that this bias exists. What matters, we believe, is whether our conclusions hold within each scene, and they do. Here we illustrate results across styles and measure accuracies across them.

ModelGenerationcartoongrayscalepaintingphotosketch
SenseNova-RL98.590.386.686.787.586.7
SenseNova-SFT98.086.581.481.585.780.3
Qwen-Image-Edit97.0*81.971.770.379.272.4
OmniGen288.867.056.566.272.269.5
Step1X-Edit-60.160.857.968.863.2
FLUX.295.265.460.963.156.460.5
FLUX.1 Kontext-67.736.545.743.148.5
BAGEL93.370.833.120.637.724.4

Soft-TIFA per scene (scenes named by their original style), averaged over the 5 styles of each scene (Appendix Table 8). * Qwen-Image.

In every scene, every model edits worse than it generates, and accuracy changes with style. The top three models are the same, in the same order, in every scene. Because the content was fixed and only the style changed, we believe this supports the idea of generator-domain bias. Though we agree that the rankings themselves may change across content and we will not claim otherwise.

Question 1

What evidence distinguishes generator-domain bias from general editing difficulty or instruction sensitivity? Do models with greater measured generation-domain concentration exhibit larger editing gaps after accounting for generation performance?

We agree that these constraints can make editing inherently more challenging, independently of generator-domain bias. To make our point clearer, we propose a complementary way to test models against generator-domain bias. For that, we suggest a continuum of increasingly specific prompts. We then argue that editing is the extreme point of this continuum: there, the source image specifies the context completely.

Specifically, we add context to the GenEval prompts step by step. Spectrum 1-3 are pure text-to-image generation, without any image conditioning:

  • Spectrum 1: the original GenEval prompt.
  • Spectrum 2: Spectrum 1 + an explicit style: "The image is in a {style} style." (5 styles: real, cartoon, painting, grayscale, sketch).
  • Spectrum 3: Spectrum 2 + a detailed description of the scene, matching our 5 Landscape scenes, e.g. "The scene is a valley with tall leafy trees filling the left half, a steep wooded slope on the right, ..." (5 scenes x 5 styles). For Texture, we instead describe the background, e.g. "The background is a flat, close-up texture of a brick wall filling the entire frame." (12 textures; no style step).
  • Spectrum 4: editing on the source image itself, i.e., our original paper's results (Tables 2 and 3).

We report Soft-TIFA results:

ModelS1: GenEval promptS2: + styleS3: + style + sceneS3 (Texture): + texture descriptionS4: edit on Landscape imageS4 (Texture): edit on Texture image
BAGEL93.392.080.689.537.371.1
Qwen-Image*97.096.692.296.275.177.7
OmniGen288.887.478.083.166.363.7
SenseNova-SFT98.096.491.497.383.187.3
SenseNova-RL98.596.994.597.387.692.0
FLUX.2 Klein95.291.767.593.461.360.9

* Spectrum 4 uses Qwen-Image-Edit.

In short: accuracy goes down with every step of specification, for every model (e.g., FLUX.2: 95.2 -> 91.7 -> 67.5 -> 61.3). Editing is the most specific form of this continuum, which is why we use the image as the final form in DOMAIN-EDIT. This supports that the generator-domain bias depends on how much we specify context. We believe that this, together with a fully specified context through image editing, strengthens our results. We will include this analysis in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.