- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Domain-Edit to test whether compositional abilities hold across different visual domains. It evaluates editing ability of eight open-source models on Texture, Landscape, and Scene subsets. The experiments measure instruction following, background preservation, and scene integration, and show that model performance and rankings can change across domains.
Critical strengths
The paper raises a valuable question. Strong generation or photo-editing scores may not reflect performance in other visual domains. Testing the same editing instructions across styles provides useful evidence. Measuring background preservation also captures failures that instruction-following scores miss.
Critical weaknesses
-
The generation-to-editing gap does not provide solid evidence of generator-domain bias. Editing also requires source-image understanding and content preservation. Training differences may affect the results as well. In Table 3, the generation model is Qwen-Image, while the editing model is Qwen-Image-Edit. These are different models with different inputs. Their score gap does not isolate generator-domain bias. The within-editing comparisons support the benchmark’s motivation. However, the claims about the generation-to-editing gap should be revised.
-
The benchmark covers a narrow range of visual contexts. Landscape contains only five underlying scenes. Each scene appears in five styles. This supports controlled style comparisons, but adds little diversity in content or layout. The findings may depend on these particular backgrounds.
-
The human evaluation is limited and incompletely reported. The context-score study uses one annotator. It reports agreement on model rankings, rather than individual images. The authors also mention human verification of 6,300 images for Soft-TIFA. However, they do not report agreement statistics or results by style. Some cross-style differences could reflect judge errors rather than editor performance.
Questions for authors
- Can authors provide more detailed analysis of the generation-to-editing gap and its relationship to generator-domain bias?
- Are the main trends consistent across more individual Landscape scenes?
- Can authors report the human verification results for Soft-TIFA, separately for each style?
3 comments
Question 1
Can authors provide more detailed analysis of the generation-to-editing gap and its relationship to generator-domain bias?
We agree that editing can be inherently more challenging than generation, independently of generator-domain bias, so we provide an additional analysis. To make our point clearer, we propose a complementary way to test models against generator-domain bias. For that, we suggest a continuum of increasingly specific prompts. We then argue that editing is the extreme point of this continuum: there, the source image specifies the context completely.
Specifically, we add context to the GenEval prompts step by step. Spectrum 1-3 are pure text-to-image generation, without any image conditioning:
- Spectrum 1: the original GenEval prompt.
- Spectrum 2: Spectrum 1 + an explicit style: "The image is in a {style} style." (5 styles: real, cartoon, painting, grayscale, sketch).
- Spectrum 3: Spectrum 2 + a detailed description of the scene, matching our 5 Landscape scenes, e.g. "The scene is a valley with tall leafy trees filling the left half, a steep wooded slope on the right, ..." (5 scenes x 5 styles). For Texture, we instead describe the background, e.g. "The background is a flat, close-up texture of a brick wall filling the entire frame." (12 textures; no style step).
- Spectrum 4: editing on the source image itself, i.e., our original paper's results (Tables 2 and 3).
We report Soft-TIFA results:
| Model | S1: GenEval prompt | S2: + style | S3: + style + scene | S3 (Texture): + texture description | S4: edit on Landscape image | S4 (Texture): edit on Texture image |
|---|---|---|---|---|---|---|
| BAGEL | 93.3 | 92.0 | 80.6 | 89.5 | 37.3 | 71.1 |
| Qwen-Image* | 97.0 | 96.6 | 92.2 | 96.2 | 75.1 | 77.7 |
| OmniGen2 | 88.8 | 87.4 | 78.0 | 83.1 | 66.3 | 63.7 |
| SenseNova-SFT | 98.0 | 96.4 | 91.4 | 97.3 | 83.1 | 87.3 |
| SenseNova-RL | 98.5 | 96.9 | 94.5 | 97.3 | 87.6 | 92.0 |
| FLUX.2 Klein | 95.2 | 91.7 | 67.5 | 93.4 | 61.3 | 60.9 |
* Spectrum 4 uses Qwen-Image-Edit.
In short: accuracy goes down with every step of specification, for every model (e.g., FLUX.2: 95.2 -> 91.7 -> 67.5 -> 61.3). Editing is the most specific form of this continuum, which is why we use the image as the final form in DOMAIN-EDIT. This supports that the generator-domain bias depends on how much we specify context. We believe that this, together with a fully specified context through image editing, strengthens our results. We will include this analysis in the paper.
Question 2
Are the main trends consistent across more individual Landscape scenes?
Yes, the main trends hold within each of the five scenes.
Note that in Landscape, each scene is fixed and only its style changes (5 scenes x 5 styles). That models do better on some sources than others already shows that this bias exists. What matters, we believe, is whether our conclusions hold within each scene, and they do. Here we illustrate results across styles and measure accuracies across them.
| Model | Generation | cartoon | grayscale | painting | photo | sketch |
|---|---|---|---|---|---|---|
| SenseNova-RL | 98.5 | 90.3 | 86.6 | 86.7 | 87.5 | 86.7 |
| SenseNova-SFT | 98.0 | 86.5 | 81.4 | 81.5 | 85.7 | 80.3 |
| Qwen-Image-Edit | 97.0* | 81.9 | 71.7 | 70.3 | 79.2 | 72.4 |
| OmniGen2 | 88.8 | 67.0 | 56.5 | 66.2 | 72.2 | 69.5 |
| Step1X-Edit | - | 60.1 | 60.8 | 57.9 | 68.8 | 63.2 |
| FLUX.2 | 95.2 | 65.4 | 60.9 | 63.1 | 56.4 | 60.5 |
| FLUX.1 Kontext | - | 67.7 | 36.5 | 45.7 | 43.1 | 48.5 |
| BAGEL | 93.3 | 70.8 | 33.1 | 20.6 | 37.7 | 24.4 |
Soft-TIFA per scene (scenes named by their original style), averaged over the 5 styles of each scene (Appendix Table 8). * Qwen-Image.
In every scene, every model edits worse than its generation's, and accuracy changes with style. The top three models are the same, in the same order, in every scene. Because the content was fixed and only the style changed, we believe this supports the idea of generator-domain bias. Though we agree that the rankings themselves may change across content and we will not claim otherwise.
Question 3
Can authors report the human verification results for Soft-TIFA, separately for each style?
We apologize for omitting these results from the paper. Single onnotator labeled each of the 6,300 edited images, and we measure how well Soft-TIFA, our metric of choice in the paper, agrees with these labels (AUROC), per style and per task:
| Source | Style | Single object | Two objects | Counting | Colors | Position | Color attr. | All tasks |
|---|---|---|---|---|---|---|---|---|
| Scene | original | NA | 0.946 | 0.979 | 0.990 | 0.969 | 0.954 | 0.964 |
| blur | 1.000 | 0.998 | 1.000 | NA | 0.957 | 0.915 | 0.982 | |
| mosaic | 1.000 | 0.936 | 1.000 | 1.000 | 0.998 | 0.969 | 0.979 | |
| game | NA | 0.985 | 1.000 | 1.000 | 0.964 | 0.851 | 0.978 | |
| cartoon | 1.000 | 0.965 | 0.994 | NA | 0.971 | 0.857 | 0.967 | |
| grayscale | 1.000 | 0.978 | 1.000 | 1.000 | 0.952 | 0.888 | 0.967 | |
| sketch | 1.000 | 0.929 | 0.968 | 1.000 | 0.965 | 0.931 | 0.963 | |
| green channel | NA | 0.961 | 0.964 | 0.898 | 0.972 | 0.961 | 0.960 | |
| painting | NA | 0.940 | 0.998 | 0.969 | 0.892 | 0.869 | 0.953 | |
| Van Gogh | 1.000 | 0.877 | 1.000 | 1.000 | 0.943 | 0.875 | 0.952 | |
| red channel | 1.000 | 0.950 | 1.000 | 0.980 | 0.917 | 0.871 | 0.946 | |
| oil painting | NA | 0.942 | 0.878 | 1.000 | 0.982 | 0.950 | 0.944 | |
| All 12 styles | 1.000 | 0.946 | 0.984 | 0.989 | 0.957 | 0.914 | 0.963 | |
| Texture | brick | 1.000 | 0.998 | 1.000 | 1.000 | 1.000 | 0.951 | 0.995 |
| paper | 1.000 | 0.995 | 1.000 | NA | 0.997 | 0.962 | 0.990 | |
| metal | 1.000 | 0.981 | 1.000 | 0.879 | 0.992 | 0.962 | 0.986 | |
| marble | 1.000 | 0.952 | 0.995 | NA | 0.946 | 0.966 | 0.972 | |
| slate | 1.000 | 0.973 | 0.986 | NA | 0.970 | 0.846 | 0.972 | |
| concrete | 1.000 | 0.949 | 0.963 | 1.000 | 0.954 | 0.986 | 0.964 | |
| fabric | 0.979 | 0.939 | 1.000 | 1.000 | 0.946 | 0.833 | 0.960 | |
| wood | 0.898 | 0.937 | 0.989 | NA | 0.977 | 0.925 | 0.958 | |
| cracked earth | 0.995 | 0.893 | 0.970 | 0.993 | 0.955 | 0.877 | 0.951 | |
| All 9 styles | 0.987 | 0.959 | 0.990 | 0.959 | 0.972 | 0.934 | 0.973 | |
| All 21 styles | 0.996 | 0.951 | 0.987 | 0.978 | 0.965 | 0.924 | 0.968 |
The scores report AUROC of Soft-TIFA (Qwen3-VL-8B) against binary human labels on BAGEL edits, per source style x GenEval task (50 images per cell and 6,300 images in total). By NA we denote that all edits in the cell succeeded, so AUROC is undefined (since the model didn't make mistakes).
Soft-TIFA agrees closely with humans: AUROC is 0.968 overall and at least 0.944 for every style, so evaluator errors are small and do not concentrate on particular styles. The weakest task is color attribution (0.924), and we will note this in the paper. We will add the full table and the annotation protocol to the appendix.
To go beyond a single annotator and BAGEL, 3 annotators labeled 360 outputs from all 8 models (Krippendorff's α = 0.84).
| Source | n | Image AUROC | Atom AUROC |
|---|---|---|---|
| Plain | 48 | 85.7 [74.0, 95.1] | 94.7 |
| Texture | 48 | 90.3 [80.9, 97.4] | 90.8 |
| Real | 48 | 88.2 [78.5, 96.1] | 92.6 |
| Cartoon | 48 | 87.3 [76.2, 95.4] | 93.6 |
| Painting | 48 | 91.4 [82.6, 98.1] | 94.2 |
| Grayscale | 48 | 87.6 [75.6, 96.1] | 91.3 |
| Sketch | 48 | 92.3 [84.4, 98.4] | 96.8 |
| Text-only (S2) | 24 | 91.7 [77.1, 100] | 93.6 |
| All | 360 | 88.7 [85.4, 91.6] | 93.3 |
Agreement is high on every source and does not depend on style (p = 0.95). Judge and humans rank the 8 models identically (Kendall's τ = 1.00), so our style comparisons and rankings hold under human labels. Brackets: 95% bootstrap CIs. Both studies and protocols will be added to the appendix.
We appreciate the reviewer for the constructive feedback.
Weakness 1
We agree. We only present them as a controlled comparison. To us, Qwen-Image serves as a reference for the generation accuracy of the model family because Qwen-Image-Edit does not support image-free generation. We will state this explicitly in the paper. We elaborate on the relation between editing and generation in the respective question (please see our response below).
Weakness 2
We agree that the number of underlying scenes in Landscape is relatively small. However, since the Scene subset already covers a broader range of object-centric scenes, Landscape is intentionally designed to investigate how models compose objects within a given spatial layout when foreground objects are absent. Collecting suitable real-world background-only images across different styles is also challenging.
Moreover, evaluating five scenes across five styles, 60 compositional instructions, and three random seeds already results in 4,500 editing samples per model (36,000 across eight models). We therefore consider this a reasonable trade-off between source-image diversity and evaluation depth in a controlled setting.
While expanding the diversity of underlying landscapes would be valuable, we emphasize that our findings regarding domain sensitivity are not based on Landscape alone.
Weakness 3
We have now run a per-sample study with three annotators, who rated scene fit (0-10) for 240 Landscape edits; 16 images where the added object was absent by majority vote were excluded.
Humans often disagree with each other on scene fit, so it is partly a matter of taste. Per sample, the judge does not match the human ratings well. Per model, it matches well: the judge and the humans rank the models almost the same way. This is also the only way we use the context score in the paper (Q4, Q5), and we will say so explicitly. Thank you for this remark.
Also, our main results do not depend on it: they use Soft-TIFA and background MSE.