- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces image editing through handwritten text and visual marks, supported by the Jasmine dataset, benchmark, and Thinker–Editor models. Its main contribution is a task and evaluation resource rather than a new model architecture.
Critical strengths
- The dataset covers 14 editing tasks and includes substantial manual quality control.
- Evaluation separates instruction understanding from editing, helping identify where existing models fail.
Critical weaknesses
- Training and evaluation share the same construction pipeline. The very high text-overlap scores do not establish generalization to unfamiliar handwriting or independently collected instructions.
- The benchmark relies on a single automated judge without reported agreement with human assessments of model outputs. Human filtering of dataset examples does not resolve this concern.
Questions for authors
- Clarification: Were splits separated by original source image, source instruction, and handwriting assets? What checks exclude near-duplicate examples across splits?
- Evidence request: How well do the judge's scores and model rankings agree with independent human ratings, particularly for spatial editing tasks?
6 comments
Q2 & W2: How is the agreement between Gemini judge and real human raters?
We would like to answer this question as comprehensive as we can, as it is asked by all of our reviewers. We re-invited our student participants to conduct human rating, and calculated the average, ranking, and the Pearson correlation between scores from the Gemini judge and real human raters. The results are presented in Tab 3.
We present the experiment process of Tab 3: (1) We referenced a paper named VIBE [1]. This work selected 2 models under benchmarking, with 100 randomly sampled outputs per model (10 samples per task across 10 tasks), to compute Pearson correlation between LLM-based judge and human raters; (2) We select 8 models. Besides our own models, the remaining models are the highest or lowest scoring models in each score band from Table 1 in our paper (e.g., Nano-Banana is the top scorer in the range 3.00~3.99, its score is 3.82). Thus, we cover all score bands, making our results as solid as possible; (3) Due to the limited time, we randomly select 280 outputs from each model (20 per task) for human rating. However, our sampling scale is still larger than VIBE; (4) We compute the MLLM average score, human average score, original ranking, current ranking, and the Pearson correlation for each model using 280 Gemini scores and 280 human scores.
We have the following findings: (1) The difference between human scores and Gemini scores is acceptable. More importantly, the relative rankings are maintained. Since these models are the highest or lowest scoring models in each score band, this indicates humans and Gemini are quite consistent in distinguishing which range the model's performance should fall on the score band; (2) Pearson correlation across all models ranges from 0.85 to 0.95, indicating strong agreement between Gemini Judge and real human raters; (3) Weaker models (e.g., Step1X-Edit) show particularly high correlation. It seems the Gemini Judge and human raters are quite consistent on determining what kind of outputs should get low scores; (4) We notice that you and another reviewer expressed interest in the Spatial Editing tasks. Thus, we conducted additional human rating on 200 samples on Spatial Editing tasks (40 samples per task) for our models. The results are presented in Tab 4, demonstrating that agreement between Gemini Judge and human raters remains strong.
Tab 3: Agreement between Gemini judge and real human raters. We compute the MLLM average score, human average score, original ranking, current ranking, and the Pearson correlation for each model using Gemini and human scores of 280 randomly selected samples.
| Model | MLLM Rating | Human Rating | Original Ranking | Current Ranking | Pearson Correlation |
|---|---|---|---|---|---|
| Step1X-Edit | 2.04 | 1.89 | 8 | 8 | 0.958 |
| FLUX.2-klein-base-4B | 2.85 | 2.69 | 7 | 7 | 0.924 |
| Qwen-Image-Edit-2509 | 3.36 | 3.08 | 6 | 6 | 0.916 |
| Nano-Banana | 3.75 | 3.70 | 5 | 5 | 0.934 |
| Jasmine-4B | 4.20 | 3.84 | 4 | 4 | 0.893 |
| Jasmine-9B | 4.33 | 4.12 | 3 | 3 | 0.907 |
| Jasmine-20B | 4.57 | 4.41 | 2 | 2 | 0.881 |
| GPT-Image-2 | 4.75 | 4.56 | 1 | 1 | 0.859 |
| Overall | 3.73 | 3.54 | - | - | 0.909 |
Tab 4: More agreement results on Spatial Editing tasks (metric: Pearson correlation).
| Model | Move | Zoom In | Zoom Out | Outpainting |
|---|---|---|---|---|
| Jasmine-4B | 0.909 | 0.916 | 0.877 | 0.870 |
| Jasmine-9B | 0.917 | 0.923 | 0.878 | 0.901 |
| Jasmine-20B | 0.894 | 0.861 | 0.882 | 0.902 |
[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.
W1: How well does our model perform on real-world user visual instructions and OOD Benchmark?
To solve your concern, we construct a new benchmark with 152 samples: 10 samples per original task + 12 new task samples. All the visual instructions are drawn / written by our student participants.
We visualize the model performance on the new real-world benchmark in Figure 6 in our paper, we strongly recommend reviewing it. It involves four challenging scenarios: (1) new hand-drawn symbol + imprecise position annotations; (2) real-user hand-written text + OOD material change requirement; (3) a new task: segmentation; (4) a hybrid task: replace + move.
We also re-run the benchmarking pipeline to get the quantitative metrics on the new real-world benchmark, with results shown in Tab 5. The MLLM-as-a-Judge pipeline covers 140 samples from the new benchmark (10 sample * 14 task). The remaining 12 samples are new tasks, and is scored by human. Our findings are as follows: our model exhibits an acceptable performance drop under noisy real-world visual instructions. Interestingly, the smaller 4B and 9B models exhibit a more obvious drop, while the 20B model remains robust.
Tab 5: Model performance on real-world user visual instructions.
| Model | Data Source | MLLM-as-a-Judge Score |
|---|---|---|
| Jasmine-20B | Original Synthetic Benchmark | 4.61 |
| New Real-World User Benchmark | 4.54 (-0.07) | |
| Jasmine-9B | Original Synthetic Benchmark | 4.49 |
| New Real-World User Benchmark | 4.26 (-0.23) | |
| Jasmine-4B | Original Synthetic Benchmark | 4.10 |
| New Real-World User Benchmark | 3.75 (-0.35) |
In addition to the real-world data, we also evaluate our model on the VIBE-Benchmark [1], which has a distribution completely different from ours. We present the results in Tab 6. The VIBE-Benchmark's input is: text prompt + image with marked target objects, and is the benchmark closest to our input structure.
Our key findings are: (1) Our model exhibits strong generalization capability. Though it performs worse than closed-source models, it surpasses existing open-source ones; (2) We achieve high scores on the first four tasks (Removal, Addition, Replacement, Translation). These tasks appeared in our dataset. However, more importantly, VIBE is a benchmark with a completely different distribution. Our model still achieves competitive scores on a new dataset, demonstrating that it really learned to pay attention to the visual annotations; (3) Especially, we achieve the highest score in the "Translation" task, which is the "Move" task in VIBE-Benchmark; (4) Most performance degradation comes from unseen tasks, which are also challenging to Jasmine-20B's base model before fine-tuning (Qwen-Image-Edit-2511). These tasks are pretty difficult, and go far beyond an unseen symbol or a new material. For instance, the "Reorientation" task requires 3D object rotation; (5) We are very satisfied with our model. Its performance greatly exceeded our expectations. We are not a foundation model team, our model could have potential limitations, please forgive it if it doesn't perform well on new tasks.
Tab 6: Model performance on VIBE-Benchmark. The "Translation" task refers to moving the target object to a designated position, not language translation. Moreover, due to VIBE's official code, the average score of the 10 tasks could not exactly equal to the overall score, as VIBE computes the overall score using the full-precision raw task scores (not two decimal places), which introduces this minor discrepancy.
| Model | Removal | Addition | Replacement | Translation | Pose Control | Reorientation | Draft Instantiation | Light Control | Flow Simulation | Billiards | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|
| FLUX.2-dev | 8.00 | 64.57 | 54.40 | 5.58 | 28.76 | 22.77 | 60.68 | 33.74 | 30.81 | 2.36 | 30.95 |
| Step1X-Edit-v1p2 | 12.59 | 33.92 | 28.17 | 13.52 | - | 25.17 | 71.48 | 25.00 | 29.67 | 0.37 | 20.20 |
| Qwen-Image-Edit-2509 | 14.38 | 55.28 | 30.13 | 14.48 | 15.14 | 17.67 | 21.38 | 28.00 | 44.40 | 2.36 | 23.85 |
| Seedream 4.0 | 93.04 | 74.02 | 79.35 | 33.29 | 58.27 | 30.37 | 72.09 | 47.00 | 43.59 | 4.11 | 51.69 |
| Seedream 4.5 | 95.82 | 81.24 | 81.82 | 48.93 | 66.79 | 20.11 | 82.33 | 50.50 | 45.55 | 2.99 | 55.46 |
| Nano Banana | 93.50 | 81.34 | 79.05 | 46.53 | 67.71 | 33.45 | 85.60 | 34.75 | 52.64 | 1.87 | 55.70 |
| Nano Banana Pro | 94.07 | 82.17 | 88.26 | 74.80 | 72.33 | 36.04 | 88.02 | 60.34 | 59.25 | 15.92 | 65.15 |
| Jasmine-20B | 91.54 | 77.51 | 82.48 | 87.09 | 17.64 | 18.72 | 20.42 | 32.28 | 52.34 | 1.58 | 48.67 |
[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.
Q1: How well are the training dataset, validation dataset, and benchmark separated from each other (in the aspect of the source images, source instructions, and handwriting assets)? What checks exclude near-duplicate examples across splits?
(1) Source Image: The training dataset, validation dataset, and benchmark have different original images and edited images, with zero overlapping.
(2) Source Instruction: For Tone Change, Style Change, Zoom Out, and Outpainting, the text instructions have overlapping parts, as the requirements for these tasks are pretty simple (e.g. "Change the time to evening"), which indeed don't need much changes. For other tasks, we deliberately included a small set of samples which have overlapping instructions during the human selection process, as their image quality was genuinely high, and the overlapping quantities are detailed in Tab 1. We also calculate DINO cosine similarity of the original images between every two samples with the same text instructions, as shown in Tab 2. The similarity is pretty low, indicating these images are not duplicate.
Tab 1: Text instruction overlapping quantities.
Tab 2: DINO cosine similarity of the original images between every two samples with the same text instructions.
(3) Handwriting Assets: To be honest, handwriting TTF style files and hand-drawn symbol templates are reused across splits. We understand your concern that this may cause generalization problems, and we indeed have an obligation to address this concern. Thus, we constructed a new real-world benchmark with real-human noisy handwriting and unseen hand-drawn symbols (please see the 3rd response "W1"). This benchmark also contains imprecise target position annotations, new unseen materials, and even new tasks. We hope this benchmark can supplement our previous benchmark which reuses handwriting TTF files and hand-drawn symbol templates.
Regarding the near-duplicate example concern, we have asked our student participants to avoid duplicate or near-duplicate sampling at the very beginning during the human selection process. Our first author has also double-checked every sample in the validation dataset and benchmark to avoid duplicate samples. Moreover, our statistics in Tab 1 & Tab 2 can also solve this concern.