acceptodds
Under review as a conference paper at ICLR 2027

Image Editing without Typing

Abstract

Text-guided Image Editing (TIE) uses a typing box to replace the button-driven interfaces in traditional software such as Adobe Photoshop, offering remarkable convenience. However, it suffers from a critical limitation: on mobile devices, the virtual keyboard occupies nearly half of the screen, which occludes or resizes the image to be edited and deteriorates user experience. We propose a potential new framework: the text box is directly eliminated, and users express all editing requirements by drawing or writing on the image with a virtual pen. We present the Jasmine Project, involving: (1) a new task NoTIE: No-Typing Image Editing. We define 14 subtasks, and conduct a user survey to identify how users express editing demands using the virtual pen; (2) a new dataset Jasmine-Dataset-15K: a high-quality 2K-resolution dataset, strictly filtered by real humans with a 50.4% high discard rate, with Style Change subtask samples exclusively reviewed by art-major students; (3) a new benchmark Jasmine-Benchmark-1K: 1,381 high-quality human-selected samples, equipped with an MLLM-as-a-Judge evaluation pipeline. We benchmark 19 SOTA models and our 3 models. All evaluation prompts are twice the length of the previous TIE work, and all scoring criteria are derived from the first-hand feedback during the human filtering process; (4) a new model series Jasmine-Model-Family: a Thinker-Editor architecture that outperforms all open-source models and ByteDance's Seedream series. We hope this work can make image editing more convenient on mobile devices, and we call for community help to extend this work.

Primary area:
datasets and benchmarks
Keywords:
image editing, pure visual prompting, hand-drawn symbols, hand-written text, typing-free editing, mobile-device-friendly editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ujoTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces image editing through handwritten text and visual marks, supported by the Jasmine dataset, benchmark, and Thinker–Editor models. Its main contribution is a task and evaluation resource rather than a new model architecture.

Critical strengths
  • The dataset covers 14 editing tasks and includes substantial manual quality control.
  • Evaluation separates instruction understanding from editing, helping identify where existing models fail.
Critical weaknesses
  • Training and evaluation share the same construction pipeline. The very high text-overlap scores do not establish generalization to unfamiliar handwriting or independently collected instructions.
  • The benchmark relies on a single automated judge without reported agreement with human assessments of model outputs. Human filtering of dataset examples does not resolve this concern.
Questions for authors
  1. Clarification: Were splits separated by original source image, source instruction, and handwriting assets? What checks exclude near-duplicate examples across splits?
  2. Evidence request: How well do the judge's scores and model rankings agree with independent human ratings, particularly for spatial editing tasks?
6 comments

Q1: How well are the training dataset, validation dataset, and benchmark separated from each other (in the aspect of the source images, source instructions, and handwriting assets)? What checks exclude near-duplicate examples across splits?

(1) Source Image: The training dataset, validation dataset, and benchmark have different original images and edited images, with zero overlapping.

(2) Source Instruction: For Tone Change, Style Change, Zoom Out, and Outpainting, the text instructions have overlapping parts, as the requirements for these tasks are pretty simple (e.g. "Change the time to evening"), which indeed don't need much changes. For other tasks, we deliberately included a small set of samples which have overlapping instructions during the human selection process, as their image quality was genuinely high, and the overlapping quantities are detailed in Tab 1. We also calculate DINO cosine similarity of the original images between every two samples with the same text instructions, as shown in Tab 2. The similarity is pretty low, indicating these images are not duplicate.

Tab 1: Text instruction overlapping quantities.

Which Two SplitsHow many text instructions are the same
Training Dataset & Validation Dataset32
Training Dataset & Benchmark76
Validation Dataset & Benchmark4

Tab 2: DINO cosine similarity of the original images between every two samples with the same text instructions.

Which Two SplitsAverage DINO (with 4 tasks)Average DINO (without 4 tasks)
Training Dataset & Validation Dataset0.3190.271
Training Dataset & Benchmark0.3460.287
Validation Dataset & Benchmark0.2830.245

(3) Handwriting Assets: To be honest, handwriting TTF style files and hand-drawn symbol templates are reused across splits. We understand your concern that this may cause generalization problems, and we indeed have an obligation to address this concern. Thus, we constructed a new real-world benchmark with real-human noisy handwriting and unseen hand-drawn symbols (please see the 3rd response "W1"). This benchmark also contains imprecise target position annotations, new unseen materials, and even new tasks. We hope this benchmark can supplement our previous benchmark which reuses handwriting TTF files and hand-drawn symbol templates.

Regarding the near-duplicate example concern, we have asked our student participants to avoid duplicate or near-duplicate sampling at the very beginning during the human selection process. Our first author has also double-checked every sample in the validation dataset and benchmark to avoid duplicate samples. Moreover, our statistics in Tab 1 & Tab 2 can also solve this concern.

Q2 & W2: How is the agreement between Gemini judge and real human raters?

We would like to answer this question as comprehensive as we can, as it is asked by all of our reviewers. We re-invited our student participants to conduct human rating, and calculated the average, ranking, and the Pearson correlation between scores from the Gemini judge and real human raters. The results are presented in Tab 3.

We present the experiment process of Tab 3: (1) We referenced a paper named VIBE [1]. This work selected 2 models under benchmarking, with 100 randomly sampled outputs per model (10 samples per task across 10 tasks), to compute Pearson correlation between LLM-based judge and human raters; (2) We select 8 models. Besides our own models, the remaining models are the highest or lowest scoring models in each score band from Table 1 in our paper (e.g., Nano-Banana is the top scorer in the range 3.00~3.99, its score is 3.82). Thus, we cover all score bands, making our results as solid as possible; (3) Due to the limited time, we randomly select 280 outputs from each model (20 per task) for human rating. However, our sampling scale is still larger than VIBE; (4) We compute the MLLM average score, human average score, original ranking, current ranking, and the Pearson correlation for each model using 280 Gemini scores and 280 human scores.

We have the following findings: (1) The difference between human scores and Gemini scores is acceptable. More importantly, the relative rankings are maintained. Since these models are the highest or lowest scoring models in each score band, this indicates humans and Gemini are quite consistent in distinguishing which range the model's performance should fall on the score band; (2) Pearson correlation across all models ranges from 0.85 to 0.95, indicating strong agreement between Gemini Judge and real human raters; (3) Weaker models (e.g., Step1X-Edit) show particularly high correlation. It seems the Gemini Judge and human raters are quite consistent on determining what kind of outputs should get low scores; (4) We notice that you and another reviewer expressed interest in the Spatial Editing tasks. Thus, we conducted additional human rating on 200 samples on Spatial Editing tasks (40 samples per task) for our models. The results are presented in Tab 4, demonstrating that agreement between Gemini Judge and human raters remains strong.

Tab 3: Agreement between Gemini judge and real human raters. We compute the MLLM average score, human average score, original ranking, current ranking, and the Pearson correlation for each model using Gemini and human scores of 280 randomly selected samples.

ModelMLLM RatingHuman RatingOriginal RankingCurrent RankingPearson Correlation
Step1X-Edit2.041.89880.958
FLUX.2-klein-base-4B2.852.69770.924
Qwen-Image-Edit-25093.363.08660.916
Nano-Banana3.753.70550.934
Jasmine-4B4.203.84440.893
Jasmine-9B4.334.12330.907
Jasmine-20B4.574.41220.881
GPT-Image-24.754.56110.859
Overall3.733.54--0.909

Tab 4: More agreement results on Spatial Editing tasks (metric: Pearson correlation).

ModelMoveZoom InZoom OutOutpainting
Jasmine-4B0.9090.9160.8770.870
Jasmine-9B0.9170.9230.8780.901
Jasmine-20B0.8940.8610.8820.902

[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.

W1: How well does our model perform on real-world user visual instructions and OOD Benchmark?

To solve your concern, we construct a new benchmark with 152 samples: 10 samples per original task + 12 new task samples. All the visual instructions are drawn / written by our student participants.

We visualize the model performance on the new real-world benchmark in Figure 6 in our paper, we strongly recommend reviewing it. It involves four challenging scenarios: (1) new hand-drawn symbol + imprecise position annotations; (2) real-user hand-written text + OOD material change requirement; (3) a new task: segmentation; (4) a hybrid task: replace + move.

We also re-run the benchmarking pipeline to get the quantitative metrics on the new real-world benchmark, with results shown in Tab 5. The MLLM-as-a-Judge pipeline covers 140 samples from the new benchmark (10 sample * 14 task). The remaining 12 samples are new tasks, and is scored by human. Our findings are as follows: our model exhibits an acceptable performance drop under noisy real-world visual instructions. Interestingly, the smaller 4B and 9B models exhibit a more obvious drop, while the 20B model remains robust.

Tab 5: Model performance on real-world user visual instructions.

ModelData SourceMLLM-as-a-Judge Score
Jasmine-20BOriginal Synthetic Benchmark4.61
New Real-World User Benchmark4.54 (-0.07)
Jasmine-9BOriginal Synthetic Benchmark4.49
New Real-World User Benchmark4.26 (-0.23)
Jasmine-4BOriginal Synthetic Benchmark4.10
New Real-World User Benchmark3.75 (-0.35)

In addition to the real-world data, we also evaluate our model on the VIBE-Benchmark [1], which has a distribution completely different from ours. We present the results in Tab 6. The VIBE-Benchmark's input is: text prompt + image with marked target objects, and is the benchmark closest to our input structure.

Our key findings are: (1) Our model exhibits strong generalization capability. Though it performs worse than closed-source models, it surpasses existing open-source ones; (2) We achieve high scores on the first four tasks (Removal, Addition, Replacement, Translation). These tasks appeared in our dataset. However, more importantly, VIBE is a benchmark with a completely different distribution. Our model still achieves competitive scores on a new dataset, demonstrating that it really learned to pay attention to the visual annotations; (3) Especially, we achieve the highest score in the "Translation" task, which is the "Move" task in VIBE-Benchmark; (4) Most performance degradation comes from unseen tasks, which are also challenging to Jasmine-20B's base model before fine-tuning (Qwen-Image-Edit-2511). These tasks are pretty difficult, and go far beyond an unseen symbol or a new material. For instance, the "Reorientation" task requires 3D object rotation; (5) We are very satisfied with our model. Its performance greatly exceeded our expectations. We are not a foundation model team, our model could have potential limitations, please forgive it if it doesn't perform well on new tasks.

Tab 6: Model performance on VIBE-Benchmark. The "Translation" task refers to moving the target object to a designated position, not language translation. Moreover, due to VIBE's official code, the average score of the 10 tasks could not exactly equal to the overall score, as VIBE computes the overall score using the full-precision raw task scores (not two decimal places), which introduces this minor discrepancy.

ModelRemovalAdditionReplacementTranslationPose ControlReorientationDraft InstantiationLight ControlFlow SimulationBilliardsOverall
FLUX.2-dev8.0064.5754.405.5828.7622.7760.6833.7430.812.3630.95
Step1X-Edit-v1p212.5933.9228.1713.52-25.1771.4825.0029.670.3720.20
Qwen-Image-Edit-250914.3855.2830.1314.4815.1417.6721.3828.0044.402.3623.85
Seedream 4.093.0474.0279.3533.2958.2730.3772.0947.0043.594.1151.69
Seedream 4.595.8281.2481.8248.9366.7920.1182.3350.5045.552.9955.46
Nano Banana93.5081.3479.0546.5367.7133.4585.6034.7552.641.8755.70
Nano Banana Pro94.0782.1788.2674.8072.3336.0488.0260.3459.2515.9265.15
Jasmine-20B91.5477.5182.4887.0917.6418.7220.4232.2852.341.5848.67

[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.

Reviewer_CeR6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes No-Typing Image Editing (NoTIE), in which images are edited without any text box, only by drawing and writing directly on the image. For instance, to remove an object, the user circles it and crosses it out. To change its color, the user circles it and writes the name of the new color. The motivation behind this task is that, on mobile devices, the virtual keyboard occupies nearly half of the screen, which occludes or resizes the image being edited and "deteriorates user experience". The authors contribute the task itself (which comprises 14 subtasks), a dataset, a benchmark, and a two-stage pipeline: (i) a Thinker that reads the marked-up image and writes a text instruction, and (ii) an image editing model (the Editor) that performs the actual edit.

Critical strengths
  1. The idea of NoTIE is novel and underexplored.
  2. Good data engineering. Building the dataset and the benchmark required a different pipeline for each task. For instance, the Add samples are created by running the Remove process in reverse. I appreciate the amount of work behind this, and I find both the data and the detailed statistics useful.
  3. Well-documented training. The training process is described in detail, and the experiments on oversampling the spatial tasks are useful and interesting.
Critical weaknesses
  1. Does NoTIE really make sense? At first, I thought it made sense for tasks such as moving or removing objects. However, for most of the other subtasks (material, color, background, tone, style, replacement, appearance, and text change), the user needs to write with a pen on the screen. My concern is that handwriting on a small screen may be even harder than typing on a virtual keyboard that appears and disappears. For example, to change the background, the user has to write the new background on the screen, and the same applies to the desired color, material, etc. I would like to see a user study comparing NoTIE with typed instructions (e.g., task completion time, success rate, and user preference) to assess whether this interaction is actually beneficial.
  2. Handwritten and out-of-distribution samples. The benchmark is well built and, as I said in the Strengths section, I recognize the amount of work behind it and its usefulness (even for traditional image editing methods). However, most subtasks require the user to write on the screen, and all the handwriting in the dataset and benchmark is synthetic. It is rendered from a fixed set of handwriting fonts rather than written by real users. It would be valuable to have a real benchmark in which users are shown the original and target images and asked to draw and write the instructions themselves.
  3. Comparison. I am concerned about the fairness of the comparison. The proposed method is the best among the open-source models, but GPT-Image-2 and the Nano-Banana models still score higher overall (e.g., 4.80 for GPT-Image-2 vs. 4.61 for Jasmine-20B), and they are the only models that solve the task end-to-end from the marked-up image, using their own reasoning. The remaining open-source and Seedream models receive the ground-truth edit instruction as text, so they skip the step of understanding the drawings. In contrast, the authors fine-tune their model specifically for NoTIE on in-distribution data, with a dedicated two-stage pipeline whose Thinker (Qwen3-VL) reproduces the ground-truth instruction almost verbatim (BLEU-4 ≈ 97, Table 2). Therefore, the models in Table 1 are not solving the same task.
  4. Aesthetics, overclaims, and tone. I find some of the paper's visual choices inappropriate for a research paper. The "Appearance Alter" and "Text Change" examples in Figure 2 (showing a weapon and an odd advertisement) are not the most suitable examples for a scientific paper. Also, I am fine with the name "Jasmine", but there is no need to add flowers everywhere in the paper as it looks unprofessional. Regarding overclaims and tone, the authors state in Appendix B (L697): "We realize that our work can change the world", and in Section 3 (L160): "..., driven by their interest and enthusiasm". Such statements are more appropriate for coursework than for a research paper at a top conference.
Questions for authors
  1. How well does the MLLM judge agree with human raters? Have you measured this agreement?
  2. Why do Tables 1 and 8 report different scores for the same model (Jasmine-20B with 2× sampling)? For example, Zoom Out is 4.06 in Table 1 but 4.17 in Table 8.
10 comments

Q1: How is the agreement between Gemini judge and real human raters?

We re-invited our student participants to conduct human rating, and calculated the Pearson correlation between scores from the Gemini judge and real human raters. The results are presented in Tab 1.

We present the experiment process of Tab 1: (1) We referenced a paper named VIBE [1]. This work selected 2 models under benchmarking, with 100 randomly sampled outputs per model (10 samples per task across 10 tasks), to compute Pearson correlation between LLM-based judge and human raters; (2) We select 8 models. Besides our own models, the remaining models are the highest or lowest scoring models in each score band from Table 1 in our paper (e.g., Nano-Banana is the top scorer in the range 3.00~3.99, its score is 3.82). Thus, we cover all score bands, making our results as solid as possible; (3) We randomly select 280 outputs from each model (20 per task) for human rating. Our sampling scale is much larger than VIBE; (4) We compute the Pearson correlation for each model using 280 Gemini scores and 280 human scores.

We have the following findings: (1) Pearson correlation across all models ranges from 0.85 to 0.95, indicating strong agreement between Gemini Judge and real human raters; (2) Weaker models (e.g., Step1X-Edit) show particularly high correlation. It seems the Gemini Judge and human raters are quite consistent on determining what kind of outputs should get low scores; (3) We notice that two of our reviewers expressed particular interest in the MLLM-human agreement in Spatial Editing tasks. We conducted additional human rating on 200 samples on Spatial Editing tasks (40 samples per task) for our models, with the results presented in Tab 2. We would like to present these results for your reviewing as well.

Tab 1: Agreement between Gemini judge and real human raters (metric: Pearson correlation).

ModelPearson Correlation
Step1X-Edit0.958
FLUX.2-klein-base-4B0.924
Qwen-Image-Edit-25090.916
Nano-Banana0.934
Jasmine-4B0.893
Jasmine-9B0.907
Jasmine-20B0.881
GPT-Image-20.859
Overall0.909

Tab 2: More agreement results on Spatial Editing tasks (metric: Pearson correlation).

ModelMoveZoom InZoom OutOutpainting
Jasmine-4B0.9090.9160.8770.870
Jasmine-9B0.9170.9230.8780.901
Jasmine-20B0.8940.8610.8820.902

[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.

Q2: Why scores are different for the same model in Table 1 and Table 8?

In Table 1 of our paper, Jasmine-20B is a composite system (32B Thinker + 20B Editor), and the Editor receives text guidance generated by Thinker. In Table 8, Jasmine-Editor-20B is a standalone model, with text guidance coming from GT editing instructions. They are actually not the same system (Thinker+Editor vs. GT+Editor) and is separately benchmarked, thus the scores are different. Moreover, Editor receives Thinker's output in Table 1 because we want to evaluate the whole composite system, and receives GT in Table 8 because we want to see the impact of different spatial task sampling times to the Editor so we need to keep the text instructions unchanged (thus we use GT).

This is our problem, we apologize for causing the misunderstanding. Thank you, sincerely, you must have read our paper carefully. We have clarified the differences in our paper (line 1258 blue text).

W1: Does NoTIE really make sense?

We notice that your main concern is users may have difficulty writing on small screens, we would like to answer this question as follows: (1) We have two types of mobile users: phone users & tablet users; (2) We acknowledge phone users will encounter small-screen writing problems. Actually, at the very beginning, we planned to provide only a lightweight, drawing-based model for phone users (only support symbols like "×", no handwriting). While, later, iPhone Duo released. We began to realize that currently small mobile devices (e.g. phone) could become large in the future, and someone may really want to write on it. This is why we ultimately provide a full-function version compatible with all mobile devices: make the NoTIE function comprehensive enough, in case users want to use a certain function one day; (3) Tablet users are our main user group at the current stage, especially designers using iPads. This paper [1] discusses the positive attitude of BMW designers towards NoTIE. They are very willing to do their 2D design work on the tablet if the NoTIE function is available.

Moreover, as you suggested, we provide a user survey in Tab 3. We invited 100 people to fill out our questionnaire and 50 people to try GPT-Image-2 and our model. It's really the upper bound of our capability. As shown in Tab 3, most people prefer NoTIE on iPad, and on phones if drawing symbols can already fulfill their demands, and the success rates of NoTIE and Typing are close. However, during the investigation, we found an important finding: users sometimes want to edit multiple objects simultaneously (circling multiple objects). If use Typing, they have to use different colors to circle different objects in order to clearly distinguish them when typing, while NoTIE hasn't this problem. NoTIE allows users to directly use arrows to specify which object should undergo which editing operation.

We also present the model latency of 2B Thinker + 4B Editor in Tab 4. To be honest, the current model is slow, as it has not undergone necessary optimizations. For example, few-step distillation has not been applied. Our model is far from a product, we are more intend to regard it as an early exploration.

Tab 3: User survey result. Here, "Typing" means users can draw circles on the image to locate target objects, but text instructions must be input through keyboard.

QuestionOptionsResult
To edit an image, do you prefer NoTIE or Typing on your iPad?NoTIE / Typing88% choose "NoTIE"
To edit an image, do you prefer NoTIE or Typing on your phone?NoTIE / Typing4% choose "NoTIE"
To edit an image, if drawing symbols can fulfill your demands, do you prefer NoTIE or Typing on your phone?NoTIE / Typing98% choose "NoTIE"
Please try GPT-Image-2. Please input by NoTIE. Do you think the editing result is successful?y / n92% choose "y"
Please try GPT-Image-2. Please input by Typing. Do you think the editing result is successful?y / n86% choose "y"
Please try our model. Please input by NoTIE. Do you think the editing result is successful?y / n74% choose "y"
Please try our model. Please input by Typing. Do you think the editing result is successful?y / n72% choose "y"
How satisfied are you with our model using NoTIE? High score means high satisfaction.1 / 2 / 3 / 4 / 53.86 on average

Tab 4: Model latency of 2B Thinker + 4B Editor.

ModelLatency (seconds per sample)
Jasmine-Thinker-2B0.85
Jasmine-Editor-4B178.72
Overall179.57

[1] Hyerim Park, Malin Eiband, Andre Luckow, and Michael Sedlmair. Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHIEA’25, pp. 1–10. ACM, April 2025. doi: 10.1145/3706599.3719802. URL http://dx.doi.org/10.1145/3706599.3719802.

Reviewer_Y3CJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes No-Typing Image Editing (NoTIE). The edit request is written or drawn directly on the image, for example a circle with a spiral to remove an object, "+object" to add one, or an arrow to move it, rather than typed as text. The motivation is mobile editing, where the on-screen keyboard occludes the image and text is a poor way to say where an edit should happen. The authors define 14 subtasks and construct a 15.7K-sample synthetic dataset, discarding about half of the raw samples during human filtering. They introduce a 1.4K-sample benchmark evaluated by a Gemini-2.5-Pro judge, and train a two-stage model in which a fine-tuned Qwen3-VL converts the annotations into a text instruction that a fine-tuned diffusion editor then executes. The 20B variant is the strongest open-source system on the benchmark and approaches, though does not reach, GPT-Image-2 and the Nano-Banana models.

Critical strengths

The central idea is simple and well motivated. Annotating the image itself is a natural way to specify an edit with a stylus, and the paper makes a reasonable case that this interface deserves study. In my view this is a real problem for mobile editing, although I am not fully convinced that drawing is preferable to spoken instructions in that setting (see Q5). The contribution is nonetheless valuable. The data work is a genuine contribution: the filtering protocol is unusually careful, and reporting discard rates by stage and subtask is more transparent than is typical for synthetic editing datasets. If the dataset and benchmark are released, I expect them to be useful.

The most persuasive results are on the spatial subtasks. On Move, every open-source baseline scores below 2 while Jasmine exceeds 4. That is precisely where a drawn arrow conveys information that a short text instruction cannot, so it supports the paper's premise more directly than the overall average.

Critical weaknesses

My principal concern is the comparison in Table 1, where models are evaluated under different input conditions:

Most baselines receive the ground-truth text instruction. GPT-Image-2 is driven through GPT-5.4. The proposed model uses its own Thinker. Some inconsistency is perhaps unavoidable for a new task, but the table is presented as a single ranking, and I do not think it supports that reading. The issue is most acute for Move. A baseline that sees only text has no access to the target location, whereas Jasmine-Editor also receives the annotated image. Part of the reported gap on the spatial tasks may therefore follow from the protocol rather than from the model.

My second concern is that the evaluation is entirely synthetic. Handwriting is rendered from fonts and shapes from a simulation library, and the training and benchmark data come from the same generation pipeline. The near-ceiling BLEU of the Thinker (approximately 96–97) suggests it has learned to invert that pipeline. Real freehand annotations on a device are likely to be considerably noisier. The Gemini judge is also not validated against human ratings, and some of the reported differences are hard to interpret without that. For instance, Jasmine-20B scores below Jasmine-9B on both Zoom In (4.01 vs. 4.29) and Zoom Out (4.06 vs. 4.29), even though it is clearly stronger elsewhere. That may be a genuine scaling effect, but with roughly 100 samples per task and a single judge, it could equally be evaluation noise, and the paper gives no means of telling the two apart.

I do not consider either issue disqualifying. Both, however, bear directly on the paper's central claims, and my assessment depends on how they are addressed.

Questions for authors

For each model in Table 1, please specify whether it receives the annotated image, the text instruction, or both. A comparison under a common protocol, at least for the spatial subtasks, would most strongly affect my assessment.

What is the agreement between the Gemini judge and human raters on a sample of the benchmark? Confidence intervals per task would also help clarify cases such as the 9B/20B reversal on Zoom In and Zoom Out.

Have the models been evaluated on annotations produced by real users on a device? Results on an existing visual-instruction benchmark such as VIBE, which the paper cites, would also help show the model is not tied to its own data pipeline.

Do the authors intend to release the dataset, benchmark, judge prompts and model weights, and is redistribution permitted under the terms of the image models used to generate the data?

Given the mobile motivation, what is the latency of the smallest configuration (2B Thinker with 4B Editor)? A brief discussion of voice input, which addresses the same keyboard limitation, would also be helpful.

8 comments

Q1 & W1: What are the inputs for each model in Table 1?

We apologize for not clearly listing the model's inputs in the benchmarking process. We have now detailed them in Tab 1 and in our paper (line 1191 blue text). For each sample, each editing model receives a text instruction, an original image, and a visual instruction image. The only difference is how the text instruction is obtained. We understand your main concern that some models may underperform due to missing access to the visual instructions. Actually, all models can see the visual instructions. Especially, in the Move task, the poor performance of open-source models is genuinely due to their failure to utilize the visual annotation information given by us.

We also notice that you hope to have a comparison under a common protocol. According to our understanding, your main demand is that all models should receive the same components (1 text + 2 images). If we are right, Table 1 in our paper is exactly the results you need. However, another reviewer has asked us to further explain why some models skip the demand understanding step, and directly receive GT text instructions. We agree that we have an obligation to explain it (please see the 6th response "A Further Discussion of W1").

Tab 1: Inputs for each model in the benchmarking process.

ModelInput
GPT-Image-2Each model receives: text instruction generated by GPT-5.4 + original image + visual instruction image
Nano-Banana seriesEach model receives: text instruction generated by Nano-Banana + original image + visual instruction image
OursEach model receives: text instruction generated by Jasmine-Thinker + original image + visual instruction image
OthersEach model receives: GT text instruction + original image + visual instruction image

Q2 & W2: How is the agreement between Gemini judge and real human raters?

We would like to answer this question as comprehensive as we can, as it is asked by all of our reviewers. We re-invited our student participants to conduct human rating, and calculated the Pearson correlation between scores from the Gemini judge and real human raters. The results are presented in Tab 2.

We present the experiment process of Tab 2: (1) We referenced a paper named VIBE [1]. This work selected 2 models under benchmarking, with 100 randomly sampled outputs per model (10 samples per task across 10 tasks), to compute Pearson correlation between LLM-based judge and human raters; (2) We select 8 models. Besides our own models, the remaining models are the highest or lowest scoring models in each score band from Table 1 in our paper (e.g., Nano-Banana is the top scorer in the range 3.00~3.99, its score is 3.82). Thus, we cover all score bands, making our results as solid as possible; (3) Due to the limited time, we randomly select 280 outputs from each model (20 per task) for human rating. However, our sampling scale is still larger than VIBE; (4) We compute the Pearson correlation for each model using 280 Gemini scores and 280 human scores.

We have the following findings: (1) Pearson correlation across all models ranges from 0.85 to 0.95, indicating strong agreement between Gemini Judge and real human raters; (2) Weaker models (e.g., Step1X-Edit) show particularly high correlation. It seems the Gemini Judge and human raters are quite consistent on determining what kind of outputs should get low scores; (3) We notice that you and another reviewer expressed interest in the Spatial Editing tasks. Thus, we conducted additional human rating on 200 samples on Spatial Editing tasks (40 samples per task) for our models. The results are presented in Tab 3, demonstrating that agreement between Gemini Judge and human raters remains strong.

We also notice that you want to know why Jasmine-9B achieves higher Zoom In and Zoom Out scores than Jasmine-20B in Table 1 in our paper. We confirm our results are reliable. This discrepancy is more likely because Jasmine-9B and Jasmine-20B are fine-tuned from different base models (Jasmine-9B from FLUX.2-klein-base-9B, Jasmine-20B from Qwen-Image-Edit-2511). However, Jasmine-20B outperforms Jasmine-9B on nearly all other tasks and in overall score.

Tab 2: Agreement between Gemini judge and real human raters (metric: Pearson correlation).

ModelPearson Correlation
Step1X-Edit0.958
FLUX.2-klein-base-4B0.924
Qwen-Image-Edit-25090.916
Nano-Banana0.934
Jasmine-4B0.893
Jasmine-9B0.907
Jasmine-20B0.881
GPT-Image-20.859
Overall0.909

Tab 3: More agreement results on Spatial Editing tasks (metric: Pearson correlation).

ModelMoveZoom InZoom OutOutpainting
Jasmine-4B0.9090.9160.8770.870
Jasmine-9B0.9170.9230.8780.901
Jasmine-20B0.8940.8610.8820.902

[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.

Q3 & W2: How well does our model perform on real-world user visual instructions and OOD Benchmark?

To solve your concern, we construct a new benchmark with 152 samples: 10 samples per original task + 12 new task samples. All the visual instructions are drawn / written by our student participants.

We visualize the model performance on the new real-world benchmark in Figure 6 in our paper, we strongly recommend reviewing it. It involves four challenging scenarios: (1) new hand-drawn symbol + imprecise position annotations; (2) real-user hand-written text + OOD material change requirement; (3) a new task: segmentation; (4) a hybrid task: replace + move.

We also re-run the benchmarking pipeline to get the quantitative metrics on the new real-world benchmark, with results shown in Tab 4. The MLLM-as-a-Judge pipeline covers 140 samples from the new benchmark (10 sample * 14 task). The remaining 12 samples are new tasks, and is scored by human. Our findings are as follows: our model exhibits an acceptable performance drop under noisy real-world visual instructions. Interestingly, the smaller 4B and 9B models exhibit a more obvious drop, while the 20B model remains robust.

Tab 4: Model performance on real-world user visual instructions.

ModelData SourceMLLM-as-a-Judge Score
Jasmine-20BOriginal Synthetic Benchmark4.61
New Real-World User Benchmark4.54 (-0.07)
Jasmine-9BOriginal Synthetic Benchmark4.49
New Real-World User Benchmark4.26 (-0.23)
Jasmine-4BOriginal Synthetic Benchmark4.10
New Real-World User Benchmark3.75 (-0.35)

In addition to the real-world data, we also follow your suggestion to evaluate our model on the VIBE-Benchmark [1], which has a distribution completely different from ours. We present the results in Tab 5. The VIBE-Benchmark's input is: text prompt + image with marked target objects, and is the benchmark closest to our input structure.

Our key findings are: (1) Our model exhibits strong generalization capability. Though it performs worse than closed-source models, it surpasses existing open-source ones; (2) We achieve high scores on the first four tasks (Removal, Addition, Replacement, Translation). These tasks appeared in our dataset. However, more importantly, VIBE is a benchmark with a completely different distribution. Our model still achieves competitive scores on a new dataset, demonstrating that it really learned to pay attention to the visual annotations; (3) Especially, we achieve the highest score in the "Translation" task, which is the "Move" task in VIBE-Benchmark; (4) Most performance degradation comes from unseen tasks, which are also challenging to Jasmine-20B's base model before fine-tuning (Qwen-Image-Edit-2511). These tasks are pretty difficult, and go far beyond an unseen symbol or a new material. For instance, the "Reorientation" task requires 3D object rotation; (5) We are very satisfied with our model. Its performance greatly exceeded our expectations. We are not a foundation model team, our model could have potential limitations, please forgive it if it doesn't perform well on new tasks.

Tab 5: Model performance on VIBE-Benchmark. The "Translation" task refers to moving the target object to a designated position, not language translation. Moreover, due to VIBE's official code, the average score of the 10 tasks could not exactly equal to the overall score, as VIBE computes the overall score using the full-precision raw task scores (not two decimal places), which introduces this minor discrepancy.

ModelRemovalAdditionReplacementTranslationPose ControlReorientationDraft InstantiationLight ControlFlow SimulationBilliardsOverall
FLUX.2-dev8.0064.5754.405.5828.7622.7760.6833.7430.812.3630.95
Step1X-Edit-v1p212.5933.9228.1713.52-25.1771.4825.0029.670.3720.20
Qwen-Image-Edit-250914.3855.2830.1314.4815.1417.6721.3828.0044.402.3623.85
Seedream 4.093.0474.0279.3533.2958.2730.3772.0947.0043.594.1151.69
Seedream 4.595.8281.2481.8248.9366.7920.1182.3350.5045.552.9955.46
Nano Banana93.5081.3479.0546.5367.7133.4585.6034.7552.641.8755.70
Nano Banana Pro94.0782.1788.2674.8072.3336.0488.0260.3459.2515.9265.15
Jasmine-20B91.5477.5182.4887.0917.6418.7220.4232.2852.341.5848.67

[1] Huanyu Zhang, Xuehai Bai, Chengzu Li, et al. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing, 2026. URL https://arxiv.org/abs/2602.01851.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.