- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses model routing for instruction-based image editing. Since editing models differ in which requests they handle well and in latency, the paper proposes to pick a model per request from the input image and instruction alone. The proposed method, EditJev, adds a small preference head on a VLM backbone that scores every candidate model in one forward pass, and combines these scores with a weighted speed score at inference. The authors claim that on two benchmarks it slightly outperforms the strongest fixed model while greatly reducing latency.
Critical strengths
S1. The problem setting is practical and clearly stated. Candidate editing models differ greatly in latency, so choosing a model per request is a meaningful problem for deployment. The authors formulate it cleanly as an objective over both quality and latency. The evaluation protocol, which precomputes benchmark results for all candidates and evaluates routing by lookup, is reproducible and can be reused directly by follow-up work.
S2. The method is explained clearly. The text and figures (especially Figure 2) make the idea easy to follow, and the method is evaluated on two benchmarks with several ablations.
Critical weaknesses
W1. The presented evidence cannot separate the benefit of sample-level routing from a simple category rule. If each model is clearly better at certain editing types, a rule built from that pattern would likely perform similarly without training a head. An MLLM given the per-model characteristics as meta information might do the same. The paper does not analyze this pattern or check whether the actual routing results follow it. It also has no comparison with rule-based routing, random routing or a per-request oracle. Without these I cannot see why a trained head is needed. Moreover, Figures 1 and 3 and the text describe the method as if each model has its own area of strength and EditJev sends each request to the model that edits it best. The quantitative results show nothing beyond choosing the faster model when the preferences are similar. Latency does drop substantially. For quality alone, however, using Qwen-Image-Edit by itself is practically sufficient. Either the claim should be narrowed or the paper needs analysis showing that routing does more than trade off latency.
W2. The differences in the results are marginal but interpreted as large, and the comparison is not fair. Quality is almost the same as the single model, and only the latency reduction can be credited. Claims that the quantitative results do not support are backed by qualitative examples instead. The ablation variants show almost no difference compared with the score gaps in Table 1, yet the authors draw conclusions from them. In addition, the MLLM routing baselines choose without knowing anything about the individual models, and I expect them to be close to random. Comparing them with the trained EditJev does not show whether a trained head is necessary.
Questions for authors
- Can you summarize which editing types each model is good at and show whether the actual routing results follow that pattern? What is the performance when routing by rule using this pattern, or when the per-model characteristics are given to an MLLM as meta information? Please also report a per-request oracle and random routing.
- Can you show that the score differences in the main results and ablations are meaningful? Variance or confidence intervals over seeds or repeated scoring are needed.
- Is the VIEScore-based preference score reliable enough? In my experience an MLLM judge gives different scores even for the same sample. If each candidate output was scored separately, there is no guarantee that the scores are comparable across candidates.
- Why do the hyperparameters differ so much between benchmarks, and how were they chosen? Pairs with a score difference below 0.5 are discarded in training. Does this not make small score differences hard for the model to predict?
- The min-max normalized speed score depends heavily on the candidate set. A single very fast but low-quality model can compress the speed scores of the remaining models. An analysis of this would be helpful.
- Please show failure cases of the routing, and discuss the deployment cost of keeping all candidates loaded on the server as a limitation.