acceptodds
Withdrawn from ICLR 2027

UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The evaluation of visual editing models remains fragmented across methods and modalities. Existing benchmarks are often tailored to specific paradigms, making fair cross-paradigm comparisons difficult, while video editing lacks reliable evaluation benchmarks. Furthermore, common automatic metrics often misalign with human preference, yet directly deploying large multimodal models (MLLMs) as evaluators incurs prohibitive computational and financial costs. We present UniEditBench, a unified benchmark for image and video editing that supports reconstruction-based and instruction-driven methods under a shared protocol. UniEditBench includes a structured taxonomy of nine image operations (Add, Remove, Replace, Change, Stroke-based, Extract, Adjust, Count, Reorder) and eight video operations, with coverage of challenging compositional tasks such as counting and spatial reordering. To enable scalable evaluation, we distill a high-capacity MLLM judge (Qwen3-VL-235B-A22B Instruct) into lightweight 4B/8B evaluators that provide multi-dimensional scoring over structural fidelity, text alignment, background consistency, naturalness, and temporal-spatial consistency (for videos). Experiments show that the distilled evaluators maintain strong agreement with human judgments and substantially reduce deployment cost relative to the teacher model. UniEditBench provides a practical and reproducible protocol for benchmarking modern visual editing methods.

Primary area:
datasets and benchmarks
Keywords:
Visual Editing, Unified Benchmark, Image and Video Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k848OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UniEditBench, a benchmark and distilled MLLM evaluation framework for joint image and video editing assessment. The benchmark curates 710 source instances (633 images and 77 videos) annotated with a unified prompt-triplet schema (psrc,ptgt,u)(p_{\text{src}}, p_{\text{tgt}}, u) (source description, target description, and editing instruction) across 9 image and 8 video editing operations (Add, Adjust, Change, Count, Extract, Remove, Reorder, Replace, and image-only Stroke). Using 25 image editors and 8 video editors, the authors generate 22,554 edited outputs and score them with a large open-weight teacher model (Qwen3-VL-235B-A22B) across 4 image dimensions (Structural Fidelity, Text-Image Alignment, Background Consistency, Naturalness) and 5 video dimensions (adding Temporal-Spatial Consistency). To reduce evaluation cost, the authors distill this teacher into compact Qwen3-VL-4B and Qwen3-VL-8B student models via a two-stage LoRA SFT curriculum (Stage 1 on image edits, Stage 2 on video edits).

Critical strengths
  1. Unified prompt-triplet schema across editing paradigms and modalities: Annotating each benchmark instance with (psrc,ptgt,u)(p_{\text{src}}, p_{\text{tgt}}, u) enables side-by-side evaluation of description-based (inversion/inversion-free), instruction-driven, and mask/stroke-guided editors across both image and video modalities under a shared operation taxonomy and multi-dimensional rubric.
  2. Practical motivation for lightweight open-source evaluators: Distilling a 235B MoE judge (requiring 8 GPUs / ~471 GB BF16 storage) into single-GPU 4B/8B evaluators addresses a genuine pain point in reproducible, low-cost benchmarking for visual editing.
Critical weaknesses
  1. Severe train-test data leakage Concern in evaluator distillation and benchmark scoring: The paper collects 710 source instances (633 images, 77 videos) and runs 25 image editors and 8 video editors to obtain 22,554 edited outputs. Nowhere in Section 3.3, Section 4, or the Appendices is a disjoint train/validation/test split of the 710 source instances or the 22,554 outputs defined! It appears that the 4B/8B student evaluators are distilled on the teacher's scores over the 22,554 benchmark outputs, and then evaluated for teacher-imitation MSE (Tables 3, 5–8) and used to produce the main benchmark leaderboards (Tables 1–2) on the exact same 710 instances / 22,554 outputs. Appendix I even acknowledges that "Editor overlap alone does not prove sample leakage; conversely, it cannot establish source-disjoint evaluation." Evaluating distillation MSE and benchmarking models on the training set of the distilled evaluator invalidates claims of out-of-distribution evaluator generalization.
  2. Insufficient human validation and tiny video benchmark scale: First, the video subset contains only 77 source videos across 8 editing operations (fewer than 10 videos per operation on average), which is far too small for a comprehensive video editing benchmark compared to existing suites (e.g.,IVEBench, FiVE-Bench). Second, the human validation study (Figure 6) only plots macroscopic average radar profiles across 5 models; as the authors themselves admit in Appendix I (lines 2178–2181), "the aggregate figure does not establish a named correlation coefficient, inter-rater agreement, or a teacher–human baseline." Without instance-level Spearman/Kendall rank correlation or pairwise agreement between the 235B teacher / 4B-8B students and human raters, low MSE with the teacher only shows that the student imitates the teacher, not that either is a valid proxy for human judgment.
  3. Unjustified two-stage curriculum claim: The paper claims a "two-stage progressive distillation strategy" (Image SFT followed by Video SFT) as a core methodological contribution that prevents catastrophic forgetting. However, as admitted in Appendix J, the experiments do not compare against joint image+video training, video-only training, or compute-matched baselines, and Stage 2 simply runs 3 additional epochs on video data only (which actually degrades image Extract MSE from 1.01 to 1.23 and worsens 8B video MSE relative to 4B: 0.79 vs. 0.73).
Questions for authors
  1. Train/Test split for evaluator distillation (Clarification — Critical): Can the authors explicitly clarify how the 710 source instances and 22,554 edited outputs were partitioned between LoRA SFT distillation training and the MSE / leaderboard evaluations in Tables 1, 2, and 3? Were the 4B/8B evaluators trained on the same 710 benchmark instances (or a subset thereof) on which Tables 1–3 are reported? If not, what are the exact train and test split sizes, and are they disjoint by source image/video and by editing model?
  2. Quantitative human-alignment metrics (Clarification): Can the authors compute and report the sample-level Spearman rank correlation (ρ\rho) and Kendall τ\tau (per dimension and overall) between human ratings, the Qwen3-VL-235B-A22B teacher, and the distilled 4B/8B students on the human-evaluated subset?
  3. Comparison with joint Image+Video SFT (Clarification): How does sequential two-stage SFT (3 epochs Image →\to 3 epochs Video) compare against standard joint multi-task SFT on mixed image and video evaluation data under an equal training token budget?
Reviewer_KXxROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniEditBench, which is a unified benchmark for image and video editing supporting reconstruction-based and instruction-driven methods, covering nine image and eight video editing operations. It distills a large MLLM judge into lightweight 4B/8B evaluators for multidimensional assessment, aiming to maintain agreement with human judgments while reducing deployment costs.

Critical strengths
  1. This work proposes a unified benchmark for image and video editing that supports reconstruction-based and instruction-driven methods under a shared protocol.
  2. This paper distills a high-capacity MLLM judge into lightweight 4B/8B evaluators that provide multi-dimensional scoring over structural fidelity, text alignment, background consistency, naturalness, and temporal-spatial consistency
Critical weaknesses
  1. The central concern is whether the proposed evaluators reliably measure editing quality. Table 3 primarily demonstrates agreement with the teacher. The consistency with human seems missing.
  2. The video coverage is limited. Only 77 video sources cover eight operations, leaving relatively few independent examples per operation.
  3. Is teacher model (Qwen3-VL-235B-A22B Instruct) aligned well with human ratings? If yes, please prove the alignment. If not, why it can be used as a teacher model?
  4. The quality of the figures in this paper is not good. Especially, Figure 2 likes a pure AI-generated figure.
Questions for authors

What are the item-level human correlations and pairwise ranking accuracies of the teacher, zero-shot students, and distilled students, separately for each modality and evaluation dimension?

Reviewer_um2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses two problems with evaluating text-guided image and video editing. Benchmarks are fragmented across modalities, and large MLLM judges are expensive to run. It introduces UniEditBench, which has 710 source instances (633 images and 77 videos). These are organized into 9 image operations and 8 video operations, and each instance carries a source prompt, a target prompt and an instruction triplet (Sec. 3.1–3.2). Each edit is scored on an integer 0–5 scale for Structural Fidelity, Text Alignment, Background Consistency and Naturalness, plus Temporal-Spatial Consistency for video, and the Overall score is their unweighted mean (Sec. 3.3, Eq. 1). To lower the cost of evaluation, the authors distill a Qwen3-VL-235B-A22B teacher into 4B and 8B students. They use two-stage LoRA (image first, then video) and supervise on both rationales and scores (Sec. 3.4). They report a drop in teacher-score MSE from 2.95/2.51 to 0.49/0.47 on images and from 5.40/4.00 to 0.73/0.79 on video (Table 3). They also benchmark 25 image editors and 8 video editors (Tables 1–2), present a 50-participant human rating study at the level of aggregate profiles (Fig. 6), and compare deployment costs (Table 4).

Critical strengths
  1. One taxonomy and one rubric for both image and video editing, plus a large evaluation of editors. Sec. 3.2 defines 9 image and 8 video operations, and Sec. 3.3 / Eq. 1 apply the same dimension-wise rubric to both modalities. With this setup the paper evaluates 25 image editors and 8 video editors, spanning reconstruction-based (inversion- and flow-based) and instruction-driven methods (Tables 1–2). Prior benchmarks such as EditVal, I2EBench, TGVE and FiVE-Bench each target a single modality, so a shared protocol that can be inspected per operation would be a useful resource for the community.
Critical weaknesses
  1. The distilled judge is never validated against human judgments, even though the paper's main claim depends on it. The Abstract states that the distilled evaluators "maintain strong agreement with human judgments", whereas Sec. 1 only describes the user study as supplying "complementary human score profiles", and the only human evidence is the aggregate radar profile in Fig. 6 (50 participants, 5 groups). App. I acknowledges that the study provides no named correlation coefficient, no inter-rater agreement and no teacher–human baseline. It also acknowledges that the human-study editors overlap with those used to build the teacher-scoring data and that the evaluation is not source-disjoint. All quantitative judge validation (Table 3, Table 5, Table 10) measures agreement with the teacher, which is in effect treated as ground truth. What is missing is per-dimension, item-level correlation between student and human scores (e.g., Spearman or Kendall correlation, or pairwise accuracy), human inter-annotator agreement as a ceiling, and a comparison under the same human protocol against existing automatic evaluators such as CLIP-based scores, VIEScore-style MLLM scoring and the zero-shot teacher itself.

  2. The protocol for measuring student–teacher agreement is underspecified and internally inconsistent. Sec. 3.4 and Eq. 2 do not say whether the MSE in Table 3 is computed on a held-out split, or whether such a split is disjoint from the training data by source asset or by editor; App. I confirms the evaluation is not source-disjoint. The per-operation values in Table 3 also differ from Appendix Table 5 for what appears to be the same configuration (e.g., zero-shot 4B: Add 2.16 vs. 1.94, Count 3.87 vs. 2.85), and Table 3 relies on an unspecified "weighted aggregation" across modalities. Finally, results are averaged over seeds 0/42/100 (App. E) without any reported variance, and App. D notes that the 0.06–0.07 gaps between the top-ranked image editors do not establish statistical separation.

Questions for authors
  1. Please report per-dimension item-level correlations (Spearman/Kendall) between the 4B and 8B students and the human ratings in Fig. 6. Please also report human inter-rater agreement and the teacher–human correlation for comparison.
  2. On what split is the Table 3 MSE computed? Is it disjoint from the training data by source asset, by editor, or both? What is the MSE on edits from editors absent from the teacher-scoring data (Sec. 3.4)?
  3. Why do the per-operation numbers in Table 3 differ from Appendix Table 5 for what appears to be the same zero-shot configuration (e.g., Count 3.87 vs. 2.85)? What exactly is the "weighted aggregation"?
  4. How does the distilled judge compare with existing cheap automatic evaluators (CLIP-based metrics, open-source MLLM judges of similar size used zero-shot) in agreement with humans?
  5. Will the benchmark data, the prompt triplets, the student weights and the teacher-scoring data be released? Under what licenses, given that part of the media is web-crawled (Sec. 3.1, Ethics Statement)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.