- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces UniEditBench, a benchmark and distilled MLLM evaluation framework for joint image and video editing assessment. The benchmark curates 710 source instances (633 images and 77 videos) annotated with a unified prompt-triplet schema (source description, target description, and editing instruction) across 9 image and 8 video editing operations (Add, Adjust, Change, Count, Extract, Remove, Reorder, Replace, and image-only Stroke). Using 25 image editors and 8 video editors, the authors generate 22,554 edited outputs and score them with a large open-weight teacher model (Qwen3-VL-235B-A22B) across 4 image dimensions (Structural Fidelity, Text-Image Alignment, Background Consistency, Naturalness) and 5 video dimensions (adding Temporal-Spatial Consistency). To reduce evaluation cost, the authors distill this teacher into compact Qwen3-VL-4B and Qwen3-VL-8B student models via a two-stage LoRA SFT curriculum (Stage 1 on image edits, Stage 2 on video edits).
Critical strengths
- Unified prompt-triplet schema across editing paradigms and modalities: Annotating each benchmark instance with enables side-by-side evaluation of description-based (inversion/inversion-free), instruction-driven, and mask/stroke-guided editors across both image and video modalities under a shared operation taxonomy and multi-dimensional rubric.
- Practical motivation for lightweight open-source evaluators: Distilling a 235B MoE judge (requiring 8 GPUs / ~471 GB BF16 storage) into single-GPU 4B/8B evaluators addresses a genuine pain point in reproducible, low-cost benchmarking for visual editing.
Critical weaknesses
- Severe train-test data leakage Concern in evaluator distillation and benchmark scoring: The paper collects 710 source instances (633 images, 77 videos) and runs 25 image editors and 8 video editors to obtain 22,554 edited outputs. Nowhere in Section 3.3, Section 4, or the Appendices is a disjoint train/validation/test split of the 710 source instances or the 22,554 outputs defined! It appears that the 4B/8B student evaluators are distilled on the teacher's scores over the 22,554 benchmark outputs, and then evaluated for teacher-imitation MSE (Tables 3, 5–8) and used to produce the main benchmark leaderboards (Tables 1–2) on the exact same 710 instances / 22,554 outputs. Appendix I even acknowledges that "Editor overlap alone does not prove sample leakage; conversely, it cannot establish source-disjoint evaluation." Evaluating distillation MSE and benchmarking models on the training set of the distilled evaluator invalidates claims of out-of-distribution evaluator generalization.
- Insufficient human validation and tiny video benchmark scale: First, the video subset contains only 77 source videos across 8 editing operations (fewer than 10 videos per operation on average), which is far too small for a comprehensive video editing benchmark compared to existing suites (e.g.,IVEBench, FiVE-Bench). Second, the human validation study (Figure 6) only plots macroscopic average radar profiles across 5 models; as the authors themselves admit in Appendix I (lines 2178–2181), "the aggregate figure does not establish a named correlation coefficient, inter-rater agreement, or a teacher–human baseline." Without instance-level Spearman/Kendall rank correlation or pairwise agreement between the 235B teacher / 4B-8B students and human raters, low MSE with the teacher only shows that the student imitates the teacher, not that either is a valid proxy for human judgment.
- Unjustified two-stage curriculum claim: The paper claims a "two-stage progressive distillation strategy" (Image SFT followed by Video SFT) as a core methodological contribution that prevents catastrophic forgetting. However, as admitted in Appendix J, the experiments do not compare against joint image+video training, video-only training, or compute-matched baselines, and Stage 2 simply runs 3 additional epochs on video data only (which actually degrades image Extract MSE from 1.01 to 1.23 and worsens 8B video MSE relative to 4B: 0.79 vs. 0.73).
Questions for authors
- Train/Test split for evaluator distillation (Clarification — Critical): Can the authors explicitly clarify how the 710 source instances and 22,554 edited outputs were partitioned between LoRA SFT distillation training and the MSE / leaderboard evaluations in Tables 1, 2, and 3? Were the 4B/8B evaluators trained on the same 710 benchmark instances (or a subset thereof) on which Tables 1–3 are reported? If not, what are the exact train and test split sizes, and are they disjoint by source image/video and by editing model?
- Quantitative human-alignment metrics (Clarification): Can the authors compute and report the sample-level Spearman rank correlation () and Kendall (per dimension and overall) between human ratings, the
Qwen3-VL-235B-A22Bteacher, and the distilled 4B/8B students on the human-evaluated subset? - Comparison with joint Image+Video SFT (Clarification): How does sequential two-stage SFT (3 epochs Image 3 epochs Video) compare against standard joint multi-task SFT on mixed image and video evaluation data under an equal training token budget?