- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MIE-Bench and MIEScore for evaluating multi-source image editing. The benchmark contains 3,000 editing instances across 16 tasks, 36,000 outputs from 12 editing models, and human scores for visual quality, instruction following, and attribute preservation. At inference, MIEScore takes the source images, editing instruction, and a candidate edited image, and predicts these three scores. It combines a natural-language evaluation skill optimized using scored trajectories and validation feedback with a Qwen3-VL-8B evaluator trained through LoRA and a regression head.
Critical strengths
The multi-source evaluation problem is reasonable.
The benchmark could be a useful community resource. Its task and generator coverage, together with separate quality, instruction-following, and preservation labels, could support training and comparing evaluators independently of the proposed model.
The experiments provide encouraging evidence for the training combination. A same-backbone ablation reports a gain from adding skill optimization, and fixed-parameter transfer tests explore its applicability to other evaluation datasets. The appendix also provides several concrete SFT hyperparameters.
Critical weaknesses
-
The reliability of the human supervision is difficult to assess. Conflicting descriptions of annotation and filtering leave the actual data-collection protocol unclear, and agreement among annotators is not established. This weakens confidence in the benchmark's central contribution. Also, calibration of score-based annotations is important, which has been shown in some technical reports (e.g., QwenImage2).
-
The reported overall scores are inconsistent with their component scores. The paper combines visual quality, instruction following, and attribute preservation using a weighted geometric mean. Under the same samples and score scale, this total cannot exceed the corresponding weighted arithmetic mean, even when scores are averaged across images. Yet, for BAGEL on Background Transfer, the reported component scores imply an upper bound of 32.317, while the reported overall score is 44.07. This cannot be explained simply by changing the order of averaging and aggregation. The paper needs to clarify whether different samples or scales were used, or correct the scores, before the resulting rankings can be trusted.
-
Generalization to unseen multi-source editing problems is not sufficiently established. The split description does not rule out shared source content and instructions across training and testing. Pooled correlations and model-level rankings also leave performance on choosing among outputs for the same task, and on unseen editors, uncertain.
4.The benefit of automatic skill optimization is not clearly isolated. Adding the final skill improves the same-backbone model, but a matched comparison with a carefully written fixed multi-source rubric is missing. The evidence therefore does not establish how much automatic search adds beyond better task-specific instructions. Comparably trained joint multi-image evaluators would also clarify the broader methodological increment.
-
The methodological contribution specific to multi-source editing is limited. The evaluation dimensions, human-rating protocol, skill optimization, and supervised score prediction are largely generic and could be applied to other image-editing evaluation settings. Although the rubric includes source-role instructions, the paper does not establish a substantial evaluation mechanism or annotation design tailored to cross-source interactions. Restricting the study to multi-source editing therefore narrows its coverage without a clearly demonstrated methodological gain; the stronger contribution appears to be the dataset's coverage.
-
RL result of MIEScore is more convinced.
Questions for authors
Which annotation protocol was actually used? Please reconcile the conflicting rater, source-image, and generator counts, and provide rating counts, exclusion logs, filtering denominators, and inter-rater reliability.
Can Tables 7–10 be regenerated from the same per-image scores using the stated aggregation rule? Please explain the Background Transfer discrepancies, report whether corrections affect Tables 2–5 or model rankings, and clarify the differences between the fine-tuned DeepSeekVL2/InternVL3.5 entries in Tables 2 and 4.
Are all outputs sharing a source-image/instruction group kept in one split, with reused source images prevented from crossing splits? Please provide group-disjoint, within-instance, unseen-editor, and source-count results with uncertainty estimates.
How much does automatic skill optimization improve over a carefully written fixed multi-source rubric and a comparable jointly multi-image evaluator under the same backbone, training data, and compute budget?
Which parts of the annotation protocol or evaluation method specifically address cross-source interactions beyond a generic editing evaluator with multiple image inputs? Can controlled experiments demonstrate a benefit from these multi-source-specific choices?
16 comments
W1&Q1. Clarification of Human Annotation Reliability and Score Calibration
2. Score calibration
To mitigate inter-subject differences in rating tendencies and scoring-scale usage, we perform subject-wise Z-score normalization as a score calibration before aggregation following the ITU-R BT.500 [1]. Similar score calibration have been adopted in existing image quality assessment studies, such as ARIQA [2] and AGIQA-3K [3], and LMM4Edit [4]. This calibration reduces systematic biases caused by individual differences in score means and variances, making ratings from different subjects more comparable while preserving their relative scoring patterns. Specifically, each subject's valid ratings are normalized using their own mean and standard deviation, followed by linear scaling to . The final Mean Opinion Score (MOS) is obtained by averaging the calibrated ratings for each image. The computation is formulated as:
\qquad z_j=\frac{1}{N_j}\sum_{i=1}^{N_j}z_{ij}, \qquad MOS_j=\frac{100(z_j+3)}{6},$$ where $r_{ij}$ denotes the raw rating assigned by the $i$-th subject to the $j$-th image, $\mu_i$ and $\sigma_i$ represent the mean and standard deviation of the ratings provided by the $i$-th subject, respectively, and $N_j$ denotes the number of valid ratings for the $j$-th image. **References** [1] H. Duan et al., "Confusing Image Quality Assessment: Toward Better Augmented Reality Experience," *IEEE Transactions on Image Processing*, vol. 31, pp. 7206–7221, 2022. [2] C. Li et al., "AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment," *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 34, no. 8, pp. 6833–6846, 2024. [3] Z. Xu et al., "LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs," Proceedings of the ACM International Conference on Multimedia, pp. 6908–6917, 2025.W2&Q2. Correction of Column Ordering Errors in Tables
W2&Q2. Correction of Column Ordering Errors in Tables
Thank you for pointing out this inconsistency. We carefully checked the original results and identified a column-ordering issue in the second subtable of Tables 9–11, rather than an error in the score calculation. Specifically, the data columns were displayed in a different order from their corresponding column headers. The original data were arranged in the following order: Assisted Drawing, Background Transfer, Filter Composition, Style Transfer, Physical Understanding, Spatial Composition, Story Generation, and Interaction. However, the column headers did not follow this order, resulting in incorrect associations between task names and reported scores.
We have corrected the column ordering in Tables 9-11 in revised PDF to ensure that all scores are properly aligned with their corresponding tasks. Importantly, the underlying calculations and experimental results remain unchanged. This issue is confined to the presentation of Tables 9–11 and does not affect any other figures, tables, or conclusions of the paper. We apologize for the confusion and thank the reviewer for helping us identify this presentation issue.
W1&Q1. Clarification of Human Annotation Reliability and Score Calibration
W1&Q1. Clarification of Human Annotation Reliability and Score Calibration
We sincerely thank the reviewer for raising this important concern regarding the reliability of human supervision and the calibration of score-based annotations. We agree that rigorous quality control, reliable human annotations, and proper score calibration are essential for ensuring the credibility of the benchmark. To address these concerns, we have further clarified our subjective data collection and processing protocols in the revised manuscript.
1. Annotation protocol
We recruited 15 human subjects to independently evaluate the edited images, following the ITU-R BT.500 [1] recommendations for subjective image quality assessment. Before the formal evaluation, all subjects received standardized training to familiarize themselves with the evaluation protocol, the three assessment dimensions (visual quality, instruction following, and attribute preservation), and the corresponding scoring criteria. They then completed a preliminary test to ensure their understanding of the evaluation requirements and scoring procedure before proceeding to the formal annotation.
The evaluation consisted of 36,000 edited images, generated from 3,000 source images using 12 editing models. All edited images were randomly shuffled to avoid potential ordering effects, and each subject independently evaluated the entire set of 36,000 images across all three dimensions. Thus, each subject provided 108,000 ratings, resulting in 1,620,000 raw subjective ratings in total.
To ensure annotation reliability, we followed the recommended procedures to identify abnormal ratings and screen unreliable subjects. An individual rating for an image is considered an outlier if it deviates from the mean rating of that image by more than 2 standard deviations for normally distributed ratings or 20 standard deviations otherwise. Following this procedure, 5.24% of the total subjective ratings were identified as outliers and removed. Specifically, among the 1,620,000 raw subjective ratings, 26,856, 28,739, and 29,295 ratings were identified as outliers and removed for visual quality, instruction following, and attribute preservation, respectively. The numbers and percentages of outlier ratings removed for each subject are reported in table below.
We further report both Kendall's coefficient of concordance (W) and the intraclass correlation coefficient (ICC(2,15)) to evaluate inter-subject agreement and the reliability of aggregated subjective scores, respectively. As shown in table below, the three evaluation dimensions achieve Kendall's W values of 0.712, 0.734, and 0.728, and ICC(2,15) values of 0.923, 0.941, and 0.938, respectively, with average values of 0.725 and 0.934. This consistently high values across all three dimensions indicate good inter-subject agreement and high reliability of the aggregated MOS scores.