- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RubricEdit-Bench, a benchmark for complex infographic editing comprising 1,483 image–instruction pairs across 15 fine-grained tasks grouped into four categories: Element, Layout, Data, and Structure. Each case is characterized by target multiplicity, reasoning demand, and source-image complexity. The paper proposes a diagnostic evaluation framework that decomposes editing requirements into atomic criteria covering instruction following (IF), preservation (PRES), and visual quality (VQ), with source-coordinate grounding, programmatic paired cropping, and localized verification. The authors further train a specialized evaluator based on Qwen3.5-9B using structured supervision generated by GPT-5.5, followed by GDPO post-training to obtain the final RL-100 evaluator. Experiments evaluate 15 image editing models and identify persistent weaknesses in coordinated layout, data, and structural modifications, as well as a gap between atomic criterion pass rates and complete task success.
Critical strengths
-
Important and practically relevant evaluation setting. Infographic editing requires maintaining consistency among text, numerical information, layout, and structural relationships. This setting exposes fine-grained errors that general-purpose image editing benchmarks may overlook, providing a valuable complement to existing evaluations.
-
Systematic task taxonomy and difficulty analysis. The benchmark covers both localized element modifications and coordinated edits involving layout, data, and structure. Its diagnostic attributes support more detailed analysis of editing difficulty. The regression analysis controls for task category and model fixed effects, helping distinguish the associations between different difficulty factors and model performance.
Critical weaknesses
-
Output-dependent atomic criterion generation may compromise the comparability of scores across models. The framework generates atomic criteria using the source image, editing instruction, and evaluated output, and allows criteria to be added based on omissions, unintended modifications, and quality issues observed in that output. Although this helps capture actual errors, different model outputs for the same source image and instruction may consequently be evaluated using checklists with different coverage, numbers of criteria, and levels of granularity. Since scores are aggregated from criterion-level judgments, these checklist differences may alter the relative weighting of requirements, causing score differences to reflect the criterion-generation process in addition to editing quality.
-
The implementation of programmatic cropping and localized verification during final evaluator inference is unclear. Section 3.4.2 and Figure 3 describe a multi-stage evaluation procedure involving criterion decomposition, source-coordinate grounding, deterministic paired cropping, and localized verification. This procedure is also used to generate training supervision. However, the specialized evaluator’s inference template in Appendix F includes only the source image, editing instruction, and edited image, and requests a structured report containing criteria, grounding boxes, judgments, and reasons. It is therefore unclear whether the final RL-100 evaluator actually performs programmatic cropping and feeds the local images into a subsequent verification stage during inference, or instead generates the entire report end-to-end from the full images.
Questions for authors
See Weaknesses