acceptodds
Under review as a conference paper at ICLR 2027

RubricEdit-Bench: Benchmarking Complex Infographic Editing with Fine-Grained Diagnostic Evaluation

Abstract

Instruction-guided image editing models have advanced rapidly, yet their capabilities on complex infographic editing remain insufficiently evaluated. Infographics tightly couple text, data, layout, and structural relations, while existing benchmarks provide limited coverage of such structured edits and holistic evaluation protocols offer little diagnostic information about specific failures. We introduce RubricEdit-Bench, a benchmark comprising 1,483 human-reviewed cases across four categories and 15 fine-grained tasks, with attributes characterizing target multiplicity, reasoning demand, and source-image complexity. RubricEdit-Bench further adopts a diagnostic evaluation protocol that decomposes each edit into typed atomic criteria and performs source-grounded, localized verification, enabling failures to be identified at the criterion level. We evaluate 15 frontier open- and closed-source editing models, with Overall scores ranging from 1.88 to 4.89 on a five-point scale. The best-performing model achieves exact success on only 85.2% of cases, indicating that complete task fulfillment remains challenging even for frontier models. Edits requiring complex reasoning show a 0.48-point performance drop compared with edits requiring little or no reasoning. Coordinated layout, data, and structural edits also remain particularly challenging. RubricEdit-Bench provides a systematic and diagnostic testbed for identifying these remaining capability gaps and guiding progress toward reliable, structure-aware image editing.

Primary area:
datasets and benchmarks
Keywords:
Instruction-Guided Image Editing, Infographic Editing, Image Editing Benchmark, Fine-Grained Diagnostic Evaluation, Vision-Language Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G2H3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RubricEdit-Bench, a benchmark for complex infographic editing comprising 1,483 image–instruction pairs across 15 fine-grained tasks grouped into four categories: Element, Layout, Data, and Structure. Each case is characterized by target multiplicity, reasoning demand, and source-image complexity. The paper proposes a diagnostic evaluation framework that decomposes editing requirements into atomic criteria covering instruction following (IF), preservation (PRES), and visual quality (VQ), with source-coordinate grounding, programmatic paired cropping, and localized verification. The authors further train a specialized evaluator based on Qwen3.5-9B using structured supervision generated by GPT-5.5, followed by GDPO post-training to obtain the final RL-100 evaluator. Experiments evaluate 15 image editing models and identify persistent weaknesses in coordinated layout, data, and structural modifications, as well as a gap between atomic criterion pass rates and complete task success.

Critical strengths
  1. Important and practically relevant evaluation setting. Infographic editing requires maintaining consistency among text, numerical information, layout, and structural relationships. This setting exposes fine-grained errors that general-purpose image editing benchmarks may overlook, providing a valuable complement to existing evaluations.

  2. Systematic task taxonomy and difficulty analysis. The benchmark covers both localized element modifications and coordinated edits involving layout, data, and structure. Its diagnostic attributes support more detailed analysis of editing difficulty. The regression analysis controls for task category and model fixed effects, helping distinguish the associations between different difficulty factors and model performance.

Critical weaknesses
  1. Output-dependent atomic criterion generation may compromise the comparability of scores across models. The framework generates atomic criteria using the source image, editing instruction, and evaluated output, and allows criteria to be added based on omissions, unintended modifications, and quality issues observed in that output. Although this helps capture actual errors, different model outputs for the same source image and instruction may consequently be evaluated using checklists with different coverage, numbers of criteria, and levels of granularity. Since scores are aggregated from criterion-level judgments, these checklist differences may alter the relative weighting of requirements, causing score differences to reflect the criterion-generation process in addition to editing quality.

  2. The implementation of programmatic cropping and localized verification during final evaluator inference is unclear. Section 3.4.2 and Figure 3 describe a multi-stage evaluation procedure involving criterion decomposition, source-coordinate grounding, deterministic paired cropping, and localized verification. This procedure is also used to generate training supervision. However, the specialized evaluator’s inference template in Appendix F includes only the source image, editing instruction, and edited image, and requests a structured report containing criteria, grounding boxes, judgments, and reasons. It is therefore unclear whether the final RL-100 evaluator actually performs programmatic cropping and feeds the local images into a subsequent verification stage during inference, or instead generates the entire report end-to-end from the full images.

Questions for authors

See Weaknesses

Reviewer_KPE3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RubricEdit-Bench evaluates instruction-guided editing of complex infographics. It introduces 1,483 human-reviewed cases covering four categories and 15 fine-grained tasks, together with annotations for target multiplicity, reasoning demand, and source complexity. The proposed evaluation protocol decomposes editing requirements into atomic criteria for instruction following, preservation, and visual quality, grounds them in source-image regions, and verifies them using paired source-output evidence. The authors also train a specialized evaluator and benchmark 15 editing models, supported by comparisons with GPT-5.5 and a human-preference diagnostic set.

The benchmark addresses a practically valuable problem and provides a well-organized task space. However, the reliability of its central evaluation methodology requires stronger validation.

Critical strengths
  • Practical relevance and useful task coverage. Infographic editing requires coordinated changes: modifying a number, label, or node may also require updating chart encodings, layout, or connections while preserving unrelated content. The benchmark captures these requirements through a reasonably comprehensive taxonomy spanning element, layout, data, and structural editing.

  • Inspectable, fine-grained evaluation. Decomposing an instruction into atomic criteria and associating judgments with localized visual evidence is a useful approach to diagnosing failures. Separating instruction following, preservation, and visual quality provides more actionable information than an aggregate score alone.

Critical weaknesses
  • The relationship between localized crop verification and the deployed evaluator is unclear. Section 3.4 describes a sequential procedure involving grounding, programmatic paired cropping, and verification using the full images and local crops. However, the specialized evaluator prompt in Appendix F receives only the two full images and the instruction, and produces criteria, bounding boxes, and judgments in one response. It is therefore unclear whether the final RL-100 evaluator used for the benchmark actually executes cropping and re-examines the resulting regions. Predicting a bounding box does not by itself establish that the judgment benefits from localized visual inspection. This ambiguity makes it difficult to assess whether the reported results validate the proposed evaluation mechanism.

  • Output-dependent criteria raise unresolved questions about score comparability. The prompts explicitly allow the edited output to influence checklist construction, including adding criteria for discovered errors or preservation risks. Consequently, different outputs for the same source and instruction may be assessed using different numbers and granularities of criteria. While this flexibility is useful for diagnosis, it can also change the effective weighting of requirements and affect both pass rates and exact case success. The paper does not sufficiently establish that these aggregate metrics remain comparable across models or robust to alternative checklist decompositions.

Questions for authors
  1. Does the final RL-100 evaluation pipeline actually perform programmatic cropping and feed the resulting crops back into the verifier, or are all judgments generated from the two full images in a single response? Please clarify the deployed procedure and provide evidence for the contribution of localized verification.

  2. How do you ensure comparable requirement coverage and granularity across different model outputs for the same case? Would the model rankings and exact-success conclusions remain stable under a shared source-and-instruction-derived checklist or another controlled rubric construction?

Satisfactory clarification and supporting evidence on these two points could materially change my assessment, and I would reconsider my rating accordingly.

Reviewer_hyjmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RubricEdit-Bench contains 1,483 human-reviewed infographic editing cases across 15 tasks in four categories: Element, Layout, Data, and Structure. Cases are labeled by target multiplicity, reasoning demand, and source complexity. The evaluator produces atomic Instruction Following (IF), Preservation (PRES), and Visual Quality (VQ) checks with source-coordinate evidence. It uses Qwen3.5-9B trained with GPT-5.5 supervision and further post-training. The paper evaluates 15 editors on 1,422 common cases, reporting dimension scores, Overall, atomic pass rate, and exact case success. Validation compares the evaluator with GPT-5.5 and human preferences and tests the utility of its reports.

Critical strengths

The benchmark covers module rearrangement, data sorting, chart conversion, and topology editing, beyond basic text and object edits. Its 15 tasks are nearly balanced and include Chinese and English. Compared with IGenBench, it tests editing with source preservation; compared with ChartM3 and VDE Bench, it brings more infographic components and dependencies into one benchmark.

Critical weaknesses
  1. The scores may not be comparable across editors. Checklists are generated after seeing each edited output (Appendices E.1 and F), so the same instruction can produce different checks and denominators. Table 6 also groups several labels into one check and overlaps local and global preservation checks. A shared-checklist comparison and a human audit of coverage, redundancy, and verdicts are needed.
  2. Human alignment is evaluated on only 88 matched pairs from a disagreement-enriched diagnostic set. The judges and selection criteria underlying these disagreements are unspecified. The small sample and unclear sampling procedure limit the conclusions about benchmark-wide human alignment; a larger, representative human evaluation is needed.
Questions for authors
  1. Are checklists fixed across editors for the same source/instruction? If not, how do scores and rankings change with a shared checklist or different split/merge rules?
  2. Please evaluate the accuracy of the predicted regions and the contribution of explicit grounding to judgment accuracy.
  3. Please provide the source datasets and their proportions in the benchmark, the number and construction procedure of task-specific infographics, and the human review criteria, including any shared review or disagreement resolution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.