- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates whether image editing systems can adjust card illustrations to designated presentation tiers (N, R, SR, SSR) while preserving subject identity. The system must be capable of both adding and reducing details, as well as attempting round-trip editing.
The authors construct 200 four-tier image groups and introduce CardHarness, a system that employs a vision-language model to diagnose issues, invoke image editors, inspect results, and retrieve experience accumulated during training. Under identical editors and a budget of three image calls, CardHarness achieves an effective progress rate of 77.5%, outperforming Best-of-3 at 59.2%. However, its exact arrival rate at the target tier is only 2.5%, and the memory module shows no advantage over its memory-free counterpart. The authors explicitly acknowledge these limitations.
Critical strengths
- Meaningful task formulation. "Visually better" does not equal "reaching a designated state." Testing both upward and downward adjustments effectively identifies systems that merely add decorations rather than exercising controlled scaling.
- Clear protocol information boundaries. Data is partitioned by subject, memory is frozen during testing, and neither target references nor final evaluators provide runtime feedback. These designs help prevent data contamination and control test-time information usage.
Critical weaknesses
- Inconsistencies between displayed data and identity protection constraints. The main text requires preserving attributes such as subject category and hair color, yet some images in the same row of Figure 3 clearly belong to different subjects, and the source and target reference in Figure 4(b) also exhibit changes in hair color. Although the supplementary material provides alternative same-subject reference groups, these discrepancies are left unexplained. If this is merely a figure selection error, it should be corrected; if such pairs are part of the actual evaluation task, identity changes may confound the results, preventing an isolated measurement of tier-control performance.
- Discrepancies in scoring methods and round-trip success definitions between the main text and supplementary material. The main text uses the CLIP ranker to define tiers and Exact matches, whereas the supplementary material states that primary comparisons are scored by GPT-5.5, with the CLIP ranker serving only for data validity checks. Furthermore, while the main text describes the 3/60 round-trip results as meeting full success conditions, Supplementary Material A.8 explicitly notes that identity preservation and artifacts across both segments were not independently verified—meaning it only indicates state closure rather than full semantic reversibility. These discrepancies alter the interpretation of the results and require unifying metric definitions and experimental explanations.
- Lack of independent validation for the evaluation rules despite their disclosure. While the supplementary material outlines prompts, thresholds, and scoring combination rules, it fails to demonstrate whether GPT-5.5's assessments of actual editing outputs align with independent human judgments. CLIP ranking accuracy cannot validate GPT-5.5's tier and progress assessments, nor can Qwen and GPT-5.5 results serve as cross-validation on the same batch of outputs since they stem from different experimental blocks. Consequently, particularly given the universally low exact arrival rates, it remains difficult to disentangle the effects of poor model control from unstable evaluation criteria.
Questions for authors
See weakness.