acceptodds
Withdrawn from ICLR 2027

CardBench-X: Benchmarking Memory-Guided Bidirectional Visual Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing image-generation benchmarks mainly evaluate prompt following, image quality, or the injection of new semantic knowledge. They rarely test whether an agent can separate visual knowledge that should remain invariant from knowledge that should be added or removed during an image transformation. We introduce CardBench-X, a benchmark for memory-guided bidirectional visual evolution. Given an input subject, a source tier, and a target tier, an agent must transform the image forward or backward across a discrete visual hier- archy while preserving identity-defining content. The bench- mark evaluates target-tier realization, knowledge retention, knowledge injection, reversibility, and transfer of transforma- tion experience across subjects and styles. We also provide CardHarness, a trace-driven baseline that stores success- ful and failed transformations, localizes over-design, under- design, identity drift, and stagnation, and retrieves validated repairs for later cases. CardBench-X offers a controlled envi- ronment for studying whether visual agents merely add more detail or actually learn reusable, direction-aware transforma- tion knowledge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U78NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates whether image editing systems can adjust card illustrations to designated presentation tiers (N, R, SR, SSR) while preserving subject identity. The system must be capable of both adding and reducing details, as well as attempting round-trip editing.

The authors construct 200 four-tier image groups and introduce CardHarness, a system that employs a vision-language model to diagnose issues, invoke image editors, inspect results, and retrieve experience accumulated during training. Under identical editors and a budget of three image calls, CardHarness achieves an effective progress rate of 77.5%, outperforming Best-of-3 at 59.2%. However, its exact arrival rate at the target tier is only 2.5%, and the memory module shows no advantage over its memory-free counterpart. The authors explicitly acknowledge these limitations.

Critical strengths
  • Meaningful task formulation. "Visually better" does not equal "reaching a designated state." Testing both upward and downward adjustments effectively identifies systems that merely add decorations rather than exercising controlled scaling.
  • Clear protocol information boundaries. Data is partitioned by subject, memory is frozen during testing, and neither target references nor final evaluators provide runtime feedback. These designs help prevent data contamination and control test-time information usage.
Critical weaknesses
  • Inconsistencies between displayed data and identity protection constraints. The main text requires preserving attributes such as subject category and hair color, yet some images in the same row of Figure 3 clearly belong to different subjects, and the source and target reference in Figure 4(b) also exhibit changes in hair color. Although the supplementary material provides alternative same-subject reference groups, these discrepancies are left unexplained. If this is merely a figure selection error, it should be corrected; if such pairs are part of the actual evaluation task, identity changes may confound the results, preventing an isolated measurement of tier-control performance.
  • Discrepancies in scoring methods and round-trip success definitions between the main text and supplementary material. The main text uses the CLIP ranker to define tiers and Exact matches, whereas the supplementary material states that primary comparisons are scored by GPT-5.5, with the CLIP ranker serving only for data validity checks. Furthermore, while the main text describes the 3/60 round-trip results as meeting full success conditions, Supplementary Material A.8 explicitly notes that identity preservation and artifacts across both segments were not independently verified—meaning it only indicates state closure rather than full semantic reversibility. These discrepancies alter the interpretation of the results and require unifying metric definitions and experimental explanations.
  • Lack of independent validation for the evaluation rules despite their disclosure. While the supplementary material outlines prompts, thresholds, and scoring combination rules, it fails to demonstrate whether GPT-5.5's assessments of actual editing outputs align with independent human judgments. CLIP ranking accuracy cannot validate GPT-5.5's tier and progress assessments, nor can Qwen and GPT-5.5 results serve as cross-validation on the same batch of outputs since they stem from different experimental blocks. Consequently, particularly given the universally low exact arrival rates, it remains difficult to disentangle the effects of poor model control from unstable evaluation criteria.
Questions for authors

See weakness.

Reviewer_EMavOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CardBench-X, a benchmark designed to evaluate whether image editors can reach a specified ordinal target state in either ascending or descending directions while preserving protected content. The benchmark contains 200 card-style illustration subjects, each with four ordinal tiers (N < R < SR < SSR), resulting in 2,400 directed transitions and 2,400 forward-backward cycles. Its evaluation considers multiple dimensions separately, including progress, exact landing, direction, content preservation, artifact risk, cycle success, and transfer from frozen training memory. The authors also introduce CardHarness, a VLM-guided editing policy based on GPT-Image-2. Under a matched three-call budget, CardHarness achieves higher valid progress than Best-of-3 (77.5% vs 59.2%). However, exact landing remains very limited, and the proposed transition memory does not yield measurable improvements.

Critical strengths
  1. The paper addresses a meaningful problem in image editing. Existing editors and benchmarks mostly emphasize adding details or improving visual preferences, while paying less attention to whether an edit reaches a specific target state, particularly when the goal is to simplify or reduce visual complexity.

  2. Formulating image editing as bidirectional ordinal state control, together with explicit forward-backward cycles, provides an interesting evaluation perspective that is largely missing from existing benchmarks.

  3. The evaluation distinguishes progress, exact landing, direction, preservation, and artifacts rather than relying on a single aggregate score. This reveals important trade-offs, such as models that achieve strong content preservation simply by making little or no meaningful change.

  4. The experimental protocol includes several measures to reduce information leakage, including subject-disjoint splits, a frozen evaluator separated from the runtime checker, hidden target references, and memory frozen before testing.

  5. The paper is relatively transparent about its negative findings, reporting low exact landing accuracy, only 3/60 successful cycles, and no measurable improvement from transition memory.

Critical weaknesses
  1. The framing of the paper is not fully supported by the experimental findings. The title and abstract highlight "memory-guided" evolution and the transfer of editing experience, but the memory experiment actually shows no improvement: 81.7 VP without memory versus 80.0 with transition memory. This negative result is not acknowledged in the abstract. As a result, memory appears to be positioned as a major contribution without sufficient empirical evidence.

  2. The results provide limited support for the claimed target-state control. CardHarness achieves only 2.5% exact landing, which is even lower than Best-of-3 (5.8%), and just 3 out of 60 trajectories satisfy the full cycle contract. These findings suggest that the approach mainly improves movement in the intended direction rather than accurately controlling the final ordinal state.

  3. The reliability of the evaluation raises several concerns. The ordinal states are defined using a CLIP linear head trained on generated images, while the runtime checker optimized by the policy is also CLIP-based, creating a risk of overfitting to the evaluation proxy. Progress is assessed by GPT-5.5, but no human validation or human-model agreement is provided. Thresholds such as artifact risk ≤ 0.45 and the requirement of "at least two dimensions" also lack justification. Furthermore, there are several inconsistencies: the "CardHarness" and "Transition memory" rows in Table 3 report different results despite apparently describing the same setting; Dir. is 83.3 in Table 2 but 65.0 in Table 3; Table 2 lists GPT-Image-2 for Mind-Brush while the text mentions Qwen-Image; and Exact landing is defined differently in the equation and the accompanying description.

  4. The experimental scale and generality are relatively limited. The main comparison includes only 120 cases, and each ablation uses 60, without confidence intervals or repeated trials. A 5-point difference on 60 cases corresponds to only 3 examples, making small ablation and memory differences difficult to interpret reliably. The baselines are restricted to Best-of-3 and Mind-Brush, without broader comparisons to iterative VLM-feedback editing methods. Moreover, the matched budget considers only image generation calls, although CardHarness also requires additional VLM calls. Finally, the benchmark focuses on one presentation axis within the card illustration domain, and the generation and quality validation of the four-tier images are not sufficiently described.

Questions for authors
  1. Are the "CardHarness" and "Transition memory" rows in Table 3 based on the same configuration? If so, what explains the different results? Also, how do the 60 evaluation cases in Table 3 relate to the 120 cases reported in Table 2?

  2. How is Exact landing calculated in practice? Does it use C_eval as specified in the equation, or a threshold applied to CLIP scores as described in the text?

  3. How were the images at the four ordinal tiers generated and quality-checked? Was any human evaluation conducted to verify their perceptual ordering and the reliability of GPT-5.5 progress judgments?

  4. Given that transition memory does not improve performance, what experimental evidence justifies emphasizing "memory-guided" editing in the title and abstract?

Reviewer_csWpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CardBench-X, a benchmark for editing card-style images across four ordered visual states (N, R, SR, SSR) in both ascending and descending directions while preserving subject identity. It contains 200 four-state ladders, which yield 2,400 directed transitions and 2,400 cycles split by subject. The authors also propose CardHarness, an agent baseline that diagnoses the current state, plans edits, validates candidates, and stores transition experience as memory.

Critical strengths
  1. The paper studies how far an edit should go, covering both enhancing and simplifying an image, rather than only enhancing it.
  2. The evaluation separates progress, direction, content preservation, and artifacts, so a method cannot score well by doing nothing or by just adding detail.
  3. The data is split by subject, so test subjects are never seen during training.
Critical weaknesses
  1. The title emphasizes memory-guided editing, but Table 3 does not show a clear benefit from memory. Why does the paper put memory at the center of the title when its contribution is not supported by the results?
  2. In Figure 3, several ladders seem to show different subjects across the four states. For example, one row of sports cards includes both basketball and soccer players, which contradicts the caption that each row keeps the subject identity fixed. Could the authors clarify how the ladders are constructed?
  3. All evaluations rely on automatic models, and there is no human study to verify whether the tier labels and scores agree with human judgment.
  4. The evaluation scale is small, so many of the differences may be within noise.
Questions for authors
  1. How does Best-of-3 select its final output from the three candidates? Why does CardHarness have a lower "Exact" rate than Best-of-3?
  2. What exactly is saved in the memory? Could the authors show a few real examples of saved records and explain how such a record changes the edit for a new case?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.