acceptodds
Withdrawn from ICLR 2027

Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Graphic Design, Procedural Memory, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yjfHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper keeps the LLM frozen and instead evolves an external library of natural-language skills for a graphic design agent. The library grows by adding skills for recurring uncovered subtasks, and improves by revising existing skills based on their failures and successes. A replay gate decides which changes are kept. Experiments show better execution success and positive win rates over a no-skill agent on several benchmarks. The ablation suggests that adding and revising skills work better together than alone.

Critical strengths
  1. The setting is realistic and hard: long-horizon tool use, editable outputs, and no reliable success oracle.
  2. The approach is practical. It needs no weight updates, adds little overhead, and the paper reports negative results honestly.
Critical weaknesses
  1. Limited novelty. The main ideas are close to existing skill and memory evolution work (e.g., ACE, MACLA, SkillAudit). The more distinctive parts are not well validated. The replay gate is never ablated. The ablation behind the "combination" claim also seems confounded, because the full system revises the newly added skills, while the "new skills only" arm does not.
  2. Moderate and hard-to-interpret gains. The same automatic judge drives evolution and evaluates the internal results, and there is no human evaluation. The improvement is mainly in task completion and execution success, while aesthetic quality barely changes. It is also unclear how failed runs are handled in some comparisons.
Questions for authors
  1. How does the system perform without the replay gate, or with a simpler acceptance rule?
  2. Do the main results hold with an independent judge, or with human evaluation?
Reviewer_oQBXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies continual adaptation for a graphic-design agent operating over 230+ professional tools (analogous to Photoshop, Illustrator, and InDesign) without updating model weights or relying on a programmatic oracle. The authors treat an external procedural memory of natural-language playbooks (SKILL.md) as the optimization target and propose a three-part offline evolution loop: (1) Widening (§3.1), which distills recurring uncovered subtasks (≥3\ge 3 occurrences) into new candidate skills; (2) Deepening (§3.2), which revises skills with ≥2\ge 2 failures by contrasting successful and failed execution traces; and (3) a Replay Gate (§3.3), which holds retrieved assets fixed and admits updates via pairwise blind judging. Across five evolution rounds on 1,406 briefs, the skill bank grows from 76 documentation-seeded skills to 139, and is evaluated against a no-skill baseline (BASE) on three foundation models across general T2I and specialized graphic-design benchmarks.

Critical strengths
  1. Practical system design in a complex tool environment: Operating a frozen VLM across 230+ graphic design tools with real asset retrieval and layered rendering is a realistic engineering setting. The two-stage retrieval, progressive disclosure via load_skill, and the separation of skill expansion from skill repair form a sensible and intuitive pipeline with modest latency overhead (+3.4%–6.2%, Fig. 6).

  2. Insightful coupling dynamics and honest failure analysis: The ablation study (Table 3) and round-by-round analysis (Fig. 5) clearly illustrate that minting rough v1 skills alone causes false-positive retrievals, and must be paired with deepening rewrites to yield a superadditive gain (58.5% win rate). Moreover, Appendix I provides a candid discussion of failure modes regarding model default biases, VLM geometric perception limits (Droste effect), and fidelity decay over long multi-step constructions.

Critical weaknesses

While the proposed framework is intuitive and interesting from a system-engineering perspective, the paradigm of evolving external natural-language skills is no longer novel, and the experimental evaluation is rough, unrigorous, and incomplete. Substantial new experiments and revisions are required before this work can be considered for publication:

  1. Limited methodological novelty and vulnerability to foundation model iteration: Evolving external natural-language playbooks or skill libraries via trajectory reflection and success/failure contrast has been extensively explored in recent literature (e.g., ACE, Dynamic Cheatsheet, SkillAudit, MACLA, Skill-Pro, and ComfyClaw, as surveyed in Appendix A). Relying on simple frequency heuristics (kmin⁡=3k_{\min}=3 for widening, m=2m=2 for deepening) offers limited algorithmic novelty. More importantly, the experiments only test older model checkpoints (Claude-Sonnet-4, Claude-Opus-4.6, and Qwen3.6-27B) and artificially restrict the Claude models to low thinking effort capped at 2,000 and 5,000 tokens (§4, Line 229). As frontier open-weight and proprietary models iterate rapidly (e.g., GPT-6, Claude Opus 5.5, Kimi K3, Qwen 3.8), native multi-step planning and tool-using capabilities have improved substantially. It is highly plausible that the tool-use failures observed under restricted thinking budgets on older models are largely resolved by newer foundation models out of the box.

  2. Missing external memory baselines and zero ablation on the Replay Gate: (a) Despite devoting three pages in Appendix A to reviewing test-time memory and skill-evolution methods (e.g., ACE, Dynamic Cheatsheet, SkillAudit, MACLA), the paper compares EVOLVE solely against a bare BASE model (no skills) and static documentation (Cold start), omitting any external memory-evolution baseline. (b) Although the Conservative Replay Gate (§3.3) is claimed as one of the three core contributions, Section 5 (Table 3) provides zero ablation on the gate itself (e.g., no comparison against accepting edits without gating or gating by absolute Graphic-Eval scores). Furthermore, Appendix F (Line 1210) reveals that each rewrite gate evaluates only 4 trajectories (2 high-scoring and 2 low-scoring), which is statistically underpowered to guarantee "no regression."

  3. Severe survivorship bias and unconvincing attribution on Qwen3.6-27B: In Table 1 and Appendix Table 4, equipping Qwen3.6-27B with EVOLVE causes severe drops in execution success rate: from 44.0% to 28.7% (-15.3%) on DPG-Bench, from 77.3% to 63.3% (-14.0%) on OpenCOLE, and from 78.0% to 64.0% (-14.0%) on BannerRequest400. Because Table 1 computes generation quality exclusively over successful outputs (Line 264), reporting a +9.67 quality gain on Qwen after filtering out 40%–71% of failed hard prompts suffers from severe survivorship bias. Attributing these drops to "local deployment timeouts and connection drops" (Appendix H, Line 1309) is unconvincing, as BASE and EVOLVE share the exact same 8xA100 vLLM setup and Fig. 6 shows only a 3.7% latency difference on completed runs.

  4. Coarse blame assignment, self-referential evaluation, and non-reproducibility: Every failed trajectory (sj<0.6s_j < 0.6) assigns blame equally to all 2–4 retrieved skills without credit assignment (Appendix E, Line 1192), risking unnecessary rewrites of healthy upstream skills. In addition, the 200-brief internal benchmark uses the exact same Graphic-Eval rubric that drives the evolution loop (Appendix D) without any human blind evaluation, and the entire 230-tool environment and datasets are proprietary and closed-source.

Questions for authors
  1. Evaluation on latest frontier models and full reasoning budgets: Have you tested EVOLVE on the latest generation of open-weight and proprietary models—such as GPT-6, Claude Opus 5.5, Kimi K3, or Qwen 3.8—as well as without the artificial low thinking effort cap (2k/5k tokens)? To what extent do stronger native tool-using capabilities in newer models already solve these multi-step execution errors without external skill evolution?

  2. External baselines and Replay Gate ablation: Why are no existing context/skill evolution methods (e.g., ACE, Dynamic Cheatsheet, SkillAudit) compared under the same budget? What is the performance on the 200-brief internal benchmark if the Replay Gate is removed or replaced with an absolute-score threshold?

  3. Survivorship bias and failure analysis on Qwen3.6-27B: If failed generations are scored as 0 to eliminate survivorship bias across all 300 prompts in Table 1, does EVOLVE still improve Qwen3.6-27B? What is the actual breakdown of failure modes causing the 14%–15.3% success rate drop when skills are injected into Qwen?

  4. Cross-model skill transfer and human evaluation: Which model acted as the solver/reflector during the 5 evolution rounds, and was the same evolved bank transferred zero-shot to all three models? Can you provide human blind preference evaluations to rule out overfitting to the Graphic-Eval VLM judge?

Reviewer_zp5sOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a technique for augmenting LLMs with an evolving skill set for image editing software. Starting from a large tool set, the system evolves skills using two operators: widening, which identifies recurring behaviors that should be encapsulated into skills, and deepening, which revises existing skills to improve their success rates. The system achieves gains over the base model across several general text-to-image benchmarks.

Critical strengths
  1. The paper studies the important problem of evolving a skill library for a frozen LLM.
  2. Empirical results demonstrate improvements over the base model across a variety of general text-to-image benchmarks.
  3. The qualitative results show substantial visual differences between the base model and the model augmented with the evolved skill set.
Critical weaknesses
  1. The paper is poorly written, making it difficult to understand important details of the method and evaluation. The term "brief" is only clearly defined at L252, despite being used frequently throughout the preceding sections. What does the trajectory mentioned at L150 consist of, specifically, what are the observations and actions? How is a skill extracted, and what does it mean to "canonicalize" it? How is a skill determined to be responsible for a bad outcome (L152), and what constitutes a poor outcome? The "replay gate" mentioned at L155 is not defined in this section. The paper also does not describe how a skill is represented in the skills.md file, despite this being central to the methodology. The method section refers to "shipping" (L184) without defining the term. What constitutes a "graded history" (L161)? Is this the VLM verifier score? What is the "heuristic" mentioned at L161? What are "paired trajectory contrasts" (L187)? These examples are drawn from just the first two paragraphs of the method section, and similar issues persist throughout the rest of the paper. This makes it difficult to understand what the method actually does and how it differs from prior work.
  2. The paper provides insufficient details about the internal benchmark. What types of prompts are included in the 300 briefs? How do the 200 human-authored briefs differ from the others? Simply stating that they use disjoint language is insufficient to establish meaningful separation between the datasets.
  3. The evaluation in Section 4.1 also lacks important details. The paper does not define what R1 through R5 represent, although I assume they refer to evolution rounds. Why is retention a useful metric to monitor? The paper should present a curve with the evolution round on the x-axis and evaluation performance on the held-out set of briefs on the y-axis, as this is central to assessing the proposed approach.
  4. It is unclear which aspects of the pipeline are specific to graphic design. The approach should be evaluated on standard benchmarks, such as those discussed in the related work section, to assess the generalizability of the methodology.
  5. The paper does not empirically compare against any prior work on skill evolution. The relationship between the proposed method and prior approaches is also unclear. I strongly suggest moving the related work section into the main paper to better establish the novelty of the contributions.
  6. The paper does not clearly ablate the complementary effects of the widening and deepening operations. It is unclear how Table 3 isolates the contributions of these two components.
  7. The paper does not investigate whether the method overfits to the VLM judge, which is also used during training. How can we determine whether the improvements reflect genuine gains in task performance rather than optimization toward the judge's preferences?
Questions for authors

See the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.