acceptodds
Under review as a conference paper at ICLR 2027

GEMS: Agent-Native Multimodal Generation with Memory and Skills

Abstract

Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose GEMS (Agent-Native Multimodal GEneration with Memory and Skills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill organizes reusable domain guidance in a shared repository, allowing diverse tasks to benefit from relevant expertise within the same generation workflow. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to match the performance of the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits. Code: https://anonymous.4open.science/r/review-code-5d4224fd95dd/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UAqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2(+1)
Soundness
3
Read review
Paper summary

This paper introduces GEMS, an agentic framework for text-to-image generation. The framework includes an iterative generate–verify–refine loop, trajectory memory, and reusable domain skills. The authors evaluate GEMS across five mainstream benchmarks and four downstream tasks using different text-to-image models. The results demonstrate consistent performance gains over the original generators.

Critical strengths
  1. The authors evaluate GEMS with two image generators across five benchmarks and four downstream tasks, showing consistent improvements. Experiments with Qwen3.5-27B as an alternative MLLM also yield comparable results on GenEval2, providing further evidence of the framework's generalizability.

  2. GEMS outperforms CRAFT on GenEval2 (63.5 vs. 55.2) while requiring fewer images per prompt (2.80 vs. 2.83). Combined with the inference cost analysis in Appendix C.3, this suggests that the gains are not simply driven by more generation attempts.

Critical weaknesses
  1. The Agent Loop closely resembles CRAFT's iterative verification and refinement process. However, the baselines do not receive the same domain skills. A skill-matched comparison is needed to show that GEMS’s design improves over existing frameworks.

  2. Figure 4 shows that retaining historical prompts, feedback, and images already improves GenEval2 from 52.4 to 59.8, while compressed experiences further increase it to 61.4. Since the Compressor also generates new strategic guidance, the contribution of compression itself remains unclear.

Questions for authors
  1. Could the authors compare GEMS with existing refinement frameworks (e.g., CRAFT) using the same automatically selected domain skills to better isolate the contributions of Agent Memory and the refinement design?

  2. Could the authors provide an ablation that separates the effect of experience compression from the additional strategic guidance generated by the Compressor?

  3. Many in-text citations are incorrectly formatted, with missing parentheses around author-year citations (e.g., "inference-time scaling Hao et al. (2023)" instead of "inference-time scaling (Hao et al., 2023)"). Please correct the citation formatting throughout the paper.

Reviewer_TdGiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEMS combines an iterative verification and prompt-refinement loop with trajectory memory and a repository of domain-specific skills to improve text-to-image generation. The system retains previous prompts, generated images, and verification results, together with concise experiences that guide subsequent refinement. It also automatically selects relevant skills to enhance the initial prompt. The framework is evaluated across nine tasks using two image generators.

Critical strengths
  1. The paper is clearly written, the diagrams are easy to understand, and the experiments cover a broad range of tasks. The ablation studies provide useful evidence for the effectiveness of the individual components. Improvements are reported with both image generators across capabilities including compositional generation, text rendering, and creative drawing.

  2. The framework has practical potential. External reasoning and iterative refinement substantially improve the performance of weaker image generators. GEMS also achieves better results than existing refinement frameworks with fewer image generations in some experiments, suggesting a useful improvement in generation efficiency.

Critical weaknesses
  1. The incremental contribution over strong existing baselines needs further clarification. The ablation studies support the effectiveness of the memory and skill modules. However, the baselines do not use the same skill repository or automatic Planner, leaving it unclear how much of GEMS’s quality and efficiency advantage over CRAFT would remain if both received the same domain guidance. In addition, adding experiences to the prompt–feedback–image history configuration improves the score from 59.8 to 61.4, but the Compressor also proposes strategies for subsequent attempts. A comparison with a separate reflection step would help distinguish the benefit of experience compression from that of additional strategic reasoning.

  2. The cost–quality trade-off relative to directly using a stronger image generator remains unclear. The appendix reports a cost advantage over other iterative refinement frameworks, but does not compare a weaker generator augmented with GEMS against a stronger generator used directly. When all LLM calls and repeated image generations are included, how do these approaches compare in total inference cost and end-to-end latency at comparable output quality? This comparison would help establish the conditions under which GEMS offers a practical advantage.

Questions for authors
  1. If CRAFT is preceded by the same automatic skill-selection and prompt-enhancement stages, how does it compare with GEMS in quality and efficiency under comparable inference budgets? Representative results on GenEval2 would help clarify the incremental contribution.

  2. Does experience compression provide a consistent advantage over retaining prompt–feedback–image history or adding a separate reflection step? Could you provide the corresponding controls and an uncertainty analysis for these incremental gains?

  3. At comparable output quality, how does a weaker generator augmented with GEMS compare with a stronger generator used directly in total inference cost and end-to-end latency? In particular, the paper highlights that Z-Image-Turbo with GEMS matches Nano Banana 2’s GenEval2 score. A comparison of the full costs of these two approaches would help assess the framework’s practical value.

I would be open to raising my rating if these concerns are convincingly addressed.

Reviewer_oQzLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEMS is an agent harness for text-to-image generation. It manages the generation process without changing the image generator's weights. It first uses a domain skill to improve the generation prompt and turns the original request into a checklist. It then generates an image, checks it, and rewrites the prompt if requirements are missing. It saves previous prompts, images, and feedback, and writes short notes about what worked and what to try next.

The experiments use two image generators and cover five general benchmarks and four domain-specific evaluations.

Critical strengths

S1. The method works with existing image generators and is tested on several tasks.

GEMS does not require retraining the image generator. The paper tests two generators across nine evaluations, studies the main components, and checks an alternative MLLM backend on GenEval2. This gives more support for the method than a small set of examples would.

S2. GEMS achieves better scores at similar image-generation budgets.

On GenEval2 with Z-Image-Turbo, GEMS scores 63.5 with an average of 2.80 generated images per prompt. CRAFT, with a reduced iteration limit, scores 55.2 at an average of 2.83 images. The cost table for Kimi K2.5 with Qwen-Image-2512 also reports a lower total estimated API cost for GEMS than for CRAFT and Maestro.

Critical weaknesses

W1. Does GEMS improve on a general-purpose agent harness?

GEMS mainly defines how the MLLM loads skills, calls the image generator and verifier, saves previous attempts, and decides whether to retry or stop. A direct baseline would be a general-purpose agent harness configured with the same MLLM, tools, and domain skill files. It could be instructed to generate an image, check it against the request, and use feedback and previous attempts to improve the next result.

The paper does not include this comparison. Removing memory or skills from GEMS shows that these components help, but does not tell us whether GEMS handles them better than a general-purpose harness. Related work already includes several of the same steps: CRAFT checks images and revises prompts, while Reflexion saves lessons from previous attempts. The missing evidence is whether GEMS's control flow and memory handling produce better images or lower costs than a general-purpose harness with the same resources.

W2. Were the skills designed specifically for the downstream benchmarks?

Three details raise this question:

  1. The four skills match the four downstream evaluations directly: Text Rendering with LongText-Bench, Spatial Intelligence with SpatialGenEval, Creative Drawing with CREA, and Aesthetic Drawing with ArtiMuse.
  2. Appendix A.3 says that information from the five general benchmarks was not used to build the skills. It does not give the same assurance for the downstream evaluations. It explicitly says that the downstream skills may have been developed with reference to domain-specific evaluation concepts.
  3. The skills were built from human-selected reference materials, drafted by Gemini, and reviewed by humans. Appendix A.3 does not identify those materials or explain what guided the revisions, so it is unclear whether the instructions were based on general domain knowledge or on the scoring rules of particular tests.

The authors state that the skills were fixed after construction and were not subsequently changed using benchmark feedback. This is useful, but it does not answer whether the benchmarks influenced their initial design. These points do not establish test leakage; they are reasons to ask how closely the skills were written around the tests.

This matters because Appendix A.5 says that the baselines did not receive the same skill files. If those files contain advice tailored to the benchmarks, better scores could come from that advice. The five general benchmarks provide some evidence that the skills help elsewhere, but they do not answer how the downstream skills were designed. A more convincing test would give the baselines the same skills and check the downstream gains on independently sourced prompts with a different evaluator.

W3. The reported score differences and stopping decisions need closer checks.

The key ablations include repeated runs, but the main table does not show the variation across runs for the strongest baselines. This matters for small differences such as 63.5 versus 62.4 for GEMS and CRAFT on GenEval2 with Z-Image-Turbo.

The system also stops when its own checker says that all requirements are met. The paper does not independently check a sample of these early-stopped outputs to show how often that judgment is correct. Finally, the full cost table covers Qwen-Image-2512, but the corresponding total costs and time per prompt are not reported for Z-Image-Turbo.

Questions for authors

Q1. How does GEMS compare with a general-purpose agent harness? Please use the same MLLM, image generator, verifier, and domain skill files under comparable total inference budgets. Give the baseline instructions to generate, check, and revise images using feedback. Does GEMS produce better images or reach the same quality at a lower cost? Which design choice accounts for any advantage? Please also test CRAFT with the same Planner and skill files.

Q2. Were downstream benchmarks used when writing the skills? Were any downstream test examples, scoring rules, or evaluator scores used to write, revise, or select the skills? Please provide the reference sources, Gemini drafting prompts, and criteria used during human review. Do the same skills still help on independently sourced prompts, judged by a different evaluator or by human reviewers who do not know which method produced each image?

Q3. Do the score differences hold across repeated runs? When the verifier says all requirements are met, are they actually met? Please report repeated-run results for GEMS and the strongest baselines. For a sample of images returned at early stopping, ask humans or an independent evaluator to check them against the original user requests. Please also report total inference costs and time per prompt for Z-Image-Turbo.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.