acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking and Evolving the Capability Boundary in Agentic Visual Generation

Abstract

Visual generators render complex scenes convincingly, yet they confidently fabricate the characters, symbols, and facts they never learned. This world-knowledge bottleneck is structural: generators learn from fixed corpora, whereas user requests are unbounded, evolving, and long-tailed. To measure and study it, we construct the first large-scale dataset SearchGen-20K, SearchGen-Corpus-1M and SearchGen-Bench featuring 20,937 production-level generation tasks paired with truth-grounded verifiers that check the requested content rather than image quality alone. Comparing open-weight generators with agent-powered frontier APIs, our analysis reveals a substantial gap of up to 35pp in handling production-level requests. Agentic visual generation, in which an agent retrieves evidence for the generator, is the natural remedy. Yet we identify two challenges: naive search can override generator capabilities due to search noise, and agent become stale when generator evolves its capability. In response, we showcase a minimal baseline approach that introduces a noise-resistant agentic protocol and co-adapts the generator and the agent. Data, code, and models are released for future research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tygcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SearchGen-20K, SearchGen-Corpus-1M, and SearchGen-Bench to study knowledge-dependent image generation. The resources combine 20,937 tasks with task-specific verification criteria, archived searches, agent trajectories, and generated images. A gate-filter-integrate protocol controls how an agent retrieves and uses external evidence. After an SFT warm-start for the agent, the baseline trains the generator with DPO and then adapts the agent using rejection-sampling fine-tuning. Experiments with Klein-4B and BAGEL examine generation quality, changes in retrieval benefit, and the value of adapting the search policy.

Critical strengths

1.The linked task and search resources are a useful contribution. The 96,848 reasoning traces, 283,493 generated images, and 159,027 archived search sessions connect agent decisions to generation outcomes. Together with task-specific checklists and reference evidence, they support reproducible experiments on factual fulfillment and retrieval behavior, beyond evaluating visual quality alone.

2.The analysis includes meaningful controls on two generator architectures. Search-Intensive scores improve from 26.4 to 31.8 for Klein-4B and from 23.4 to 26.8 for BAGEL. The fixed-evidence diagnostics in Appendix D.5 show how retrieval benefit changes after generator training, while the gate, filter, and integration ablations in Appendix D.6 demonstrate that each protocol component matters. These experiments provide useful insight into when retrieval helps or harms generation.

Critical weaknesses

1.Independent validation of the co-training gains is limited. The same fine-tuned Qwen3-VL-8B judge provides the generator preferences, selects agent trajectories, and scores the final systems. The human study reports encouraging aggregate agreement (Spearman rho = 0.87 on 500 prompt-image pairs), but does not establish agreement on the smaller differences between training stages in Table 5. This leaves uncertainty about whether independent evaluation would confirm the reported gains, particularly on TextualSearch, where judge agreement is reported to be weaker.

2.The adaptation analysis does not fully separate initial generator-specific specialization from readaptation after generator evolution. Section 3.2 states that the initial SFT agent is not adapted to either generator. Table 7 and Appendix D.5 support generator-specific adaptation and changing retrieval benefits, but do not test an agent previously adapted to the base generator and then transferred unchanged to the DPO generator. Consequently, the results provide less direct support for the claim that generator updates make an already adapted agent stale.

Questions for authors

1.Does the 500-pair human study include held-out outputs from the different co-training stages? Can you report a paired human or independent-judge comparison of the SFT and final systems on the same Search-Intensive prompts, including TextualSearch?

2.Have you evaluated an agent adapted using the base generator’s feedback and then transferred unchanged to the DPO generator? Comparing it with the DPO-adapted agent would help distinguish initial specialization from the need to readapt after a generator update.

3.There are unresolved cross-references at lines 798 and 958 that should be corrected so readers can locate the intended discussions.

Reviewer_rtWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the world-knowledge limitations of text-to-image generators, which can produce visually convincing images but struggle with unfamiliar entities and factual details. To study this problem, the authors introduce SearchGen-20K, a dataset of 20,937 knowledge-intensive generation tasks, along with SearchGen-Bench for evaluation and SearchGen-Corpus-1M for reproducible retrieval. They show that blindly incorporating search results can sometimes harm generation quality. To address this issue, the authors propose a gate–filter–integrate protocol that helps an agent decide when to search, which references to select, and how to use them. They further introduce a co-training framework that improves the generator through search-conditioned DPO and then adapts the agent to the updated generator through RFT. Experiments on Klein-4B and Bagel demonstrate improvements in generation quality, supporting the potential of jointly adapting search agents and image generators.

Critical strengths

1. Task and evaluation design. The prompt-specific checklists and rubrics assess factual and entity-level requirements alongside image quality. Archived retrieval results allow systems to be compared using fixed search evidence. Together, these resources provide a structured setting for studying knowledge-dependent visual generation.

2. Search and adaptation analyses. The paper documents cases in which search harms generation, reports stagewise results on two generator architectures, examines changes in retrieval usefulness and search decisions, and ablates the gate, filter, and integration steps. These analyses add context to the reported score changes and the behavior of the search agent.

Critical weaknesses

1. Insufficient comparison with existing trained search agents. The paper identifies Gen-Searcher as its closest related work but does not provide a direct experimental comparison. GenEvolve, which also trains image-generation agents using search, visual references, and generation feedback, is not discussed. A controlled comparison, or a discussion of the practical barriers to such a comparison, would help clarify the incremental benefits of the proposed co-training framework over prior agent-only approaches.

2. Limited evaluation of iterative co-training. The paper performs only one round of generator DPO followed by agent RFT. While this demonstrates the benefit of a single adaptation cycle, it remains unclear whether the updated agent can further improve the generator in subsequent rounds. Given the paper's emphasis on evolving capabilities through co-training, evaluating additional iterations would help determine whether the improvements continue, saturate, or degrade.

3. Insufficient analysis and independent validation of performance gains. Table 5 reports aggregate scores without breaking down the improvements in knowledge-sensitive metrics such as Checklist and Rubric, making it difficult to distinguish better factual grounding from general image-quality improvements. Furthermore, the same VLM judge is used for training feedback and final evaluation, raising concerns about potential judge-specific optimization. The final system outperforms the Gemini-3-Flash agent by only 0.6–0.7 points, but the statistical uncertainty and independent validation of this small advantage are not reported.

Questions for authors

1. Comparison with existing trained search agents. Can the authors compare their approach with Gen-Searcher and GenEvolve on SearchGen-Bench using the same downstream generator, comparable retrieval budgets, and evaluation protocol? This would help clarify the incremental contribution of the proposed co-training framework over existing agent-only approaches.

2. Multi-round co-training. Have the authors explored additional rounds of alternating generator DPO and agent RFT? Do the improvements continue across multiple cycles, or do they saturate or degrade? How do retrieval benefits and the agent's search decisions evolve over these iterations?

3. Generator-specific agent adaptation. Table 7 only evaluates an RFT agent trained with the DPO generator. Can the authors also train an RFT agent with the base generator and cross-evaluate both RFT agents on the base and DPO generators? This would provide stronger evidence that the learned search strategies are generator-specific.

4. Knowledge-specific improvements and evaluation reliability. Could the authors report stagewise Checklist and Rubric scores in Table 5, separately for VisualSearch and TextualSearch? This would help clarify whether the improvements primarily reflect better knowledge grounding or general image quality. Additionally, could the authors provide paired confidence intervals for the stagewise gains and the small advantage over the Gemini-3-Flash agent, and validate these results using an independent VLM judge or blinded human evaluation?

Reviewer_9vUfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the SearchGen suite of resources, designed to evaluate the boundaries of world knowledge in image generation models using production-grade text-to-image prompts and ground-truth-verified checkers. It also proposes a minimal baseline for agentic visual generation: a Vision-Language Model (VLM) agent determines whether to perform a search, filters the retrieved evidence, and integrates it into the input for the image generator. Through joint training—utilizing Direct Preference Optimization (DPO) for the generator and Reinforcement Fine-Tuning (RFT) for the agent—both the agent and the generator are enabled to co-adapt as their respective capability boundaries evolve.

Critical strengths
  1. This paper introduces SearchGen-20K, SearchGen-Corpus-1M, and SearchGen-Bench, which establish a ground-truth-based evaluation framework and an offline, reproducible search environment tailored for production-grade visual generation requests.

  2. Using this benchmark, the paper demonstrates that visual generators face a significant "world-knowledge bottleneck"—evidenced by a substantial performance gap between familiar requests and knowledge-intensive ones—a phenomenon particularly pronounced in open-weight generative models.

Critical weaknesses
  1. Although the authors have released the code (anonymized), the dataset constitutes the primary contribution of this work; however, there is no indication that the full dataset will be released, which diminishes the paper's overall contribution.

  2. The paper acknowledges that "TextualSearch" remains a challenge; specifically, the performance of both agents using Klein-4B-DPO falls below that of the "no-search" baseline. This indicates that the current strategy fails to fully resolve the issues related to textual facts and the rendering of text within images.

Questions for authors

See Critical Weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Responses

We sincerely thank the Area Chair and all three reviewers for their thoughtful feedback.

For convenience of ACs, we summarize the rebuttal progress below.


Highlights of Reviewer Feedback

First, we are encouraged that all three reviewers recognized the value of our work and all lean toward acceptance (score-3). The positive feedback are:

  • A valuable benchmark and reproducible research environment. Reviewer 9vUf highlighted the "ground-truth-based evaluation framework" and "offline, reproducible search environment." Reviewer rtWm emphasized that prompt-specific checklists, rubrics, and archived retrieval provide a "structured setting for studying knowledge-dependent visual generation." Reviewer tygc likewise described the linked resources as "a useful contribution." SearchGen connects 20,937 tasks to task-specific verification criteria, 96,848 reasoning traces, 283,493 generated images, and 159,027 archived search sessions, supporting research on both generation outcomes and the decisions that produce them.

  • An important diagnosis of current generators. Reviewer 9vUf recognized the evidence for a significant "world-knowledge bottleneck," particularly in open-weight generators. Reviewers rtWm and tygc valued the analysis of when search helps or harms generation. Together, these observations identify a practical challenge: a visually convincing image can still depict the wrong identity, symbol, or fact, and relevant retrieved evidence can introduce unwanted content.

  • Informative controls and a useful baseline. Reviewers rtWm and tygc highlighted the stagewise analysis across two generator architectures and the gate, filter, and integration ablations. Reviewer tygc specifically noted that these experiments offer "useful insight into when retrieval helps or harms generation." The original results demonstrate improvements on Search-Intensive prompts for both Klein-4B and Bagel, providing a concrete starting point for further work.


Summary of Responses to Main Concerns

Our responses clarify the contribution, address the concerns, and provide additional experiment results and analyses.

  • Resource availability and contribution scope — Reviewers 9vUf and rtWm. We clarified that the paper contributes useful benchmarking, valuable and rich resources, problem diagnosis, and a deliberately minimal co-training baseline. We emphasized that establishing SOTA system for agentic image generation is not the focus of the work.

  • Evaluation reliability and the nature of the gains — Reviewers rtWm and tygc. We set out paired independent-judge evaluation, uncertainty estimates, and separate knowledge-sensitive and image-quality components for VisualSearch and TextualSearch. These comparisons directly target the concern about using the same judge for training and evaluation.

  • Generator-specific adaptation — Reviewers rtWm and tygc. We specified the complete crossover between agents trained with base- and DPO-generator feedback and their evaluation on both generators. This directly targets the missing transfer comparison and the reviewer also acknowledged it.

  • Comparisons and further co-training — Reviewer rtWm. We detailed shared-generator comparisons with Gen-Searcher and GenEvolve and a further Klein co-training cycle, alongside complementary multi-round evidence from Wan. These extensions address the requested comparisons and continuation analysis.

We incorporated the additional results and analyses in the revised manuscript (bottom of the appendix) and correct a cross-reference typo.


Research Enabled by Our Work

The reviewers have agreed on the value of this work:

  • SearchGen provides infrastructure for learning when to retrieve, selecting useful evidence, integrating references without copying irrelevant content, and adapting agents to different generators.

  • Its archived searches support controlled comparisons without live-search drift, while linked trajectories, images, and verifiers support supervised learning, preference learning, and future agentic reinforcement learning.

  • The findings and insights into how frontier models failed, and analyses on TextualSearch failures, provide concrete targets for future research.


Final Appeal

We respectfully ask the Area Chair to support acceptance based on the contribution recognized across the reviews. Our work makes the gap between visual plausibility and factual fulfillment measurable and gives the community the resources to study it. We believe these contributions offer lasting value to researchers developing more knowledgeable and reliable image-generation systems.


Thank you all for the patience, support and service.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.