- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SearchGen-20K, SearchGen-Corpus-1M, and SearchGen-Bench to study knowledge-dependent image generation. The resources combine 20,937 tasks with task-specific verification criteria, archived searches, agent trajectories, and generated images. A gate-filter-integrate protocol controls how an agent retrieves and uses external evidence. After an SFT warm-start for the agent, the baseline trains the generator with DPO and then adapts the agent using rejection-sampling fine-tuning. Experiments with Klein-4B and BAGEL examine generation quality, changes in retrieval benefit, and the value of adapting the search policy.
Critical strengths
1.The linked task and search resources are a useful contribution. The 96,848 reasoning traces, 283,493 generated images, and 159,027 archived search sessions connect agent decisions to generation outcomes. Together with task-specific checklists and reference evidence, they support reproducible experiments on factual fulfillment and retrieval behavior, beyond evaluating visual quality alone.
2.The analysis includes meaningful controls on two generator architectures. Search-Intensive scores improve from 26.4 to 31.8 for Klein-4B and from 23.4 to 26.8 for BAGEL. The fixed-evidence diagnostics in Appendix D.5 show how retrieval benefit changes after generator training, while the gate, filter, and integration ablations in Appendix D.6 demonstrate that each protocol component matters. These experiments provide useful insight into when retrieval helps or harms generation.
Critical weaknesses
1.Independent validation of the co-training gains is limited. The same fine-tuned Qwen3-VL-8B judge provides the generator preferences, selects agent trajectories, and scores the final systems. The human study reports encouraging aggregate agreement (Spearman rho = 0.87 on 500 prompt-image pairs), but does not establish agreement on the smaller differences between training stages in Table 5. This leaves uncertainty about whether independent evaluation would confirm the reported gains, particularly on TextualSearch, where judge agreement is reported to be weaker.
2.The adaptation analysis does not fully separate initial generator-specific specialization from readaptation after generator evolution. Section 3.2 states that the initial SFT agent is not adapted to either generator. Table 7 and Appendix D.5 support generator-specific adaptation and changing retrieval benefits, but do not test an agent previously adapted to the base generator and then transferred unchanged to the DPO generator. Consequently, the results provide less direct support for the claim that generator updates make an already adapted agent stale.
Questions for authors
1.Does the 500-pair human study include held-out outputs from the different co-training stages? Can you report a paired human or independent-judge comparison of the SFT and final systems on the same Search-Intensive prompts, including TextualSearch?
2.Have you evaluated an agent adapted using the base generator’s feedback and then transferred unchanged to the DPO generator? Comparing it with the DPO-adapted agent would help distinguish initial specialization from the need to readapt after a generator update.
3.There are unresolved cross-references at lines 798 and 958 that should be corrected so readers can locate the intended discussions.
Summary of Rebuttal Responses
We sincerely thank the Area Chair and all three reviewers for their thoughtful feedback.
For convenience of ACs, we summarize the rebuttal progress below.
Highlights of Reviewer Feedback
First, we are encouraged that all three reviewers recognized the value of our work and all lean toward acceptance (score-3). The positive feedback are:
A valuable benchmark and reproducible research environment. Reviewer
9vUfhighlighted the "ground-truth-based evaluation framework" and "offline, reproducible search environment." ReviewerrtWmemphasized that prompt-specific checklists, rubrics, and archived retrieval provide a "structured setting for studying knowledge-dependent visual generation." Reviewertygclikewise described the linked resources as "a useful contribution." SearchGen connects 20,937 tasks to task-specific verification criteria, 96,848 reasoning traces, 283,493 generated images, and 159,027 archived search sessions, supporting research on both generation outcomes and the decisions that produce them.An important diagnosis of current generators. Reviewer
9vUfrecognized the evidence for a significant "world-knowledge bottleneck," particularly in open-weight generators. ReviewersrtWmandtygcvalued the analysis of when search helps or harms generation. Together, these observations identify a practical challenge: a visually convincing image can still depict the wrong identity, symbol, or fact, and relevant retrieved evidence can introduce unwanted content.Informative controls and a useful baseline. Reviewers
rtWmandtygchighlighted the stagewise analysis across two generator architectures and the gate, filter, and integration ablations. Reviewertygcspecifically noted that these experiments offer "useful insight into when retrieval helps or harms generation." The original results demonstrate improvements on Search-Intensive prompts for both Klein-4B and Bagel, providing a concrete starting point for further work.Summary of Responses to Main Concerns
Our responses clarify the contribution, address the concerns, and provide additional experiment results and analyses.
Resource availability and contribution scope — Reviewers
9vUfandrtWm. We clarified that the paper contributes useful benchmarking, valuable and rich resources, problem diagnosis, and a deliberately minimal co-training baseline. We emphasized that establishing SOTA system for agentic image generation is not the focus of the work.Evaluation reliability and the nature of the gains — Reviewers
rtWmandtygc. We set out paired independent-judge evaluation, uncertainty estimates, and separate knowledge-sensitive and image-quality components for VisualSearch and TextualSearch. These comparisons directly target the concern about using the same judge for training and evaluation.Generator-specific adaptation — Reviewers
rtWmandtygc. We specified the complete crossover between agents trained with base- and DPO-generator feedback and their evaluation on both generators. This directly targets the missing transfer comparison and the reviewer also acknowledged it.Comparisons and further co-training — Reviewer
rtWm. We detailed shared-generator comparisons with Gen-Searcher and GenEvolve and a further Klein co-training cycle, alongside complementary multi-round evidence from Wan. These extensions address the requested comparisons and continuation analysis.We incorporated the additional results and analyses in the revised manuscript (bottom of the appendix) and correct a cross-reference typo.
Research Enabled by Our Work
The reviewers have agreed on the value of this work:
SearchGen provides infrastructure for learning when to retrieve, selecting useful evidence, integrating references without copying irrelevant content, and adapting agents to different generators.
Its archived searches support controlled comparisons without live-search drift, while linked trajectories, images, and verifiers support supervised learning, preference learning, and future agentic reinforcement learning.
The findings and insights into how frontier models failed, and analyses on TextualSearch failures, provide concrete targets for future research.
Final Appeal
We respectfully ask the Area Chair to support acceptance based on the contribution recognized across the reviews. Our work makes the gap between visual plausibility and factual fulfillment measurable and gives the community the resources to study it. We believe these contributions offer lasting value to researchers developing more knowledgeable and reliable image-generation systems.
Thank you all for the patience, support and service.
The Authors