acceptodds
Under review as a conference paper at ICLR 2027

Agentic Image Generation in the Wild

Abstract

Agentic image generation pairs an image generator with a search agent that retrieves references from the web, extending generation beyond the generator's training data. Real-world requests, however, often concern entities that appeared only days ago. We call this setting Image Generation in the Wild. In this setting, the web offers limited visual evidence, search results are crowded with easily confused entities, and the target is rarely shown on its own. To study this setting, we build an automated pipeline that synthesizes prompts from recent headlines and keeps only those whose real search results are highly confusing, yielding requests that are hard to solve yet natural to ask. With it, we construct Savanna, a benchmark of 500 prompts across 16 subcategories. On Savanna, even harness frameworks built on the frontier models retrieve a correct reference for only about half of the prompts. We then propose Locate-Corroborate-Isolate, a paradigm in which the agent first pins down the target's identity through text search, then verifies candidate images against multiple sources, and finally crops a clean reference for the generator. Trained on 15K trajectories that follow this paradigm, Ranger-8B achieves a search score of 67.2, outperforming harnesses equipped with frontier models. It also achieves the best generation quality with every generator we test, and surpasses prior agentic methods on other image generation benchmarks. We will release the pipeline, benchmark, and training data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Image Generation, Vision-Language Model, Search Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YERLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a practical problem that image generators struggle with new entities that didn't appear in the training data and might appear alongside other entities because they are relatively new. The paper contributes a benchmark named Savanna of 500 prompts across 16 subcategories. These prompts are challenging and directly retrieving evidence from the internet is likely to fail to retrieve the correct ones to facilitate the image generator. The authors thus propose a paradigm named Locate-Corroborate-Isolate, where agents verify the retrieved evidence with multiple sources, then crop the reference images for use by image generators. The paper also shows that training models on the trajectories following this paradigm can outperform the harnesses with frontier models and yield the best generation quality.

Critical strengths

S1. This paper identifies a challenging and practical problem of using agents to retrieve new entities for image generators. The challenges in this setup are also summarized into three aspects and lead to a solid benchmark where frontier models also struggle to retrieve correct evidence in this setup.

S2. This paper contributes a benchmark and a paradigm to improve agent performance in evidence retrieval and follow-up improvement in image generation with well-retrieved evidence. The methods are well-motivated and the performance improvement is effective, outperforming a wide range of baselines and generalizing well.

Critical weaknesses

W1. The paradigm is well-motivated and its effectiveness is validated via extensive experiments. The effects of different components or stages might need further exploration, for example, how performance varies if applied without cropping or without verification, although the additional computational costs of such ablation studies may be a consideration.

W2. The Corroborate stage appears to be a potential bottleneck in the proposed bottleneck. The current paradigm uses cross-modal verification and cross-image agreement, which might not be reliable in the proposed setup. The example in Figure 4 also presents a subtle entity, which might be challenging for VLMs to correctly identify as the desired evidence. Is there any measure to control or examine the quality at this stage?

Questions for authors

Overall I think the strengths outweigh the weaknesses and the major concerns are listed in the weaknesses. Minor question: the paper claims that the generation is bounded by reference quality, and it might be interesting to visualize the correlation between reference quality and generation quality for the same generator, and it will better support this claim.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YERL

Thank you for your feedback and valuable comments. Below is our response to your concerns.


W1 [How much does each LCI stage exactly contribute?]

(1) Isolate cropping ablation (Sec. 4.4, Fig. 7): On the 102 prompts where Ranger-8B calls crop_image, passing the cropped target region instead of the full retrieved image raises target identity scores by +6.3+6.3 points (28.1→34.428.1 \to 34.4) on Qwen-Image-Edit-2511 and +3.8+3.8 points (47.8→51.647.8 \to 51.6) on Nano Banana Pro.

(2) Controlled 5K SFT stage ablations (Sec. 4.5, Tab. 3): We randomly sample 5,000 trajectories from Ranger-15K and remove the corresponding steps while keeping the original trajectory's final result using Gemini-3.1-Pro (w/o Locate searches images directly without prior text search; w/o Corroborate directly selects the target image without verification steps). We use these trajectories to fine-tune Qwen3-VL-8B (see the table below; Sec. 4.5, Tab. 3):

  • Full LCI (59.7) vs. w/o Locate (45.6): Without Locate, the search score is limited by the recall of direct image search, dropping most in Culture (−26.3-26.3) and Entertainment (−15.9-15.9).
  • Full LCI (59.7) vs. w/o Corroborate (48.0): Without Corroborate, the agent fails to filter distractors in search results, dropping most in Fashion (−18.1-18.1) and Sports (−13.3-13.3).
MethodSFT DataLocateCorrob.IsolateEnt.Spo.Fas.Pro.Cul.Avg.
Qwen3-VL-8BNone✓\checkmark✓\checkmark✓\checkmark41.035.029.841.528.135.1
w/o Locate5K–✓\checkmark✓\checkmark48.248.351.148.931.645.6
w/o Corroborate5K✓\checkmark–✓\checkmark52.845.043.651.147.448.0
Ranger-8B5K✓\checkmark✓\checkmark✓\checkmark64.158.361.756.457.959.7

W2 [How reliably does Corroborate select the right image from noisy candidates?]

We check all candidate images returned by image_search on 100 sampled prompts to measure how often the agent picks the right reference once a correct target image appears in the search results (see the table below; App. M, Tab. 12). Ranger-8B correctly selects 67 out of 74 candidates (90.5%), while Gen-Searcher with the same backbone only selects 39 of 76 candidates (51.3%). This confirms that Corroborate helps the model select the target image more accurately.

MethodImg Searches / PromptCandidate Hit RateSelection PrecisionFinal Hit Rate
Gen-Searcher1.6076.0%51.3%39.0%
Ranger-8B (Ours)1.3474.0%90.5%67.0%

Minor Question [How strongly does retrieved reference quality affect final image generation?]

We show this in three ways in our paper:

(1) With vs. without Isolate on the same retrieved image (Sec. 4.4, Fig. 7): Fixing the generator and the retrieved source image on the 102 cropped prompts, cropping out surrounding distractors raises target identity scores by +6.3+6.3 points (28.1→34.428.1 \to 34.4) on Qwen-Image-Edit-2511 and +3.8+3.8 points (47.8→51.647.8 \to 51.6) on Nano Banana Pro.

(2) Same generator with references from different methods (Sec. 4.2, Tab. 1): Fixing Nano Banana Pro, generation scores increase consistently with reference Search scores from 26.1 (built-in search) and 31.6 (Unify-Agent, 23.2 Search) to 47.3 (Gen-Searcher, 43.5 Search) and 58.9 (Ranger-8B, 67.2 Search). Generation scores on Qwen-Image-Edit-2511 follow the same trend, rising from 24.9 to 46.8.

(3) Visual comparisons in the Appendix (App. J, Fig. 9 & App. L, Fig. 11): We also provide side-by-side visual results for the same generator in the Appendix. App. J (Fig. 9) shows Nano Banana Pro outputs using references retrieved by different methods across all five domains, and App. L (Fig. 11) shows paired outputs generated from full vs. cropped references.

We hope this addresses your concerns and are happy to discuss further.

Thank you for your response; my confusion has been resolved, and I will maintain my positive rating.

Glad to hear our response resolved your questions! Thanks again for taking the time to engage with our rebuttal and for your support of our work.

Reviewer_36gaOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates image generation for novel entities or events which emerge recently. In order to study this emergent situation, this study proposes a pipeline to create a benchmark on the fly by collecting news articles, extracting entities from text and image search, and filtering to ensure difficulties. Based on the benchmark data, an agentic image generation model is trained to follow the supervision of the training split of the data. Experiments show that existing model struggle, leading to the necessity to tune a model using the proposed training recipe.

Critical strengths
  • This work is investigating an important topic of image generation of novel entities and events. The pipeline is engineered to collect news events on the fly followed by entity extraction and filtering to control the difficulties level.
Critical weaknesses
  • Entity control: Although the experiments are designed to avoid the leakage using time-stamp, it does not have a control of leakage regarding entities or events. A particular entity might have appeared for long under similar events, and thus, it is not clear whether such leakage might have an impact.
  • Image retrieval: The final image generation relies on reference image retrieval, but no mention of the set of images. If they come from the retrieved images from news, this is additional leakage regarding the objects, so that a model have to select only from a narrow subset of an image pool. The image pool should be also controlled to see whether an object is never seen before or not.
  • Controlled difficulties: This work should control the dataset by differentiating hard and easy set by controlling the filtering step. Also, it is better to investigate how the threshold might have an impact to fine-tuning of a model.
Questions for authors

See the critical weaknesses.

The critical weaknesses regarding the task design is resolved after the rebuttal.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 36ga

Thank you for your feedback and valuable comments. Below is our response to your concerns, and we hope these clarifications and experiments will help you reconsider your rating.


W1 [Are training data disjoint from Savanna, and do test prompts cover newly unveiled July entities?]

(1) Zero overlap in target entities and final reference answers: We check SHA-256 hashes of all 68,355 training images against Savanna and manually inspect all trajectories with similar keywords. Target entities and final selected reference images have zero overlap. Only 3 prompts (0.6%) have unselected candidate thumbnails in intermediate searches. Removing these 3 prompts (n=497n=497, see the table below), Ranger-8B still achieves 67.0 Search (vs. 67.2) and 46.8 / 58.8 Generation (vs. 46.8 / 58.9). We will replace these trajectories in the final release.

(2) 95.2% of Savanna targets are first released in July 2026: Checking the first release date of all 500 targets shows that 98.4% (492/500) are released in 2026 and 95.2% (476/500) in July 2026. On these 476 July-release prompts (see the table below), Ranger-8B achieves 67.1 Search (vs. 67.2 overall) and leads all baselines by +22.3+22.3 to +32.6+32.6 points. Our 50 synthesis meta queries (App. E, Tab. 5) focus on release events (first look, debut, unveiling, launch) and exclude review terms (roundup, recap, retrospective).

MethodSearch (500)Search (497)Search (July 476)Qwen-IE (500 / 497)NB Pro (500 / 497)
Mind-Brush35.835.434.529.5 / 29.342.2 / 42.0
SCOPE41.141.140.230.2 / 30.245.1 / 45.1
Gen-Searcher43.543.243.337.9 / 37.747.3 / 47.1
GenClaw45.145.344.834.2 / 34.344.2 / 44.4
Ranger-8B67.267.067.146.8 / 46.858.9 / 58.8

W2 [Do evaluated agents retrieve from a pre-collected image pool or the open web?]

Evaluated agents never retrieve from a pre-collected image pool; every method searches the live open web from scratch (Sec. 2.2, Sec. 3.1 & App. D, Tab. 4):

  • Synthesis candidates stay on the evaluation side: The 8 candidate images from benchmark construction are used only to compute retrieval confusion κ(u)\kappa(u) and select the ground-truth reference image for the judge. Neither the candidates nor the ground-truth reference is shown to evaluated agents.
  • Live open-web search at test time: At inference time, each agent receives only the text prompt and searches live Google Search and Google Images via Serper.

W3 [How do models perform on subsets of different difficulties, and why filter training queries at κ(u)≥0.5\kappa(u) \ge 0.5?]

(1) Results across difficulty levels (Sec. 4.4, Fig. 6 & App. M, Tab. 11): Grouping all 500 prompts from easier to harder by retrieval confusion (κ(u)=0.50,0.625,0.75,0.875\kappa(u) = 0.50, 0.625, 0.75, 0.875), Ranger-8B leads all baselines by at least 16 points in every level (72.3 at κ=0.50\kappa=0.50 and 56.4 at κ=0.875\kappa=0.875; see the table below). On the Text-oriented (n=399n=399) and Image-oriented (n=101n=101) branches, Ranger-8B reaches 66.0 / 70.3, outperforming all baselines by 15 to 24 points.

(2) Why we filter queries at κ(u)≥0.5\kappa(u) \ge 0.5 (Sec. 2.2 & Sec. 3.2): We use this threshold to ensure that collected queries exhibit semantic crowding challenge, where at least half of the top search results are distractors. When κ(u)<0.5\kappa(u) < 0.5, most returned images already show the target, so a simple first-turn search succeeds. Filtering at κ(u)≥0.5\kappa(u) \ge 0.5 keeps the benchmark challenging and ensures that SFT trajectories contain meaningful verification steps for the model to learn.

Methodκ=0.50\kappa=0.50κ=0.625\kappa=0.625κ=0.75\kappa=0.75κ=0.875\kappa=0.875Search (Text / Img)Qwen-IE (Text / Img)NB Pro (Text / Img)
Unify-Agent22.923.129.316.023.2 / 23.224.9 / 24.931.7 / 31.1
Mind-Brush37.337.540.832.737.2 / 30.429.6 / 28.842.6 / 39.9
GenEvolve37.323.147.143.637.8 / 37.235.1 / 30.942.7 / 42.5
SCOPE42.248.150.335.942.3 / 36.130.6 / 27.945.3 / 43.2
Gen-Searcher30.131.755.448.744.7 / 38.838.8 / 34.648.6 / 41.7
GenClaw44.649.052.941.043.3 / 49.633.7 / 36.043.6 / 45.7
Ranger-8B72.367.378.356.466.0 / 70.346.2 / 48.157.8 / 62.8

We hope this addresses your concerns and kindly invite you to reconsider your rating.

Thank you for the response. I think the explanation makes the contribution clearer, and I've flipped my decision to the positive side.

Thank you for reading through our response and flipping your score! We're really glad the explanation made our contribution clearer, and we truly appreciate you engaging with us in the discussion!

Reviewer_tjcbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies image generation for recent entities whose reference images are difficult to find or identify. It introduces Savanna, a benchmark with 500 prompts, and the Locate-Corroborate-Isolate strategy for searching, checking, and cropping references. Ranger-8B is trained on about 15K teacher trajectories. It improves search and generation on Savanna and shows gains over the included agentic baselines on two existing benchmarks.

Critical strengths

Savanna studies a practical problem: finding the correct entity among similar or cluttered images. Separate search and generation scores help explain failures. Human checking of all benchmark samples and the judge-agreement study add useful validation.

Ranger improves the reported search score from 51.5 to 67.2 and gives better generation with both tested generators. Results on Mind-Bench and KnowGen provide further evidence beyond Savanna. Together, these results support a useful benchmark and trained-system contribution.

Critical weaknesses

Earlier methods already use text evidence to guide image search, and Gen-Searcher trains the same backbone for search and reference selection. Ranger also changes the training data and supervision. Figures 5-6 do not separate these factors. The cropping comparison is useful, but controlled tests for Locate and Corroborate would make the explanation stronger. This limits the claim about why the method works, while the full-system improvement remains valuable.

Different news dates do not guarantee different entities or reference images between training and test data. A new article can also describe an older entity. Appendix G provides useful validation of the date filter, but does not establish fully equivalent historical search conditions. An overlap audit and more precise wording would improve confidence. The current evidence does not show that leakage occurred.

Questions for authors

Did you check overlapping entities and duplicate reference images between Ranger-15K and Savanna? How many test entities were first introduced in July rather than only reported then? Were search restrictions and tool budgets comparable across methods? Please specify the evaluator model and how human-validation cases were selected. How does base Qwen3-VL-8B perform with the same LCI instructions and tools before training? If available, controlled Locate/Corroborate comparisons would help explain the gains.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tjcb [Part I]

Thank you for your feedback and valuable comments. Below is our response to your concerns.


W1, Q5 & Q6 [How much does each LCI stage exactly contribute?]

(1) Original Qwen3-VL-8B with LCI prompt and tools (Q5): We evaluate the original Qwen3-VL-8B-Instruct with the LCI prompt and tools on all 500 Savanna prompts, which only gets 35.1 Search score (see the table below, Row 1).

(2) Controlled 5K SFT stage ablations (W1, Q6): We randomly sample 5,000 trajectories from Ranger-15K and remove the corresponding steps while keeping the original trajectory's final result using Gemini-3.1-Pro (w/o Locate searches images directly without prior text search; w/o Corroborate directly selects the target image without verification steps). We use these trajectories to fine-tune Qwen3-VL-8B (see the table below; Sec. 4.5, Tab. 3):

  • Full LCI (59.7) vs. w/o Locate (45.6): Without Locate, the search score is limited by the recall of direct image search, dropping most in Culture (−26.3-26.3) and Entertainment (−15.9-15.9).
  • Full LCI (59.7) vs. w/o Corroborate (48.0): Without Corroborate, the agent fails to filter distractors in search results, dropping most in Fashion (−18.1-18.1) and Sports (−13.3-13.3).
MethodSFT DataLocateCorrob.IsolateEnt.Spo.Fas.Pro.Cul.Avg.
Qwen3-VL-8BNone✓\checkmark✓\checkmark✓\checkmark41.035.029.841.528.135.1
w/o Locate5K–✓\checkmark✓\checkmark48.248.351.148.931.645.6
w/o Corroborate5K✓\checkmark–✓\checkmark52.845.043.651.147.448.0
Ranger-8B5K✓\checkmark✓\checkmark✓\checkmark64.158.361.756.457.959.7

W2, Q1 & Q2 [Are training data disjoint from Savanna, and do test prompts cover newly unveiled July entities?]

(1) Zero overlap in target entities and final reference answers (W2, Q1): We check SHA-256 hashes of all 68,355 training images against Savanna and manually inspect all trajectories with similar keywords. Target entities and final selected reference images have zero overlap. Across the whole dataset, only 3 prompts (0.6%) have unselected candidate thumbnails that appear during intermediate searches on related topics. Removing these 3 prompts (n=497n=497, see the table below), Ranger-8B still achieves 67.0 Search (vs. 67.2) and 46.8 / 58.8 Generation (vs. 46.8 / 58.9). We will replace these trajectories with newly collected ones in the final release.

(2) 95.2% of Savanna targets are first released in July 2026 (Q2): Checking the first public release date of all 500 targets shows that 98.4% (492/500) are released in 2026 and 95.2% (476/500) are first released in July 2026. On these 476 July-release prompts (see the table below), Ranger-8B achieves 67.1 Search (vs. 67.2 overall) and leads all baselines by +22.3+22.3 to +32.6+32.6 points. During synthesis, our 50 meta queries (App. E, Tab. 5) focus on release events (first look, debut, unveiling, launch) and exclude review terms (roundup, recap, retrospective).

MethodSearch (All 500)Search (Clean 497)Search (July Debut 476)Search (Text / Img)Gen. Qwen-IE (500 / 497)Gen. NB Pro (500 / 497)
Mind-Brush (GPT-5.1)35.835.434.537.2 / 30.429.5 / 29.342.2 / 42.0
SCOPE (GPT-5.4)41.141.140.242.3 / 36.130.2 / 30.245.1 / 45.1
Gen-Searcher (Qwen3-VL-8B)43.543.243.344.7 / 38.837.9 / 37.747.3 / 47.1
GenClaw (Claude-Opus-4.6)45.145.344.843.3 / 49.634.2 / 34.344.2 / 44.4
Ranger-8B (Ours)67.267.067.166.0 / 70.346.8 / 46.858.9 / 58.8
AuthorsOpenReview ↗

Response to Reviewer tjcb [Part II]

Q3 [Do all baselines share the same search backend under fair tool budgets?]

All agentic baselines use the same Serper Google Search and Google Images backend (up to 5 images per call). Ranger-8B uses a stricter call limit than prior methods (8 calls vs. 10 for Gen-Searcher/GenEvolve and 60 for GenClaw) and averages only 2.50 search calls per prompt (see the table below; Sec. 4.1, App. M Tab. 10).

MethodBackboneAvailable ToolsMax BudgetAvg. Text / Img SearchAvg. Total CallsSearch Score
Mind-BrushGPT-5.1text_search, image_search21.00 / 1.002.0035.8
SCOPEGPT-5.4text_search, image_search121.39 / 2.233.6241.1
GenClawOpus-4.6search, todo_write, format_prompt601.03 / 1.035.3845.1
Unify-AgentBAGEL-14Btext_search, image_search121.88 / 2.344.2223.2
GenEvolveQwen3-VL-8Btext_search, image_search, query_knowledge100.21 / 1.733.9037.5
Gen-SearcherQwen3-VL-8Btext_search, image_search, browse101.01 / 1.572.5943.5
Ranger-8B (Ours)Qwen3-VL-8Btext_search, image_search, web_fetch, crop_image81.10 / 1.402.8667.2

Q4 [Which judge model scores Savanna across the 100 stratified validation prompts?]

All Savanna evaluations use Gemini-3.1-Pro (gemini-3.1-pro-preview, temperature 0; prompts in App. F Tab. 6–7). Across 100 random prompts (20 per domain), it reaches 98.0% agreement with human experts on search and identity scores and 94.5% on context rubrics (See Sec. 4.1, App. H Tab. 9).

We hope this addresses your concerns and are happy to discuss further.

Dear Reviewer tjcb,

The authors have posted their rebuttal. Could you please take a look and let us know whether your concerns have been addressed, or if you have any remaining questions or concerns?

Your feedback during the discussion would be greatly appreciated. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response and Summary of Manuscript Revisions

Dear Reviewers, AC, SAC, and PC,

Thank you for reviewing our paper. Below we summarize our responses to the reviewers.

Paper Overview and Strengths

[Paper Overview]: We study agentic image generation for newly emerging entities that are absent from generator weights (temporal novelty), mixed with distractors in search results (semantic crowding), or shown alongside other objects (spatial entanglement). We build Savanna, a news-driven benchmark of 500 prompts across 16 subcategories that scores reference retrieval separately from image generation. We also propose Locate-Corroborate-Isolate (LCI), which finds entity facts via text search, checks image candidates against page metadata, and crops the target region. Fine-tuning Qwen3-VL-8B on 15,119 verified trajectories (Ranger-15K) gives Ranger-8B, reaching 67.2 Search on Savanna (vs. 51.5 for the strongest frontier harness and 43.5 for Gen-Searcher) and top Generation scores with Qwen-Image-Edit-2511 (46.8) and Nano Banana Pro (58.9).

[Strengths]: The reviewers note that retrieving newly emerged entities from noisy web results is a practical, well-motivated problem (tjcb, YERL, 36ga). They also recognize our three-challenge formulation, automated benchmark pipeline, human-validated evaluation, and Ranger-8B's gains over frontier harnesses on Savanna, Mind-Bench, and KnowGen (tjcb, YERL, 36ga).

Responses to Key Concerns

1. Method Ablations and Comparison Fairness (tjcb, YERL)

[Search backend and tool budgets] (tjcb Q3): All baselines use the same Serper Google Search and Google Images backend. Ranger-8B uses a stricter call limit than prior methods (8 calls vs. 10–60) and averages 2.50 search calls per prompt (App. M, Tab. 10).

[Contribution of each LCI stage] (tjcb W1, Q5, Q6; YERL W1): Both Locate and Corroborate are necessary, and prompting alone is not enough. Without SFT, Qwen3-VL-8B with the LCI prompt and tools gets 35.1 Search. In 5K SFT ablations on the same prompts, full LCI reaches 59.7 Search, outperforming w/o Locate (45.6) and w/o Corroborate (48.0) (Sec. 4.5, Tab. 3).

[Reliability of Corroborate selection] (tjcb W1; YERL W2): On 100 sampled prompts, Ranger-8B and Gen-Searcher find a valid candidate at similar rates (74.0% vs. 76.0%), but Ranger-8B selects the right reference in 90.5% (67/74) of cases vs. 51.3% (39/76) for Gen-Searcher (App. M, Tab. 12).

[Reference quality vs. image generation] (YERL Minor Q): Better references improve both scores and visual results. First, on the same retrieved image, Isolate cropping raises identity scores by +6.3+6.3 on Qwen-Image-Edit-2511 and +3.8+3.8 on Nano Banana Pro (Sec. 4.4, Fig. 7; App. L, Fig. 11). Second, across methods, generation scores rise with Search scores from 26.1 to 58.9 on Nano Banana Pro and from 24.9 to 46.8 on Qwen-Image-Edit-2511 (Sec. 4.2, Tab. 1; App. J, Fig. 9).

2. Benchmark Design and Evaluation Details (tjcb, 36ga)

[Open-web retrieval at test time] (36ga W2): Evaluated agents never access a pre-collected image pool; every method receives only the text prompt and searches the live open web via Serper from scratch (Sec. 2.2, 3.1 & App. D, Tab. 4).

[Train-test separation] (tjcb W2, Q1; 36ga W1): Target entities and final references in Ranger-15K have zero overlap with Savanna. Across 68,355 training images, only 3 prompts (0.6%) have unselected candidate thumbnails in intermediate searches. Removing them leaves Ranger-8B almost unchanged (67.0 Search; 46.8 / 58.8 Generation), and we will replace these trajectories in the final release.

[Newly unveiled July 2026 entities] (tjcb Q2; 36ga W1): 95.2% (476/500) of Savanna targets are first released in July 2026, and 98.4% (492/500) in 2026. On the 476 July-release prompts, Ranger-8B achieves 67.1 Search (vs. 67.2 overall) and leads all baselines by +22.3+22.3 to +32.6+32.6 points (Sec. 2.2 & App. E, Tab. 5).

[Difficulty levels and filtering at κ(u)≥0.5\kappa(u) \ge 0.5] (36ga W3): Filtering at κ(u)≥0.5\kappa(u) \ge 0.5 ensures semantic crowding, keeping the benchmark challenging and providing meaningful verification steps in SFT trajectories (Sec. 2.2 & 3.2). Across the four confusion levels (κ=0.50\kappa=0.50 to 0.8750.875), Ranger-8B leads all baselines by at least 16 points in every level (72.3 at κ=0.50\kappa=0.50, 56.4 at κ=0.875\kappa=0.875; Sec. 4.4, Fig. 6), and reaches 66.0 / 70.3 on the Text- and Image-oriented branches (App. M, Tab. 11).

[Judge model and human validation] (tjcb Q4): All evaluations use Gemini-3.1-Pro (temperature 0), reaching 98.0% agreement with humans on search and identity scores and 94.5% on context rubrics across 100 random prompts (Sec. 4.1, App. F & App. H, Tab. 9).

PDF revisions (Sec. 2.2, 3.1, 4.1, 4.5 Tab. 3, App. H, App. M Tab. 10–12) are color-coded (blue: tjcb, green: YERL, amber: 36ga, lilac: shared).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.