acceptodds
Under review as a conference paper at ICLR 2027

AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation

Abstract

Recent image generation models can take multiple reference images as input and combine them into a new image. However, multi-reference image generation remains challenging: models may omit or duplicate subjects from the references, or produce images in which multiple subjects appear unnaturally copied and pasted. Recent work has proposed image generation agents that combine image generation models, reasoning models, and a harness, which is an executable program that specifies how reference images are interpreted, how generation is performed, how outputs are diagnosed, and how the final image is selected. In multi-reference generation, however, references play different roles and outputs must satisfy many criteria at once, such as fidelity to each reference and the naturalness of the whole image, so many parts of the harness could be improved, from how references are processed to how outputs are diagnosed. This makes it hard to predict which changes will improve performance and by how much, and good harnesses difficult to design by hand; indeed, human-written harnesses vary widely in performance. We therefore propose AutoRef, which optimizes the harness automatically while keeping both models frozen: a coding agent iteratively rewrites the harness code. AutoRef separates the tasks whose feedback informs proposals from the tasks used to select candidates, and continues the search from a beam of the top-ranked harnesses on the selection tasks. Using this procedure, we discover AutoRef-Harness, which improves the open-weight FLUX.2 [klein] 4B from 5.72 to 7.37 (out of 10) on held-out four-reference tasks of the MultiBanana benchmark, matching or exceeding proprietary models including Nano Banana Pro and GPT-Image-1.5. Without re-optimization, the same harness also improves results when the generator, number of references, benchmark, evaluator, or reasoning model differs from those used in the search. We release our code and AutoRef-Harness at https://anonymous.4open.science/r/AutoRef-C506.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image generation, multi reference, agent, harness optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vfr6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether automatically optimizing the harness around frozen models can improve multi-reference image generation. The proposed method, AutoRef, uses a coding agent to modify harness implementations based on training-task scores, generated images, and execution traces. Candidate harnesses are selected using separate validation tasks, with multiple candidates retained as starting points for subsequent iterations. The resulting AutoRef-Harness combines reference-grounded prompt rewriting, two structurally different draft prompts, complaint-directed revision, and failure-aware selection within a budget of three image generations. Experiments demonstrate improvements on MultiBanana across multiple generators, alongside transfer to different reference counts, the unseen OmniContext benchmark, another evaluator, and another reasoning model. Component ablations and human preference evaluations provide additional evidence of its effectiveness.

Critical strengths
  1. The problem is practically relevant, and the contribution is clearly defined. Multi-reference image generation requires coordinating subject identity, attribute binding, scene composition, and style constraints. By optimizing the executable program surrounding frozen models, the paper demonstrates a practical way to improve generation quality without updating model parameters.
  2. The experimental coverage is extensive and includes several transfer evaluations. The evaluation extends beyond the generator and benchmark used during search to other generators, different reference counts, OmniContext, and another reasoning model. These experiments provide meaningful evidence that the discovered harness can transfer beyond its search setting.
Critical weaknesses
  1. The main automatic evaluation lacks cross-validation using different types of measurement tools. MultiBanana’s five dimensions cover instruction alignment, reference consistency, background–subject compatibility, physical realism, and visual quality, but all are assessed by the same VLM. Consequently, the reported scores do not clearly separate improvements in identity preservation, fine-grained reference fidelity, and aesthetic quality. The human evaluation and alternative VLM evaluator partially address this limitation. Nevertheless, complementary measurements, such as face identity similarity on suitable subsets and HPS-based preference scores, could provide evidence from different assessment mechanisms. Their applicability to non-face subjects and stylized images should be explicitly discussed.

  2. The stability of the gains on stronger generators remains insufficiently established. The discovered harness is primarily applied to FLUX.2 [klein] 4B and 9B, and Qwen-Image-Edit-2511. Although the paper compares against proprietary models, showing that a smaller generator with a harness approaches a stronger model does not establish that the stronger model would also benefit from the harness. It remains unclear whether the gains primarily compensate for limitations in reference understanding and constraint following, or also improve generators with stronger native multi-reference capabilities. A controlled comparison of direct generation, Best-of-3, and AutoRef-Harness on a stronger generator would help establish this boundary. The choice of model should account for availability at submission time; missing comparisons with contemporaneous models should not, by themselves, be grounds for rejection.

Questions for authors
  1. On the independence and interpretability of evaluation metrics. While the paper includes human preference evaluation, its main automatic evaluation relies on VLM judgments. Have the authors considered complementary domain-specific evaluators, such as ArcFace/LVFace for identity similarity on suitable face-related tasks, and HPS for visual preference? Do improvements under these metrics agree with the VLM score improvements? This would help distinguish gains in subject identity preservation, reference fidelity, and aesthetic quality.

  2. On effectiveness with stronger generators. Could the authors evaluate AutoRef-Harness on stronger multi-reference generators, such as higher-capacity FLUX.2 variants, Qwen-Image-3.0, or Nano Banana 2, where available and appropriate to the submission timeline? In particular, a comparison against direct generation and Best-of-3 under identical inputs and matched generation budgets would help determine whether the gains primarily compensate for weaker generators or remain consistent with stronger native capabilities.

  3. On potential overfitting to fixed evaluation criteria. MultiBanana uses the same scoring dimensions and prompt template across training, validation, and testing, and these criteria are accessible to Claude when it proposes candidate harnesses. Although validation scores and feedback are withheld, could the search trajectory still progressively adapt to the fixed evaluator’s preferences or blind spots? The paper already provides cross-evaluator and human evaluations; could the authors further clarify how strongly these results rule out this possibility? For example, do successive harnesses improve consistently under independent evaluators or human judgments throughout the search, rather than only at the final checkpoint?

3 comments

Thank you for the constructive review. We answer each question below. Table and figure numbers refer to the revised PDF. Differences are paired, with 95% bootstrap intervals over the test tasks.

Q1, W1

Have the authors considered complementary domain-specific evaluators, such as ArcFace/LVFace for identity similarity on suitable face-related tasks, and HPS for visual preference? Do improvements under these metrics agree with the VLM score improvements? This would help distinguish gains in subject identity preservation, reference fidelity, and aesthetic quality.

For identity similarity, we used DINO similarity, which is widely used to measure subject fidelity in subject-driven generation [1], instead of ArcFace/LVFace: face-recognition models apply only to human faces visible in both the reference and the output, whereas DINO measures person, object, and background references, including non-face subjects and stylized images, with one measure. We re-evaluated the base generator and AutoRef-Harness on the 133 test tasks with evaluators that were not used in the search (Appendix H.8, Table 18).

What it measuresEvaluatorTasksVanillaAutoRef-HarnessDifference (95% CI)Win rate
Overall scoreQwen3-VL-8B (used in the search)1335.727.37+1.64 [+1.19, +2.07]69.9%
Overall scoreGPT-5 (independent VLM)1335.236.99+1.76 [+1.51, +2.01]87.2%
Overall scoreGemini 2.5 Flash (independent VLM)1333.474.92+1.45 [+1.09, +1.83]71.4%
Subject identity preservationDINO (person references)1240.5300.515−0.014 [−0.044, +0.015]55.6%
Reference fidelityDINO (object references)1170.5850.611+0.026 [−0.015, +0.069]53.8%
Reference fidelityDINO (background references)640.4200.486+0.066 [+0.012, +0.120]59.4%
Aesthetic qualityHPSv313310.2410.14−0.11 [−0.34, +0.12]39.1%

GPT-5 and Gemini 2.5 Flash reproduce the gain, with intervals far from zero, so it is not specific to the evaluator used in the search. The domain-specific evaluators distinguish its source, as you suggested: reference fidelity is preserved and improves for background references, while the HPSv3 preference score does not increase. The gain is in following the instruction and integrating the references, including the background, into one scene.

Q2

In particular, a comparison against direct generation and Best-of-3 under identical inputs and matched generation budgets would help determine whether the gains primarily compensate for weaker generators or remain consistent with stronger native capabilities.

We added this comparison on Nano Banana 2 under identical inputs (MultiBanana with six references, 116 tasks) and a matched generation budget of three images (Appendix H.6, Table 16). The harness was optimized for FLUX.2 [klein] 4B with four references and is applied unchanged.

MethodGen.ObjectLocalGlobalBackgroundAvg.AutoRef-Harness minus this row (95% CI)p
Nano Banana 2 (direct generation)16.907.525.985.326.43+0.20 [−0.20, +0.58]0.32
+ Best-of-336.087.246.195.266.19+0.43 [+0.06, +0.82]0.025
+ AutoRef-Harness37.287.166.445.626.62––

It gives the highest score, and under the matched budget it is significantly better than Best-of-3. Best-of-3 falls below direct generation, so the benefit does not come from sampling more. The gain over direct generation is smaller than on FLUX.2 [klein] 4B, FLUX.2 [klein] 9B, and Qwen-Image-Edit-2511 (four references; Figure 1(c)) and is not statistically significant. So the gain is larger for the weaker, open-weight generators, and a harness that was never optimized for Nano Banana 2 or for six references does not hurt it.

Q3

For example, do successive harnesses improve consistently under independent evaluators or human judgments throughout the search, rather than only at the final checkpoint?

We re-scored the validation-best harness of each search iteration with an independent evaluator on the 133 test tasks (Appendix H.8, Table 19). Under GPT-5, the score increases over the search, from 5.23 to 6.99, and the final harness has the highest score, so the improvement is not limited to the final checkpoint.

IterationQwen3-VL-8B (used in the search)GPT-5 (independent)
0 (Vanilla)5.725.23
16.096.24
26.806.71
37.026.96
47.236.92
57.376.99

Q3

Although validation scores and feedback are withheld, could the search trajectory still progressively adapt to the fixed evaluator’s preferences or blind spots?

To check this, we compared, for each iteration, the score from the search evaluator with the score from GPT-5, which was not used in the search (table above). If the search had only adapted to the fixed evaluator's preferences or blind spots, GPT-5 would not show the improvement. It does: the score increases over the search, and the final harness has the highest score. The final harness also improves OmniContext under its GPT-4.1 evaluator, and human raters prefer it (Section 6.6, Figure 6). In iterations 4 and 5, the gain under GPT-5 is smaller than under the search evaluator (table above), so a part of it may be specific to the search evaluator.

We hope this answers your questions. Thank you again for the suggestions.

[1] Ruiz, et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.

Reviewer_vfr6OpenReview ↗

Response to Authors

Thank you for the detailed responses and additional experiments, which have addressed my concerns. I have no further questions and will maintain my rating of weak acceptance.

Reviewer_2P3xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoRef is a pipeline that searches executable harnesses to generate multi-reference images using separate feedback and selection task splits and maintaining multiple parent harnesses through beam search. The final selected harness comprises four components – reference grounded prompting, diverse drafts, complaint-directed revision, and failure-aware selection. This resulted in an empirically useful framework for harness search around the frozen models.

Critical strengths
  1. Autoref reports substantial gains on the MultiBanana benchmark: 7.37 versus 5.72 for vanilla, 6.01 for best-of-three, and 7.02 for IPR, as shown in Table 6 (under average column). The three leave-one-component-out ablations with 3 images support the contribution of harness components. Human evaluation also favors it over the base generator.

  2. Transfer of performance over unseen benchmarks without fine-tuning and with a different evaluator depicts breadth of the pipeline’s utility. However, testing on OmniContext shows a modest advantage over sampling, with average scores across 8 categories (see tables 10-11) of 8.85 for Autoref, 8.64 for best-of-three, and 8.3 for Vanilla, with Vanilla outperforming Autoref in two categories.

Critical weaknesses
  1. Authors have not established the incremental contribution of the optimizer. Compared with MetaHarness, the novel contribution is selection separation achieved through a train/val task split and a beam-parent rule. And when comparing with Meta-Harness, there is no clear control for the selection separation and parent beam variables, though for the greedy-to-beam framework, the beam retention test is more controlled. In Figure 6, in iteration 4, a candidate scoring 7.05 is selected over one scoring 7.04, based on only 48 validation tasks. Here, a statistical stability test is expected to address this concern.

  2. Authors need to tighten Comparative quality and efficiency claims. Their framework’s MultiBanana margins over IPR and nine-image Idea2Img are 0.35 and 0.25, which are smaller gaps, and no uncertainty estimates are given for these. Even human evaluation is not compared to iterative harness baselines like IPR and Idea2Img. Regarding efficiency, AutoRef uses multiple conditional calls to LLMs, whereas best-of-three uses one selection call (see Section 5). The study does not report Total tokens, latency, or cost. The small OmniContext margin further complicates the claim of superiority.

Questions for authors
  1. How stable are the search rankings when the pipeline is run on several seeds for fixed components and decided constraints? Can you run a controlled experiment to isolate split separation, keeping the parent rule fixed?

  2. Can you run and report paired confidence intervals for Autoref’s score differences against IPR, Idea2Img, and best-of-three (on OmniContext), and also report total LLM reasoning tokens, latency, and costs?

  3. How were the 50 tasks and four raters selected, randomly or by any strategy, and how were their votes aggregated? We would also like to know about the reliability of inter-rater agreement.

3 comments

Thank you for the constructive review. We answer each point below. Table and figure numbers refer to the revised PDF. All differences are paired, with 95% bootstrap intervals over the test tasks (Appendix H.7), and are computed from unrounded scores.

Q1

Can you run a controlled experiment to isolate split separation, keeping the parent rule fixed?

We added Unsplit Search, which removes only split separation from AutoRef and keeps the parent rule fixed (top-B by score, B=2), with the same proposer, candidate count, and task-evaluation budget.

SearchSplit separationParent beamParent ruleTasks per candidate (proposal-visible + selection-only)Test Avg.AutoRef minus this row (95% CI)
Meta-Harness✗–chosen by the proposer48 + 06.23+1.13 [+0.70, +1.57]
Greedy Search✓✗ (B=1)top-B by score48 + 486.67+0.70 [+0.30, +1.11]
Unsplit Search✗✓ (B=2)top-B by score96 + 06.93+0.43 [+0.09, +0.77]
AutoRef✓✓ (B=2)top-B by score48 + 487.37–

With the parent rule fixed, removing split separation lowers the test score, and the interval excludes zero. (Section 6.4, Figure 4 Right; Appendix G.2)

W1

Authors have not established the incremental contribution of the optimizer. Compared with MetaHarness, the novel contribution is selection separation achieved through a train/val task split and a beam-parent rule.

The table above establishes this. With everything else matched, selection separation (AutoRef vs. Unsplit Search) and the parent beam (AutoRef vs. Greedy Search) each raise the test score of the discovered harness significantly. Even AutoRef's second-best harness outperforms the final harnesses of Meta-Harness, Greedy Search, and Unsplit Search (Figure 4). Held-out selection and beams are established for prompt tuning; our contribution is to show that they make harness optimization work under a noisy visual evaluator. The selected harness is a contribution in its own right, and its evidence does not depend on the optimizer comparison: it is evaluated on test tasks never used in the search, and removing any of its components lowers the score (Section 6.5, Table 4).

Q1

How stable are the search rankings when the pipeline is run on several seeds for fixed components and decided constraints?

Due to time and cost constraints (about 50 hours per run, plus API costs), we could not rerun the full pipeline on several seeds. The existing run gives two kinds of evidence. First, the improvement does not rest on one candidate: the first-ranked harness improves on the held-out test split at every iteration (Table 19), and at least four of AutoRef's 20 candidates score above the final harnesses of Meta-Harness, Greedy Search, and Unsplit Search. Second, the rankings are stable to the choice of the selection tasks (next answer).

W1

In Figure 6, in iteration 4, a candidate scoring 7.05 is selected over one scoring 7.04, based on only 48 validation tasks. Here, a statistical stability test is expected to address this concern.

We agree that a gap of 0.01 on 48 tasks carries no information, and the search does not rely on it. We added the test (Appendix D.2, Tables 7 and 9): we scored all 20 search candidates on the 133 test tasks and redrew 48 selection tasks from them 10,000 times.

CandidateTest (133 tasks)Ranked first across redraws
AutoRef-Harness (first on validation)7.3756%
Second-best candidate7.2318%
Remaining 18 candidates6.09–7.1526% in total
IterationValidation-best candidate remains firstRemains in the beam (top 2)
13%14%
273%97%
340%74%
445%73%
587%96%
Chance25%50%

The top two candidates are more stable than the top one (second table above), which is why AutoRef keeps two parents. The selected harness is the best of the 20 candidates on test and ranks first about three times as often as the runner-up; in iterations 2–5, the validation-best candidate stays in the beam in most redraws (iteration 1, early in the search, where the candidates are not yet refined and still close, is the exception). In Figure 7, the 7.05 and 7.04 candidates belong to a second lineage; the final harness descends from the first-ranked candidate of that iteration (7.30), not from the 7.05 or the 7.04 candidate. The beam mattered in the other direction: the final harness descends from the candidate ranked second at iteration 3.

Q2

Can you run and report paired confidence intervals for Autoref’s score differences against IPR, Idea2Img, and best-of-three (on OmniContext), and also report total LLM reasoning tokens, latency, and costs?

Paired confidence intervals. We added them for MultiBanana and OmniContext (Appendix H.7, Table 17).

BenchmarkAutoRef-Harness versusScoreDifference95% CIp
MultiBanana (7.37)IPR7.02+0.35[+0.00, +0.70]0.049
MultiBanana (7.37)Idea2Img (9 images)7.12+0.25[−0.11, +0.60]0.17
MultiBanana (7.37)Best-of-36.01+1.36[+0.92, +1.77]<0.001
OmniContext (8.85)Best-of-38.64+0.21[−0.03, +0.47]0.084
OmniContext (8.85)Vanilla8.30+0.55[+0.25, +0.87]<0.001

AutoRef-Harness has the highest score in every comparison. On MultiBanana, the gain over Best-of-3 is significant, the gain over IPR is at the 5% threshold, and AutoRef-Harness matches Idea2Img with one-third of the images (at three images, Idea2Img scores 6.42; Table 10). On OmniContext, which was never used in the search, the gain over Vanilla is significant, and the gain over Best-of-3 is not. We ran IPR and Idea2Img on MultiBanana only.

Total LLM reasoning tokens, latency, and costs. We added Figure 5 (Section 6.4) and Figure 11 (Appendix H.5, all harnesses), which plot the average score of AutoRef-Harness, IPR, Idea2Img, Best-of-3, and other baselines against reasoning-model tokens, end-to-end latency, API cost, and generated images per task. The full harness scores higher than Idea2Img with one-third of the images and no more reasoning tokens or API cost; it uses more reasoning tokens than IPR. Its grounding-only component, also discovered in the search, offers a low-cost option when run alone: it is better and cheaper than Best-of-3 on all four measures. With the open-weight Qwen3-VL-32B as the reasoning model, the harness needs no API and still reaches 6.90 (Figure 4 Left).

W2

Authors need to tighten Comparative quality and efficiency claims. Their framework’s MultiBanana margins over IPR and nine-image Idea2Img are 0.35 and 0.25, which are smaller gaps, and no uncertainty estimates are given for these.

We added the uncertainty estimates and aligned the wording with them.

  • Comparative quality. AutoRef-Harness has the highest score among all harnesses, and we now report the intervals next to this statement. The margins over IPR and nine-image Idea2Img are small, as you note: the margin over IPR is at the 5% threshold, and the interval for Idea2Img includes zero, so we say that AutoRef-Harness matches Idea2Img with one third of the images (Section 6.4).
  • Efficiency. AutoRef-Harness uses multiple reasoning calls per task, where Best-of-3 uses one selection call, and Figure 5 now shows tokens used in reasoning calls. Our efficiency claim is that the full harness (highest score) and its grounding-only component (low cost) each extend the Pareto front of existing harnesses (Section 6.4, Figure 5). The grounding-only component was also discovered in the search and is not a human-written baseline: run alone with one image, it scores higher than Best-of-3 with three (6.91 vs. 6.01).

W2

Even human evaluation is not compared to iterative harness baselines like IPR and Idea2Img.

That is correct: our human evaluation compares AutoRef-Harness with generators (Section 6.6, Figure 6). Its purpose is to test whether the gains measured by the automatic evaluator hold for human raters, and they do: raters prefer AutoRef-Harness over the base generator, FLUX.2 [klein] 9B, and Seedream 4.5 (each p < 0.0001). For IPR and Idea2Img, the comparison rests on the automatic scores and the intervals above.

W2

The small OmniContext margin further complicates the claim of superiority.

OmniContext was never used in the search, has a different evaluator, and leaves little headroom: the base generator already scores 8.30 out of 10. Without re-optimization, AutoRef-Harness still improves it significantly and has the highest score among Vanilla, Best-of-3, and AutoRef-Harness. We do not claim a significant gain over Best-of-3 there (p = 0.084), and Vanilla is higher in two of the eight task types, as you note (Table 3).

Q3

How were the 50 tasks and four raters selected, randomly or by any strategy, and how were their votes aggregated? We would also like to know about the reliability of inter-rater agreement.

  • How the 50 tasks were selected. They were sampled randomly from the held-out test split.
  • How the four raters were selected. They are machine learning experts. They rated blind: method names were hidden, and the pair and left/right order were randomized.
  • How their votes were aggregated. We pool all votes: 4 raters × 50 tasks = 200 votes per baseline, without majority voting. We allowed a tie only when the outputs could not be distinguished. Intervals are a bootstrap over tasks.
  • Reliability of inter-rater agreement. Pairwise agreement is 66.2%: over the six rater pairs, among the comparisons where both raters picked a winner (ties excluded), the share where they picked the same output. This is in line with the human–human agreement reported in prior studies of human preference for generated images: 63.5% in HPS [1], and 65.3% (annotators) and 71.2% (researchers) in ImageReward [2]. Raters often disagree on individual tasks, so we rely on pooled votes, not on per-task majorities. These details, the rating interface (Figure 13), and the win and loss rates with 95% confidence intervals (Table 20) are in Appendix I.

We hope these additions address your concerns, and we would be grateful if you would reconsider your rating.

[1] Wu, et al. Human Preference Score: Better Aligning Text-to-Image Models with Human Preference. ICCV 2023.
[2] Xu, et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023.

Reviewer_FUtgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoRef searches for executable code that coordinates frozen reasoning and image-generation models for multi-reference generation. A coding agent reads training-task scores, evaluator explanations, execution traces, and generated images, then proposes changes to the harness. A separate validation split selects a beam of candidates; its scores and outputs are hidden from the proposer. The resulting harness uses reference-grounded prompts to generate two structurally different drafts, regenerates from explicit complaints about the better draft, and applies failure-aware pairwise selection. It uses three image generations per task. On the four-reference MultiBanana test split, it raises FLUX.2 [klein] 4B's score from 5.72 to 7.37. The paper tests transfer across generators, reference counts, benchmarks, evaluators, and reasoning models, and includes component ablations and human preference comparisons.

Critical strengths
  1. The discovered workflow addresses practical failures in multi-reference generation, and its decisions are easy to inspect. The search trace and worked execution example show how reference grounding, complementary drafts, explicit complaints, and conservative selection fit together. The component ablations also show that the improvement comes from more than drawing additional independent samples.

  2. The transfer tests cover another generator family, unseen reference counts, OmniContext with a different evaluator, and an open-weight reasoning model. Matching image counts in the baseline comparisons helps control the sampling budget. Human preferences provide further support: against the base FLUX.2 [klein] 4B, the harness wins 70% of comparisons and loses 17%.

Critical weaknesses
  1. The paper makes a stronger case for the final harness than for the search procedure that found it. Code evolution, held-out selection, and beam search are established mechanisms. The Meta-Harness-to-Greedy comparison changes both the proposal/selection split and the choice of parents, so it cannot isolate the benefit of task separation. Greedy-to-AutoRef is a more focused comparison, but the paper reports one five-iteration search trajectory without variation across independent proposer runs or task splits. Selection also repeatedly uses the same 48 validation tasks, making sensitivity to that split worth testing. A single grounded generation already scores 6.91 in Table 4, compared with 7.37 for the full system. It remains unclear how reliably automatic search improves on strong, simple grounded workflows, or how much each search choice contributes when total evaluation resources are matched (Sections 4, 6.4-6.5; Appendices C, D, and G.2).

  2. The overall score does not establish an advantage in reference fidelity or total cost. AutoRef improves reference consistency over its base generator, but its 7.15 in Table 6 is below Nano Banana Pro's 7.74 and GPT-Image-1.5's 7.63. Human evaluation asks for overall preference and gives a close result against Nano Banana Pro: 46% wins and 40% losses. Evaluation focused on identity and attribute preservation, with uncertainty estimates, would help explain what the aggregate gain represents. The image-count-matched experiments control generator sampling, but the three-image harness also calls a reasoning model. The paper does not report reasoning calls or tokens, end-to-end latency, monetary cost, or search cost. These omissions make it difficult to assess the quality-cost advantage over simple grounding and proprietary systems (Tables 1, 4, and 6; Section 6.6; Appendix G.1).

Questions for authors
  1. Please separate the effects of task separation and beam retention while controlling the proposer, parent-selection policy, candidate count, and total task-evaluation resources. How many proposal-visible and selection-only evaluations does each search baseline in Appendix G.2 receive?

  2. How much do the discovered harness and the method ranking vary across independent proposer/search runs, generation samples, and train/validation splits? Please give uncertainty estimates for the main paired comparisons, especially 7.37 versus 7.20 and the comparisons with the strongest human-written harnesses. Please also explain how human ratings are aggregated and report inter-rater agreement.

  3. Can you evaluate per-reference identity and attribute preservation with human judgments or another independently validated measure? Table 6 shows lower reference consistency than the proprietary baselines despite a higher overall average. Which fidelity errors remain, and are there task types where improved realism or background integration comes at the expense of reference preservation?

  4. What are the full runtime and search costs, including reasoning calls and tokens, latency, API cost or hardware usage, and image generations? Please compare quality and cost with the 6.91 grounding-only variant, a simple grounded multi-sample workflow, and proprietary generators given comparable prompt adaptation.

5 comments

Thank you for the detailed review. We answer each point below. Table and figure numbers refer to the revised PDF. All differences are paired, with 95% bootstrap intervals over the 133 test tasks (Appendix H.7), and are computed from unrounded scores.

W1

The paper makes a stronger case for the final harness than for the search procedure that found it. Code evolution, held-out selection, and beam search are established mechanisms.

We state what we claim for the search procedure and the final harness.

The search procedure. Held-out selection and beam search are established for prompt tuning within a fixed program, and we do not claim them as new mechanisms. Our claim is that these design choices make harness optimization work when failures cannot be verified by labels or tests and the only feedback is a noisy visual evaluator. The controlled comparison you asked for (Section 6.4, Figure 4 Right) supports this claim: removing task separation or beam retention lowers the test score of the discovered harness significantly. To our knowledge, AutoRef is the first harness optimization method for image generation.

The final harness is a fixed program, so its evidence does not depend on how it was found. We added paired intervals to the component ablation of Table 4 (Section 6.5), on the 133 test tasks, which were never used in the search:

Harness (rows of Table 4)Avg.AutoRef-Harness minus this row (95% CI)
Full harness (AutoRef-Harness)7.37–
− structurally diverse drafts (§5.2)7.01+0.36 [+0.00, +0.70]
− complaint-directed revision (§5.3)6.99+0.38 [+0.02, +0.75]
− failure-aware selection (§5.4)6.93+0.44 [+0.09, +0.79]
Grounding only (§5.1 alone, one image)6.91+0.46 [+0.13, +0.81]
Selection only (no reference-grounded prompting)6.15+1.22 [+0.76, +1.68]
Generator only (Vanilla)5.72+1.64 [+1.19, +2.07]

Removing any component lowers the score; only the interval for structurally diverse drafts touches zero. The full harness also improves on grounding only, and the interval excludes zero. Grounding only is not a human-written baseline: it is reference-grounded prompting (§5.1), a component discovered in the search, run alone with one image.

Q1, W1

Please separate the effects of task separation and beam retention while controlling the proposer, parent-selection policy, candidate count, and total task-evaluation resources. How many proposal-visible and selection-only evaluations does each search baseline in Appendix G.2 receive?

We added Unsplit Search, which removes only task separation from AutoRef. Greedy Search removes only beam retention. Greedy Search, Unsplit Search, and AutoRef share the proposer, the parent-selection policy (top-B by score), the candidate count (20), and the total task-evaluation resources (96 tasks per candidate).

SearchTask separationBeam retentionProposal-visible evals per candidateSelection-only evals per candidateTotal task evaluationsTest Avg.AutoRef minus this row (95% CI)
Meta-Harness✗–4809606.23+1.13 [+0.70, +1.57]
Greedy Search✓✗ (B=1)48481,9206.67+0.70 [+0.30, +1.11]
Unsplit Search✗✓ (B=2)9601,9206.93+0.43 [+0.09, +0.77]
AutoRef✓✓ (B=2)48481,9207.37–

Task separation and beam retention each contribute: removing task separation (Unsplit Search) or beam retention (Greedy Search) lowers the test score significantly. Even AutoRef's second-best harness (7.23) outperforms the final harnesses of Meta-Harness, Greedy Search, and Unsplit Search (Figure 4 Right), which also draw more images per task than AutoRef-Harness (Table 10). Each search was run once (see Q2), so the intervals are over the test tasks. Meta-Harness follows its own protocol (no selection split; the proposer chooses the parent) and is not a resource-matched comparison. (Section 6.4; Appendix G.2)

Q2, W1

How much do the discovered harness and the method ranking vary across independent proposer/search runs, generation samples, and train/validation splits?

For independent proposer/search runs, we could not rerun the search due to time and cost constraints (about 50 hours per run, plus API costs), so we ran each search variant once. To check how much the discovered harness and the method ranking vary across train/validation splits and generation samples, we redrew the selection tasks 10,000 times and computed paired intervals over the test tasks (details below).

  • Train/validation splits. We tested whether the same harness would be selected from the same 20 candidates if the selection tasks were different. We scored all 20 search candidates on the 133 test tasks and redrew 48 selection tasks from them 10,000 times (12 per task type, as in the validation split). The selected harness is the best of the 20 on test and ranks first in 56% of the redraws; the runner-up ranks first in 18%. The validation ranking tracks the test ranking (Spearman ρ = 0.84; for the training ranking, 0.74). Details: Appendix D.2, Tables 7–9.
  • Generation samples. Each task is scored from one independent sample, so the paired intervals over tasks already include generation randomness for every comparison in this response.

W1

Selection also repeatedly uses the same 48 validation tasks, making sensitivity to that split worth testing.

To test the sensitivity to that split, we re-ranked the four candidates of each iteration on 48 redrawn selection tasks, 10,000 times (Appendix D.2, Table 9):

IterationValidation-best candidate remains firstRemains in the beam (top 2)
13%14%
273%97%
340%74%
445%73%
587%96%
Chance25%50%

In iterations 2–5, the validation-best candidate stays in the beam in most redraws; in iteration 1, early in the search, the candidates are not yet refined and still close. The top two candidates are more stable than the top one, which is why AutoRef keeps two parents: the final harness descends from the candidate ranked second at iteration 3 (Figure 7), which a search with a single parent would have dropped.

Q2

Please give uncertainty estimates for the main paired comparisons, especially 7.37 versus 7.20 and the comparisons with the strongest human-written harnesses.

We added paired bootstrap intervals for the main comparisons (Appendix H.7, Table 17).

AutoRef-Harness (7.37) versusScoreDifference95% CIp
Nano Banana Pro7.20+0.17[−0.23, +0.55]0.41
Idea2Img (9 images)7.12+0.25[−0.11, +0.60]0.17
IPR7.02+0.35[+0.00, +0.70]0.049
Best-of-36.01+1.36[+0.92, +1.77]<0.001
Vanilla5.72+1.64[+1.19, +2.07]<0.001
  • 7.37 versus 7.20 (AutoRef-Harness versus Nano Banana Pro). The interval of the difference includes zero: with a 4B open-weight generator, AutoRef-Harness matches Nano Banana Pro.
  • Comparisons with the strongest human-written harnesses (IPR and Idea2Img). The gain over IPR is at the 5% threshold. For Idea2Img, the interval includes zero: AutoRef-Harness matches it with one-third of the images (at three images, Idea2Img scores 6.42; Table 10).

The gains over the base generator and Best-of-3 are significant, and AutoRef-Harness has the highest score in every comparison. Error bars are now shown in Figures 1, 4, 5, 10, 11, and 12.

Q2, W2

Please also explain how human ratings are aggregated and report inter-rater agreement.

We added the aggregation rule, the protocol, and the inter-rater agreement (Appendix I).

  • How human ratings are aggregated. We pool all votes: 4 raters × 50 tasks = 200 votes per baseline, without majority voting. Intervals are a bootstrap over tasks.
  • Inter-rater agreement. Pairwise agreement is 66.2%: over the six rater pairs, among the comparisons where both raters picked a winner (ties excluded), the share where they picked the same output. This is in line with the human–human agreement reported in prior studies of human preference for generated images: 63.5% in HPS [1], and 65.3% (annotators) and 71.2% (researchers) in ImageReward [2]. Raters often disagree on individual tasks, so we rely on pooled votes rather than per-task majorities.
  • Protocol. The 50 tasks were randomly sampled from the test split, and the four raters are machine learning experts. They rated blind: method names were hidden, and the pair and left/right order were randomized. A tie was allowed only when the outputs could not be distinguished.
BaselineWinLossWin − loss (95% CI)p
FLUX.2 [klein] 4B (base generator)70.0%17.0%+53.0 [+39.5, +66.0]<0.0001
FLUX.2 [klein] 9B64.5%24.0%+40.5 [+24.5, +55.5]<0.0001
Seedream 4.563.0%27.0%+36.0 [+22.5, +49.0]<0.0001
Nano Banana Pro46.0%40.5%+5.5 [−9.5, +20.5]0.52

Human raters prefer AutoRef-Harness over the base generator, the larger FLUX.2 [klein] 9B, and the proprietary Seedream 4.5. Against Nano Banana Pro, the result is close, as you note (W2): AutoRef-Harness and Nano Banana Pro are on par. (Section 6.6, Figure 6; Appendix I, Table 20)

Q3, W2

Can you evaluate per-reference identity and attribute preservation with human judgments or another independently validated measure?

We added DINO similarity between each reference and the output (Appendix H.8, Table 18). It is widely used to measure subject fidelity in subject-driven generation [3] and does not depend on the VLM evaluator. We localize each target in the reference and the output with Grounding DINO and SAM 2.1, then compare the DINOv2-L features of the two crops.

Per-reference similarity (DINO)TasksVanillaAutoRef-HarnessDifference (95% CI)
Person references1240.5300.515−0.014 [−0.044, +0.015]
Object references1170.5850.611+0.026 [−0.015, +0.069]
Background references640.4200.486+0.066 [+0.012, +0.120]

By this measure, reference preservation is not traded for realism: preservation of person and object references is unchanged, and preservation of background references improves.

Q3, W2

Table 6 shows lower reference consistency than the proprietary baselines despite a higher overall average.

As you note, reference consistency is below Nano Banana Pro (Table 10). Over the base generator, it improves significantly. To show what the aggregate gain represents, we compared the five criteria with the base generator and with Nano Banana Pro:

Criterion (columns of Table 10)AutoRef-HarnessMinus Vanilla (95% CI)Minus Nano Banana Pro (95% CI)
Instruction alignment (Inst.)6.57+1.32 [+0.93, +1.72]−0.37 [−0.76, +0.02]
Reference consistency (Ref.)7.15+1.28 [+0.74, +1.81]−0.59 [−1.01, −0.17]
Background–subject match (BG)7.20+2.16 [+1.51, +2.80]+0.58 [+0.03, +1.14]
Physical realism (Real.)7.77+2.13 [+1.55, +2.69]+0.98 [+0.45, +1.52]
Visual quality (Qual.)8.16+1.33 [+0.85, +1.80]+0.22 [−0.19, +0.64]
Average7.37+1.64 [+1.19, +2.07]+0.17 [−0.23, +0.55]

Over the base generator, all five criteria improve. AutoRef-Harness and Nano Banana Pro reach the same average through different strengths: Nano Banana Pro preserves the references better, and AutoRef-Harness integrates them better into one scene (background–subject match, physical realism). Reference consistency alone misses this integration: it stays high even when the references are pasted as-is and do not form one coherent scene (Appendix J, Figure 14), which is why we compare methods using the average of the five criteria.

[1] Wu, et al. Human Preference Score: Better Aligning Text-to-Image Models with Human Preference. ICCV 2023.
[2] Xu, et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023.
[3] Ruiz, et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reviews. Changes in the PDF are in blue, and table and figure numbers below refer to it.

Paper summary

AutoRef is, to our knowledge, the first harness optimization method for image generation: unlike math or coding, the only feedback is a noisy visual evaluator. With the generator and the reasoning model frozen, the discovered harness (a) brings the open-weight FLUX.2 [klein] 4B to the level of Nano Banana Pro and (b) transfers unchanged to other generators, reference counts, benchmarks, evaluators, and reasoning models. Task separation and beam retention are the design choices that make the search work under noisy feedback.

Key results

The harness (i) improves the base generator by a large margin, confirmed by independent VLMs and human raters, and (ii) has the highest score among all human-written and searched harnesses. (iii) Under matched resources, task separation and beam retention each improve the discovered harness.

We addressed the main concerns as follows.

1. Effects of task separation and beam retention (Reviewers FUtg, 2P3x)

We added Unsplit Search, which removes only task separation from AutoRef; Greedy Search removes only beam retention. Under the same proposer, parent-selection policy, candidate count, and task-evaluation resources, removing task separation or beam retention lowers the test score significantly. Even AutoRef's second-best harness outperforms the final harnesses of Meta-Harness, Greedy Search, and Unsplit Search (Section 6.4, Figure 4 Right).

2. Stability across seeds and train/validation splits (FUtg, 2P3x)

We scored all 20 search candidates on the 133 test tasks and redrew the 48 selection tasks 10,000 times. The selected harness is also the best on test and ranks first far more often than any other candidate, and selected harnesses improve on the test split at every iteration. We did not rerun the search with new seeds (about 50 hours per run, plus API costs); the evidence for the final harness does not depend on it: it is a fixed program evaluated on unseen test tasks (Appendix D.2; Table 19).

3. Uncertainty estimates for the main paired comparisons (FUtg, 2P3x)

We added paired comparisons with 95% confidence intervals. The gains over the base generator and Best-of-3 are significant, and the gain over IPR is at the 5% threshold (p = 0.049). AutoRef-Harness matches Nano Banana Pro with a 4B open-weight generator and Idea2Img with one-third of the images (Appendix H.7, Table 17).

4. Runtime and search costs, and comparison with simple grounded workflows (FUtg, 2P3x)

We added per-task costs (images, reasoning tokens, latency, and API cost) and the search cost. The full harness extends the Pareto front of existing harnesses on all four measures and achieves the highest score. Its grounding-only component, which was also discovered in the search, extends the front as a low-cost option when run alone. A simple grounded multi-sample workflow does not reach the full harness, so the gain of the full harness does not come from sampling more. With the open-weight Qwen3-VL-32B, the harness needs no API and still reaches 6.90 (Section 6.4, Figures 4 and 5; Appendix D.1, Table 6).

5. Independent evaluators and reference consistency (vfr6, FUtg)

We added evaluations with independent VLMs, DINO similarity, and HPSv3. GPT-5 and Gemini 2.5 Flash reproduce the improvement. Reference fidelity is preserved (DINO), and the gain comes from integrating references into one scene, not from aesthetic quality (HPSv3). Reference consistency improves significantly over the base generator. Against Nano Banana Pro, it is lower, but background–subject match and physical realism are higher, and human raters rate AutoRef-Harness on par (Appendix H.8, Table 18; Table 10).

6. Effectiveness with stronger generators (vfr6)

We added Nano Banana 2 with direct generation, Best-of-3, and AutoRef-Harness under matched generation budgets. The harness, transferred unchanged, gives the highest score and significantly outperforms Best-of-3, which falls below direct generation. Its gain over direct generation (+0.20) is smaller than for the open-weight generators (+1.25 to +1.65) (Appendix H.6, Table 16).

7. Potential overfitting to the fixed evaluator over the search (vfr6)

We re-scored the harness of each search iteration with an independent evaluator, GPT-5. The score increases over the search, from 5.23 to 6.99, and is highest for the final harness (Appendix H.8, Table 19).

8. Human rating aggregation and inter-rater agreement (FUtg, 2P3x)

We added the protocol (blind; votes pooled over the four raters), the inter-rater agreement (66.2%), and 95% confidence intervals. AutoRef-Harness is preferred over the base generator, FLUX.2 [klein] 9B, and Seedream 4.5 (each p < 0.0001), and is on par with Nano Banana Pro (Figure 6; Appendix I, Table 20).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.