- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether automatically optimizing the harness around frozen models can improve multi-reference image generation. The proposed method, AutoRef, uses a coding agent to modify harness implementations based on training-task scores, generated images, and execution traces. Candidate harnesses are selected using separate validation tasks, with multiple candidates retained as starting points for subsequent iterations. The resulting AutoRef-Harness combines reference-grounded prompt rewriting, two structurally different draft prompts, complaint-directed revision, and failure-aware selection within a budget of three image generations. Experiments demonstrate improvements on MultiBanana across multiple generators, alongside transfer to different reference counts, the unseen OmniContext benchmark, another evaluator, and another reasoning model. Component ablations and human preference evaluations provide additional evidence of its effectiveness.
Critical strengths
- The problem is practically relevant, and the contribution is clearly defined. Multi-reference image generation requires coordinating subject identity, attribute binding, scene composition, and style constraints. By optimizing the executable program surrounding frozen models, the paper demonstrates a practical way to improve generation quality without updating model parameters.
- The experimental coverage is extensive and includes several transfer evaluations. The evaluation extends beyond the generator and benchmark used during search to other generators, different reference counts, OmniContext, and another reasoning model. These experiments provide meaningful evidence that the discovered harness can transfer beyond its search setting.
Critical weaknesses
-
The main automatic evaluation lacks cross-validation using different types of measurement tools. MultiBanana’s five dimensions cover instruction alignment, reference consistency, background–subject compatibility, physical realism, and visual quality, but all are assessed by the same VLM. Consequently, the reported scores do not clearly separate improvements in identity preservation, fine-grained reference fidelity, and aesthetic quality. The human evaluation and alternative VLM evaluator partially address this limitation. Nevertheless, complementary measurements, such as face identity similarity on suitable subsets and HPS-based preference scores, could provide evidence from different assessment mechanisms. Their applicability to non-face subjects and stylized images should be explicitly discussed.
-
The stability of the gains on stronger generators remains insufficiently established. The discovered harness is primarily applied to FLUX.2 [klein] 4B and 9B, and Qwen-Image-Edit-2511. Although the paper compares against proprietary models, showing that a smaller generator with a harness approaches a stronger model does not establish that the stronger model would also benefit from the harness. It remains unclear whether the gains primarily compensate for limitations in reference understanding and constraint following, or also improve generators with stronger native multi-reference capabilities. A controlled comparison of direct generation, Best-of-3, and AutoRef-Harness on a stronger generator would help establish this boundary. The choice of model should account for availability at submission time; missing comparisons with contemporaneous models should not, by themselves, be grounds for rejection.
Questions for authors
-
On the independence and interpretability of evaluation metrics. While the paper includes human preference evaluation, its main automatic evaluation relies on VLM judgments. Have the authors considered complementary domain-specific evaluators, such as ArcFace/LVFace for identity similarity on suitable face-related tasks, and HPS for visual preference? Do improvements under these metrics agree with the VLM score improvements? This would help distinguish gains in subject identity preservation, reference fidelity, and aesthetic quality.
-
On effectiveness with stronger generators. Could the authors evaluate AutoRef-Harness on stronger multi-reference generators, such as higher-capacity FLUX.2 variants, Qwen-Image-3.0, or Nano Banana 2, where available and appropriate to the submission timeline? In particular, a comparison against direct generation and Best-of-3 under identical inputs and matched generation budgets would help determine whether the gains primarily compensate for weaker generators or remain consistent with stronger native capabilities.
-
On potential overfitting to fixed evaluation criteria. MultiBanana uses the same scoring dimensions and prompt template across training, validation, and testing, and these criteria are accessible to Claude when it proposes candidate harnesses. Although validation scores and feedback are withheld, could the search trajectory still progressively adapt to the fixed evaluator’s preferences or blind spots? The paper already provides cross-evaluator and human evaluations; could the authors further clarify how strongly these results rule out this possibility? For example, do successive harnesses improve consistently under independent evaluators or human judgments throughout the search, rather than only at the final checkpoint?
3 comments
Q3
For example, do successive harnesses improve consistently under independent evaluators or human judgments throughout the search, rather than only at the final checkpoint?
We re-scored the validation-best harness of each search iteration with an independent evaluator on the 133 test tasks (Appendix H.8, Table 19). Under GPT-5, the score increases over the search, from 5.23 to 6.99, and the final harness has the highest score, so the improvement is not limited to the final checkpoint.
| Iteration | Qwen3-VL-8B (used in the search) | GPT-5 (independent) |
|---|---|---|
| 0 (Vanilla) | 5.72 | 5.23 |
| 1 | 6.09 | 6.24 |
| 2 | 6.80 | 6.71 |
| 3 | 7.02 | 6.96 |
| 4 | 7.23 | 6.92 |
| 5 | 7.37 | 6.99 |
Q3
Although validation scores and feedback are withheld, could the search trajectory still progressively adapt to the fixed evaluator’s preferences or blind spots?
To check this, we compared, for each iteration, the score from the search evaluator with the score from GPT-5, which was not used in the search (table above). If the search had only adapted to the fixed evaluator's preferences or blind spots, GPT-5 would not show the improvement. It does: the score increases over the search, and the final harness has the highest score. The final harness also improves OmniContext under its GPT-4.1 evaluator, and human raters prefer it (Section 6.6, Figure 6). In iterations 4 and 5, the gain under GPT-5 is smaller than under the search evaluator (table above), so a part of it may be specific to the search evaluator.
We hope this answers your questions. Thank you again for the suggestions.
[1] Ruiz, et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.
Response to Authors
Thank you for the detailed responses and additional experiments, which have addressed my concerns. I have no further questions and will maintain my rating of weak acceptance.
Thank you for the constructive review. We answer each question below. Table and figure numbers refer to the revised PDF. Differences are paired, with 95% bootstrap intervals over the test tasks.
Q1, W1
For identity similarity, we used DINO similarity, which is widely used to measure subject fidelity in subject-driven generation [1], instead of ArcFace/LVFace: face-recognition models apply only to human faces visible in both the reference and the output, whereas DINO measures person, object, and background references, including non-face subjects and stylized images, with one measure. We re-evaluated the base generator and AutoRef-Harness on the 133 test tasks with evaluators that were not used in the search (Appendix H.8, Table 18).
GPT-5 and Gemini 2.5 Flash reproduce the gain, with intervals far from zero, so it is not specific to the evaluator used in the search. The domain-specific evaluators distinguish its source, as you suggested: reference fidelity is preserved and improves for background references, while the HPSv3 preference score does not increase. The gain is in following the instruction and integrating the references, including the background, into one scene.
Q2
We added this comparison on Nano Banana 2 under identical inputs (MultiBanana with six references, 116 tasks) and a matched generation budget of three images (Appendix H.6, Table 16). The harness was optimized for FLUX.2 [klein] 4B with four references and is applied unchanged.
It gives the highest score, and under the matched budget it is significantly better than Best-of-3. Best-of-3 falls below direct generation, so the benefit does not come from sampling more. The gain over direct generation is smaller than on FLUX.2 [klein] 4B, FLUX.2 [klein] 9B, and Qwen-Image-Edit-2511 (four references; Figure 1(c)) and is not statistically significant. So the gain is larger for the weaker, open-weight generators, and a harness that was never optimized for Nano Banana 2 or for six references does not hurt it.