acceptodds
Under review as a conference paper at ICLR 2027

EvoGen-Harness: Learning Where and How to Evolve Image-Generation Harnesses

16 views

Abstract

Modern text-to-image (T2I) systems can be improved without modifying generator parameters by adapting the external system around frozen generators. However, existing approaches typically optimize a predefined dimension, such as prompts, routing, or workflows, restricting the space in which generation failures can be corrected. Allowing multiple generator-external responsibilities to evolve provides a broader adaptation space, but introduces a new challenge: visual feedback reveals what failed, but not where persistent evolution should occur or how this space should be explored efficiently. We introduce EVOGEN-HARNESS, a generator-agnostic framework for multi-responsibility image-generation harness evolution, together with TRACE (Trajectory-Relative Attribution and Coordinated Evolution). TRACE aggregates evidence across stochastic executions, uses failure attribution as a search prior to focus candidate updates, and progressively re-attributes residual failures to coordinate evolution across responsibilities, while NO-PATCH and held-out validation prevent unnecessary or harmful updates. Across GenEval2, T2I-CompBench++, and WISE, EVOGEN-HARNESS improves over the strongest evaluated baselines by +0.2633, +0.0720, and +0.0752, respectively, while achieving 87.9–91.4% attribution recall, 94.8% NO-PATCH accuracy, and only 1.9% regression. These results demonstrate that attribution-guided multi-responsibility evolution can substantially enhance frozen T2I systems beyond single-dimension adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Generation Agents, Self-Evolving Agents, Agent Harness Engineering, Text-to-Image Generation, Generative AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oRqTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EvoGen-Harness, which improves frozen text-to-image generators by evolving five persistent external responsibilities: Policy, Tools, Skills, Middleware, and Memory. Its TRACE mechanism combines trajectory-based attribution, localized edit proposals, matched stochastic evaluation, progressive re-attribution, and validation before persistence. Experiments report substantial improvements on three benchmarks, supported by component ablations, controlled attribution tests, and evaluations across three frozen generators.

Critical strengths
  1. A consequential formulation of external system improvement. The paper distinguishes identifying a visual failure from deciding which persistent system component should change. This broadens the correction space beyond predefined prompt or routing optimization and addresses a practical challenge in adaptive generation systems.

  2. Strong reported results with complementary validation. The framework improves performance across three benchmarks, while ablations support the roles of attribution, matched evaluation, and selective persistence. The separation between evolution data and frozen benchmark evaluation makes these improvements meaningful evidence of cross-task generalization.

Critical weaknesses
  1. The necessity of multi-responsibility evolution is not directly isolated. Comparisons against existing systems do not replace controlled single-responsibility baselines using the same initial harness and evolution budget. Consequently, it remains unclear how much improvement comes specifically from the broader edit space rather than additional search or a stronger execution framework.

  2. Evidence for reliable attribution and persistence remains limited. Attribution is primarily evaluated through artificially injected faults, while each update uses at most three held-out and two preservation examples for validation. These settings leave uncertainty about whether the reported attribution accuracy and low regression rate extend to naturally occurring failures and accumulated persistent updates.

Questions for authors
  1. Clarification: How does “w/o Attribution Guidance” differ from “Full-Space Search” in candidate generation, search budget, and progressive re-localization? Understanding why these variants obtain 0.6070 and 0.7112, respectively, would materially affect my assessment of attribution’s contribution.

  2. Clarification: Is the reported Held-out Gain measured on examples used to accept updates, or on a separate evaluation set? Please also clarify the sample sizes and construction of the attribution and final regression evaluations, as these determine how broadly the reliability claims can be interpreted.

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer oRqT,

As the discussion period is ending soon, we would appreciate any final feedback on our rebuttal, particularly whether any concerns remain unresolved.

Thank you for your time!

Reviewer_ZpUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Proposes evolving policy, tools, skills, middleware, and memory around frozen image generators.
  2. TRACE attributes repeated failures, proposes targeted changes, and validates updates before retaining them.
  3. Experiments report improvements across three benchmarks and three generator backbones.
Critical strengths
  1. Broad reported gains. Improvements span several benchmarks and generators, with evolution data separated from final evaluation.
  2. Sensible update safeguards. Matched executions, abstention, and preservation checks address stochastic failures and harmful updates.
Critical weaknesses
  1. Implementation details are incomplete. The tool inventory, initial and final harnesses, and executable interpretation of edits are unclear.
  2. Controls do not isolate the contribution. Fixed-harness, single-component evolution, and verifier-retry comparisons are missing under matched resources.
  3. Attribution tests are under-specified. Intervention recipes and sample counts are missing, limiting interpretation of attribution accuracy.
  4. Reliability needs stronger evidence. Main results lack repeated-run uncertainty, and independent evaluation of retained updates is unclear.
Questions for authors
  1. Can you provide the tools, initial/final harnesses, and executable edit specifications?
  2. How does TRACE compare with fixed-harness, single-component, and verifier-retry controls under matched budgets?
  3. How were attribution interventions constructed, and how many were evaluated?
  4. How stable are results across evolution runs? Are regression tests independent of update-approval examples?
1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer ZpUn,

As the discussion period is ending soon, we would appreciate any final feedback on our rebuttal, particularly whether any concerns remain unresolved.

Thank you for your time!

Reviewer_moM7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoGen-Harness improves text-to-image generation by evolving persistent external records while keeping the generator, language models, visual verifiers, and execution controller fixed. Its TRACE procedure aggregates failures across stochastic executions, identifies promising responsibilities to edit, and evaluates candidate changes under matched random seeds. These edits modify requirement rules, capability descriptions, reusable procedures, orchestration settings, or cross-task experience. Evolution takes place on a separate corpus, and the resulting harness is frozen for final benchmark evaluation.

The problem is relevant, the method is coherent, and the reported improvements are substantial. However, the evaluation does not establish whether these gains persist with stronger contemporary generators or how much benefit the framework offers over simpler alternatives under matched budgets. The specific value of evolving multiple responsibilities also remains insufficiently isolated. These limitations in experimental design and contribution attribution lead me to recommend weak rejection.

Critical strengths
  1. A relevant problem and reasonable design. Aggregating repeated executions, comparing edits under matched seeds, and validating persistent changes form a coherent approach to improving stochastic image-generation systems.

  2. Substantial experimental improvements. The method reports clear gains on GenEval2, T2I-CompBench++, and WISE, with additional comparisons using three frozen generators. The qualitative examples show improvements in counting, attribute binding, and spatial composition.

  3. Detailed analysis of the search strategy. The paper includes component ablations, controlled failure cases, and search-budget comparisons. TRACE closely matches full-space search quality while reducing evaluated candidates from 20 to 8 per evolution task.

Critical weaknesses
  1. Limited evaluation with stronger contemporary generators. The framework is evaluated on FLUX.1-dev, Janus-Pro, and the original Qwen-Image. It remains unclear whether the substantial gains persist when the generator already handles compositional constraints well.

  2. Insufficient comparison with simpler alternatives under matched budgets. Although prompt optimization baselines are included, there is no explicit comparison with fixed-harness best-of-N sampling. A simple refinement baseline using a comparably capable LLM would also help establish the value of the additional complexity.

  3. The benefit of evolving multiple responsibilities is not sufficiently isolated. Existing ablations support TRACE's search components, but do not establish its advantage over prompt-only or single-responsibility evolution using the same initial harness and evolution budget.

Questions for authors
  1. Does the framework retain substantial gains with a stronger contemporary generator, particularly on counting, attribute binding, and spatial relations that the generator already handles well?

  2. How does the frozen evolved harness compare with best-of-N sampling from the original harness, using the same visual verifier for selection under matched deployment budgets, including selection costs?

  3. How does full harness evolution compare with a representative prompt-only or single-responsibility strategy using the same generator, LLM, initial harness, data, and evolution budget?

  4. Could the authors provide a complete execution example showing the initial harness, an actual record edit, and the resulting changes to prompts, tool calls, or control flow? This would clarify how the fixed controller interprets the edited records.

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer moM7,

As the discussion period is ending soon, we would appreciate any final feedback on our rebuttal, particularly whether any concerns remain unresolved.

Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.