- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EvoGen-Harness, which improves frozen text-to-image generators by evolving five persistent external responsibilities: Policy, Tools, Skills, Middleware, and Memory. Its TRACE mechanism combines trajectory-based attribution, localized edit proposals, matched stochastic evaluation, progressive re-attribution, and validation before persistence. Experiments report substantial improvements on three benchmarks, supported by component ablations, controlled attribution tests, and evaluations across three frozen generators.
Critical strengths
-
A consequential formulation of external system improvement. The paper distinguishes identifying a visual failure from deciding which persistent system component should change. This broadens the correction space beyond predefined prompt or routing optimization and addresses a practical challenge in adaptive generation systems.
-
Strong reported results with complementary validation. The framework improves performance across three benchmarks, while ablations support the roles of attribution, matched evaluation, and selective persistence. The separation between evolution data and frozen benchmark evaluation makes these improvements meaningful evidence of cross-task generalization.
Critical weaknesses
-
The necessity of multi-responsibility evolution is not directly isolated. Comparisons against existing systems do not replace controlled single-responsibility baselines using the same initial harness and evolution budget. Consequently, it remains unclear how much improvement comes specifically from the broader edit space rather than additional search or a stronger execution framework.
-
Evidence for reliable attribution and persistence remains limited. Attribution is primarily evaluated through artificially injected faults, while each update uses at most three held-out and two preservation examples for validation. These settings leave uncertainty about whether the reported attribution accuracy and low regression rate extend to naturally occurring failures and accumulated persistent updates.
Questions for authors
-
Clarification: How does “w/o Attribution Guidance” differ from “Full-Space Search” in candidate generation, search budget, and progressive re-localization? Understanding why these variants obtain 0.6070 and 0.7112, respectively, would materially affect my assessment of attribution’s contribution.
-
Clarification: Is the reported Held-out Gain measured on examples used to accept updates, or on a separate evaluation set? Please also clarify the sample sizes and construction of the attribution and final regression evaluations, as these determine how broadly the reliability claims can be interpreted.
Follow-up on our rebuttal
Dear Reviewer oRqT,
As the discussion period is ending soon, we would appreciate any final feedback on our rebuttal, particularly whether any concerns remain unresolved.
Thank you for your time!