- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a controller (the author called it as "Harness") for t2i generation that decides whether to keep the first image or try a fix, such as rewriting the prompt or sampling again. It records how much each fix helped in similar past cases, weighs that experience by how relevant and reliable it is, and decides whether running another test is worth the cost.
Critical strengths
- Asking when past experience should be trusted seems timing and interesting.
- Measuring each intervention's improvement relative to the original image is a sensible way to separate a helpful intervention from an easy prompt.
- The method is tested on several different generators.
Critical weaknesses
- The main results are scored by the same vision model that the system optimizes against, so the gains may not reflect real quality.
- Improvements on external benchmarks seems small. For example, on GenEval 2, gains over Raw are often a fraction of a point. A simple baseline is missing, generate the same number of images and let the same verifier pick the best one.
- The intervention space is limited. Regenerating or rewriting the prompt feels trivial and does not show the potential of a self-evolving harness.
Questions for authors
- Results with different evaluators?
- Perform against a best-of-N baseline that generates the same number of images and uses the same verifier to choose?
9 comments
Part 2
Table 3. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with PLaMo 2.1-8B-VL.
| B | TSR Raw | TSR PS | TSR SP | TSR ER | TSR ECHL | CSR Raw | CSR PS | CSR SP | CSR ER | CSR ECHL | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | 1 | 36.94 | 40.00 | 32.78 | - | 40.83 | 47.83 | 51.14 | 40.81 | - | 51.97 |
| 2 | 36.94 | 40.00 | 32.78 | 42.78 | 43.61 | 47.83 | 51.14 | 40.81 | 53.08 | 53.47 | |
| 4 | 36.94 | 40.00 | 32.78 | 42.78 | 45.28 | 47.83 | 51.14 | 40.81 | 53.08 | 55.58 | |
| SD3.5M | 1 | 62.78 | 63.33 | 63.06 | - | 64.72 | 75.67 | 75.56 | 74.42 | - | 77.08 |
| 2 | 62.78 | 63.33 | 63.06 | 63.61 | 66.39 | 75.67 | 75.56 | 74.42 | 77.25 | 78.39 | |
| 4 | 62.78 | 63.33 | 63.06 | 63.61 | 67.22 | 75.67 | 75.56 | 74.42 | 77.25 | 79.08 | |
| FLUX.2 Klein | 1 | 61.39 | 60.00 | 61.94 | - | 62.50 | 76.44 | 74.94 | 76.89 | - | 77.11 |
| 2 | 61.39 | 60.00 | 61.94 | 63.06 | 63.61 | 76.44 | 74.94 | 76.89 | 78.08 | 78.19 | |
| 4 | 61.39 | 60.00 | 61.94 | 63.06 | 65.56 | 76.44 | 74.94 | 76.89 | 78.08 | 79.56 | |
| Z-Image-Turbo | 1 | 68.89 | 65.83 | 62.22 | - | 69.17 | 81.72 | 77.61 | 75.81 | - | 81.78 |
| 2 | 68.89 | 65.83 | 62.22 | 68.89 | 69.44 | 81.72 | 77.61 | 75.81 | 82.11 | 82.31 | |
| 4 | 68.89 | 65.83 | 62.22 | 68.89 | 69.17 | 81.72 | 77.61 | 75.81 | 82.11 | 82.22 | |
| Janus-Pro-7B | 1 | 55.83 | 45.83 | 8.06 | - | 56.11 | 67.69 | 54.39 | 25.36 | - | 68.08 |
| 2 | 55.83 | 45.83 | 8.06 | 57.22 | 56.11 | 67.69 | 54.39 | 25.36 | 68.58 | 68.08 | |
| 4 | 55.83 | 45.83 | 8.06 | 57.22 | 57.50 | 67.69 | 54.39 | 25.36 | 68.58 | 69.47 |
Table 4. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with GLM-4V-9B.
| B | TSR Raw | TSR PS | TSR SP | TSR ER | TSR ECHL | CSR Raw | CSR PS | CSR SP | CSR ER | CSR ECHL | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | 1 | 22.78 | 19.17 | 14.44 | - | 24.17 | 33.08 | 29.08 | 24.03 | - | 34.33 |
| 2 | 22.78 | 19.17 | 14.44 | 24.17 | 24.72 | 33.08 | 29.08 | 24.03 | 34.72 | 35.97 | |
| 4 | 22.78 | 19.17 | 14.44 | 24.17 | 28.06 | 33.08 | 29.08 | 24.03 | 34.72 | 38.03 | |
| SD3.5M | 1 | 51.11 | 47.22 | 49.17 | - | 52.50 | 67.11 | 63.06 | 62.50 | - | 68.94 |
| 2 | 51.11 | 47.22 | 49.17 | 55.00 | 54.17 | 67.11 | 63.06 | 62.50 | 71.50 | 70.58 | |
| 4 | 51.11 | 47.22 | 49.17 | 55.00 | 55.56 | 67.11 | 63.06 | 62.50 | 71.50 | 71.89 | |
| FLUX.2 Klein | 1 | 60.56 | 57.78 | 55.56 | - | 61.39 | 72.72 | 70.86 | 65.50 | - | 72.92 |
| 2 | 60.56 | 57.78 | 55.56 | 60.83 | 62.78 | 72.72 | 70.86 | 65.50 | 74.33 | 74.69 | |
| 4 | 60.56 | 57.78 | 55.56 | 60.83 | 63.89 | 72.72 | 70.86 | 65.50 | 74.33 | 75.44 | |
| Z-Image-Turbo | 1 | 61.94 | 53.89 | 49.17 | - | 63.33 | 75.83 | 69.72 | 65.94 | - | 76.69 |
| 2 | 61.94 | 53.89 | 49.17 | 65.28 | 65.28 | 75.83 | 69.72 | 65.94 | 78.28 | 78.56 | |
| 4 | 61.94 | 53.89 | 49.17 | 65.28 | 66.11 | 75.83 | 69.72 | 65.94 | 78.28 | 79.72 | |
| Janus-Pro-7B | 1 | 48.33 | 39.72 | 2.50 | - | 50.83 | 59.36 | 48.58 | 14.25 | - | 61.56 |
| 2 | 48.33 | 39.72 | 2.50 | 52.22 | 50.83 | 59.36 | 48.58 | 14.25 | 63.14 | 62.61 | |
| 4 | 48.33 | 39.72 | 2.50 | 52.22 | 53.33 | 59.36 | 48.58 | 14.25 | 63.14 | 64.28 |
Table 5. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with Llama-3.2-11B-Vision.
| B | TSR Raw | TSR PS | TSR SP | TSR ER | TSR ECHL | CSR Raw | CSR PS | CSR SP | CSR ER | CSR ECHL | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | 1 | 25.00 | 26.11 | 21.11 | - | 29.44 | 35.72 | 36.92 | 30.64 | - | 39.86 |
| 2 | 25.00 | 26.11 | 21.11 | 30.56 | 31.39 | 35.72 | 36.92 | 30.64 | 41.56 | 42.08 | |
| 4 | 25.00 | 26.11 | 21.11 | 30.56 | 34.17 | 35.72 | 36.92 | 30.64 | 41.56 | 44.81 | |
| SD3.5M | 1 | 59.72 | 59.17 | 55.28 | - | 62.78 | 73.00 | 72.00 | 68.78 | - | 75.44 |
| 2 | 59.72 | 59.17 | 55.28 | 65.00 | 65.00 | 73.00 | 72.00 | 68.78 | 78.03 | 77.75 | |
| 4 | 59.72 | 59.17 | 55.28 | 65.00 | 68.06 | 73.00 | 72.00 | 68.78 | 78.03 | 79.64 | |
| FLUX.2 Klein | 1 | 60.28 | 59.72 | 60.56 | - | 60.83 | 74.97 | 73.83 | 75.31 | - | 75.72 |
| 2 | 60.28 | 59.72 | 60.56 | 62.22 | 63.89 | 74.97 | 73.83 | 75.31 | 76.44 | 77.72 | |
| 4 | 60.28 | 59.72 | 60.56 | 62.22 | 65.56 | 74.97 | 73.83 | 75.31 | 76.44 | 78.50 | |
| Z-Image-Turbo | 1 | 61.39 | 61.94 | 55.00 | - | 62.78 | 76.78 | 75.72 | 69.69 | - | 77.64 |
| 2 | 61.39 | 61.94 | 55.00 | 64.17 | 64.72 | 76.78 | 75.72 | 69.69 | 79.22 | 79.58 | |
| 4 | 61.39 | 61.94 | 55.00 | 64.17 | 64.72 | 76.78 | 75.72 | 69.69 | 79.22 | 80.03 | |
| Janus-Pro-7B | 1 | 48.89 | 41.11 | 5.28 | - | 50.56 | 61.75 | 50.00 | 19.25 | - | 63.78 |
| 2 | 48.89 | 41.11 | 5.28 | 52.78 | 50.56 | 61.75 | 50.00 | 19.25 | 66.69 | 63.78 | |
| 4 | 48.89 | 41.11 | 5.28 | 52.78 | 53.33 | 61.75 | 50.00 | 19.25 | 66.69 | 66.81 |
Part 3
Table 6. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with Qwen3.8-27B.
| B | TSR Raw | TSR PS | TSR SP | TSR ER | TSR ECHL | CSR Raw | CSR PS | CSR SP | CSR ER | CSR ECHL | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | 1 | 23.06 | 26.11 | 20.00 | - | 26.94 | 36.08 | 37.25 | 30.50 | - | 39.56 |
| 2 | 23.06 | 26.11 | 20.00 | 28.06 | 28.33 | 36.08 | 37.25 | 30.50 | 40.39 | 41.08 | |
| 4 | 23.06 | 26.11 | 20.00 | 28.06 | 30.83 | 36.08 | 37.25 | 30.50 | 40.39 | 43.42 | |
| SD3.5M | 1 | 54.72 | 46.94 | 46.39 | - | 55.56 | 70.61 | 63.36 | 62.58 | - | 71.22 |
| 2 | 54.72 | 46.94 | 46.39 | 60.00 | 58.33 | 70.61 | 63.36 | 62.58 | 75.72 | 73.28 | |
| 4 | 54.72 | 46.94 | 46.39 | 60.00 | 60.56 | 70.61 | 63.36 | 62.58 | 75.72 | 76.44 | |
| FLUX.2 Klein | 1 | 56.67 | 56.94 | 57.22 | - | 58.06 | 68.25 | 66.61 | 68.42 | - | 68.75 |
| 2 | 56.67 | 56.94 | 57.22 | 60.83 | 61.11 | 68.25 | 66.61 | 68.42 | 70.28 | 71.19 | |
| 4 | 56.67 | 56.94 | 57.22 | 60.83 | 62.22 | 68.25 | 66.61 | 68.42 | 70.28 | 71.94 | |
| Z-Image-Turbo | 1 | 70.56 | 70.28 | 61.67 | - | 72.22 | 78.44 | 79.06 | 72.28 | - | 79.81 |
| 2 | 70.56 | 70.28 | 61.67 | 73.33 | 74.17 | 78.44 | 79.06 | 72.28 | 81.06 | 81.89 | |
| 4 | 70.56 | 70.28 | 61.67 | 73.33 | 75.00 | 78.44 | 79.06 | 72.28 | 81.06 | 82.72 | |
| Janus-Pro-7B | 1 | 46.11 | 39.44 | 5.28 | - | 49.17 | 60.33 | 49.42 | 15.42 | - | 63.25 |
| 2 | 46.11 | 39.44 | 5.28 | 50.83 | 49.17 | 60.33 | 49.42 | 15.42 | 63.75 | 64.19 | |
| 4 | 46.11 | 39.44 | 5.28 | 50.83 | 51.39 | 60.33 | 49.42 | 15.42 | 63.75 | 65.53 |
W2 (On GenEval 2)
We agree that the absolute improvements on external benchmarks are relatively modest, and would like to provide additional context to clarify their practical significance. Specifically, the achievable gains are inherently bounded by the capabilities of the frozen backbone models and the corrective capacity of the available intervention set. And the value of T2I-SEH, lies in unlocking these existing capabilities effectively through selective, budget-controlled intervention, without requiring generator retraining. It helps determine when additional computation is worthwhile, which intervention to apply, and when to retain the original image. Furthermore, to assess whether the observed improvements are consistent, we conduct additional experiments across a total of five independent seeds.
Table 7. Internal Success Rate (ISR), GenEval 2 Soft-TIFA and action cost across five independent random seeds, reporting the mean and standard deviation.
| Generator | B | ISR Raw | ISR PS | ISR SP | ISR ER | ISR ECHL | Soft-TIFA Raw | Soft-TIFA PS | Soft-TIFA SP | Soft-TIFA ER | Soft-TIFA ECHL | Cost ECHL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | 1 | 61.15±0.89 | 83.48±1.11 | 51.08±1.41 | – | 75.95±3.84 | 7.97±0.39 | 6.93±0.37 | 5.35±0.39 | – | 8.38±0.36 | 0.15±0.08 |
| 2 | 61.15±0.89 | 83.48±1.11 | 51.08±1.41 | 76.50±0.46 | 89.03±0.68 | 7.97±0.39 | 6.93±0.37 | 5.35±0.39 | 8.15±0.50 | 8.59±0.27 | 0.28±0.01 | |
| 4 | 61.15±0.89 | 83.48±1.11 | 51.08±1.41 | 76.50±0.46 | 91.95±0.73 | 7.97±0.39 | 6.93±0.37 | 5.35±0.39 | 8.15±0.50 | 8.73±0.37 | 0.34±0.02 | |
| SD3.5M | 1 | 79.98±0.64 | 90.43±0.49 | 72.35±2.54 | – | 92.10±1.66 | 19.04±1.06 | 16.31±0.48 | 11.53±0.63 | – | 19.73±1.11 | 0.12±0.02 |
| 2 | 79.98±0.64 | 90.43±0.49 | 72.35±2.54 | 91.25±0.81 | 96.20±0.34 | 19.04±1.06 | 16.31±0.48 | 11.53±0.63 | 20.22±1.04 | 19.92±1.02 | 0.16±0.01 | |
| 4 | 79.98±0.64 | 90.43±0.49 | 72.35±2.54 | 91.25±0.81 | 97.80±0.45 | 19.04±1.06 | 16.31±0.48 | 11.53±0.63 | 20.22±1.04 | 20.06±1.03 | 0.19±0.01 | |
| FLUX.2 Klein | 1 | 89.55±0.65 | 97.35±0.38 | 95.28±0.85 | – | 98.83±0.29 | 25.48±0.85 | 24.77±0.30 | 23.51±0.54 | – | 26.78±0.90 | 0.09±0.01 |
| 2 | 89.55±0.65 | 97.35±0.38 | 95.28±0.85 | 94.98±0.43 | 99.63±0.15 | 25.48±0.85 | 24.77±0.30 | 23.51±0.54 | 26.14±0.89 | 26.84±0.88 | 0.10±0.01 | |
| 4 | 89.55±0.65 | 97.35±0.38 | 95.28±0.85 | 94.98±0.43 | 99.70±0.14 | 25.48±0.85 | 24.77±0.30 | 23.51±0.54 | 26.14±0.89 | 26.84±0.88 | 0.10±0.01 | |
| Z-Image-Turbo | 1 | 94.83±0.38 | 98.45±0.60 | 97.23±0.45 | – | 99.05±0.34 | 31.50±0.79 | 28.74±0.82 | 28.42±0.97 | – | 32.26±0.77 | 0.04±0.01 |
| 2 | 94.83±0.38 | 98.45±0.60 | 97.23±0.45 | 97.13±0.39 | 99.83±0.14 | 31.50±0.79 | 28.74±0.82 | 28.42±0.97 | 31.76±0.84 | 32.27±0.79 | 0.05±0.00 | |
| 4 | 94.83±0.38 | 98.45±0.60 | 97.23±0.45 | 97.13±0.39 | 99.90±0.06 | 31.50±0.79 | 28.74±0.82 | 28.42±0.97 | 31.76±0.84 | 32.27±0.79 | 0.05±0.00 | |
| Janus-Pro-7B | 1 | 67.48±1.92 | 80.25±0.94 | 34.05±1.24 | – | 86.10±0.64 | 13.26±0.37 | 11.68±0.52 | 5.19±0.25 | – | 13.73±0.50 | 0.09±0.05 |
| 2 | 67.48±1.92 | 80.25±0.94 | 34.05±1.24 | 78.85±1.15 | 89.03±0.69 | 13.26±0.37 | 11.68±0.52 | 5.19±0.25 | 14.24±0.29 | 14.35±0.25 | 0.22±0.02 | |
| 4 | 67.48±1.92 | 80.25±0.94 | 34.05±1.24 | 78.85±1.15 | 90.98±0.72 | 13.26±0.37 | 11.68±0.52 | 5.19±0.25 | 14.24±0.29 | 14.47±0.27 | 0.25±0.02 |
Part 1
Dear Reviewer Q6j8,
Thank you for your efforts and time on our work, for your insightful comments and for your constructive suggestions. We are more than delighted that you consider the motivation of our work "timely and interesting", consider the proposed methodology "sensible", and recognize the evaluation across multiple generators as a key strength of our work. We fully agree with your constructive comments, which have helped us further improve our work. In the following, we provide detailed, point-by-point responses to each of them.
In Table 1, we employ Qwen2.5-VL-3B as both the internal observer and the atomic evaluator to assess the Task Success Rate and Constraint Satisfaction Rate, primarily to examine the effectiveness of internal decision-making. We fully agree with your suggestion to incorporate independent evaluators for a more comprehensive and robust assessment. We additionally introduce six vision models of varying parameter scales developed by different organizations as independent evaluators: HyperCLOVAX-SEED-Vision-Instruct-3B, InternVL3.5-4B, PLaMo 2.1-8B-VL, GLM-4V-9B, Llama-3.2-11B-Vision and Qwen3.8-27B. Although the absolute scores vary somewhat under independent evaluation, the overall trend remains consistent, confirming that T2I-SEH selects more appropriate interventions while incurring lower action costs.
Table 1. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with HyperCLOVAX-SEED-Vision-Instruct-3B.
Table 2. Task Success Rate (TSR) and Constraint Satisfaction Rate (CSR) evaluated with InternVL3.5-4B.