- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Failure-Guided Preference Supervision (F-GPS). It analyzes the generation failures of the target text-to-image model and distills recurring error patterns into executable visual programs. These programs are then used to construct Rejected Images with targeted biases, providing more effective preference supervision for Diffusion Preference Optimization. The method achieves fairly consistent performance improvements on image composition generation and visual text rendering tasks.
Critical strengths
-
The problem is clearly defined.
-
The experimental details are clearly described.
Critical weaknesses
-
The novelty of the paper is very limited. In essence, it is a reorganization of the data used in preference optimization learning. The only new part is organizing recurring and reusable failure cases, but all the methods involved are existing techniques.
-
The paper uses GPT-4o to analyze and interpret failure cases. However, is such interpretation really reliable? Could other models also work, and would more advanced models be better? Does this interpretation consider all possible visual elements?
Questions for authors
-
The paper contains statements such as "The contribution therefore lies not in rendering or compositing itself, but in distilling observed model failures into reusable programs for constructing preference supervision". I do not know what such statements are trying to emphasize.
-
The paper essentially reorganizes failure cases and forms reusable programs. Can the corresponding successful cases also be processed in this way? Why must it be failure cases?
-
The readability of the paper is poor. There are many borrowed concepts that seem plausible but are not clearly defined. Such expressions make it unfriendly for readers to understand the core intent of the authors. In addition, the figures and the organization of the presentation rely too much on LLM tools, but the effect of the presentation is not good either.
-
For the evaluation using VLM-as-a-Judge, the paper relies on a single model, Qwen3-VL-8B-Instruct. Such evaluation is not reliable enough, especially because human trustworthiness assessment is missing.
-
Based on the experimental results and the corresponding visualizations, why do the basic preference optimization methods sometimes perform worse than the base model?
-
Are the failure programs transferable? Can the failure programs extracted from failure cases on SD v1.5 be applied to other models?
11 comments
Rebuttal
To substantiate this contribution, we have added component ablations investigating both the source of failure cues and the mechanism used to construct rejected examples. We compare predefined visual perturbations, LLM-suggested failures, F-GPS-identified failures with model-based editing, and the complete F-GPS pipeline. All configurations use the same preferred images, conditioning prompts, and SDXL Diffusion-DPO optimization settings.
Table 12: Component ablation for image composition (Appendix I.5).
| Method | T2I-CompBench++ BLIP | T2I-CompBench++ GIT | TIFA BLIP | TIFA GIT |
|---|---|---|---|---|
| Base | 78.08 | 75.30 | 88.60 | 87.20 |
| Predefined Visual Perturbations + Executable Programs | 71.71 | 74.04 | 74.91 | 73.03 |
| LLM-Suggested Failures + Model-Based Editing | 43.84 | 30.86 | 40.78 | 33.22 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 60.49 | 52.54 | 61.73 | 53.14 |
| F-GPS-Identified Failure Cues + Executable Programs | 81.55 | 79.30 | 92.30 | 90.80 |
Table 13(a): Component ablation on CVTG (visual text rendering).
| Method | EM | NEM | PSS | WP | WR | WF1 |
|---|---|---|---|---|---|---|
| Base | 24.2 | 27.0 | 77.9 | 50.1 | 50.9 | 48.6 |
| Predefined Visual Perturbations + Executable Programs | 7.6 | 7.8 | 74.2 | 57.0 | 32.8 | 38.1 |
| LLM-Suggested Failures + Model-Based Editing | 16.0 | 16.0 | 70.1 | 49.5 | 28.4 | 33.0 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 6.0 | 6.0 | 67.7 | 43.4 | 21.6 | 26.3 |
| F-GPS-Identified Failure Cues + Executable Programs | 32.0 | 32.8 | 81.3 | 58.2 | 59.0 | 56.8 |
Table 13(b): Component ablation on MARIO-Eval (visual text rendering).
| Method | EM | NEM | PSS | WP | WR | WF1 |
|---|---|---|---|---|---|---|
| Base | 0.8 | 0.8 | 39.8 | 3.1 | 3.7 | 3.1 |
| Predefined Visual Perturbations + Executable Programs | 20.8 | 23.8 | 75.6 | 42.4 | 43.8 | 41.0 |
| LLM-Suggested Failures + Model-Based Editing | 14.4 | 16.6 | 64.1 | 28.9 | 26.5 | 26.4 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 15.4 | 17.2 | 64.0 | 31.3 | 28.3 | 28.2 |
| F-GPS-Identified Failure Cues + Executable Programs | 24.9 | 26.1 | 76.8 | 48.1 | 46.7 | 45.8 |
Tables 12–13 use SDXL-Base-1.0 with Diffusion-DPO. Model-Based Editing denotes instruction-guided editing with FLUX.2-klein-4B.
The complete F-GPS pipeline achieves the best results across the reported composition and text-rendering metrics. In particular, predefined executable perturbations substantially underperform F-GPS, suggesting that the source of visual deviations matters beyond the use of executable transformations alone. Similarly, using F-GPS-identified failure cues with model-based editing performs worse than the complete pipeline, highlighting the importance of how the identified failures are operationalized.
To better understand these differences, we additionally evaluate 500 matched preferred–rejected pairs per method and task using GPT-6.1 Sol with medium reasoning effort. The evaluation considers four criteria: intended alteration realization, new prompt violations, unrequested changes, and whether the rejected image is visually better than the preferred image.
Rebuttal
Table 14(a): Negative-example validation for image composition.
| Evaluation criterion | Negative construction | Yes (%) | No (%) | Unclear (%) |
|---|---|---|---|---|
| All intended changes visibly present? | LLM-Suggested Failures + Model-Based Editing | 0.4 | 98.4 | 1.2 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 3.6 | 90.2 | 6.2 | |
| F-GPS-Identified Failure Cues + Executable Programs | 80.8 | 15.8 | 3.4 | |
| New clear prompt violation? | LLM-Suggested Failures + Model-Based Editing | 2.4 | 63.8 | 33.8 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 27.4 | 52.4 | 20.2 | |
| F-GPS-Identified Failure Cues + Executable Programs | 58.2 | 20.4 | 21.4 | |
| Unrequested changes? | LLM-Suggested Failures + Model-Based Editing | 82.0 | 18.0 | 0.0 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 78.2 | 20.4 | 1.4 | |
| F-GPS-Identified Failure Cues + Executable Programs | 10.2 | 89.2 | 0.6 | |
| Rejected image visually better? | LLM-Suggested Failures + Model-Based Editing | 7.0 | 92.6 | 0.4 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 0.2 | 99.6 | 0.2 | |
| F-GPS-Identified Failure Cues + Executable Programs | 0.2 | 99.6 | 0.2 |
Table 14(b): Negative-example validation for visual text rendering.
| Evaluation criterion | Negative construction | Yes (%) | No (%) | Unclear (%) |
|---|---|---|---|---|
| All intended changes visibly present? | LLM-Suggested Failures + Model-Based Editing | 2.6 | 95.6 | 1.8 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 6.2 | 90.0 | 3.8 | |
| F-GPS-Identified Failure Cues + Executable Programs | 84.4 | 7.4 | 8.2 | |
| New clear prompt violation? | LLM-Suggested Failures + Model-Based Editing | 26.4 | 34.0 | 39.6 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 36.6 | 32.4 | 31.0 | |
| F-GPS-Identified Failure Cues + Executable Programs | 51.6 | 23.0 | 25.4 | |
| Unrequested changes? | LLM-Suggested Failures + Model-Based Editing | 54.8 | 45.2 | 0.0 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 71.6 | 28.2 | 0.2 | |
| F-GPS-Identified Failure Cues + Executable Programs | 0.0 | 100.0 | 0.0 | |
| Rejected image visually better? | LLM-Suggested Failures + Model-Based Editing | 17.4 | 82.6 | 0.0 |
| F-GPS-Identified Failure Cues + Model-Based Editing | 10.4 | 89.6 | 0.0 | |
| F-GPS-Identified Failure Cues + Executable Programs | 0.0 | 100.0 | 0.0 |
Table 14 reports Yes/No/Unclear judgments; lower Yes is desirable for unrequested changes and visually better negatives. The complete evaluation protocol is described in Appendix J.2.
These additional analyses show that executable F-GPS programs realize the intended alterations in 80.8%/84.4% of composition/text-rendering pairs, compared with 3.6%/6.2% for model-based editing using the same F-GPS-identified failure cues. They also introduce clearer prompt violations with substantially fewer unrequested changes.
Taken together, the downstream performance ablations and negative-example validation provide complementary evidence for the importance of both model-specific failure information and controlled executable negative construction. This supports our central motivation: rather than presenting the model with arbitrary rejected examples, F-GPS constructs preference contrasts that explicitly reflect the types of failures observed in its own generations.
We have also clarified the corresponding methodological statements in the revised manuscript (Appendix I.5, Tables 12–14).
Rebuttal
Reviewer iki6
We sincerely thank the reviewer for their careful evaluation of our work, for acknowledging the clearly defined problem and detailed experimental setup, and for providing constructive feedback. We have strengthened the manuscript with additional experiments, analyses, and clarifications addressing the reviewer's concerns.
1. Methodological novelty and contribution of failure-guided preference supervision
We apologize that the intended methodological contribution was not sufficiently clear in the original presentation. We would like to emphasize that F-GPS aims to provide the target model with preference supervision that explicitly reflects its own observed failures, rather than simply constructing additional generic negative examples. Our motivation is to enable the model to learn not only what a preferred image should look like, but also which specific visual deviations, corresponding to its observed weaknesses, should be avoided.
To achieve this, F-GPS (i) identifies sample-specific failures in the target model's generations, (ii) atomizes and consolidates recurring failure patterns into reusable atomic visual cues, (iii) operationalizes these cues as parameterized executable programs, and (iv) applies compositions of these programs to editable preferred examples. This enables the construction of controlled preferred–rejected contrasts that deliberately expose the model to the types of mistakes identified in its own generations.
Compared with approaches such as D-Fusion, OSPO, and RealAlign, our distinguishing contribution is therefore not the rendering or compositing mechanism itself, but the distillation of target-model failures into a reusable executable failure-program library for constructing informative preference supervision.