acceptodds
Under review as a conference paper at ICLR 2027

Learning to Repair: Skill Optimization for Multi-Round Text-to-Image Generation

Abstract

Multi-round text-to-image generation uses feedback to correct compositional errors, yet additional refinement does not always lead to better images. Progress depends on choosing suitable repair actions and ordering them so that later operations preserve earlier corrections. How can repair experience be turned into reusable guidance for these decisions? We address this question through skill optimization, with a focus on designing the space of guidance to be learned. Different failure types respond differently to editing and regeneration, while interactions between repairs make their order consequential. Motivated by these observations, we structure the critic's natural-language skill around failure diagnosis, type-conditional action choice, preservation requirements, and repair priorities, with generator-specific observations stored separately. Starting from human-designed guidance, we use feedback from a small collection of historical trajectories to refine the skill while keeping the underlying models frozen. Experiments on GenEval, GenEval 2, and ConceptMix show improvements over the IRG baseline. Removing either action or ordering guidance reduces the effectiveness of the optimized skill, while transfer from GenEval2 to ConceptMix shows that the learned guidance retains value beyond its source dataset. These findings support structured skill optimization as a way to improve compositional generation by learning how to repair.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Generaion, Agentic Generation, VLM Skill Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UtHUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper optimizes structured natural-language guidance for a VLM critic in multi-round text-to-image generation. The guidance addresses repair actions and their ordering and is refined using trajectory feedback while all model weights remain frozen. Experiments on GenEval, GenEval 2, and ConceptMix show improvements over IRG.

Critical strengths

The method uses feedback from previous repair trajectories to refine reusable critic guidance while keeping all model weights frozen.

Critical weaknesses
  • The specific contribution of the structured skill design is not sufficiently established. Although the component-removal ablations show that action and ordering guidance contribute to the final system, they do not isolate the benefit of the prescribed structure over critic-instruction optimization without that structure, using the same trajectory feedback and optimization budget. In addition, the rationale for choosing 25 training examples and 25 validation examples, and the procedure used to select or construct them, are not clearly described. Further details would help assess the design of the optimization data and improve reproducibility.
  • Generalizability across generators and editors remains untested. The proposed method is evaluated only with Qwen-Image and Qwen-Image-Edit. Since effective repair actions and ordering may depend on these models’ capabilities, it remains unclear whether the learned guidance transfers to other generator–editor combinations, or whether repeating the optimization with those models yields similar improvements. This limits the evidence for the method’s broader applicability.
Questions for authors
  • Can the critic combine multiple repairs into a single editing instruction? EdiVal-Agent [A] reports higher instruction-following success for single-shot multi-instruction editing than for sequential editing with Qwen-Image-Edit under its evaluation protocol. Has the paper compared jointly addressing multiple errors with addressing them sequentially according to the learned repair ordering? Reporting both final constraint satisfaction and editing cost would help clarify when explicit repair ordering is beneficial.
  • What motivated the choice of 25 training examples and 25 validation examples? How were these examples selected or constructed, and what considerations guided their coverage of failure types and difficulty levels?
  • Could the proposed skill structure and optimization procedure be adapted to improve other aspects of text-to-image quality, such as aesthetics?

[A] Chen et al. EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing. ICLR 2026.

Reviewer_gNYJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The task is to generate an image satisfying some requirements, specified in prompt. The paper introduces a skill that uses several frozen VLM, which refines the generated image so it better satisfies the conditions. Essentially, skill specifies what call (prompt plus image) to send to VLM, get the result (text or image), and decides what next call to make.

The skill is a text, which includes:

  • failure types (how to classify problem into category like "spatial error"),
  • repair methods (guidance on what next action to take and what prompt to give to VLM)
  • repair order (if several conditions are not satisfied, what takes priority)
  • generator notes (notes on the model that generates the first image)

The paper gives an optimization procedure for these four parts of the skill.

Critical strengths

The paper is well-written. The construction of the skill and its optimization procedure seem reasonable. Experimental results (Tables 1-3) support the claim that the method is SOTA.

Critical weaknesses

I do not see any major weaknesses.

But I am not an expert in the field: I have some understanding of image generation, but it is the first paper on skill optimization that I have read. I do not know the established standards for such papers, and will orient on other reviews.

Questions for authors
  1. Am I right that each dataset (Geneval 1-2, ConceptMix) is split into train and test datasets, and you optimize the skill using only train dataset and after that evaluate it on test dataset?
  2. Does it make sense to report for each method how many calls to VLM were made during training and the average number of calls during inference?
Reviewer_y4NeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In multi-round text-to-image generation, a VLM checks each image and decides whether to edit or regenerate, but extra rounds do not always help: a poor action wastes a round, and a later repair can undo constraints that were already satisfied. Building on IRG, the paper freezes the generator, editor and VLM and optimizes only the critic's natural-language skill document, which is fixed to four sections: failure diagnosis, failure-type-conditioned action choice (including what satisfied constraints to preserve), repair order, and generator notes.

An optimizer LLM reads serialized trajectory records and proposes additions, deletions and replacements, which are accepted only if the validation gain exceeds κσ^\kappa\hat{\sigma} (κ\kappa times the bootstrap standard error of the validation mean), using 25 training and 25 validation examples. With Qwen-Image and Qwen-Image-Edit as generator and editor, the method improves over IRG on GenEval (0.81→0.89), GenEval 2 (71.5→79.6) and ConceptMix (+4.3–7.4 points across complexity levels).

Critical strengths
  • The problem is practical. In multi-round repair, "which repair action to use and which failure to address first" (lines 039–040) does affect the outcome, and Figure 1 clearly illustrates how a later repair can undo an earlier one. Optimizing only the text instructions, without touching model weights, is cheap to deploy.
  • The improvement over IRG is consistent across three benchmarks: on GenEval it is concentrated in counting, position and attribute binding (+13, +11, +16 points), on ConceptMix it holds at all seven complexity levels, and the skill transferred from GenEval 2 to ConceptMix still exceeds the manual instructions by 5.4 points (at k = 5, the only setting reported for transfer).
  • The authors are fairly candid about limitations: they state that the fix rates in Figure 3 are observational and that Figure 1 is only a motivating example, acknowledge that GEMS and CRAFT score higher on position and verbs on GenEval 2, and note that Figure 4c counts only images and not total inference cost.
Critical weaknesses
  • Location: Section 1 (lines 051–052), Section 2.2 (lines 137–139), Section 3.3 (lines 260–281, 302–311), Section 4.2 (line 408), Figure 4c, Section 4.3 (lines 417–427) This is my main concern: the methodological novelty lies in the four-section structure of the skill space, but no experiment shows that this structure itself helps. The optimization procedure largely follows SkillOpt (Yang et al., 2026): the fast update loop with an epoch-level meta-update and the validation-based acceptance rule are both attributed to that work (lines 303–311), and Section 2.2 calls it the closest related work. The ADD/DEL/REP edit operator and the edit budget that decreases over training (lines 276–280) also appear in SkillOpt, which uses add/delete/replace edits under a decaying edit budget, but they are not attributed to it. Line 408 says "adding SkillOpt to the IRG framework," and the legend of Figure 4c reads "Flash + SkillOpt (Ours)," using the name of the cited prior work, which makes it hard for readers to tell what this paper contributes. The introduction argues that optimizing the whole instruction document as an undifferentiated prompt obscures which decision an update targets (lines 051–052), but this control is missing: directly optimizing the unsectioned IRG instructions with the same optimizer, budget and data. The ablation in Figure 4a removes the repair-order or action-choice guidance post hoc from an already optimized skill, and the authors note that these variants are not separately optimized (lines 419–421); this shows that the learned text is useful, not that the four-section structure makes optimization more effective. This ablation is also reported only for ConceptMix at k = 5, without confidence intervals or repeated runs. The sTs_T (failure types) and sGs_G (generator notes) sections are not ablated either. I suggest adding an unstructured SkillOpt baseline and per-section ablations, and stating clearly in the main text how this work differs from SkillOpt.

  • Location: Section 3.2 (lines 237–238), Section 3.3 (lines 274–281, 293, 302, 308–311), Section 4.1 (lines 349–364), Tables 1–3, Reproducibility statement (lines 502–505) Key experimental settings are not described, which affects both the interpretation and the reproducibility of the results; the Reproducibility statement (lines 502–505) points to Sections 3 and 4, but the items below are not there. (1) The paper has no appendix: the final learned skill documents and the optimizer prompts are not given, and it is not described how the IRG critic instructions used as s0s_0 (line 293) were organized into the four sections. None of the following is reported: the optimizer LLM (line 302), the value of κ\kappa (line 310), Lmax⁡L_{\max} and the section budgets bqb_q with their schedule (lines 274–280), the number of optimization epochs, or how many candidate edits were evaluated and accepted. (2) Lines 357–359 list three VLMs, but Tables 1–3 do not say which one was used for the main results, and no result for GPT-5. 4-mini appears anywhere in the paper. (3) The inference-time-scaling baselines in Tables 1 and 2 (Original, Rewrite, Promptist, Search, Maestro, CRAFT and GEMS) are listed under Qwen-Image, but all 91 of their reported values are identical to those that the GEMS paper (He et al., 2026b; its Appendix Tables 4–5 and the GenEval2 column of its Table 1) reports for Qwen-Image-2512. GEMS also states that Search was run with 5 parallel samples, whereas lines 355–357 state that parallel-sampling baselines are compute-matched to 3.2 / 3.4 / 3.4 calls. Please state the source and settings of every baseline, and clarify which Qwen-Image version is used for IRG and the proposed method; if the settings differ, the numbers in the tables are not directly comparable. (4) It is unclear where the 25 training and 25 validation examples come from. Lines 363–364 state that these optimization examples are "not the test sets," but results are reported on the "full benchmark test sets" (lines 362–363): if these 50 examples are drawn from the benchmarks themselves, were they excluded from evaluation? If they come from elsewhere, what data are they? Is a separate skill optimized for each benchmark, and is one ConceptMix skill used for all seven k levels? Moreover, the diagnostic data used in Section 3.2 to design the skill space are 501 failed ConceptMix cases (lines 237–238); how do they relate to the ConceptMix test set?

  • Location: Section 3.1 (lines 201–203), Section 4.1 (lines 353–357), Tables 1–3, Figure 4c (lines 429–461) The budget consumed by the optimized skill is not reported for the main results, so it is unclear whether IRG and the optimized skill are compared at matched cost. The loop runs for at most 4–5 rounds, stops early when all constraints are satisfied or STOP is chosen, and returns the highest-scoring image among all rounds (lines 201–203). Under this best-of-rounds selection, a skill could improve results without making better individual repair decisions, e.g., by choosing STOP less often (more candidate images per prompt) or by regenerating more often (more diverse candidates for the verifier to choose from). Figure 4c partly addresses the first possibility: the last point of the optimized Flash curve lies at fewer images per prompt than the last points of the Flash and Pro curves under the original instructions. However, Figure 4c does not state the benchmark or which setting of Tables 1–3 it corresponds to, so it is unclear whether this holds for the main results. Lines 355–357 give the average numbers of generation and editing calls used to compute-match parallel-sampling baselines (3.2 / 3.4 / 3.4), but do not say whether they refer to IRG or to the optimized skill (see also point (3) of the previous weakness regarding the Search numbers). Figure 4c counts only images; the cost of VLM calls and of the optimization stage is not reported either. I suggest reporting, for each benchmark, the average number of rounds, the action distribution (EDIT/REGEN/BACK/STOP), generation/editing calls and VLM calls for IRG and for the optimized skill separately, as well as the cost of the optimization stage.

  • Location: References (lines 512–518, 543–548, 642–647, 655–657), lines 049, 119, 137, 304, 311, Table 1 (Minor) Three pairs of reference entries are identical in content but are cited as two separate works under different year suffixes: Z-Image (Cai et al., 2025a/b), GenEval (Ghosh et al., 2023a/b) and SkillOpt (Yang et al., 2026a/b). The main text mixes the two keys; for example, lines 049 and 137 cite Yang 2026b, while lines 304 and 311 cite Yang 2026a. The 46 entries therefore correspond to only 43 papers. Please merge the duplicate entries. In addition, the VisualPrompter entry lacks a year, which results in a year-less "Wu et al." on line 119, and the SkillRouter entry has an incomplete arXiv identifier ("arXiv–2603", lines 655–657).

Questions for authors
  1. What are the results of directly optimizing the unsectioned IRG instructions (i.e., the original SkillOpt) with the same optimizer, budget and training/validation data? What are the results when each of the four sections (sTs_T, sMs_M, sOs_O, sGs_G) is removed and the skill is re-optimized? How does the optimization procedure in this paper differ from SkillOpt?
  2. Which model is the optimizer LLM? What are the values of κ\kappa, Lmax⁡L_{\max} and bqb_q (including its schedule), and the number of optimization epochs? How many candidate edits were evaluated and accepted? Can the final skill documents, and the four-section version of the initial IRG instructions, be provided? Which VLM is used in Tables 1–3? Were the baseline numbers in Tables 1–2 taken from GEMS (He et al., 2026b), and which Qwen-Image version is used for IRG and the proposed method?
  3. Where exactly do the 25 training and 25 validation examples come from? If they are drawn from the benchmarks themselves, were they removed from the test sets on which results are reported? Is a separate skill optimized for each benchmark, and for each ConceptMix k? How do the 501 diagnostic ConceptMix cases relate to the ConceptMix test set?
  4. For each benchmark, what are the average numbers of rounds, the action distribution, generation/editing calls and VLM calls for IRG and for the optimized skill? Which benchmark does Figure 4c use?
  5. Could the revision merge the three pairs of duplicate reference entries, unify the citation keys in the main text, add the missing year to the VisualPrompter entry, and complete the SkillRouter arXiv identifier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.