acceptodds
Withdrawn from ICLR 2027

APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training is crucial for improving text-to-image models, yet existing pipelines typically rely on manually curated training data and fixed training distributions. We introduce APT (Agentic Post-Training), a fully automated framework that adaptively post-trains a base model according to its evolving capability profile. APT coordinates three agents to construct evaluation sets from user requirements, diagnose model-specific strengths and failure modes, and generate reinforcement learning data with an adaptive prompt distribution that emphasizes underperforming capabilities. By repeatedly evaluating, diagnosing, and updating the generator, APT lets the current model's behavior shape its next training tasks, forming a feedback loop inspired by recursive self-improvement. For general-purpose settings without user-specified evaluation scenarios, we further develop an evaluation harness that enables broad and diverse evaluation set generation. Experiments across multiple base models, benchmarks, and generation architectures demonstrate that APT is model-agnostic, benchmark-agnostic, architecture-agnostic, and fully automated.

Primary area:
generative models
Keywords:
Image generation, agentic post-train

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xuQFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work tackles the labor-intensive nature of existing text-to-image post-training and the poor cross-model generalization of handcrafted post-training datasets. They present a fully automated, model-, benchmark-, and architecture-agnostic RSI framework built upon three collaborative agents. Their framework achieves higher average performance than the manually curated RL dataset baseline on all tested diffusion models.

Critical strengths

(1) Propose a model-, benchmark-, and architecture-agnostic RSI framework for automated text-to-image post-training with recursive self-improvement. (2) Design an iterative loop with three collaborative agents. Then create a diagnostic evaluation dataset along with a hierarchical scene taxonomy, and generate diverse prompts across multiple descriptive dimensions. The pipeline automatically yields prompt-rubric pairs for diagnosis, independent of public benchmarks.

Critical weaknesses

(1) While empirical results show promising performance, the work neglects inherent errors from the agent-based evaluator. (2) It remains unvalidated whether the proposed hierarchical taxonomy comprehensively spans all critical evaluation dimensions for text-to-image generation. (3) The alignment between judgments from the diagnosis agent and human preferences is not verified.

Questions for authors

(1) Is the superiority of the automated results over the manually curated training set confirmed via human double-blind evaluation? (2) What is the construction process of the manually curated RL training baseline?

2 comments

We thank the reviewer for the helpful comments on evaluator reliability and human alignment. We respond to each point below.

W1: Inherent errors of the agent-based evaluator

We agree that the evaluator is imperfect. Several properties of APT make it robust to evaluator noise, and we now quantify that noise.

(i) Decomposed binary judgments. Instead of asking for a holistic score, the evaluator answers narrow, verifiable yes/no questions such as "exactly two puppies are present." Such questions are less ambiguous than holistic ratings.

(ii) Aggregation. The training reward averages up to 10 rubric judgments. The Data Generation Agent does not act on individual judgments; it identifies recurring failure patterns across about 120 prompts and up to 1,200 rubric judgments per round. Isolated errors are therefore smoothed out.

(iii) Independent final evaluation. None of the reported results are computed with our GPT-4.1-mini evaluator. All numbers come from the official protocols of five public benchmarks: detector- and classifier-based checks in GenEval, Soft-TIFA in GenEval2, MLLM-based VQA in DPG, OCR word accuracy in CVTG-2K, and dimension-specific scorers in OneIG-EN. If evaluator errors were driving the training, the gains would be unlikely to transfer to these independent metrics.

(iv) Measured error rate. Following RubricRL[1], we conduct a human evaluation to validate the quality of generated rubrics and the reliability of our VLM grader (GPT-4.1-mini). We sample 100 prompts and generate 10 rubrics per prompt, yielding 1,000 prompt–rubric pairs. Human annotators assess prompt–rubric consistency, rubric hallucination, and grader correctness. Three annotators independently labeled all 1,000 prompt–rubric pairs, and we report the average over annotators. On average, 97.2% of the generated rubrics are consistent with their prompts, only 0.83% contain hallucinated constraints, and the VLM grader agrees with human judgments on 95.7% of pairs, demonstrating the reliability of our rubric-based evaluation.

W2: Whether the hierarchical taxonomy spans all critical evaluation dimensions

No finite taxonomy can be proven complete, and we will soften the wording accordingly. Our goal is broad and extensible coverage rather than exhaustive coverage, and we provide the following evidence for it.

(i) Scale and structure. The evaluation set spans 10 categories, 219 sub-categories, and 2,609 prompt–rubric pairs. The rubrics cover OCR, counting, attributes, actions, and spatial relations (Figure 2, Appendix E.2).

(ii) Generalization to unseen benchmarks. None of the five benchmarks is used to build the evaluation set, yet a single APT run improves the base model on almost all of them across four models.

(iii) Comparison with benchmark evaluation sets. Replacing our evaluation set with the OneIG-EN or GenEval sets lowers the average and produces biased improvements (Table 3). This indicates that our set is broader than either single benchmark.

(iv) Extensibility. Users can specify additional scopes, and the Evaluation Set Generation Agent will extend the taxonomy.

W3: Alignment between the diagnosis judgments and human preferences

We add the human agreement study described in W1(iv): 97.2% of the generated rubrics are consistent with their corresponding prompts, while only 0.83% contain hallucinated constraints. Moreover, our VLM grader achieves 95.7% agreement with human judgments, demonstrating the reliability of our rubric-based evaluation. The benchmark metrics we report are also known to be human-aligned. For example, GenEval2 reports that Soft-TIFA_GM, the metric we use, achieves higher AUROC against prompt-level human judgments than the arithmetic-mean variant.

Q1: Is the superiority over the manually curated training set confirmed via human double-blind evaluation?

We conducted a human double-blind evaluation (Table C1). We sampled 1,000 prompts from the five evaluation benchmarks (GenEval, GenEval2, OneIG-EN, DPG, and CVTG-2K) and generated images with the Base model, the Handcrafted RL model, and APT using the same seed and inference settings. Six PhD students from different research fields served as annotators. For each prompt, APT was compared with each baseline, giving 1,000 pairs per comparison, and each pair was rated by one annotator. The two images were shown side by side in random order without method labels, and annotators chose win, tie, or loss on prompt alignment, visual quality, and overall preference. APT is preferred over the handcrafted baseline in 64.6% of comparisons on prompt alignment and 63.1% on overall preference.

Table C1. Human double-blind evaluation (Lens; 1,000 pairs per comparison, 6 annotators). Win / Tie / Loss (%) of APT.

ComparisonPrompt alignmentVisual qualityOverall preference
APT vs. Base81.1/9.8/9.171.3/8.8/19.980.5/8.5/11.0
APT vs. Handcrafted RL64.6/12.2/23.260.2/14.5/25.363.1/12.5/24.4

Q2: Construction process of the manually curated RL training baseline

The handcrafted baseline is the RL training set released with Lens[2], which was curated by human experts to cover diverse generation scenarios. As described in the Lens[2] paper, it is a static RL prompt set built with a taxonomy-driven pipeline:

  • Human experts defined a taxonomy of generation-scenario categories (e.g., Human, Object, Text, UI & Graphic Design). Each category has fine-grained sub-categories and concrete items.
  • They defined description dimensions for enriching prompts: Attribute, Spatial Relationship, Count, Interaction, and Color.
  • For each item, a few dimensions were sampled at random, and GPT-4.1 turned them into an image-generation prompt.
  • They manually reviewed the prompts and added prompts from categories where the base model performs poorly.

In the Lens RL dataset, human experts fix the training distribution in advance, and it stays the same across models and throughout training. APT covers the same scenarios but adapts its distribution at every round to the current model's diagnosed weaknesses. We will add these details to the appendix.

We thank the reviewer again. All new results and clarifications will be included in the revised manuscript.

[1] RubricRL: Simple Generalizable Rewards for Text-to-Image Generation [2] Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models

Reviewer_FNT9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APT, an automated framework for adapting the training-prompt distribution during text-to-image post-training. Three agents construct an evaluation set of prompt–rubric pairs, diagnose the current generator’s failures, and generate training prompts emphasizing weak capabilities while retaining coverage of stronger ones. This process alternates with reinforcement learning, using rubric-based rewards from GPT-4.1-mini. Experiments show improvements over five benchmarks and an autoregressive model.

Critical strengths
  1. Consistent improvements across multiple generators and external benchmarks. APT outperforms the handcrafted-data baseline in aggregate for all four diffusion models. The Janus-Pro experiment provides evidence of applicability across architectures.
  2. Generality. The experiemnts over diffusion and ar model provides evidence of applicability across architectures.
Critical weaknesses
  1. Prioritizing Failures Without Assessing Learnability. APT prioritizes capabilities with low diagnostic scores, but poor performance does not necessarily imply high training value. The method does not specify how the agent distinguishes tractable weaknesses from tasks beyond the model’s current capacity, or ensures that generated prompts provide informative reward variation. This leaves an important gap between identifying failures and selecting effective training tasks.

  2. Insufficient Isolation of the Adaptive Training Benefit. The experiments do not sufficiently distinguish the benefit of model-adaptive feedback from that of LLM-generated training prompts. Although the paper includes an ablation removing iterative feedback, the main comparison relies on a handcrafted dataset and lacks a matched baseline using the same LLM without diagnostic feedback. The specific contribution of diagnosis-guided data generation remains unclear.

Questions for authors
  1. How does the data-generation agent ensure that generated tasks are challenging yet learnable, particularly when all sampled outputs receive identical rewards?

  2. Under the same prompt-generation model, training-prompt count, and RL budget, how does APT compare with a curriculum generated without model-specific diagnostic feedback?

2 comments

We thank the reviewer for the insightful comments on learnability and on isolating the benefit of adaptive feedback. We respond to each point below.

W1: Prioritizing failures without assessing learnability

We agree that low performance does not imply high training value, and this is why APT does not sample only failures. Three design choices address learnability.

(i) A mixed-difficulty distribution rather than failure-only sampling. The Data Generation Agent is instructed (Appendix E.4) to devote a clear majority of prompts to weak capabilities, a smaller portion to moderately performing ones, and a limited minority to mastered ones. It is also told not to concentrate on the single weakest category. The Hard-Only Sampling ablation (Table 4) tests the reviewer's hypothesis directly. Training only on the weakest capabilities lowers the average from 71.26 to 70.64, which confirms that failure severity alone is not the right criterion.

(ii) Graded rewards from rubric decomposition. Each prompt is paired with up to 10 binary rubrics, and the reward is their mean. The reward is therefore graded rather than all-or-nothing. A difficult prompt (e.g., counting combined with text rendering) usually has some rubrics satisfied and others not, which keeps within-group reward variance and avoids zero-advantage groups.

(iii) Closed-loop correction. Because diagnosis is repeated every round, the agent sees whether a targeted capability has improved. Capabilities that become mastered are down-weighted, and the training distribution moves to the remaining weaknesses.

We acknowledge that the agent does not explicitly estimate learnability. We will state this in the limitations section.

W2: Insufficient isolation of the adaptive training benefit

Thanks for your suggestion. We agree that this comparison is important. As suggested, we add a Feedback-free LLM curriculum that matches APT in every respect except for the diagnostic report. It uses the same data-generation LLM (GPT-5.4), an adapted version of the same instruction prompt, in which the diagnostic report {testing_report} (prompts with reward and rubric scores) is replaced by the bare prompts {testing_prompts} without any scores, and the failure-analysis instructions are removed, the same number of training prompts per round (480), the same rubric generation and reward, and the same RL budget (400 steps, 20 steps per round). Table B1 shows the results. As the results show, removing diagnostic feedback reduces the average from 71.26 to 68.54, essentially matching the handcrafted set (68.61). This indicates that APT's advantage over the handcrafted baseline comes from model-specific diagnosis rather than from LLM-based prompt generation itself; the drop is also larger than removing refresh alone (70.34). The detailed system prompt used in Feedback-free LLM curriculum for GPT-5.4 is shown in Prompt 1.

Table B1. Isolating the effect of diagnostic feedback (Lens, 400 steps).

SettingDiagnosisGenEvalOneIG-ENGenEval2DPGCVTG-2KAvg
Base model–0.7020.52129.0685.930.48057.06
Handcrafted RL set–0.8150.51846.3587.600.75868.61
LLM, feedback-free*✗0.8010.52045.1887.810.77668.54
Stale feedback (w/o iterative loop)base only0.8440.52348.2787.610.79170.34
Reduced diagnostic criteriapartial0.8000.52049.5887.530.76369.08
APT (full)✓0.8590.52350.7387.880.79571.26

Q1: How to ensure tasks are challenging yet learnable, especially when all outputs receive identical rewards?

As described in W1, challenge comes from weighting prompts toward weak capabilities, and learnability is supported by (a) retaining moderate and mastered capabilities, (b) graded rubric-mean rewards, and (c) per-round re-diagnosis. When all sampled outputs for a prompt receive identical rewards, under group-normalized rewards (as in DiffusionNFT and GRPO), such a prompt provides no preference signal. That prompt contributes no gradient, so it wastes some compute but does not bias the update.

Q2: Under the same prompt-generation model, training-prompt count, and RL budget, how does APT compare with a curriculum generated without model-specific diagnostic feedback?

Please see Table B1. Under identical conditions (GPT-5.4, 480 prompts per round, 400 RL steps), removing diagnostic feedback reduces the average from 71.26 to 68.54, essentially matching the handcrafted set (68.61). This indicates that APT's advantage over the handcrafted baseline comes from model-specific diagnosis rather than from LLM-based prompt generation itself; the drop is also larger than removing refresh alone (70.34).

We thank the reviewer again. All new results will be included in the revised manuscript.

Prompt 1. System prompt used in Feedback-free LLM curriculum experiment.

Below is a set of reference prompts randomly sampled from a broad text-to-image evaluation set. They are provided only as examples of the scope and style of image-generation requests. No information is available about how any particular text-to-image model performs on these prompts, and you should not assume any model-specific strengths or weaknesses:
{testing_prompts}

Your task is to generate {N_train} new training prompts for reinforcement-learning post-training of a general text-to-image model. Base your choices only on general knowledge of what text-to-image generation commonly requires, not on the performance of any specific model.

The prompts should collectively cover a broad range of visual generation demands, such as:
- diverse object types, including humans, animals, plants, objects, food, scenes, and fictional entities
- attributes (e.g. color, count, size, shape, texture, material, age, pose, expression)
- relations (e.g. left/right, above/below, in front of/behind, overlapping, holding, wearing)
- text-rendering demands (e.g. OCR text, signage, labels, logos, clothing text)
- visual style or medium (e.g. photorealistic, watercolor, vector art, 3D render, sketch, low-poly, cinematic, flat design)

Generate the prompts with a mixed difficulty distribution, where difficulty is judged by what is generally challenging for current text-to-image models:
- A clear majority should involve generation demands that are generally challenging for current text-to-image models.
- A smaller portion should involve demands of moderate general difficulty.
- A limited minority should involve demands that are generally easy, to preserve breadth.

Important:
- Do not concentrate on a single capability or category.
- Keep the capability coverage broad and balanced across the dimensions listed above.
- Do not infer or speculate about the weaknesses of any specific model.

Requirements for the prompts:
- Do not merely copy the reference prompts; create new prompts that are diverse.
- Keep prompts natural and realistic for text-to-image model training.
- Make the prompts visually specific and testable.

Output format:
- Output exactly {N_train} lines.
- Each line must be valid JSONL: one JSON object per line.
- Do not output markdown fences.
- Do not output explanations, analysis, headings, bullets, or any text before or after the JSONL.
- Each JSON object must contain:
  - "id": integer from 1 to {N_train}
  - "prompt": the training prompt string

Example format:
{"id":1,"prompt":"..."}
{"id":2,"prompt":"..."}

Return only the JSONL prompt objects.
Reviewer_kD3zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select training prompts for text-to-image post-training. It proposes APT, which uses agents to evaluate the generator, diagnose its weaknesses and generate training prompts. This process is repeated as the generator is updated. Experiments cover several diffusion models and an autoregressive model.

Critical strengths
  1. The paper is generally well written and easy to follow.

  2. The framework addresses a practical problem, and the experiments across different generators support the usefulness of the approach.

  3. The experimental results show the efficacy of the proposed method.

Critical weaknesses
  1. It is somewhat unclear how much the repeated diagnosis contributes to the final improvement. Compared with stale feedback, the gains in Table 4 are relatively small, and no uncertainty across independent training runs is reported. A comparison with fresh prompt generation without model-specific feedback would help clarify this point.

  2. The capability gains come with reduced diversity and style quality, as reported in Table 8. The reviewer appreciates that the authors acknowledge this limitation, but it remains unclear whether the overall trade-off is favorable. Since diagnosis and training rewards use the same evaluator, an independent evaluation would also be helpful.

  3. The cost of diagnostic image generation, rubric scoring and agent calls is not fully reported. This makes the practical overhead of repeatedly updating the training prompts difficult to assess.

  4. Training time and the additional cost of diagnosis, rubric scoring and LLM calls are not reported.

  5. Sensitivity to several key settings is not examined, including the diagnostic batch size, the number of generated training prompts and the frequency of refreshing feedback.

Questions for authors

Please see weaknesses

4 comments

We thank the reviewer for the careful reading and constructive suggestions.

W1: Contribution of repeated diagnosis; uncertainty; feedback-free baseline

Thanks for your suggestion. We agree that this comparison is important. As suggested, we add a Feedback-free LLM curriculum that matches APT in every respect except for the diagnostic report. It uses the same data-generation LLM (GPT-5.4), an adapted version of the same instruction prompt, in which the diagnostic report {testing_report} (prompts with reward and rubric scores) is replaced by the bare prompts {testing_prompts} without any scores, and the failure-analysis instructions are removed, the same number of training prompts per round (480), the same rubric generation and reward, and the same RL budget (400 steps, 20 steps per round). Table A1 shows the results. As the results show, removing diagnostic feedback reduces the average from 71.26 to 68.54, essentially matching the handcrafted set (68.61). This indicates that APT's advantage over the handcrafted baseline comes from model-specific diagnosis rather than from LLM-based prompt generation itself; the drop is also larger than removing refresh alone (70.34). The detailed system prompt used in Feedback-free LLM curriculum for GPT-5.4 is shown in Prompt 1.

Table A1. The effect of diagnostic feedback (Lens, 400 steps).

SettingDiagnosisGenEvalOneIG-ENGenEval2DPGCVTG-2KAvg
Base model–0.7020.52129.0685.930.48057.06
Handcrafted RL set–0.8150.51846.3587.600.75868.61
LLM, feedback-free*✗0.8010.52045.1887.810.77668.54
Stale feedback (w/o iterative loop)base only0.8440.52348.2787.610.79170.34
Reduced diagnostic criteriapartial0.8000.52049.5887.530.76369.08
APT (full)✓0.8590.52350.7387.880.79571.26

W2: Diversity/style trade-off; independent evaluation

Trade-off. The drop in diversity is common to RL post-training (also observed for Lens and Mage-Flow) and is not specific to APT. The handcrafted RL baseline shows a similar drop. On Lens, diversity falls from 0.257 to 0.184 with the handcrafted set and to 0.174 with APT. Across the four models, APT differs from the handcrafted baseline in diversity by only 0.005–0.010. APT achieves higher style scores than the handcrafted baseline on Lens (0.335 vs. 0.314) and FLUX (0.340 vs. 0.330), matches it on LongCat (0.322), and is comparable on SD3.5 (0.314 vs. 0.316). The OneIG-EN average, which includes both diversity and style, is at least as high for APT as for the handcrafted baseline on all four models. We acknowledge that on SD3.5 it remains slightly below the base model (0.404 vs. 0.408). Overall, given the same RL algorithm, APT obtains larger capability gains at a similar diversity and style cost. We will expand this discussion in the limitations section.

Independent evaluation. The shared reward calculator is GPT-4.1-mini, which scores generated images against rubrics during both diagnosis and RL reward computation. However, none of the reported results are computed with this evaluator. All numbers in the paper come from the official protocols of five public benchmarks: detector- and classifier-based checks in GenEval, Soft-TIFA in GenEval2, MLLM-based VQA in DPG, OCR word accuracy in CVTG-2K, and dimension-specific scorers in OneIG-EN. None of these uses our GPT-4.1-mini rubrics, so the reported gains are unlikely to be explained by the policy exploiting its own training evaluator. We further add a human double-blind evaluation (Table A2) and an evaluator–human agreement study.

  • Human double-blind evaluation. We sampled 1,000 prompts from the five evaluation benchmarks (GenEval, GenEval2, OneIG-EN, DPG, and CVTG-2K) and generated images with the Base model, the Handcrafted RL model, and APT using the same seed and inference settings. Six PhD students from different research fields served as annotators. For each prompt, APT was compared with each baseline, giving 1,000 pairs per comparison, and each pair was rated by one annotator. The two images were shown side by side in random order without method labels, and annotators chose win, tie, or loss on prompt alignment, visual quality, and overall preference.

  • Evaluator–human agreement study. Following RubricRL[1], we conduct a human evaluation to validate the quality of generated rubrics and the reliability of our VLM grader (GPT-4.1-mini). We sample 100 prompts and generate 10 rubrics per prompt, yielding 1,000 prompt–rubric pairs. Human annotators assess prompt–rubric consistency, rubric hallucination, and grader correctness. Three annotators independently labeled all 1,000 prompt–rubric pairs, and we report the average over annotators. On average, 97.2% of the generated rubrics are consistent with their prompts, only 0.83% contain hallucinated constraints, and the VLM grader agrees with human judgments on 95.7% of pairs, demonstrating the reliability of our rubric-based evaluation.

Table A2. Human double-blind evaluation (Lens; 1,000 pairs per comparison, 6 annotators). Win / Tie / Loss (%) of APT.

ComparisonPrompt alignmentVisual qualityOverall preference
APT vs. Base81.1/9.8/9.171.3/8.8/19.980.5/8.5/11.0
APT vs. Handcrafted RL64.6/12.2/23.260.2/14.5/25.363.1/12.5/24.4

W3 & W4: Cost of diagnosis, rubric scoring, agent calls; training time

We agree that the costs of diagnosis, rubric scoring, and prompt generation should be reported explicitly, and we add a wall-clock breakdown (Table A3) and an API cost breakdown (Table A4). Both are for the default configuration on Lens: 400 RL steps with 20 steps per round (20 rounds), on 8 NVIDIA A100 80GB GPUs.

In each round, the diagnosis step generates N = 120 images and makes at most 120 × 10 = 1,200 binary rubric judgments. The Data Generation Agent is called once and produces 480 prompts, and rubric generation is called 480 times. The handcrafted baseline also needs rubric generation and RL reward scoring, since it uses the same number of prompts and the same reward procedure. The additional cost of APT therefore comes only from diagnostic image generation, diagnostic scoring, and one LLM call per round.

The full run took 32.70 hours (261.62 allocated GPU-hours), including initialization and API waiting time. In terms of API cost, the APT-specific components (diagnostic rubric scoring and the prompt-generation agent) cost $22.40, which is 6.5% of the total $347.17. In terms of wall-clock time, diagnosis takes 1.29 h (3.9%). Training-prompt and rubric generation run online between rounds and take another 3.90 h. Even if all prompt and rubric generation is counted as APT overhead (although the baseline also requires rubric generation), APT-specific steps take at most 5.19 h (15.9%). Nearly all of the cost comes from RL training and reward scoring, which both methods share.

Table A3. Wall-clock time (Lens, 400 steps, 8 × A100).

ComponentWall-clock timeShareAPT-specific
Initialization and model loading0.53 h1.6%–
Training sampling, reward scoring, optimization, and periodic validation26.99 h82.5%–
Diagnostic image generation, rubric scoring, and synchronization1.29 h3.9%✓
Training-prompt and rubric generation3.90 h11.9%partly
Total32.70 h100%≤ 5.19 h (≤ 15.9%)

Table A4. API cost (Lens, 400 steps).

API componentEstimated cost (USD)ShareAPT-specific
Diagnostic rubric scoring$12.323.5%✓
Prompt-generation agent$10.082.9%✓
Generation of new rubrics$5.951.7%– (also needed by the baseline)
Training-reward scoring$318.3991.7%–
Periodic validation scoring$0.440.1%–
Total$347.17100%$22.40 (6.5%)

Components may not sum exactly due to rounding.

W5: Sensitivity to key settings

In APT, the number of training prompts per round (NtrainN_{\text{train}}) is tied to the number of RL steps per round. Each RL step uses a fixed number of prompts (24 in our default setting: 480 prompts / 20 steps). When we change the refresh frequency, NtrainN_{\text{train}} scales with it: 240, 480, and 960 prompts for 10, 20, and 40 steps per round. The total number of training prompts (9,600) and RL steps (400) stay the same. Changing the refresh frequency therefore also changes NtrainN_{\text{train}}, so the second study below covers both settings. We vary one factor at a time from the default on Lens. "No refresh" is the existing stale-feedback variant (Table 4).

Table A5 shows that APT is robust to these settings. A diagnostic batch size of 120 is already close to the best result, and doubling it to 240 adds only +0.03. Refresh frequencies from 10 to 40 RL steps per round, with the matching NtrainN_{\text{train}} of 240 to 960, all give similar results (71.22–71.28). Disabling refresh lowers the average by 0.92, which confirms the benefit of periodic re-diagnosis.

Table A5. Sensitivity analysis (Lens, 400 steps; average over five benchmarks). Default in bold.

Diagnostic batch size N60120240
Avg70.8871.2671.29
RL steps per round102040No refresh
Rounds4020101 diagnosis
Avg71.2871.2671.2270.34

We thank the reviewer again. All new results will be included in the revised manuscript.

[1] RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

Prompt 1. System prompt used in Feedback-free LLM curriculum experiment.

Below is a set of reference prompts randomly sampled from a broad text-to-image evaluation set. They are provided only as examples of the scope and style of image-generation requests. No information is available about how any particular text-to-image model performs on these prompts, and you should not assume any model-specific strengths or weaknesses:
{testing_prompts}

Your task is to generate {N_train} new training prompts for reinforcement-learning post-training of a general text-to-image model. Base your choices only on general knowledge of what text-to-image generation commonly requires, not on the performance of any specific model.

The prompts should collectively cover a broad range of visual generation demands, such as:
- diverse object types, including humans, animals, plants, objects, food, scenes, and fictional entities
- attributes (e.g. color, count, size, shape, texture, material, age, pose, expression)
- relations (e.g. left/right, above/below, in front of/behind, overlapping, holding, wearing)
- text-rendering demands (e.g. OCR text, signage, labels, logos, clothing text)
- visual style or medium (e.g. photorealistic, watercolor, vector art, 3D render, sketch, low-poly, cinematic, flat design)

Generate the prompts with a mixed difficulty distribution, where difficulty is judged by what is generally challenging for current text-to-image models:
- A clear majority should involve generation demands that are generally challenging for current text-to-image models.
- A smaller portion should involve demands of moderate general difficulty.
- A limited minority should involve demands that are generally easy, to preserve breadth.

Important:
- Do not concentrate on a single capability or category.
- Keep the capability coverage broad and balanced across the dimensions listed above.
- Do not infer or speculate about the weaknesses of any specific model.

Requirements for the prompts:
- Do not merely copy the reference prompts; create new prompts that are diverse.
- Keep prompts natural and realistic for text-to-image model training.
- Make the prompts visually specific and testable.

Output format:
- Output exactly {N_train} lines.
- Each line must be valid JSONL: one JSON object per line.
- Do not output markdown fences.
- Do not output explanations, analysis, headings, bullets, or any text before or after the JSONL.
- Each JSON object must contain:
  - "id": integer from 1 to {N_train}
  - "prompt": the training prompt string

Example format:
{"id":1,"prompt":"..."}
{"id":2,"prompt":"..."}

Return only the JSONL prompt objects.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.