acceptodds
Under review as a conference paper at ICLR 2027

Internalizing Agent Experience into Diffusion Model Weights via On-Policy Context Distillation

Abstract

Wrapping an image generation model in an agentic harness can effectively boost Text-to-Image task performance: the harness can leverage memory, skills, workflow orchestration, result verification, and iterative refinement to continually construct and revise prompts, thereby eliciting better images. These gains, however, remain external to the diffusion model and are realized only while the full harness runs. We propose Diffusion On-Policy Context Distillation (D-OPCD), which treats the agent-improved prompt as privileged context and distills the knowledge encoded in the agent harness into the weights of the diffusion model, so that the model retains part of the harness's benefit when conditioned on the original query alone. Using a Text-to-Image agent equipped with our proposed Auto Skill Evolver (ASE), we show that D-OPCD can internalize harness capabilities into the generator's weights, raising the average direct-generation score from 60.52 to 65.09 across four benchmarks. With this knowledge absorbed into the weights, the harness can shed its saturated skills and resume evolving: a second ASE round on the updated generator improves on a skill-free harness by additional 1.83 points, pointing toward text-to-image systems in which harness and model keep improving each other through continual co-evolution.

Primary area:
generative models
Keywords:
Text-to-image generation, Agentic image generation, Harness–model co-evolution, Test-time learning, Diffusion on-policy context distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GE9HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether improvements obtained by a text-to-image agent harness can be transferred into the generator’s weights. Its Auto Skill Evolver (ASE) converts task episodes into insights and reusable prompt-rewriting skills. Diffusion On-Policy Context Distillation (D-OPCD) then trains a query-conditioned student to match an EMA teacher conditioned on both the query and the harness-produced prompt, using velocity matching on detached student-generated trajectory states.

Experiments use Z-Image-Turbo with benchmark-specific LoRA adaptation on GenEval, GenEval2, WISE, and R2I-Bench. D-OPCD improves the average direct-generation score from 60.52 to 65.09 and outperforms the tested SFT, Diffusion-DPO, and D-OPSD baselines. The paper also examines teacher conditioning, context filtering and diversity, OOD transfer, and a subsequent round of skill evolution after updating the generator and resetting the harness.

Critical strengths
  1. A useful interface between agent experience and generator learning, with consistent direct-generation improvements. Using the original query and the harness-selected prompt avoids requiring the full agent at deployment. Table 2 reports improvements over the base generator on all four benchmarks, with an average gain of 4.57 points and a 1.71-point advantage over SFT. These results support the practical possibility of transferring some prompt-mediated harness benefit into generator weights.

  2. The experimental protocol separates several important sources of evidence. The paper distinguishes training, selection, and final evaluation tasks; freezes the selected harness during replay; and compares training methods using the same retained task records. Table 3 tests matched versus shuffled teacher context, while Table 5 evaluates separately held-out OOD categories. The explicit discussion of filtering confounds and the limited number of co-evolution rounds also helps delimit what the experiments establish.

Critical weaknesses
  1. The comparative advantage is not yet sufficiently established, particularly against D-OPSD. Appendix D.3 describes D-OPSD as using a detached clean-output endpoint target from Qwen3-VL-4B-Instruct. The original method instead specifies velocity matching and explicitly replaces the VLM’s language-model weights with Qwen3-4B weights to mitigate conditioning-feature mismatch. The manuscript does not establish whether these implementation details were preserved or whether the endpoint formulation includes equivalent timestep weighting. This matters because D-OPSD is the closest algorithmic comparator. See Jiang et al., Eq. 7 and Appendix C.

  2. Moreover, Appendix D.4 specifies one training seed, and in-distribution evaluation uses one image per task without uncertainty estimates. Some advantages are small: the WISE difference between D-OPCD and D-OPSD corresponds to four additional successes among 350 tasks. Matching 2,000 optimizer steps also does not establish matched computation across objectives involving different numbers of model evaluations; effective batch size and actual training costs are not reported.

  3. The experiments do not isolate internalization of accumulated skills from ordinary prompt enhancement, or establish that resetting old skills is warranted. All distillation records come from the evolved harness. There is no corresponding distillation control using prompts from the skill-free harness or an experience-free rewriter. Matched-versus-shuffled context establishes the importance of task relevance, but does not isolate the contribution of ASE’s accumulated experience. This distinction is consequential because the objective closely follows existing on-policy distillation constructions, making the experience-specific findings central to the contribution. Likewise, Eq. 9 compares the updated skill-free harness with the original skill-equipped harness, but does not test whether the original skills remain beneficial on the updated generator.

Questions for authors
  1. Baseline clarification: Did the D-OPSD implementation use the original paper’s Qwen3 language-weight replacement within the multimodal encoder? Please specify its exact loss and timestep weighting, and explain how the “clean-output endpoint target” in Appendix D.3 relates to the original velocity-matching objective.

  2. Attribution clarification: Are there existing comparisons that distill prompts from the skill-free harness or an experience-free rewriter under the same protocol? If not, what evidence distinguishes learning from accumulated ASE experience from learning ordinary task-specific prompt enhancement?

  3. Reset clarification: Was the original skill library evaluated with the D-OPCD-updated generator? This comparison could determine whether the old skills are actually redundant and whether resetting them is preferable to retaining them.

5 comments

We appreciate your detailed feedback and constructive questions.

Q1: The implementation of the D-OPSD baseline is insufficiently clarified.

Thank you for requesting these implementation details. We apologize for not explaining the implementation clearly enough in the original manuscript. Our implementation uses the same loading helper as the official D-OPSD code to load the generator’s Qwen3 text-encoder weights into Qwen3-VL’s language-model component, retaining its vision encoder and connector weights.

The loss also follows clean-endpoint matching in the official released training code. On shared student states,

Lendpoint(t)=(1−t)2∥vs−sg(vT)∥22L_{\mathrm{endpoint}}(t)=(1-t)^2\|v_s-\mathrm{sg}(v_T)\|_2^2

It therefore corresponds to timestep-weighted velocity matching, rather than the paper’s unweighted objective. At t=0,0.1,0.25,0.5t=0,0.1,0.25,0.5, the weights are 1,0.81,0.5625,0.251,0.81,0.5625,0.25, respectively. We will clarify the encoder replacement, exact loss, and timestep weighting in Appendix D.3, and document the official code version and adaptations to our task records.

To further assess the sensitivity of our results to this objective choice, we additionally trained D-OPCD using the same endpoint-loss formulation as D-OPSD. We include the reported D-OPSD results below for reference.

MethodLossGenEval ↑GenEval2 ↑WISE ↑R2I-Bench ↑Avg. ↑
D-OPSDEndpoint80.7276.4848.8644.9662.75
D-OPCDVelocity81.2882.245046.8365.09
D-OPCDEndpoint85.981.9449.7145.1265.67

With endpoint matching, D-OPCD achieves an average score of 65.67 and still exceeds the reported D-OPSD scores on all four benchmarks. These results suggest that D-OPCD’s advantage is not specific to the unweighted velocity objective. We will also include this comparison in Appendix D.3.

Q2. The reported gains need stronger evidence of robustness across training and generation seeds, together with uncertainty estimates and significance tests.

Thank you for raising this point. We have expanded our experiments to examine variation in both training and image generation, and added uncertainty estimates and significance tests.

We retain the original results obtained with seed 42 and add training seeds 43 and 44. For each training seed, we select a checkpoint using the validation set and evaluate it with generation seeds 42, 43, and 44. Each checkpoint therefore produces three images per test task, giving nine training–generation combinations for each method on each benchmark.

We first average the three generation-seed scores for each trained model, then report the mean and sample standard deviation across training seeds. The full results are shown below, with standard deviations in parentheses. Avg gives equal weight to the four benchmarks.

MethodGenEvalGenEval2WISER2I-BenchAvg
SFT75.50 (3.23)78.67 (1.56)46.29 (1.51)44.45 (0.82)61.23 (1.77)
DPO76.30 (2.14)78.25 (0.19)47.94 (0.52)45.23 (0.54)61.93 (0.83)
D-OPSD76.91 (1.97)73.69 (1.49)47.46 (2.01)45.79 (0.58)60.96 (1.29)
D-OPCD83.73 (2.55)81.77 (1.04)52.32 (0.65)47.27 (0.76)66.27 (0.92)

D-OPCD has the highest mean score on all four benchmarks. Its overall score is 66.27, exceeding SFT, DPO, and D-OPSD by 5.04, 4.34, and 5.31 percentage points, respectively. It also has the highest overall score for each training seed after averaging over the three generation seeds. On WISE, where you raised this concern, the mean scores are 52.32 for D-OPCD and 47.46 for D-OPSD—a difference of 4.86 points, with a positive difference for every training seed.

We also assess the overall score differences using two-sided paired t-tests, pairing results by training seed, and report the corresponding 95% confidence intervals:

ComparisonImprovement (pp)95% CIp
D-OPCD − SFT5.04[1.56, 8.53]0.0248
D-OPCD − DPO4.34[0.90, 7.78]0.0323
D-OPCD − D-OPSD5.31[1.14, 9.48]0.0317

All three confidence intervals lie above zero, and the corresponding p-values are below 0.05. These results are consistent with the overall average advantage observed across training seeds.

We will add the full experimental protocol, multi-seed results, and statistical analysis to the experimental section and Appendix D.4 of the revised manuscript.

Q3: Equal optimizer updates do not ensure matched computational costs, and effective batch sizes and actual training costs are not reported.

Thank you for pointing this out. Using the same 2,000 optimizer updates does not ensure matched computational costs across methods. However, we do not believe that matching compute necessarily provides a fairer comparison. Different methods have different computational requirements and use the training data with different levels of efficiency. For example, given the same dataset, SFT may begin to overfit after consuming less compute. We therefore set a common, sufficiently large upper limit on optimizer updates and use a held-out validation set to select the best checkpoint for each method before evaluating it on the test set. This allows us to compare the performance each method achieves within the same training horizon.

Representative related work also adopts similar experimental controls. For example, Diffusion-DPO [1] uses the same schedule length when comparing SFT with DPO. D-OPSD [2] likewise specifies a fixed training horizon of 1,000 iterations for its LoRA setting and, in Appendix C, reports training costs of 5 GPU hours for SFT and 9 GPU hours for D-OPSD, comparing their performance despite the difference in compute. These examples show that using a common training schedule and comparing methods with different training costs are both established practices; matching actual compute is not a uniform prerequisite for all performance comparisons.

We also apologize for not clearly documenting the effective batch size and training costs in the appendix. For a configuration with an effective batch size of 4, using four NVIDIA A100 80GB GPUs for a complete 2,000-update training run, the cost per run is as follows:

MethodBatch sizeTraining cost per run (GPU hours)
SFT44
DPO416
D-OPSD418
D-OPCD417

We will add the training configurations, the basis for reporting training costs, and the validation-based checkpoint selection protocol to the experimental section and Appendix D.4 of the revised manuscript.

[1] Diffusion Model Alignment Using Direct Preference Optimization. https://arxiv.org/abs/2311.12908

[2] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. https://arxiv.org/abs/2605.05204v3

Q4: Does D-OPCD internalize accumulated ASE experience beyond ordinary task-specific prompt enhancement?

Thank you for raising this attribution question. We added a skill-free scaffold distillation control and reused the second generator-update results reported in our response to Reviewer 1LwT, Q3. We will include these results in the revised manuscript. Our experiments indicate that D-OPCD internalizes the agent’s overall experience, benefiting from both ordinary scaffold refinement and the additional experience accumulated through ASE. We believe that the value of ASE lies not only in providing transferable experience beyond scaffold refinement but also in enabling renewed harness adaptation that supports further generator improvement.

Experience used for internalizationGenEvalGenEval2WISER2I-BenchAvg.
Skill-free scaffold78.1681.4251.1446.2764.25
ASE: first round81.2882.2450.0046.8365.09
ASE: second round89.7483.7451.4347.2668.04

Specifically, distilling skill-free scaffold experience achieves an average score of 64.25, while first-round ASE experience raises it to 65.09, with gains on three of four benchmarks. This supports an additional transferable benefit from accumulated ASE experience beyond ordinary scaffold refinement. After the second update described in our response to Reviewer 1LwT, Q3, the average reaches 68.04, improving by 2.95 points with gains on all four benchmarks. While this gain includes additional optimization, it supports the value of renewed harness adaptation in providing experience for further generator improvement.

Q5: Was the original skill library evaluated with the D-OPCD-updated generator to assess whether the old skills are redundant and resetting is preferable to retention?

Thank you for raising this question. We added an experiment evaluating the D-OPCD-updated generator with and without the original skill library.

BenchmarkWith old skillsWithout old skillsDifference (pp)
GenEval91.2485.57+5.67
GenEval288.5288.38+0.14
WISE81.4380.86+0.57
R2I-Bench66.7970.18−3.39
Overall82.0081.25+0.75

The results show that the old skills retain utility after generator internalization. However, removing them yields comparable overall performance, with a difference of 0.75 percentage points, and their effects depend on the dataset.

Importantly, our motivation for resetting the skill library was to enable continued evolution after generator internalization, rather than to improve immediate inference performance over retaining the old skills. The original library was the best snapshot selected by held-out performance during the first ASE stage, and further evolution during that stage resulted in lower performance. After internalizing the accumulated experience into the generator, resetting provides a fresh starting point for learning skills around the updated generator’s capabilities and remaining limitations. This design aims to move beyond the previously saturated evolutionary trajectory and sustain further improvement, as supported by our subsequent skill-evolution and second generator-update results. Reset is not required in practical deployment, where useful old skills can still be retained.

We agree that “now-redundant” is imprecise and could misleadingly suggest that the old skills have become useless. We will replace it with “previously saturated” to clarify that skill evolution had reached a plateau with the original generator, while the old skills can still provide value after the update.

Reviewer_1LwTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates whether it’s possible to internalize the gains established by an agent harness guiding a diffusion model into its weights. The paper proposes D-OPCD, a distillation method to distill the gains of the harness into the base model, using the harness-produced prompt as privileged context for the teacher. The paper proposes ASE (Auto Skill Evolver) to learn from insights as the harness completes tasks and to generate query-prompt pairs where prompts can act as privileged context.

Critical strengths
  • The paper addresses a relevant problem by transferring some benefits of an agentic image generation workflow into the generator’s weights. Conditioning the teacher on harness-produced prompts provides a practical interface for learning from agent experience, while allowing the student to generate directly from the original request at inference time.
  • The paper has through experimental coverage and compares against various baselines like SFT, DPO and D-OPSD on multiple benchmarks. Along with that the paper performs reasonable ablations and evaluates both with and without its ASE skill evolution.
Critical weaknesses
  • ASE builds on existing work in continual and experiential learning for agents, as the paper acknowledges. On-policy context distillation and on-policy self-distillation for diffusion models have also been formulated separately. The specific contribution therefore lies in using ASE-generated prompts as privileged teacher context along student-generated denoising trajectories. However, ASE provides incremental improvements over the harness without learned skills, which already accounts for most of the improvement over direct generation.
  • SFT recovers a substantial portion of the improvement achieved by D-OPCD, leaving a relatively modest additional benefit from the proposed method. Furthermore, the authors perform only one generator update followed by another round of skill evolution, without updating the generator again using the newly evolved agent. The experiments therefore do not establish whether improvements persist across repeated update-and-reset cycles.
  • The paper would benefit from human validation of the generated images, since the reported improvements are measured using automated evaluators. A blinded comparison of outputs from the base generator, SFT, and D-OPCD would help determine whether human evaluators also observe improvements in faithfulness to the original request and overall visual quality.
Questions for authors
  • How sensitive is ASE to task-stream order? Please evaluate multiple shuffled task orders and report the variance in final performance.
  • What is the mean number of image regenerations and verification cycles per task for the skill-free and ASE-equipped harnesses?. This would help understand the efficiency of the method in practical settings
  • What happens after a second generator update? Are earlier improvements retained?
3 comments

We sincerely thank you for your careful review and helpful suggestions.

Q1. The contribution appears incremental, as experiential learning and on-policy distillation have already been established in prior work.

Thank you for discussing the novelty of our work. We acknowledge that our approach builds on existing experiential learning and on-policy distillation methods. However, our contribution goes beyond an incremental extension or a simple combination of these methods: we introduce and empirically validate a co-evolution learning task in which external experience accumulated by the harness becomes a persistent capability of the generator under query-only inference, and the harness then resumes experience accumulation around the updated generator. We demonstrate this transfer across four benchmarks and provide empirical support for continued improvement through this cycle. We firmly believe that introducing and validating this learning task is a meaningful contribution.

Q2: Does D-OPCD provide a substantial benefit beyond SFT?

Thank you for raising this concern. To better assess the magnitude and consistency of D-OPCD’s advantage, we added experiments with multiple training and evaluation seeds following Reviewer GE9H’s suggestion, and a human pairwise comparison study in response to your Weakness 3.

First, as detailed in our response to Reviewer GE9H, Q2, the repeated experiments show a larger and consistent advantage over SFT across all four benchmarks. The table reports mean scores, with standard deviations in parentheses.

MethodGenEvalGenEval2WISER2I-BenchAvg.
SFT75.50 (3.23)78.67 (1.56)46.29 (1.51)44.45 (0.82)61.23 (1.77)
DPO76.30 (2.14)78.25 (0.19)47.94 (0.52)45.23 (0.54)61.93 (0.83)
D-OPSD76.91 (1.97)73.69 (1.49)47.46 (2.01)45.79 (0.58)60.96 (1.29)
D-OPCD83.73 (2.55)81.77 (1.04)52.32 (0.65)47.27 (0.76)66.27 (0.92)

D-OPCD achieves the highest mean score on all four benchmarks, with an overall average of 66.27 versus 61.23 for SFT, 61.93 for DPO, and 60.96 for D-OPSD. Its overall advantage over each baseline is statistically significant (p<0.05p<0.05).

Second, human evaluation corroborates this advantage beyond automated metrics. As detailed in our response to Q4, D-OPCD achieves a 74.00% human preference score against SFT, with 126 wins, 44 ties, and 30 losses across 200 requests. Our qualitative inspection also suggests that SFT outputs are more prone to visible tearing artifacts and blurring, consistent with the human preference for D-OPCD when jointly considering request adherence and overall visual quality.

Q3. The experiments include only one generator update followed by skill evolution, leaving sustained improvement across repeated update-and-reset cycles unestablished.

Thank you for raising this important question. We believe the goal of this work is to connect external experience internalization with renewed harness adaptation and provide initial empirical evidence for co-evolution between the harness and the generator in text-to-image tasks. Accordingly, the original experiments focused on one transfer step and subsequent skill evolution around the updated generator. Longer controlled cycles would help further examine this process, while long-term outcomes in deployment also depend on changing task distributions, feedback quality, and compute budgets, requiring broader investigation.

Nevertheless, we agree that a second generator update is an important step toward strengthening the current evidence. We therefore added an experiment using the skill library S1 evolved on G1 to collect fresh query–prompt pairs, followed by a second D-OPCD update initialized from G1 to obtain G2. The table reports query-only generation scores under the same evaluation protocol.

Generator stageGenEvalGenEval2WISER2I-BenchAvg.
After the first generator update (G1)81.2882.2450.0046.8365.09
After the second generator update (G2)89.7483.7451.4347.2668.04

G2 improves the average by 2.95 points, with gains on all four benchmarks, supporting further improvement through renewed harness adaptation and generator internalization. We will report these results in the revised manuscript.

Q4: Do human evaluators also observe improvements in faithfulness to the original request and overall visual quality?

Thank you for this suggestion. To complement the automated evaluation, we added a human pairwise comparison study and will include its protocol and results in the revised manuscript.

We randomly sampled 200 test requests, with 50 each from GenEval, GenEval2, WISE, and R2I Bench. For each request, we paired D-OPCD with SFT, Flow-DPO, and D-OPSD, yielding 600 image pairs. The annotator viewed the original request and two images with method identities hidden, then selected the better image based jointly on request adherence and overall visual quality, with ties allowed. Presentation order and left–right positions were randomized; neither enhanced prompts nor automated scores were shown.

Wins, ties, and losses below are reported from D-OPCD’s perspective. The preference score is defined as (W+0.5T)/(W+T+L)(W+0.5T)/(W+T+L).

BaselineD-OPCD win / tie / lossD-OPCD preference score
SFT126 / 44 / 3074.00%
Flow-DPO108 / 50 / 4266.50%
D-OPSD109 / 54 / 3768.00%

D-OPCD achieves human preference scores of 74.00%, 66.50%, and 68.00% against SFT, Flow-DPO, and D-OPSD, respectively, indicating that human evaluators favor D-OPCD’s outputs when jointly considering adherence to the original request and overall visual appearance. Our qualitative inspection also suggests that SFT outputs are more prone to visible tearing artifacts and blurring, consistent with the human preference for D-OPCD.

Q5: How sensitive is ASE to task-stream order?

Thank you for this suggestion. We conducted two additional ASE evolution experiments with task streams randomly shuffled using seeds 17 and 73, supplementing the original seed-42 experiment. For each run, we evaluated the base model within the harness equipped with the skill library produced by that ASE evolution run on the benchmark test sets. The table reports the individual test scores and their mean and sample standard deviation across all three task orders. The standard deviations are 1.36, 0.97, 2.86, and 1.32 percentage points on GenEval, GenEval2, WISE, and R2I-Bench, respectively. These results suggest that ASE is reasonably robust to task-stream order across the tested permutations. We will include these results in the revised manuscript.

BenchmarkASE (Seed 42)ASE (Seed 17)ASE (Seed 73)Mean ± SD (3 Seeds)
GenEval87.6389.6990.2189.18 ± 1.36
GenEval288.8387.4789.3688.55 ± 0.97
WISE81.4380.2985.7182.48 ± 2.86
R2I-Bench69.4166.8668.7068.32 ± 1.32

Q6. What are the mean numbers of image regenerations and verification cycles per task for the skill-free and ASE-equipped harnesses?

Thank you for asking about practical efficiency. To address this question, we added an analysis of generation records for all 1,024 matched test tasks across the four benchmarks. In the table, Regen. and Verify. denote mean regenerations and verification cycles per task, respectively; each pair lists the skill-free harness (SF) first and the ASE-equipped harness second. Regenerations equal total generations minus the initial generation. Verification-cycle counts are inferred from generation counts under the standard workflow, where each generation is followed by one verification cycle.

BenchmarkTasksRegen. (SF / ASE)Verify. (SF / ASE)
GenEval1940.1443 / 0.05671.1443 / 1.0567
GenEval22802.1286 / 1.70363.1286 / 2.7036
WISE3500.9486 / 0.46001.9486 / 1.4600
R2I2001.5900 / 1.54002.5900 / 2.5400
Overall1,0241.2441 / 0.93462.2441 / 1.9346

Overall results are weighted by task count. ASE reduces mean regenerations on all four benchmarks, with an overall reduction of approximately 24.9% in regenerations and 13.8% in total generations. These results indicate that accumulated skills reduce the need for repeated generation within a task. We will include these statistics and counting conventions in the final version of the paper to help readers understand the method’s efficiency in practical settings.

Thank you, these address my concerns. However, I will keep my current rating for the paper

Reviewer_6BykOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distills prompts selected by an image-generation agent into a diffusion model. An EMA teacher sees the query and selected prompt, while a student sees only the query and learns along its own denoising trajectories. Z-Image-Turbo is adapted separately on four benchmarks, followed by another round of skill learning with an empty library.

Critical strengths

The direct-generation gains are convincing. Table 2 raises the mean from 60.52 to 65.09, exceeding SFT by 1.71 points with matched task records and a separate selection split. Appendix E.3 compares original and selected prompts with the same seed on retained training tasks, scoring both against the original requirements. Together with the shuffled-prompt result in Table 3, this supports transferring useful prompt changes into the generator. I see the use of agent-selected prompts as the main contribution beyond the distillation methods in D-OPSD and OPCD [1, 2].

[1] D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. https://arxiv.org/abs/2605.05204v3 [2] On-Policy Context Distillation for Language Models. https://arxiv.org/abs/2602.12275v2

Critical weaknesses
  1. My main concern is the claim that learned skills have been internalized. In Table 1, the skill-free harness already improves generation by 18.76 points, compared with 21.31 for the skill-equipped harness. All distillation records come from the latter. These scores do not tell us whether the learned skills produce better training prompts than the refinement already performed by the skill-free harness. The prompt-transfer result is supported, but the contribution of accumulated skills remains unclear.
  2. The introduction calls the old skills "now-redundant", yet Table 1 never evaluates the updated generator with them. Its mean gap of +0.50 compares the updated agent with an empty library against the original agent with skills, and is driven by R2I. New skills help on three benchmarks but lower R2I from 73.68 to 69.45. This shows that further skill learning can help after an update; it does not explain why the original library should be discarded.
Questions for authors

See weakness.

2 comments

We sincerely appreciate the thoughtful and constructive feedback.

Q1: Does accumulated ASE experience provide additional transferable benefits beyond skill-free scaffold refinement?

Thank you for raising this point. We added an experiment and found that accumulated ASE experience provides transferable benefits beyond ordinary scaffold refinement. Importantly, we believe that ASE’s value extends beyond its advantage over a skill-free scaffold in a single round: it enables renewed experience accumulation around the updated generator, supporting continued improvement through harness–generator co-evolution.

To directly assess the contribution of accumulated skills, we added an experiment that trains the generator using D-OPCD on query–prompt pairs produced by the skill-free scaffold. Internalizing this experience achieves an average score of 64.25, while first-round ASE experience raises it to 65.09. This comparison shows that ordinary scaffold refinement provides useful training experience, while accumulated ASE skills contribute an additional transferable benefit. All scores below measure direct generation from the original query, without the harness or external skills at inference time.

Experience used for internalizationGenEvalGenEval2WISER2I-BenchAvg.
Skill-free scaffold78.1681.4251.1446.2764.25
ASE: first round81.2882.2450.0046.8365.09
ASE: second round89.7483.7451.4347.2668.04

Beyond this first-update benefit, ASE allows experience accumulation to continue around the updated generator. Following Reviewer 1LwT’s suggestion (Q3), we used the skill library evolved on G1 to collect fresh query–prompt pairs and performed a second D-OPCD update initialized from G1. The resulting generator G2 achieves an average score of 68.04, improving by 2.95 points over G1, with gains on all four benchmarks. This result supports continued improvement through renewed harness adaptation and further generator optimization.

Together, these experiments clarify ASE’s contribution: it provides additional transferable experience beyond scaffold refinement and renews experience around the updated generator to support further improvement. We will include these results and clarify this distinction in the revised manuscript.

Q2: Do the original skills remain useful after the generator update, and why reset the skill library?

Thank you for raising this question. We added a comparison using the updated generator with and without the original skill library. Retaining the old skills achieves an overall score of 82.00, compared with 81.25 when removing them. These results show that the original skills remain useful after generator internalization, although the overall difference is modest and their effects vary across benchmarks.

BenchmarkWith old skillsWithout old skillsDifference (pp)
GenEval91.2485.57+5.67
GenEval288.5288.38+0.14
WISE81.4380.86+0.57
R2I-Bench66.7970.18−3.39
Overall82.0081.25+0.75

Our motivation for resetting the skill library concerns continued evolution. The original library was the best snapshot selected by held-out performance during the first ASE stage, and further evolution during that stage resulted in lower performance. After internalizing the accumulated experience into the generator, we reset the library to provide a fresh starting point for skill learning around the updated generator’s capabilities and remaining limitations. This strategy was intended to help evolution move beyond the previously saturated trajectory. It does not establish that resetting is preferable to retaining and further evolving the old library, and reset is not required in practical deployment.

We agree that “now-redundant” in the introduction is imprecise and could suggest that the old skills have lost their utility after the generator update. We will replace it with “previously saturated” to clarify that skill evolution had reached a plateau with the original generator, while the old skills can still provide value after the update.

Dear Reviewer,

Thank you again for your thoughtful comments. We are writing to gently check whether you have had the opportunity to review our rebuttal, and whether our responses have properly addressed your concerns. We would be deeply grateful if you would consider revising your score should you find our replies satisfactory.

Best regards, The Authors

Comments on the paper

1 comment

We thank the reviewers for recognizing the value of transferring agent experience into the generator and our gains across four benchmarks. We address the shared concerns below.

Q1. The contribution of accumulated ASE experience beyond ordinary scaffold refinement remains unclear.

We added a skill-free scaffold distillation control and found that accumulated skills provide a transferable benefit beyond ordinary prompt refinement. The skill-free scaffold still verifies images and refines prompts within each task. Training D-OPCD on its query–prompt pairs yielded an average score of 64.25. Using first-round experience from the Auto Skill Evolver (ASE) raised the average to 65.09, with gains on three of four benchmarks. This 0.84-point advantage persists when the generator sees only the original query, without the harness or skill library at inference.

We believe ASE’s value extends beyond this first-round advantage. After the generator internalizes earlier experience, ASE can learn new skills around the updated generator and supply fresh training examples. Our second D-OPCD update raised the query-only average from 65.09 to 68.04. This 2.95-point gain includes additional optimization and supports our broader claim that renewed harness adaptation can supply useful experience for further generator improvement.

Q2. The original experiments include only one generator update and leave the rationale for resetting the skill library unclear.

We added a second D-OPCD update using fresh query–prompt pairs collected with the skills learned on the first updated generator (G1). Starting from G1, we trained G2 on these pairs. G2 outperformed G1 on all four benchmarks under query-only inference, showing that renewed harness adaptation can support further generator improvement after the first transfer.

We also tested the original library on G1. Retaining the old skills achieves an overall score of 82.00, compared with 81.25 when removing them. These results show that the original skills remain useful after generator internalization, although the overall difference is modest and their effects vary across benchmarks. However, our motivation for resetting the skill library concerns continued evolution rather than immediate performance gains. Further evolution past the best held-out snapshot had lowered performance on the original generator. Resetting gave ASE a fresh starting point for learning skills suited to G1. Whether further evolving the old library would work as well remains open. We will replace “now-redundant” with “previously saturated” to clarify this distinction.

Q3. The reported gains need stronger evidence across training and generation seeds, task orders, and human judgments.

We added two training seeds and evaluated each selected checkpoint with three generation seeds. The advantage over SFT was larger than in the original comparison and held across training seeds. D-OPCD had the highest mean score on all four benchmarks, averaging 66.27 versus 61.23 for SFT, 61.93 for Flow-DPO, and 60.96 for D-OPSD. All three overall advantages were significant in two-sided t-tests paired by training seed after averaging over generation seeds (p < 0.05). The gain over SFT was 5.04 points (95% CI 1.56 to 8.53). On WISE, where the original comparison with D-OPSD was close, D-OPCD led for every training seed, by 4.86 points on average. We also reran ASE on two shuffled task orders. Across all three orders, benchmark scores had standard deviations of 0.97 to 2.86 points, suggesting robustness to the tested permutations.

We also conducted a blinded human comparison on 200 randomly sampled requests, 50 per benchmark. One annotator judged all 600 pairs on request adherence and overall visual quality. Against SFT, D-OPCD recorded 126 wins, 44 ties, and 30 losses. Counting ties as half a win, its preference scores were 74.0% against SFT, 66.5% against Flow-DPO, and 68.0% against D-OPSD. Human evaluation thus corroborates D-OPCD’s advantage beyond the automated metrics.

Q4. The contribution appears incremental, as experiential learning and on-policy distillation have already been established in prior work.

We acknowledge that our approach builds on experiential learning and on-policy distillation. Yet we believe our contribution goes beyond combining these methods. We introduce and empirically validate a co-evolution learning task in which external experience accumulated by the harness becomes a persistent capability of the generator under query-only inference. The harness then resumes experience accumulation as it adapts to the updated generator.

We demonstrate this transfer across four benchmarks. After the first generator update, the harness learns new skills and gathers fresh experience that supports a second update, improving query-only generation on all four benchmarks. Our central contribution is to formulate this learning task and demonstrate that improvement can continue through this cycle.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.