acceptodds
Under review as a conference paper at ICLR 2027

Learning from Reflection: On-Policy Self-Distillation for Text-to-Image Generation

Abstract

Unified multimodal models can diagnose errors in their own generations, providing image-level feedback for text-to-image post-training. However, existing methods typically use this feedback as sparse rewards to optimize the model, making it difficult to assign credit to individual generation states across timesteps. We propose Text-to-Image On-Policy Self-Distillation (T2I-OPSD), which turns textual reflections into dense state-level supervision, providing explicit velocity correction targets to improve prompt alignment. Specifically, the current model acts as both student and self-teacher, with the latter additionally conditioned on the student's generated image and textual reflection. By matching their velocity predictions at shared student-induced states, we turn image-specific feedback into dense directional supervision that shapes the student's velocity field for subsequent prompt-only generation. Furthermore, to stabilize supervision as the student evolves, we use reference-centered residual distillation to interpolate self-teacher predictions with those of a frozen prompt-only reference. We evaluate T2I-OPSD on multiple text-to-image benchmarks, achieving leading performance in compositional generation without additional sampling steps or inference memory.

Primary area:
generative models
Keywords:
Text-to-image generation, Unified multimodal models, On-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ux4NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

T2I-OPSD transfers reflection-conditioned supervision into a prompt-only text-to-image model. Starting from a reflection-capable AlphaGRPO checkpoint, the model generates an image and textual reflection, then constructs noisy query states by forward resampling from the generated endpoint. The prompt-only student matches a same-parameter teacher conditioned on the image and reflection. Interpolation with a frozen prompt-only reference stabilizes the targets. The resulting model requires no reflection or second generation round at inference. Evaluation covers compositional benchmarks, preference metrics, and a user study.

Critical strengths
  1. Meaningful improvements without additional inference-time refinement. The method addresses the useful problem of internalizing an existing correction capability. Relative to the same-backbone AlphaGRPO model, GenEval improves from 77.17 to 85.04 and DPG-Bench from 86.05 to 88.77. These gains support practical value beyond simply spending an additional generation round at deployment.

  2. Complementary evaluation supports the overall training recipe. Improvements extend to TIIF-Bench and preference metrics, and the method receives 52% of choices in a randomized anonymous four-way user study. Reference-weight, teacher-update, and SFT comparisons also expose unstable alternatives. Together, this is stronger evidence than a single benchmark improvement or selected qualitative examples.

Critical weaknesses
  1. The independent contribution of textual reflection is not isolated. The teacher receives both the generated image and reflection, and its image CFG scale differs from rollout generation. Without an image-only teacher and removed or shuffled reflections under matched CFG and reference mixing, the gains cannot be attributed specifically to meaningful reflection. This is consequential because reflection is the central distinction in the paper’s framing, whereas image-conditioned privileged distillation could explain part of the improvement.

  2. The proposed co-drift mechanism is not conclusively established. Eq. 9 decomposes target refresh but does not establish instability. A declining teacher-versus-student win rate can reflect a stronger student rather than decreasing absolute correction quality. Similar online, EMA, and frozen-teacher curves help rule out a purely parameter-refresh explanation, but do not distinguish degraded reflections from reduced relative headroom. The training-stability observations are valuable; their proposed causal interpretation requires more direct evidence.

Questions for authors
  1. Under matched CFG and reference mixing, how does an image-only teacher compare with image-plus-reflection conditioning? What changes when reflections are removed or shuffled?
  2. On a fixed image/prompt evaluation set, does absolute teacher correction quality decline during training, or does only its relative advantage over the improving student decline?
  3. For the user study, please clarify participant consent and ethics-review approval or exemption status. The stated anonymity and randomization do not specify this status.
2 comments

Thank you for recognizing the practical gains without additional inference-time refinement and the breadth of the evaluation. We have added teacher-conditioning ablations and fixed-input correction analysis to address your concerns about textual reflection and co-drift. We have also corrected the image-CFG description and explicitly documented participant consent and ethics approval for the user study.

W1 / Q1. Isolating textual reflection under matched settings. Can the authors compare image-only and image-plus-reflection teachers, and remove or shuffle textual reflections under matched CFG and reference mixing? Otherwise, image-conditioned privileged distillation may explain the improvement.

Thank you for this suggestion. We apologize for the confusion caused by the original CFG description. In the original manuscript, we mistakenly reported the code’s default image CFG value of 1.0 as a rollout setting. This setting has no effect on student rollouts because they use only the prompt with text CFG 4.0 and receive neither an input image nor textual reflection. The image CFG scale of 2.0 belongs to the additional SRR round used to evaluate AlphaGRPO+SRR. During distillation, teacher velocity targets are computed without image CFG, so the reported 1.0-to-2.0 difference does not represent stronger image guidance for the teacher. We have clarified these settings in Sec. 4.1 and Table 2.

We also agree that controlled ablations are needed to distinguish the contributions of textual reflection and visual context. To address this, we compare the full method with four variants: image only, shuffled textual reflection, textual reflection only, and shuffled images. All variants share the same initialization, reference-mixing coefficient, teacher update strategy, and other training settings. Only the teacher-conditioning inputs change.

The table below summarizes student ImageReward at selected training steps. Fig. 7(a) shows the full training curves.

Teacher conditioningStep 0Step 50Step 100Step 150Step 200
Image + textual reflection (Ours)1.0661.4271.4361.5041.513
Shuffled textual reflection1.0661.2541.2831.3351.391
Shuffled images1.0661.1991.1941.2191.232
Textual reflection only1.0661.2501.2661.2581.244
Image only1.066————

Step 0 reports the shared initialization score before training. “—” denotes training collapse. ImageReward is used only for monitoring and does not enter the training objective.

The full method achieves the highest final student ImageReward. Keeping the image input but shuffling textual reflection reduces the step-200 score from 1.513 to 1.391, showing that sample-specific textual feedback contributes beyond image conditioning. Removing or shuffling the image also lowers performance, supporting the additional value of correctly paired visual context. Training remains stable with textual reflection alone, whereas the image-only variant collapses after an initial improvement and exhibits substantially larger gradient norms. This suggests that removing textual reflection shifts the teacher’s conditioning distribution and may impair its ability to provide stable corrective supervision. Together, these results support the complementary roles of visual context and textual reflection in providing effective teacher supervision.

In the revised manuscript:

  • Sec. 3.1, p. 4: Clarified the roles of visual context and textual reflection.
  • Sec. 4.1, pp. 7–8, and Table 2, p. 16: Corrected the CFG configuration description.
  • Sec. 4.4 and Fig. 7(a), p. 10: Added and analyzed all four teacher-conditioning variants.

W2 / Q2. Evidence for co-drift and absolute correction quality. Eq. 9 decomposes target refresh but does not prove instability. On a fixed prompt–image evaluation set, does teacher correction quality actually decline, or does its advantage diminish because the student improves? Similar online, EMA, and frozen-teacher curves do not by themselves resolve this distinction.

Thank you for pointing out the limits of our original co-drift analysis. To distinguish changes in teacher correction quality from a loss of corrective benefit as the student improves, we evaluate different teacher checkpoints on two fixed sets of prompt–image pairs generated by the initial and step-200 students. Within each set, all teacher checkpoints use the same input images, correction settings, and correction seeds. The table below summarizes the mean ImageReward gains over the original inputs, while Sec. 4.5 and Fig. 7(b) provide a more complete visualization and analysis, including absolute corrected-image scores and 95% confidence intervals for the gains. We have also revised the co-drift discussion in light of these results.

Table: Mean ImageReward gain over the fixed student inputs.

Fixed student inputs / Teacher step020406080100120140160180200
Student 0+0.1372+0.1637+0.1222+0.1088+0.1406+0.2043+0.1923+0.2072+0.1053+0.1227+0.1448
Student 200−0.1323−0.1858−0.2328−0.2292−0.1847−0.1473−0.1257−0.0851−0.1356−0.1417−0.1929

Columns indicate teacher checkpoint steps. Student 0 and Student 200 denote fixed input images generated by the initial and step-200 students, with mean input ImageReward scores of approximately 1.078 and 1.523, respectively. Gains are paired differences between corrected and original input-image scores.

On either fixed set, the corrected-image score does not decline consistently across teacher checkpoints. However, the correction benefit differs between the two input sets. All 11 teacher checkpoints yield positive mean gains on initial student inputs, while their mean gains on step-200 student inputs are consistently negative. Thus, the teacher remains useful for the initial student's outputs while, on average, degrading the stronger student's outputs according to ImageReward. The fixed-input evaluation therefore shows that the teacher’s corrections remain beneficial for the initial student but reduce ImageReward on the stronger student’s outputs, providing direct evidence that corrective supervision can become detrimental as the student improves.

Eq. 9 characterizes the cross-step coupling between the student and its supervision: student updates change the generated images and teacher-conditioning context, which in turn change the targets used for subsequent student updates. Our fixed-input evaluation shows that teacher corrections can become detrimental to the stronger student. Under this coupling, continued distillation from harmful corrective targets can degrade the student, whose altered outputs then affect subsequent supervision and potentially reinforce the degradation. This is the co-drift risk we aim to explain, and the collapse under weak reference anchoring in Fig. 6(a) is consistent with this interpretation. We have clarified this connection in the introduction, method discussion, and conclusion.

In the revised manuscript:

  • Sec. 3.3, pp. 5–6: Revised the co-drift discussion following the target-refresh decomposition.
  • Sec. 4.5 and Fig. 7(b), p. 10: Added fixed-input absolute correction scores and paired gains.
  • Sec. 4.4 and Fig. 6(a), p. 9: Reported the reference-anchoring training dynamics.
  • Appendix E.2, pp. 18–19: Clarified the relative nature of teacher win rates and linked to the fixed-input analysis.
  • Introduction, p. 2, and Conclusion, p. 10: Reframed the claim as analysis of co-drift risk.

Q3. Consent and ethics-review status. Please clarify whether participants provided informed consent and whether the user study received ethics approval or a formal exemption. Anonymization and randomization alone do not specify these conditions.

Thank you for requesting this clarification. All participants provided informed consent before the evaluation, and the study has received ethics approval. We have added this statement explicitly to the user-study paragraph in Sec. 4.2.

In the revised manuscript:

  • Sec. 4.2, p. 8, lines 420–426: Explicitly stated participant informed consent and ethics approval.
Reviewer_9pppOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper attempts to apply OPSD to image generation, transferring a unified multimodal model’s existing self-reflective refinement capability into image generation that does not require a second generation round. The generated reflections serve as privileged conditioning for the teacher model to correct the student model’s generation trajectories. Experiments on an AlphaGRPO-trained BAGEL checkpoint show that further OPSD training improves multiple overall generation metrics over both the original model and the original model with inference-time SRR.

Critical strengths
  1. The paper explores combining OPSD with image generation, using AlphaGRPO-generated reflections as privileged conditioning for the teacher model to correct the student model’s generation trajectories.
  2. The OPSD-trained model is evaluated on multiple benchmarks and outperforms both the original model and its inference-time SRR variant on several metrics.
Critical weaknesses
  1. The method is evaluated only on BAGEL initialized from an AlphaGRPO checkpoint, without experiments on other backbones. Although outperforming two-round SRR refinement with single-round generation is practically valuable, SRR is an inference strategy, so this comparison is insufficient to establish an advantage over other post-training methods.
  2. The TIIF-Bench results in Table 1 are inconsistent with those in Tables 4 and 6. The authors should clarify the data sources and evaluation protocols underlying these results and explain any differences in experimental settings.
Questions for authors
  1. The comparison with alternative training methods is limited; the main comparison is effectively against SRR, which is an inference strategy. Could the authors include additional post-training methods with comparable settings, such as representative OPD methods, and compare their training costs?
  2. Could the authors explain the discrepancies between the TIIF-Bench results in Table 1 and those in Tables 4 and 6? If they arise from different experimental settings, please specify those differences.
3 comments

Thank you for recognizing the improvements in prompt-only generation and the value of transferring an existing refinement capability into training supervision. We have added validation on SANA with Reflect-DiT, comparisons with four additional BAGEL-based post-training methods, and a training-time comparison on eight H20 GPUs. We have also clarified the sources and evaluators underlying the differing TIIF-Bench baseline results.

W1(a). Applicability beyond BAGEL. The method is evaluated only on BAGEL initialized from AlphaGRPO. Can the distillation approach also be applied to another backbone?

Thank you for this suggestion. We have added an experiment adapting T2I-OPSD to SANA initialized from a Reflect-DiT checkpoint with reflection-conditioned generation capability, as reported in Appendix F.

We use the same training dataset as in the BAGEL experiments and generate images at 1024 × 1024 resolution with 20 sampling steps and a CFG scale of 4.5. We train LoRA adapters in the denoising backbone, with rank and alpha both set to 32 and a learning rate of 10−410^{-4}. The 200-step run uses eight NVIDIA H20 GPUs and requires approximately 42.9 H20 GPU-hours.

The table below summarizes the student ImageReward curve at selected training steps, with the full curve shown in Fig. 10.

Table: Student ImageReward during training on SANA / Reflect-DiT.

Training step020406080100120140160180200
ImageReward0.98001.00061.01121.02741.03351.04421.04411.05871.06251.08531.0943

Despite intermediate fluctuations, the student's reward curve shows an overall upward trend, rising from its initial value of 0.9800 to 1.0943 at step 200. The trained student retains prompt-only generation without reflection or iterative correction at inference, providing additional evidence that our distillation approach can be applied beyond BAGEL.

In the revised manuscript:

  • Sec. 4.1, p. 8: Added a pointer to the additional-backbone experiment.
  • Appendix F and Fig. 10, p. 19: Reported the SANA / Reflect-DiT setup and training dynamics.

W1(b) / Q1. Post-training baselines and training cost. SRR is an inference strategy, so outperforming SRR alone does not establish an advantage over other post-training methods. Can the authors add comparable post-training baselines and compare their training costs?

We agree that the comparison with AlphaGRPO+SRR primarily demonstrates the benefit of avoiding inference-time refinement. To broaden the post-training comparison, we have added Self-SpectraReward, UniCorn, BAGEL-RecA, and SRUM, using their publicly released checkpoints. These methods share the BAGEL backbone.

MethodTIIF-Short ↑TIIF-Long ↑DPG-Bench ↑GenEval ↑WISE Verified ↑
SRUM72.2572.8585.0672.660.37
UniCorn74.7072.9086.8382.000.40
BAGEL-RecA77.5677.7085.2982.400.38
Self-SpectraReward80.7180.9187.5584.300.39
T2I-OPSD83.3583.2888.7785.040.41

T2I-OPSD achieves the highest scores among these methods on all five reported columns. Relative to Self-SpectraReward, the strongest added baseline on TIIF-Bench, DPG-Bench, and GenEval, the gains are 2.64 / 2.37 points on TIIF-Short / Long, 1.22 points on DPG-Bench, and 0.74 points on GenEval. Table 1 also reports preference metrics from DrawBench generations, where T2I-OPSD achieves higher ImageReward, PickScore, and HPSv2 scores than these added baselines.

We additionally compare T2I-OPSD and Self-SpectraReward, both online post-training methods, on eight NVIDIA H20 GPUs. We report the actual training time for each method:

MethodGPUsTraining time (hours)
Self-SpectraReward8 × H2058.27
T2I-OPSD8 × H2014.40

Under the respective training configurations, T2I-OPSD takes approximately 75.3% less time while obtaining higher benchmark scores. UniCorn, SRUM, and BAGEL-RecA train on offline-collected data, so we include them in the performance comparison but exclude them from this comparison of online post-training time.

In the revised manuscript:

  • Table 1, p. 7: Added four BAGEL-based post-training baselines.
  • Secs. 4.1–4.2, p. 8: Described checkpoint usage and analyzed the quantitative comparisons.
  • Appendix F.1, pp. 19–20, and Fig. 11, p. 20: Reported training time on eight H20 GPUs.

W2 / Q2. Inconsistent TIIF-Bench results across tables. Why do the TIIF-Bench results in Table 1 differ from those in Tables 4 and 6? Which data sources, evaluation splits, and scoring models underlie these results?

Thank you for pointing this out. All TIIF-Bench scores in Table 1 are evaluated with Qwen2.5-VL-72B, including our reproduced results. For SD3, FLUX.1-dev, Show-o, and Janus-Pro, however, Tables 4 and 6 directly quote the GPT-4o scores reported in the TIIF-Bench paper. Both sets of published results use testmini, so the cross-table differences come from the scoring model rather than the evaluation split. The corresponding values are listed below.

External baselineTable 1: Qwen2.5-VL-72B, Short / LongTables 4 / 6: GPT-4o, Short / Long
SD364.79 / 64.7967.46 / 66.09
FLUX.1-dev66.24 / 66.7271.09 / 71.78
Show-o57.34 / 59.2459.72 / 58.86
Janus-Pro65.38 / 61.1066.50 / 65.02

Specifically, our reproduced results for BAGEL, AlphaGRPO, AlphaGRPO+SRR, and T2I-OPSD all use Qwen2.5-VL-72B and remain consistent across the tables.

We apologize that the original manuscript did not make these sources sufficiently clear. We have added the evaluation protocol in Sec. 4.1 and explained the sources of the four external baselines in Appendix I, making the evaluator difference explicit.

In the revised manuscript:

  • Sec. 4.1, p. 8: Specified the testmini split and Qwen2.5-VL-72B evaluator for the cited main-table results.
  • Appendix I, pp. 20–21: Identified the four external baselines and explained the GPT-4o versus Qwen2.5-VL-72B sources.
  • Table 1, p. 7; Tables 4 and 6, pp. 23–24: Provided the corresponding results.

Thanks to the authors for their detailed response and experiments. The newly added comparisons of post-training methods and clarification of the TIIF-Bench evaluation discrepancies have addressed my concerns. I will maintain my original rating of 3 (Weak acceptance).

Reviewer_mJSvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to turn a unified model's textual diagnosis of its own generated images into training supervision. The proposed Text-to-Image On-Policy Self-Distillation (T2I-OPSD) provides more direct training guidance than image-level rewards while avoiding the inference cost of repeated reflection and regeneration. During training, the model serves as both a prompt-only student and a teacher additionally conditioned on the generated image and its textual reflection. The student matches the teacher’s velocity predictions, blended with a frozen reference to stabilise training. The resulting model generates images without reflection or additional generation rounds at inference.

Critical strengths
  • This work connects two useful capabilities of unified multimodal models: inspecting an image and generating one. Distilling the predictions of a reflection-conditioned teacher offers a plausible route from descriptive feedback to state-level training targets. Retaining only the prompt-conditioned student at deployment directly addresses the cost of repeated reflective refinement.
  • The main empirical results support improved compositional generation with a single generation round. Relative to AlphaGRPO, T2I-OPSD shows substantial improvements on GenEval, DPG-Bench, and TIIF-short as reported in Table 1.
Critical weaknesses
  • The empirical evidence for whether the model improves because of the reflection itself is insufficient. The teacher also sees the generated image and uses stronger guidance (CFG 1.0->2.0), which could contribute to the gains. I would suggest additional ablation experiments that remove or shuffle the reflection while keeping everything else unchanged to separate these effects. This would help establish stronger support to the central claim that reflection provides useful corrective supervision.
  • The co-drift explanation is plausible, but the evidence does not distinguish deteriorating feedback from a diminishing need for correction as the student improves. A declining teacher win rate measures the teacher’s relative advantage, rather than the accuracy of its reflections. The anchoring experiments demonstrate improved training stability, but do not establish that unreliable reflective feedback is the cause of instability.
Questions for authors
  • What evidence distinguishes the proposed co-drift failure from a teacher's reflections becoming less useful as the student improves?
5 comments

Thank you for recognizing the value of converting reflection-conditioned predictions into state-level supervision and retaining prompt-only generation at deployment. Your comments helped us clarify the role of textual reflection and the interpretation of co-drift. We have added four teacher-conditioning ablations and a fixed-input evaluation of correction effectiveness, corrected the description of image CFG, and revised the mechanism discussion to distinguish empirical observations from the proposed explanation.

W1. Contribution of textual reflection. The teacher receives both the generated image and textual reflection, and the original setup appears to give it stronger image CFG. How much of the improvement comes from meaningful textual reflection? What happens when reflection is removed or shuffled?

Thank you for pointing out the unclear CFG description. In our implementation, student rollouts use only the prompt, without an input image or textual reflection, and the text CFG scale is 4.0. The image CFG scale of 1.0 is a default option in the code that has no effect on these prompt-only rollouts, while the scale of 2.0 is used only for the additional SRR round when evaluating AlphaGRPO+SRR. We apologize for the confusion caused by presenting these values as training settings in the original manuscript. To clarify, the teacher velocity targets are computed without image CFG during distillation, so the reported difference does not reflect stronger image guidance for the teacher. We have corrected this description in Sec. 4.1 and Table 2.

We have also added four teacher-conditioning ablations in Fig. 7(a): shuffled textual reflection, shuffled images, textual reflection only, and image only. All variants use the same reference-mixing coefficient, teacher update strategy, initialization, and all other training settings, differing only in the teacher-conditioning inputs. The table below summarizes student ImageReward at selected training steps; Fig. 7(a) shows the full training curves.

Teacher conditioningStep 0Step 50Step 100Step 150Step 200
Image + textual reflection (Ours)1.0661.4271.4361.5041.513
Shuffled textual reflection1.0661.2541.2831.3351.391
Shuffled images1.0661.1991.1941.2191.232
Textual reflection only1.0661.2501.2661.2581.244
Image only1.066————

Step 0 reports the shared initialization score before training. “—” denotes training collapse. ImageReward is used only for monitoring and does not enter the training objective.

The full method achieves the highest final student ImageReward and maintains a clear advantage through most of training. Shuffling textual reflection reduces performance even though image context remains available: at step 200, student ImageReward falls from 1.513 to 1.391. This supports the contribution of sample-specific textual feedback beyond image conditioning. Shuffling images also reduces performance, highlighting the importance of correctly paired visual reflection. Textual reflection without image context supports stable training but gives lower performance, while the image-only variant improves initially and then collapses, accompanied by substantially larger gradient norms that indicate optimization instability.

These results support complementary contributions from visual context and textual reflection. We have made this distinction explicit in Sec. 3.1: the generated image supplies visual context, and the textual reflection supplies feedback on discrepancies between the image and the prompt. We therefore attribute the full method's gains to their joint use, rather than to textual reflection in isolation.

In the revised manuscript:

  • Sec. 3.1, p. 4: Distinguished visual context from textual reflection in teacher conditioning.
  • Sec. 4.1, pp. 7–8, and Table 2, p. 16: Corrected the training and SRR evaluation CFG descriptions.
  • Sec. 4.4 and Fig. 7(a), p. 10: Added the four conditioning ablations and their analysis.

W2 / Q1. Co-drift versus diminishing corrective benefit. A declining teacher win rate measures relative advantage, not reflection accuracy. What evidence distinguishes deteriorating feedback from a stronger student that benefits less from correction? Do the anchoring results establish that unreliable feedback causes instability?

Thank you for highlighting this distinction. To examine whether the teacher itself becomes less effective or simply loses its advantage as the student improves, we evaluate different teacher checkpoints on two fixed sets of prompt–image pairs generated by the initial and step-200 students. Within each set, all teacher checkpoints use the same input images, correction settings, and correction seeds. The table below summarizes the mean ImageReward gains, while Sec. 4.5 and Fig. 7(b) provide a more complete visualization and analysis, including absolute corrected-image scores and 95% confidence intervals for the gains.

Table: Mean ImageReward gain over the fixed student inputs.

Fixed student inputs / Teacher step020406080100120140160180200
Student 0+0.1372+0.1637+0.1222+0.1088+0.1406+0.2043+0.1923+0.2072+0.1053+0.1227+0.1448
Student 200−0.1323−0.1858−0.2328−0.2292−0.1847−0.1473−0.1257−0.0851−0.1356−0.1417−0.1929

Columns indicate teacher checkpoint steps. Student 0 and Student 200 denote fixed input images generated by the initial and step-200 students, with mean input ImageReward scores of approximately 1.078 and 1.523, respectively. Gains are paired differences between corrected and original input-image scores.

Across all 11 teacher checkpoints, correction yields positive mean gains on initial student inputs but negative mean gains on step-200 inputs. Meanwhile, the absolute corrected-image scores do not show a consistent decline across teacher checkpoints on either fixed set. Together, these observations suggest that throughout training, the teacher remains helpful for the initial student, yet its corrections no longer benefit the stronger student and can instead lower ImageReward.

Furthermore, we have revised the co-drift discussion to clarify the risk of continuing to distill from teacher corrections after they lose their benefit for the current student. Since student-generated images also determine the teacher's conditioning context, updates toward ineffective targets can alter subsequent supervision and create a harmful feedback loop. The fixed-input results show that corrections can become detrimental to the stronger student, while the collapse under weak reference anchoring in Fig. 6(a) supports the motivation for limiting reliance on these targets. We apologize that our original wording did not clearly distinguish the accuracy of reflections from the effectiveness of the teacher's corrections. We have clarified this distinction in the introduction, method discussion, and conclusion, presenting co-drift as a potential failure mechanism supported by these observations.

In the revised manuscript:

  • Sec. 4.5 and Fig. 7(b), p. 10: Added fixed-input absolute scores and paired correction gains.
  • Sec. 3.3, pp. 5–6: Reframed co-drift as a potential feedback-driven failure mechanism.
  • Sec. 4.4 and Fig. 6(a), p. 9: Related reference anchoring to the observed training dynamics.
  • Appendix E.2, pp. 18–19: Clarified that teacher win rate measures relative corrective advantage.
  • Introduction, p. 2, and Conclusion, p. 10: Aligned the claims with the revised interpretation.

Dear Reviewer,

Thank you again for your thoughtful review and constructive feedback. As the discussion period is nearing its end, we would greatly appreciate your feedback on whether our responses and the revisions to the manuscript address your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.