acceptodds
Withdrawn from ICLR 2027

Adaptive Self-Correction for Multi-Turn Interleaved Text-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-correction in interleaved text-image generation requires deciding when to refine an image, fall back to a new attempt, or stop, based on the evolving generation history. We introduce *Adaptive Self-Correction Policy Optimization* (ASPO), an on-policy reinforcement learning framework that learns a variable-length policy over reasoning, correction actions, and image synthesis. Cold-start supervision establishes the correction behaviors; reinforcement learning then jointly optimizes the language-action and image-generation branches using only final-image rewards. Each intermediate image provides context for subsequent decisions. For the rectified-flow image branch, ASPO replays one cached stochastic denoising transition per generated image, yielding a tractable transition-level policy surrogate for optimization from the final trajectory outcome. We also introduce *CorrectionBench*, comprising 350 instances that evaluate recovery from flawed interleaved contexts. ASPO improves overall generation performance on GENIUS and GenEval++ over supervised initialization. On CorrectionBench, it achieves gains of 7.9 points in automated evaluation and 7.53 points in blinded human evaluation over the same baseline. Behavioral analyses and component ablations further support the learned correction behavior and the contribution of image-policy optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
interleaved text-image generation, adaptive self-correction, multimodal generative models, reinforcement learning, policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KfWkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ASPO an adaptive self-correction policy for interleaved text-image generation. It allows a multimodal model to choose between refining a generation, a re-generation fallback, and termination. A new CorrectionBench is introduced to evaluate this task. Against multiple image generation baselines it receives superior quality scores.

Critical strengths

1.) The overall idea of introducing an adaptive self-correction policy appears a useful contribution. Formulating the solution as a learning problem across "refinement", "re-generation fallback", and "termination" also makes sense and is solid.

2.) Introduces CorrectionBench to evaluate the specified task.

Critical weaknesses

1.) I have various theoretical concerns with how the paper learned a tractable policy term for each image-emitting decision. Essentially, ASPO wishes to maximize the expected reward E[R(x_final)], via a surrogate loss L_IM. Is this surrogate aligned with the desired objective? Additionally, by adding noise to an intermediate latent subsequent velocity predictions may receive inputs that differ from what was seen during pre-training. So, what prevents RL fine-tuning from damaging the learned velocity field? While it makes intuitive sense to some degree, a more formal theoretical analysis would provide greater clarity.

2.) ASPO naturally uses more compute across multiple fallbacks/refinements, is it fair to compare it against single-turn image generation models? How would ASPO fare against naive Best-of-N generation with similar compute? The key point is, are improvements over baselines simply due to more compute?

3.) CorrectionBench gives a flawed context asking ASPO to correct it. However, given an image that is correct, it does not test if ASPO can choose “terminal” rather than unnecessarily refining/regenerating the image.

Questions for authors

Please see Critical Weaknesses 1-3, which contain my main questions

Reviewer_932MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the self-correction problem in multi-turn interleaved text–image generation with unified multimodal models and proposes the ASPO method. The method formulates generation as a history-conditioned variable-length policy that supports local refinement, regeneration, or termination. It adopts a two-stage training procedure: first, cold-start supervised fine-tuning (SFT) on 28,842 constructed interleaved training trajectories; then, on-policy reinforcement learning based on the final-image reward to jointly optimize the language-action policy and the image-generation branch. To address the difficulty of obtaining log probabilities from Rectified Flow, the paper proposes a stochastic transition replay technique to construct a policy surrogate. The authors also construct CorrectionBench, a benchmark containing 350 instances. Experiments show that ASPO improves the automatic and human evaluation scores on CorrectionBench by 7.9 and 7.53 points, respectively, and also improves the overall scores on GENIUS and GenEval++.

Critical strengths
  1. The problem studied in this paper is of practical value. It unifies local refinement, fallback (regeneration), and the termination in multi-turn interleaved text–image generation into a history-conditioned variable-length policy. Compared with fixed, pre-defined generation pipelines, this method allows the model to autonomously select the next operation based on the current image and the preceding generation history, tightly coupling subsequent image generation with the timing of termination.
  2. The proposed training framework that jointly optimizes the language-action policy and the image-generation policy is technically innovative. The paper not only uses reinforcement learning to optimize the model’s correction-action selection, but also attempts to propagate the reward signal from the final image to the branch that actually performs image generation. The proposed stochastic transition replay provides a feasible solution to the problem that the Rectified Flow sampling process cannot directly provide the log probabilities required by standard policy gradients, and the ablation studies in the paper also support this design.
  3. The experimental results provide preliminary support for the effectiveness of the method. The paper evaluates overall image generation capability on GENIUS and GenEval++, and constructs CorrectionBench, which contains 350 instances. Compared with the SFT baseline, ASPO improves the automatic evaluation score on CorrectionBench by 7.9 points and the human evaluation score by 7.53 points. In addition, the paper conducts relatively comprehensive ablation studies that demonstrate the effectiveness of the current design
Critical weaknesses
  1. The overall score on GENIUS masks regressions on some tasks, and ASPO’s gains are inconsistent across different tasks and across the RC, VC, and AQ metrics. For Implicit Pattern, RC/VC/AQ changes from 68.02/43.84/42.44 to 70.35/36.30/55.81; for Symbolic Constraint, it changes from 51.96/32.06/39.87 to 55.23/27.10/41.18, among others. Multiple VC metrics decline or show no improvement, and the AQ score on Prior-Conflicting also drops by 6.50 points. The same issue also exists on GenEval++. The paper needs to further explain whether these regressions are related to the reinforcement learning objective, the reward signal, or the training process, and analyze why some capabilities improve while others deteriorate.
  2. The reinforcement learning reward model Qwen3.6-27B receives only the original prompt and the final image, but some tasks in CorrectionBench rely on previous images, preferences, or contextual relationships. The same trajectory-level advantage is applied to the text, action, and image-generation steps across the entire trajectory, making it difficult to distinguish which steps actually improve the final result. The reward ablation also shows that per-round rewards increase GenEval++ from 0.429 to 0.436 but decrease CorrectionBench from 71.55 to 70.94, indicating that the reward design involves a trade-off across tasks.
  3. ASPO constructs stochastic denoising transitions and uses their log probabilities to construct a policy optimization surrogate for image generation. However, this surrogate is not the probability of the complete denoising path or the marginal likelihood of the final image. Meanwhile, the paper does not adequately analyze the impact of factors such as the selected denoising timestep, the number of stochastic transitions, and noise settings on training effectiveness, nor does it sufficiently demonstrate what independent advantage this surrogate has over existing reinforcement learning methods for flow matching or diffusion models.
  4. On CorrectionBench, the paper mainly compares with the base model and different SFT variants, lacking direct comparisons with related methods such as R3 and UniGRPO on the same correction benchmark.
Questions for authors
  1. Although ASPO improves the overall scores on GENIUS and GenEval++, some tasks show clear regressions, such as declines in multiple VC metrics on GENIUS. Could the authors explain the reasons for these regressions and clarify whether the results are stable?
  2. Qwen3.6-27B receives only the original prompt and the final image, but CorrectionBench contains 230 multi-image context tasks. How do the authors ensure that the reward model can accurately evaluate results that depend on historical context? In addition, what evidence shows that the trajectory-level reward can effectively guide decision-making and image generation at each round?
  3. ASPO uses a single stochastic denoising transition to construct an image-policy optimization surrogate. Could the authors explain the rationale for selecting the denoising timestep and noise settings, and explain how the existing experiments verify the reliability of this surrogate and its independent contribution relative to existing reinforcement learning methods for image generation?
  4. The paper does not conduct direct comparisons with related methods such as R3 and UniGRPO on CorrectionBench. Could the authors add such experiments?
Reviewer_EmFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-turn self-correction in interleaved text-image generation. The proposed ASPO framework jointly optimizes decisions to refine, fall back, or terminate, together with image generation. Starting from supervised fine-tuning on 28,842 synthetic trajectories, it applies a final-image reward and a transition-level surrogate for the image policy. The paper also introduces CorrectionBench and reports improvements on generation and correction benchmarks. In particular, the human evaluation supports a substantial improvement over SFT. The main remaining question is whether these gains establish the value of adaptive correction decisions beyond additional generation attempts and joint model optimization.

Critical strengths
  1. A meaningful formulation of adaptive correction. Section 3.1 jointly models refinement, fallback, and termination with image generation, rather than prescribing a fixed multi-turn schedule. This targets an important limitation of current interleaved generators and offers a useful framework for studying correction decisions.

  2. The main improvement has credible human support. Appendix B.4 evaluates all 350 CorrectionBench instances using three evaluators who did not participate in construction or training. The 7.53-point improvement over SFT, with a 95% interval of [6.21, 8.89], supports the claimed improvement beyond the model-based judge. The action and image-policy ablations further strengthen the positive evidence for the overall system.

Critical weaknesses
  1. The experiments do not isolate the benefit of adaptive decisions at a matched inference budget. Increased refine/fallback usage in Table 4 shows a behavioral change, but does not establish that the policy chooses better actions than fixed refinement, regeneration, or best-of-N using the same backbone and number of image generations. This matters because refine and fallback use the same generator and full history, distinguished by an action token. Without these internal controls, the improvement could partly reflect additional generation attempts and model optimization rather than useful state-dependent decision making.

  2. The correction evaluation does not directly measure preservation of previously correct content. In Appendix B.3, the judge receives a Hint describing the desired image and the final output, but not the original erroneous image or full history; the human evaluation follows the same FP/AQ rubric. These results support final-image correctness, but may also reward regeneration that changes previously correct regions, identities, or historical constraints. Since local repair and history-aware correction are central motivations, this evaluation gap materially limits the interpretation of the otherwise encouraging results.

Questions for authors
  1. Targeted validation: At the same number of image generations, how does ASPO compare with fixed refinement, regeneration, and best-of-N on the same SFT backbone? What is the distribution of actual generation counts, and what happens when refine/fallback are merged into one correction action?
  2. Evaluation clarification: How many instances belong to each CorrectionBench action subset, and how are their scores aggregated? Does the Hint specify content that must be preserved? A focused original-versus-corrected-image assessment on the refinement subset would help determine whether the gains reflect successful repair rather than replacement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.