acceptodds
Withdrawn from ICLR 2027

DuetGRPO: Co-Adaptive Joint Reinforcement Learning for Reasoning-Driven Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint training allows the language and visual policies in reasoning-driven image generation to co-adapt prompt formulation and image synthesis. However, existing joint training faces a central challenge: relying solely on image rewards induces reward mismatch, as rendering quality is an unreliable proxy for reasoning and prompt correctness. In this paper, we propose DuetGRPO a joint reinforcement learning framework that combines modality-specific optimization with shared alignment feedback to support online co-adaptation of language and visual policies. To supervise language outputs directly, we use a text-only evaluator to assess reasoning correctness and check generation prompts against a candidate-independent checklist of instruction requirements. We route this text reward exclusively to the language policy and share image–text alignment feedback across both policies, aligning their joint updates with distinct objectives: the language policy learns to reason correctly and formulate prompts that satisfy the user instruction and can be faithfully rendered, while the visual policy learns to follow those prompts. Experiments improve WISE Overall from 0.72 to 0.77 and T2I-CoReBench Overall from 59.1 to 60.7. Improvements are also observed when this recipe transfers from modular MLLM + diffusion transformer (DiT) pipelines to a mixture-of-transformers (MoT) architecture.

Primary area:
generative models
Keywords:
Reinforcement Learning, Unified Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hAuFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DUETGRPO, which supplies direct rewards for reasoning correctness and generation-prompt fidelity. This textual supervision is combined with shared image–text alignment feedback to enable online co-adaptation of the language and visual policies. Relative gains from the GRPO stage are reported on WISE (+0.05) and T2I-CoReBench (+1.6).

Critical strengths
  1. The core motivation of directly checking both the reasoning trace and the generation prompt against the original user instruction is reasonable and addresses a gap in existing think-then-generate pipelines.
  2. The joint RL recipe that integrates composite textual supervision with shared image–text alignment feedback is supported by carefully designed ablation studies, providing empirical verification of the proposed combination.
Critical weaknesses
  1. The reasoning reward, which constitutes the primary claimed motivation, can be viewed as a straightforward instantiation of multi-reward or process-reward ideas. This raises concerns about the degree of novelty and the extent of new inspiration offered to the community.
  2. The additional data construction and training cost do not yield state-of-the-art results, nor do they produce substantially larger relative (SFT -> RL) improvements than those observed for other baseline methods. This raises questions about the efficiency and necessity of the reasoning-reward component.
  3. Several introduced modules (e.g., checklist construction for the text-only evaluator and the selection of coefficients for combining textual rewards) rely on manually designed components. Such design choices may introduce instability when assessing the effectiveness and robustness of the overall approach.
Questions for authors
  1. Could the authors provide deeper discussion of the experimental results beyond numerical values? For example, Table 2 shows that DUETGRPO behaves quite differently across categories. Why does this variation occur, and what guidance does it offer regarding the conditions under which reasoning rewards are most effective?
  2. Please clarify the key differences between DUETGRPO and Dual-GRPO. Both methods follow the think-then-generate paradigm. From the presented material, the reward for reasoning correctness appears to be the principal addition; clarification on whether this constitutes a more substantial technical distinction would be helpful.
7 comments

Weakness 1 & Question 2: The reasoning reward, which constitutes the primary ...

We thank the reviewer for the opportunity to clarify the distinction. DuetGRPO differs from the original Dual-GRPO in the supervision target and the definition and assignment of rewards.

1. Direct supervision of reasoning and prompt correctness. In the original Dual-GRPO, the language policy is rewarded by averaging the semantic consistency scores between generated images and the original user instruction; the reasoning and refined prompt are not directly checked for factual correctness. (Dual-GRPO, Sec. 3.3) DuetGRPO instead directly evaluates reasoning correctness, generation-prompt coverage, and prompt correctness with a text-only evaluator, using a candidate-independent checklist for prompt evaluation. This supplies correctness feedback on language outputs independently of rendering quality.

2. Separate checks of instruction satisfaction and rendering fidelity. Dual-GRPO uses image-based semantic feedback for the language policy and a combination of aesthetic, physical-consistency, and semantic scores for the DiT. DuetGRPO checks language outputs against the user instruction and measures image alignment against the generated prompt. The language policy receives Rtext+R‾alignR^{\mathrm{text}}+\overline{R}^{\mathrm{align}}, while the visual policy receives only RalignR^{\mathrm{align}}. This explicitly rewards both a correct description of what should be generated and faithful rendering of that description.

The matched ablations support the value of these complementary signals: removing either the text reward or the language policy's alignment feedback reduces WISE Overall from 0.77 to 0.75 (Table 4 of the manuscript). For a stricter baseline comparison, the revised Dual-GRPO baseline in Table R2-2 of our response to Reviewer 2 additionally receives our text reward and uses generated prompts for CLIPScore; the distinctions above concern the published Dual-GRPO objective.

Regarding the gradient-routing question, with the full rewards retained, detached conditioning achieves 0.77, compared with 0.75 for coupled gradients. Appendix F analyzes early, archived runs in which visual-loss gradients introduced additional language updates; these unrestricted extra updates were constrained before the experiments in Table 4 of the manuscript. The diagnostics identify a potential source of optimization imbalance, while the ablation in Table 4 of the manuscript evaluates the effect of conditioning-gradient routing under the revised schedule.

Weakness 2: The additional data construction and training ...

We thank the reviewer for raising the question of whether the additional supervision justifies its cost. All experiments in Table 4 of the manuscript use the same training data and SFT initialization, so the data-construction cost is shared across these comparisons. Regarding final performance, DuetGRPO achieves the highest Overall scores among the methods compared on both WISE (0.77) and T2I-CoReBench (60.7). While the starting backbone leaves room for improvement in some subcategories, these overall results support the effectiveness of our training recipe.

To further examine the contribution of the text reward and its combination with joint optimization, we replace text reward with VLM-based language reward . All scores below are WISE Overall:

VLM-based reward with a frozen DiT: We train the autoregressive (AR) policy using UnifiedReward as the sole reward while keep clip score training DiT. This configuration yields 0.72 → 0.74, whereas text reward+clip score yields 0.72 → 0.77 (Table R3-1). This comparison supports the benefit of directly evaluating reasoning correctness and generation-prompt fidelity using text alone, independently of rendered-image quality, over the tested VLM-based feedback.

Table R3-1. Comparison of UnifiedReward as a language reward.

Training variantAR training rewardCulturalTimeSpaceBiologyPhysicsChemistryOverall
DuetGRPO (ours)Text reward + CLIPScore0.810.820.800.620.720.740.77
UnifiedReward baselineUnifiedReward0.800.760.710.650.740.710.74

The matched ablation in Table 4 of the manuscript isolates the text reward's contribution during joint optimization: removing it reduces Overall from 0.77 to 0.75, with the remaining configuration unchanged. Together, these results show a measurable benefit from direct text supervision and further gains from combining it with online joint optimization. Our text reward specifically supervises reasoning and prompt correctness, while a separate image–text alignment reward supports coordination with the renderer. The shared training data and these controlled comparisons provide evidence for the reward's empirical value.

Weakness 3: Several introduced modules ...

We thank the reviewer for highlighting the need to validate the scoring rules and reward coefficients. Each checklist is generated by a fixed evaluator from the instruction and its reference explanation, cached, and shared across candidates, allowing semantically equivalent realizations. Manual review of 6,000 CoT–PE pairs on held-out WISE instructions shows agreement with human judgments: checklist coverage achieves Pearson correlations of 0.82/0.77 and ICCs of 0.81/0.79 for DuetGRPO/SFT, respectively, while prompt and reasoning correctness achieve κ = 0.80–0.86 (Table R2-4 in our response to Reviewer 2).

The coefficient sweep in Table R3-2 further shows that WISE Overall remains within 0.73–0.74 across the tested nonzero coefficients, with multiple settings reaching 0.74. Setting the coefficient of reasoning correctness (rir_i), prompt coverage (pip_i), or prompt correctness (hih_i) to zero yields 0.73, 0.72, and 0.72, respectively. These results support retaining all three components and indicate limited sensitivity of aggregate performance to the tested coefficient values.

Table R3-2. Ablation of text-reward coefficients in DuetGRPO (ours). Bold values indicate the best result within each coefficient sweep, including ties.

CoefficientValueCulturalTimeSpaceBiologyPhysicsChemistryOverall
rir_i00.790.790.730.630.600.660.73
rir_i1/50.800.790.730.640.700.670.74
rir_i2/50.800.800.740.640.690.670.74
rir_i3/50.790.780.720.630.710.730.74
pip_i00.790.770.760.680.630.600.72
pip_i1/50.790.790.720.630.670.650.73
pip_i2/50.800.800.740.640.690.670.74
pip_i3/50.800.790.720.640.700.670.74
hih_i00.780.790.730.630.570.640.72
hih_i1/50.790.810.720.660.650.650.73
hih_i2/50.790.790.730.630.670.660.73
hih_i1/20.800.800.740.640.700.670.74
hih_i3/50.790.810.770.620.680.700.74
Reviewer_LBZAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DuetGRPO addresses reward mismatch in reasoning-driven image generation: image quality does not reliably establish whether preceding reasoning is correct. A fixed text evaluator scores reasoning and generation-prompt fidelity; the language policy receives this reward plus image–prompt alignment, while the visual policy receives alignment alone. Both policies update online with detached conditioning. On ThinkGen, WISE improves from 0.72 to 0.77 and T2I-CoReBench from 59.1 to 60.7; ablations and a BAGEL experiment provide additional validation.

Critical strengths
  1. Separating instruction-level textual correctness from rendering fidelity addresses a real weakness of image-only feedback. Sharing alignment rewards retains feedback about what the evolving renderer can realize.
  2. Table 4 reports three-seed reward, freezing, and gradient-routing controls: the full method reaches 0.77±0.0063 versus approximately 0.75 for either language-reward ablation. Gains on T2I-CoReBench and BAGEL provide additional evidence beyond the primary setup.
Critical weaknesses
  1. The incremental contribution is insufficiently established against close precedents. GoT-R1 (ICLR 2026) already directly evaluates reasoning and combines it with image-consistency rewards; PromptRL (ICML 2026) already jointly trains policies with shared rewards and separate gradients. The particular checklist and reward assignment remain distinct, but a GoT-R1-inspired reward control is missing. AlphaGRPO (ICML 2026), a relevant BAGEL-based joint RL method with decompositional verification, is also neither discussed nor compared.
  2. The adapted baselines do not convincingly establish comparative superiority. UniGRPO/Dual-GRPO use CLIP against the original, potentially implicit instruction, while DuetGRPO additionally receives Gemini factual supervision and aligns images to explicit generation prompts (§4.1). This disclosed comparison does not isolate the benefit of the proposed assignment from the stronger supervision. A competitive VLM image-reward baseline and closer text-plus-image reward controls would materially strengthen the contribution claim.
Questions for authors
  1. Comparative evidence: Does the advantage persist against a GoT-R1-inspired text-plus-image reward control and AlphaGRPO on BAGEL, with matched initialization/data/budget and competitive, available reward evaluators?
  2. Core claim: How well do the text scores agree with independent expert judgments, and does factual reasoning accuracy improve from SFT to DuetGRPO on held-out instructions, separately from final-image quality?
  3. Clarification: In Table 4, are language optimizer-step counts, KL applications, and loss scales matched between coupled and detached training? Appendix F reports many extra visual-driven language updates in separate archived runs; please clarify whether that scheduling difference also affects Table 4.
5 comments

Weakness 1 & Question 1: The incremental contribution is insufficiently ...

We thank the reviewer. DuetGRPO differs from AlphaGRPO in the outputs it verifies and the rewards assigned to each policy. AlphaGRPO's DVReward decomposes user requests into atomic questions about image semantics and perceptual quality, which an MLLM answers from the generated image. Its joint GRPO objective uses the resulting image-based advantage to update both the AR and diffusion policies. (AlphaGRPO, Secs. 4.1–4.2)

DuetGRPO directly supervises the intermediate language outputs. A text-only evaluator checks reasoning correctness, generation-prompt coverage, and prompt correctness, using a candidate-independent checklist of instruction requirements for prompt evaluation. This text reward is assigned exclusively to the language policy, which also receives image–text alignment feedback; the visual policy receives only alignment feedback, computed between each image and its generated prompt. Thus, the language policy receives explicit correctness supervision independent of rendering quality, while shared alignment feedback supports online co-adaptation.

We compared DuetGRPO with AlphaGRPO on BAGEL using matched training data and training volume. DuetGRPO achieves higher WISE Overall (0.66 vs. 0.64), with its largest advantages in Space (+0.08) and Chemistry (+0.07). It also improves Time (+0.03) and Cultural (+0.01), matches Biology at 0.50, and is 0.02 lower in Physics (Table R2-1). The gains across four of the six categories provide empirical support for our training recipe, which combines direct supervision of reasoning and prompt correctness with shared image–text alignment feedback.

Both methods are evaluated using the updated WISE protocol used throughout our manuscript (Qwen3.5-35B-A3B), which removes the separate aesthetic component of legacy WiScore; scores obtained under different evaluation protocols are therefore not directly comparable. (WISE protocol)

We don't reproduce GoT-R1 as GoT-R1's reward implementation is not publicly released. (Official repository)

Table R2-1. AlphaGRPO vs DuetGRPO (ours) on BAGEL.

Training StageCulturalTimeSpaceBiologyPhysicsChemistryOverall
SFT0.790.540.510.470.570.630.64
DuetGRPO (ours)0.780.600.570.500.610.660.66
AlphaGRPO0.770.570.490.500.630.590.64

Weakness 2: The adapted baselines do not ...

We thank the reviewer for highlighting the reward configurations in the baseline comparisons. We have standardized the reward setup across UniGRPO, Dual-GRPO, and DuetGRPO: CLIPScore is computed against each method’s generated prompts, and both baselines additionally receive the text reward. The manuscript has been updated accordingly. WISE Overall increases from 0.74 to 0.75 for Dual-GRPO and remains 0.73 for UniGRPO at the reported precision (Table R2-2). UniGRPO improves in other categories, but a decline in Cultural, which carries 40% of the overall weight, offsets these gains.

Table R2-2. Revised comparison of UniGRPO, Dual-GRPO, and DuetGRPO (ours).

Training variantLanguage rewardVisual rewardGradientsInitialOverallΔ
UniGRPORtext+CLIPScoreR^{\mathrm{text}}+\mathrm{CLIPScore}CLIPScorecoupled0.720.73+0.01
Dual-GRPORtext+CLIPScoreR^{\mathrm{text}}+\mathrm{CLIPScore}CLIPScoredetached0.720.75+0.03
DuetGRPO (ours)Rtext+CLIPScore‾R^{\mathrm{text}}+\overline{\mathrm{CLIPScore}}CLIPScoredetached0.720.77+0.05

We also added a VLM image-reward baseline using UnifiedReward in place of CLIPScore as language reward during training, as shown in Table R2-3. It improves WISE Overall from 0.72 to 0.74, compared with 0.77 for DuetGRPO. In this configuration, the judge assesses image–prompt consistency without explicitly supervising the factual validity of the preceding reasoning, and its feedback may reflect evaluator(VLM used by UnifiedReward) biases. DuetGRPO additionally supervises reasoning correctness and prompt fidelity directly. These baseline comparisons assess the complete training recipe; our matched ablations separately examine the contributions of text supervision, shared alignment feedback, and gradient routing.

Table R2-3. Comparison of UnifiedReward as a language reward.

Training variantAR training rewardCulturalTimeSpaceBiologyPhysicsChemistryOverall
DuetGRPO (ours)Text reward + CLIPScore0.810.820.800.620.720.740.77
UnifiedReward baselineUnifiedReward0.800.760.710.650.740.710.74

Question 2: How well do the text scores ...

We thank the reviewer for this suggestion. We evaluated the textual reasoning and generation prompts directly on held-out instructions, separately from final-image quality. We sampled three CoT–prompt enhancement (PE) pairs per instruction from each of the SFT and DuetGRPO checkpoints on 1,000 WISE instructions held out from training, yielding 6,000 pairs in total. All pairs were manually reviewed for their textual content, and the human annotations served as the reference for evaluating the text scorer.

Regarding agreement with human judgments, Table R2-4 reports scorer–human agreement accuracies of 93%/91% for prompt correctness and 91%/88% for reasoning correctness for DuetGRPO/SFT, respectively. The corresponding κ values are 0.86/0.81 and 0.86/0.80. Checklist coverage achieves Pearson correlations of 0.82/0.77 and ICCs of 0.81/0.79. These statistics validate the text scorer against human judgments independently of image-level evaluation.

Table R2-4. Agreement between the text scorer and human annotations on held-out WISE outputs. Bold values indicate the best result for each criterion across checkpoints, including ties.

CheckpointCriterionPearsonκICCAgreement accuracy
DuetGRPO (ours)Prompt correctness0.880.86-0.93
DuetGRPO (ours)Reasoning correctness0.860.86-0.91
DuetGRPO (ours)Checklist coverage0.82-0.810.89
SFTPrompt correctness0.810.81-0.91
SFTReasoning correctness0.790.80-0.88
SFTChecklist coverage0.77-0.790.86
Reviewer_j89YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the inability of image rewards to directly assess language-output correctness in reasoning-driven image generation. It proposes DuetGRPO, a joint reinforcement learning framework that evaluates reasoning and generation prompts with a text-only evaluator. Text rewards supervise the language policy, while image–text alignment feedback is shared across language and visual policies. Both policies update from fresh trajectories, with conditioning gradients from the visual loss into the language policy detached. Experiments report improvements over SFT initialization on WISE and T2I-CoReBench, supported by internal ablations and application to BAGEL.

Critical strengths

The internal ablations are comprehensive. This work examines removing text rewards or alignment feedback from the language policy, freezing either policy, and changing gradient coupling. These ablations help identify the contributions of individual design choices. In addition, the main internal controls report results across three training seeds, strengthening the experimental credibility.

Critical weaknesses

1. The reliability of the text reward lacks direct validation. The text reward relies primarily on Gemini’s judgments, without directly demonstrating that it accurately identifies substantive errors while accepting diverse valid solutions. Since it's a central contribution, the authors need to demonstrate its reliability to substantiate the interpretation that reasoning and prompt correctness improve.

2. The independent value of online co-adaptation is not isolated. Freezing either policy in Table 4 also reduces the trainable components, leaving unclear whether the gains arise from optimizing both modules or from their continual adaptation to each other. Offline prompt optimization, staged training, or alternating updates could also improve both policies. The authors need to show what additional benefit online co-adaptation provides.

Questions for authors

See Weaknesses.

3 comments

We sincerely thank the reviewer for their valuable feedback. We appreciate the chance to clarify the raised important questions regarding the reliability of the text reward and value of online co-adaptation.

Weakness 1: The reliability of the text reward ...

Thank you for raising this concern. Our text reward is based on human-validated checklists and predefined scoring rules. Gemini assists with checklist construction and performs automated scoring. Each checklist is derived from the instruction and its reference explanation, manually validated, and shared across candidates. We will make this human-validation step explicit in the manuscript.

To demonstrate that the proposed text reward can accurately identify substantive errors while accepting various effective solutions, we use the weights of SFT and RL for inference on WISE which are hold-out instructions independent of training data, with a rollout set to 3, to obtain a total of 6000 COTs and corresponding PEs, which are then manually reviewed. Based on the manual review results as GT, calculate the accuracy, Kappa coefficient, Pearson coefficient and ICC (for continuous values) of PE correction, Reasoning correction, and checklist coverage, as reported in Table 1. The reuslts highlight the alignment with human preferences of our proposed text reward.

To examine dependence on Gemini, we additionally trained a model using Qwen3.5-35B-A3B as the text-reward scorer, retaining the same checklists and scoring rules. The resulting checkpoint achieves 0.75 WISE Overall, compared with 0.77 using Gemini (detach). The detailed evaluation results are shown in Table 2. Experimental results show that our proposed text reward is of good robustness.

Table 1. Agreement between the text scorer and human annotations on held-out WISE outputs. Bold values indicate the best result for each criterion across checkpoints, including ties.

CheckpointCriterionPearsonκICCAgreement accuracy
DuetGRPO (ours)Prompt correctness0.880.86-0.93
DuetGRPO (ours)Reasoning correctness0.860.86-0.91
DuetGRPO (ours)Checklist coverage0.82-0.810.89
SFTPrompt correctness0.810.81-0.91
SFTReasoning correctness0.790.80-0.88
SFTChecklist coverage0.77-0.790.86

Table 2. Comparison of text-reward scorers for DuetGRPO (ours).

Training StageCulturalTimeSpaceBiologyPhysicsChemistryOverall
DuetGRPO (Qwen) (ours)0.800.750.800.610.680.780.76
DuetGRPO (ours)0.810.820.800.620.720.740.77

Weakness 2: The independent value of online ...

To distinguish the benefit of online co-adaptation from that of training both modules, we added two complementary controls alongside the freezing ablations in Table 4:

Staged training: Both policies are optimized in separate stages. This baseline underperforms DuetGRPO despite training both modules as shown in table 1, supporting the benefit of continual mutual adaptation over the tested staged schedule.

Offline prompt optimization: We use identical Gemini-optimized text for each WISE instruction with the SFT and DuetGRPO checkpoints. Each checkpoint processes this fixed text through its MLLM to obtain hidden states that condition its corresponding DiT. DuetGRPO still achieves a higher score. Together, these controls support the added value of online co-adaptation in strengthening coordination between the language and visual policies as shown in table 2, with gains over staged training and benefits that persist even under identical offline-optimized prompts.

Table 1. WISE evaluation of staged training and DuetGRPO (ours).

Training StageCulturalTimeSpaceBiologyPhysicsChemistryOverall
Staged Training0.800.750.800.610.680.780.75
DuetGRPO (ours)0.810.820.800.620.720.740.77

Table 2. WISE evaluation comparasion in different training stage employing the same offline optimized prompts.

Training StageCulturalTimeSpaceBiologyPhysicsChemistryOverall
SFT0.830.790.730.590.660.670.75
DuetGRPO (ours)0.870.810.840.760.890.840.85
Staged Training0.840.820.780.660.760.710.78

Dear Reviewer j89Y,

Thank you again for your thoughtful review and constructive questions about the reliability of the text reward and the value of online co-adaptation.

We would like to briefly follow up on our submitted rebuttal and highlight the additional evidence relevant to these concerns:

  • Text-reward reliability: We manually evaluated 6,000 reasoning–prompt pairs from 1,000 held-out WISE instructions. The text scorer achieves agreement accuracies of 88%–93% with human annotations for reasoning and prompt correctness. We also examined dependence on Gemini by training with an open-source Qwen text scorer while retaining the same checklists and scoring rules.
  • Value of online co-adaptation: DuetGRPO achieves 0.77 WISE Overall, compared with 0.75 for staged training that also optimizes both policies. In the separate experiment using identical offline-optimized prompts, DuetGRPO achieves 0.85, compared with 0.78 for staged training and 0.75 for SFT. These controls provide additional evidence for the benefits of continued co-adaptation in the evaluated settings.

When you have an opportunity, we would be grateful if you could review these additions and let us know whether any concerns remain. We would be happy to provide further clarification. If this evidence strengthens your assessment of the paper, we would also sincerely appreciate your reconsideration of the score.

Thank you again for your time and careful consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reading and insightful, constructive comments. We provide point-by-point responses below and have revised the manuscript accordingly, with the corresponding changes highlighted in red for ease of review. We hope that our responses and revisions address the reviewers' concerns and clarify our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.