acceptodds
Withdrawn from ICLR 2027

InterGRPO: Global-Local Credit Assignment for Multi-Unit Interleaved Text–Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-unit interleaved text-image generation aims to produce coherent sequences of alternating textual and visual content, such as visual storytelling generation, requiring semantic and visual consistency across units as well as coherence of the complete sequence. Yet models trained primarily through supervised fine-tuning have limited ability to improve these qualities from generation feedback. Although recent reinforcement learning methods offer a potential remedy, existing methods primarily optimize a single image or an atomic text-image unit. Extending them to multi-unit outputs poses two challenges: a trajectory-level reward cannot identify which visual units should be reinforced, and uncertain quality judgments can provide unreliable optimization signals.To address this challenge, we introduce InterGRPO, an online group-based policy optimization framework tailored to multi-unit interleaved text-image generation. InterGRPO employs global-local credit assignment to optimize individual visual units while preserving sequence-level coherence. However, scoring hallucinations is widespread in general vision-language models for image quality evaluation, leading to inaccurate quality scores. To address this issue, we further develop an Uncertainty-Calibrated Reward (UC-Reward) model, which distills calibrated score distributions from multiple task-specific teachers. Experiments on SIL-Bench and ISG-Bench show that InterGRPO outperforms the strongest baseline by 2.5% and 4.5%, respectively, while improving cross-unit consistency, logical coherence, and overall generation quality.

Primary area:
generative models
Keywords:
Text-Image Interleaved Generation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rKVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes InterGRPO, an RL framework for improving models that generate sequences of alternating text and images. It argues that each generated image should receive feedback based on both its individual quality and the coherence of the entire sequence. It also introduces UC-Reward, a reward model designed to reduce unreliable quality assessments from vision-language models. The results show that InterGRPO improves performance over the ILLUME-X baseline on some benchmarks.

Critical strengths

The paper is generally well-written and clearly organized, making the motivation and proposed methodology easy to follow. The introduction of UC-Reward is also an interesting aspect of the work. In particular, the use of uncertainty-aware score distribution distillation and repeated evaluation of ambiguous samples may provide a reasonable approach to improving reward reliability, although the entire reward calibration pipeline seems too complicated and requires further ablation to validate.

Critical weaknesses

The proposed InterGRPO framework has limited methodological novelty, as its core global-local credit assignment mechanism is essentially a straightforward extension of standard GRPO. Specifically, the method combines a trajectory-level global advantage with an image-level local advantage through a weighted sum, while retaining the standard GRPO optimization objective. Incorporating local and global rewards is a well-established idea in reinforcement learning, and the proposed formulation does not introduce a fundamentally new credit assignment mechanism. Moreover, the local reward is simply an aesthetic score assigned to each generated image, which is then applied uniformly to its diffusion steps. This does not explicitly address the more challenging credit assignment problem arising from dependencies across different generated units, such as determining which earlier image or generation decision is responsible for inconsistencies in later outputs. The fixed weighting between global and local advantages also appears heuristic and lacks sufficient justification. While adapting GRPO to multi-unit interleaved generation is practically useful, the proposed training framework appears largely incremental, and its claimed methodological contribution is not sufficiently distinguished from existing RL approaches that incorporate rewards at different granularities.

Questions for authors

The current ablation study (Figure 4) primarily compares training reward trajectories under different configurations. However, improvements in training reward do not necessarily translate into better generation quality, particularly when the reward model itself is being changed. Can the authors report downstream benchmark results (e.g., SIL-Bench and ISG-Bench) for global-only GRPO with UC-Reward, global-local GRPO with UC-Reward, and ideally local-only GRPO? This would clarify whether the reported improvements primarily originate from the proposed credit assignment mechanism or from the stronger reward model. In addition, evaluating these variants using a common independent evaluator would provide stronger evidence of the effectiveness of the proposed optimization framework.

Reviewer_gp7zOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InterGRPO, a reinforcement learning framework for interleaved image text generation. InterGRPO combines separately normalized sequence-level and image-level advantages to optimize image generation, while using global rewards for text generation. Besides, authors proposes UC-Reward, which distills scoring distributions from multiple teachers and incorporates uncertainty-based repeated evaluation, human calibration, and high-quality references to improve reward reliability. Authors also introduce a new benchmark SIL-Bench, and experiments on this benchmark and ISG-Bench demonstrates that their method outperform previous baselines significantly.

Critical strengths
  1. The problem is important, the motivation is well described and the paper is easy to follow.
  2. The method is simple but intuitive.
  3. The experimental results are promising.
Critical weaknesses
  1. This paper is missing one important baselines: Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization (Accepted to NeurIPS 2025).
  2. I cannot see the contribution of each parts through current experimental results. Specifically, the paper does not clearly establish how much improvement comes from global–local advantage assignment versus the reward model and additional supervision. I would suggest to provide a budget-matched comparisons with global-only GRPO and a combined global/local reward broadcast to all units.
  3. From my understanding, the local reward measures only image aesthetics, which does not necessarily reflect semantic correctness or text–image alignment.
  4. The gains are modest and uneven. On SIL-Bench, the overall score increases from 5.66 to 5.80, while image–text accuracy and single-image accuracy slightly decline.
  5. This paper is missing many important experimental settings, including dataset sizes and splits, teacher identities, human-annotation protocols, calibration thresholds, and several optimization settings.
Questions for authors

See above.

Reviewer_X62xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning for multi-unit interleaved text-image generation and proposes InterGRPO together with UC-Reward.

  • InterGRPO combines trajectory-level global advantages with image-unit-level local advantages to address the credit assignment issue in conventional GRPO, where different units within the same trajectory receive the same reward signal.
  • UC-Reward aims to provide more reliable reward supervision through multiple teacher models, repeated scoring, human calibration, and reference-based anchoring. The authors
    • evaluate the proposed approach on SIL-Bench and ISG-Bench, reporting improvements over ILLUME-X
    • providing additional analyses of the reward design.

Overall, I find the problem meaningful and the proposed approach reasonable. The experimental results are encouraging to some extent. However, I am not fully convinced that the current evidence supports the main methodological claim

  • whether the global-local credit assignment provides a clear benefit over simpler RL alternatives with the same reward model.
  • The novelty positioning, experimental controls, and reporting details could also be strengthened.

I therefore lean toward a weak reject.

Critical strengths
  • Multi-unit interleaved generation requires both high-quality individual outputs and consistency across the entire sequence.
  • A single trajectory-level reward may fail to distinguish strong and weak image units within the same generation. Introducing finer-grained feedback is therefore well motivated.
  • The results show some promising improvements.
    • On SIL-Bench, the image sequence accuracy score improves from 5.30 to 5.67, while the overall performance on ISG-Bench also increases.
      • These improvements are relevant to the paper's motivation of improving multi-unit generation.
      • The additional analyses in Figures 4 and 5 are also useful for understanding the behavior of the proposed reward design and reward model.
Critical weaknesses
  • The methodological novelty is not sufficiently established.

    • Combining global and local feedback is a reasonable design choice, but closely related ideas have been explored in prior work on multimodal interleaved generation and structured credit assignment, such as UniGRPO, BRAID, and InterleaveThinker.
      • The paper does not sufficiently discuss these connections or provide direct comparisons. As a result, it is difficult to determine what the key methodological advantage of InterGRPO is over existing approaches.
  • The current ablations do not fully support the central claim.

    • Figure 4 compares training reward curves under different reward configurations, but the corresponding checkpoints are not evaluated on the final benchmarks.
      • Since UC-Reward itself may contribute substantially to the improvements, it remains unclear how much of the gain comes specifically from global-local credit assignment.
      • In particular, a comparison against global-only GRPO using the same backbone, reward model, and training budget is missing. Without such a controlled comparison, the benefit of the proposed credit assignment mechanism is not convincingly isolated.
  • Some improvements are relatively modest or concentrated, and several quantitative claims need clarification.

    • The overall SIL-Bench score increases from 5.66 to 5.80, while some individual metrics slightly decrease.

      • The reported improvement in image sequence accuracy is inconsistent with Table 1. The claim about outperforming the strongest baseline is also misleading, since InterGRPO does not outperform Gemini in Table 1.
    • On ISG-Bench, a substantial portion of the improvement comes from the 3D Scene category, with smaller gains in most other categories.

Questions for authors

Please refer to the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.