acceptodds
Under review as a conference paper at ICLR 2027

Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Models

Abstract

Despite promising multimodal understanding results, discrete diffusion models reason predominantly in text, using images as inputs but not as output for visual thinking that supports grounding and spatial reasoning. We introduce GRPO-based post-training that enables a unified discrete diffusion model to first generate an intermediate reasoning image and then use it to guide textual reasoning, while addressing two key challenges. First, to reduce visual rollout costs during RL, we leverage bidirectional attention for **localized visual editing**, denoising only a small subset of tokens while preserving all the other tokens. Second, we introduce **modality-factorized credit assignment** with segment-specific rewards, ensuring that image updates exclude future text while text updates use the completed image. Across five multimodal benchmarks, **LocFac-RL** improves average accuracy over standard GRPO by on LaViDa-O and on MMaDA-Parallel. Compared with global editing on the same backbones, it reduces training time by and while improving average accuracy by and , respectively.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4WMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LocFac-RL, a GRPO-based post-training framework that enables unified discrete diffusion models to generate an intermediate edited image before textual reasoning. To reduce visual rollout cost, the model predicts a bounding box and denoises only the image tokens inside that region (localized editing). To address the conditioning mismatch caused by bidirectional attention, the authors propose modality-factorized credit assignment, which updates the image and text segments separately with segment-specific rewards. Experiments on LaViDa-O and MMaDA-Parallel across five benchmarks show that LocFac-RL outperforms standard GRPO while reducing training time compared to global editing.

Critical strengths
  1. Exploiting the bidirectional attention of DDMs for localized editing during RL rollouts is a natural and reasonable idea. The conditioning mismatch between image rollouts and actor updates is also a valid issue for RL with unified DDMs.
  2. The paper provides ablations on each component (Tables 2 and 3) and comparisons with training-free fast sampling methods, and the experiments are clearly organized.
Critical weaknesses
  1. There is no guarantee that the localized edit is relevant to the question. The bounding box receives no reward, and the image update is trained only with CLIP similarity to the ground-truth image, without the answer reward. Evidence of question-relevant edits is limited to a few qualitative examples (Fig. 4) and a global CLIP score, which the authors admit can remain high even when the edit is wrong.
  2. In the efficiency analysis (Fig. 3a), the low LPIPS of local editing may come from copying the input outside the box, or even from making no edit at all. Since global LPIPS is dominated by the unchanged region, it cannot show whether the edit itself is correct. A no-edit baseline and a region-level metric (e.g., LPIPS inside the box) are needed.
  3. The visual reward requires ground-truth reasoning images, which limits the scalability of the method to datasets without such annotations.
  4. The comparison is limited to standard GRPO and an AR baseline (Anole). Existing RL methods for unified DDMs, such as ParaRL proposed with MMaDA-Parallel, are not compared, so it is unclear how much LocFac-RL improves over prior RL approaches designed for the same backbones.
Questions for authors
  1. Beyond the global LPIPS in Fig. 3a, how does local editing perform on metrics that reflect the edit itself, such as LPIPS or CLIP similarity computed only inside the edited region?
  2. Does the final answer actually depend on the generated image? How does the accuracy change if the generated image is replaced with the original input image at inference time?
  3. On LaViDa-O, ChartQA drops from 38.68 to 19.64 after SFT and recovers to 67.12 after RL, which accounts for most of the average gain. What is the reason for this drop and recovery? Is the gain from improved visual reasoning, or from recovering the answer format?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4WMj (1/2)

We thank the reviewer for the review. We also thank the reviewer for finding the local editing idea natural, the conditioning mismatch valid and the ablations well organized.

W1: Is the edit relevant to the question; does the answer depend on the generated image?

Once the model is trained for visual thinking, its answer depends on the image, and RL makes the edit more relevant to the question.

(1) Does the answer depend on the image? Before and after SFT. Base LaViDa-O cannot use visual thinking without training for it. Even the reference reasoning image gives it no gain. We trained two SFT models from base LaViDa-O for one epoch on the same 15K ThinkMorph rows. The text-only model learns input image → reasoning and answer. The interleaved model also learns to generate the reference image, and its text step sees that image. All three models answer the same 200 validation prompts, which neither SFT model was trained on, one example at a time. The answer step receives the input and a second image. As the reviewer suggests, we replace the second image with the original input and compare with the reference reasoning image.

ModelReference − input, pp [95% CI]McNemar p
Base LaViDa-O*+1.0 [−3.5, +5.5]0.82
Text-only SFT−1.0 [−5.5, +3.5]0.82
Interleaved SFT+9.0 [+4.0, +15.0]0.003

After interleaved SFT, the answer depends on the image: the reasoning image raises accuracy by 9.0 pp, and by 25.7 pp on Spatial_Navigation (p < 0.001). Both SFT models see the same textual traces, so this dependence comes from training the visual step.

(2) Is the edit relevant to the question? After RL. The reference box marks the region where the reference reasoning image differs from the input, that is, the region the visual step of this question changes (App. C). RL moves the predicted box toward it. Grounding IoU rises from 0.716 after SFT to 0.826 after LocFac-RL (Table 4), although the box receives no reward. Fig. 4 and App. G.1 show edits that cover the question region and complete the missing content. Changing only the visual step, with everything else fixed, also changes answer accuracy. Local instead of global editing raises the LaViDa-O average from 42.06 to 43.77 (Table 2b). CLIP similarity scores the whole image, which is mostly copied from the input under local editing. It therefore changes little (0.825 → 0.831), as Sec. 5.4 notes. IoU shows the gain in relevance more clearly.

The revision will add the SFT control to Sec. 5.4 and reports IoU as the measure of question relevance.

W2: Low LPIPS in Fig. 3a may come from copying the input or from making no edit; region-level metrics and a no-edit baseline.

Copying the input outside the box is the purpose of local editing, and inside the box the edit changes the image. As the reviewer asks, we ran a new region-level analysis with a no-edit baseline. We used LocFac-RL on the 200 ThinkMorph validation prompts, AlexNet LPIPS and the box the model predicts. Every generated image passes through the VQ tokenizer and decoder. The no-edit baseline is therefore the input passed through the same tokenizer and decoder, so both sides carry the same reconstruction loss.

RegionLPIPS toNo editLocal editGlobal edit
Outside the boxinput0.0930.0940.140
Inside the boxinput0.0990.129–
Whole imagereference image0.1330.132–
  • Outside the box, local editing keeps every image token of the input: 0 tokens change in any of the 199 local edits. Its LPIPS to the input therefore equals the no-edit baseline. Global editing changes this region (0.140). It also moves it further from the reference image than local editing does (+0.011, 95% CI [+0.006, +0.017]).
  • Inside the box, the local edit changes the input more than no edit does, on average (0.129 vs. 0.099).
  • Whole image, the tokenizer round trip alone gives LPIPS 0.133 to the reference image. Whole-image LPIPS is thus dominated by reconstruction loss that every method shares. Fig. 3a compares rollout methods that all pass through this tokenizer.

In the revision, Fig. 3a will be added with the no-edit baseline and report LPIPS inside and outside the box.

AuthorsOpenReview ↗

Response to Reviewer 4WMj (2/2)

W3: The visual reward requires ground-truth reasoning images, which limits scalability.

RL in our recipe already trains on examples without a reference image. The 1K ArxivQA GRPO examples have none. Their image is generated in the rollout. It receives no image reward. The image advantage is exactly 0. These examples train the model through the answer and format rewards. A GRPO dataset can therefore mix examples with and without reference images. The image reward and the cold-start SFT do use reasoning images (ThinkMorph, ZebraCoT). App. C derives the edit regions from them automatically. We will state this as a limitation.

W4: No comparison with RL methods for unified DDMs such as ParaRL.

ParaRL and LocFac-RL solve different problems. ParaRL rewards alignment between reasoning text and a final output image for generation. We train an intermediate image and a textual answer for understanding. ParaRL was proposed with MMaDA-Parallel for thinking-aware image generation and editing. There, text and image are denoised in parallel. Semantic rewards along the denoising trajectory align the reasoning text with the final output image. In our setting, the image is an intermediate step. Rollouts are sequential: image first, then text. Local edits avoid regenerating the full canvas. ParaRL's text-image alignment reward has no direct counterpart for answer correctness. The two methods are complementary. They could be combined, for example with ParaRL-style trajectory rewards for the image segment of our factorized update. We will add this discussion to the related work.

Q1, Q2: Answered in W2 / W1, respectively.

Q3: LaViDa-O ChartQA drops from 38.68 to 19.64 after SFT and rises to 67.12 after RL. Visual reasoning or answer format?

The SFT drop is answer format, and the RL gain is not only format. With format held fixed, LocFac-RL still scores highest on ChartQA.

  • Format. In the Table 1 evaluation, base and SFT return no answer for 92% and 87% of ChartQA questions. RL trains a format reward (Sec. 4.1), and the RL checkpoints answer 96–99%.
  • Beyond format. We re-evaluated base, SFT, GRPO and LocFac-RL on all 2,500 ChartQA test questions with a 16-token answer budget and the input image only. Every checkpoint then answers at least 96% of the questions. LocFac-RL scores highest of the four (72.12 relaxed accuracy), 2.76 pp above GRPO (p < 0.001).
  • Without ChartQA. LocFac-RL also keeps the highest LaViDa-O average in Table 1. Over the other four benchmarks, it averages 37.94 vs. 35.12 for GRPO.

We will add this analysis to Sec. 5.2.

Thank you for the additional experiments and clarifications. Some of my concerns have been addressed, but a few remain.

  1. Regarding Q2, the response compares the reference image with the input image. However, my question was whether the image generated by the model itself actually helps the answer. If possible, please report the results of LocFac-RL and of GRPO trained from the same SFT checkpoint when the generated image is replaced with the original input image.
  2. IoU improvement. The bounding box receives no reward during RL, yet the grounding IoU increases from 0.716 to 0.826. Do the box tokens receive any gradient during RL? If not, what drives this increase? It would also be helpful to report the IoU of the other LaViDa-O variants (Image-only RL and Text-only RL in Table 3, Unified in Table 2a, and Global Edit in Table 2b), to see which update the improvement comes from.

We thank the reviewer for the follow-up and apologize for answering Q2 with the wrong comparison.

Does the model's own generated image help the answer? (Q2)

We replaced each model's own generated image with the original input image. Both RL models start from the same SFT checkpoint. We used 1,000 ThinkMorph examples not used as RL prompts.

ModelInput image (%)Own generated image (%)Δ pp [95% CI], McNemar p
SFT55.155.5+0.4 [−1.6, +2.4], p = 0.77
GRPO54.754.4−0.3 [−2.4, +1.8], p = 0.86
LocFac-RL52.152.9+0.8 [−1.3, +2.9], p = 0.52

The gap here is small, and we expected that. Our model was trained with SFT and RL on ThinkMorph itself, so it has already learned to read these images directly. A model that is already good at a task has little room to gain from a helper image.

A cleaner test: give our edits to a model that was not trained on ThinkMorph. To check whether the edited images help on their own, we gave LocFac-RL's edited images to the LaViDa-O base model, which was never fine-tuned on ThinkMorph. We changed only the input image. We scored it on all ThinkMorph Spatial Navigation rows that were not in RL training data of LocFacRL.

Image shown to the answer modelAcc. (%)Δ vs. input image, pp [95% CI], McNemar p
Input image only11.0–
Whole-image edit by LocFac-RL18.2+7.1 [+4.2, +10.3], p = 9e-6
Local edit by LocFac-RL14.5+3.5, p = 0.003

This shows that the edits help a model that never saw this data. Swapping only the image raises accuracy by +7.1pp, a 65% relative gain. The result is statistically significant on a paired test.

In short, the model trained on ThinkMorph no longer needs its own edits, but those edits still help a model that wasn't trained on the data. We will add this experiment to the main paper.

Do the box tokens receive gradient during RL?

The box tokens receive gradient from visual reward. As mentioned in the paper, visual quality indirectly reflects the grounding accuracy. The experimental result also proves this point. Among all variants, local edit which uses bounding boxes show improvement in mean IoU, whereas other variants do not. We will clarify this in the paper.

CheckpointMean IoU
SFT0.716
Factorized, local edit (LocFac RL)0.826
Unified, local edit0.811
Unified, global Edit0.703
Image-only RL, global edit0.664
Text-only RL (No Edit)0.680

We thank the reviewer again for these precise questions. We will add these results into the main paper to supplement how grounding accuracy improves after RL.

Reviewer_uQDcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that existing unified discrete diffusion models rely solely on textual reasoning in multimodal tasks: images serve only as inputs and are never generated as intermediate reasoning steps, which limits their ability to support grounding and spatial reasoning. To address this, the paper proposes LocFac-RL, a post-training method consisting of supervised fine-tuning (SFT) and GRPO, which enables the model to first predict the region to be edited, then generate an intermediate reasoning image by editing within that region, and finally perform textual reasoning conditioned on the edited image. In the GRPO phase, the authors introduce two mechanisms. First, localized visual editing leverages bidirectional attention to denoise only the image tokens inside the predicted grounding region while keeping the surrounding tokens fixed, thereby reducing visual rollout costs. Second, the authors observe that during actor updates, bidirectional attention allows image tokens to attend to response text that does not yet exist when the image is generated in the rollout. To address this conditioning mismatch, modality-factorized credit assignment splits each response into separate image and text updates: the image update excludes the subsequent response text from the conditioning context and uses only the visual reward, while the text update conditions on the completed image and uses the visual reward together with the answer and format rewards. Across five multimodal benchmarks, the method achieves higher average accuracy than standard GRPO on both LaViDa-O and MMaDA-Parallel. With factorized credit assignment applied in both settings, local editing reduces training time while improving accuracy compared with global editing.

Critical strengths
  1. Local editing makes effective use of the structural properties of discrete diffusion models and yields clear efficiency gains. Leveraging bidirectional attention, the method denoises only the image tokens inside the predicted grounding region, which is difficult for autoregressive models due to their causal structure. Compared with global editing, this design both reduces training time and improves accuracy on both backbones (Table 2b), and it can be combined with training-free fast sampling methods (Fig. 3). This is the most distinctive contribution of the paper with respect to discrete diffusion models.

  2. A reusable automatic grounding-region annotation pipeline. The paper uses image registration to automatically add edit-region annotations to existing interleaved reasoning data (Appendix C), without manual labeling, and grounding IoU improves consistently across training stages (Table 4). The pipeline can be transferred to other interleaved reasoning datasets and is of practical value for future work.

Critical weaknesses
  1. The core hypothesis that visual thinking improves reasoning is not experimentally verified.The current experiments show that the overall training pipeline outperforms standard GRPO, but they do not show that the improvement comes from the intermediate images.

(1) No clean “no visual thinking” control. For the Text Only RL variant in Table 3, the paper only states that it uses textual reasoning in RL rollouts (Section 5.3); it does not state whether the model still generates intermediate images at evaluation time. Moreover, this variant presumably starts from the same SFT model trained with visual editing as Both (LocFac-RL). Therefore, the gap of about 4 points on LaViDa-O (39.83 vs. 43.77) only indicates that optimizing both modalities during RL is more effective.

(2) No evidence linking the intermediate images to answer correctness. For LaViDa-O, average accuracy rises from 30.05 (SFT) to 43.77 (LocFac-RL) (Table 1), whereas CLIP similarity only changes from 0.825 to 0.831 (Table 4), and the authors acknowledge in Section 5.4 that this metric “does not by itself establish the correctness or usefulness of the visual reasoning step.” The IoU improvement only indicates more accurate localization, which does not imply that the edited content helps answer the question.

(3) The gains from the SFT stage on MMaDA-Parallel and ANOLE cannot be attributed either. The SFT data contain both textual reasoning traces and visual edits, so it is unclear which part the improvement comes from.

  1. The image-side reward design is inconsistent with the motivation that intermediate images should serve downstream reasoning, and it is undefined for part of the data.

(1) The image side is not driven by answer correctness. Under modality-factorized credit assignment, the image advantage A^I uses only the visual reward R_1=r_"vis" (CLIP similarity to the ground-truth image).Whether the final answer is correct influences only the textual reasoning and does not encourage the generation of intermediate images that help solve the problem. The image side is therefore effectively trained to imitate the reference image.

(2) The visual reward is undefined for ArxivQA samples. The 1K ArxivQA samples used in the GRPO stage have no intermediate reasoning images, and the paper does not explain how r_"vis" is computed for them; the cited prior works, Omni-R1 (Cheng et al., 2026) and NV-CoT (Zhao et al., 2026a), do not provide intermediate images for ArxivQA either. If r_"vis" is set to a constant, A^I is always zero for these samples; if the input image is used as the reference, the reward favors making no edit.

(3) The empirical effect of factorized credit assignment is unstable. Under global editing, it lowers the average accuracy of MMaDA-Parallel from 22.15 (Table 1, GRPO with unified credit assignment) to 17.92 (Table 2b), but improves LaViDa-O under the same setting. The authors do not provide a mechanistic explanation.

Questions for authors

Clarification:

  1. How is r_"vis" computed for the ArxivQA samples?
  2. In the factorized update, which segment do the grounding bounding-box tokens belong to, and which advantage do they use? Section 4.1 states that “we do not assign a separate reward to the grounding bounding box.” What training signal then drives the IoU increase from 0.716 to 0.826 after RL?
  3. Does the Text Only RL variant generate intermediate images at evaluation time?
  4. Why does factorized credit assignment affect the two backbones in opposite directions under global editing? 5.The introduction states that “existing unified discrete diffusion models remain predominantly text-centric”, but the cited LLaDA2.0-Uni (AI et al., 2026) has already explored interleaved reasoning with intermediate image generation within a discrete diffusion framework. Given this, which models does this claim refer to? What is the novelty of this work relative to LLaDA2.0-Uni?

Suggestions:

  1. Add a control that uses only textual reasoning traces from the SFT stage onward, and an ablation that removes or replaces the intermediate images at evaluation time.
  2. Report answer accuracy on the held-out ThinkMorph examples.

Minor: The bold markings in Table 1 contain errors and follow no consistent rule, and the caption does not define them: (1) In the ChartQA column of the ANOLE group, 5.8 (+SFT) is bolded, whereas the best value in that group is 7.3 (base model). (2) In the Average Accuracy column, 30.05 (LaViDa-O + SFT) is bolded although it is lower than the base model’s 31.21; the GRPO averages of both discrete diffusion groups (40.57 and 22.15) are also bolded although they are lower than the corresponding LocFac-RL results (43.77 and 24.79). (3) Decimal places are inconsistent, e.g., the ChartQA score of MMaDA-Parallel + LocFac-RL is written as “4.6”, while the other values in the same group use two decimal places.

The authors are encouraged to adopt a consistent rule (e.g., bold for the best and underline for the second best) and to define the bold and gray-shaded entries in the caption.

5 comments
AuthorsOpenReview ↗

**Response to Reviewer uQDc (1/3)**

We thank the reviewer for the careful review and the concrete suggestions. We first answer the questions.

W1: Is the hypothesis that visual thinking improves reasoning verified?

Yes, at both training stages. SFT teaches the model to use the reasoning image. During RL, changing only how the visual step is trained changes answer accuracy.

(1) Before and after SFT: a clean "no visual thinking" control (W1(1), W1(3)). LaViDa-O cannot use visual thinking without training for it. Even the reference reasoning image gives the base model no gain. We trained two SFT models from base LaViDa-O for one epoch on the same 15K ThinkMorph rows. The text-only model learns input image → reasoning and answer. The interleaved model also learns to generate the reference image, and its text step sees that image. All three models answer the same 200 validation prompts, which neither SFT model was trained on, one example at a time. The answer step receives the input and a second image: the input again, or the reference reasoning image.

ModelReference − input again, pp [95% CI]McNemar p
Base LaViDa-O*+1.0 [−3.5, +5.5]0.82
Text-only SFT−1.0 [−5.5, +3.5]0.82
Interleaved SFT+9.0 [+4.0, +15.0]0.003

Only interleaved SFT teaches the model to use the reasoning image. On Spatial Navigation the gain is +25.7 pp (p < 0.001). Both SFT models see the same textual traces. The ability to use the image therefore comes from training the visual step, not from the textual traces.

(2) After RL: the visual step becomes more precise, and accuracy follows it. RL then sharpens the visual step that SFT built. Grounding IoU rises from 0.716 after SFT to 0.826 after LocFac-RL (Table 4). Fig. 4 and App. G.1 show edits that cover the question region and complete the missing content. Changing only the visual step, with everything else fixed, also changes answer accuracy. Local instead of global editing raises the average from 42.06 to 43.77 on LaViDa-O and from 17.92 to 24.79 on MMaDA-Parallel (Table 2b). Scoring the image update without the later text, with segment-specific rewards, raises it from 41.72 to 43.77 and from 23.69 to 24.79 (Table 2a). Answer accuracy therefore tracks how the visual step is trained.

CLIP similarity hardly moves because it scores the whole image. In a local edit, about 82% of the image lies outside the edit box and is copied from the input. A better edit inside the box therefore changes the score little, as Sec. 5.4 notes. Grounding IoU and answer accuracy show the improvement more clearly.

During revision, we will adds the SFT control to Sec. 5. and state that whole-image CLIP similarity is insensitive to local edits.

AuthorsOpenReview ↗

**Response to Reviewer uQDc (2/3)**

W2 (1): The image advantage uses only r_vis, so answer correctness does not drive image generation.

Keeping the answer reward off the image tokens is what makes the image signal informative. Each segment is rewarded by the signal it determines.

  • r_vis is a function of the generated image alone. Within a GRPO group, its differences come only from the image tokens. Its advantage is therefore a direct signal for them.
  • The answer is decided after the image, by a long sampled text step. The same image can be followed by a correct or a wrong answer. As an image signal, the answer advantage therefore carries the sampling noise of the text step. It also rewards an image that adds nothing, such as an unchanged copy of the input, whenever the text step answers correctly anyway.

Matching the reference visual step is thus the intended objective of the image segment. RL samples edits around the SFT policy and reinforces those closer to the reference.

Table 2a measures the alternative. In its Unified rows, image tokens also receive the answer reward (these rows also score the image in the joint context). With local editing, this lowers accuracy from 43.77 to 41.72 on LaViDa-O and from 24.79 to 23.69 on MMaDA-Parallel.

W2 (2): How is r_vis computed for ArxivQA?

The ArxivQA rows do not reward a no-edit image. The trainer computes the CLIP reward only on rows that have a reference image. It writes 0 for the 1K ArxivQA rows. Sec. 5.1 and App. D.2 will state this.

W2 (3): Why does factorized credit assignment move the two backbones in opposite directions under global editing?

The sign depends on where the image gradient lands, and the two backbones differ exactly there. In the factorized update, image tokens are trained by r_vis only. Under global editing, every image token is regenerated, so this r_vis gradient is large. Under local editing, about 82% of the canvas lies outside the box and is copied from the input. These tokens are easy to predict, so the gradient is small.

  • LaViDa-O processes image-generation tokens with a separate tower, with its own attention and FFN weights (width 2,048 vs. 4,096 for text). The r_vis gradient therefore lands mainly on the generation weights, and the text step keeps its weights. The factorized update helps under both editing modes: 40.57 → 42.06 under global editing.
  • MMaDA-Parallel uses one set of weights for all modalities. The large r_vis gradient then moves the same weights the text step uses, toward an objective that is not the answer. Accuracy falls most on MMMU (17.67 → 9.89) and V* (27.75 → 11.00). In the unified update, the image tokens also carry the answer reward, so their gradient is tied to answer correctness, like the text step's.
  • Under local editing, the r_vis gradient is small and does not pull the shared weights away. Factorization then helps both backbones: 41.72 → 43.77 and 23.69 → 24.79 (Table 2a). This is also why LocFac-RL pairs factorized updates with local editing. Sec. 5.3 will give this explanation with the per-benchmark numbers.
AuthorsOpenReview ↗

Response to Reviewer uQDc (3/3)

Q1 How is r_vis computed for ArxivQA?

Answered in W2 (2).

Q2: Which segment do the box tokens belong to, and which advantage do they use? What drives the IoU change?

The box tokens belong to neither segment. They receive no advantage or gradient. Grounding is therefore learned in SFT, not in RL. The box is predicted in a separate grounding pass. This pass is one forward pass without gradients. It takes an argmax over the four masked LOC positions and parses the result into coordinates. The coordinates only select which image tokens are masked in the local-edit rollout. Neither actor-update input contains the LOC tokens.

Q3: Does Text-only RL generate intermediate images at evaluation?

Yes. Text-only RL generates the intermediate image at evaluation.

Q4: Why does factorized credit assignment move the two backbones in opposite directions under global editing?

Answered in W2 (3).

Reviewer_6RcXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LocFac-RL trains discrete diffusion models to generate a locally edited image before answering a question, using modality-factorized updates to prevent leakage of future text. Experiments on two diffusion backbones show improved average performance and shorter training step times compared with global editing using the same factorized updates.

Critical strengths
  • Regional editing for interleaved text–image reasoning seems valuable.
Critical weaknesses
  • The method restricts visual reasoning to a single editing stage, but does not establish whether this is sufficient for questions requiring successive visual operations or recovery from an incorrect edit. The authors should compare no-edit, single-edit, and iterative-editing variants under matched computational budgets to justify this design.

  • Existing frameworks such as VeRL-Omni [1] already support RL post-training for diffusion and omni models, making the framework-level novelty insufficiently clear.

  • Preventing image updates from accessing future text primarily enforces consistency between training and rollout conditioning, and does not by itself establish a novel credit assignment method, for example [2]

[1] https://github.com/verl-project/verl-omni

[2] https://github.com/verl-project/verl-omni/blob/0fb3c91c5db457b3bc7aee72ba7e4146e336467b/verl_omni/pipelines/bagel_flow_grpo/bagel_sft_model.py#L1128

Questions for authors
  • Why is the reasoning process restricted to a single visual-to-textual transition? The paper describes the approach as “interleaved visual–textual reasoning,” but only evaluates one such transition, without testing multiple rounds of alternating visual and textual reasoning.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6RcX (1/2)

We sincerely thank the reviewer for their time and effort for reviewing this paper. Below are the rebuttals on the raised weaknesses. We hope this addresses your concerns and happy to provide additional rebuttal results if the reviewer deems insufficient.

W1: One edit round is enough. A second round is not distinguishable from one edit for any trained checkpoint.

Adding a second round of image editing adds up 1.5–1.6× the time compared to one edit, without performance gain. Moreover, our paper states that the single transition is a scope of the paper. The Introduction and Sec. 4.1 states the same scope.

Edit vs. No edit Discrete diffusion model cannot use visual thinking without training for it. Base LaViDa-O gains nothing even from the reference reasoning image. We tested base LaViDa-O and two SFT models trained from it for one epoch on the same 15K ThinkMorph rows. The text-only model learns input image → reasoning and answer. The interleaved model also learns to generate the reference image, and its text step sees that image. Each model answers the 200 validation prompts, which neither of the models saw. The answer step always receives two images: the input, then the input again, the model's own edit or the reference image.

ModelInput againOwn editReferenceReference − input, pp [95% CI], p
Base LaViDa-O*19.021.020.0+1.0 [−3.5, +5.5], p = 0.82
Text-only SFT62.062.561.0−1.0 [−5.5, +3.5], p = 0.82
Interleaved SFT57.058.5†66.0+9.0 [+4.0, +15.0], p = 0.003

Of these models, only the one trained with the reference as its second image gains from it. Its own edit is not distinguishable from the input (+1.5 pp, p = 0.61). More rounds therefore first need training that teaches the model to use each new image.

We ran the requested comparison with LaViDa-O LocFac-RL at measured compute. We used the 200 ThinkMorph validation prompts and decoded one example at a time. Times are per example on one A100.

VariantTime (s)Acc. (%)Δ pp [95% CI], McNemar p
One local edit, 64 steps37.054.5−2.0 [−6.0, +2.5] vs. no edit, p = 0.52
Two edit rounds (re-ground, edit again)≈5957.5+3.0 [−2.0, +8.0] vs. one edit, p = 0.33
One edit, 2× denoising steps≈5653.5−1.0 [−6.0, +4.0] vs. one edit, p = 0.85
AuthorsOpenReview ↗

Response to Reviewer 6RcX (2/2)

W2: Framework novelty relative to VeRL-Omni.

VeRL-Omni does not cover our contribution. It contains no masked discrete diffusion model. It has no RL objective over masked tokens. The cited line builds an attention mask for supervised training only.

Timing. Under the ICLR 2027 reviewer guidelines, this is contemporaneous work. We will still cite and discuss it.

What it covers. VeRL-Omni supports the following models:

  • continuous diffusion/flow generators (e.g. Qwen-Image, Wan2.2, SD3.5, FLUX)
  • unified models that pair an autoregressive text backbone with a flow image head (BAGEL with FlowGRPO)
  • omni models (e.g. Qwen3-Omni)

It includes no masked discrete diffusion model such as LaViDa-O or MMaDA-Parallel. Its RL algorithms are FlowGRPO, DanceGRPO and DiffusionNFT. They compute likelihoods along flow/SDE trajectories, not over masked tokens.

The problem we address is specific to masked DDMs. The GRPO actor update (Eq. 2) re-masks tokens of the whole response. It scores them with bidirectional attention. Image tokens therefore attend to answer text that did not exist when the image was sampled. LocFac-RL scores the image segment without the later text. It gives each segment its own reward. Its rollouts are local edits. Local edits rely on bidirectional denoising of a masked region given the rest of the image (Sec. 4.2).

We will replace "Systematization of Multi-modal RL with Discrete Diffusion" with "GRPO for image-then-text reasoning in unified masked discrete diffusion models".

W3:Training-Rollout inconsistency inconsistency arises from bidirectional attention, so it does not exist in autoregressive RL.

In an autoregressive model, causal attention hides later text from every image token, so the update scores each token in the same context as the rollout. A masked diffusion model attends in both directions. A unified update scores the whole response in one pass, so image tokens see text that was generated after them. The update then optimizes image tokens under a context the rollout never sampled from. Its importance ratio is blind to the gap, because old and new log-probabilities share that context. A GRPO recipe carried over from autoregressive models therefore leaves this bias in place, as the Introduction states for DDMs. Excluding future text removes it, and we measured it directly (every 95% CI above 0).

Excluding future text removes this bias, and we measured it directly (every 95% CI above 0). Scoring image tokens in the joint context puts more masked image tokens outside the clip band. The extra share is +1.1 to +1.3 pp for local and +2.6 pp for global editing. Every 95% CI is above 0.

In a unified update, old and new log-probabilities of image tokens are both computed in the joint context. The importance ratio therefore never shows the gap. The image tokens are optimized under a conditional that the rollout did not sample from.

We measured the gap under the exact mask of our RL update. The mask selects 15% of image tokens i.i.d. How large is the bias? We take the same rollouts and the same weights. We score each masked image token twice: without the later text, as in the rollout, and with it, as in a unified update. If the later text did not matter, the two probabilities would match. We count the share of tokens whose probability ratio falls outside the training clip band [0.8, 1.2]. We use the training mask (15% of image tokens, i.i.d.) on the 200 ThinkMorph validation prompts. bf16 noise alone puts 1.0–1.5% (local) and 3.8–3.9% (global) of tokens outside the band.

Adding the later text raises this share by +1.32 pp [+1.18, +1.48] (local) and +2.60 pp [+2.36, +2.87] (global) for LocFac-RL. SFT shows the same, on all three tasks. The later text therefore shifts the image-token probabilities that a unified update optimizes, and that update cannot see the shift. Table 2a measures removing it together with segment-specific rewards: LaViDa-O rises from 41.72 to 43.77, MMaDA-Parallel from 23.69 to 24.79.

In the revision, we will rename the component as "modality-factorized updates". It has two stated parts:

  • (a) Inference-consistent conditioning. The image update is scored without the later response text, as in the rollout.
  • (b) Segment-specific rewards. Image tokens get an advantage from r_vis only. Text tokens get an advantage from the answer and format rewards only.

Dear Reviewer 6RcX,

Thank you again for your review. As the discussion period is coming to an end, we would like to check whether our response has addressed your concerns.

In our response, we:

  1. ran the requested compute-matched comparison of no edit, one edit and iterative editing, with measured A100 time (W1)
  2. clarified the novelty relative to VeRL-Omni, which has no masked discrete diffusion model and no RL objective over masked tokens (W2)
  3. measured the conditioning mismatch that our image update removes, under the exact training mask (W3)
  4. committed to the requested wording changes ("image-then-text reasoning", "modality-factorized updates").

If any of these points remains unresolved, please let us know. We will answer before the deadline. If our response has resolved your concerns, we would be grateful if you would consider raising your score. We would also appreciate knowing the reason for the ethics flag, so that we can address it.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.