- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LocFac-RL, a GRPO-based post-training framework that enables unified discrete diffusion models to generate an intermediate edited image before textual reasoning. To reduce visual rollout cost, the model predicts a bounding box and denoises only the image tokens inside that region (localized editing). To address the conditioning mismatch caused by bidirectional attention, the authors propose modality-factorized credit assignment, which updates the image and text segments separately with segment-specific rewards. Experiments on LaViDa-O and MMaDA-Parallel across five benchmarks show that LocFac-RL outperforms standard GRPO while reducing training time compared to global editing.
Critical strengths
- Exploiting the bidirectional attention of DDMs for localized editing during RL rollouts is a natural and reasonable idea. The conditioning mismatch between image rollouts and actor updates is also a valid issue for RL with unified DDMs.
- The paper provides ablations on each component (Tables 2 and 3) and comparisons with training-free fast sampling methods, and the experiments are clearly organized.
Critical weaknesses
- There is no guarantee that the localized edit is relevant to the question. The bounding box receives no reward, and the image update is trained only with CLIP similarity to the ground-truth image, without the answer reward. Evidence of question-relevant edits is limited to a few qualitative examples (Fig. 4) and a global CLIP score, which the authors admit can remain high even when the edit is wrong.
- In the efficiency analysis (Fig. 3a), the low LPIPS of local editing may come from copying the input outside the box, or even from making no edit at all. Since global LPIPS is dominated by the unchanged region, it cannot show whether the edit itself is correct. A no-edit baseline and a region-level metric (e.g., LPIPS inside the box) are needed.
- The visual reward requires ground-truth reasoning images, which limits the scalability of the method to datasets without such annotations.
- The comparison is limited to standard GRPO and an AR baseline (Anole). Existing RL methods for unified DDMs, such as ParaRL proposed with MMaDA-Parallel, are not compared, so it is unclear how much LocFac-RL improves over prior RL approaches designed for the same backbones.
Questions for authors
- Beyond the global LPIPS in Fig. 3a, how does local editing perform on metrics that reflect the edit itself, such as LPIPS or CLIP similarity computed only inside the edited region?
- Does the final answer actually depend on the generated image? How does the accuracy change if the generated image is replaced with the original input image at inference time?
- On LaViDa-O, ChartQA drops from 38.68 to 19.64 after SFT and recovers to 67.12 after RL, which accounts for most of the average gain. What is the reason for this drop and recovery? Is the gain from improved visual reasoning, or from recovering the answer format?
4 comments
Response to Reviewer 4WMj (2/2)
W3: The visual reward requires ground-truth reasoning images, which limits scalability.
RL in our recipe already trains on examples without a reference image. The 1K ArxivQA GRPO examples have none. Their image is generated in the rollout. It receives no image reward. The image advantage is exactly 0. These examples train the model through the answer and format rewards. A GRPO dataset can therefore mix examples with and without reference images. The image reward and the cold-start SFT do use reasoning images (ThinkMorph, ZebraCoT). App. C derives the edit regions from them automatically. We will state this as a limitation.
W4: No comparison with RL methods for unified DDMs such as ParaRL.
ParaRL and LocFac-RL solve different problems. ParaRL rewards alignment between reasoning text and a final output image for generation. We train an intermediate image and a textual answer for understanding. ParaRL was proposed with MMaDA-Parallel for thinking-aware image generation and editing. There, text and image are denoised in parallel. Semantic rewards along the denoising trajectory align the reasoning text with the final output image. In our setting, the image is an intermediate step. Rollouts are sequential: image first, then text. Local edits avoid regenerating the full canvas. ParaRL's text-image alignment reward has no direct counterpart for answer correctness. The two methods are complementary. They could be combined, for example with ParaRL-style trajectory rewards for the image segment of our factorized update. We will add this discussion to the related work.
Q1, Q2: Answered in W2 / W1, respectively.
Q3: LaViDa-O ChartQA drops from 38.68 to 19.64 after SFT and rises to 67.12 after RL. Visual reasoning or answer format?
The SFT drop is answer format, and the RL gain is not only format. With format held fixed, LocFac-RL still scores highest on ChartQA.
- Format. In the Table 1 evaluation, base and SFT return no answer for 92% and 87% of ChartQA questions. RL trains a format reward (Sec. 4.1), and the RL checkpoints answer 96–99%.
- Beyond format. We re-evaluated base, SFT, GRPO and LocFac-RL on all 2,500 ChartQA test questions with a 16-token answer budget and the input image only. Every checkpoint then answers at least 96% of the questions. LocFac-RL scores highest of the four (72.12 relaxed accuracy), 2.76 pp above GRPO (p < 0.001).
- Without ChartQA. LocFac-RL also keeps the highest LaViDa-O average in Table 1. Over the other four benchmarks, it averages 37.94 vs. 35.12 for GRPO.
We will add this analysis to Sec. 5.2.
Thank you for the additional experiments and clarifications. Some of my concerns have been addressed, but a few remain.
- Regarding Q2, the response compares the reference image with the input image. However, my question was whether the image generated by the model itself actually helps the answer. If possible, please report the results of LocFac-RL and of GRPO trained from the same SFT checkpoint when the generated image is replaced with the original input image.
- IoU improvement. The bounding box receives no reward during RL, yet the grounding IoU increases from 0.716 to 0.826. Do the box tokens receive any gradient during RL? If not, what drives this increase? It would also be helpful to report the IoU of the other LaViDa-O variants (Image-only RL and Text-only RL in Table 3, Unified in Table 2a, and Global Edit in Table 2b), to see which update the improvement comes from.
We thank the reviewer for the follow-up and apologize for answering Q2 with the wrong comparison.
Does the model's own generated image help the answer? (Q2)
We replaced each model's own generated image with the original input image. Both RL models start from the same SFT checkpoint. We used 1,000 ThinkMorph examples not used as RL prompts.
| Model | Input image (%) | Own generated image (%) | Δ pp [95% CI], McNemar p |
|---|---|---|---|
| SFT | 55.1 | 55.5 | +0.4 [−1.6, +2.4], p = 0.77 |
| GRPO | 54.7 | 54.4 | −0.3 [−2.4, +1.8], p = 0.86 |
| LocFac-RL | 52.1 | 52.9 | +0.8 [−1.3, +2.9], p = 0.52 |
The gap here is small, and we expected that. Our model was trained with SFT and RL on ThinkMorph itself, so it has already learned to read these images directly. A model that is already good at a task has little room to gain from a helper image.
A cleaner test: give our edits to a model that was not trained on ThinkMorph. To check whether the edited images help on their own, we gave LocFac-RL's edited images to the LaViDa-O base model, which was never fine-tuned on ThinkMorph. We changed only the input image. We scored it on all ThinkMorph Spatial Navigation rows that were not in RL training data of LocFacRL.
| Image shown to the answer model | Acc. (%) | Δ vs. input image, pp [95% CI], McNemar p |
|---|---|---|
| Input image only | 11.0 | – |
| Whole-image edit by LocFac-RL | 18.2 | +7.1 [+4.2, +10.3], p = 9e-6 |
| Local edit by LocFac-RL | 14.5 | +3.5, p = 0.003 |
This shows that the edits help a model that never saw this data. Swapping only the image raises accuracy by +7.1pp, a 65% relative gain. The result is statistically significant on a paired test.
In short, the model trained on ThinkMorph no longer needs its own edits, but those edits still help a model that wasn't trained on the data. We will add this experiment to the main paper.
Do the box tokens receive gradient during RL?
The box tokens receive gradient from visual reward. As mentioned in the paper, visual quality indirectly reflects the grounding accuracy. The experimental result also proves this point. Among all variants, local edit which uses bounding boxes show improvement in mean IoU, whereas other variants do not. We will clarify this in the paper.
| Checkpoint | Mean IoU |
|---|---|
| SFT | 0.716 |
| Factorized, local edit (LocFac RL) | 0.826 |
| Unified, local edit | 0.811 |
| Unified, global Edit | 0.703 |
| Image-only RL, global edit | 0.664 |
| Text-only RL (No Edit) | 0.680 |
We thank the reviewer again for these precise questions. We will add these results into the main paper to supplement how grounding accuracy improves after RL.
Response to Reviewer 4WMj (1/2)
We thank the reviewer for the review. We also thank the reviewer for finding the local editing idea natural, the conditioning mismatch valid and the ablations well organized.
W1: Is the edit relevant to the question; does the answer depend on the generated image?
Once the model is trained for visual thinking, its answer depends on the image, and RL makes the edit more relevant to the question.
(1) Does the answer depend on the image? Before and after SFT. Base LaViDa-O cannot use visual thinking without training for it. Even the reference reasoning image gives it no gain. We trained two SFT models from base LaViDa-O for one epoch on the same 15K ThinkMorph rows. The text-only model learns input image → reasoning and answer. The interleaved model also learns to generate the reference image, and its text step sees that image. All three models answer the same 200 validation prompts, which neither SFT model was trained on, one example at a time. The answer step receives the input and a second image. As the reviewer suggests, we replace the second image with the original input and compare with the reference reasoning image.
After interleaved SFT, the answer depends on the image: the reasoning image raises accuracy by 9.0 pp, and by 25.7 pp on Spatial_Navigation (p < 0.001). Both SFT models see the same textual traces, so this dependence comes from training the visual step.
(2) Is the edit relevant to the question? After RL. The reference box marks the region where the reference reasoning image differs from the input, that is, the region the visual step of this question changes (App. C). RL moves the predicted box toward it. Grounding IoU rises from 0.716 after SFT to 0.826 after LocFac-RL (Table 4), although the box receives no reward. Fig. 4 and App. G.1 show edits that cover the question region and complete the missing content. Changing only the visual step, with everything else fixed, also changes answer accuracy. Local instead of global editing raises the LaViDa-O average from 42.06 to 43.77 (Table 2b). CLIP similarity scores the whole image, which is mostly copied from the input under local editing. It therefore changes little (0.825 → 0.831), as Sec. 5.4 notes. IoU shows the gain in relevance more clearly.
The revision will add the SFT control to Sec. 5.4 and reports IoU as the measure of question relevance.
W2: Low LPIPS in Fig. 3a may come from copying the input or from making no edit; region-level metrics and a no-edit baseline.
Copying the input outside the box is the purpose of local editing, and inside the box the edit changes the image. As the reviewer asks, we ran a new region-level analysis with a no-edit baseline. We used LocFac-RL on the 200 ThinkMorph validation prompts, AlexNet LPIPS and the box the model predicts. Every generated image passes through the VQ tokenizer and decoder. The no-edit baseline is therefore the input passed through the same tokenizer and decoder, so both sides carry the same reconstruction loss.
In the revision, Fig. 3a will be added with the no-edit baseline and report LPIPS inside and outside the box.