- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses image restoration with an emphasis on preserving readable and accurate text. During restoration, a VLM generates textual guidance for a DiT, revising that guidance as it observes intermediate results. The VLM first undergoes SFT on approximately 161,000 examples containing CoT. The two models are then jointly optimized through RL, using GRPO for the VLM and DiffusionNFT for the DiT. Experiments on SA-Text and Real-Text evaluate both text recognition and image quality, with the full method outperforming UniT and TeReDiff on most reported metrics.
Critical strengths
- Broad empirical evaluation. The paper compares against a large number of text-aware restoration methods. Improvements are observed with ABCNet v2 as well as TESTR, even though ABCNet is not used as the reward model during RL.
- A practically meaningful question. Recovering an image that looks plausible is insufficient when incorrect characters change the meaning of its text. The paper addresses this concrete limitation of image restoration and evaluates the readability of the recovered text alongside overall image quality.
Critical weaknesses
- Repeated VLM inference adds a substantial deployment cost. The default restoration process requires multiple calls to a 7B VLM, each generating a reasoning and answer sequence. This additional dependence on a large VLM remains a practical limitation for applications.
- The benefit of continuing to optimize the VLM after SFT is not convincingly established. This paper explains the degradation under VLM-only RL as "VLM drift", arguing that the VLM sacrifices transcription accuracy to accommodate a frozen DiT's generation biases. However, it lacks a direct comparison of VLM reading accuracy across SFT, VLM-only RL, and joint RL Moreover, in Table 10, joint RL improves the mean F1 over DiT-only RL by just 0.11 percentage points on ABCNet v2 and 0.21 on TESTR. These gains are small relative to the reported variation. Since joint optimization is the paper’s central contribution, stronger evidence is needed to demonstrate the additional benefit of updating the VLM through RL over keeping it frozen after SFT.
Questions for authors
- What specific capabilities does GRPO improve in the VLM, and is there any evidence that these improvements lead to consistent additional gains in image restoration? For example, comparing the VLM's transcription accuracy and final restoration quality across VLM-only RL, DiT-only RL, and joint RL could help clarify this point.
- Why is a VLM needed to provide textual guidance, and why is joint training the VLM necessary? Dedicated OCR systems, such as PaddleOCR and TESTR, already provide text recognition capabilities. Why would their outputs not be sufficient as guidance for the DiT, and what additional value justifies using a VLM and further improving it through joint training?
- What distinguishes the method from UniT beyond joint optimization? UniT already uses a VLM to provide textual guidance and revises that guidance using intermediate OCR feedback during restoration. Please clarify the additional methodological contribution of this paper beyond joint training.