- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a methodology for text image restoration. Main contribution is a pipeline that takes strength of two approaches. First approach is to have a recognition base method, that exploits or uses character priors to recognize a character in degraded images. Naturally this is limited by the data it is trained to recognize. Second is to have stroke based reconstruction, but this doesn't have an understanding of letters/characters so it may miss strokes not visible in the degraded image. The core contribution is choosing per character between these two strategies. The selected characters then guide a full image restorer. Another contribution is the introduction of MText500, which is a real-world benchmark of 500 images in seven scripts. They report the best OCR accuracy in most settlings, but especially under heavy degradation.
Critical strengths
- Relevant problem with good results. The paper shows a real problem, when text is badly degraded, what should the model trust? Visual features or learned priors based on real characters.
- The provided method gets best OCR accuracy especially under heavy degradation on both synthetic and real data. Thorough quantatiative comparisons are done.
- MText500 is a new benchmark that contains 500 real-world image with seven scripts. It is split by degradation level, and if the characters were seen in training. This could be a useful resource.
Critical weaknesses
- Ablations contradict the main contribution. The key idea is "choosing" per character between the two strategies. Both strategies seem like they are already established in prior works, so the selection is the main contribution. However, Tables 3 and 4 are showing that it doesnt add anything substantial. In light degradation visual branch seems to be better than full system. Under heavy degradation the full model is barely better than visual only branch. This is the main reason for the recommendation.
- Layout augmentation is not ablated to see how much retraining with layout augmentation improved results. Would training the baselines with this augmentation improve their results substantially?
Questions for authors
Clarifications that could change recommendation
-
Need better validation for main contribution of selection.
-
What happens if baseline is retrained on your data with layout augmentation.
9 comments
Response to uF5k (Part2/2) : The Effect of Layout Augmentation
Q2. The Effect of Layout Augmentation
Thank you for suggesting that we retrain the baseline on our data. We retrained TiGeSR restorer on our data with layout augmentation. The retrained TiGeSR improves on the original baseline across all six metrics under both light and heavy degradation. OCR-A increases from 76.16% to 76.84% under light degradation and from 64.35% to 65.56% under heavy degradation.
TiGeSR — light degradation
| Mask | Restorer | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DISTS ↓ | FID ↓ | OCR-A ↑ |
|---|---|---|---|---|---|---|---|
| TiGeSR | Original | 21.08 | 0.6698 | 0.3023 | 0.2033 | 39.31 | 76.16 |
| TiGeSR | w/ Aug | 21.34 | 0.6823 | 0.2830 | 0.1912 | 36.47 | 76.84 |
TiGeSR — heavy degradation
| Mask | Restorer | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DISTS ↓ | FID ↓ | OCR-A ↑ |
|---|---|---|---|---|---|---|---|
| TiGeSR | Original | 18.22 | 0.4943 | 0.4114 | 0.2631 | 54.99 | 64.35 |
| TiGeSR | w/ Aug | 18.43 | 0.5142 | 0.3908 | 0.2495 | 51.50 | 65.56 |
To isolate the contribution of layout augmentation, we also trained the GRASP restorer on the same data with and without layout augmentation, keeping all other training settings fixed.
GRASP — light degradation
| Mask | Restorer | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DISTS ↓ | FID ↓ | OCR-A ↑ |
|---|---|---|---|---|---|---|---|
| Ours | w/o Aug | 21.81 | 0.7148 | 0.2620 | 0.1829 | 33.89 | 78.44 |
| Ours | w/ Aug | 22.33 | 0.7443 | 0.2344 | 0.1594 | 29.39 | 79.00 |
GRASP — heavy degradation
| Mask | Restorer | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DISTS ↓ | FID ↓ | OCR-A ↑ |
|---|---|---|---|---|---|---|---|
| Ours | w/o Aug | 18.19 | 0.5253 | 0.3851 | 0.2509 | 48.15 | 68.59 |
| Ours | w/ Aug | 18.64 | 0.5513 | 0.3595 | 0.2277 | 43.89 | 69.88 |
Layout augmentation improves all six metrics for GRASP at both degradation levels. OCR-A rises from 78.44% to 79.00% under light degradation and from 68.59% to 69.88% under heavy degradation. With augmentation, GRASP also outperforms the retrained TiGeSR baseline across all six metrics at both degradation levels.
Thank you for retraining TiGeSR on your data and for the layout augmentation ablation. Retraining improves the baseline, but GRASP still outperforms it, so this resolves my concern about the comparison. I note, however, that the visual-only branch in Tables 3 and 4 also outperforms the retrained TiGeSR. This supports visual reconstruction more than selection, which is the main point of the paper. Still, I consider Q2 resolved by the retrained results.
Response to uF5k (Part1/2) : Contribution of GRASP
Dear Reviewer uF5k,
Thank you for recognizing the practical importance of our work. We also appreciate your comments and suggestions. We address your questions point by point below.
Q1. Contribution of GRASP
(1) Contribution of the selection scheme
We agree that both visual reconstruction and prior-based restoration have strong precedents, and our own previous work has also explored both directions. From early CNN-based reconstruction to methods using GAN and diffusion priors, these approaches have substantially advanced natural image restoration. However, the trade-off between fidelity and visual quality has long been a problem for image restoration tasks, especially for highly-structured objects like faces and text, since minor artifacts like distorted strokes can be extremely noticeable.
Existing efforts to preserve both fidelity and visual quality typically fuse the features of two branches (for example, the encoder features delivered by skip connections and the decoder features with generative priors), where the fusion scheme is learned in a data-driven manner on synthetic data and struggles to generalize to complex real-world degradations. Moreover, contrary to the ideal of simultaneously preserving fidelity and visual quality, the fusion of sometimes conflicting features from the two branches often leads to artifacts instead.
GRASP decouples the decision to use a prior from glyph reconstruction. For each character, the visual branch reconstructs a candidate glyph from the image alone, and the prior branch produces a second candidate using the OCR-predicted character identity. A separate selector uses visual features and OCR confidence to choose between the two candidates for that character.
(2) Other contributions
Beyond the selection scheme, another technical contribution lies in the design of layout augmentation. At inference, the selected glyphs form a spatial text mask that guides the restorer. During training, layout augmentation keeps the ground-truth image and its text mask aligned while varying text placement relative to the degraded input. This encourages the restorer to follow the mask. Q2 reports the ablation.
(3) Explanation of Table 3
We agree with the reviewer's reading of Table 3. Under light degradation, Visual-only outperforms GRASP with predicted OCR on all six metrics in both subsets. Tables 3 and 4 were designed to test the intuition behind our selection strategy. We expect Visual to be preferable under light degradation on both Known and Unknown, and under heavy degradation on Unknown, where out-of-vocabulary characters have no matching prior. Under heavy degradation on Known, we expect Prior to help when strokes are missing, while Visual can still be preferable for individual characters whose strokes remain legible. The strong performance of Visual-only under light degradation is therefore consistent with our design.
Input OCR label errors appear to explain much of GRASP's small shortfall in OCR-A under light degradation. With predicted labels, GRASP scores 79.00% versus 79.17% for Visual-only. Correcting only the labels while retaining the original OCR confidence raises GRASP from 81.10% to 81.35% on Light-Known and from 76.87% to 77.07% on Light-Unknown. Both scores then slightly exceed Visual-only (81.33% and 76.99%). Under heavy degradation, the fixed-branch results in Table 4 follow the expected pattern. Prior-only has higher OCR-A on Known, and Visual-only on Unknown. GRASP slightly exceeds the stronger fixed branch in each subset.
One limitation of our original evaluation is that we defined Unknown at the text-line level and pooled the characters within those lines. At each degradation level, the Unknown subset contains 2,120 in-vocabulary and 5,455 out-of-vocabulary (OOV) characters. We should have reported their recognition accuracy separately. We provide that breakdown in our response to Reviewer 3aJ2 (Q2, Table Q2-1). Tables Q2-2 and Q2-3 also show that GRASP retains some robustness when input OCR misclassifies an OOV character as known. Even though this error bypasses the automatic visual fallback, the selector still chooses Visual in 140 of 190 such cases under light degradation and 131 of 222 under heavy degradation.