- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a high-quality large-scale dataset (RealIR-1M) and a benchmark (RealIR-Bench) for general-purpose image restoration. The dataset covers 9 common real-world degradation types (blur, noise, low-light, etc.) and contains a substantial number of samples (~1.65M). With this dataset, the paper proposes a real-world image restoration model, RealRestorer, by fine-tuning Step1X-Edit.
Critical strengths
1.
The engineering contribution looks solid and is described in enough detail to follow: a per-degradation synthesis pipeline, CLIP retrieval prompts, and the full training configuration (optimizer, schedule, steps, batch, hardware, etc.), with a plan to release the model and the pipeline. The high-quality large-scale dataset for general-purpose image restoration is also a main strength. If the dataset and the benchmark are also released, I think they could be useful to the community by themselves.
Critical weaknesses
1.
First of all, the text defines LPS = 1 - LPIPS (higher is better), but I think that the "LPS" columns in Tables 1 and 2 appear to indicate LPIPS itself (with a down arrow), and the Table 2 caption calls the same column "LPIPS". Since I was confused interpreting the table, please use notations consistently.
As far as I understand, the final score FS is a product of two terms: FS = 0.2 x (1 - LPIPS) x RS, where the (1 - LPIPS) measures how much the output stays consistent with the input and the RS measures how much degradation is removed. On the "1 - LPIPS" term, the proposed model is the worst among all open-source models: its average "1-LPIPS" is 1-0.445=0.555, while the other open-source models range from 0.675(=1-0.325) to 0.798(=1-0.202) (Table 2, Avg Total).
That is, I think that its first-ranked FS comes from the removal term instead of consistency term "1 - LPIPS": its RS of 1.312 is much higher than the next-ranked open-source model Qwen (0.931), which is large enough to outweigh the worst consistency in the product ( {(1.312-0.931)/0.931}x100 ~ 41% higher in RS ). In other words, the model seems to win by changing the image the most, not by preserving it, while the abstract presents consistency preservation as a central property.
Thus, I think there is an internal tension that the text does not discuss and this is the main reason for my recommendation.
2.
Second, the state-of-the-art claim does not seem matched by the comparison set, and a key part of the data pipeline is not specified. As far as I checked, every quantitative table (Tables 1, 2, 3, and 8) compares only against image editing models. That is, no dedicated restoration or all-in-one restoration model appears in any table, even on FoundIR (all-in-one restore), Flare7K++ (Deflare), and UHDM (Demoiré) (Tables 3 and 8), where such models are the established reference points. Small restoration models (EnlightenGAN, Retinexformer) appear only in a qualitative figure (Figure 2).
That is, I think "achieving state-of-the-art performance" (Abstract) is supported only within the editing-model family, and even there the model is second on average PSNR on the public benchmarks (20.88 vs. 22.21; Table 8). In addition, the clean references for the real-world training pairs are generated by "appropriate editing models" (Appendix A.2), but their identity is not given anywhere.
Thus, in my opinion, the paper could not support its central claims until the comparison model set is extended or the claims are narrowed. Also, please specify which models generated the clean references.
Questions for authors
- Section 3.2 says that "the intermediate checkpoint with the best generalization capability" is selected, and Figure 6 plots the performance against training steps on RealIR-Bench. Was this curve measured on the RealIR-Bench evaluation set itself, or on a separate validation set? If the former, I think the reported checkpoint could have been selected on the same benchmark used for Tables 1 and 2, which is a selection step the baseline editing models were not given. Could you state the exact selection rule and the data it was computed on?
- As far as I understand, both RealIR-Bench and the real-world training data are collected from the internet (Sec. 3.1, 4.1). Was any deduplication performed between the two? Could you report the near-duplicate rate if possible?
- The benchmark size is 464 in the abstract and Sec. 4.1 but 456 in Table 7. And the training-sample counts are different across Sec. 4.4 (about 1M / 100K), Table 5 (1,567,492 / 87,585), and Table 6 (1.5M / 80K). Could you please write these numbers accurately and consistently?
- I was confused by several citations that appear to come from unrelated fields: Benz (2017) is a solar-physics review but cited for lens flare, Tong & White (1996) and Al-Salem et al. (2015) are about plastics degradation. Could you clarify or correct these references? And please check all the references in the paper once more, so that every citation actually points to a work in the related field.
Minor issues:
- Typos and formatting: "Apeendix F" (Sec. 4.4), "removalremoval" (Table 8 caption), and a broken sentence ("...consistency. which means...") in the Figure 7 caption.